{"id":6870,"date":"2020-10-29T21:41:22","date_gmt":"2020-10-30T01:41:22","guid":{"rendered":"https:\/\/courses.ideate.cmu.edu\/60-212\/f2020\/?p=6870"},"modified":"2020-11-01T22:36:10","modified_gmt":"2020-11-02T03:36:10","slug":"sweetcorn-mask","status":"publish","type":"post","link":"https:\/\/courses.ideate.cmu.edu\/60-212\/f2020\/bkrummenandrew-cmu-edu\/10\/29\/sweetcorn-mask\/","title":{"rendered":"sweetcorn-mask"},"content":{"rendered":"<h3 style=\"text-align: center\"><a href=\"https:\/\/childrens-stories.glitch.me\">Children&#8217;s Stories<\/a><\/h3>\n<blockquote><p>You are your own virtual teacher as you generate your own children&#8217;s stories. Open your mouth to show each new word.<\/p><\/blockquote>\n<p>I determine if the user is opening their mouth by comparing the distance between their lips to the distance between the top of their head and the bottom of their head.<\/p>\n<p><img loading=\"lazy\" class=\"aligncenter size-full wp-image-6876\" src=\"https:\/\/courses.ideate.cmu.edu\/60-212\/f2020\/wp-content\/uploads\/2020\/10\/mouthOpenSensing.gif\" alt=\"\" width=\"621\" height=\"384\" \/><\/p>\n<p>Above is a very early demonstration of this in my debug-mode, which also shows the indices of the vertices that I use to draw the face. I am using <a href=\"https:\/\/www.beyond-reality-face.com\/\"><em>Beyond Reality Face<\/em><\/a> for face tracking.<\/p>\n<p>Below, I had started filling in features using the given vertices. Most features were very simple to turn into shapes by using <em>beginShape()<\/em> and <em>endShape(CLOSE)<\/em> around a for loop across each point and creating a <em>curveVertex()<\/em> at the coordinates of that point. A few features had to be estimated or otherwise fiddled with.<\/p>\n<p><img loading=\"lazy\" class=\"aligncenter size-full wp-image-6875\" src=\"https:\/\/courses.ideate.cmu.edu\/60-212\/f2020\/wp-content\/uploads\/2020\/10\/ezgif-4-bca7b70ffbb2.gif\" alt=\"\" width=\"449\" height=\"453\" \/><\/p>\n<p>For example, <em>Beyond Reality <\/em>does not supply the top of the head, so I estimated it using the width of the face and drew an ellipse. I had to angle it to match the angle of the face with<em> atan2()<\/em>. At those same angles, I added a couple hair arcs for bangs and a couple triangles for pigtails.<\/p>\n<p>The nose vertices, as you can see in the debug-view, make up two curves: the bridge and the underside of the nose. I could have made the nose simply those two strokes, but I thought it would be kind of cute to create a triangle from the top of the bridge to the two ends of the nostrils.<\/p>\n<p>There were no pupils given either, so I had to guess where those would be. Luckily that&#8217;s pretty predictable if I assume the character will be looking straight ahead. The center of each circle was placed at the average of the coordinates of two diagonal vertices in each eye. The radius of the pupil is some fraction of the distance between those two points. It looked kind of boring without eyelashes, so I added a line anchored at each eye vertex and angled it away from the pupil using <em>atan2() <\/em>and some simple math.<\/p>\n<p>The face still seemed a little bare at this point, so I added cheeks using calculations similar to those of the pupils. The center of the blush circle is the average of the coordinates of the side of the face and the bridge of the nose. The radius is some fraction of the distance between the two\u2014this one mattered more than with the pupils, since as you turn your face, the size of one side becomes a lot larger than that of the other side.<\/p>\n<p>The last thing I did was add a simple neck and body using a rectangle and a quadrilateral, with proportions relative to the head width. I had to use the width, rather than the height (the more intuitive option), as the height of the head changes greatly when you open your mouth and it would be silly to have the size of your neck and body change with it.<\/p>\n<p>I then focused on text generation using <em><a href=\"https:\/\/rednoise.org\/rita\/index.php\">RiTa.js<\/a>, <\/em>which I fed a bunch of children&#8217;s stories from <a href=\"https:\/\/americanliterature.com\/short-stories-for-children\">this page <\/a>of short stories for children. When you open your mouth, the next word of a generated sentence (stripped of punctuation) appears. I wrote these sentences with Lingdong&#8217;s <a href=\"https:\/\/github.com\/LingDong-\/p5-hershey-js\">p5.hershey.js.\u00a0<\/a>Originally, I had these words simply appearing, but it would have been a shame if I hadn&#8217;t utilized <a href=\"https:\/\/en.wikipedia.org\/wiki\/Hershey_fonts\">hershey fonts<\/a> to at least some of their potential.<\/p>\n<p>Animating the text was one of the trickiest parts of this project, but by modifying Lingdong&#8217;s code to have an extra <em>putChar()<\/em> function called\u00a0<em>putAnimatedChar(), <\/em>which takes a time parameter and draws vertices only if time elapsed is over a certain multiple of their index. With this, I could successfully draw words, but only by drawing all the words at the same time. To solve this, I made a boolean parameter for <em>putText(),\u00a0 <\/em>which determines if the text passed to it should be animated. I broke the text into two strings in an array: one of previous words and one of the current word. The former passes <em>false <\/em>and the latter passes <em>true<\/em>. I use <em>estimateTextWidth() <\/em>to determine the leftward translation of the previous words.<\/p>\n<div id='gallery-1' class='gallery galleryid-6870 gallery-columns-3 gallery-size-full'><figure class='gallery-item'>\n\t\t\t<div class='gallery-icon portrait'>\n\t\t\t\t<a href='https:\/\/courses.ideate.cmu.edu\/60-212\/f2020\/bkrummenandrew-cmu-edu\/10\/29\/sweetcorn-mask\/31-597x1108\/'><img width=\"597\" height=\"1108\" src=\"https:\/\/courses.ideate.cmu.edu\/60-212\/f2020\/wp-content\/uploads\/2020\/10\/31-597\u00d71108.gif\" class=\"attachment-full size-full\" alt=\"\" loading=\"lazy\" \/><\/a>\n\t\t\t<\/div><\/figure><figure class='gallery-item'>\n\t\t\t<div class='gallery-icon portrait'>\n\t\t\t\t<a href='https:\/\/courses.ideate.cmu.edu\/60-212\/f2020\/bkrummenandrew-cmu-edu\/10\/29\/sweetcorn-mask\/ezgif-4-a8c109f3aa9e\/'><img width=\"600\" height=\"1112\" src=\"https:\/\/courses.ideate.cmu.edu\/60-212\/f2020\/wp-content\/uploads\/2020\/10\/ezgif-4-a8c109f3aa9e.gif\" class=\"attachment-full size-full\" alt=\"\" loading=\"lazy\" \/><\/a>\n\t\t\t<\/div><\/figure><figure class='gallery-item'>\n\t\t\t<div class='gallery-icon portrait'>\n\t\t\t\t<a href='https:\/\/courses.ideate.cmu.edu\/60-212\/f2020\/bkrummenandrew-cmu-edu\/10\/29\/sweetcorn-mask\/attachment\/16040837696768327551097402277033\/'><img width=\"600\" height=\"1109\" src=\"https:\/\/courses.ideate.cmu.edu\/60-212\/f2020\/wp-content\/uploads\/2020\/10\/16040837696768327551097402277033.gif\" class=\"attachment-full size-full\" alt=\"\" loading=\"lazy\" \/><\/a>\n\t\t\t<\/div><\/figure>\n\t\t<\/div>\n\n<p>Following the advice of a surprising amount of people, I made it my last endeavor to synthesize a voice for this character. I was reminded of the time Everest Pipkin told my EMS class about how Nintendo created <a href=\"https:\/\/animalcrossing.fandom.com\/wiki\/Language\">Animalese<\/a> in Animal Crossing and also of the <a href=\"https:\/\/www.youtube.com\/watch?v=ss2hULhXf04\">teacher&#8217;s voice in Charlie Brown<\/a>, which Golan mentioned in office hours. I figured I could make something nice using <a href=\"https:\/\/tonejs.github.io\/\"><em>Tone.js<\/em><\/a> and formants for vowels I would obtain through <em>RiTa&#8217;s<\/em> <em>getPhonemes(). <\/em>I was somewhat successful in this, though the resulting sound is far less charming and far more sinister than I&#8217;d hoped. Below is a video that demonstrates this result.<\/p>\n<div id=\"kgvid_kgvid_0_wrapper\" class=\"kgvid_wrapper\">\n\t\t\t<div id=\"video_kgvid_0_div\" class=\"fitvidsignore kgvid_videodiv\" data-id=\"kgvid_0\" data-kgvid_video_vars=\"{&quot;id&quot;:&quot;kgvid_0&quot;,&quot;attachment_id&quot;:6928,&quot;player_type&quot;:&quot;Video.js&quot;,&quot;width&quot;:&quot;640&quot;,&quot;height&quot;:360,&quot;fullwidth&quot;:&quot;true&quot;,&quot;countable&quot;:true,&quot;count_views&quot;:&quot;start_complete&quot;,&quot;start&quot;:&quot;&quot;,&quot;autoplay&quot;:&quot;false&quot;,&quot;pauseothervideos&quot;:&quot;true&quot;,&quot;set_volume&quot;:1,&quot;muted&quot;:&quot;false&quot;,&quot;meta&quot;:true,&quot;endofvideooverlay&quot;:false,&quot;resize&quot;:&quot;true&quot;,&quot;auto_res&quot;:&quot;automatic&quot;,&quot;pixel_ratio&quot;:&quot;true&quot;,&quot;right_click&quot;:&quot;on&quot;,&quot;playback_rate&quot;:&quot;false&quot;,&quot;title&quot;:&quot;RPReplay_Final1604251068&quot;,&quot;nativecontrolsfortouch&quot;:&quot;false&quot;,&quot;locale&quot;:&quot;en&quot;,&quot;enable_resolutions_plugin&quot;:false}\" itemprop=\"video\" itemscope itemtype=\"https:\/\/schema.org\/VideoObject\"><meta itemprop=\"embedUrl\" content=\"https:\/\/courses.ideate.cmu.edu\/60-212\/f2020\/?attachment_id=6928&amp;kgvid_video_embed[enable]=true\" \/><meta itemprop=\"contentUrl\" content=\"https:\/\/courses.ideate.cmu.edu\/60-212\/f2020\/wp-content\/uploads\/2020\/10\/RPReplay_Final1604251068.mov\" \/><meta itemprop=\"name\" content=\"RPReplay_Final1604251068\" \/><meta itemprop=\"description\" content=\"Video\" \/><meta itemprop=\"uploadDate\" content=\"2020-11-01T12:49:01-05:00\" \/>\n\t\t\t\t<video id=\"video_kgvid_0\" playsinline controls preload=\"metadata\" width=\"640\" height=\"360\" class=\"fitvidsignore video-js kg-video-js-skin\">\n\t\t\t\t\t<source src=\"https:\/\/courses.ideate.cmu.edu\/60-212\/f2020\/wp-content\/uploads\/2020\/10\/RPReplay_Final1604251068.mov?id=0\" type=\"video\/mp4\" data-res=\"1092p\">\n\t\t\t\t<\/video>\n\t\t\t<\/div>\n\t\t\t<div style=\"display:none;\" id=\"video_kgvid_0_meta\" class=\"kgvid_video_meta kgvid_video_meta_hover \">\n\t\t\t\t<span class='kgvid_meta_icons'><\/span>\n\t\t\t\t<span id='video_kgvid_0_title' class='kgvid_title'>RPReplay_Final1604251068<\/span>\n<\/div>\n\t\t<\/div>\n<p>To be more in line with the affect I had hoped for, I implemented a slightly-modified version of Josh Simmons&#8217; <a href=\"https:\/\/github.com\/Acedio\/animalese.js\"><em>Animalese.js <\/em><\/a>project, in which he uses javascript audio and a <em>.wav<\/em> asset to create Animal Crossing-type speech based on text input. I slowed the voice down a bit and had to change a few things to make it compatible with\u00a0my current program<em>, <\/em>but the conversion was simple enough. Each time your mouth opens, I feed the <em>Animalese<\/em> object the current word. It has a lot of charm, is very cute, and makes me much happier. Below is a video which contains the generated sound and a gif of one generated story.<\/p>\n<p><iframe title=\"Face Tracking Children&#039;s Stories\" width=\"840\" height=\"473\" src=\"https:\/\/www.youtube.com\/embed\/NdTUIJ60Btk?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen><\/iframe><\/p>\n<p><img loading=\"lazy\" class=\"aligncenter size-full wp-image-6940\" src=\"https:\/\/courses.ideate.cmu.edu\/60-212\/f2020\/wp-content\/uploads\/2020\/10\/finalstorygif.gif\" alt=\"\" width=\"600\" height=\"291\" \/><\/p>\n<p><a href=\"https:\/\/glitch.com\/edit\/#!\/childrens-stories\">Link to code<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Children&#8217;s Stories You are your own virtual teacher as you generate your own children&#8217;s stories. Open your mouth to show each new word. I determine if the user is opening their mouth by comparing the distance between their lips to the distance between the top of their head and the bottom of their head. Above &hellip; <a href=\"https:\/\/courses.ideate.cmu.edu\/60-212\/f2020\/bkrummenandrew-cmu-edu\/10\/29\/sweetcorn-mask\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;sweetcorn-mask&#8221;<\/span><\/a><\/p>\n","protected":false},"author":46,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[63],"tags":[],"_links":{"self":[{"href":"https:\/\/courses.ideate.cmu.edu\/60-212\/f2020\/wp-json\/wp\/v2\/posts\/6870"}],"collection":[{"href":"https:\/\/courses.ideate.cmu.edu\/60-212\/f2020\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/courses.ideate.cmu.edu\/60-212\/f2020\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/courses.ideate.cmu.edu\/60-212\/f2020\/wp-json\/wp\/v2\/users\/46"}],"replies":[{"embeddable":true,"href":"https:\/\/courses.ideate.cmu.edu\/60-212\/f2020\/wp-json\/wp\/v2\/comments?post=6870"}],"version-history":[{"count":8,"href":"https:\/\/courses.ideate.cmu.edu\/60-212\/f2020\/wp-json\/wp\/v2\/posts\/6870\/revisions"}],"predecessor-version":[{"id":6941,"href":"https:\/\/courses.ideate.cmu.edu\/60-212\/f2020\/wp-json\/wp\/v2\/posts\/6870\/revisions\/6941"}],"wp:attachment":[{"href":"https:\/\/courses.ideate.cmu.edu\/60-212\/f2020\/wp-json\/wp\/v2\/media?parent=6870"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/courses.ideate.cmu.edu\/60-212\/f2020\/wp-json\/wp\/v2\/categories?post=6870"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/courses.ideate.cmu.edu\/60-212\/f2020\/wp-json\/wp\/v2\/tags?post=6870"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}