{"id":1367,"date":"2022-02-28T01:51:30","date_gmt":"2022-02-28T06:51:30","guid":{"rendered":"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/?page_id=1367"},"modified":"2022-02-28T01:52:27","modified_gmt":"2022-02-28T06:52:27","slug":"machine-learning-ii","status":"publish","type":"page","link":"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/daily-notes\/02-28-machine-learning\/machine-learning-ii\/","title":{"rendered":"Machine Learning II"},"content":{"rendered":"<p><iframe loading=\"lazy\" title=\"Entangled II\" src=\"https:\/\/player.vimeo.com\/video\/345875600?dnt=1&amp;app_id=122963\" width=\"840\" height=\"473\" frameborder=\"0\" allowfullscreen=\"allowfullscreen\" data-mce-fragment=\"1\"><\/iframe><\/p>\n<p><a href=\"https:\/\/vimeo.com\/345875600\"><em>Entangled II<\/em><\/a>\u00a0by Scott Eaton.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-1368\" src=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/ezgif-1-5fb6ee312e.gif\" alt=\"\" width=\"540\" height=\"540\" \/><\/p>\n<p><em>Performance II: Skin on Repeat,<\/em>\u00a0NFT by\u00a0<a href=\"http:\/\/www.chrystalding.com\/\">Chrystal Y. Ding.<\/a><br \/>\nDing is an artist\/writer \u201cinterested in fluctuations in identity and embodiment\u201d. She \u201cuses machine learning to explore the impact of trauma &amp; future technology on identity.\u201d<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-1369\" src=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2022-02-28-at-1.28.32-AM.png\" alt=\"\" width=\"1878\" height=\"868\" srcset=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2022-02-28-at-1.28.32-AM.png 1878w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2022-02-28-at-1.28.32-AM-640x296.png 640w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2022-02-28-at-1.28.32-AM-1024x473.png 1024w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2022-02-28-at-1.28.32-AM-768x355.png 768w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2022-02-28-at-1.28.32-AM-1536x710.png 1536w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2022-02-28-at-1.28.32-AM-1200x555.png 1200w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/p>\n<p><em><strong><a href=\"https:\/\/twitter.com\/zachwhalen\/status\/1331460387615682560\">StyleGAN2 trained on comics<\/a><\/strong><\/em>, study by Zach Whelan.<\/p>\n<hr \/>\n<h1>Agenda<\/h1>\n<h5>ML Image Synthesis<\/h5>\n<ul>\n<li>An Explanation of Convolution<\/li>\n<li>Text-to-Image and Google Colab<\/li>\n<li>A\u00a0<a href=\"https:\/\/twitter.com\/gwenckatz\/status\/1381652071695351810\">caveat about AI colorization<\/a><\/li>\n<\/ul>\n<h5>A Sampler of Diverse uses of ML in Art<\/h5>\n<ul>\n<li>Human-Robot Interaction<\/li>\n<li>Neural Nets in Games<\/li>\n<li>Poetry<\/li>\n<li>Music<\/li>\n<li>Film<\/li>\n<\/ul>\n<hr \/>\n<h3>Convolution: The Foundation of Convolutional Neural Nets<\/h3>\n<p><strong><em>Convolution<\/em><\/strong>\u00a0is a fancy-sounding name for an simple image-processing operation (such as blurring, sharpening, or edge detection). It underlies\u00a0<em>all<\/em>\u00a0of the recent work in Deep Convolutional Neural Nets and is worth taking a moment to understand. In the simplest terms, a\u00a0<em>convolution<\/em>\u00a0is an operation in which:<\/p>\n<ul>\n<li>each given pixel\u00a0<em>(1\u00d71 gray)<\/em><\/li>\n<li>in a destination image\u00a0<em>(cyan)<\/em><\/li>\n<li>is computed by taking a weighted average of<\/li>\n<li>a\u00a0<em>neighborhood<\/em>\u00a0of corresponding pixels<em>\u00a0(3\u00d73 gray)<\/em><\/li>\n<li>from a source image<em>\u00a0(blue):<\/em><\/li>\n<\/ul>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-1370\" src=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/same_padding_no_strides.gif\" alt=\"\" width=\"395\" height=\"449\" \/><\/p>\n<p>Let\u2019s\u00a0<strong><a href=\"https:\/\/www.youtube.com\/watch?v=8rrHTtUzyZA&amp;t=72s\">watch this nice explanation<\/a><\/strong>\u00a0of \u201cConvolution\u201d from 1:12\u20132:50, and 4:31\u20136:13:<\/p>\n<p><iframe loading=\"lazy\" title=\"Convolutions in image processing | Week 1 | MIT 18.S191 Fall 2020 | Grant Sanderson\" src=\"https:\/\/www.youtube.com\/embed\/8rrHTtUzyZA?start=72&amp;feature=oembed\" width=\"840\" height=\"473\" frameborder=\"0\" allowfullscreen=\"allowfullscreen\" data-mce-fragment=\"1\"><\/iframe><\/p>\n<p><a href=\"https:\/\/setosa.io\/ev\/image-kernels\/\"><strong>Here\u2019s an interactive explanation<\/strong><\/a>\u00a0that should make this even more clear:<\/p>\n<p><a href=\"https:\/\/setosa.io\/ev\/image-kernels\/\"><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-1371\" src=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2021-04-14-at-3.40.06-AM-1536x654-1.png\" alt=\"\" width=\"1536\" height=\"654\" srcset=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2021-04-14-at-3.40.06-AM-1536x654-1.png 1536w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2021-04-14-at-3.40.06-AM-1536x654-1-640x273.png 640w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2021-04-14-at-3.40.06-AM-1536x654-1-1024x436.png 1024w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2021-04-14-at-3.40.06-AM-1536x654-1-768x327.png 768w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2021-04-14-at-3.40.06-AM-1536x654-1-1200x511.png 1200w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/a><\/p>\n<p>&nbsp;<\/p>\n<p>Convolutional Neural Networks (CNNs) use convolutions at their lowest levels to detect features like points and edges. At the next-higher levels, the results of these detections (\u201cactivations\u201d) are grouped in ever-more-complex combinations (<em>textures\u2192patterns\u2192parts\u2192objects<\/em>). This\u00a0<strong><a href=\"https:\/\/poloclub.github.io\/cnn-explainer\/\">CNN Explainer<\/a><\/strong>\u00a0does a good job of opening the hood:<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-1372\" src=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/convlayer_detailedview_demo.gif\" alt=\"\" width=\"614\" height=\"327\" \/><\/p>\n<hr \/>\n<h3>Image-to-Text: Bleeding Edge Techniques<\/h3>\n<p>For some time, AI researchers have worked on<strong>\u00a0automatic captioning systems<\/strong>, computationally deriving text from images. Here are the results of one system called\u00a0<em><a href=\"https:\/\/medium.com\/@samim\/generating-stories-about-images-d163ba41e4ed\">Neural Storyteller<\/a><\/em>, developed in 2015. Note that the language model used to generate the text for these was\u00a0<em>trained on romance novels<\/em>.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-1373\" src=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2021-04-14-at-2.37.39-AM-2048x995-1.png\" alt=\"\" width=\"2048\" height=\"995\" srcset=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2021-04-14-at-2.37.39-AM-2048x995-1.png 2048w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2021-04-14-at-2.37.39-AM-2048x995-1-640x311.png 640w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2021-04-14-at-2.37.39-AM-2048x995-1-1024x498.png 1024w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2021-04-14-at-2.37.39-AM-2048x995-1-768x373.png 768w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2021-04-14-at-2.37.39-AM-2048x995-1-1536x746.png 1536w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2021-04-14-at-2.37.39-AM-2048x995-1-1200x583.png 1200w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-1374\" src=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/1_CCF_gTs16d_FmosXZAb_UA-2048x993-1.jpg\" alt=\"\" width=\"2048\" height=\"993\" srcset=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/1_CCF_gTs16d_FmosXZAb_UA-2048x993-1.jpg 2048w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/1_CCF_gTs16d_FmosXZAb_UA-2048x993-1-640x310.jpg 640w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/1_CCF_gTs16d_FmosXZAb_UA-2048x993-1-1024x497.jpg 1024w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/1_CCF_gTs16d_FmosXZAb_UA-2048x993-1-768x372.jpg 768w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/1_CCF_gTs16d_FmosXZAb_UA-2048x993-1-1536x745.jpg 1536w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/1_CCF_gTs16d_FmosXZAb_UA-2048x993-1-1200x582.jpg 1200w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/p>\n<p>But what about the opposite? Is it possible to go from text to images? The\u00a0<strong><a href=\"https:\/\/www.wordseye.com\/\">Wordseye<\/a><\/strong>\u00a0system allows you to do this (in the browser). It does not use machine learning to generate the images; rather, it combines a large library of readymade 3D models according to the nouns and prepositions in the provided caption.<\/p>\n<p><iframe loading=\"lazy\" title=\"Welcome to WordsEye\" src=\"https:\/\/www.youtube.com\/embed\/P3PsSJx3QtA?start=28&amp;feature=oembed\" width=\"840\" height=\"630\" frameborder=\"0\" allowfullscreen=\"allowfullscreen\" data-mce-fragment=\"1\"><\/iframe><\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-1375\" src=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2021-04-14-at-2.48.46-AM.png\" alt=\"\" width=\"1246\" height=\"1038\" srcset=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2021-04-14-at-2.48.46-AM.png 1246w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2021-04-14-at-2.48.46-AM-576x480.png 576w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2021-04-14-at-2.48.46-AM-1024x853.png 1024w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2021-04-14-at-2.48.46-AM-768x640.png 768w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2021-04-14-at-2.48.46-AM-1200x1000.png 1200w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/p>\n<p>In 2018, the AttnGAN network was developed. From text descriptions, it could synthesize images of things similar to ones it had been trained on\u2014 generally, objects in the ImageNet database.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-1388\" src=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/1_45RxSlzrE271jkXNqo55bA.png\" alt=\"\" width=\"714\" height=\"268\" srcset=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/1_45RxSlzrE271jkXNqo55bA.png 714w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/1_45RxSlzrE271jkXNqo55bA-640x240.png 640w\" sizes=\"(max-width: 714px) 85vw, 714px\" \/><\/p>\n<p>Things have advanced significantly. Last year, OpenAI <a href=\"https:\/\/openai.com\/blog\/dall-e\/\">released a new model called \u201cDall-E\u201d<\/a>\u00a0that could contruct novel forms in unexpected ways. Here it is synthesizing images of \u201can armchair in the shape of an avocado\u201d:<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-1387\" src=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2021-04-14-at-3.18.47-AM-2048x1571-1-1.png\" alt=\"\" width=\"2048\" height=\"1571\" srcset=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2021-04-14-at-3.18.47-AM-2048x1571-1-1.png 2048w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2021-04-14-at-3.18.47-AM-2048x1571-1-1-626x480.png 626w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2021-04-14-at-3.18.47-AM-2048x1571-1-1-1024x786.png 1024w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2021-04-14-at-3.18.47-AM-2048x1571-1-1-768x589.png 768w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2021-04-14-at-3.18.47-AM-2048x1571-1-1-1536x1178.png 1536w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2021-04-14-at-3.18.47-AM-2048x1571-1-1-1200x921.png 1200w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/p>\n<p>Let\u2019s take a moment to experiment with three extremely recent code repositories that build on this work. These have all been developed in the last month or so, and will give us the opportunity to run a project in a\u00a0<em>Google Colab Notebook<\/em>, which is how many ML algorithms are developed:<\/p>\n<ul>\n<li>DeepDaze (<strong><a href=\"https:\/\/github.com\/lucidrains\/deep-daze\">Github<\/a><\/strong>; use \u201cSimplified [Colab] Notebook\u201d)<\/li>\n<li>BigSleep (<strong><a href=\"https:\/\/github.com\/lucidrains\/big-sleep\">Github<\/a><\/strong>; use \u201cSimplified [Colab] Notebook\u201d or\u00a0<a href=\"https:\/\/colab.research.google.com\/drive\/1NCceX2mbiKOSlAd_o7IU7nA9UskKN5WR?usp=sharing\"><strong>Colab<\/strong><\/a>)<\/li>\n<li>Aleph2Image (<strong><a href=\"https:\/\/colab.research.google.com\/drive\/1oA1fZP7N1uPBxwbGIvOEXbTsq2ORa9vb\">Colab<\/a><\/strong>)<\/li>\n<\/ul>\n<p>Here is a comparison of how each of these algorithms interpreted the sentence \u201ccute frogs eating pizza\u201d that I provided to them. How would you describe the difference in their results?<\/p>\n<p><em><strong>DeepDaze<\/strong><\/em>\u00a0(\u201ccute frogs eating pizza\u201d):<br \/>\n<img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-1377\" src=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/cute-frogs-eating-pizza-deepdaze.png\" alt=\"\" width=\"512\" height=\"512\" srcset=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/cute-frogs-eating-pizza-deepdaze.png 512w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/cute-frogs-eating-pizza-deepdaze-480x480.png 480w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/cute-frogs-eating-pizza-deepdaze-150x150.png 150w\" sizes=\"(max-width: 512px) 85vw, 512px\" \/><\/p>\n<p><em><strong>BigSleep<\/strong><\/em>\u00a0(\u201ccute frogs eating pizza\u201d):<br \/>\n<img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-1378\" src=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/cute-frogs-eating-pizza-bigsleep.png\" alt=\"\" width=\"512\" height=\"512\" srcset=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/cute-frogs-eating-pizza-bigsleep.png 512w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/cute-frogs-eating-pizza-bigsleep-480x480.png 480w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/cute-frogs-eating-pizza-bigsleep-150x150.png 150w\" sizes=\"(max-width: 512px) 85vw, 512px\" \/><\/p>\n<p><strong><em>Aleph2Image<\/em><\/strong>\u00a0(\u201ccute frogs eating pizza\u201d):<br \/>\n<img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-1379\" src=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Ey6Vpz7VUAcGdKH.png\" alt=\"\" width=\"512\" height=\"512\" srcset=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Ey6Vpz7VUAcGdKH.png 512w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Ey6Vpz7VUAcGdKH-480x480.png 480w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Ey6Vpz7VUAcGdKH-150x150.png 150w\" sizes=\"(max-width: 512px) 85vw, 512px\" \/><\/p>\n<p>Here\u2019s a comparative study Simon Colton made of their output:<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-1380\" src=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Evatf3CWgAAVR7T.jpg\" alt=\"\" width=\"1024\" height=\"768\" srcset=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Evatf3CWgAAVR7T.jpg 1024w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Evatf3CWgAAVR7T-640x480.jpg 640w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Evatf3CWgAAVR7T-768x576.jpg 768w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-1381\" src=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Evatko5XEAEb-is.jpg\" alt=\"\" width=\"1024\" height=\"768\" srcset=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Evatko5XEAEb-is.jpg 1024w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Evatko5XEAEb-is-640x480.jpg 640w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Evatko5XEAEb-is-768x576.jpg 768w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/p>\n<p>&nbsp;<\/p>\n<hr \/>\n<p>Artist Trevor Paglen collaborated with the Kronos Quartet on\u00a0<em>Sight Machine<\/em>\u00a0(2018), a performance in which the musicians were analyzed by (projected) computer vision systems. \u201c<em>Sight Machine<\/em>\u2019s premise is simple and strange.\u00a0 The Kronos Quartet performs on stage, and Paglen runs a series of computer vision algorithms on the live video of this performance; behind the musicians, he then projects a video that these algorithms generate.\u00a0 Paglen is an artist best known for his photography of spy satellites, military drones, and secret intelligence facilities, and in the past couple of years, he has begun exploring the social ramifications of artificial intelligence.\u201d<\/p>\n<p><iframe loading=\"lazy\" title=\"Holland Festival 2018: Sight Machine - Trevor Paglen &amp; Kronos Quartet\" src=\"https:\/\/www.youtube.com\/embed\/5mg3MXETfj4?feature=oembed\" width=\"840\" height=\"473\" frameborder=\"0\" allowfullscreen=\"allowfullscreen\" data-mce-fragment=\"1\"><\/iframe><\/p>\n<p><iframe loading=\"lazy\" title=\"Trevor Paglen's Sight Machine with Kronos Quartet and Obscura Digital\" src=\"https:\/\/player.vimeo.com\/video\/205149078?dnt=1&amp;app_id=122963\" width=\"840\" height=\"473\" frameborder=\"0\" allowfullscreen=\"allowfullscreen\" data-mce-fragment=\"1\"><\/iframe><\/p>\n<hr \/>\n<h3>Critical Appery<\/h3>\n<p><strong><em>White Collar Crime Risk Zones<\/em><\/strong>\u00a0by Sam Lavigne et al. uses machine learning to predict where financial crimes are mostly likely to occur across the US. The project is realized as a website, smartphone app and gallery installation.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-1386\" src=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/white-collar-1.png\" alt=\"\" width=\"1800\" height=\"1010\" srcset=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/white-collar-1.png 1800w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/white-collar-1-640x359.png 640w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/white-collar-1-1024x575.png 1024w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/white-collar-1-768x431.png 768w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/white-collar-1-1536x862.png 1536w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/white-collar-1-1200x673.png 1200w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/p>\n<p><iframe loading=\"lazy\" title=\"White Collar Crime Risk Zones Sam Lavigne (The New Inquiry)\" src=\"https:\/\/www.youtube.com\/embed\/Wj8meXnHSPo?feature=oembed\" width=\"840\" height=\"473\" frameborder=\"0\" allowfullscreen=\"allowfullscreen\" data-mce-fragment=\"1\"><\/iframe><\/p>\n<hr \/>\n<h3>Absurd machine vision<\/h3>\n<p><em><a href=\"https:\/\/notnot.home.xs4all.nl\/pareidolia\/pareidolia.html\"><strong>Pareidolia<\/strong><\/a><\/em>, Maria Verstappen &amp; Erwin Driessens, 2019. [<strong><em><a href=\"https:\/\/notnot.home.xs4all.nl\/pareidolia\/pareidolia-movie.html\">VIDEO<\/a><\/em><\/strong>]<br \/>\nIn the artwork\u00a0<em>Pareidolia<\/em>, facial detection is applied to grains of sand. A fully automated robot search engine examines the grains of sand in situ. When the machine finds a face in one of the grains, the portrait is photographed and displayed on a large screen.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-1382\" src=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/pareidolia.png\" alt=\"\" width=\"1280\" height=\"848\" srcset=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/pareidolia.png 1280w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/pareidolia-640x424.png 640w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/pareidolia-1024x678.png 1024w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/pareidolia-768x509.png 768w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/pareidolia-1200x795.png 1200w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/p>\n<hr \/>\n<h3>Text Synthesis and Poetry<\/h3>\n<p>Theo Lutz is credited as the author of the first text classifiable as a work of electronic literature. Here is his computer-generated \u201c<em><a href=\"https:\/\/zkm.de\/en\/artwork\/stochastic-texts\"><strong>Stochastische Texte,<\/strong><\/a><\/em>\u201d 1959:<\/p>\n<figure id=\"attachment_1383\" aria-describedby=\"caption-attachment-1383\" style=\"width: 1365px\" class=\"wp-caption alignnone\"><img decoding=\"async\" loading=\"lazy\" class=\"size-full wp-image-1383\" src=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/ZKM000163526.jpg\" alt=\"\" width=\"1365\" height=\"1024\" srcset=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/ZKM000163526.jpg 1365w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/ZKM000163526-640x480.jpg 640w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/ZKM000163526-1024x768.jpg 1024w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/ZKM000163526-768x576.jpg 768w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/ZKM000163526-1200x900.jpg 1200w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><figcaption id=\"caption-attachment-1383\" class=\"wp-caption-text\">Theo Lutz, \u00bbStochastische Texte\u00ab, 1959, computergeneriertes Gedicht ; Computer: Zuse Z22; Ausgabe: Fernschreiber, ca. 80 x 20 cm<\/figcaption><\/figure>\n<p>Computer poetry (~1959) preceded computer graphics and visual computer art (~1964) by a few years. The first computers could only manipulate symbols (like letters and numbers). Here\u2019s more early computer poetry by Brion Gysin (1960):<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-1384\" src=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2021-04-14-at-5.41.43-AM.png\" alt=\"\" width=\"1574\" height=\"888\" srcset=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2021-04-14-at-5.41.43-AM.png 1574w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2021-04-14-at-5.41.43-AM-640x361.png 640w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2021-04-14-at-5.41.43-AM-1024x578.png 1024w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2021-04-14-at-5.41.43-AM-768x433.png 768w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2021-04-14-at-5.41.43-AM-1536x867.png 1536w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/Screen-Shot-2021-04-14-at-5.41.43-AM-1200x677.png 1200w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/p>\n<p>Janelle Shane is particularly well-known for her\u00a0<strong><a href=\"https:\/\/aiweirdness.com\/post\/160776374467\/new-paint-colors-invented-by-neural-network\">humorous 2017 project<\/a><\/strong>\u00a0in which she trained a neural network to generate the names and colors of new paints:<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-1385\" src=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/janelle_shane_colornames-293.jpg\" alt=\"\" width=\"1427\" height=\"942\" srcset=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/janelle_shane_colornames-293.jpg 1427w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/janelle_shane_colornames-293-640x422.jpg 640w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/janelle_shane_colornames-293-1024x676.jpg 1024w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/janelle_shane_colornames-293-768x507.jpg 768w, https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-content\/uploads\/2022\/02\/janelle_shane_colornames-293-1200x792.jpg 1200w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/p>\n<p>The world\u2019s foremost computational poet is probably <strong><em><a href=\"https:\/\/www.decontextualize.com\/\">Allison Parrish<\/a><\/em><\/strong>.\u00a0 Here is an astoundingly good video from 2015; let\u2019s watch a few minutes starting from 1:30-4:30.<\/p>\n<p><iframe loading=\"lazy\" title=\"Eyeo 2015 &amp;ndash; Allison Parrish\" src=\"https:\/\/player.vimeo.com\/video\/134734729?dnt=1&amp;app_id=122963\" width=\"840\" height=\"473\" frameborder=\"0\" allowfullscreen=\"allowfullscreen\" data-mce-fragment=\"1\"><\/iframe><\/p>\n<hr \/>\n<h3>Music<\/h3>\n<p><em><a href=\"https:\/\/experiments.withgoogle.com\/ai\/drum-machine\/view\/\"><strong>The Infinite Drum Machine<\/strong><\/a><\/em>, Manny Tan &amp; Kyle McDonald (2017). \u201cThis experiment uses machine learning to organize thousands of everyday sounds. The computer wasn\u2019t given any descriptions or tags \u2013 only the audio. Using a technique called t-SNE, the computer placed similar sounds closer together. You can use the map to explore neighborhoods of similar sounds and even make beats using the drum sequencer.\u201d<\/p>\n<p><iframe loading=\"lazy\" title=\"A.I. Experiments: The Infinite Drum Machine\" src=\"https:\/\/www.youtube.com\/embed\/9x-_My5yjQY?feature=oembed\" width=\"840\" height=\"473\" frameborder=\"0\" allowfullscreen=\"allowfullscreen\" data-mce-fragment=\"1\"><\/iframe><\/p>\n<hr \/>\n<h3>Filmmaking<\/h3>\n<p>Ross Goodwin:\u00a0<a href=\"https:\/\/www.youtube.com\/watch?v=LY7x2Ihqjmc\"><strong><em>Sunspring, A Sci-Fi Short Film<\/em><\/strong><\/a><\/p>\n<p><iframe loading=\"lazy\" title=\"Sunspring | A Sci-Fi Short Film Starring Thomas Middleditch\" src=\"https:\/\/www.youtube.com\/embed\/LY7x2Ihqjmc?feature=oembed\" width=\"840\" height=\"473\" frameborder=\"0\" allowfullscreen=\"allowfullscreen\" data-mce-fragment=\"1\"><\/iframe><\/p>\n<h3><\/h3>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Entangled II\u00a0by Scott Eaton. Performance II: Skin on Repeat,\u00a0NFT by\u00a0Chrystal Y. Ding. Ding is an artist\/writer \u201cinterested in fluctuations in identity and embodiment\u201d. She \u201cuses machine learning to explore the impact of trauma &amp; future technology on identity.\u201d StyleGAN2 trained on comics, study by Zach Whelan. Agenda ML Image Synthesis An Explanation of Convolution Text-to-Image &hellip; <a href=\"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/daily-notes\/02-28-machine-learning\/machine-learning-ii\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Machine Learning II&#8221;<\/span><\/a><\/p>\n","protected":false},"author":3,"featured_media":0,"parent":1363,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":[],"_links":{"self":[{"href":"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-json\/wp\/v2\/pages\/1367"}],"collection":[{"href":"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-json\/wp\/v2\/comments?post=1367"}],"version-history":[{"count":2,"href":"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-json\/wp\/v2\/pages\/1367\/revisions"}],"predecessor-version":[{"id":1390,"href":"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-json\/wp\/v2\/pages\/1367\/revisions\/1390"}],"up":[{"embeddable":true,"href":"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-json\/wp\/v2\/pages\/1363"}],"wp:attachment":[{"href":"https:\/\/courses.ideate.cmu.edu\/60-212\/s2022\/wp-json\/wp\/v2\/media?parent=1367"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}