My final project uses the webcam to generate simple animations that will follow motion/gestures of the user through a red color marker. For the best experience, make sure to only have one red object in front of the camera at a time.

Instructions:

1. grab something red (any relatively small red object would work best, or just pull up a red image on your phone)

2. hold it within range of the webcam, with the red facing the camera

3. wave/draw/play

   *to see the animations alone, you can turn the camera image on/off by pressing camera button

   press on shape 1, shape 2, and shape 3 to see different shapes

   works best in a brightly lit room, so the computer can recognize the red object

sketch

// Sophie Chen
// sophiec@andrew.cmu.edu
// 15-104 Final Project
// Dec 2018

// This program uses the webcam and generates a simple animation that will
// follow a red color marker. For best user experience, make sure to only have
// one red object in front of the camera at a time.

// declare variables
var myCaptureDevice;
var xarray = []; // array to store the value of x positions
var yarray = []; // array to store the value of y positions
var limitSize = 50; // length limit of x and y arrays
var cameraOn = 1; // camera switch - default on
var shapeOne = 1; // default starts with shape one
var shapeTwo = -1; // shape two starts false
var shapeThree = -1; // shape three starts false
var turtle; // define turtle graphics variable


// setup loads the webcam input and initializes turtle graphics
function setup() {
    createCanvas(600, 430);
    myCaptureDevice = createCapture(VIDEO);
    myCaptureDevice.size(600, 430);
    myCaptureDevice.hide();
    turtle = makeTurtle(0, 0);
    turtle.penDown();
    turtle.setColor(255);
}

// this function tests for color as a condition
function isColor(c) {
	return (c instanceof Array);
}

// this function draws the animations on top of the camera image
function draw() {
    myCaptureDevice.loadPixels();
    // if cameraOn is true, load camera output
    // if cameraOn is false, camera output is not visible, load black background
    if (cameraOn === 1){
        image(myCaptureDevice, 0, 0);
    } else {
        background(0);
    }

    //call functions that draw the buttons
    drawCamButton();
    drawShapeButton();
    drawShapeTwoButton();
    drawShapeThreeButton();

    //declare variables used to calculate centerpoint of red object/marker
    var xMin = 0; // x value minimum 
    var xMax = 600; // x value maximum
    var yMin = 0; // y value minimum
    var yMax = 430; // y value maximum

    // for loop that draws the shape animations
    for (var a = 0; a < xarray.length; a++){ 
        // declare color and size variables based on forloop
        var size = (50 / xarray.length) * a;
        var r = map(a, 0, xarray.length, 0, 255);
        var g = map(a, 0, xarray.length, 0, 255);
        var b = map(a, 0, xarray.length, 0, 255);

        // Shape 1: filled ellipse
        if (shapeOne === 1){
            shapeTwo = -1;
            shapeThree = -1;
            noStroke();
            fill(r, g, 255);
            ellipse(xarray[a], yarray[a], size, size);
        }


        // Shape 2: outlined ellipse
        if (shapeTwo === 1) {
            shapeOne = -1;
            shapeThree = -1;
            noFill();
            stroke(r, g, b);
            strokeWeight(1);
            ellipse(xarray[a], yarray[a], size, size);
        }

        
        // Shape 3: turtle graphics
        if (shapeThree === 1) {
            shapeOne = -1;
            shapeTwo = -1;
            turtle.setColor(color(205, 255, 10));
            turtle.goto(xarray[a], yarray[a]);
            turtle.forward(25);
            turtle.right(90);  
        }
    }

    // get the color value of every 5 pixels of webcam output
    for (var i = 0; i < width; i += 5){
        for (var j = 0; j < height; j+= 5) {
            var currentColor = myCaptureDevice.get(i, j);
            // targetColor: color(255, 0, 0);

            // calculate the difference between current color and target color
            var dr = red(currentColor) - 255;
            var dg = green(currentColor) - 0;
            var db =  blue(currentColor) - 0;
            
            // if current color is close enough to target color (~120), calculate
            // center point of the red area
            if (isColor(currentColor)){
               var dist = sqrt(sq(dr) + sq(dg) + sq(db));
               if (dist < 120) {
                    // find center point of red marker
                    if (i > xMin){ 
                        xMin = i;
                    }
                    if (i < xMax){
                        xMax = i;
                    }

                    if (j > yMin){
                        yMin = j;
                    }
                    if (j < yMax){
                        yMax = j;
                    } 
                }   
    		}
    	}
    }

    // push the newly discovered x, y into the array 
    xarray.push((xMin + xMax) / 2);
    yarray.push((yMin + yMax) / 2); 
    
    // if array is full, pop something out from the beginning
    while (xarray.length > limitSize) {
        xarray.shift();
        yarray.shift();
    }
}

// functions to trigger responses of buttons pressed
function mouseClicked(){
    // if camera button is pressed, toggle on/off
    if (mouseX > 10 & mouseX < 60 && mouseY > 10 && mouseY < 30){
        cameraOn = -cameraOn;
    }
    // if shape 1 button is pressed, show shape 1, disable shape 2 & 3
    if (mouseX > 10 && mouseX < 60 && mouseY > 20 && mouseY < 60){
        shapeOne = 1;
        shapeTwo = -1;
        shapeThree = -1;
    }
    // if shape 2 button is pressed, show shape 2, disable shape 1 & 3
    if (mouseX > 10 && mouseX < 60 && mouseY > 60 && mouseY < 90){
        shapeTwo = 1;
        shapeOne = -1;
        shapeThree = -1;
    }
    // if shape 3 button is pressed, show shape 3, disable shape 1 & 2
    if (mouseX > 10 && mouseX < 60 && mouseY > 90 && mouseY < 120){
        shapeThree = 1;
        shapeTwo = -1;
        shapeOne = -1;
    }

}

// camera button
function drawCamButton(){
    fill(255);
    stroke(0);
    rect(10, 10, 50, 20);
    noStroke();
    fill(0);
    text('camera', 15, 23);
}

// shape 1 button
function drawShapeButton(){
    fill(255);
    stroke(0);
    rect(10, 40, 50, 20);
    noStroke();
    fill(0);
    text('shape 1', 15, 54);
}

// shape 2 button
function drawShapeTwoButton(){
    fill(255);
    stroke(0);
    rect(10, 70, 50, 20);
    noStroke();
    fill(0);
    text('shape 2', 15, 85);
}

// shape 3 button
function drawShapeThreeButton(){
    fill(255);
    stroke(0);
    rect(10, 100, 50, 20);
    noStroke();
    fill(0);
    text('shape 3', 15, 116);
}

//////////////////////////////////////////////////////////////////////////////
function turtleLeft(d){this.angle-=d;}function turtleRight(d){this.angle+=d;}
function turtleForward(p){var rad=radians(this.angle);var newx=this.x+cos(rad)*p;
var newy=this.y+sin(rad)*p;this.goto(newx,newy);}function turtleBack(p){
this.forward(-p);}function turtlePenDown(){this.penIsDown=true;}
function turtlePenUp(){this.penIsDown = false;}function turtleGoTo(x,y){
if(this.penIsDown){stroke(this.color);strokeWeight(this.weight);
line(this.x,this.y,x,y);}this.x = x;this.y = y;}function turtleDistTo(x,y){
return sqrt(sq(this.x-x)+sq(this.y-y));}function turtleAngleTo(x,y){
var absAngle=degrees(atan2(y-this.y,x-this.x));
var angle=((absAngle-this.angle)+360)%360.0;return angle;}
function turtleTurnToward(x,y,d){var angle = this.angleTo(x,y);if(angle< 180){
this.angle+=d;}else{this.angle-=d;}}function turtleSetColor(c){this.color=c;}
function turtleSetWeight(w){this.weight=w;}function turtleFace(angle){
this.angle = angle;}function makeTurtle(tx,ty){var turtle={x:tx,y:ty,
angle:0.0,penIsDown:true,color:color(128),weight:1,left:turtleLeft,
right:turtleRight,forward:turtleForward, back:turtleBack,penDown:turtlePenDown,
penUp:turtlePenUp,goto:turtleGoTo, angleto:turtleAngleTo,
turnToward:turtleTurnToward,distanceTo:turtleDistTo, angleTo:turtleAngleTo,
setColor:turtleSetColor, setWeight:turtleSetWeight,face:turtleFace};
return turtle;}

I really enjoyed the camera interaction aspect of the text rain assignment, which is why I wanted to work more with live cam for this project and create something where the user has even more direct control. Overall this project was a lot more challenging than I expected, but that also made it more rewarding when I finally got it to work. I’m glad I went with color recognition for the marker because it allows for precise control and is more forgiving towards the user in terms of what environment they should be in when using this program. The most time-consuming and unexpected challenge was the lagging and freezing that comes with working with so many pixels, so trying to figure out what was causing the freezing took a lot longer than changing the code to fix it. Since the animations are on the simple side, I decided to include 3 different options. Ideally the animations would’ve been more complex, that’s something I hope to keep working on in the future. Below are screenshots from me using the program to give an idea of how it would look like when it’s working.

For my final project, I would like to create animations that react to movement and gestures in front of a camera. I study projection design in the school of drama, so I’m really interested in possibilities between a performer and live reactive animations. I hope to create a prototype that can be possibly expanded to a larger scale or used for live performances/installations in the future. I’m not entirely sure if the final visual result will include the camera input of the users or consist of just the animations alone, but I think that’s something I will decide as the animations solidify. Since this project will be receiving its camera input through a webcam, I think it will rely on the viewer to be in front of a solid color background wearing solid color clothing. The actual animation content will be abstract and simple, and may be created through turtle graphics.

rough sketch

For my final project, I wanted to dive deeper into animations that interact with or respond to a live camera input. As I was doing research I came across this project called webcam piano created by Memo Akten, where the users’ gestures not only trigger animations but also different keys of the piano, creating their own visual and sound sequence. Another project that I found is Fauna by Dr. Woohoo, an immersive art experience where a floating fauna seems to be just on screen but when people stand in front of it, it starts moving and engaging with them. Both projects are triggered by what’s in front of the camera, the main difference being that with the webcam piano the users see themselves and the animations on top, whereas with Fauna users don’t see themselves on screen at all, merely the animations that are triggered by their movements/presence. This made me think a lot about what it means when the users are able to see themselves vs. when they are not, and what I want to do for my project.

Fauna

Webcam Piano

Link to webcam piano: http://www.memo.tv/portfolio/webcam-piano-2/

Link to Fauna: http://blog.drwoohoo.com/currents-2010-playing-with-fauna/

Laetitia Sonami

Laetitia Sonami is a sound artist, performer, and composer of interactive electronic music based in San Francisco. What initially drew my attention is “The Lady’s Glove”, an instrument she developed herself, which triggers and manipulates sound in live performance. This instrument is worn on her right hand, and is a black glove made of mesh integrated with a lot of different sensors such as micro-switches, pressure pads, ultrasonic receivers, and light sensors, just to name a few. The signals the glove receive are connected to a hardware named Sensorlab, which is then mapped onto MAX MSP running on a computer, which connects it to pre-stored sounds.

Sonami said that through creating this instrument, she was trying to figure out at which point does a controller become an instrument. I found what she said to be very insightful and thought provoking, especially applicable today when there are so many tools around us that we can just use to generate things without even putting much thought into it. Sonami concludes that when a software starts adapting to the controller, it becomes more of a symbiosis between the controller, the code, and the software. I think her glove does successfully embody that and qualify as an instrument, not just a controller/generator.

(jump to 10:34 for Sonami performing with the glove)

sketch

// Sophie Chen
// Section C
// sophiec@andrew.cmu.edu
// Project 11

var turtle;

function setup() {
    createCanvas(480, 480);
    background(0, 0, 0);    
    
}


function draw() {
    
    turtle = makeTurtle(width / 2 + 50, height + 50)
    //gradient color
    var g = map(mouseY, 0, height, 255, 0);
    var r = map(mouseY, 0, width, 10, 155);
    var col = color(r, g, 200);

    turtle.setColor(col);
    turtle.setWeight(2);
    var navY = map(mouseY, 0, 480, 480, 0); // navigation Y to scale of canvas
    var navX = map(mouseX, 0, 480, 480, 0); // navigation X to scale of canvas

    // not visible, sets distance between blobs
    for (var i = 0; i < 100; i++) {
        turtle.penUp();
        turtle.forward(navY / 2);
        turtle.left(90.4);
        turtle.forward(navX); 
        turtle.penDown(); 

    // blobs along path of previous for loop
        for (var j = 0; j < 100; j++){
        	turtle.setWeight(0.3);
        	turtle.right(200.5);
        	turtle.forward(100);
        	turtle.right(19.1);
        }

    }
 
}

/////////////////////////////////////////////

function turtleLeft(d){this.angle-=d;}function turtleRight(d){this.angle+=d;}
function turtleForward(p){var rad=radians(this.angle);var newx=this.x+cos(rad)*p;
var newy=this.y+sin(rad)*p;this.goto(newx,newy);}function turtleBack(p){
this.forward(-p);}function turtlePenDown(){this.penIsDown=true;}
function turtlePenUp(){this.penIsDown = false;}function turtleGoTo(x,y){
if(this.penIsDown){stroke(this.color);strokeWeight(this.weight);
line(this.x,this.y,x,y);}this.x = x;this.y = y;}function turtleDistTo(x,y){
return sqrt(sq(this.x-x)+sq(this.y-y));}function turtleAngleTo(x,y){
var absAngle=degrees(atan2(y-this.y,x-this.x));
var angle=((absAngle-this.angle)+360)%360.0;return angle;}
function turtleTurnToward(x,y,d){var angle = this.angleTo(x,y);if(angle< 180){
this.angle+=d;}else{this.angle-=d;}}function turtleSetColor(c){this.color=c;}
function turtleSetWeight(w){this.weight=w;}function turtleFace(angle){
this.angle = angle;}function makeTurtle(tx,ty){var turtle={x:tx,y:ty,
angle:0.0,penIsDown:true,color:color(128),weight:1,left:turtleLeft,
right:turtleRight,forward:turtleForward, back:turtleBack,penDown:turtlePenDown,
penUp:turtlePenUp,goto:turtleGoTo, angleto:turtleAngleTo,
turnToward:turtleTurnToward,distanceTo:turtleDistTo, angleTo:turtleAngleTo,
setColor:turtleSetColor, setWeight:turtleSetWeight,face:turtleFace};
return turtle;}

For this project I wanted to use turtle graphics to create something that allows the user to draw something that looks 3d, made up of the 2d patterns. I tested out a lot of different variations and ended up liking this one the most (the slower you move the mouse, the smoother the gradient). Overall I had a lot of fun with this, I’m definitely a lot more comfortable with turtle graphics now.

sketch

// Sophie Chen
// Section C
// sophiec@andrew.cmu.edu
// generative landscape

var terrainSpeed = 0.0004;
var terrainDetail = 0.005;
var clouds = [];

function setup() {
    createCanvas(480, 250);
      // create an initial clouds
    for (var i = 0; i < 10; i++){
        var rx = random(width);
        clouds[i] = makeClouds(rx);
    }
    
}

function draw() {
    background(150, 200, 200, 10);
    sun();
    terrain();
    terrain2();
    terrain3();
    terrain4();
    terrain5();
    

    updateClouds();
    removeClouds();
    addRandomClouds(); 

}

function sun(){
    noStroke();
    fill(255, 160, 110);
    ellipse(350, 230, 220, 220);

}
function updateClouds(){
    // Update the cloud positions
    for (var i = 0; i < clouds.length; i++){
        clouds[i].move();
        clouds[i].display();
    }
}


function removeClouds(){
    var cloudsToKeep = [];
    for (var i = 0; i < clouds.length; i++){
        if (clouds[i].x + clouds[i].breadth > 0) {
            cloudsToKeep.push(clouds[i]);
        }
    }
    clouds = cloudsToKeep;
}


function addRandomClouds() {
    // half half probability of new cloud
    var newCloudLikelihood = 0.005; 
    if (random(0,1) < newCloudLikelihood) {
        clouds.push(makeClouds(width));
    }
}


// move cloud positions
function cloudMove() {
    this.x += this.speed;
}
    

// draw clouds
function cloudDisplay() {
    var bHeight = this.x; 
    fill(255, 255, 255); 
    noStroke(); 
    push();
    translate(this.x, this.y);
    ellipse(0, 0, this.diam, this.diam);
    ellipse(0 - 20, 0 + 5, this.diam / 1.3, this.diam / 1.3);
    ellipse(0 + 15, 0, this.diam, this.diam);

    pop();
}

function makeClouds(birthLocationX) {
    var cloud = {x: birthLocationX,
    	        y: random(0, height / 2),
    	        diam: random(30, 60),
                breadth: 50,
                speed: -1.0,
                move: cloudMove,
                display: cloudDisplay}
    return cloud;
}

// noise terrains

function terrain(){
    noFill(); 
    stroke(0); 
    beginShape();
    for (var x = 0; x < width; x++) {
        var t = (x * terrainDetail) + (millis() * terrainSpeed);
        var y = map(noise(t - 0.05), 0, 1, 0, height);
        vertex(x, y); 
    }
    endShape();   
}

function terrain2(){
    noFill(); 
    stroke(0);
    beginShape(); 
    for (var x = 0; x < width; x++) {
        var t = (x * terrainDetail) + (millis() * terrainSpeed);
        var y = map(noise(t - 0.1), 0, 1, 0, height);
        vertex(x, y + 5); 
    }
    endShape();
}

function terrain3(){
    noFill(); 
    stroke(0);
    beginShape(); 
    for (var x = 0; x < width; x++) {
        var t = (x * terrainDetail) + (millis() * terrainSpeed);
        var y = map(noise(t - 0.15), 0, 1, 0, height);
        vertex(x, y + 10); 
    }
    endShape();
}

function terrain4(){
    noFill(); 
    stroke(0);
    beginShape(); 
    for (var x = 0; x < width; x++) {
        var t = (x * terrainDetail) + (millis() * terrainSpeed);
        var y = map(noise(t - 0.2), 0, 1, 0, height);
        vertex(x, y + 15); 
    }
    endShape();
}

function terrain5(){
    noFill();
    stroke(0);
    beginShape(); 
    for (var x = 0; x < width; x++) {
        var t = (x * terrainDetail) + (millis() * terrainSpeed);
        var y = map(noise(t - .25), 0, 1, 0, height);
        vertex(x, y + 20); 
    }
    endShape();
}

I’m glad that I got to play more with noise in this project. I tried to create a mix of 2d and 3d to create more contrast and depth since everything is going in the same direction. My favorite part is the interaction between the terrain and the sun, overall I think it turned out better than I expected.

Yael Braha

Originally from Rome, Italy, Yael Braha is a creative director, designer, film maker, educator and fine artist. She is currently working as a multimedia director at the Moment Factory. Braha has worked on many projects across the country and across the world, ranging from intimate immersive experiences to projection mapping large-scale shows. One of her projects that caught my attention is Kontinuum, a 3-floor underground multimedia interactive installation that’s built in an actual subway station under construction. Created for the 150th anniversary of Canada’s confederation, this installation combines lighting, video projections, and music to create an interactive experience that scans the viewers in 3D. Throughout the installation, as viewers travel through the environment, they have choices of which path to take and ends with viewers watching their own 3D image aboard the “invisible train”.

Video documenting Kontinuum

What really stood out to me about this project is its ability to be instantly personal to every viewer as they can literally see themselves be a part of it. Braha strives to push the boundaries of art and technology through generative animations, coding, and kinetic sculptures, and I think she successfully did that with this project.

Yael Braha’s website: http://www.yaelbraha.com/

sketch

//Sophie Chen
//Section C
//sophiec@andrew.cmu.edu
//project-09

var underlyingImage;
var p = 70; // text starting size

function preload() {
    var myImageURL = "https://i.imgur.com/jr1cT80.jpg";
    underlyingImage = loadImage(myImageURL);
}

function setup() {
    createCanvas(400, 600);
    background(0);
    underlyingImage.loadPixels();
    frameRate(10);
}

function draw() {
    var coordX = random(width); 
    var coordY = random(height);
    var x = constrain(floor(coordX), 0, width-1);
    var y = constrain(floor(coordY), 0, height-1);
    var color = underlyingImage.get(x, y);
    noStroke();
    p -= 0.2; // each text's size decreases by 0.2
    textSize(p);
    fill(color);
    text('lil', coordX, coordY);

    // stop decreasing text size once it reaches 20
    if (p < 20){ 
    	p = 20;
    }
   
    
}

I enjoyed this project a lot, I used this picture of my friend Sabrina. The word “lil” is what forms the image- I made it start pretty big to block out the general color scheme, and as the image continues to draw it gets smaller and smaller so the image is more detailed, which I personally found more interesting/enjoyable to watch than when I had the text stay the same size the entire time.

original photo

Neil Mendoza’s Eyeo Festival lecture

For this week I looked at Alex Kaplan’s looking outwards post about Neil Mendoza. Neil Mendoza is an artist who has an MA in math and computer science from Oxford University and MFA in design media art from UCLA. One of his projects that particularly stood out is “Hamster Powered Hamster Drawing Machine”, where he built a contraption that, when a hamster runs on its wheel, draws a picture of a running hamster. I really resonated with Alex’s comment about how Mendoza opened her mind to the different possibilities of programming and code outside of the digital world. So far, I’ve been looking at and thinking about code in a very digital sense. Even when it is interactive, I think about it as the user vs. digital interactions, whereas Mendoza’s work integrates code to create physical and analog pieces. This opens a ton of new possibilities, especially for artists, to create unique and meaningful work.

Video of Hamster Drawing Machine

Drawing in the Air – Mimi Son & Elliot Woods

In this lecture, Mimi Son (South Korea) and Elliot Woods (UK) talk about their ongoing work between the realms of material and immaterial, creating visual objects that explore the unpredictable attributes of things when they interact with technology. Son and Woods founded Kimchi and Chips, an experimental art studio based in Seoul that combines the disciplines of code form, material, concept, and mechanism. Kimchi and Chips Studio creates installations and dialogues across the globe. They play with material, space and light, while developing new patterns and theories. Son and Woods are very interested in manipulating physical materials and spaces that already exist through technology. Through combining the physical materials with the immaterial (things that are not traditionally “material” – physical, visible, tangible, etc) like drawing, space, and motion, they explore and investigate new technologies, and evaluate the relationship between knowledge of reality and experience. For instance for one of their projects, they projected and mapped light to bamboo trees and created visual forms within the tree. The light triggered the growth of the trees over time as well – starting a dialogue between the trees and people. I find their work to be very thoughtful and fruitful, ultimately creating meaningful and purposeful work.

Lecture Video