Experimental Capture • Fall 2022
CMU College of Fine Arts • Profs. Golan Levin & Nica Ross
My interest is in between natural/organic/physics movement and technological/synthesis/computational movement.
Suminagashi is Japanese-style ink painting. With ink and water spread out, it creates a figure-ground relationship and temporal relationship. Slit scanning is also about the direction of the movement and the direction of time. So I wonder what happens when two collide. How water waves move/ how pixel move. What’s the relationship? Or do I create a new relationship?
For my final project, I am really interested in making my second project (the one of the veins) interactive. I am thinking of using Touch Designer and and Infrared camera to be able to isolate and track someone’s veins as they stand in front of a screen. I am unsure if I want the final projection to be just of someone’s veins, or of their veins layered on top of their reflection.
For this project, I am hoping to construct an interactive art device that captures the clothes colors of passersby using Kinect and openFrameworks.
Using Kinect, the program should be able to detect the human bodies passing by the device. Then the color pixels of the person can be extracted and processed. The program will start off with an empty white palette; as more and more people pass by, the color pixels from their body are detected and shown on the palette for a period of time. The people interacting with the art basically fills in the palette over time.
Although people visit and share the same physical space, every one of them gets a unique experience out of the space. For example, members of a household share the living room, dining room, and kitchen, but how they interact with the communal space varies. The same goes for offices, classrooms, or any other shared space.
For my final project, I plan to build an interactive visualization of people’s gaze patterns in a shared space through Mixed Reality. Using Meta Quest Pro, I will collect the eye gazes of people who use the space. Then, after data collection, I will create an overlay visualization of these gaze data mapped on to the 3D model of the space (which will resemble Shopify’s gaze heatmap). Instead of revealing all gaze data at once, however, individual gaze patterns will be revealed only when the viewer looks at the same part, sharing gazes in a similar way to Kyle McDonald’s Sharing Faces. In other words, if you look at something in a 3D scene, and if other people in the same space happened to have looked at the same thing in the past, the system would show what else those people have looked as well.
I’m not sure what the final form of the project would look like yet. It could be an interactive experience in the final exhibition.
For my final project I want to motion capture hair. My self defined goal to capture (for now is just my) hair and movement in an actually experimental way. This is a combination of my other two projects and an attempt at pushing my experience with experimental capture. I don’t want to capture things in with the utmost degree of reality as I tried to with my hair clump scans but I also want a sort of immediate visual recognition.
At the moment I’m hoping to do a sort of photogrammetry video/ volumetric capture of my hair as I move or dance through a space and then key out anything that is not (for example) red in all of the textures. I’ve been looking into possibly using agisoft metashape to do the photogrammetry with and ffmpeg to process image textures with. To be honest, I’m not sure exactly how the technical pipeline will work.
– I’m not concerned with reality, hair simulation, etc., I am a little concerned/interested in how the subject of hair or movement may affect my exploration. I also don’t foresee the ability to be super expansive.
– Alternatively I want to stick motion capture reflective tape pieces onto my hair and use the ideate motion capture studio to get data.
For the final project, I have a couple of ideas that are coming from previous projects.
Option #1: Audio Capturing/ I am considering analyzing a specific genre of music (Black Female Hip Hop) to create a typology/amalgamation/poem capturing words, phrases sentences.
Option #2: Music Video/ using footage from the edgertronic / photomosh / etc. using audio to accompany the experiments to create a narrative.
Option#3: Vignettes: I would use the edgertronic and a DSLR to capture moments of bodies interacting to create a little narrative.
I plan on making a slurp detector. I will be basing this off of Sam Lavigne’s videogrep – but for slurps. Google Teachable Machine also has a function to train an audio model, so I plan on extracting 1 second audio clips from my previous footage to get a prototype slurp model.
For my final project, I want to extend (and finalize) the work done for my second project, which explored the UR5 as an expressive capturing device using real time communication. Because of limitations and struggles with RoboDK, networking and scheduling I didn’t have the opportunity to see this project fully realized. I learned a lot from my explorations with my second project, from understanding the basics of robot arms to learning about real-time control mechanisms.
Goal #1: I want to perfect a system to capture myself or others as a subject without a need for a cameraman or woman. This is in essence a completion of my second project, but also a refinement of the systems I was already working with. I will be working with TCP/IP control of the robot using Python.
Goal #2: I want this system to support storytelling. I think having the ability of having a robot autonomously follow you provides an interesting avenue for a creative story that can be told. I don’t want this project just to be a capturing device, but I want to get to the core of what I want to capture. I think that question still needs refinement, and part of my final project will be getting towards that point. But for Goal 2, I need to finish Goal 1.
What has been done? What needs to be done?
I was able to control the robot through TCP/IP and Python relatively well, I need to extend the system I have to support real time interaction through camera feedback. But the networking aspect of that, which is the trickiest part for me, has been figured out. So this weekend I will try my best to finalize this system (Goal 1), so I can begin hacking away at Goal 2 after Thanksgiving.
For the final project, I’ll be creating a machine that prints out the space over time. Using Kinect, each image captured over time will be saved in the database, divided up by the depth of the object. Images will be processed to only have the object contours for simplicity. The user then can press the button to print out what Kinect has been seeing since the past till the moment the button was pressed in one image with a timestamp. The prints this machine will provide will be in a form of generative collage in a sense that objects to be selected for each depth would be chosen at random among the collection.
In terms of where this project would live, I imagine this machine to be placed at a high-traffic public location like a subway station. In these public spaces, many people with diverse range of belongings travel across the same space over different time. Capturing what use to be there over different period of time in one frame and delivering it on a physical paper upon anyone’s request would give a greater perspective to the audience on how diverse of the crowd and objects we’re sharing the common space with.
Why a Waorb?
Pictures capture a likeness, but not the feeling of someone. I want a device so that I can feel the warmth of specific moments. So I can know my friends or lovers touched what I’m touching now and feel the warmth of their bodies when they’re far away. I want to be able to let others hold this same device and know they’re able to experience an intimate part of this moment too. I want a Waorb.
What will I have for final critique?
In short: a completed Waorb and documentation.
In long (Deliverables):
Where am I right now?
Constructing the final Waorb. I have the components I need and am building the Waorb. I have finished the magnetic clasp (which will be supplemented with another type of closure), and have begun constructing and installing the heating module.