60-461/761: Experimental Capture
CMU School of Art / IDeATe, Spring 2020 • Profs. Golan Levin & Nica Ross
Here & There is small series of scenes depicting domesticity during quarantine. The avatars in the scenes embody the hope, dream, and opportunity of other places, speeding through locales that contrast to the stillness and current constraints of home life.
banner image:
Here & There is small series of scenes depicting domesticity during quarantine. The avatars in the scenes embody the hope, dream, and opportunity of other places, speeding through locales that contrast to the stillness and current constraints of home life.
Using a Kinect Sensor, TouchDesigner (a visual programming language), and access to IP Cams I built a system that transformed a person to become a mediated avatar, that through it’s body displayed other parts of the world.
____________________________________________
This project has taken a few turns – I originally pitched a sort of digital camera obscura effect that would use the motions of my body in my space to digitally reveal other places; think body paintbrush. What I attempted to build was a system that would use layers of feedback on top of itself while still clearly being able to see each layer. I had hoped the use of a projector would aide in the layering effect allowing most of the visual content to conform to the geometry of my room. Unfortunately TouchDesigner does not support projector output on the free license.
The next iteration was to focus more on the computer vision aspect of the piece and implement hand tracking that I would be able to paint objects surfaces in my room that would reveal portals to other places. The triggering and isolation specific content proved difficult, was working on gesture recognition in TD that would trigger the effect on whether my hand was open or closed. I am confident with more time and development I still could realize this system.
This led to the current incarnation that flipped the layers. Instead of layering behind my body I opted to cover my body entirely rendering myself something entirely mediated.
Further work on this project would expand the series to stage more scenes, as well develop a more fool proof overlay of the body and limit artifact.
This project was the first time I worked with the Kinect sensor and much of the R&D was seeing what could be accomplished in TD and how best to maniuplate the sensor data. There are countless of things to explore with it and Im excited to see what else I can build with it and continue to build off this project.
Making slow but steady progress with my project. Have roughed in the body tracking system to allow for feedback created by movements to reveal another image on top of the background.
Next steps:
For the rest of the semester I’ll be working on a system in my living room, using projection, streaming Ip cams and the Kinect. Given the lack of ability to travel and venture out to public places I thought to bring the places to me by creating something akin to the camera obscura effect, difference being rather than a projection of my immediate outside surroundings it can be anywhere in the world. Using the Kinect to track my motions and TouchDesigner, my hope is to use my body as a way to navigate space and create portals to other streaming locations from the trails of my own local movement. 
This past weekend in my normal scroll of the internet I happened upon a post by Philippe Dubost (@philou.cc) who has been posting shots using light painting with a pixel stick to write phrases in Montreal.
He has also posted a video tutorial on how to create the same effect using TouchDesigner.
So I set it up and my girlfriend and I spent some time playing around the apartment drawing with phones and flashlights. What became interesting was that since it was looking for the brightest spot, a reflection on a wall would also trigger the feeback/line to continue.
We than started painting our faces with light, with led to using the flash of a camera to imprint a silhouetted reflection.
I realized that doing this was the inverse of what photograms were. Rather creating exposures by blocking light, this process was producing similar effects by reflecting light.
I first thought of the aesthetic of looking at old digital photos and how due to the advances in imaging technology and greater resolution, the old photos often appear blurry and small. So I decided to go into my facebook for some old photos. I found a photo I uploaded over 10 years ago that depicted my friends and I, with the reflection of other friends taking the photo. Taking a new photo of that picture on my monitor I was able to see a reflection of my current surroundings similar to the mirrored reflection in the original photo. Repeating this process with the new capture established more visual feedback with the original image quality as well as the reflections in my own monitor. Enter sentence about time and place and the reflections of it here 😉
Through an empty wine bottle
1) Use Slit-scanning to document a person as their body moves through frames at different rates of time, to create a depiction of a person as hyper-dimensional
2) Use the kinect to alter a person’s shadow form and/or image to document, capture, play with where the person existed in space just moments before.
3) 3d model of a person that will be compressed over a duration until the model fully degrades
Port Explorer is a typology of personal electronics utilizing macro photogrammetry to capture and model unseen spaces that we carry with us daily. The collection presented is comprised of the charging ports of people’s personal devices: USB-C, Apple Lightning, and USB-A.
Rather than a reflection of an individual, I believe these captures to be a refracted representation. Not much personal information may be derived from these images, but the use and collected grime, lint and dust found in these spaces has unique personal ties and presented as a typology they become the personality of the images.
It was my aim to capture and represent these spaces as micro-verses that the viewer could navigate and explore. Using the Bebird otoscope and compiling into Agisoft’s Metashape, I brought the resulting 3d models into TouchDesigner to create a gallery of 3d responsive models. The viewer can control and navigate each model by clicking and dragging with a 3 button mouse.
Macro Photogrammetry is quite difficult. The Bebird Otoscope has a very narrow depth of field, and I was also trying to capture cavities externally given the size of the Otoscope. Below is a picture of the set up I used, which did stabilize and to a certain extent mechanize the capture so that I can adjust and line up my next shot somewhat accurately. A more precise rig that could hold the subject and Camera, while also having full 3 axis manipulation of both, as well as controlling distance would have been the most ideal. Given the challenges of this type of capture I did scale up from phone ports to USB ports, which proved less challenging given the extra space and larger cavity.
I believe there is more to explore around this project. By understanding the limitations of my set up and my methodology, and was there more time, I would have liked to further investigate computer cables (DVI, VGA, etc.) as a typology that presents and abstracts their various topographies as micro-verses. There is a gendered component to this work that I am also interested in further exploring. Specifically investigating, colloquially termed “Male” cables that at macro scale are unique bodies and cavities that are not dis-similar from their “female” counterparts.