Project Objective:
Make a convincing narrative MV to the song Truisms 4 Dummies by Headache using a moving depth video.
I’ve been wanting to make something out of this song for a while, and I wanted to use the noisy output of a depth camera to convey an abstract narrative of the lyrics.
Inspo:
Faith Kim (2017) —Pittonkatonk festival immersive 3D project 
I used the OAK-D Camera since it was compatible with my Mac and a Ronin S3 Mini gimbal for the most minimal setup/rig for capturing moving depth footage. Much of the work process was to get the technology to work. There weren’t too many ‘beginner-friendly’ resources out there with the OAK-D to TouchDesigner workflows but with help from A LOT of people (Nica, Alex, Kelly, Emmanuel,..) I got it to do the things. (Thank you)
Direct Grayscale Depth Video Output
Touchdesignering…. 
TD takes data from the OAK camera (via a select operator) & Reorder TOP with vertical and horizontal ramps to map positions for instancing. This data flows to a Resolution TOP for formatting, then to a Null TOP, which passes it to a Geometry COMP (driven by a circle SOP and constant material) for instancing, and finally renders it to the screen.
How it looks with the gimbal setup.
Test vid of Lilian walking the walk
After getting the camera & rig to work successfully I went to shoot some clips for the song. Since the lyrics of the song are somewhat deep but also unserious, I thought matching those tones would be convincing.
Some test footage in different spaces:
Bathroom Mirror Find — Cubic space through angles of mirror reflection..
Theory: The OAK-D camera I used utilizes stereo vision algorithms (calculates depth by determining the disparity between corresponding points in the left and right images) instead of an infrared (IR) structured light system or Time-of-Flight (ToF) which uses the calculation of how light bounces off in space. So when it is placed where the mirror and the wall meet, it is still able to recognize the reflections and outputs it as another wall. (Could this be used to make virtual 3d space of mirrored spaces? mirrored realities? underwater? water reflections?)
Final Video (Final Video Featuring my friend Lucas)