Due to my frustration with my output in this class, art in general, and life, I am definitely behind schedule with my final project. Ultimately, I’ve decided to switch directions and work on something that may be less interesting as a capture method, but is more meaningful to me personally (and will hopefully be more meaningful to the viewer as well.)

Over the past couple weeks, I had my roommate Katy pose several times for Tae Kwon Do photogrammetry. We tried it with 5 people (the entire isolation pod) taking photos from different angles, but this led to background noise that reduced Katy herself to a sad blob. This is where I was at last Thursday, from which point I’ve worked on three further plans.

Plan A: A few days later we tried again with two photographers, which a lot better but still not perfect. The model of Katy’s body lacked detail and the nunchuks she was holding never rendered correctly. Even using Metashape on her gaming GPU with 16GB of RAM, it did not have enough memory to attempt a higher-quality mesh. I’m considering attempting to model in the nunchucks myself using Maya.

Plan B: I tried extracting frames from a 20-second video of Katy holding a kick (I had to learn FFMPEG to do this, and command line tools are scary) which output over a thousand high-quality images. Metashape is thought long and hard about these, but it cut off Katy’s head. Overall, this process has been frustrating because I’ve made decent photogrammetry of my own face before, so I don’t know what I’m doing wrong this time. Maybe the lighting was optimal in the STUDIO but not here, or maybe Katy was shifting her balance slightly which can’t really be helped.

Plan C: The content I am interested in working with is my old family video and photos. I want to create a virtual space representing what my little brother’s room would have looked like in early 2008. I’ve been experimenting with the Colab notebook for 3D photo inpainting, which produces awesome images, but the actual 3D objects it produces are fairly useless height maps. Therefore, the space will incorporate two techniques. First, I will ask my parents to take photogrammetry-style pictures of my brother’s old stuffed animal dog. No matter how fucked up the model is, I will use it. In fact, it’ll be better if it’s a complete mess. I’ll also incorporate illustration by attempting to redraw the rug he had, which had train track graphics on it, and the rest of his stuffed animals. Overall, I hope this combination of 3D and 2D elements would make for an interesting virtual space.

 

Making slow but steady progress with my project. Have roughed in the body tracking system to allow for feedback created by movements to reveal another image on top of the background.

Next steps:

• Refining edges of the feedback system with the body tracking
• Building a playback system to pipe in ip cams
• Calibrating projector output with kinect input.

I have continued to refine my mapmaking process and am  pleased with the 2D results in getting everything to run very quickly with a larger dataset, more like what I will have in performance.

However, I have been trying to think of a more easily interactive way to view the content that is created during the show. Currently, I am making a wordpress page for the performance that populates the images into a 3D spinning globe plugin. While this plugin is a good starting point, I am trying to get inside of its three.js  components in order to customize it further to my visual needs. Which would be repeated images, and more images in the grid pattern of the globe, and that it would already rotate on its own (are at least some initial ideas I have). Below is a video of a work in progress of the 3D gallery element.

I am mostly on track with my project. I managed to make a Colab notebook based on Detectron2’s tutorial notebook which has:

• A widget text box that takes in a URL
• Uses youtube-dl to download the video from the URL
• Uses Detectron2 on (for now) one of the frames
• Outputs the mask based on the images (as a numpy array which can be used with openCV)

What I need to do next is:

• Differentiate what segmentation info I care about rather than all info at once
• Use the mask to segment my photo
• Collage the output photos
• Create logic to protect against bad input
• Create a front end interface

I am still deciding on what I want the output collages style to be.

 

For my final project, I’m going to be making a short 360 video set in my grandparents’ home of 50 years, featuring mostly still photos (no animation) and my grandparents’ voices. Right off the bat, I’ll say: I am behind where I would like to be, because it took me so long to decide on a narrow topic. I still think I can finish it, but without as much time to polish as I’d like.

I decided to focus on stories from Christmastime, specifically of adapting family traditions for unusual circumstances. There are a few reasons for this. First, a hugely disproportionate number of our family photos and photos in that house are of Christmas celebrations. So I can really fill the space with these images, from all angles. Second, it’s a theme we have multiple interesting stories for, and one that feels very relevant to our current situation. It may not be Christmas, but families are adapting their rituals and traditions for the quarantine, often in ways that resemble the stories I’ve collected.

Once I landed on this theme, I set several things in motion that are currently still underway: I told my grandparents about this topic and arranged a phone call with them in which I will record their voices for the project, and I also reached out to my mom and aunt (who are in possession of all our old physical and digital photo albums) for ALL the Christmas photos they have. I should be getting those in late this week, and doing the editing early next week.

In the meantime, enjoy the following images of my family at Christmas, and recordings of my grandparents telling some holiday stories (recorded last semester for a different project, but I may just end up using them because the quality is pretty good).

I won’t likely be able to use all of the photos I’m going to be sent, but I bet I can fit a lot in. The film will be “set” in the living room, where all of these photos were taken.

I got face tracking working with Zoom!

It requires at least two computers: one with a camera that streams video through Zoom to the second, which processes the video and runs the face tracking software, then send the video back to the first.

Here’s what happens on the computer running the capture:

1. The section of the screen with the speaker’s video from zoom is clipped through OBS.
2. OBS creates a virtual cam with this section of the screen.
3. openFrameworks uses the virtual camera stream from OBS as its input and runs a face tracking library to detect faces and draw over them.
4. A second OBS instance captures the openFrameworks window and creates a second virtual camera, which is used as the input for Zoom.

This is what results for the person on the laptop:

There’s definitely some latency in the system, but it appears as if most of it is through Zoom and unavoidable. The virtual cameras and openFrameworks program have nearly no lag.

For multiple inputs, it becomes a bit more tricky. I tried running the face detection program on Zoom’s grid view of participants but found it to struggle to find more than one face at a time. The issue didn’t appear to be related to the size of the videos, as enlarging the capture area didn’t have an effect. I think it has something to do with the multiple “windows” with black bars between; the classifier likely wasn’t trained on video input with this format.

The work around I found was to create multiple OBS instances and virtual cameras, so each is recording just the section of screen with the participant’s video. Then in openFrameworks I run the face tracker on each of these video streams individually, creating multiple outputs. The limitation of this method is the number of virtual cameras that can be created; the OBS plugin currently only supports four, which means the game will be four players max.