WHAT
Previously…
Person In Time_ Cooking in Concert
I want to capture these memorable last moments with my college friends in a form where I can visit later on in the years when I want to reminisce.
I decided to document a representative “group activity” which is how me and my roommates move around and work together in our small kitchen by setting up a GoPro camera and observing our moving patterns.
I then recreated a 3d version of scene so that we can view the movements more immersively. I wanted to work with depth data in this next iteration of the project, so I used a Kinect this time.
Azure Kinect
Kinect is a device designed for computer vision that has depth sensing, RGB camera, and spatial audio capabilities.
My Demo
WHO
After my first iteration capture of a moment when we cook together, I wanted to choose another moment of interaction. With the Kinect I wanted to spatially capture my interaction with my friends when we regularly meet every Saturday 7pm at our friend’s apartment to watch k-dramas together. The setting is very particular because we watch in the living room on the tv we hooked up the computer to and all four of us sit in a row on the futon.
I was especially drawn to this 3D view of the capture and want to bring it into Unity so I could add additional properties like words from our conversation and who is addressing who and so on.
HOW
Now comes my struggle…:’)
I had recorded the capture in a file called mkv, which is a format that includes both depth and color data. In order to bring it into Unity to visualize this I would need to transform each frame of data as a ply, or point clouds.
I used this Point Cloud Player tool by keijiro to display the ply files in Unity. And I managed to get the example scene working with the given files.
However, I faced a lot of trouble converting the mkv recording into a folder of ply files. Initially, it just looked like this splash of points when I opened it in Blender.
After bringing it into MeshLab and playing with the colors and angles, I do see some form of face. However, the points weirdly collapse in the middle like we are being sucked out of space.
Nevertheless I still brought it into Unity, but the points are very faint and I could not quite tell if the points above are correctly displayed below.
Next Steps…
- Find alternative methods to convert to ply files
- Try to fix my current python code
- Or, try this transformation_example on the Github (I am stuck trying to build the project using Visual Studio, so that I can actually run it)
- Bring it into Unity
- Add text objects for the conversations