FacePlace explores the intersection of real and virtual worlds, through the examination, digital representation, and sonification of faces. Utilizing FaceOSC, interfaced through Max, we were able to capture single frames, or a picture, of guests’ faces. Although the goals of this project were mostly technical, the experience has proven to be quite thought provoking. The idea of “scanning” your face, only for it to exist in a virtual world leaves the installation’s concept very open ended.

These frames were sent to exist in a virtual world, powered by Max, and experienced in an HTC Vive. Simultaneously, the raw data received over OSC from FaceOSC was analyzed in Max, and then scaled to be appropriately used to control multiple parameters of processors.

Adrian worked on producing the VR space, creating the virtual world, and handling technical issues surrounding the VR delivery, communication between devices, as well as over-all user experience design.

Joey worked hand in hand with Adrian on user experience design, as well as splicing data from FaceOSC (supplying Adrian with matrices over jit.net.send objects), and utilizing this data to create a unique sonic environment for each face, using custom sound files. In order to process these sound files into the virtual world, the HOA library (RIP)  was used for spatialization.

First tests of a virtual world in the Vive, powered by Max.

Testing the VR world, using Seymour as our test subject.

First playtests of the installation.

Experience photos to come.

Git Repository

For my first project, I utilized a Microsoft Kinect to gather movement information on a person, within a space, and apply the tracking information to parameters on a granular synthesis. At the end of the pipeline, the resulting sound was then processed into a Lissajous pattern and project onto a wall behind the performance space.

To farm the information from the Kinect, I used Processing, an open-sourced graphical library, which was programmed to collect data from the Kinect, parse out only the left hand, right hand, and head X-Y Coordinates. Then, using OSC I sent this bundle of coordinates to Max MSP.

Using the UDPRECEIVE object, I received data from Processing, containing coordinates.

Next, I scaled these numbers to be useful by the granular synthesis parameters.

Scaling the data which I am receiving from the Kinect, to be useful for granular synthesis parameters.

Originally, I started with a rather bland audio file to be processed, however, I realized soon that I would need a more complex sound file to really accentuate the differences in the granular synthesis. To do this, I used a file of wind chimes, which proved to be much more textual and rewarding.

Below is a video of the piece in work, as well as the max patch and processing files.

IMG_1881

https://gist.github.com/jsantillo/9441d9acfa79fbc7548cc4aeaf5d0150.js