For Project 2 I wanted to use the Kinect in some way and expand upon my previous project of controlling sound and visuals. My original goal was to have motion generating sound with a basic human outline on the screen and including lighting from the overhead lights in the classroom. However, I ended up changing my idea, and got rid of the lighting idea, since I wanted to learn more about creating cool visuals in Max, getting inspired by some really aesthetic visualizers on YouTube.

For the project, I used the dp.kinect2 object, along with the starter patch shared in class for the Kinect to help with getting the Kinect data. I wanted to learn more about particle systems since the visuals created with them always look super cool, so I added a hand controlled particle system as a visual, with some help from a YouTube tutorial. At this point everything visual was very fluid, so I wanted something stationary in the image as well, so I used jit.gl.mesh and jit.gl.gridshape to create a stationary shape that makes it seem like you’re in a spiral.

For the sounds, I wanted them to be simple, to contrast the visuals, but also controllable. I ended up having both hands controlling the frequencies of two different sounds, each going to a different channel. I mapped the coordinates of the hands to reasonable frequencies, and fiddled around in order to have controlling the pitch on each hand be pretty reasonable. I played around with using the head to create a tremolo effect, but I didn’t like the sound created, so I scrapped it.

Having done this, I wanted to add more to the visuals, so I had the colors of the particle system and the color of the shape change with the sound. I had different components of the sound controlling the RGB values of the particle system, and had the same components plus the position of the head control the color of the shape.

Here’s a video of how it works:

And here’s the patch:

For this project, I decided to take camera input and use different characteristics of the video to generate sound. I split the input into 4 matrices, and used different characteristics of each of the 4 matrices and mapped them to 4 different sounds. The characteristics I used included the hue, saturation, and lightness values as well as the RGB values of the images. I also wanted to add a visual component, so I used jit.sobel for edge detection on all 4 parts, with 2 of the parts remaining as the output of the edge detection, and the other 2 parts having the edge detection output blended with the video with the brightness and saturation varying based on the original video input. I then stitched the parts back together to get a cool distorted image to go along with the eerie sound that is generated.

Here is my patch:

Here is a video that shows the patch working:

I decided to experiment a bit with pfft~ and see what kind of output I could get by combining some techniques that were covered. I decided to take in 3 input sounds, with 2 of the inputs being cross-synthesized, and the third being convolved with the output. I used the examples in the pfft~ reference for help with this. The sound ended up being pretty cool. I then took the peak amplitude of the output, scaled it, and used it to control the saturation of input video from the camera. Overall, the output seemed pretty neat. I’ve posted the gists for the patches and a video of the output below:

Main patch:

fft~ subpatch:

I decided to use an impulse recording from Baker, an impulse recording from West Wing, and then two parts of different songs: Bohemian Rhapsody by Queen and the Theme Song from Schindler’s List. I decided to try to pick different genres to see what kind of result would occur. My initial recording was a part from Thunder by Imagine Dragons. The ending result was very unexpected, but it sounds cool! The order through which I did the different iterations of convolution: Schindler’s List Theme, Bohemian Rhapsody, West Wing, Baker

West Wing:

Baker:

Bohemian Rhapsody:

Schindler’s List:

Thunder (Initial):

Final Result:

The time delay video patches that we did in class kind of made me think of how it might look if someone was hallucinating, so I decided to play around with that idea. I added on random dimension, saturation, contrast, and brightness changing along with time delay and feedback to get an interesting result that I think goes along with my theme.

The video shows what the patch results in, and here is the patch below.

I decided to use Thesaurus.com to generate synonyms for words, and see how far from the initial word definition that multiple iterations would take me, essentially destroying the original meaning. I had a constraint, similar to the Google Image recursion video we saw in class. If I saw a word once, I would not go back to it, and would instead pick the next best synonym. I started with the arbitrary word “bright”, and generated synonyms for it, and so on for 40 iterations. Here is a selection of the iterations:

Fifth iteration: “flaming”

Tenth iteration: “bloom”

Twentieth iteration: “smirk”

Thirtieth iteration: “bomb”

Fortieth iteration: “bombing run”

As can be seen, by the tenth iteration, the meaning was already distorted, and it continued on from there. Here’s a link to a PowerPoint showing each iteration, if anyone wants to see the specifics!