This project was an exploration of how the Kinect might be used to map pre-rendered and generative audio-responsive projections onto the faces of instruments.
The patch uses adjustable maximum and minimum Kinect depth map thresholds to isolate the desired object/projection surface, and subsequently uses the created depth map as a mask. This forces the video content to show through only in the shape of the isolated surface.
The patch is less precise on instruments which expose a large part of the body, as, for example, legs tend to inhabit similar depth thresholds as the face of a guitar; it is better suited to instruments with larger faces that obscure most of the body, such as cellos and upright basses.
In attempting to map to the surface of a guitar, I also toyed around with other uses for the patch, which include this animated depth shadow, which places the video mapped shadow of the performer on the wall, creating the potential for visual duets between any number of performers and mediated versions of their bodies.
I plan to continue exploring how to make this patch more precise on a variety of instruments, possibly by pairing this existing process with computer vision, motion tracking, and/or IR sensor elements.
For my project 1 I decided to try and make an instrument out of my computer. I separated out the keys into distinct regions and assigned them midi values based on where they were on the keyboard. I then used these note values in different modes to produce different sounds. I also included a boomerang effect that allows the user to record a short piece of audio and then the patch loops it and repeatedly plays it. I created ten drum sound effects by filtering noise in different ways. The main instrument portion is a square wave filtered in a similar way to make the note sound less harsh. The last mode is a saw tooth tremolo that repeated plays the same not so long as it is held. The launch pad is polyphonic and can export the sound in the loop buffer.
A short example piece that has been layered three times
For project 1, I used a Kinect to control the motion of a particle system using my hand. I am very interested in different applications of motion tracking and I think this was a good introduction to help me learn how the Kinect works. Here is a download link to a video showing my project in real time: IMG_1479
I used this tutorial to help me create particles from an image that I would then control using input from the Kinect. To control the Kinect I used output from the object dp.kinect2. This took me a while to set up initially. I wanted to have the system use real-time image input from the Kinect as the image – that did not end up working quite like I wanted so I stuck to using one preset image.
This patch analyzes audio in three ways and represents the information through a LED light pattern.
The 30 LED lights are grouped into three subgroups, where there’s an inner layer consist of two lights, a middle layer consist of a ring of 10 lights, and an outer layer of 18 lights. Changes of colors or brightness happen from the inner ring to the outer ring, so that the light propagates outwards.
The audio amplitude controls the brightness(saturation) of the colors. The ratio of lower frequency to higher frequency controls a color picker, which determines the RGB values. Then the values are being sent to the three layers with different amount of delay.
For assignment 4, I decided to utilize the fft object and create an audio effect patch that would mimic the sound of recent favorite music genre ‘Vaporwave’.
I created a noise reduction subpatch. Furthermore, a degrading fft subpatch is also used and linked to the output, playing along with the other signal. After these two subpatches, the audio signal is then run through the original patch where it is stretched out in real time (slowed down). This is done by using the delay effect.
I also added another simple visual presentation, that is very similar to the one we made in class. A japanese city pop song was used in demonstration, to achieve that ‘vaporwave aesthetic’.
Another demo: https://soundcloud.com/thewx/assignment4-demo/s-K8iMc
This assignment had us look at signal processing utilizing the pfft~ object.
Personally, I wanted to do something interesting with the amplitude and phase data that the pfft~ object provides that we had not yet tried. The end result was a fairly straight forward use of signal information to create a reactive video — similar to what we achieved in class — in which the amplitude and phase were used as an external effect on a noise matrix.
I chose to use noise because I found it easier to produce a particle effect using the draw points in the jit,gl.mesh object.
The patch takes the amplitude and phase data from the incoming audio — in this case, from a microphone — and captures each as a number value (using snapshot~ rather than poltocar~ to convert the signal information after some minor processing). The number value is then used as a set of parameters defining the location of an attracting force on the particles — causing them to moving around the screen.
I also added a more extreme set of attraction forces which use the amplitude and phase information to govern how strong the particles are attracted or rejected to the center of the video window. When turned on, the particles become more erratic due to the constantly changing values which limits its applications — but I like it as an effect to instantly intensify the drama of the visual effect.
I am interested in developing this patch further with a set of filters and gates to create a combination audio/visual instrument. I would also like to refine the way in which the particles are acted on by different forces to create a more fine-tuned reactive effect.
A video of the patch in action (sorry about the clipping):
In this project, I connect a multislider object to a pfft instance to allow for real-time volume adjustment across the frequency spectrum. The multislider updates a matrix which is read via the jit.peek object inside of a pfft subpatch. The subpatch reads the appropriate value of this matrix for the current bucket, and adjusts the amplitude accordingly. This amplitude is written into a separate matrix, which itself is read by the main patch to create a visualization of the amplitude across the frequency spectrum.
At first, the multislider had as many sliders as buckets. However, this was too cumbersome to easily manipulate, and so I reduced the number of sliders, having one slider control the equalization for multiple buckets. At first I divided these equally, but this lead to the problem of the first few buckets controlling the majority of the sound, and the last few controlling none. This stems from the fact that humans are better at distinguishing low frequencies from each other. Approximating the psychoacoustic curve from class as a logarithmic curve, I assigned volume sliders to buckets based on the logarithm of the bucket. After doing this, I was happy with what portion of the frequency spectrum was controlled by each slider.
(Also interesting: to visualize the frequency, I took the logarithm of the absolute value of the amplitude. In the graph, the points you see are actually mostly negative – this means the original amplitudes were less than one. I took the logarithm to take away peaks – the lowest frequencies always registered as way louder and so kinda wrecked the visualization.)
This is the code for the subpatch running inside pfft.
For this assignment I used two Pokemon models to represent the frequency spectrum. Larvitar is displayed when the sound frequency is lower, and Pikachu is shown when the frequency is higher. The scale of the models vary based on the amplitude for each frequency. In the video, the audio frequency changes from 1000Hz to 3000Hz, we can clearly see a greater number of Larvitars at the beginning, and Pikachus gradually take over the space toward the end of the video.
For this assignment, I created my take on a spectrum visualizer using jit.poke~. The visualizer works by doing spectral analysis on a number of different audio samples and writes the data into either the red, green, or blue channel of a matrix corresponding to the frequency bins. The result is the visualization of 3 audio signals simultaneously. I added a rudimentary spectral degrader to demonstrate how different effects are displayed on the visualizer. The audio used in the examples are short loops that I wrote. (My favorite is the 8-bit lead synth on the blue channel due to the pitch bending.)
I use fft to build a vocoder controlled by three keysliders. Here’s a quick demo of how it works! In the demo I played a clip from “I’m sitting in a room” at different speeds plus at different directions.