For this project, I continued my work on the first project and added a tap for bpm and pose recognition using machine learning and the leap motion controller.
I kept the same overall layout with a video feed being stripped into separate RGB colorplanes and then moving them against each other but instead of having a single looping video I created a playlist of videos which can be switched by making a fist. I also altered the playback speed of the video using the position of the right palm over the sensor.
Instead of using the problematic beat detection object from the first version, I instead built a simple tap for bpm. I did this through a timer and some zl functions.
If I were to continue this further I would look into more interesting parameters to tweak as well as finding some ways to add some more visual diversity.
For the final project I decided to further explore the connection between motion and sound. I incorporated data from the Myo armband into a music synthesizer that used several techniques I have learned from this class.
The synthesizer is composed of two main parts: the motion data reading section and the music control section. I used an online myo-osc communication application (https://github.com/samyk/myo-osc) and udp messaging to read the armband data. I am able to obtain normalized quaternion metrics as well as several gesture readings. These data laid a solid foundation for a stable translation from motion to sound.
I selected pitch, playback speed, timbre and reverberation as the manipulation parameters. I downloaded music as separate instrument stems so that I can play with the parameters on individual track without interfering with the overall music flow. After many trials, I eventually had the following mapping relationships:
The up/down motion of the arm will change the pitch of the timbani instrument.
The left/right motion of the arm will change the playback speed of both timbani and percussion part of the music.
The fist/rest gesture will switch between piano-based and bass-based core melody.
The rotation motion of the arm will change the reverberation delay time of the piano melody.
I recorded a section of the generated music, which is shown below:
For my second project, I decided to continue using the leap motion device, but for visual purposes. I decided to create an object generator. The object’s position, size and color are all manipulated through gestures and positions of the hands. I was able to incorporate topics we learnt in class such as Machine Learning, Open GL, etc in my project as well.
Here is a short demonstration:
Like project 1, I created my main patch from scratch:
I modified the visual subpath of the leap motion help file:
This is modified patch of the machine learning sam starter and training patch:
For this past semester, I have been conducting a research project under Prof. Susan Finger to install projection systems around the IDeATe Hunt Basement to create a platform for students in the animation, game design, and intelligent environment minors to publicly display their work. Therefore, my projects for Twisted Signals revolved around creating demos for specifically the interactive projection system using Max. My first project, a virtual ball pit, was a good exercise in learning on how to use the Kinect but was not really a conceptually heavy demo. Therefore, for my second project, I wanted to make a system that would actually teach the users something.
The concept that I settled on was to make a system that allowed users to interact with the Hunt Swiss Poster collection, an extensive set of extraordinary Swiss design posters that are housed in the Hunt Library which very students know exist. Originally, I had planned on using the Kinect to allow users to “draw something” using a colored depth map that would then get processed to display the closest Swiss design poster. However, in my early protoyping, it was starting to get apparent that the interaction was not as obvious as it could be, which was leading to a weaker installation. Moreover, as I have had to borrow all of my equipment from IDeATe for every project, I ran into the issue that every Kinect and my specific computer was checked out for the time span that I needed to work on this project. Therefore, I had to pivot.
While planning the projection installation, we were hit with the news that the Kinect was no longer going to be produced. As I was forced to work without a Kinect anyway, I decided to work on creating an interesting interaction with just an RGB camera which thankfully will probably always be produced. Additionally, I realized that, although being a far more difficult path, the best possible way for users to interact with these Swiss posters was to be a literal part of them, which would mean every single poster would have to be designed uniquely. However, this direction would also result in an avenue where several students could choose to participate in this project if they are lacking in their ideas for projects.
Therefore, for my Project 2, I created two different Swiss poster exhibits as well as a very simple UI that an IDeATe staff member would use when turning on the projection system each morning. Each exhibit has an interaction display that mimics a Swiss poster design that is placed next to the original Swiss poster, some information about the poster, and some information about the project.
I decided to build off my patch from project 1 and continue on my journey of sound synthesis and instrument creation. I cleaned up the patch and made a presentation view for easier interaction and I added three new instruments. The first is a take on FM synthesis and clipping effects that resulted in a harmonized growling sound. For the second instrument, I took the same clipping effect and applied it to a long resonance filter over a click to create a kind of rounded square wave. For the last instrument I took the resonant click and filtered it in a new way to create a sound similar to a church organ.
Here is a sample loop I created using the new instruments and the drums from my old patch.
For this project, I used spectral analysis along with machine learning to create a system for chord recognition. The system works by writing FFT frequency bin amplitudes into a matrix, then taking “snapshots” of the matrix and outputting the snapshot as a list, then sending these lists to the ml.svm object for categorization. While the system could easily work with any audio source, for this demonstration I made a simple polyphonic synth using sawtooth oscillators and a MIDI controller to play chords for the system to analyze. The challenge with this project was devising a system for processing the data from the FFT matrix and molding it into a form that is usable by the SVM but still contains enough data to identify specific chord spectra. The algorithm is powerful enough to recognize, for example, the difference between a C major chord and a C minor chord, if given enough training data.
In this demonstration I show how to train the SVM and how to map new chords. At the end I show that the system is not aware that a chord played an octave higher is not recognized. This can be fixed easily by simply mapping one chord played in several octaves (for example play C major chords with roots C3, C4, and C5 as state 1, D3 D4 D5 state 2, etc.)
For this project I wanted to explore the possibility for using voice to text to control Max. After a lot of research and trial and error I found that speech processing is better suited to other programs, like Processing or Google APIs. So the project transformed in to small “performance” piece, where with a little behind the scenes magic, the user can ask to know their future.
For this project I explored the connection between movement and music, and essentially created my own theremin, which is an instrument that controls the frequency and amplitude of sounds using hand movement.
I used Leap Motion sensor to read the absolute position of my left hand along the z (vertical) axis, and the range of that data stream is translated into 8 MIDI notes from C3 to C4. The velocities of my right ring finger are normalized and then mapped onto the computer system’s volume scale, so the faster my right hand moves, the higher the volume will be.
I also added a slowly rotating noise point cloud to create some visual atmosphere. The note change will be reflected in the color change of the visualization, and volume change will alter the cloud size.
For my project I wanted to see what I could do by modifying mesh points in a 3D model. This for me was an exploration of how Max reads 3D models, and I ended up with a crashing patch that took in a model, and distorted it using a .mov file. It used the normals of the model to extrude the vertices, and each point was being modified in accordance to the video. I eventually tried to use a pfft to get it to react to sound. It created really cool textures, but unfortunately I can’t get the patch to open. Below are screenshots of what I had.
I based this off of the duck distortion patch and this one I found on the internet:
For this project I tried making reactive visuals for rave music. There are two main components, the video of a 3D model dancing and an actual 3D model jumping around in space. The video is separated into R, G and B planes which are then moved around in sync with the music. The 3D model is distorted by a jit.catch on the signal and is bounced around on beat to the incoming audio.
The beat/bpm detection was done by an external object that was rather inaccurate but still created some cool visual effects.
One thing I really wanted to add to this project that would have made it a lot more interesting is having it fade in and out of these separate visual components based on the activity of the low-end. Since rave music is largely driven by kick drums, moments in a song where the kick drum or bass is absent are generally tense and dramatic. Being able to have the visuals correspond to this moment would have been really key. I tried to start this in measuring the difference in peakamps on beat with a low pass filtered signal but couldn’t find a meaningful delta. I then tried to simply map the amplitude of the filtered signal to the alpha channels of the layers but the 3D model would respond to a change in alpha values.
Overall, I think I could greatly improve on this project by more accurately measuring beats/bpm and getting the triggering/fading working. Below is a low-res recording of the visuals as well as the pasted patch.
