In my case, I planned to convert water movement into music. Specifically, I decided to manipulate data about a form of water that we all are familiar with: rain. By using Node.js via Max, I was able to retrieve weather data from the MetaWeather API. I chose to take in numbers from four major cities: Seoul, New York, London, and Brasilia. Even though there was a variety of interesting data, I decided to only use each cities’ current weather state from this API.

Along with retrieving weather conditions, I set up four micro servos using two Arduinos in the ceiling of our Media Lab. Also, I placed a black rectangular container filled with water, approximately three meters below these servos. The servos centered perpendicular to center points of nonexistent four sub-rectangles of the container. With this system, I used my weather data to rotate the motors of servos for dropping water into the container. By doing so, the droplets caused water movement and a webcam recognized those movements to trigger musical notes. I not only used data from a specific form of water but also implemented a system where physical movements of water will make sound signals.

Following is the process of making music in this setup. At first, I used Max to convert weather status data into integers that I assigned. These numbers will control how fast the micro servos move in order to drop water into the container. As servos finish letting go of water droplets, a webcam set to view the water surface of the container will read in changing water wave movement as the drops splash into the container. The video from the webcam is manipulated so that four sub-rectangles each correspond to different musical notes. Also, within those four matrices, five rows of matrices were once again divided so that rows from the left to right corresponds to each notes’ -2 octave up to +2 octave, respectively.

There are a few important components of this project.

• Attaching a funnel (narrow side blocked) as a carrier of water onto a servo
• Having light shine onto the water surface in the container
• Running two Arduinos in order to use four servos (one Arduino supports only two servos)
• Using a black container as it is effective for Max to recognize any movements that happen in a webcam’s frame
• Implementing variations in musical notes corresponding to water movement

With this system, a never-ending music can be made by water from rain (water) data. Compared to music that we listen on a daily basis, this water music can sound boring. However, if we think about it running for more than a few days, then one musical note created by a drop of water every five minutes is truly meaningful. Below are the screenshots of the main patches:

One of the most difficult parts of this project was building the container to hold water. I used black acrylic plates and cut sidewalls using a table saw from TechSpark. With the parts cut, I glued them using acrylic cement onto a plate that worked as a floor. Finally, I guaranteed the container to be watertight by applying silicon onto the sides of the container’s interior.

Overall, I really enjoyed how I could use a very fundamental element to make music. As much as I now understand why it is difficult to deal with water and electronics, I believe that water is a great source of inspiration for the field of music. There will be a broader spectrum to the usage of water when more people realize that water can be utilized for creative works. For that reason, I would love to continue exploring ways to incorporate water with art and music.

Link to this project’s Google Doc: https://drive.google.com/open?id=172NFpCFe8TwA5Eb65xX9gccARMKjXd-N

With this project, I wanted to study and apply the fundamentals of ambisonics and spatial audio. During my research prior to starting the project, I found other various related projects. One of those was a video of an orchestra session recorded in 360 degrees. It was interesting to experience the sound differences as I moved the scene around with my trackpad. Along with the video, another project showed a train station setting where the sound was vibrant as a train passed along the platform. Interestingly, I was more intrigued by the fact that I was able to hear a conversation in the back of where I was facing.

Based on those great ambisonics examples, I decided to create a Max project that incorporates ambisonics with the Leap Motion controller. I used both the Higher-Order Ambisonics Library and the Leapmotion for Max library.

I first gathered two types of leap motion data using the library. First, the hand/palm data. By using this data, I made Max to recognize two gestural movements: slide and pinch. When I slid my hand, I allowed my sound sources to rotate in circle that represented equally spaced loudspeakers. This is significant because I can not only let the audience hear the differences between each sample but also converge each sound to become one combined music with a faster sliding movement. Also when I pinch my hand, the sources would either get further or closer to the center of the circle. This technique will help listeners to notice the spatialization of sounds. In order to change the gestural motions, I set a certain y-axis value so that a higher point movement will correlate to slide motions and the lower to recognize pinching gestures.

The second type of data utilized was fingertip positions. This data was used to make another mode where I controlled the sound sources with my fingers. By moving the sources in this way, I had more freedom over the location of my sounds and managed what listeners would specifically hear. Moreover, I was able to research on how differing coordinate points of signals can affect the synthesis of sounds to meld together.

In order to amalgamate two modes, I made a switch to turn on the finger movements with the key “d” and hand/palm gestures with the key “f.” However, I didn’t allow both modes to happen at the same time because of the strict distinction between the two types of data and it wouldn’t help a performer to easily understand the differences in using the two modes.

For testing the ambisonics, I used a jazz piece called “What Is This Thing Called Love” by Jesper Buhl Trio and a country song named “Carolina In The Pines” by Pretty Saro. Both acquired from Cambridge Music Technology Database, these music are multitrack files that contain musical stems of a song. I specifically chose songs that have five multitracks and imported a total of ten stems to correlate with my sound source movements. Choosing different genres was consequential because I had the capability of either distinguishing or merging the two types of genre via spatialization.

Below is a screenshot of the main patch and a video in working:

Despite what I learned over the process of making this project, I notice that there are improvements that can be resulted in the future. First, any leap motion-controlled movements should be applicable to both left and right-hand users. As I was researching about the Leap Motion controller applications, I read a lot about how left-hand dominant individuals had hardships using most resources. Along with that concern, it may be useful to provide graphic visualizations of hand movements to audiences. Most listeners will be mesmerized by the spatialization of sound in performance, but having to see the action will gradually affect listeners to notice what is creating the difference compared to normalzx stereo sounds.

By integrating two types of data, any performer can use their musical stems to create ambisonic music. I figure that this can be a helpful framework for those to be creative for sound spatialization events. Furthermore, I have learned a lot about ambisonics and how it can significantly help the field of virtual reality to be more realistic. Even though this project is more applicable to creative performances, I believe that it is a near-future technology that is relevant in many technological medium.

First and foremost, I decided to Fourier transform a song called “Open” by Rhye. The reason for choosing this music was that it has tonal and rhythmic contrast as it started with a mellow melody, gradually increasing to an upbeat. Since I used the transformed matrix data for the scaling of my graphical shapes, I didn’t want certain high amplitudes to completely go off the limits of the window dimension.

In order to make a sound visualization similar to the structure of the outer space, I created a 3D sphere in the center of the window. I used jit.gl.gridshape instead of jit.gl.plato that we used in class in order to render 3D objects. For changing the color, I incorporated the color parameter function from jit.mo’s organic movement patch and used min, max, and mean values of the return values of the function. Also, I worked with the mean value of the transformed matrix data to scale the size of the sphere.

Then, I created a ring system around the planet by using the position and rotation parameter functions from the jit.mo’s help patch. I tested values in these functions to make an encircling ring around the sphere. The most important aspect of these rings is that the most inner ring corresponds to the lowest fourth of the frequency range and the following rings also relate to its section in the range. I was able to achieve this by using jit.scissors and dissecting the matrix range into four parts.

Lastly, I wanted to display meteors in the outer space. In order to accomplish this I tested various point visuals based on the transformed matrix. Instead of jit.gl.gridshape, I utilized jit.gl.plato for this illustration because I only needed points that move in certain flow.

Another significant part of this project is that it not only uses FFT for sound visualization, but also for audio processing. I used one of our lecture examples on noise gating so that I can provide a control for the user to modulate the scaling value of the sound visualization. Since noise above certain limit will be lost, this is a good way for the visualization to be within the window resolution.

By working on this project, I was very glad to make my own visualizer by using FFT. Furthermore, the final outcome made me think that it looks a lot like the expansion or the explosion of the universe.

https://drive.google.com/open?id=1BAh3ae9IVZ0IWv25_XLpeLN_j36FyX_d

To obtain a response from underwater, I had to consider an ideal location for collecting it. At first, I thought of the school’s swimming pool as the only option. However, I realized that swimming pools may have noise from in and outside of the pool. Hence, I decided to use my bathtub at home to replicate an underwater environment. By using a GoPro Hero 5 from the Hunt Library, I tested various impulses responses underwater. Few of those options were popping a balloon, splashing one of the sides of my hand, and splashing the palm. Multiple trials led me to conclude that splashing one the sides of my hand brings about the best impulse response underwater.

In order to gather a response from my body, I came to realize that my mouth is both a spacious and an accessible space. I put a recorder in my mouth after a breathe in to make my mouth more spacious. Then, I inserted a small roll of tissue in one of my nostrils to block any sounds from dispersing and left the other nostril open for my clap signal to go in through it. By allowing the input signal to transfer over to my mouth (acoustic space) through one of the nostrils, the recorder was able to collect the impulse response of a clap. I also considered popping a balloon, but the response was insignificant for applying convolution.

For one of the experimental sound convolutions, I wanted to test my comical senses further. This led to try convolving an impulse fart sound to my original signal. I looked on YouTube for various fart sounds and after multiple testings, I figured that a short and clean fart produced better convolution than dirtier farts. This allowed the original signal’s beats to be replaced by farts. As for the next experimental sound, I decided to make a more serious outcome. Due to my love for rain drop sounds, I acquired a sample from YouTube and processed it in Audacity. I used the normalization effect to make the drops more audible and the phaser effect to give it a sci-fi mood. As I was processing, I hoped this impulse response can make a rap song more dreamy and instrumental.

I chose “Bad Guy” by Billie Eilish for its hard drops and beats to convolve with the first three impulse responses. Whereas for the last response, I selected “Congratulations” by Post Malone due to its party-style rap. Since both signals had a lot of bass, I ran a high pass filter in Audacity so that gain would not exceed and produce noise when I convolve with the impulse responses.

Even though testing my mouth as an acoustic space and using a fart sound for an impulse response may sound stupid, I am very satisfied with the final outcome. Because of the fact that those not only make the listeners laugh, but also allowed me to enjoy going through the convolution processes. For the other two impulse responses, I was glad to hear the result matching what I had expected before convolving.

The size of sound files were too large to be embedded on this post so the final output sounds can accessed in the Google Drive. Link to this assignment’s Google Drive: https://drive.google.com/drive/u/1/folders/17hUPip3c3mFj_vhtDdBA8zJTC0KVkWea

Inspired by the astonishing work of art, I realized that input signals with less variations of all sound component truly show the power of the system that it goes through. In other words, if a boring instrumental music processed in a system became a very sophisticated signal, then it implies the greater quality and complexity of the signal processing system.

For this assignment, I wanted to prove that little details in a system can develop interesting differentiations to an original signal. For the sake of choosing a song that sounds simple, I decided to input the music video of “To Build a Home” by The Cinematic Orchestra.

In Max, I processed the audio and video signals separately. For the audio component, I incorporated a tapped delay line system. Compared to an ordinary tapped delay system, I considered sounds from two channels individually to bring about a gradual variation in a song. Moreover, one of the channel sounds focused only on the first half of the sample part when the other dealt with the second half. This processing resulted into an intriguing echo that translated from one side of a stereo to another. For the video component, I utilized the patches from our lecture. The video system is a combination of infinite feedback and slit scan. The resulting output led to look like, to my best description, a mosaic flip book.

I was very satisfied that a dull input became a dreamy video / audio. Also, I was glad to check my hypothesis that little details matter in a system for there to be an interesting result. However, I wonder that the video may have resulted in a better way if I added another video on top the original.

This assignment helped me think more about suitable parameters in a system to bring about better results. Since certain values completely distort a signal, I had to test different values to find the right fit for a system. Along with that, I would like to learn more later in the course about how many channels in sound signals can create a more spatial hearing experience.

My final work is a combination of processed videos and sounds. Both of those initial signals are what I actually recorded during my trip to Japan this past summer. The base video is a time-lapse video that shows my view of looking outward from a moving train. The starting audio is a voice memo that recorded sounds of a train station.

To take on my idea of using old systems, I processed my videos through a CRT (Cathode Ray Tube) TV. Inspired by Nam Jun Paik, a renowned media artist, I wanted to input HD time-lapsed videos into the TV to find out if the time-lapse component may smooth out and any color variations might happen due to the effects of picture-tubes. For the case of sounds, I tried out a new technique that I read about on the Internet called “amp slaving.” This method was used by old guitarists who wanted to take her favorite amp’s tone and make it louder. In my case, I connected four amps and didn’t give any stereo effects because I considered that my original sound already had too much noise. By using this method, I expected distortions to be added to my original signal. Below is the final video / sound that iterated a total of fifteen times.

As much as I feel accomplished by feeding signals from our era into a “retro” system and trying out a sound system approach used by past guitarists, I have learned very important lessons that I could have done to better my outcome.

• Video Processing
  • Processing a video and filming it on camera results to a decrease in the video’s size dimension. Finding a method to retain its size rather than cropping out parts will result to a finer delivery of the full content.
• Sound Processing
  • Feeding sound signals into amps repeatedly may cause noise that are difficult for easy-listening.
  • Iterating initial sound signals made by professional artists are more likely to converge to a melodic sound than sounds from our everyday situations, which already have a combination of various sound disturbance.

Overall, I am glad to share the fact that an old system still has useful values when utilized as a feedback system. Also, incorporating my own signals into an assignment allowed me to appreciate my past experiences and to provide with a hint of what I felt at that time to other people. If chances allow, I would love to further research about how other human senses such as, smell and touch, can be processed recursively through variations in time and space via diverse systems.