For this project, Alec and I decided to merge our efforts of audiovisualization and pitch correction. We realized that the pitch correction system Alec created could be used to pinpoint a user’s pitch and use that value to manipulate variables within the visualization, so we decided to make a Rock Band-style game where the player sings along to none other than Twinkle Twinkle Little Star.

My contribution to the project was primarily on the visual end. I took in the variables that Alec’s patcher gave me and represented them using jit.gl.sketch and jit.gl.text within a js object. In addition to the point cloud that expands whenever the player sings, I modified the particle system to change the hue of the particles in correspondence with the note sung by the player. At the bottom of the screen, I added a player cursor – which has its y-position determined by the note sung by the player and a fixed-length tail that shows the past sung notes – and a scrolling bar of upcoming notes in the song. I then added a score counter and a method of state-switching between gameplay and game over screens.

This Drive folder has my contributions, including all of the javascript class files,

and this Drive folder hold all the files for our project as a whole.

Here’s a gist for the visualization patcher, although it won’t be of much use without the js files:

For project 2 I wanted to take advantage of the Media Lab’s 8-channel sound system to create an immersive experience for a listener. Using the HOA Library, I generate lissajous patterns in sonic space as well as allow a user to control the exact placement of one of the three sounds.

In order to emphasize the movement of the sounds, I also have uplighting for the loudspeakers, where each sound corresponds to either red, green, or blue and the moving average amplitude of a signal coming out of a speaker dictates the color values for the lights.

The sounds that are played include sounds made from granular synthesis using parameters based on an accelerometer sent through a Raspberry Pi as well as other effects applied to audio files controlled by a Seaboard (done by Ramin Akhavijou and Rob Keller).

This Google Drive folder includes all of the Max Patches for our project.
https://drive.google.com/open?id=1WZH1nr-ARBmZOF9gPrks3_Oh1Q5mJTS8
The top-level patch that incorporates everything with a (somewhat organized) presentation view is main.maxpat. Most of my work (aside from putting together main.maxpat) is in ambisonics.maxpat, which in turn has several subpatches and sub-subpatches and so on.
ambisonics.maxpat is what receives sounds, sound position coordinates, and outputs data to speakers and lights. poly voicecontrol is where the positioning of an individual sound is handled. placement.maxpat calculates the position for a sound (using coord.maxpat) and spatialize.maxpat contains calls to the HOA Library to calculate the signals that should come out of each speaker channel. These are sent to poly light to calculate the light channel value and write it into the appropriate cell of a global matrix. The global matrix is accessed in ambisonics.maxpat to the lights.

Here’s a rough video of our project in action

In this project, I used an accelerometer to get triple axis data in order to control different parameters in music and light. For receiving the data from the accelerometer, I connected it to the Raspberry Pi (The python code for Pi and accelerometer is written hereunder). After getting data from the microcontroller, I sent the data to my computer using wifi. In order to do that, I added some python codes which connects the Pi and computer to the same network and port. Next, I converted the received data to another format which is readable for Max by using itoa. Then, I used “fromsymbole object” in order to convert the symbol to numeric data. By using unpack, I was able to get xyz data from the accelerometer to my computer. Moreover, I helped in some parts of the music patch to have an acceptable sound which interacts with light as well.

Here is the python code for accelerometer:

import time
import board
import busio
import adafruit_mma8451

Initialize I2C bus.

i2c = busio.I2C(board.SCL, board.SDA)

Initialize MMA8451 module.

sensor = adafruit_mma8451.MMA8451(i2c)

Optionally change the address if it’s not the default:

#sensor = adafruit_mma8451.MMA8451(i2c, address=0x1C)

Optionally change the range from its default of +/-4G:

#sensor.range = adafruit_mma8451.RANGE_2G # +/- 2G
#sensor.range = adafruit_mma8451.RANGE_4G # +/- 4G (default)
#sensor.range = adafruit_mma8451.RANGE_8G # +/- 8G

Optionally change the data rate from its default of 800hz:

#sensor.data_rate = adafruit_mma8451.DATARATE_800HZ # 800Hz (default)
#sensor.data_rate = adafruit_mma8451.DATARATE_400HZ # 400Hz
#sensor.data_rate = adafruit_mma8451.DATARATE_200HZ # 200Hz
#sensor.data_rate = adafruit_mma8451.DATARATE_100HZ # 100Hz
#sensor.data_rate = adafruit_mma8451.DATARATE_50HZ # 50Hz
#sensor.data_rate = adafruit_mma8451.DATARATE_12_5HZ # 12.5Hz
#sensor.data_rate = adafruit_mma8451.DATARATE_6_25HZ # 6.25Hz
#sensor.data_rate = adafruit_mma8451.DATARATE_1_56HZ # 1.56Hz

Main loop to print the acceleration and orientation every second.

while True:
x, y, z = sensor.acceleration
print(‘Acceleration: x={0:0.3f}m/s^2 y={1:0.3f}m/s^2 z={2:0.3f}m/s^2’.format(x, y, z))
orientation = sensor.orientation

Orientation is one of these values:

– PL_PUF: Portrait, up, front

– PL_PUB: Portrait, up, back

– PL_PDF: Portrait, down, front

– PL_PDB: Portrait, down, back

– PL_LRF: Landscape, right, front

– PL_LRB: Landscape, right, back

– PL_LLF: Landscape, left, front

– PL_LLB: Landscape, left, back

print(‘Orientation: ‘, end=”)
if orientation == adafruit_mma8451.PL_PUF:
print(‘Portrait, up, front’)
elif orientation == adafruit_mma8451.PL_PUB:
print(‘Portrait, up, back’)
elif orientation == adafruit_mma8451.PL_PDF:
print(‘Portrait, down, front’)
elif orientation == adafruit_mma8451.PL_PDB:
print(‘Portrait, down, back’)
elif orientation == adafruit_mma8451.PL_LRF:
print(‘Landscape, right, front’)
elif orientation == adafruit_mma8451.PL_LRB:
print(‘Landscape, right, back’)
elif orientation == adafruit_mma8451.PL_LLF:
print(‘Landscape, left, front’)
elif orientation == adafruit_mma8451.PL_LLB:
print(‘Landscape, left, back’)

And here is the code for sending data from Pi to Max using wifi:

import socket

from time import sleep

from time import time

host = ‘….’

port = 5560

def setupSocket():

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

s.connect((host, port))

return s

For Project 2 I wanted to use the Kinect in some way and expand upon my previous project of controlling sound and visuals. My original goal was to have motion generating sound with a basic human outline on the screen and including lighting from the overhead lights in the classroom. However, I ended up changing my idea, and got rid of the lighting idea, since I wanted to learn more about creating cool visuals in Max, getting inspired by some really aesthetic visualizers on YouTube.

For the project, I used the dp.kinect2 object, along with the starter patch shared in class for the Kinect to help with getting the Kinect data. I wanted to learn more about particle systems since the visuals created with them always look super cool, so I added a hand controlled particle system as a visual, with some help from a YouTube tutorial. At this point everything visual was very fluid, so I wanted something stationary in the image as well, so I used jit.gl.mesh and jit.gl.gridshape to create a stationary shape that makes it seem like you’re in a spiral.

For the sounds, I wanted them to be simple, to contrast the visuals, but also controllable. I ended up having both hands controlling the frequencies of two different sounds, each going to a different channel. I mapped the coordinates of the hands to reasonable frequencies, and fiddled around in order to have controlling the pitch on each hand be pretty reasonable. I played around with using the head to create a tremolo effect, but I didn’t like the sound created, so I scrapped it.

Having done this, I wanted to add more to the visuals, so I had the colors of the particle system and the color of the shape change with the sound. I had different components of the sound controlling the RGB values of the particle system, and had the same components plus the position of the head control the color of the shape.

Here’s a video of how it works:

And here’s the patch: