This assignment had us look at signal processing utilizing the pfft~ object.

Personally, I wanted to do something interesting with the amplitude and phase data that the pfft~ object provides that we had not yet tried. The end result was a fairly straight forward use of signal information to create a reactive video — similar to what we achieved in class — in which the amplitude and phase were used as an external effect on a noise matrix.

I chose to use noise because I found it easier to produce a particle effect using the draw points in the jit,gl.mesh object.

The patch takes the amplitude and phase data from the incoming audio — in this case, from a microphone — and captures each as a number value (using snapshot~ rather than poltocar~ to convert the signal information after some minor processing). The number value is then used as a set of parameters defining the location of an attracting force on the particles — causing them to moving around the screen.

I also added a more extreme set of attraction forces which use the amplitude and phase information to govern how strong the particles are attracted or rejected to the center of the video window. When turned on, the particles become more erratic due to the constantly changing values which limits its applications — but I like it as an effect to instantly intensify the drama of the visual effect.

I am interested in developing this patch further with a set of filters and gates to create a combination audio/visual instrument. I would also like to refine the way in which the particles are acted on by different forces to create a more fine-tuned reactive effect.

A video of the patch in action (sorry about the clipping):

In this project, I connect a multislider object to a pfft instance to allow for real-time volume adjustment across the frequency spectrum. The multislider updates a matrix which is read via the jit.peek object inside of a pfft subpatch. The subpatch reads the appropriate value of this matrix for the current bucket, and adjusts the amplitude accordingly. This amplitude is written into a separate matrix, which itself is read by the main patch to create a visualization of the amplitude across the frequency spectrum.

At first, the multislider had as many sliders as buckets. However, this was too cumbersome to easily manipulate, and so I reduced the number of sliders, having one slider control the equalization for multiple buckets. At first I divided these equally, but this lead to the problem of the first few buckets controlling the majority of the sound, and the last few controlling none. This stems from the fact that humans are better at distinguishing low frequencies from each other. Approximating the psychoacoustic curve from class as a logarithmic curve, I assigned volume sliders to buckets based on the logarithm of the bucket. After doing this, I was happy with what portion of the frequency spectrum was controlled by each slider.

(Also interesting: to visualize the frequency, I took the logarithm of the absolute value of the amplitude. In the graph, the points you see are actually mostly negative – this means the original amplitudes were less than one. I took the logarithm to take away peaks – the lowest frequencies always registered as way louder and so kinda wrecked the visualization.)

This is the code for the subpatch running inside pfft.

And this is the code for the main patch.

For this assignment I used two Pokemon models to represent the frequency spectrum. Larvitar is displayed when the sound frequency is lower, and Pikachu is shown when the frequency is higher. The scale of the models vary based on the amplitude for each frequency. In the video, the audio frequency changes from 1000Hz to 3000Hz, we can clearly see a greater number of Larvitars at the beginning, and Pikachus gradually take over the space toward the end of the video.

I use fft to build a vocoder controlled by three keysliders. Here’s a quick demo of how it works! In the demo I played a clip from “I’m sitting in a room” at different speeds plus at different directions.

----------begin_max5_patcher----------
2170.3oc6aszaiiiD9ryuBAg8zBmX9RulaKv.rKVf9zt6gEMVzfVl1gSKI5U
RtSmYv3e6CeIYIGKElHEmd5YZf1wjhR0W8whUwpn7ubyB+0huxp789AuO5sX
wubyhE5tTcrv1dgeN8qoYzJ8v7qE61kw7WZtTU8ixFp9a5o3PNuHiUqGM7Tm
hC0m2qoq5G2yL.vmWT668+rWcOsN8ddwtOUxRqMCf..2AV5EjDdGP9uP3ROD
Q0i5y1ajuQCHw5e51H+1GVIMmUyJ+Dqft1.Yf5R+5M2n9XoiZeNqphtqU8qY
es1Hr8rBG4DzKiSFkPv33yIDbjlPPWlPh8eEJsDoqYkGeg5WtXCqa6shh5sz
TC02ouJ9Oq6Chuqo6J9N8nZ6nOWgFhqj2WAMyeom+1LAs1EpKhDXXPMqguLq
AA9c.bgzRReo+VIWJsWCgxdP9XehQT0VI7DkaN5gdarkr7iCzRXRxcgx+EGI
oH3XVTvWkIkb9Qh4WnWjArV1JJyoZ3G9xLWTFJqoE6FkPfxETARRv34ALlYB
JZDyjqjinp8L1FOn2eA9BMgFfbmCuTVJDhzrWTvXbHddWosOi9XFup93jBXE
77Nbtz2T+eCOslKJnkONNEAzrBLFo+iwNCCFXIWRC12PqoVs2p9K7Sy36ain
24BJNkmwZ4zOPSkwaEU268O9weX0+ohUVs5exjWe0eWHjg389wR9WXq92OHo
OoI0+RqVUq9.aCmtpRbnXS0pRgH+1pB996n7sVP0HnOyKLPldXCWn5oy.xDh
8c3aMtkS3rh5OUUSqYMJUW3uvXbaTsak7UbC4bxMPQCedh4LqO3xEL0kLIwa
FP+K2Dq5iJGBU7T+dW8+eflwqeb3ArmKetU2y2NfzEk7cbI6IY+c0M.PMCKw
U5mq5+vpxNTpfa6v5cUszqOTvK1Y2SDA7TrHihnrsjFdWRa6Cm7pKKolQUyx
2Kr5E57AoVecIQXmLFGGaEYYhG1kIV2SJxA0LF8pcyWrcZ5Ydh2t5nG4cKPK
LhnbJF.0af0D0cnXrjj4bWFozL1QuagxnDf6RhUVqtteiv4e+FAAFWcgZGep
vBivCQyHOj9XphGdG2pkQ0Ql8WYaLjlGNiZ9e8nG7cSqQIZScLQGoCiFUqCl
S6d9tiuMos9BzZiYNdbqb7Lp040hsuMZ8n5afQQQAXyj8n5K58ICBzUJChXi
5iQlLMGctGGb8Rg32OLXTXr6LX3exfWfAAX2Yvn+jAeJCFZJrlaLX70iAelc
2b2TJC.5kVF.GhDFDg5xiIiwiIyaw.RE44xLLeBWsUlUoz34AZ4lUpuull94
6lGax97F34cwAfl8Aaph6.E.H75vK6n7hqNOXcTY3AbvXzPvUopr6umVIJOp
Rz9auES1sYEqWFEO9louNEwtlwpu+3KLexqEQgCMEViL5gjPfyKS84pL9FV4
jR4po28krJ4ZUpplJ87pZV1XV0fCahQ0pdW7r1V97G4l87.f1zzuvStaAvIW
ona62ts9n2WDoBIuda0g0ueIwizTNAqWABAitu.zrlOK8gipSD8cOS9H8hov
Qc9.I+9OmVq9ZqWy34zBC9F0QANxTnMyIo+V3oX3mbW9I7ObdJP5SUyQOEvn
um7T.M6OwMOEeGT8qF80r80mwSA5aTOE5kwuUdIHtrcBHbtcRv94MzziSh0F
I2EXndMcj4PLLpn5yKtGyWyrtJgroUCaGd0YdtYvlCtx9tPXNvZx.ong9Cma
d6I46na9tVA5A3mwKN+ExSKEU+84kJwgxzFjYKWxoSX0eCqplWPOc3qs413A
aGTNeydgbF2JwfXnZtEGfzSwxVfjjjnPaWlyf8IyXthuXGvGzNnIInnqkfHt
P4n4PPtnRjfwlawg50sXigIFAZaMUrE5B1hlCVHvkIVxbHIrq78jkDxEIAlC
IAcwXcTan.XfxpgfzwsIPRaqohMfq9tlpjvwtJIzPrPHIriWRUqH...QyiWR
bjq.DOD.ifnN.T0BIciGGOS.LzU.RFDfQQcAnrUhjAQjYBfAtBvfg.XLA2Af
caMUr4hGrjYvNG4hYDdNDjSrc+.kmGZBqCFQhAlWBRcBS5VSEaDWvFZTrQzl
pHhdibVGf5VSEat.swPlkmBSh0ucsQFuQpVmxRqTkjnbz3qBTgNBUyeF.pn2
bnB6GBnUxvqijgOUxfIIYXhqA1G1VpITtMubPaq2EvBibGrHDZDvBuBf0cr1
s0ba4CcJ8p4XijvnI6wuYRa183CcYyIv.m73adgO.mZMUr4Tjx4HlLj35hrg
SWzn2Dn1Ws5EWro0TwF1w0TCGMwD+vBstslJzbxG9rLA4RtXPj6loVVXVLSm
7p6F2bOypaSAun62+EVYk84qQkeN8mDZWhQK0M4EllXcyR1W3Mimn6gVldOu
lkVenzTruuFaNfI8urixhCbar.IebisnfmUjs1i4O+QSMC0plT82ROjU2mxT
kcLUjIJ+DuPgc14UL2n2c7wud2VdVl9dN+2qSS4E82UR2vO8d3n9EwXGt5Qi
R.vjP0iFCRhADy2jcA6II6cAats3.HHQM1XBBQLeCC.DzktKT6Oekk8+3z3n
E6L0SEE09yyTcDBh8hxFaDIvRZG+gZQqh0r4AyzfgW5njPC0knJ4Ho4aI80u
duRFRiwOHJnoB+F6pkVK8Ro3Nuzu94xIRdCH5txnct2Tc4+KS86R4V3Xl.ce
iU9n9cVIPlBa6uHIeZZJqnt2LXLJ.FoTqnXXLTajDjDPBr52rn.OnVF3kKJD
U6U+Va+dvH1Eyh9FvRtFhi.ZNNFEEh02lj0iQIus1y8zVo7hQJIihigVkM.h
wvyo0d2UhDjDrVOihCBhLeCgfH2VHLhAnS9B7YYr7yu0nXPBzvh..BFadJXX
X7q15UdO+5M+FlmFofC
-----------end_max5_patcher-----------

I’ve used this assignment as an opportunity to continue my explorations of the writings of Virginia Woolf as transformed by digital mediums, as well as to better understand how to use a spectral system to create audio-responsive visuals.

I began by reviewing Jesse’s original shapes/ducks/text patch and reconstructing through the components of the PFFT~ one by one in order to understand how they work together toward being written into a matrix. I subsequently created a system outside of the PFFT~ subpatch which randomly pulls a series of lines from Woolf’s novels and renders them as 3D text to a jit.gl sequence.

The only extant recording of Woolf speaking about the identities of words activates the PFFT~ process, and the resulting signals control the scale of the text. The movement of the text uses Jesse’s original Positions subpatch, but filtered through a new matrix used to control the number of lines which appear at any given time.

At the top of her recording, Woolf says, “Words…are full of echoes, memories, associations…” and I aimed to create a visual experience which reflects this statement as an interplay between her own spoken and written words.

I spent some time altering various parameters – size of the matrices, size of the text, amount of time allotted to the trail, swapping the placement and scaling matrices, etc – in order to achieve different effects. Some examples of those experiments are below.

Here’s the recording of Virginia Woolf.

Here’s the gist.