3D audio visualization with opengl

For this project I made an audio visualizer that manipulates 3D models. In this case, when the signal goes above a certain threshold, the model is stretched in a new random direction. Additionally, using an fft, the signal produces a color based on the loudest detected amplitude, with red being mapped to low frequencies, blue to mid and green to high. Here is the max patch that accomplishes this:

<pre><code>
----------begin_max5_patcher----------
3692.3ocyc00iqhaF95Yk1+CnndWmcJFrMPunZWscqzdc6Es5npQjDmLbFBv
BjyG6pd9sWa+ZRLL.wPvYxEmSFCNfed7qe86G1N+w2+cOrZc9WXUqb9qNev4
gG9C9UdPdMwUdn4BOr5P7W1jFWIq3pL1myW+wUOptWM6K0xq+wj5m9bR117O
67iYwGXNUuDukcpdEw0adIIa+ykrM0vaDSeh7niGk9j6iNHLV7gm2StN+2lu
TU8WSYxm9omS1wC4GqSY0xViWykS1JqFuk8C9dmpLTy5uVvf23p0wY6W8H+w
c9cvefIYMOOj7h+uu+6Dex+3wqlVJbXUa9gcGSSq1TxXYmpwt7rZAOIq0OUl
DmZ.W4EH4JWOIW4OItB0CWgiFlqFhib0QPUxuKqOh2VZ07YkJ1pgt30OIk8I
VYURdld64gUwEEZW+A8ujfk+Xt7YE734qkjAWy670JYeJo4QfOe43RNUVy4w
ik.67EJdk1SJeKqL6Xx4NeneuooctGHDDScCApmH+vkn0EvkQ1mlu4U1VcZh
yrErrjrhRVEKqNtV0FOe+srcwGSqetEe59T+UXW7F1ve89EndX09xjs4YhFR
alWb8lW4GbPRPgZiIYUxhK56qyEg37yP2shC1iUqiKEcLqAgyScXqpyySaeu
yeyT1tZ08KRxx5Rn04Eib2xj8uL12dcN+tGF8wKuU0yGyfa+LeTd8yUwepCy
WGmlpF224M7k3rjCw0r5Dn+vy87cYYwb.+BWcPdZZabC25S8cqsbw6MrOmrs
9E4Kqk3A+KjTzHXs5b291j8rp5NWrNdeUmK8VcG7qcbsZf7y0rCEobzzoFsl
7n0fVcsksuQGslaxObfOn37.RcUm+R0l3BlyO8O+4e8Wc947ySlbA8mCoCEF
ASAUnH0.XcY8Anh2pJ0U+VZpSC59czlYoaa+zHczSjHBJL3z8US+znJZ1jKz
fa0j9Tb4IN6XfOWrDQn9Wh3Ty5PBdxSLIDvbjEk47Lj4ZDXjulEmwNvpph2y
5Wb77b3N+IzUJJpXTOfQCfoVBmEihFfQa2qNxr6ukw8t8xpsMepMw+J6qKBc
6K+eL8JF3iGfrwiS1IBUb88wHcBt2WcBUrTNcxYxEoqfBB9jfqnuvaf9Bx38
Eu0Ef6hA.0462mxLi8hTJhAOlvKpZCZmlPIu2slaC.XZxaDL6UTeDlEM.woY
vYZR1flUHoDQEFfRqxOVtooonzB5zgd3lKUmjcxN7ObZBntU7kjsaYYuAxaS
pDTAXzn4xAStoiMsoSt6Z5DSa5z6tlN0zlt+h1za4DI2Q4zM4o4bKuUCr3dg
EuMQ2R4VUS9+Hnw49D1m3EgeT7WTu.LQ9WQgt9QbiL53uZmmgWyyvKxEEQEe
SeW9WEC+E+RnK9LttGAW6rxSw.cebzqRQYdQd4I+nexOBp0Y+1WI7Ua6yf6o
OGWWWlr9XMnUoUHIlnqSC5gN2u+0woJ+kNMg33NXc1OrSM+EO5ShfxsO8oM7
VTY7XwWBQCEAXBEIPkS.YohEm6vwWh21dde5yh1pH1L2znx8mcPiwFQAx.r4
CrgWDD6sqNZaAzgYiVya1wZjEG9+EGTfqID.Ec1v4kf.H2IDv93ZliuAL.1S
x.yJ3z98w.3QB35ip+cyngZm0b8myLjzQf06D2YKhzmFCZ3kid+tz73AEVP8
FdZdCawYO118ruMWxy2qgvlozUujWzkIO4mu6bWxb4MkhItNY4GKklY24oX5
tmuPtt.GEErnDFJ3NmvpXoZywOSRi.RYzEZ3IBO+LSd6XtBV7qWwrBgpYBPm
mbXAl2D4OL0ASGLPf0d+XQ1qeyopfSLZVdOKtDEJmeHjtPVgg7tDWN.64eyX
ueX1ica3LezxNAA5NWe2weOwgn0yNKa4BAyQhVJic8Lzbj6kAsqOtaGq7zvV
merJ9PQUKdsGxiR7kDFU9AhtTNOSoFnyaTy4rv3Rh6nNNhi.t.9vObgTYQuW
bb7hDfeDL5ATCsbD.9FS.qOVWmmMFRIDPPGE0jFhNIiXlH0+BJMtYB6+q78+
x18iuD1PJCb.WgCWnQ9jq0OtYyEb8J7mznPN.TyAx4yx.udgbn1TWkGhkuM5
41wv4gp+.obaEU9ahvPaffhh0VL0BjacDEkcBeaTjBqPNDrrNCAXe8qTQx4Y
.5tR5P8cC8VD1SykDuHUf8zaRSyNXqN7hqJgHjP.kqAmFkcsiubuL+M0AdCv
2Hk8PH3Ce22BgoOjsOa.ENlaqTl7Rb11T0hX14GEYe6XkC5rWwhbJtI+HrTj
7FaZRX8RQBfgEmCov0lZkfKDI4alxOEiIWnnuHVwdJRSUn3EVITZKqjWL40w
mRErjfBQ3yKZoB9hOc7TQwk7KS9hcXtKaLkB0DkyCKkwTdgyVqIBjaQp4p5o
E8NadFWC8VvXnQnUvFUEsRcWpHnPu01n1YkIdlEJYwagAU+S4.rm93WpFO1l
P.zwpUfn2RQIye9Y07EjgaQFu2DVdG91FuYTycBUl6.VEOqzf41WnSVcivWQ
9qr2Fmx9hxQHVFKMvcuYELx9.JNrej5u3HsjOid9AQv8cc7LIFFf6bQnEZDB
9dI6+5DAxjXYrzDg+cBQTD+piz.Wm0oGY547nOEmHeok4XXwwfbWpEEgWvz2
CV9VjK1KWh6WfLbA0dMjAhrTjA89hLJYau.UDBKRHb.BDKVpQIdj6KlfeEVl
XC1cA9.SjSMfI.e3uXRF36K9nHuJQXHy+4RCU7wvZ7.1jG99KEeftO4i+sg7
gq+xxGnn6A9P4Kbiytfivv1g9RN9hboZ9pfBWpzHgBe+b7cHZZSbYcdQd52L
HuZnP+Y67Vu7wHq0mpj8YhLkd5utY1hraW82DIZT9OWSnErbF3vkZkIFMAVw
h7S68AS+6AFojP3hFnDShSxLBEpO3sDLknQwPYDCfm+vvhz3ullTUaxvMXwH
hfD6DMM4JRe4y6ZiaOLegJr8xrqOB8MhnpZ00tMYi38DW90KFq4MoIEuvDam
bwkk9+nty13532rYBDUenMtj3rO3zxw3uG+ONl85SwI6z1rUxp7ZRFvbwG2l
jKthdMf8f2osgBjZk169qUo44Ec1eWq1jmwscq9YwFxenMBiXmpWUWx3hEpm
d681RQB+NaxKKa0BZWm7xj8IbplC080ubn5Tyry9SK8XYyFiumauNth2EcLK
QDGMPNnSM1kmll+YXOWH1a5481ZDGsCmhIG+Qtp6Co7PbV8EfjXjiAXUqCgy
jadspyKqpfA6+mOzsC6gU+1w3zj5uNRSs4cogUza2zjxdnpWR1UOvKRgXs6N
3lQ552QJBCg1UTs+3bW4pJyffo8gzHt.YMY9QoDzdCJi5q4LsjvnyIcOROr+
5bRYkZIKSyJ07x5Wxc7LnOo47.HZoRMy8zA1iXOQvJiqXWlHTImco15CFrZ4
fF1MiK9Mt8Nk4NZo6XPt.h7Mco71GMuDrLe6NEaRPMcU8fy.DVKQtpzZuDNx
6cYcRhVGrr1E4TckYITAgcGrkZrKxHKxnMa5Pp9hnZAXTezDXzl3VXJopN6s
5qwNaREpV6c9thj6tAlan52tcwiZsgjGZWK2det2yVVd0gjsE4b2PTsDkuKD
OMGYHsf8a2ey8JtXLRDceFfD2ohDjZIX.wSPuj8vhmmQXgNUrD34KF5PgnnQ
gUIQ6RDJnuB0WIX6.ggovaWhBAyG6CugVGGXKN+fn1gevT7oV+sBKhEagUvB
z5IPB3zKYwwfA1AKD+ae+hPwn2kfBMZ5CAk4CDAI9IHBrMtcIXUVDgB6oDjT
ULR9PBe2I.Qh3cPWAAnUxtP4hhknIO0.k5oAE8R1EJHifxj5UnMBhRgs.We.
JA1DJFo3ax5JDK9UBrRc48IP3taUvORq2RT3cFgnYhPek45vVVSu.Dzg.Xk2
fsI.CLZh3.7TgXTnTAGFbdRujEwBwHrPmIV7C0whrjEwB1Dk2h0PhdsLvXXW
W4QJLlBwUWch1JKZYzbQkdhEAxzPSTDjTYv+DDBI05IKZYvbQwLw53XhfQMH
A2ZHC1thYdlHlELYMbQQRoLjqT5RujkwxEExBbm4veDRcH7ACbZUxGrqsQ0f
csqMvH+jEmTCSyBhldIvXnFb1pjWjZ0N4QfSWBzc.P8lqvIn0PujEwhqQyMQ
l6bSjVyMQrpRCpQlvRCmLVnvTSvXoyErHRLx9t.uYhj.cjDXejbY2J7mrRAX
4e2r3WgXn0tjGRcD4CdBJKZQfZjQdhUI2zl8k.m8K9JEcPNdZWBqLNm.KHXr
UU9QMxJCTzTwoXeZRNeF2AXQVxhXwDnDMWm2gcpjdo2YjDLWjfUt8En8KiRS
IHHsgHHZKjn6BThrDJ8I2KnjPtNwRrpCythkDihOF0exgfFAmAlPVfvgtZ4D
poDbh6Pv8UB1JvDpZKD4+9yAX50wAjliHLu6.rbs8mvTA1FKFYik3DMXZpSv
gpC4IOsjX0tDJRktA0gbqUs.iXl9jn41moPlVIKhEhwQEbZ8YJsgfpCJDH21
kTlboNYpZWBiT+NXohgEJ38mDHyPghNIn5PkkrHVLx8SO7jslT8iqAVcP3nh
YkU6WvFMPiN4ItaTsG1RQenUGn0UE9.XYxgeuQ0dXKE81EKFAkI2q.IAQstz
0JXw74aTNvQSN7FHUhfIDulMOIQUxhfgZIv3BZwtsfgXGvDFEc6wB1NXASI2
drfrjPF1+1iEiV.LyTKFrHB0JXWbfVbbfPfysHsDTiraVcQQVU3B6Q0Dtjkr
HVLZpkfIu3O3ZiUmO5P1Z.+djkrHXH1ZdRoUjD3GghlQ8T6hErw4TycNcLJC
wZUzhnw2njAfmMZTapc8hVDMFoRF4Oyz.5AmnG5krHVLxOLb3LC5nJULTXYQ
X4LwXzj9dyalRpl49Tis1Wsd5676JsD.c+8jt6ukz876H8H+FR+le+nksA9+
8+svX76A
-----------end_max5_patcher-----------
</code></pre>
view raw gistfile1.txt hosted with ❤ by GitHub

But in addition to max I needed to write an opengl shader to manipulate the model. I also decided to handle lighting and color in a shader as well. Here is that shader:

<jittershader name="spikes">
<description>
Shader that manipulates the verticies of a geometry
</description>
<param name="positionX" type ="float" default="0.0" />
<param name="positionY" type ="float" default="0.0" />
<param name="rotateX" type ="float" default="0.0" />
<param name="rotateY" type ="float" default="0.0" />
<param name="extension" type ="float" default="0.0" />
<param name="red" type ="float" default="0.0" />
<param name="green" type ="float" default="0.0" />
<param name="blue" type ="float" default="0.0" />
<language name="glsl" version="1.0">
<bind param="positionX" program="vp" />
<bind param="positionY" program="vp" />
<bind param="rotateX" program="vp" />
<bind param="rotateY" program="vp" />
<bind param="extension" program="vp" />
<bind param="red" program="fp" />
<bind param="green" program="fp" />
<bind param="blue" program="fp" />
<program name="vp" type="vertex">
<![CDATA[
//
// Vertex shader of for the spikes
//
// define variables here:
// things to note: varying means the scope extends to
// both the vertex and fragment shaders
varying vec3 vNormal;
varying vec3 Position;
uniform float positionX;
uniform float positionY;
uniform float rotateX;
uniform float rotateY;
uniform float extension;
// creates a smooth effect for the spikes in a certain
// direction, eliminates some randomness
float rand(vec3 co) {
float a = 0.8;
return fract(sin(co.x/a)*sin(co.y/a)*sin(co.x/a)*sin(co.y/a)*sin(co.z/a)*sin(co.z/a));
}
void main(void)
{
vNormal = normalize(gl_Normal.xyz);
vec3 position = gl_Vertex.xyz;
vec3 newPosition = position;
vec3 n = normalize(position);
// calculates a rotation matrix to apply to the vertices
mat3 Xchange = mat3(1.0,0.0,0.0,
0.0,cos(rotateX/3.14),-sin(rotateX/3.14),
0.0,sin(rotateX/3.14),cos(rotateX/3.14));
mat3 Ychange = mat3(cos(rotateY/3.14),0.0,sin(rotateY/3.14),
0.0, 1.0, 0.0,
-sin(rotateY/3.14),0.0,cos(rotateY/3.14));
//distance from (0.0,0.0,0.0)
float dist = pow(dot(position,position),0.5);
//this is the actual spikes now
vec3 position3 = vec3(positionX, positionY, 0.0);
//position3 *= Xchange;
//position3 *= Ychange;
float adjust = extension * rand(position.xyz) * (dot(vNormal,position3)*5.0);
newPosition += vNormal * adjust;
//applies the matriz to both the vertex and normal vector
//newPosition *= Xchange;
//newPosition *= Ychange;
//vNormal *= Xchange;
//vNormal *= Ychange;
gl_Position = gl_ProjectionMatrix * gl_ModelViewMatrix * vec4(newPosition, 1.0);
Position = newPosition;
}
]]>
</program>
<program name="fp" type="fragment">
<![CDATA[
varying vec3 vNormal;
varying vec3 Position;
uniform float red;
uniform float green;
uniform float blue;
void main(void) {
// this is the default color of the model: black
gl_FragColor = vec4(0.0,0.0,0.0,1.0);
// lets add the top light!
vec3 color = vec3(red, green, blue);
vec3 lightPos = vec3(0.0,1.0,0.2);
lightPos = normalize(lightPos);
float dProd = dot(vNormal, lightPos);
if (dProd > 0.0) {
gl_FragColor += vec4(color,1.0) * dProd;
}
//now for the bottom light, which is the exact opposite of the top
vec3 opColor = vec3(1.0-red,1.0-green,1.0-blue);
vec3 lightPos2 = vec3(0.0,-1.0,-0.2);
lightPos2 = normalize(lightPos2);
float dProd2 = dot(vNormal, lightPos2);
if (dProd < 0.0) {
gl_FragColor += vec4(opColor,1.0) * dProd2;
}
}
]]>
</program>
</language>
</jittershader>
view raw gistfile1.txt hosted with ❤ by GitHub