Not discussed in class: hiding things in plain sight sometimes using asshat mobile phone “security”.  Any physical penetration tester could find/defeat these within minutes but a drunk person at a party would not do so well.  John Wick had a good plan.

Foley artists creating the sounds of weather.

The Arduino guide to handling debounce on switches.

A mapping of western scale notes to frequencies.

The Sound Noise Device that uses cameras to create animation and sound.

It’s easy to create sounds without speakers:

• music from steppers
• A “top ten” list of arduino music projects (what I was looking for in class)
• historical tube doorbells and someone what lost their mind
• whirring fans (I demoed these in class)

Please watch the shorter ones, I linked to a couple of > 1 hour sources for future reference.

Experimental / avant-garde sound

Mark Applebaum’s experimental instruments and scoring.

Nikoli Voinov who composed music by drawing on paper, creating animation that made sound.

Musique concrete using early technology to record and modify sound, including the original soundtrack to “Doctor Who”.  Some great examples of recording found sound and reusing it for music.

Avant-garde and futurism is a rather wide grouping, like saying “rock” or “country”:

• Experimental composition with traditional instruments including John Cage
• Luigi Russolo

Using street technology to change and create new genres of music.  Entertainment and environmental sounds can come from other contexts with the use of equipment to record, store, modify, and replay.

Turntables used to create hip hop and the 1.5 hour documentary.

The Orchestral Hit

The Amen Break

Genre conflation

This is a counter to sampling and using electronics to invent new music.  Instruments from one style of music are used to perform a style of music from a completely unrelated genre.  My favorite example is “pirate metal” or the band Orkestra Obsolete playing famous pop music.

Why is sound important?  We have binaural hearing and can point to the direction of a sound without any practice.

How do you “close your earlids” when you go to sleep?  How do children learn to speak?   Leonard Bernstein’s experiments with making sounds like a child and how common those sounds might be across cultures.

Close your eyes after following links in this section. don’t worry about the visual details and information, this is learning to understand sound and signals.

Classes of sounds (one view)

Signals / alerts — short sounds that transfer information

• automobile informational sounds How many of these sounds are intentionally generated and how many are practical sound effects that let us know we’ve clicked in our seat belt?
• walk/don’t walk beep boops
• fire department / police codes on dispatch channels
• collection that includes a game tone from Starcraft
• doorbell database and an example of the ones used in the 50s and 60s in my neighborhood.

Songs and patterns that transfer information over time

We have a history of using air raid sirens from WWI and WWII as a means to notify the population of an area of an event or condition.

• air raid siren, dual pitch  — my borough uses one of these at 9:45pm to signal curfew hours for children.  The local volunteer fire department have a different one they use to alert volunteers to a fire.
• tornado sirens used to notify a town/area of a possible tornado

Music and entertainment

• ringing phones are so unique that we map and learn our own ringtones using songs.  Golan Levin’s mobile phone concert was only possible because phones had ring tones that couldn’t be changed.
• Star Trek had one of the earliest catalogs of special effects sounds used to alert viewers of plot elements and activity.
• Professional companies that sell sound libraries.
• Orbital and Underworld are two electronic bands using dual internal tempos, one near the rate of resting heart beat the other near the rate of active (dancing) heart beat.

Psychological effects of sound

Is it genetics that cause us to respond to the sound of a crying human baby?  Can you think of an “angry” noise?  A “happy” noise? a “relaxing” noise?

Sketches, Fritzing, and 3d models

Sketches from sound-class-1 (including one I DID NOT show in class) and Fritzing a speaker to a transistor.

3d printed air raid sirens.

“The Demo”, showing off the mouse, chord keyboard, and social media.

Accessibility / Inclusion

Microsoft’s Inclusion and a PDF copy of their book.

What is accessible?

Are 30mm arcade buttons are accessible? Do they simply nterrupt or do they provide constant state?  Are the buttons convex or concave?  How high are the guards around the buttons?  If you want to use Universal Design, how do you decide how big the button should be and where it’s located?

What is wrong with the E-Stop button in A10?

• Unlit
• Recessed button “hidden” in a guard
• No signage on the wall like we have with fire extinguishers

Are controls like buttons the wrong answer?  Is better output the way to go?

tactile maps

presentation of research data on tactile map comparison

tactile graphics using “swell paper”

3d printing reference objects for the blind — what does a snowflake look like?  A butterfly?  A sailboat?

Assignment/schedule

Kinetic crit on 29 Oct.  Thursday office hours + Thursday class is a work day.

Interrupts and kinetic interaction

Interrupts as inputs to state machines
Human interruption as part of a UI
Environmental interruption as part of interaction, state machines that can self-correct or shutdown
Interrupts that limit motion — end-stop switches

Types of Input

monophonic + skill: wind instruments, percussion

polyphonic + tech: keyboards, pianos, organs, strings
anthropomorphic: respond to human condition, heart rate, blood pressure, galvanic skin, breath rate, pulse rate, visual interpretation of secondary movements: eye twitch, touching your face, blinking

Golan Levin “Opto-Isolator”

Serial Communication

SPI/I2C and complex communications protocols
How we get complex data from sensors – a lot of this is hidden in libraries
Unique IDs
Simple controls for complex output: neopixel
SparkFun’s version: Qwiic

Interrupts

Show examples of interrupt code in the environment
switches on mobiles
remote controls for the projectors
complex interrupt systems in video game controllers
rotary encoder (we’ll do a demo later in the semester)
for now, we only use digital inputs for interrupts

Code samples, show how an interrupt can be used to toggle a state by one increment compared to holding down a switch and falling through a number of states.
Note that holding down the switch means the interrupt service routine (ISR) only functions once
Compare to using delay() to sample data every so many units of time.

Use an interrupt to stop a task that takes a long time, say a long for() or while() loop, by adjusting the terminating conditions

Question: What if you were playing mp3 files or video, how would you use interrupts as part of the interface?

zip file with some example interrupts

Haptic/touch vs.  objects moving in space

touching a person vs. moving an object – touch is more personal, requires little energy.   Touch can be wrapped in a robotic device, ex: Paro (wiki) trade show demo.

Moving things typically requires an external power source, Arduino can only provide 5V at a few milliamps.

For moving things, use basic physics

• gears — FAQ on 3d printing gears
• levers — drive a lever with a solenoid, transfer short movement to long
• pulleys – drive with DC motors or steppers

For the purposes of this class we’re looking at physical representation of information over time at a small scale.  Think tabletop or handheld representations.

Some examples:

• vibration: usually done with “tactors” but also a feature of mobile devices and handheld devices.  Mobile phones can vibrate, some tools for electronics vibrate as a way of sending notification.
• thumps and pokes:  using motion from a solenoid or servo to relay information with pressure or tapping motions
• temperature: peltier pads (what we use to cool CPUs and GPUs) that can heat/cool, flowing water or air that is heated or cooled.  Electric heaters or coolers are probably too complex for this class.  Dry ice is another option but can be hazardous to work with.
• symbols: Braille terminals

Adam Savage’s DIY costume cooling vest for cosplay and a commercial alternative.

Physical representation of information over time

Using motion over time

• signal encodings of language: Morse code
• pattern recognition: what motion feels like walking? Running? Being happy or sad?
• meaning is generated by content that changes over time
• School for Poetic Computation

Coaching vs. grading

Think  about coaching, providing good feedback and encouragement to take a positive action.

Example: sports trainer that monitors your HR, BP, breathing rate, and hydration and knows your training course.  It encourages you to do better instead of punishing you for not doing enough.

Example: music “coach” that helps you learn to perform music. Watches your body and helps you correct form/posture.  Reminds you that you are always performing, even when you’re just practicing a scale or an etude.

Alice Miller’s “For Your Own Good“, a criticism arguing that we replace the pedagogy of punishment  with support for learning, using the German pedagogy that gave rise to support of fascism as one study.

Consider the size of physical control related to other controls and the context for the controls.

Is there physical feedback that you’ve used a control or do you have to look at a screen to know the effects of your control change?

Is physical feedback encoded — one buzz is a phone call, two buzzes is a text message — or is it just an alert to request that you look at a screen?

Controls can be stylistic or even skeumorphic: cars that look like airplanes vs. boxy economy cars.

We fly spacecraft with computers, there are no joysticks ala Star Wars or any other movie that uses WWII airplane controls to navigate in space.   However, on the Enterprise…

MIX MECHANICAL AND OTHER CONTROLS WHERE APPROPRIATE Mechanical controls are better for some uses, though they can’t as easily serve multiple functions. Nonmechanical controls, like touch-screen buttons, are easier to change into other controls but don’t offer the same kind of haptic feedback, making them impossible to identify without looking at them and creating questions about whether they’ve been actuated. Design interfaces with an appropriate combination that best fits the various uses and characteristics.

– Shedroff, Nathan. Make It So: Interaction Design Lessons from Science Fiction (p. 26). Rosenfeld Media. Kindle Edition.

Data filtering and cleanup

look at input over time for kinetic outputs

How do you smooth data, similar to what I showed on the whiteboard?

Reading assignment

Arudino tutotorial on simple smoothing

20 minute tutorial on smoothing analog input

Find some examples of data over time that you can interact with, not just respond to.  ex: weather forecast changes your todo list so your outdoor errands happen when it’s not raining.

Even more interesting to me is when we try to use language to express what we see, thereby translating visual cognition into an audible expression and back again. (As we are aiming to do with our projects in this course.)

This idea reminds me of a study that found that Russian speakers, who have separate words to distinguish between light and dark blue, are quicker to recognize these subtle differences than English speakers when shown two different shades, thus indicating language affecting visual perception right here on our own blue planet.

Article from the National Academy of Science

On the other end of the same cycle, Vox did an interesting piece looking at the evolution of words for color in language across different cultures, beginning almost always with just light and dark, then next to red before blue and green.

I’m curious if there is any way to actively adapt the interconnection between visual and linguistic cognition for use in interface design, or to create new connections by building a new vocabulary to map optical cues to concepts that do not have representations in the visual spectrum.