VR Headsets and vision problems:

https://www.bbc.com/news/technology-52992675

https://www.aao.org/eye-health/tips-prevention/are-virtual-reality-headsets-safe-eyes

Sand Tables and Interaction:

Sand Noise Device – please watch

A History of Sand Tables – this is long, but if you’re interested in how to visualize things in 3D this is a good starting point.

A brief aside on visual skeuomorphism

A skeuomorph is a object derived from past objects and contains past visual cues that have no meaning in current culture.  Ex: the “phone” icon on my Android looks like the physical handsets we stopped using in the 90s.

Think of a derivative object that retains ornamental design cues (attributes) from structures that were inherent to the original. Examples include pottery embellished with imitation rivets reminiscent of similar pots made of metal and a software calendar that imitates the appearance of binding on a paper desk calendar.  Or the “save” icon that looks like a floppy drive, that a Japanese student thought looked like a drink dispenser:

“Why is the Excel save mark a vending machine? “

When you are creating visual representations, are you using your culture’s history of symbolism?

States of being, replacing sound with visuals

Visualizing sound

Visualizing music with images

ASL vs Braille

ASL replaces a word with a motion while Braille replaces each letter with a pattern with a shape similar to the letter.  Before Braille, books for the blind used large raised letters that you would trace by hand.

Early experiments with live animation and MIDI inputs with devices manipulated by the audience.

Console displays in video games are a different way to display information.  This is metadata about your status.  health, money, power, speed.

Wipeout series brought commercial racing context to a SF racing game.  Music for Wipeout XL was mostly bands who were played or performed in rave spaces.  There were in-game billboards for real products.  The visuals were designed by The Designers Republic, a well-known design firm.

Rez was a completely different video game and hard to explain.  The controls are simple, but as you play you dynamically create new visual and audio structures.

Cultural Contexts

What colors mean in US culture vs. Asiatic, Arabic, African countries

ex: white wedding dress or white funeral dress?

colors important because they show wealth: purple for royalty, gold.  There were “dyer guilds” that kept secret their methods for creating a dye.

Color of money — the US is one of the few countries where all denominations are the same size and same color.

Color of food:  In the south we eat “dirty” rice, or ask for “debris” on our po-boy.

Reading/Viewing assignment while you plan your crit project

Walk This Way navigational guidance

Deaf Architecture

How quickly can you change your appearance in public?  Same retired-CIA agent comments on disguise in movies.  The second one is a good guide for understanding what you’re seeing in entertainment.

Reading body language.

Visual feedback

The less types and amount of visual feedback you give the better

Ex: an analog clock needs hours and minutes, but does it need seconds?  Microseconds?  Centuries?  Do digital clocks (ex: alarm clocks) display seconds?

What do the clocks in my kitchen need to show me?  Why are there so many clocks that aren’t in sync?  I have:

• Zojirushi water boiler on the counter.
• Over the window/sink, vintage analog clock stuck at 5:30
• Zojirushi rice maker
• iPad (we use for music in the kitchen)
• Microwave
• Stove

If the power goes out I have to reset each of them, there’s no external clock.  This clock costs $20 and sinks with the US atomic clock radio broadcast.

Another watch with no second hand and no minute marks.  Originally designed for the blind, but now a popular visual style.

Did Susan Kare invent emoji?  Her first icon designs for the Mac were done on a square grid as she knew needlepoint.  She went on to do graphic design for General Magic.

Visual states

I look at these fundamental types of visual state:

• color
• motion
• intensity

The type of display is also important: led, a light, a screen, a moving object in 3 dimensions, a swinging/flashing metronome.

Complex visual states

• typeface
• language
• icons
• images

Weekend assignment in two parts

First, please read chapters 3, 4, and 5 of Make It So.

Second, look around your world and find some good and bad examples of visual display of information.  Your home, your car, public transit, etc.    Example:  I think my Eone Bradley watch changed how I think about time.  I can only tell time to “about minutes”, it’s “about 4:55” or “about 4:15”.  This has changed how I think about time being analog, not digital like on my phone.

These can be mechanical sounds made with solenoids, motors, or fans; or sounds created from within the Teensy.

For amplifying sounds sent to speakers there is also the option of using a power transistor the same way you’d power a large motor.

https://www.color-blindness.com/coblis-color-blindness-simulator/

How sound effects are used in music:

MIA, this is the song I heard while we were waiting on a concert to let in.  I couldn’t hear the music but I could hear the FX:

Kraftwerk using samples of bicycles in Tour de France:

There’s also a link to genre conflation in MIA’s work.  She’s not from an American city where she grew up with hip-hop, she found a lot of different genres of music she liked and tied them all together.  Here’s a video shot in Morocco of Saudis (and others) drifting and marching in the street:

Why are these machines so quiet?  Why doesn’t it sound like a Chuck E Cheese or an arcade?

A mapping of pitches to frequencies https://pages.mtu.edu/~suits/notefreqs.html

It’s easy to create sounds without speakers:

• music made with  steppers
• A “top ten” list of arduino music projects
• historical tube doorbells and someone what lost their mind

Start looking at wider range of interaction with sites like Buxton’s collection of interaction design devices:

https://www.microsoft.com/buxtoncollection/browse.aspx

This is a counter to sampling and using electronics to invent new music.  Instruments from one style of music are used to perform a style of music from a completely unrelated genre.  My favorite examples are “pirate metal” or the band Orkestra Obsolete playing famous pop music.

Another take on this is how Kraftwerk inspired so many artist in other genres.  Probably the most influential is a sample from “Trans Europe Express” used by Africa Bambatta as the base melody for “Planet Rock“.

I’m skipping EDM, IDM, dub, liquid trap, future soul, and so many types of music.  Again, there’s an entire class in this topic, but Ishkur’s Guide is one place to start exploring the genealogy of music: http://music.ishkur.com/

From the earliest days of theater, foley artists have made a wide variety of sounds by hand (or foot):  https://www.youtube.com/watch?v=UO3N_PRIgX0

Mini-assignment:  What sounds would you add to your house to make it more accessible?

This is a good time to use push buttons or other inputs to trigger sound and another input to define the type of sound.

This is also where interrupts are very useful.

My  phone *doesn’t* do this well.  If I am listening to music and someone rings my doorbell at home, my phone continues to play music *and* the doorbell notification sound at the same time.  What it should do is stop the music, play the notification, then give me the opportunity to talk to the person at the door, then continue playing music.

Due next class.

Using MQS is pretty easy, here’s an example:

https://github.com/TeensyUser/doc/wiki/Audio-Example-using-MQS-on-Teensy-4.0-or-4.1

The Teensy can drive a tiny speaker but it’s not very loud.  Use the amplifier I distributed to drive a small speaker.

Psychological effects of sound

Is it genetics that cause us to respond to the sound of a crying human baby?  Can you think of an “angry” noise?  A “happy” noise? a “relaxing” noise?

Experimental / avant-garde sound

Mark Applebaum’s experimental instruments and scoring.

Nikoli Voinov who composed music by drawing on paper, creating animation that made sound.

The Variophone: https://www.youtube.com/watch?v=4r4WqAf-X8Y

Musique concrete using early technology to record and modify sound, including the original soundtrack to “Doctor Who”.  Some great examples of recording found sound and reusing it for music.

Some experimental music is a do-over of something from a previous generation.  Brian Eno and Robert Fripp “invented” Frippertronics, but people have been experimenting with looped tape for decades.

Live demonstration: https://www.youtube.com/watch?v=kaKgj9DqxhE

Fripp/Eno live performance: https://www.youtube.com/watch?v=xso_RoigibA

Some music you should listen to as background music while you’re doing other tasks.  Avant-garde and futurism is a rather wide grouping, like saying “rock” or “country”:

• Experimental composition with traditional instruments including John Cage
• Luigi Russolo

Aphex Twin (Richard James) wants to make music instruments that don’t exist, so he creates them with synthesizers:  https://www.youtube.com/watch?v=_AWIqXzvX-U

The Development of Rap/Hip-Hop

Using street technology to change and create new genres of music.  Entertainment and environmental sounds can come from other contexts with the use of equipment to record, store, modify, and replay.

Turntables used to create hip hop and the 1.5 hour documentary.

The Orchestral Hit.  Please watch all of this as it goes in to early interaction design

The Amen Break.  Also watch all of this, it’s not only a great story about a break beat but how to tell a story about music using film/editing techniques.

Rob Base (of Rob Base and DJ EZ Rock  https://www.youtube.com/watch?v=phOW-CZJWT0) teaching DJ at The New school:  https://www.youtube.com/watch?v=I15WVyhoCHo

Assignment

Assignment 7: sound with meaning

Why is sound important?  How do you close your “ear-lids” when you go to sleep?  Is sound our first evolved sense that is effectively an interrupt for an external event not touching our skin?

If you’re interested in a deep dive in to sound design:”Designing with Sound: Fundamentals for Products and Services” starts out with a vocab for sound design: https://amzn.to/3JiAikD

We’ve been thinking about sound as philosophy for quite awhile, how do we learn to make sounds as children?  https://www.youtube.com/watch?v=8fHi36dvTdE&feature=youtu.be&t=920

We learn to make some sounds as children but can never learn them as an adult learning another language.  The Russian vowel Bbl is one example

You can learn to have “perfect pitch” if you start as a child: https://www.youtube.com/watch?v=816VLQNdPMM

Close your eyes after following links in this section.  don’t worry about the visual details and information, this is learning to understand sound and signals

Some of these examples are long, 10-20min.  I’m only including ones worth listening to from start to finish.

classes of sounds (one view)

Signals and alerts – short sounds that transfer information

• automobile warning buzzers
  https://www.soundsnap.com/tags/car_interiors
• walk/don’t walk beep boop and how crazy we were about it at CMU
  https://www.youtube.com/watch?v=dcE6gn-JLbw
• fire department / police codes on dispatch channels
  http://www.policeinterceptor.com/emerg.htm
• Emergency Broadcast System, a pre-cursor to things like “Amber Alert” https://www.youtube.com/watch?v=vb5KZnNvXBM
• collection that includes a game tone https://www.zedge.net/find/ringtones/dispatch
• doorbell: https://www.youtube.com/watch?v=sbCyxgLgmEE
  https://www.youtube.com/watch?v=fUVj0BnC8Gw
• ringing phone: https://www.youtube.com/watch?v=3GnoCz5mw3M

INSERT photo of our doorbell

Information over time – songs and patterns

– information, many of them skeuomorphic

– air raid siren, dual pitch:

( my borough uses this as the 15 minute warning on curfew for minors!)

– tornado sirens:  https://www.youtube.com/watch?v=Djqi86jMl5g

– Victorian houses used remote bells to alert servants and family, bell pull at the front door to let people know someone has arrived

DIY air raid siren, there are a lot of these on Thingiverse (equiv)  https://www.youtube.com/watch?v=ZVDmbFXFTj0

Music and entertainment

• ringing phones are so unique that we map and learn our own ringtones using songs.  Golan Levin’s mobile phone concert was only possible because phones had ring tones that couldn’t be changed.
• Star Trek had one of the earliest catalogs of special effects sounds used to alert viewers of plot elements and activity.
• Professional companies that sell sound libraries. sizzling fajitas podcast
• Chili’s used the sound of sizzling fajitas to sell them, not their flavor:
  https://99percentinvisible.org/episode/the-sizzle/

We like things that reflect heartbeat rhythm, this goes back to dance for thousands of years, continue to do this in today’s music with multiple tempo compositions

Orbital and Underworld are two electronic bands using dual internal tempos, one near the rate of resting heart beat the other near the rate of active (dancing) heart beat.

mini-assignments:

Reading: Make it So Chapters 6, 10

Make your Teensy make some noise

Crit will happen during class on 3 Mar, 2022.

How to read a data sheet using L293 vs. tutorials

How would you do your project differently using a bridge?  Is coasting a useful feature?

How do interrupts work with motion?

How do state machines work with interrupts and motion?

Using chips to simply interrupts

Let’s say you have to interrupts, and you only care if both of them happen at the same time.   You could feed both inputs to an AND chip that we have in A10 and feed the output to a single input interrupt.  Likewise, if you have two inputs and you care if either one go off and will do the same thing for each, use an OR chip.

Use state variables to change solenoid / motor based on an interrupt

Talk about crits and elevator pitches.

Simple assignment

• Verify that your Teensy boards are working and that you can download sketches.
• Skim the Teensy 4.1 page and think about things you can do in future projects / crits.  Update Looking Out if you find something interesting using the Teensy chips in ixd projects.