VR Headsets and vision problems:

https://www.bbc.com/news/technology-52992675

https://www.aao.org/eye-health/tips-prevention/are-virtual-reality-headsets-safe-eyes

Sand Tables and Interaction:

Sand Noise Device – please watch

A History of Sand Tables – this is long, but if you’re interested in how to visualize things in 3D this is a good starting point.

A brief aside on visual skeuomorphism

A skeuomorph is a object derived from past objects and contains past visual cues that have no meaning in current culture.  Ex: the “phone” icon on my Android looks like the physical handsets we stopped using in the 90s.

Think of a derivative object that retains ornamental design cues (attributes) from structures that were inherent to the original. Examples include pottery embellished with imitation rivets reminiscent of similar pots made of metal and a software calendar that imitates the appearance of binding on a paper desk calendar.  Or the “save” icon that looks like a floppy drive, that a Japanese student thought looked like a drink dispenser:

“Why is the Excel save mark a vending machine? “

When you are creating visual representations, are you using your culture’s history of symbolism?

States of being, replacing sound with visuals

Visualizing sound

Visualizing music with images

ASL vs Braille

ASL replaces a word with a motion while Braille replaces each letter with a pattern with a shape similar to the letter.  Before Braille, books for the blind used large raised letters that you would trace by hand.

Early experiments with live animation and MIDI inputs with devices manipulated by the audience.

Console displays in video games are a different way to display information.  This is metadata about your status.  health, money, power, speed.

Wipeout series brought commercial racing context to a SF racing game.  Music for Wipeout XL was mostly bands who were played or performed in rave spaces.  There were in-game billboards for real products.  The visuals were designed by The Designers Republic, a well-known design firm.

Rez was a completely different video game and hard to explain.  The controls are simple, but as you play you dynamically create new visual and audio structures.

Cultural Contexts

What colors mean in US culture vs. Asiatic, Arabic, African countries

ex: white wedding dress or white funeral dress?

colors important because they show wealth: purple for royalty, gold.  There were “dyer guilds” that kept secret their methods for creating a dye.

Color of money — the US is one of the few countries where all denominations are the same size and same color.

Color of food:  In the south we eat “dirty” rice, or ask for “debris” on our po-boy.

Reading/Viewing assignment while you plan your crit project

Walk This Way navigational guidance

Deaf Architecture

How quickly can you change your appearance in public?  Same retired-CIA agent comments on disguise in movies.  The second one is a good guide for understanding what you’re seeing in entertainment.

Reading body language.

Visual feedback

The less types and amount of visual feedback you give the better

Ex: an analog clock needs hours and minutes, but does it need seconds?  Microseconds?  Centuries?  Do digital clocks (ex: alarm clocks) display seconds?

What do the clocks in my kitchen need to show me?  Why are there so many clocks that aren’t in sync?  I have:

• Zojirushi water boiler on the counter.
• Over the window/sink, vintage analog clock stuck at 5:30
• Zojirushi rice maker
• iPad (we use for music in the kitchen)
• Microwave
• Stove

If the power goes out I have to reset each of them, there’s no external clock.  This clock costs $20 and sinks with the US atomic clock radio broadcast.

Another watch with no second hand and no minute marks.  Originally designed for the blind, but now a popular visual style.

Did Susan Kare invent emoji?  Her first icon designs for the Mac were done on a square grid as she knew needlepoint.  She went on to do graphic design for General Magic.

Visual states

I look at these fundamental types of visual state:

• color
• motion
• intensity

The type of display is also important: led, a light, a screen, a moving object in 3 dimensions, a swinging/flashing metronome.

Complex visual states

• typeface
• language
• icons
• images

Weekend assignment in two parts

First, please read chapters 3, 4, and 5 of Make It So.

Second, look around your world and find some good and bad examples of visual display of information.  Your home, your car, public transit, etc.    Example:  I think my Eone Bradley watch changed how I think about time.  I can only tell time to “about minutes”, it’s “about 4:55” or “about 4:15”.  This has changed how I think about time being analog, not digital like on my phone.

https://www.color-blindness.com/coblis-color-blindness-simulator/

How sound effects are used in music:

MIA, this is the song I heard while we were waiting on a concert to let in.  I couldn’t hear the music but I could hear the FX:

Kraftwerk using samples of bicycles in Tour de France:

There’s also a link to genre conflation in MIA’s work.  She’s not from an American city where she grew up with hip-hop, she found a lot of different genres of music she liked and tied them all together.  Here’s a video shot in Morocco of Saudis (and others) drifting and marching in the street:

Why are these machines so quiet?  Why doesn’t it sound like a Chuck E Cheese or an arcade?

A mapping of pitches to frequencies https://pages.mtu.edu/~suits/notefreqs.html

It’s easy to create sounds without speakers:

• music made with  steppers
• A “top ten” list of arduino music projects
• historical tube doorbells and someone what lost their mind

Start looking at wider range of interaction with sites like Buxton’s collection of interaction design devices:

https://www.microsoft.com/buxtoncollection/browse.aspx

This is a counter to sampling and using electronics to invent new music.  Instruments from one style of music are used to perform a style of music from a completely unrelated genre.  My favorite examples are “pirate metal” or the band Orkestra Obsolete playing famous pop music.

Another take on this is how Kraftwerk inspired so many artist in other genres.  Probably the most influential is a sample from “Trans Europe Express” used by Africa Bambatta as the base melody for “Planet Rock“.

I’m skipping EDM, IDM, dub, liquid trap, future soul, and so many types of music.  Again, there’s an entire class in this topic, but Ishkur’s Guide is one place to start exploring the genealogy of music: http://music.ishkur.com/

From the earliest days of theater, foley artists have made a wide variety of sounds by hand (or foot):  https://www.youtube.com/watch?v=UO3N_PRIgX0

Mini-assignment:  What sounds would you add to your house to make it more accessible?

Using MQS is pretty easy, here’s an example:

https://github.com/TeensyUser/doc/wiki/Audio-Example-using-MQS-on-Teensy-4.0-or-4.1

The Teensy can drive a tiny speaker but it’s not very loud.  Use the amplifier I distributed to drive a small speaker.

Psychological effects of sound

Is it genetics that cause us to respond to the sound of a crying human baby?  Can you think of an “angry” noise?  A “happy” noise? a “relaxing” noise?

Experimental / avant-garde sound

Mark Applebaum’s experimental instruments and scoring.

Nikoli Voinov who composed music by drawing on paper, creating animation that made sound.

The Variophone: https://www.youtube.com/watch?v=4r4WqAf-X8Y

Musique concrete using early technology to record and modify sound, including the original soundtrack to “Doctor Who”.  Some great examples of recording found sound and reusing it for music.

Some experimental music is a do-over of something from a previous generation.  Brian Eno and Robert Fripp “invented” Frippertronics, but people have been experimenting with looped tape for decades.

Live demonstration: https://www.youtube.com/watch?v=kaKgj9DqxhE

Fripp/Eno live performance: https://www.youtube.com/watch?v=xso_RoigibA

Some music you should listen to as background music while you’re doing other tasks.  Avant-garde and futurism is a rather wide grouping, like saying “rock” or “country”:

• Experimental composition with traditional instruments including John Cage
• Luigi Russolo

Aphex Twin (Richard James) wants to make music instruments that don’t exist, so he creates them with synthesizers:  https://www.youtube.com/watch?v=_AWIqXzvX-U

The Development of Rap/Hip-Hop

Using street technology to change and create new genres of music.  Entertainment and environmental sounds can come from other contexts with the use of equipment to record, store, modify, and replay.

Turntables used to create hip hop and the 1.5 hour documentary.

The Orchestral Hit.  Please watch all of this as it goes in to early interaction design

The Amen Break.  Also watch all of this, it’s not only a great story about a break beat but how to tell a story about music using film/editing techniques.

Rob Base (of Rob Base and DJ EZ Rock  https://www.youtube.com/watch?v=phOW-CZJWT0) teaching DJ at The New school:  https://www.youtube.com/watch?v=I15WVyhoCHo

Assignment

Assignment 7: sound with meaning

Why is sound important?  How do you close your “ear-lids” when you go to sleep?  Is sound our first evolved sense that is effectively an interrupt for an external event not touching our skin?

If you’re interested in a deep dive in to sound design:”Designing with Sound: Fundamentals for Products and Services” starts out with a vocab for sound design: https://amzn.to/3JiAikD

We’ve been thinking about sound as philosophy for quite awhile, how do we learn to make sounds as children?  https://www.youtube.com/watch?v=8fHi36dvTdE&feature=youtu.be&t=920

We learn to make some sounds as children but can never learn them as an adult learning another language.  The Russian vowel Bbl is one example

You can learn to have “perfect pitch” if you start as a child: https://www.youtube.com/watch?v=816VLQNdPMM

Close your eyes after following links in this section.  don’t worry about the visual details and information, this is learning to understand sound and signals

Some of these examples are long, 10-20min.  I’m only including ones worth listening to from start to finish.

classes of sounds (one view)

Signals and alerts – short sounds that transfer information

• automobile warning buzzers
  https://www.soundsnap.com/tags/car_interiors
• walk/don’t walk beep boop and how crazy we were about it at CMU
  https://www.youtube.com/watch?v=dcE6gn-JLbw
• fire department / police codes on dispatch channels
  http://www.policeinterceptor.com/emerg.htm
• Emergency Broadcast System, a pre-cursor to things like “Amber Alert” https://www.youtube.com/watch?v=vb5KZnNvXBM
• collection that includes a game tone https://www.zedge.net/find/ringtones/dispatch
• doorbell: https://www.youtube.com/watch?v=sbCyxgLgmEE
  https://www.youtube.com/watch?v=fUVj0BnC8Gw
• ringing phone: https://www.youtube.com/watch?v=3GnoCz5mw3M

INSERT photo of our doorbell

Information over time – songs and patterns

– information, many of them skeuomorphic

– air raid siren, dual pitch:

( my borough uses this as the 15 minute warning on curfew for minors!)

– tornado sirens:  https://www.youtube.com/watch?v=Djqi86jMl5g

– Victorian houses used remote bells to alert servants and family, bell pull at the front door to let people know someone has arrived

DIY air raid siren, there are a lot of these on Thingiverse (equiv)  https://www.youtube.com/watch?v=ZVDmbFXFTj0

Music and entertainment

• ringing phones are so unique that we map and learn our own ringtones using songs.  Golan Levin’s mobile phone concert was only possible because phones had ring tones that couldn’t be changed.
• Star Trek had one of the earliest catalogs of special effects sounds used to alert viewers of plot elements and activity.
• Professional companies that sell sound libraries. sizzling fajitas podcast
• Chili’s used the sound of sizzling fajitas to sell them, not their flavor:
  https://99percentinvisible.org/episode/the-sizzle/

We like things that reflect heartbeat rhythm, this goes back to dance for thousands of years, continue to do this in today’s music with multiple tempo compositions

Orbital and Underworld are two electronic bands using dual internal tempos, one near the rate of resting heart beat the other near the rate of active (dancing) heart beat.

mini-assignments:

Reading: Make it So Chapters 6, 10

Make your Teensy make some noise

How to read a data sheet using L293 vs. tutorials

How would you do your project differently using a bridge?  Is coasting a useful feature?

How do interrupts work with motion?

How do state machines work with interrupts and motion?

Using chips to simply interrupts

Let’s say you have to interrupts, and you only care if both of them happen at the same time.   You could feed both inputs to an AND chip that we have in A10 and feed the output to a single input interrupt.  Likewise, if you have two inputs and you care if either one go off and will do the same thing for each, use an OR chip.

Use state variables to change solenoid / motor based on an interrupt

Talk about crits and elevator pitches.

Simple assignment

• Verify that your Teensy boards are working and that you can download sketches.
• Skim the Teensy 4.1 page and think about things you can do in future projects / crits.  Update Looking Out if you find something interesting using the Teensy chips in ixd projects.

Why do things move?  How can we express emotion?

American Sign Language (ASL) is not English as we speak English, or like they speak in books.  (The wiki entry goes in to deep linguistic details.)

Emotional, artistic expression is done solely through movement.  These examples have audio for people who can’t read ASL:

ASL poetry Sign off Finals – “drop it like its hot”

Cardi B, Bodak Yellow ASL Interpretation

From “Hamilton:  Can you sign as fast as Daveed Diggs can rap? “Guns and Ships“.  “Alexander Hamilton” by a performance group with correct costumes.

ASL in entertainment

Troy Kotsur, a deaf actor, was hired to play a Tusken Raider and create their sign language.

Expression through artistic grammars

Bill Shannon has invented a new vocabulary for dance performed with canes.

• Music video for RJD2’s “Work it Out“
• Commercial for VISA
• “Body Language“, an autobiographical short film
• The Shannon Technique, his vocabulary of dance

Demo Using transistors and diodes

• rotate a DC motor
• use a solenoid
• use a servo

Fritzing sketch, compressed with gzip: arduino-IRLB8721-simple.fzz

Intro to Electricity and Power

Start with example using light and power

100w incandescent bulb

23w LED bulb

120v 20a electrical circuit

wattage — what does it really mean?  Is a 100W incandescent bulb brighter than a 23W LED?

Compute amps based on wattage and 120v power.

Same brightness but different amounts of power, meaning LEDs are more efficient:

100W incandescent bulb ~1A

24W LED bulb ~0.2A

How things move

Think about power in the physical world:

voltage => weight

amps => speed

I weigh 85 kilos, how “dangerous” am I if I bump in to you while I’m walking?  While I’m running?

Is voltage dangerous or is amperage dangerous?

Bicycles vs. cars. Hit by a car or a bike at 10mph, which is worse? Is it really voltage or amps?

Handheld 50 KW stun gun.  500 times as strong as strong as a 100W light bulb.  Why doesn’t it kill people?

Think about volts vs. amps.  Where does the power come from?

Mine uses two 9V batteries, the ones you use in consumer electronics.   50KW is more like 50KV at 1 Amp, or 100Kv at 0.5 Amps.

Where we get power

A/C wall power to drive power supplies (PSUs).

DC power from a PSU, from 5V to 12V, we use 6V in class.

USB power converted by Arduinos or other devices

Arduino pins only have a tiny amount of power: 5V at 20 mA (milliamp).

How motors work

How to determine power requirements

How to drive servos vs. DC motors

Spring-loaded solenoids that require constant power to stay open/closed

How do you lock a motor and prevent it from turning?

What happens if you spin a DC motor?  It’s now a DC generator and could fry your Arduino.  We “fix” this using diodes in the circuits.

For moving things, use basic physics

• gears — FAQ on 3d printing gears
• levers — drive a lever with a solenoid, transfer short movement to long
• pulleys – drive with DC motors or stepper

A great guide for animated mechanisms:  http://507movements.com/index02.html

Reading/Watching Assignment

“The Secret Life of” was a British TV series by Tim Hunkin.  He now has a Youtube channel and a new series, “The Secret Life of Components“.  Each of the eight episodes is about 45 minutes long.  Pick the one you know the least about, watch it, and post comments here or in Looking Outward

Going forward, verify that your camera is not set to “HD” in Zoom.  I disabled this tonight and things went much better than expected.

Correction for previous lecture!  You do use Serial.print() and Serial.write() to send info to p5.js.  When you’re doing this, you can’t use Serial methods for “printf debugging”.

Types of Input

Using sound / music as an example:

• monophonic: wind instruments, voice.  Easy to detect pitch/volume.

polyphonic: keyboards, pianos, organs, stringed instruments.  Hard to detect pitch/volume of a single note, but easy to detect volume of a collection of instruments.

Anthropomorphic inputs

respond to human state/condition:  blood pressure, galvanic skin response, breath rate, pulse rate.

Visual interpretation of secondary movements: eye twitch, touching your face, blinking

Tracking “rapid eye movement” when eyes are closed, like REM when you’re asleep is used in PTSD therapy EMDR. https://en.wikipedia.org/wiki/Eye_movement_desensitization_and_reprocessing.  This is a recently developed form of therapy and treatment for PTSD and requires trained human therapists.

Alternatively, you can make art out it:  http://www.flong.com/archive/projects/optoisolator/index.html

SparkFun’s biometric sensors:  https://www.sparkfun.com/categories/146

More kinetic output

There is a lot of work in this area to provide accessibility for people with limited vision or blind people.  Please watch all of these, we didn’t have time to show them all in class.

• tactile maps for the blind
• presentation of research data on tactile map comparison
• tactile graphics using “swell paper”
• 3d printing reference objects for the blind — what does a snowflake look like?  A butterfly?  A sailboat?

Accessibility and HCI

In 1968, Doug Englebart demonstrated the first “workstation”.  It’s a long watch but I think at least the first half will give you a lot of ideas on how to pitch a novel technological concept.  The important bit about accessibility   starts about 30 minutes in when they show the interface. https://www.youtube.com/watch?v=yJDv-zdhzMY

Using this workstation requires sight, hearing, and functioning hands/fingers/feet.

Accessibility vs. Inclusion

What makes something accessible?  Is Universal Design also accessibility?

Inclusion ==> inviting, making someone want to participate. How do you invite someone to provide input / direction?

Are 30mm arcade buttons are accessible? Interrupt or constant? Convex or concave? If you want to use Universal Design, how do you decide how big the button should be and where it’s located?

There’s a wide variety of arcade push buttons.  Are controls like buttons the wrong answer?  Is a better way to collect input?

Physical interaction with temperature

There are some serious things to consider when working with temperature.  Basically, “what happens if this breaks, has bugs, or completely fails?”  Can someone be injured?

Maybe you need temperature control in your project? Adam Savage made 2001:Space Odyssey suits for Comic-Con that required a costume cooling vest.  He actually made two, the other was for Astronaut Chris Hadfield(!) who provides good feedback.  This is another one to watch all the way through, there’s a lot of design/fabrication skills shown off.

There’s also a commercial alternative for performers.

Coaching vs. grading

Think  about coaching, providing good feedback and encouragement to take a positive action.

Example: sports trainer that monitors your HR, BP, breathing rate, and hydration and knows your training course.  It encourages you to do better instead of punishing you for not doing enough.

Example: music “coach” that helps you learn to perform music. Watches your body and helps you correct form/posture.  Reminds you that you are always performing, even when you’re just practicing a scale or an etude.

Alice Miller’s “For Your Own Good“, a criticism arguing that we replace the pedagogy of punishment  with support for learning, using the German pedagogy that gave rise to support of fascism as one study

Assignment 5 has requirements in its own post.