Class notes, 6 Oct 2020

Haptic/touch vs. objects moving in space

touching a person vs. moving an object

person presses a button vs. wind blowing a windmill

touching a person vs. moving an object – touch is more personal, requires little energy.

Touch can be wrapped in a robotic device, ex: Paro (wiki) trade show demo.

Moving things typically requires an external power source, Arduino can only provide 5V at a few milliamps.

energy required to move large things vs. small things

explain watts vs. amps vs voltage vs. movement

For moving things, use basic physics

  • gears — FAQ on 3d printing gears
  • levers — drive a lever with a solenoid, transfer short movement to long
  • pulleys – drive with DC motors or stepper

Input classification, serial communication, interrupts.

Types of Input

monophonic + skill: wind instruments, percussion

polyphonic + tech: keyboards, pianos, organs, strings
anthropomorphic: respond to human condition, blood pressure, galvanic skin, breath rate, pulse rate, visual interpretation of secondary movements: eye twitch, touching your face, blinking

“rapid eye movement” when eyes are closed, used in PTSD therapy called “EMDR”:

SparkFun biometric sensors:

UPMC lobby temperature check:  upmc-lobby-temperature-check

Serial Communication

SPI/I2C and complex communications protocols
How we get complex data from sensors – a lot of this is hidden in libraries
Unique IDs
Simple controls for complex output: neopixel
SparkFun’s version: Qwiic


Show examples of interrupt code in the environment
switches on mobiles
remote controls for the projectors, TVs, stereos
little to no latency
complex interrupt systems in video game controllers
rotary encoder (we’ll do a demo later in the semester)
for now, we only use digital inputs for interrupts

Code samples, show how an interrupt can be used to toggle a state by one increment compared to holding down a switch and falling through a number of states.
Note that holding down the switch means the interrupt service routine (ISR) only functions once
Compare to using delay() to sample data every so many units of time.

Use an interrupt to stop a task that takes a long time, say a long for() or while() loop, by adjusting the terminating conditions

Question: What if you were playing mp3 files or video, how would you use interrupts as part of the interface?

zip file with some examples:  interrupts-1-pot

Class 10, 1 October notes


voting – go vote!  If you think you’ll be late for class, please drop me email in advance.

celebrate non-US holidays:  Chuseok and other Fall-holidays are a perfectly fine reason to miss class

Update:  another banned assignment is navigation belts as they have been done to death.

Physical interaction with temperature

Adam Savage’s DIY costume cooling vest for cosplay and a commercial alternative.

Consider medical/physically safety with devices that touch the skin.  It’s really easy to get burned/frozen with Pletier plates and ice.  Instead of smoke, use soap bubbles and a fan.

Coaching vs. grading

Think  about coaching, providing good feedback and encouragement to take a positive action instead of negative feedback.   In Total Control Training we use positive feedback and coaching.  Instead of saying, “You did that wrong, you were too slow going in to the corner,do it again” we say “That’s good, now do it again with a little more throttle as you go in to the corner.”

You want your interaction to be one that invites people and makes them want to interact.  If you do a good job, people will wait in line to use your interaction:

Example: sports trainer that monitors your HR, BP, breathing rate, and hydration and knows your training course.  It encourages you to do better instead of punishing you for not doing enough.

Example: music “coach” that helps you learn to perform music. Watches your body and helps you correct form/posture.  Reminds you that you are always performing, even when you’re just practicing a scale or an etude.

Something to read over the break:  Alice Miller’s “For Your Own Good“, a criticism arguing that we replace the pedagogy of punishment  with support for learning, using the German pedagogy that gave rise to support of fascism as one study.

Class notes 29 Sep 2020 – start kinetics


A10 dates updated on blog

reminder, campus is closed after Thanksgiving

looking for parts, try octopart search engine:

do people want 3d prints or lasercuts or both?

Discuss reading of Make It So chapters

Start Kinetic discussion

size of physical control vs. input

size of physical control vs. output

tactile controls are great for fine control/refinement/detailed feedback

but can get lost in the shuffle — NASA controls are laid out based on the physical design of the space shuttle

or can be stylistic / skeumorphic. Why does a Starfleet vessel have touchscreens (LCARS) everywhere but the warp engines are driven by an 19th century ship’s throttle?

MIX MECHANICAL AND OTHER CONTROLS WHERE APPROPRIATE Mechanical controls are better for some uses, though they can’t as easily serve multiple functions. Nonmechanical controls, like touch-screen buttons, are easier to change into other controls but don’t offer the same kind of haptic feedback, making them impossible to identify without looking at them and creating questions about whether they’ve been actuated. Design interfaces with an appropriate combination that best fits the various uses and characteristics.

look at inputs for kinetic outputs

median vs. mean

std devs see wiki for details:

how to read complex sensors over serial protocols: I2C, SPI, MIDI

look at data smoothing / filtering

simple smoothing:

break down the types of kinetic interaction

focusing on output

– vibration

– thumps, pokes

– temperature? peltier boards

– symbols: Braille

Then over time

– signal encodings, morse

– pattern recognition: what does walk feel like? Run? Crying? Laughing?

– earthquake pattern recognition

– meaning generated by content that changes over time, poetry

Class notes, 17 Sep 2020

Class Admin

I’ve made a request that Tuesday’s work class be held in A10.  Will email an update.

Looking Outword comments

dark skies as a source of crowd-sourced weather data, scrape data from,-79.8655/us12/en

Interfering with sensors goes back to “Dazzle Camouflag”:

Virag Varga’s research on devices that communicate with one another through skin:

State machine transitions

Documenting a state machine: Omnigraffle (mac) vs. SmartDraw (win10) vs. ??? (linux) vs. whiteboard

A finite(explain) state machine needs states, transitions, and actions (transitions that do things)

Example: UI button that changes screen color has a state variable with the current color

state machines in video games, hierarchies of state machines

how do we define a transition from one state to another?

– segue: pan, wipe, blur

– alert:

– microwave ding when it’s finished heating

– countdown timer to start an event (from waiting -> running)

– elevator alerts

what can visual output replace? sound? motion?

babymonitor that translates sound to video

GFCI sound vs. click

replace sound warning with video flash


how can we replace sound from a state machine with visual queues?

what sounds are important in state machines and what sound are decorations?

keyclicks on/off phone

annoying car noises

car sounds effects

Reading/research links not discussed in class

Discuss tempo of sound

visualizing sound waves

visualizing music:

Vibrant Data Visualizations of Famous Classical Music Scores Burst with Color

Visualization of language



Visual communication examples

Arrival — read “The Story of Your Life”

the movie version:

99pi episodes 50 “Deafspace” (Gaulludet) and 126 “Walk This Way” (wayfinding)

When your senses get crossed, Synesthesia

Class 5, 15 Sep – State changes

Class Admin

who needs access to A10? We can only do it Tue night during class hours.

bookstore has not shipped yet. we will get tracking #’s when they do.

This is in the kit: ELEGOO UNO Project Super Starter Kit with Tutorial and UNO R3 Compatible with Arduino IDE

trying to deal with lost class time: an assignment for next tue, then a crit on the 24th that includes the assignment?

Review Assignments

what went right?

what went wrong?

The “thing like a pot” is a rotary encoder

Finite vs. abstract state machines

finite: what we’re doing in class, fixed states and paths

abstract: can modify itself to adapt to changes in data.

What’s a state machine in nature with interaction?

Reactive state machines:

Hourglass timer – set time, start, stop, reset

Arcade game – attract / play / reward

gambling devices slot machines

Murphy bed attached to lights

Interactive state machines:

content predictors on Amazon/Netflix/eBay, “if you like…”

advertising predictors, “if you like xxx we want you to buy yyy”

microwave that can adjust cooking time based on humidity (I have one of these!)

traffic lights with IR detectors for EMS vehicles

Picker robots

Music instrument as state machine:

What if state machines were smart(er)? What are interesting problems for them to solve?

What are predictive states machines? console gaming turned in to physical gaming

auto-aiming / target tracking

catch a frisbee (dogs know calculus)

Can made predictions based on history/sensors? Your car notices you’re a sloppy driver (exhausted) and limits your speed. Duolingo detects you’re having vocab problems and changes your exercises (this is an ASM, arguably).

Visual changes in state

  • Doppler shift in nature
  • lives / health in a video game
  • traffic slowdowns in Waze
  • intensity of light => time of day

Mini-assignment for Thursday

post some interesting state machines to Looking Outward

Class Notes: 8 Sep, 10 Sep

 Start Visual


Crits/projects due on Tuesdays.  Reading/listening/watching assignments due Thursdays.

Arduino / P5.js configuration

Install Arduino, run test hello world sketch

Install p5.js and serial control:

Dan Schiffman has great tutorials.

Discuss visual feedback

The less types and amount of visual feedback you give the better

Ex: a clock needs hours and minutes, but does it need seconds? Microseconds? Centuries?

What do the clocks in my kitchen need to show me?  Why do I have so many clocks in appliances that can set an action for a specific time in the future?

Combining the visual and haptic, the Bradley watch.

Did Susan Kare invent emoji?  Her work at Apple

led to her work at General Magic, making “stamps” for the cutting-edge Magic Link “PDA”.

Fundamental types of visual state

  • color
  • motion
  • intensity
  • type of display: led, a light, a screen, a moving object in 3 dimensions, metronome

Complex types of visual state

  • typeface
  • language
  • icons
  • images

Reading assignment

how many lights do you have in your kitchen?

Make It So: Forward, Chapter 1, Chapter 3. (Chapter 2 if you have time.)

Displaying State

But first, story telling

Tell a story, not a novel, not an elevator pitch.

A story has at least one actor, a beginning, a plot, and an end.

In interaction design “plot” is the interaction.

What is the plot of a doorbell?  Someone visits, they want to get your attention (as you don’t spend all day staring at the door), they push a button, an action happens, and you know someone is at the door.

How do doorbells work for people without hearing?  What did we do before electricity?

A brief aside on visual skeuomorphism

But first, visual skeuomorphism.  A skeuomorph is a object derived from past objects and contians past visual cues that have no meaning in current culture.  Ex: the “phone” icon on my phone that looks like the physical handsets we stopped using in the 90s.

is a derivative object that retains ornamental design cues (attributes) from structures that were inherent to the original.[3] Examples include pottery embellished with imitation rivets reminiscent of similar pots made of metal[4] and a software calendar that imitates the appearance of binding on a paper desk calendar.  Or the “save” icon that looks like a floppy drive, that a Japanese student thought looked like a drink dispenser:


When you are creating visual representations, are you using your culture’s history of symbolism?

States visualization

Console displays in video games – metadata about your status. health, money, power, speed.

Why do some of the games with great graphics still use bar graphs to display raw data?  We could hear our speed.  Our character/avatar could change color.  Why do we just have bar graphs?

  • Wipeout:
  • Rez:

What do colors mean?  What states do they reflect?

  • ex: white wedding dress or white funeral dress?
  • colors important because they show wealth: purple for royalty, gold, dichromatic glass.  Purple as an expensive color to make so only the rich/powerful could afford clothes dyed purple. (“The Secret Lives of Color“, “Why are Jeans Blue?
  • color of money — bills with different colors are easier to sort/find
  • color of food, white is good/clean.  In the south we have “dirty rice” where we mix leftovers with fresh, “clean” rice.
  • the colors of Victorian houses


Class 2 notes, 3 Sep, 2020

Introductions and Admin

Introduce ourselves and describe what we want out of the class

Class blog — I have to “promote” you after you sign on for the first time.

A10 cannot be used outside of class time. I’ll see if I can create a “class time” for MTI

kits are available:

How do we handle the end of the semester?

Pick My Office Hours


revisit tigoe’s page — what was physical interaction and what was reaction? Do you see any themes in projects that aren’t allowed? is there something you’re tired of seeing?

reading assignment — what makes something interactive?

does interaction require emotion or just the appearance of emotion? When do we go from “robot” to “uncanny valley” to “human”?

Musicians and DJs working a crowd — who is interacting and who is driving the interaction?

so what is physical interaction? When/where did it start?

Physical Interaction History

Beekeepers – explain the complexity of a hive and how we’re only just now (past 20 years) discovering how bees vote to make hive-wide decisions. “Honeybee Democracy” is a great introduction to how science works and how to write about science:

Alchemists – trying to make things happen with substances they don’t understand. If you don’t know about elements and that lead and gold are elements, what decisions are making to interact with substances? People knew that honey bee “propolis” helped prevent infection thousands of years before we discovered bacteria. Sometimes working in ignorance still leads to progress.

New music and dance styles based on sampling and analog synths: Moog, ARP 2600, hip-hop. Early hip-hop was limited to record players and mics; early synths were insanely expensive and shared at fancy studios. Grand Wizard Theodore *invented* scratching:

Console games with physical feedback, Rez Trance Vibrator:

Dance Dance Revolution from arcades to console games

Driving assistance with vehicles: AI? Interaction? What if the car won’t let me turn, start, or stop? How can I disable safety features?

Flight assistance software in commercial aircraft. Is autopilot interactive?

Flight assistance for military aircraft: self-guiding drones, incoming missile warnings for helicopter pilots. Interaction or reaction?

Near-future Interaction

the focus of this class – we’re prototyping for five years out

interacting with intelligent systems that we don’t completely understand and that can make decisions against our will or with results we don’t appreciate

  • car that thinks you’re too intoxicated, sleepy, or incompetent to drive, so it refuses to move
  • home automation system that won’t open the doors and let you leave because the particulate count in the air is hazardous and you’re not wearing a mask
  • police equipment that won’t let you fire at unarmed civilians — pistol trapped in a holster, safety wont go off.
  • fire engines that won’t engage fires that cannot be contained
  • entertainment systems that can filter content as part of mental-health

We’re giving “agency” to interactive devices:

“Near-future” is a popular base of pop-culture: MCU set a few years from now

Blade Runner 2049 is a bit farther in the future.

practical effects used as input devices to imagined systems of the future

Adam Savage gets a no-spoilers tour of the Blade Runner 2049 prop shop.

show Adam Savage with prop manager, start around 4:12, skip to 14:30, skip to 18:30

This is not new, in the past we talked about the future and technology. “Design for Dreaming”, released in 1956, is all about the future. It’s also all about the sexism in the 1950s, so you can skip to these points, to get the important points: start at 3:30; skip to 6; skip to 8

Good drama is about storytelling — what if interactive devices are characters in the story?

  • Colossus: The Forbin Project (1970) — our super computer meets the Soviet supercomputer and they plan our future
  • 2001 — HAL 9000, the ship’s computer that decides it does not want to follow the plan laid out by humans
  • War Games (1983) — A computer controls all of America’s nuclear weapons. Would you like to play a game? the only way to win is to not play the game.
  • Alien — MOTHER, a semi-intelligent computer that controls spacecraft while humans are in cryosleep.
  • ST:TNG’s Data — a walking mobile phone smarter than spacecraft computers?
  • (Terminator movies don’t count — killing spree robots, not interaction)
  • Farscape — Moya is semi-intelligent spacecraft used by the main characters, and can refuse orders or do things because of her own desire, not that of her passengers.

Assignment 2, due at 11:59 the day before class, so these are due Monday, 7 September

First, order your kit

This is a pen-and-paper assignment to go over posting to the class blog and what a good submission looks like. If you want to make something with hardware, that’s fine, but it’s not required. Science fiction is ok, fantasy is not. ever. NO UNICORNS.

1.) A smart doorbell, similar to the smart thermostat I talked about on Tuesday, or

2.) Make a useful practical effect by giving interaction to a thing that doesn’t currently interact with humans. For inspiration, watch Blade Runner 2049. Yes, it’s 3 hours long but OMFG. Look at the physical interfaces between intelligent systems, replicants, humans, AI, and agency for androids.

Prepare the presentation as you would for a crit. Look at previous classes for inspiration

Class 1 notes, 1 Sep 2020

Remote Learning

No need to leave video on unless we’re having a group discussion so we can raise hands.

Most lectures (starting next week) will be recorded and take roughly half of the class time.

Classroom discussions will not be recorded.

Take-home kits will be available sometime next week. We’ll let you know when they can be ordered, please don’t call the bookstore.

Projects will be turned in on the blog and demoed via Zoom. Live or pre-recorded demos?

Office hours as needed — email me if you’re stuck and want to have a 1:1 Skype

How do you feel about Canvas?

Undergraduate Requirements

Undergrads have a pre-req of Intro to Physical Computing. If you haven’t taken the pre-req but think you know enough to keep up with MTI assignments, stay for this intro then talk to me after class.

Graduate students

Comfortable working with Arduino, equivalent knowledge of Intro to Phys Comp.

Skills needed by all students


  • Arduino IDE in C++ or CircuitPython. VSC is also an option if you prefer.
  • for() vs. while() loops
  • arrays (C++)
  • switch/case
  • analog vs digital voltages
  • dictionaries (Python)
  • integer vs. float (C++)
  • interrupts
  • Understand how/why to use state machines

Physical Computing skills

  • Analog vs. digital electronics
  • External power — wall outlets and batteries and power supplies
  • The difference between servo motors, regular motor, solenoids
  • Can do more than cut-and-paste a sketch and change a few variable names
  • Can read/create basic sketches in Fritzing

Other Useful Skills

p5.js (javascript) can be used in this class

Python on Arduino is allowable as a substitute for C++, but I can’t help you debug

BeagleBone Black and Raspberry Pi may be used as a substitute for Arduino, again I can’t help you debug

If you want to learn this sort of technology but can’t take this class, feel free to email me for some pointers on how to get started on your own time. There are some “Arduino 101” kits that you could do over winter break that I would count for PhysComp in the future.

Introduction to the Class / go over Administration document

We’ll do real introductions on Thursday

Originally a required class for a MTID degree, that program has ended but SoA and IDeATe like this class.

What we’re doing differently in a remote class with additional readings and responses/projects.

Syllabus and grading: weekly assignments, three crits, final crit, class participation

Note taking vs. the class blog — will post my notes to the blog after class

Class Theme – “Accessiblity”

Improving the human condition by improving living spaces with tangible interaction design.

Tweaking Nathan Shedroff’s list:

  • Assist
  • Enhance
  • Improve
  • Qualify
  • Sense

Class Theme – Quarantine Quitchen

A few months ago, TV personality Alton Brown and his wife, Elizabeth Ingram, started doing a youtube series called “Quarantine Quitchen”. He’s a famous chef, filmmaker, and host of competition cooking shows, she’s an interior designer who does restaurants. Every Tuesday evening (during our class), they make dinner based on whatever they can find in the fridge, freezer, or pantry. And drink some alcohol. I’m lifting their idea of “living in a quarantine” and changing class assignments to reflect being stuck at home, having limited access to physical social events, and social distancing.

Introduction to Tangible Interaction

Reaction vs. Interaction

Classic thermostat (temperature sensor and on/off switch) vs. smart thermostat (PID controller or AI google hive mind)

Tangent: explain how PID is different from a sensor + relay

Explain how machine learning (open ended system) is different from PID (a closed loop system)


What if we had a smart (AI) thermostat? NEST is on the edge of this.

  • change heating/cooling controls based on history of changes made by people
  • change temperature related to outside environment: tomorrow is hot, turn the AC on earlier
  • react to weather changes: storm front rolls through from the north, temp drops almost 10C
  • modulate temp based on who is in the house: I like it warm, spouse likes it cold
  • modulate temp based on predicted activities: “they always stay up late on Friday”
  • error control: “never let the house go below 50F” to prevent pipes from freezing

What if we had a smart door bell? What would it do? This will be an assignment on Thursday.

Short History of Physical Computing and Interaction Design

Physical computing and tangible interaction design are recently created fields but there is a history of how we got here. The key point is the size (scale) of computing hardware.

    • water powered tools and windmills, literally physical computing
      • beginning of the PID idea, centrifuges to maintain speed in grain mills
      • water mills that stop pumping water when the reservoir is full
    • industrial revolution
      • early punch-card computing to control looms in 1725
      • Korsakov proposes using punched cards to store data in 1832
      • steam engines that can react to malfunctions (more physical computing)
      • sophisticated PID in factories to run mills and lathes
    • (skipping mainframes that only crunched numbers)
    • transistors as second industrial revolution
      • the first computers that didn’t fill buildings
      • 1976: first use of “human-computer interaction” in a published paper
      • 1983: The Psychology of Human-Computer Interaction brings the concept to the general computing community
    • Early arcade games introduced haptic output or feedback: helicopter arcade game that shakes when you are shot (early 1980s)
      • haptic technology:
      • tactor: hardware that conveys a feeling of touch, pressure, or vibration
    • Modems broadband access over the phone
      • Interaction moves from an isolated software package to a network of software packages
      • People can interact with other people at distant locations via BBS or USENET or email
      • People can interact with systems, first online games and dating services
      • Still no physical interaction
    • Mobile phones — getting closer!
      • contains sensors, CPU, network access
      • has output in the form of image, sound, and vibration
    • Arduino, 2005
      • first affordable, usable, embedded controller
      • opened up a market of input/output hardware
      • set the space for Rpi, BBB, etc
    • Five years from now
      • where we’re thinking in this class
      • experimental projects that show new interactions
      • reading assignment is showing us a few more years ahead

What can we do in this class?

Study physical computing and interaction

Look at near future concepts

Design, build, and demonstrate physically interactive devices and systems