Sound in Physical Interactions

There is a music staircase in a mall in my hometown in China. I couldn’t find any videos for that mall, but here is something similar. Stepping on different stairs makes different notes.


This is one of the golf training aids that uses sounds to indicate something is wrong with your swing. This particular example is for if your swing is too long, but there are some other ones that monitor rotational angles and separation between upper and lower body.

This one does not have much interaction but I just wanted to put it here. Japanese singing toilets were very interesting…

Sound in Physical Interactions

  1. VR Art installation by Ray McClure

2. Microsoft Soundscape – Map delivered in 3D sound

Link for more information

3. Some interesting links:

Deep learning monitors human activity based on sound alone, able to parse whether a person in a room is sitting, standing, walking or falling.

Apple’s approach to harmonize sound with physical interactions by thinking of them as audio-haptic experiences

Sound in Physical Interaction Examples

Korean rice cooker: when the rice is ready, the voice says “Delicious rice is ready to be served” and the lid automatically opens. Based on time/water level one sets, the rice cooker serves the most ideal type of rice.

Theft alarm in stores: When people walk out with a product without paying, the alarm detects it and makes a loud noise.

Automatic Cat feeder at my house: whenever it is user’s set time, the voice recording of me saying “time to eat kitties!”(which I trained my cats to come and eat when I say this) plays automatically, and the chamber inside the feeder rotates and sends out cat food.

Not related to this topic, but I also made something similar in my last semester’s class, which is an automatic feeder that detects cats movement and also a cat exerciser.

Sound in Physical Interaction

Secret Knock Detecting Door Lock:

This device can recognize a secret knock pattern that’s up to 20 knocks long and unlock the door if it is correct! Instructions on how to make it can be found here.

The world’s deepest bin

This trash can makes it sound like garbage thrown in is taking a really long time. It made throwing trash away more fun and even got some people to pick up litter.

Urban Lights Contacts:

This interactive sound installation changes the sound output based on electrostatic connections; in other words, it will make varying sounds from how close people are and if people are touching. This is a really cool concept, but definitely something that would give me anxiety to try because of COVID.

Example: data over time to interact with

GIS: “A geographic information system (GIS) is a framework for gathering, managing, and analyzing data. Rooted in the science of geography, GIS integrates many types of data. It analyzes spatial location and organizes layers of information into visualizations using maps and 3D scenes. ​With this unique capability, GIS reveals deeper insights into data, such as patterns, relationships, and situations—helping users make smarter decisions. ”

There are some examples in the link where GIS is used for monitor changes and forecasting.

Kinetic Interaction Examples

This swing analyzer is put on golf gloves to sense and analyze golf swings  by detecting motions.


This smart night light automatically lights up when someone walk by it within a certain distance, and it turns off either after a set period of time after the person leaves.

An alarm clock that is turned off only after the person gets off the bed and stands on the pad.

Kinetic Interaction Examples

  1. Games that make use of device gyroscope / accelerometer for input control
  2. Project Soli – close range radar for fine motor controls w/o physical hardware
  3. Ultrahaptics – feeling without touching; providing haptic feedback through ultrasound
  4. Shape changing controller based on drag force experienced
  5. Theremin
  6. Text rain
  7. Posture training device
  8. Dynamic VR display (Rhizomatiks Research)

Kinetic Interaction Examples

These “music gloves” by Imogen Heat help users to create music much more seamlessly and in the moment than using a keyboard or soundboard. By just moving hands up and down, left and right, tilting, pinching fingers, and pointing the user can change the volume, pitch, tone, and filtering.


This “Kinetic Wall” by Cupra changes shape so that where the user is looking moves from wall panels to windows to glimpse behind the wall.

This “Bloomframe” window at the push of a button will transform from a window to an open air balcony.

The”Sharifi-ha House” is a house that will open or close rooms based on the season/temperature. When it is warm out, the rooms will be very open and get a lot of natural light and air while when it is cold, the house will close itself so that the minimal air is leaked. It can also be changed by users choice, but the website didn’t really specify this.

State machines

A. Ambient Umbrella

It glows when it is going to rain to remind the owner to take it out.

B. Cube

Cube is an intuitive, tangible interface for smart homes. By touching, lifting, tilting and turning, Cube allows users to adjust technological appliances such as lighting, temperature and music using a single interface.

C. Water Pilly

Water Pilly constitutes a mechanical flower that opens and closes to offer pills to people suffering from humidity-related asthma.