Make it So Ch 3, 4, 5 – James Kyle

Thoughts:

I thought the conversation about virtual projections was especially interesting given the state of virtual reality in the modern tech world. I think it is interesting to think about real world applications of virtual projections and I think things like the Metaverse are the closest thing we will get in the near future. I think it’s also interesting to think about the application of virtual reality type equipment not only in virtual environments but also in exploring real world places and things. With haptic feedback and camera technology nowadays, I think it is realistic to think that the next step of virtual projection type technology is not only visual but also touch based.

 

The conversation about color was also interesting to me because giving meaning to colors as an engineer is not something I think about everyday but affects the way I interact with my projects. For example, the color of wires in electrical diagrams plays a big part in how easy they are to understand. Using red wires for everything connecting to power and black wires for everything connecting to ground is something that is second nature to me having worked with electrical components for a while but it seems like an arbitrary decision. There are lots of other color standards for indicating signal types flowing through specific wires but it’s kind of interesting to think about the origin of those color schemes similar to the reasons why blue displays signify futuristic tech.

 

 

Doorbell for the Blind

Concept:

The concept for this device was to create a doorbell that could be used navigated and used with little to no eyesight required. My goal was to eliminate the component of doorbell interactions where the user has to navigate on the door to find the button and ring the doorbell. I ended up settling on a doormat that senses when a person is present and then greets the user and prompts them to stomp a specified amount of times for desired interactions. The actions were made to resemble those of a traditional answering machine: stomp once to hear the address, stomp twice to ring the doorbell, stomp three times to hear the options again. Going further into the design, I think it would be helpful to add a message component so the user could leave a message to the resident if they want. I think it would also be helpful for security sake to incorporate a notification system to the resident is notified of any users on their doorstep and a picture is captured of whoever is there.

 

Electrical Schematic:

 

Demo Video:

 

Code:

/*
   Warning Machine for Something Approaching
*/

#include <Audio.h>
#include <Wire.h>
#include <SPI.h>
#include <SD.h>
#include <SerialFlash.h>

AudioPlaySdWav           playWav;

AudioOutputMQS           mqs1;           //xy=625,279
AudioConnection          patchCord1(playWav, 0, mqs1, 0);
AudioConnection          patchCord2(playWav, 1, mqs1, 1);


// Use these with the Teensy 3.5 & 3.6 SD card
#define SDCARD_CS_PIN     BUILTIN_SDCARD
#define SDCARD_MOSI_PIN   11  // not actually used
#define SDCARD_SCK_PIN    13  // not actually used

#define pressurePlate     14



String message = "";
int stompCounter = 0;


void setup() {
  // put your setup code here, to run once:

  Serial.begin(9600);

  // Audio connections require memory to work.  For more
  // detailed information, see the MemoryAndCpuUsage example
  AudioMemory(8);

  // Comment these out if not using the audio adaptor board.
  // This may wait forever if the SDA & SCL pins lack
  // pullup resistors

  SPI.setMOSI(SDCARD_MOSI_PIN);
  SPI.setSCK(SDCARD_SCK_PIN);
  if (!(SD.begin(SDCARD_CS_PIN))) {
    // stop here, but print a message repetitively
    while (1) {
      Serial.println("Unable to access the SD card");
      delay(500);
    }
  }


  pinMode(pressurePlate, INPUT);


}

void loop() {
  // put your main code here, to run repeatedly

  bool someoneOnPressurePlate = doPlatePresenceCheck();

  if (someoneOnPressurePlate) {
    doPressurePlateCheck();
  }

  
  Serial.print("Pressure Plate Reading: ");
  Serial.print(analogRead(pressurePlate));
  Serial.print("\t");
  Serial.print("Stomp Counter: ");
  Serial.print(stompCounter);
  Serial.print("\t");
  //  Serial.print("Pressure Plate Pressence: ");
  //  Serial.print(someoneOnPressurePlate);
  //  Serial.print("\t");

  Serial.print("Message: ");
  Serial.print(message);
  Serial.print("\t");
  Serial.println("");
  delay(5);


}

void playSound(const char *filename) {
  playWav.play(filename);
  delay(25);
  while (playWav.isPlaying()) {

  }
}

bool doPlatePresenceCheck() {

  int standingThreshold = 600;
  int standingTimeThreshold = 2000; // [ms]
  static bool timingStand = false;
  static bool someoneOnPressurePlate = false;
  static unsigned long standingTime = 0;


  int plateReading = analogRead(pressurePlate);

  if (plateReading > standingThreshold) {
    if (!timingStand) {
      timingStand = true;
      standingTime = millis();
    }
  } else if (plateReading < standingThreshold) {
    timingStand = false;
    someoneOnPressurePlate = false;
  }

  if (timingStand && millis() - standingTime > standingTimeThreshold) {
    if (!someoneOnPressurePlate) {
      playSound("Initial Greeting.WAV");
      playSound("Options Sound.WAV");
    }
    timingStand = false;
    someoneOnPressurePlate = true;
  }

  return someoneOnPressurePlate;

}


void doPressurePlateCheck() {

  int debounceTime = 300;
  int stompThreshold = 900;
  int countTimerThreshold = 3000; // [ms]
  static bool DEBOUNCE = false;
  static bool timeDebounce = false;
  static bool countStart = false;
  static unsigned long debounceTimer = 0;
  static unsigned long countTimer = 0;

  int pressureReading = analogRead(pressurePlate);


  if (!DEBOUNCE && pressureReading > stompThreshold) {
    if (!countStart) {
      countStart = true;
      countTimer = millis();
    }

    DEBOUNCE = true;
    stompCounter += 1;
  }

  if (millis() - countTimer > countTimerThreshold || stompCounter > 3) {
    actionBasedOnStompCount(stompCounter);
    countStart = false;
    stompCounter = 0;
  }


  if (DEBOUNCE && timeDebounce) {
    timeDebounce = false;
    debounceTimer = millis();
  }

  if (millis() - debounceTimer > debounceTime) {
    DEBOUNCE = false;
    timeDebounce = true;
  }

}


void actionBasedOnStompCount(int stompCount) {
  if (stompCount == 1) {
    //user wants to read address
    message = "Read Address";

    playSound("Address Sound.WAV");
    
  } else if (stompCount == 2) {
    //user wants to ring doorbell
    message = "Ring Doorbell";

    playSound("Doorbell Sound.WAV");
    
  } else if (stompCount == 3) {
    //user wants to leave message
    message = "Reading Options Again";

    playSound("Options Sound.WAV");

  }
}

 

Class Notes, 5 Apr 2022

Discuss readings

VR Headsets and vision problems:

https://www.bbc.com/news/technology-52992675

https://www.aao.org/eye-health/tips-prevention/are-virtual-reality-headsets-safe-eyes

Sand Tables and Interaction:

Sand Noise Device – please watch

A History of Sand Tables – this is long, but if you’re interested in how to visualize things in 3D this is a good starting point.

A brief aside on visual skeuomorphism

A skeuomorph is a object derived from past objects and contains past visual cues that have no meaning in current culture.  Ex: the “phone” icon on my Android looks like the physical handsets we stopped using in the 90s.

Think of a derivative object that retains ornamental design cues (attributes) from structures that were inherent to the original. Examples include pottery embellished with imitation rivets reminiscent of similar pots made of metal and a software calendar that imitates the appearance of binding on a paper desk calendar.  Or the “save” icon that looks like a floppy drive, that a Japanese student thought looked like a drink dispenser:

“Why is the Excel save mark a vending machine? “

When you are creating visual representations, are you using your culture’s history of symbolism?

States of being, replacing sound with visuals

Visualizing sound

Visualizing music with images

ASL vs Braille

ASL replaces a word with a motion while Braille replaces each letter with a pattern with a shape similar to the letter.  Before Braille, books for the blind used large raised letters that you would trace by hand.

Early experiments with live animation and MIDI inputs with devices manipulated by the audience.

Console displays in video games are a different way to display information.  This is metadata about your status.  health, money, power, speed.

Wipeout series brought commercial racing context to a SF racing game.  Music for Wipeout XL was mostly bands who were played or performed in rave spaces.  There were in-game billboards for real products.  The visuals were designed by The Designers Republic, a well-known design firm.

Rez was a completely different video game and hard to explain.  The controls are simple, but as you play you dynamically create new visual and audio structures.

Cultural Contexts

What colors mean in US culture vs. Asiatic, Arabic, African countries

ex: white wedding dress or white funeral dress?

colors important because they show wealth: purple for royalty, gold.  There were “dyer guilds” that kept secret their methods for creating a dye.

Color of money — the US is one of the few countries where all denominations are the same size and same color.

Color of food:  In the south we eat “dirty” rice, or ask for “debris” on our po-boy.

Reading/Viewing assignment while you plan your crit project

Walk This Way navigational guidance

Deaf Architecture

How quickly can you change your appearance in public?  Same retired-CIA agent comments on disguise in movies.  The second one is a good guide for understanding what you’re seeing in entertainment.

Reading body language.

Make It So Chapters 3, 4, & 5

Chapter 3:

Throughout the chapter, there were three things that were portrayed as critical to an interface’s functionality and perception. The first of these categories was font and its use to convey information through a screen. Font is super important because it conveys information to the user through it’s appearance. Fonts that are more block-like and all uppercase tend to be harder to read and mimic early style computers making the user seem more intelligent or knowledgeable. In addition, fonts that have more character like serif fonts are more readable and can be used to convey very important information that needs to be read. The size of the font is also important providing insight into the importance of the information. All of these characteristics add up to convey information about the importance and urgency of the words being displayed without the user even having to read them.

In addition to font, color is also stressed as a very important visual aspect when it comes to displays. One aspect mentioned throughout chapter 3 is the idea of the blue and glow phenomena that conveys futuristic text. This is interesting because whenever I hear blue and glow I think of hologram which is very futuristic highlighting the cultural implications of this color. In addition, colors like red on a display signify more urgent or error-like messages. While this isn’t always the case, red is described as having a sense of danger. This is an interesting interpretation because red in other cultures also signifies good luck or happiness which is generally thought of to be the opposite of danger. This difference in color perception is very interesting because it brings up the idea that it is pretty much impossible to create a perfectly perceptible user interface. While we can come to a general consensus on what means what, it is very hard to create a display that means the same thing regardless of culture especially when it comes to color schemes.

Finally, in addition to color and font size, one large aspect of displays is the user interaction through buttons or cursors. One example brought up in chapter 3 is the Star Trek LCARS interface that uses shadows to give the digital buttons on the screen a sort of depth and animation. This not only makes them feel more life-like but also gives feedback as to whether the button was pressed or not. These types of visual cues that are missed when translating from physical to digital interfaces are key in user interaction and is something that can contribute to a device being amazing or completely unusable.

Chapter 4:

Volumetric projections are very good at portraying information in such a way that humans are comfortable with. Since we see the world in 3D, VP’s allow the user to interact with the information in such a way that mimics their natural perspective. Where this can become an issue, however, is in the realness of the projection. As discussed in chapter 4, VP’s can be very tricky if they are made too realistic because the user might actually think there is something there when in reality there isn’t. While this isn’t all bad for most applications, when something that has to be physical for the safety of the user is made virtual (like a step or floor) this can create circumstances ready for injury.  In addition to safety and trickery, VP’s also have to portray the 3D information in such a way that looks natural and mimics the proportions of regular space. Otherwise the information might seem very abnormal and not as informative.

Chapter 5:

Gestural interfaces are very intuitive for the most part but also have the caveat of moving the entire body to do one single action. While there are seven distinct motions set by physical intuition but also movies and TV, these motions require sustained use of the arms at heights around heart level requiring lots of strength and endurance to keep up. This results in the user getting very tired after a small amount of use. So while gestural interfaces might seem cool, they are mostly just used for futuristic looking interfaces.

Good and Bad Visual Feedback

Doors:

Many doors that I have encountered are very poorly designed in that they don’t indicate which direction they are supposed to be opened or where they are hinged. While this might look good from an aesthetics stand point, it hinders the ability to quickly recognize which side of the door to push or pull and whether or not to push or pull. One example of a very poorly designed door are those entering the UC from the football field entrance. While they do a good job of showing where the door hits a stop and thus whether it will be a push or pull open, they hide their hinges very well and have no physical indicators to persuade users to one side of the door or another. This has on many occasions resulted in me walking right into the door and being immediately brought to a halt because it’s the wrong side. On doors with a turn handle or ones with a push to open bar that has a direction to is, the side to push or pull from is made much more clear.

Assignment: Visual

The assignment is two parts- one, reading chapters 3-5 in Make It So, and two, examples of good and bad visuals.

 

Readings

Chapter 3: Visual Interfaces

The authors discuss visuals in the context of user interfaces and controls, identifying cinematic cliches such as the heavy use of blue glow, transparencies, and circular screens to suggest futuristic technology. The drivers coming from the needs of film include post-production work, legibility (movie watchers can only read so much text on a screen so fast), and narratives (communicating additional information about the scenario or cultures of of the users, dramatic tension, attractive/entertaining visuals for audiences).

Chapter 4: Volumetric Projection

The volumetric projection, aka projections of massless, moving 3d imagery, is used as a futuristic means of communicating, despite various practical drawbacks (need for stealth/privacy, and need for sorting out eye contact).. Authors address that 2d information representation may be better than 3d and challenges with VP, including that past VP remains very similar across time, and that deviating from the tried-and-true qualities may lead to confusing and failure to fulfill audience expectations.

Chapter 5: Gesture

The authors discuss examples of using gestures to provide input  to a system and challenges associated with their use, given that gestural interface systems need to understand intent and need to be able to be manipulated to do a variety of tasks, many of which are abstract. Minority Report, which we discussed in class, was referenced. The authors list 7 major gestures used in Hollywood, which are intuitive to our understanding of working with tactile objects in the real world (the book says pinch and spread for scaling has no physical analog, but it is very commonly used today for the manipulation of touchscreens). They discuss the limitations of a ‘natural’ user interface working off of gestures- also, in class, we discuss how gestures can differ between cultures, even for gestures that we may consider ‘natural.’

Good and Bad Visuals

Unusual Stop Sign: On my commute, there is a stop sign at a mostly L-intersection that had a second sign beneath it saying “Except for Right Turn” – meaning that those approaching the  T can turn. It is located at a unique condition, where the L-intersection has a fenced off road to one side (which gives it the look of a T-intersection). Given that the area is a bottleneck, this sign encourages that traffic continues to flow, especially since it seems the majority of people do turn right. I have thought it was weird since I first saw it, but I cannot think of a better way to communicate to drivers. It is good to show the STOP, in case the driver is turning left. Then, as the driver prepares to stop, they read “except for right turn” and those turning right keep moving.

3d or 2d: We had a seminar presentation about modelling hazards and infrastructural risks in the event of natural disasters. One student asked the presenter about 3d modelling and topography, and the presenter showed an example of using 3d, but also said that when one works with data in 3d (4d, really, if we are considering time as an integral dimension to these predictive models), the computational lag is so great that it cannot perform under the tight time constraints sometimes needed for natural disaster risk assessment and disaster response. Therefore, working in 3d may actually not be suitable or appropriate, depending on the scope and scale of the problem.

Class Notes, 31 Mar 2022

Start Visual

Visual feedback

The less types and amount of visual feedback you give the better

Ex: an analog clock needs hours and minutes, but does it need seconds?  Microseconds?  Centuries?  Do digital clocks (ex: alarm clocks) display seconds?

What do the clocks in my kitchen need to show me?  Why are there so many clocks that aren’t in sync?  I have:

  • Zojirushi water boiler on the counter.
  • Over the window/sink, vintage analog clock stuck at 5:30
  • Zojirushi rice maker
  • iPad (we use for music in the kitchen)
  • Microwave
  • Stove

If the power goes out I have to reset each of them, there’s no external clock.  This clock costs $20 and sinks with the US atomic clock radio broadcast.

Another watch with no second hand and no minute marks.  Originally designed for the blind, but now a popular visual style.

Did Susan Kare invent emoji?  Her first icon designs for the Mac were done on a square grid as she knew needlepoint.  She went on to do graphic design for General Magic.

Visual states

I look at these fundamental types of visual state:

  • color
  • motion
  • intensity

The type of display is also important: led, a light, a screen, a moving object in 3 dimensions, a swinging/flashing metronome.

Complex visual states

  • typeface
  • language
  • icons
  • images

Weekend assignment in two parts

First, please read chapters 3, 4, and 5 of Make It So.

Second, look around your world and find some good and bad examples of visual display of information.  Your home, your car, public transit, etc.    Example:  I think my Eone Bradley watch changed how I think about time.  I can only tell time to “about minutes”, it’s “about 4:55” or “about 4:15”.  This has changed how I think about time being analog, not digital like on my phone.

Crit 2: Sound

This project uses a color sensor (TCS34725) to determine whether or not a banana is ripe. The sketch includes a button for the user to press when ready to evaluate the banana. One series of beeps indicates the banana is ready to eat, another series indicates the banana is not ready to eat. Since there is a data smoothing component, there is also a noise that identifies when new readings are dramatically different from the average reading, and notifies the user that the sensor is still calculating.

#include <Wire.h>
#include "Adafruit_TCS34725.h"

/* Example code for the Adafruit TCS34725 breakout library */

/* Connect SCL    to analog 5
   Connect SDA    to analog 4
   Connect VDD    to 3.3V DC
   Connect GROUND to common ground */

/* Initialise with default values (int time = 2.4ms, gain = 1x) */
Adafruit_TCS34725 tcs = Adafruit_TCS34725();

//* Initialise with specific int time and gain values *
//Adafruit_TCS34725 tcs = Adafruit_TCS34725(TCS34725_INTEGRATIONTIME_500MS, TCS34725_GAIN_1X);

int speakerpin=8;
int buttonpin=2;


// data smoothing
const int numReadings = 10;

int readings[numReadings];      // the readings from the analog input
int readIndex = 0;              // the index of the current reading
int total = 0;                  // the running total
int averageb = 0;                // the average


void setup(void) {
  Serial.begin(9600);
  for (int thisReading = 0; thisReading < numReadings; thisReading++) {
    readings[thisReading] = 0;
  }

  pinMode(buttonpin,INPUT_PULLUP);

  if (tcs.begin()) {
    Serial.println("Found sensor");
  } else {
    Serial.println("No TCS34725 found ... check your connections");
    while (1);
  }

  // Now we're ready to get readings!
}

void loop(void) {
  uint16_t r, g, b, c, colorTemp, lux;

  tcs.getRawData(&r, &g, &b, &c);
  // colorTemp = tcs.calculateColorTemperature(r, g, b);
  colorTemp = tcs.calculateColorTemperature_dn40(r, g, b, c);
  lux = tcs.calculateLux(r, g, b);


    // subtract the last reading:
  total = total - readings[readIndex];
  // read from the sensor:
  readings[readIndex] = b;
  Serial.println(b);
  // add the reading to the total:
  total = total + readings[readIndex];
  // advance to the next position in the array:
  readIndex = readIndex + 1;

  // if we're at the end of the array...
  if (readIndex >= numReadings) {
    // ...wrap around to the beginning:
    readIndex = 0;
  }

  // calculate the average:
  averageb = total / numReadings;
  // send it to the computer as ASCII digits
  Serial.println(averageb);
  delay(1);        // delay in between reads for stability



  Serial.print("Color Temp: "); Serial.print(colorTemp, DEC); Serial.print(" K - ");
  Serial.print("Lux: "); Serial.print(lux, DEC); Serial.print(" - ");
  Serial.print("R: "); Serial.print(r, DEC); Serial.print(" ");
  Serial.print("G: "); Serial.print(g, DEC); Serial.print(" ");
  Serial.print("B: "); Serial.print(b, DEC); Serial.print(" ");
  Serial.print("C: "); Serial.print(c, DEC); Serial.print(" ");
  Serial.println(" ");
delay(100); delay(100);   delay(100);   

  int buttonvalue=digitalRead(buttonpin);
  Serial.println(buttonvalue);

  if (buttonvalue == HIGH) {
    Serial.println("button is pressed");
    if (averageb>=20){
      Serial.println("not ready");
      tone(speakerpin,293.66,1000);
      delay(1000);
      noTone(speakerpin);
      delay(500);
      tone(speakerpin,293.66,1000);
      delay(1000);
      noTone(speakerpin);
    } else if (b-averageb>=10) {
      Serial.println("calculating");
      tone(speakerpin,200,500);
      delay(500);
      noTone(speakerpin);
      delay(500);
    } else {
      Serial.println("ready to eat");
      tone(speakerpin,329.63,250);
      delay(250);
      noTone(speakerpin);
      delay(250);
      tone(speakerpin,311.13,250);
      noTone(speakerpin);
      delay(1000);
      tone(speakerpin,329.63,1000);
      noTone(speakerpin);
      delay(1000);
      }

  } else {
    Serial.println("button is not pressed");
  }





}

 

Room Indicator for the Blind

Description:

The purpose of this device is to assist people in locating themselves when they enter a new room in their home or other location. The basic idea is that when someone enters a room, a speaker will relay what room they are entering. This is accomplished by tracking the user’s position within a house through IR break beam sensors. One set of IR break beams is installed at each doorway or opening where the user can transition from room to room. When a break is detected in one of the beams, the system checks the current sensor being tripped against the previous room the user was in and adjusts the current room accordingly. Additionally, each room has its own sound that plays indicating to the user what room they are entering upon walking in. Finally, the system sends the current room number to a p5.js script that shows a visual layout of the floor plan with the current room highlighted in green.

Images:

Videos:

Demo of physical build

Code:

 

/*
   Crit 2: Location Feedback w/Sound
   Judson Kyle
   judsonk

   Description:
*/

#include <Audio.h>
#include <Wire.h>
#include <SPI.h>
#include <SD.h>
#include <SerialFlash.h>

AudioPlaySdWav           playWav1;
AudioMixer4              mixer1;         //xy=647,408
AudioOutputMQS           mqs1;           //xy=625,279
AudioConnection          patchCord1(playWav1, 0, mqs1, 0);
AudioConnection          patchCord2(playWav1, 1, mqs1, 1);

#define SDCARD_CS_PIN    BUILTIN_SDCARD
#define SDCARD_MOSI_PIN  11  // not actually used
#define SDCARD_SCK_PIN   13  // not actually used

#define TRIGGER0   17
#define TRIGGER1   18
#define TRIGGER2   19
#define TRIGGER3   20
#define TRIGGER4   21
#define TRIGGER5   33
#define TRIGGER6   33
#define TRIGGER7   33
#define TRIGGER8   33
#define TRIGGER9   33
#define TRIGGER10  33
#define TRIGGER11  33
#define TRIGGER12  33

const char* outside = "Outside.WAV";
const char* sunRoom = "Sun Room.WAV";
const char* livingRoom = "Living Room.WAV";
const char* bed1 = "Bedroom 1.WAV";
const char* bed2 = "Bedroom 2.WAV";
const char* bed3 = "Bedroom 3.WAV";
const char* kitchen = "Kitchen.WAV";
const char* laundryRoom = "Laundry Room.WAV";
const char* diningRoom = "Dining Room.WAV";
const char* bath1 = "Bathroom 1.WAV";
const char* bath2 = "Bathroom 2.WAV";
const char* hall1 = "Hallway 1.WAV";
const char* hall2 = "Hallway 2.WAV";

//char *roomNames[13] = { outside,    sunRoom,      livingRoom,
//                        bed1,       bed2,         bed3,
//                        kitchen,    laundryRoom,  diningRoom,
//                        bath1,      bath2,        hall1,
//                        hall2};

int triggerPins[13] = { TRIGGER0,   TRIGGER2,   TRIGGER3,   TRIGGER4,
                        TRIGGER4,   TRIGGER5,   TRIGGER6,   TRIGGER7,
                        TRIGGER8,   TRIGGER9,   TRIGGER10,  TRIGGER11,
                        TRIGGER12
                      };
int numTriggers = 13;
int currTriggerOn = 0;
int prevTriggerOn = -1;
int prevRoom = 0;
volatile int currRoom = 0;

unsigned long currTime = 0;
unsigned long debounceTime = 100;
volatile unsigned long startTime = 0;
volatile bool DEBOUNCE = false;

const char* currFileName = outside;

void setup(void)
{
  // Wait for at least 3 seconds for the USB serial connection
  Serial.begin (9600);

  AudioMemory(8);

  pinMode(TRIGGER0, INPUT);
  pinMode(TRIGGER1, INPUT);
  pinMode(TRIGGER2, INPUT);
  pinMode(TRIGGER3, INPUT);
  pinMode(TRIGGER4, INPUT);

  SPI.setMOSI(SDCARD_MOSI_PIN);
  SPI.setSCK(SDCARD_SCK_PIN);
  if (!(SD.begin(SDCARD_CS_PIN))) {
    // stop here, but print a message repetitively
    while (1) {
      Serial.println("Unable to access the SD card");
      delay(500);
    }
  }

}

void loop (void)
{
  currTime = millis();

  if (!DEBOUNCE && !playWav1.isPlaying()) {
    updateCurrRoom();
    DEBOUNCE = true;
    startTime = currTime;
  }

  if (currTriggerOn != prevTriggerOn) {
    updateCurrRoom();
  }

  if (prevRoom != currRoom) {
    if (playWav1.isPlaying()) {
      playWav1.togglePlayPause();
    }
    playWav1.play(currFileName);
    Serial.println(currRoom);
  }

  prevRoom = currRoom;
  prevTriggerOn = currTriggerOn;

  //Update debounce state
  DEBOUNCE = abs(currTime - startTime) < debounceTime;
}

void updateCurrRoom() {
  switch (currRoom) {
    case 0:
      if (digitalRead(TRIGGER0) > 0) {
        currRoom = 1;
        currFileName = sunRoom;
      }
      break;
    case 1:
      if (digitalRead(TRIGGER0) > 0) {
        currRoom = 0;
        currFileName = outside;
      }
      else if (digitalRead(TRIGGER1) > 0) {
        currRoom = 2;
        currFileName = livingRoom;
      }
      break;
    case 2:
      if (digitalRead(TRIGGER2) > 0) {
        currRoom = 8;
        currFileName = diningRoom;
      }
      else if (digitalRead(TRIGGER1) > 0) {
        currRoom = 1;
        currFileName = sunRoom;
      }
      break;
    case 3:
      if (digitalRead(TRIGGER3) > 0) {
        currRoom = 8;
        currFileName = diningRoom;
      }
      break;
    case 8:
      if (digitalRead(TRIGGER2) > 0) {
        currRoom = 2;
        currFileName = livingRoom;
      }
      else if (digitalRead(TRIGGER4) > 0) {
        currRoom = 9;
        currFileName = bath1;
      }
      else if (digitalRead(TRIGGER3) > 0) {
        currRoom = 3;
        currFileName = bed1;
      }
      break;
    case 9:
      if (digitalRead(TRIGGER4) > 0) {
        currRoom = 8;
        currFileName = diningRoom;
      }
      break;
    default:
      break;
  }
}

 

let serial;
let latestData = "waiting for data";
var outData;

var minWidth = 600;   //set min width and height for canvas
var minHeight = 400;
var width, height;    // actual width and height for the sketch
var squareWidth = 100;

var greenColor = '#64d65d';
var redColor = '#d6220c';

var currRoom = 0;

class Room {
  constructor(length, height, xOffset, yOffset, color, name) {
    this.scale = 40;
    this.length = length*this.scale;
    this.height = height*this.scale;
    this.xOffset = xOffset*this.scale;
    this.yOffset = yOffset*this.scale;
    this.color = color; 
    this.name = name;
  }
  getX(canvasWidth) {
    return (canvasWidth - 12*this.scale)/2 + this.xOffset;
  }
  getY(canvasHeight) {
    return (canvasHeight + 17*this.scale)/2 - this.yOffset - this.height;
  }
}

let sunRoom = new Room(6, 2, 0, 0, redColor, "Sun Room");
let livingRoom = new Room(6, 5, 0, 2, redColor, "Living Room");
let dinningRoom = new Room(6, 4, 0, 7, redColor, "Dinning Room");
let bed1 = new Room(6, 7, 6, 0, redColor, "Bedroom 1");
let bed2 = new Room(6, 5, 6, 11, redColor, "Bedroom 2");
let bed3 = new Room(5, 3, 0, 11, redColor, "Bedroom 3");
let kitchen = new Room(4, 3, 2, 14, redColor, "Kitchen");
let laundryRoom = new Room(2, 1, 0, 16, redColor, "Laundry Room");
let bath1 = new Room(5, 4, 7, 7, redColor, "Bathroom 1");
let bath2 = new Room(2, 2, 0, 14, redColor, "Bathroom 2");
let hall1 = new Room(1, 4, 6, 7, redColor, "Hall 1");
let hall2 = new Room(1, 3, 5, 11, redColor, "Hall 2");
let outside = new Room(0, 0, 0, 0, redColor, "Outside");


var rooms = [ outside, sunRoom,  livingRoom, 
              bed1,     bed2,        bed3, 
              kitchen,  laundryRoom, dinningRoom,
              bath1,    bath2, 
              hall1,    hall2];

var scale = 10;


function setup() {
 // set the canvas to match the window size
 if (window.innerWidth > minWidth){
  width = window.innerWidth;
  } else {
    width = minWidth;
  }
  if (window.innerHeight > minHeight) {
    height = window.innerHeight;
  } else {
    height = minHeight;
  }

  originX = width/2 - 5.5*scale;
  originY = height/2 - 8.5*scale;

  //set up canvas
  createCanvas(width, height);
  noStroke();

 serial = new p5.SerialPort();

 serial.list();
 serial.open('/dev/tty.usbmodem114760901');

 serial.on('connected', serverConnected);

 serial.on('list', gotList);

 serial.on('data', gotData);

 serial.on('error', gotError);

 serial.on('open', gotOpen);

 serial.on('close', gotClose);
}

function serverConnected() {
 print("Connected to Server");
}

function gotList(thelist) {
 print("List of Serial Ports:");

 for (let i = 0; i < thelist.length; i++) {
  print(i + " " + thelist[i]);
 }
}

function gotOpen() {
 print("Serial Port is Open");
}

function gotClose(){
 print("Serial Port is Closed");
 latestData = "Serial Port is Closed";
}

function gotError(theerror) {
 print(theerror);
}

function gotData() {
  let currentString = serial.readLine();
  trim(currentString);
  if (!currentString) return;
  console.log(currentString);
  latestData = Number(currentString);
  currRoom = latestData;
  for (i = 0; i < rooms.length; i++) {
    rooms[i].color = redColor;
  }
  rooms[currRoom].color = greenColor;
}

function draw() {  
  background(0);

  //Draw canvas
  strokeWeight(4);
  stroke('white');
  fill('black');
  noStroke();

  fill(255);
  textSize(16);
  text("House Map", 30, 30);
  text("Current Room: " + rooms[currRoom].name, 30, 50);    // displaying the input

  //Draw Rooms
  strokeWeight(4);
  stroke('black');
  // fill(redColor);
  for (i = 0; i < rooms.length; i++) {
    fill(rooms[i].color);
    rect(rooms[i].getX(width), rooms[i].getY(height), rooms[i].length, rooms[i].height);
  }

}

function mousePressed() {
  var currX = mouseX;
  var currY = mouseY;

  for (i = 0; i < rooms.length; i++) {
    var roomX = rooms[i].getX(width);
    var roomY = rooms[i].getY(height);

    var check1 = (currX > roomX) && (currX < roomX + rooms[i].length);
    var check2 = (currY > roomY) && (currY < roomY + rooms[i].height);
    
    if (check1 && check2) {
      currRoom = i;
      rooms[i].color = greenColor;
    }
    else {
      rooms[i].color = redColor;
    }
  }
  
}

function mouseDragged() {

}

Electrical Schematic:

 

Piano Visualizer: Combining Sound and Visualization

These have become popular on my Youtube suggestions list:

There are multiple resources for piano visualization. They can be mesmerizing to watch. They are also used for interactive, remote piano instruction. When I started playing piano again after about 15 years, I found that videos were more helpful than sheet music (which I had to re-remember how to read).

https://piano-visualizer.com/home