What We Built

We created a braille alarm clock. The alarm clock is fully functional by someone who is both without sight and hard of hearing. The user can check and reset the current time, as well as set the time for when they’d like their alarm to go off, and it will reliably do so.

Final clock with bed shaker and original alarm clock on presentation day

Top interface of clock with 12 red buttons in a circle for the clock, and 2 red buttons on the left (alarm and clock), and 2 green buttons (set and clear)

Right side of prototype with solenoid and rocker switch

Back of clock with acrylic flower details and power plug

Additional flower acrylic details at the front

Alycia testing out our final device. (Seated to the right of the a table is Alycia, a middle-aged woman with long hair. She is pressing buttons on a pink box that acts as a clock with brailled on it)

Narrative Sketch

Alicia has a big day tomorrow and wants to get an early start in the morning. She goes to her alarm clock, presses the arcade button marked Alarm and then shifts her hand to the clock buttons. She presses the fifth button, and then the sixth button: 5:30.She then presses a different button market Set. Just to double check that she didn’t make a mistake, she presses the Alarm button once more. Now the exciting part… she shifts her hand to the right wall of the clock, feeling for a circle opening with felt under it. Two of her fingers lightly rest on the felt and she waits. Tap, Tap, Tap, Tap, Tap. 5. There’s a short pause. Tap, Tap, Tap. 3. There’s a much pause. 0. A quick Tap-Tap. Knowing that the end of the sequence is always marked by the Tap-Tap, she goes to bed content.

Come morning time, the bed shaker violently rocks her bed waking her up. She gets out of bed, goes to the alarm clock and switches the switch from ON to OFF. The bed shaker stops immediately. She wants to check the time just to ensure she didn’t miss anything in her day, so she presses Clock and moves her fingers to the felt again. 5 taps, 3 taps, and 1 tap. 5:31am. She nods with satisfaction and starts her day.

How We Got Here (Prototype and Process)

This prototype was designed to help answer the design question: How do we make an alarm clock for a blind person that is hard of hearing?

Prototype

First meeting discussion possible ideas

Second meeting discussing clock ideas

Final 3 ideas for prototyping

Prototype 1: Two clocks where each number is a button. The first clock would be for setting the hours, and the second clock would be for setting the minutes.

Prototype 2: A finger slider, where you put your finger on a slider and the top of your finger would slide across different braille numbers. When you got to the number you wanted first, you would press set, and that would set the hours. Then for the minutes, you would slide to the number you wanted and that would set the minutes.

Prototype 3: The third idea is similar to a rotary phone where you would put your finger in a small hole and circle around all the numbers to the number you were looking for. In each number spot would be braille that the user can read, and they would use the rotary system to set the hours and then minutes of their alarm.

Showing Alycia the prototypes on prototype day

Our 3 prototype interfaces

testing prototypes with Alycia

Alycia showing us how to make braille on dymo tape
(Alycia is using a poking device and a braille grid to press indentations onto sticky tape)

All of our prototypes were feasible, yet how we moved forward was solely to the discretion of our client and what her personal preference was in terms of the alarm. She chose Prototype 1, where the interface is all buttons in a circle.

We were a bit surprised that the client chose the option she did. When she went over her reasoning, she explained that practicality and ease-of-use in the long term were her main decision factors. The other ideas were fun and exciting, but there’s much less that could go wrong with simple buttons. She also gave feedback that was eye-opening in the sense that we were underestimating her skills ability to a great extent. Things such as only needing one button clock interface and not two, and needing less braille options, her intuit knowledge that could help her use the alarm with ease. Initially when designing and speaking amongst each other, we were approaching the prototype as if we had just lost our vision and were encountering the world for the first time. However, after talking with Alyicia, we were able to revise our prototypes and make them not only braille friendly, but also extremely intuitive to her needs.

Feedback we took from the critique was to make it only one clock interface, as well as to change the button size. Alicia opted for smaller buttons around the clock as those would decrease the overall size of the alarm. She also mentioned the need for there to be ample space around each button as that would ensure there was enough room for her dymo tape to stick on. Thus, we made sure to account for the braille attachments and their length and width in each button design. She also told us she had no need for a Snooze button, nor did she need an option to set any time more than the multiples of five. This type of feedback was pivotal to our further discussion of how to implement the alarm clock successfully.

Something we hadn’t even considered before she brought it up, was being able to tell the normal time of the day using the alarm clock. She informed us this was normal practice on her existing braille alarm clock that didn’t work that well. Initially we chose not to focus on this as the clock module should have been able to reliably keep the time and there would be no reason to doubt it. However, after much discussion as a team and with our professor, we reimplemented it as a feature.

Process

Deciding on how to improve prototype after showing to Alycia

Final interface design for the clock

Lasercut plywood to build clock

Painting the plywood

The programmable portion of the alarm clock was based on a finite state machine, a concept that David was familiar with from his ECE studies. This style of design was chosen because an alarm clock enters through many states during its usage, and transitioning between states based on inputs (and generating outputs at the right time) are all central ideas to an FSM. The final product ended up with seven states, and it worked extremely well- it allowed for coding details much easier, as it helped partition up the overall complicated structure into smaller, more manageable pieces. That being said, coding did have its fair share of difficulties. Converting a button into a toggle-able input was surprisingly complex; it involved debouncing a button to prevent erroneous multiple readings upon press and release of the button, as well as checking to make sure when a button is pressed and held (and not causing the incorrect interpretation of multiple button presses). The code also involved understanding a RTC (real-time clock) library, which had its own hidden difficulties (we had spent nearly an hour panicking as to why the clock was not working until we realized we hadn’t written the code to actually START the clock!). With respect to the Gantt chart, there was some divergence from the original plan, because testing the code proved to be more reliant upon the hardware it worked with- for example, it was impossible to check if anything related to the clock was working if there was no clock module, which we did not receive until much later than anticipated. Nonetheless, the code was still completed and thoroughly debugged.

All wiring of the clock

Putting all components into the fingerjoint base

There were a few tricky components to the circuitry, such as the solenoid and the interfacing of the bed shaker and its power supply. The solenoid needed to be powered directly by a 12VDC power supply, yet it had to be controlled by the Arduino; as such, a transistor circuit was used to act as a “switch” that the Arduino could control to allow for power to the solenoid. Interfacing with the bed shaker proved to be very complicated, due to the strange peculiarity that it ran off of 9VAC. Creating a switch for 9V, being alternating current as well, was rather difficult, since that implied it would require a power supply that could not be unified in any way with the Arduino (unlike the solenoid, whose 12VDC power supply could also be used for the Arduino). Controlling an AC power supply also required the use of a relay, which was in essence, a switch for AC power supplies, similar to the transistor. The relay proved to also be troublesome in that it would draw too much current from the Arduino should it be directly connected, and that meant that another transistor circuit was needed for the relay- in essence, a switch for the switch. It was certainly a confusing set of circuitry- a memorable moment was when the transistor was actually fried due to an incorrect wiring, which then caused subsequent correct wirings to not work as well.

Conclusions and Lessons Learned 

We learned throughout the process that planning was very important to ensure that we could get all the different parts done well. Since we each had different responsibilities, it was important that we communicated our schedules well and adjusted our work plan as needed so that everyone in the group could work on their responsibilities. Especially for our code-heavy project, we found that it was always better to test early and test more. There would often be small edge cases that would mess with how our clock worked. Changes also needed to be made physically, such as adding another button or changing where the buttons were placed, which would help make using the device easier for Alycia. Additionally, we realized that we could make the best device only if we knew what the client actually needed and wanted. Rather than making assumptions, we always made sure to ask Alycia on what she thought would work the best for her as well as what she wanted from the project. There were times when her responses reassured our progress, and other where we were surprised by what she said and would have to rethink our ideas. In the end, we were grateful that Alycia was very cooperative and responsive to us during this project as we were able to produce a product that she liked as well. 

A common, but important piece of advice that was given was that “Though it’s for a blind person, for presentations sake, there could have been some visual labels.” This would allow for those who were NOT Alycia (eg. caretakers, friends, etc.) to be able to understand what was what on the clock. This would be a simple change, but extremely necessary, since as of now, we only know which buttons are what simply because we are the creators. There were also comments on the alarm clock having “other features that many clocks have (i.e., snooze).” This was also a fair point, and was considered by our team; given more time, this definitely could have been done. While we considered the situation of having too many inputs and causing confusion rather than more features, simple thoughts like “Different shape on the “12” button” to help find/distinguish the clock buttons easier would be useful and helpful to implement.

Technical Details

Schematic and Block Diagram

Code

/* Project Title: Tactile Alarm Clock
 * Creator: David Wu, Noni Shelton, and Catherine Liu
 * 
 * This code is similar to a finite state machine for
 * an alarm clock. There are 12 button inputs for the 12 
 * numbers on the clock. There are 4 other button inputs 
 * for miscellaneous purposes, being the alarm, clock, 
 * set, and clear buttons. The 16 button inputs are each 
 * involved in 2 arrays to facilitate checking for button
 * presses and button holding. An on/off input is used to 
 * determine if the alarm is turned on or not. An output signal
 * to a solenoid allows it to tap the time in an specified, 
 * encoded manner. The code outputs its alarm signal at a certain
 * pin, and for this project, this is intended to connect the
 * bed shaker to its power supply. 
 * 
 * Pin Mapping:
 * Clock 12 : 12    Clock 6  : 6
 * Clock 1  : 13    Clock 7  : 7
 * Clock 2  : 2     Clock 8  : 8
 * Clock 3  : 3     Clock 9  : 9
 * Clock 4  : 4     Clock 10 : 10
 * Clock 5  : 5     Clock 11 : 11
 * Doing Alarm               : 27
 * Doing Clock               : 26
 * Set                       : 24
 * Clear                     : 22
 * Alarm On/Off              : 28
 * Solenoid                  : 30
 * Alarm output (bed shaker) : 33
 * 
 */

#include <Wire.h>
#include "RTClib.h"

RTC_DS3231 rtc;

// clock 12 inputs
const int CLOCKPIN0 = 12;
const int CLOCKPIN1 = 13;
const int CLOCKPIN2 = 2;
const int CLOCKPIN3 = 3;
const int CLOCKPIN4 = 4;
const int CLOCKPIN5 = 5;
const int CLOCKPIN6 = 6;
const int CLOCKPIN7 = 7;
const int CLOCKPIN8 = 8;
const int CLOCKPIN9 = 9;
const int CLOCKPIN10 = 10;
const int CLOCKPIN11 = 11;

// the miscellaneous buttons
const int DOING_ALARMPIN = 27;
const int DOING_CLOCKPIN = 26;
const int SETPIN = 24;
const int CLEARPIN = 22;

// alarm on/off
const int ON_OFFPIN = 28;

// outputs
const int SOLEPIN = 30;
const int BEDSHAKERPIN = 33;

// state management
int currState = 0;
int nextState = 0;
const int defaultState = 0;
const int alarmState = 1;
const int clockState = 2;
const int setAlarmHourState = 3;
const int setClockHourState = 4;
const int setAlarmMinState = 5;
const int setClockMinState = 6;
const int alarmTriggered = 7;

// clock 12 inputs
int clockInput[12];
int clockPressed[12];

// the miscellaneous buttons
// misc inputs: doing_alarm, doing_clock, set, clear
int miscInput[4]; 
int miscPressed[4];

// alarm on/off
bool isOn = true;

// time management
// arbitrary default setting when clock is first turned on
const int DEFAULT_HOUR = 7;
int alarmHour = DEFAULT_HOUR;
int alarmMin = 0;
int tempHour = DEFAULT_HOUR;
int tempMin = 0;

void setup() {
  pinMode(CLOCKPIN0, INPUT);
  pinMode(CLOCKPIN1, INPUT);
  pinMode(CLOCKPIN2, INPUT);
  pinMode(CLOCKPIN3, INPUT);
  pinMode(CLOCKPIN4, INPUT);
  pinMode(CLOCKPIN5, INPUT);
  pinMode(CLOCKPIN6, INPUT);
  pinMode(CLOCKPIN7, INPUT);
  pinMode(CLOCKPIN8, INPUT);
  pinMode(CLOCKPIN9, INPUT);
  pinMode(CLOCKPIN10, INPUT);
  pinMode(CLOCKPIN11, INPUT);
  pinMode(DOING_ALARMPIN, INPUT);
  pinMode(DOING_CLOCKPIN, INPUT);
  pinMode(SETPIN, INPUT);
  pinMode(CLEARPIN, INPUT);
  pinMode(ON_OFFPIN, INPUT);
  pinMode(SOLEPIN, OUTPUT);
  pinMode(BEDSHAKERPIN, OUTPUT);

  // ensure rtc is found and we can start
  if (!rtc.begin()) {
    while (1);
  }

  // initialize an arbitrary time (12:34)
  // rtc.adjust(DateTime(F(__DATE__), F(__TIME__)));
  rtc.adjust(DateTime(2000, 1, 1, 12, 34, 0));
  Serial.begin(9600);
}

void loop() {
  // clock 12 inputs
  clockInput[0] = digitalRead(CLOCKPIN0);
  clockInput[1] = digitalRead(CLOCKPIN1);
  clockInput[2] = digitalRead(CLOCKPIN2);
  clockInput[3] = digitalRead(CLOCKPIN3);
  clockInput[4] = digitalRead(CLOCKPIN4);
  clockInput[5] = digitalRead(CLOCKPIN5);
  clockInput[6] = digitalRead(CLOCKPIN6);
  clockInput[7] = digitalRead(CLOCKPIN7);
  clockInput[8] = digitalRead(CLOCKPIN8);
  clockInput[9] = digitalRead(CLOCKPIN9);
  clockInput[10] = digitalRead(CLOCKPIN10);
  clockInput[11] = digitalRead(CLOCKPIN11);

  // misc inputs: doing_alarm, doing_clock, set, clear
  miscInput[0] = !digitalRead(DOING_ALARMPIN); // normally closed button
  miscInput[1] = !digitalRead(DOING_CLOCKPIN); // normally closed button
  miscInput[2] = digitalRead(SETPIN); 
  miscInput[3] = digitalRead(CLEARPIN); 
  
  // alarm on/off
  isOn = digitalRead(ON_OFFPIN);

  // find if/which clockInput was pressed
  int trigClockButton = -1;
  for (int i = 0; i < 12; i++) {
    if (clockInput[i] == 1) {
      trigClockButton = i;
    }
  }

  // find if/which miscInput was pressed
  int trigMiscButton = -1;
  for (int i = 0; i < 4; i++) {
    if (miscInput[i] == 1) {
      trigMiscButton = i;
    }
  }    

  // when to start output
  if (isOn) {
    DateTime now = rtc.now();
    if ((convert12Hour(now.hour()) == alarmHour) &&
        (now.minute() == alarmMin)  &&
        (now.second() < 5)) {
       currState = alarmTriggered;
    }
  }

  // figure out next state and output
  if (currState == alarmTriggered) {
    if(!isOn) {
      digitalWrite(BEDSHAKERPIN, LOW);
      nextState = defaultState;     
    }
    else {
      digitalWrite(BEDSHAKERPIN, HIGH);
      nextState = alarmTriggered;
    }
  }
  else if (trigMiscButton != -1)
    handlePressedMiscButton(trigMiscButton);
  else if (trigClockButton != -1) 
    handlePressedClockButton(trigClockButton);
  else 
    nextState = currState;
    
  // update unpressed buttons
  // when a clockInput is unpressed and was pressed before
  for (int i = 0; i < 12; i++) {
    if (!clockInput[i] && clockPressed[i]) {
      delay(30);
    }
  }
  // when a miscInput is unpressed and was pressed before
  for (int i = 0; i < 4; i++) {
    if (!miscInput[i] && miscPressed[i]) {
      delay(30);
    }
  }  
  
  //update info
  clockPressed[0] = clockInput[0];
  clockPressed[1] = clockInput[1];
  clockPressed[2] = clockInput[2];
  clockPressed[3] = clockInput[3];
  clockPressed[4] = clockInput[4];
  clockPressed[5] = clockInput[5];
  clockPressed[6] = clockInput[6];
  clockPressed[7] = clockInput[7];
  clockPressed[8] = clockInput[8];
  clockPressed[9] = clockInput[9];
  clockPressed[10] = clockInput[10];
  clockPressed[11] = clockInput[11];

  miscPressed[0] = miscInput[0];
  miscPressed[1] = miscInput[1];
  miscPressed[2] = miscInput[2];
  miscPressed[3] = miscInput[3]; 
  
  currState = nextState;
}

void handlePressedClockButton (int index) {
  // when the clockInput is pressed and was not pressed before
  if (!clockPressed[index]) {
    delay(30);
    switch (currState) {
      // if alarm is triggered, ignore everything
      case alarmTriggered:
        break;
      case alarmState:
        tempHour = index;
        if (tempHour == 0) tempHour = 12;
        nextState = setAlarmHourState;
        break;
      case clockState:
        tempHour = index;
        if (tempHour == 0) tempHour = 12;
        nextState = setClockHourState;
        break;
      case setAlarmHourState:
        tempMin = index * 5;
        nextState = setAlarmMinState;
        break;
      case setClockHourState:
        tempMin = index * 5;
        nextState = setClockMinState;
        break;
      default: 
        nextState = currState;
        break;
    }
  }
}

void handlePressedMiscButton (int index) {
  // misc inputs: doing_alarm, doing_clock, set, clear
  // when the miscInput is pressed and was not pressed before
  if (!miscPressed[index]) {
    delay(30);
    // if alarm is triggered, ignore everything
    if (currState == alarmTriggered) {
      nextState = alarmTriggered;
    }
    // doing_alarm
    else if (index == 0) {
      solenoidTapping(alarmHour, alarmMin);
      nextState = alarmState;
    }
    // doing_clock
    else if (index == 1) {
      DateTime now = rtc.now();
      solenoidTapping(convert12Hour(now.hour()), now.minute());
      nextState = clockState;
    }
    else {
      switch (currState) {
        case alarmState:
        case setAlarmHourState:
          if (index == 2) {
            // not complete inputs
            tempHour = DEFAULT_HOUR;
            tempMin = 0;
            nextState = defaultState;
          }
          else if (index == 3) {
            tempHour = DEFAULT_HOUR;
            tempMin = 0;
            nextState = alarmState;
          }
          break;
        case clockState:
        case setClockHourState:
          if (index == 2) {
            // not complete inputs
            tempHour = DEFAULT_HOUR;
            tempMin = 0;
            nextState = defaultState;
          }
          else if (index == 3) {
            tempHour = DEFAULT_HOUR;
            tempMin = 0;
            nextState = clockState;
          }
          break;
        case setAlarmMinState:
          if (index == 2) {
            alarmHour = tempHour;
            alarmMin = tempMin;
            nextState = defaultState;
          }
          else if (index == 3) {
            tempHour = DEFAULT_HOUR;
            tempMin = 0;
            nextState = alarmState;
          }
          break;
        case setClockMinState:
          if (index == 2) {
            // DateTime(YYYY, M, D, h, m, s)
            rtc.adjust(DateTime(2000, 1, 1, tempHour, tempMin, 0));
            nextState = defaultState;
          }
          else if (index == 3) {
            tempHour = DEFAULT_HOUR;
            tempMin = 0;
            nextState = clockState;
          }
          break;
        default: 
          nextState = currState;
          break;
      }
    }
  }
}

// Taps: Hours, Tens place of Mins, Ones place of Mins
// Ex. 9:47 is 9 taps, 4 taps, 7 taps
void solenoidTapping (int timeHour, int timeMinute) {
  for (int i = 0; i < timeHour; i++) {
    digitalWrite(SOLEPIN, HIGH);
    delay(50);
    digitalWrite(SOLEPIN, LOW);
    delay(450);
  }
  delay(2000);
  int minuteTens = timeMinute / 10;
  int minuteOnes = timeMinute % 10;
  for (int i = 0; i < minuteTens; i++) {
    digitalWrite(SOLEPIN, HIGH);
    delay(50);
    digitalWrite(SOLEPIN, LOW);
    delay(450);
  }
  delay(2000);
  for (int i = 0; i < minuteOnes; i++) {
    digitalWrite(SOLEPIN, HIGH);
    delay(50);
    digitalWrite(SOLEPIN, LOW);
    delay(450);
  }
  delay(2000);
  // a quick TAP-TAP to signal done tapping
  digitalWrite(SOLEPIN, HIGH);
  delay(50);
  digitalWrite(SOLEPIN, LOW);
  delay(100);
  digitalWrite(SOLEPIN, HIGH);
  delay(50);
  digitalWrite(SOLEPIN, LOW);
  delay(100);
}

// converts hours from 24-hour format to 12-hour format
int convert12Hour(int timeHour) {
  // later half of day
  if (timeHour > 12) 
    return timeHour - 12;
  // midnight
  else if (timeHour == 0)
    return 12;
  else
    return timeHour;
}

The topic of our final project is to design and build a device for our client with physical difficulties. We get to understand our client’s needs by interviewing and project progress meetings during the 6-week project period and keep improving our product according to information we got from conversations and feedback from the client. Our client is Haleigh, and here is the documentation of our first interview: Interview with Haleigh. Haleigh has difficulties moving and twisting her hands and fingers freely, and she doesn’t have enough grip to hold utensils as well as paintbrushes; she loves painting. Based on this information, we built a wearable device that can hold and rotate utensils for her.

2. Final Product

The wearable device we built is a wristband that carries the electronics and a servo motor with a connector to hold and drive different utensils. We crocheted the wristband with light purple yarn to ensure it’s light-weighted and comfortable for our client; and yes, purple is Haleigh’s favorite color. We also choose Velcro to tighten and loosen the wristband thus reducing the wearing difficulty. The utensils and pastel holder are designed, and 3D printed based on our client’s preferred position and rotational angles. To make the control most uncomplicated, we created a discrete button to trigger the utensil rotation, so Haleigh can use one of her hands to use a utensil and control the rotation with another hand.

Utensil Holder with A Spoon

Utensil Holder with A Fork

Whenever Haleigh is, for example, eating a meal, her mother would easily be able to attach the velcro wrist straps on the device and then serve her the food. While before Haleigh mostly ate with her hand, which is still difficult due to her limited wrist movement, with the device she would able to relax more. She would be able to change between the spoon or fork fixtures depending on what she’s eating and be able to have her meal mostly by herself if she so chose to. Every time she hits the button, the servo motor would consistently rotate the utensil towards the food and then towards her, allowing her to eat with more ease.

3. Prototypes and Modifications

3.1 Prototyping

After the first interview, we proposed three essential question for our project:

1. Wearable part – How to make the wristband easy to ware, comfortable, but also sturdy enough to hold devices and utensils;
2. Electronic part – How to select a proper sensor to convert the change of pose to some data, and how the data can use to control the movement of utensils;
3. Utensil part – How to create a universal connector that fits different utensils and connect them.

We discussed these questions and came up with an initial idea sketch:

Prototyping Sketch

Based on our client’s way of holding things and her grip, we proposed a glove-like design with multiple supports: writs, palm, and curlicue for the wristband. Such a method helps to average dispense the gravity of devices and electronics to the whole hand; the wristband could be knitted.

Prototyping – Wristband

For the utensil, we first designed a box-like holder for the disposable tableware – the user can just stab the tail of utensils into the holder, and the holder could fix the utensil by friction and magnetism. However, because of lacking consideration of the size of the magnets, we failed to install magnets to the utensils. Still, we found that the friction could be enough to hold every tableware (made of plastic, not metal).

Prototyping – Utensil Holder

At the end of the prototyping phase, we put parts together and created a rough version of our utensil holder.

The First Prototype

The prototype enables us to check the feasibility of our design. The fun part is that all three parts – electronics, wristband, and utensil holder – work somewhat well as a whole; the wristband can hold the electronics, the electronics can drive the utensils, and the utensil and its hold aren’t too heavy for the servo. Then we brought this initial product to the first critic presentation to get more feedback about potential improvements to our design.

We got very constructive and helpful suggestions from out client and critics during the prototype presentation:

1. Wristband: The wristband can be knitted with thinner yarn, making the band lighter and better ventilated. It’s better to make the wristband cover the whole back of the hand to avoid any hard part making the hand uncomfortable. We can still use a cushion to support the device, as the knitting part is too soft to mount electronics and servo.
2. Utensils: The angle of utensils should be well considered because now we are using the utensil differently: utensils like forks and spoons should somewhat curve towards the mouth. So, we should consider designing utensils instead of using disposable ones.
3. Electronics: The accelerometer is too sensitive to the gesture changes, which might lead to unwilling utensil rotations. We could go with more accessible solutions like using a button to control spoon and fork motions. We could separate the button from the wristband so that our client could use the product more accessible. We could establish wireless communications between the button and electronics on the wristband.

3.2 Interim Modifications

Based on the feedback from the prototype critic, we redesigned the wristband, electronics, and utensils (and the holder). We then invite our client to the Hunt Library again to see if our modifications fit her gestures and needs.

We did a significant improvement on lightness and softness for the wristband part. Still, it’s not sturdy enough – it’s not perfectly wrapping around the hand, so the device on it might move around unwillingly and frequently. So we decide to redesign the belt to make tighten the wristband.

Interim Modification – Wearing the Wristband

Interim Modification – the Wristband Front

Interim Modification – the Wristband Back

The button is working pretty well; the cap is large enough for our client, and the communication between the button and utensil servo seems established, although disconnection happens sometimes. The next step for the button is to design a container for it.

Interim Modification – the Control Button

Interim Modification – the Pastel Holder Angle

3.3 Finalize the Product

With the interim modifications, we used our last week to make final improvements to our design. We mounted electronics onto the wristband, assembled things, did some final tests, and finished the project.

Final Modifications

Utensil Holder Redesign

Final Product

4. Conclusion and Lesson Learned

From the final critique, there were many things that we discussed with both the reviewers and with Haleigh that we could look at to further improve the project. For instance, one point that one of the reviewers brought up with us and on the written feedback sheet was how “[we] could have pushed further with liquid and food protection – a removable, washable cover would have helped polish [it] off”. Another reviewer had also commented that “the 3d printed utensils should be reconsidered” as “the material may not be entirely safe for use with foods”. While the food safety of the project was something that we had considered, given the form of our project and how we had to go about making the specific parts and their shapes, it was difficult to find a way to implement them given our budget and resources. A solution to this that we wish we looked more into was trying to make primarily holders that would be able to perform the same task but with any utensil the same way we made the universal writing implement holder. Another reviewer brought up how it would be better “if the wrist piece had a little sleeve that could contain the electronic pieces”. This is an easy solution which we feel we could accomplish with either another piece of the foam that there currently is or with an additional crochet piece, either of which could be attached with velcro. Another one of the comments that were made by the reviewers was that they were “worried [the device] will turn while on her wrist”, which is something we think that we addressed in our meetings with Haleigh and the customizations that we were able to make to make it as suited to her as possible. 

Overall, despite the numerous points for reconsideration that there are to be considered in evaluating our project, we still were able to learn a lot through this experience of working with a person with a disability. Designing for another client as opposed to ourselves, it was very interesting going about the process of taking into consideration a different set of needs and restrictions which are outside of our own experiences. It surprised us how we had to translate the movements and abilities based on how Haleigh would be able to approach the same tasks. It was interesting to re-evaluate the way that we did something that is almost second nature to us, like eating with a fork and spoon and how these motions can be applied to the way Haleigh can move.

Looking at the project overall, we would go about choosing different materials for the attachments and also consider how to go about making the attachments more universal. Given an unlimited amount of time and resources, our main priority would be to making the attachments food safe and try increase the flexibility of the structure for both comfort and lightening the weight.

5. Technical Details

/*
  Utensil Holder
*/
/*
  This code is for the button, which enables it to remotely control the motor
*/
/*
  Hardware Pinout Connection
  Arduino Uno        nRF24L01
       3.3V ---------- VCC
       GND ----------- GND
       7  ------------ CE
       8  ------------ CSN
       11 ------------ MOSI
       12 ------------ MISO
       13 ------------ SCK
________________________________________
  Arduino Uno         Button
       GND ---------- negative
       2  ----------- positive
*/

#include <Wire.h>
#include <SPI.h>
#include <nRF24L01.h>
#include <RF24.h>

const int buttonPin = 2; // the number of the pushbutton pin
int buttonState = 0;
const byte address[6] = "00001";

RF24 radio(7, 8); // CE, CSN

void setup() {
  pinMode(buttonPin, INPUT_PULLUP);
  Serial.begin(9600);
  Wire.begin();
  radio.begin();
  radio.openWritingPipe(address);
  radio.setPALevel(RF24_PA_MIN);
  radio.stopListening();
}

void loop() {
  buttonState = digitalRead(buttonPin);
  Serial.println(buttonState);
  radio.write(&buttonState, sizeof(buttonState));
}

/*
  Utensil Holder
*/
/*
  This code is for the motor, which enables it to be controlled by the button
*/
/*
  Hardware Pinout Connection
  Arduino Pro Micro        nRF24L01
          VCC -------------- VCC
          GND -------------- GND
          9  --------------- CE
          10  -------------- CSN
          16 --------------- MOSI
          14 --------------- MISO
          15 --------------- SCK
________________________________________
  Arduino Pro Micro       Servo Motor
          VCC -------------- VCC
          GND -------------- GND
          8  --------------- signal
*/

#include <Servo.h>
#include <Wire.h>
#include <SPI.h>
#include <nRF24L01.h>
#include <RF24.h>

int buttonState = 0;
int tracker = 1; //tracking which direction we want the motor to rotate
int btn_prev = buttonState; //track the last buttonState to avoid multiple count of signal
const byte address[6] = "00001";

RF24 radio(9, 10); // CE, CSN
Servo myservo;

void setup() {
  pinMode(buttonPin, INPUT_PULLUP);
  myservo.attach(8);
  Serial.begin(9600);
  Wire.begin();
  radio.begin();
  radio.openReadingPipe(0, address);
  radio.setPALevel(RF24_PA_MIN);
  radio.startListening();
  btn_prev = 1;
  myservo.write(90);
}

void loop() {
  if (radio.available())
  {
    radio.read(&buttonState, sizeof(buttonState));
    Serial.println(buttonState);
    if (buttonState == LOW && btn_prev == 1) { 
      if (tracker == 1) {
        for (double pos = 90; pos <= 180; pos += 0.5) { //rotate from 90 to 180 degrees
          myservo.write(pos);              
          delay(15);  // waits 15ms for the servo to reach the position
        }
        tracker = -tracker;
      }
      else {
        for (double pos = 180; pos >= 90; pos -= 0.5) { //rotate from 180 to 90 degrees
          myservo.write(pos);              
          delay(15);
        }
        tracker = -tracker;
      }
    }  
  }  
  btn_prev = buttonState;
}

What We Built:

Our project is designed to allow our client Allie to type a price into a keypad and then have the amount of money they need to be dispensed to them as well as the amount of change left over displayed. This cash dispenser can be attached to our client’s wheelchair. There is also a reload mechanism that allows our client to reload the machine with money. Our specific device was designed to dispense $10 and $20 bills. 

A general view of our machine. It is a pink box with two sets of money rollers inside. Each roller compose of two tubes covered and connected by a black fabric with two motors attached to the right side.

Infrared Sensors are depicted on the left and Infrared Detectors are shown on the right. These pair of sensors get blocked by the bills dispensed by the module and once the bill stops blocking the infrared light, the Arduino adds one to the amount of bill dispensed.

An LCD display is shown to the left with “price: $” printed on it, a keypad installed in a pink box is shown to the right with the numbers 0 to 9, a star key, and a pound sign arranged onto it in a 3 by 4 matrix. When a key is pressed on the keypad, the display shows the updated number and when done, the user presses the pound sign to enter the price needed to be dispensed. The star key is used to delete unwanted numbers.

A red push button is placed to the right of the money roller. When this button is pressed, both yellow motors attached to the rollers will begin driving (the top one drives faster than the bottom to ensure the fabric doesn’t create slack between the rollers) to reload the money.

A bill is shown being dispensed by the money roller module. The module is made with two tubes and a black fabric that wraps around both of them. Bills are stored between the rolled-up fabric and motors are driven to dispense and reload the bills.

A video that demonstrates the entire process of dispensing money from the view of the user.

A video that demonstrates the entire process of dispensing money from the view of the shop clerks.

Narrative Sketch:

When Allie goes out shopping and is about to pay for her groceries, she would turn to the Money Dispenser sitting on her desk on top of the wheelchair. She types in the price of the item using the keypad and the motors run and dispense the correct number of 10- and 20-dollar bills. The display screen reads the change Allie would get back. When Allie gets home, she and her helper would unlock the cabinet in the front, and then press the red button which says reload 10s and reload the 20s to reload the machine for purchases in the future.

How we got here:

Prototype:

This prototype was designed to help answer the question: how to most effectively dispense only one bill at a time?

Our first prototype was based on how a printer dispenses paper and utilized multiple roller mechanisms that would dispense a bill from a stack. Basically a spring would push up a stack of bills to a certain height and then a rolling cylinder with grippy material would push the bill forward. We then moved on to our second prototype which is the design that we ended up using with two rollers and fabric pulled between them. This design is essentially a piece of fabric taped to two roller and as its winded, bills are inserted between the folds of the fabric. The fabric is then unwound to dispense the bills.

Our original bill-dispensing prototype

The first prototype required us to make a spring which we did by wrapping wire around a clamp.

First rolling mechanism with only two motors and one set of rollers.

Our first prototype of the bill dispensing mechanism was not able to consistently roll out a single bill at a time.

Prototype of the new bill dispensing mechanism without fabric attached

Testing the new bill-dispensing mechanism

Assembling our prototype for our final design

Most of our prototyping process was spent figuring out if it was possible for us to create a money dispenser, specifically one that would only dispense one bill at a time. We first modeled our mechanism after a printer that utilized a set of rollers that would only allow one bill to pass through at a time. However, we found that this mechanism required high precision tolerance between the rollers which was difficult for us to consistently achieve. We then switched to our final mechanism which instead of storing the bills in a stack where they could stick together, layered the bills separately inside a roll of fabric.

We received feedback that our original mechanism was flawed because we did not have a way to detect if the correct number of bills had been dispensed, which we solved by integrating IR sensors into the design. However, we weren’t able to react to the feedback to make the mechanism smaller due to how much testing that would require and the time limit we were under. The biggest (good) surprise we encountered was how smoothly our second mechanism worked after having so much difficulty with tweaking the first one.

Process:

Constructing the bill rollers using laser-cut and 3D-printed parts

Fully constructed bill rollers using standardized 3D-printed and laser-cut parts.

Figuring out the row and column pin-mapping for the keypad.

Wiring the IR receiver and transducer on a breadboard for initial readings

Integrating multiple pieces of hardware into a circuit with the Arduino Mega.

Constructing the back panel of the box

The biggest change in our design moving forward from our prototype presentation was switching our bill dispensing mechanism to a simpler two-roller setup. We found a mechanism online that used two rollers driven by motors to reload and dispense bills from a spool of fabric. This mechanism gave us more control over how many bills were released at once. It was also more appropriate for Allie’s needs since she told us that she gets about 60 dollars each week in the form of 3 twenty-dollar bills, so she didn’t necessarily need a device that could fit a large stack of bills. 

We also had to make a decision about how our device could track the number of dispensed bills from each compartment. Initially, we planned to time how long it would take to dispense a single bill, but that would require the bills to be evenly spaced out when loaded into the rollers. Instead, we chose to use an IR transmitter and receiver that detects a decrease in the IR signal as a bill exits the money rollers. Similarly, we had to find a method to determine when a given compartment was out of bills. At first we planned to use an RGB color sensor to detect a color change when a roller reached a white piece of tape at the end of the fabric. However, we ended up using an IR sensor so that we would only have to work with one signal value rather than the three values that an RGB sensor outputs. 

Our biggest breakthrough was figuring out how to prevent fabric slack from bunching up between the two rollers when running the motors. We found the best way to maintain good tension in the fabric roll was to drive the bottom roller faster than the top roller while dispensing and drive the top roller faster while reloading. This allowed us to smoothly and consistently dispense/reload bills. 

One mistake we made was waiting too long to start arranging our pieces of hardware into the outer shell of the box. We spent a lot of time fine-tuning the IR sensor thresholds outside the final cash box. This took up a lot of time, and we had to redo the IR signal thresholds in our code anyway because the box provided a much darker environment. If we had put our entire device into the box earlier on, we would have still been able to fine-tune the performance of our hardware and would have had more time for wire management inside the final form.

Gantt Chart Schedule:

Gantt chart timeline from prototype critique until final presentation.

Our team followed the sequence of tasks on the Gantt chart pretty closely. However, we underestimated how long it would take us to complete certain tasks. There ended up being a lot more hardware than initially expected, so the process of wiring, writing code, and introducing each component into the circuit took longer than expected. Rather than finishing the “implementing circuit” phase by 4/24, we didn’t finish integrating all the electronics and the roller mechanisms until 4/29. This only gave us 3 days to put all of our hardware into the outer box.

Conclusions and Lessons Learned:

This project has been a difficult but rewarding experience. We have run into a lot of problems during every stage of the process, including revamping the entire mechanism after the prototype review, but we were able to put together the mechanism in the end. The box itself during the final review still had a lot of room for improvement. As pointed out by comments such as “the fabrication is flimsy with a less-than-ideal UI”, “a bit big”, and “[Allie] can’t see what comes out”, the fabrication was a last-minute process and had flaws such as its size and fragility. We also planned on making a locked cabinet in the front but couldn’t get around to installing it. Additionally, the box orients itself away from Allie which prevents her from seeing the money being dispensed. A great solution that our critic brought up was adding a “swivel” to the bottom of the device so that it can “turn towards Allie” first, then when Allie makes sure it is the right amount, she can turn the machine towards the shop staff. Moreover, the problem of size was also discussed heavily in the critique. One critic suggested using “bill recognition” and some forms of servo motor that directs the money out or back into the loop so that we can use one module for all kinds of bills instead of two modules for two types of bills. This will not only increase the utility of the machine but also downsize it to a reasonable and usable form. Another critic pointed out is the messy “inner components” that should be organized which might reduce the size of the box further. However, despite the shortcomings, our project still got some praise for tackling such a difficult problem, dealing with single sheets of paper. Some critics said that the “motor system works well” and contains “clever paper handling” which we are happy to see.

Working with Allie has been a phenomenal experience as she was extremely helpful during our creation process. Allie shared us some hobbies with us and even some of the posters she made in her free time. We were able to learn to incorporate ergonomic designs into the project that caters to Allie’s specific needs, and through that, we learned more about disability beyond the project as we realized that disability is not just simply a medical definition but also a social problem. Some items and facilities are simply not designed for those with different needs and people are handicapped by the products and that is an external problem instead of an internal one. In the future, we will ask more questions working with people with disabilities and have more meetings with them to get a better sense of the user’s experience.

Working on Zoom certainly brought up some difficulties during the interview as Allie had some issues with her microphone, but overall, we as a team were able to get together in person for the most part and work in the Physical Computing lab together without any issues.

Photo with Allie at the final critique.

Technical Details:

Cash dispenser electrical schematic

Cash dispenser block diagram

Code:

/* Money Box Code - Andy Jiang, Cassie Rausch, Lynn Rushkin
 *  
 * How it works:
 *  Use the keypad to enter the price you would like to pay:
      '#' = enter (E)
      '*' = delete (D)
      type in your price as dollars and cents (i.e. 1600 --> $16.00, 4567 --> $45.67)
      if you mess up a digit then click '*' to delete the last digit
      press # when you have your price entered
   The code calculates how many 10 and 20 dollar bills need to be dispensed. As the device dispenses the bills, "dispensing..." will show up on LCD display. The IR transducer/receiver pair on the $10 and $20 compartments will detect how many bills have been dispensed and once the correct amount has been paid the motors of the compartments will stop running. The change amount is calculated based on how much was paid and the entered price. This amount will pop up on the LCD display. If one bill compartment is empty(detected by IR sensor)then the number of bills needed will be recalculated and dispensed from the compartment that still has bills. 

   Sources:
   1. Keypad code from Terrance Ou's "Math Buddy" code (https://courses.ideate.cmu.edu/60-223/s2022/work/math-buddy/)
   2. LCD display code from https://courses.ideate.cmu.edu/60-223/s2022/tutorials/I2C-lcd

  //----------------pin mapping ----------------------
  Pin mapping:
    Arduino Mega pin | type   | description
    -----------------|--------|-------------
    A1                 input     IR receiver 
                                 (detect dispensed $20 bills)
    A2                 input     IR receiver 
                                 (detect dispensed $10 bills)
    A4                 input     IR sensor 
                                 (detects empty $10 bill compartment) 
    A5                 input     IR sensor 
                                 (detects empty $20 bill compartment) 
    2                  output    toy gear box motor 1 
                                 (top roller of $10 bill compartment)
    3                  output    toy gear box motor 1 
                                 (top roller of $10 bill compartment) 
    5                  output    toy gear box motor 2 
                                 (bottom roller of $10 bill compartment)
    6                  output    toy gear box motor 2 
                                 (bottom roller of $10 bill compartment)
    9                  output    toy gear box motor 3 
                                 (top roller of $20 bill compartment)
    10                 output    toy gear box motor 3 
                                 (top roller of $20 bill compartment)
    11                 output    toy gear box motor 4 
                                 (bottom roller of $20 bill compartment)
    12                 output    toy gear box motor 4 
                                 (bottom roller of $20 bill compartment)
    25                 input     button 1 (for reloading $20 bills) 
    27                 input     button 2(for reloading $10 bills)
    39                 input     keypad (COL 2)
    41                 input     keypad (ROW 4)
    43                 input     keypad (COL 3)
    45                 input     keypad (ROW 1)
    47                 input     keypad (COL 1)
    49                 input     keypad (ROW 2)
    51                 input     keypad (ROW 3)
    SDA                          LCD display
    SCL                          LCD diaplay

   Note: toy gear box motors are connected to Arduino Mega through a dual motor driver
*/

//--------------initialize LCD----------------------
#include <Wire.h>
#include <LiquidCrystal_I2C.h>
LiquidCrystal_I2C screen(0x27, 16, 2);

//---------------------initialize keypad---------------------
#include <Keypad.h>
const byte ROWS = 4; // four rows
const byte COLS = 3; // three columns

char keys[ROWS][COLS] = {
  {'1', '2', '3'},
  {'4', '5', '6'},
  {'7', '8', '9'},
  {'D', '0', 'E'}
};


byte rowPins[ROWS] = {45, 49, 51, 41}; // connect to the row pinouts of the keypad
byte colPins[COLS] = {47, 39, 43};    // connect to the column pinouts of the keypad


Keypad keypad = Keypad( makeKeymap(keys), rowPins, colPins, ROWS, COLS );

//----------- initialize motors ----------------------------------
//pins for 20s top
int motorControlPin1 = 12;
int motorControlPin2 = 11;

//pins for 20s bottom
int motorControlPin3 = 2;
int motorControlPin4 = 3;

//pins for 10s top
int motorControlPin5 = 5;
int motorControlPin6 = 6;

//pins for 10s bottom
int motorControlPin7 = 9;
int motorControlPin8 = 10;

//---------------initialize IR receivers for bill detection ------------------
const int IRPin_20s = A1;
const int IRPin_10s = A2;
unsigned long LastIRTime;

int IRVal_20s; //reading of the sensor (0 to 1023)
bool currentIR_20s = 0; //is the IR sensor covered currently? (1 = yes, 0 = no)
bool prevIR_20s = 0; //was the IR sensor covered previously? (1 = yes, 0 = no)

int IRVal_10s;
bool currentIR_10s = 0;
bool prevIR_10s = 0;

int dispensedTwenties = 0; //counts how many bills have been dispensed from 20s compartment
int dispensedTens = 0;

//-------------initialize IR sensor variables for empty compartment detection -------
const int tenIR_empty = A4;
const int twenIR_empty = A5;
int tenIR, twenIR;
bool EmptyModule_20s = 0; //is the $20 compartment empty? (1 = yes, 0 = no)
bool EmptyModule_10s = 0;

// -----------initialize reload buttons ------------------------
const int tens_button = 27;
const int twens_button = 25;

int twens_buttonVal; //on or off values of the button reading
int tens_buttonVal;

//----------- initialize bill calculating variables --------------
char key; //keypad reading
String price = "";
float finalPrice = 0;
int billsTwenties = 0; //number of $20 bills we need to dispense
int billsTens = 0; //number of $10 bills we need to dispense
float change = 0;

// -----------calibrate these values -----------------------------

//motor speed dispense
const int twenTopSpeed_dispense = 100;
const int twenBotSpeed_dispense = 140;

const int tenTopSpeed_dispense = 100;
const int tenBotSpeed_dispense = 140;

//motor speed reloading
const int twenTopSpeed_reload = 150;
const int twenBotSpeed_reload = 80;

const int tenTopSpeed_reload = 150;
const int tenBotSpeed_reload = 60;

/*IR bill dispense threshold - any IR reading below these values means
a bill is passing */
const int twenBills_threshold = 50;
const int tenBills_threshold = 60;

/*IR empty module threshold - any IR reading above these values means 
the bill compartment has been emptied.*/
const int empty_threshold20s = 200;
const int empty_threshold10s = 180;

//------------setup and main loop---------------------------------

void setup() {
  Serial.begin(9600);

  // setup LCD
  screen.init();
  screen.backlight();
  screen.home();

  pinMode(motorControlPin1, OUTPUT);
  pinMode(motorControlPin2, OUTPUT);

  pinMode(motorControlPin3, OUTPUT);
  pinMode(motorControlPin4, OUTPUT);

  pinMode(motorControlPin5, OUTPUT);
  pinMode(motorControlPin6, OUTPUT);

  pinMode(motorControlPin7, OUTPUT);
  pinMode(motorControlPin8, OUTPUT);

  pinMode(tenIR_empty, INPUT);
  pinMode(twenIR_empty, INPUT);
  pinMode(IRPin_20s, INPUT);
  pinMode(IRPin_10s, INPUT);
  pinMode(twens_button, INPUT);
  pinMode(tens_button, INPUT);
}

void loop() {
  enteredPrice();
  if (key == 'E') {
    delay(1000);
    calculateBills();
    dispenseBills();
    stop10s_motor();
    stop20s_motor();
    delay(1000);
    calculateChange();
    resetVariables();
  }
 
  checkReload();
}

//------------functions -------------------------------------
void clearLastInput() {
  int lastIdx = price.length() - 1;
  screen.setCursor(lastIdx + 8, 0);
  screen.print(" ");
}

void enteredPrice() { //displays the price typed into the keypad on LCD
  screen.home();
  screen.print("price: $");
  key = keypad.getKey();
  if (key != NO_KEY
      && key != 'D' && key != 'E'
      && price.length() < 20) {
    price += key;
    screen.setCursor(8, 0);
    screen.print(price);
  }
  if (key == 'D') {
    clearLastInput();
    price.remove(price.length() - 1); //removes the last digit from the price string
  }

  else if (key == 'E') {
    finalPrice = price.toFloat() / 100;
    screen.setCursor(8, 0);
    screen.print(finalPrice);
    price = "";
  }
}

void calculateBills() { //calculates the number of 10 and 20 dollar bills to ideally dispense
  billsTwenties = finalPrice / 20; //integer divide to get how many twenties we need to dispense
  float remainder = finalPrice - 20 * billsTwenties;
  if (remainder > 10) {
    billsTwenties += 1;
  }
  else if (remainder > 0 ) {
    billsTens = 1;
  }

  /*Serial.print("bills twenties:");
  Serial.println(billsTwenties);
  Serial.print("bills Tens:");
  Serial.println(billsTens);*/
}

void run20s_motor() { //drives the top and bottom motors of the $20 compartment to dispense
  //do an initial acceleration (at highest speed) to overcome internal gear resistance, 
  //and then drive the motors at a reasonable speed
  
  if (digitalRead(motorControlPin1) == 0 && digitalRead(motorControlPin4) == 0) {
    analogWrite(motorControlPin1, 255);
    analogWrite(motorControlPin4, 255);
    delay(300);
  }

  analogWrite(motorControlPin1, twenTopSpeed_dispense);
  analogWrite(motorControlPin2, 0);

  analogWrite(motorControlPin3, 0);
  analogWrite(motorControlPin4, twenBotSpeed_dispense);

}
void stop20s_motor() {
  analogWrite(motorControlPin1, 0);
  analogWrite(motorControlPin2, 0);

  analogWrite(motorControlPin3, 0);
  analogWrite(motorControlPin4, 0);
}
void run10s_motor() {
  if (digitalRead(motorControlPin6) == 0 && digitalRead(motorControlPin8) == 0) {
    analogWrite(motorControlPin6, 255);
    analogWrite(motorControlPin8, 255);
    delay(300);
  }
  analogWrite(motorControlPin5, 0);
  analogWrite(motorControlPin6, tenTopSpeed_dispense);

  analogWrite(motorControlPin7, 0);
  analogWrite(motorControlPin8, tenBotSpeed_dispense);

}

void stop10s_motor() {
  analogWrite(motorControlPin5, 0);
  analogWrite(motorControlPin6, 0);

  analogWrite(motorControlPin7, 0);
  analogWrite(motorControlPin8, 0);
}

void checkIR() { //updates boolean values of whether a bill is passing
  //check IR readings in $20 compartment
  IRVal_20s = analogRead(IRPin_20s);
  if (IRVal_20s < twenBills_threshold) {
    currentIR_20s = 1;
  }
  else {
    currentIR_20s = 0;
  }

  //check IR in $10 compartment
  IRVal_10s = analogRead(IRPin_10s);
  if (IRVal_10s < tenBills_threshold) {
    currentIR_10s = 1;
  }
  else {
    currentIR_10s = 0;
  }
}

void count_billsdispensed() { /*uses the change in currentIR_10s value from 1 --> 0 
  to determine that a bill has passed*/
  if (millis() - LastIRTime >= 400) {
    LastIRTime = millis();
    checkIR();
    if (currentIR_20s == 0 && prevIR_20s == 1) {
      dispensedTwenties += 1;
    }
    prevIR_20s = currentIR_20s;
    if (currentIR_10s == 0 && prevIR_10s == 1) {
      dispensedTens += 1;
    }
    prevIR_10s = currentIR_10s;
    /*
      Serial.print("dispensed tens: ");
      Serial.println(dispensedTens);
    */
  }
}

void moduleStatus() { //check if either bill compartment has been emptied
  tenIR = analogRead(tenIR_empty);
  twenIR = analogRead(twenIR_empty);
  //Serial.println(tenIR); // push the most recent value to the computer
  if (tenIR >= empty_threshold10s) {
    EmptyModule_10s = 1;
  }
  if (twenIR >= empty_threshold20s) {
    EmptyModule_20s = 1;
  }
}

void dispenseBills() {
  screen.clear();
  screen.home();
  screen.print("dispensing...");
  
  //the following while loop will continue to run as long as there are bills needed to be dispensed to pay the entered price and at least one bill compartment hasn't been emptied
  while ((billsTwenties > dispensedTwenties or billsTens > dispensedTens) && (EmptyModule_20s == 0 or EmptyModule_10s == 0)) {
    if (billsTwenties > dispensedTwenties and EmptyModule_20s == 0) {
      run20s_motor();
    }
    if (billsTwenties > dispensedTwenties and EmptyModule_20s == 1) {
      stop20s_motor();
      billsTens += 2 * (billsTwenties - dispensedTwenties);
      billsTwenties = dispensedTwenties;
    }
    if (billsTwenties == dispensedTwenties) {
      stop20s_motor();
    }
    if (billsTens > dispensedTens and EmptyModule_10s == 0) {
      run10s_motor();
    }
    if (billsTens > dispensedTens and EmptyModule_10s == 1) {
      stop10s_motor();
      billsTwenties += ((billsTens - dispensedTens) / 2) + 1 ;
      billsTens = dispensedTens;
    }
    if (billsTens == dispensedTens) {
      stop10s_motor();
    }
    //check whether the 10s or 20s modules are empty
    moduleStatus();

    //check how many 10s and 20s have been dispensed
    count_billsdispensed();
  }
}

void calculateChange() {
  change = (dispensedTwenties * 20 + dispensedTens * 10) - finalPrice;
  screen.clear();
  screen.setCursor(0, 0);
  screen.print("change: $");
  screen.setCursor(9, 0);
  screen.print(change);
  delay(3000);
  screen.clear();
}

void checkReload() { //checks if buttons are being pressed. If so, then motors of the bill compartment that correspond to the button will be driven in the opposite direction to reload money into the rollers. 
  twens_buttonVal = digitalRead(twens_button);
  tens_buttonVal = digitalRead(tens_button);
  if (twens_buttonVal == 1) {
    if (digitalRead(motorControlPin2) == 0 && digitalRead(motorControlPin3) == 0) {
      analogWrite(motorControlPin2, 255);
      analogWrite(motorControlPin3, 255);
      delay(300);
    }

    analogWrite(motorControlPin1, 0);
    analogWrite(motorControlPin2, twenTopSpeed_reload);

    analogWrite(motorControlPin3, twenBotSpeed_reload);
    analogWrite(motorControlPin4, 0);

  }

  if (twens_buttonVal == 0) {
    stop20s_motor();
  }

  if (tens_buttonVal == 1) {
    //10
    if (digitalRead(motorControlPin5) == 0 && digitalRead(motorControlPin7) == 0) {
      analogWrite(motorControlPin5, 255);
      analogWrite(motorControlPin7, 200);
      delay(300);
    }

    analogWrite(motorControlPin5, tenTopSpeed_reload);
    analogWrite(motorControlPin6, 0);

    analogWrite(motorControlPin7, tenBotSpeed_reload);
    analogWrite(motorControlPin8, 0);
  }
  
  if (tens_buttonVal == 0) {
    stop10s_motor();
  }
}

void resetVariables() {
  key;
  price = "";
  finalPrice = 0;
  billsTwenties = 0;
  billsTens = 0;
  change = 0;
  dispensedTwenties = 0; //counts how many bills have been dispensed from 20s compartment
  dispensedTens = 0;
  EmptyModule_20s = 0;
  EmptyModule_10s = 0;
}

Design Files

Money box case zip

Money dispenser zip

Keypad box

The Inspiration Capture Device (ICD) is the product of weeks of collaboration with our team’s client, Mary. After meeting with Mary multiple times in person and through the phone, we determined that Mary, as a writer, values the inspiration that can come at a moment’s notice highly. Thus, we wanted to make her an assistive device that allowed her to quickly and easily capture whatever kind of inspiration reaches her in her daily life so that it could help her writing process.  For some more background information on our project, please refer to this post: https://courses.ideate.cmu.edu/60-223/s2022/work/team-callisto-interview-documentations/

What we built

The ICD captures ‘inspiration’ in two different ways- through a picture taken on a small camera or through a voice memo recorded on an audio recorder. The ICD is meant to do this while being as accessible as possible for Mary; the device is split into two parts (a remote controller and a necklace) to make it easy for her to interact with. At the mere press of a button, Mary can take a picture of whatever she is currently facing or turn on a voice recorder on her neck to quickly capture what’s on her mind for future use.

Camera component of the ICD’s remote controller

Inside the remote you’ll notice various wires as well as an Arduino Uno and a transceiver on the left near the camera

Top view of the remote controller, featuring two buttons with their LED lights meant to indicate current remote activity

The necklace part of the ICD featuring a green LED

The above video is a compilation of clips relating to the ICD’s capabilities.

Image of Juan’s hand that was captured in the ICD demo video.

Narrative

While Mary is out shopping, she notices that a beautiful unfamiliar kind of bird has perched itself on a table in front of her. She pushes the ICD’s red button to take a picture of the bird so she could find out what type of bird it is later. After doing so, she decides to write something about the bird.

Later in the week, she suddenly gets a fantastic idea for a short story  while she’s out and about in Pittsburgh. To ensure she doesn’t forget this idea, she presses the ICD’s blue button and says her idea out loud. She then presses the button again to stop and save the recording. Hours later, Mary uploads the file on her computer, listens to her voice memo, and starts getting to work on an amazing story.

Prototype and Process

Our prototype was broken into four parts so that we could test each component before combining them.

The four components included the remote control, the microphone, the camera, and the necklace. The remote control included large buttons connected to LEDs and wireless transmitters, the microphone and the camera were each attached to an SD card, and the necklace included a casing for internal components and a chain.

The remote control component of the ICD prototype

The microphone component of the ICD prototype

The camera component of the ICD prototype

The necklace component of the ICD prototype

The above video showcases the ICD’s simple user interface.

Prototyping

The initial sketches for the necklace part of the ICD

Through our interview with Mary, we found that she is creative and a writer but cannot write down her thoughts quickly, so we created a solution that would allow her to record herself the moment she receives inspiration.

The initial sketch for the remote control part of the ICD

After talking with Mary about our initial sketches and solutions, she suggested that we also include a camera capability because she needs the help of someone else to use her phone camera. We also decided to create a separate remote that housed the camera but also triggered the microphone in the necklace wirelessly to make it more accessible for Mary.

The initial sketch for the attachment of remote

We were trying to figure out the best way to attach the remote to the wheelchair so that it would be easy for Mary to press to record her life. We found that it might be best for remote attachment works like a watch to fix on the wheelchair controller.

Started a simple setup to work on the microphone component using a microsd card reader module and a microphone module

Upon recording the audio noticed that there was a lot of noise. In order to investigate where the noise was coming from we hooked the output of the microphone to an oscilloscope. With some testing we realized that the noise was coming from writing to the SD card reader.

During the prototype critique, we asked Mary to interact with the microphone set up, focusing on how she would insert and remove the microsd card from the reader. We found that reader module was not optimized for Mary’s hands as the sd card was too small and the push mechanism to remove was difficult. And so, we invested in a larger SD card reader.

Unfortunately, the noise still persisted. We switched the arduino uno for an arduino pro micro to see if the board made any difference. Our final design would also use a pro micro as it would allow the microphone to fit in the necklace holder.

The above change did not fix the noise either. As a last resort we scrapped the SD card readers and invested in a module(vs1053)  that had the optimizations required to cleanly record audio. Given more time we would have fully recorded audio with little noise, but instead the audio files were written to the SD card without any audio.

There were also some issues with other components; a moment of frustration regarding the transceivers in our project should be mentioned: Juan spent an unideal amount of time attempting to debug issues with the wireless transceivers. Unsure if the issue was hardware, software, or wiring-related, hours were lost on his part trying to fix the transceiver problem and could’ve gone towards other aspects of the project. This narrow focus on solving the problem in retrospect was a mistake- the time spent trying to resolve the issue (which ended up being due to a broken transceiver) could have instead been spent assisting Sumayya with the voice recorder component of the project. By the time the solution to the transceiver problem was discovered, it was simply too late to assist with other aspects of the project. This served as a learning experience regarding debugging physical computing components as well as knowing when to cut your losses. In addition, we also had some issues with the ICD’s physical case.

The failed versions of the cases

To make sure the two cases are easy for Mary to use, we have to make them both as small and light as possible. Therefore, every electronic component should be located at its specific place. We did a lot of tests to decrease the size.

Prototyping Process Findings

Though we each worked on separate components of the ICD, the prototyping process helped teach us a variety of lessons not only about our individual components, but also about working on a physical computing project in general. From questions regarding how our overall project would be powered to what final form it would take, there were several moments throughout the prototyping process where we had to answer questions as a team rather than individually. This was somewhat surprising-even though we were all working on vastly different pieces of technology, the choices we each made on our parts had effects on the overall team project that were sometimes subtle and other times drastic. It was partly due to this that our final version was incomplete- we didn’t truly begin accommodating each other’s work until the final stages of the project. By that point, it was too late to make some much needed adjustments, such as making the ICD’s case less sharp and improving the ICD’s attachment to a wheelchair which were recommended to us from our critiques. We did not intentionally ignore this feedback, but unfortunately we ran out of time to address most of it.

Gantt Tracking

Our team’s Gantt chart- visualizing our originally planned project schedule

Overall, we were able to roughly follow our Gantt chart to about 75% accuracy. Divergences from our schedule were mainly due to longer than expected debugging times, slow-arriving components that were stuck in transit for longer than preferred, as well as other miscellaneous issues such as group members falling ill during some days.

Conclusions and Lessons Learned

Feedback

Some feedback that stood out to us were the following:

• “Develop ergonomics, far reach in objectives file transferring”
• “There are several steps involved in retrieving files; but, those could be incorporated into an end-of-day routine. The device is a bit large and the sharp corners could be uncomfortable.”
• “They really should have considered how their client would use this and added some personality of their client into it.”
• “A good idea. Make attachment to chair more stable.”

The reason why these pieces of feedback stood out to us was because they reinforced ongoing discussions we were already having regarding the direction of our project. Adding “personality” and making file transferring simpler, for instance, were goals we wanted to achieve but didn’t really figure out how to do so. This was partly due to delays in our planned schedule, and ultimately by the time we shifted our focus to the form of our project there was too little time to implement some of the recommended changes. That being said, all of the feedback received was insightful and extremely helpful for possible future iterations of our project.

Thoughts on Remote Work

In general, it was smooth to work remotely. At the first beginning, we lost contact with Mary but we got into a collaboration with her successfully soon. Sometimes we did different parts of the project individually but we still cared about the work from each other. Everyone put a lot of effort into this project and often continued working after class alone. If we had another chance to work together remotely, it might be a good way for better collaboration by updating the process once after work.

Takeaways from working with Mary

This journey looked like we were doing a device to help Mary, but literally it was Mary who inspired us a lot. She never gave up her love towards life though her mobility was limited. She had her own hobbies and work, her life was so adorable and she got a lot of sense of achievement during her tutorial work. “I really enjoy my life.” Mary said in her first interview with us. At that time we four were all touched by her strong positive vibe.

Concluding Thoughts

Even though our final prototype was somewhat distant from our original vision, the process we all went through while working on the prototype was very educational and insightful. We each learned different things about not just our components, but about the process of prototyping a physical computing project as well as working as a team in general.

Given a second go at the project knowing what we know now, our biggest changes would have to do with how we worked on the project in general. Our decision to split the ICD into four separate parts, one for each member, seemed like an obvious one at first but in retrospect proved to be a hindrance in the final stages of our prototyping process. Since we were all the resident “expert” on the part we were working on within the group, having a group member absent severely hampered our ability to move the project forward. This hurt us a lot after carnival, where some members of the group got sick and as a result had to miss some work sessions- negatively impacting the whole group’s progress.

A possible solution would be to assign two members to each component. This would greatly help with the aforementioned absence problem, and it would have also had positive effects on each component itself since there would be two minds tasked to making progress on each part instead of just one. Additionally, we would also resolve certain prototype issues, like choosing a battery and final form, as soon as possible instead of saving that for the very end given the importance of accessibility for our client.

Some other takeaways include establishing a debugging process for components early on, not underestimating the soldering process, properly gauging capabilities/knowledge early on in the process, and removing/combining features earlier! These takeaways were largely individual- Juan’s takeaway (the debugging process) was the result of too many hours spent trying to debug what ended up being broken components. Sumayya’s, on the other hand, centered on her experience tackling the complicated voice recording system. Instead of pivoting to a different direction earlier, her various attempts to get the component working backfired as other components that were behind in terms of progress would have made major use of an extra mind. Yongwen and Tristan also had their fair share of takeaways stemming from their assigned components. Overall, however, we can simply say that each one of us learned something valuable about our very own approaches to different problems.

Technical details

Diagrams

Remote Controller Diagram

Necklace Diagram

Code

// Thanks to ArduCAM Mini demo (C)2018 by Lee, found at http://www.ArduCAM.com
/*Image Capture Device - Remote/Camera Code,
Juan Meza, Tristan Hineman
This code operates the remote's various functionalities, including
how it takes user input, how it sends data wirelessly, and how it takes 
and saves images taken from the device. 
*/

//SPI Mosi-11, Miso-12, Sck-13

#include <nRF24L01.h>
#include <RF24.h>
#include <EEPROM.h>

#include <Wire.h>
#include <ArduCAM.h>
#include <SPI.h>
#include <SD.h>
#include "memorysaver.h"

#define BUTTON 8
#define BUTTON_BLUE 2
#define LED_BLUE 6 //blue LED
#define LED_RED 5 //red LED
#define SD_CS 9
#define FRAMES_NUM    0x00

const byte address[6] = "42069";
bool isButtonPressed = false;
unsigned long buttonTime;
int ButtonPresses = 0;

// set pin 7 as the slave select for the digital pot:
const int CS = 7;

RF24 radio(3, 4); // CE, CSN

//camera
//This demo can only work on OV5640_MINI_5MP_PLUS or OV5642_MINI_5MP_PLUS platform.
#if !(defined (OV2640_MINI_2MP_PLUS))
#error Please select the hardware platform and camera module in the ../libraries/ArduCAM/memorysaver.h file
#endif


bool is_header = false;
int total_time = 0;

#if defined (OV2640_MINI_2MP_PLUS)
ArduCAM myCAM( OV2640, CS );
#endif

uint8_t read_fifo_burst(ArduCAM myCAM);

void setup() {

  pinMode(BUTTON, INPUT_PULLUP);
  pinMode(BUTTON_BLUE, INPUT_PULLUP);
  pinMode(LED_BLUE, OUTPUT);
  pinMode(LED_RED, OUTPUT);

  buttonTime = millis();

  radio.begin();
  radio.openWritingPipe(address);
  radio.setPALevel(RF24_PA_MIN);
  radio.stopListening();

  // camera
  uint8_t vid, pid;
  uint8_t temp;
  
#if defined(__SAM3X8E__)
  Wire1.begin();
#else
  Wire.begin();
#endif

  Serial.begin(115200);
  
  Serial.println(F("ArduCAM Start!"));
  // set the CS as an output:
  pinMode(CS, OUTPUT);
  digitalWrite(CS, HIGH);
  // initialize SPI:
  SPI.begin();
  //Reset the CPLD
  myCAM.write_reg(0x07, 0x80);
  delay(100);
  myCAM.write_reg(0x07, 0x00);
  delay(100);
   
   
  while (1) {
    //Check if the ArduCAM SPI bus is OK
    myCAM.write_reg(ARDUCHIP_TEST1, 0x55);
    temp = myCAM.read_reg(ARDUCHIP_TEST1);
    if (temp != 0x55)
    {
      Serial.println(F("SPI interface Error!"));
      delay(1000); continue;
    } else {
      Serial.println(F("SPI interface OK.")); break;
    }
  }


#if defined (OV2640_MINI_2MP_PLUS)
  while (1) {
    //Check if the camera module type is OV2640
    myCAM.wrSensorReg8_8(0xff, 0x01);
    myCAM.rdSensorReg8_8(OV2640_CHIPID_HIGH, &vid);
    myCAM.rdSensorReg8_8(OV2640_CHIPID_LOW, &pid);
    if ((vid != 0x26 ) && (( pid != 0x41 ) || ( pid != 0x42 ))) {
      Serial.println(F("ACK CMD Can't find OV2640 module!"));
      delay(1000); continue;
    }
    else {
      Serial.println(F("ACK CMD OV2640 detected.")); break;
    }
  }
  
#endif


  //Initialize SD Card
  while (!SD.begin(SD_CS))
  {
    Serial.println(F("SD Card Error!")); delay(1000);
  }
  Serial.println(F("SD Card detected."));
  //Change to JPEG capture mode and initialize the OV5640 module
  myCAM.set_format(JPEG);
  myCAM.InitCAM();
  myCAM.clear_fifo_flag();
  myCAM.write_reg(ARDUCHIP_FRAMES, FRAMES_NUM);
}


void loop() {

  myCAM.flush_fifo();
  myCAM.clear_fifo_flag();
#if defined (OV2640_MINI_2MP_PLUS)
  myCAM.OV2640_set_JPEG_size(OV2640_1600x1200);
#endif

  int buttonVal = digitalRead(BUTTON);
  //Serial.println(buttonVal);
  //as a reminder, when a button is being pressed buttonVal turns to zero

  int buttonBlueVal = digitalRead(BUTTON_BLUE);

  if (buttonVal == 0) {
    Serial.println("red is being pressed");
    digitalWrite(LED_RED, HIGH);

    myCAM.start_capture();
    Serial.println(F("start capture."));
    total_time = millis();
    while ( !myCAM.get_bit(ARDUCHIP_TRIG, CAP_DONE_MASK));
    
    Serial.println(F("CAM Capture Done."));
    total_time = millis() - total_time;
    Serial.print(F("capture total_time used (in miliseconds):"));
    Serial.println(total_time, DEC);
    total_time = millis();
    
    read_fifo_burst(myCAM);
    total_time = millis() - total_time;
    Serial.print(F("save capture total_time used (in miliseconds):"));
    Serial.println(total_time, DEC);
    //Clear the capture done flag
    myCAM.clear_fifo_flag();
    digitalWrite(LED_RED, LOW); //turn off to signify that image has been taken
    
  }

  if (buttonBlueVal == 0) {
    Serial.println("blue is being pressed");
    const char text[] = "ACTIVATE NECKLACE";
    radio.write(&text, sizeof(text));
    digitalWrite(LED_BLUE, HIGH);
    delay(500); //do some blinking first
    digitalWrite(LED_BLUE, LOW);
  }

  if ((buttonVal == 0 || buttonBlueVal == 0) && millis() - buttonTime >= 500) { //button is being pressed for reasonable time

    //Serial.println(millis()-buttonTime);
    ButtonPresses++;
    buttonTime = millis(); //reset buttonTime for next button press
    //Serial.println("Button is being pressed.");
    Serial.println("Button has been pressed this many times:");
    Serial.println(ButtonPresses);

  }

  else {
    //Serial.println("Button is NOT being pressed.");
  }

  if (ButtonPresses >= 1) {
    //digitalWrite(LED1,HIGH); //on, LOW for off

  }

  if (ButtonPresses == 2) {
    //digitalWrite(LED2,HIGH);

  }

  if (ButtonPresses > 2) { //more than 2,  reset!
    Serial.println("TIME TO RESET");

    digitalWrite(LED_BLUE, LOW);
    digitalWrite(LED_RED, LOW);

    delay(300); //do some blinking first

    digitalWrite(LED_BLUE, HIGH);
    digitalWrite(LED_RED, HIGH);

    delay(300);

    digitalWrite(LED_BLUE, LOW);
    digitalWrite(LED_RED, LOW);

    ButtonPresses = 0; //reset total button presses
  }
}

uint8_t read_fifo_burst(ArduCAM myCAM)
{
  uint8_t temp = 0, temp_last = 0;
  uint32_t length = 0;
  static int i = 0;
  static int k;
  char str[16];
  File outFile;
  byte buf[256];
  length = myCAM.read_fifo_length();
  Serial.print(F("The fifo length is :"));
  Serial.println(length, DEC);
  
  if (length >= MAX_FIFO_SIZE) //8M
  {
    Serial.println("Over size.");
    return 0;
  }
  if (length == 0 ) //0 kb
  {
    Serial.println(F("Size is 0."));
    return 0;
  }
  
  myCAM.CS_LOW();
  myCAM.set_fifo_burst();//Set fifo burst mode
  i = 0;
  while ( length-- )
  {
    temp_last = temp;
    temp =  SPI.transfer(0x00);
    //Read JPEG data from FIFO
    if ( (temp == 0xD9) && (temp_last == 0xFF) ) //If find the end ,break while,
    {
      buf[i++] = temp;  //save the last  0XD9
      //Write the remain bytes in the buffer
      myCAM.CS_HIGH();
      outFile.write(buf, i);
      //Close the file
      outFile.close();
      Serial.println(F("OK"));
      is_header = false;
      myCAM.CS_LOW();
      myCAM.set_fifo_burst();
      i = 0;
    }
    if (is_header == true)
    {
      //Write image data to buffer if not full
      if (i < 256)
        buf[i++] = temp;
      else
      {
        //Write 256 bytes image data to file
        myCAM.CS_HIGH();
        outFile.write(buf, 256);
        i = 0;
        buf[i++] = temp;
        myCAM.CS_LOW();
        myCAM.set_fifo_burst();
      }
    }
    
    else if ((temp == 0xD8) & (temp_last == 0xFF))
    {
      is_header = true;
      myCAM.CS_HIGH();
      //Create a avi file
      EEPROM.get(0, k);
      k = k + 1;
      EEPROM.update(0, k);
      
      Serial.print("Picture number=");
      Serial.println(k);
      itoa(k, str, 10);
      strcat(str, ".jpg");
      
      //Open the new file
      outFile = SD.open(str, O_WRITE | O_CREAT | O_TRUNC);
      
      if (! outFile)
      {
        Serial.println(F("File open failed"));
        while (1);
      }
      
      myCAM.CS_LOW();
      myCAM.set_fifo_burst();
      buf[i++] = temp_last;
      buf[i++] = temp;
    }
  }
  myCAM.CS_HIGH();
  return 1;
}

/*               
* Thanks to Dejan Nedelkovski, www.HowToMechatronics.com
/*Image Capture Device - "Necklace" Code,
Juan Meza 
This code operates our necklace's wireless capabilities.
It where the necklace should look for a signal and 
performs an action after finding said signal. 
*/


#include <SPI.h>
#include <nRF24L01.h>
#include <RF24.h>
#include <string.h>

#define LED_TEST 3 //green temp LED

const byte address[6] = "42069";
RF24 radio(7, 8); // CE, CSN

void setup() {

  pinMode(LED_TEST,OUTPUT);
  
  Serial.begin(115200); 
  radio.begin(); 
  radio.openReadingPipe(0, address); 
  radio.setPALevel(RF24_PA_MIN); 
  radio.startListening(); 
  
  Serial.println("Receiver is ready to receive...");
  digitalWrite(LED_TEST,HIGH);
  delay(300); 
  digitalWrite(LED_TEST,LOW); 
  
}

void loop() {
  
  if (radio.available()) {
    char text[32] = "";
    radio.read(&text, sizeof(text));
    Serial.println(text);

    String key = "ACTIVATE NECKLACE";

    if (String(text) == key){
      //blink LED! 
      digitalWrite(LED_TEST,LOW); 
      digitalWrite(LED_TEST,LOW); 
      delay(300); 
      digitalWrite(LED_TEST,HIGH);
      digitalWrite(LED_TEST,HIGH);
      delay(300);
    }
  }
  
}

Special thanks to Professor Zacharias and everyone else who helped us get so far!

The goal of this project was to create a specialized keyboard and mouse for our design client, Kim, to make gaming on her computer more accessible. We first interviewed Kim to understand her daily routine and challenges. From there we ideated different assistive devices to address these daily problems. We had learned that Kim has a physical disability that limits the use of her right hand so she depends solely on her left hand when playing games on her laptop. Kim can grasp objects with her right hand but needs the assistance of her left hand to release her grip. We wanted to address this challenge by designing a keyboard catered to her left hand and a joystick that has a release mechanism for her right hand. Hence, we built this gaming device catered to her needs and preferences. More information about our interview with Kim can be found here: https://courses.ideate.cmu.edu/60-223/s2022/work/team-dione-interview-documentations/

Final Product

a. Description

Our assistive Gaming Keyboard and Mouse device was designed to make gaming on the computer more accessible to our client Kim. The keyboard was created to Kim’s personal preferences and controls keys on the commonly used in gaming. Its interface inspired by classic arcade machines with the goal of making the gaming experience more enjoyable for Kim. We experimented with different layouts and shapes for the keyboard and finalized on a design that had optimal comfort and usability for Kim. The overall goal for this keyboard was to make the buttons used to play games more accessible for Kim’s left hand. In addition to the keyboard we also designed a mouse for Kim’s right hand. The mouse is a 3D printed joystick that controls the cursor and it has an inflatable plastic bag around it which helps our client release and firmly grasp the joystick.

b. Images

Front view of the device

start game by using joystick and playing snake game with keyboard

perspective view of the arcade keyboard

perspective view of the inflatable joystick

side view of the joystick

detail view of the convex 3d printed shell and led buttons

c. Narrative Sketch

When Kim wants to play a game on her laptop, she plugs in the USB-C cable into her laptop and presses the on switch. She knows the gaming device is up and running once the LED arcade buttons light up. She then controls the joystick with her right hand and uses the keyboard with her left hand. When she wants to release her hand from the joystick she presses the bottom left button on the keyboard. This button activates the air pump which pumps air into the joystick for 20 seconds. Once the joystick is inflated, Kim is able to easily remove her hand from the joystick. Then when Kim wants to use the joystick again she can press the bottom left button on the keyboard to deflate it. When Kim is done playing games she presses the switch off and unplugs the USB-C cable from her laptop.

Prototype and Process

This prototype was created to answer the design question of how each feature meets Kim’s needs and how we should change it to her convenience. The goal for our prototype was to create a works like prototype, focusing on the electronics and individual parts of the system performing the mechanisms we ideated.

Our prototype consisted of three different features, the joystick, the keyboard and the inflatable mechanism. The joystick was a functioning joystick that controlled the laptop cursor, the keyboard focused on the placement of the buttons and the ease of use for Kim and the inflatable mechanism tested its functionality for Kim’s use.

This is the first iteration of the keyboard. The labels show the letters the buttons were originally mapped to test and debug

wiring and testing the joystick on the Arduino micro

back side wiring for the arcade button keyboard

The prototype was effective in identifying areas of improvement for our design. Our initial keyboard iteration had a leveled surface where all the buttons laid flat. We had Kim test this design by playing a game so we can observe what adjustments to make. Kim was elated to see the colorful arcade buttons however her discomfort was evident when trying to navigate the buttons. She struggled with where to place her hand and some buttons seemed inconvenient to reach. Moving forward we wanted to experiment with the keyboard’s ergonomics to optimize comfort and usability for Kim. From Kim’s feedback we redesigned the keyboard to have a curvature on its surface and changed the layout of the buttons.

The joystick we used for our prototype was a small Arduino joystick from the physical computing lab. Our goal was to get the code and joystick to work for the prototype so moving forward we could experiment with its size, shape, and orientation. The prototype was functional in its control over the laptop. We could not receive much feedback on its current design from Kim since it was a functional prototype. However, we did have Kim grasp different sized objects with her right hand so we could gain a better understanding on the ideal shape for the joystick.

The third component to our prototype was the inflating mechanism. We designed a separate device to demonstrate the inflating and deflating concept for the joystick. This device consisted of an air pump, tubing, and a balloon. Kim tested if this mechanism was effective in releasing her hand so we could understand what size and shape would be ideal moving forward. From our observation and Kim’s feedback the balloon was effective but the air pump was too slow. Moving forward we sought out different air pumps that could pump air at a faster rate and experimented with more durable materials than a balloon.

figuring how to transit to a larger air pump

testing the fitting of airbag and joystick extension

This image shows all of the wires that had to be connected to the Arduino. We used tape to label the pin each wire should be connected to keep the wires organize in case they became detached.

This diagram represents the layout of the buttons and the number we assigned to each one of them to stay organized during the debugging process.

The two images above were diagrams we created for the debugging process. We created a system to test each button and ensure its code and wiring was correct

One of the final iterations o the keyboard with the convex surface prior to being painted.

testing the new air pump with the Arduino uno

The early prototyping stages of this project were focused on creating a device that works. After we developed a functioning keyboard, air pump and joystick we shifted our focus to the ergonomics. After the prototype critique and user testing we had many new ideas on how to improve the overall design to increase comfort and accessibility. We wanted to experiment with the size and shape of the keyboard. We drew inspiration from different keyboards we saw online and created a new model with a curved surface. In addition to redesigning the keyboard, we wanted to find a more durable joystick to use. With a more durable joy-stick we were able to model a 3-D printed shell personalized to Kim’s preferences that was easier for her to grip. Throughout the redesign process we had many conversations with Kim to receive more feedback on these new ideas. This bore fruit as we added contour lines on it to ensure that her hand does not slide off and we decided to include the factor of visual appeal to the keyboard.

To improve the user interface of the device we wanted to add a visual indicator to signal to the use the device is on. Our initial idea was to add an LED strip around the perimeter of the keyboard that would light up when Kim switched the device on. Then we learned there are arcade buttons that had LED lights we decided to move forward with these instead. In our final iteration of the project we programmed the LED buttons to light up when the device is on.

The inflatable mechanism went through multiple revisions. After the prototype critique we wanted to create a more durable and faster inflatable mechanism. We considered using water instead of air to pump it but decided air would be a wiser choice since we are dealing with electronics. We consulted Professor Zach about experimenting with other materials instead of a balloon such as casting a silicon bag which would be soft and would avoid issues where the air leaks out. To pursue this option, we met with Professor Garth and spent a class discussing this with him. However, he said the main difficulty would be to cast the silicon bag in the shape we wanted as the silicon gel would have to be constantly rotated in all directions to create our preferred shape. He suggested we adopt any existing product that serves the same purpose and this helped us in fixating on inflatable plastic bags that have a compartment with them. This was perfect as it properly wrapped around the joystick and inflated to the required volume for Kim’s needs.

There were a couple of issues during the development of the final product. Our first 3d print of the curved surface was not strong and was disintegrating so, we printed another with a thicker width to ensure structural integrity. One of the potentiometers of the joystick broke so only one axis worked for the joystick. We had to buy another one post the final presentation. Our schedule was delayed by a few days due to the revisions in our design as there were products like the joystick and the buttons that had to be ordered online. A few days before the final presentation, the micro-Arduino’s port broke while we were testing it which was stressful and we required a replacement.

Gantt Tracking 

This is the Gantt chart we used to plan and track our progress throughout the assignment. Even though we tried to follow it diligiently, we could not plan for certain problems that occurred. Some unforeseen challenges like the micro Arduino breaking set us behind schedule and the 3-D print deforming. However, it was useful to have this schedule to reference to make sure we were maximizing our work time and knew what tasks to complete next.

Conclusions

The final critique provided us with a lot of valuable feedback and new perspectives on the direction and potential for the mouse and gaming machine. When we first sketched the inflatable pump, we were all skeptical about its feasibility. In the end, we were happy to develop a mechanism that accomplished what we imagined. Even though this idea shows promise there is a lot of potential for improvement. One common issue that was raised about the pneumatic pump bag was “Pneumatics are slightly leaky” and “Pneumatic pump/bag on lever fragile to pinholes”. We were concerned about the longevity of the air bag and the pipe connections into the air bag but we opted to proceed with it to ensure the completion of a workable project. However, we did note that for further considerations of the project, we would explore other materials and better pneumatic fittings for the inflatable mechanism.

We spent the latter of this project working to improve the keyboard’s design, specifically its shape and layout. The feedback from the critique opened our eyes to other aspects of the design that can be refined. One person mentioned that the wires that connected the joystick and keyboard looked messy. In the future we would hope to have the keyboard and joystick be connected wirelessly. A wireless connection would improve the aesthetic and usability of the gaming device. Another piece of criticism we received was about the lack of labels on the buttons. Including labels would improve the user interface of the keyboard. Another design critique was that, “The clear box is a little messy with acrylic cement”. One of the critiquers gave us feedback with regard to the motive of using a clear acrylic box for the mount of the joystick and our rationale for glueing it together with acrylic cement instead of screws. We chose the acrylic box to display the inner mechanism which added to the overall aesthetic of the project however, the acrylic cement smudged the clarity of the acryclic build in certain places. This would be an area of improvement for further versions of the same project.

Designing for other people is a very rewarding experience. We all enjoyed working with Kim to create a device she was excited to use and could help her in her everyday life. Kim may have physical limitations but she does not allow it to inhibit her lifestyle. She has so many hobbies and interests I think we all felt inspired to do more of what we love after talking to her. It felt so special to have the opportunity to create a device that can make Kim’s hobbies and interests more accessible to her. The world is not designed for people with disabilities so it was rewarding to make Kim something that was for her. Seeing Kim smile and laugh using the gaming device made all the work we put in worth it.

The timeline of this project gave each of us exposure to the product design process. From the interview to the final product it is rewarding to see how much we accomplished in a short amount of time. For some of us, it was our first experience designing for a client. It was challenging at times to design for some one else when they are not present. We had to make decisions that we believed were in our client’s best interest but sometimes it felt like a guessing game. The feedback we did receive from Kim gave us direction and propelled us forward throughout the project. It felt like most of our ideas were hers and it was our responsibility to make them come to life. Compared to the last assignment the pressure of designing for a client weighs much heavier than designing for yourself. We not only wanted to deliver a working project but also a project that met Kim’s expectations. In the end we were able to develop a product that our client enjoyed using however with more user testing and feedback it could be improved.

Technical details

Schematic and block diagram

Code

• the project title,
• (optionally) your names,
• a description (short or long) of what the code does,
• any description of pin mapping that would be useful to somebody else trying to recreate your work,
• appropriate credit to any code source that you incorporated into your project, and
• (optionally) a license notice (i.e. copyright, CC BY-SA 4.0, the MIT License, release it to the public domain, or just follow your heart). If you have written code that you wish to keep strictly proprietary for any reason, please speak with the instructor about an exception to this documentation requirement.

//Gaming Keyboard and Mouse 

//This code controls the air pump, keyboard, and joystick. When the user turns the switch on the LED buttons turn on. There are 9 buttons on the keyboard and the code maps each of the buttons to a key. When the code reads the buttons as high then the code presses the key it is mapped to. After the button is done being pressed the code then reads the button pin as low and releases the key that was being pressed. This ensures that the key is only pressed once when the button is pressed once. The code for the joystick takes in the inputs from the potentiometer along the x and y axis and outputs a direction the computer mouse should move in. The code for the air pump inflates air when the bottom left button on the keyboard is pressed. The next time this button is pressed the air pump deflates the air bag surrounding the joystick. 

/*
Pin Mapping:
Arduino Micro Pin | type   | description 
------------------|--------|------------
A1                  Output   LED

A2                  Input    Joystick X axis

A3                  Input    Joystick Y axis

2                   Input    Switch(controlled whether the keyboard was on or off)

3                   Output   Pump

4                   Output   Pump

5                   Input    Button(controls shift key)

6                   Input    Button(controls space key)

7                   Input    Button(controls right arrow key)

8                   Input    Button(controls return key)

9                   Input    Button(controls up key)

10                  Input    Button(controls down arrow key)

14                  Input    Button(controls left arrow key)

15                  Input    Button(controls delete key)

16                  Input    Button(controls if pump is on or off)


*/


//https://docs.arduino.cc/tutorials/micro/keyboard-press


#include <Keyboard.h>
#include <Mouse.h>

//on button
//switch
const int onButton = 2;

//buttons 
//LED BUTTONS
const int upButton = 9;
int previousupButtonState = LOW;
const int downButton = 10;
int previousdownButtonState = LOW;
const int rightButton = 7;
int previousrightButtonState = LOW;
const int leftButton = 14;
int previousleftButtonState = LOW;
const int spaceButton = 6;
int previousspaceButtonState = LOW;
const int shiftButton = 5;
int previousshiftButtonState = HIGH;
const int returnButton = 8;
int previousreturnButtonState = HIGH;
const int deleteButton = 15;
int previousdeleteButtonState = HIGH;

//LED
const int LED = A1;


//constants for air pump control
const int pumpButton = 16;
int buttonState = HIGH;//pump input
const int PUMP1 = 3;
const int PUMP2 = 4;
bool pressVal = false;
bool AIRIN = 0;
bool pumpOn = false;
unsigned long timecount = 0;//this is to check whether the pump is on Periodically

//Constants for joystick control
//  pin numbers for switch, joystick axes, and LED:
const int xAxis = A2;         // joystick X axis
const int yAxis = A3;         // joystick Y axis
// parameters for reading the joystick:
int range = 12;               // output range of X or Y movement
int responseDelay = 5;        // response delay of the mouse, in ms
int threshold = range / 4;    // resting threshold
int center = range / 2;       // resting position value
 // whether or not to control the mouse
bool mouseIsActive = false;   
int lastSwitchState = LOW;   









void setup() {
   pinMode(onButton, INPUT); 
   pinMode(LED, OUTPUT);  
   pinMode(upButton, INPUT_PULLUP);   
   pinMode(downButton, INPUT_PULLUP); 
   pinMode(leftButton, INPUT_PULLUP); 
   pinMode(rightButton, INPUT_PULLUP); 
   pinMode(spaceButton, INPUT_PULLUP);   
   pinMode(shiftButton, INPUT_PULLUP); 
   pinMode(returnButton, INPUT_PULLUP); 
   pinMode(deleteButton, INPUT_PULLUP);  
   pinMode(pumpButton, INPUT_PULLUP);
   pinMode(PUMP1, OUTPUT); 
   pinMode(PUMP1, OUTPUT);
   
   Mouse.begin();   
   Keyboard.begin();
   Serial.begin(9600);
}

void loop() {
  //code for the keyboard
//Serial.println(" working");
Serial.println(digitalRead(onButton));
if (digitalRead(onButton) == HIGH){
  digitalWrite(LED, HIGH);
//upbutton:218/UP
  int upbuttonState = digitalRead(upButton);
   //replaces button press with UP arrow  
  if (upbuttonState == HIGH && previousupButtonState == LOW) {      
     Keyboard.press(218); 
     //delay(50);
   }
  if (upbuttonState == LOW && previousupButtonState == HIGH) {
    Keyboard.release(218);
    delay(50);
  }
   previousupButtonState = upbuttonState;
   
//downbutton:217/DOWN
  int downbuttonState = digitalRead(downButton); 
   if (downbuttonState == HIGH && previousdownButtonState == LOW) {      
     Keyboard.press(217); 
     delay(50);
   }
  if (downbuttonState == LOW && previousdownButtonState == HIGH) {
    Keyboard.release(217);
    delay(50);
  }
   previousdownButtonState = downbuttonState;

//right button:215/RIGHT
  int rightbuttonState = digitalRead(rightButton);
   if (rightbuttonState == HIGH && previousrightButtonState == LOW) {      
     Keyboard.press(215); 
     delay(50);
   }
  if (rightbuttonState == LOW && previousrightButtonState == HIGH) {
    Keyboard.release(215);
    delay(50);
  }
   previousrightButtonState = rightbuttonState;

//leftbutton:216/LEFT
  int leftbuttonState = digitalRead(leftButton);
   if (leftbuttonState == HIGH && previousleftButtonState == LOW) {      
     Keyboard.press(216); 
     delay(50);
   }
  if (leftbuttonState == LOW && previousleftButtonState == HIGH) {
    Keyboard.release(216);
    delay(50);
  }
   previousleftButtonState = leftbuttonState;
   

//spacebutton: 32
  int spacebuttonState = digitalRead(spaceButton);
   if (spacebuttonState == HIGH && previousspaceButtonState == LOW) {      
     Keyboard.press(32); 
     delay(50);
   }
  if (spacebuttonState == LOW && previousspaceButtonState == HIGH) {
    Keyboard.release(32);
    delay(50);
  }
   previousspaceButtonState = spacebuttonState;

//shiftbutton:224
  int shiftbuttonState = digitalRead(shiftButton);
   if (shiftbuttonState == LOW && previousshiftButtonState == HIGH) {      
     Keyboard.press(224); 
     delay(50);
   }
  if (shiftbuttonState == HIGH && previousshiftButtonState == LOW) {
    Keyboard.release(224);
    delay(50);
  }
   previousshiftButtonState = shiftbuttonState;


//returnbutton:176
  int returnbuttonState = digitalRead(returnButton);
   if (returnbuttonState == LOW && previousreturnButtonState == HIGH) {      
     Keyboard.press(176); 
     delay(50);
   }
  if (returnbuttonState == HIGH && previousreturnButtonState == LOW) {
    Keyboard.release(176);
    delay(50);
  }
   previousreturnButtonState = returnbuttonState;


//deletebutton:BACKSPACE(8)
  int deletebuttonState = digitalRead(deleteButton);
   if (deletebuttonState == LOW && previousdeleteButtonState == HIGH) {      
     Keyboard.press(8); 
     delay(50);
   }
  if (deletebuttonState == HIGH && previousdeleteButtonState == LOW) {
    Keyboard.release(8);
    delay(50);
  }
   previousdeleteButtonState = deletebuttonState;

}
if (digitalRead(onButton) == LOW){
  digitalWrite(LED, LOW);
}




//code for pump
  buttonState = digitalRead(pumpButton);
  if (pumpOn == true){
    if (millis()-timecount > 30000){
      pumpOn = false;
      digitalWrite(PUMP1,LOW);//turn motor off
      digitalWrite(PUMP2,LOW);
    }
  }
  if (buttonState == LOW ){
    Serial.println("the pumpbutton is pressed");
    timecount = millis();
    pumpOn = true;
    if(pressVal == false){
      pressVal = true;
      digitalWrite(PUMP1,HIGH);//turn motor on, starts inflating
      digitalWrite(PUMP2,LOW);
    } 
    else {
      pressVal = false;
      digitalWrite(PUMP1,LOW);//turn motor on, starts deflating
      digitalWrite(PUMP2,HIGH);
    }
  }

//Code for the joystick

  int switchState = digitalRead(onButton);
  // if it's changed and it's high, toggle the mouse state:
  if (switchState != lastSwitchState) {
    if (switchState == HIGH) {
      mouseIsActive = !mouseIsActive;
    }
  }
  // save switch state for next comparison:
  lastSwitchState = switchState;

  // read and scale the two axes:
  int xReading = readAxis(xAxis);
  int yReading = readAxis(yAxis);

  // if the mouse control state is active, move the mouse:
  if (mouseIsActive) {
    Mouse.move(xReading, yReading, 0);
  } else { 
     //else the mouse button is not pressed:
    // if the mouse is pressed, release it:
    if (Mouse.isPressed(MOUSE_LEFT)) {
      Mouse.release(MOUSE_LEFT);
    }
  }

  delay(50);


}// end of the code

int readAxis(int thisAxis) {
  // read the analog input:
  int reading = analogRead(thisAxis);

  // map the reading from the analog input range to the output range:
  reading = map(reading, 0, 1023, 0, range);

  // if the output reading is outside from the rest position threshold, use it:
  int distance = reading - center;

  if (abs(distance) < threshold) {
    distance = 0;
  }

  // return the distance for this axis:
  return distance;
}

The project objective is to design an assistive device for our client with physical difficulties. On Mar. 21, 2022, we held a short interview with our client, Haleigh, on the ground floor of Hunt Library; the talk gave us a chance to better understand Haleigh’s daily life, interests, difficulties, and some of her expectations towards a device that could make her life colorful. We also discussed some of our initial ideas and dived into details that specifically fit our client’s needs.

Interview With Haleigh

Agenda

1. Greetings and Self-introduction

We started our interview with a short self-introduction. It’s not in a very formal form as we already know each other in the project launch class in the previous week. We basically start with a short greeting outside the Hunt Library, talk about some activities before coming to this interview, and walk together into the open classroom located on the ground floor.

2. Project Introduction

After sitting down, we provided an overview of this project; we stated that our goal is to make a prototype of an assistant device for Haleigh within the 6-week timeframe; we also clarified that the project is an explorative one rather than a well-crafted commercial one. Before starting the interview, we asked our client and her mom if we could record our conversation and take photos during our conversion, and they agreed.

3. Understanding Needs

We started the interview by talking about our client’s routine daily life. We acknowledged that she’s interested in painting and reading during the conversation, and she also likes the wordle game. Our client and her mom also told us there are many difficulties in everyday life, like lacking grip and twisting on her hand, limited vision, and needing help when eating due to the posture problem. One of our members also holds Haleigh’s hand under consent to see how much grip she has.

4. Random Chat

The fourth part of the interview is a random chat. We don’t have preset topics but just started with a general question of what was a memorable thing in her life. The conversation was going freely; everyone on the table joined the talk and shared many exciting things like culture, favorite weather, and daily life, not just Haleigh talking and we listening aside. The random chat made us understand our client better and gave us a more vivid image of our client. There are occasional silent moments during the conversation, but we all tried to bring up new topics and make the talk continue.

5. Conclusion and Appreciation

The conversation lasted about 45 minutes. We appreciated our client’s time coming for this interview and her patience in answering our questions. Haleigh also told us she’s happy that have someone listening to her. In the end, we told them that instead of doing a one-time interview, we regard this meeting as a starting point so that we can keep in touch with our client through text when we have ideas or further questions. And we will send her our product at the end of this semester if possible.

Interview Notes

Summary / Takeaways

The interview provided us with a better understanding of our client; we got to know her daily life and interests and found some important points related to Haleigh’s difficulty that repeatedly appeared during the conversation.

1. Grip issue

The grip issue, and twist issue on Haleigh’s two hands cause one of the major inconveniences in her everyday life. She couldn’t drink easily because of lack of grip; we found it was her mom holding the cup for her and using a straw for drinking water. Our client also mentioned she likes pizza and steak, but she couldn’t cut them because of the grip issue and some twisting difficulties on her wrist; that means carrying and using any utensils is a heavy task for our clients. The same issue also limited her chance to work on her interests like painting, due to the problem of holding pens, brushes, or some similar tools.

Hearing this, we discussed the possibility of creating a device that integrates daily tools like utensils; it doesn’t need a lot of grip or precise gestures to use. One of our members held both of Haleigh’s hands under her and her mom’s consent and found that the amount of grip was good. Haleigh’s mom also showed us the gesture that Haleigh holds things most easily, and we took a photo of that pose. Thus we have empirical information on Haleigh’s grip and twist issue and enable us to work toward our client’s specific needs.

Gesture of Holding Things

Untensil Device Sketch

2. Skin issue

When talking about our favorite weather, our client said that she likes sunny days with a smile on her face. She doesn’t like cloudy days because she always like to stay outside and enjoy the sunshine and fresh air. But her mom mentioned that she couldn’t stay outside too long becase the solar radiation would hurt her skin and harm her eyes. We noticed that when hearing that the sunlight was “bad” to her, Haleigh’s face turned a little sad. We don’t want her deprived of the chance to enjoy the great weather and outdoor activities, so we arose another idea of designing a sun angle responsive “umbrella” on our client’s chair that can block the solar radiation when staying outside. This might need heavier engineering, but we kept it as one of our project directions.

Responsive Shading Sketch

Reflection

The meeting followed the agenda very well, we started the conversation smoothly and it went pretty well. Although there were some silence in that short 45 minutes, we all tried to drop new topics and keep the conversation going. Our client was so nice, trying to answer our questions in detail and sharing many exciting moments in her life. We are also impressed by Haleigh’s optimism and shinny smiles.

The interview was also productive; we got important information from our client like her interests, difficulties, and some expectations. With this knowledge, we can design a device that meets her needs better.

This is the documentation for the two interviews our team, team Dione, conducted with our client Kim and her father Bob. The purpose of this interview was to understand the Kim’s habits, hobbies and uncover any daily challenges she experiences that we could potentially solve. After we compiled a list of challenges, we devised ways to address them with assistive devices. Our goal was to gain Kim’s perspective on what it’s like living with a disability, how it impacts her daily routine and better understand her relationship with her assertive devices. With this in mind, we were able to ideate devices that Kim would be excited to use and that could make a difference in her every day life.

Meeting Agenda

The first meeting on March 21st includes basic understanding of client’s interests, needs and preference. We’ve prepared the following list of questions to ask:

• Introductions
  • Each team member introduce themselves
  • Have Kim introduce herself
• Living habits
  • Describe a typical day in your life
  • What parts of your day do you look forward to?
  • Are there any parts of your daily routine do you dread doing?
• Hobbies/Interests/Favorite Activities
  • How do you like to spend your free time?
  • What are some of your hobbies?
    • Are there any challenges or frustrations you face when doing these hobbies?
  • What are some of your interests?
    • Favorite foods, color, movies, books, games
    • Are there any problems that prevent you from pursuing any of these interests?
  • What extracurricular activities are you apart of?
  • Are there any hobbies or activities that you’ve always wanted to try but haven’t had the chance?

The second meeting on March 25th involves better understanding of client’s physical ability through demonstrations and artifact representations. It is to clarify confusion and misunderstanding in the first interview. Prior to the meeting we’ve prepare to ask Kim to perform the following task:

• To paint with Acrylic on Canvas to see how she uses brushes, how her father assist her during the process
• Play games on laptop or type on keyboard to show what keys and functions she usually uses

Questions for meeting 2:

• Questions about Kim’s experience using a keyboard
  • What games do you like playing on your laptop?
  • Which keys do you typically use on the laptop?
  • Aside from games, what else do you use your laptop for?
  • Do you use the mouse or the track pad?
  • Do you face any discomfort when typing on the key board or when using the mouse?
  • Do you use the keyboard to type our essays, letters, or emails?
• Questions about Kim’s experience with arts and crafts, specifically painting
  • How do you set up your art supplies to paint?
  • Do you paint at home or only in class?
    • What prevents you from painting at home?
  • What discomfort or challenges do you face when painting?
  • How do you clean up your art supplies?
  • How do you situate the canvas?

Summary And Documentations

The following text are points summarizing from the interview:

Most of our conversation with Kim focused on her love for painting and playing video games on her computer. From these conversations we learned Kim has limited motion in her right arm, she cannot bend her elbow and her hand can only grasp lightly but could not release. Due to these restraints, Kim heavily depends on her left hand when painting and playing video games. Kim and her father were excited by the potential to create a device that can help her use her right hand again. However, after learning about Kim’s extensive physical therapy journey to try and regain motor control over her right hand, we pivoted from that idea. Instead, we looked at the potential to create a device to help Kim use her left hand more efficiently and decrease Kim’s dependence on others when doing her hobbies.

Kim used to be righthanded but she is skilled with her left hand in craft, typing and writing, through our observation during the second Interview. In the first Interview we became acquainted with Kim’s lifestyle, she told us about her interests, hobbies and weekly routine. These are the notes we wrote down during the interview:

• Kim is enrolled in a cooking class
  • Enjoys making desserts
    • Loves the make and decorate cakes
    • Challenges
      • Cannot cut, slice, or dice due to the lack of control in her right hand
      • Maneuvering around the kitchen-kitchen layout
  • Helps out her Dad when he is cooking
    • They enjoy making pizza together
• Kim is in an art class
  • Enjoys Painting
    • Art class has special supplies that is catered towards artists with disabilities
    • Usually paints on a canvas laying flat on a table
      • Wants to try painting on a canvas when it’s on an easel
    • What Kim likes to paint
      • Flowers
      • Horses
      • Buildings
      • Landscape
      • Looking at the picture and draw on it
    • Challenges
      • Setting up the supplies requires assistance
      • Cleaning the brushes
      • Changing the dirty water out with clean water
      • positioning the canvas
      • Sometimes hard time to come up with inspiration
  • Enjoys pottery
  • A painting on the office
  • Likes to make arts and crafts
    • Crafts: necklace, bracelet
    • made a small miniature design of a her dream bedroom for her art class
    • Purse: beads hanging on the purse
• In a computer class
  • Loves playing games on her computer and tablet 
    • Candy crash
    • Card games
      • solitare
      • wants to play with physical cards but has not tried that yet
    • Bigger computers desktop
    • Tablet at home
      • Special equipment for placing the tablets
• Her school takes weekly field trips that she enjoys 
  • Movies
  • zoo
• Interests
  • Favorite movies: titanic, Mamamia Cinderella
  • Favorite music: country music, soft rocks
  • Loves the color red 
• Kim does therapy horse back riding every week
  • They have specialized equipment to get her from her wheelchair onto the horse
  • Kim loves horses

Kim’s Acrylic Painting during Interview

The way Kim is holding the brush with her left hand

Kim’s pottery with paint

Kim’s bedroom miniature that she made by herself

Kim’s painting of Titanic(her favorite movie) during her art class

Bracelet made by Kim

Discussions and Ideation Reviews

Kim enjoys spending her free time playing video games on her tablet and computer. When we learned Kim has limited motor control over her right hand we were inquisitive about her experience using a keyboard with one hand. The layout of the keyboard is designed for users to use with two hands. We thought there was potential to create a personalized keyboard for Kim to use when playing video games on the computer. This keyboard would have a layout specialized for only using your left hand and would only include keys that Kim uses when gaming. In the second interview, we pitched this design to Kim to gauge whether she would want or use this device. After pitching the specialized keyboard to Kim we learned that she doesn’t really use the keys when gaming on her computer since her computer is touch screen. She was still interested in the concept and the potential of using keys rather than touch screen.

The second design was inspired by Kim’s passion for painting. Kim described how much she enjoys painting in her art class and even demonstrated painting to us during the second interview. From our observation, Kim needs assistance setting up the painting supplies, changing out the water to clean the brushes, and cleaning up the supplies. Kim also struggles with cleaning the brushes and keeping the supplies secure since she can only use one hand. After observing these challenges Kim faces when painting, we began to devise ways we could mitigate them. We developed some ideas to create an electronic paint pallet that could be controlled by Kim with a set of buttons. The pallet can be rotated by Kim using a joystick(the pallet has LEDs next to each crevice that light up has it is being turned). The pallet can expel dirty water and refill with clean water from a tank of water stored within the device. The pallet has holes to attach each paint tube underneath and seal the paint. Kim can control how much paint is dispensed onto the pallet using the set of control buttons. When Kim rotates the pallet using the joystick and chooses the paint tube she wants to dispense(using the joystick and the selection is indicated by which LED lights up) she presses a button to confirm the selection. Once she confirms the selection, the desired paint is dispensed into the pallet. These are just some of the original features we came up with to assist Kim while painting. There is potential to come up with other ways to make painting a more independent task for Kim.

Team Callisto had the opportunity to speak with Mary D’Ottavio to design an assistive device for Mary’s daily life. Through the Ideate: Physical Computing course at CMU we were able to meet with Mary and ask her about how she navigates her life with her physical disabilities. The Team consisting of Sumayya, Yongwen, Juan and Tristan held multiple interviews with Mary to really understand what type of device would benefit her. On Wednesday, March 23 Mary told the team about her love for sports, writing, card collecting and emphasized some issues regarding accessing items that are far out of reach. After some brainstorming, the team decided to meet with Mary again on March 27 to go over their ideas and designs for another brainstorming session. 

Agenda:

Interview 1:

1. General introduction of the team and Mary
2. Thank for Mary for taking the time to do the interview, and ask what encouraged to join the project
3. Briefly explain project and establish the expectations we have
4. Ask what she does in daily life (try to branch off of the reason she joined the project)
5. Ask about hobbies
6. Ask about difficulties she has in her daily life
7. Ask about if she has any ideas for projects
8. Brainstorm some potential solutions to problems that Mary may have mentioned from the last few questions
9. Summarize the meeting
10. Thank Mary for coming and let her know we will keep her updated about our ideas.

Interview 2:

1. Short recap of last meeting, briefly discussing any key takeaways [5-10 Minutes]
2. Discuss any topics/ideas Mary may have thought of since last meeting 
3. Go over our current project ideas [5 minutes per idea, 20 minutes total] 
4. Get feedback from Mary, ideally try to narrow down to 1-2 most appealing ideas
5. Decide whether or not to move forward with one specific project, or to keep brainstorming

Summary and Takeaways:

Through the interviews we learned a lot about Mary’s activities and daily life. We were inspired by her strive to be independent and the many techniques she’s used to do her tasks on her own as much as possible. Although we were happy to learn Mary is very independent, this initially made it difficult to come with solutions that Mary would truly benefit from. It is why our first project idea was a solution to help with difficulties related to physical concerns such as reaching for items at the back of a closet or refrigerator. During the first interview we had discussed solutions to this problem but after a conversation with the professor, we realized our solution was very mechanical. And so, we went back to our notes from the first meeting and held a second interview with Mary. 

During the second interview, we clarified the purpose of the project was to find solutions that involve electronics. We eventually learned that Mary loves to write whenever she gets the chance. We wondered if Mary has difficulty noting ideas that she may randomly think of. As a result we suggested a device that could record notes by the press of a button. This way, Mary doesn’t have to worry about the time and difficulty of typing ideas. Instead, she simply scrolls through her recordings, minimizing hand movement and maximizing her ability to write more stories. 

Images of Mary’s fridge from the first Interview. We wanted to see the layout of her fridge when we were discussing the accessibility problem with items at the far back.

Design inspired from our second interview. This device is a recorder that would help Mary note ideas whenever and wherever.

Thoughts:

The two meetings we had with Mary both went well. Mary is a very positive lady who loves her life so much, which inspires us not only for this assignment but also for our life attitude. We followed the agenda to carry on the meeting and Mary cooperated with us nicely. Unfortunately we could not meet in person for the second interview. But even over the phone, we had a productive discussion about the pros and cons of different project ideas. We appreciated Mary’s honesty and feedback on our ideas as it greatly helped us revise our designs. Since everything went so well, we ended the meeting about 10 minutes earlier than expected. The second meeting was truly productive which made us know better about Mary’s needs and helped us decide on the final design: the voice recorder device. All the team members are very excited about this idea and hope the project goes well and benefits Mary in the future.

Our goal with the project is to create a personally tailored device that makes one aspect of our client’s life better. To better meet this goal, we first needed to know more about the life of our client, Allie. Because of the inconveniences of travel for Allie, we decided to meet online for the interview on Monday, March 21st.

Agenda

We decided to follow a loose schedule for the interview as topics can pivot depending on the conversation. The schedule goes as follows:

Icebreaker (5-6 min)

• Introduce ourselves and talk about one of our hobbies
• Ask Allie about her hobbies
• What drew her to participate/volunteer for this project
• Also, ask Allie if we can record the meeting

Explaining the project (5 min)

• We are trying to build a helpful divide, but we are not professionals. We will try our best, but the final product might not work.
• Give a quick timeline overview of the process; April 6th Prototype Crit; May 2nd Final Crit
• Ask Allie if she has any general questions regarding the project.

Discussing potential ideas (35 min)

• Ask Allie to draw or describe her daily routine
• Allie might have some ideas on what she wants to see built, might leave some time for that
• Then ask her about which tasks she enjoys doing, dislikes doing, or finds challenging to do; something she enjoyed doing and had to give up because of inconveniences
• Ask her to elaborate on the challenging tasks and start coming up with potential ideas as a group
• Mockup solutions with materials at hand or have a whiteboard on zoom so everyone can draw on it

Wrap Up and Conclusion

• Thank Allie for her time
• Final documentation
• Reiterate schedule if needed

Summary + Takeaways

We started the meeting by talking about our hobbies as an icebreaker. We learned that Allie is a very creative and social person. She said that she likes designing posters on the computer, painting, shopping, and going out to eat. When we asked about her daily routine, she told us that she spends most of her day on the computer, designing with canva and taking skillshare classes. As we got deeper into the conversation, Allie told us that she relies a lot on her helper to assist with day-to-day tasks, and she had some ideas for assistive devices that could give her more independence. She said that a grabber that could pick up objects that she drops off her bed at night could be useful, or an automatic wallet or cash box that could hand out the right amount of money to a cashier and receive the change in return when she goes shopping. 

Brainstorming ideas with Allie: Automatic money dispenser for shopping and self-correcting spoon to reduce spilling food

We also chatted about the different sources of inspiration for Allie’s posters. She told us that she creates posters for anything from stores to movies to her favorite foods. In a follow-up email, she showed us some of the posters she has made. Allie seems to draw a lot of her artistic inspiration from the world around her, so we also started to brainstorm ideas for devices that could challenge her to create posters related to different topics or objects.

Babe movie poster (top) and Toys R US store poster (bottom) designed by Allie

Post-Meeting / Team Thoughts

We all enjoyed being able to connect and chat with Allie! Going into the interview, we believed that Allie would have a variety of issues to tackle due to the extent of her disability. However, some of the problems she presented were past our experience level. We quickly realized that it would be tough for us to create devices to retrieve dropped items, pick specific items out of cabinets, or allow her to use a spoon without spilling. It was disheartening to tell her that we may not be capable of solving those types of problems, but Allie was so positive and friendly that it made it easy to work with her. 

After the meeting, we had plenty of ideas to work on, but once we started breaking down these ideas, they all seemed fairly difficult. A money box that could pay for items and receive change seemed like the best idea in terms of practicality and usefulness to Allie. However, we realized it could be difficult to make the money dispenser consistently work. We also came up with an idea to create an animated Tamagotchi-like game that would give her new ideas for poster designs. We thought this device could be cute and fun for Allie, but this didn’t include very many physical computing components. Finally, we brainstormed a design for a self-correcting spoon. This concept was interesting and useful for Allie’s day-to-day life, but the mechanics would be difficult to achieve on a small scale.  We decided to focus on the money box idea moving forward as it seemed to be the most feasible and could be simplified if need be. 

Overall, we all agreed that the interview with Allie went smoothly. We followed our agenda for the most part, but we also didn’t force ourselves to stick to a rigid schedule so that we had room to let the conversation flow naturally. This helped us come up with more ideas and get to know Allie better as a person. If we were to do anything again, we would have asked Allie about some of the other hobbies she mentioned during the icebreaker that we didn’t have time to discuss in-depth. This information could have given us even more ideas for assistive devices to consider.

For the final project in Physical Computing, we have formed a team of three (Team I/O) in order to create an accessible device for someone with physical disabilities. Our end goal is to create something that is useful and relevant for our client, something driven by her specific wants or needs so she can take home a device unique to her. Our client is Alycia, a blind woman who is also hard of hearing. The purpose of this meeting was threefold: to get to know her and introduce ourselves further, to get to know her wants that would make her life easier, and to get to know her needs – the immediate things that need to be addressed in her life. Present at the meeting was Alycia, Noni, Catherine, and David. It was held over zoom on Monday, March 21st at 10:30am. 

Our Meeting Agenda

The meeting agenda was intentionally loosely structured. We wanted to allow room to run in whichever direction we were taken, while also being able to hear enough of Alycia’s needs and wants to ideate a device that addresses them. Here is what we initially planned:

1. Introductions and icebreaker [everyone answers questions] (5-10mins)
   1. Introduce yourself
   2. Tell us a fun fact 
2. Entering conversation [client answers] (30-40mins)
   1. Why did you volunteer for this project? What was of interest to you?
   2. Do you have any activities in your daily life that are difficult, frustrating, or otherwise seem like they could use an intervention?
   3. What are some activities that you enjoy doing?
   4. What are the main challenges you face that you would want to be addressed?
   5. If you could have anything to improve your daily experience, what would it be?
   6. What’s your daily schedule?
3. Conclusion (2 mins)
   1. Thanking Alycia for her time 
   2. Do you have any questions for us? Or last minute thoughts/things to add?

Meeting Summary and Major Takeaways

We started off the meeting by asking Alycia why she chose to take part in this project, to which she said that she wanted to connect with more people and help raise disability awareness. She also mentioned that there are many products that claim to help people with disabilities, but without actually speaking with one and finding out what they need help with, these products oftentimes cause more harm than good.

We continued this conversation by asking Alycia if there are any activities in her current life that she finds difficult or frustrating. This led to one of our main ideas: a braille-friendly alarm clock. Alycia explained that her current alarm clock is an analog one that connects to a bed shaker. However, it is hard to set the alarm time accurately as she has to feel for the hands’ position while adjusting the clock so it is hard to set the exact time she needs. This often leads to her alarm going off five minutes before she actually wants to wake up. She has also tried talking clocks, but because she is hard of hearing, those don’t work well for her either. Alycia also showed us what her alarm clocks looked like, which we took screenshots of through zoom.

Alycia’s analog alarm clock with two plugs connected to the bed shaker and power source

Example of bed shaker Alycia uses that vibrates when the alarm goes off

Through this, we found that Alycia is a very diligent person who keeps to a strict schedule. Whenever her alarm rings, she gets up immediately and carries out her day, which is why having a well-working alarm clock is very important to her. She also told us she wakes up at 5:30 am every day and we were curious why she did that, so we asked if she could describe what she does in a day. As she was describing her day, she started talking about how she spends a lot of time figuring out how to find directions to her microwave meals.

Because of her debilitating eyesight, she can’t see the box directions and relies on an app to scan the packaging for cooking directions and read it out for her. The app she uses to scan for directions is called Seeing AI, developed by Microsoft. The app is constantly improving and new features are being added, which is very helpful for Alycia Even so, it’s hard for her to position the phone correctly so that the app can see the directions, and she doesn’t know which side the directions are on so she has to continuously rotate the box.

Apart from these two main frustrations, she says that she is able to do most things well and doesn’t need too much help. She spends her days going to class, watching hockey games, playing dice world, and taking care of her cats. There are also many new developing technologies like seeing AI that she finds which greatly help her through her daily activities.

Post-Meeting Thoughts and Reflection

In general, the meeting went very well. We managed to get some good ideas to work with, which was the main goal we had when we entered the interview. Alycia provided us with a clear problem she had right off the bat, allowing us a good starting point, albeit skipping through the initial process of finding what her general life was like and “base scoping” ideas. That being said, it was a bit difficult to get off this initial path once we were on it, and there was a bit of concern that, should this initial idea not work, that we would not have anything to work with. Thus, we tried generalizing the topic more to break away from going down one idea too quickly; we went back to our initial agenda of finding what her life was like, and what she typically did. This allowed us to investigate and find other ideas, like the issue with scanning and packaging. As such, our ideation process went better, though still with concerns that we walked away with only two directions. Further discussion afterwards revealed that we should have asked more about certain details, such as the specifications of the bed shaker, or more about her life in general. This would allow us to explore other ideas to these problems, like wearables as an alternative alarm clock, or entirely different directions, such as something involving her cat or other aspects of her life.