This project was a 7-week challenge to create a device that can provide assistance to a person living with a disability. We collaborated with Community Living and Support Services (CLASS), a local Pittsburgh nonprofit, to gain insight into the existing stage of assistive devices, proper methods and practices when working with clients that live with disabilities, and also to connect us to Keith, our client for the project.

Keith, our client, was a machinist who suffered a traumatic head injury, resulting in headaches, reduced mobility in his left side, and forgetfulness. Beyond that, he is a lovable grouch and has an appreciation for cheesecake. We documented our initial interview with Keith here.

What we built

Keith’s Explosive Schedule Reminder is a handheld phone-sized device that reminds him of his schedule. A screen on one side displays his upcoming scheduled event and a buzzer inside beeps when event is five minutes away from starting.

Keith’s Explosive Schedule Reminder

Keith’s Explosive Schedule Reminder

Keith’s device to scale

Internal parts of the device

Explosive side detail

Property of side detail

Battery pack detail

Battery pack detail

When Keith is at CLASS, he goes out for a smoke break. After he’s done, he’s having trouble remembering where he is supposed to report to. He grabs his reminder device and reads a display screen that says Math Class along with the location, time, and instructor. With this information, he is able to easily navigate back to the class on his own.

How we got here (prototype and process)

a. Prototype

Prototype 1 Demo

Prototype 1

Prototype 1 Deconstructed

Prototype 1 scale

Prototype 2

These two prototypes were designed to help answer the design question: What form and user-experience considerations are suitable for Keith’s device?

The first prototype was created using a white board material and blue LCD screen, which cycles through Keith’s schedule every few seconds. The second prototype is fixed to a chipboard, and reads out a class description, as well as vibrating.

The first prototype was specifically intended to show Keith what the device we were making would look like, so that we could further understand what limitations he might have towards adopting this new device. Size, readability, and usability were paramount. The second prototype was intended to show Keith what alert features we could incorporate, between audio and haptic feedback.

Our strongest piece of feedback from Keith was that he didn’t particularly care about the device itself, but wanted it to be red and read “explosive” on it. It was a frustrating critique session because Keith wasn’t very enthusiastic about the device, but we were able to ascertain that he could not read the LCD screen due to low contrast, but did not mind the size and shape of the device we were making for him. The feedback gave us a direction to move forward, but was demoralizing due to Keith’s disinterest. We hoped that by customizing the device to his preferences, he may be more excited about his device. Something we got from Bill was that the vibration motor wasn’t as important as an audio feature. Keith may not carry the device on his person at all times, and also may be wearing too many layers to feel a vibrating motor. We decided to limit our scope and focus on audio instead.

b. Process

Initial ideation was inspired by Bill’s sharing of Keith’s schedule at CLASS:

Keith’s Schedule

We found out that Keith had a pretty static schedule (with the only variations being instructors or irregular meetings off campus like with CMU). However, we also discovered that Keith keeps his schedule crumpled up in his pocket. This inspired us to make a device that he can’t lose or destroy, which he can keep on his person to remind him of his schedule, as well as reading it out when a scheduled item is close. Here is the very initial ideation sketch we made:

Ideation sketches

Some process drawing on the scale and form:

Whiteboard sketching of early idea

We initially conducted all tests and prototyping with a sound shield for the Arduino Uno, hence the scale and lack of real prototypes in the first stages of the project

Working with the e-ink display upon switching from the LCD after Keith’s feedback:

E-ink display testing

Soldering pins to the Adafruit board after receiving it late in the process:

Bhairavi soldering pin heads onto the Adafruit board

The craft and creation for the housing of the device was the largest challenge outside of electrical components, as we want the appearance to be appealing to Keith.

Solidworks model

Process for painting the box

Progress tracker:

Gantt Chart

This is the chart we used to track our process. Unfortunately, we did not actually receive parts until Monday the 24th, which lead to a lot of technical difficulties in the final phases of our project. We planned throughout the project to add a voice feature to the device to remind Keith of his classes, like his Alexa does at home. However, given our size limitation, we sought out an Adafruit board and music-maker in order to achieve this. We hinged our entire project on this, and the fact that we received the board so late, as well as frying it the night before, caused us to end up incorporating a buzzer and Arduino Micro instead. This decision was made out of desperation, and ideally we would have had more time to actually work with the final parts together.

4. Conclusions and lessons learned

Our largest realization through the making of this process was that Adafruit parts are not reliable, and that we should’ve had more contingent plans in place instead of putting our final prototype together the week before. We experienced a lot of extraneous circumstances that prevented our final product from being as successful as initially intended, but ironically had stumbled upon a product that might’ve done Keith more good on accident.

One of the most repeated pieces of feedback we got with our device was that Keith does not need to keep his device on him at all times. It could, instead, live in his cubby all day and serve as a location to ground him. If he forgets where to go, he will always know to return to his cubby to look at the device’s screen. From both critique feedback and Keith’s own admissions, we realized the primary feature of the device should be the screen, rather than the voice reminders. If we continue with this line of feedback, we might consider investing in a larger screen so that Keith can read the screen from a longer distance, as well as having a pop-up stand that allows the device to stand up on its own. Additionally, we could consider installing a re-chargeable battery and a button to toggle between voice/buzzer states and a muted state. These are all features that are dependent on the primary function of the device, which change based on what role it might have in Keith’s life.

Something that we learned pretty early on was that Keith was a client that was pretty content with his life. His schedules and habits weren’t bothering him, and he wasn’t particularly enthusiastic about making any big changes. This made interviewing him difficult, as we didn’t have a great grounding point for him to focus on ideating or considering making changes. Something promising brought up during our final critique was how our “final” device felt more like a late-stage prototype. This final device could very well be a device that Keith incorporates into his life for a few weeks and gives us concrete feedback on, which we can then respond to and adjust for the next stage of the device. This could be a good step to take for Keith, or any candidate that perhaps needs something physical to actually try out, rather than just imagining, which we were having him do previously.

The one particular strength of our final prototype was the visual design, which was something we’d identified early on as a feature that posed extremely high value to Keith in his decision to actually adopt any new technology. Creating any device to “assist” in someone’s life is by definition rather invasive, so we saw making that transition as personalized and meaningful as possible as a primary goal for whatever our device ended up being. As such, we spent a lot of time talking to him about his preferences in terms of color and decoration. In the final critique, we discussed possible expansion points for this project, and whether it had any standing as a product on mass production. IoT and such. The device can be applied to a variety of settings that choose to stay phone-free, such as summer camps or hospitals. A head administrator can hard code schedules into a series of devices using a master device or app and assign them to people. For settings specifically like CLASS, customizability for each device is paramount and can be maintained in this model.

One last concluding tidbit to this story is the fact that Keith loved his device. He was absolutely blown away by it, even though it was not necessarily to the fidelity that we had hoped we would be able to present. It just goes to show how much customizability matters when creating personalized devices, and would likely increase the device’s effectiveness tenfold (if we actually got it to work).

Keith’s Thumbs Up

5. Technical details

a. Schematic and block diagram

Original Block diagram:

Final Block Diagram

Block Diagram of final project

Schematic of our final project.

 

b. Code

// Keith's Explosive Schedule Reminder
// Team Members: Ethan Xu, Bhairavi Chandersekhar, Caleb Sun
// Code displays upcoming event information for Keith on an Adafruit ThinkInk e-display, as well as buzzing 5 minutes before an event

#include <RTClib.h>
#include <Wire.h>
#include "Adafruit_ThinkInk.h"

#define EPD_CS      9 // white
#define EPD_DC      10 // purple
#define SRAM_CS     6 // brown
#define EPD_RESET   8 // can set to -1 and share with microcontroller Reset! // RST blue
#define EPD_BUSY    7 // can set to -1 to not use a pin (will wait a fixed delay) // BUSY blue

#define buzzerPin 18

RTC_DS3231 rtc;
char t[32];
int h;
int m;
int d;

// 2.13" Monochrome displays with 250x122 pixels and SSD1680 chipset
ThinkInk_213_Mono_BN display(EPD_DC, EPD_RESET, EPD_CS, SRAM_CS, EPD_BUSY);

void setup() {
  pinMode(buzzerPin,OUTPUT);
  Serial.begin(115200);
  while (!Serial) { delay(10); }
  Serial.println("Adafruit EPD full update test in mono");
  display.begin(THINKINK_MONO);
  rtc.begin();
}

void testdrawtext(const char *text, uint16_t color) {
  display.setCursor(0, 0);
  display.setTextColor(color);
  display.setTextWrap(true);
  display.print(text);
}

void loop() {
  //setting up RTC module
  DateTime now = rtc.now();
  sprintf(t, "%02d:%02d:%02d %02d/%02d/%02d", now.hour(), now.minute(), now.second(), now.day(), now.month(), now.year());
  h=now.hour();
  m=now.minute();
  d=now.day();

  if((h==9) && (m==25) && (d=="Tuesday")){
  bitClear(DDRD,5);
  bitClear(DDRB,0);
  digitalWrite(buzzerPin,HIGH);
  delay (300);
  digitalWrite(buzzerPin,LOW);
  delay (200);
  digitalWrite(buzzerPin,HIGH);
  delay (300);
  digitalWrite(buzzerPin,LOW);
  delay (200);
  digitalWrite(buzzerPin,HIGH);
  delay (300);
  digitalWrite(buzzerPin,LOW);
  delay (200);
  display.clearBuffer();
  display.setTextSize(3);
  // spaces in testdrawtext are for center-aligning text; have to be done manually
  testdrawtext("    Math       at 9:30AM     Room 111   w/ Genevieve", EPD_BLACK);
  display.display();
  }
  //delay of 5 seconds is to serve as safeguard against e-ink display frying if updated too quickly
  delay(5000);

  if((h==10) && (m==55) && (d=="Tuesday")){
  digitalWrite(buzzerPin,HIGH);
  delay (300);
  digitalWrite(buzzerPin,LOW);
  delay (200);
  digitalWrite(buzzerPin,HIGH);
  delay (300);
  digitalWrite(buzzerPin,LOW);
  delay (200);
  digitalWrite(buzzerPin,HIGH);
  delay (300);
  digitalWrite(buzzerPin,LOW);
  delay (200);
  display.clearBuffer();
  display.setTextSize(3);
  testdrawtext("    Lunch      at 11:00AM    Room Main   w/ Courtney", EPD_BLACK);
  display.display();
  }

  delay(5000);

  if((h==11) && (m==55) && (d=="Tuesday")){
  digitalWrite(buzzerPin,HIGH);
  delay (300);
  digitalWrite(buzzerPin,LOW);
  delay (200);
  digitalWrite(buzzerPin,HIGH);
  delay (300);
  digitalWrite(buzzerPin,LOW);
  delay (200);
  digitalWrite(buzzerPin,HIGH);
  delay (300);
  digitalWrite(buzzerPin,LOW);
  delay (200);
  display.clearBuffer();
  display.setTextSize(3);
  testdrawtext(" Stress Mgmt   at 12:00PM    Room 111   w/ Michelle", EPD_BLACK);
  display.display();
  }

  delay(5000);  

  if((h==12) && (m==55) && (d=="Tuesday")){
  digitalWrite(buzzerPin,HIGH);
  delay (300);
  digitalWrite(buzzerPin,LOW);
  delay (200);
  digitalWrite(buzzerPin,HIGH);
  delay (300);
  digitalWrite(buzzerPin,LOW);
  delay (200);
  digitalWrite(buzzerPin,HIGH);
  delay (300);
  digitalWrite(buzzerPin,LOW);
  delay (200);
  display.clearBuffer();
  display.setTextSize(3);
  testdrawtext("    Music      at 1:00PM     Room 111    w/ Maggie", EPD_BLACK);
  display.display();
  }

  delay(5000);

  if((h==13) && (m==55) && (d=="Tuesday")){
  digitalWrite(buzzerPin,HIGH);
  delay (300);
  digitalWrite(buzzerPin,LOW);
  delay (200);
  digitalWrite(buzzerPin,HIGH);
  delay (300);
  digitalWrite(buzzerPin,LOW);
  delay (200);
  digitalWrite(buzzerPin,HIGH);
  delay (300);
  digitalWrite(buzzerPin,LOW);
  delay (200);
  display.clearBuffer();
  display.setTextSize(3);
  testdrawtext("  Wii Games    at 2:00PM     Room 111      w/ Bill", EPD_BLACK);
  display.display();
  delay (5000);
}
}

c. Design files

Keith Stl Files

Our client Mark lives with a visual impairment and mild memory loss due to a traumatic brain injury. From our conversation with Mark and his caretaker Bill, we learned that Mark’s day-to-day life can be improved if he had a device that helped remind him of his daily schedule in a way that integrates seamlessly into his existing habits. We met with Mark over the course of the project, twice in person and once over the phone. During these meetings, Mark gave us crucial feedback to fine-tune this project to his needs.

What We Built:

We created a handheld playback device that stores and plays pre-recorded audio messages. Each message is assigned to a specific hour of the day. When Mark picks up the device and holds it up to his ear, the device plays the message corresponding to the current hour. If Mark wants to hear reminders scheduled for other hours of the day, there is a dial located near the user’s thumb that allows for scrolling forwards or backwards through all 24 hours of the day. The device is designed so the user can comfortably scroll the dial whilst holding the device to the ear to listen for audio cues rather than a visual display to navigate through the hours. The housing is specifically formed to Mark’s hands and uses ergonomic touch-points (areas of the device that come in contact with the user) to comfortably and intuitively use the device.

Image of the final product being held. (1)

Image of the final product being held. (2)

Final CAD model of the playback device (including finger loop, speaker & thumb wheel) (3)

Final CAD model with cover removed to reveal interior (4)

Quick tour of the SolidWorks file (5)

Narrative:

BEEP BEEP BEEP! Mark’s 8AM alarm goes off. Mark lets out a large yawn, grabs his Portable Schedule Playback Device, and raises it to his ear.

“8AM: Good morning! Remember to grab breakfast with Paul at Starbucks.”

Ah that’s right! As he is gets ready, he starts to wonder what he has in store for the rest of the day. He puts the device to his ear and dials the device forward to 1 pm.

“1 PM: Remember to grab groceries.”

Oh yes, I need to buy cabbages for my homemade Korean kimchi recipe. He puts the device away in his pocket. “Wait, what restaurant am I meeting Paul at again?” He raises the device to his ear once more:

“8AM: Good morning! Remember to grab breakfast with Paul at Starbucks.”

Starbucks! That’s right. I think I’ll try their new Unicorn Galactic Frappuccino Supreme.

How We Got Here:

Prototypes & Processes:

We developed our first prototype to answer a question we all had on our minds: for someone with limited vision, what is a shape that is easy for them to distinguish, handle and interact with?

With that question in mind, we set a few goals that we thought were important to the overall functionality and concept of the device.

• Mark should be able to pick up the device from wherever it’s stored/held (in his pocket, on his belt, on his cane…), raise it up to his ear and hear the message instantly without any fumbling around/trying to figure out the device’s orientation, where the dial is, where the speaker is, etc.
• Mark should be able to line up the speaker to his ear quickly and comfortably so that he is able to hear the audio message with no time wasted.
• The size and weight of the device shouldn’t add too much burden to Mark’s existing everyday carry that it discourages him to bring and use the device on a daily basis.

What that, our first prototypes were ergonomic mockups of the device. They all follow the idea that Mark would raise the device to his ear and hear the audio message corresponding with the current hour. He can also scroll forward or backwards in the day to hear his previous/future schedule. Initial prototypes looked like this:

Side-profile of ergonomic prototype #1 made of foam. This design intends for the index finger to use the scroll wheel and navigate to the right recording.

Side-profile of ergonomic prototype #2 made of foam. This design intends for the thumb to use the scroll wheel and navigate to the right recording.

Side-profile of ergonomic prototype #3 made of foam. This design intends for the index finger to use the scroll wheel (now vertical) and navigate to the right recording.

Side-profile of ergonomic prototype #4 made of foam. This design intends for the thumb to use the scroll wheel (now vertical) and navigate to the right recording. This ended up being the closest to the final design of the device.

From these prototypes, we were able to make some decisions on the overall shape of the device, as well as some bigger aspects such as the placement of the dial (if it would be used by the thumb or the index finger). From there, we moved towards some more high-fidelity models. Some criteria that we set for ourselves were that the dial should be used by the thumb, and finger grooves should be implemented for better grip and to help Mark quickly identify the orientation of the device and make corrections.

High-fidelity ergonomic model (without working speaker) with horizontal thumb scroll. Gripped to show fit in hand.

High-fidelity ergonomic model (without working speaker) with horizontal thumb scroll. Open hand to show ergonomics.

Prototype that we presented to Mark to hold and play with.

Getting closer… the shapes of these prototype lends a bit more space for components on the inside and has finger grooves pronounced enough that it helps the user find a good grip on it. In the prototype directly above, we implemented a speaker unit and rotary encoder dial to get ourselves a bit closer to the look and feel of the final product – as well as starting to think about the placing of components. At this point, we were advised that the housing is just too small to house anything, if we were to keep to this form factor moving forwards. At this point, we also got an Arduino UNO to read file on an SD card and play audio files stored within, so we integrated that into the foam prototype.

Speaker demonstrating playback of a pre-recorded audio file.

Final prototype given to Mark to test with working speaker to get a sense of the volume adjustments, etc.

This ended up being the prototype that we showed Mark to get feedback on.

Overall, Mark was happy with how the device felt and our planned user process. In particular, Mark enjoyed the size and form factor, the finger grooves, and the placement of the thumb dial. Upon explaining our proposed user scenario, we also learned that Mark actually owns a cassette player at home, which his staff uses to relay their planned schedule on to him. We explained to him specifically the function of the device and how it’s able to keep track of time, how it’s able to keep a record of Mark’s schedule for the whole day, and how Mark is able to hear his current activity by raising the device to his ear or scrolling the dial. Cameron also brought with him his skateboard controller for Mark to try out, since we had it on us. He ended up really enjoying how it felt in the hands – especially the placement and orientation of the dial which spun vertically, and a hole/ring for the index finger to hook onto. With the proposed user process, Mark noted that the skateboard controller feels more appropriate for that interaction, especially because of how easy it is to scroll the dial on the skateboard controller.

Cameron’s skateboard controller that Mark ended up liking more than our foam prototypes!

Hearing Mark’s feedback was incredibly helpful, especially because we were able to ask him what he specifically enjoys about these different proposed shapes. In the end, the shape of the skateboard controller set the tone for what future iterations of the housing would look like. As said before, we were advised that the size and number of components we had to integrate would become a problem if we kept the same form factor, so we made sure to start working on a formal audit of all the parts we would include, their size, number of pins, etc. Generally, the prototype feedback session went exactly how we wished it would, and it was very helpful to get Mark’s approval and say on the ergonomics of the device, as well as the proposed user process. Moving forwards, we began to modify the shape of the device through some really quick prototyping, whilst trying to reserve as much space as possible for the internals.

Cardboard cutout of the final housing. It is thicker than the skateboard to accommodate for all the internal components.

Another angle of the cardboard prototype of the final housing for the device.

Moving forward, we split up figuring out the components between each person. Stanley worked on the tilt switch and IR sensor, both of which are responsible for getting the device to play an audio message as soon as Mark lifts up the device to his ear, as well as ordering and figuring out a rotary encoder we had ordered, Lily worked on a charging module so that we could get a charging lithium ion battery instead of having to replace an alkaline battery constantly as well as a RTC module so that the device can keep track of the current hour, and Cameron worked on the microphone, speaker and a music-maker shield so that we could record audio, separate it, and play it.

Breadboarded wiring of IR sensor and tilt switch for initial “hold-up-to-ear-and-listen” mechanism.

Here’s the first attempt at getting the IR sensor to work with the tilt switch to mockup the put-up-to-ear-to-listen interaction. In this example, the tilt switch, when activated, provides all power for the components on the breadboard. The tilt switch was surprisingly responsive and consistent, that we didn’t have to do any additional work to make it so that as soon as the device was put up to the ear, the IR sensor would start reading and send a signal to cause an output (which in this case was turning on a white LED).

Breadboarded wiring of music maker shield (deprecated) and detented rotary encoder for “scroll-and-browse” mechanism.

Here’s Cameron’s first time getting the rotary encoder to work with a speaker. Each detent corresponds with 1 hour difference. Spinning upwards scrolls forward in the day, and spinning downwards scrolls backwards in the day. At this point, all audio files are pre-recorded and stored in an SD card. At the beginning of each message, a robot voice reads out the hour that has been reached, followed by the corresponding voice message. If nothing is played, the user can scroll past, which overrides the current message being played instantly. At this point, we were able to 3D print a quick test housing to get a better idea of size and scale, ergonomics, and feasibility in terms of stuffing in electronics.

Initial print of our final housing. Notice that it closely resembles the skateboard remote.

It’s rough, but the bottom half of the body can actually be held in place without any external mechanical fasteners, and can be popped open to access the inside. This enclosure turned out to be too small, especially because of how deep the finger grooves were.

At this point, we were very short on time due to external factors that made it difficult to work consistently. So we went ahead and modeled and printed a second iteration that was a little larger in size, and had shallower finger grooves and fillets to afford more space in the interior. This ended up being the final housing that we worked with.

Note how the finger grooves cuts into the interior of the playback device significantly

Final CAD models of the previous iteration

A comparison between the past and most recent iteration of Mark’s playback device. Note the difference in size and proportions.

Second higher quality prints of the final housing.

Final prints of the device.

Again, since we were short on time, we decided to work in parallel, where some of us began to work on the final construction of the internal components (soldering everything to an internal power supply and Arduino Pro Micro), and others on the code and main functionality. We decided to scrap the microphone recorder function. The only way we could record audio and separate it was through a shield, which was simply way too big to fit in the device. Other components we found had other caveats, such as only being able to record 10s of audio, which we didn’t even know if we could separate and assign to different hours. Instead, we switched to an MP3 player that could read MP3 files (as opposed to the SD card reader which could only read WAV files that had to be encoded in a specific way via third-party software). We did this in hopes that this couldn’t compromise the recording portion of the device too much… on the assumption that Mark’s staff could be trained to use their portable devices to record MP3 audio into an SD card. It’s really not ideal, but it’s what we had to do with the amount of time we have available. Here’s how soldering went:

Final soldering of Arduino Micro, tilt switch, IR sensor, battery, MP3 player, and speaker.

Soldering process part 1: tilt switch, latching switch, voltage regulator, battery cap, MP3 SD card reader, speaker unit

Soldering process part 2: RTC module and rotary encoder (hidden behind), and test fitting into the housing.

Conclusions

In the end, we did not have a working prototype to demonstrate. Although each component was soldered and in the right place, the transition from Arduino Uno to Arduino Micro brought unforeseen behavior and hardware failures beyond our control. The transition was an inevitable bottleneck of the process and perhaps too much faith was put into the swiftness of such a transition. That said, our progress still evidenced lots of hard work and intention, and we were able to still receive great feedback about the concept and fabrication of the device.

Overall, our final product was well received. A few people noted that the housing had good ergonomics in reference to the overall ‘hold’ of the device, as well as the placement of the thumb dial and index finger loop. Here is what one student said, “The device is actually very comfortable! I have relatively small hands and found the size and feel of the turning dial to be really comfortable to use.” We came a long way from the foam prototypes that we showed Mark. Even though we ultimately adopted a design derived from Cameron’s skateboard controller, the final size, shape, and feel came from many considerations and incessant iterating. To our delight, most people interacted with the device the way we intended. Each person seemed to instantly recognize the important touch-points that made the device effective. They held the device as intended, scrolled the wheel with their thumb, and raised the device to their ear in exactly the spot that it was designed for. We were also told that the tangible aspect of the whole housing was much appreciated, and that they could see how our focus on designing around user interaction played a large role in the conceptualization, design, and fabrication of the project as a whole.

When we handed the device to Mark, we saw that he had a bit of trouble holding it correctly at first, which was most likely due to a tradeoff we had to make in order to save space–removing the finger grooves. This was perhaps the biggest drawback in our final design, since the finger grooves were so well-received by Mark in our initial prototype presentation. Unfortunately, the breadboard was ever-so-slightly too wide and took up the entire width of the housing, requiring us to pivot the design. Since day one, space consideration was always an obstacle we were aware about with this project. We spent a lot of time planning for this and ultimately theorized that all the components could fit. Even though the size of the housing did inflate a bit from original design and we had to get rid of the finger grooves, we believe the final design and shape of the device trumped the slightly bulky feel and discomfort.

We also received positive responses in terms of the concept of the device, and how we streamlined the device to not add extra burden to Mark’s existing everyday-carry, habits, and muscle memory. Here were some features that were applauded: having the device play audio as soon as it’s raised and put up to the ear, the ability for the user to scroll forwards and backwards throughout the day, and the hatch to access the internals and replace the battery. Overall, our biggest goal was to design a device that (A) worked, and (B) added minimal training and cognitive load to use the device. In hindsight it would also have been nice to experiment with different textures to distinguish different parts of the device, since after all this device is meant to be used without the need for sight. Of course, it is difficult to test out just how successful our device would be without a finalized working product (which is a point that came up in all of our critiques and we fully acknowledge). However, since each design aspect for this device was made with evidence-based backing and consideration, I think the purpose and ultimate integration in Mark’s life was transparent.

From this project, we were able to experience the hard yet humbling feeling of putting in hours and hours of work and not getting a working product at the end. One of the biggest obstacles we had was deciding on the best method for audio recording and playback. There are so many projects online with this feature, but only so many work with our space constraints. Recording audio ended up being infinitely harder than playing audio, which we realized (a bit too late) can only be done by a handful of hardware components that we didn’t have. There were many obstacles and thus considerations we had to make throughout this project, but the brick wall that slammed in our faces at the tail of the project was transitioning from an Arduino Uno to Arduino Pro Micro. Code, designed specifically for the Micro, failed to upload due to the device unrecognizable by any of our computers. As soon as we pressed “Upload,” the Pro Micro would disconnect. We tried four different Micros, checked and resoldered all our connections, and refactored our code twice, yet still the device refused to connect. We were unaware of the extreme unreliability of the Pro Micro and had no time to change our design, hence our ultimate failure. Our mistake was not having a secondary “all-organs-out” version of our circuit so that we could at least show the functionality of our device. This failure hit us all pretty heavily, but ultimately it was a good lesson learned.

Technical Details

Block Diagram:

Circuit Diagram:

Code:

// Title: Portable Schedule Playback Device by Apples Group 
// Author: Lily, Stan, Cameron
// Date: 5/05/23
// Description: This code handles the logic for playing back the appropriate pre-recorded
// recording stored in the microSD card depending on what hour of the day the dial is set to. 
// When the code boots for the first time, the rotary encoder is set to the current hour of the
// day (read from the RTC) and thus this is the first recording that is played back. Each
// recording is prefaced with a voice telling the time of the day. The code handles interruptions
// at any point in the recording/preface so the user can quickly browse to the desired recording.
// 
//
//                   -------------------------------------------------
//                   |   Role      |   Variable Name   | Arduino Pin |
//                   -------------------------------------------------
//  Rotary Encoder   |  Right Pin  |   encoderPinA     |      2      | https://www.nobleusa.com/pdf/xre.pdf
//                   |  Left Pin   |   encoderPinB     |      3      |
//                   -------------------------------------------------
//    MP3 Playback   |  RX Pin     |   pinDfpRX        |      10     | https://markus-wobisch.blogspot.com/search?q=DFPlayer
//                   |  TX Pin     |   pinDfpTX        |      11     |
//                   |  Busy Pin   |   pinDfpBusy      |      12     |
//                   -------------------------------------------------
//             RTC   |  SCL Pin    |   pinSCL          |      A5     |
//                   |  SDA Pin    |   pinSDA          |      A4     |
//                   -------------------------------------------------

the project title,
(optionally) your names,
a description (short or long) of what the code does,
any description of pin mapping that would be useful to somebody else trying to recreate your work,
appropriate credit to any code source that you incorporated into your project, and

//


                  /* GLOBALS */

// Rotary Encoder Globals //
#include <Keyboard.h>
enum PinAssignments {
  encoderPinA = 2,   // right
  encoderPinB = 3,   // left
};

volatile unsigned int encoderPos = 0;  // a counter for the dial
unsigned int lastReportedPos = 1;   // change management
static boolean rotating = false;    // debounce management
volatile int rotationDelt;
boolean A_set = false;
boolean B_set = false;
boolean clicked = false;
volatile int rotationPos = 1;

// MP3 Playback Globals //
#include "SoftwareSerial.h"
const byte pinDfpRX = 10;  
const byte pinDfpTX = 11;  
const byte pinDfpBusy = 12;  
const byte dfpVolume = 0x11; // set volume of DFPLayer - in range: 0x00-0x30 - default=0x30  
SoftwareSerial mySerial(pinDfpRX, pinDfpTX);
# define Start_Byte 0x7E
# define Version_Byte 0xFF
# define Command_Length 0x06
# define End_Byte 0xEF
# define Acknowledge 0x00 //Returns info with command 0x41 [0x01: info, 0x00: no info]

// RTC Globals //
const byte pinSCL = A5;
const byte pinSDA = A4;
#include <RTClib.h>
RTC_DS3231 rtc;


// Other Globals //
bool start = true;
bool startRot = true;
const byte nfiles = 6;     // number of mp3 files on SD card  
static byte ifile = 0;      // number of file played next
unsigned long tellTimeDelay = 800;
bool playMemo = false;
unsigned long memoDelayTimeStamp = 0;
static byte nextFile = 0;





                  /* FUNCTIONS */

// MP3 Playback Functions //
//plays a file named Par2 from folder Par1
void dfpExecute(byte CMD, byte Par1, byte Par2) {
  # define Start_Byte   0x7E  
  # define Version_Byte  0xFF  
  # define Command_Length 0x06  
  # define Acknowledge  0x00   
  # define End_Byte    0xEF  
  // Calculate the checksum (2 bytes)  
  uint16_t checksum = -(Version_Byte + Command_Length + CMD + Acknowledge + Par1 + Par2);  
  // Build the command line  
  uint8_t Command_line[10] = { Start_Byte, Version_Byte, Command_Length, CMD, Acknowledge,  
         Par1, Par2, highByte(checksum), lowByte(checksum), End_Byte};

  // Send the command line to DFPlayer  
  for (byte i=0; i<10; i++) mySerial.write( Command_line[i]);  
}


// Rotary Encoder Functions //
// Interrupt on A changing state
void doEncoderA() {
  // debounce
  if ( rotating ) delay (1);  // wait a little until the bouncing is done

  // Test transition, did things really change?
  if ( digitalRead(encoderPinA) != A_set ) { // debounce once more
    A_set = !A_set;

    // adjust counter + if A leads B
    if ( A_set && !B_set )
      // encoderPos += 1;
      rotationDelt = 1;

    rotating = false;  // no more debouncing until loop() hits again
  }
}

// Interrupt on B changing state, same as A above
void doEncoderB() {
  if ( rotating ) delay (1);
  if ( digitalRead(encoderPinB) != B_set ) {
    B_set = !B_set;
    //  adjust counter - 1 if B leads A
    if ( B_set && !A_set )
      //encoderPos -= 1;
      rotationDelt = -1;

    rotating = false;
  }
}


// Print Debugging //
// formats and prints the time of the day (AM/PM)
void printTime(byte ifile){
  if(ifile == 12){
    Serial.print(ifile);
  }
  else{
    Serial.print(ifile % 12);    
  }

  if(ifile >= 12){
    Serial.println("PM");
  }
  else{
    Serial.println("AM");
  }

}






void setup() {
  // Rotary Encoder Setup
  pinMode(encoderPinA, INPUT);
  pinMode(encoderPinB, INPUT);
  // turn on pullup resistors
  digitalWrite(encoderPinA, HIGH);
  digitalWrite(encoderPinB, HIGH);
  // encoder pin on interrupt 0 (pin 2)
  attachInterrupt(0, doEncoderA, CHANGE);
  // encoder pin on interrupt 1 (pin 3)
  attachInterrupt(1, doEncoderB, CHANGE);
  Keyboard.begin();
  // set the encoder position to the current
  // hour of the day
  rotationPos = rtc.now().hour();

  //RTC Setup
  pinMode(pinSCL, OUTPUT);
  pinMode(pinSDA, INPUT);
  rtc.begin();       // initialize rtc


  Serial.println("\n\n\n\n\n\n\n---------------------\n\n");
  Serial.print("The hour of the day is ");
  printTime(rtc.now().hour());


  //MP3 Player
  mySerial.begin(9600);
  pinMode(pinDfpBusy, INPUT);   // init Busy pin from DFPlayer (lo: file is playing / hi: no file playing)  
  dfpExecute(0x3F, 0x00, 0x00);  // Send request for initialization parameters  
  delay(5000);
  dfpExecute(0x06,0x00,dfpVolume); // set volume DL=0x00-0x30, default=0x30  
  delay(30);            // have >20ms delays between commands  

}





void loop() {
  unsigned long currTime = millis();

  // speaker plays the MEMO (after tellTimeDelay milliseconds have passed)
  if(playMemo && (currTime - memoDelayTimeStamp) > tellTimeDelay){
    Serial.print("Playing memo for ");
    printTime(ifile);
    dfpExecute(0x0F,0x02, nextFile);
    playMemo = false;
  }
  
  rotating = true;  // reset the debouncer

  // register a left turn
  if(rotationDelt == -1) {
    Keyboard.press(KEY_LEFT_ARROW);
    rotationPos -= 1;
    if(rotationPos < 1){
      rotationPos = 24; // we wrap around to 24 when we go below 1
    }
  }

  // register a right turn
  else if(rotationDelt == 1) {
    Keyboard.press(KEY_RIGHT_ARROW);

    if(rotationPos > 24){
      rotationPos = 1; // we wrap around to 1 when we go over 24
    }
  }
  

  // if the encoder changed position...
  if(startRot || rotationDelt != 0) {

    Serial.println(rotationPos);
    startRot = false;

    ifile = rotationPos;
    Serial.print("Telling time for ");
    printTime(ifile);

    // ...speaker tells the TIME
    dfpExecute(0x0F,0x01,ifile);
    playMemo = true;
    nextFile = ifile;
    memoDelayTimeStamp = currTime;
  }

  Keyboard.releaseAll();
  rotationDelt = 0;

}

CAD Files:

Mark’s Playback Device SW Files

For this project, we were tasked to create an assistive device for a client with a disability. Our class focuses on small electronics and interaction, so we had to find a way to use those skills to create a device that would helpful for our client. We were connected to our client through CLASS, a local organization that provides a myriad of services for folks with disabilities. Over the course of six weeks, we met with our client, designed prototypes, tested prototypes, and finally created the final product. You can read more about the initial interview here.

What we built

We met with our client Darnisha and she told us how she would often set her phone down and forget where she had placed it. Additionally, due to her mobility, she had trouble reaching over to turn off her alarm on her phone in the mornings. The solution that we came up with solves both these problems with one device. The device hooks onto her wheelchair armrest and has a phone holder that lights up when the phone is not there to remind her to locate her phone. The device can unhook from the chair and sit on her bedside table to act as an alarm clock that is turned off by waving her hand over a sensor.

The final phone holder and alarm.

This video demonstrates the alarm function. The alarm is preset to Darnisha’s schedule, as it is very consistent and does not change often. When the alarm goes off, Darnisha turns it off by simply waving her hand over the top of the device.

Here, you can see the phone holder function. There are flashing rainbow lights that display when the phone is not in the slot to remind Darnisha that her phone is not on her. With the help of an IR sensor, when the phone is placed in the slot, the lights turn off.

The top view of the device. Here, you can see the small IR sensor in the middle of the phone slot, and the ultrasonic ranger on the side which turns the alarm off. Also shown here is the handle, which was laser cut acrylic that we then bent with a heat gun to fit the curve of Darnisha’s wheelchair armrest.

Darnisha’s phone lays diagonally in the slot to allow for her to easily grab the phone from underneath.

We made the back panel of the device removable in order to access the electronics inside. We achieved this by using small “T” joints with a bolt and nut. Additionally, we used a piano hinge so that the handle can lift up, slide over the armrest, and the device hangs over the side.

Show here is the bottom of the device, where there is a panel mount battery pack to easily change batteries. This is also where the ON/OFF switch is located.

A narrative sketch –

When Darnisha gets up in the morning she reaches out to her side table where her phone holder and alarm sits. As the alarm goes off the display screen says Get Up!!! She waves on top of the ultrasonic sensor to turn off the alarm. 

After she is all ready for her day, she attaches her phone holder to her wheelchair which helps her keep her phone at a more accessible position and avoids her from misplacing her phone. As Darnisha picks up the phone to use the LED strip lights up indicating her that the phone is missing from its position.

How we got here

Prototype

The fabrication prototype was designed to determine the best way for the device to attach to Darnisha’s wheelchair. The electrical prototype was designed to determine a couple questions: would the alarm be loud enough for Darnisha? Did Darnisha want to set her alarm each day or have it preset in the device? What color and brightness would be best for Darnisha?

The first prototype was designed to sit on top of the armrest, and have the ability to swing out over Darnisha’s lap to act as a phone stand while she uses her phone. Additionally, we had an electrical prototype of both the LED function and the alarm function. The LEDs are connected to an IR proximity sensor. When the sensor is covered, the lights are off, and when it is not covered, the lights turn on. The alarm was shown on an LCD screen. The alarm time was inputted by and numerical keypad. The alarm is turned off by waving over an ultrasonic ranger.

The cardboard prototype of the physical device.

This prototype was meant to sit on top of Darnisha’s armrest.

The top of the device swivels out to provide a phone prop for Darnisha to use while on her phone.

The prototype for the LED lights that turn on and off based on the IR sensor input.

The prototype for the alarm turned off by an ultrasonic ranger.

Showing Darnisha our first prototype design.

Here, we realized that having the device on top of her armrest would inhibit her use of the armrest. She decided that the swivel part wasn’t necessary for her and it would be more useful on the side of her chair.

We also tried out some alternative handle designs.

Throughout the prototyping process, we tested a variety of ways for the device to attach to the wheelchair, including it sitting on top of the armrest, connecting to a bar below the armrest, and hooking over the armrest, which is what we ended up going with. This choice still allowed for comfortable use of the armrest, and the device was still close enough that Darnisha could easily grab her phone. We also used this time to get as many measurements of the armrest as possible so that we could make the fit perfect, as this would be the last time we would see Darnisha in person before the final critique.

As far as the electronics, Darnisha decided that the buzzer we had used would be loud enough and wouldn’t need volume control. We learned that her schedule is very rigid and doesn’t change often, so she wanted the alarm times to be preset in the device. This meant that we would get rid of the keypad. Finally, we asked about the light brightness and color. Darnisha liked the brightness of the lights, and asked for them to be rainbow and flashing.

 Process

Once we received feedback on the prototypes, we got into the process of creating the final device.

Assembling the laser cut pieces. Somehow, all the pieces and components fit together on the first try, which was really exciting.

Here we tested out bending acrylic with a heat gun on a scrap piece.

Once we heated the acrylic till it was pliable, we bent around a PVC pipe that roughly matched the curve and diameter of the armrest.

Once we had mostly assembled the box, we spray painted it Darnisha’s favorite color – pink!

Then, we began assembling all the electronics inside. The holes and slots that were cut for the components were almost an exact fit, so it took some patience so get everything into the right spot.

Here, you can see everything in place, except for the back panel. The back panel is secured with t-joints with small nuts and bolts. It’s very secure, but can be removed if necessary to get to the electronics.

We got good feedback from Darnisha in terms of her preference for position, light, and usage during the prototype crit and her being in person helped us get the wheelchair measurements, which made it easier for us to design the attachment to the correct size. However after digitally prototyping the model and assembling it, we realized that there was an issue with the size of the phone holder. Even after measuring Darnisha’s phone size we should’ve added some allowance in the width and height, taking in consideration if she changes her phone that is bigger. Phone in the space was not laying flat because of the size issue but we framed it as a very happy accident by saying that ” Darnisha’s phone actually doesn’t lay flat, but that’s a feature and not a bug because it makes it easier to grab out of the device.” This was the major breakthrough during our process.

After analyzing the whole product we also realized that the LED strip was exposed and that it might not be the best decision in terms of safety for the product and the user. However we had fabricated and assembled the whole product, but we definitely noted this point down for future consideration.

The Gantt Chart was very helpful in aiding us to plan and divide our work. We didn’t diverge from it and managed to keep to our original goals as we gave enough space for each task and left a decent amount of room at the end to do some final touch ups. For the tasks we split 2 v 1 depending on the task. For example, Mo and I started modeling the device and making laser cutter files while Michelle opted to combine our already tested code in bits and pieces and solder the electronics. The effort was distributed fairly and we switched places wherever needed. Throughout the process we made sure to check in with each other and cover up for the other if needed so that there is no lag with the work.

Conclusions and lessons learned

Salient Findings from Final Critique

Our final critique was very formative and valuable to us, and we definitely learned a lot from it. Most of the feedback seemed to be in the vein that the device was very solid, and a few more tweaks or features could improve it.

Time display is rudimentary. Day of week could also be helpful.

We had a few people point out that the time display could be improved by adding additional information, such as the day of the week. There was also a similar comment about the digits in the display being formatted somewhat unconventionally, due to the lack of leading 0’s before each digit. This is definitely something that may seem like a small detail, but in practice could greatly improve user experience with a simple fix. In a future iteration, we would update the code to do so.

I think it would be helpful if the device showed its current charge since it’s battery-powered — or if it could be recharged.

Adding an indicator for current charge is something that had never crossed our minds while we were designing the product, but would definitely be extremely useful in practice. During critique, it was pointed out that this is especially important, because having a device unknowingly die out in the middle of the night may be problematic since it would prevent the alarm from going off in the mornings. Adding a battery percentage, or even a low-battery indicator would help Darnisha prepare in advance for this.

I appreciated the design and approachability of the product, the lights and motion sensor to turn off the alarm were well planned out while remaining fun and remaining true to the client’s personality. The strawberries were a cute touch

We were really happy to receive this feedback, because it was really important to us that this device would be something that would be fun for Darnisha to be a part of and that would be really useful to her in practice. We put a lot of time and effort in our design considerations, so it was great to see this re-affirmed.

I was mildly worried about the durability of the design and the security of the phone throughout perhaps a busier day.

We felt that this feedback was definitely a very relevant concern, especially after learning from Darnisha that there may be some difficulty keeping her phone in the holder when she boards the ACCESS transit service. In the future, we could use a more durable material or add additional reinforcements, other than just the wood glue.

Major Takeaways

1.  Often, devices and tools for disabled folks are often made by taking something else and tweaking it to fit their needs. It was very cool to be able to create a device fully for our client’s needs from the beginning. We could really take her abilities and desires into account and make it perfect for her, and not worry about making it “marketable” in any way.
2. Collaboration is crucial: Designing something for a person with a disability is not a solo endeavor, and instead requires a lot of listening and collaboration to truly understand their needs. This project was not only made for Darnisha, but with Darnisha.
3. Small details matter: When designing something for a person with a disability, small details can make a big difference. Something as simple as the placement of a sensor or simple measurements can have a significant impact on the user’s experience. It is important to pay attention to these details and make adjustments as needed. Something challenging in this process was building the device to the exact dimensions of Darnisha’s wheelchair and phone size. In the end, while the device fit and came together well, we could’ve taken even greater care to get the exact dimensions.

Concluding Thoughts

Overall, it was a super amazing experience to get to work on this project with Darnisha. It was really cool getting to build this device from beginning to end, all the way from designing the device to putting together a final product that she loved and was customized to her needs. We learned a lot about fabrication and the importance of considering all the small details of a design and a user’s needs. With such a complex project, we also learned to break down big tasks into smaller ones and to keep things modular, whether it be in the code or general planning. We also learned the importance of building extra time into our schedule and to account for last-minute issues that might come up.

If we were to do the project a second time around, we’d double-check measurements even more than we already did to ensure that everything fit perfectly. This would also involve checking fit more carefully before assembling the project, so that we could make adjustments as needed. We’d also make sure to take into consideration the space that would be taken up by joints & connections at the beginning of the process. If we had more time, we’d also pay closer attention to different usability elements, such as the LCD time display and add more features, such as adjusting the alarm, snooze button, etc.

Technical details

Block Diagram

Schematic

Code

/*
* Phone Holder and Alarm
* Team Peaches: Mo Cambron, Juhi Kedia, Michelle Liu
* 
* Code for a wheelchair attachment that lights up an LED strip
* there's no phone blocking the proximity sensor and has
* a motion-activated alarm clock functionality.
* 
* Pin Map:
  variable name | mode | pin | description
  -----------------------------------------------
  PING_PIN | input | 3 | reads & emits ultrasonic ranger data
  LED_PIN | output | 7 | LED strip
  PROXIMITY_PIN | input | A0 | reads IR sensor data
  BUZZER_PIN | output | 13 | makes buzzing noise
* 
* Credits:
* RTC alarm clock code & reference: https://adafruit.github.io/RTClib/html/class_r_t_c___d_s3231.html
* 
* Understanding how the RTC needs to be coded and how the information
* from the RTC can be displayed on the LCD:
* https://www.youtube.com/watch?v=aJcncPB0GHg
* 
*/

#include <Wire.h>
#include <RTClib.h>
#include <LiquidCrystal_I2C.h>
#include <PololuLedStrip.h>

#define I2C_ADDR 0x27 //LCD i2c stuff
#define PING_PIN 3
#define LED_PIN 7
#define PROXIMITY_PIN A0
#define BUZZER_PIN 13;
#define MAX_DISTANCE 200
#define LED_COUNT 27
#define PROX_THRESH 850

PololuLedStrip<LED_PIN> ledStrip1;
LiquidCrystal_I2C lcd(I2C_ADDR, 16, 2);
RTC_DS3231 myRTC; //Wiring of the RTC (CLK,DAT,RST)
//If  you change the wiring change the pins here also

// Alarm clock hours
const int WEEKEND = 12;
const int WEEKDAY = 7;

// note: change for demo purposes; should be 0 in actual version
int alarmMin = 0;

int proxState0 = 0;

unsigned long rainbowDelay = 7;

rgb_color on[LED_COUNT];
rgb_color off[LED_COUNT];

const int ALARM_TIMES[7] = {WEEKEND, WEEKDAY, WEEKDAY, WEEKDAY, WEEKDAY, WEEKDAY, WEEKEND};
int RAINBOW[7][3] =
{
  {255, 0, 0}, // Red
  {255, 30, 0}, // Orange
  {255, 200, 0}, // Yellow
  {0, 255, 0}, // Green
  {0, 0, 255}, // Blue
  {75, 0, 220}, // Indigo
  {148, 0, 211} // Violet
};

int hue = 0;

void setup() {
  lcd.init();
  Serial.begin(9600);
  lcd.begin (16, 2); //Initialize the LCD
  lcd.backlight();

  pinMode(PING_PIN, OUTPUT);
  pinMode(PROXIMITY_PIN, INPUT);

  if (!myRTC.begin()) {
    Serial.println(" RTC Module not Present");
    while (1);
  }

  // Uncomment on initial RTC setup to set the initial time
  // comment back out and reupload after initial setup, in order
  // to prevent RTC from resetting every time
  
  //  if (myRTC.lostPower()) {
  //    Serial.println("RTC power failure, reset the time!");
  //    // automatically sets the RTC to the date & time on PC this sketch was compiled
  //    //    myRTC.adjust(DateTime(F(__DATE__), F(__TIME__)));
  //    // rtc.adjust(DateTime(2023, 2, 20, 3, 0, 0));
  //  } else {
  //    myRTC.adjust(DateTime(F(__DATE__), F(__TIME__)));
  //    //    rtc.adjust(DateTime(2023, 3, 2, 10, 0, 0));
  //  }

  //  myRTC.adjust(DateTime(2023, 4, 26, 10, 0, 0));

  int* curColor = RAINBOW[hue];
  for (int i = 0; i < LED_COUNT; i++) {
    // on is red
    on[i] = rgb_color(curColor[0], curColor[1], curColor[2]);
  }
  for (int i = 0; i < LED_COUNT; i++) {
    // off is clear/nothing
    off[i] = rgb_color(0, 0, 0);
  }

  DateTime now = myRTC.now();
  Serial.print(ALARM_TIMES[now.dayOfTheWeek()]);
  myRTC.setAlarm1(
    DateTime(now.year(), now.month(), now.day(), ALARM_TIMES[now.dayOfTheWeek()], alarmMin, 0), // Set alarm time to 7 am
    DS3231_A1_Date // Set alarm to trigger when date & time match
  );
}

// Helper function to get Ultrasonic Ranger distance
int getDist() {
  pinMode(PING_PIN, OUTPUT);

  // Create a wave by alternating a HIGH and LOW pulse
  digitalWrite(PING_PIN, LOW);
  delayMicroseconds(2);
  digitalWrite(PING_PIN, HIGH);
  delayMicroseconds(5);
  digitalWrite(PING_PIN, LOW);

  // The same pin is used to read the signal from the PING))):
  pinMode(PING_PIN, INPUT);
  return pulseIn(PING_PIN, HIGH);
}

// Convert military time to AM/PM
int militaryToRegular(int militaryHour) {
  if (militaryHour == 0) {
    return 12;
  } else if (militaryHour <= 12) {
    return militaryHour;
  } else {
    return militaryHour - 12;
  }
}

// Display date on LCD
void displayDate() {
  DateTime now = myRTC.now();
  DateTime tomorrow = now + TimeSpan(1, 0, 0, 0);
  lcd.setCursor(0, 0);
  lcd.print(now.month());
  lcd.print("/");
  lcd.print(now.day());
  lcd.print("/");
  lcd.print(now.year());
  lcd.setCursor(0, 1);
  lcd.print(militaryToRegular(now.hour()));
  lcd.print(":");
  lcd.print(now.minute());
  lcd.print(":");
  lcd.print(now.second());
  lcd.print(now.hour() < 12 ? "am" : "pm");
}

unsigned long tm = 0;


// Flash rainbow colors on LED strip
void rainbow()
{
  if (tm % rainbowDelay == 0) {
    int* curColor = RAINBOW[hue % 7];
    for (uint16_t i = 0; i < LED_COUNT; i++)
    {
      on[i] = rgb_color(curColor[0],  curColor[1],  curColor[2]);
    }
    hue++;
  }
  tm += 1;
}

void  loop() {
  delay(100);
  DateTime now = myRTC.now();
  DateTime tomorrow = now + TimeSpan(1, 0, 0, 0);
  long val = getDist();
  int readVal0 = analogRead(PROXIMITY_PIN);
  proxState0 = readVal0 > PROX_THRESH ? 1 : 0; // whether or not proximity sensor is covered
  rainbow();

  if (proxState0 == 1) {
    ledStrip1.write(on, LED_COUNT);
  } else {
    Serial.println("off");
    ledStrip1.write(off, LED_COUNT);
  }

  displayDate();

  if (myRTC.alarmFired(1)) {
    while (val > 500) {
      tone(BUZZER_PIN, 1000); //You can modify the tone or make your own  sound
      delay(100);
      tone(BUZZER_PIN, 2000);
      delay(100);
      lcd.clear();
      lcd.print("Get up !!!"); //Message to show when the alarm is ringing
      val = getDist();
    }
    lcd.clear();
    myRTC.clearAlarm(1);
    myRTC.disableAlarm(1);

    // set alarm for tomorrow
    myRTC.setAlarm1(
      DateTime(tomorrow.year(), tomorrow.month(), tomorrow.day(), ALARM_TIMES[tomorrow.dayOfTheWeek()], 0, 0),
      DS3231_A1_Date // Set alarm to trigger when date & time match
    );
    noTone(BUZZER_PIN);
    delay(100);
  }
}

Design Files

Here, you can access the design files used to model and laser cut the device: Design Files

We are Team Guava, and our client is Shaude.

Over the weeks, we talked to and interacted with Shaude to get to know her better and to understand what she does. From those conversations, we were able to find and target aspects of her life that we could help to improve, such as picking fallen things up, helping her paint, and assisting her in communicating, especially in the morning.

Link to previous post: https://courses.ideate.cmu.edu/60-223/s2023/work/interview-with-shaude-team-guava/

1. What We Built

By having two boxes that can communicate with each other at a distance, we can give our client, Shaude, the ability to communicate with someone in her home much easier than a smartphone. In particular, we had one device for the client’s bedside that could send up to six predetermined messages for specific needs and two adjustable gradient indicators for both general mood and health to another device that could be placed anywhere in the home for the ease of the caretaker. We also made it so both devices run based on a wall outlet, removing the need to change batteries in the product.

Controller (left) and Display (Right)

Controller – Custom Slider Cap, Arcade Buttons, Sticker Papers

Controller – Internal Wiring

Display – LCD

Display – Internal Wiring

By having two boxes that can communicate with each other at a distance, we can give our client, Shaude, the ability to communicate with someone in her home much easier than a smartphone. In particular, we had one device for the client’s bedside that could send up to six predetermined messages for specific needs and two adjustable gradient indicators for both general mood and health to another device that could be placed anywhere in the home for the ease of the caretaker. We also made it so both devices run based on a wall outlet, removing the need to change batteries in the product.

Video of Our Devices Working as Intended.

Narrative sketch of how our product works

2. Prototypes and Processes

Our first prototypes were built in order to answer the following questions: 

• What inputs and outputs we wanted to use?
  • Such as how should Shaude enter her emotions and how should they be shown
• What colors and gradients made the most sense for out purpose 
  • Should we use light color or brightness?
• What scale should the device be at
• What should be the layout of the device 

We made a lot of prototypes testing the hardware that we would be using, such as the radio, the potentiometers and the Neo pixel bulbs. These were practical prototypes to see how the coding and functionality would look like. We also printed prototypes of the housing so we would be able to better address scale. Some prototypes were simple drawings exploring how different aspects of the outputs might be, such as drawing of different light gradients or LED configurations.

These sketches show our thought process behind our questions, and some things we wanted to explore in our prototyping.

Neopixel bulb light up for the first time (emotional)

This was our first version of slider to lights. This simple setup controlled the red blue and green value on the Neopixel. You can see that they all run to the same bulb and effect each other. The position on the slider effects the brightness of each color in the bulb. We were happy with this experiment because it was much like mixing paint, but with light, and it was fun to play with.

Above is second iteration of the potentiometer to lights. We tested mapping specific gradients to the lights. Here you can green to red as well as red to blue. You may notice the weird jump from blue to red in the lights. This was a coding issue we needed to battle and happens because the red value is much stronger than the blue.

LED strip alternative to neopixel attempt

This idea correlated to a sketch seen above, where the amount of mood or health corelates to a number of bulbs illuminated on the LED strip. This idea was scraped because it seemed a little confusing and added a layer of complexity that was unnecessary. This porotype was never brought to a fully functional level.

Above you can see a video of us testing the connection between two Arduinos using Arduino. The input is the press of a button and the output is a message displaying on an LCD screen.

physical prototypes

To test how big we wanted to control panel to be we printed a few panel to show to Shaude for layout and scale. We also wanted to test using button caps to make the buttons differentiated and easier to press. These caps did not end up working because they attached onto the tops of the buttons and were not deep enough to create a snug fit.

We learned the most from our prototypes after discussing the outcomes with Shaude. After showing her the scale of the models we learned that she needed much bigger buttons, and a big display box for her to see what the options on the box were. She also noted that she preferred a flat down view of the box rather than a slanted one. She enjoyed the potentiometers and the feedback of the lights. We talked to her about what colors she connected with different expressions and moods. We were then able to use this information to decide on our final color gradients that would be most descriptive and helpful to her. The range of these colors were red to blue for agitated to calm and green to red for health to ill. We also discussed her favorite colors and such so we could design the box to be most pleasing to her later on. Her favorite colors by the way are purple and pink, and she really likes dogs. 

During our prototyping process we were surprised by how the neopixel colors values are not all equal. The power and brightness of the red really overpowered the blue and we had difficulty mapping the values to get a noticeable gradient from blue to red. If we had more time to prototype we would have liked to explore other visual methods and hardware for showing the gradients and values of colors. We had one drawn idea of utilizing strip LEDs to use both color and value to show mood and health (see above), but it seemed to add a layer of confusion, especially with calm and agitation. We chose to continue with the Neopixel bulbs because they were simple to read and true to our original intent. Overall going further with our design we took the work from our prototypes and simply made them better suited for Shaude.

3. Conclusions and Reflections

Overall, our group thought that our product successfully fulfilled our client’s identified needs. Through interview and iteration, we were able to come up with a device that allowed the communication of both specific desires and more general moods in a concise and effective way. However, the process did involve challenges, not the least of which was working with our client. Working in an environment like CMU, we are often surrounded by people of similar dispositions who know what we can accomplish. This project allowed us to work with someone with a disability who had none of these attributes. In particular, we often found it hard when speaking or working with Shaude to get actionable feedback or find specific problems that were within our skills to address. This unexpected difficulty forced us to change how we interacted with Shaude in our second interview. Instead of open-ended questions where we expected her to give us actionable answers, we provided her with a list of possible options or user interfaces. This narrowed the scope of choices and discussion, allowing her to point out any general issues in all the possibilities. For instance, when we were giving her options on how the buttons were laid out, she expressed that even the most prominent option we had was too small. This informed us of a significant issue we must be aware of. Next time, I would like to schedule periodic meetings with Shaude at the beginning of the project to increase her involvement in our process and interview her caretaker to get a feel for their opinion on the output device.

The final crit was an informative and valuable process. Our team felt that those present gave good feedback on what we did well and what we could have done better as a final project. In particular, the one we felt was most true was regarding the layout of the mood and health potentiometers, “The ‘mood’ light felt backward, and the red/blue color scheme was hard to decipher any middle ground.” This was a comment that a couple of people gave us because the mood and health sliders went opposite directions to increase/decrease, which needed to be clarified, and the spectrum for the mood light (red to blue) was hard to see any interim values. With more time, we would fix this by flipping one of the potentiometers so that both are the same and choosing better colors to the spectrum between. Another comment we got was to improve the “Robustness of the device. Add texture or images to buttons, which is something we would want to do. We were considering making our buttons at one time, which could have allowed us to do this. Making it even easier by allowing buttons to be differentiated by feel or image instead of just clarifying what each does by text. Another useful feedback was about having a message on the receiving device for too long. This could be addressed by “adding some kind of ‘acknowledge’ button to the remote receiver, which would then silence the message and blank out the screen.” This is something that we should have considered and is a problem. Because it teaches the recipient that messages are not current, decreasing the device’s effectiveness. We did have difficulties with the radio, and getting two-way communication instead of one-way would be a challenge that we felt would have taken more time than we had. Finally, someone commented, “Bedside table space can be limited, so a slightly smaller bedside device” would be better. We agree and think that its height might also become an issue. If we could do it again, the solution would be to cut the arcade buttons in half, which would have worked because the buttons’ electrical and physical components could be separated. The buttons took up the majority of the space inside the input box.

To sum up, this device solves a real problem in Shaude’s life. Hopefully, it gets used, and no issues pop up over long periods of use, but we don’t think they will. It was an exciting experience that Team Guava was happy to be a part of.

Final project, Shaude, and Team Guava at final Critique

4. Technical Details

Block Diagrams:

Block diagram showing input and output flow for Shaude’s portion of communicator

Output Device Block Diagram

Receiver block diagram

Electrical Schematics:

Electrical Schematic of Input device for Mood communicator

Receiver Wiring diagram

Code:

/*
 * Mood Communicator, input code, final project 
 * By: Andres Montemayor, Evette Lacomb, Dunn Zhang

 Description: The code below takes in inputs from buttons, and sliding potentiometers
 then, displays a gradient from red to blue and red to green based on the pot values
 it also transmits those values over a NRF24L01 radio to another arduino

 Pin Map:
  POTPIN1     | input    | potentiometer: sets mood gradient
  POTPIN2     | input    | potentiometer: sets health gradient
  BUTTON1     | input    | reads when message 1 button pressed, sets integer for radio to send
  BUTTON2    | input        | reads when message 2 button pressed, sets integer for radio to send
  BUTTON3    | input        | reads when message 3 button pressed, sets integer for radio to send
  BUTTON4    | input        | reads when message 4 button pressed, sets integer for radio to send
  BUTTON5    | input        | reads when message 5 button pressed, sets integer for radio to send
  BUTTON6    | input        | reads when message 6 button pressed, sets integer for radio to send
  moodLed   | output        | outputs rgb value to neopixel to show potentiometer value on spectrum(red->blue)
  healthLed | output       | outputs rgb value to neopixel to show potentiometer value on spectrum(red->green)

NRF24L01 pin map can be seen below 
 * 
 * radio pin    Arduino Uno/Nano pin    Arduino Micro pin
 * VCC          3.3V                    3.3V
 * GND          GND                     GND
 * CE           7                       7
 * CSN          8                       8
 * MOSI         11                      MO
 * MISO         12                      MI
 * SCK          13                      SCK

 * modified from code by: Alton Olson, Vicky Zhou, Seema Kamath, Robert Zacharias, Andres Montemayor, Evette Lacomb
 Credit:
 Used example libraries from relevant libraries for relevant sections
 for radio code, it was provided by Robert Zacharias and then later adjusted by team to better fulfill project
 parameters.
 * 
 */

//Libraries in use in this code
#include <SPI.h>
#include <nRF24L01.h>
#include <RF24.h>
#include <Wire.h>
#include <PololuLedStrip.h>
// numbers in <> are pin numbers for the leds to display mood and health, reference for what is transmitted
PololuLedStrip<2> moodLed;
PololuLedStrip<4> healthLed;

//LED initialize count. We use a single neopixel so only one, if use a strip would be more
#define LED_COUNT 1
rgb_color moodcolors[1];
rgb_color healthcolors[1];

//define integers for colors map from sliding potentiometer
int red;
int red2;
int blue;
int green;

// Initialize the pins for the buttons, potentiometer, radio, single 
const int BUTTON1 = 3;
const int BUTTON2 = 5;
const int BUTTON3 = 6;
const int BUTTON4 = A2;
const int BUTTON5 = A3;
const int BUTTON6 = A4;

const int POTPIN1 = A0;
const int POTPIN2 = A1;

const int RADIO_CE = 7;
const int RADIO_CSN = 8;

int button = 0;
int POT1 = -1;
int POT2 = -1;
int send[] = {0,0,0};

//initiallize and name radio, address needs to be 6 digit and same on recieve and transmit
RF24 radio(RADIO_CE, RADIO_CSN);
const byte address[6] = "00001";

void setup() {
  // radio startup
  radio.begin();
  radio.openWritingPipe(address);
  radio.setPALevel(RF24_PA_HIGH);
  radio.stopListening();
  Wire.begin();
  // Set the pins for the potentiometers & buttons
  pinMode(BUTTON1, INPUT);
  pinMode(BUTTON2, INPUT);
  pinMode(BUTTON3, INPUT);
  pinMode(BUTTON4, INPUT);
  pinMode(BUTTON5, INPUT);
  pinMode(BUTTON6, INPUT);

  pinMode(POTPIN1, INPUT);
  pinMode(POTPIN2,INPUT);
}

void loop() {
  // read analog values from sliding pots
  POT1 = map(analogRead(POTPIN1),0,1023,0,255);
  POT2 = map(analogRead(POTPIN2),0,1023,0,255);

  //map sliding pots to colors for leds
  red = map(POT1, 0, 255, 7, 255);
  blue = map(POT1, 0, 255, 10, 0);

  red2 = map(POT2, 0, 255, 0, 255);
  green = map(POT2, 0, 255, 255, 0);

  //display color
  moodcolors[0] = rgb_color(0, red, blue);
  healthcolors[0] = rgb_color(green, red2, 0);

  moodLed.write(moodcolors, 1);
  healthLed.write(healthcolors, 1);
  // check which button is pressed to see what integer should be transmitted over radio
  if (digitalRead(BUTTON1) == LOW) {
    button = 1;
  }
  if (digitalRead(BUTTON2) == LOW) {
    button = 2;
  }
  if (digitalRead(BUTTON3) == LOW) {
    button = 3;
  }
  if (digitalRead(BUTTON4) == LOW) {
    button = 4;
  }
  if (digitalRead(BUTTON5) == LOW) {
    button = 5;
  }
  if (digitalRead(BUTTON6) == LOW) {
    button = 6;
  }
  //redefine arrey before sending based on inputs 
  send[0]=POT1;
  send[1]=POT2;
  send[2]=button;

  // transmit values
  radio.write(send, sizeof(send));
  //delay so time to update led correctly
  delay(50);
}

//PROJECT 3 GUAVAS: OUTPUT CODE
//Dunn Zhang, Andres Montemayor, Evette LaComb
//SHAUDE's mood box RECEIVE code - takes the radio signal from another box and displays the inputs through screen and lights  


//Pinmap:
/*  pin       mode         description      
    4         output       mood Led 
    3         output       health Led           
    SDA       output       LCD screen 
    SCL       output       LCD screen 
    7         CE           radio
    8         CSN          radio
    11        MOSI         radio
    12        MISO         radio
    13        SCK          radio

NRF24L01 pinmap can be seen below 
  radio pin    Arduino Uno/Nano pin    Arduino Micro pin
  VCC          3.3V                    3.3V
  GND          GND                     GND
  CE           7                       7
  CSN          8                       8
  MOSI         11                      MO
  MISO         12                      MI
  SCK          13                      SCK
 
 * modified from code by: Alton Olson, Vicky Zhou, Seema Kamath
 * Robert Zacharias 5-23

*/

//USE THIS CODE FOR RECIEVE
#include <SPI.h>
#include <nRF24L01.h>
#include <RF24.h>
#include <Wire.h>
#include <LiquidCrystal_I2C.h>
/// lights vvv 

#include <PololuLedStrip.h>
PololuLedStrip<4> moodLed;
PololuLedStrip<3> healthLed;

#define LED_COUNT 1
rgb_color moodcolors[1];
rgb_color healthcolors[1];

int red;
int red2;
int blue;
int green;
//lights ^^^

//Initialize the LCD library with the I2C address, number of columns and rows. call it lcd
LiquidCrystal_I2C lcd(0x27, 20, 4);

const int RADIO_CE_PIN = 7;
const int RADIO_CSN_PIN = 8;

int POT1;
int POT2;
int button;
int message = -1;

RF24 radio(RADIO_CE_PIN, RADIO_CSN_PIN);
const byte address[6] = "00001";

void setup() {
  Serial.begin(9600);

  radio.begin();
  radio.openReadingPipe(0, address);
  radio.setPALevel(RF24_PA_HIGH);
  radio.startListening();
  lcd.init();
  lcd.backlight();
}

void loop() {
  // get radio input from transmitter
  if (radio.available()) {
    int readVal[3];  
    radio.read(&readVal, sizeof(readVal));

    POT1 = readVal[0];
    POT2 = readVal[1];
    button = readVal[2];
    Serial.println(POT1);
    
    //depending on the button signal show message 
    if (readVal[2] == 5 && message != 5) {
      message = 5;
      lcd.clear();
      lcd.setCursor(0, 0);
      lcd.print("Can you help me");
      lcd.setCursor(0, 1);
      lcd.print("get up?");
    }
    if (readVal[2] == 2 && message != 2) {
      message = 2;
      lcd.clear();
      lcd.setCursor(0, 0);
      lcd.print("Can I have some");
      lcd.setCursor(0, 1);
      lcd.print("time to myself?");
    }
    if (readVal[2] == 3 && message != 3) {
      message = 3;
      lcd.clear();
      lcd.setCursor(0, 0);
      lcd.print("I want to get");
      lcd.setCursor(0, 1);
      lcd.print("outside today.");
    }
    if (readVal[2] == 6 && message != 6) {
      message = 6;
      lcd.clear();
      lcd.setCursor(0, 0);
      lcd.print("Could you get");
      lcd.setCursor(0, 1);
      lcd.print("me some water?");
    }
    if (readVal[2] == 4 && message != 4) {
      message = 4;
      lcd.clear();
      lcd.setCursor(0, 0);
      lcd.print("I can't wait");
      lcd.setCursor(0, 1);
      lcd.print("for breakfast!");
    }
    if (readVal[2] == 1 && message != 1) {
      message = 1;
      lcd.clear();
      lcd.setCursor(0, 0);
      lcd.print("Please be patient");
      lcd.setCursor(0, 1);
      lcd.print("with me today.");
    }
    //colored LEDs:
    int red = map(POT1, 0, 255, 7, 255);
    int blue = map(POT1, 0, 255, 10, 0);

    int red2 = map(POT2, 0, 255, 0, 255);
    int green = map(POT2, 0, 255, 255, 0);

    moodcolors[0] = rgb_color(0, red, blue);
    healthcolors[0] = rgb_color(green, red2, 0);

    moodLed.write(moodcolors, 1);
    healthLed.write(healthcolors, 1);

    delay(150);

  }
}

Introduction

In this class, we worked collaboratively with our client  Bill, who has achromatopsia and is sensitive to light. We aimed to create an assistive device to help him with his daily activities. After conducting several interviews with Bill, we learned that he loves cooking but has difficulty determining if the meat is cooked through due to his inability to see color changes. As a result, we decided to create an assistive cooking device to help him cook safely. We aim to provide Bill with a device that will enable him to cook confidently and safely.

What we Built

Our project is an assistive device designed to help our client Bill with his cooking activities. As mentioned, Bill has achromatopsia and is sensitive to light, which makes it difficult for him to determine if meat is cooked through due to his inability to see color changes. To address this challenge, our device has two systems. The first system is a temperature sensing system that allows Bill to input the desired cooking temperature using a keypad. The device will then display both the set temperature and the current temperature detected by the probe on two seperate screens. The second system is a timer that countdown the cooking time for Bill’s dish and alerts him when it’s completed. Together, these two systems help Bill cook safely and confidently without worrying about undercooked or overcooked meat.

“Story”

Bill wakes up early on a Sunday morning feeling energized and ready to cook up a storm in the kitchen. He decides to make a special breakfast of pancakes and bacon, and he can’t wait to use his new assistive cooking device. He pulls out his ingredients, preheats the stove, and starts cooking the bacon. With his device by his side, he inputs the desired temperature and sets the timer for the bacon. As the bacon cooks, Bill takes advantage of the time to prepare the pancake batter. He uses the probe to check the internal temperature, and the device displays the readings on the screen. The bacon is perfectly cooked, and Bill is thrilled with the results. Next, it’s time to cook the pancakes. Bill sets the temperature for the griddle and starts cooking. He sets the timer for each batch, knowing that he won’t have to worry about overcooking or undercooking them. The device beeps to signal when it’s time to flip the pancakes, and Bill easily flips them. Finally, it’s time to plate the food. Bill arranges the pancakes and bacon on the plate and pours a generous maple syrup. He sits down to enjoy his meal, feeling proud of himself for cooking such a delicious breakfast. Thanks to his new assistive cooking device, Bill can cook safely and confidently without worrying about undercooked or overcooked meat. He feels empowered and independent in the kitchen and can’t wait to try out new recipes with his device by his side.

How we got There

• Prototype

We started by prototyping the inputs and outputs of the meat-cooking assistive device. We were thinking about and exploring how different buttons, screens, LEDs, and temperature sensor work with one another and are housed and interacted with by Bill.

Ideating on the form of the object and how Bill may interact with it.

A version with screen to compliment the voice from the speaker, and a more tangible button system for inputting the temperature.

A version with screen to compliment the voice from the speaker, and a more tangible button system for inputting the temperature.

Figuring out handheld versions. Thinking about features it might have and how they may be interacted with.

A reference that Bill shared of his kitchen space, and the shelf the device will live on.

After the prototype critique, we asked more questions to Bill because we wanted to design a device that truly fits his preferences. Our questions for Bill after these initial prototypes revolved around:

1. Where would he like it? Storage? 
   • On the countertop
   • As a complete object (fully handheld) 
   • Something that fits in a drawer
   • Something that goes on a backsplash
   • Should it be wipable/ cleanable? 
2. What kind of inputs would he prefer? 
   • More touch screen style 
   • Different kinds of potentiometers 
   • Keypad  
   • The dirtiness of fingers may be a consideration
   • Does he want to set each degree of temperature or scroll through a menu? 
   • Does he want the additional feature of a cooking timer?
3. What kind of outputs to tell him that it is ready
   • Audio
   • Visual? Zach said he had a high screen
   • Light? How many bulbs are on kind of thing

Incorporating the feedback we received during the critiques, we made several changes to our design. We added a voice output feature for the current temperature and included a separate timer feature. We also made the box housing generally pretty dark for higher contrast. During the refining process, we asked Bill many questions about his cooking preferences and needs. We learned that while he does cook in a dark kitchen and likes sound output, he prefers to read numbers from a screen. He also mentioned that white text on a black background and a comfortable font size would work well for him. We found a 7-segment number display that would meet his needs and decided to make the housing and aesthetic of the box generally pretty dark to provide higher contrast.

Initially, we had considered including a progress bar or feedback feature to let Bill know how close his meat was to reaching the target temperature. However, Bill said it wouldn’t be something he would particularly need, so we decided to focus on providing him with the current and target temperatures on the two displays instead. Regarding input, we discussed the practicality of a keypad layout and decided to include a little button connecting to the speaker to voice the current temperature when pressed. We also included a separate timer feature with an hour and minute dials and a similar screen and button for voicing the feature.

Considering Bill wants to put this device on his shelf above the stove, we designed to used a power cable that allows Bill to plug it in and use the device easily. We also added an on and off switch just in case he doesn’t like to plug in and out every time but still can turn the device on and off.

Another feedback we took from the critique is that adding a hinge on the device to organize the probe is necessary. So we 3D modeled and printed a hinge that could exactly hold the probe and mounted it to the top of the device.

•   Process

Our Built process followed our initially outlaid schedule very closely, we did not experience a lot of delays or deviations from the initial design plan aside from the electronics taking 1 day extra to finish and debug.

3D Printed Hinge for the thermistor

Laser-cut the box with black acrylic. and soldered electronics placed side by side.

We have a double panel for the front to cover the ‘ABCD’ row of the keypad, which we will not use.

Soldered all electronics together to avoid unstable connections.

Glue the components on the box.

Overall the build process was what we expected. We stuck closely to our initial design and the electronics cooperated with the vision of our final product. There were a few strange library conflict with our text to speech library but that was debugged rather quickly. All of our box parts fit well and the foresight to use clearly labeled stranded wire made assembly very easy.

Conclusions and Lessons Learned

Starting with the final critique. We were overall pleased with the final presentation and critique of the project. There were some last minute software/ electric related bugs but other than that, our device performed as expected. 

The feedback that we received was roughly split into 2 categories: design and execution. The design related feedback was very insightful as at that point in time, we had been staring at this particular design for a couple weeks, having fresh perspectives was nice. Much of the feedback centered around the user friendliness of the design stating that :”Some of the usability was unclear, it could benefit from some labels to clarify what each numbers are corresponding to and a volume potentiometer.

“ and “Would be helpful if the buttons read out their purposes as well as the corresponding figures, as well as having the displays physically labeled”. The core of the feedback that complimented the design on its sleekness and minimal look also criticized the confusing nature of the placement of the buttons. It would probably be a good idea to label the buttons as it does not really detract from the overall look or function of the device. It was an issue of running out of time that we chose not to mess with the overall design too much. In addition, the displays being physically labeled would probably preclude the need for labeling the buttons. We also had a design comment about the placement of the thermometer over the speaker cutout. “Placement of thermometer rest in proximity to speaker holes presents moisture challenges and could be made more ergonomic.” This was very interesting as we had not considered the potential for the thermistor to become wet and leak into the speaker port. Given that both the port and the mount are easily moved. A future interaction would definitely address this. There was also a comment about adding some rubber feet to the device to prevent it from slipping. This would probably be a good idea on a commercial version but we designed the device to fit on a specific shelf where slipping when pushing a button would not cause any problems.

Finally on execution notes. A lot of notes were on the buggy timer and screen flashing which we were aware of potential fixes already. However, there were some software notes that we had not considered. One interesting note is that “Needs warning beeps at T–5°, solid alarm tone at T°, and a constant (but dismissable) BURN ALERT at something like T+10°.” While designing the functionality of the product, we only considered a warning when the temperature was met with the assumption that Bill would be able to head and see it. If it’s the case that he is in another room, we did not consider that possibility. A future iteration would include an approaching temperature chime as well as a burn warning that is sustained in order to address these cases. I think this was just a situation where we did not consider the possibility of him missing the complete chime. 

Overall the feedback was very insightful and having many experienced perspectives really helped us understand the advantages and disadvantages of the design choices that we made.

Working with Bill was an overall good experience. We think given that he was the coordinator and CLASS staff, our experience might be different from other groups working with CLASS members. With Bill specifically, something like not being able to see the doneness of meat was not a complication that we had ever considered with color blindness. Having him as a client made us aware of a lot of issues that partial vision deficiency could cause for a person even if it’s not as serious as full blindness. We are grateful for the opportunity to work with him. We also learned valuable lessons working with and interacting with the other clients in the project as well as the CLASS trip. Prior to this project, we did not have a lot of experience interacting with the disabled community and this project gave a lot of insight into their lives and struggles. 

Finally, some notes about this project. This project ran smoothly overall and the team worked well together. The division of tasks was more or less equal and people were able to bring their specialized skills to make the project what it was. Given the opportunity to do it again, our process would be very similar. We built it flex time in the event that something went wrong which helped us out a lot in the end. This final project was an excellent close to a very interesting and engaging class.

Technical Details

• Schematic and Block Diagram

• Codes

/*
Project Name: Bill's Cooking Buddy Helper
Project Members: Sapna Tayal, Angie (Chuyi) Wang, Harry Ren
Code Author: Harry Ren
- Assistive Meat Thermometer and Cooking timer
- Keypad input for target temperature
- All values displayed on 7 segments and can be read via speaker

Controller: Arduino Mega

Pin Mapping:
A0: Keypad pin 1
A1: Keypad pin 2
A2: Keypad pin 3
A3: Keypad pin 4
10: Keypad pin 5
11: Keypad pin 6
12: Keypad pin 7
13: Keypad pin 8

2: Current Temp Button
3: Target Temp Button
4: Timer Button

A8: Hour Potentiometer Pin
A9: Minute Potentiometer Pin

6: Main Audio Amp Pin
7: Inverted Audio Amp Pin

A10: Temp Probe Pin

Note: all 7 Segment pins connect to I2C Bus (master-slave) System
SDA: 7 Segment main SDA
SCL: 7 Segment main SCL

Portions of code are adapted from the following sources
- https://forum.arduino.cc/t/another-bbq-thermometer-wifi-enabled-with-push-notifications/225467
- https://forum.arduino.cc/t/countdown-timer-and-adafruit-4-digit-7-segment-display-w-i2c-backpack/364882/2
- https://www.instructables.com/Arduino-Text-to-Speech-Converter-Using-LM386-Talki/
relevant functions are credited
code by Harry Ren at Carnegie Mellon University (hjren@andrew.cmu.edu)
released by author to public domain, April 2023
*/


#include <Keypad.h>
#include <OneWire.h>
#include <DallasTemperature.h>
#include "Talkie.h"
#include "Vocab_US_Large.h"
#include <Wire.h>  // Enable this line if using Arduino Uno, Mega, etc.
#include <Adafruit_GFX.h>
#include "Adafruit_LEDBackpack.h"
#include <math.h>

//Lines 12-19 adapted from https://forum.arduino.cc/t/another-bbq-thermometer-wifi-enabled-with-push-notifications/225467/2
float air;          //  temperature
float meat;         //   Unused from sample code
float vin = 4.999;  //  DC Voltage as measured with DMM between +5V and GND
float r2 = 100000;  // Resistance in ohms of your fixed resistor

float A = 0.00004556837992;     // "A" Coeffecient in Steinhart-Hart Equation
float B = 0.0002634285613;      // "B"
float C = 0.00000005286310982;  // "C"

const int sw1 = 2;  //current temp button
const int sw2 = 3;  //tgt temp button
const int sw3 = 4;  //timer button
const int sw4 = 5; //io switch

Talkie voice;  //init speaker

// Define the pins for the 7-segment display: depreciated
const int digitPins[] = { 2, 3, 4, 5 };
const int segmentPins[] = { 6, 7, 8, 9, 10, 11, 12 };

// Define the keypad
const byte ROWS = 4;
const byte COLS = 4;
char keys[ROWS][COLS] = {
  { '1', '2', '3', 'A' },
  { '4', '5', '6', 'B' },
  { '7', '8', '9', 'C' },
  { '*', '0', '#', 'D' }
};
byte rowPins[ROWS] = { A0, A1, A2, A3 };  //keypad pins
byte colPins[COLS] = { 10, 11, 13, 12 };  //keypad pins
Keypad keypad = Keypad(makeKeymap(keys), rowPins, colPins, ROWS, COLS);


Adafruit_7segment mat = Adafruit_7segment();     //curr 7seg
Adafruit_7segment mat1 = Adafruit_7segment();    //tgt 7seg
Adafruit_7segment matrix = Adafruit_7segment();  //timer 7seg

int number = 0;  //curr tgt
int temp = 0;    //curr temp

int startMinute = 4;   //  Modify these defines to
int startSecond = 59;  // change the timer interval
int minutes = startMinute;
int seconds = startSecond;
bool notStarted = 0;  //timer started bool flag

unsigned long previousSecondMillis = 0UL;
long oneSecond = 1000UL;  // milliseconds per second

const int POT_PIN_H = A8;
const int POT_PIN_M = A9;
int digit_H = 0;
int digit_M = 0;


void setup() {

  Serial.begin(9600);

  //init button pins
  pinMode(sw1, INPUT);
  pinMode(sw2, INPUT);
  pinMode(sw3, INPUT);

  //init 7 segments
  mat.begin(0x70);
  mat1.begin(0x71);
  matrix.begin(0x72);
  mat.print(number);
  mat1.print(temp);

  matrix.writeDigitRaw(2,0x2);
  
  // Set the digit pins as outputs: depreciated
  for (int i = 0; i < 4; i++) {
    pinMode(digitPins[i], OUTPUT);
  }

  // Set the segment pins as outputs: depreciated
  for (int i = 0; i < 7; i++) {
    pinMode(segmentPins[i], OUTPUT);
  }

//init speaker code: depreciated
#if defined(TEENSYDUINO)
  pinMode(5, OUTPUT);
  digitalWrite(5, HIGH);  //Enable Amplified PROP shield
#endif
}


void loop() {
  //read switches
  int switch1 = digitalRead(sw1);
  int switch2 = digitalRead(sw2);
  int switch3 = digitalRead(sw3);
  int switch4 = digitalRead(sw4);

  //BUTTON CODE
  if (switch1 == 1) {
    readnum(number);
  }
  if (switch2 == 1) {
    readnum((int)air);
  }
  if (switch3 == 1) {
    notStarted = 1;
    //previousSecondMillis = 0UL;
  }
  

  //TEMP PROBE CODE
  //read and update current temp
  readSensors();
  mat1.print((int)air);

  //KEYPAD CODE
  // Wait for a key press
  char key = keypad.getKey();
  char printednum = '0';
  // If a key was pressed, display and read the corresponding number
  if (key != NO_KEY) {
    int newnumber = getNumberForKey(key);
    saynum(newnumber);
    if (newnumber == 14) {
      number = 0;
      mat.clear();
    }
    //update the 7 seg number
    else if (number < 1000 && newnumber != 14 && newnumber != 15) {
      number = number * 10 + newnumber;
    }
    mat.print(number);
    //Serial.println(number);
  }
  if (switch4==1){
    mat.clear();
    mat1.clear();
    
  }
  mat.writeDisplay();
  mat1.writeDisplay();

  //TIMER CODE
  //Setting timer
  if (notStarted == 0) {
    int potVal_H = analogRead(POT_PIN_H);
    int potVal_M = analogRead(POT_PIN_M);

    digit_H = map(potVal_H, 1023, 0, 0, 30);
    digit_M = map(potVal_M, 1023, 0, 0, 59);

    startMinute = digit_H;  
    startSecond = digit_M;  
    minutes = startMinute;
    seconds = startSecond;
    matrix.writeDigitNum(0, (minutes / 10));
    matrix.writeDigitNum(1, (minutes % 10));
    matrix.writeDigitNum(3, (seconds / 10));
    matrix.writeDigitNum(4, (seconds % 10));
    if (switch4==1){
    matrix.clear();

  }
    matrix.writeDisplay();
  }

  else {
    // Countdown timer
    //adapted from https://forum.arduino.cc/t/countdown-timer-and-adafruit-4-digit-7-segment-display-w-i2c-backpack/364882/2
    // --------- Run this every second ------------------
    if (millis() - previousSecondMillis >= oneSecond) {
      //update 7 seg per second
      matrix.writeDigitNum(0, (minutes / 10));
      matrix.writeDigitNum(1, (minutes % 10));
      matrix.writeDigitNum(3, (seconds / 10));
      matrix.writeDigitNum(4, (seconds % 10));
      if (switch4==1){
    matrix.clear(); 
  }
      matrix.writeDisplay();
      if (seconds-- == 0) {
        if (minutes == 0) {
          //If timer reaches 0
          notStarted = 0;
          voice.say(sp2_COMPLETE);
          //minutes = startMinute;
          //seconds = startSecond;
          delay(1000);
        } else {
          // rollever from n:00 to (n-1):59
          minutes -= 1;
          seconds = 59;
        }
      }

      previousSecondMillis += oneSecond;
    }
  }
}

//reads a 4 dig number on speaker
void readnum(int number) {
  if (number > 999) {
    saynum((number / 1000) % 10);
  }
  if (number > 99) {
    saynum((number / 100) % 10);
  }
  if (number > 9) {
    saynum((number / 10) % 10);
  }
  if (number > -1) {
    saynum(number % 10);
  }
}

//reads a single number
void saynum(int n) {
  switch (n) {
    case 1:
      voice.say(sp2_ONE);
      return;
    case 2:
      voice.say(sp2_TWO);
      return;
    case 3:
      voice.say(sp2_THREE);
      return;
    case 4:
      voice.say(sp2_FOUR);
      return;
    case 5:
      voice.say(sp2_FIVE);
      return;
    case 6:
      voice.say(sp2_SIX);
      return;
    case 7:
      voice.say(sp2_SEVEN);
      return;
    case 8:
      voice.say(sp2_EIGHT);
      return;
    case 9:
      voice.say(sp2_NINE);
      return;
    case 0:
      voice.say(sp2_ZERO);
      return;
    case 14:
      voice.say(sp3_CLEAR);
      return;
    case 15:
      //readnum();
      return;

    default: return;
  }
}


// Convert the key to a number (casting caused problems)
int getNumberForKey(char key) {
  switch (key) {
    case '0': return 0;
    case '1': return 1;
    case '2': return 2;
    case '3': return 3;
    case '4': return 4;
    case '5': return 5;
    case '6': return 6;
    case '7': return 7;
    case '8': return 8;
    case '9': return 9;
    case 'A': return 10;
    case 'B': return 11;
    case 'C': return 12;
    case 'D': return 13;
    case '*': return 14;
    case '#': return 15;
    default: return 0;
  }
}


//Depreciated Function
void displayNumber(int number) {
  int digits[4];
  digits[0] = number % 10;
  digits[1] = (number / 10) % 10;
  digits[2] = (number / 100) % 10;
  digits[3] = (number / 1000) % 10;
}

// Reads and updates current temp
// adapted from https://forum.arduino.cc/t/another-bbq-thermometer-wifi-enabled-with-push-notifications/225467
void readSensors() {
  float a0 = analogRead(A10);                            // This reads the "voltage" value on A0. Value is actually divided into 1024 steps from 0-1023.
  float v0 = a0 * .0049;                                 // Converts A0 value to an actual voltage (5.0V / 1024 steps = .0049V/step.
  float r0 = (((r2 * vin) / v0) - r2);                   // Calculates resistance value of thermistor based on fixed resistor value and measured voltage
  float logr0 = log(r0);                                 // Natural log of thermistor resistance used in Steinhart-Hart Equation
  float logcubed0 = logr0 * logr0 * logr0;               // The cube of the above value
  float k0 = 1.0 / (A + (B * logr0) + (C * logcubed0));  // Steinhart-Hart Equation to calculate temperature in Kelvin
  float f0 = (1.8 * (k0 - 273)) + 32;                    // Convert temperature to Fahrenheit

  if (isnan(f0))  // If value is not a number, assign an arbitrary value
  {
    air = int(1);
  } else {
    air = f0;  // Otherwise use the calculated value
  }
}

For the final project of the Intro to Physical Computing class, our Team Alycai, is working with Keith, a brain injured patient from Pittsburgh, and developing an assistive device that will be useful and relevant to his experience. To better understand his wants and needs, and to get to know Keith better, we conducted an informational interview with him. The interview was conducted at Community Living and Support Services (CLASS), a non-profit organization where Keith attends classes for part of the day, and members of our group: Cable, Bhairabi, and Ethan chose to interview Keith at a time after his classes were over. The purpose of our interview was to gain a better understanding of various aspects of Keith’s daily life, the things he enjoys doing, and the limitations he has in these areas.

Meeting Keith:

Introduction/icebreaker

• Name, interest, major, fun fact
• Ask client to share the same
• Ask the client why they volunteered for this project. 
  • What drew them to participate?

• Explanation and clarification about project goals
  • We are:
    • Trying to build a little gadget that makes something in *your* life easier or more fun.
    • Going to see him again on April 5th weeks to get his thoughts on what we’re making, and we’ll have a product for him by May. 
    • Taking about seven weeks to go from this meeting to a reasonably high-fidelity final product
    • Documenting our process
  • We are not:
    • Professional engineers who are experienced in making polished devices
    • Planning to build something that will be sold in stores
    • Building something for a lot of people, so the final is specifically geared for you
    • Inventing new parts; we have limits to both components and skill set
  • Today, we will just be asking you about your life and some of the tasks you do to get a better idea of what we will be making in the future. There’s no right or wrong answers, and there’s no pressure to answer anything you are uncomfortable with. Let us know if you have any questions throughout the process or need anything that would make this conversation easier.
  • By the end of this interview, we want to be on the same page with you about the general idea of what we make. We will be sketching ideas along the way, and don’t hold back if you have any ideas or are excited about/dislike any ideas that we have.
  • Are you ready? Have any questions?
  • If you want to get in touch with us for any reason talk to Bill, he has our contact info
• Questions:
  • What are some things that you enjoy doing?
    • Are there any challenges that prevent you from doing that?
    • Have they gotten harder over time?
    • What do you like about them?
  • What does your daily routine look like?
    • Are there any tasks that are difficult to do in your everyday routine?
    • What makes them difficult?
  • What is something you wish you could do differently in your everyday life?
  • What are some things you wish you could do more?
    • Ask if they’d be willing to demonstrate a daily task or a series of tasks to you during the meeting. They can actually do the task if it’s something simple like getting a glass of water, or they may pantomime the task. The closer to reality this action is, the more likely you are to find a particular aspect of the task that’s harder than it needs to be.
    • Use the simple creative act of drawing to bring out ideas: ask the client to draw a map or cartoon of their daily life and show it to you on camera, or have them narrate the action and do the drawing yourself in a way that’s visible to the group. It could be a map (literal map, showing travels), a sequence of cartoon panels (first I do this, then I do this), or take a more abstract form.
    • In general, using a shared drawing-surface tool (such as a whiteboard in Zoom, or a piece of paper in real life) so that everybody can simultaneously contribute to and see others’ ideas is a great idea.
  • For the final prototype, we also want it to be something that you like to have around. What are your favorite colors?
    • What are some things that you like? Cars? Brands? Movies? Something that you own?
• Conclude
  • Thank your client for their time, make sure they have your contact information, and take any final documentation images/drawings/notes you may need to wrap up before you sign off.
  • One last thing, want to come up with a team name? 
    • Ideas: Keith and the rest, Keith and Co
  • Reiterate the overall project schedule if you feel it would be necessary to clarify before leaving.

Summary and Major Takeway:

• A lot of Keith’s problems already have solutions that rely upon other people or existing devices
• Keith believes that he doesn’t exactly need solutions for these problems as he has already found solutions
• The biggest problems that Keith struggle with are unable to be addressed through physical computing; ie major headaches and a desire to buy weed
• We may need to discuss Keith’s struggles with someone who knows and understands Keith, rather than Keith himself, to understand what it’s like to need to help Keith with his everyday life (and how to create a device that can alleviate some of that work)

After Meeting Thoughts:

Our meeting followed our agenda but wasn’t very fruitful. Keith’s main problem is his headaches, which are an extremely difficult problem to solve. Additionally, a lot of our questions were potentially too probing or too vague, since his response to a vast majority of them were “I don’t know” or “I don’t care”. His main proposed solutions involved growing marijuana to help with his headaches, which is obviously a solution we cannot help with. Our probing questions into his daily activities were not very revealing as to what types of devices might help him, so we wound up relying primarily on Bill’s suggestions of a device to help Keith put his watch on and to help Keith remember his schedule. In hindsight, it would have been nice to have Bill in the room while we were meeting with Keith, as Keith’s traumatic brain injury seems to sometimes make him an unreliable narrator of his own experience. 

For our Intro to Physical Computing class’s final project, our group, the Guavas, is working with Shaude, an individual from Pittsburgh with cerebral palsy, to develop an assistive device that’s useful and relevant to her experience. To better understand her wants and needs, and get to know Shaude better, we conducted an informational interview with her. It was conducted at Community Living And Support Services (CLASS), a nonprofit organization Shaude spends her time taking classes at during the day, with the members of our group: Sapna, Evie, Dunn and Andres. Our goal for the meeting was to gain a better understanding of aspects of Shaude’s daily routine, tasks or activities she enjoys doing, and spaces for an assistive device during these.

Meeting Agenda

10:00 – Introductions/Ice Breakers. Continue the meeting that started in class the previous week in order to allow a more comfortable interview environment as well as begin to understand her desires and needs

10:20 – Discuss the initial ideas we had that may help her and the initial hopes on what she hoped to gain by participating in this program as well as timeline for project.

10:40 – Discuss how she goes about her day, investigate a walkthrough of normal schedule from wake up to go to sleep in order to find areas in daily life that may need assistance

10:55 – Discuss her likes and frustrations in life. What are some things she really enjoys doing we could make more common, are there any interactions that you have with people we could make easier

11:20 – Spend rest of time discussing takeaways from interview with interviewee as well as see if we can document real life actions of Shaude, i.e. take photos of chair, her range of motion, her doing normal actions.

Gantt Chart detailing the timeline of final project as well as deadline

Summary and Key Takeaways

Rough notes from the meeting.

With Shaude we talked about her daily tasks, what she has the most difficulty with, and how she copes. Shaude has Cerebral Palsy, uses a motorized wheelchair and has the most difficulty with her left foot. In her arms she has the most range of motion, but has struggle gripping with her right hand. She mostly uses her left hand. Besides her mobility issues, Shaude opened up about recently losing her best friend, her great aunt and her troubles coping with grief and mood. She notes that she wishes she could communicate her mood better with her mother, roommate and others helping her. She also wishes to understand them better in turn. She notes that she has the most trouble with her mood in the morning. She says that she is foggy in the mornings, it is harder in the morning for her to communicate her needs to others.

Ideation for mood display, to convey current feelings/desires without words.

In her daily tasks she noted the reach holding her back from independence. Since she is in a wheelchair she needs someone to help her retrieve anything she drops. She notes that this is most commonly a problem when she is reading a book. She first needs to request a book for someone, and then notes she will often drop a book and need help retrieving it. With her hobby of art and painting we asked her to demonstrate how she goes through this process. She didn’t note any difficulty with her process, but we noted that she needs to bend over in a seemingly uncomfortable way.

Client demonstrating current method of use for paint brush, allow designers to understand range of motion and physical limitations

Shaude holding a paintbrush to show us grip strength and positioning

Lastly we asked about how she gets from place to place. She uses a service called access, but notes she often has difficulty with it because she gets cold while waiting and has difficulty getting warmer. Besides all of these issues we got a good list of her favorite TV shows, games and books. We hope to pursue a feasible solution to one of these problems while making it personal to her and her tastes.

After Meeting Thoughts

Overall, the interview with Shaude was great, and stayed mostly according to our meeting agenda. The conversation was easy to start and she was very open to communication and chatting with us. There are instances where she brought up ideas that are potentially unachievable, such as those involving dogs, money, and traveling. The team thought it would have been better if the meeting was more physically involved, rather than verbally. For example, we asked her, with respect, to demonstrate her arms’ range of motion, and we should have done more of these where we can understand how she lives her life. She also talked a lot about her passion about art and reading, and we thought it would be very insightful to see the art she makes and the book she reads, so the team can understand more about her and her personality.

Meeting agenda: 

• Introduction/ice breaker
  • Name, major, year, interests
  • Talk about daily routine
  • Ask about her daily routine
  • Ask why she wants to participate in the project
  • Explain our personal assistive device projects
• Explanation and clarification about project goals
  • We are:
    • Trying to build prototype useful devices
    • Taking about seven weeks to go from this meeting to a reasonably high-fidelity final product
    • Documenting our process
  • We are not:
    • Professional technologists who are experienced in making polished products
    • Planning to build something that will be sold commercially
    • Constrained by any practicality outside of usefulness to the person we’re designing for
• Quickly go over timeline again

Project timeline.

• Understanding needs and thinking of possible technological interventions
  • Interests
  • Daily routine
  • Ask directly about difficulties
  • Follow-up asking for demonstration
  • Ask if there is anything she used to enjoy that has now become difficult
  • Experiment with quick prototypes
• Conclude
  • Thank client for their time, make sure they have your contact information.

Meeting summary and major takeaways:

First, we did some brief introductions and went over the goals of the project again. We explained what we are trying to accomplish, and what we are not trying to accomplish with the project. During this part of the discussion, we also talked about the devices we each made for our personal assistive devices to give her an idea of the types of projects we could make for this course. Then, we went over the timeline of the whole project to make sure we are all on the same page.

Going over the timeline with Darnisha.

We discovered a lot about Darnisha’s interests. She is a very creative person and likes to express that in many different ways. She enjoys painting, watercolor, collage, cut and paste, and other arts and crafts. She also loves fashion (She had just won “Best Dressed” at a camp over the weekend!). She particularly enjoys accessorizing with jewelry and beads. Her favorite colors are pink and purple. She really enjoys these activities, but her limited mobility can sometimes make it difficult for her to fully enjoy them.

We also discussed a bit of her daily routine. Some things she mentioned that she has difficulty with is putting on socks and shoes, because she is unable to bend over or pull them up. She also mentioned that she struggles to remember where she puts her phone down sometimes.

Once we had discussed some of her interests and daily routines, we talked a bit more about specific ideas. We talked about some difficulties with arts and crafts. She mentioned that she enjoys using various cut out shapes in her creations, but struggles with the action of cutting. She said that she typically uses a few set shapes in her crafts— circles, squares, rectangles, and triangles. We talked a bit about ways that we could help her cut out shapes. 

Darnisha also enjoys wearing a lot of beaded jewelry and expressed that she would like to be able to make them herself. Currently, she needs someone else to create beaded jewelry for her, because she’s unable to both hold a string still and put beads on it. We had brought some craft supplies to make small prototypes, so we asked her to string some beads on a pipe cleaner so we could see how she is currently able to handle beads. She was able to pick up the beads and string them on the pipe cleaner as long as the pipe cleaner was rigid and held in place.

Darnisha demonstrating how she uses beads.

Darnisha stringing beads on a pipe cleaner.

Finally, we discussed some ways of input and output that she is comfortable with. She prefers light as an indicator. She mentioned that she can get startled easily, so it would probably be best to not use sound. She is used to switches and joysticks, and is comfortable with typing. 

We left off with making sure that there weren’t any unanswered questions, and that we were all on the same page. As we left, Darnisha showed us one of her paintings of a beautiful ocean scene at night that was on display in the hallway.

Our thoughts after holding the meeting and discussing as a team:

Our meeting with Darnisha was great. We started off by introducing ourselves which perhaps built the conversation itself. The introductions led us into discussing our interests which made it easier for us to target areas of interest for Darnisha. Talking about and showing her our own previous projects also helped us explain the scope and context of the course, as well as this project, more clearly.

We followed our meeting agenda as our client was actually answering very to the point. Next time, we might want to ask additional follow up questions more related to her abilities as during the interview we might have spoken a lot about her interests and activities she might need help with, but we missed out on a couple of opportunities for her to engage physically with something. For example, Darnisha said that she is able to paint pretty independently, but it might’ve been beneficial to see her engage with a paint brush to see if we could make the process even easier for her.

We covered a good range with the questions we asked to Darnisha as they led from her daily routine to her interests, involving things she needs assistance with. As mentioned earlier, we could have asked her to engage more physically with things to help us know better about how she does those things in particular. As we did an activity with her about making a bracelet for her using beads and that gave us a good insight into her ability with her upper body and hands.

We as a group had a uniform opinion about the interview as we all had similar ways of questioning the client which made us realize we were actually targeting the same realm. 

Introduction

For our Intro to Physical Computing class’s final project, our group, the Durians, is working with Bill, an individual from Pittsburgh with Achromatopsia, to develop an assistive device that’s useful and relevant to his experience. To better understand his wants and needs, and get to know Bill better, we conducted an informational interview with Bill. It was conducted via zoom with the members of our group: Sapna, Harry, and Angie. Our goal for the meeting was to gain a better understanding of aspects of Bill’s daily routine, tasks or activities he enjoys doing, and spaces for an assistive device during these.

Meeting Agenda:

Given that Bill has participated in this project before and is the primary liaison for the other clients, we had a relatively straightforward interview plan and anticipated that the interview would run quickly with straightforward answers.

• Introductions: names & background
• Quick rundown of project and purpose
• Bill’s Disability description and history of the disability
• What he struggles with day to day & problems that he has come across as a result of his disability
• Specific instances of struggle with disability
• Something that he wanted to do when he was younger
• What he wants from this project
• What previous groups from this class have done for him
• Any specific requests for devices (since he knows the project pretty well)
• Next steps

Meeting summary and major takeaways:

About the Disability: From the interview, we learned a lot about Bill’s Disability. He has severe light sensitivity and took the meeting in the dark in his house. He expressed discomfort in rooms with bright ambient light, especially outdoors in sunlight or looking at a bright screen. He is also legally blind with 20/200 vision which makes facial recognition, and looking at text at distances difficult. Finally, He has Achromatopsia. Which is full color blindness.

These vision problems were birth conditions so he has lived with these all his life. He recalls challenges from growing up like wanting to be able to play baseball but not being able to see pitches clearly.

Bill seems to have the most day to day complications with issues pertaining to color. He told us that the previous IDEATE team built a clothing color identifier to help him select his wardrobe. He explicitly wanted to address an issue he has with cooking with color blindness.

Since he cant see colors, it is impossible for him to tell the doneness of meat. He would like something to help identify when meat is cooked through appropriately. Here is a quick sketch of a potential devices from the meeting:

Rough sketch of Meat Thermometer Device

As a quick aside from the project, Bill mentioned that one of the important things he wants us to take from this project is experience interacting with the disabled community. We hope that this experience, along with class visits to CLASS and interacting with other clients gives us this opportunity to experience a subset of the community that is often marginalized and overlooked.

Thoughts and Discussions

• The overall interview experience was smooth. Bill is well spoken and has plenty of experience with this project.
• The conversation was good. We got mostly all the basic information we needed to start our ideation. But we may need more specific information from Bill, for example, the specific value of screen/daylight he is sensitive to.
• Bill seemed pretty set on an idea of something that we could do that interacted specifically with his disability. In hindsight, we probably should have also focused on general assistive devices that could help with quality of life improvements, not just vision based devices.
• We intend on following up with him with alternate ideas that we came up with not related to color blindness.
• Our group had a team member shuffle, we will likely contact Bill again with the full team in order to catch everyone up

Team Name: Apples

Team Members: Cameron, Stan, Lily

In this course, 60-223: Intro to Physical Computing, our team is working with Mark, a member of Pittsburgh’s Community, Living, and Support Services (CLASS). Mark is 60 years old and lives with diminished eyesight and a brain injury that affects his memory. We held our interview with Mark over the phone on Monday, March 20, 2023. The goal of this interview was to learn about Mark’s likes, dislikes, habits, struggles, disability, and most importantly how our group could assist him by collaborating on a single physical computing gadget that will benefit him.

Meeting Agenda

Introduction/Icebreaker:

• Introduce by name & share interests (what is your hobby and why?)

Explanation & clarification about project goals:

• We are students learning about building simple electronic devices to help people in their everyday lives (give examples?) Right now we’re doing a project to help build a device that can assist someone who is currently living with a disability. 
• Our goal with you today is to talk to you about your everyday life, and understand your hobbies, your routines, the things you like to do, and then understand the things that bother you or may be difficult for you – and then over the next 7 weeks, we’ll build a gadget or device that might help solve that problem.
• To preface, we are not professionals who are aiming to build a product which will be sold commercially. We are students learning how to design and build tools and gadgets with simple electronics, and the gadget we’re going to make with you is going to be made specifically for your needs and desires only. 
• Before we start, why have you chosen to volunteer in this project? What do you hope to gain from this?
• Remember to ask for questions

Understanding needs:

Helping the client to start thinking about daily routines and activities. 

• What time do you normally wake up, and from there, how do you usually go about your day?
• What’s one activity that you do everyday that you think is really important to you?
• What’s one activity that you would like to do more? Why is that important to you? 

Product-oriented questions. Things, devices that already help the client. 

• What are some things that you do or products that you use to help support…etc. 
• What are some must-haves that you carry with you everyday or things that you keep at home to make life easier?
• What’s one thing that’s with you right now that you could tell us about? How it helps you in daily life, how you found it, how often do you replace it…

Direct questions about needs and desires.

• Are there any small things that bother you that happen everyday you wish could be changed or you wish could be improved?
• What’s one thing that happens everyday that annoys you? How do you wish that could be improved?
• What is one thing that you enjoy doing everyday that has become more difficult?
  • Why is this important to you
  • Why has it become more difficult

Closing:

• Thank you so much for participating in our interview.
• What we’ll do now  is we’ll take some time and think about the things that you talked to us about today, and start building that gadget that will help you…. We’ll talk to each other again in a few weeks time, but by that time we’ll have a mock-up of a gadget that we’ll be prepared to show you. 
• Remember to ask for questions. 

Meeting Summary & Major Takeaways

Our interview with Mark helped us gain some insight into his habits, schedule, the things he enjoys doing, and the things he finds difficult. From the start, we got a sense that Mark is someone who has a strong sense of routines and is very consistent. Before Mark’s traumatic injury due to a car accident about 40 years ago, Mark was attending Duquesne University on a full music scholarship. Yet today, he still loves playing the drums. We aren’t sure what he was playing/studying in Duquesne exactly – but his continued passion for the drums is quite telling of his consistency, and other aspects that may be helpful to us (such as fine motor skills and a strong sense of rhythm & percussion). On that topic, Mark also makes sure he stays active, which he achieves by going on walks for more than two hours every single day. When he goes outside, some things he carries with him include his thick-lensed glasses, his monocular, and a cane. For outdoor activities where he needs assistance, such as going to buy groceries, Mark usually goes with a member of staff. When we asked Mark if there were any particular annoyances and hindrances in his daily routines, Mark did not voice any particular needs or wants. Upon consulting Bill however, we learned that Mark does have a hard time remembering the schedules which are set by his staff, despite his consistency. Some other observations we made throughout the meeting, and when some of us interacted with him during class was that he also has trouble accessing information that may seem trivial to us, such as the time and weather. To access this information, Mark usually has to call a hotline, or ask someone near him. With that, we hope to develop a solution that provides Mark with easy access to this information, without adding too much burden to his existing routines, and things he has to carry. 

Thoughts

Overall, our interview with Mark was very insightful and interesting. We were able to follow through with the questions we had planned, and learned more than we hoped to know, which provided us a lot of insight into what kind of person Mark is – his interests, hobbies, likings, etc. This was especially important because our interview with Mark was done over the phone. That said, we did have a hard time initially trying to pin down a need and want that we could work on, which is probably a product of just not knowing Mark well enough to begin with. With Bill’s input however (where Bill has known Mark for 15 years), we were able to learn some things we weren’t able to during the interview, and that set us on a path where we feel like we could design something impactful and feasible for Mark.