This project was a 7-week challenge to create a device that can provide assistance to a person living with a disability. We collaborated with Community Living and Support Services (CLASS), a local Pittsburgh nonprofit, to gain insight into the existing stage of assistive devices, proper methods and practices when working with clients that live with disabilities, and also to connect us to Keith, our client for the project.

Keith, our client, was a machinist who suffered a traumatic head injury, resulting in headaches, reduced mobility in his left side, and forgetfulness. Beyond that, he is a lovable grouch and has an appreciation for cheesecake. We documented our initial interview with Keith here.

What we built

Keith’s Explosive Schedule Reminder is a handheld phone-sized device that reminds him of his schedule. A screen on one side displays his upcoming scheduled event and a buzzer inside beeps when event is five minutes away from starting.

Keith’s Explosive Schedule Reminder

Keith’s Explosive Schedule Reminder

Keith’s device to scale

Internal parts of the device

Explosive side detail

Property of side detail

Battery pack detail

Battery pack detail

When Keith is at CLASS, he goes out for a smoke break. After he’s done, he’s having trouble remembering where he is supposed to report to. He grabs his reminder device and reads a display screen that says Math Class along with the location, time, and instructor. With this information, he is able to easily navigate back to the class on his own.

How we got here (prototype and process)

a. Prototype

Prototype 1 Demo

Prototype 1

Prototype 1 Deconstructed

Prototype 1 scale

Prototype 2

These two prototypes were designed to help answer the design question: What form and user-experience considerations are suitable for Keith’s device?

The first prototype was created using a white board material and blue LCD screen, which cycles through Keith’s schedule every few seconds. The second prototype is fixed to a chipboard, and reads out a class description, as well as vibrating.

The first prototype was specifically intended to show Keith what the device we were making would look like, so that we could further understand what limitations he might have towards adopting this new device. Size, readability, and usability were paramount. The second prototype was intended to show Keith what alert features we could incorporate, between audio and haptic feedback.

Our strongest piece of feedback from Keith was that he didn’t particularly care about the device itself, but wanted it to be red and read “explosive” on it. It was a frustrating critique session because Keith wasn’t very enthusiastic about the device, but we were able to ascertain that he could not read the LCD screen due to low contrast, but did not mind the size and shape of the device we were making for him. The feedback gave us a direction to move forward, but was demoralizing due to Keith’s disinterest. We hoped that by customizing the device to his preferences, he may be more excited about his device. Something we got from Bill was that the vibration motor wasn’t as important as an audio feature. Keith may not carry the device on his person at all times, and also may be wearing too many layers to feel a vibrating motor. We decided to limit our scope and focus on audio instead.

b. Process

Initial ideation was inspired by Bill’s sharing of Keith’s schedule at CLASS:

Keith’s Schedule

We found out that Keith had a pretty static schedule (with the only variations being instructors or irregular meetings off campus like with CMU). However, we also discovered that Keith keeps his schedule crumpled up in his pocket. This inspired us to make a device that he can’t lose or destroy, which he can keep on his person to remind him of his schedule, as well as reading it out when a scheduled item is close. Here is the very initial ideation sketch we made:

Ideation sketches

Some process drawing on the scale and form:

Whiteboard sketching of early idea

We initially conducted all tests and prototyping with a sound shield for the Arduino Uno, hence the scale and lack of real prototypes in the first stages of the project

Working with the e-ink display upon switching from the LCD after Keith’s feedback:

E-ink display testing

Soldering pins to the Adafruit board after receiving it late in the process:

Bhairavi soldering pin heads onto the Adafruit board

The craft and creation for the housing of the device was the largest challenge outside of electrical components, as we want the appearance to be appealing to Keith.

Solidworks model

Process for painting the box

Progress tracker:

Gantt Chart

This is the chart we used to track our process. Unfortunately, we did not actually receive parts until Monday the 24th, which lead to a lot of technical difficulties in the final phases of our project. We planned throughout the project to add a voice feature to the device to remind Keith of his classes, like his Alexa does at home. However, given our size limitation, we sought out an Adafruit board and music-maker in order to achieve this. We hinged our entire project on this, and the fact that we received the board so late, as well as frying it the night before, caused us to end up incorporating a buzzer and Arduino Micro instead. This decision was made out of desperation, and ideally we would have had more time to actually work with the final parts together.

4. Conclusions and lessons learned

Our largest realization through the making of this process was that Adafruit parts are not reliable, and that we should’ve had more contingent plans in place instead of putting our final prototype together the week before. We experienced a lot of extraneous circumstances that prevented our final product from being as successful as initially intended, but ironically had stumbled upon a product that might’ve done Keith more good on accident.

One of the most repeated pieces of feedback we got with our device was that Keith does not need to keep his device on him at all times. It could, instead, live in his cubby all day and serve as a location to ground him. If he forgets where to go, he will always know to return to his cubby to look at the device’s screen. From both critique feedback and Keith’s own admissions, we realized the primary feature of the device should be the screen, rather than the voice reminders. If we continue with this line of feedback, we might consider investing in a larger screen so that Keith can read the screen from a longer distance, as well as having a pop-up stand that allows the device to stand up on its own. Additionally, we could consider installing a re-chargeable battery and a button to toggle between voice/buzzer states and a muted state. These are all features that are dependent on the primary function of the device, which change based on what role it might have in Keith’s life.

Something that we learned pretty early on was that Keith was a client that was pretty content with his life. His schedules and habits weren’t bothering him, and he wasn’t particularly enthusiastic about making any big changes. This made interviewing him difficult, as we didn’t have a great grounding point for him to focus on ideating or considering making changes. Something promising brought up during our final critique was how our “final” device felt more like a late-stage prototype. This final device could very well be a device that Keith incorporates into his life for a few weeks and gives us concrete feedback on, which we can then respond to and adjust for the next stage of the device. This could be a good step to take for Keith, or any candidate that perhaps needs something physical to actually try out, rather than just imagining, which we were having him do previously.

The one particular strength of our final prototype was the visual design, which was something we’d identified early on as a feature that posed extremely high value to Keith in his decision to actually adopt any new technology. Creating any device to “assist” in someone’s life is by definition rather invasive, so we saw making that transition as personalized and meaningful as possible as a primary goal for whatever our device ended up being. As such, we spent a lot of time talking to him about his preferences in terms of color and decoration. In the final critique, we discussed possible expansion points for this project, and whether it had any standing as a product on mass production. IoT and such. The device can be applied to a variety of settings that choose to stay phone-free, such as summer camps or hospitals. A head administrator can hard code schedules into a series of devices using a master device or app and assign them to people. For settings specifically like CLASS, customizability for each device is paramount and can be maintained in this model.

One last concluding tidbit to this story is the fact that Keith loved his device. He was absolutely blown away by it, even though it was not necessarily to the fidelity that we had hoped we would be able to present. It just goes to show how much customizability matters when creating personalized devices, and would likely increase the device’s effectiveness tenfold (if we actually got it to work).

Keith’s Thumbs Up

5. Technical details

a. Schematic and block diagram

Original Block diagram:

Final Block Diagram

Block Diagram of final project

Schematic of our final project.

 

b. Code

// Keith's Explosive Schedule Reminder
// Team Members: Ethan Xu, Bhairavi Chandersekhar, Caleb Sun
// Code displays upcoming event information for Keith on an Adafruit ThinkInk e-display, as well as buzzing 5 minutes before an event

#include <RTClib.h>
#include <Wire.h>
#include "Adafruit_ThinkInk.h"

#define EPD_CS      9 // white
#define EPD_DC      10 // purple
#define SRAM_CS     6 // brown
#define EPD_RESET   8 // can set to -1 and share with microcontroller Reset! // RST blue
#define EPD_BUSY    7 // can set to -1 to not use a pin (will wait a fixed delay) // BUSY blue

#define buzzerPin 18

RTC_DS3231 rtc;
char t[32];
int h;
int m;
int d;

// 2.13" Monochrome displays with 250x122 pixels and SSD1680 chipset
ThinkInk_213_Mono_BN display(EPD_DC, EPD_RESET, EPD_CS, SRAM_CS, EPD_BUSY);

void setup() {
  pinMode(buzzerPin,OUTPUT);
  Serial.begin(115200);
  while (!Serial) { delay(10); }
  Serial.println("Adafruit EPD full update test in mono");
  display.begin(THINKINK_MONO);
  rtc.begin();
}

void testdrawtext(const char *text, uint16_t color) {
  display.setCursor(0, 0);
  display.setTextColor(color);
  display.setTextWrap(true);
  display.print(text);
}

void loop() {
  //setting up RTC module
  DateTime now = rtc.now();
  sprintf(t, "%02d:%02d:%02d %02d/%02d/%02d", now.hour(), now.minute(), now.second(), now.day(), now.month(), now.year());
  h=now.hour();
  m=now.minute();
  d=now.day();

  if((h==9) && (m==25) && (d=="Tuesday")){
  bitClear(DDRD,5);
  bitClear(DDRB,0);
  digitalWrite(buzzerPin,HIGH);
  delay (300);
  digitalWrite(buzzerPin,LOW);
  delay (200);
  digitalWrite(buzzerPin,HIGH);
  delay (300);
  digitalWrite(buzzerPin,LOW);
  delay (200);
  digitalWrite(buzzerPin,HIGH);
  delay (300);
  digitalWrite(buzzerPin,LOW);
  delay (200);
  display.clearBuffer();
  display.setTextSize(3);
  // spaces in testdrawtext are for center-aligning text; have to be done manually
  testdrawtext("    Math       at 9:30AM     Room 111   w/ Genevieve", EPD_BLACK);
  display.display();
  }
  //delay of 5 seconds is to serve as safeguard against e-ink display frying if updated too quickly
  delay(5000);

  if((h==10) && (m==55) && (d=="Tuesday")){
  digitalWrite(buzzerPin,HIGH);
  delay (300);
  digitalWrite(buzzerPin,LOW);
  delay (200);
  digitalWrite(buzzerPin,HIGH);
  delay (300);
  digitalWrite(buzzerPin,LOW);
  delay (200);
  digitalWrite(buzzerPin,HIGH);
  delay (300);
  digitalWrite(buzzerPin,LOW);
  delay (200);
  display.clearBuffer();
  display.setTextSize(3);
  testdrawtext("    Lunch      at 11:00AM    Room Main   w/ Courtney", EPD_BLACK);
  display.display();
  }

  delay(5000);

  if((h==11) && (m==55) && (d=="Tuesday")){
  digitalWrite(buzzerPin,HIGH);
  delay (300);
  digitalWrite(buzzerPin,LOW);
  delay (200);
  digitalWrite(buzzerPin,HIGH);
  delay (300);
  digitalWrite(buzzerPin,LOW);
  delay (200);
  digitalWrite(buzzerPin,HIGH);
  delay (300);
  digitalWrite(buzzerPin,LOW);
  delay (200);
  display.clearBuffer();
  display.setTextSize(3);
  testdrawtext(" Stress Mgmt   at 12:00PM    Room 111   w/ Michelle", EPD_BLACK);
  display.display();
  }

  delay(5000);  

  if((h==12) && (m==55) && (d=="Tuesday")){
  digitalWrite(buzzerPin,HIGH);
  delay (300);
  digitalWrite(buzzerPin,LOW);
  delay (200);
  digitalWrite(buzzerPin,HIGH);
  delay (300);
  digitalWrite(buzzerPin,LOW);
  delay (200);
  digitalWrite(buzzerPin,HIGH);
  delay (300);
  digitalWrite(buzzerPin,LOW);
  delay (200);
  display.clearBuffer();
  display.setTextSize(3);
  testdrawtext("    Music      at 1:00PM     Room 111    w/ Maggie", EPD_BLACK);
  display.display();
  }

  delay(5000);

  if((h==13) && (m==55) && (d=="Tuesday")){
  digitalWrite(buzzerPin,HIGH);
  delay (300);
  digitalWrite(buzzerPin,LOW);
  delay (200);
  digitalWrite(buzzerPin,HIGH);
  delay (300);
  digitalWrite(buzzerPin,LOW);
  delay (200);
  digitalWrite(buzzerPin,HIGH);
  delay (300);
  digitalWrite(buzzerPin,LOW);
  delay (200);
  display.clearBuffer();
  display.setTextSize(3);
  testdrawtext("  Wii Games    at 2:00PM     Room 111      w/ Bill", EPD_BLACK);
  display.display();
  delay (5000);
}
}

c. Design files

Keith Stl Files

Our client Mark lives with a visual impairment and mild memory loss due to a traumatic brain injury. From our conversation with Mark and his caretaker Bill, we learned that Mark’s day-to-day life can be improved if he had a device that helped remind him of his daily schedule in a way that integrates seamlessly into his existing habits. We met with Mark over the course of the project, twice in person and once over the phone. During these meetings, Mark gave us crucial feedback to fine-tune this project to his needs.

What We Built:

We created a handheld playback device that stores and plays pre-recorded audio messages. Each message is assigned to a specific hour of the day. When Mark picks up the device and holds it up to his ear, the device plays the message corresponding to the current hour. If Mark wants to hear reminders scheduled for other hours of the day, there is a dial located near the user’s thumb that allows for scrolling forwards or backwards through all 24 hours of the day. The device is designed so the user can comfortably scroll the dial whilst holding the device to the ear to listen for audio cues rather than a visual display to navigate through the hours. The housing is specifically formed to Mark’s hands and uses ergonomic touch-points (areas of the device that come in contact with the user) to comfortably and intuitively use the device.

Image of the final product being held. (1)

Image of the final product being held. (2)

Final CAD model of the playback device (including finger loop, speaker & thumb wheel) (3)

Final CAD model with cover removed to reveal interior (4)

Quick tour of the SolidWorks file (5)

Narrative:

BEEP BEEP BEEP! Mark’s 8AM alarm goes off. Mark lets out a large yawn, grabs his Portable Schedule Playback Device, and raises it to his ear.

“8AM: Good morning! Remember to grab breakfast with Paul at Starbucks.”

Ah that’s right! As he is gets ready, he starts to wonder what he has in store for the rest of the day. He puts the device to his ear and dials the device forward to 1 pm.

“1 PM: Remember to grab groceries.”

Oh yes, I need to buy cabbages for my homemade Korean kimchi recipe. He puts the device away in his pocket. “Wait, what restaurant am I meeting Paul at again?” He raises the device to his ear once more:

“8AM: Good morning! Remember to grab breakfast with Paul at Starbucks.”

Starbucks! That’s right. I think I’ll try their new Unicorn Galactic Frappuccino Supreme.

How We Got Here:

Prototypes & Processes:

We developed our first prototype to answer a question we all had on our minds: for someone with limited vision, what is a shape that is easy for them to distinguish, handle and interact with?

With that question in mind, we set a few goals that we thought were important to the overall functionality and concept of the device.

• Mark should be able to pick up the device from wherever it’s stored/held (in his pocket, on his belt, on his cane…), raise it up to his ear and hear the message instantly without any fumbling around/trying to figure out the device’s orientation, where the dial is, where the speaker is, etc.
• Mark should be able to line up the speaker to his ear quickly and comfortably so that he is able to hear the audio message with no time wasted.
• The size and weight of the device shouldn’t add too much burden to Mark’s existing everyday carry that it discourages him to bring and use the device on a daily basis.

What that, our first prototypes were ergonomic mockups of the device. They all follow the idea that Mark would raise the device to his ear and hear the audio message corresponding with the current hour. He can also scroll forward or backwards in the day to hear his previous/future schedule. Initial prototypes looked like this:

Side-profile of ergonomic prototype #1 made of foam. This design intends for the index finger to use the scroll wheel and navigate to the right recording.

Side-profile of ergonomic prototype #2 made of foam. This design intends for the thumb to use the scroll wheel and navigate to the right recording.

Side-profile of ergonomic prototype #3 made of foam. This design intends for the index finger to use the scroll wheel (now vertical) and navigate to the right recording.

Side-profile of ergonomic prototype #4 made of foam. This design intends for the thumb to use the scroll wheel (now vertical) and navigate to the right recording. This ended up being the closest to the final design of the device.

From these prototypes, we were able to make some decisions on the overall shape of the device, as well as some bigger aspects such as the placement of the dial (if it would be used by the thumb or the index finger). From there, we moved towards some more high-fidelity models. Some criteria that we set for ourselves were that the dial should be used by the thumb, and finger grooves should be implemented for better grip and to help Mark quickly identify the orientation of the device and make corrections.

High-fidelity ergonomic model (without working speaker) with horizontal thumb scroll. Gripped to show fit in hand.

High-fidelity ergonomic model (without working speaker) with horizontal thumb scroll. Open hand to show ergonomics.

Prototype that we presented to Mark to hold and play with.

Getting closer… the shapes of these prototype lends a bit more space for components on the inside and has finger grooves pronounced enough that it helps the user find a good grip on it. In the prototype directly above, we implemented a speaker unit and rotary encoder dial to get ourselves a bit closer to the look and feel of the final product – as well as starting to think about the placing of components. At this point, we were advised that the housing is just too small to house anything, if we were to keep to this form factor moving forwards. At this point, we also got an Arduino UNO to read file on an SD card and play audio files stored within, so we integrated that into the foam prototype.

Speaker demonstrating playback of a pre-recorded audio file.

Final prototype given to Mark to test with working speaker to get a sense of the volume adjustments, etc.

This ended up being the prototype that we showed Mark to get feedback on.

Overall, Mark was happy with how the device felt and our planned user process. In particular, Mark enjoyed the size and form factor, the finger grooves, and the placement of the thumb dial. Upon explaining our proposed user scenario, we also learned that Mark actually owns a cassette player at home, which his staff uses to relay their planned schedule on to him. We explained to him specifically the function of the device and how it’s able to keep track of time, how it’s able to keep a record of Mark’s schedule for the whole day, and how Mark is able to hear his current activity by raising the device to his ear or scrolling the dial. Cameron also brought with him his skateboard controller for Mark to try out, since we had it on us. He ended up really enjoying how it felt in the hands – especially the placement and orientation of the dial which spun vertically, and a hole/ring for the index finger to hook onto. With the proposed user process, Mark noted that the skateboard controller feels more appropriate for that interaction, especially because of how easy it is to scroll the dial on the skateboard controller.

Cameron’s skateboard controller that Mark ended up liking more than our foam prototypes!

Hearing Mark’s feedback was incredibly helpful, especially because we were able to ask him what he specifically enjoys about these different proposed shapes. In the end, the shape of the skateboard controller set the tone for what future iterations of the housing would look like. As said before, we were advised that the size and number of components we had to integrate would become a problem if we kept the same form factor, so we made sure to start working on a formal audit of all the parts we would include, their size, number of pins, etc. Generally, the prototype feedback session went exactly how we wished it would, and it was very helpful to get Mark’s approval and say on the ergonomics of the device, as well as the proposed user process. Moving forwards, we began to modify the shape of the device through some really quick prototyping, whilst trying to reserve as much space as possible for the internals.

Cardboard cutout of the final housing. It is thicker than the skateboard to accommodate for all the internal components.

Another angle of the cardboard prototype of the final housing for the device.

Moving forward, we split up figuring out the components between each person. Stanley worked on the tilt switch and IR sensor, both of which are responsible for getting the device to play an audio message as soon as Mark lifts up the device to his ear, as well as ordering and figuring out a rotary encoder we had ordered, Lily worked on a charging module so that we could get a charging lithium ion battery instead of having to replace an alkaline battery constantly as well as a RTC module so that the device can keep track of the current hour, and Cameron worked on the microphone, speaker and a music-maker shield so that we could record audio, separate it, and play it.

Breadboarded wiring of IR sensor and tilt switch for initial “hold-up-to-ear-and-listen” mechanism.

Here’s the first attempt at getting the IR sensor to work with the tilt switch to mockup the put-up-to-ear-to-listen interaction. In this example, the tilt switch, when activated, provides all power for the components on the breadboard. The tilt switch was surprisingly responsive and consistent, that we didn’t have to do any additional work to make it so that as soon as the device was put up to the ear, the IR sensor would start reading and send a signal to cause an output (which in this case was turning on a white LED).

Breadboarded wiring of music maker shield (deprecated) and detented rotary encoder for “scroll-and-browse” mechanism.

Here’s Cameron’s first time getting the rotary encoder to work with a speaker. Each detent corresponds with 1 hour difference. Spinning upwards scrolls forward in the day, and spinning downwards scrolls backwards in the day. At this point, all audio files are pre-recorded and stored in an SD card. At the beginning of each message, a robot voice reads out the hour that has been reached, followed by the corresponding voice message. If nothing is played, the user can scroll past, which overrides the current message being played instantly. At this point, we were able to 3D print a quick test housing to get a better idea of size and scale, ergonomics, and feasibility in terms of stuffing in electronics.

Initial print of our final housing. Notice that it closely resembles the skateboard remote.

It’s rough, but the bottom half of the body can actually be held in place without any external mechanical fasteners, and can be popped open to access the inside. This enclosure turned out to be too small, especially because of how deep the finger grooves were.

At this point, we were very short on time due to external factors that made it difficult to work consistently. So we went ahead and modeled and printed a second iteration that was a little larger in size, and had shallower finger grooves and fillets to afford more space in the interior. This ended up being the final housing that we worked with.

Note how the finger grooves cuts into the interior of the playback device significantly

Final CAD models of the previous iteration

A comparison between the past and most recent iteration of Mark’s playback device. Note the difference in size and proportions.

Second higher quality prints of the final housing.

Final prints of the device.

Again, since we were short on time, we decided to work in parallel, where some of us began to work on the final construction of the internal components (soldering everything to an internal power supply and Arduino Pro Micro), and others on the code and main functionality. We decided to scrap the microphone recorder function. The only way we could record audio and separate it was through a shield, which was simply way too big to fit in the device. Other components we found had other caveats, such as only being able to record 10s of audio, which we didn’t even know if we could separate and assign to different hours. Instead, we switched to an MP3 player that could read MP3 files (as opposed to the SD card reader which could only read WAV files that had to be encoded in a specific way via third-party software). We did this in hopes that this couldn’t compromise the recording portion of the device too much… on the assumption that Mark’s staff could be trained to use their portable devices to record MP3 audio into an SD card. It’s really not ideal, but it’s what we had to do with the amount of time we have available. Here’s how soldering went:

Final soldering of Arduino Micro, tilt switch, IR sensor, battery, MP3 player, and speaker.

Soldering process part 1: tilt switch, latching switch, voltage regulator, battery cap, MP3 SD card reader, speaker unit

Soldering process part 2: RTC module and rotary encoder (hidden behind), and test fitting into the housing.

Conclusions

In the end, we did not have a working prototype to demonstrate. Although each component was soldered and in the right place, the transition from Arduino Uno to Arduino Micro brought unforeseen behavior and hardware failures beyond our control. The transition was an inevitable bottleneck of the process and perhaps too much faith was put into the swiftness of such a transition. That said, our progress still evidenced lots of hard work and intention, and we were able to still receive great feedback about the concept and fabrication of the device.

Overall, our final product was well received. A few people noted that the housing had good ergonomics in reference to the overall ‘hold’ of the device, as well as the placement of the thumb dial and index finger loop. Here is what one student said, “The device is actually very comfortable! I have relatively small hands and found the size and feel of the turning dial to be really comfortable to use.” We came a long way from the foam prototypes that we showed Mark. Even though we ultimately adopted a design derived from Cameron’s skateboard controller, the final size, shape, and feel came from many considerations and incessant iterating. To our delight, most people interacted with the device the way we intended. Each person seemed to instantly recognize the important touch-points that made the device effective. They held the device as intended, scrolled the wheel with their thumb, and raised the device to their ear in exactly the spot that it was designed for. We were also told that the tangible aspect of the whole housing was much appreciated, and that they could see how our focus on designing around user interaction played a large role in the conceptualization, design, and fabrication of the project as a whole.

When we handed the device to Mark, we saw that he had a bit of trouble holding it correctly at first, which was most likely due to a tradeoff we had to make in order to save space–removing the finger grooves. This was perhaps the biggest drawback in our final design, since the finger grooves were so well-received by Mark in our initial prototype presentation. Unfortunately, the breadboard was ever-so-slightly too wide and took up the entire width of the housing, requiring us to pivot the design. Since day one, space consideration was always an obstacle we were aware about with this project. We spent a lot of time planning for this and ultimately theorized that all the components could fit. Even though the size of the housing did inflate a bit from original design and we had to get rid of the finger grooves, we believe the final design and shape of the device trumped the slightly bulky feel and discomfort.

We also received positive responses in terms of the concept of the device, and how we streamlined the device to not add extra burden to Mark’s existing everyday-carry, habits, and muscle memory. Here were some features that were applauded: having the device play audio as soon as it’s raised and put up to the ear, the ability for the user to scroll forwards and backwards throughout the day, and the hatch to access the internals and replace the battery. Overall, our biggest goal was to design a device that (A) worked, and (B) added minimal training and cognitive load to use the device. In hindsight it would also have been nice to experiment with different textures to distinguish different parts of the device, since after all this device is meant to be used without the need for sight. Of course, it is difficult to test out just how successful our device would be without a finalized working product (which is a point that came up in all of our critiques and we fully acknowledge). However, since each design aspect for this device was made with evidence-based backing and consideration, I think the purpose and ultimate integration in Mark’s life was transparent.

From this project, we were able to experience the hard yet humbling feeling of putting in hours and hours of work and not getting a working product at the end. One of the biggest obstacles we had was deciding on the best method for audio recording and playback. There are so many projects online with this feature, but only so many work with our space constraints. Recording audio ended up being infinitely harder than playing audio, which we realized (a bit too late) can only be done by a handful of hardware components that we didn’t have. There were many obstacles and thus considerations we had to make throughout this project, but the brick wall that slammed in our faces at the tail of the project was transitioning from an Arduino Uno to Arduino Pro Micro. Code, designed specifically for the Micro, failed to upload due to the device unrecognizable by any of our computers. As soon as we pressed “Upload,” the Pro Micro would disconnect. We tried four different Micros, checked and resoldered all our connections, and refactored our code twice, yet still the device refused to connect. We were unaware of the extreme unreliability of the Pro Micro and had no time to change our design, hence our ultimate failure. Our mistake was not having a secondary “all-organs-out” version of our circuit so that we could at least show the functionality of our device. This failure hit us all pretty heavily, but ultimately it was a good lesson learned.

Technical Details

Block Diagram:

Circuit Diagram:

Code:

// Title: Portable Schedule Playback Device by Apples Group 
// Author: Lily, Stan, Cameron
// Date: 5/05/23
// Description: This code handles the logic for playing back the appropriate pre-recorded
// recording stored in the microSD card depending on what hour of the day the dial is set to. 
// When the code boots for the first time, the rotary encoder is set to the current hour of the
// day (read from the RTC) and thus this is the first recording that is played back. Each
// recording is prefaced with a voice telling the time of the day. The code handles interruptions
// at any point in the recording/preface so the user can quickly browse to the desired recording.
// 
//
//                   -------------------------------------------------
//                   |   Role      |   Variable Name   | Arduino Pin |
//                   -------------------------------------------------
//  Rotary Encoder   |  Right Pin  |   encoderPinA     |      2      | https://www.nobleusa.com/pdf/xre.pdf
//                   |  Left Pin   |   encoderPinB     |      3      |
//                   -------------------------------------------------
//    MP3 Playback   |  RX Pin     |   pinDfpRX        |      10     | https://markus-wobisch.blogspot.com/search?q=DFPlayer
//                   |  TX Pin     |   pinDfpTX        |      11     |
//                   |  Busy Pin   |   pinDfpBusy      |      12     |
//                   -------------------------------------------------
//             RTC   |  SCL Pin    |   pinSCL          |      A5     |
//                   |  SDA Pin    |   pinSDA          |      A4     |
//                   -------------------------------------------------

the project title,
(optionally) your names,
a description (short or long) of what the code does,
any description of pin mapping that would be useful to somebody else trying to recreate your work,
appropriate credit to any code source that you incorporated into your project, and

//


                  /* GLOBALS */

// Rotary Encoder Globals //
#include <Keyboard.h>
enum PinAssignments {
  encoderPinA = 2,   // right
  encoderPinB = 3,   // left
};

volatile unsigned int encoderPos = 0;  // a counter for the dial
unsigned int lastReportedPos = 1;   // change management
static boolean rotating = false;    // debounce management
volatile int rotationDelt;
boolean A_set = false;
boolean B_set = false;
boolean clicked = false;
volatile int rotationPos = 1;

// MP3 Playback Globals //
#include "SoftwareSerial.h"
const byte pinDfpRX = 10;  
const byte pinDfpTX = 11;  
const byte pinDfpBusy = 12;  
const byte dfpVolume = 0x11; // set volume of DFPLayer - in range: 0x00-0x30 - default=0x30  
SoftwareSerial mySerial(pinDfpRX, pinDfpTX);
# define Start_Byte 0x7E
# define Version_Byte 0xFF
# define Command_Length 0x06
# define End_Byte 0xEF
# define Acknowledge 0x00 //Returns info with command 0x41 [0x01: info, 0x00: no info]

// RTC Globals //
const byte pinSCL = A5;
const byte pinSDA = A4;
#include <RTClib.h>
RTC_DS3231 rtc;


// Other Globals //
bool start = true;
bool startRot = true;
const byte nfiles = 6;     // number of mp3 files on SD card  
static byte ifile = 0;      // number of file played next
unsigned long tellTimeDelay = 800;
bool playMemo = false;
unsigned long memoDelayTimeStamp = 0;
static byte nextFile = 0;





                  /* FUNCTIONS */

// MP3 Playback Functions //
//plays a file named Par2 from folder Par1
void dfpExecute(byte CMD, byte Par1, byte Par2) {
  # define Start_Byte   0x7E  
  # define Version_Byte  0xFF  
  # define Command_Length 0x06  
  # define Acknowledge  0x00   
  # define End_Byte    0xEF  
  // Calculate the checksum (2 bytes)  
  uint16_t checksum = -(Version_Byte + Command_Length + CMD + Acknowledge + Par1 + Par2);  
  // Build the command line  
  uint8_t Command_line[10] = { Start_Byte, Version_Byte, Command_Length, CMD, Acknowledge,  
         Par1, Par2, highByte(checksum), lowByte(checksum), End_Byte};

  // Send the command line to DFPlayer  
  for (byte i=0; i<10; i++) mySerial.write( Command_line[i]);  
}


// Rotary Encoder Functions //
// Interrupt on A changing state
void doEncoderA() {
  // debounce
  if ( rotating ) delay (1);  // wait a little until the bouncing is done

  // Test transition, did things really change?
  if ( digitalRead(encoderPinA) != A_set ) { // debounce once more
    A_set = !A_set;

    // adjust counter + if A leads B
    if ( A_set && !B_set )
      // encoderPos += 1;
      rotationDelt = 1;

    rotating = false;  // no more debouncing until loop() hits again
  }
}

// Interrupt on B changing state, same as A above
void doEncoderB() {
  if ( rotating ) delay (1);
  if ( digitalRead(encoderPinB) != B_set ) {
    B_set = !B_set;
    //  adjust counter - 1 if B leads A
    if ( B_set && !A_set )
      //encoderPos -= 1;
      rotationDelt = -1;

    rotating = false;
  }
}


// Print Debugging //
// formats and prints the time of the day (AM/PM)
void printTime(byte ifile){
  if(ifile == 12){
    Serial.print(ifile);
  }
  else{
    Serial.print(ifile % 12);    
  }

  if(ifile >= 12){
    Serial.println("PM");
  }
  else{
    Serial.println("AM");
  }

}






void setup() {
  // Rotary Encoder Setup
  pinMode(encoderPinA, INPUT);
  pinMode(encoderPinB, INPUT);
  // turn on pullup resistors
  digitalWrite(encoderPinA, HIGH);
  digitalWrite(encoderPinB, HIGH);
  // encoder pin on interrupt 0 (pin 2)
  attachInterrupt(0, doEncoderA, CHANGE);
  // encoder pin on interrupt 1 (pin 3)
  attachInterrupt(1, doEncoderB, CHANGE);
  Keyboard.begin();
  // set the encoder position to the current
  // hour of the day
  rotationPos = rtc.now().hour();

  //RTC Setup
  pinMode(pinSCL, OUTPUT);
  pinMode(pinSDA, INPUT);
  rtc.begin();       // initialize rtc


  Serial.println("\n\n\n\n\n\n\n---------------------\n\n");
  Serial.print("The hour of the day is ");
  printTime(rtc.now().hour());


  //MP3 Player
  mySerial.begin(9600);
  pinMode(pinDfpBusy, INPUT);   // init Busy pin from DFPlayer (lo: file is playing / hi: no file playing)  
  dfpExecute(0x3F, 0x00, 0x00);  // Send request for initialization parameters  
  delay(5000);
  dfpExecute(0x06,0x00,dfpVolume); // set volume DL=0x00-0x30, default=0x30  
  delay(30);            // have >20ms delays between commands  

}





void loop() {
  unsigned long currTime = millis();

  // speaker plays the MEMO (after tellTimeDelay milliseconds have passed)
  if(playMemo && (currTime - memoDelayTimeStamp) > tellTimeDelay){
    Serial.print("Playing memo for ");
    printTime(ifile);
    dfpExecute(0x0F,0x02, nextFile);
    playMemo = false;
  }
  
  rotating = true;  // reset the debouncer

  // register a left turn
  if(rotationDelt == -1) {
    Keyboard.press(KEY_LEFT_ARROW);
    rotationPos -= 1;
    if(rotationPos < 1){
      rotationPos = 24; // we wrap around to 24 when we go below 1
    }
  }

  // register a right turn
  else if(rotationDelt == 1) {
    Keyboard.press(KEY_RIGHT_ARROW);

    if(rotationPos > 24){
      rotationPos = 1; // we wrap around to 1 when we go over 24
    }
  }
  

  // if the encoder changed position...
  if(startRot || rotationDelt != 0) {

    Serial.println(rotationPos);
    startRot = false;

    ifile = rotationPos;
    Serial.print("Telling time for ");
    printTime(ifile);

    // ...speaker tells the TIME
    dfpExecute(0x0F,0x01,ifile);
    playMemo = true;
    nextFile = ifile;
    memoDelayTimeStamp = currTime;
  }

  Keyboard.releaseAll();
  rotationDelt = 0;

}

CAD Files:

Mark’s Playback Device SW Files

For this project, we were tasked to create an assistive device for a client with a disability. Our class focuses on small electronics and interaction, so we had to find a way to use those skills to create a device that would helpful for our client. We were connected to our client through CLASS, a local organization that provides a myriad of services for folks with disabilities. Over the course of six weeks, we met with our client, designed prototypes, tested prototypes, and finally created the final product. You can read more about the initial interview here.

What we built

We met with our client Darnisha and she told us how she would often set her phone down and forget where she had placed it. Additionally, due to her mobility, she had trouble reaching over to turn off her alarm on her phone in the mornings. The solution that we came up with solves both these problems with one device. The device hooks onto her wheelchair armrest and has a phone holder that lights up when the phone is not there to remind her to locate her phone. The device can unhook from the chair and sit on her bedside table to act as an alarm clock that is turned off by waving her hand over a sensor.

The final phone holder and alarm.

This video demonstrates the alarm function. The alarm is preset to Darnisha’s schedule, as it is very consistent and does not change often. When the alarm goes off, Darnisha turns it off by simply waving her hand over the top of the device.

Here, you can see the phone holder function. There are flashing rainbow lights that display when the phone is not in the slot to remind Darnisha that her phone is not on her. With the help of an IR sensor, when the phone is placed in the slot, the lights turn off.

The top view of the device. Here, you can see the small IR sensor in the middle of the phone slot, and the ultrasonic ranger on the side which turns the alarm off. Also shown here is the handle, which was laser cut acrylic that we then bent with a heat gun to fit the curve of Darnisha’s wheelchair armrest.

Darnisha’s phone lays diagonally in the slot to allow for her to easily grab the phone from underneath.

We made the back panel of the device removable in order to access the electronics inside. We achieved this by using small “T” joints with a bolt and nut. Additionally, we used a piano hinge so that the handle can lift up, slide over the armrest, and the device hangs over the side.

Show here is the bottom of the device, where there is a panel mount battery pack to easily change batteries. This is also where the ON/OFF switch is located.

A narrative sketch –

When Darnisha gets up in the morning she reaches out to her side table where her phone holder and alarm sits. As the alarm goes off the display screen says Get Up!!! She waves on top of the ultrasonic sensor to turn off the alarm. 

After she is all ready for her day, she attaches her phone holder to her wheelchair which helps her keep her phone at a more accessible position and avoids her from misplacing her phone. As Darnisha picks up the phone to use the LED strip lights up indicating her that the phone is missing from its position.

How we got here

Prototype

The fabrication prototype was designed to determine the best way for the device to attach to Darnisha’s wheelchair. The electrical prototype was designed to determine a couple questions: would the alarm be loud enough for Darnisha? Did Darnisha want to set her alarm each day or have it preset in the device? What color and brightness would be best for Darnisha?

The first prototype was designed to sit on top of the armrest, and have the ability to swing out over Darnisha’s lap to act as a phone stand while she uses her phone. Additionally, we had an electrical prototype of both the LED function and the alarm function. The LEDs are connected to an IR proximity sensor. When the sensor is covered, the lights are off, and when it is not covered, the lights turn on. The alarm was shown on an LCD screen. The alarm time was inputted by and numerical keypad. The alarm is turned off by waving over an ultrasonic ranger.

The cardboard prototype of the physical device.

This prototype was meant to sit on top of Darnisha’s armrest.

The top of the device swivels out to provide a phone prop for Darnisha to use while on her phone.

The prototype for the LED lights that turn on and off based on the IR sensor input.

The prototype for the alarm turned off by an ultrasonic ranger.

Showing Darnisha our first prototype design.

Here, we realized that having the device on top of her armrest would inhibit her use of the armrest. She decided that the swivel part wasn’t necessary for her and it would be more useful on the side of her chair.

We also tried out some alternative handle designs.

Throughout the prototyping process, we tested a variety of ways for the device to attach to the wheelchair, including it sitting on top of the armrest, connecting to a bar below the armrest, and hooking over the armrest, which is what we ended up going with. This choice still allowed for comfortable use of the armrest, and the device was still close enough that Darnisha could easily grab her phone. We also used this time to get as many measurements of the armrest as possible so that we could make the fit perfect, as this would be the last time we would see Darnisha in person before the final critique.

As far as the electronics, Darnisha decided that the buzzer we had used would be loud enough and wouldn’t need volume control. We learned that her schedule is very rigid and doesn’t change often, so she wanted the alarm times to be preset in the device. This meant that we would get rid of the keypad. Finally, we asked about the light brightness and color. Darnisha liked the brightness of the lights, and asked for them to be rainbow and flashing.

 Process

Once we received feedback on the prototypes, we got into the process of creating the final device.

Assembling the laser cut pieces. Somehow, all the pieces and components fit together on the first try, which was really exciting.

Here we tested out bending acrylic with a heat gun on a scrap piece.

Once we heated the acrylic till it was pliable, we bent around a PVC pipe that roughly matched the curve and diameter of the armrest.

Once we had mostly assembled the box, we spray painted it Darnisha’s favorite color – pink!

Then, we began assembling all the electronics inside. The holes and slots that were cut for the components were almost an exact fit, so it took some patience so get everything into the right spot.

Here, you can see everything in place, except for the back panel. The back panel is secured with t-joints with small nuts and bolts. It’s very secure, but can be removed if necessary to get to the electronics.

We got good feedback from Darnisha in terms of her preference for position, light, and usage during the prototype crit and her being in person helped us get the wheelchair measurements, which made it easier for us to design the attachment to the correct size. However after digitally prototyping the model and assembling it, we realized that there was an issue with the size of the phone holder. Even after measuring Darnisha’s phone size we should’ve added some allowance in the width and height, taking in consideration if she changes her phone that is bigger. Phone in the space was not laying flat because of the size issue but we framed it as a very happy accident by saying that ” Darnisha’s phone actually doesn’t lay flat, but that’s a feature and not a bug because it makes it easier to grab out of the device.” This was the major breakthrough during our process.

After analyzing the whole product we also realized that the LED strip was exposed and that it might not be the best decision in terms of safety for the product and the user. However we had fabricated and assembled the whole product, but we definitely noted this point down for future consideration.

The Gantt Chart was very helpful in aiding us to plan and divide our work. We didn’t diverge from it and managed to keep to our original goals as we gave enough space for each task and left a decent amount of room at the end to do some final touch ups. For the tasks we split 2 v 1 depending on the task. For example, Mo and I started modeling the device and making laser cutter files while Michelle opted to combine our already tested code in bits and pieces and solder the electronics. The effort was distributed fairly and we switched places wherever needed. Throughout the process we made sure to check in with each other and cover up for the other if needed so that there is no lag with the work.

Conclusions and lessons learned

Salient Findings from Final Critique

Our final critique was very formative and valuable to us, and we definitely learned a lot from it. Most of the feedback seemed to be in the vein that the device was very solid, and a few more tweaks or features could improve it.

Time display is rudimentary. Day of week could also be helpful.

We had a few people point out that the time display could be improved by adding additional information, such as the day of the week. There was also a similar comment about the digits in the display being formatted somewhat unconventionally, due to the lack of leading 0’s before each digit. This is definitely something that may seem like a small detail, but in practice could greatly improve user experience with a simple fix. In a future iteration, we would update the code to do so.

I think it would be helpful if the device showed its current charge since it’s battery-powered — or if it could be recharged.

Adding an indicator for current charge is something that had never crossed our minds while we were designing the product, but would definitely be extremely useful in practice. During critique, it was pointed out that this is especially important, because having a device unknowingly die out in the middle of the night may be problematic since it would prevent the alarm from going off in the mornings. Adding a battery percentage, or even a low-battery indicator would help Darnisha prepare in advance for this.

I appreciated the design and approachability of the product, the lights and motion sensor to turn off the alarm were well planned out while remaining fun and remaining true to the client’s personality. The strawberries were a cute touch

We were really happy to receive this feedback, because it was really important to us that this device would be something that would be fun for Darnisha to be a part of and that would be really useful to her in practice. We put a lot of time and effort in our design considerations, so it was great to see this re-affirmed.

I was mildly worried about the durability of the design and the security of the phone throughout perhaps a busier day.

We felt that this feedback was definitely a very relevant concern, especially after learning from Darnisha that there may be some difficulty keeping her phone in the holder when she boards the ACCESS transit service. In the future, we could use a more durable material or add additional reinforcements, other than just the wood glue.

Major Takeaways

1.  Often, devices and tools for disabled folks are often made by taking something else and tweaking it to fit their needs. It was very cool to be able to create a device fully for our client’s needs from the beginning. We could really take her abilities and desires into account and make it perfect for her, and not worry about making it “marketable” in any way.
2. Collaboration is crucial: Designing something for a person with a disability is not a solo endeavor, and instead requires a lot of listening and collaboration to truly understand their needs. This project was not only made for Darnisha, but with Darnisha.
3. Small details matter: When designing something for a person with a disability, small details can make a big difference. Something as simple as the placement of a sensor or simple measurements can have a significant impact on the user’s experience. It is important to pay attention to these details and make adjustments as needed. Something challenging in this process was building the device to the exact dimensions of Darnisha’s wheelchair and phone size. In the end, while the device fit and came together well, we could’ve taken even greater care to get the exact dimensions.

Concluding Thoughts

Overall, it was a super amazing experience to get to work on this project with Darnisha. It was really cool getting to build this device from beginning to end, all the way from designing the device to putting together a final product that she loved and was customized to her needs. We learned a lot about fabrication and the importance of considering all the small details of a design and a user’s needs. With such a complex project, we also learned to break down big tasks into smaller ones and to keep things modular, whether it be in the code or general planning. We also learned the importance of building extra time into our schedule and to account for last-minute issues that might come up.

If we were to do the project a second time around, we’d double-check measurements even more than we already did to ensure that everything fit perfectly. This would also involve checking fit more carefully before assembling the project, so that we could make adjustments as needed. We’d also make sure to take into consideration the space that would be taken up by joints & connections at the beginning of the process. If we had more time, we’d also pay closer attention to different usability elements, such as the LCD time display and add more features, such as adjusting the alarm, snooze button, etc.

Technical details

Block Diagram

Schematic

Code

/*
* Phone Holder and Alarm
* Team Peaches: Mo Cambron, Juhi Kedia, Michelle Liu
* 
* Code for a wheelchair attachment that lights up an LED strip
* there's no phone blocking the proximity sensor and has
* a motion-activated alarm clock functionality.
* 
* Pin Map:
  variable name | mode | pin | description
  -----------------------------------------------
  PING_PIN | input | 3 | reads & emits ultrasonic ranger data
  LED_PIN | output | 7 | LED strip
  PROXIMITY_PIN | input | A0 | reads IR sensor data
  BUZZER_PIN | output | 13 | makes buzzing noise
* 
* Credits:
* RTC alarm clock code & reference: https://adafruit.github.io/RTClib/html/class_r_t_c___d_s3231.html
* 
* Understanding how the RTC needs to be coded and how the information
* from the RTC can be displayed on the LCD:
* https://www.youtube.com/watch?v=aJcncPB0GHg
* 
*/

#include <Wire.h>
#include <RTClib.h>
#include <LiquidCrystal_I2C.h>
#include <PololuLedStrip.h>

#define I2C_ADDR 0x27 //LCD i2c stuff
#define PING_PIN 3
#define LED_PIN 7
#define PROXIMITY_PIN A0
#define BUZZER_PIN 13;
#define MAX_DISTANCE 200
#define LED_COUNT 27
#define PROX_THRESH 850

PololuLedStrip<LED_PIN> ledStrip1;
LiquidCrystal_I2C lcd(I2C_ADDR, 16, 2);
RTC_DS3231 myRTC; //Wiring of the RTC (CLK,DAT,RST)
//If  you change the wiring change the pins here also

// Alarm clock hours
const int WEEKEND = 12;
const int WEEKDAY = 7;

// note: change for demo purposes; should be 0 in actual version
int alarmMin = 0;

int proxState0 = 0;

unsigned long rainbowDelay = 7;

rgb_color on[LED_COUNT];
rgb_color off[LED_COUNT];

const int ALARM_TIMES[7] = {WEEKEND, WEEKDAY, WEEKDAY, WEEKDAY, WEEKDAY, WEEKDAY, WEEKEND};
int RAINBOW[7][3] =
{
  {255, 0, 0}, // Red
  {255, 30, 0}, // Orange
  {255, 200, 0}, // Yellow
  {0, 255, 0}, // Green
  {0, 0, 255}, // Blue
  {75, 0, 220}, // Indigo
  {148, 0, 211} // Violet
};

int hue = 0;

void setup() {
  lcd.init();
  Serial.begin(9600);
  lcd.begin (16, 2); //Initialize the LCD
  lcd.backlight();

  pinMode(PING_PIN, OUTPUT);
  pinMode(PROXIMITY_PIN, INPUT);

  if (!myRTC.begin()) {
    Serial.println(" RTC Module not Present");
    while (1);
  }

  // Uncomment on initial RTC setup to set the initial time
  // comment back out and reupload after initial setup, in order
  // to prevent RTC from resetting every time
  
  //  if (myRTC.lostPower()) {
  //    Serial.println("RTC power failure, reset the time!");
  //    // automatically sets the RTC to the date & time on PC this sketch was compiled
  //    //    myRTC.adjust(DateTime(F(__DATE__), F(__TIME__)));
  //    // rtc.adjust(DateTime(2023, 2, 20, 3, 0, 0));
  //  } else {
  //    myRTC.adjust(DateTime(F(__DATE__), F(__TIME__)));
  //    //    rtc.adjust(DateTime(2023, 3, 2, 10, 0, 0));
  //  }

  //  myRTC.adjust(DateTime(2023, 4, 26, 10, 0, 0));

  int* curColor = RAINBOW[hue];
  for (int i = 0; i < LED_COUNT; i++) {
    // on is red
    on[i] = rgb_color(curColor[0], curColor[1], curColor[2]);
  }
  for (int i = 0; i < LED_COUNT; i++) {
    // off is clear/nothing
    off[i] = rgb_color(0, 0, 0);
  }

  DateTime now = myRTC.now();
  Serial.print(ALARM_TIMES[now.dayOfTheWeek()]);
  myRTC.setAlarm1(
    DateTime(now.year(), now.month(), now.day(), ALARM_TIMES[now.dayOfTheWeek()], alarmMin, 0), // Set alarm time to 7 am
    DS3231_A1_Date // Set alarm to trigger when date & time match
  );
}

// Helper function to get Ultrasonic Ranger distance
int getDist() {
  pinMode(PING_PIN, OUTPUT);

  // Create a wave by alternating a HIGH and LOW pulse
  digitalWrite(PING_PIN, LOW);
  delayMicroseconds(2);
  digitalWrite(PING_PIN, HIGH);
  delayMicroseconds(5);
  digitalWrite(PING_PIN, LOW);

  // The same pin is used to read the signal from the PING))):
  pinMode(PING_PIN, INPUT);
  return pulseIn(PING_PIN, HIGH);
}

// Convert military time to AM/PM
int militaryToRegular(int militaryHour) {
  if (militaryHour == 0) {
    return 12;
  } else if (militaryHour <= 12) {
    return militaryHour;
  } else {
    return militaryHour - 12;
  }
}

// Display date on LCD
void displayDate() {
  DateTime now = myRTC.now();
  DateTime tomorrow = now + TimeSpan(1, 0, 0, 0);
  lcd.setCursor(0, 0);
  lcd.print(now.month());
  lcd.print("/");
  lcd.print(now.day());
  lcd.print("/");
  lcd.print(now.year());
  lcd.setCursor(0, 1);
  lcd.print(militaryToRegular(now.hour()));
  lcd.print(":");
  lcd.print(now.minute());
  lcd.print(":");
  lcd.print(now.second());
  lcd.print(now.hour() < 12 ? "am" : "pm");
}

unsigned long tm = 0;


// Flash rainbow colors on LED strip
void rainbow()
{
  if (tm % rainbowDelay == 0) {
    int* curColor = RAINBOW[hue % 7];
    for (uint16_t i = 0; i < LED_COUNT; i++)
    {
      on[i] = rgb_color(curColor[0],  curColor[1],  curColor[2]);
    }
    hue++;
  }
  tm += 1;
}

void  loop() {
  delay(100);
  DateTime now = myRTC.now();
  DateTime tomorrow = now + TimeSpan(1, 0, 0, 0);
  long val = getDist();
  int readVal0 = analogRead(PROXIMITY_PIN);
  proxState0 = readVal0 > PROX_THRESH ? 1 : 0; // whether or not proximity sensor is covered
  rainbow();

  if (proxState0 == 1) {
    ledStrip1.write(on, LED_COUNT);
  } else {
    Serial.println("off");
    ledStrip1.write(off, LED_COUNT);
  }

  displayDate();

  if (myRTC.alarmFired(1)) {
    while (val > 500) {
      tone(BUZZER_PIN, 1000); //You can modify the tone or make your own  sound
      delay(100);
      tone(BUZZER_PIN, 2000);
      delay(100);
      lcd.clear();
      lcd.print("Get up !!!"); //Message to show when the alarm is ringing
      val = getDist();
    }
    lcd.clear();
    myRTC.clearAlarm(1);
    myRTC.disableAlarm(1);

    // set alarm for tomorrow
    myRTC.setAlarm1(
      DateTime(tomorrow.year(), tomorrow.month(), tomorrow.day(), ALARM_TIMES[tomorrow.dayOfTheWeek()], 0, 0),
      DS3231_A1_Date // Set alarm to trigger when date & time match
    );
    noTone(BUZZER_PIN);
    delay(100);
  }
}

Design Files

Here, you can access the design files used to model and laser cut the device: Design Files

We are Team Guava, and our client is Shaude.

Over the weeks, we talked to and interacted with Shaude to get to know her better and to understand what she does. From those conversations, we were able to find and target aspects of her life that we could help to improve, such as picking fallen things up, helping her paint, and assisting her in communicating, especially in the morning.

Link to previous post: https://courses.ideate.cmu.edu/60-223/s2023/work/interview-with-shaude-team-guava/

1. What We Built

By having two boxes that can communicate with each other at a distance, we can give our client, Shaude, the ability to communicate with someone in her home much easier than a smartphone. In particular, we had one device for the client’s bedside that could send up to six predetermined messages for specific needs and two adjustable gradient indicators for both general mood and health to another device that could be placed anywhere in the home for the ease of the caretaker. We also made it so both devices run based on a wall outlet, removing the need to change batteries in the product.

Controller (left) and Display (Right)

Controller – Custom Slider Cap, Arcade Buttons, Sticker Papers

Controller – Internal Wiring

Display – LCD

Display – Internal Wiring

By having two boxes that can communicate with each other at a distance, we can give our client, Shaude, the ability to communicate with someone in her home much easier than a smartphone. In particular, we had one device for the client’s bedside that could send up to six predetermined messages for specific needs and two adjustable gradient indicators for both general mood and health to another device that could be placed anywhere in the home for the ease of the caretaker. We also made it so both devices run based on a wall outlet, removing the need to change batteries in the product.

Video of Our Devices Working as Intended.

Narrative sketch of how our product works

2. Prototypes and Processes

Our first prototypes were built in order to answer the following questions: 

• What inputs and outputs we wanted to use?
  • Such as how should Shaude enter her emotions and how should they be shown
• What colors and gradients made the most sense for out purpose 
  • Should we use light color or brightness?
• What scale should the device be at
• What should be the layout of the device 

We made a lot of prototypes testing the hardware that we would be using, such as the radio, the potentiometers and the Neo pixel bulbs. These were practical prototypes to see how the coding and functionality would look like. We also printed prototypes of the housing so we would be able to better address scale. Some prototypes were simple drawings exploring how different aspects of the outputs might be, such as drawing of different light gradients or LED configurations.

These sketches show our thought process behind our questions, and some things we wanted to explore in our prototyping.

Neopixel bulb light up for the first time (emotional)

This was our first version of slider to lights. This simple setup controlled the red blue and green value on the Neopixel. You can see that they all run to the same bulb and effect each other. The position on the slider effects the brightness of each color in the bulb. We were happy with this experiment because it was much like mixing paint, but with light, and it was fun to play with.

Above is second iteration of the potentiometer to lights. We tested mapping specific gradients to the lights. Here you can green to red as well as red to blue. You may notice the weird jump from blue to red in the lights. This was a coding issue we needed to battle and happens because the red value is much stronger than the blue.

LED strip alternative to neopixel attempt

This idea correlated to a sketch seen above, where the amount of mood or health corelates to a number of bulbs illuminated on the LED strip. This idea was scraped because it seemed a little confusing and added a layer of complexity that was unnecessary. This porotype was never brought to a fully functional level.

Above you can see a video of us testing the connection between two Arduinos using Arduino. The input is the press of a button and the output is a message displaying on an LCD screen.

physical prototypes

To test how big we wanted to control panel to be we printed a few panel to show to Shaude for layout and scale. We also wanted to test using button caps to make the buttons differentiated and easier to press. These caps did not end up working because they attached onto the tops of the buttons and were not deep enough to create a snug fit.

We learned the most from our prototypes after discussing the outcomes with Shaude. After showing her the scale of the models we learned that she needed much bigger buttons, and a big display box for her to see what the options on the box were. She also noted that she preferred a flat down view of the box rather than a slanted one. She enjoyed the potentiometers and the feedback of the lights. We talked to her about what colors she connected with different expressions and moods. We were then able to use this information to decide on our final color gradients that would be most descriptive and helpful to her. The range of these colors were red to blue for agitated to calm and green to red for health to ill. We also discussed her favorite colors and such so we could design the box to be most pleasing to her later on. Her favorite colors by the way are purple and pink, and she really likes dogs. 

During our prototyping process we were surprised by how the neopixel colors values are not all equal. The power and brightness of the red really overpowered the blue and we had difficulty mapping the values to get a noticeable gradient from blue to red. If we had more time to prototype we would have liked to explore other visual methods and hardware for showing the gradients and values of colors. We had one drawn idea of utilizing strip LEDs to use both color and value to show mood and health (see above), but it seemed to add a layer of confusion, especially with calm and agitation. We chose to continue with the Neopixel bulbs because they were simple to read and true to our original intent. Overall going further with our design we took the work from our prototypes and simply made them better suited for Shaude.

3. Conclusions and Reflections

Overall, our group thought that our product successfully fulfilled our client’s identified needs. Through interview and iteration, we were able to come up with a device that allowed the communication of both specific desires and more general moods in a concise and effective way. However, the process did involve challenges, not the least of which was working with our client. Working in an environment like CMU, we are often surrounded by people of similar dispositions who know what we can accomplish. This project allowed us to work with someone with a disability who had none of these attributes. In particular, we often found it hard when speaking or working with Shaude to get actionable feedback or find specific problems that were within our skills to address. This unexpected difficulty forced us to change how we interacted with Shaude in our second interview. Instead of open-ended questions where we expected her to give us actionable answers, we provided her with a list of possible options or user interfaces. This narrowed the scope of choices and discussion, allowing her to point out any general issues in all the possibilities. For instance, when we were giving her options on how the buttons were laid out, she expressed that even the most prominent option we had was too small. This informed us of a significant issue we must be aware of. Next time, I would like to schedule periodic meetings with Shaude at the beginning of the project to increase her involvement in our process and interview her caretaker to get a feel for their opinion on the output device.

The final crit was an informative and valuable process. Our team felt that those present gave good feedback on what we did well and what we could have done better as a final project. In particular, the one we felt was most true was regarding the layout of the mood and health potentiometers, “The ‘mood’ light felt backward, and the red/blue color scheme was hard to decipher any middle ground.” This was a comment that a couple of people gave us because the mood and health sliders went opposite directions to increase/decrease, which needed to be clarified, and the spectrum for the mood light (red to blue) was hard to see any interim values. With more time, we would fix this by flipping one of the potentiometers so that both are the same and choosing better colors to the spectrum between. Another comment we got was to improve the “Robustness of the device. Add texture or images to buttons, which is something we would want to do. We were considering making our buttons at one time, which could have allowed us to do this. Making it even easier by allowing buttons to be differentiated by feel or image instead of just clarifying what each does by text. Another useful feedback was about having a message on the receiving device for too long. This could be addressed by “adding some kind of ‘acknowledge’ button to the remote receiver, which would then silence the message and blank out the screen.” This is something that we should have considered and is a problem. Because it teaches the recipient that messages are not current, decreasing the device’s effectiveness. We did have difficulties with the radio, and getting two-way communication instead of one-way would be a challenge that we felt would have taken more time than we had. Finally, someone commented, “Bedside table space can be limited, so a slightly smaller bedside device” would be better. We agree and think that its height might also become an issue. If we could do it again, the solution would be to cut the arcade buttons in half, which would have worked because the buttons’ electrical and physical components could be separated. The buttons took up the majority of the space inside the input box.

To sum up, this device solves a real problem in Shaude’s life. Hopefully, it gets used, and no issues pop up over long periods of use, but we don’t think they will. It was an exciting experience that Team Guava was happy to be a part of.

Final project, Shaude, and Team Guava at final Critique

4. Technical Details

Block Diagrams:

Block diagram showing input and output flow for Shaude’s portion of communicator

Output Device Block Diagram

Receiver block diagram

Electrical Schematics:

Electrical Schematic of Input device for Mood communicator

Receiver Wiring diagram

Code:

/*
 * Mood Communicator, input code, final project 
 * By: Andres Montemayor, Evette Lacomb, Dunn Zhang

 Description: The code below takes in inputs from buttons, and sliding potentiometers
 then, displays a gradient from red to blue and red to green based on the pot values
 it also transmits those values over a NRF24L01 radio to another arduino

 Pin Map:
  POTPIN1     | input    | potentiometer: sets mood gradient
  POTPIN2     | input    | potentiometer: sets health gradient
  BUTTON1     | input    | reads when message 1 button pressed, sets integer for radio to send
  BUTTON2    | input        | reads when message 2 button pressed, sets integer for radio to send
  BUTTON3    | input        | reads when message 3 button pressed, sets integer for radio to send
  BUTTON4    | input        | reads when message 4 button pressed, sets integer for radio to send
  BUTTON5    | input        | reads when message 5 button pressed, sets integer for radio to send
  BUTTON6    | input        | reads when message 6 button pressed, sets integer for radio to send
  moodLed   | output        | outputs rgb value to neopixel to show potentiometer value on spectrum(red->blue)
  healthLed | output       | outputs rgb value to neopixel to show potentiometer value on spectrum(red->green)

NRF24L01 pin map can be seen below 
 * 
 * radio pin    Arduino Uno/Nano pin    Arduino Micro pin
 * VCC          3.3V                    3.3V
 * GND          GND                     GND
 * CE           7                       7
 * CSN          8                       8
 * MOSI         11                      MO
 * MISO         12                      MI
 * SCK          13                      SCK

 * modified from code by: Alton Olson, Vicky Zhou, Seema Kamath, Robert Zacharias, Andres Montemayor, Evette Lacomb
 Credit:
 Used example libraries from relevant libraries for relevant sections
 for radio code, it was provided by Robert Zacharias and then later adjusted by team to better fulfill project
 parameters.
 * 
 */

//Libraries in use in this code
#include <SPI.h>
#include <nRF24L01.h>
#include <RF24.h>
#include <Wire.h>
#include <PololuLedStrip.h>
// numbers in <> are pin numbers for the leds to display mood and health, reference for what is transmitted
PololuLedStrip<2> moodLed;
PololuLedStrip<4> healthLed;

//LED initialize count. We use a single neopixel so only one, if use a strip would be more
#define LED_COUNT 1
rgb_color moodcolors[1];
rgb_color healthcolors[1];

//define integers for colors map from sliding potentiometer
int red;
int red2;
int blue;
int green;

// Initialize the pins for the buttons, potentiometer, radio, single 
const int BUTTON1 = 3;
const int BUTTON2 = 5;
const int BUTTON3 = 6;
const int BUTTON4 = A2;
const int BUTTON5 = A3;
const int BUTTON6 = A4;

const int POTPIN1 = A0;
const int POTPIN2 = A1;

const int RADIO_CE = 7;
const int RADIO_CSN = 8;

int button = 0;
int POT1 = -1;
int POT2 = -1;
int send[] = {0,0,0};

//initiallize and name radio, address needs to be 6 digit and same on recieve and transmit
RF24 radio(RADIO_CE, RADIO_CSN);
const byte address[6] = "00001";

void setup() {
  // radio startup
  radio.begin();
  radio.openWritingPipe(address);
  radio.setPALevel(RF24_PA_HIGH);
  radio.stopListening();
  Wire.begin();
  // Set the pins for the potentiometers & buttons
  pinMode(BUTTON1, INPUT);
  pinMode(BUTTON2, INPUT);
  pinMode(BUTTON3, INPUT);
  pinMode(BUTTON4, INPUT);
  pinMode(BUTTON5, INPUT);
  pinMode(BUTTON6, INPUT);

  pinMode(POTPIN1, INPUT);
  pinMode(POTPIN2,INPUT);
}

void loop() {
  // read analog values from sliding pots
  POT1 = map(analogRead(POTPIN1),0,1023,0,255);
  POT2 = map(analogRead(POTPIN2),0,1023,0,255);

  //map sliding pots to colors for leds
  red = map(POT1, 0, 255, 7, 255);
  blue = map(POT1, 0, 255, 10, 0);

  red2 = map(POT2, 0, 255, 0, 255);
  green = map(POT2, 0, 255, 255, 0);

  //display color
  moodcolors[0] = rgb_color(0, red, blue);
  healthcolors[0] = rgb_color(green, red2, 0);

  moodLed.write(moodcolors, 1);
  healthLed.write(healthcolors, 1);
  // check which button is pressed to see what integer should be transmitted over radio
  if (digitalRead(BUTTON1) == LOW) {
    button = 1;
  }
  if (digitalRead(BUTTON2) == LOW) {
    button = 2;
  }
  if (digitalRead(BUTTON3) == LOW) {
    button = 3;
  }
  if (digitalRead(BUTTON4) == LOW) {
    button = 4;
  }
  if (digitalRead(BUTTON5) == LOW) {
    button = 5;
  }
  if (digitalRead(BUTTON6) == LOW) {
    button = 6;
  }
  //redefine arrey before sending based on inputs 
  send[0]=POT1;
  send[1]=POT2;
  send[2]=button;

  // transmit values
  radio.write(send, sizeof(send));
  //delay so time to update led correctly
  delay(50);
}

//PROJECT 3 GUAVAS: OUTPUT CODE
//Dunn Zhang, Andres Montemayor, Evette LaComb
//SHAUDE's mood box RECEIVE code - takes the radio signal from another box and displays the inputs through screen and lights  


//Pinmap:
/*  pin       mode         description      
    4         output       mood Led 
    3         output       health Led           
    SDA       output       LCD screen 
    SCL       output       LCD screen 
    7         CE           radio
    8         CSN          radio
    11        MOSI         radio
    12        MISO         radio
    13        SCK          radio

NRF24L01 pinmap can be seen below 
  radio pin    Arduino Uno/Nano pin    Arduino Micro pin
  VCC          3.3V                    3.3V
  GND          GND                     GND
  CE           7                       7
  CSN          8                       8
  MOSI         11                      MO
  MISO         12                      MI
  SCK          13                      SCK
 
 * modified from code by: Alton Olson, Vicky Zhou, Seema Kamath
 * Robert Zacharias 5-23

*/

//USE THIS CODE FOR RECIEVE
#include <SPI.h>
#include <nRF24L01.h>
#include <RF24.h>
#include <Wire.h>
#include <LiquidCrystal_I2C.h>
/// lights vvv 

#include <PololuLedStrip.h>
PololuLedStrip<4> moodLed;
PololuLedStrip<3> healthLed;

#define LED_COUNT 1
rgb_color moodcolors[1];
rgb_color healthcolors[1];

int red;
int red2;
int blue;
int green;
//lights ^^^

//Initialize the LCD library with the I2C address, number of columns and rows. call it lcd
LiquidCrystal_I2C lcd(0x27, 20, 4);

const int RADIO_CE_PIN = 7;
const int RADIO_CSN_PIN = 8;

int POT1;
int POT2;
int button;
int message = -1;

RF24 radio(RADIO_CE_PIN, RADIO_CSN_PIN);
const byte address[6] = "00001";

void setup() {
  Serial.begin(9600);

  radio.begin();
  radio.openReadingPipe(0, address);
  radio.setPALevel(RF24_PA_HIGH);
  radio.startListening();
  lcd.init();
  lcd.backlight();
}

void loop() {
  // get radio input from transmitter
  if (radio.available()) {
    int readVal[3];  
    radio.read(&readVal, sizeof(readVal));

    POT1 = readVal[0];
    POT2 = readVal[1];
    button = readVal[2];
    Serial.println(POT1);
    
    //depending on the button signal show message 
    if (readVal[2] == 5 && message != 5) {
      message = 5;
      lcd.clear();
      lcd.setCursor(0, 0);
      lcd.print("Can you help me");
      lcd.setCursor(0, 1);
      lcd.print("get up?");
    }
    if (readVal[2] == 2 && message != 2) {
      message = 2;
      lcd.clear();
      lcd.setCursor(0, 0);
      lcd.print("Can I have some");
      lcd.setCursor(0, 1);
      lcd.print("time to myself?");
    }
    if (readVal[2] == 3 && message != 3) {
      message = 3;
      lcd.clear();
      lcd.setCursor(0, 0);
      lcd.print("I want to get");
      lcd.setCursor(0, 1);
      lcd.print("outside today.");
    }
    if (readVal[2] == 6 && message != 6) {
      message = 6;
      lcd.clear();
      lcd.setCursor(0, 0);
      lcd.print("Could you get");
      lcd.setCursor(0, 1);
      lcd.print("me some water?");
    }
    if (readVal[2] == 4 && message != 4) {
      message = 4;
      lcd.clear();
      lcd.setCursor(0, 0);
      lcd.print("I can't wait");
      lcd.setCursor(0, 1);
      lcd.print("for breakfast!");
    }
    if (readVal[2] == 1 && message != 1) {
      message = 1;
      lcd.clear();
      lcd.setCursor(0, 0);
      lcd.print("Please be patient");
      lcd.setCursor(0, 1);
      lcd.print("with me today.");
    }
    //colored LEDs:
    int red = map(POT1, 0, 255, 7, 255);
    int blue = map(POT1, 0, 255, 10, 0);

    int red2 = map(POT2, 0, 255, 0, 255);
    int green = map(POT2, 0, 255, 255, 0);

    moodcolors[0] = rgb_color(0, red, blue);
    healthcolors[0] = rgb_color(green, red2, 0);

    moodLed.write(moodcolors, 1);
    healthLed.write(healthcolors, 1);

    delay(150);

  }
}