We are Team MARYa, (Mohan, Andrea, Roly, Yingyang) a group of four Carnegie Mellon University students taking an Introduction to Physical Computing course. For our final project, we were given the task of pairing up with an older person, Maria and creating a useful device for her that implements an Arduino and other physical computing elements. At our initial meeting, we discovered Maria had a habit of throwing papers on the floor instead of organizing them. Keeping this in mind, we created a prototype  for a shelf that would sort your papers for you. Be warned, our prototype is very different from our final product which is documented here. Due to some constraints, we had to take our project in a different direction.

What We Built:

This shelf is helping Maria to easily organise her documents in her home working space. Each section on the shelf has a category with a number and match to the button. The shelf section will be blink by pressing the button. It will be stopped blinking by receiving the document in the right shelf section.

Overall

Botton Layouts

Shelf Construction

OLYMPUS DIGITAL CAMERA

Maria is busy working in her home office. She wants to quickly organise her documents on her messy desk. After reading her latest bank invoice, she presses the button “3”, which identifies as “Finance” section. She saw the section 3 on the shelf starts to blink so she put the letter into the right section. All the action takes Maria less than 2 secs. Maria is so happy that she can quickly organise her document without thinking where is the “Finance” section. Maria starts to like to organise her documents and she can finally find her documents in the right category in her shelf!

How We Got Here:

As was previously mentioned, our project took a sudden turn near the end of the process. We realized that the mechanical aspects of our filing cabinet (including a small elevator to place papers in shelves) were proving far too finicky to finish on time.

Roly here is removing the old supports for the elevator mechanism after trying to solve mechanical issues.

Due to the nature of timing belts and how precise they need to be to produce the motion desired, the elevator was experiencing problems with keeping the belts tensioned enough. These belts were essential to the elevator’s movement since they provided the force that spun the lead screws, which in turn lifted or lowered the tray.

With this big change, the team needed to decide on an alternative option that kept the spirit of the idea. The first idea came almost right away. We wanted to help Maria make sure her papers got filed, but we opted to make it a device to help modify behavior rather than perform the task itself.

This was taken moments after we decided that we had to change our idea. Written on the table were some notes on the logistics of the new idea.

These LED’s became a central part of our new idea.

This is a sensor similar to the one we used to detect when paper was placed in a shelf. The clear bulb emits infrared light, and when an object bounces it back to the black bulb, the sensor knows how far the object is.

Our device was changed slightly to include LED lights on the side of the shelves. These lights would illuminate the shelf where Maria was supposed to put a piece of paper (she would first pick the filing category with a button). Inside of the shelf, some device would sense that she put the paper there and turn off the light. This version of the device was now more focused on reinforcing good habits of paper filing. By having Maria file her papers immediately, we were trying to break the habit of simply leaving it for later. But one problem was left: how to incentivize this.

This was our inspiration for an incentive: a gumball machine. Maria had wanted some sort of candy dispenser before, so we figured we could bring it back in a similar form to this.

Based on one of our first meetings with Maria, we decided to use candy as the incentive for filing. Maria had initially proposed that we make her an automatic candy dispenser, but we decided upon our current project since it seemed more useful to her. By bringing back the candy idea, we added a little inside humor between us and created a reward system for filing.

First test of the lighting and paper detection!

Once we got the shelf lighting and paper detection system working, we began designing the reward system. It was going to be a simple box on top of the shelf that contained a mechanism to dispense candy. Unfortunately, when we were going to laser cut the box, the USB that its file was on got corrupted and we lost it permanently.

In the end, we were only able to get the lighting and paper detection system working, and it was very finicky. We were slightly disappointed to the project turn out like this, but we simply did not have time to fix our ranging technical issues by the day of the presentation.

The final circuit board for the shelving unit. It was somewhat cramped, but it worked (mostly).

Throughout this process, we learned a lot about working on teams with vastly different skill sets. One of our members was an architect, one was a costume designer, and the other two were Information Systems students. We had to be very thorough in our meetings to ensure that this very widespread team was always on the same page. We also had to learn to keep frustrations in check, seeing as we were working hard and late up until the last minute.

Patience was key throughout this whole process. We were working very hard and had to make sure not to clash heads.

Overall, this project was a semi-success in that we were able to finish something for Maria. We would have been disappointed to not have anything at all to present to her or the class. However, through diligence and taking responsibility for our own sections of work, we pushed through to the end.

Conclusions and Lessons Learned:

Assemble process

From the problems we as a team encountered at final assblem stage, we learn a important lesson that for the problem we never worked before, there should be enough time set aside for it, different parts could be delivered late, for example the belt we had at last minute won’t fit the mechanism we already had. Also building a quick prototype from what we have might be a good idea to identify problems need to be resolved. If time constrain is a issue, different scenarios could be developed in advance.

Final Presentation feedback

We get a lot of useful feedback during crit that would help us identify problems and know better about outside opinion, overall people like the project. Although we change our idea at last minute, it is actually from the rewarding system idea we had long time ago. In terms of helping Maria on her habit, reminding system would potentially help her more on change her habbits, as she wrote in comment, it’s more a human/motivation issue than a mechanical issue, how we address it is very interesting, and how the client would use it is a very important post-evaluation part too. There are advice from students suggesting that visual reminder(blinking LED)might be substitued by audio which is more “annoying” and some people feel like bliking LED is not very comfortable to them, therefore this is a debating point which might need to discuss with our “client”. Right now the shelf is not painted and LED stripe is exposed, both need to be worked on in future.

Overall, our team learned a lot from this project. We worked closely with Maria trying to identify her needs in daily life that herself might even not aware of. We tried to start from the point of “organizing paper work” and develop a system/mechanism that would work to help her better on it. We stumbled on some issues and succeed on other issues, both are good lessons we learned.

Technical Details:

/*
 * Lighted Paper Shelf by Andrea Chung
 * 
 * When a button is pushed, its corresponding shelf section will
 * light up and blink until a photocell sensor reads that something 
 * has been inserted into that shelf.
 * 
 * The circuit:
 *  - 6 photocell sensors attached to pins A0-A5
 *  - 6 corresponding buttons attached to pins 2-7
 *  - LED strip attached to pin 8
 */

//photocell sensor pins
int sensor1 = A0; 
int sensor2 = A1;
int sensor3 = A2;
int sensor4 = A3;
int sensor5 = A4;
int sensor6 = A5;
//button pins
const int BUT1 = 2;
const int BUT2 = 3;
const int BUT3 = 4;
const int BUT4 = 5;
const int BUT5 = 6;
const int BUT6 = 7;
//filtering values for the sensor readings
float avgVal = 0;  // this variable, of type float (decimal), holds the filtered sensor value
const float PREV_WEIGHT = 0.8; // this variable, of type float (decimal), determines the previously filtered value's weight.
const float CUR_WEIGHT = 1 - PREV_WEIGHT; 
const float threshold = 5.0;
//timer setup for blinking LEDs
unsigned long previousMillis = 0;
const long interval = 1000;
//indicattors for light on or off
bool on1 = false;
bool on2 = false;
bool on3 = false;
bool on4 = false;
bool on5 = false;
bool on6 = false;

//LED strip setup
#include <PololuLedStrip.h>

PololuLedStrip<8> ledStrip; 
// set up how many LEDs on the strip you would like to control
#define LED_COUNT 48
rgb_color colors[LED_COUNT];

void setup() {
  //setup sensor pins as input
  pinMode(sensor1, INPUT);
  pinMode(sensor2, INPUT);
  pinMode(sensor3, INPUT);
  pinMode(sensor4, INPUT);
  pinMode(sensor5, INPUT);
  pinMode(sensor6, INPUT);
  //setup button pins as pullup 
  pinMode(BUT1, INPUT_PULLUP);
  pinMode(BUT2, INPUT_PULLUP);
  pinMode(BUT3, INPUT_PULLUP);
  pinMode(BUT4, INPUT_PULLUP);
  pinMode(BUT5, INPUT_PULLUP);
  pinMode(BUT6, INPUT_PULLUP);
  Serial.begin(9600); // starts serial communication at 9,600 baud (the rate)
}

void loop() {
  //read button values
  int but1Val = digitalRead(BUT1);
  int but2Val = digitalRead(BUT2);
  int but3Val = digitalRead(BUT3);
  int but4Val = digitalRead(BUT4);
  int but5Val = digitalRead(BUT5);
  int but6Val = digitalRead(BUT6);

  rgb_color color;

  //current time passed
  unsigned long currentMillis = millis();

  //if button 1 pressed set as on and read sensor value
  if (but1Val == LOW) {
    on1 = true;
    avgVal = (float)analogRead(sensor1);
    Serial.println("button1");
  }

  //if LED is set as on make light blink until the sensor
  //reading average changes suddenly
  if (on1 == true){
    //make led blink
    if (currentMillis - previousMillis >= interval) {
      // save the last time you blinked the LED
      previousMillis = currentMillis;
  
      // if the LED is off turn it on and vice-versa:
      if (color.red > 0) {
        color.red = 0;
        color.green = 0;
        color.blue = 0;
      } 
      else {
        color.red = 255;
        color.green = 0;
        color.blue = 0;
      }
      for (uint16_t i = 0; i < 8; i++)
      {
        colors[i] = color;
      }
      ledStrip.write(colors, LED_COUNT);
    }
    //equations taken from Phys Comp course website credit to Joseph Paetz
    float cur_reading = (float)analogRead(sensor1);
    avgVal = avgVal * PREV_WEIGHT + cur_reading * CUR_WEIGHT;
    Serial.println(cur_reading);
    //turn LED off and set as off if the reading changes over a set threshold
    if (abs(avgVal - cur_reading) > threshold){
      color.red = 0;
      color.green = 0;
      color.blue = 0;
      for (uint16_t i = 0; i < 8; i++)
      {
        colors[i] = color;
      }
      ledStrip.write(colors, LED_COUNT);
      on1 = false;
    }
  }

  //if button 2 pressed set as on and read sensor value
  if (but2Val == LOW) {
    on2 = true;
    avgVal = analogRead(sensor2);
    Serial.println("button2");
  }
  
  if (on2 == true){
    //make led blink
    if (currentMillis - previousMillis >= interval) {
      // save the last time you blinked the LED
      previousMillis = currentMillis;
  
      // if the LED is off turn it on and vice-versa:
      if (color.red == 255) {
        color.red = 0;
        color.green = 0;
        color.blue = 0;
      } 
      else {
        color.red = 255;
        color.green = 255;
        color.blue = 0;
      }
      for (uint16_t i = 8; i < 16; i++)
      {
        colors[i] = color;
      }
      ledStrip.write(colors, LED_COUNT);
    }
    float cur_reading = (float)analogRead(sensor2);
    avgVal = avgVal * PREV_WEIGHT + cur_reading * CUR_WEIGHT;
    Serial.println(cur_reading);
    if (abs(avgVal - cur_reading) > threshold){
      color.red = 0;
      color.green = 0;
      color.blue = 0;
      for (uint16_t i = 8; i < 16; i++)
      {
        colors[i] = color;
      }
      ledStrip.write(colors, LED_COUNT);
      on2 = false;
    }
  }

  //if button 3 pressed set as on and read sensor value
  if (but3Val == LOW) {
    on3 = true;
    avgVal = analogRead(sensor3);
    Serial.println("button3");
  }

  if (on3 == true){
    //make led blink
    if (currentMillis - previousMillis >= interval) {
      // save the last time you blinked the LED
      previousMillis = currentMillis;
  
      // if the LED is off turn it on and vice-versa:
      if (color.red == 255) {
        color.red = 0;
        color.green = 0;
        color.blue = 0;
      } 
      else {
        color.red = 255;
        color.green = 255;
        color.blue = 0;
      }
      for (uint16_t i = 16; i < 24; i++)
      {
        colors[i] = color;
      }
      ledStrip.write(colors, LED_COUNT);
    }
    float cur_reading = (float)analogRead(sensor3);
    avgVal = avgVal * PREV_WEIGHT + cur_reading * CUR_WEIGHT;
    Serial.println(cur_reading*10);
    if (abs(avgVal - cur_reading) > threshold){
      color.red = 0;
      color.green = 0;
      color.blue = 0;
      for (uint16_t i = 16; i < 24; i++)
      {
        colors[i] = color;
      }
      ledStrip.write(colors, LED_COUNT);
      on3 = false;
    }
  }

  //if button 4 pressed set as on and read sensor value
  if (but4Val == LOW) {
    on4 = true;
    avgVal = analogRead(sensor4);
    Serial.println("button4");
  }

  if (on4 == true){
    //make led blink
    if (currentMillis - previousMillis >= interval) {
      // save the last time you blinked the LED
      previousMillis = currentMillis;
  
      // if the LED is off turn it on and vice-versa:
      if (color.blue == 255) {
        color.red = 0;
        color.green = 0;
        color.blue = 0;
      } 
      else {
        color.red = 0;
        color.green = 0;
        color.blue = 255;
      }
      for (uint16_t i = 24; i < 32; i++)
      {
        colors[i] = color;
      }
      ledStrip.write(colors, LED_COUNT);
    }
    float cur_reading = (float)analogRead(sensor4);
    avgVal = avgVal * PREV_WEIGHT + cur_reading * CUR_WEIGHT;
    Serial.println(cur_reading);
    if (abs(avgVal - cur_reading) > threshold){
      color.red = 0;
      color.green = 0;
      color.blue = 0;
      for (uint16_t i = 24; i < 32; i++)
      {
        colors[i] = color;
      }
      ledStrip.write(colors, LED_COUNT);
      on4 = false;
    }
  }

  //if button 5 pressed set as on and read sensor value
  if (but5Val == LOW) {
    on5 = true;
    avgVal = analogRead(sensor5);
    Serial.println("button5");
  }

  if (on5 == true){
    //make led blink
    if (currentMillis - previousMillis >= interval) {
      // save the last time you blinked the LED
      previousMillis = currentMillis;
  
      // if the LED is off turn it on and vice-versa:
      if (color.blue == 255) {
        color.red = 0;
        color.green = 0;
        color.blue = 0;
      } 
      else {
        color.red = 0;
        color.green = 0;
        color.blue = 255;
      }
      for (uint16_t i = 32; i < 40; i++)
      {
        colors[i] = color;
      }
      ledStrip.write(colors, LED_COUNT);
    }
    float cur_reading = (float)analogRead(sensor5);
    avgVal = avgVal * PREV_WEIGHT + cur_reading * CUR_WEIGHT;
    Serial.println(cur_reading);
    if (abs(avgVal - cur_reading) > threshold){
      color.red = 0;
      color.green = 0;
      color.blue = 0;
      for (uint16_t i = 32; i < 40; i++)
      {
        colors[i] = color;
      }
      ledStrip.write(colors, LED_COUNT);
      on5 = false;
    }
  }

  //if button 6 pressed set as on and read sensor value
  if(but6Val == LOW) {
    on6 = true;
    avgVal = analogRead(sensor6);
    Serial.println("button6");
  }
  
  if (on6 == true){
    //make led blink
    if (currentMillis - previousMillis >= interval) {
      // save the last time you blinked the LED
      previousMillis = currentMillis;
  
      // if the LED is off turn it on and vice-versa:
      if (color.blue == 255) {
        color.red = 0;
        color.green = 0;
        color.blue = 0;
      } 
      else {
        color.red = 0;
        color.green = 0;
        color.blue = 255;
      }
      for (uint16_t i = 40; i < 48; i++)
      {
        colors[i] = color;
      }
      ledStrip.write(colors, LED_COUNT);
    }
    float cur_reading = (float)analogRead(sensor6);
    avgVal = avgVal * PREV_WEIGHT + cur_reading * CUR_WEIGHT;
    Serial.println(cur_reading);
    if (abs(avgVal - cur_reading) > threshold){
      color.red = 0;
      color.green = 0;
      color.blue = 0;
      for (uint16_t i = 40; i < 48  ; i++)
      {
        colors[i] = color;
      }
      ledStrip.write(colors, LED_COUNT);
      on6 = false;
    }
  }
  else{
    color.red = 0;
      color.green = 0;
      color.blue = 0;
      for (uint16_t i = 40; i < 48  ; i++)
      {
        colors[i] = color;
      }
      ledStrip.write(colors, LED_COUNT);
  }
}

Schematic

For the final project of Introduction to Physical Computing, The Weavers, which consists of Jenny, Megan, and Ghalya, were assigned to create an assistive device for Rebecca Herbert. Rebecca loves to weave and the most tedious task in weaving is winding the yarn into a ball from its original loose packaging. Not only does this tedious hurt her wrist, but she also has to estimate the amount of yarn that she has. This documents the process of creating a motorized ball winder that would save Rebecca from doing this task manually.

Link to Prototype Documentation: https://courses.ideate.cmu.edu/60-223/f2018/work/team-weavers-prototype-documentation/

Link to Initial Interview: https://courses.ideate.cmu.edu/60-223/f2018/work/interview-documentation/

What We Built

Details of Ball Holder

The Automatic Ball Winder is composed of three main components. First, the screen and buttons allow the user to interact with the device and input yards of yarn needed as well as emergency stop, reset, and continue buttons that let the user change his/her mind. Next is a distance encoder. The user feeds the yarn through metal feeder loops and around a disk. The disk measures distance by counting the number of rotations while the metal feeder loops keep the yarn at the right tension and height so it doesn’t fall off. When the user presses “start”, the motor that is attached to the yarn holder will spin, spinning both the encoder as well as winding yarn onto the holder. If the yarn falls off or encounters a knot, the device will automatically shut off to reduce error.

Ball Winder in Action with Yarn Umbrella.

Imagine Rebecca wants to start a new weaving project, but her hands are hurting too much to use the manual ballwinder, and since she’s working with silk she can’t afford to ball up too much more than what is necessary. She goes to her Automatic Ballwinder, and presses the “reset” button, which makes it say “Welcome Rebecca”, and “Yarn to wind:”. She turns the knob until she gets to the desired amount to wind, presses start, and goes to make herself some tea.

After about 15 minutes, she comes back and sees that the ballwinder is done, and she screws off the top so she can use the yarn there. She still needs some more for another project, so she prepares another skein and heads off to start her project.

Threading Yarn Through Distance Encoder

Putting String onto Ball Holder

Unscrewing Ball Holder

Ball Holder Unscrewed

Details of Distance Encoder and Buttons

Wire Thread Feeders

How We Got Here

During the initial meeting with Rebecca, it was clear that she loved to weave. She vocalized how strongly she disliked the ball winding process and how commercial ball winders were upwards of 300 dollars. Since that meeting, Team Weavers decided the product that would best fit her lifestyle would be an automatic ball winder that would tell her distance of yarn wound and save her wrists from carpal tunnel.

Initially, the main thing that we focused on was the criss-cross yarn pattern that the handheld ballwinder created. With the winding method of the motor, the pattern was mostly parallel. After a few days, we decided and Rebecca assured us that the yarn pattern was not as important as the distance encoding and the automatic function of the winding. At the end of the project, we found that consistent tension in the yarn and speed without erratic stop and start would create a decent yarn pattern that would not entangle.

Manual Ball WInder Yarn Pattern

Prototype Yarn Pattern

Final Yarn Pattern: Consistent Tension and Speed

To get an accurate distance reading for the yarn, we created a disk that the yarn wrapped around. When the motor for the yarn holder spun, it would pull on the yarn around the disk and turn the encoding disk. Counting the number of turns would give an accurate reading of yarn wound on the ball. Some of the problems we encountered with the encoder were that the yarn kept falling out of the disk, making the reading inaccurate. The ball winder would spin and pull yarn onto the ball, but the encoder would not spin. This was a big problem because not only would the device not know that the yarn was being added and therefore stop, but it would deceive Rebecca on how much yarn was wound onto the ball.

Testing the Length of String After Finished.

To deal with this problem, we turned to both software and the physical form of the device. In terms of software, we decided that if the motor was running and the encoder was not, then the yarn had fallen off and to switch to an “emergency stop” mode. During this mode, the user can read how much yarn is currently wound as well as continue the winding process or restart. One of the physical changes was to make the shape of the encoder have a larger lip and therefore less chance to fall out. The second preventative physical measure was to create wire guides that served as a way to feed the yarn at the right height as well as create the right pull tension so that the yarn would not fall out. Also, the wire guides instead of the prototype wood guards make it easy to remove the yarn without being at the end of the string.

Initial Prototype of Distance Encoder

Second Iteration of Distance Encoder: Lowered yarn holders to prevent yarn from slipping out.

Final Distance Encoder: Metal wires allow for easy usage and larger gap in the disk prevents slipping.

Finally, instead of an overall level box for the shape of the device, we decided to create levels. After much testing, we found that having the encoder be at an elevated surface that feeds the string directly parallel to the yarn winder would increase the tension in the string and encourage the string not to be found on the bottom of the yarn holder or fall off the track.

Original Top for the device.

Second Iteration of Box

Moving from the prototype to the finished device had some aesthetic and physical challenges as well. We ended up choosing white acrylic for the overall form of the box and white plastic to create the yarn holder to mimic the original feeling of the handheld ball winder. Also, the screen updates the user as to how many yards have been wound, if the emergency stop was triggered, and if the winding has been completed.

Original Wood Prototype

Prototype Ball Winder with Ball

Megan Working on Soldering the Final Board

We spent awhile soldering the components in, to make the project more permanent and portable; in addition, we glued the box shut for easy portability.

Digital Sketch of Final Device Structure

For the above design, we went with white acrylic to make it seem more attractive and presentable, and we used the teeth to make our box easier to assemble and sturdier. We added some space for buttons, and a port in the back for Rebecca to plug in her winder.

Making of Acrylic Box

The laser cutter did not fully cut through the acrylic, so we had to try and rip a lot of pieces out- though it was difficult, we wound up getting most to come out and by using only a little extra acrylic and a filer, we were able to get the box made.

Welcome Screen

Winding Screen

We created a few “states” for the screen- two are shown above- one for the intro, where we welcome Rebecca, one where we ask how much she wants wound, one that shows how much has been wound, one if our winder emergency stops, and one for when it finishes.

Conclusions and Lessons Learned

The most rewarding thing about this project was the final critique. When we showed the project to Rebecca, something that she has been wanting for quite some time, she was so excited to bring the device home to use. The biggest motivator to making sure this device was working flawlessly was knowing that Rebecca would be using this daily and it would make such a positive impact in her life. Even during the creation process, many people who also had interests in fabrics and weaving reaffirmed the benefit of this device to us. We were surprised how much this device meant to her/other weavers and it was satisfying to create a project that had a purpose.

We also gained some important insights from others during the final critique process. One person suggested to “[have] a flap to keep the yarn on the disk” which would make it hard for the string to fall out as well. To fix this problem, Rebecca suggested we use a piece of tape would be an easy way to incorporate this idea. Another suggestion was to have an “adjustable speed, so customers can change the tightness themselves”. This would be an interesting component to add in case different types of yarn break under different pressures. Thus, having an adjustable speed or tension setting would account for more types of fibers.

Had we had more time to perfect the device, we would have tested the length of string over 600 yards. Even though Rebecca typically uses between 250-450 yards of string, we want to make sure that no matter how long the string is that the reading is accurate. During this entire process, one of the most educational aspects was realizing to test components as we go. After having the device begin to smoke up after plugging in power to our initial final product with a sultered final board, we had to go back to our board and take apart each component to isolate the problem. We found that testing each piece as we went was a much easier way to deal with this problem in the future.

Final Presentation with Rebecca.

Technical details

Code

/*
Automatic Ball Winder

Jenny Han
Ghalya Alsanea
Megan Roche

The following code uses an Arduino, stepper moter, two rotary encoders, buttons
and LCD Display.

It maintains four states: Welcome, Winding, Emergency Stop, and Finished.
Welcome allows users to input the amount of yarn that the user needs.
Winding starts the motor to collect the yarn into a ball as well as 
counting amountof yarn wound.
Emergency Stop detects if the yarn fell off and displays how much 
yarn was wound at that instant.
Finished stops the motor when the distance wanted is achieved.

*/

#include <Encoder.h>
#include <AccelStepper.h>
#include <Wire.h>
#include <LiquidCrystal_I2C.h>

LiquidCrystal_I2C lcd(0x27, 16, 2);
unsigned long timer = 0;
long int stepsToDo = 0;

//output rotary encoder: distance measuring  
const int O_CLK = 2;
const int O_DT = 3;
const int O_SW = 4;

//input rotary encoder: distance input 
const int I_CLK = 5;
const int I_DT = 6;
const int I_SW = 7;

//buttons for emergency stop and continue
const int ESTOP = 11;
const int CONTB = 12;

long val;

//current state that the device is in
bool stateWelcome = false;
bool stateWind = false;
bool stateStop = false;
bool stateFinished = false;
bool winding = false;

//variables to keep track of yarn wound
long oldPosition  = -999;
long OLDLENGTH = -1;
long NEWLENGTH;
int CURRENTD;
long TOTALD = 0;
long int finalLen = 0;


//stepper motor pins
const int STEP_PIN = 8;
const int DIR_PIN = 9;
AccelStepper winderMotor(1, STEP_PIN, DIR_PIN);

//initialize encoders
Encoder encLength(O_CLK, O_DT);
Encoder encInput(I_CLK, I_DT);

void setup() {
  lcd.init();
  //Serial.begin(9600);
  lcd.backlight();
  lcd.setCursor(0, 0);
  lcd.print("Welcome Rebecca!");
  delay(4000);
  lcd.clear();
  pinMode(O_SW, INPUT);
  pinMode(I_SW, INPUT);
  winderMotor.setMaxSpeed(1000);
  winderMotor.setSpeed(400);

  pinMode(ESTOP, INPUT_PULLUP);
  pinMode(CONTB, INPUT_PULLUP);

  stateWelcome = true;
}

void loop() {
  //run welcome function
  if (stateWelcome) {
    stateWelcomeFunction();
  }
  //run winding 
  if (stateWind) {
    stateWindFunction();
  }
  //run emergency stop
  if (stateStop) {
    stateStopFunction();

  }
  //run finished screen
  if (stateFinished) {
    stateFinishedFunction();
  }
}

//takes in distance desired
void stateWelcomeFunction() {
  long newPosition = encInput.read();
  if (val < 0) {
    val = 0;
  }
  //starts at 350, average yarn length needed
  if (newPosition != oldPosition) {
    oldPosition = newPosition;
    val = 350 - 5* (newPosition/4);
    lcd.clear();
    lcd.leftToRight();
    lcd.setCursor(0, 0);
    lcd.print("Yards needed:"); 
    lcd.setCursor(0, 1);
    lcd.print(val);
  
  }
  //cannot go negative
  if (!digitalRead(I_SW)) {
    TOTALD = val;
    lcd.clear();
    lcd.leftToRight();
    lcd.setCursor(0, 0);
    lcd.print("Winding ");
    lcd.setCursor(0, 1);
    lcd.print(TOTALD);
    lcd.print(" Yards.");
    delay(2000);
    stateWelcome = false;
    stateWind = true;
  }
}

//starts winding motor and counts distance
//checks if yarn fell off
void stateWindFunction() {
  int rotNeeded = TOTALD * 6;
  int oldD = rotNeeded / 6;

  while (rotNeeded > 0) {
    //if stop pressed then go to state stop function
    bool stateStopButton = digitalRead(ESTOP);
    if (stateStopButton == 0) {
      stateWind = false;
      stateStop = true;
      break;
    }
    //every four seconds check if string fell off
    if (millis() - timer >= 4000 ) {  
      // this happens once per 4 seconds
      NEWLENGTH = encLength.read();
      if ((NEWLENGTH == OLDLENGTH) && (NEWLENGTH != 0)) {
        stateWind = false;
        stateStop = true;
        break;
      }
      timer = millis();
    }
    NEWLENGTH = encLength.read();
    
    //calls for motor to run
    for (int i = 0; i < 700; i++) {
      winderMotor.runSpeed();
    }
    //update LCD screen
    if ((NEWLENGTH != OLDLENGTH)) { 
      OLDLENGTH = NEWLENGTH;
      rotNeeded = (TOTALD * 6) - floor(abs(NEWLENGTH / 800));
      CURRENTD = rotNeeded / 6;
      if (CURRENTD != oldD) {
        oldD = CURRENTD;
        lcd.clear();
        lcd.setCursor(0, 0);
        lcd.print("TO GO : ");
        lcd.setCursor(0, 1);
        lcd.print(CURRENTD );
        lcd.print(" yards");
      }
      if (rotNeeded == 0) {
        stateWind = false;
        stateFinished = true;
      }
    }
  }

}

//finished state, restart button is the reset pin
void stateFinishedFunction() {
  //if button press
  lcd.clear();
  lcd.setCursor(0, 0);
  lcd.print("Finished!");
  lcd.setCursor(0, 1);
  lcd.print("Wound ");
  lcd.print(TOTALD);
  lcd.print("yds");

}

//emergecy stop state, restart button is the reset pin
void stateStopFunction() {
  lcd.clear();
  lcd.setCursor(0, 0);
  lcd.print("STOPPED!");
  lcd.setCursor(0, 1);
  lcd.print("Wound ");
  lcd.print(TOTALD - CURRENTD - 1);
  lcd.print("yds");
  int contbutton = digitalRead(CONTB);
  //continue
  if (contbutton == 0) {
    delay(100);
    stateStop = false;
    stateWind = true;
    stateFinished=false;
    stateWelcome = false;

  }
}

Schematic and design files

Design Files

For this project, our group of three were tasked with creating a device that would help make the life of our older friend, Jim (as he prefers to go by), better. Though we’re limited by what we’ve learned so far, we were excited to face this task on and to help make something that would survive past the end of this class that would benefit someone else. From our initial first meeting, it is clear to see that the device that would benefit Jim the most would be a device to make sure he doesn’t forget what he need when he leaves the house. We also have another post highlighting what occurred during the first meeting here as well as this post of where our prototype was at about three weeks ago.

WHAT WE BUILT

Description

Our project is a box that is to be installed by Jim’s front door that contains a screen that will display Jim’s items of the day and a motion sensor that triggers the device when he approaches the door. The Google API parses through Jim’s Google Calendar to identify a list of common events or take a list of items from the description of non-common events to display in a list format on his door. Whenever he walks approximately 3 meters by the door, the motion sensor will trip, turning on the display that will remind him to make sure he has all the items he needs for the day.

Final Photographs

Overall Content Photo

Standard opening screen

Barrel jack power and ground connection at the base of the box with motion sensor clearly visible

Temporary velcro closing mechanism

Items list screen(1/3)

Keyboard screen (2/3)

Goodbye screen (1/3)

Narrated Sketch of the Intended Use

Every morning, Jim wakes up to a busy day filled with community activities and long board meetings. Those days often require Jim to bring with him many important supplies, without which his day is not as productive. Often, Jim makes note of the things he needs in his Google Calendar, but that is often in a place that is not as easily accessible to him when he is rushing out the door. Imagine this scenario: Jim is walking out the door in the morning, hoping he remembers everything he needs for the day. Just then, a motion detector by the door detects his presence, and, in response, an LED touchscreen lights up, asking Jim if he has everything he needs for the day based on the entries in his calendar. If the answer is yes, then Jim touches the on-screen button to indicate as such, and off he goes, day’s supplies in tow. This is Jim’s daily reminder box helping make his life easier and more productive, one notification at a time.

HOW WE GOT HERE

Our original idea was to have the ESP connect to a server that parses data from Google API to interpret Jim’s calendar for the day. Then the items for those events would be communicated back to the ESP to be displayed in the form of a checklist on the TFT Display as a reminder when he leaves the house.

Sketch of one of the earliest planning diagrams when the Arduino was incorporated

Decided not to set up server. We ended up getting recommended to avoid setting up a server, however, because it would have been really difficult and cost money to maintain. Instead we redesigned the system to parse all data locally and on-premise, which had the trade-off of taking up significantly more memory on the ESP32, out IoT chip.

Ordering a second TFT Display. The next part of the process was to adapt the Adafruit ILI9341 library to do exactly what we needed on the touchscreen display. After the prototype we ordered a slightly bigger screen based off the request of Jim. This served to be useful in the long run since it allowed two people to work n the screens simultaneously and independently. Catherine’s biggest problem was getting the touch to respond properly either due to wearing down the screen or something wrong with the smaller screen.

Attempt#1 for coordinates and basic code with display

While she tinkered with the bigger screen, Lexi experimented with displaying icons to improve visibility of Jim’s items.

Updating the screen based on input from the Serial Monitor

Initially and for demo purposes, we set up the touchscreen to behave like the Google API by pulling data from the Serial monitor. When the screen would sense something written into the Serial monitor, it would perform a Serial.read() to pull that data and print it on the touchscreen.

Serial and display communicating

Getting a keyboard on the display through the ESP. We discovered that there were many problems implementing touch onto the TFT display when connected to the ESP.

Tft display with touchscreen keyboard

We needed to have the keyboard so that Jim could input the verification required to access his wifi once he moved houses. This is because the ESP will not be able to communicate with anything else in the system until it is connected to the WiFi. Ultimately we decided that it’d be better to prioritize the touchscreen keyboard and connected the display to the Arduino which would in turn communicate with the ESP through the RX and TX pins. This also resulted in the scrapping of the icons due the example pictureEmbeded only working with an ESP.

Icon conversion issues. The first problem with the icon displaying was that it took Lexi a week to realize the example code, Adafruit ILI9341’s pictureEmbeded, wasn’t designed to work on Arduino Unos, which she had been testing with. This code doesn’t work with an Arduino, however, so we had to scrap it. Even before we scrapped it though, we were running into the issue that the image converter Lexi was using to the transfer the pixel measurements of the icon into hex codes kept returning values of at least 15,500 values. uint16_t is able to store at most 14,000 values and experimenting with image sizing and various settings would either increase the number of values or have it stay the same. We ended up ditching the entire idea before she had to attempt to find another image converting method.

Image converted into hex codes

Router to connect ESP to Wifi. It would have been useful to discover earlier that a router could be set up to bypass the issue of the ESP connecting to CMU-SECURE. This is due to the way the ESP handles the wifi verification. Since it is much harder to connect to WEP-Enterprise networks, testing anything on campus was impossible. It would have been nice to realize that a router was an option a few days before we did. At that point, there had already been a few days of Catherine testing various bits of code that Sana wrote and implementing at Catherine’s house due to Sana using the school wifi at home.

Memory Problems with the ESP. The ESP32, despite being a chip that can access the internet, holds a very small amount of memory. Accessing the Google Calendar API returned an extremely long string of characters, which was large enough to cause overflow in memory. As a result, the ESP32 turned out being the wrong device for us to use for this project.

Soldering all the wires together to make the connections inside the box more durable

Testing the power and ground side of the DC Jack using the multimeter

Velcro-ing the box closed. Originally we were thinking about gluing the box closed with everything inside it.

The box post-soldering pre-velcroing

Due to the impermanent nature of our box at the time of the demo, we decided to velcro the box closed. This was because velcro is the easiest way of closing the box without prior planning and worked pretty well for what we needed. We realized by doing this that it could actually benefit the box for it to have the option of opening in case anything goes wrong/something gets jostled when Jim moves between houses. As a result, we decided to add latches and a hinge to attach to the final one but didn’t manage to get them in time for documentation.

Velcro-sealed box

CONCLUSIONS

The final crit proved to be a very helpful time for us to learn about good features to add to the project. Overall response to the box itself was very positive- both from older friends and from peers. Overall, it seemed as if both groups were very willing to use the box in daily life. Some feedback was that it may have been better to change the design of the box so that it draws a little more attention to itself- like a coat of varnish and maybe a blinking LED light and some sound (these were all suggestions that came from 3 separate people). While we did what our client wanted (these were all ideas we discussed with him), they are certainly ideas we would have implemented.

Additionally, the idea of the demo in its simple form- without the Google calendar implementation- also garnered a good amount of support. It appeared to be a good solution for those who didn’t use a google calendar, though of course there was also the opposing side of the same argument- that the keyboard was too small and the Google Calendar would have worked much better. This reasoning, however, often came from the general sentiment that the screen itself was too small. We’d already bought the largest, reasonably priced screen that we could find, and going any bigger would have given us marginal improvements of size with dramatic price increases. By the end of this discussion, there was a mention that this product would have been great on an iPad, which then begged the question- why not mount an iPad by the door instead? Why this project? To us, this is because an iPad is a one-size-fits-all, expensive solution, whereas our box is a this-size-fits-Jim, more cost-effective solution.

Overall, this project is a quite simple idea, but, under the hood, the complexities of this project were so great that factors like time, WiFi access, and constraints in memory played a big role in making this project much harder to realize. While we were able to put together a final product that satisfied the needs of our client, we would have been much happier with a product that would have been able to access WiFi, a circumstance we couldn’t control. Testing and overall product development were severely impedimented by this, despite the fact that most of the code was written and only needed to be tested. Everything is currently in progress and will soon be updated to meet our standards of functionality, as well as Jim’s. Looking forward, one thing to change would be to ensure that the WiFi chip we are using has much more memory and is able to effectively access the internet from where it is being tested, and to pre-check that the devices being used are always compatible. Compatibility played a large role in the effectiveness of this project. Because IoT projects require hardware, connectivity, and software to work together seamlessly, the presence of many roadblocks in this path made a final product much more difficult to develop with the resources on hand. Still, the learning curve, though steep, was important and taught us a lot about the importance of developing products that are viable and useful. With a larger timeframe, however, we are fully confident that we can produce a fully functioning reminder box for Jim by the time spring semester rolls around.

TECHNICAL DETAILS

Code

The following code is a completely locally-hosted, on-premise code block operating only with the Arduino Uno R3, which takes entries based on the next day to add to the calendar:

/* this code is written for the Arduino component of the reminder box, and is the demo code, 
 * which allows the user to manually enter information about the things he/she needs for the 
 * next day. The reminder box pulls information on the day of the week (set based on an example 
 * schedule here) and prompts the user (whenever the motion sensor is tripped) to check off 
 * whether they have everything or if they want to add anything to the list of things they need
 * for tomorrow.
 */

//libraries
#include <Adafruit_GFX.h>
#include <SPI.h>
#include <Wire.h>
#include <Adafruit_ILI9341.h>
#include <TouchScreen.h>
#include <stdint.h>
#include <DS3231.h>

//Touchscreen pins
#define YP A2
#define XM A3
#define YM 8
#define XP 9

//Calibration data
#define TS_MINX 150
#define TS_MINY 120
#define TS_MAXX 920
#define TS_MAXY 940

#define MINPRESSURE 10
#define MAXPRESSURE 1000

#define IsWithin(x, a, b) ((x>=a)&&(x<=b))

TouchScreen ts = TouchScreen(XP, YP, XM, YM, 600);

#define TFT_CS 10
#define TFT_DC 9
Adafruit_ILI9341 tft = Adafruit_ILI9341(TFT_CS, TFT_DC);

DS3231 rtc(SDA, SCL);

String dailyStuff[7] = {"\n agenda\n coat\n umbrella",
                       "\n wallet\n documents",
                       "\n gym bag\n dog treats",
                       "\n files\n pen drive",
                       "\n cell phone\n passport", 
                       "\n car keys\n bags",
                       "\n art supplies\n laptop charger"};

// define variables for if he has everything and the new and old strings
bool hasEverything = false;
String old = "";
String updated = "";
String printed = updated;
String add = "";
String tomorrowString = "";
String tb = "";
int day;

//define LED and motion sensor pins
const int MOTION_SENSOR = 3;
const int LED = 4;
// set boolean for startup
bool remind = false;

int index = 0;

int bx;
int by;

bool scrollUp;
bool scrollDown;
bool adding;

const char textLimit = 25;
char MyBuffer[textLimit];

#define FRAME_X 210
#define FRAME_Y 180
#define FRAME_W 50
#define FRAME_H 40

#define GREENBUTTON_X (FRAME_X + FRAME_W)
#define GREENBUTTON_Y FRAME_Y
#define GREENBUTTON_W FRAME_W
#define GREENBUTTON_H FRAME_H

void drawFrame() {
  tft.drawRect(GREENBUTTON_X, GREENBUTTON_Y + 7, FRAME_W, FRAME_H, ILI9341_BLACK);
}

void addMode() {
  keyboard();
  while (adding) {
    GetKeyPress(MyBuffer);
  }
  yesBtn();
}

// updates the string needed for the day based on the time
void dayUpdate() {
  if (rtc.getTime().hour == 23 && rtc.getTime().min == 58) {
    tb = tomorrowString;
    day = rtc.getTime().dow;
    updated = dailyStuff[day] + tb;
    tomorrowString = "";
  }
}

void checkScroll() {
  TSPoint p = ts.getPoint();
  bx = map(p.y, TS_MINX, TS_MAXX, 0, tft.width());
  by = map(p.x, TS_MINY, TS_MAXY, tft.height(),0);
  if ((260 < bx  && bx < 300) && (65 < by && by < 100)) {
    scrollUp = true;
  }
  else if ((260 < bx && bx < 300) && (100 < by && by < 135)) {
    scrollDown = true;
  }
  if (scrollDown) {
    index = updated.indexOf("\n ");
    if (index == -1) {
      printed = updated.substring(index + 2);
      tft.println(printed);
    }
    scrollDown = false;
  }

  if (scrollUp) {
    index = updated.indexOf(printed);
    if (index != 0) {
      printed = updated.lastIndexOf("\n ", index);
      tft.println(printed);
    }
    scrollUp = false;
  }
}

//arrays for keyboard
const char Mobile_KB[3][13] PROGMEM = {
  {0, 13, 10, 'Q', 'W', 'E', 'R', 'T', 'Y', 'U', 'I', 'O', 'P'},
  {1, 12, 9, 'A', 'S', 'D', 'F', 'G', 'H', 'J', 'K', 'L'},
  {3, 10, 7, 'Z', 'X', 'C', 'V', 'B', 'N', 'M'},
};

const char Mobile_NumKeys[3][13] PROGMEM = {
  {0, 13, 10, '1', '2', '3', '4', '5', '6', '7', '8', '9', '0'},
  {0, 13, 10, '-', '/', ':', ';', '(', ')', '$', '&', '@', '"'},
  {5, 8, 5, '.', '\,', '?', '!', '\''}
};

const char Mobile_SymKeys[3][13] PROGMEM = {
  {0, 13, 10, '[', ']', '{', '}', '#', '%', '^', '*', '+', '='},
  {4, 9, 6, '_', '\\', '|', '~', '<', '>'}, //4
  {5, 8, 5, '.', '\,', '?', '!', '\''}
};

//function that actually makes the keyboard look like a keyboard (has buttons and prints letters)
void MakeKB_Button(const char type[][13])
{
  tft.setTextSize(2);
  tft.setTextColor(ILI9341_WHITE);
  for (int y = 0; y < 3; y++)
  {
    int ShiftRight = 15 * pgm_read_byte(&(type[y][0]));
    for (int x = 3; x < 13; x++)
    {
      if (x >= pgm_read_byte(&(type[y][1]))) break;

      drawButton(15 + (30 * (x - 3)) + ShiftRight, 100 + (30 * y), 20, 25); // this will draw the button on the screen by so many pixels
      tft.setCursor(20 + (30 * (x - 3)) + ShiftRight, 105 + (30 * y));
      tft.print(char(pgm_read_byte(&(type[y][x]))));
    }
  }
  //ShiftKey
  drawButton(15, 160, 35, 25);
  tft.setCursor(27, 168);
  tft.print('^');

  //Special Characters
  drawButton(15, 190, 35, 25);
  tft.setCursor(21, 195);
  tft.print(F("SP"));

  //BackSpace
  drawButton(270, 160, 35, 25);
  tft.setCursor(276, 165);
  tft.print(F("BS"));

  //Return
  drawButton(270, 190, 35, 25);
  tft.setCursor(276, 195);
  tft.print(F("RT"));

  //Spacebar
  drawButton(60, 190, 200, 25);
  tft.setCursor(105, 195);
  tft.print(F("SPACE BAR"));
}

//function that draws buttons
void drawButton(int x, int y, int w, int h)
{
  // grey
  tft.fillRoundRect(x - 3, y + 3, w, h, 3, ILI9341_LIGHTGREY); //Button Shading

  // white
  tft.fillRoundRect(x, y, w, h, 3, ILI9341_WHITE);// outter button color

  //red
  tft.fillRoundRect(x + 1, y + 1, w - 1 * 2, h - 1 * 2, 3, ILI9341_RED); //inner button color
}

void GetKeyPress(char * textBuffer)
{
  static bool shift = false, special = false, back = false, lastSp = false, lastSh = false;
  static int bufIndex = 0;
  TSPoint p = ts.getPoint();
  Serial.print("bx = "); Serial.print(bx); Serial.print(" by = "); Serial.println(by);
  Serial.println(p.z);
  Serial.println("we're here");
  if (p.z >= 0)
  {
    //ShiftKey
    if (TouchButton(-10, 160, 30, 25))
    {
      shift = !shift;
      delay(200);
    }

    //Special Characters
    if (TouchButton(-10, 190, 30, 25))
    {
      special = !special;
      delay(200);
    }

    if (special != lastSp || shift != lastSh) //Start of changing the keys and make it look different
    {
      if (special)
      {
        if (shift)
        {
          tft.fillScreen(ILI9341_BLUE);
          MakeKB_Button(Mobile_SymKeys);
        }
        else
        {
          tft.fillScreen(ILI9341_BLUE);
          MakeKB_Button(Mobile_NumKeys);
        }
      }
      else
      {
        tft.fillScreen(ILI9341_BLUE);
        MakeKB_Button(Mobile_KB);
        tft.setTextColor(0xffff, 0xf800);
      }

      if (special)
        tft.setTextColor(0x0FF0, 0xf800);
      else
        tft.setTextColor(0xFFFF, 0xf800);

      tft.setCursor(21, 195);
      tft.print(F("SP"));

      if (shift)
        tft.setTextColor(0x0FF0, 0xf800);
      else
        tft.setTextColor(0xffff, 0xf800);

      tft.setCursor(27, 168);
      tft.print('^');

      lastSh = shift;
      lastSp = special;
      lastSh = shift;
    }//End of changing the keys and make it look different

    for (int y = 0; y < 3; y++)
    {
      int ShiftRight;
      if (special)
      {
        if (shift) //special+shift = symbol
          ShiftRight = 15 * pgm_read_byte(&(Mobile_SymKeys[y][0]));
        else //special only = number
          ShiftRight = 15 * pgm_read_byte(&(Mobile_NumKeys[y][0]));
      }
      else //alphabet
        ShiftRight = 15 * pgm_read_byte(&(Mobile_KB[y][0]));

      for (int x = 3; x < 13; x++)
      {
        if (x >=  (special ? (shift ? pgm_read_byte(&(Mobile_SymKeys[y][1])) : pgm_read_byte(&(Mobile_NumKeys[y][1]))) : pgm_read_byte(&(Mobile_KB[y][1])) )) break;

        if (TouchButton((33 * (x - 3)) + ShiftRight - 14, 100 + (30 * y), 20, 25)) // this will draw the button on the screen by so many pixels
        {
          if (bufIndex < (textLimit - 1))
          {
            delay(200);

            if (special)
            {
              if (shift)
                textBuffer[bufIndex] = pgm_read_byte(&(Mobile_SymKeys[y][x]));
              else
                textBuffer[bufIndex] = pgm_read_byte(&(Mobile_NumKeys[y][x]));
            }
            else
              textBuffer[bufIndex] = (pgm_read_byte(&(Mobile_KB[y][x])) + (shift ? 0 : ('a' - 'A')));

            bufIndex++;
          }
          break;
        }
      }
    }

    //Spacebar
    if (TouchButton(35, 190, 200, 25))
    {
      textBuffer[bufIndex++] = ' ';
      delay(200);
    }

    //BackSpace
    if (TouchButton(268, 160, 30, 25))
    {
      if ((bufIndex) > 0)
        bufIndex--;
      textBuffer[bufIndex] = 0;
      tft.setTextColor(0, ILI9341_BLUE);
      tft.setCursor(15, 80);
      tft.print(F("                          "));
      delay(200);
    }

    //Return
    if (TouchButton(268, 190, 30, 25))
    {
      tomorrowString = tomorrowString + textBuffer + "\n ";
      Serial.println(textBuffer);
      Serial.println("we got here too");
      adding = false;
      while (bufIndex > 0)
      {
        bufIndex--;
        textBuffer[bufIndex] = 0;
      }

      tft.setTextColor(0, ILI9341_BLUE);
      tft.setCursor(15, 80);
      tft.print(F("                         "));
      Serial.println(tomorrowString);
      yesBtn();
    }
  }
  tft.setTextColor(0xffff, 0xf800);
  tft.setCursor(15, 80);
  tft.print(textBuffer);
}

//Makes you know if the point that you touched is in the range computed by x,y,w,h
byte TouchButton(int x, int y, int w, int h)
{
  int X, Y;
  // Retrieve a point
  TSPoint p = ts.getPoint();
  Y = map(p.x, TS_MINY, TS_MAXY, tft.height(), 0);
  X = map(p.y, TS_MINX, TS_MAXX, 0, tft.width());

  return (IsWithin(X, x, x + w) & IsWithin(Y, y, y + h));
}

void keyboard() {
  tft.fillScreen(ILI9341_BLUE);
  MakeKB_Button(Mobile_KB);
  GetKeyPress(MyBuffer);
}

void yesBtn()
{
  tft.fillScreen(ILI9341_BLUE);
  //Asking Jim text
  tft.setCursor(10, 10);
  tft.setTextColor(ILI9341_WHITE);
  tft.setTextSize(2);
  tft.println("Hey Jim! Do you have      what you need for today?");
  //Printing item list
  tft.setTextSize(2);
  tft.println(updated);
  //Scroll buttons
  tft.fillRect(GREENBUTTON_X + 6, 60, GREENBUTTON_W - 12, 70, ILI9341_LIGHTGREY);
  tft.drawRect(GREENBUTTON_X + 6, 60, GREENBUTTON_W - 12, 35, ILI9341_BLACK);
  tft.drawRect(GREENBUTTON_X + 6, 95, GREENBUTTON_W - 12, 35, ILI9341_BLACK);
  //Scroll up arrow
  tft.setCursor(GREENBUTTON_X + 17, 65);
  tft.setTextColor(ILI9341_BLACK);
  tft.setTextSize(3);
  tft.println((char)24);
  //Scroll down arrow
  tft.setCursor(GREENBUTTON_X + 17, 100);
  tft.setTextColor(ILI9341_BLACK);
  tft.setTextSize(3);
  tft.println((char)25);
  //Add item button
  tft.fillRect(GREENBUTTON_X, 138, GREENBUTTON_W, GREENBUTTON_H, ILI9341_DARKGREEN);
  tft.drawRect(GREENBUTTON_X, 138, GREENBUTTON_W, GREENBUTTON_H, ILI9341_BLACK);
  tft.setCursor(GREENBUTTON_X + 7, 150);
  tft.setTextColor(ILI9341_WHITE);
  tft.setTextSize(2);
  tft.println("ADD");
  //YES button
  tft.fillRect(GREENBUTTON_X, GREENBUTTON_Y + 7, GREENBUTTON_W, GREENBUTTON_H, ILI9341_RED);
  drawFrame();
  tft.setCursor(GREENBUTTON_X + 7 , GREENBUTTON_Y + (GREENBUTTON_H / 2));
  tft.setTextColor(ILI9341_WHITE);
  tft.setTextSize(2);
  tft.println("YES");
}

void byeBtn()
{
  //Have a good day text
  tft.fillScreen(ILI9341_BLUE);
  tft.setCursor(10, 10);
  tft.setTextColor(ILI9341_WHITE);
  tft.setTextSize(4);
  tft.println("Great! Have  a good day");
  //bye button
  tft.fillRect(GREENBUTTON_X, GREENBUTTON_Y + 7, GREENBUTTON_W, GREENBUTTON_H, ILI9341_GREEN);
  drawFrame();
  tft.setCursor(GREENBUTTON_X + 7 , GREENBUTTON_Y + (GREENBUTTON_H / 2));
  tft.setTextColor(ILI9341_WHITE);
  tft.setTextSize(2);
  tft.println("BYE");
}

void setup() {
  // Set up serial monitor
  Serial.begin(9600);
  pinMode(MOTION_SENSOR, INPUT);
  pinMode(LED, OUTPUT);
  tft.begin();
  rtc.begin();
  // Set up the screen at first
  tft.setRotation(1);
  yesBtn();
}

void loop() {

  dayUpdate();
  
  if (digitalRead(MOTION_SENSOR) == HIGH) {
    digitalWrite(LED, HIGH);
    //Serial.println("motion sensor triggered");
    remind = true;
  }
  else if (digitalRead(MOTION_SENSOR) == LOW) {
    digitalWrite(LED, LOW);
    remind = false;
  }

  if (remind == false) {
    // fills the screen with black to save power 
    // and also not draw attention when not needed
    tft.fillScreen(ILI9341_BLACK);
    while (remind == false) {
      if (digitalRead(MOTION_SENSOR) == HIGH) {
        digitalWrite(LED, HIGH);
        //Serial.println("motion sensor triggered");
        remind = true;
      }
      else if (digitalRead(MOTION_SENSOR) == LOW) {
        digitalWrite(LED, LOW);
        remind = false;
      } 
    }
  }

  //only do things if the motion sensor gets tripped
  if (remind == true) {
    // first read the screen to see if he did anything
  // Retrieve a point
  TSPoint p = ts.getPoint();

  //See if there's any touch data
  if (p.z > MINPRESSURE && p.z < MAXPRESSURE) {
    //Scale using the calibration #'s and rotate coordinate system
    bx = map(p.y, TS_MINX, TS_MAXX, 0, tft.width());
    by = map(p.x, TS_MINY, TS_MAXY, tft.height(),0);
    if (hasEverything == false) {
      // if he presses the button
      if ((255 < bx && bx < 310) && (192 < by && by < 230)) {
        // set hasEverything to true and execute that part of the code
        hasEverything = true;
      }
      if (hasEverything == true) {
      byeBtn();
      delay(5000);
      hasEverything = false;
      yesBtn();
  }
    }
  }

   p = ts.getPoint();

  //See if there's any touch data
  if (p.z > MINPRESSURE && p.z < MAXPRESSURE) {
    //Scale using the calibration #'s and rotate coordinate system
    bx = map(p.y, TS_MINX, TS_MAXX, 0, tft.width());
    by = map(p.x, TS_MINY, TS_MAXY, tft.height(),0);
    Serial.print("bx = "); Serial.print(bx); Serial.print(" by = "); Serial.println(by);

    // pressed add button
    if (255 < bx && bx < 310 && 143 < by && by < 181) {
      bx = 10;
      by = 10;
      adding = true;
      addMode();
    }
    checkScroll();
  }

  // if he adds stuff from serial monitor
  if (Serial.available() > 0) {
    //// upload it onto the main display
    add = Serial.readStringUntil('\n');
    updated = updated + '\n' + ' ' + add;
    Serial.println(updated);
    yesBtn();
  }

  }

  
}

NOTE: After it is tested and confirmed to work with the Arduino, we will be adding in the IoT version of this code, which connects with Google API to return events without the need for input.

Schematic and Design Files

Schematic of the box at final demo

Joanne wanted to have control of her front yard lamp from inside her house. She also wanted her front steps to be better lit at night. We made a modular system that triggers her linear switch box for the lamp and also lights up LED tape for her stairs when she flips a switch inside her house. This system gives Joanne complete control of her front yard lights from a single switch inside. See here for documentation of our prototype and see here for documentation of our first meeting with Joanne.

What We Built

This overall photo shows all components of the project.

This video shows our project in action. When the user grips the photocell and blocks light from reaching it, the system is triggered and it turns on the LED strip and the servo motor rotates to move the toggle switch up. When our device is actually implemented at Joanne’s house the photocell won’t be squeezed by the user, rather the photocell will be focused on a lightbulb that is controlled by Joanne from inside her house.

The LED strip and servo motor assembly are the two main features of this project.

The electronic hardware that deal with voltage logistics and computation are shown in this photo.

This photo shows the 3D printed assembly that the servo motor operates. The part in grey is a model of Joanne’s linear toggle switch. The part in black is what we will implement at her house to drive the toggle switch.

Here is a closeup of the rack and pinion mechanism that is used to translate the servo’s rotational motion into linear. This is an important feature because we are using the servo’s rotational output to activate the toggle switch which requires linear motion.

The photocell is placed in a 3D printed housing to focus its view on the lightbulb at Joanne’s house that will be triggering the whole system.

Narrative Sketch

It is a cold winter night and Joanne is in her house when her brother comes home. She wants to turn on her front yard lights for him but one of the front yard lamps is controlled by a switch located outside. She isn’t dressed properly to go outside into the freezing weather conditions. Just as she begins to go put on her warm coat she realizes that two students from Carnegie Mellon just implemented a yard light system that allows her to control all of her front yard lights by a single switch inside. Bursting with delight, she smiles as she flips the switch that controls one of the lamps on her front yard. The photocell that the students place inside of this lamp changes resistance value due to the increased light from the lamp turning on. This triggers her newly implemented yard light system to do two things. One, rotate the servo motor which pushes the linear toggle switch up and turns on the second lamp on her front yard. Two, turn on the LED light strip and light up the previously unlit section of her front yard path. Her brother walks inside and then she flips her control switch again and a split second later all the lights on her front yard are back off.

How We Got Here

After our first prototype we knew we would have to order a lot of parts for the final device. Once essential part was the waterproof servo. Once this arrived the first thing we did was test it.

The new servo runs on a maximum of 6.8 volts and we used a power supply to test it.

It was a little different than the servo we were used to using because it had 270 degrees of rotation instead of 180. We knew that we would have to eventually adjust our code to control it but the more challenging problem was to change our mechanical design of our prototype’s 3D printed assembly to fit with the larger dimensions of the new servo. First, we took measurements of the servo and modeled it in SolidWorks.

We used a caliper to measure the servo and we only modeled the important features. We ignored all the little fillets and complex grooves.

Although it didn’t take that long to model the servo, it was still time wasted because we later found an exact CAD model on the servo manufacturer’s website. Having the CAD of our servo made it easier to design around.  For our new design, we wanted to ensure that we were getting the maximum amount of torque output from our servo so that there would be no issue in terms of strength with moving the linear switch at Joanne’s house. To do this, we had to optimize the size of the gear that would attach to the servo. The smaller the gear, the higher the torque output is. Knowing that the maximum range of rotation for our servo is 270 degrees and that the range of motion for the linear switch is about 0.8 inches, we calculated how small the pitch diameter of our gear could be. We ended up with a pretty small gear that only had 8 teeth on it. It was only after we implemented this new gear into our CAD assembly that we realized the lower bound of our gear size was actually constrained by the width of the servo. Unless we made drastic design changes to our assembly, the gear diameter had to be slightly larger than the width of the servo or else it wouldn’t be able to mesh with the rack. So we designed a larger gear and then moved on to making design adjustments of other parts in our assembly. These adjustments included:

1. Slightly reducing the dimensions on the rack for smoother gliding within its track
2. Making the grip for the switch bigger and enclosed to maximize the contact area with the switch
3. Raising the location of the servo relative to the rack so that it started at the top of the rack and allowed us to optimized the length of the rack.
4. Moving the connection point for our rack and pinion assembly from the front face of the switch box cover to the side face.
5. Reducing the weight of the rack and pinion assembly by cutting down unused wall space.
6. Minor changes to increase structural integrity.

Final assembly of the rack and pinion mechanism that the servo drives.

We also made a significant design change to our switch box cover which is what our rack and pinion assembly will be mounted to. Our prototype cover enclosed the entire switch box. For the final design, we decided that covering the entire switch box was unnecessary and only increased the chances of our cover not fitting properly over the box. So we designed the new cover to be like a small hat for the switch box with a small wall coming down the side for the rack and pinion assembly to be bolted to. This was a minimalist solution to a problem we were very concerned about. The switch box had a lot of protruding bolts, inconvenient fillets, and other complexities induced by deterioration over time. We were worried that even with accurate measurements of the box, these features could throw off the fit of our cover and therefore, cause our switch grip to be misaligned. We think our simplified box cover greatly increases the probability of a proper fit but we still plan to bring some files and other tools with us when we implement our device at Joanne’s house in case additional adjustments need to be made.

Full assembly of Joanne’s linear toggle switch and the device that we designed to control this toggle switch. The part that is a darker grey shade plus the cylindrical component coming out of the bottom of it is a model of Joanne’s switch. The rest is what we will be implementing.

We used an Ultimaker to 3D print all of our parts.

Full assembly of our 3D printed parts. The part in grey is a model of Joanne’s switch and the parts in black constitute our solutions to controlling this switch with a servo.

To test the performance of our assembly we made a simple circuit to control the servo with a push button.

Our solution to control the linear toggle switch was now complete. We then moved onto the other main component of our project: controlling the LED light strip. We first hooked up our LED strip to a power supply to ensure that it was working.

Our LED strip runs on 12 volts. The lights are a cool white color.

We used an n-channel mosfet to turn on and off the LED strip. We were unsure about how to properly use a mosfet so we did a little research online and made a simple circuit on a breadboard and tried a few different configurations until it worked.

Test circuit using our mosfet.

With all of our hardware and electronic components properly working, we could then integrate everything together. To do this we first had to cut our electrical cords and wires to the proper lengths dictated by the measurements we had previously taken of Joanne’s front yard. To determine these lengths, it helped to visualize the layout of  our entire system so we drew a rough sketch of it on a table.

Each component was methodically placed on the table to be in the relative position that they would be when we actually implement the entire system at Joanne’s house. This diagram helped us determine the four different cord lengths that we needed.

We cut the electrical cords to conservative lengths and then we were ready to solder everything together. We soldered a few connections but before long, it became hard to keep track of everything so we drew another diagram to visualize the entire circuit.

This diagram gave us a visual of almost all the connections that needed to be soldered.

We spent a whole night soldering but by the morning our circuit was complete and all components were working properly.

Completed circuit with all connections soldered.

Joanne and Rory at the final crit. Claire unfortunately could not be present because she got very sick the night before.

Conclusions and Lessons Learned

From the final crit we added two major changes and one minor one. The first being a potentiometer which told the code what number was the threshold between night and day. This was a simple addition until the Potentiometer was accidentally plugged into 12v and not the 5v. This broke the Arduino Nano and it had to be removed and a new one soldered in its place. The second major point was allowing Joanne the ability to choose what turning on her front porch light would trigger: the LEDs, the lever, both or none. The simplest method to complete this is having two switches, on representing the lever and one representing the LEDs. If both are switched off turning on the front porch light does nothing. If both are switched on turning on the front porch light triggers both. A challenge was incorporating this addition into the design that would maintain integrity in the weather. A decision was reached that the switches will mount through the top of the waterproof box. The final slight adjustment was making use of the other half of the breadboard by adding indicator lights, a set for daytime and nighttime, a set for the lever pushed and pulled and a set for when the LEDs were on and when they were off in order to help with install and troubleshooting.

From the final crit we received approval from many people. One person said that our device “really solved the problem” and that this could be adapted to solve a similar problem he is having. He was concerned that the “connections seem delicate and could malfunction”. This brings us to our discovery on how critical visual communication was (see the next paragraph). I believe the delicate parts he referred to were the small thin wires on the breadboard itself, something that would be protected by the waterproof box and since the waterproof box wasn’t present at the crit his concern makes sense. But maybe he was psychic and detected the problem that took Zack and Claire a good hour to figure out why the system suddenly electrically stopped working.

Many others shared the concern of weatherproofing and durability.

Some honorable mentions:

“This would be applicable in my home, but I don’t live in a house”

“Good luck installing it :)”

“A creative solution that I will really appreciate”

Something that was a major communication bump early in the process was the first showing of the prototype. Because we lost a group member and a part of the project along with her, we attached a button to the breadboard to show the motor being triggered. Without the photoresistor giving the visual information we were trying to communicate the idea didn’t get across easily. Instead the visual of the button trumped the verbal explanation. The big takeaway was to be as straightforward and simplistic in the explanations for the best possibly clarity. Next time we would have jumped immediately from the final crit to an analysis of the next step, purchasing the switches and getting it ready for installation. Since we waited till the weekend we were unable to install in time and pushed the date back to January. If we could do anything differently it would be Claire not needing to go to the E.R. the night before the project was due.

The overall takeaway is that there is always something you overlook and time needs to be allotted for the mistake. Successfully managing a project is not defined in the ability to predict mistakes but to accommodate for them.

Schematic

Code

/*
 * Yard Lights
 * Claire Mildred and Rory Hubbard
 *
 * This code activates and LED strip and a servo motor depending on a reading
 * from a photocell. When the photocell reading is below a certain threshold
 * the LED strip will be turned off and the servo will rotate to the 0
 * position. When the photocell is higher than the threshold the LED strip will
 * be turned on and the servo will rotate 80 degrees. No rights reserved.
 * 
*/

#include <Servo.h>
Servo myservo;

// establish pins
const int PHOTOCELL_READ = A0;
const int LED_CONTROL = 5;

// establish photocell threshold value to turn on or off the system
const int THRESHOLD = 800;

bool down; // this variable will keep track of the position of the servo
int photoVal; // initialize a new integer to store the photocell value

void setup() {
  pinMode(PHOTOCELL_READ, INPUT);
  pinMode(LED_CONTROL,OUTPUT);
  myservo.attach(3,500,2500); // pulse width range of this servo is 500 - 2500
  // microseconds
  myservo.write(0); // start with servo down
  down = true;
  //Serial.begin(9600);
}

void loop() {

  photoVal = analogRead(PHOTOCELL_READ); // acquire a reading from the
  // photocell every loop
  // Serial.println(photoVal);

  if (photoVal >= THRESHOLD && down == true) {
    myservo.write(80);
    down = false; // servo has moved the toggle switch up
    digitalWrite(LED_CONTROL,HIGH); // turn on LED strip   
    delay(100); // little delay to deter flickering and allow servo to reach   
    // final position
  }
  
  else if (photoVal < THRESHOLD && down == false) {
    myservo.write(0);
    down = true; // servo has moved the toggle switch down
    digitalWrite(LED_CONTROL,LOW); // turn off LED strip    
    delay(100); // little delay to deter flickering and allow servo to reach
    // final position
  }

  // no else statement because we only need to write to our servo and LED's
  // one time when it changes state

}

For our final project, our team worked with Joseph, a semi-retired attorney and a Parkinson’s patient, in a collaborative effort to design and develop an assistive device that would be useful to him every day. Over the past several weeks, Team Joseph created a detachable cane assist that focused on mitigating some of Joseph’s freezing episodes, which are involuntary and temporary blocks of movement that can limit his everyday mobility. Typically, Joseph experiences freezing episodes when he is due for his next dose of medication. This accessory will be useful to him during this time when he may be more vulnerable to freezing.

For more information about our process, please read our prototyping documentation and our meeting documentation with Joseph.

What we built

This detachable cane accessory provides a visual cue (a red laser on the ground) and a haptic cue (vibrating disks). Both of these cues gives Joseph a specific target to focus on, which can help him move past a freezing episode. The cane accessory is a 3-D printed box that has buttons at the top to control the power (on/off switch), the visual cue (laser), and haptic cue (vibrating disks). In case Joseph needs to change the rechargeable batteries, we implemented a hinge and a latch that can open the box. As Joseph travels frequently to visit family across the country, we ensured that the batteries were TSA-safe.

We 3D printed a custom box that can attach to any of Joseph’s many canes. Joseph likes to buy new canes for different needs (e.g. a cane for snowy/slippery weather, a cane for walking on inclines, a cane for collapsible travel purposes, etc.)

Luckily, we were able to find a spare cane in the Phys Comp lab. This helped us decide early on that we wanted a detachable feature, rather than built in directly to one of Joseph’s canes.

We ordered spring clips online. However, one roadblock that we ran into was that the clips were slightly too large for the cane we had at hand. Since this was a standardized size (and the next size up/down would not be appropriate), we decided to implement a velcro band that can attach to the cane as well, and keep it from falling off forward, which was an incident that we had during one of our demos.

The very bottom contents of our box. This contains our soldered wirings to a protoboard to increase the box’s durability in the case that the cane is dropped. It also contains an Arduino Nano (power source not pictured).

The next layer of contents in our box. We used a scrap piece of thin 3D printer material to separate the backend wirings from the rest of the box. Pictured is the laser that is positioned at an angle to provide a visual cue that Joseph can follow and step over.

Top layer of contents in our box. We have a 3×2 battery pack powered by rechargeable dry-cell batteries that are TSA-approved. To keep the battery from jiggling around in the box, we cut a piece of foam to secure all of the box’s contents in place. We placed the battery at the top of the box for ease of reach once Joseph needs to change the batteries.

A close-up view of the laser that we embedded into the box. We used a hot glue gun to secure its position in case of a fall. On the inside right part of the box, we also glued 5 vibrating disks that will vibrate the box and the handle to provide Joseph a haptic cue. We tested the vibrations on Joseph to ensure that they were strong enough.

The top of the box contains three input buttons and switches. The switch turns the assistive device on/off; the off option allows us to save battery when Joseph does not need the assistance. The red button turns on the laser, and the black button turns on the vibrations. We made these cues independent of another per Joseph’s suggestion.

The above video shows how each of the buttons work. We designed the buttons to sit on top of the box so that Joseph could reach them with minimal movement (we confirmed with him that while he was in a freezing episode, he was able to move his fingers to reach these buttons).

Despite being a Parkinson’s patient, Joseph is still very active and travels frequently to visit his family in San Francisco. When Joseph goes to the airport, he brings a different cane with him — a collapsible one that will fit in his carry-on bag. Joseph also brings his detachable Cane Assist that can attach onto any one of his canes flexibly. He goes through TSA successfully.

In the latter half of the day, Joseph’s symptoms tend to worsen. Before his dopamine medication is due, he is more prone to freezing episodes. When he is navigating the airport, Joseph experiences a freezing episode, where he struggles to coordinate where to move next. He turns on the Cane Assist and pushes the laser button, as well as the vibrator button, which provides him both visual and haptic cues to stimulate his brain and help direct his next steps. With the Cane Assist’s help, Joseph is able to move past the episode. He remembers to take his medication and successfully reaches his destination.

How we got here

Along the journey of building Joseph’s Cane Assist, we made mistakes, discoveries, and pivots.

One big challenge that our team faced was figuring out how to build a custom box for Joseph. Since none of us had experience with 3D fabrication, we had to obtain outside help (thank you to one of Catherine’s friends).

parts measurement

box design details

We faced several 3D printing mishaps, like the printer jamming overnight, so in the meantime we built an acrylic backup prototype.

As a backup, we laser cut an acrylic box to fit all of our hardware. We wanted it to be a closed-box solution.

However, after laser cutting the pieces and gluing them together, we realized that the acrylic box (although visually appealing) was extremely heavy and would add significant weight to the cane. Consequently we reinvested our efforts into 3D printing our box. Luckily, one of our classmates, Roly, offered to print our box for us with his at-home 3D printer (shoutout to Roly — thank you!).

Our 3D printed box (at last) with Roly’s printing help. This material was much lighter and didn’t add noticeable weight to the cane.

The 3D printed box that Roly helped us print out was lightweight and much better suited to our needs, especially considering that we needed to add in a battery pack (significant weight) and other hardware components. However, finding a suitable hardware solution was not the last of our worries.

Another significant challenge we faced was finding a suitable battery pack. Because of our electrical components (laser, vibrators) we needed at least 6 batteries, or 7.2 volts. In the classroom, we were able to find some spare battery holders, which were either 6 in a line, or in a 3×2 configuration. Some of these holders were faulty and/or had physical gaps in the electrical connection.

This was our 3×2 battery pack that had a missing electrical connection. Zach kindly helped us solder our wire directly onto an electrical connection to get it working. However, during a demo, our box was dropped and knocked some things astray in our battery holder.

Zach helped us solder a wire to correct some strange electrical disconnections in this battery holder. During our first client presentation, our Cane Assist was dropped and we determined that the battery pack was an issue again. We had to last-minute re-order a new battery pack in this 3×2 configuration (due to space constraints in our box). We also worked on ensuring that our box would not fall during usage again.

To remedy this problem, our team found some spare foam and cut a hole in the middle to fit out battery pack. This foam would act as a physical cushion that would hopefully absorb impact in case the cane assist was dropped. The foam would also prevent the battery from making unnecessarily loud vibrating sounds when the vibrator button was turned on.

We added foam stuffing to prevent the battery from moving around inside the box and making excessive noise when the vibrating button was turned on.

We also realized that the cane assist had fallen during the demo because the spring clamps that we ordered did not perfectly fit the cane that we had at hand. Since these sizes were standardized, we were unable to find a size that was a closer fit to the cane diameter. Instead, we installed a velcro strap at the back of the box that would act as a third flexible clamp to hold the box securely to the cane.

We installed a velcro strap to the back of the box to more securely fasten it to any cane.

We thought adding a velcro strap would be the best solution, because cane diameters cane vary, and it would not be possible to find a single clamp that fits all the different sizes of Joseph’s many different canes.

Since this cane assist is a detachable solution that would be carried around, we wanted our batteries to be rechargeable. We purchased dry-cell rechargeable batteries on Amazon that were TSA-friendly. However, another challenge was designing the 3D-printed box such that it could be opened and closed, since Joseph would need to replace the batteries from time to time.

Per Zach’s suggestion, we devised a hinge-and-latch solution, where the front wall of the box would have a hinge that could allow the wall the swing open. There would be an additional latch at the outside of the box’s front that would keep the “door” shut.

Our sketch of the hinge-and-latch solution to make our box accessible, so that Joseph can replace the batteries when needed.

We had some extra help on this part. (Thanks Zach)

The finished hinge-and-latch solution.

We decided to purchase a pinless plastic hinge to reduce mechanical complexity. Combined with the latch, installed at the top of the door and the box’s side, the hinge-and-latch mechanism worked fluidly and easily, without popping open and keeping all the contents of the box in place.

Lastly, another challenge that we had was getting the laser to position correctly inside our box. If the laser was installing pointing straight down, it would not give sufficient distance from the cane to be a walking cue that Joseph could use to “step” over.

Using some complex measuring tools (a protractor), we tried to measure at what angle the laser should be pointing to give sufficient distance from the cane and be a walking target.

Despite our best measuring efforts, it was difficult to build out a model in our 3D modeling software that would allow a sufficient angle. Previously, we had a separate part that would help anchor the angling component, but this ended up being too large and taking up valuable space inside our cane assist box. As an alternative solution, we filed the hole for the laser to be slightly larger, and used a hot glue gun to properly anchor the laser at the right angle.

Conclusion and lessons learned

For most of us, this was our first client project. From this project, we learned that the initial client meeting is an incredibly important meeting. Our first meeting with Joseph lasted well over two hours, and while we didn’t stick to a script, perhaps in the future we would better outline the structure of client meetings, so that it’s clear for both sides what information needs to be exchanged. Joseph came into the meeting very prepared, with printouts of design ideas he already had and suggestions/solutions for cane accessories that we could build. During the meeting, it felt like we had a lot of ideas that we could potentially build; however, afterwards, we realized that many of these were mechanical solutions, and that it would be difficult to incorporate an electrical component. Therefore, for initial client meetings, we learned that it’s important to direct the conversation to learn about the client’s problems rather than solutions right away; this means understanding his background, habits, and pain points. Honing specifically in on understanding these points would have better helped expand the breadth of potential design solutions we could have worked on.

Prior to the initial meeting, it also would have been helpful for us to have thoroughly researched the symptoms of Parkinson’s disease beforehand. We had been mostly focused on gathering interview questions for Joseph, but hadn’t known what kinds of problems that the client had (perhaps it’s a good idea to give a brief background of each client prior to the initial meeting for future Phys Comp students). With some research beforehand, we believe we could have also helped expand the breadth of design ideas by asking Joseph how he experiences some specific symptoms associated with Parkinson’s.

During our final crit, we received mostly positive feedback. Some people mentioned that this idea is one that could potentially be monetized or purchased by a medical device company, which was encouraging to hear. We also received some concerns about weatherproofing — while one of our initial goals was to keep the box weather proof, we realized that it would not be possible after we devised the latch-and-hinge mechanism (the box would not be watertight with a door). For future iterations of the project, we would look into designing a custom box that would be resistant to Pittsburgh’s random weather. Additionally, we would add more custom functionality to the pacing of the vibrations, perhaps with a potentiometer, such that Joseph can tune the rhythm to his liking.

Overall, we really enjoyed working on this project, and we’re excited to hand-off the project to Joseph so he is able to use it as an everyday assistive device.

Technical details

/*
 * Project Title: Cool Cane
 * Name: Catherine Yu, Linda Xia, Joey Santillo
 * Description: There are two buttons. One controls on/off the haptic cue(vibrator) and 
 * the other controls on/off of the visual cue(laser).
 */


//PIN ASSIGNMENTS
const int LASERPIN = 5;
const int VIBRATORPIN1 = 8;
const int VIBRATORPIN2 = 9;
const int VIBRATORPIN3 = 10;
const int VIBRATORPIN4 = 11;
const int VIBRATORBUTTON = 4;
const int LASERBUTTON = 3;

//CUSTOMIZABLE VARIABLES
const int vibrateInterval = 1000;
const int debounceDelay = 10;

//VARIABLES NEEDED FOR LATER
bool LASERNeeded = false;
unsigned long lastLASERTime = 0;
int lastLASERState = HIGH;
bool VIBRATENeeded = false;
bool vibrating = false;
unsigned long lastCUETime = 0;
int lastCUEState = HIGH;
unsigned long currTime = 0;

void changeVibrate(){
  if (currTime-lastLASERTime>=vibrateInterval){
    if (vibrating){
      digitalWrite(VIBRATORPIN1, LOW);
      digitalWrite(VIBRATORPIN2, LOW);
      digitalWrite(VIBRATORPIN3, LOW);
      digitalWrite(VIBRATORPIN4, LOW);
      vibrating = false;
    }
    else{
      digitalWrite(VIBRATORPIN1, HIGH);
      digitalWrite(VIBRATORPIN2, HIGH);
      digitalWrite(VIBRATORPIN3, HIGH);
      digitalWrite(VIBRATORPIN4, HIGH);
      vibrating = true;
    }
    lastLASERTime = currTime;
  } 
  else{
    if(vibrating){
      digitalWrite(VIBRATORPIN1, HIGH);
      digitalWrite(VIBRATORPIN2, HIGH);
      digitalWrite(VIBRATORPIN3, HIGH);
      digitalWrite(VIBRATORPIN4, HIGH);
    }
    else{
      digitalWrite(VIBRATORPIN1, LOW);
      digitalWrite(VIBRATORPIN2, LOW);
      digitalWrite(VIBRATORPIN3, LOW);
      digitalWrite(VIBRATORPIN4, LOW);
    }
  }
}

bool checkDebounce(String indicator, int state){
   //returns true if needs an update 
   //false otherwise
  if (indicator == "LASER"){
    if (currTime-lastLASERTime>=debounceDelay && state != lastLASERState){
      lastLASERState = state;
      lastLASERTime = currTime;
      return true;
    }
    return false;
  }
  else{
    if (currTime-lastCUETime>=debounceDelay && state != lastCUEState){
      lastCUEState = state;
      lastCUETime = currTime;
      return true;
    }
    return false;
  }
}

void setup() {
  Serial.begin(9600);
  pinMode(LASERPIN,OUTPUT);
  pinMode(VIBRATORPIN1,OUTPUT);
  pinMode(VIBRATORPIN2,OUTPUT);
  pinMode(VIBRATORPIN3,OUTPUT);
  pinMode(VIBRATORPIN4,OUTPUT);
  pinMode(VIBRATORBUTTON,INPUT_PULLUP);
  pinMode(LASERBUTTON,INPUT_PULLUP);
}

void loop() {
  currTime = millis();
  if(LASERNeeded){
    digitalWrite(LASERPIN, HIGH);
  }
  else{
    digitalWrite(LASERPIN, LOW);
  }
  if(VIBRATENeeded){
    changeVibrate();
  }
  else{
    digitalWrite(VIBRATORPIN1, LOW);
    digitalWrite(VIBRATORPIN2, LOW);
    digitalWrite(VIBRATORPIN3, LOW);
    digitalWrite(VIBRATORPIN4, LOW);
  }
  if (checkDebounce("LASER", digitalRead(LASERBUTTON)) && digitalRead(LASERBUTTON)==LOW ){
    if (!LASERNeeded){
      LASERNeeded = true;
    }
    else{
      LASERNeeded = false;
    }
  }
  if (checkDebounce("VIBTRATOR",digitalRead(VIBRATORBUTTON)) && digitalRead(VIBRATORBUTTON)==LOW){
    if (!VIBRATENeeded){
      VIBRATENeeded = true;
    }
    else{
      VIBRATENeeded = false;
    }
  }
}

Schematic

You can view the Google drive link to our 3D models (STL files) here.

Thank you, Joseph!

Our team had a wonderful time working with Joseph on this cane assist.

This project is a Bingo-style game for Jeffrey to play with his 6-year-old granddaughter, Stella. This game, partially disguised as trivia, would be used as a tool to spark conversations about interests and personal topics as a way to get both young kids and adult family members to know each other better. See here for documentation of our initial meeting with Jeffrey and see here for documentation of our prototype.

What we built

We built an electronic board-game, with very simple rules, which could be used for both a 73 year old and a 6 year old. The game-play is a mix between Trivial Pursuit, Bingo and Truth or Dare.

Before the game starts, each Player writes down a set of questions for the other player to answer. The use of this analog method is to create a constantly evolving game, which grows and complexifies as Stella does. Player 1 asks a question from their stack of cards, and player 2 picks a spot on the board and attempts to answer. If answered correctly, player one presses their green button which locks the location on the button pad (as seen in the image below). If the answer is wrong, however, the illuminated button disappears, and player 2 must try again in the next round. This format continue until one player obtains 5 lit-up buttons in a row.

For a long game play, penalty cards such as “Tell me a story in song” or “Invent a short dance routine”, may be given to the person who has answered wrong at the end of each turn. For a short game play, the penalty card may be given at the end of the game.

Player 1 reading a question card, while player 2 chooses a position on the button pad.

Detail shot of the LCD screen, giving the players indications: “Green asks, Pink answers.” You also get a sense of the box’s transparency, which was a customer request made by Jeffrey.

Detail shot of the button pad, correct and incorrect buttons, as well as the LCD screen. 

Detail shot of the speakers and box design on the back face.

Narrative Sketch

Stella gets dropped off by her single father at Jeffrey’s house, for their weekly Wednesday hangout. Stella would like to play a game, but Jeffrey isn’t sure how to successfully entertain a 6-year girl. Instead of having his wife help him out, he pulls out their customized board-game.

Stella likes to start, because she enjoys the pretty colors the button pad generates. As the game starts, a rainbow pattern illuminates the board, making Stella smile.

The first step of the game is for each player to write down a list of personalized questions on their assigned cards, which can be re-used or re-written as pleased. The aim of these questions is to generate conversation, and learn something from one another. Then, Stella starts by pressing a button of her chosen location on the button pad. Jeffrey asks her a question from his green deck of cards .

Player 1 chooses a location on the board.

Stella attempts to answer the question. If answered correctly, Jeffrey will select the green button, locking in the position on the buttonpad. If the answer is incorrect, Jeffrey presses the red button, and the button’s light goes out. Stella can then try again during the next round. In a long-play game, Stella would then have to perform a penalty found on the white stack of cards.

Player 2 reads the card from their stack, and prompts Player 1 to answer. If correct, Player 2 presses the “correct” button, and the button pad stays lit.

After Stella’s turn, Jeffrey can pick a position and attempt to answer a question. The game goes on until a player is able to line up 5 of their colored buttons on the pad. During a short play game, the loser must perform a penalty found on the white stack of cards, only at the end of the game.

When the game is over, the orange “reset” button can be pressed, a rainbow pattern appears and the board is cleared. 

How we got here

Initial Meeting and Brainstorm

On November 1st, 2018, our group – Chloé Desaulles, Jiatian Sun, Jianxiao Ge –  firstly met up with Jeffrey, who is an older man participating in the CMU Osher program, with the intention of building a useful device for him.

We had a meeting and interviewed Jeffrey in the university cafe. After the first interview, we started by brainstorming ideas to make Jeffrey’s life easier or more pleasurable. He did not seem to need anything fixed, so we tried to look into ways of enhancing good things in his life. It became clear very quickly that he kept gravitating towards his granddaughter Stella, and the time he and his wife get to spend with her. We additionally noticed that Jeffrey might not perfectly know how to interact with a six-year-old, often letting his wife entertain her.

Our main challenge was creating an interesting and fun game for such a wide age gap. Making a game intuitive and worth playing was a hard thinking exercise considering none of us had any game design experience. In addition, we needed to add physical computing elements into the game, which made it even harder.

Finally, we were inspired by the traditional game Gomoku & Bingo and decided to create a similar game with trivia elements, which match Jeffrey’s interests, and a simple and colorful button interface, with a changeable gameplay so the game can be adaptable as Stella grows up.

Preliminary sketches of our game board.

Prototype and the Second Meeting

After settling on the concept and gameplay, we started working on the layout and board design, option for big buttons with fun interactions, which would make the game more interesting to a 6-year-old. However, because large buttons took up too much space, we ended up with a medium sized button.

Working on the conceptualization and physical design/implementation of the game board.

We then started programming the colored buttons, LCD screen, and button pads separately and finally got together to merge our individual codes.

Jianxiao and Jiatian working on merging code sections.

The LCD screen welcoming Jeffrey and Stella to their custom game.

One of the biggest challenges we met during prototype was the core part of our game, the button pad. The RGB keypad we needed was out of stock at that time. We contacted the manufacturer Adafruit and received a reply that it would be two weeks before we could have the keypad available. Therefore, we had to order the LED version of the pad as an alternative in order to finish the prototype on time.

Setup of our first button pad trial with LED keypad.

Button Pad is able to light up according to the color of LED beneath it.

The completed first prototype.

We met Jeffrey again after the prototype presentation. Jeffrey seemed very excited by our prototype and believed he and Stella would have fun playing the game. He was also happy about our choices in tactility and color and even suggested we add sound (which is something we had been shying away from because we thought it might be irritating Jeffrey and his wife). What’s more, he gave us a lot of input when it came to the aesthetics of the board, suggesting we should make it out of clear acrylic so that Stella could see the colorful wiring on the inside. Some other fabrication advice he and Zach gave us included building a box to hold question and penalty cards. The crit was useful for our team and allowed us to move confidently forward with our design.

Jeffrey giving us feedback on our prototype.

Final Fabrication

After the second meeting, we adjusted the game based on the feedback. For example, combine the card box with the game board, add the game sound effect, and remove people’s name from the instruction on the LCD screen to involve other family members as well.

Chloé and Jiatian adjusting the game after the second meeting.

Another good news was that RGB Keypad, which was out of stock before, had been replenished. With the help of Zach, we got the keypad in time and immediately started to adjust the code as well as the soldering and testing.

Soldering four 4*4 RGB Keypads together.

At the same time, we started the design of the game box and material preparation. According to Jeffrey’s suggestion, we chose the white fog-faced acrylic as the main material to highlight the colorful buttons and create a light and transparent visual effect.

We sketched the game board by hand and determined the general layout. After actually measuring the size of each part, we drew it exactly one-to-one and then redrew it in our laptop using Autocad and Adobe Illustrator.

Rough hand sketch of game board design and dimensions.

Drawing it one-to-one after measuring the size of each part.

While Jiatian was in charge of the code adjustment and part testing,  Chloé and Jianxiao took the acrylic to the laser cut studio. The laser cutting process was not as smooth as expected. Because of the measuring error, the buttons and the LCD screen couldn’t be embedded smoothly. So we measured again, adjusted the size and then carried out a second laser cut, finally got the panel in line with the requirements and started the assembly.

During the assembly, we found that the card box was smaller than expected, so the card could only be erected. But we thought it would make it easier to pull out the cards and make the game go more smoothly, so we didn’t redo the box.

On December 4th, a month after the initial meeting, we had the final presentation and met Jeffrey again. We were delighted to see that he was very pleased with our final product. He played the game with us and was ready to give it to his granddaughter Stella as a Christmas present.

Jeffrey playing the game with us during the final desk crits.

Here is our team! (From left to right: Jiatian Sun, Jeffrey, Chloé Desaulles, Jianxiao Ge)

Conclusions and lessons learned

To our surprise, almost all of the oral feedbacks we received during the final crit were positive, whether it was about the experience of the game or the design of the box itself. (which, of course, could be because people were too shy to make suggestions in person). We were happy to see that older people seemed to be very interested in the game. Some of them even thought that it had commercial value. Since the question cards can be written by the players themselves, it gives the game more possibilities. It can be a bridge for family members to communicate and learn from each other, even if there is a huge age gap.

After the crit, we also see a lot of very good advice from the written feedback. For example, “It would be useful to add the name/instructions onto the game.” Because of the novel mechanics of the game, people may be confused when they start to play. Although we try to give each step a hint on the LCD screen, some instructions are not clear and accurate due to the size of the screen. At the same time, an extra game description may help people understand the game faster.

Here is another one: “It could be smaller/thinner – effects portability. Would be nice if users could choose which colors they are in the button pad – maybe they have a favorite color.” The final volume of the game box was indeed larger than we expected. In order to prevent the electronic components from being unable to fit, we reserved much space during the box design. But when we actually assembled it, we found that most of the space in the box was empty. If we can make full use of space, the game can be greatly improved in portability (although the portability may be not that important for a family game box). For the color part, we did take Jeffrey and Stella’s preferences into account and set up keyboard lighting based on their favorite colors. It might be a better idea to let players themselves choose the own lighting colors.

We didn’t have much experience working with older people before, but the process of working with Jeffrey was very pleasant. He was very talkative and provided us with many practical suggestions, which helped us adjust in different stages and finally completed a work that satisfied us all. Due to the time limit, our initial interview could only be held in the university cafe. Although we talked a lot, it seemed difficult for us to get Jeffrey’s demand quickly from his conversation, which made us struggle about where to start in the brainstorm. Next time we have the opportunity to cooperate with the elderly, maybe we will choose their home as the interview place. On the one hand, it can make them feel more comfortable and be willing to talk more, on the other hand, it is more convenient for us to observe the details of their life so that we can understand their needs more quickly.

All in all, this project is an interesting interdisciplinary cooperation experience. We had a lot of brainstorming in the early stage and repeatedly discussed and improved the design scheme according to Jeffrey and Zach’s feedback. In the post-fabrication stage, we divided the work according to everyone’s background and expertise, and finally achieved satisfactory results. If you have any more questions or suggestions, please feel free to contact us by email: jianxiag@andrew.cmu.edu, thanks!

Technical details

Schematic

Code

/* 
 *  This code makes reference to sample code of Adafruit seesaw liborary
 * whose source code can be viewed at https://github.com/adafruit/Adafruit_Seesaw
 * 
 * Game Logic part Author: Caroline Sun & Jianxiao Ge
 * This is a game is similar to Bingo. To create a enjoyable gaming experience, we add extra
 * features like playing sound sound to the game.
 */
 
#include "Adafruit_NeoTrellis.h"
#include <LiquidCrystal_I2C.h>
LiquidCrystal_I2C lcd(0x27, 16, 2);

#define DEBUG 1

// TONES  ==========================================
// Start by defining the relationship between 
//       note, period, &  frequency. 
#define  c     3830    // 261 Hz 
#define  d     3400    // 294 Hz 
#define  e     3038    // 329 Hz 
#define  f     2864    // 349 Hz 
#define  g     2550    // 392 Hz 
#define  a     2272    // 440 Hz 
#define  b     2028    // 493 Hz 
#define  C     1912    // 523 Hz 
// Define a special note, 'R', to represent a rest
#define  R     0

#define Y_DIM 8 //number of rows of key
#define X_DIM 8 //number of columns of keys

//Define game states
#define QUESTION 0
#define ANSWER 1
#define NEXT 2

//Define Answer to Questions
#define YESJ 2
#define YESS 4
#define NOJ 3
#define NOS 5

//Define button pins
#define RESET 6
#define NEXTJ 5
#define NEXTS 6

//Define player states
#define STELLA 0
#define JEFFREY 1

//Define Board Value
#define MIDDLE 3
#define INVALID (-1)
#define BINGOSIZE 5

//Define speaker pins
#define SPEAKER 9

//Color value
int val = 125;

//Declare game States
int isStart;
int state;
int player;
int currSquare;
int scoreS;
int scoreJ;
int SIZE = 8 ;
int totalSize = SIZE * SIZE;
int board[64];

// Set overall Speaker configuration
long tempo = 10000;
int pause = 1000;
int rest_count = 100;


// MELODY and TIMING  =======================================
//  melody[] is an array of notes, accompanied by beats[], 
//  which sets each note's relative length (higher #, longer note) 

//This is the tune of correct answer
int successMelody[] = {  c,  e,  g,  C, C, C};
int successBeats[]  = { 8, 8, 8,  8, 8,8}; 
int SUCCESS_MAX_COUNT = sizeof(successMelody) / 2; 

//This is the tune of wrong answer
int failureMelody[] = {  g,  e,  d,  c, c};
int failureBeats[]  = { 8, 8,  8,  8, 8}; 
int FAILURE_MAX_COUNT = sizeof(failureMelody) / 2; 

//This is the tune of winning
int winMelody[] = {  e,  e,  e,  e, f, f, f, f,  g,g, g, g, g, g, g, g, f, f, f, f,  e,e, e, e, d, d, d, d};
int winBeats[]  = { 16, 16, 16, 16, 16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16};
int WIN_MAX_COUNT = sizeof(winMelody) / 2; 

//This is the tune of reset
int resetMelody[] = {  c,  c,  c,  c, g, g, g, g,  c,  c,  c,  c, g, g, g, g};
int resetBeats[]  = { 16, 16, 16, 16, 16,16,16,16, 16, 16, 16, 16,16,16,16,16};
int RESET_MAX_COUNT = sizeof(resetMelody) / 2; 


// PLAY TONE FUNCTION  ==============================================
// Pulse the speaker to play a tone for a particular duration
void playTone(int tone_, int beat, int duration) {
  long elapsed_time = 0;
  if (tone_ > 0) { // if this isn't a Rest beat, while the tone has 
    //  played less long than 'duration', pulse speaker HIGH and LOW
    while (elapsed_time < duration) {

      digitalWrite(SPEAKER,HIGH);
      delayMicroseconds(tone_ / 2);

      // DOWN
      digitalWrite(SPEAKER, LOW);
      delayMicroseconds(tone_ / 2);

      // Keep track of how long we pulsed
      elapsed_time += (tone_);
    } 
  }
  else { // Rest beat; loop times delay
    for (int j = 0; j < rest_count; j++) { // See NOTE on rest_count
      delayMicroseconds(duration);  
    }                                
  }                                 
}

//Play Melody function
void playMelody(int* pitches, int* beats, int count){
  // Set up a counter to pull from melody[] and beats[]
  int tone_,beat,duration;
  for (int i=0; i<count; i++) {
    tone_ = pitches[i];
    beat = beats[i];

    duration = beat * tempo; // Set up timing

    playTone(tone_,beat,duration); 
    // A pause between notes...
    delayMicroseconds(pause);
  }
}

//create a matrix of trellis panels
Adafruit_NeoTrellis t_array[Y_DIM/4][X_DIM/4] = {
  
  { Adafruit_NeoTrellis(0x2E), Adafruit_NeoTrellis(0x2F) },

  { Adafruit_NeoTrellis(0x30), Adafruit_NeoTrellis(0x31) }
  
};


//pass this matrix to the multitrellis object
Adafruit_MultiTrellis trellis((Adafruit_NeoTrellis *)t_array, Y_DIM/4, X_DIM/4);


//Print Board Function for testing
void printBoard(){
  Serial.println("board");
  for(int i = 0;i < totalSize;i++){
    Serial.print(board[i]);
    Serial.print(" ");
  }
  Serial.println("");
}

//Helper function that cleans up a array
void cleanArr(int *arr, int n){
  for(int i = 0; i < n;i++){
    arr[i] = 0;
  }
}


//Write a two-line message to the LCD screen
void writemessage(String message1, String message2){
   lcd.backlight();
   lcd.setCursor(0, 0);
   lcd.print(message1);
   lcd.setCursor(0, 1);
   lcd.print(message2);
}

//Turn on the button first being pressed
int turnOnFirstPressed(int num){
  int val = 125;
  if(player == JEFFREY) val = 255;
  trellis.setPixelColor(num, Wheel(val));
  trellis.show();
  return num;
}


// Input a value 0 to 255 to get a color value.
// The colors are a transition r - g - b - back to r.
uint32_t Wheel(byte WheelPos) {
  if(WheelPos < 85) {
   return seesaw_NeoPixel::Color(WheelPos * 3, 255 - WheelPos * 3, 0);
  } else if(WheelPos < 170) {
   WheelPos -= 85;
   return seesaw_NeoPixel::Color(255 - WheelPos * 3, 0, WheelPos * 3);
  } else {
   WheelPos -= 170;
   return seesaw_NeoPixel::Color(0, WheelPos * 3, 255 - WheelPos * 3);
  }
  return 0;
}


//Check if one index is in bound of a board
bool inBound(int x, int y, int width, int height){
  return x>=0 && y >=0 && x < width && y <height; 
}

//define a callback for key presses
TrellisCallback blink(keyEvent evt){
  
  if(evt.bit.EDGE == SEESAW_KEYPAD_EDGE_RISING){
    //Wheel(map(evt.bit.NUM, 0, X_DIM*Y_DIM, 0, 255))
    Serial.print("blink! ");
    Serial.print(evt.bit.NUM);
    Serial.print(" ");
    Serial.print(val);
    Serial.println(" prit");
    trellis.setPixelColor(evt.bit.NUM, Wheel(val));
    val = val>125? 125: 225;
    
  }//on rising
  else if(evt.bit.EDGE == SEESAW_KEYPAD_EDGE_FALLING)
    trellis.setPixelColor(evt.bit.NUM, 0); //off falling
    
  trellis.show();
  return 0;
}

//The function that checks if there exists winner on the board
bool checkWinner(){
  int rowCorrect[2];
  int colCorrect[2];
  int diagDownCorrect[2];
  int diagUpCorrect[2];
  int currCont[2] = {0,0};
  
  cleanArr(rowCorrect,2);
  cleanArr(colCorrect,2);
  cleanArr(diagUpCorrect,2);
  cleanArr(diagDownCorrect,2);
  cleanArr(currCont, 2);

  //Check if there is bingo in a row
  for(int i = 0; i < SIZE; i++){
    for(int j = 1; j < SIZE; j++){
      int currP = board[i * SIZE + j];
      if(board[i * SIZE + j] == board[i * SIZE + j -1] && board[i * SIZE + j]!=INVALID){
        currCont[currP] += 1;
        if(currCont[currP] > rowCorrect[currP]) rowCorrect[currP] = currCont[currP];
      }
      else{
        cleanArr(currCont, 2);
      }
    }
  }

  cleanArr(currCont, 2);
  //Check if there is Bingo in a column
  for(int j = 0; j < SIZE; j++){
    for(int i = 1; i < SIZE; i++){
      int currP = board[i * SIZE + j];
      if(board[i * SIZE + j] == board[(i-1) * SIZE + j] && board[i * SIZE + j]!=INVALID ){
        currCont[currP] += 1;
        if(currCont[currP] > colCorrect[currP]) colCorrect[currP] = currCont[currP];
      }
      else{
        cleanArr(currCont, 2);
      }
    }
  }


  //Check if there is bingo in a diagonal
  for(int xShift = -SIZE +1; xShift<SIZE;xShift++){
  cleanArr(currCont, 2);
  for(int diagX = 1; diagX<SIZE;diagX++){
    int xPos = diagX + xShift;
    int yPos = diagX;
    int prevX = xPos -1;
    int prevY = yPos -1;
    if(!inBound(xPos,yPos,SIZE,SIZE) || !inBound(prevX,prevY,SIZE,SIZE)){
      cleanArr(currCont,2);
      continue;
    }
    int currP = board[yPos * SIZE + xPos];
    if(currP == board[prevY * SIZE + prevX] && currP!=INVALID){
        currCont[currP] += 1;
        if(currCont[currP] > diagDownCorrect[currP]) diagDownCorrect[currP] = currCont[currP];
      }
      else{
        cleanArr(currCont, 2);
      }
  }
  }

  for(int xShift = 0; xShift<2 * SIZE -1;xShift++){
  cleanArr(currCont, 2);
  for(int diagX = 1; diagX<SIZE;diagX++){
    int xPos = -diagX + xShift;
    int yPos = diagX;
    int prevX = xPos +1;
    int prevY = yPos -1;
    if(!inBound(xPos,yPos,SIZE,SIZE) || !inBound(prevX,prevY,SIZE,SIZE)){
      cleanArr(currCont,2);
      continue;
    }
    int currP = board[yPos * SIZE + xPos];
    if(currP == board[prevY* SIZE + prevX] && currP!=INVALID){
        currCont[currP] += 1;
        if(currCont[currP] > diagUpCorrect[currP]) diagUpCorrect[currP] = currCont[currP];
      }
      else{
        cleanArr(currCont, 2);
      }
  }
  }

  int sScore = max(colCorrect[STELLA],max(rowCorrect[STELLA],max(diagDownCorrect[STELLA],diagUpCorrect[STELLA])));
  int jScore = max(colCorrect[JEFFREY],max(rowCorrect[JEFFREY],max(diagDownCorrect[JEFFREY],diagUpCorrect[JEFFREY])));
  
  //Display the winner of the game
  if(sScore >= BINGOSIZE-1 && jScore >=BINGOSIZE-1){
    Serial.println("Both of you wins!");
    writemessage("  Both of you  ","    Wins!   ");
    playMelody(winMelody,winBeats,WIN_MAX_COUNT);
    return true;
  }
  if(sScore >= BINGOSIZE-1){
    Serial.println("Pink wins!");
    writemessage("     Pink       ","     Wins!      ");
    playMelody(winMelody,winBeats,WIN_MAX_COUNT);
    return true;
  }
  if(jScore >= BINGOSIZE-1){
    Serial.println("Green wins!");
    writemessage("     Green    ","      Wins!      ");
    playMelody(winMelody,winBeats,WIN_MAX_COUNT);
    return true;
  }
  return false;
}

//The callBack function on the Trellis Buttons
//This function tells the Button to update the game state, 
//if it is pressed when someone is answering question
TrellisCallback gameControl(keyEvent evt){
  if(isStart){
      // go through every button
   if(state == QUESTION){
      if(evt.bit.EDGE == SEESAW_KEYPAD_EDGE_RISING){
        currSquare = turnOnFirstPressed(evt.bit.NUM);
        if(currSquare>=0){
          state = ANSWER;
        }
     }
   }
  }
}

//Setup the game
void setup() {
  Serial.begin(9600);
  //while(!Serial);

  if(!trellis.begin()){
    Serial.println("failed to begin trellis");
    while(1);
  }

  //Register all buttons
  pinMode(YESJ,INPUT_PULLUP);
  pinMode(YESS,INPUT_PULLUP);
  pinMode(NOJ,INPUT_PULLUP);
  pinMode(NOS,INPUT_PULLUP);
  pinMode(NEXTJ,INPUT_PULLUP);
  pinMode(NEXTS,INPUT_PULLUP);
  pinMode(RESET,INPUT_PULLUP);

  //Register speaker
  pinMode(SPEAKER,OUTPUT);

  //Initialize LCD
  lcd.init();                      // initialize the lcd
  // Print a message to the LCD.
  lcd.backlight();
  lcd.setCursor(0, 0);
  lcd.print("Pink & Green");
  lcd.setCursor(0, 1);
  lcd.print("Welcome to BINGO");

  //Clear board
  for(int i = 0; i< SIZE * SIZE;i++){
    board[i] = -1;
  }
  
  //Go through each button on the board and display different color
  for(int i=0; i<Y_DIM*X_DIM; i++){
      trellis.setPixelColor(i, Wheel(map(i, 0, X_DIM*Y_DIM, 0, 255))); 
      trellis.show();
      delay(20);
  }

  //Setup the trellis button pad
  for(int y=0; y<Y_DIM; y++){
    for(int x=0; x<X_DIM; x++){
      //activate rising and falling edges on all keys
      trellis.activateKey(x, y, SEESAW_KEYPAD_EDGE_RISING, true);
      trellis.activateKey(x, y, SEESAW_KEYPAD_EDGE_FALLING, true);
      trellis.registerCallback(x, y,gameControl);
      trellis.setPixelColor(x, y, 0x000000); //addressed with x,y
      trellis.show(); //show all LEDs
      delay(20);
    }
  }

  //Randomize a player to start the game
  float r = random()%2;
  if( r > 0){
    player = STELLA;
  }
  else{
    player = JEFFREY;
  }

  //Reset Game State
  isStart = 1;
  state = QUESTION;
  scoreS = 0;
  scoreJ = 0;
  currSquare = -1;

 //Display welcome message
  writemessage("Pink & Green","Welcome to BINGO");

}


void loop() {
  trellis.read();
  delay(20);

  //If Rest Button is pressed, reset the game
  if(digitalRead(RESET)<1){
    writemessage("     Reset!     ","                ");
    delay(500);
    playMelody(resetMelody,resetBeats,RESET_MAX_COUNT);
    setup();
    return;
  }
  //let the game continue only if the game has not ended yet
  if(isStart){
   //If it is question and question time
   if(state == QUESTION){
    
    //Display messages showing states
    if(player ==STELLA)
      writemessage("   Green asks     ", "  Pink answers    ");
    else
      writemessage("   Pink asks  ", "  Green answers   ");
   }
   //If it is revealing answer state,
   else if(state == ANSWER){
    if(player==STELLA){
      writemessage("   Is Pink's    ", " answer correct? ");
      //If Pink's answer is correct
      if(digitalRead(YESJ)<1){
        board[currSquare] = STELLA;
        Serial.print("Pink places at position: ");
        Serial.println(currSquare);
        printBoard();
        //play music
        playMelody(successMelody,successBeats,SUCCESS_MAX_COUNT);
        //Change game state to next round
        currSquare = -1;
        state = QUESTION;
        isStart = !checkWinner();
        player = !player;
      }
      //If Pink's answer is wrong
      else if(digitalRead(NOJ)<1){
        trellis.setPixelColor(currSquare, 0);
        trellis.show();
        //play music
        playMelody(failureMelody,failureBeats,FAILURE_MAX_COUNT);
        //change game state to next round
        currSquare = -1;
        state = QUESTION;
        isStart = !checkWinner();
        player = !player;
      }
    }
    else{
      writemessage(" Is Green's  ", "answer correct? ");
      //If Green's answer is correct
      if(digitalRead(YESS)<1){
        board[currSquare] = JEFFREY;
        Serial.print("Green places at position: ");
        Serial.println(currSquare);
        printBoard();
        playMelody(successMelody,successBeats,SUCCESS_MAX_COUNT);
        currSquare = -1;
        state = QUESTION;
        isStart = !checkWinner();
        player = !player;
      }
      //If Green's answer is wrong
      else if(digitalRead(NOS)<1){
        trellis.setPixelColor(currSquare, 0);
        trellis.show();
        playMelody(failureMelody,failureBeats,FAILURE_MAX_COUNT);
        currSquare = -1;
        state = QUESTION;
        isStart = !checkWinner();
        player = !player;
        
      }
    }
   }
  }
  
}

Chloé Desaulles

Jiatian Sun

Jianxiao Ge