Interactive iPad Board by Team Richard: Final Documentation

For this project, we were tasked with creating an assistive device for an older person we were paired with from CMU’s Osher Institute. We were paired with Richard and met with him in his home to talk to him more about his day to day life, what tasks he had trouble with, and areas he felt could be improved by the creation of some sort of gadget or device, documentation from which you can find here. Based off of what we learned from our discussion with Richard, we began to build an interactive iPad stand that would make his process of making drawings for jewelry commissions easier and more efficient. First, we built a prototype, documentation from which you can find here. We then began building a final model, the documentation for which can be found below.

What We Built:

We built an interactive iPad stand for Richard that allows him to position his iPad in different directions and angles while keeping a log of his different commissions and tracking how long he’s been working on drawings for each one. On the mechanical spectrum, the stand is comprised of a baseboard made of wood, on which a joint that allows the top plate to move up and down, into which a ball and socket joint is screwed, which connects to the top board via a plate screwed into the plastic of the top material. The top board is made of a plastic composite that was milled on both sides, and has slots for Richard’s iPad and Apple Pencil. The electronics consist of a screen, buttons to navigate the menu on the screen, and a rotary encoder to input names for each commission.

Final Images

Photo showing scale and proportion of device from the front.

Overall image showing scale and proportion, and the back of the device.

Close up showing the different pockets and finger slots milled into the front of the board

Up close shot showing the two joints, which connected the base to the top board, in action.

Close up of added acrylic semi circles that could be rotated to keep the iPad from falling out during use. You can also see the paper backing that was glued to the back of where the iPad rests so that the t screws attaching the joint to the board were covered.

Close up of corners on base and top board hand rounded so as to be comfortable to use for a long period of time.

The board can be positioned and different heights and angles, and is adjusted manually, as shown in this photo.

This video shows some of the menu’s functionality, including entering a new project, saving then resuming said project, and then archiving it once it’s finished.

The use for the three buttons depends on which menu screen is being shown. The function of each button is shown above it on the screen.

A rotary encoder was used to input names for each project that was being timed. These names were inputted on the “Start New” screen.

The device can be placed on your lap or on a table, and your iPad rests into the board while you draw. The pocket in the board is milled for Richard’s 12″ iPad, the one being used for reference in this photo is a 9″ iPad.

Here’s a hypothetical situation for its use: Richard and his wife have just finished dinner and now are sitting on the couch watching the news, as they do many nights. Richard wants to work on drawings for a new jewelry commission of a necklace charm he’s just received for a client, Anna, so he grabs his iPad stand, puts it on his lap and begins.

He starts by turning on the screen embedded in the top board of the stand, and navigates to input a new project. Using the rotary encoder, he names the project “Anna”, hits the center button to go navigate to and start the timer, and begins to draw on his iPad. When he’s finished drawing for the night, he hits the “Archive” button, then switches the screen off. The next night, he starts up the timer where it left off and continues to draw Anna’s necklace charm.

How We Got Here:

In our chart we’d hoped to have the electronic and design components prototyped much earlier in the process so we could spend time at the end combining the two, doing lots of testing, and showing a functional MDF prototype to Richard before moving to the final material. We ended up deviating significantly from the timeline we set up in our Gantt chart. Our software was much more complicated and time consuming to write than expected, and it took longer than we’d expected for the mills to be completed due to the number of people using the mill, so we ended up working on both the electronic and design portions of the device up until the end of the project.

Initial sketch I used as a guideline to create the 3D model for milling the back of the board for the electronics to fit into.

It was challenging to update the design for the final mill into the new plastic we were using because of time restraints. Because of the line to use the CNC machine, we needed to submit the final mill file the day after we got the prototype fully milled for it to be finished in time. The other was that we weren’t able to get our electronics wired together as quickly as we’d expected due to software issues, so we weren’t able to mount them into the MDF prototype and actually test all of the components together as we’d initially hoped to.

Notes from measuring the electronics against the MDF prototype.

When we milled the back of the MDF prototype, we did our best to gauge what measurements were correct and which should be changed. I realized it would be difficult to mill at a depth where everything would be submerged in the material, and changed my design for the back to include a larger pocket for the electronics to be drilled into. Because of this, I could no longer plan to just screw an acrylic cover over the components and instead began to plan a vacuum-formed plastic piece that would cover the components.

The plastic composite we were using for our final prototype after being milled, before any hand finishing.

Our functional design/mechanical prototype featuring a baseboard made from scrap wood, but that kept the device from tipping, and a joint that interfaced with the top board and allowed for a great degree of movement for positioning.

Our initial attempt at combining the electronic and design components of the device immediately revealed some problems with fitting the pieces together.

A lot of finishing work went into the design, including rounding all the edges and sanding both the base and the top board, and spray painting the top, making a paper backing to cover the t-screws holding the top board in place, and paper, acrylic, and vacuum formed styrene covers for the electronics. If I’d done this a second time I would have made the top board grey so that it wouldn’t get dirty so quickly.

During crit we had Richard hold the device on his lap as he might at home. From his feedback, we decided to make the baseboard a bit wider, and to round the edges of the wood more for his comfort.

Due to wrong initial dimensions and changing components, some of what was milled needed to be altered. I did this by dremmeling the pockets/holes until they were roughly the desired size/shape. The screen pocket was just an issue of depth for the component, not for its acrylic cover, so I dremmeled a deeper pocket for the component, then used a glue specifically for acrylic to bond the component to it’s cover, and screwed the cover into the plastic.

The screen pocket and hole for the power switch after being dremmeled, before their new components were added back in.

We encountered a lot of issues with the electronics, particularly after we’d solder them all together. We’d initially soldered all of our wires into the Arduino, but after we continued to have problems, we ended up cutting the wires and moving back to sticking the wires into the header pins mounted on the arduino.

When working on developing the display software, we encountered difficulties with the internal arduino timing and the rotary input

After many hours analyzing the code to determine what was causing issues with our rotary input, we determined that it was due to a library conflict, and rewrote the code to utilize a different library- resulting in an encoder that “worjs”.

Developing the software was a lot more difficult than anticipated. Previous to receiving the screen, we focused on creating separate files of code for the logic of each individual function of the device- rotary input, timing, and data storage. Unfortunately, combining perfectly functional separate programs into one doesn’t always work out perfectly; we experienced library and compilation conflicts that caused us to have to re-write large sections of code for it to be compatible with other functions of the device.

Conclusions

While a lot of the feedback we got from the crit revolved around things we already knew were issues – in particular, that “some components weren’t secured/working right”, and that because our ball socket wore out during the crit the top board was “floppy”, some of the feedback brought up things we hadn’t even considered. Someone brought up that we should “consider whether the client was right or left handed” because our current positioning of the electronics was on the left side of the board.

Two people brought up the issue that maybe using an app or timer already on the iPad would be more effective, with someone saying that “the UI for tracking projects was clunky and would be better solved by an app on the iPad”. While this is a good point, and there may be more sophisticated timer options in the app store, this was something we’d talked about with Richard when we’d first met with each other, and while he acknowledged that it was something he could do on his iPad, he liked the idea of it being somewhere he wouldn’t have to minimize his drawing app to access.

It was interesting to work with Richard and to create something that tailored to his needs and interests specifically. Making something for the purpose of a crit is very different from the process of making something to last someone for a while. I think if we were able to get a stronger ball and socket joint, as the one we used wore out incredibly quickly, the mechanical and design aspects of this project would last a good amount of time and be reliable for him to draw on.
We went into this process thinking both the design and the electronic aspects were a lot less complicated than they actually ended up being, and so we didn’t front-load the work as much as we should have. Our design elements ended up needing a lot more finishing work than was initially expected, and because we were handing it off to a client as opposed to it just being for crit, I felt more compelled to do the work to make it look nice and functional. If we were to do this project again knowing what we know now, I think we would make a big effort to frontload as much as possible, and get electronics and a basic design prototype in place as quickly as we could so that we could focus more on testing and making this stable enough to last Richard a long time.

Technical Details

Code Submission

/***************************************************
  Adrien Krupenkin, 2019
  Operates a menu-based system that allows the user to 
  input and track projects they are working on, to be displayed
  on a TFT screen.
 ****************************************************/


#include "SPI.h"
//#include "Adafruit_GFX.h"
#include "Adafruit_ILI9341.h"
#include <Encoder.h>
#include <Fonts/FreeSerif24pt7b.h>
#include <Fonts/FreeSans18pt7b.h>
#include <Fonts/FreeSans12pt7b.h>
const int button1Pin = 5;
const int button2Pin = 6;
const int button3Pin = 7;
const int offPin = 3;
int menuPage = 0;
int currentprojectNumber = 0;
int pastprojectNumber = 0;
int scrollVal = 0;
int positionVal;
bool button1Val = 1;
bool button2Val = 1;
bool button3Val = 1;
bool onoff = 1;
char letter;
const int encoderA = 2; // CLK pin of Rotary Enocoder
const int encoderB = 3; // DT pin of Rotary Enocoder
const int BUT_ENTER = 4; // button pin of Rotary Enocoder
String projects[15];
String pastprojects[32];
String projectTimes[15];
String pastprojectTimes[32];
// variables for rotary encoder
int aLastState;
// didn't implement debouncing
int encoder_pos = 1;
int n = 0;
int seconds[15]={0,0,0,0,0,0,0,0,0,0,0,0,0,0,0};
int minutes[15]={0,0,0,0,0,0,0,0,0,0,0,0,0,0,0};
int hours[15]={0,0,0,0,0,0,0,0,0,0,0,0,0,0,0};
int lim = 0;
unsigned long count;
unsigned long offcount = 0;
bool timer[15];

// letters, numbers, space, characters
const int str_len = 44;
char alphabet[44] = "abcdefghijklmnopqrstuvwxyz0123456789 !?,.:-";
String message = "";

// Declare encoder (from library)
Encoder encoder(encoderA, encoderB);



// For the Adafruit shield, these are the default.
#define TFT_DC 9
#define TFT_CS 10

// Use hardware SPI (on Uno, #13, #12, #11) and the above for CS/DC
Adafruit_ILI9341 tft = Adafruit_ILI9341(TFT_CS, TFT_DC);


void setup() {
  pinMode(offPin, INPUT_PULLUP);
  pinMode(BUT_ENTER, INPUT_PULLUP);
  Serial.begin(9600);
  aLastState = digitalRead(encoderA); //initialize variable for rotary encoder
  Serial.println("ILI9341 Test!");
  pinMode(button1Pin, INPUT_PULLUP);
  pinMode(button2Pin, INPUT_PULLUP);
  pinMode(button3Pin, INPUT_PULLUP);
  tft.begin();
  int encoder_pos;
  tft.fillScreen(ILI9341_BLACK);
  tft.setFont(&FreeSans18pt7b);

}


void loop(void) {

  //Home Screen
  if (menuPage == 0) {
    tft.setRotation(1);
    tft.setCursor(20, 60);
    tft.setTextColor(ILI9341_GREEN);  tft.setTextSize(1);
    tft.setFont(&FreeSerif24pt7b);
    tft.println("Hello Richard!");
    tft.setFont(&FreeSans18pt7b);
    tft.setCursor(35, 115);
    tft.setTextColor(ILI9341_WHITE); tft.setTextSize(1);
    tft.println("What would you");
    tft.setCursor(35, 145);
    tft.println("like to do today?");
    tft.setFont(&FreeSans12pt7b);
    tft.setTextColor(ILI9341_YELLOW);    tft.setTextSize(1);
    tft.setCursor(15, 215);
    tft.println("Resume");
    tft.setCursor(135, 215);
    tft.println("New");
    tft.setCursor(215, 215);
    tft.println("History");
    button1Val = digitalRead(button1Pin);
    button2Val = digitalRead(button2Pin);
    button3Val = digitalRead(button3Pin);
    if (button1Val == 0) {
      menuPage = 1;
      tft.fillScreen(ILI9341_BLACK);
    }
    if (button2Val == 0) {
      menuPage = 3;
      tft.fillScreen(ILI9341_BLACK);
    }
    if (button3Val == 0) {
      menuPage = 4;
      tft.fillScreen(ILI9341_BLACK);
    }
  }

  //Resume Menu
  if (menuPage == 1) {
    tft.setRotation(1);
    tft.setCursor(20, 60);
    tft.setTextColor(ILI9341_GREEN);  tft.setTextSize(1);
    tft.setFont(&FreeSerif24pt7b);
    tft.println("Project Menu");
    if (0 <= n && n <= currentprojectNumber)
    {
      tft.setFont(&FreeSans12pt7b);
      tft.setTextColor(ILI9341_WHITE);    tft.setTextSize(1);
      positionVal = positionVal + 20 - scrollVal;
      tft.setCursor(15, 115);
      tft.print(projects[n]);
      //will later edit to list name and time
    }
    tft.setCursor(15, 215);
    tft.setFont(&FreeSans12pt7b);
    tft.setTextColor(ILI9341_YELLOW);    tft.setTextSize(1);
    tft.println("Previous");
    tft.setCursor(115, 215);
    tft.println("Select");
    tft.setCursor(215, 215);
    tft.println("Next");
    button1Val = digitalRead(button1Pin);
    button2Val = digitalRead(button2Pin);
    button3Val = digitalRead(button3Pin);
    if (button1Val == 0) {
      n = n - 1;
      tft.fillScreen(ILI9341_BLACK);
    }
    if (button2Val == 0) {
      menuPage = 2;
      count = 0;
      tft.fillScreen(ILI9341_BLACK);

    }
    if (button3Val == 0) {
      scrollVal = scrollVal - 25;
      tft.fillScreen(ILI9341_BLACK);
      n = n + 1;
    }
  }

  //Resume Project
  if (menuPage == 2) {
    tft.setRotation(1);
    tft.setCursor(20, 60);
    tft.setTextColor(ILI9341_GREEN);  tft.setTextSize(1);
    tft.setFont(&FreeSerif24pt7b);
    tft.println(projects[n]);
    tft.setFont(&FreeSans18pt7b);
    tft.setCursor(35, 115);
    tft.setTextColor(ILI9341_WHITE); tft.setTextSize(1);

    if (timer[n] == 1) {
      if (count + 1000 < millis()) {
      //timer operation
    
        //delay (1000);
        seconds[n] = seconds[n] + 1;
        if (seconds[n] == 60)
        {
          seconds[n] = 0;
          minutes[n] = minutes[n] + 1;
        }
        if (minutes[n] == 60)
        {
          minutes[n] = 0;
          hours[n] = hours[n] + 1;
        }
        String throwawaystring = " ";
        String colon = ":";
        String timercount = throwawaystring + hours[n] + colon + minutes[n] + colon + seconds[n];
        projectTimes[n] = timercount;
        tft.fillRect(35, 80, 160, 50, ILI9341_BLACK);
        tft.setCursor(35, 115);
        count = millis();
        tft.setTextColor(ILI9341_WHITE); tft.setTextSize(1);
        tft.println(projectTimes[n]);
      }
    }
    tft.setFont(&FreeSans12pt7b);
    tft.setTextColor(ILI9341_YELLOW);    tft.setTextSize(1);
    tft.setCursor(15, 215);
    tft.println("Home");
    tft.setCursor(115, 215);
    tft.println("Timer");
    tft.setCursor(215, 215);
    tft.println("Archive");
    button1Val = digitalRead(button1Pin);
    button2Val = digitalRead(button2Pin);
    button3Val = digitalRead(button3Pin);
    if (button1Val == 0) {
      menuPage = 0;
      tft.fillScreen(ILI9341_BLACK);
    }
    if (button2Val == 0) {
      timer[n] = !timer[n];
      //timer operation
    }
    if (button3Val == 0) {
      menuPage = 4;
      tft.fillScreen(ILI9341_BLACK);
      pastprojectNumber = pastprojectNumber + 1;
      pastprojects[pastprojectNumber] = projects[n];
      pastprojectTimes[pastprojectNumber] = projectTimes[n];
      for (int j = n; j <= currentprojectNumber; j++) {
        projects[j] = projects[j + 1];
        projectTimes[j] = projectTimes[j + 1];
        seconds[j]=seconds[j + 1];
        minutes[j]=minutes[j + 1];
        hours[j]=hours[j + 1];
      }
      currentprojectNumber = currentprojectNumber - 1;
      //save project time and add to list
    }
  }

  //New Project
  if (menuPage == 3) {
    tft.setRotation(1);
    tft.setCursor(20, 60);
    tft.setTextColor(ILI9341_GREEN);  
    tft.setTextSize(1);
    tft.setFont(&FreeSerif24pt7b);
    tft.println("Start New");
    tft.setFont(&FreeSans18pt7b);
    tft.setTextColor(ILI9341_YELLOW);  tft.setTextSize(1);
    //Rotary code here

    int newPos = encoder.read();
    Serial.println(newPos);
    lim = newPos/4 % str_len;
    // lim is the index of the characters in the variable "alphabet"
    if (lim < 0) {
      lim += str_len; // loop around to the first index of the characters, 
      // so you don't have to move all the way back from z to a
    }
    if (encoder_pos != newPos) { // if encoder is being rotated
      //lim = newPos;
      // lim is the index of the characters in the variable "alphabet"
      encoder_pos = newPos;
      char letter = alphabet[lim];
      tft.setFont(&FreeSans18pt7b);
      tft.setTextColor(ILI9341_RED);  tft.setTextSize(1);
      tft.setCursor(100, 100);
      tft.fillRect(100, 75, 30, 40, ILI9341_BLACK);
      tft.print(letter);
      Serial.println("here");
    }
    if (digitalRead(BUT_ENTER) == 0) { // if encoder button is pressed, i.e. to select a char
      delay(500); // so you don't accidentally select the letter many times
      message = message + alphabet[lim]; // add the selected letter to the message
      tft.setCursor(30, 135);
      tft.setTextColor(ILI9341_WHITE);  tft.setTextSize(1);
      tft.println(message);
      Serial.print(message);
    }

    tft.setFont(&FreeSans12pt7b);
    tft.setTextColor(ILI9341_YELLOW);    tft.setTextSize(1);
    tft.setCursor(115, 215);
    tft.println("Enter");
    tft.setCursor(215, 215);
    tft.println("Cancel");
    button1Val = digitalRead(button1Pin);
    button2Val = digitalRead(button2Pin);
    button3Val = digitalRead(button3Pin);
    if (button1Val == 0) {
      //backspace operation
    }
    if (button2Val == 0) {
      menuPage = 1;
      projects[currentprojectNumber] = message;
      message = "";
      currentprojectNumber = currentprojectNumber + 1;
      tft.fillScreen(ILI9341_BLACK);
    }
    if (button3Val == 0) {
      menuPage = 0;
      tft.fillScreen(ILI9341_BLACK);
    }
  }

  //History Menu
  if (menuPage == 4) {
    positionVal = 100 - scrollVal;
    tft.setRotation(1);
    tft.setCursor(20, 60);
    tft.setTextColor(ILI9341_GREEN);  tft.setTextSize(1);
    tft.setFont(&FreeSerif24pt7b);
    tft.println("History");
    for (int y = 0; y <= pastprojectNumber; y++)
    {
      tft.setFont(&FreeSans12pt7b);
      tft.setTextColor(ILI9341_WHITE);    tft.setTextSize(1);
      if (60 < positionVal && positionVal < 215) {
        tft.setCursor(15, positionVal);
        tft.print(pastprojects[y]);
        tft.print(pastprojectTimes[y]);
        //will later edit to list name and time
      }
      positionVal = positionVal + 25;
    }

    tft.setFont(&FreeSans12pt7b);
    tft.setTextColor(ILI9341_YELLOW);    tft.setTextSize(1);
    tft.setCursor(15, 215);
    tft.println("Up");
    tft.setCursor(115, 215);
    tft.println("Home");
    tft.setCursor(215, 215);
    tft.println("Down");
    button1Val = digitalRead(button1Pin);
    button2Val = digitalRead(button2Pin);
    button3Val = digitalRead(button3Pin);
    if (button1Val == 0) {
      scrollVal = scrollVal + 25;
      tft.fillScreen(ILI9341_BLACK);
    }
    if (button2Val == 0) {
      menuPage = 0;
      scrollVal = 0;
      tft.fillScreen(ILI9341_BLACK);
    }
    if (button3Val == 0) {
      scrollVal = scrollVal - 25;
      tft.fillScreen(ILI9341_BLACK);
    }
  }

}