Final Project: Emotionally Intelligent Home Assistant

Abstract: Existing home assistants improve interaction in the home by using speech recognition and intelligent natural language processing. However, they lack the emotional intelligence needed to listen, motivate and support the user and provide support towards a healthy mind. My project proposes the use of a custom google assistant through AIY voice kit to detect emotion, provide social support and generate empathy in the user.

The project builds on existing technology and can potentially replace the current Google home assistant.

Hardware:

AIY Voice kit and a mobile browser


Systems Diagram:

DEMO:

References

 

Smart Desk

The goal of this project is to add interactive functionality to an adjustable standing desk. Countless times my environment interrupts my workflow when I’m at home. I use my room as a home office, and it is important that my housemates and friends know when I am on an important call and cannot be interrupted. At the same time, I often find myself getting carried away and spending hours in front of my task without moving at all. The small systems on this desk aim to both aid the user in time management, but also help remind them that it is healthy to stop and take breaks.

 

Three Main Systems

1.A ultrasonic sensor is visible under the table and will detect if a person is standing at the desk. If there is someone there, it triggers a neopixel light to turn on. If someone is standing there for more than 52 minutes, the neopixel turns red. This is meant to remind them to take a short break from the given task before continuing.

Note: the timer for the standing person is set to 5 seconds for demo purposes. If used for real tasks, the timer would be set for 52 minutes.

2. A task management system uses a RFID reader and RFID credit cards to set timers on the users task. The user swipes a box across the reader and a timer is set. This timer is visualized by the LEDs inside of the box turning on. When the time is about to end for the given task, the LEDs will blink before turning off. The user is only permitted to light up one task at a time, so they have a visual reminder of which task they should be working on. I wanted to use color-blind friendly colors, but our lab ran out of led colors.

Note: the timer for each task is set to 5 seconds for demo purposes. If used for real tasks, the timer can be set to the time the user desires to spend on each task.

3. A led, ideally placed outside of your office door, is used to tell your housemates or if it is okay for them to come in and interrupt you or not. The switch would be placed on the desk and the user will choose to set it to green or red.

Actual Project:

Ultrasonic sensor

RFID task organizer

RFID Reader Wiring

Inside card box

Busy/free switch 

 

Smart Desk 

Future Work

While my desk was a simple prototype of a potential product, there are a few future ideas that resulted from the creation process.

  • Using the task box light system in an open office space to keep your team organized. Oftentimes, teams are working on multiple projects in parallel and this system could be used to visually communicate which project each individual is currently working on.
  • Add a temperature controlling functionality (a fan that automatically blows when the room reaches a certain temperature)
  • Expand the neopixel light system so it can be used as the main source of light while working. This will make the “take a break” alert more prominent and harder to avoid. I also want to make the alert blink as a warning before fully turning red.

Tap Happy

Final Presentation Setup:

 

^ Two patches, one has two tappable areas, the other has three.

^ Swapping out the patches to test.

^ Design samples on the various colors of canvas. The texture was a bit too rough for this application.

     

^ Process of making the patches

First Tests with taps + conductive ink:

Example of two user animations, created on presentation day:

Arduino Code:
//--------CAP1188 SETUP --------
#include
#include
#include
#define CAP1188_SENSITIVITY 0x1F // Setting the sensitivity....lower the number the more sensitive!!
static const int CAP1188_RESET = 12; // Reset Pin is used for I2C or SPI
Adafruit_CAP1188 cap = Adafruit_CAP1188();
//------------------------------

const int numOfCaps = 3;

//-----CAP TOUCH Variables-----
uint8_t touched;
uint8_t light;
int capState[numOfCaps] = {0, 0, 0};
int lastCapState[numOfCaps] = {0, 0, 0};
int capCurrent[numOfCaps] = {false, false, false};

//-----TIMING N@T------
int pinState[numOfCaps] = {1, 1, 1}; //debouncing
bool timing[numOfCaps] = {false, false, false};
uint32_t dbTime = 20;
uint32_t eTime[numOfCaps] = {};
uint32_t dbStart[numOfCaps] = {};
uint32_t start[numOfCaps] = {};
uint32_t sendTime[numOfCaps] = {};

//-----TIMING BREAK-----
static const int btnPin = 2;
int btnState;
int lastBtnState;

void setup() {
Serial.begin(115200);
Serial.println("CAP1188 test!");
pinMode(btnPin, INPUT);
// Initialize the sensor, if using i2c you can pass in the i2c address
if (!cap.begin()) {
//if (!cap.begin()) {
Serial.println("CAP1188 not found");
while (1);
}

Serial.println("CAP1188 found!");
cap.writeRegister(CAP1188_SENSITIVITY, 0x6F);

}

void loop() {
touches(); // sensing for touches
for (int i = 0; i < numOfCaps; i++) {
touchOnce(i);
debounceBtn(i);
timeTest(i);
}
endSession();
}

void touches() {
touched = cap.touched();
light = cap.readRegister(0x3);

if (touched == 0) {
// No touch detected
return;
}

for (uint8_t i = 0; i < 8; i++) {
if (touched &amp; (1 &lt;<i> dbTime) {// db timer has elapsed
capState[n] = pinState[n]; // button state is valid
}
//+++++++++++++++++++++++++++++++++++
}</i>

void timeTest(int t) {
if (capState[t] == 0 &amp;&amp; !timing[t]) { //if button state is NOT pressed and timing is NOT happening
sendTime[t] = 0;
start[t] = millis(); // start timing
timing[t] = true;
}

if (capState[t] == 1 &amp;&amp; timing[t]) { //if button state is pressed and timing is happening
eTime[t] = (millis() - start[t]); // record elapsed time
sendTime[t] = eTime[t];
timing[t] = false;
}
}

void endSession() {
// read the pushbutton input pin:
btnState = digitalRead(btnPin);

// compare the buttonState to its previous state
if (btnState != lastBtnState) {
// if the state has changed, increment the counter
if (btnState == HIGH) {
Serial.println("E");
}
// Delay a little bit to avoid bouncing
delay(50);
}
// save the current state as the last state, for next time through the loop
lastBtnState = btnState;
}

void serialButtonCount() {
Serial.print(capCurrent[2]);
Serial.print(",");
Serial.print(capCurrent[1]);
Serial.print(",");
Serial.print(capCurrent[0]);
// Serial.print(" ");
Serial.print(",");
Serial.print(sendTime[2]);
Serial.print(",");
Serial.print(sendTime[1]);
Serial.print(",");
Serial.println(sendTime[0]);
}
Processing Code Two Tap:
import gifAnimation.*;
import processing.serial.*;

float x, y;
color col1, col2, col3;
color[] colorVal = new color[10];
String[] values;

GifMaker gifExport;
int frames = 0;

int bOne; //button one
int bTwo; //button two
int bThree; //button three
int tOne; //time one
int tTwo; //time two
int tThree; //time three

float colMap1, colMap2, colMap3;
float growMap1, growMap2, growMap3;

int endState = 0;
int lastEndState;

int xpos1, ypos1, xpos2, ypos2, xpos3, ypos3;
float radius1 = 100;
float radius2 = 100;
float radius3 = 100;

int pos1x = 200;
int pos1y = 550;
int pos2x = 400;
int pos2y = 250;
int pos3x = 400;
int pos3y = 550;

Circle myCircle;

int num = 0;

void setup() {
size(800, 800);
frameRate(12);
noFill();
ellipseMode(CENTER);
//colorMode(RGB, 255);

//update the serial port index based on your setup
Serial arduino = new Serial(this, Serial.list()[3], 115200);
arduino.bufferUntil('\n');

ellipseMode(RADIUS);
myCircle = new Circle(pos1x, pos1y, pos2x, pos2y, pos3x, pos3y);

gifExport = new GifMaker(this, "export.gif");
gifExport.setRepeat(0); // make it an "endless" animation
//gifExport.setTransparent(0, 0, 0); // make black the transparent color. every black pixel in the animation will be transparent
}

void draw() {

background(240, 240, 240);
noFill();

tapEvent();

fill(col2);
noStroke();
ellipse(pos2x, pos2y, 100, 100);

fill(col3);
noStroke();
ellipse(pos3x, pos3y, 100, 100);
//println(gifExport);
gifExport.setDelay(1);
gifExport.addFrame();

saveGif();
}

void tapEvent() {
colMap1 = map(tOne, 100, 2000, 255, 150);
colMap2 = map(tTwo, 100, 2000, 255, 150);

if (bThree == 1) {
col3 = color(colMap1, colMap2, 255);
myCircle.display();
myCircle.grow();
} else if (bThree == 0) {
col3 = color(240, 240, 240);
}
if (bTwo == 1) {
col2 = color(colMap1, colMap2, 255);
myCircle.display();
myCircle.grow();
} else if (bTwo == 0) {
col2 = color(240, 240, 240);
}
if (bOne == 1) {
col1 = color(colMap1, colMap2, 255);
myCircle.display();
myCircle.grow();
} else if (bOne == 0) {
col1 = color(240, 240, 240);
}
}

void saveGif() {
if (endState != lastEndState) {
// if the state has changed, increment the counter
if (endState == 1) {
boolean finished = gifExport.finish(); //exports gif
num++;
println(finished);
if (finished == true) {
gifExport = new GifMaker(this, "export" +str(num)+ ".gif");
gifExport.setRepeat(0); // make it an "endless" animation
}
endState = 0;
}
// Delay a little bit to avoid bouncing
delay(50);
}
// save the current state as the last state, for next time through the loop
lastEndState = endState;
}

void serialEvent(Serial p) { // code from Rebecca Fienbrink
String rawString = p.readString(); //read the string from serial
rawString = rawString.trim(); //trim any unwanted empty spaces
try { //split the string into an array of 2 value (e.g. "0,127" will become ["0","127"]
values = rawString.split(",");
//for(int i =0; i 1) {
bOne = int(values[0]); //convert strings to int
bTwo = int(values[1]); //convert strings to int
bThree = int(values[2]); //convert strings to int
tOne = int(values[3]);
tTwo = int(values[4]);
tThree = int(values[5]);
//println(values[0] + "," + values[1] + ","+ values[2] );
//println(values[3] + "," + values[4] + ","+ values[5] );
} else if (values.length == 1) {
endState = 1;
//println(end);
} else {
endState = 0;
}
}
catch(Exception e) {
println("Error parsing string from Serial:");
e.printStackTrace();
}
}

class Circle {
// The Constructor is defined with arguments.
Circle(int x1, int y1, int x2, int y2, int x3, int y3) {

xpos1 = x1;
ypos1 = y1;
xpos2 = x2;
ypos2 = y2;
xpos3 = x3;
ypos3 = y3;
}

void display() {
stroke(0);
strokeWeight (3);
noFill();
//ellipse(xpos1, ypos1, radius1, radius1);
strokeWeight (3);
noFill();
ellipse(xpos2, ypos2, radius2, radius2);
strokeWeight (3);
noFill();
ellipse(xpos3, ypos3, radius3, radius3);
}

void grow() {
growMap1 = map(tOne, 100, 8000, 1, 4);
growMap2 = map(tTwo, 100, 8000, 1, 4);
growMap3 = map(tThree, 100, 8000, 1, 4);
//if (bOne == 1) {
// radius1 = radius1 + growMap1;
//}
if (bTwo ==1) {
radius2 = radius2 + growMap2;
}
else if (bThree == 1) {
radius3 = radius3 + growMap3;
}
}
}
Processing Code Three Tap:
import gifAnimation.*;
import processing.serial.*;

float x, y;
color col1, col2, col3;
color[] colorVal = new color[10];
String[] values;

GifMaker gifExport;
int frames = 0;

int bOne; //button one
int bTwo; //button two
int bThree; //button three
int tOne; //time one
int tTwo; //time two
int tThree; //time three

float colMap1, colMap2, colMap3;
float growMap1, growMap2, growMap3;

int endState = 0;
int lastEndState;

int xpos1, ypos1, xpos2, ypos2, xpos3, ypos3;
float radius1 = 100;
float radius2 = 100;
float radius3 = 100;

int pos1x = 200;
int pos1y = 550;
int pos2x = 400;
int pos2y = 250;
int pos3x = 600;
int pos3y = 550;

Circle myCircle;

int num = 0;

void setup() {
size(800, 800);
frameRate(12);
noFill();
ellipseMode(CENTER);
//colorMode(RGB, 255);

//update the serial port index based on your setup
Serial arduino = new Serial(this, Serial.list()[3], 115200);
arduino.bufferUntil('\n');

ellipseMode(RADIUS);
myCircle = new Circle(pos1x, pos1y, pos2x, pos2y, pos3x, pos3y);

gifExport = new GifMaker(this, "export.gif");
gifExport.setRepeat(0); // make it an "endless" animation
//gifExport.setTransparent(0, 0, 0); // make black the transparent color. every black pixel in the animation will be transparent
}

void draw() {

background(240, 240, 240);
noFill();

tapEvent();

fill(col1);
noStroke();
ellipse(pos1x, pos1y, 100, 100);

fill(col2);
noStroke();
ellipse(pos2x, pos2y, 100, 100);

fill(col3);
noStroke();
ellipse(pos3x, pos3y, 100, 100);
//println(gifExport);
gifExport.setDelay(1);
gifExport.addFrame();

saveGif();
}

void tapEvent() {
colMap1 = map(tOne, 100, 2000, 255, 150);
colMap2 = map(tTwo, 100, 2000, 255, 150);
colMap3 = map(tThree, 100, 2000, 255, 150);

if (bThree == 1) {
col3 = color(colMap1, colMap2, colMap3);
myCircle.display();
myCircle.grow();
} else if (bThree == 0) {
col3 = color(240, 240, 240);
}
if (bTwo == 1) {
col2 = color(colMap1, colMap2, colMap3);
myCircle.display();
myCircle.grow();
} else if (bTwo == 0) {
col2 = color(240, 240, 240);
}
if (bOne == 1) {
col1 = color(colMap1, colMap2, colMap3);
myCircle.display();
myCircle.grow();
} else if (bOne == 0) {
col1 = color(240, 240, 240);
}
}

void saveGif() {
if (endState != lastEndState) {
// if the state has changed, increment the counter
if (endState == 1) {
boolean finished = gifExport.finish(); //exports gif
num++;
println(finished);
if (finished == true) {
gifExport = new GifMaker(this, "export" +str(num)+ ".gif");
gifExport.setRepeat(0); // make it an "endless" animation
}
endState = 0;
}
// Delay a little bit to avoid bouncing
delay(50);
}
// save the current state as the last state, for next time through the loop
lastEndState = endState;
}

void serialEvent(Serial p) { // code from Rebecca Fienbrink
String rawString = p.readString(); //read the string from serial
rawString = rawString.trim(); //trim any unwanted empty spaces
try { //split the string into an array of 2 value (e.g. "0,127" will become ["0","127"]
values = rawString.split(",");
//for(int i =0; i 1) {
bOne = int(values[0]); //convert strings to int
bTwo = int(values[1]); //convert strings to int
bThree = int(values[2]); //convert strings to int
tOne = int(values[3]);
tTwo = int(values[4]);
tThree = int(values[5]);
//println(values[0] + "," + values[1] + ","+ values[2] );
//println(values[3] + "," + values[4] + ","+ values[5] );
} else if (values.length == 1) {
endState = 1;
//println(end);
} else {
endState = 0;
}
}
catch(Exception e) {
println("Error parsing string from Serial:");
e.printStackTrace();
}
}

class Circle {
// The Constructor is defined with arguments.
Circle(int x1, int y1, int x2, int y2, int x3, int y3) {

xpos1 = x1;
ypos1 = y1;
xpos2 = x2;
ypos2 = y2;
xpos3 = x3;
ypos3 = y3;
}

void display() {
stroke(0);
strokeWeight (3);
noFill();
ellipse(xpos1, ypos1, radius1, radius1);
strokeWeight (3);
noFill();
ellipse(xpos2, ypos2, radius2, radius2);
strokeWeight (3);
noFill();
ellipse(xpos3, ypos3, radius3, radius3);
}

void grow() {
growMap1 = map(tOne, 100, 8000, 1, 4);
growMap2 = map(tTwo, 100, 8000, 1, 4);
growMap3 = map(tThree, 100, 8000, 1, 4);
if (bOne == 1) {
radius1 = radius1 + growMap1;
} else if (bTwo ==1) {
radius2 = radius2 + growMap2;
} else if (bThree == 1) {
radius3 = radius3 + growMap3;
}
}
}

Future Considerations:

I want to continue to work on this project. Considerations for the patch include fabric type and design of the tappable areas. For the Processing end of things, I’d like to explore what else is possible with the data being sent and well as build a state machine and initialization sequence so that the system knows how many tappable areas are present when plugged in so that two Processing sketches are not needed. On the Arduino side of things, I would like use timing to save the GIF animations rather than a button press.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Vinyl & controller setups are large and expensive.

 

 

Audio Player

Two audio track crossfade without beat/tempo match.

 

 

interactive textile interface – acrylic and silver-nano inks on polyester substrate.

 

 

Bluno Nano and CapSense arduino library.

Bluno can easily connect arduino sensors to Android & iOS. Although, this project is not serially connected to Unity on Android, the example code connects to an app created in Android Studio and the Bluno connects to Unity on iOS.

This program uses an initialization stage to calculate a baseline for each screen
printed touch sensor, then uses a multiplier to calculate a touch threshold. To
increasethe correctness of the sensor data, I will implement a touch calibration
step into the setup sequence.

#include 

// infinEight Driver
// Ty Van de Zande 2018

/*
 * CapitiveSense Library Demo Sketch
 * Paul Badger 2008
 * Uses a high value resistor e.g. 10M between send pin and receive pin
 * Resistor effects sensitivity, experiment with values, 50K - 50M. Larger resistor values yield larger sensor values.
 * Receive pin is the sensor pin - try different amounts of foil/metal on this pin
 */


// Arcitecture
// TBD

static int IN1 = 18;
static int IN2 = 17;
static int IN3 = 16;

static int ledGROUND = 23;
static int ledONE    = 21;
static int ledTWO    = 20;
static int ledTHREE  = 19;


int SENSE1;
int SENSE2;
int SENSE3;

long THRESH1;
long THRESH2;
long THRESH3;

float mult = 1.7;



void setup()                    
{
   pinMode(ledONE, OUTPUT);
   pinMode(ledTWO, OUTPUT);
   pinMode(ledTHREE, OUTPUT);
   pinMode(ledGROUND, OUTPUT);
   
   
   Serial.begin(9600);
   Serial.println("Prepping");
   initializeSensors();
}

void loop()                    
{
  updateSensors();
  //printSensors();
  digitalWrite(ledONE, LOW);
  digitalWrite(ledTWO, LOW);
  digitalWrite(ledTHREE, LOW);
  areWeTouched();     
  delay(10);                    
}

void  areWeTouched()
{
  if(SENSE1 > THRESH1 || SENSE1 == -2){
//    printSensors();
    digitalWrite(ledONE, HIGH);
      Serial.println("3");
  };
  if(SENSE2 > THRESH2 || SENSE2 == -2){
//    printSensors();
      digitalWrite(ledTWO, HIGH);
      Serial.println("2");
  };
  if(SENSE3 > THRESH3  || SENSE3 == -2){
//    printSensors();
      digitalWrite(ledTHREE, HIGH);
      Serial.println("1");
  };
}



void printThresh(int one, int two, int three)
{
  Serial.print(one);
  Serial.print(" . ");
  Serial.print(two);
  Serial.print(" . ");
  Serial.print(three);
  Serial.println(" ");
  
}



void updateSensors()
{
    SENSE1 = touchRead(IN1);
    SENSE2 = touchRead(IN2);
    SENSE3 = touchRead(IN3);
    // Array not working???
//     int SENSESTATES[] = {SENSE1, SENSE2, SENSE3, SENSE4};
//     int lisLEN = sizeof(SENSESTATES);
//     for(int i = 0; i < lisLEN; i++){
//        Serial.print(i);
//        Serial.print(":  ");
//        Serial.print(SENSESTATES[i]);      
//     }
}

void printSensors()
{
    Serial.print(SENSE1);
    Serial.print(" . ");
    Serial.print(SENSE2);
    Serial.print(" . ");
    Serial.print(SENSE3);
    Serial.println(" ");
}


void initializeSensors()
{
  int cts = 104;
  //int mult = 20;
  
  long temp1 = 0;
  long temp2 = 0;
  long temp3 = 0;

  for(int i = 0; i < 20; i++){
    updateSensors();
    //printSensors();
  }
  
  Serial.println("Collecting Summer Readings");
  for(int i = 0; i < cts; i++){
    if (i % 4 == 0) { Serial.print("|"); }
    updateSensors();
    temp1 += SENSE1;
    temp2 += SENSE2;
    temp3 += SENSE3;
  }

  Serial.println(" ");
  Serial.println("Averaging thresholds");
  THRESH1 =  mult * (temp1 / cts);
  THRESH2 =  mult * (temp2 / cts);
  THRESH3 =  mult * (temp3 / cts);
  printThresh(THRESH1, THRESH2, THRESH3);
  printThresh(THRESH1/mult, THRESH2/mult, THRESH3/mult);
  Serial.println(" ");
  digitalWrite(ledONE, HIGH);
  delay(80);
  digitalWrite(ledTWO, HIGH);
  delay(80);
  digitalWrite(ledTHREE, HIGH);
  delay(80);
  Serial.println("Ready!");
  
}

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Thank you to people who helped, and others!!!!
Golan Levin
Claire Hentschker
Zachary Rapaport
Gray Crawford
Daiki Itoh
Lucas Ochoa
Lucy Yu
Jake Scherlis
Imin Yeh
Jesse Klein
Dan Lockton
FRFAF
URO-SURF

Final Project: Running Companion!

This project used a blob code to animate a character that runs with you. By wearing bands of a certain color, the computer can determine what pace goal you want to set, and how fast you’re running. Using the difference between these two paces, it determines the location of the animal companion on the screen. If the pace is slower than the pace goal, then the animal will move to the left of the screen, as if it is running faster than you. If the pace is faster than the pace goal, the animal will move to the right of the screen, as if it is running slower than you.

The amount of bands that you wear determines what animal companion you get, and what your pace goal is. The animal runs at the pace goal.

This sort of pace-keeper allows for runners to be more conscious about their cadence, and makes fixing their cadence easier.

 

The project in use:

At the final presentation:

Here’s my final code, you have to open it in processing:

BlobsforanimalsforShow

Arduino memory / p5.js

If you apply power to the Metro through the power jack, you can disconnect/reconnect from p5.js.   There’s a tiny power switch between the USB port and power connector that needs to be flipped to handle external power.

  1. Connect Arduino to computer, download sketch.
  2. Disconnect Arduino, collect data
  3. Connect Arduino to computer, run p5.js connection and read data

Here’s a simple Arduino sketch and p5.js script: memory-example