Images

Here are the three PC nodes in the network…

Jarvis (Iron Man reference), which is the system I usually work at…

Phoenix, which was the codename for this laptop when it was being developed. It sits in the same room as Jarvis, so their temps have effects on each other…

… and Tars (Interstellar reference). This hosts a Minecraft server and resides in my basement… quite dusty at the moment thanks to the kitchen renovation taking place directly above…

Here’s the the Arduino-powered gateway, which can be connected to any PC to interface with the Arduino MQTT bridge. It has been repurposed from the previous project.

The front of the device…

The rear of the device…

And here are a shot of the network in action…

Here’s a shot of the 3 PC nodes interacting with each other. Right now, they’re all running great, sending each other encouraging messages.

And here’s a shot of what might happen if you neglect your own personal health too much, leading the PCs to go on-strike…

While this is just “Wizard of Ozzed” for the sake of having a quick and dirty example (as well as some humor in my choice of commands…), this is something that can happen if you don’t take care of yourself.

Description

There’s quite a bit to cover here, so bear with me.

As I had stated in the briefing for my previous project, I tend to take much better care of my computers than I do myself (especially when it comes to sleep schedule). The previous idea was one way of attempting to remedy this, but it had some shortcomings. Namely, it was far too easy to game the system with regard to reporting temperatures of the systems, and it had no connection to the systems in order to assist in dealing with certain situations (i.e., when a system is running too hot, what do you do?).

The goal is to allow any and all nodes in the network to communicate their vitals to each other. There are 3 PC nodes (all running Manjaro Linux), as well as a human node. This human node consists of an Arduino (for capturing my vitals at any given moment using the same sensors as in the last project), as well as a PC for reading the output of the Arduino and sending it to the other nodes in the network via the MQTT protocol.

Most of the time, the systems simply communicate their vitals with each other and send each other little messages based on what may be happening at that moment. For instance, one PC may be under heavy load, but its temperatures are in-check. The other PCs will see that and send a message to spur it on further–something along the lines of, “Hey, you’re doing great!” (because especially in times like these, we can all use a little bit of affirmation).

In another instance, a PC may be under very little load, but the temps could still be high. This would indicate a serious problem with that machine (perhaps a fan failure or serious heat in the room). This would cause the other PCs in the network to “sound the alarm” and notify me that something is seriously wrong.

Certain audible responses are triggered based on what the situation is. For instance, if a PC is “sounding the alarm,” it will ring the terminal bell and say something along the lines of “HELP! I need somebody! Not just anybody!” (in a text-to-speech voice for added comedic effect).

Now, that only covers PC-to-PC interaction. At any given point, I can enter the network and report my own vitals via the Arduino connected to a PC. Like with the last project, these are Galvanic Skin Response/Resistance (GSR), heart rate, when I went to sleep the previous night, and for how long I slept. If my GSR is too low or heart rate is too high (indicating that I am stressed), then the PCs will see this and offer words of encouragement.

On the other hand, if I went to sleep at an unreasonable hour the previous night, then that will count as a strike. Strikes last until the end of the week. If I accumulate three strikes, then the computers go on-strike and refuse to perform certain actions. Pulling an all-nighter results in an automatic three strikes.

Progress

One of the things that required quite a bit of trial and error was pulling the necessary data from terminal output. I was simply capturing the output of the sensors command (to get the CPU temperature) and nvidia-smi (to get GPU temperature and utilization, since I have Nvidia GPUs in my systems.

Initially, when the GPU was under 100% load, it would read as 00. This was an indexing issue and was easy to fix.

This happened for no apparent reason. It’s possible that the program just received garbage data from nvidia-smi…

This didn’t go over well with my parents… more on that later…

Since there wasn’t a bunch to do on the Arduino side of things and most of my issues were in Python, that’s all I have for in-progress photo documentation, sadly.

Process Reflection

As with everything I’ve done thus far, time management was a major issue. I’m going to save the details on that, though, as I’ve gone into it already in documentation for my previous works.

Getting the PCs to talk to each other over MQTT was far easier than I had anticipated thanks to Garth Zeglin’s work on the MQTT bridge and server for IDeATe. He did the majority of the dirty work, and there was no need to reinvent the wheel, so I borrowed liberally from his original code.

Something much more difficult was getting the system to consistently read the same sensor data. I’m not sure why, but occasionally, nvidia-smi will output garbage for the GPU utilization. This led to crashes that were initially very hard to track down, but after a while, I simply added an empty string check where necessary–the crashes are gone now.

Another difficult part was figuring out what to actually do based on certain scenarios. I wanted to have a somewhat whimsical feel to the PC reactions regardless of what the situation was. Having heard RZach use a somewhat comical text-to-speech voice during class throughout the semester to notify us when our presentation time slot was ending, I decided to follow a similar path and have the machines output responses to me via text-to-speech.

Unfortunately, that did not go over well with my parents when they heard Tars say “Seth, what the hell are you doing? I’m burning over here!” in a robotic voice… right in the middle of their Netflix-watching time. As a result, that code has been omitted for the time being.

Perhaps when I get my own place, I can revisit that. For now, I have to stick with more discrete notifications like plain text messages over MQTT.

Code

Here are the scripts that run on the 3 PC nodes. They are all very similar, but have minor differences to account for Intel vs. AMD CPU (different way of reporting temps), as well as GPU vitals reporting.

For Jarvis…

#!/usr/bin/env python3
"""A PyQt5 GUI utility to monitor and send MQTT server messages."""

################################################################
# Written in 2018-2020 by Garth Zeglin <garthz@cmu.edu>

# To the extent possible under law, the author has dedicated all copyright
# and related and neighboring rights to this software to the public domain
# worldwide. This software is distributed without any warranty.

# You should have received a copy of the CC0 Public Domain Dedication along with this software.
# If not, see <http://creativecommons.org/publicdomain/zero/1.0/>.

################################################################
# standard Python libraries
from __future__ import print_function
import os, sys, struct, time, logging, functools, queue, signal, getpass, psutil

# documentation: https://doc.qt.io/qt-5/index.html
# documentation: https://www.riverbankcomputing.com/static/Docs/PyQt5/index.html
from PyQt5 import QtCore, QtGui, QtWidgets, QtNetwork

# documentation: https://www.eclipse.org/paho/clients/python/docs/
import paho.mqtt.client as mqtt

# default logging output
log = logging.getLogger('main')

# logger to pass to the MQTT library
mqtt_log = logging.getLogger('mqtt')
mqtt_log.setLevel(logging.WARNING)

# IDeATE server instances, as per https://mqtt.ideate.cmu.edu/#ports

ideate_ports = { 8884 : '16-223',
                 8885 : '16-375',
                 8886 : '60-223',
                 8887 : '62-362',
}

mqtt_rc_codes = ['Success', 'Incorrect protocol version', 'Invalid client identifier', 'Server unavailable', 'Bad username or password', 'Not authorized']
counter = 0


################################################################
class MainApp(object):
    """Main application object holding any non-GUI related state."""

    def __init__(self):

        # Attach a handler to the keyboard interrupt (control-C).
        signal.signal(signal.SIGINT, self._sigint_handler)

        # load any available persistent application settings
        QtCore.QCoreApplication.setOrganizationName("IDeATe")
        QtCore.QCoreApplication.setOrganizationDomain("ideate.cmu.edu")
        QtCore.QCoreApplication.setApplicationName('mqtt_monitor')
        self.settings = QtCore.QSettings()

        # uncomment to restore 'factory defaults'
        # self.settings.clear()

        # MQTT server settings
        self.hostname = self.settings.value('mqtt_host', 'mqtt.ideate.cmu.edu')
        self.systemHostname = os.popen('uname -n').read().strip()
        self.portnum  = self.settings.value('mqtt_port', 8887)
        self.portnum = int(self.portnum)
        self.username = self.settings.value('mqtt_user', 'students')
        self.password = self.settings.value('mqtt_password', 'Arduino')

        # Create a default subscription based on the username.  The hash mark is a wildcard.
        username = getpass.getuser()
        self.subscription = self.settings.value('mqtt_subscription', username + '/#')

        # default message to send
        self.topic = self.settings.value('mqtt_topic', username)
        self.payload = self.settings.value('mqtt_payload', 'hello')

        # Initialize the MQTT client system
        self.client = mqtt.Client()
        self.client.enable_logger(mqtt_log)
        self.client.on_log = self.on_log
        self.client.on_connect = self.on_connect
        self.client.on_disconnect = self.on_disconnect
        self.client.on_message = self.on_message
        self.client.tls_set()
        return

    ################################################################
    def app_is_exiting(self):
        if self.client.is_connected():
            self.client.disconnect()
            self.client.loop_stop()

    def _sigint_handler(self, signal, frame):
        print("Keyboard interrupt caught, running close handlers...")
        self.app_is_exiting()
        sys.exit(0)

    ################################################################
    def set_server_name(self, name):
        self.hostname = name
        self.settings.setValue('mqtt_host', name)

    def set_server_port(self, value):
        self.portnum = value
        self.settings.setValue('mqtt_port', self.portnum)

    def set_username(self, name):
        self.username = name
        self.settings.setValue('mqtt_user', name)

    def set_password(self, name):
        self.password = name
        self.settings.setValue('mqtt_password', name)

    def connect_to_mqtt_server(self):
        if self.client.is_connected():
            return
        else:
            if self.portnum is None:
                print("Warning: Please specify the server port before attempting connection.")
            else:
                print("Info: Initiating MQTT connection to %s:%d" % (self.hostname, self.portnum))
                self.client.username_pw_set(self.username, self.password)
                self.client.connect_async(self.hostname, self.portnum)
                self.client.loop_start()

    def disconnect_from_mqtt_server(self):
        if self.client.is_connected():
            self.client.disconnect()
        else:
            print("Warning: Not connected.")
        self.client.loop_stop()

    ################################################################
    # The callback for when the broker responds to our connection request.
    def on_connect(self, client, userdata, flags, rc):
        print("Info: Connected to server with with flags: %s, result code: %s" % (flags, rc))

        if rc == 0:
            print("Info: Connection succeeded.")

        elif rc > 0:
            if rc < len(mqtt_rc_codes):
                print("Warning: Connection failed with error: %s", mqtt_rc_codes[rc])
            else:
                print("Warning: Connection failed with unknown error %d", rc)

        # Subscribing in on_connect() means that if we lose the connection and reconnect then subscriptions will be renewed.
        client.subscribe(self.subscription)
        return

    # The callback for when the broker responds with error messages.
    def on_log(client, userdata, level, buf):
        print("Info: on_log level %s: %s", level, userdata)
        return

    def on_disconnect(self, client, userdata, rc):
        print("Info: disconnected")
        return

    # The callback for when a message has been received on a topic to which this
    # client is subscribed.  The message variable is a MQTTMessage that describes
    # all of the message parameters.

    # Some useful MQTTMessage fields: topic, payload, qos, retain, mid, properties.
    #   The payload is a binary string (bytes).
    #   qos is an integer quality of service indicator (0,1, or 2)
    #   mid is an integer message ID.

    def on_message(self, client, userdata, msg):
        global counter
        if msg.topic == "leffen/pc_vitals":
            if counter == 5:
                counter = 0
                msgList = str(msg.payload)
                msgList = msgList.split(",")
                msgList[-1] = msgList[-1][0:2]
                msgList[-1] = msgList[-1].strip()
                if msgList[0][2:] != self.systemHostname:
                    if (float(msgList[1]) < 40.0 and float(msgList[2]) >= 85.0) or \
                        (int(msgList[3]) >= 85 and (len(msgList) < 5 or (msgList[4] != "" and int(msgList[4]) < 40))):
                        # This is the critical case, where something is probably seriously wrong with the system...
                        sys.stdout.write('\a') # Ring terminal bell
                        sys.stdout.flush()
                        self.send_message("leffen/pc_chatter", "Hey %s you good? No? I'm gonna call the Fire Apartment then... WOO-ooo-WOO-ooo!" % msgList[0])
                    elif (float(msgList[1]) >= 40.0 and float(msgList[2]) >= 85.0) or \
                        (int(msgList[3]) >= 85 and (len(msgList) < 5 or (msgList[4] != "" and int(msgList[4]) >= 40))):
                        # This is the case where the system is under load and getting a bit hot...
                        self.send_message("leffen/pc_chatter", "%s looks like you're running a little hot... why not take a break? I think you've earned it." % msgList[0])
                    elif (float(msgList[1]) >= 80.0 and float(msgList[2]) < 85.0) or \
                        (int(msgList[3]) < 85 and (len(msgList) < 5 or (msgList[4] != "" and int(msgList[4]) >= 80 or \
                            int(msgList[4]) == 0))):
                        # This is the case where the system is under load and temps are in-check...
                        self.send_message("leffen/pc_chatter", "Looks like you're on it %s... keep it up!" % msgList[0])
        elif msg.topic == "leffen/human_vitals":
            msgList = str(msg.payload)
            msgList = msgList.split(",")
            if (float(msgList[0]) < 100 or float(msgList[1]) >= 130):
                self.send_message("leffen/to_human", "Hey man, looks like you need a break.")
            if (int(msgList[2]) == 0):
                self.send_message("leffen/to_human", "Alright, if you aren't gonna take care of yourself, then we're going on strike!")
            elif (int(msgList[3]) < 7 or int(msgList[3]) >= 12):
                self.send_message("leffen/to_human", "Come on, man. Get better sleep!")
        counter += 1                
        print("{%s} %s" % (msg.topic, msg.payload))
        return

    ################################################################
    def set_subscription(self, sub):
        if self.client.is_connected():
            self.client.unsubscribe(self.subscription)
            try:
                self.client.subscribe(sub)
                self.subscription = sub
                self.settings.setValue('mqtt_subscription', sub)
            except ValueError:
                print("Error: Invalid subscription string, not changed.")
                self.client.subscribe(self.subscription)
        else:
            self.subscription = sub
            self.settings.setValue('mqtt_subscription', sub)

    def set_topic(self, sub):
        self.topic = sub
        self.settings.setValue('mqtt_topic', sub)

    def send_message(self, topic, payload):
        if self.client.is_connected():
            self.client.publish(topic, payload)
        else:
            print("Error: Not connected.")
        self.payload = payload
        self.settings.setValue('mqtt_payload', payload)

    ################################################################

def main():
    # Optionally add an additional root log handler to stream messages to the
    # terminal console.
    if False:
        console_handler = logging.StreamHandler()
        console_handler.setLevel(logging.DEBUG)
        console_handler.setFormatter(logging.Formatter('%(levelname)s:%(name)s: %(message)s'))
        logging.getLogger().addHandler(console_handler)

    # initialize the Qt system itself
    app = QtWidgets.QApplication(sys.argv)

    # create the main application controller
    main = MainApp()
    main.connect_to_mqtt_server()

    # run the event loop until the user is done
    print("Info: Starting event loop.")
    while True:
        rawCPUTemp = os.popen('sensors').read()
        rawGPU = os.popen('nvidia-smi').read()
        cpuTempLine = rawCPUTemp.splitlines()[5].strip()
        gpuTempLine = rawGPU.splitlines()[9].strip()
        gpuUtilLine = rawGPU.splitlines()[9].strip()
        cpuUtil = psutil.cpu_percent()
        cpuTempstr = cpuTempLine[15:19]
        gpuTempstr = gpuTempLine[8:10]
        cpuUtilstr = str(cpuUtil)
        gpuUtilstr = gpuUtilLine[60:63]
        cpuTemp = float(cpuTempstr)
        gpuTemp = float(gpuTempstr)
        main.send_message("leffen/pc_vitals", main.systemHostname + "," +
            cpuUtilstr + "," + cpuTempstr + "," + gpuTempstr + "," + gpuUtilstr)
        time.sleep(10)
    sys.exit(app.exec_())

################################################################
# Main script follows. This sequence is executed when the script is initiated
# from the command line.

if __name__ == "__main__":
    main()

For Phoenix…

#!/usr/bin/env python3
"""A PyQt5 GUI utility to monitor and send MQTT server messages."""

################################################################
# Written in 2018-2020 by Garth Zeglin <garthz@cmu.edu>

# To the extent possible under law, the author has dedicated all copyright
# and related and neighboring rights to this software to the public domain
# worldwide. This software is distributed without any warranty.

# You should have received a copy of the CC0 Public Domain Dedication along with this software.
# If not, see <http://creativecommons.org/publicdomain/zero/1.0/>.

################################################################
# standard Python libraries
from __future__ import print_function
import os, sys, struct, time, logging, functools, queue, signal, getpass, psutil

# documentation: https://doc.qt.io/qt-5/index.html
# documentation: https://www.riverbankcomputing.com/static/Docs/PyQt5/index.html
from PyQt5 import QtCore, QtGui, QtWidgets, QtNetwork

# documentation: https://www.eclipse.org/paho/clients/python/docs/
import paho.mqtt.client as mqtt

# default logging output
log = logging.getLogger('main')

# logger to pass to the MQTT library
mqtt_log = logging.getLogger('mqtt')
mqtt_log.setLevel(logging.WARNING)

# IDeATE server instances, as per https://mqtt.ideate.cmu.edu/#ports

ideate_ports = { 8884 : '16-223',
                 8885 : '16-375',
                 8886 : '60-223',
                 8887 : '62-362',
}

mqtt_rc_codes = ['Success', 'Incorrect protocol version', 'Invalid client identifier', 'Server unavailable', 'Bad username or password', 'Not authorized']
counter = 0


################################################################
class MainApp(object):
    """Main application object holding any non-GUI related state."""

    def __init__(self):

        # Attach a handler to the keyboard interrupt (control-C).
        signal.signal(signal.SIGINT, self._sigint_handler)

        # load any available persistent application settings
        QtCore.QCoreApplication.setOrganizationName("IDeATe")
        QtCore.QCoreApplication.setOrganizationDomain("ideate.cmu.edu")
        QtCore.QCoreApplication.setApplicationName('mqtt_monitor')
        self.settings = QtCore.QSettings()

        # uncomment to restore 'factory defaults'
        # self.settings.clear()

        # MQTT server settings
        self.hostname = self.settings.value('mqtt_host', 'mqtt.ideate.cmu.edu')
        self.systemHostname = os.popen('uname -n').read().strip()
        self.portnum  = self.settings.value('mqtt_port', 8887)
        self.portnum = int(self.portnum)
        self.username = self.settings.value('mqtt_user', 'students')
        self.password = self.settings.value('mqtt_password', 'Arduino')

        # Create a default subscription based on the username.  The hash mark is a wildcard.
        username = getpass.getuser()
        self.subscription = self.settings.value('mqtt_subscription', username + '/#')

        # default message to send
        self.topic = self.settings.value('mqtt_topic', username)
        self.payload = self.settings.value('mqtt_payload', 'hello')

        # Initialize the MQTT client system
        self.client = mqtt.Client()
        self.client.enable_logger(mqtt_log)
        self.client.on_log = self.on_log
        self.client.on_connect = self.on_connect
        self.client.on_disconnect = self.on_disconnect
        self.client.on_message = self.on_message
        self.client.tls_set()
        return

    ################################################################
    def app_is_exiting(self):
        if self.client.is_connected():
            self.client.disconnect()
            self.client.loop_stop()

    def _sigint_handler(self, signal, frame):
        print("Keyboard interrupt caught, running close handlers...")
        self.app_is_exiting()
        sys.exit(0)

    ################################################################
    def set_server_name(self, name):
        self.hostname = name
        self.settings.setValue('mqtt_host', name)

    def set_server_port(self, value):
        self.portnum = value
        self.settings.setValue('mqtt_port', self.portnum)

    def set_username(self, name):
        self.username = name
        self.settings.setValue('mqtt_user', name)

    def set_password(self, name):
        self.password = name
        self.settings.setValue('mqtt_password', name)

    def connect_to_mqtt_server(self):
        if self.client.is_connected():
            return
        else:
            if self.portnum is None:
                print("Warning: Please specify the server port before attempting connection.")
            else:
                print("Info: Initiating MQTT connection to %s:%d" % (self.hostname, self.portnum))
                self.client.username_pw_set(self.username, self.password)
                self.client.connect_async(self.hostname, self.portnum)
                self.client.loop_start()

    def disconnect_from_mqtt_server(self):
        if self.client.is_connected():
            self.client.disconnect()
        else:
            print("Warning: Not connected.")
        self.client.loop_stop()

    ################################################################
    # The callback for when the broker responds to our connection request.
    def on_connect(self, client, userdata, flags, rc):
        print("Info: Connected to server with with flags: %s, result code: %s" % (flags, rc))

        if rc == 0:
            print("Info: Connection succeeded.")

        elif rc > 0:
            if rc < len(mqtt_rc_codes):
                print("Warning: Connection failed with error: %s", mqtt_rc_codes[rc])
            else:
                print("Warning: Connection failed with unknown error %d", rc)

        # Subscribing in on_connect() means that if we lose the connection and reconnect then subscriptions will be renewed.
        client.subscribe(self.subscription)
        return

    # The callback for when the broker responds with error messages.
    def on_log(client, userdata, level, buf):
        print("Info: on_log level %s: %s", level, userdata)
        return

    def on_disconnect(self, client, userdata, rc):
        print("Info: disconnected")
        return

    # The callback for when a message has been received on a topic to which this
    # client is subscribed.  The message variable is a MQTTMessage that describes
    # all of the message parameters.

    # Some useful MQTTMessage fields: topic, payload, qos, retain, mid, properties.
    #   The payload is a binary string (bytes).
    #   qos is an integer quality of service indicator (0,1, or 2)
    #   mid is an integer message ID.

    def on_message(self, client, userdata, msg):
        global counter
        if msg.topic == "leffen/pc_vitals":
            if counter == 5:
                counter = 0
                msgList = str(msg.payload)
                msgList = msgList.split(",")
                msgList[-1] = msgList[-1][0:2]
                msgList[-1] = msgList[-1].strip()
                if msgList[0][2:] != self.systemHostname:
                    if (float(msgList[1]) < 40.0 and float(msgList[2]) >= 85.0) or \
                        (int(msgList[3]) >= 85 and (len(msgList) < 5 or (msgList[4] != "" and int(msgList[4]) < 40))):
                        # This is the critical case, where something is probably seriously wrong with the system...
                        sys.stdout.write('\a') # Ring terminal bell
                        sys.stdout.flush()
                        self.send_message("leffen/pc_chatter", "Hey %s you good? No? I'm gonna call the Fire Apartment then... WOO-ooo-WOO-ooo!" % msgList[0])
                    elif (float(msgList[1]) >= 40.0 and float(msgList[2]) >= 85.0) or \
                        (int(msgList[3]) >= 85 and (len(msgList) < 5 or (msgList[4] != "" and int(msgList[4]) >= 40))):
                        # This is the case where the system is under load and getting a bit hot...
                        self.send_message("leffen/pc_chatter", "%s looks like you're running a little hot... why not take a break? I think you've earned it." % msgList[0])
                    elif (float(msgList[1]) >= 70.0 and float(msgList[2]) < 85.0) or \
                        (int(msgList[3]) < 85 and (len(msgList) < 5 or (msgList[4] != "" and int(msgList[4]) >= 80))):
                        # This is the case where the system is under load and temps are in-check...
                        self.send_message("leffen/pc_chatter", "Looks like you're on it %s... keep it up!" % msgList[0])
        elif msg.topic == "leffen/human_vitals":
            msgList = str(msg.payload)
            msgList = msgList.split(",")
            if (float(msgList[0]) < 100 or float(msgList[1]) >= 130):
                self.send_message("leffen/to_human", "Hey man, looks like you need a break.")
            if (int(msgList[2]) == 0):
                self.send_message("leffen/to_human", "Alright, if you aren't gonna take care of yourself, then we're going on strike!")
            elif (int(msgList[3]) < 7 or int(msgList[3]) >= 12):
                self.send_message("leffen/to_human", "Come on, man. Get better sleep!")
        counter += 1
        print("{%s} %s" % (msg.topic, msg.payload))
        return

    ################################################################
    def set_subscription(self, sub):
        if self.client.is_connected():
            self.client.unsubscribe(self.subscription)
            try:
                self.client.subscribe(sub)
                self.subscription = sub
                self.settings.setValue('mqtt_subscription', sub)
            except ValueError:
                print("Error: Invalid subscription string, not changed.")
                self.client.subscribe(self.subscription)
        else:
            self.subscription = sub
            self.settings.setValue('mqtt_subscription', sub)

    def set_topic(self, sub):
        self.topic = sub
        self.settings.setValue('mqtt_topic', sub)

    def send_message(self, topic, payload):
        if self.client.is_connected():
            self.client.publish(topic, payload)
        else:
            print("Error: Not connected.")
        self.payload = payload
        self.settings.setValue('mqtt_payload', payload)

    ################################################################

def main():
    # Optionally add an additional root log handler to stream messages to the
    # terminal console.
    if False:
        console_handler = logging.StreamHandler()
        console_handler.setLevel(logging.DEBUG)
        console_handler.setFormatter(logging.Formatter('%(levelname)s:%(name)s: %(message)s'))
        logging.getLogger().addHandler(console_handler)

    # initialize the Qt system itself
    app = QtWidgets.QApplication(sys.argv)

    # create the main application controller
    main = MainApp()
    main.connect_to_mqtt_server()

    # run the event loop until the user is done
    print("Info: Starting event loop.")
    #rawCPU = os.system("sensors"
    while True:
        rawCPUTemp = os.popen('sensors').read()
        rawGPU = os.popen('nvidia-smi').read()
        cpuTempLine = rawCPUTemp.splitlines()[19].strip()
        gpuTempLine = rawGPU.splitlines()[9].strip()
        cpuUtil = psutil.cpu_percent()
        gpuUtilLine = rawGPU.splitlines()[9].strip()
        cpuTempstr = cpuTempLine[16:20]
        gpuTempstr = gpuTempLine[8:10]
        cpuUtilstr = str(cpuUtil)
        gpuUtilstr = gpuUtilLine[60:63]
        cpuTemp = float(cpuTempstr)
        gpuTemp = float(gpuTempstr)
        cpuUtil = float(cpuUtilstr)
        main.send_message("leffen/pc_vitals", main.systemHostname + "," +
            cpuUtilstr + "," + cpuTempstr + "," + gpuTempstr + "," + gpuUtilstr)
        time.sleep(10)
    sys.exit(app.exec_())

################################################################
# Main script follows. This sequence is executed when the script is initiated
# from the command line.

if __name__ == "__main__":
    main()

and for Tars…

#!/usr/bin/env python3
"""A PyQt5 GUI utility to monitor and send MQTT server messages."""

################################################################
# Written in 2018-2020 by Garth Zeglin <garthz@cmu.edu>

# To the extent possible under law, the author has dedicated all copyright
# and related and neighboring rights to this software to the public domain
# worldwide. This software is distributed without any warranty.

# You should have received a copy of the CC0 Public Domain Dedication along with this software.
# If not, see <http://creativecommons.org/publicdomain/zero/1.0/>.

################################################################
# standard Python libraries
from __future__ import print_function
import os, sys, struct, time, logging, functools, queue, signal, getpass, psutil

# documentation: https://doc.qt.io/qt-5/index.html
# documentation: https://www.riverbankcomputing.com/static/Docs/PyQt5/index.html
from PyQt5 import QtCore, QtGui, QtWidgets, QtNetwork

# documentation: https://www.eclipse.org/paho/clients/python/docs/
import paho.mqtt.client as mqtt

# default logging output
log = logging.getLogger('main')

# logger to pass to the MQTT library
mqtt_log = logging.getLogger('mqtt')
mqtt_log.setLevel(logging.WARNING)

# IDeATE server instances, as per https://mqtt.ideate.cmu.edu/#ports

ideate_ports = { 8884 : '16-223',
                 8885 : '16-375',
                 8886 : '60-223',
                 8887 : '62-362',
}

mqtt_rc_codes = ['Success', 'Incorrect protocol version', 'Invalid client identifier', 'Server unavailable', 'Bad username or password', 'Not authorized']
counter = 0


################################################################
class MainApp(object):
    """Main application object holding any non-GUI related state."""

    def __init__(self):

        # Attach a handler to the keyboard interrupt (control-C).
        signal.signal(signal.SIGINT, self._sigint_handler)

        # load any available persistent application settings
        QtCore.QCoreApplication.setOrganizationName("IDeATe")
        QtCore.QCoreApplication.setOrganizationDomain("ideate.cmu.edu")
        QtCore.QCoreApplication.setApplicationName('mqtt_monitor')
        self.settings = QtCore.QSettings()

        # uncomment to restore 'factory defaults'
        # self.settings.clear()

        # MQTT server settings
        self.hostname = self.settings.value('mqtt_host', 'mqtt.ideate.cmu.edu')
        self.systemHostname = os.popen('uname -n').read().strip()
        self.portnum  = self.settings.value('mqtt_port', 8887)
        self.portnum = int(self.portnum)
        self.username = self.settings.value('mqtt_user', 'students')
        self.password = self.settings.value('mqtt_password', 'Arduino')

        # Create a default subscription based on the username.  The hash mark is a wildcard.
        username = getpass.getuser()
        self.subscription = self.settings.value('mqtt_subscription', username + '/#')

        # default message to send
        self.topic = self.settings.value('mqtt_topic', username)
        self.payload = self.settings.value('mqtt_payload', 'hello')

        # Initialize the MQTT client system
        self.client = mqtt.Client()
        self.client.enable_logger(mqtt_log)
        self.client.on_log = self.on_log
        self.client.on_connect = self.on_connect
        self.client.on_disconnect = self.on_disconnect
        self.client.on_message = self.on_message
        self.client.tls_set()
        return

    ################################################################
    def app_is_exiting(self):
        if self.client.is_connected():
            self.client.disconnect()
            self.client.loop_stop()

    def _sigint_handler(self, signal, frame):
        print("Keyboard interrupt caught, running close handlers...")
        self.app_is_exiting()
        sys.exit(0)

    ################################################################
    def set_server_name(self, name):
        self.hostname = name
        self.settings.setValue('mqtt_host', name)

    def set_server_port(self, value):
        self.portnum = value
        self.settings.setValue('mqtt_port', self.portnum)

    def set_username(self, name):
        self.username = name
        self.settings.setValue('mqtt_user', name)

    def set_password(self, name):
        self.password = name
        self.settings.setValue('mqtt_password', name)

    def connect_to_mqtt_server(self):
        if self.client.is_connected():
            return
        else:
            if self.portnum is None:
                print("Warning: Please specify the server port before attempting connection.")
            else:
                print("Info: Initiating MQTT connection to %s:%d" % (self.hostname, self.portnum))
                self.client.username_pw_set(self.username, self.password)
                self.client.connect_async(self.hostname, self.portnum)
                self.client.loop_start()

    def disconnect_from_mqtt_server(self):
        if self.client.is_connected():
            self.client.disconnect()
        else:
            print("Warning: Not connected.")
        self.client.loop_stop()

    ################################################################
    # The callback for when the broker responds to our connection request.
    def on_connect(self, client, userdata, flags, rc):
        print("Info: Connected to server with with flags: %s, result code: %s" % (flags, rc))

        if rc == 0:
            print("Info: Connection succeeded.")

        elif rc > 0:
            if rc < len(mqtt_rc_codes):
                print("Warning: Connection failed with error: %s", mqtt_rc_codes[rc])
            else:
                print("Warning: Connection failed with unknown error %d", rc)

        # Subscribing in on_connect() means that if we lose the connection and reconnect then subscriptions will be renewed.
        client.subscribe(self.subscription)
        return

    # The callback for when the broker responds with error messages.
    def on_log(client, userdata, level, buf):
        print("Info: on_log level %s: %s", level, userdata)
        return

    def on_disconnect(self, client, userdata, rc):
        print("Info: disconnected")
        return

    # The callback for when a message has been received on a topic to which this
    # client is subscribed.  The message variable is a MQTTMessage that describes
    # all of the message parameters.

    # Some useful MQTTMessage fields: topic, payload, qos, retain, mid, properties.
    #   The payload is a binary string (bytes).
    #   qos is an integer quality of service indicator (0,1, or 2)
    #   mid is an integer message ID.

    def on_message(self, client, userdata, msg):
        global counter
        if msg.topic == "leffen/pc_vitals":
            if counter == 6:
                counter = 0
                msgList = str(msg.payload)
                msgList = msgList.split(",")
                msgList[-1] = msgList[-1][0:2]
                msgList[-1] = msgList[-1].strip()
                if msgList[0][2:] != self.systemHostname:
                    if (float(msgList[1]) < 40.0 and float(msgList[2]) >= 85.0) or \
                        (int(msgList[3]) >= 85 and (len(msgList) < 5 or (msgList[4] != "" and int(msgList[4]) < 40))):
                        # This is the critical case, where something is probably seriously wrong with the system...
                        sys.stdout.write('\a') # Ring terminal bell
                        sys.stdout.flush()
                        self.send_message("leffen/pc_chatter", "Hey %s you good? No? I'm gonna call the Fire Apartment then... WOO-ooo-WOO-ooo!" % msgList[0])
                    elif (float(msgList[1]) >= 40.0 and float(msgList[2]) >= 85.0) or \
                        (int(msgList[3]) >= 85 and (len(msgList) < 5 or (msgList[4] != "" and int(msgList[4]) >= 40))):
                        # This is the case where the system is under load and getting a bit hot...
                        self.send_message("leffen/pc_chatter", "%s looks like you're running a little hot... why not take a break? I think you've earned it." % msgList[0])
                    elif (float(msgList[1]) >= 80.0 and float(msgList[2]) < 85.0) or \
                        (int(msgList[3]) < 85 and (len(msgList) < 5 or (msgList[4] != "" and int(msgList[4]) >= 80))):
                        # This is the case where the system is under load and temps are in-check...
                        self.send_message("leffen/pc_chatter", "Looks like you're on it %s... keep it up!" % msgList[0])
        elif msg.topic == "leffen/human_vitals":
            msgList = str(msg.payload)
            msgList = msgList.split(",")
            if (float(msgList[0]) < 100 or float(msgList[1]) >= 130):
                self.send_message("leffen/to_human", "Hey man, looks like you need a break.")
            if (int(msgList[2]) == 0):
                self.send_message("leffen/to_human", "Alright, if you aren't gonna take care of yourself, then we're going on strike!")
            elif (int(msgList[3]) < 7 or int(msgList[3]) >= 12):
                self.send_message("leffen/to_human", "Come on, man. Get better sleep!")
        counter += 1
        print("{%s} %s" % (msg.topic, msg.payload))
        return

    ################################################################
    def set_subscription(self, sub):
        if self.client.is_connected():
            self.client.unsubscribe(self.subscription)
            try:
                self.client.subscribe(sub)
                self.subscription = sub
                self.settings.setValue('mqtt_subscription', sub)
            except ValueError:
                print("Error: Invalid subscription string, not changed.")
                self.client.subscribe(self.subscription)
        else:
            self.subscription = sub
            self.settings.setValue('mqtt_subscription', sub)

    def set_topic(self, sub):
        self.topic = sub
        self.settings.setValue('mqtt_topic', sub)

    def send_message(self, topic, payload):
        if self.client.is_connected():
            self.client.publish(topic, payload)
        else:
            print("Error: Not connected.")
        self.payload = payload
        self.settings.setValue('mqtt_payload', payload)

    ################################################################

def main():
    # Optionally add an additional root log handler to stream messages to the
    # terminal console.
    if False:
        console_handler = logging.StreamHandler()
        console_handler.setLevel(logging.DEBUG)
        console_handler.setFormatter(logging.Formatter('%(levelname)s:%(name)s: %(message)s'))
        logging.getLogger().addHandler(console_handler)

    # initialize the Qt system itself
    app = QtWidgets.QApplication(sys.argv)

    # create the main application controller
    main = MainApp()
    main.connect_to_mqtt_server()

    # run the event loop until the user is done
    print("Info: Starting event loop.")
    while True:
        rawCPUTemp = os.popen('sensors').read()
        rawGPU = os.popen('nvidia-smi').read()
        cpuTempLine = rawCPUTemp.splitlines()[31].strip()
        gpuTempLine = rawGPU.splitlines()[9].strip()
        cpuUtil = psutil.cpu_percent()
        cpuTempstr = cpuTempLine[15:19]
        gpuTempstr = gpuTempLine[8:10]
        cpuUtilstr = str(cpuUtil)
        cpuTemp = float(cpuTempstr)
        gpuTemp = float(gpuTempstr)
        main.send_message("leffen/pc_vitals", main.systemHostname + "," +
            cpuUtilstr + "," + cpuTempstr + "," + gpuTempstr)
        time.sleep(10)
    sys.exit(app.exec_())

################################################################
# Main script follows. This sequence is executed when the script is initiated
# from the command line.

if __name__ == "__main__":
    main()

Here’s the code that runs on the Arduino. It’s nothing fancy–it’s just sending the relevant data as a comma-separated string over serial, which is then picked up by the computer that the Arduino is connected to.

/*
   Human Gateway (62-362 Project 3)
   Seth Geiser (sgeiser)

   Collaboration: Same as Project 2

   GSR Sensor: https://wiki.seeedstudio.com/Grove-GSR_Sensor/
   MAX30102 PulseOx: Example 5: Heart-rate
   Keypad: Same as Project 1, Workaholic's Clock
   DFPlayer Mini: Same as Project 1, Workaholic's Clock

   This code initializes a keypad, 20x4 I2C LCD, Pulse
   Oximeter, Grove GSR sensor, and a DFRobot DFPlayer
   Mini. The keypad is for entering when I went to sleep an
   how long I slept. The GSR sensor measures galvanic skin
   response, similar to a polygraph test. The Pulse Oximeter
   is currently only being used as a heart-rate monitor,
   since the SPO2 functionality appears to not be
   working right now (tested with the SPO2 example
   code).

   Pin mapping:

   pin      | mode       | description
   ---------|------------|------------
   SDA/SCL   I2C          MAX30102 PulseOx + 20x4 I2C LCD
   2-8       Drive/Sense  Keypad
   9         RX           DFPlayer Mini TX
   10        TX           DFPlayer Mini RX
   A0        INPUT        GSR Sensor (through Grove shield)
*/

#include <SoftwareSerial.h>
#include <DFRobotDFPlayerMini.h>
#include <Wire.h>
#include <LiquidCrystal_I2C.h>
#include <Keypad.h>
#include <MAX30105.h>
#include <heartRate.h>

SoftwareSerial mySoftwareSerial(9, 10); // RX, TX
DFRobotDFPlayerMini myDFPlayer;

MAX30105 particleSensor;

LiquidCrystal_I2C lcd(0x27, 20, 4);
// set the LCD address to 0x27 for 20x4 display

char customKey = 0;
const byte ROWS = 4;
const byte COLS = 3;
char hexaKeys[ROWS][COLS] = {
  {'1', '2', '3'},
  {'4', '5', '6'},
  {'7', '8', '9'},
  {'*', '0', '#'}
};

byte rowPins[ROWS] = {5, 3, 2, 7};
byte colPins[COLS] = {4, 8, 6};
Keypad customKeypad = Keypad(makeKeymap(hexaKeys), rowPins, colPins, ROWS, COLS);

const byte RATE_SIZE = 4; // Increase this for more averaging. 4 is good.
byte rates[RATE_SIZE]; // Array of heart rates
byte rateSpot = 0;
long lastBeat = 0; // Time at which the last beat occurred

float beatsPerMinute;
int beatAvg;

const int GSR = A0;
int sensorValue = 0;
int gsr_average = 0;

int j = 0;
int sleepStart;
int sleepAmount;

// For accepting input from the keypad, these get converted
// to the ints above...
char sleep[2];

void(* resetFunc) (void) = 0; // reset the Arduino when done
void setup() {
  lcd.init();
  lcd.backlight();
  mySoftwareSerial.begin(9600);
  Serial.begin(115200);
  Serial.println("Human,Arduino UNO R3 SMD Connected...");

  lcd.setCursor(0, 0);
  lcd.print(F("Arduino UNO R3 SMD"));
  lcd.setCursor(0, 1);
  lcd.print(F("Starting DFPlayer..."));
  lcd.setCursor(0, 2);
  lcd.print(F("Starting MAX30102..."));
  lcd.setCursor(0, 3);
  lcd.print(F("Please wait..."));
  delay(2000);
  // Initialize sensor
  if (!particleSensor.begin(Wire, I2C_SPEED_FAST))
    // Use default I2C port, 400kHz speed
  {
    lcd.clear();
    lcd.setCursor(0, 0);
    lcd.print(F("MAX30102 not found."));
    lcd.setCursor(0, 1);
    lcd.print(F("Please check wiring."));
    while (1);
  }

  particleSensor.setup();
  // Configure sensor with default settings
  particleSensor.setPulseAmplitudeRed(0x0A);
  // Turn Red LED to low to indicate sensor is running
  particleSensor.setPulseAmplitudeGreen(0);
  // Turn off Green LED

  if (!myDFPlayer.begin(mySoftwareSerial)) {
    // Use softwareSerial to communicate with mp3.
    lcd.clear();
    lcd.setCursor(0, 0);
    lcd.print(F("Unable to begin:"));
    lcd.setCursor(0, 1);
    lcd.print(F("Please check wiring!"));
    lcd.setCursor(0, 2);
    lcd.print(F("Please insert SD!"));
  }
  lcd.setCursor(0, 3);
  lcd.print(F("DFPlayerMini online."));
  myDFPlayer.volume(20);  // Set volume value. From 0 to 30
  delay(2000);
  lcd.clear();
  lcd.setCursor(0, 0);
  lcd.print(F("CMU IDeATe PhysComp"));
  lcd.setCursor(0, 1);
  lcd.print(F("Human Gateway"));
  lcd.setCursor(0, 2);
  lcd.print(F("Powered by sgeiser"));
  lcd.setCursor(0, 3);
  lcd.print(F("BOOTING DEVICE..."));
  delay(2500);
  myDFPlayer.play(1); // 0001_ps2.mp3
  delay(2500);
  lcd.clear();
  customKey = customKeypad.getKey();
  lcd.clear();
  lcd.setCursor(0, 0);
  lcd.print(F("Input when you went"));
  lcd.setCursor(0, 1);
  lcd.print(F("to sleep and how"));
  lcd.setCursor(0, 2);
  lcd.print(F("long you slept in"));
  lcd.setCursor(0, 3);
  lcd.print(F("24-hour time."));
  delay(5000);
  lcd.clear();
  lcd.setCursor(0, 0);
  lcd.print(F("If any value is less"));
  lcd.setCursor(0, 1);
  lcd.print(F("than 10, then prefix"));
  lcd.setCursor(0, 2);
  lcd.print(F("with a zero."));
  delay(5000);
  lcd.clear();
  inputSleepPattern();
}

void loop() {
  customKey = customKeypad.getKey();
  long sum = 0;
  for (int i = 0; i < 10; i++) {
    sensorValue = analogRead(GSR);
    sum += sensorValue;
  }
  gsr_average = sum / 10;
  long irValue = particleSensor.getIR();

  if (checkForBeat(irValue) == true)
  {
    // We sensed a beat!
    long delta = millis() - lastBeat;
    lastBeat = millis();

    beatsPerMinute = 60 / (delta / 1000.0);

    if (beatsPerMinute < 255 && beatsPerMinute > 20)
    {
      rates[rateSpot++] = (byte)beatsPerMinute;
      // Store this reading in the array
      rateSpot %= RATE_SIZE; // Wrap variable

      // Take average of readings
      beatAvg = 0;
      for (byte x = 0 ; x < RATE_SIZE ; x++)
        beatAvg += rates[x];
      beatAvg /= RATE_SIZE;
    }
  }
  lcd.setCursor(0, 0);
  lcd.print((String)"Average GSR: " + gsr_average + "    ");
  lcd.setCursor(0, 1);
  lcd.print((String)"Average BPM: " + beatAvg + "    ");
  lcd.setCursor(0, 2);
  lcd.print(F("#: Assess vitals"));
  if (customKey == '#') {
    lcd.clear();
    assess();
  }
}

void inputSleepStart() {
  lcd.setCursor(0, 0);
  lcd.print(F("Sleep start: "));
  while (j < 2) {
    char key = customKeypad.getKey();
    if (key) {
      sleep[j++] = key;
      lcd.print(key);
    }
    key = 0;
  }
  sleepStart = atoi(sleep);
  // Convert the char array to int and store it. Note
  // that atoi is deprecated, am going to change it soon.
  j = 0;
}

void inputSleepAmount() {
  lcd.setCursor(0, 1);
  lcd.print(F("Sleep amount: "));
  while (j < 2) {
    char key = customKeypad.getKey();
    if (key) {
      sleep[j++] = key;
      lcd.print(key);
    }
    key = 0;
  }
  sleepAmount = atoi(sleep);
  // Convert the char array to int and store it. Note
  // that atoi is deprecated, am going to change it soon.
  j = 0;
}

// I am not fond of how I wrote these two functions.
// I hope to condense them.

void inputSleepPattern() {
  inputSleepStart();
  inputSleepAmount();
  lcd.clear();
}

void assess() {
  lcd.clear();
  lcd.setCursor(0, 0);
  lcd.setCursor(0, 2);
  lcd.print(F("Checking your vitals"));
  delay(2500);
  if (gsr_average > 500) {
    lcd.setCursor(0, 3);
    lcd.print(F("Could not read GSR."));
    delay(2500);
  }
  if (beatAvg <= 40) {
    lcd.setCursor(0, 3);
    lcd.print(F("Could not get pulse."));
    delay(2500);
  }
  if ((gsr_average < 150 || beatAvg >= 130) && (sleepStart <= 6 || (sleepAmount < 7 || sleepAmount > 12))) {
    allBad();
  }
  else if (gsr_average < 150 || beatAvg >= 130) {
    stressed();
  }
  else if (sleepStart <= 6 || (sleepAmount < 7 || sleepAmount > 12)) {
    badSleep();
  }
  else {
    allGood();
  }
  Serial.println(String(gsr_average) + "," + String(beatAvg) + "," + String(sleepStart) + "," + String(sleepAmount));
  resetFunc();
}

void badSleep() {
  lcd.setCursor(0, 3);
  lcd.print(F("Get better sleep!   "));
  delay(5000);
}

void stressed() {
  lcd.setCursor(0, 3);
  lcd.print(F("You're stressed..."));
  delay(2500);
  lcd.setCursor(0, 3);
  lcd.print(F("Take a break, dude."));
  delay(5000);
}

void allBad() {
  lcd.clear();
  lcd.setCursor(0, 0);
  lcd.print(F("Oh my God, it's all"));
  lcd.setCursor(0, 1);
  lcd.print(F("broken! Seriously,"));
  lcd.setCursor(0, 2);
  lcd.print(F("you need to get on"));
  lcd.setCursor(0, 3);
  lcd.print(F("top of this..."));
  myDFPlayer.play(5); // 0005_battery_low.mp3
  delay(5000);
}

void allGood() {
  lcd.setCursor(0, 3);
  lcd.print(F("Your vitals are OK!"));
  delay(2500);
  lcd.clear();
  lcd.setCursor(0, 0);
  lcd.print(F("Nicely done! Your"));
  lcd.setCursor(0, 1);
  lcd.print(F("vitals look alright,"));
  lcd.setCursor(0, 2);
  lcd.print(F("and your systems are"));
  lcd.setCursor(0, 3);
  lcd.print(F("running smoothly!"));
  myDFPlayer.play(3); // 0003_scatman_world.mp3
  delay(13000);
}

Process Images:

Above are the 3 different positions I was considering when setting up the laptop.

Above are the ways that I setup the camera for the self-portraits.

Me checking the settings of the app to make sure that it was working properly.

Reflection:

While doing this project I definitely felt conscious of the way that I interacted with my space, with how my hair looked, and how I was dressed. I did my best to make sure that this did not impact the actions I made but it was something I felt more aware of. The laptop had a camera protector on it but here I was exposing myself 24/7. I was also thinking about how people might judge me once they paid more attention to the times that I woke up or went to sleep. 

In the beginning of this project I was using the laptop and phone that I always use, but at some point they had to be moved so that I could use them. So I did not really like that because that also meant that the pictures could not be taken in the exact same position. Finding a free security app for my iPhone and MacBook was a long process, I was not successful with finding one for my phone but I eventually found one for my MacBook. Since I needed my MacBook more I tried to connect the camera on my phone with the security app on my laptop but unfortunately the app did not detect the motion from my phone. I remembered that I had an extra laptop and if that I could get a charger for it I could use that without having to interfere with it. So I went on another hunt to find an app that could work with it since it was a Dell laptop and came across iSpyConnect. This app took awhile for me to figure out how to use and was much more complicated compared to the one I had used on my MacBook, but it worked out eventually. Since I interacted with the area in front of the camera a good bit, this resulted in over 3000 pictures that were taken, having to sift through all of those to decide the final 50 took awhile and I also kept going back and forth on which part of the walking in and out process that I wanted to use. It was also an interesting process setting the camera up to take the self portraits in the different spaces, depending on the space it took awhile to figure out. The easiest part about this project was definitely coming in and out of my room haha.

Perry mentioned looking at all the pictures in which I interacted with the space in a certain way, for instance right when I put my hand on the door to open, or when I came into my room with a mask on. This is an interesting idea that I would have liked to include as well, to see the way in which I interacted with my space differently or in the same manner.

Video:

This video shows me placing dolls on a board and pressing a button which spawns them into my Unity scene. This is only a prototype, which is far from perfect, and you can see me fiddle with the position of the blue doll (and later the red doll) after the Arduino failed to pick them up.

Description:

Place dolls on a board and push the ‘spawn’ button, the dolls on said board will then appear in a digital environment (created with Unity). Each doll can be differentiated by their color, both in their physical and digital representation. When the ‘spawn’  button is pushed, the digital environment will be emptied and then refilled with the dolls placed on the board (if any). In the digital environment, the dolls are given instructions to move to random positions inside given boundaries.

Each doll holds a different resistor that can be read by the Arduino, which is what allows the Arduino to determine the dolls color. The Arduino then sends that information to Unity, where it is interpreted and carried out.

Music: Brian Eno – New Space Music (Visualizer)

Progress Images:

These images include some of the 3d models I made as well as  parts of the fabrication process.

Reflection:

There are many events in our lives that we share with our family and friends. Under quarantine however, we are no longer able to share the same physical spaces with one another as we once could. It was my goal, for this project, to simulate these life events as they may have appeared before quarantine. Where representations of our family and friends would be able to share a virtual environment in lieu of a physical one.

I chose to create a funeral setting after attending a zoom call in memorial of my late uncle. I began to picture what my own funeral might look like, and who I hoped would attend.

So I made a board that would essentially act as the invite list, and a few dolls that would represent the people you might invite.

This project is only a prototype of the initial idea. The environment I made was very open to interpretation, and not necessarily personal. In a future iteration, I would make it so that, rather than color, each doll would represent a specific person in your life. I feel that this would drastically change how a person would interact with this project. Specifically, I believe that it would make placing the dolls on the board a much more personal experience. There are many other levels of customization I could hope to incorporate, but to create a literal connection between the dolls at your disposal and the people in your life is perhaps the most significant.

I was glad to be able to incorporate Unity in this project. Getting the Arduino to communicate with Unity was a good learning experience, and I believe will be useful in the future. I used SolidWorks to create the models in the scene as well.  It was very satisfying to be able to bring together these different skills I’ve developed over the semester into one project.

Code:

Arduino:

int pin_1 = A0;
int pin_2 = A1;
int pin_3 = A2;
int pin_4 = A3;

int sim_button = 5;

int array_size = 4;
int pin_list[] = {pin_1, pin_2, pin_3, pin_4};
float pin_readings[] = {0.0, 0.0, 0.0, 0.0};
int send_to_unity[] = {0, 0, 0, 0};

int pin; 
float pin_reading = 0.0;

int raw = 0;
int Vin = 5;
float Vout = 0;
float R_1K = 1000;
float R_unknown = 0;
float buffer = 0;


void setup() {
  Serial.begin(9600);
  
  pinMode(pin_1, INPUT);
  pinMode(pin_2, INPUT);
  pinMode(pin_3, INPUT);
  pinMode(pin_4, INPUT);
  pinMode(sim_button, INPUT);
}

void loop() {
  if (digitalRead(sim_button) == HIGH) {
    for (int i = 0; i < array_size; i++) {
      pin = pin_list[i];
      pin_reading = resistor_read(pin);
      pin_readings[i] = pin_reading;
      send_to_unity[i] = doll_num(pin_reading);
    }
    
    for (int i = 0; i < array_size; i++) {
      Serial.write(i);
      Serial.flush();
      Serial.write(send_to_unity[i]);
      Serial.flush();
    }    
    delay(1000);
  }
}

float resistor_read(int pin) {
  raw = analogRead(pin);
  if(raw){
    buffer = raw * Vin;
    Vout = (buffer)/1024.0;
    buffer = (Vin/Vout) - 1;
    R_unknown = R_1K * buffer;
    return(R_unknown);
  } else {
    return(0.0);
  }
}

int doll_num(float resistor_val) {
  if (0.0 <= resistor_val && resistor_val < 5.0) {
    //0 empty
    return 0;
  } else if (90.0 < resistor_val && resistor_val < 150.0) {
    //100 red
    return 1;
  } else if (300.0 < resistor_val && resistor_val < 500.0) {
    //330 yellow
    return 2;
  } else if (1900.0 < resistor_val && resistor_val < 2600.0) {
    //2k white
    return 3;
  } else if (4000.0 < resistor_val && resistor_val < 6000.0) {
    //5k1 blue
    return 4;
  } else {
    return 0;
  }
}

Unity:

spawn_dolls:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using System.IO.Ports;

public class spawn_dolls : MonoBehaviour
{
    
    public SerialPort sp = new SerialPort("/dev/cu.usbmodem14101", 9600);
    public GameObject doll_prefab;

    private int pin_num = 0;
    private int pin_name = 0;

    private GameObject[] doll_list;

    public Material color;

    void Start()
    {
        sp.Open();
        sp.ReadTimeout = 1;

        doll_list = new GameObject[] {null, null, null, null};
    }

    // Update is called once per frame
    void Update()
    {
        if (sp.IsOpen) {
            try 
            { 
                // read the incoming byte:
                pin_num = sp.ReadByte();
                pin_name = sp.ReadByte();

                if (pin_num == 0) {
                    //first doll, delete last batch
                    for (int i = 0; i < 4; i++){
                        if (doll_list[i] != null) {
                            Destroy(doll_list[i]);
                            doll_list[i] = null;
                        }
                    }
                }

                if (pin_name != 0) {
                    spawn_doll(pin_num, pin_name);
                }

            }
            catch (System.Exception)
            {

            }
        }
    }


    void spawn_doll(int pin_num, int pin_name){

        Vector3 position = new Vector3(Random.Range(-300.0F, 300.0F), 100, Random.Range(-150.0F, 200.0F));//position
        GameObject doll_clone = (GameObject) Instantiate(doll_prefab, position, Quaternion.identity);

        GameObject body = doll_clone.transform.GetChild(0).gameObject;
        GameObject hands = doll_clone.transform.GetChild(1).gameObject;
        GameObject head = doll_clone.transform.GetChild(2).GetChild(0).gameObject;

        Renderer body_render = body.GetComponent<Renderer>();
        Renderer hands_render = hands.GetComponent<Renderer>();
        Renderer head_render = head.GetComponent<Renderer>();


        if (pin_name == 1) {
            body_render.material.color = Color.red;
            hands_render.material.color = Color.red;
            head_render.material.color = Color.red;
            doll_clone.transform.localScale = new Vector3(0.9F,0.9F,0.9F);

        } else if (pin_name == 2) {
            body_render.material.color = Color.yellow;
            hands_render.material.color = Color.yellow;
            head_render.material.color = Color.yellow;
            doll_clone.transform.localScale = new Vector3(0.92F,0.92F,0.92F);

        } else if (pin_name == 3) {
            body_render.material.color = Color.white;
            hands_render.material.color = Color.white;
            head_render.material.color = Color.white;
            doll_clone.transform.localScale = new Vector3(0.94F,0.94F,0.94F);

        } else if (pin_name == 4) {
            body_render.material.color = Color.blue;
            hands_render.material.color = Color.blue;
            head_render.material.color = Color.blue;
            doll_clone.transform.localScale = new Vector3(0.96F,0.96F,0.96F);

        } else {
            body_render.material.color = Color.black;
            hands_render.material.color = Color.black;
            head_render.material.color = Color.black;
            doll_clone.transform.localScale = new Vector3(1F,1F,1F);
        }

        doll_list[pin_num] = doll_clone;
    }
}

doll_movement:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class doll_movement : MonoBehaviour
{   
    private GameObject head;
    private GameObject hands;
    private GameObject body;
    private GameObject head_axis;
    private GameObject neck_axis;

    public float xMax;
    public float zMax;
    public float xMin;
    public float zMin;

    public float moveSpeed = 2;
    public float turnSpeed = 45;
    
    private Vector3 position = Vector3.zero;

    private float timer;
    private float timer_limit;

    private int move_bool;

    void Start()
    {
        body  = transform.GetChild(0).gameObject;
        hands  = transform.GetChild(1).gameObject;
        head_axis  = transform.GetChild(2).gameObject;
        head  = transform.GetChild(2).GetChild(0).gameObject;
        neck_axis  = transform.GetChild(3).gameObject;

        timer_limit = Random.Range(1.0f, 10.0f);
        move_bool = Random.Range(0, 4);

        RandomizePosition();

    }

    void Update()
    {
        timer += Time.deltaTime;
        
        if (transform.localPosition.x > xMax) {
            RandomizePosition();
            timer = 0.0f;
            timer_limit = Random.Range(1.0f, 10.0f);
            move_bool = Random.Range(0, 4);
        }
        if (transform.localPosition.x < xMin) {
            RandomizePosition();
            timer = 0.0f;
            timer_limit = Random.Range(1.0f, 10.0f);
            move_bool = Random.Range(0, 4);
        }
        if (transform.localPosition.z > zMax) {
            RandomizePosition();
            timer = 0.0f;
            timer_limit = Random.Range(1.0f, 10.0f);
            move_bool = Random.Range(0, 4);
        }
        if (transform.localPosition.z < zMin) {
            RandomizePosition();
            timer = 0.0f;
            timer_limit = Random.Range(1.0f, 10.0f);
            move_bool = Random.Range(0, 4);
        }

        if (timer > timer_limit) {
            RandomizePosition();
            timer = 0.0f;
            timer_limit = Random.Range(1.0f, 10.0f);
            move_bool = Random.Range(0, 4);      
        }

        if (move_bool != 3) {
            MoveGameObject();
        }

    }

    private void RandomizePosition()
    {
        position = new Vector3(Random.Range(-200, 200),transform.position.y, Random.Range(-200, 200));
    }

    private void MoveGameObject ()
    {
        Vector3 angle = position - transform.position;
        if(angle.y != 0)angle.y = 0;
        if(position.y != transform.position.y)position.y = transform.position.y;
        if(transform.position != Vector3.MoveTowards(transform.position, position, moveSpeed * Time.deltaTime))
        {
            transform.position = Vector3.MoveTowards(transform.position, position, moveSpeed * Time.deltaTime);
            transform.rotation = Quaternion.RotateTowards(transform.rotation,Quaternion.LookRotation(angle),turnSpeed * Time.deltaTime);
        }
    }

}

Resources used:

• https://www.circuitbasics.com/arduino-ohm-meter/
• https://www.alanzucconi.com/2015/10/07/how-to-integrate-arduino-with-unity/
• https://answers.unity.com/questions/336663/random-movement-staying-in-an-area.html
• https://answers.unity.com/questions/1456945/how-to-make-a-gameobject-move-randomly-withing-a-b.html

Final Documentation 12-14-20

(1. Gumball machine with finished capsules inside.)

(2. The four capsule types and what they contain.)

(3. All of the insert suggestion slips laid out.)

(4. Another picture of the gumball machine, empty.)

(5. Capsules in hand, for size comparison.)

(6. An example of multiple capsule types combined.)

(7. Video demo.)

Narrative Description:

Gachaplunk is a collection of capsule toys designed to be used as instruments, with the ability to combine and mix the capsules to produce increasingly complex and unique instruments the more capsules one has. The capsules are contained in a gumball machine placed in a public place, meant to be obtained one at a time and played with on one’s own time. The nature of the gumball machine means that there is an element of chance in obtaining a capsule outside the user’s control. Each capsule is a toy within itself, containing a component that allows it to be interacted with alone or together with multiple other capsules if desired. Each capsule also contains a fortune cookie-inspired “suggestion”, which provides a thematic idea in order to inspire new modes of use. Through the interlocking systems of physical component, thematic idea and the ways these can be extended and combined, each user can create systems completely specific to themselves and their own collection. This is designed to create an environment for exploration, chance, abstract thinking, and growth over time.

Five Process Images:

(1. Spray-painting the capsules.)

(2. My original ideation for the project.)

(3. Original order of capsules were much too large for the machine.)

(4. Development of the ideas and themes of each of the capsule colors.)

(5. Production difficulties in applying sandpaper to the spray-painted surface of the capsules.)

Process Reflection:

This project was a breath of fresh air compared to the last two, which were much more programming base. I really latched onto the idea of modular design with the theme of “Emergent Complexity” with this unit. Because of this, however, my process had to be much more abstract and open-ended. There wasn’t quite as linear of a process as transducing data. The modularity aspect required me to come up with many separate ideas, and consider how this would affect each other idea individually and collectively. This ended up taking a lot of time, because I wanted each of the four capsule colors to be pretty distinct in how they encouraged physical engagement (via their component), and how they inspired artistic/performance ideas (via the suggestion slips). Overall, I believe this was pretty successful, and I like what I ended up with. I believe I am more comfortable being open ended in my instruction – providing suggestions rather than strict instructions.

The actual act of fabrication was relatively new to this project as well, which I enjoyed. I did have initial difficulties finding the right kind of capsules. But I loved the colors they turned out with. I do believe I want to continue working on this project, possibly adding more capsule types, or just refining the idea in general.

90% Complete Critique: 12-2-20

I have spent the last week and a half developing the the theoretical and thematic aspects of the work, figuring out how the system “works”, so to speak.

A larger order of clear spherical capsules is supposed to be coming in tomorrow, but I actually did happen to find some by digging through a drawer of mine. In this image, I’ve outlined the four different colors of capsules and what they contain. Each of the four colors has a physical component, providing a unique method of interaction. The red capsules contain ball bearings, facilitating “shaker” like percussive interaction. The yellow capsules have a small sandpaper strip on the outside, encouraging two capsules to be struck or scratched together. The blue capsules come with a string “connector” which allows two capsules to be connected together. The green capsules are filled with a ball of modeling clay to act as a weight, which adds mass as an element to be played around with when combining capsule components.

Capsules also contain “suggestions” written on strips of paper to inspire methods of using the capsules as instruments.

Here are some prototypes for the “Red” and “Blue” type capsules:

I’ve also obtained the materials necessary for completing the rest of the capsules and finishing the project.

Draft project statement/narrative description –

Gumball Machine Modular Instruments (temporary title) is a collection of capsule toys designed to be used as instruments, with the ability to combine and mix the capsules to produce increasingly complex and unique instruments the more capsules one has. The capsules are contained in a gumball machine placed in a public place, meant to be obtained one at a time and played with on one’s own time. The nature of the gumball machine means that there is an element of chance in obtaining a capsule outside the user’s control. Each capsule is a toy within itself, containing a component that allows it to be interacted with alone or together with multiple other capsules if desired. Each capsule also contains a fortune cookie-inspired “suggestion”, which provides a thematic idea in order to inspire new modes of use. Through the interlocking systems of physical component and thematic idea and the ways these can be extended and combined, each user can create systems completely specific to themselves and their own collection.

Maquette Presentation: 11-23-20

The gumball machine, successfully lockpicked.

I ordered some round, colored plastic capsules. Unfortunately, I assumed when they were referenced as 2 inches in circumference, they were actually 2 inches in diameter… I will need to find a new source of capsules, probably around 1 inch diameter.

Here is a video of me testing the gumball machine, which does require quarters, with some regular acorn shaped capsules:

Things I already have:

Gumball machine, lockpicked and usable

Ball Bearings

Magnets

Things I still need to get:

Colored, spherical capsules, ~1 inch diameter (a source still needs to be found for these)

Other fillings for the capsules, once the system is more fully developed.