Project Objectives

Lunch Time for the Puppy is an interactive children’s fabric book. The book is made from felt and other fabric material with different textures, and is embedded with soft sensors and electronic output elements for creating a rich storytelling experience. We produced a proof of concept prototype of the design by making one page from the book for demonstration. Our vision for the project is to have an entire book fabricated with similar methods with the main character, the detachable interactive puppy. The complete story would contain various scenarios of the puppy at places around the house.

Interaction and Story Outline

The puppy is designed to be detachable and responsive, able to be reattached to various positions throughout the fabric book. The fabric book contains different scenes of a puppy’s day and each page corresponds to different behaviors of a puppy. For example, petting the puppy in the shower is different from petting the puppy before going for a walk.

Creative Design Opportunities

With our successful implementation and modification of the methods provided by the Multi-Touch Kit (reference 1) research paper, we believe that by using this technique, capacitive sensing with off-the-shelf microcontroller board can be used for prototyping and designing touch-screen-like interactions for fabric, static flat surfaces and surfaces of other 3D objects. The technique can be used for augmenting children’s fabric books with digital interactions additional to the original material based interactions. Soft sensor grid can be integrated with the design of other soft materials for a more unified look of the book.
We see the possibility of creating open-ended storylines with adding different states of the behaviors. The rich interactive quality also helps reduce screen time for children.

Prototyping Process

In this section, we will discuss sensor iterations and software modifications during the prototyping process.

Sensor Iterations

We fabricated our sensors using instructions from the original research project. The materials and tools we used were copper tapes, think double sided tapes, paper, scissors, and exacto knives.

The very first grid we made was of size 3×3. Copper tapes are taped to opposite sides of a thin plastic sheet. We made this to test if the provided multi-touch kit works at all.

We looked for ways to ease the fabrication process. We found the sensor design guideline and printed out the patterns for easy tracing. After a few attempts we fabricated the sensors using the following process:

• Cut off a piece of paper that contains a strip of the grid from the printed pattern
• Tape the strip to do the back of the conductive tape
• Use the taped conductive tape along the traces
• Use the exacto knife to cut the vertical traces for the think lines connecting diamonds
• Use the scissors to cut along the diamonds
• Peal off the backing of the conductive tape to tape it with the guidance of the printed pattern

The second grid we made was of size 6×6. The top layer of the copper tapes are taped on plastic sheet. The bottom layer of the copper tapes are taped on the paper. And the plastic sheet is taped on top of the paper. This sensor grid had issues with not recognizing light touches, so we suspected that it was caused by the gaps between the plastic sheet and the paper.

The third and the final grid we made was still of size 6×6. The copper tapes are taped the same as the first sensor grid: copper tapes are taped to opposite sides of a thin plastic sheet. This sensor worked relatively reliable and light touches could be detected.

Software Modifications

We built our software implementation using the Processing sketch from the original research project. Here is what the sketch does:

• Read analog values from the pins
• Set Baseline
• Use BlobDetection to identify&locate touches
• Use OpenCV to visualize the touches

We extracted the result of blob detection to do gesture recognitions.

Initially, we wanted to be able to implement gestures recognitions of the set below.

We then ran into some troubles when trying to detect sliding behavior, partly because of the grid is actually low resolution, and it cannot slide from touching multiple positions at the same time. Thus we decided to put sliding apart and use the other interactions.

Above is the API we ended up implementing. Note that we use the number of continuous taps to detect slides. Using the available gestures, we were able to map them to output behaviors shown below.

Outcomes

In this section, we will discuss the successes and failures of our choices, things we’ve learned, and how future work would further improve the results.

Successes

• Successful implementation of the multi-touch kit with software modification tailored to our purpose: With the limited resources we had, a lot of our time were spent on how to fabricate the sensor grid. Though our hand-cutting method is nowhere near scalable, it is sufficient for a small sensing area like the one we have.
• Exploration of interactive fabric book with more complex interactions: Earlier works on tangible interactive books have focused on using simple e-textile inputs(e.g. buttons, sliders, and etc.) and thermochromic paint. We believe that our project is a proof of concept for interactive fabric book with un-constrained interaction sites. In other words, the entire page or a detachable piece of the page could act like a touch screen. Further research would be needed to determine effects of such interactions.
• Variations of behaviors: To bring the puppy more to life, we implemented a very naive state machine so that the puppy has different states that represent different moods. The states are very limited right now and not all state changes have corresponding outputs, future works could further the complexity of the state machine.

Failures

• Sensor grid robustness: The sensor grid has a low resolution because our sensor grid is very small. With a 6×6 grid, there are 36 ‘pixels’ in total, but a usual touch contact can easily cover up to 10 pixels. The use of blob detection also introduces errors like a single touch with a too large of a contact area could be detected as two separate touches. We suspect that some of these detection errors might be eliminated with a larger sensor grid, i.e. a larger sensing surface. One may asks about the diamond size and spacings, we chose the size and spacings as recommended by the original research project. Further works could look into replacing blob detections with some other algorithms to detect touches.
• Missing opportunities provided by the multi-touch kit: Many current interactions we hae can be done with simpler inputs. For example, taps/presses can be detected using a fabric button or capacitive touch sensing. Though we do use touch coordinates to decide whether the touches are the same as before for each time loop() is called, we are not taking advantages of the rich possibilities offered by location information. Further works could explore more gestures such as direction slides, pinches, and other gestures that a 2D touch screen can recognize. One thing to keep in mind here is that although we are mimicking a touch screen using fabric, we should still remember the unique tangible interaction opportunities that soft fabric brings. For example, pinching fabric is drastically different from a hard touch screen as the fabric will be pinched/folded as well.
• Unnatural interactions compared with interacting with a real puppy: Right now the sensor grid is placed behind the ear of the puppy and thus makes the interaction of reaching to the back very natural. It was a decision between the aesthetics and the normality of interactions, and we chose to hide the sensor grid. A limitation of the multi-touch kit is that fingers must directly contact with the top conductive layer. The question about using resistive sensing technologies instead of the capacitive one that we chose was raised, and we believe that resistive sensing complicates the fabrication process and causes a more unnatural interaction as it requires firmly pressings on the sensor. For future work, one could use conductive thread to directly sew the sensors onto the fabric and to have the sensors at more natural locations.

Due to limited resources, we weren’t able to integrate as many different textures as we hoped for. Moving forward, we believe that adding more textures to the body of the puppy would allow a more diverse tactile experience.

Sources

• Hardware requirements and schematics can be found from the tutorial by the original research project.
• Software source code(README has instructions on using the arduino and processing sketches.):

Group member contributions

Equal contributions:

• Preliminary research
• Project scope definement
• Storybook storyline
• Interaction design
• Testing/prototyping the sensor grid
• Weekly reports
• Final system troubleshooting

Catherine’s additional contribution:

• Software implementation
• Troubleshooting on a breadboard

Yanwen’s additional contribution:

• Fabrication of the fabric page
• Integration of soft technology components with fabric materials

References

Narjes Pourjafarian, Anusha Withana, Joseph A. Paradiso, and Jürgen Steimle. 2019. Multi-Touch Kit: A Do-It-Yourself Technique for Capacitive Multi-Touch Sensing Using a Commodity Microcontroller. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (UIST ’19). Association for Computing Machinery, New York, NY, USA, 1071–1083. DOI:https://doi-org.proxy.library.cmu.edu/10.1145/3332165.3347895

Jie Qi and Leah Buechley. 2010. Electronic popables: exploring paper-based computing through an interactive pop-up book. In Proceedings of the fourth international conference on Tangible, embedded, and embodied interaction (TEI ’10). Association for Computing Machinery, New York, NY, USA, 121–128. DOI:https://doi-org.proxy.library.cmu.edu/10.1145/1709886.1709909

Irene Posch. 2021. Crafting Stories: Smart and Electronic Textile Craftsmanship for Interactive Books. In Proceedings of the Fifteenth International Conference on Tangible, Embedded, and Embodied Interaction (TEI ’21). Association for Computing Machinery, New York, NY, USA, Article 100, 1–12. DOI:https://doi-org.proxy.library.cmu.edu/10.1145/3430524.3446076

Project Objectives:

grubo is a set of soft pneumatic actuators compatible with the Lego Technic system. The kit consists of expandable silicone air chambers, Lego-compatible connector pieces, and tubing and syringes to inflate the air chambers. Using the grubo pieces, builders can expand the functionality of their Lego creations with soft pneumatic actuation.

Creative Design Opportunities:

We imagine that the findings from our project can be implemented in various contexts, they are:

Within an educational institution: We imagine this set to be useful in providing a low barrier of entry for students in elementary schools to introduce students to pneumatic actuation through a method of creative play with which they are already familiar. Students may be given kits with Lego and grubo components, as well as instructions to guide students through projects that take advantage of the pneumatic actuation. This could also be implemented within other educational facilities like children’s museums, where interaction and collaborative play is encouraged.

A Lego Technic Set Add-on: A second potential avenue is to productize the kit. This concept is similar to existing Lego expansion kits that can add, for example, motorized functionality to already existing kits as well as new creations. In addition to being included in add-on kits, grubo components could also be incorporated into novel Lego sets, using the soft actuators to meet a specific designed end. We imagine this could interest established Lego fans.

grubo in Sculpture: The movement of the silicone components could also be used for biomimetic, sculptural ends . We named this project, somewhat facetiously, as grubo: a portmanteau of the words “grub” and “Lego,” because of the parts’ movement evoking images of larvae. Artists could employ these more complex versions of these connectors in artistic works to create dynamic, lifelike movements.

Outcomes:

Over the course of this project we developed a proof of concept for a kit of soft, inflatable, Lego-compatible components. While we only produced one one type of inflatable (one that bends when inflated) and one type of Lego connector (a 5-unit long male-male connector), we created parts that lay a clear groundwork for a whole line of similar pieces. The novel components also add new functionality to a Lego kit of parts, adding flexible, pneumatic actuation Technic-style builds.

We also developed a method for producing silicone inflatables with geometrically complex inner cavities. By making paraffin wax internal negatives and then melting them out of the cured silicone parts, we were able to produce monolithic silicone parts with internal cavities too large to remove rigid negatives from.

While we regard this project as an overall success, we have left notable room for improvement. By using a single, round connection point, the silicone parts can rotate around the air nozzles without much user control. Given the non-rotationally-symmetric nature of the silicone parts, this results in Lego builds that could be challenging to keep in a consistent state. Furthermore, the bistable connector nozzle is significantly narrower than the mouth of the plastic nozzles, a design choice that increases stability of the connection, but makes it challenging to easily mount the silicone pieces onto the plastic nozzles. This would likely prove a particular challenge for children, a target demographic of these toys. Additionally, we put not thought into how the syringes interact with the Lego system, so the syringes feel very unintegrated into the wider kit. Lastly, given the narrow range of parts we developed, the system lacks flexibility. Future iterations of this project would benefit from prioritizing solutions to these issues.

Citations:

While we looked at a lot of resources developing this project, we took primary inspiration from the following three:

1. Chao Zhang, Pingan Zhu, Yangqiao Lin, Zhongdong Jiao, Jun Zou. “Modular Soft Robotics: Modular Units, Connection Mechanisms, and Applications.” Advanced Intelligent Systems, Volume 2, Issue 6, page 1900166. 2020. doi: 10.1002/aisy.201900166.
   This paper presents a number of methods for creating modular soft robots. We found the high-level design specifications of our bistable connector from this paper.
2. Jun-Young Lee, Jaemin Eom, Woo-Young Choi, Kyu-Jin Cho. “Soft LEGO: Bottom-Up Design Platform for Soft Robotics.” 2018 IEEE/RSJ International Conference on Intelligent Robotics and Systems, Madrid, Spain, pages 7513-7520. 2018. doi: 10.1109/IROS.2018.8593546.
   A primary conceptual inspiration for our project. This paper presents another Lego-compatible inflatable soft actuator. These parts are compatible with the bricks rather than the Technic elements, and we through there was room to do more with the concept than was being done in this paper.
3. Golan Levin, Shawn Sims. Free Universal Construction Toy. 2012. url: http://fffff.at/free-universal-construction-kit/.
   The Free Universal Construction kit was another conceptual inspiration for our project. Similar to the previous citation, this work presents a kit of parts that can expand the functionality of the Lego (and other branded) building systems.
4. X. Lu, W. Xu, X. Li. “A Soft Robotic Tongue–Mechatronic Design and Surface Reconstruction.” IEEE/ASME Transactions on Mechatronics, Volume 22, Issue 5, Pages 2102-2110. October 2017. doi: 10.1109/TMECH.2017.2748606.
   The Soft Robotic Tongue provided a primary inspiration for the inflatable component we produced. This paper presents inflatable actuators as part of the tongue that, when inflated, provide a bending motion along one plane. While the actuators in the cited paper are more complex than what we produced, the basic design was adapted for the purposes of our project.

Technical Documentation:

A folder containing zips of our CAD files can be found here.

Contributions:

Sebastian: I focused primarily on developing the inflatable components. I designed the 3D-printed molds. I also figured out how to make the meltable wax internal components, and poured most of the casts.

Elena: I designed the hard plastic piece, focusing on the rigid bistable nozzles and adjusting tolerances for Lego compatibility. I also experimented with the bistable connection necks to find the diameters that worked the best.

Together: Both of us worked together to develop the concept for the project and to carry out early balloon and silicone experiments. Towards the end, we both worked together to make edits/fill in gaps left by the other with regards to part manufacturing and documentation.

We used the Multi-Touch Kit Software Toolkit and attached it to the back of the left ear of the puppy. The only other digital intput is a fabric button hidden underneath the puppy’s belly.

For outputs, we combined visual, audio, and haptic feedbacks using LEDs, the laptop speaker, and vibration motors.

To minimize the expected feeling of having some changes after doing something, we implmented a realatively naive state machine so that the puppy could be in different moods.

Due to time constraint, we used conductive copper tape instead of conductive yarn to fabricate the sensor grid. Between the aesthetics and the normality of interactions, we chose aesthetics to hide the sensor grid. If we fabricated the sensor grid using conductive yarn, we could interact with the top of the furry ears with gestures that are more similar to how one would pet a puppy.

Due to material constraint, we weren’t able to integrate as many different textures as we would like. Adding more textures to the body of the puppy allows a mroe diverse tactile experience.

This is only a prototype of a single page, we invisioned a puppy-themed interactive book of which the interactive puppy is like a bookmark. Every page is a different setting, and placing the puppy on the page triggers the start of the interaction to tell stories about puppy’s different behaviors and reactions in different settings.

circle:

“pie”

single cup fold:

multiple cupping folds:

circle control:

“spork”

lined spoon:

short spoon control:

winged spoon:

control winged spoon:

spoon:

control spoon:

rectangles:

Factors affecting repeatability:

Corners: I noticed that the cuts at the intersection of the oval and the rectangle sometimes had slits just by error of hand-cutting. This is likely what created inconsistencies in the folding of these pieces. In the spoon attempt above, the rectangular section folded similarly each time.

initial curvature of pieces: when the sheets come out of the printer, some pieces are slightly warped (from heat) to begin with. When put on the hot plate, this caused some parts of the shape to not be touching the hot plate at all, which affects how it is heated.

imperfections in line thickness/weight: most consistent results came from samples that went through the printer. this ink was the most opaque and was uniform in thickness all the way through, unlike hand-drawn lines. Thicker lines responded faster and more prominently than thin lines.

Takeaways

• Lined samples act as an expedited method of folding which allows for more direction/control vs. the randomized creasing seen in control groups
• Material should be placed where the heat lamp is centered due to weaker light distribution as it radiates outwards
• If the plate is too hot, the material will instantly melt and morph with bubbles
• Other pieces can’t attach together without external pressure

Replicable Hinge Fold

Circular Cup Fold

Final Spoon Folds and Forms

This week we finalized the folds we are including in our final report and are doing one more lab with the printed shrinky dink paper to make sure the folds are replicable. We plan on finishing up photos and design language system graphics for the final this week. Below are the final forms.

01 Pinch Fold | Trapezoidal

02 Cupping fold | Perimeter

03 Spork Attempt | Defined Edge and Crinkle Folds

This shape was pretty consistent among the 3 attempts.

4/29 Pinch and Crinkle Folds

Why: Taking one look at this project, one might immediately think “more plastic utensils?” but the reason we decided to focus on utensils isn’t because of the materiality of polystyrene rather for the unique forms it can morph into under heat. We find potential in the application of low-touch, heat-activated morphing in the medical field where many tools are single-use for sanitary purposes. Because our knowledge base isn’t in medicine, we chose to explore the system of kitchen tools due to their universality and their uniqueness in shape and bends. We see this as a starting point to discover more accurate and technical patterns that can evolve with the shrinky dink polymer.

Weekend Update: We continue creating a system of folds for our final set of utensils

New Research:

Precedent of organizing/naming curved origami from “British Origami Society”

Iterating on the spoon design:

Ray used some thick tin foil to test the fold design. Thick tin foil bends more similarly to the shink paper compared to paper. This makes it a better tool for prototyping curving designs.

This week we finalized the folds we are including in our final report and are doing one more lab with the printed shrinky dink paper to make sure the folds are replicable. We plan on finishing up photos and design language system graphics for the final this week. Below are the final forms.

01 Pinch Fold | Trapezoidal

02 Cupping fold | Perimeter

03 Spork Attempt | Defined Edge and Crinkle Folds

4/29 Pinch and Crinkle Folds

Why: Taking one look at this project, one might immediately think “more plastic utensils?” but the reason we decided to focus on utensils isn’t because of the materiality of polystyrene rather for the unique forms it can morph into under heat. We find potential in the application of low-touch, heat-activated morphing in the medical field where many tools are single-use for sanitary purposes. Because our knowledge base isn’t in medicine, we chose to explore the system of kitchen tools due to their universality and their uniqueness in shape and bends. We see this as a starting point to discover more accurate and technical patterns that can evolve with the shrinky dink polymer.

Weekend Update: We continue creating a system of folds for our final set of utensils

New Research:

Precedent of organizing/naming curved origami from “British Origami Society”

Iterating on the spoon design:

Ray used some thick tin foil to test the fold design. Thick tin foil bends more similarly to the shink paper compared to paper. This makes it a better tool for prototyping curving designs.

Inflatables

Given the holes resulting from bubbles in our last attempt at casting silicone inflatables, we worked on a redesigned mold. This new mold added a Lego-compatible connector and reoriented the molds to create a larger area to pour the silicone. We made casts in both molds, but the mold with the open flat side proved to be the most successful.

Bistable Connectors

We found that the thinnest lip of 1.1 mm was the most successful, producing a successful and repeatable invert upon part-insertion.

Silicone

We cast a prototype for the bistable connection. The cast came out nicely, but we still need to test it with the plastic nozzle to make sure it works properly. If it doesn’t we’ll need to assess whether the issue is a result of the design of the silicone component, the plastic bit, or both.

Wax

We cast the internal pieces of one of our inflatable components from paraffin wax. After a bit of experimentation, we produced a successful cast of high enough quality that we can use it for a silicone mold.

This cast was produced in two parts, a base piece and an auxiliary piece, connected to each other by an 1/8in dowel. The base piece cast beautifully on the first attempt. Our first attempt of the other part failed the first try, as some bits of the piece’s geometry broke off when we removed it from the mold. In our second attempt, we coated the plastic mold in a release agent before pouring the wax. With the release agent, we were able to easily separate the wax cast from the mold after hardening. We will attempt to use this wax piece to make a silicone cast this week.

Weekend Update: We continue creating a system of folds for our final set of utensils

New Research:

Precedent of organizing/naming curved origami from “British Origami Society”

Iterating on the spoon design:

Ray used some thick tin foil to test the fold design. Thick tin foil bends more similarly to the shink paper compared to paper. This makes it a better tool for prototyping curving designs.

We tried adding a fabric layer(felt, muslin, and t-shirt fabric) on top of the sensing grid to see if detections can still be made. Unfortunately it did not work, and even layering a sheet of a piece of paper significantly weakens the raw data.

After looking into the library together with Garth, we took another look into the blob detection library used. The next step on the software side would be writing code to detect swiping and the speed of swiping. We plan to use the normalized coordinates of the blob’s center returned by the blob detection function.

We discussed on possible page designs and materials needed. Below is a rough sketch of the possibilities.

For the upcoming week, we will work on different tasks:

• Catherine: Work on the software.
• Yanwen: Investigate outputs mappings.

New Research:

Precedent of organizing/naming curved origami from “British Origami Society”

Iterating on the spoon design:

Ray used some thick tin foil to test the fold design. Thick tin foil bends more similarly to the shink paper compared to paper. This makes it a better tool for prototyping curving designs.