Sketch Garden
One hundred recorded attempts to draw the same flower in a drawing puzzle game.
Author: Jacqui
Jacqui – Final
Sketch Garden
One hundred recorded attempts to draw the same flower in a drawing puzzle game.
For CMU’s latest puzzle hunt, I wrote a drawing puzzle game called Stroke of Genius (pictured above,playable here) with my friend Tom Wildenhain. Players are presented with a simple drawing tool and asked to draw three images (tracing guidelines are provided). The catch is that every colored “paint” has a different unexpected behavior. For example, pink strokes follow your cursor when you finish drawing, green strokes self-destruct as you add other strokes to the canvas, and yellow strokes are duplicated rotated 180 degrees about their start point. This idea of an antagonistic drawing tool was inspired by Ben Fry’s fugpaint, but as far as I know such a tool has never before been turned into a game.
During the puzzle hunt over 500 participants played Stroke of Genius, and I recorded their keystrokes. I took 100 attempts at the first level (which gives users 3 strokes to draw a particular flower) and organized them in a grid based on what approach they took, with the more successful attempts closer to the center. This format of displaying many drawings in a grid was inspired in part by Aaron Koblin’s Sheep Market.
It’s interesting to compare the different attempts at the puzzle, especially when you’ve solved it yourself, because in many cases the player’s thought process is clear through their strokes. You can see when people stop and realized they’ve made a fatal error, or become frustrated and angrily scribble over everything. The attempts are quite different, but they’re still definitely drawing the same flower. I find the final result pretty mesmerizing and easy to watch for a long time. However, there’s still a lot of interesting data that I haven’t used yet, and analyses I haven’t performed.
Sketch Garden is accessible live here.
Though they aren’t part of the main project, you may also be interested in viewing 100 attempts to solve the other levels of Stroke of Genius. These are not sorted and only minimally filtered: fish level attempts and boat level attempts.
In case you’re interested, my reference correct solution to the flower puzzle is below. One can also complete the level by drawing green first (with a much longer tail at the beginning) and then drawing pink, making sure to stop in the center. Note that since the stroke layering matters, green must be drawn before yellow.
Making Of
I was originally working on a different project for this course, and treating Stroke of Genius purely as a personal project for the puzzle hunt. I spoke to Golan a few days before the hunt and expressed that I was unsatisfied with my current project. He advised me to turn Stroke of Genius into my final project, adding some code to the tool that would log participants’ strokes so I could play their drawings back later. I loved this idea, and quickly implemented it. Over 1,400 people participated in the hunt, so I ended up with over 18,000 attempts across all levels of the game.
I was originally going to do this project about the final level, the Boat, because it was the most difficult. But when I started looking through the data, I found that the boat recordings were not as informative as the flower recordings. The puzzle was so hard that most attempts were given up on early, and when an attempt failed, it was hard to tell what the participant was trying to do. The flower level, by contrast, is the simplest puzzle-wise. The antagonistic tool is just enough to be annoying. There are a few common mistakes that people make, and they are all pretty apparent to an outside observer. This gave the flower recordings the understandability I wanted, so I decided to use them for the project instead.
However, I couldn’t use all of the thousands of flower attempts I had recorded, so I filtered out several undesirable properties: drawing very little before giving up, drawing more than the allowed three strokes (because a solution could look correct but be invisibly incorrect), using the undo button (convenient for the game, but makes the recording less clear), using entirely the wrong paints (I didn’t want to introduce other colors, like blue or purple, to the grid), etc. From the remaining attempts, I randomly chose the 100 I would use in the grid. I chose a 10×10 grid because I wanted to have many densely-packed flowers, but would have to make them smaller than I wanted to fit into an 11×11.
Once I knew which attempts I was using, I needed to order them. I initially wanted to use tSNE to sort them, but had trouble getting that to work and also wasn’t sure it would use the criteria I had in mind. So I manually (and somewhat coarsely) organized them according to what I perceived to be their general approaches. For example, in the upper left, you have the people who started with pink and gave up quickly after. In the center right, you have people who drew green and then pink (a valid approach), but ended up having their leaf erased by their other strokes. The best submissions are in the center (or slightly left and top of center, since we had SO many green attempts).
I’m really pleased with the final result; from a distance the whole thing is aesthetically pleasing, but up close each flower has its own story to tell. I like the one in the second row and seventh column, because you can really see their “damnit” moment when their green stroke starts to self-destruct. Meanwhile in the eighth row and third column, their green starts to self-destruct and they just keep going. They get all the way done with pink before sadly wandering their cursor around the screen.
I have so much data from this project that so far went unused, and I hope to do something else interesting with it when I have a bit more time. Perhaps a heat map of people’s mouse locations? Or combine it with the game itself so once you complete a level, it shows you “People who solved it like you.” With enough filters and manual selection, I also might be able to get a presentable grid out of the fish or boat data.
Final Project Update
For my final project, I’m going to be making a short 360 video set in my grandparents’ home of 50 years, featuring mostly still photos (no animation) and my grandparents’ voices. Right off the bat, I’ll say: I am behind where I would like to be, because it took me so long to decide on a narrow topic. I still think I can finish it, but without as much time to polish as I’d like.
I decided to focus on stories from Christmastime, specifically of adapting family traditions for unusual circumstances. There are a few reasons for this. First, a hugely disproportionate number of our family photos and photos in that house are of Christmas celebrations. So I can really fill the space with these images, from all angles. Second, it’s a theme we have multiple interesting stories for, and one that feels very relevant to our current situation. It may not be Christmas, but families are adapting their rituals and traditions for the quarantine, often in ways that resemble the stories I’ve collected.
Once I landed on this theme, I set several things in motion that are currently still underway: I told my grandparents about this topic and arranged a phone call with them in which I will record their voices for the project, and I also reached out to my mom and aunt (who are in possession of all our old physical and digital photo albums) for ALL the Christmas photos they have. I should be getting those in late this week, and doing the editing early next week.
In the meantime, enjoy the following images of my family at Christmas, and recordings of my grandparents telling some holiday stories (recorded last semester for a different project, but I may just end up using them because the quality is pretty good).
I won’t likely be able to use all of the photos I’m going to be sent, but I bet I can fit a lot in. The film will be “set” in the living room, where all of these photos were taken.
Objects of My Isolation
Headphones – I used to have to wear these most of my waking life. When I went out, I’d wear them for my Misophonia, and when I was at home, I’d wear them so whatever I listened to wouldn’t distract my roommate. Now that I’m always at home and my roommate is gone, I never really need them. But I find myself still listening to most things through my headphones because I’ve gotten so used to the way it makes everything sound.
Seaweed – My roommate never came back to campus after Spring Break, so she left behind a lot that wasn’t worth mailing back. I’ve inherited several things from her this way, including a lot of food that I typically wouldn’t buy. I never really ate seaweed plain before this, but now I have a huge 10-pack!
Whiteboard – This whiteboard has been on the door of my room since freshman year, but since the quarantine it’s been repurposed for 15-251, the course for which I’m a teaching assistant. I use it to hold office hours and recitations (sometimes paired with a document camera). It’s actually pretty effective.
Microwave – Previously, I got most of my meals from campus eateries or local restaurants. I used this microwave to make tea or sometimes reheat leftovers. Now I’m using it multiple times a day. From pasta cups to mug cakes, much of my food comes out of this microwave.
Blinds – Most of my time at home used to be during the night. Since there wasn’t sunlight, I didn’t need to open the blinds, and so I closed them for my privacy. They probably went months without being opened. But now I sometimes open them to look outside or just let light in. I also find myself much more aware of where the sun is during the day because it affects the lighting in my video calls.
Webcam – I didn’t use my webcam for much before this, so it always made me a little uncomfortable to see that little green light turn on. I’d have to perfectly stage every video call. Now I’m 100% over that. I’ll hold a meeting from my bed while wearing my pajamas, and that light doesn’t phase me at all.
Microwave – Previously, I got most of my meals from campus eateries or local restaurants. I used this microwave to make tea or sometimes reheat leftovers. Now I’m using it multiple times a day. From pasta cups to mug cakes, much of my food comes out of this microwave.
Shoes – These were and are my go-to pair of shoes. I used to spend most of the day in them, but now I’m almost always barefoot. When I want to go outside, I have to get out these shoes and a new pair of socks, and it feels like a major event.
Polyhedral Dice – My dad and sister and I have been playing D’n’D together over Skype since before the pandemic, but since it started we have a lot more time. We’ve spent many hours playing it in the past few weeks, and I look forward to our sessions as a much-needed social time. We each have our own set of dice to roll from our respective locations.
School ID – This used to be extremely valuable to me. I used it to buy food. I needed it to get into my room (and several other academic buildings). It was a library card and a bus pass. I’ve had it with me almost every day for the past 4 years. Now most of its applications are gone. I still take it with me when I leave my room, but it’s a lot less useful otherwise. And I’m getting bummed out by the expiration date.
April Plan
As per our earlier discussions, I plan to make a project (a short 360 video) using the data I’ve already collected of my grandparents and their home (360 photos, old family photographs, and audio recordings). Though I could see this project featuring multiple rooms in the house, I will only focus on one room for the end of this semester. If I expand the project later, this room could be a single installment or “episode”. I guess you could call this a final project, but I’m not sure how final/polished it will be. I’m mostly using it as an exercise in editing for documentary.
I also intend to complete offerings as much as I can, and go back to do some of the older ones I missed (they seem fun, but this has been a busy week for me).
Panoramic App Misuse
I don’t know if this is app misuse so much as just… unusual app use, but it captures images of things in a way unlike they actually are, so I’ll count it. I used my phone’s native panoramic camera to do a few experiments.
Un-bending Bendy Things
For these images I sought out real-world items that were curved and tried to move the camera as I photographed them so that they came out straight in the final image. It only really worked with things that were curved in a horizontal plane, because my app did not allow me to rotate my camera in any other direction. Here are some of my results:
Drawing with Light
I brought a flashlight out to photograph a wall at night, and tried moving it in various ways as I took the panorama.
Some “Learning to Love You More” Assignments
- Take a flash photo under your bed.
When I got to CMU, I purposefully raised my bed to have more storage space underneath. Unfortunately, as I am mid move-out, it isn’t as densely packed as usual. The assignment also says “We are looking for photos that depict the space between the bottom of the bed and the floor, please do not send us photos if your bed is flush against the floor or if it is a loft bed.” and since I actively use the space for storage I’m not sure how much it counts. But there’s no harm done in taking the picture.
2. Ask your family to describe what you do.
– My Mom: [she] builds and uses existing interfaces so that she can create beauty or amusement for or unique expressions of or enhancements to human life through the medium of technology
– My Dad: I imagine that Jacqui wakes up and checks her e-mail, making any responses that are necessary in a timely manner. She cleans up, dresses and grabs a pre-made breakfast item and bottle of water on the way to her first class or appointment. In between classes or meetings, she will chat on-line to colleagues about assignments, or about any of the various extra-curricular activities in which she involved. Lunch will be an ad-hoc affair as opportunity allows. As the afternoon progresses, there will be time for some computer work, as well as some on-line diversions for U-Tube, games and topics of interest. Dinner will quite often be part of an extra-curricular group meeting or discussion among friends. In the evening, there might be a movie or a rehearsal to attend, followed by a late night of scholastic work occasionally punctuated with a board game or puzzle activity among friends.
– My youngest sister Elise (17): I think you go to classes to learn how to code better, do better graphics, etc. I think you make lines of code as part of a bigger program to make it function. I think you work digitally to make both film projects and computer animations.
– My Grandmother: I would say that my granddaughter works on the computer a lot, and she likes to do film, and she likes games. And she tries to create them in the computer. That’s all I know how to say.
I didn’t read her full description until after I had asked several people, so I have more than 3, each with kind of different interpretations of the assignment.
Playing with Distortion
To achieve some IRL distortion, I looked around my room and neighborhood for just about anything reflective or refractive I could find (that wasn’t meant for this, like a mirror or camera lens). None of my images are works of art so much as casual experiments, but it was fun thinking about how to approach this!
I made sort-of-triptych of distorted pictures through a glass jar held at several different heights. I love my eye in the first one!
For some physical distortion, I smushed my face with some clear plastic.
My favorite distorted image I captured, though, is this one. It’s me reflected in an outdoor electrical box. The metal is only kind of reflective, and it blurs more horizontally than vertically, which gives it a feeling of movement, though you can tell from the raindrops that the camera was still.
It looks almost like it could just be a weird camera filter, but no–it’s a sheet of metal.
Cameras on Tools
I was inspired by some of the moving camera projects, especially the one with the camera on the shovel. It’s such a simple concept that still provided an interesting (and enjoyable) perspective on a common tool. This got me thinking of other tools/devices I’d like to mount a camera to:
- A hammer – I think it would be really delightful to repeatedly slam the viewer down towards a surface. As long as one could effectively stabilize the camera.
- A pottery wheel – I thought it would be fun to watch a lump of clay sitting still in front of your be transformed by a rapidly spinning hand. And apparently, I’m not the only one who thinks so, because this one has been done by Eric Landon. It’s a pretty neat video!
- A paintbrush – This idea kind of reminds me of the interactive art project from last spring where you could draw with a stylus and “ride” it in real time with a VR headset. That really let you imagine yourself as one with your own writing instrument. But I think it could be differently informative to look down the handle of a paintbrush with a regular camera. You could watch the bristles as they are dunked into paint and rubbed across the paper in different directions. I think that moving camera would be interesting to apply to this subject because we are so used to thinking of painting/drawing as a person moving their tool across a surface. If the surface is the thing moving, would people watching be able to identify what is being painted? It could also help some people appreciate the differences in brushes and painting techniques more, by highlighting how some bristles are more flexible or more absorbent. Maybe I just still have typologies on the brain, but a set of recordings with different brushes and different paints or papers could be very interesting.
Person in Time Project Ideas
These all feel only tangentially related to the prompt, but I also think they all sound like a lot of fun, so…
- Exquisite Choreo – You know the game where someone draws a picture, then someone else describes the picture, then a third person draws a picture based just on the description, then someone else describes that, and so on? It’s like that but with dance. I would do a few rounds: someone dances to a clip of music and I record them, then someone else watches that video and tries to write a description for the movement. I show that description to another dancer, et cetera. Alternatively, I just do a series of video with the instruction “Copy this Movement,” like telephone. Alternatively, I just do Exquisite Corpse but with dance, asking “What should come next?” with limited context.
- Maybe more of a typology, but… there is this awful door into my residence hall that abruptly stops after you pull it a few inches open. I have enjoyed watching people’s disappointed faces when trying to open this door, especially when I can tell this has happened to them before. I’d love to photograph/film a series of reactions to this. Maybe I could even rig something to do that automatically?
- Study for Fifteen Points but YOU are the points – I am currently obsessed with Study for Fifteen Points and I thought it could be fun to recreate it with people instead of robot arms. There are a few ways to do this: firstly, I could give each person a stick with a light at the end, teach them how they should move it, and have them “perform” this somewhere. The “person” they depict could be larger than life, outdoors in the evening somewhere on campus. Could be fun. Alternatively, I could make this something of a game for fewer people, where they are each controlling an interesting point or two and trying to make something that a computer will recognize as a walking. This would force people to consider how their joints move.