Cre·ate (verb) is a participatory video installation that invites the audience to create their own butterfly that follows the path of their hands using their hands. Regardless of background, anyone can create a butterfly of their own that flies along a distinctive path they defined.
Inspiration:
As AI based art generation tools like Dall E become more accessible to the public, I started to wonder what it means to create. What makes one to be the author of the creation? If an audience prompted the software like Dall E, which is developed by another person, and the software generates the artwork, does this work created by the audience or a developer or the software? Through this project, I wanted to explore what makes creation belong to an individual and how much of an authorship do the viewers take into consideration when seeing a collection of generated creations.
What I decide to capture:
Since creations are often made with hands, I decided to capture the movement of the creator’s hand during the act of creation. I also decided to make this an interactive audience participatory project instead of a project focusing on an artisan’s hands to challenge the general public’s misbelief in art making that only artists can make art.
Inspiration work:
Design-IO’s Connected Worlds project from 2015 is a real-time interactive piece that captures people’s pose and motion along with object placement then responds with changes in visuals on surrounding screens. Although my project doesn’t have a real-time interaction component that allows audience to change the positions of butterflies at any given time, both projects embodies the idea of inviting the audience to make changes in the piece.
Capture System Workflow:
To make audience be the creator, I developed a workflow where the creation happens upon a hand motion of opening a fist is detected (as if you’re showing something that you caught or made with your hand).
I used mediapipe and open CV to gather hand gesture and position information. The path a new butterfly follows would be determined by the trace of audience’s thumb root joint. When the audience is ready to create a new butterfly that would be following their hand trace, they would close their fingers to make a fist then open up the hand to release the butterfly. Doing so would write the path list to a file, which would then be read by Blender python at each frame to generate a new butterfly with an animation key points of the hand path locations. I limited the path list length so that Blender won’t freeze importing and creating large number of frame key points.
(Recorded Trace Shown)
(Pipeline view of interaction above)
I made the butterfly model and used color ramp node in shader graph to randomize the color of the generated butterfly. I randomized the color with intention of giving a bit of diversity in the generated butterflies, but now that I’m looking back, perhaps I should have the butterflies to take the creator’s visual information as part of the generation so that each butterfly can be more customized to the creator. One idea of this would be having the butterfly wing texture to use creator’s hand image as a base color.
(Installation Views)
As also mentioned in my project proposal, I chose butterfly as an object to be created because creation starts with one’s idea. Butterfly often symbolizes metamorphosis and soul/mind, which fits well to this project’s exploration of act of creation (bringing an idea to a physical space) and authorship (idea/concept). The collective colorful glow of the generated butterflies let the viewers to both appreciate the group of butterflies from far or identify the specific butterfly one has created based on the path it follows.
Reflection:
Despite my initial prediction that the viewers would be able to easily identify the butterfly they created based on the trace they drew by moving their hand, I realized this wasn’t the case especially if there are more butterflies flying around on screen. As I mentioned earlier, I wish I made the function to generate butterflies with more customization so that anyone can easily find their creation while also appreciating the overall harmony of their work being part of those of many others. Even though this project was a great opportunity for me to learn and explore mediapipe and hand tracking using open cv, I also realized I could’ve utilized the hand tracking data more than just gather a past locations of a specific point of the hand. If I work on this project again, I would also take the data of how people move their hands in addition to where they move their hands. I think exploring how would bring more interesting insights to people in time, as different people move their bodies in a different manner even if they’re prompted to do the same movement.