Motion capture technology (also known as Mo-cap) is the process of digitally recording the movement of an object. It translates the motion trajectory into data and models. This technology is widely used in entertainment, medical applications, military, and arts industries. Maybe the word doesn’t sound so familiar to you, but you must have come into contact with it, either directly or indirectly. For example, the famous movie “Avatar” uses a lot of motion capture, including full-body gesture tracking, hand tracking, and even facial expression tracking.
Source: Avatar Blog
Motion Capture in the Art Industry
Nowadays, audiences have become increasingly demanding and want to have diverse experiences, understand, interact, and share artwork. Many art institutions face the challenges of digital innovation, especially in the pandemic era. This article mainly talks about the change motion capture brings to the museums, artists, audiences; the advantages and issues to consider to use the technology.
In 2020, The Robins Nature Center reopened and offered a diverse, interactive multimedia experience to the public. The exhibition used camera and motion capture technology to build a touch-responsive virtual diving pool to invite visitors to engage, creating an immersive viewing experience through the audiences’ movements. In addition to making artworks responsive to audiences, some museums try to provide a high degree of interactivity, allowing the audiences to become artists and creators. For instance, History Colorado Center used motion capture technology to process images of visitors, revealing their thermal features and contours. The images, like signatures, are integrated into a new scene to interact with the new audiences.
Generally, motion capture meets the need and creates interactive installations to help the museum educate the audiences, inspire the artists to make innovations, and provide audiences with more opportunities.
Motion Capture Helps Museums Better Communicate Information
Early in 2015, the Metropolitan Museum experimented with digitizing a Renaissance sculptural masterpiece, Adam, with performers in motion-captured suits controlling digital Adam on set. Since the sculpture had previously been shattered by gravity and time, artists Farrington and the Met Museum invested a lot of time in restoration and intensive research to develop this live performance installation. The Hidden cameras, microphones, and speakers captured the movements and voices of the performers and allowed viewers to talk with the digital Adam.
Source: The Met Museum
Through this way, audiences can explore the complexity of the sculpture and dynamically investigate, research, and ask questions about the details. The Metropolitan Museum’s revolutionary exhibition provides an interactive and immersive experience to educate their audiences better.
Motion Capture Inspires Artists to Innovate
At the same time, motion capture gives artists more diverse forms, more inspiration, and more accessibility. For example, choreographer Merce Cunningham used motion capture for solo dance demonstrations. The camera and software tracked the dancers’ movements and created images with 48 small devices worn on knuckles, wrists, and fingertips. Even when he was in a wheelchair in his later years, he could still control the images with his hands, showing the audience behind the screen the fusion of visual arts and technology.
In most cases, artists use motion capture in two ways. On the one hand, they could choose to create an avatar and then have performers outfitted with motion-tracking suits to manipulate that avatar in real-time. On the other hand, they can also create a virtual scene, throwing real-person images into the background. Artist Lu Yang chose the latter approach. She used the Noitom Motion Tracking Suit to integrate the performer’s figure into an animated scene. Specifically, the on-site cameras and software would dynamically switch between the reality and virtual space to incorporate the performer and the lively scene. It is also worth mentioning that in this process, the performer’s image can be duplicated into many copies to communicate with different audiences at the same time. The form of audiovisual performance is pioneering, showing the dynamic online art experience possibilities. As we can see, motion capture can inspire artists to explore different art forms.
In contemporary art, interactive technology like motion capture is an inevitable trend, which records the movements of artists or performers, and allows audiences to watch from different angles, anywhere, and at any time.
Motion Capture Makes Audiences Become Artists
In addition to the benefits that motion capture brings to artists and museums, the core is serving the audience and giving them a better experience. Future You, the exhibition at the Barbican Center, does a great job of explaining how to fully engage the audience and inspire them to explore the possibilities of art and even create their art. This groundbreaking exhibition uses Mocap from the audience’s perspective: it presents some abstract shapes, allowing the audience to set up the different sizes, styles, color, material type, opacity, density, and other data, and then utilizes one-to-one movement mapping to synthesize figures. In this way, audiences can create a synthetic version of themselves. Each robotic reflection is unique because it is generated from tens of thousands of possible iterations, providing visitors with a personalized, exceptional interactive experience. Even the audiences without any art background can feel the joy of being a creator and have more desire to share. With over 88,000 visitors, the exhibition became the Barbican Center’s most successful exhibition.
Source: Universal Everything
In the current age, many organizations want to involve audiences as artists. However, the software would be an issue to consider when presenting touchless motion-based exhibitions. The most common software is Kinect, Intel and Stereolabs ZED. Their design principles include tracking visitor activity through skeletal tracking or blob tracking, detecting gestures and poses. Visitors need to stand in specific positions to make gestures to engage with the LED screen.
Indeed, more and more motion capture technology is being used in zoos, science centers, children’s museums, and art galleries; the technology develops from a single-user experience to a shared multi-user immersive technology for audiences to deeply participate in the exhibition. Before making a blind decision, organizations need to consider whether their exhibit is suitable for motion capture technology and the pros and cons of this technology.
Advantages and Issues to Consider
“We need to be open to new ideas and understand that failure at some points is totally expected. We need to help people share what they’ve learned from failure and move on to the next idea.” – Lisa Tessarowicz
Mocap has opened a new door for the art industry. Still, there is no correct answer as to whether art organizations need to adopt this technology because Mocap’s advantages and disadvantages need to be considered and the organization’s own infrastructure and conditions.
To begin with, it is easy to see that the audiences’ direct interaction with the artwork will lead to deeper participation and understanding of the arts. Typically, only the researchers can use the magnifying glass to figure out the details; motion capture technology helps the audiences achieve this goal through multi-dimensional exploration. Unlike 3D modeling, motion capture provides more spontaneous behavior from the viewer’s perspective, and it can recreate complex movements and realistic physical interactions accurately. The Case Study for Animating Virtual Humans uses martial arts as an experiment to show how motion capture performs better than 3D modeling. Mocap can record more complex gestures, capture more details, and present smoother movements.
Along with the convenience, art managers should consider several potential problems of mocap. First, museums are free-choice learning environments, so every interaction and participation need to be clear and intuitive. Otherwise, most visitors will leave directly to another exhibit. Therefore, organizations need to carefully consider whether they can provide the most detailed and understandable guidance for their audiences. In other words, the Mocap needs to be accessible and easy to understand. Some cultural organizations have removed physical equipment and eliminated the training phase for mocap to invite the audiences to interact directly with the digital art. For example, art organizations can display digital paintings on interactive screens, allowing viewers to use body movements and gestures to paint images.
What is more, the mocap technology requires specific hardware and special software programs to generate and process the data, which means it can be an expensive investment for many arts organizations. In addition to purchasing equipment, organizations also need to consider the costs of human resources and the space limitation. In most cases, the organization needs to have technicians to support the operation of the mocap and solve problems and provide help for the audiences. Furthermore, the organization also needs to have enough space for the audience to wave, swing, and interact. The mocap camera will only capture a limited vision if the area is too narrow.
Last but not least, mocap is closely related to ethical issues. Instead of using mocap to enhance interactivity, some art organizations choose to use it as a tool to collect data, monitor the audience’s behaviors, and improve customer service. Some art organizations will use motion capture not as a work of art to enhance interactivity but as a management and data processing tool to monitor their audience and improve their customer service and visitor experience. For example, facial and movement tracking, a branch of motion capture, is used by the Palace Museum in Beijing to navigate visitors, reminding them where to avoid queues and where they haven’t yet arrived. The technology sounds helpful to improve visitors’ experiences. Still, at the same time, it means that the motion capture will record the whereabouts and movements of the tourists at any time.
Additionally, Bologna’s Civic Museum combines mocap and AI technology to track visitors automatically. The integrated technology can analyze each visitor’s movements, perspectives, facial expressions, sentiments, and demographic information. Hence, these museums’ use of motion capture has led some people to believe that motion capture has ethical issues and privacy concerns. Therefore, art organizations must carefully consider balancing service customer experience, information leakage, and data processing to improve the visitor’s viewing experience effectively.
Conclusion
The rise of the Internet and new technologies has affected the digital transformation of all industries, forcing all arts and cultural organizations to diversify their practices and innovate in forms to attract larger audience groups. As a widespread technology, Mocap can be combined with AI, robotics, 3D modeling, and other fields to help museums educate audiences, inspire artists, and increase audience engagement. The technology gives the arts organization more possibilities, but its potential problems are also worth considering.
Resources
Ashley Lee Wong, “Lu Yang Live Motion-Capture Performances at Rockbund Art Museum Shanghai.” Sedition Art, published on September 28, 2018. https://www.seditionart.com/magazine/lu-yang-live-motion-capture-performances-at-rockbund-art-museum-shanghai
Avatar, “Behind the Scenes Look at the Motion Capture Technology Used in AVATAR.” Avatar blog, published on May 20, 2010. https://avatarblog.typepad.com/avatar-blog/2010/05/behind-the-scenes-look-at-the-motion-capture-technology-used-in-avatar.html#:~:text=Traditional%20movies%20use%20a%20sound,actors%20and%20their%20facial%20expressions.
Bruce Bisbey, “What Does a Motion Capture Artist Do? (In the entertainment industry, what does a motion capture artist do?)” Linkedin, published on September 5, 2018. https://bit.ly/3BQ3e0F
Çağlar Akman & Tolga Sönmez, “Body Motion Capture and Applications” in Decision Support Systems and Industrial IoT in Smart Grid, Factories, and Cities (IGI Global, 2021), 181-188.
History Colorado, “What’s Your Story?” Accessed on February 24, 2022. https://www.historycolorado.org/exhibit/whats-your-story
Jim Spadaccini, “Touchless Gesture-Based Exhibits, Part Two: Full-Body Interaction.” Ideum, published on April 22, 2020. https://ideum.com/news/touchless-interaction-public-spaces-part2
Katherine Abbott, “Merce Cunningham Lives on in Motion Capture Experience at Jacob’s Pillow.” The Berkshire Eagle, published on August 15, 2019. https://www.berkshireeagle.com/archives/merce-cunningham-lives-on-in-motion-capture-experience-at-jacobs-pillow/article_c100695a-6a22-5d43-88e2-66dfc070ea47.html
Indiana University, “I Think Audiences Are Ready for This: Motion Capture and Modern Dance.” Newswise, published on September 21, 2006. https://www.newswise.com/articles/i-think-audiences-are-ready-for-this-motion-capture-and-modern-dance
Manuel Charr, “What Implications Are There With Facial Recognition Technology For Museums?” MuseumNext, published on September 25, 2019. https://www.museumnext.com/article/what-implications-are-there-with-facial-recognition-technology-for-museums/
Matt Pyke, “Future You Digital Installation.” Communication Arts, accessed on February 24, 2022. https://www.commarts.com/exhibit/future-you
Maymont, “New James River Experience Will Delight, Educate and Inspire All Ages.” Published on March 3, 2020. https://maymont.org/news/media-release-robins-nature-center-reopening/
Meryl Cates, “Experiencing the Return.” The Met Museum, published on July 24, 2015. https://www.metmuseum.org/blogs/met-live-arts/2015/experiencing-the-return
MetaObjects, “Lu Yang: Delusional World Virtual Motion Capture Performance.” Accessed on February 25, 2022. https://metaobjects.org/work/lu-yang-delusional-world/
MIRALab, “Case Study for Animating Virtual Human.” Virtual Multimodal Museum. Accessed on February 22, 2022. https://www.vi-mm.eu/project/motion-capture-for-tai-chi-case-study-for-animating-virtual-human/
Pascal Vella, “The Digital Transformation and Museums: New Challenges, Practices and Uses.” Dot Museum, published on January 27, 2022. https://welcome.museum/interview-the-digital-transformation-and-museums-new-challenges-practices-and-uses/
Tim Boutelle, “Interactive Museum Media: Camera & Gesture Interaction.” RLMG, accessed on February 20, 2022. http://rlmg.com/Covid/InteractiveMuseumMedia_Camera_Gesture.pdf