{"id":778,"date":"2022-03-02T17:12:46","date_gmt":"2022-03-02T22:12:46","guid":{"rendered":"https:\/\/courses.ideate.cmu.edu\/62-830\/s2022\/?p=778"},"modified":"2022-03-02T17:12:46","modified_gmt":"2022-03-02T22:12:46","slug":"has-motion-capture-technology-helped-or-hindered-arts-organizations","status":"publish","type":"post","link":"https:\/\/courses.ideate.cmu.edu\/62-830\/s2022\/?p=778","title":{"rendered":"Has motion capture technology helped or hindered arts organizations?"},"content":{"rendered":"\n

Motion capture technology (also known as Mo-cap) is the process of digitally recording the movement of an object. It translates the motion trajectory into data and models. This technology is widely used in entertainment, medical applications, military, and arts industries. Maybe the word doesn’t sound so familiar to you, but you must have come into contact with it, either directly or indirectly. For example, the famous movie “Avatar<\/em><\/a>” uses a lot of motion capture, including full-body gesture tracking, hand tracking, and even facial expression tracking.\u00a0<\/p>\n\n\n\n

\"\"
Figure 1: Mocap tracks facial expression in Avatar<\/em>.
Source: Avatar Blog<\/a><\/figcaption><\/figure><\/div>\n\n\n\n

<\/p>\n\n\n\n

Motion Capture in the Art Industry<\/strong><\/p>\n\n\n\n

Nowadays, audiences have become increasingly demanding and want to have diverse experiences, understand, interact, and share artwork. Many art institutions face the challenges of digital innovation, especially in the pandemic era.\u00a0This article mainly talks about the change motion capture brings to the museums, artists, audiences; the advantages and issues to consider to use the technology. <\/p>\n\n\n\n

In 2020, The Robins Nature Center<\/a> reopened and offered a diverse, interactive multimedia experience to the public. The exhibition used camera and motion capture technology to build a touch-responsive virtual diving pool to invite visitors to engage, creating an immersive viewing experience through the audiences’ movements. In addition to making artworks responsive to audiences, some museums try to provide a high degree of interactivity, allowing the audiences to become artists and creators. For instance, History Colorado Center<\/a> used motion capture technology to process images of visitors, revealing their thermal features and contours. The images, like signatures, are integrated into a new scene to interact with the new audiences.<\/p>\n\n\n\n

Generally, motion capture meets the need and creates interactive installations to help the museum educate the audiences, inspire the artists to make innovations, and provide audiences with more opportunities. <\/p>\n\n\n\n

\"\"
Figure 2: Infographic three perspectives of Mocap in the art industry. Source: Author.<\/figcaption><\/figure><\/div>\n\n\n\n

<\/p>\n\n\n\n

Motion Capture Helps Museums Better Communicate Information<\/strong><\/p>\n\n\n\n

Early in 2015, the Metropolitan Museum<\/a> experimented with digitizing a Renaissance sculptural masterpiece, Adam, with performers in motion-captured suits controlling digital Adam on set. Since the sculpture had previously been shattered by gravity and time, artists Farrington and the Met Museum invested a lot of time in restoration and intensive research to develop this live performance installation. The Hidden cameras, microphones, and speakers captured the movements and voices of the performers and allowed viewers to talk with the digital Adam.\u00a0<\/p>\n\n\n\n

\"\"
Figure 3: The performer uses motion capture to control the digital Adam.
Source: The Met Museum<\/a><\/figcaption><\/figure><\/div>\n\n\n\n

Through this way, audiences can explore the complexity of the sculpture and dynamically investigate, research, and ask questions about the details. The Metropolitan Museum’s revolutionary exhibition provides an interactive and immersive experience to educate their audiences better. <\/p>\n\n\n\n

<\/p>\n\n\n\n

Motion Capture Inspires Artists to Innovate<\/strong><\/p>\n\n\n\n

At the same time, motion capture gives artists more diverse forms, more inspiration, and more accessibility. For example, choreographer Merce Cunningham<\/a> used motion capture for solo dance demonstrations. The camera and software tracked the dancers\u2019 movements and created images with 48 small devices worn on knuckles, wrists, and fingertips. Even when he was in a wheelchair in his later years, he could still control the images with his hands, showing the audience behind the screen the fusion of visual arts and technology.\u00a0<\/p>\n\n\n\n

In most cases, artists use motion capture in two ways. On the one hand, they could choose to create an avatar and then have performers outfitted with motion-tracking suits to manipulate that avatar in real-time. On the other hand, they can also create a virtual scene, throwing real-person images into the background. Artist Lu Yang<\/a> chose the latter approach. She used the Noitom Motion Tracking Suit<\/a> to integrate the performer’s figure into an animated scene. Specifically, the on-site cameras and software would dynamically switch between the reality and virtual space to incorporate the performer and the lively scene. It is also worth mentioning that in this process, the performer’s image can be duplicated into many copies to communicate with different audiences at the same time. The form of audiovisual performance is pioneering, showing the dynamic online art experience possibilities. As we can see, motion capture can inspire artists to explore different art forms.\u00a0<\/p>\n\n\n\n

In contemporary art, interactive technology like motion capture is an inevitable trend, which records the movements of artists or performers, and allows audiences to watch from different angles, anywhere, and at any time.\u00a0<\/p>\n\n\n\n

<\/p>\n\n\n\n

Motion Capture Makes Audiences Become Artists<\/strong><\/p>\n\n\n\n

In addition to the benefits that motion capture brings to artists and museums, the core is serving the audience and giving them a better experience. Future You<\/em>, the exhibition at the Barbican Center<\/a>, does a great job of explaining how to fully engage the audience and inspire them to explore the possibilities of art and even create their art. This groundbreaking exhibition uses Mocap from the audience’s perspective: it presents some abstract shapes, allowing the audience to set up the different sizes, styles, color, material type, opacity, density, and other data, and then utilizes one-to-one movement mapping to synthesize figures. In this way, audiences can create a synthetic version of themselves. Each robotic reflection is unique because it is generated from tens of thousands of possible iterations, providing visitors with a personalized, exceptional interactive experience. Even the audiences without any art background can feel the joy of being a creator and have more desire to share. With over 88,000 visitors, the exhibition became the Barbican Center’s most successful exhibition.<\/p>\n\n\n\n

\"\"
Figure 4: Visitors interact with the screen in the Future You<\/em> exhibition.
Source: Universal Everything<\/a><\/figcaption><\/figure><\/div>\n\n\n\n

In the current age, many organizations want to involve audiences as artists. However, the software would be an issue to consider when presenting touchless motion-based exhibitions. The most common software is Kinect, Intel and Stereolabs ZED<\/a>. Their design principles include tracking visitor activity through skeletal tracking or blob tracking, detecting gestures and poses. Visitors need to stand in specific positions to make gestures to engage with the LED screen.\u00a0<\/p>\n\n\n\n

Indeed, more and more motion capture technology is being used in zoos, science centers, children’s museums, and art galleries; the technology develops from a single-user experience to a shared multi-user immersive technology for audiences to deeply participate in the exhibition. Before making a blind decision, organizations need to consider whether their exhibit is suitable for motion capture technology and the pros and cons of this technology.<\/p>\n\n\n\n

<\/p>\n\n\n\n

Advantages and Issues to Consider<\/strong><\/p>\n\n\n\n

\u201cWe need to be open to new ideas and understand that failure at some points is totally expected. We need to help people share what they\u2019ve learned from failure and move on to the next idea.\u201d<\/em> \u2013 Lisa Tessarowicz<\/em><\/p><\/blockquote>\n\n\n\n

Mocap has opened a new door for the art industry. Still, there is no correct answer as to whether art organizations need to adopt this technology because Mocap’s advantages and disadvantages need to be considered and the organization’s own infrastructure and conditions.<\/p>\n\n\n\n

\"\"
Figure 5: Infographic showing advantages and issues to consider. Source: Author.<\/figcaption><\/figure><\/div>\n\n\n\n

To begin with, it is easy to see that the audiences\u2019 direct interaction with the artwork will lead to deeper participation and understanding of the arts. Typically, only the researchers can use the magnifying glass to figure out the details; motion capture technology helps the audiences achieve this goal through multi-dimensional exploration. Unlike 3D modeling, motion capture provides more spontaneous behavior from the viewer’s perspective, and it can recreate complex movements and realistic physical interactions accurately. The Case Study for Animating Virtual Humans<\/a> uses martial arts as an experiment to show how motion capture performs better than 3D modeling. Mocap can record more complex gestures, capture more details, and present smoother movements.\u00a0<\/p>\n\n\n\n

\"\"
Figure 6: Using motion capture technology to do martial arts. Source: VIMM<\/a><\/figcaption><\/figure><\/div>\n\n\n\n

<\/p>\n\n\n\n

Along with the convenience, art managers should consider several potential problems of mocap. First, museums are free-choice learning environments, so every interaction and participation need to be clear and intuitive. Otherwise, most visitors will leave directly to another exhibit. Therefore, organizations need to carefully consider whether they can provide the most detailed and understandable guidance for their audiences. In other words, the Mocap needs to be accessible and easy to understand. Some cultural organizations have removed physical equipment and eliminated the training phase for mocap to invite the audiences to interact directly with the digital art. For example, art organizations can display digital paintings on interactive screens, allowing viewers to use body movements and gestures to paint images. <\/p>\n\n\n\n

What is more, the mocap technology requires specific hardware and special software programs to generate and process the data, which means it can be an expensive investment for many arts organizations. In addition to purchasing equipment, organizations also need to consider the costs of human resources and the space limitation. In most cases, the organization needs to have technicians to support the operation of the mocap and solve problems and provide help for the audiences. Furthermore, the organization also needs to have enough space for the audience to wave, swing, and interact. The mocap camera will only capture a limited vision if the area is too narrow. <\/p>\n\n\n\n

Last but not least, mocap is closely related to ethical issues. Instead of using mocap to enhance interactivity, some art organizations choose to use it as a tool to collect data, monitor the audience’s behaviors, and improve customer service. Some art organizations will use motion capture not as a work of art to enhance interactivity but as a management and data processing tool to monitor their audience and improve their customer service and visitor experience. For example, facial and movement tracking, a branch of motion capture, is used by the Palace Museum in Beijing<\/a> to navigate visitors, reminding them where to avoid queues and where they haven\u2019t yet arrived. The technology sounds helpful to improve visitors’ experiences. Still, at the same time, it means that the motion capture will record the whereabouts and movements of the tourists at any time.\u00a0<\/p>\n\n\n\n

Additionally, Bologna’s Civic Museum<\/a> combines mocap and AI technology to track visitors automatically. The integrated technology can analyze each visitor\u2019s movements, perspectives, facial expressions, sentiments, and demographic information. Hence, these museums’ use of motion capture has led some people to believe that motion capture has ethical issues and privacy concerns. Therefore, art organizations must carefully consider balancing service customer experience, information leakage, and data processing to improve the visitor’s viewing experience effectively.<\/p>\n\n\n\n

<\/p>\n\n\n\n

Conclusion<\/strong><\/p>\n\n\n\n

The rise of the Internet and new technologies has affected the digital transformation of all industries, forcing all arts and cultural organizations to diversify their practices and innovate in forms to attract larger audience groups. As a widespread technology, Mocap can be combined with AI, robotics, 3D modeling, and other fields to help museums educate audiences, inspire artists, and increase audience engagement. The technology gives the arts organization more possibilities, but its potential problems are also worth considering.<\/p>\n\n\n\n

<\/p>\n\n\n\n

<\/p>\n\n\n\n

Resources<\/strong><\/p>\n\n\n\n

Ashley Lee Wong, \u201cLu Yang Live Motion-Capture Performances at Rockbund Art\u00a0Museum Shanghai.\u201d Sedition Art, published on September 28, 2018. https:\/\/www.seditionart.com\/magazine\/lu-yang-live-motion-capture-performances-at-rockbund-art-museum-shanghai<\/a>\u00a0<\/p>\n\n\n\n

Avatar, \u201cBehind the Scenes Look at the Motion Capture Technology Used in AVATAR.\u201d\u00a0Avatar blog, published on May 20, 2010. https:\/\/avatarblog.typepad.com\/avatar-blog\/2010\/05\/behind-the-scenes-look-at-the-motion-capture-technology-used-in-avatar.html#:~:text=Traditional%20movies%20use%20a%20sound,actors%20and%20their%20facial%20expressions<\/a>.<\/p>\n\n\n\n

Bruce Bisbey, \u201cWhat Does a Motion Capture Artist Do? (In the entertainment industry, what does a motion capture artist do?)\u201d Linkedin, published on September 5, 2018. https:\/\/bit.ly\/3BQ3e0F<\/a>\u00a0<\/p>\n\n\n\n

\u00c7a\u011flar Akman & Tolga S\u00f6nmez, \u201cBody Motion Capture and Applications\u201d in Decision\u00a0Support Systems and Industrial IoT in Smart Grid, Factories, and Cities<\/em> (IGI Global, 2021), 181-188.<\/p>\n\n\n\n

History Colorado, \u201cWhat\u2019s Your Story?\u201d Accessed on February 24, 2022.\u00a0https:\/\/www.historycolorado.org\/exhibit\/whats-your-story<\/a><\/p>\n\n\n\n

Jim Spadaccini, \u201cTouchless Gesture-Based Exhibits, Part Two: Full-Body Interaction.\u201d\u00a0Ideum, published on April 22, 2020. https:\/\/ideum.com\/news\/touchless-interaction-public-spaces-part2<\/a>\u00a0<\/p>\n\n\n\n

Katherine Abbott, \u201cMerce Cunningham Lives on in Motion Capture Experience at\u00a0Jacob\u2019s Pillow.\u201d The Berkshire Eagle, published on August 15, 2019. https:\/\/www.berkshireeagle.com\/archives\/merce-cunningham-lives-on-in-motion-capture-experience-at-jacobs-pillow\/article_c100695a-6a22-5d43-88e2-66dfc070ea47.html<\/a>\u00a0<\/p>\n\n\n\n

Indiana University, \u201cI Think Audiences Are Ready for This: Motion Capture and Modern\u00a0Dance.\u201d Newswise, published on September 21, 2006.\u00a0https:\/\/www.newswise.com\/articles\/i-think-audiences-are-ready-for-this-motion-capture-and-modern-dance<\/a><\/p>\n\n\n\n

Manuel Charr, \u201cWhat Implications Are There With Facial Recognition Technology For\u00a0Museums?\u201d MuseumNext, published on September 25, 2019. https:\/\/www.museumnext.com\/article\/what-implications-are-there-with-facial-recognition-technology-for-museums\/<\/a>\u00a0<\/p>\n\n\n\n

Matt Pyke, \u201cFuture You Digital Installation.\u201d Communication Arts, accessed on February\u00a024, 2022. https:\/\/www.commarts.com\/exhibit\/future-you<\/a>\u00a0<\/p>\n\n\n\n

Maymont, \u201cNew James River Experience Will Delight, Educate and Inspire All Ages.\u201d\u00a0Published on March 3, 2020. https:\/\/maymont.org\/news\/media-release-robins-nature-center-reopening\/<\/a>\u00a0<\/p>\n\n\n\n

Meryl Cates, \u201cExperiencing the Return.\u201d The Met Museum, published on July 24, 2015.\u00a0https:\/\/www.metmuseum.org\/blogs\/met-live-arts\/2015\/experiencing-the-return<\/a><\/p>\n\n\n\n

MetaObjects, \u201cLu Yang: Delusional World Virtual Motion Capture Performance.\u201d\u00a0Accessed on February 25, 2022. https:\/\/metaobjects.org\/work\/lu-yang-delusional-world\/<\/a>\u00a0<\/p>\n\n\n\n

MIRALab, \u201cCase Study for Animating Virtual Human.\u201d Virtual Multimodal Museum.\u00a0Accessed on February 22, 2022. https:\/\/www.vi-mm.eu\/project\/motion-capture-for-tai-chi-case-study-for-animating-virtual-human\/<\/a>\u00a0<\/p>\n\n\n\n

Pascal Vella, \u201cThe Digital Transformation and Museums: New Challenges, Practices\u00a0and Uses.\u201d Dot Museum, published on January 27, 2022.\u00a0 https:\/\/welcome.museum\/interview-the-digital-transformation-and-museums-new-challenges-practices-and-uses\/<\/a>\u00a0<\/p>\n\n\n\n

Tim Boutelle, \u201cInteractive Museum Media: Camera & Gesture Interaction.\u201d RLMG,\u00a0accessed on February 20, 2022. http:\/\/rlmg.com\/Covid\/InteractiveMuseumMedia_Camera_Gesture.pdf<\/a>\u00a0<\/p>\n\n\n\n

<\/p>\n","protected":false},"excerpt":{"rendered":"

Motion capture technology (also known as Mo-cap) is the process of digitally recording the movement of an object. It translates the motion trajectory into data and models. This technology is widely used in entertainment, medical applications, military, and arts industries. Maybe the word doesn’t sound so familiar to you, but you must have come into […]<\/p>\n","protected":false},"author":40,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[60,59],"_links":{"self":[{"href":"https:\/\/courses.ideate.cmu.edu\/62-830\/s2022\/index.php?rest_route=\/wp\/v2\/posts\/778"}],"collection":[{"href":"https:\/\/courses.ideate.cmu.edu\/62-830\/s2022\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/courses.ideate.cmu.edu\/62-830\/s2022\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/courses.ideate.cmu.edu\/62-830\/s2022\/index.php?rest_route=\/wp\/v2\/users\/40"}],"replies":[{"embeddable":true,"href":"https:\/\/courses.ideate.cmu.edu\/62-830\/s2022\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=778"}],"version-history":[{"count":2,"href":"https:\/\/courses.ideate.cmu.edu\/62-830\/s2022\/index.php?rest_route=\/wp\/v2\/posts\/778\/revisions"}],"predecessor-version":[{"id":827,"href":"https:\/\/courses.ideate.cmu.edu\/62-830\/s2022\/index.php?rest_route=\/wp\/v2\/posts\/778\/revisions\/827"}],"wp:attachment":[{"href":"https:\/\/courses.ideate.cmu.edu\/62-830\/s2022\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=778"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/courses.ideate.cmu.edu\/62-830\/s2022\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=778"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/courses.ideate.cmu.edu\/62-830\/s2022\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=778"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}