With the rapid development of technology, people’s lives are increasingly dependent on the Internet, and the boundaries between virtual and reality are blurring. We create the metaverse, a virtual world where people can work, play, and socialize. Since the virtual world is similar to the real world, will there be virtual residents besides the real human? Will there be virtual superstars who can get the fervent love of fans like the real idols? It sounds strange, but virtual idols are more popular these days than you can imagine.

What is a virtual idol?

Virtual idols are performers who do not have a living body and only exist in the virtual world. They have their own personalities, can sing, dance, and create more daily life possibilities like real humans. The technologies to create a virtual idol includes 3D modeling, artificial intelligence, motion capture, and virtual reality/augmented reality. 

The concept of virtual idols has a history of around thirty years. Since virtual idols are a synonym for all virtual performers, there are different types of virtual idols. The simplest version of virtual idols is mainly animation characters. As an IP product that the animation company created, fans learn more about their personalities and backstories through anime and related games, music, and live concerts online. With the development of 3D models and real-time motion capture software, virtual idols become more like real humans in appearance and are able to make more real-time interactions. The virtual influencers, also known as vtubers, have a real person to control the figure, live-stream behind an animated face. Some other virtual idols are avatars of real idols in the virtual world to make more interactions with fans. 

In this article, we will focus on one of the cases of virtual idols. They are customized idols that do not have a specific performer behind them, can sing any song that you want and be whatever you want them to be. The representative figures are Hatsune Miku and Luo Tianyi. As “superstars” who have sung on the stages of many concerts and music festivals, performed with real artists, and have countless fans around the world, they are actually software voicebank developed by Yamaha Vocaloid: an AI singing synthesis technology that can simulate human’s voice and produce songs by computer.

How were these idols created?

Yamaha Vocaloid is software to produce music by computer. It is like a “Siri-meets-Garage”. It records human voice in short samples, and these samples are stored in a database and constructed into a keyboard-style instrument within the Vocaloid software. Then anyone could produce their own songs by inputting lyrics and an instrumental melody into the software. Now, by incorporating the technology of deep learning and neural network, Vocaloid can do a better job to analyze the recordings of the real vocalists’ songs and speech. It can make the tunes and timbre closer to the vocalist, include the unique mannerisms and nuances, and clone the voice from just a few minutes of recordings. 

The Vocaloid idols will take voice samples from a specific voice actor/actress. But its persona and music productions are completely independent of the voice provider. After having the initial image and character set by the company, their music could come from any creator who has purchased the software. Fans around the world will continuously add their detailed personalities and backstories.

Famous Vocaloid idols and their performance

Hatsune Miku is one of the most famous Vocaloid idols. The 16 years old girl with long, turquoise twintails, whose name means “The first sound of the future”, has millions of followers on social media platforms. She was created in 2007 and takes voice samples from Japanese voice actress Saki Fujita. Now she has released over 100,000 songs, 170,000 YouTube videos, and 1,000,000 artworks. Similarly, there is another popular Vocaloid idol in China, Luo Tianyi. She is a 15 years old girl dressed in Chinese elements who can detect “the song of your heart”.

Besides performing in the virtual world, these idols also attend music festivals and concerts like real singers. By using holograms or augmented reality, they are able to show up on stages and perform with real performance. Miku has opened for Lady Gaga’s tour and attended the Coachella festival. She also has live concerts worldwide every year. She toured Europe in 2020, performing in cities such as London and Paris. In the concert, Miku was projected onto the screen with live bands and backup dancers. The audience in all sorts of fancy outfits gathered in the venue, screaming and waving the glow sticks like every normal concert.

As a popular idol in China, Luo Tianyi also appears on various shows. She had a live performance accompanied by world-famous pianist Lang lang using the hologram in 2019 and performed on China Central Television Spring Festival Gala, which is the most popular show on the Chinese new year’s eve in 2021. 

The live performance with Vocaloid idols has many advantages. Since the performers and stage settings are virtual, they’re not bound by physical costume changes and breaks in a setlist. They can change the costume and stage backdrop in a second. Some complex and exaggerated settings and effects can also be achieved. Although there is no need to cover the star’s travel and lodging fees and stage setting expenses, it is not a cost-saving performance. For the two hours’ Luo Tianyi performance, a team of 200 people from China and Japan worked for roughly six months. They used 3D modeling and motion capture to create the appearance and personalities. During the performance, a backstage voice dubber and motion-capture actress allowed Tianyi to interact with Lang Lang. Despite its complexity, it is still interesting to try things like that to break the boundaries between virtual and reality.

How can virtual idols change the entertainment industry?

Firstly, we should know why Vocaloid virtual idols are so popular. As virtual beings, they are perfect positive figures. A real idol may do things that do not match your imagination or even have scandals. But a virtual idol will never make you frustrated. Besides, they will always have time and energy, and stable vocal skills to provide you with a perfect performance. Because they do not need to consider physical function, they can make some performances that real idols can not complete. The design of props and scenes in the performances can also be more innovative. Besides, the Vocaloid technology can not only create idols for animation fans but also “clone” the deceased legendary singers. Yamaha has presented a live performance of a new song by Hibari Misora, a singer who passed away thirty years ago.

In addition, Vocaloid idols are idols created by everyone. They have the advantage of embodying not only the dreams of followers but also their creative skills. Since there are only basic settings, you can create your own version of Miku or Tianyi by writing stories or making artworks of them. Every fan is a part of the creation. That is why Miku has been popular in the 15 years since she was born — there are always new fans to bring different works. Virtual idols are not bound to a specific genre of music. Different songwriters give her different colors. Even songs in the live concerts are written by fans. Vocaloid company opens the use of the idols’ voices and images to the community, and they will check the most popular songs online and contact the writer to ask for permission of using them. Since the creators are mainly fans, it is an easy process for the management team. Some of Miku’s concerts were promoted with songwriting contests. Miku would perform the winning entry and sing songs that were written by individuals with no direct tie to her company. It may be a trend in the music industry, which is a decentralized model of music production. Non-professional people use virtual idols as a carrier to create great works and be heard while keeping this virtual idol in the spotlight. It is difficult for real idols to achieve the win-win situation.

The copyright issues

Since the essence of this model is to create works by fans, it is important to talk about the copyright. In the case of Vocaloid idols, the right of using Vocaloid software belongs to Yamaha, and the avatars belong to their respected companies. The companies control the intellectual property and brand image. However, it is hard to control the fan-made artworks due to their wide presence and rapid spread on the Internet. As a result, many Vocaloid idol companies do not profit by controlling what fans can do with their characters or clamping down on pirating or illegal downloads. For example, Miku’s company opens the right of using the image. As said in the previous part, it mainly gets profit by making promotions through fanfiction, choosing the best works such as songs and videos, and partnering with big companies such as SEGA and Toyota to make games, throw concerts, and create ad campaigns.

However, copyright is still a big issue for most virtual idols, especially for those who represent the real person. For example, if an avatar is created based on a real idol, the right to use the image after the artist leaves the company needs to be discussed. Besides, the creation of a virtual idol needs a team of engineers and artists. If the group of creators comes from different parties, the distribution of the rights can be complex.

The future of virtual idols

Having the advantages of perfect personalities, excellent performance skills, and opportunities for fans to make customizations, virtual idols may be a trend in the entertainment industry. In the future, there will be more collaborative performances by real and virtual idols. Although the current virtual idols are sometimes stiff in their voices and require the assistance of real humans to complete the interactions, as technology advances, these easily accessible idols will become an indispensable part of the entertainment industry. 

Resources

Cryer, Hirun. “Nothing Could Have Prepared Me for Seeing Hatsune Miku Live.” USgamer.net. USgamer, January 24, 2020. https://www.usgamer.net/articles/nothing-could-have-prepared-me-for-seeing-hatsune-miku-live. 

Hamamoto, Ben. “Making Miku.” Institute for the future, April 10, 2018. https://www.iftf.org/future-now/article-detail/making-miku/. 

Jenkins, Mark. “This Singer Is Part Hologram, Part Avatar, and Might Be the Pop Star of the Future.” The Washington Post. WP Company, July 8, 2018. https://www.washingtonpost.com/entertainment/music/this-singer-is-part-hologram-part-avatar-and-might-be-the-pop-star-of-the-future/2018/07/05/e2557cdc-7ed3-11e8-b660-4d0f9f0351f1_story.html. 

Ke, Jiayun. “Live Performances with Entertainers Who Aren’t Alive.” SHINE, February 24, 2021. https://www.shine.cn/news/in-focus/2102245044/. 

“Legal Implications.” Vocaloid Wiki. Accessed May 4, 2022. https://vocaloid.fandom.com/wiki/Legal_Implications. 

Li, Donna. “The Rise of Vocaloid, a Voice Synthesizing Software That Has Revolutionized the Digital Music Industry.” The Science Survey, June 22, 2021. https://thesciencesurvey.com/arts-entertainment/2021/06/22/the-rise-of-vocaloid-a-voice-synthesizing-software-that-has-revolutionized-the-digital-music-industry/. 

“Luo Tianyi.” Vocaloid Wiki. Accessed May 4, 2022. https://vocaloid.fandom.com/wiki/Luo_Tianyi#cite_note-article1-13. 

Poole, Robert Michael. “Fans Interact with Holographic Superstar Hatsune Miku.” Japan Today, September 2, 2011. https://japantoday.com/category/features/fans-interact-with-holographic-superstar-hatsune-miku. 

Tangermann, Victor. “Thousands of Chinese Fans Paid to See a Virtual Hologram Singer Live.” Futurism. Futurism, March 4, 2019. https://futurism.com/virtual-idol-hologram-luo-tianyi-lang-lang-live. 

Vale, Chris. “An Alternate Dimension: Virtual Idols – New Technology – UK.” An Alternate Dimension: Virtual Idols – New Technology – UK. Rouse, December 3, 2020. https://www.mondaq.com/uk/new-technology/1011984/an-alternate-dimension-virtual-idols. 

“VOCALOID: from Hatsune Miku to Hibari Misora.” Music Technology Group, January 27, 2020. 2.https://www.upf.edu/web/mtg/news/-/asset_publisher/WM181VyAQipW/content/id/231857712/#.YnGAw9PMJEY. 

“Who Is Hatsune Miku?” About HATSUNE MIKU | CRYPTON FUTURE MEDIA. Accessed May 4, 2022. https://ec.crypton.co.jp/pages/prod/virtualsinger/cv01_us. 

“Yamaha Vocaloid:AI Faithfully Reproduces Singing of Legendary Japanese Vocalist Hibari Misora.” Yamaha VOCALOID:AI Faithfully Reproduces Singing of Legendary Japanese Vocalist Hibari Misora – News Releases – Yamaha Corporation. Accessed May 4, 2022. https://www.yamaha.com/en/news_release/2019/19100801/. 

Ye-eun, Jie. “K-Pop Idols Bond with Gen Z and Alpha Fans on Metaverse Platforms.” The Korea Herald. The Korea Herald, March 20, 2022. http://www.koreaherald.com/view.php?ud=20220320000143. 

Yu, Cheng. “Virtual Idol Means Music to Stock Market’s Ears.” Chinadaily.com.cn, January 5, 2022. http://global.chinadaily.com.cn/a/202201/05/WS61d4f406a310cdd39bc7f1dc.html. 

Zhang, Jane. “Virtual Idols Leave Everything to Their Fans’ Imagination.” South China Morning Post, March 1, 2019. https://www.scmp.com/tech/article/2188238/chinas-virtual-idols-meet-their-fans-intersection-entertainment-and-technology. 

Zoladz, Lindsay. “Hatsune Miku Is a Piece of Software. She May Also Be the Future of Music.” Vulture, November 19, 2014. https://www.vulture.com/2014/11/hatsune-miku-the-future-of-music.html. 

We live in a world full of data. Data brings information, knowledge, and wisdom, as presented by the DIKW pyramids. Based on large sets of data, we can add value and enrich it through productive analysis, and finally, we can get wisdom. With technology developing, getting trends and insights from unorganized raw data becomes easier and easier for individuals and organizations. Today, most for-profit companies collect data and use analytical tools to target customers, reduce costs, and make optimal choices. As nonprofits, arts organizations also rely on databases to collect audience information and get a better understanding of them. People are still exploring ways to apply new technologies to analyze data precisely. As one of the hottest emerging concepts at the moment, AI can do more than simulate human behavior. In the article, we will explore how to integrate machine learning, a type of artificial intelligence that has strong analytical and forecasting skills, into art organizations, better “learn” from your data of audience and achieve their mission.

What is machine learning?

Machine learning is a branch of artificial intelligence that applies more to the data analysis field. One of its core concepts is to use machines to learn from data and make predictions. It is a way to “use computers to answer questions”. The computer will learn from a training dataset, which is the historical data we already had, build models and find patterns, and use the model in the test dataset to evaluate if it fits the data. For example, we have a dataset of three different species of iris and their features of sepal length and sepal width. We can use this dataset to let computers get the correlation model between the features of sepal and species. By observing the sepal width and length, the computer can help us to identify the species if we get a new flower. We may simply find the pattern and classify the species ourselves, but sometimes we may not have the ability to handle data with many features and seemingly no pattern, so machine learning can be helpful.

Machine learning is commonly used in fields of image and video recognition, prediction, and personalized push notification on social media platforms. Companies mainly apply this method to build price strategies by mining historical price data and making customer segments. In the arts and entertainment industry, it is also useful in marketing research. It would be helpful to get insights into who our customers are, why they choose us and how they will be more satisfied and loyal in any organization.

Data in art organizations

Arts organizations already find benefits in using data-driven methods to engage the audience. Around 90% of nonprofits are collecting data but half of them are not sure how to use it according to The State of Data in the Nonprofit Sector report. 85% of respondents in the Nonprofit Trend report say they “use insights from marketing and engagement data to target outreach efforts. Organizations are used to applying data tools to collect customer data and build stronger relationships. It is easy to find the information of ticket buyers such as email, address, what kinds of performances they like, or how much they spend on your organization’s events. And that helps organizations to make customer segments, know preferences, and reach out to them directly. Based on a research of nonprofits, the most common use of audience data is using the contact details to send out newsletters. And less than 40% of the respondents use the audience data to personalize campaigns and inform the process of creating artworks.

Machine learning in audience analysis

Art organizations should analyze data more deeply. With new technologies, organizations can get data that is bigger and multiple forms and find better insights in audience analysis. Machine learning makes it possible to find patterns in disorganized audience behavior data. We will mainly focus on several applications this technology can bring to art organizations, build better relationships, make recommendations and help improve the services and productions. 

Profile your customers for a better relationship

Customer Relationship Management(CRM) is software to manage all the relationships in a company. Arts organizations are benefiting from it by managing all the disparate activities such as ticketing, fundraising, and data reporting, which makes it easier to target the audience. For example, it allows organizations to turn the transaction record to the relationship. You can easily find the first-time buyers and send them emails about the upcoming event to improve their return rates.

Machine learning offers more possibilities for CRM. It can leverage the data from “what” to “why” based on the CRM platform. Besides using the sales data to predict future profits, it is also useful in profiling your audience. Accurate customer segments bring better connections. Integrating the big data and machine learning method into CRM enables more types of data to the platform and plays an important role in understanding audience behaviors. By mining deeply the historical ticketing sales data, it helps to find more patterns in buyers’ habits and predict the probabilities of whether they will return. 

Organizations can use the results for more accurate segmentation and send different newsletters to audiences with different return rates or buying habits, which enables them to make deeper connections and improve their loyalty. For unstructured data and free text fields, It is difficult to extract common points with traditional statistical methods. However, it is possible to set an algorithm to find out any inquiries, complaints, and references for specific shows. It is also useful in fundraising. Donors are giving because they want the organizations to meet their needs. By getting an accurate profile, we can know more about donors’ interests and explore how the organizations can introduce programs to donors and help them to pursue their needs. It helps to cultivate long-term relationships and encourage giving.

The arts industry has already applied it in customer segmentation. Purple Seven is the leading theater and art data analytics company to use big data to help find insights for organizations. By combining an organization’s CRM with external big data of performing art consumption, Purple Seven can identify which bookers have the greatest likelihood of returning, what their audience is interested in, how to reach them, and where to find new audiences. 

Make personalized recommendations

Machine learning’s application in CRM makes customization possible. A personalized recommendation system is a technique to provide recommendations based on historical behavior data. It is a common application by using big data to analyze user behavior, particularly for online streams like YouTube and Netflix. Netflix uses the watch history of other users who have similar tastes to recommend what you may be most interested in watching next. It excels in analyzing audience profiles and pushing personalized products. To maximize the satisfaction of viewers with different preferences, it even cut out several different versions of the trailer and distributed them to different people through the user profile mastered by big data. The benefit is also significant. Netflix Over 75% of its viewer activity is based on personalized recommendations. They earned over 1 billion due to the recommendation system accounting for over 80% of the content streamed on the platform. 

Unlike the entertainment industry, traditional art organizations focus on off-line events. Currently, the personalized recommendation system is used for galleries. There is software using big data and machine learning to predict the artworks’ price for customers and make personalized recommendations, such as Arternal, Artbase, and Artcloud. Although few recommendation systems are widely used in the theater industry, data scientists are making models to gain insights into the artistic preferences of customers and similarities between performances to make personalized recommendations.

Improve content and service

Organizations want to know not only why the audience chooses us, but also their whole experience, and which parts or details raise their interests or make them satisfied. Analyzing audience behaviors in the events can help us achieve this goal. For the online platform, the user’s clicks, length of stay, and comments will be analyzed by algorithms to find out the pattern of which types of plots are popular. The investment of productions is also based on the prediction of the main characters and plots. The millions of visits per day can bring a large number of data samples to the platform to support the analysis and prediction, to provide the audience with more desired works. 

Museums use a similar method in exploring how the visitors interact with the venue. The British Museum uses machine learning to collect data of visitor experience. They can find out how people experience its exhibitions: what routes they take, what they engage with, how many minutes they take at each installation, and which pieces they choose to ignore. By doing this, the museum can get the point of interest of the audience, know what is working and what is not.  It can be more targeted to making the promotion and event and exhibition design in the future. For example, if more people stay a long time during similar topic paintings, it may be interesting to consider an education tour for these artworks. It helps to increase visitation, harness social outcomes and deliver efficiencies.

Theaters can hardly get the audience’s review during the events, but show lovers love to post their feelings on social media. Theater can get a view of what the audience likes and dislikes about the show by analyzing the unstructured comments on the internet.

Conclusion

After introducing the applications, we found that their commonality is to analyze the behavior of the audience, discover the characteristics of the audience, and use this to engage more audiences, build loyalty and expand the impact of the organization. That is why we use data analytics tools. We want to find a measurable way to know our audience, to discover who they are, why they are here and what they want, and broaden it and connect with it in innovative ways. Arts organizations always value audience feedback, but previously we could only do this using common sense, observation, and interviews of individuals. Now machine gives us a chance to get more comprehensive information and discover hidden insights from a large amount of data. With machine learning, it is easier to get a clear profile of the audience and their story with your organizations and help improve the organization and build deeper relationships in a targeted way. A better understanding of your audience is also a better understanding of your organization. For small art organizations, there may be barriers such as being hard to get a big data sample. Although the pattern will be more accurate with more data, it will still be useful to know more about your audience. 

It is the trend in the world to use big data. Art organizations will not miss the chance to be involved in this trend for an efficient way to make connections with audiences. Arts organizations will gradually discover the importance of data and get used to using analytical tools to discover insights and understand the audience.

Resources

Abernethy, Jacob D., Cyrus Anderson, Alex Chojnacki, Chengyu Dai, John Dryden, Eric M. Schwartz, Wenbo Shen, Jonathan C. Stroud, Laura Burdick, Sheng Yang and Daniel T. Zhang. “Data Science in Service of Performing Arts: Applying Machine Learning to Predicting Audience Preferences.” ArXiv abs/1611.05788 (2016): n. Pag.

“Art Gallery Software Market 2021 High Growth Forecast Due to Rising Demand and Future Trends.” ZNews Africa, February 25, 2022. https://znewsafrica.com/space/318680/art-gallery-software-market-2021-high-growth-forecast-due-to-rising-demand-and-future-trends-artbase-art-galleria-art-systems-masterpiece/.

Carlsson, Rebecca. “Big Data and Museums.” MuseumNext, March 30, 2021. https://www.museumnext.com/article/big-data-and-museums/.

Day, Adrienne. “Data-Driven Connections for a Better World (SSIR).” Using Data and Technology to Create World-Changing Connections Between Nonprofits and Their Supporters, 2020. https://ssir.org/articles/entry/data_driven_connections_for_a_better_world.

“How CRM Can Help You Outperform National Arts Industry Revenue Benchmarks.” NAMP, May 15, 2019. https://namp.americansforthearts.org/2019/05/15/how-crm-can-help-you-outperform-national-arts-industry-revenue-benchmarks.

“How Netflix Used Big Data and Analytics to Generate Billions.” Selerity, September 27, 2021. https://seleritysas.com/blog/2019/04/05/how-netflix-used-big-data-and-analytics-to-generate-billions/.

Midura, Danine. “Leveraging Machine Learning Using CRM Data.” Technology Advisors, July 14, 2021. https://www.techadv.com/blog/leveraging-machine-learning-using-crm-data.

“New Audience.” Purple Seven. Accessed March 2, 2022. New Audience. 2022. Ebook. Purple Seven. https://purplesevenanalytics.com/wp-content/uploads/2018/10/New-Audiences_InfoCosts.pdf.

Sauravdeb. “Introduction to Machine Learning: Iris Dataset.” Medium. Medium, February 18, 2022. https://medium.com/@sauravdeb98/introduction-to-machine-learning-iris-dataset-58f2f30f966e.

“The British Museum Is Using Big Data to Help Visitors Learn More about History.” Microsoft News Centre UK, July 4, 2017. https://news.microsoft.com/en-gb/2017/07/04/the-british-museum-is-using-big-data-to-help-visitors-learn-more-about-history/.

“The National Gallery Predicts the Future with Artificial Intelligence.” Digital meets Culture, September 14, 2017. https://www.digitalmeetsculture.net/article/the-national-gallery-predicts-the-future-with-artificial-intelligence/.

“The State of Data in the Nonprofit Sector.” everyaction. Accessed March 2, 2022. https://cdn2.hubspot.net/hubfs/433841/The_State_of_Data_in_The_Nonprofit_Sector.pdf.

“Uses of Machine Learning: List of Top 10 Uses of Machine Learning.” EDUCBA, March 2, 2021. https://www.educba.com/uses-of-machine-learning/.

Villaespesa, Elena. “Digital Culture 2014 – How Arts Organisations Use Audience Data.” arts&metrcs, December 8, 2014. https://www.artsmetrics.com/en/how-arts-organisations-use-audience-data/.

“What Is the Data, Information, Knowledge, Wisdom (DIKW) Pyramid?” Ontotext, October 22, 2020. https://www.ontotext.com/knowledgehub/fundamentals/dikw-pyramid/.

Yu, Allen. “How Netflix Uses AI and Machine Learning.” Medium. Becoming Human: Artificial Intelligence Magazine, October 1, 2019. https://becominghuman.ai/how-netflix-uses-ai-and-machine-learning-a087614630fe.