In a world where Siri can set alarms, give us directions, and look things up, shouldn’t machine translation be better by now? How does machine translation work? Why are museums still using human translation? And most importantly, what happens when machine translation is good enough that museums and other arts enterprises can use it without human oversight? 

This article explores how machine translation works and how it came to be in its current state, as well as how we evaluate state-of-the-art machine translation technology. Finally, this article examines how improvements in machine translation could disrupt and transform the museum industry.

How does machine translation work?

So, how does machine translation work? Underpinning machine translation is a field called natural language processing, or NLP for short. This area of study uses tools of computational linguistics combined with modern advances in computer science in order to “learn, understand, and produce human language content.” While NLP has progressed greatly since its inception, one foundational technique that is still used in some forms today is called “bag of words”, where all the words in a document are added up and each word’s frequency is calculated. This concept is a great illustration of many of the techniques used in NLP – counting words and calculating their relationship to other words within the same document or within a corpus of documents. 

Two other concepts to highlight within NLP are part-of-speech tagging and named entity recognition. Part-of-speech tagging is a computational linguistics tool in which descriptors (the tags) are assigned to each word’s role within a sentence. This allows for semantic understanding of the role each word plays, which is critical to grokking the full meaning of a sentence. Another important concept in NLP is named entity recognition, which is when a word or phrase that contains important information is identified and categorized. We need to be able to differentiate a name or location as distinct from an ordinary noun or pronoun to understand a sentence, and named entity recognition allows for that.

The techniques described above offer a window into how the field of natural language processing works and are illustrative of concepts integral to machine translation. Machine translation historically used statistical learning techniques similar to those described above. Launched in April 2006, Google Translate used a corpus of billions of words in both the origin and target languages to help craft computer-generated translations. While it is not the only company offering cutting-edge translation technology, Google has been an industry leader for many years.

With improvement in computing hardware as well as machine learning technology, Google was able to improve its translation service. In 2016, it launched a new version of its machine translation service that used neural machine translation instead of statistical models. Artificial neural networks are a type of machine learning algorithm that has quickly become useful in many industries. Using neural machine translation means that the model itself is made up of layers and layers of simple linear regression models. By combining these simple models, it mimics the way a human brain works in sending, translating, and receiving signals (hence why these models are referred to as “neural networks”). 

Google wasn’t done in 2016; although their current translation model still uses neural machine translation, they’ve made strides in other areas. Previously, they needed copious amounts of data in every language that they wanted to translate, but in using neural network machine learning techniques, their models can now perform better on low-resource languages, meaning they perform well even if they were not trained on very much data for that language. In 2019, Google released a massively multilingual, massive neural machine translation (M4) that was pretrained on over 25 billion sentence pairs and as a result, performed even better on low-resource languages. They have continued to focus on improving this low-resource language ability in order to expand their machine translation offerings, and as such released an updated model in 2020.

How do we know machine translation is improving?

So, machine translation works by taking natural language processing techniques, training a complex model with many layers of simpler models, and then inputting new text on which the model uses what it has learned. But how do researchers know that machine translation is improving? 

 There are a few metrics that exist for machine translation, and Google uses one called BLEU, which stands for “bilingual evaluation understudy”. The paper establishing the method was published in 2002, but despite its age the metric is still being used to evaluate cutting-edge language models. BLEU calculates a translation closeness score by evaluating each sentence as a unit and comparing them on a weighted average of words, allowing for other word choices that make sense in context. The score is then reported as a number between 0 and 1 with 1 being the highest score possible. Google has continued to evaluate its models using BLEU so that updated models can be easily compared to their predecessors.

Despite the progress that has been made, it’s notable to compare machine translation capabilities with natural language processing tools that we use in English every day. As mentioned in the introduction, people take for granted that they can use speech-to-text on their phone, or that they can interface with a chatbot on a website. Notably, a company in Canada called Cohere just raised $125 million to perform similar NLP tasks, including content moderation, conversational artificial intelligence, and search support. Another example of the expansion of English language model tools comes in the form of a startup called Forefront. Two weeks after the launch of an open-source large language model called GPT-NeoX-20b, Forefront announced that it was offering fine-tuning services to make the use of GPT-NeoX-20b more accessible. All these markers of progress demonstrate that NLP in English is rapidly advancing and expanding. 

Why isn’t machine translation good enough?

If AI can write papers and chat with us, why can’t it fluently translate my words into any language without errors? One of the shortcomings of large language models is that they require copious amounts of data to be effective. While there has been considerable effort to train machine translation models, there just isn’t as much training data available in many languages as there is in English. Despite Google’s best efforts to train their translation models to perform well on low-resource languages, they still aren’t nearly as good as the models that use English. And not only are the models not as good, they are also insufficient for professional use even in high-resource languages. As researcher Aihua Zhu states, current machine translation technology fails to meet the needs of use in professional contexts.

Despite the clear advantage that English has over other languages, there are still numerous shortcomings that exist with English large language models. Because many of them were trained on text from the internet, they tend to have biases such as stereotypes towards people of minority gender or ethnic identities. A recent illustration of this is that YouTube’s automated captioning service was spotted using profanity in captions of videos for young children. So while language models in English perform well enough to be useful to large swathes of society, they still have limitations. Comparatively, the limitations of multilingual language models are greater and do not allow for as much integration into high-level societal tasks, although they are very useful for many people (in 2016 Google stated that Google Translate has 500 million users and translates over 1 billion words a day). 

How are museums translating their material?

Turning back to the museum industry, we consider the current state of translation resources available to them. How are museums currently offering material in multiple languages? Many museums are employing professional translation services. For instance, the Field Museum in Chicago uses a company called Multilingual Connections, and the Denver Botanical Gardens, South Florida Science Center, and the Metropolitan Museum of Art use a company called Eriksen Translation. That is, museums must pay for professional human translators in order to offer material in multiple languages. When the Children’s Discovery Museum of San Jose decided in 2015 to translate materials so that they could engage more with Latino visitors, they decided to exclusively use human translations rather than machine translation because they wanted to ensure that their materials were true to the “spirit of the words,” not just that it offered visitors the gist of what was going on. 

While all of these are museums in the United States, it also seems that countries in Europe use a similar model of hiring professional translators – two contract bidding announcements by the Museum of the Quai Branly – Jacques Chirac and the Museum of Bastia in France demonstrate that human translation is still the industry standard in Europe as well. 

A great illustration of the current state of machine translation as useful but not quite good enough comes in an announcement for a translation sprint. Europeana Pro is an organization funded by the European Union that is dedicated to preserving and promoting cultural heritage. In their 2020 invitation to join sprint to translate foundational documents into more languages, the organizers of the sprint instructed participants to use a number of machine translation tools only “as long as you review the resulting text.” This instruction underlines the themes explored by this paper: machine translation is good, but not yet good enough for professional use without human review.

How will museums be affected when machine translation achieves parity with human translation?

So machine translation clearly is not yet good enough – but how could it change once the technology catches up to the need? User experience is naturally a priority for museums, and translation is an integral part of that. There are many museums in America that could better serve their non-English-speaking populations by expanding their offerings. And it isn’t just in serving their visitors; museums could also benefit greatly from better machine translation for their collections. A team of researchers from Beijing Jiaotong University used machine translation to evaluate ancient Chinese texts and translate them into English, demonstrating the promise of machine translation as a way to preserve and expand collection offerings. A research group based out of the University of California Los Angeles (UCLA) has a program dedicated to using automated translation to analyze and understand cuneiform languages on a series of tablets from southern Mesopotamia. A widely-used museum database software called Axiell has multilingual fields embedded in its data structures, and offers an automated translation tool so that curators have some idea of what they’re looking at even if it is in a language they don’t understand.

Another team of researchers commented on translation in museums as a narrative and a means by which to convey identity. They explore how bad translation can be seen as a failure of the museum and argue that the quality of the translation not only has an impact on the museum, but also on the message that the museum wants to convey. A professor and researcher from the United Kingdom synthesized the benefits of translation for museums into two central themes: economic value and social inclusivity. However, given that a professional translation service is often required, that is currently not always an option for museums. 

While professional translation services are the norm, some museums are starting to explore machine translation as a potential tool. The Computer History Museum conducted an experiment with artificial intelligence during which they used machine translation on a number of their audio and video files. While it is unclear if they have put it into effect so far, they underscored many of the points made in this article by commenting that machine translation promises to make museums and collections “more accessible to speakers of languages other than English.” 

All of this evidence points to the potential disruption and change that could occur for museums when machine translation becomes a ubiquitous option for professional-level translation. Besides machine translation’s promise of museums’ ability to better serve communities in America, the international museum industry also thrives off the ability to offer people a window into culture and identity that isn’t possible without accurate translations. The significant reduction of costs that would occur if machine translation improved would offer museums the ability to broaden their offerings and expand their visitor experience beyond their current capability.

References

Axiell. n.d. “Axiell – Bringing Culture and Knowledge to Life.” Axiell. Accessed March 2, 2022a. https://www.axiell.com/.

Axiell. n.d. “Interface Functionality: Editing or Translating Adlib Interface Texts.” Accessed March 2, 2022b. http://documentation.axiell.com/alm/en/ds_tmeditingtranslatingtexts.html#autotranslate.

Axiell. n.d. “Translations Manager: Introduction.” Accessed March 2, 2022c. http://documentation.axiell.com/alm/en/ds_tmintroduction.html.

Bapna, Ankur, and Orhan Firat. 2019. “Exploring Massively Multilingual, Massive Neural Machine Translation.” Google AI Blog (blog). October 11, 2019. http://ai.googleblog.com/2019/10/exploring-massively-multilingual.html.

Chen, Chia-Li, and Min-Hsiu Liao. 2017. “National Identity, International Visitors: Narration and Translation of the Taipei 228 Memorial Museum.” Museum and Society 15 (1): 56–68. https://doi.org/10.29311/mas.v15i1.662.

“Cohere.” n.d. Cohere. Accessed March 2, 2022. https://cohere.ai.

David C. Brock. 2022. “A Museum’s Experience With AI.” CHM. February 3, 2022. https://computerhistory.org/blog/a-museums-experience-with-ai/.

Eriksen Translations Inc. 2019. “Museum Audience Engagement: Translation Strategies to Promote Diversity.” Eriksen Translations Inc. September 19, 2019. https://eriksen.com/arts-culture/museum-audience-engagement-translation-strategies/.

Eriksen Translations Inc. n.d. “Translating and Typesetting the Met Guides into 6 Languages.” Eriksen Translations Inc. Accessed March 2, 2022a. https://eriksen.com/work/met-guides/.

Eriksen Translations Inc. n.d. “Translating Exhibition Materials for the South Florida Science Center.” Eriksen Translations Inc. Accessed March 2, 2022b. https://eriksen.com/work/exhibition-displays/.

Eriksen Translations Inc. n.d. “Translating the Garden Tool Mobile Website for Denver Botanic Gardens.” Eriksen Translations Inc. Accessed March 2, 2022c. https://eriksen.com/work/garden-tool-mobile-website/.

Eriksen Translations Inc. n.d. “Translation & Localization Services NYC | Eriksen Translations.” Eriksen Translations Inc. Accessed March 2, 2022d. https://eriksen.com/.

“Forefront: Fine-Tune and Deploy GPT-J, GPT-13B, and GPT-NeoX.” n.d. Accessed March 2, 2022. https://www.forefront.ai/helloforefront.com/.

Field Museum. 2018. “About.” Text. Field Museum. May 14, 2018. https://www.fieldmuseum.org/about.

Furui, S., T. Kikuchi, Y. Shinnaka, and C. Hori. 2004. “Speech-to-Text and Speech-to-Speech Summarization of Spontaneous Speech.” IEEE Transactions on Speech and Audio Processing 12 (4): 401–8. https://doi.org/10.1109/TSA.2004.828699.

Goel, Aman. 2018. “How Does Siri Work? The Science Behind Siri.” Magoosh Data Science Blog (blog). February 3, 2018. https://magoosh.com/data-science/siri-work-science-behind-siri/.

Hirschberg, Julia, and Christopher D. Manning. 2015. “Advances in Natural Language Processing.” Science 349 (6245): 261–66. https://doi.org/10.1126/science.aaa8685.

IBM Cloud Education. 2021. “What Are Neural Networks?” August 3, 2021. https://www.ibm.com/cloud/learn/neural-networks.

Lalwani, Tarun, Shashank Bhalotia, Ashish Pal, Vasundhara Rathod, and Shreya Bisen. 2018. “Implementation of a Chatbot System Using AI and NLP.” SSRN Scholarly Paper ID 3531782. Rochester, NY: Social Science Research Network. https://doi.org/10.2139/ssrn.3531782.

Leahy, Connor. 2022. “Announcing GPT-NeoX-20B.” EleutherAI Blog. February 2, 2022. https://blog.eleuther.ai/announcing-20b/.

Liao, Min-Hsiu. 2018. “Museums and Creative Industries: The Contribution of Translation Studies.” January 2018. https://jostrans.org/issue29/art_liao.php.

Marshall, Christopher. 2020. “What Is Named Entity Recognition (NER) and How Can I Use It?” Super.AI (blog). June 2, 2020. https://medium.com/mysuperai/what-is-named-entity-recognition-ner-and-how-can-i-use-it-2b68cf6f545d.

Martin, Jenni, and Marilee Jennings. 2015. “Tomorrow’s Museum: Multilingual Audiences and the Learning Institution.” Museums & Social Issues 10 (1): 83–94. https://doi.org/10.1179/1559689314Z.00000000034.

Matas, Ariadna. 2020. “OpenGLAM Translation Sprint at Europeana 2020.” Europeana Pro. September 2020. https://pro.europeana.eu/event/openglam-translation-sprint-at-europeana-2020.

Melisa Palferro. 2018. “Different Approaches to Museum Translation.” #ucreatewetranslate (blog). March 8, 2018. https://www.ucreatewetranslate.com/different-approaches-museum-translation/.

Menon, Yasmin, and Will Lach. 2021. “Creating Accessibility in Museums with High Quality Translations.” Eriksen Translations Inc. November 19, 2021. https://eriksen.com/arts-culture/accessibility-museums-high-quality-translations/.

Mingyu, Lu, and Si Xianzhu. 2010. “Application of Machine Translation to Chinese-English Translation of Relic Texts in Museum.” In 2010 International Conference on Intelligent System Design and Engineering Application, 1:355–58. https://doi.org/10.1109/ISDEA.2010.4.

Mitkov, Ruslan. 2004. The Oxford Handbook of Computational Linguistics. OUP Oxford. https://books.google.com/books?hl=en&lr=&id=yl6AnaKtVAkC&oi=fnd&pg=PA219&dq=part+of+speech+tagging&ots=_VVi1buLDj&sig=y-3BFecOVtTtptCNFrS8Br-EFGE#v=onepage&q=part%20of%20speech%20tagging&f=false.

MTAAC Team. n.d. “Machine Translation and Automated Analysis of Cuneiform Languages.” Machine Translation and Automated Analysis of Cuneiform Languages. Accessed March 2, 2022. https://cdli-gh.github.io/mtaac/.

Multilingual Connections. n.d. “Museum Translation Services.” Multilingual Connections. Accessed March 2, 2022. https://multilingualconnections.com/industry-experience/museums-cultural-institutions/.

Nadeem, Moin, Anna Bethke, and Siva Reddy. 2020. “StereoSet: Measuring Stereotypical Bias in Pretrained Language Models.” ArXiv:2004.09456 [Cs], April. http://arxiv.org/abs/2004.09456.

Och, Franz. n.d. “Statistical Machine Translation Live.” Google AI Blog (blog). Accessed March 2, 2022. http://ai.googleblog.com/2006/04/statistical-machine-translation-live.html.

Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. “Bleu: A Method for Automatic Evaluation of Machine Translation.” In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, 311–18. Philadelphia, Pennsylvania, USA: Association for Computational Linguistics. https://doi.org/10.3115/1073083.1073135.

Scott, Josh. 2022. “Cohere Closes $125 Million USD Series B Round Led by Tiger Global.” BetaKit (blog). February 15, 2022. https://betakit.com/cohere-closes-125-million-usd-series-b-round-led-by-tiger-global/.

Siddhant, Aditya, Melvin Johnson, Henry Tsai, Naveen Ari, Jason Riesa, Ankur Bapna, Orhan Firat, and Karthik Raman. 2020. “Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation.” Proceedings of the AAAI Conference on Artificial Intelligence 34 (05): 8854–61. https://doi.org/10.1609/aaai.v34i05.6414.

Simonite, Tom. 2022. “YouTube’s Captions Insert Explicit Language in Kids’ Videos.” Wired, February 24, 2022. https://www.wired.com/story/youtubes-captions-insert-explicit-language-kids-videos/?utm_source=nl&utm_brand=wired&utm_mailing=WIR_FastForward_022822&utm_campaign=aud-dev&utm_medium=email&utm_content=WIR_FastForward_022822&bxid=60a682d211af1a6455755091&cndid=65158197&esrc=bouncexmulti_second%20&source=EDT_WIR_NEWSLETTER_0_TRANSPORTATION_ZZ&mbid=mbid%3DCRMWIR012019%0A%0A&utm_term=WIR_Transportation.

Turovsky, Barak. 2016a. “Ten Years of Google Translate.” Google. April 28, 2016. https://blog.google/products/translate/ten-years-of-google-translate/.

Turovsky, Barak. 2016b. “Found in Translation: More Accurate, Fluent Sentences in Google Translate.” Google. November 15, 2016. https://blog.google/products/translate/found-translation-more-accurate-fluent-sentences-google-translate/.

Zhu, Aihua. 2021. “Man-Machine Translation—Future of Computer-Assisted Translation.” Journal of Physics: Conference Series 1861 (1). http://dx.doi.org/10.1088/1742-6596/1861/1/012088.

Dalí Lives is an advanced instance of a growing phenomenon with the potential to disrupt both the art world and technologically mediated inter personal relationships. Ghostbots (sometimes called Griefbots) are an impending challenge for post mortem privacy and the arts space, and have found exploration in some recent socially conscious science fiction. This technology is defined loosely as efforts to use digital versions of an individual to recreate them in a way that can survive after the death of the individual. The first of these to gain major traction, Eternime, is now defunct after a period of media hype:

Two successors in the space, HereAfter AI and Replika AI, were both founded by computer scientists as projects to recreate specific individuals that they were hoping to memorialize. The ultimate fates of each of these projects is telling as to the future of this space. Replika does not recreate individuals, but rather uses input from individuals to create a digital friend. HereAfter AI is essentially a digital diary. For something like Dalí Lives to become mainstream for artists and for people seeking a form of digital immortality, something in between HereAfter and Replika needs to become widely available. HereAfter seeks to recreate an individual, while Replika applies machine learning to create generative conversations that an individual will respond well to. There is no bridge between this at the moment that is available without the resources of an entire museum. This has wide implications for who has the right to a digital immortality that is similar to the gate kept nature of most advanced AI – at the moment it is limited to those rich enough to hire a team of computer scientists and pay for the computing power, or to people who are already famous enough that organizations are willing to put in the effort and time to recreate them.

The consequences of poorly executed generative AI are vast – as it could lead to an individual or artist producing nonsensical or harmful conversations during an artistic expression or during conversations with close relatives. In scenarios where the artist or individual is deceased, there is no recourse if an AI is not representative of who they are, and there is a possible future to this technology where the quality of your digital afterlife is mediated by wealth. This is already present, as HereAfter AI and Mind Bank AI (discussed later) are very expensive. HereAfter AI relies on a paid, tier subscription model to maintain a digital afterlife. Like a Medieval Indulgence.

There is one other mass market Ghostbot/Griefbot that I found, Mind Bank AI, which markets itself as a “Digital Twin”. Mind Bank AI is a pretty shady company. Like HereAfter, I am unconvinced that their privacy practices are ready to handle the implications of a technology explicitly designed to gather the most intimate stories of a person’s life.

Mind Bank AI plays a slight of hand. It does not yet seem to use generative conversations. Instead, it draws people in with vague promises of self help and an eventual digital twin once AI technology has reached the point where it is possible. However, Dalí Lives proves that it is already largely possible, it is just gate kept by wealth and access to enough data about a person.

Mind Bank AI and Eternime are/were designed to be used for extended periods, possibly years, to gather enough data to recreate an individual. This is a reason why this technology is particularly suited to disrupt the arts space, as artists are likely to have vast banks of carefully crafted and stored records of their work, public appearances, and thought processes. Ghostbots, Griefbots, and Digital Twins are in very nascent stages, and are something to be watched along with privacy laws and the increased access to AI to people without vast resources and technology knowledge.

Bibliography

A Grief Bot Could “Resurrect” My Dead Husband. No Thanks. | HuffPost Life. (n.d.). Retrieved February 25, 2022, from https://www.huffpost.com/entry/grief-bots-death-loved-ones_n_5ad5193fe4b016a07e9f9a2d

Ahmad, M. A. (2016). After death: Big data and the promise of resurrection by proxy. Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, 397–408.

A Son’s Race to Give His Dying Father Artificial Immortality | WIRED. (n.d.). Retrieved February 25, 2022, from https://www.wired.com/story/a-sons-race-to-give-his-dying-father-artificial-immortality/

Eternime. (n.d.). Retrieved February 25, 2022, from https://web.archive.org/web/20180317190713/http://eterni.me/

Eternime. (2018, March 17). https://web.archive.org/web/20180317190713/http://eterni.me/

Eternime Alpha Two on Vimeo. (n.d.). Retrieved February 25, 2022, from https://vimeo.com/142260863

Eterni.me—Become Virtually Immortal. (2014, January 30). https://web.archive.org/web/20140130073423/http://eterni.me/

Figueroa, M. (2022, January 18). Ghostbots, the Quest for Digital Immortality and the Law—JURIST – Commentary—Legal News & Commentary. https://www.jurist.org/commentary/2022/01/mauricio-figueroa-ghostbots-digital-immortality-law/

HereAfter AI — Save the Stories of Someone You Love. (n.d.). Retrieved February 25, 2022, from https://www.hereafter.ai/

‘Hey, Google! Let me talk to my departed father.’ (n.d.). Washington Post. Retrieved February 25, 2022, from https://www.washingtonpost.com/technology/2019/08/29/hey-google-let-me-talk-my-departed-father/

LifeNaut. (n.d.). LifeNaut. Retrieved February 25, 2022, from https://www.lifenaut.com/

Replika. (n.d.). Retrieved February 25, 2022, from https://replika.ai/

Replika—Virtual AI Friend on the App Store. (n.d.). Retrieved February 25, 2022, from https://apps.apple.com/us/app/replika-your-ai-friend/id1158555867

Roman Mazurenko on the App Store. (n.d.). Retrieved February 25, 2022, from https://apps.apple.com/us/app/roman-mazurenko/id958946383

Selinger, E. (2018, March 13). Can Bots Help Us Deal with Grief? Medium. https://medium.com/s/when-robots-rule-the-world/can-bots-help-us-deal-with-grief-3de488cae96

Speak, Memory. (n.d.). Retrieved February 25, 2022, from https://www.theverge.com/a/luka-artificial-intelligence-memorial-roman-mazurenko-bot

Tech company lets you chat with your dead loved ones | Metro News. (n.d.). Retrieved February 25, 2022, from https://metro.co.uk/2021/12/16/ai-company-lets-you-chat-with-your-dead-loved-ones-15781759/

The Griefbot That Could Change How We Mourn. (n.d.). Retrieved February 25, 2022, from https://www.thedailybeast.com/the-griefbot-that-could-change-how-we-mourn

The Journey to Digital Immortality | by Marius Ursache | Medium. (n.d.). Retrieved February 25, 2022, from https://medium.com/@mariusursache/the-journey-to-digital-immortality-33fcbd79949

Eterni.me links to this post from their now defunct url. It is 7 years old and seems to be the only remnant of the company left. I think it would be noteworthy to do a “where are they now” of the founders.

Watch Black Mirror | Netflix Official Site. (n.d.). Retrieved February 25, 2022, from https://www.netflix.com/title/70264888

When his father died, this technologist created a chatbot, so his kids could talk to their grandfather | CBC Radio. (n.d.). Retrieved February 25, 2022, from https://www.cbc.ca/radio/spark/386-attention-residue-new-design-ethics-and-more-1.4548894/when-his-father-died-this-technologist-created-a-chatbot-so-his-kids-could-talk-to-their-grandfather-1.4548924