In 2019, Microsoft announced its AI for Cultural Heritage program, aiming to preserve and treasure people, places, and historical artifacts through AI technology and collaboration with cultural nonprofit organizations.
“Preserving cultural heritage isn’t something that is solely nice to have or nice to do, it’s sometimes imperative to the well-being of the world’s societies.”
– Brad Smith, the President of Microsoft
According to Poonam Gandhi, a professor at Flame University, culture and heritages shape human society and provide a sense of identity. Specifically, cultural heritages can stimulate tourism, which hence benefits the local economies, educate people with culture and history, and enhance social inclusiveness.
The importance of preserving cultural heritage is obvious, meanwhile, people are facing difficulties in the preservation process. So, what can AI and other modern technologies actually do in the process of preserving cultural heritages that makes Microsoft and other organizations invest in them?
Antiques Restoration
Due to natural disasters or man-made damages during the war, lots of precious historical heritages had been broken into pieces. Considering the time and cost efficiency, plus the difficulty of lacking pieces of evidence of their original looks, it’s very difficult to repair those historical artifacts and might need the efforts of several generations. Just like solving a big jigsaw puzzle without knowing what it should look like, everything starts from scratch and guesses.
RePAIR
The ancient Roman city of Pompeii which was destroyed and buried by the eruption of Mount Vesuvius in A.D. 79, is now only left with the incomplete architecture and thousands of irregular fragments of ancient artifacts, such as vases and frescoes.
The RePAIR (Reconstructing the Past: Artificial Intelligence and Robotics meet Cultural Heritage) project is proposed by a group of scientists at the Italian Institute of Technology (IIT) to help with the time-consuming and labor-costly physical reconstruction process of Pompeii restoration. The technologies involved in the project are robotics, 3-D scanning, machine-learning algorithms and artificial intelligence.
The first component is about scanning the fresco puzzle pieces using high-tech computers, using machine-learning algorithms to solve the puzzles, and having human expertise to guide the AI at the same time.
A team of archeologists from the University of Lausanne, who have attempted to solve the puzzle in the past, along with other professional art historians and scholars input their current knowledge about the Pompeii puzzles into the AI algorithm, then let it evolve and study by itself based on those examples, and apply to a larger scale. The computer software will compare all the fragments in pairs and evaluate their degree of similarity based on the shapes, the fitness, and how the illustrations on the pieces match up, which helps draw a picture of what the original complete piece looks like.
The second component brings robots into the workplace. The robot should be able to scan those pieces on its own, and move the fragments together as a complete one based on the paired results generated by AI. In the end, it will reassemble the pieces together into a finished “puzzle”. Designed by Nikolaos Tsagarakis and his colleagues at the Humanoid & Human Centered Mechatronics Lab at IIT in Genoa, the robot will look like an arm that has a similar size as an average person, weighing 25 to 30 kilograms and lengthening 80 to 100 centimeters. A soft glove hand will be attached to the arm to grab and move the fragments with extreme care, and gather information about them while holding them in hand. The robot will help save a large amount of time and human resources. It doesn’t need to sleep or eat, but can work 24 hours a day without rest. Antonio Bicchi, a senior scientist at IIT in Genoa and the chair of robotics at the University of Pisa is in charge of the design of the robot’s hand, and says the ideal version of it in the future will also be able to allow humans to wear them and gather data during the process.
The project and its techniques are still at the experimental stage. The two experimental subjects will be the frescoes from the ceiling of the House of the Painters at Work, in the Insula of the Chaste Lovers, and the frescoes from the Schola Armaturarum. At the end of the project, the finished frescoes will be exhibited to the public for visitors to experience ancient Pompeii and for researchers to study. And this is the first time in the archeological field that AI will be used at such a large scale, and robotic hands will be responsible for such a large amount of pieces.
Therefore, the progress it can contribute to cultural heritage restoration and human history is significant and meaningful. “If the project succeeds, the technology can be used to reconstruct a variety of fragmented cultural artifacts whose restoration has remained ‘out of human reach’, ” said the project coordinator Marcello Pelillo, a professor of computer science and artificial intelligence at the University of Venice.
The RePAIR scientists aim to install the robot in Pompeii by the spring or summer of 2022. And hopefully, this project will succeed and benefit other cultural sites in the world that need to be repaired.
Scan4Reco
Another EU-funded project called Scan4Reco is also a great example of using artificial intelligence-assisted technologies for cultural heritages’ preservations.
They first use depth-scanning cameras to create a high-resolution 3D replica to transfer the artifact into digital files. Then, more information about the object’s underlying layers which cannot be witnessed directly by human eyes is added to the research by the sensor data.
“Very specialized sensors used in the conservation domain carry out non-destructive diagnostic analysis of the different layers that is equivalent to the more destructive chemical ones,” the deputy project coordinator Dr. Anastasios Drosou explains.
The sensor data is combined with predictive modeling technologies using AI, whose deep learning algorithms simulate the artifact’s appearance as it ages. Therefore, it can suggest conservators with accordingly treatments to protect the artifacts before their conditions get more terrible, or to minimize the existing damages and keep their current looks. According to Dr. Drosou, when the system detects a scratch or some color fading in a painting, the system would advise the conservators to apply a particular chemical on the scratch, or paint or cleaner in the other area.
Digitalization
Another beneficial and impactful progress that technology brings to the field is the digitalization of cultural heritages, where laser scanning and photogrammetry play essential roles. Having digital cultural heritage will help record history, establish a digital library for researchers and future generations to learn about them, and enrich cultural vibrancy.
The Mogao Caves, located in Gansu province in northwestern China, with over 2,000 sculptures and 45,000 square meters of unique and fragile painted murals, all of which are between 600 and 1,600 years old, is facing the challenge of preserving them in their current conditions.
The caves are great demonstrations of the history of the Silk Road as well as related commercial trades and cultural exchanges across countries. However, due to thousands of years of natural disasters and damages, as well as the overloaded tourists after its development and reopening as a historical visiting site, the speed of painted statues damage and fresco degradation of Mogao Cave is dramatic.
To better preserve this UNESCO World Heritage Site, the researchers and scientists in China decided to use Laser Scanning and Multi-baseline Rotating Photogrammetry to turn the statues and frescos into digital formats, and store them in a safe data library for future studies and possible virtual exhibitions to let people appreciate the caves on their mobile devices with the similar experience as if they were in the caves.
- Photogrammetry: a sophisticated process by which information is extracted from photographs to create accurate three-dimensional maps and models. Using ultra-high-resolution aerial photographs, this practice combines UAV-mounted overhead sensors with powerful GIS mapping systems to create dynamic, measurable documents for a number of real-world situations and uses. (Source: Science Direct)
- Laser Scanning: an automatic, direct measurement of 3D points that can accurately measure and collect data from objects, surfaces, buildings, and landscapes. Laser scanners collect information in the form of point cloud data, which consists of millions of 3D coordinates (XYZ coordinates). (Source: Science Direct)
Another historical site that has been damaged and needed further protection and repair is the Notre-Dame Cathedral which was burnt by fire in 2019. The government is planning for its reconstruction and says hoping to finish by 2024.
Thanks to the digital scanning that they previously did before the fire, the team are now able to create a 3D BIM model of the Notre-Dame Cathedral as it existed before. Then, using recent reality capture scans to create the post-fire 3D model using a 3D design software called Autodesk, and comparing these two versions to rebuild the Cathedral.
Since the roof which was the most burnt part of the Cathedral was made of wood, this cultural tragedy rings the alarm for other historical buildings with wood structures. Performing 3D scanning and modeling to have digital forms of the heritages on file would be a great kind of look-ahead preparation for future possible accidents.
Conclusion
AI and other advanced technologies can help with the restoration and digitalization of tangible cultural heritage. They work as assistants of people on those works that cannot be achieved by human abilities or are time-consuming. They also work as tools for people to utilize and advance their working process to a higher level.
We’ve already seen some great progress that utilizing technologies in cultural heritages’ protections from the above projects and examples. In the future, we can also have a positive prospect that technologies would be able to make more achievements in this area by human inventions and under human supervision.
Bibliography
“AI for Cultural heritage.” Microsoft. Accessed February 25, 2022. https://www.microsoft.com/en-us/ai/ai-for-cultural-heritage.
“An Introduction to Drone Photogrammetry.” Mapware. Accessed March 4, 2022. https://mapware.ai/an-introduction-to-drone-photogrammetry/?gclid=CjwKCAiAjoeRBhAJEiwAYY3nDBfZ-wtHG1wd1Xv4sfiGfTEjObRb2Pm-CkaRFOXQycgG5RP7R-1R0BoCm-cQAvD_BwE#whatisutm_term=photogrammetry&utm_campaign=Traffic_Clicks_Global+%5BJoe%5D&utm_source=adwords&utm_medium=ppc&hsa_acc=7859616296&hsa_cam=15446499960&hsa_grp=136083379128&hsa_ad=566158081175&hsa_src=g&hsa_tgt=kwd319349969229&hsa_kw=photogrammetry&hsa_mt=p&hsa_net=adwords&hsa_ver=3.
Brandon, Elissaveta M. “How AI and Robotics are Reconstructing A 2,000-year-old Fresco in Pompeii.” Fast Company. December 23, 2021.
“Cutting Edge | Protecting and Preserving Cultural Diversity in the Digital Era.” UNESCO. October 28, 2020. https://en.unesco.org/news/cutting-edge-protecting-and-preserving-cultural-diversity-digital-era.
Denbo, Seth. “Digitalizing Cultural Heritage: Historic Preservation and The Mogao Caves in China.” Perspectives on History. December 1, 2016. https://www.historians.org/publications-anddirectories/perspectives-on-history/december-2016/digitizing-cultural-heritage-historic-preservation-and-the-mogao-caves-in-china.
“‘Digital Surrogates’ Assist in the Preservation of Cultural Artefacts.” CORDIS. February 19, 2019. https://cordis.europa.eu/article/id/248968-digital-surrogates-assist-in-the-preservation-of-cultural-artefacts.
Gandhi, Poonam. “World Heritage – Why Preserve, Conserve and Protect?” TravelWorld.com. April 20, 2021. https://travel.economictimes.indiatimes.com/speakeasy/world-heritage-day/4891.
“How Technology Is Preserving Our Cultural Heritage.” Reader’s Digest. Accessed February 25, 2022. https://www.readersdigest.co.uk/inspire/life/how-technology-is-preserving-our-cultural-heritage.
Horgan, Rob. “How Digital Technology is Aiding Notre Dame Rebuild, Two Years since Tragic Blaze.” New Civil Engineer. April 15, 2021. https://www.newcivilengineer.com/latest/how-digital-technology-is-aiding-notre-dame-rebuild-two-years-since-tragic-blaze-15-04-2021/#:~:text=In%20a%20bid%20to%20aid,existed%20before%20the%20catastrophic%20event.
Ibaraki, Stephen. “Artificial Intelligence For Good: Preserving Our Cultural Heritage.” Forbes. March 28, 2019. https://www.forbes.com/sites/cognitiveworld/2019/03/28/artificial-intelligence-for-good-preserving-our-cultural-heritage/?sh=bf2912f4e960.
“Laser Scanning Reveals Cathedral’s Mysteries | National Geographic.” National Geographic. June 23, 2015. Video, 4:10. https://youtu.be/jAi29udFMKw.
“Laser Scanning.” ScienceDirect. Accessed March 4, 2022. https://www.sciencedirect.com/topics/earth-and-planetary-sciences/laser-scanning.
Pinkowski, Jen. “Pompeii’s Ruins to Be Reconstructed by Robot.” Scientific American.December 6, 2021. https://www.scientificamerican.com/article/pompeiis-ruins-to-be-reconstructed-by-robot/.
“RePAIR Project.” Pompeii Sites. September 2, 2021. Video, 4:49. https://youtu.be/v9rpoLL5eLM.
“Reconstructing the Past: Artificial Intelligence and Robotics Meet Cultural Heritage.” CORDIS. Accessed February 25, 2022. https://cordis.europa.eu/project/id/964854.
“The RePAIR Project Begins – Robotics and Digitisation at the Service of Archaeology.” Pompeii. Accessed February 25, 2022. http://pompeiisites.org/en/comunicati/the-repair-project-begins-robotics-and-digitisation-at-the-service-of-archaeology/.
Werkheiser, Greg. “Artificial Intelligence: Cultural Heritage Required.” Cultural Heritage Partners. Accessed February 25, 2022. http://www.culturalheritagepartners.com/artificial-intelligence-cultural-heritage-required/.
“What Is Laser Scanning and How Can It Be Used?” TopoDot. June 11, 2020. https://blog.topodot.com/what-is-laser-scanning-and-how-can-it-be-used/.
Wiggers, Kyle. “Microsoft Launches AI for Cultural Heritage Program.” Venture Beat. July 11, 2019. https://venturebeat.com/2019/07/11/microsoft-launches-ai-for-cultural-heritage-program/.