Technology that makes art

“Machines that paint art” are probably not the first words that comes to mind when you think about the type of project that a Dutch bank and advertising firm would create together, but in 2014 that’s exactly what happened with The Next Rembrandt project .

The Dutch bank ING was looking for a way to modernize their sponsorship of Dutch arts and culture and so they began working with advertising firm J. Walter Thompson on a way to combine the latest technology with historical arts and culture. The idea they landed on was to see if a machine could learn the artistic style of famous 17^th century painter Rembrandt Van Rijn and then use that data to create a new original Rembrandt painting.

A team of experts in art and technology from places such as Microsoft, TU Delft The Rembrandt house museum and, The Mauritshuis and led by JWT’s creative director Bos Korsten were put together to tackle the daunting challenge. The first step was to build a database of all Rembrandts works of art. For this, they used high quality digital photos and 3D digital scans of 346 paintings. These were then up-scaled even further by learning algorithms they had created to study as much of the paintings detail as possible, while also resizing the images to a consistent size of 150GB. Once they had built the database, they decided to further study the details in the portraits Rembrandt painted since they made up the majority of his works, offering the biggest pool of data. Further research determined Rembrandt painted the majority of these portraits between 1632 and 1642. The next step in refining the data was to look at the demographics of these portraits. This revealed that 49% were of women and 51% of men. The team then started examining every detail of Rembrandt’s male portraits to see which features occurred the most. This analysis let to the conclusion that the new original Rembrandt painting should be a portrait of a Caucasian male with facial hair who is roughly thirty to forty years old wearing dark clothing with a collar along with a hat and facing to the right.

The challenge for the team was then how to use this data to teach a machine to paint just like Rembrandt. As Developer Ben Haanstra put it, “We had to create a whole painting using just data from Rembrandt’s paintings”. To do this the team created software that let the computer understand how Rembrandt painted based on his use of painting materials, composition and geometry. Then the computer used facial recognition software that used various algorithms along with statistical analysis to identify, classify, and create a “typical” Rembrandt eye, nose, mouth, hat, and collar. Now that the computer could create original facial features they needed a way to build a complete face as Rembrandt would of. The team created an algorithm that measured over 60 facial data points of Rembrandt’s paintings including the distance between the eyes, distance from the eyes to the nose, and the proportions of each facial feature compared to one another. With this information, the computer could now arrange, scale, and place all facial features to create a complete face. To render the final painting image took over 500 hours.

However, an image on a computer screen is not a painting. A painting has texture, brush strokes, and paint layers build up. To replicate Rembrandt’s brushstrokes the team went back and studied the 3D scans of his paintings. For this, the team used algorithms that analyzed the layers of paint and the texture on the canvas. That information was then used by the computer to create a height map which it would use produce the new Rembrandt. Containing 148 million pixels the final work of art was painted using a 3D printer that uses an elevated printing technique and paint based UV ink which it put down in 14 separate layers. The result is an original Rembrandt painting created hundreds of years after his death.

If the simple fact that a computer created an artistic masterpiece is not enough, the project does address a real world problem. Accidents happen. Paintings get lost or stolen, destroyed by natural disasters or just degrade over time. The Next Rembrandt project offers the possibility that when something like this does happen to an irreplaceable work of art there is a way to repair or replace it. While these occurrences are not common, when they do happen it can be devastating for such important artifacts to be lost. The context of the project and the insight it offers is also very interesting. We live in a technological world where we use technology for so many things. But most technology is used to make life easier or provide instant communication. When do we ever see technology that produces art Vs. humans using technology as a tool to create art. What if we put the technological resources used to do things like create the smartphone or developing the internet to use creating art? Would we see a machine on the stage at Symphony hall creating the most beautiful music we have ever heard? This idea of technology and art merging was conveyed best in a quote by team member Ron Augustus of Microsoft, “Data is used by many people today to help them be more efficient and knowledgeable about their daily, work, and about the decisions they need to make. But in this project it’s also used to make life itself more beautiful. It really touches the human soul.” This project also fits in directly with the theme of our Physical Computing Course. The Next Rembrandt is in its rawest form a computer sensing information from the physical world, analyzing that data and making computations, and then outputting the results back to the physical world in the form of a painting. While it is enormously more complex and did involve some human input along the way I stand by the fact that it is the same as having three encoders connected to an Arduino with each encoder mapped to either red, green, or blue. Also connected to the Arduino is a RGB led. As the encoders are turned to different numerical values, the Arduino receives physical data and then combines it as a rendered color in the RGB LED. In both The Next Rembrandt and the example, I gave above physical data goes in to a computer. Computation takes place. New data is output back to the world in the form of a physical change. The Next Rembrandt is a perfect example of the potential power of physical computing devices in the not so distant future.