So what you listen to when drawing is important. I listen to Indian classical music, Gregorian chant, and unknown composers like György Ligeti, Leo Ornstein, and Terry Riley. Music fuels my creativity and allows me to be fully present in the moment.
For years I've been obsessed with how my paintings sound. The AI revolution has led me to seek out experts who can help me. My first contact was Radhika Dirks, an expert in AI and quantum computing. We had a few Zoom sessions with her and she said that as far as she knew, no AI program could help me. Instead, she suggested creating a visual alphabet that matched colors to the musical chords I heard in my mind.
I thought it might be a way to increase my creativity. It is also based on the unconscious alphabet idea that has influenced my art throughout my career.
We met with musicians and AI experts to create a visual alphabet
I started looking for musicians to collaborate with and met Anthony Cardella, a young and incredibly talented pianist in Los Angeles. He was a PhD student at his USC, and he happened to know and sometimes play many of the unknown composers I listen to when I paint.
We have started a collaboration. We sat down and examined my drawings together. I zoomed in on the colors in Photoshop and felt the notes. Then I say to Anthony. For example, I said, I think this is the color of the B note. He hit his B and I said, “No, that's not it. Why don't you try a B sharp?” After a few tries, he suddenly started hitting the right notes. I knew it because the colors started vibrating for me. Together, we tabulated the codes for 40 colors.
Shortly thereafter, through mutual connections, I met an AI researcher named Jonah Lynch. He works at the intersection of digital humanities and machine learning. I invited him to my ranch in central California and explained the work I had done and how I painted. We had long discussions about art, poetry, and creating AI algorithms that you can type code into.
He developed a program that would “read” my drawings and convert them into music. I gave him the main colors I used in each picture and the codes he heard when he saw those colors. Jonah watched videos of me drawing and studied the movements. He created software that moved my hand, sampled the image of the painting according to my hand movements, and assigned each color sampled from the painting to a corresponding code. This sequence of codes was then fed into the neural network, which memorized most of the last code. He inspired a network of 500 years of keyboard music to create pages of sheet music, “dreaming up” new sequences based on color code sequences and the history of Western music.
When I heard the music playing, tears came to my eyes. It was just a rough reproduction of something I heard while I was painting, but I thought, “This is it.”
I took the music back to my pianist, Anthony. Amazingly, I was able to point to the sheet music and tell him what song I was listening to while drawing. And he says, “Yes, I can see that from the code.” Indian ragas, Gregorian chant, Ligeti, Ornstein, they were all there.
Still, at that stage music consisted primarily of a series of chords. Anthony said he could make a melody with a little rearranging.
AI is still a tool that requires human oversight
We have composed music for several paintings and performed it for audiences all over the world. Last month, I held a concert at the Forest Lawn Museum in Los Angeles, where I also exhibited some of my paintings. While Anthony was performing, the audience was able to admire the paintings, which was a very profound experience. Several people cried.
At the launch of my latest exhibition during the opening week of the Venice Biennale, Anthony performed the world premiere of the sonata “Only Through Time'', a sonata he composed inspired by my paintings. and performed to a live audience. After the performance, I talked to some people and they said that they could clearly see where the colors and musical notes intersect in the painting. It was something they had never experienced.
I know that many people are very afraid of AI, but I also believe that AI is a tool that requires human oversight. It's not a means to an end. Still, it opened up a lot of possibilities and enhanced my creative process. Without that, I don't know if I would have been able to truly unleash the musicality of my paintings. Listen to the sonata below.
