FRIDA, which stands for Framework and Robotics Initiative for Developing Art, is a robotic arm that collaborates with humans to create paintings on canvases, using a paintbrush.
FRIDA is named in honor of the famous painter Frida Kahlo, one of the most important female artists of the 20th century.
The project was developed under the leadership of Schaldenbrand, along with Jean Oh and Jim McCann, who are members of The Robotics Institute from Carnegie Mellon University.
FRIDA is a tool that can be used to enhance the creative process, allowing artists to explore new possibilities and push the boundaries of their craft.
The model
FRIDA’s operation relies on AI models similar to those powering tools like OpenAI’s ChatGPT and DALL-E 2, using prompts to generate text or images. Taking things a step further, FRIDA produces physical paintings through its embodied robotic system.
While most AI bots do not have a corresponding physical robot, FRIDA does. And that’s what makes this one so special, for it is able to produce its results in real life on paper.
FRIDA utilizes machine learning to simulate how it would paint an image using brushstrokes, evaluating its progress as it works. To bridge the gap between simulation and reality, it uses the “Real2Sim2Real” methodology providing a high-fidelity painting simulation environment.
“Real2Sim2Real” is an approach that involves training a machine learning model in a simulated environment (Sim2) and then transferring that model to the real world (Real), followed by retraining the model in the real environment to improve its performance (Real2).
How it works
Initially, the robot spends over an hour learning how to use its paintbrush. It then employs large vision-language models, such as OpenAI’s CLIP, which are trained on vast datasets containing paired text and images scraped from the internet, to comprehend the input it receives.
To avoid an American or Western bias, they fed the models’ headlines from news articles to provide a sense of global events and trained them on images and text that better represent diverse cultures.
With a brush and paint, FRIDA makes its first strokes, periodically using an overhead camera to capture an image of the painting. This image provided by the camera helps FRIDA evaluate its progress and refine its plan, when needed. The entire process may take several hours.
Throughout the creative process, the robot uses AI to communicate with the artists. They can provide inputs to FRIDA in various forms, such as a text description, other artworks for stylistic inspiration, or by uploading a photo.
Furthermore, the group responsible for FRIDA is investigating other forms of input, including audio. As an illustration, they played the song “Dancing Queen” by ABBA and instructed FRIDA to create a painting inspired by the song.
The resulting works of art produced by FRIDA have a whimsical and impressionistic style. Even if FRIDA makes mistakes during the process, it creatively incorporates the errors into the final product, resulting in a unique and expressive painting.
Conclusion, future research
FRIDA is an amazing robotic painting system, but it is not an artist in the traditional sense. Rather, it serves as a tool for artistic collaboration and facilitates a partnership between humans and technology in the creation of works of art.
The team behind FRIDA is also working to address some of the limitations of current large vision-language models by continuously refining the models they use.
