How a Tweet inspired Instapainting founder Chris Chen to build a master painter robot controlled by Twitch viewers.
Since he couldn’t paint like a master, Chris Chen made a painter robot that could.
Chen is the founder of Instapainting, a company that hires accomplished artists to turn photographs into paintings on canvas. When a Twitter user attempted to steal a customer by offering to have a robot paint the customer’s image, Chen was inspired to make a painter robot of his own.
Three weeks and $250 later, Chen had his first prototype. His bot quickly evolved from mimicking human painters to taking paint stroke commands from live TwitchTV users.
“I don’t think there was ever a real, physical world collaborative Twitch stream,” said Chen.
So he started one.
For years Twitch has been the social video platform of choice for gamers — a place to chat with fellow enthusiasts, stream play-throughs or watch eSports pros battle it out in gaming tournaments. When Twitch switched on new flavors of programming last fall, millions of art fans tuned in to watch a nine-day marathon of “The Joy of Painting,” the 1980s TV show hosted by frizzy-haired, mellow-talking painter Bob Ross.
It was Chen’s second Twitch Paints livestream collaboration. In the debut episode, which made the front page of Hacker News, Twitch commenters had to write code commands to tell the robot where to paint.
For the second show, which attracted more than six thousand Twitch viewers, Chen made it easier, allowing viewers to control the robot painter by typing simple x and y axis coordinates. The finished work is being sold via Ebay with proceeds from the auction going to Watsi, an organization that uses crowd-funding to provide medical care to people around the world.
A History of Robot Painters
Chen’s may be the first painting robot that is crowd controlled via the internet, but it’s not the first robot artist.
The Basel Academy of Art and Design’s “mobile sensory image production mechanism” BNJMN began signing his name to original works in 2013, and E-David (“Drawing Apparatus for Vivid Interactive Display”) started creating visually optimized works of art from the University of Konstanz in 2009. But it was Chen who took the robot art game to the next level of crowd-control creativity.
Chen became curious about crowdsourcing the painting process after seeing a group of Twitch users collaborate on a more scientific project.
“I saw a Twitch Installs Arch Linux project, and that gave me the idea,” said Chen.
Linux is an operating system like Apple OS X or Microsoft Windows, but unlike those systems, Linux is free and open-source. During the Twitch stream, the most popular keystroke every five seconds in a Twitch chat was used to command an Arch Linux virtual machine, giving an entire community of Twitch users control over a single machine. Goals for that stream included booting Arch Linux from the hard disk and configuring a fully working X server.
Building a Rembrandt Robot
Though the technology may looks and sounds complex, Chen said the execution was simple. When an artist draws, the movements are recorded digitally in three dimensions: X and Y (up, down and across the canvas) and Z (for applying pen pressure to the canvas).
The robot then plays the movements back to recreate the painting. Because it skips lulls in movement, the robot can recreate a precise replica faster than humans.
“The robot uses two stepper motors and a servo motor,” explained Chen. “It is modified from a 2D plotting system,” which creates images by moving a pen or instrument across a canvas. This plotting system is essentially the X and Y axis the robot uses to place paint on the canvas.
He added a water brush and DC peristaltic pump so the robot could wet the watercolor palette and apply the paint to the paper.
“The motors are connected to an Arduino Due, and that can plug into a computer where additional software is run to control it and interface with control inputs like a Wacom tablet, mouse or even a Myo armband,” he said.
According to Chen, it was equally simple to connect the robot to Twitch.
“For the Twitch Paints project, the computer was substituted with a Raspberry Pi, which sent commands to the robot,” he explained. “It was hooked up to read commands from the chat window and access a special database to read commands from the point-and-click GUI control.”
The graphical user interface (GUI) allowed viewers to add coordinates in the chat window rather than type elaborate software code instructions.
This futuristic technology has the potential to help anyone paint like a pro, and Chen has considered experimenting with new designs for the robot that would include using analog feedback and automatic color mixing.
For now, he’s focusing on Instapainting, helping it expand the types of handmade art it offers.
“While the robot is cool, it’s not going to replace the breadth and depth of human-designed art any time soon.”