Oriental Ink Painting with a Computer Instead of a Brush

Traditional occidental painting techniques like watercolor or oil build an image from many layered brush strokes. You don't usually notice the individual strokes unless you stand very close. But in traditional oriental ink painting, called sumi-e, the brush strokes are the painting. Each stroke is made to capture maximum information about the subject.

examples of individual brush strokes — Example brush strokes in typical shapes, from Xie et al.

In its simplicity, sumi-e poses a challenge to computer-generated artistry. The computer can't simply add and layer a large number of standard strokes, because the shape of each brush stroke is entirely dependent on the subject. And subjects can be infinite.

Now three computer science researchers from the Tokyo Institute of Technology have cracked the sumi-e puzzle by training a digital ink brush with a technique called reinforcement learning.

digital brush learning how to paint — Iterations in the learning process of the digital brush, from Xie et al.

They trained their model brush not on entire images (like a bird) but only on individual brush strokes (like the curve of the bird's belly). They digitized eighty natural brush strokes and gave them to the model to mimic, along with a function that rewarded smoother shapes. (To your correspondent's dismay, "reward functions" in computer science don't actually hand out cookies.)

After the sumi-e brush program had been trained, the researchers tested it with a greater variety of natural brush strokes. Finally, they used it to create full paintings from photos--though they had to lend a helping hand, by manually drawing contours onto the photos for the program to find and fill.

edinburgh, real photo and rendered in sumi-e — Top: photo of Edinburgh. Bottom: painting rendered by the sumi-e program, from Xie et al.

Some digital painting programs attempt to satisfy artists by simulating all the physics of the painting process, from the exact angle of the brush to the absorption of ink into paper. Xie et al.'s sumi-e program is from a different family of methods, called "stroke-based rendering," whose aim isn't to create a believable experience for the user, but merely a believable end product. It's hard to argue with the results.

The research is posted on arXiv and has just been presented at the International Conference on Machine Learning in Edinburgh, Scotland. It's not yet available for public use--though if you're sufficiently math-savvy you could probably recreate the program yourself.