Skip to content

Learning to Paint with Model-based Deep Reinforcement Learning

Notifications You must be signed in to change notification settings

ruinnight/LearningToPaint

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Learning to Paint with Model-based Deep Reinforcement Learning

---> https://arxiv.org/abs/1903.04411

Abstract

We show how to teach machines to paint like human painters, who can use a few strokes to create fantastic paintings. By combining the neural renderer and model-based Deep Reinforcement Learning (DRL), our agent can decompose texture-rich images into strokes and make long-term plans. For each stroke, the agent directly determines the position and color of the stroke. Excellent visual effect can be achieved using hundreds of strokes. The training process does not require experience of human painting or stroke tracking data.

You can easily use colaboratory to have a try.

DemoDemoDemo DemoDemoDemo

YouTube bilibili

Architecture

Installation

Use anaconda to manage environment

$ conda create -n py36 python=3.6
$ source activate py36
$ git clone https://github.com/hzwer/LearningToPaint.git
$ cd LearningToPaint

Dependencies

pip3 install opencv-python==0.4.1
pip3 install tensorboardX
pip3 install opencv-python

Testing

Make sure there are renderer.pkl and actor.pkl before testing.

You can download a trained neural renderer and a CelebA actor for test: renderer.pkl and actor.pkl

$ wget "https://drive.google.com/uc?export=download&id=1-7dVdjCIZIxh8hHJnGTK-RA1-jL1tor4" -O renderer.pkl
$ wget "https://drive.google.com/uc?export=download&id=1a3vpKgjCVXHON4P7wodqhCgCMPgg1KeR" -O actor.pkl
$ python3 baseline/test.py --max_step=100 --actor=actor.pkl --renderer=renderer.pkl --img=image/test.png --divide=4
$ ffmpeg -r 30 -f image2 -i output/generated%d.png -s 512x512 -vcodec mpeg4 video.mp4 -q:v 0 -q:a 0
(make a painting process video)

We also provide with some other neural renderers and agents, you can use them instead of renderer.pkl to train the agent:

triangle.pkl --- actor_triangle.pkl;

round.pkl --- actor_round.pkl;

bezierwotrans.pkl

Training

Datasets

Download the CelebA dataset and put the aligned images in data/img_align_celeba/******.jpg

Neural Renderer

To create a differentiable painting environment, we need train the neural renderer firstly.

$ python3 baseline/train_renderer.py
$ tensorboard --logdir train_log --port=6006
(The training process will be shown at http://127.0.0.1:6006)

Paint Agent

After the neural renderer looks good enough, we can begin training the agent.

$ python3 baseline/train.py --max_step=200 --debug --batch_size=96
(A step contains 5 strokes in default.)
$ tensorboard --logdir train_log --port=6006

Results

Painting process in different datasets

## FAQ **Why does your demo look better than the results in the paper?**

In our demo, after painting the outline of each image, we divide it into 4 × 4 patches to paint parallelly to get a high resolution.

Your main difference from Geometrize

Our research is to explore how to make machines learn to use painting tools. Our implementation is a combination of reinforcement learning and computer vision. If you are interested, welcome to see our paper.

Resources

Contributors

If you find this repository useful for your research, please cite the following paper:

@article{huang2019learning,
  title={Learning to Paint with Model-based Deep Reinforcement Learning},
  author={Huang, Zhewei and Heng, Wen and Zhou, Shuchang},
  journal={arXiv preprint arXiv:1903.04411},
  year={2019}
}

About

Learning to Paint with Model-based Deep Reinforcement Learning

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 80.6%
  • Jupyter Notebook 19.4%