libcom: everything about image composition

Introduction

libcom is an image composition toolbox. The goal of image composition is inserting one foreground into a background image to get a realistic composite image. Generally speaking, image composition could be used to combine the visual elements from different images.

libcom covers various related tasks in the field of image composition, including image harmonization, painterly image harmonization, shadow generation, object placement, generative composition, quality evaluation, etc. For each task, we integrate one or two selected methods considering both efficiency and effectiveness. The selected methods will be continuously updated upon the emergence of better methods.

Main Parts

• get_composite_image generates composite images using naive copy-and-paste followed by image blending.
• OPAScoreModel is an object placement assessment model that evaluates the rationality of object placement by predicting a rationality scores.
• FOPAHeatMapModel can predict the rationality scores for all locations with a pair of background and scaled foreground as input in a single forward pass.
• color_transfer tranfers the color of foreground to fit background scene using reinhard algorithm.
• ImageHarmonizationModel contains several pretrained models for image harmonization, which aims to adjust the illumination statistics of foreground to fit background.
• PainterlyHarmonizationModel contains serveral pretrained models for painterly image harmonization, which aims to adjust the foreground style of the painterly composite image to make it compatible with the background.
• HarmonyScoreModel predicts harmony score for a composite image, in which larger harmony score implies more harmonious composite image.
• InharmoniousLocalizationModel supports the localization of the inharmonious region in a synthetic image.
• FOSScoreModel contains two foreground object search models, which can be used to evaluate the compatibility between foreground and background in terms of geometry and semantics.
• ControlComModel is a controllable image composition model, which unifies image blending and image harmonization in one diffusion model.
• ShadowGenerationModel takes in deshadowed composite image and foreground object mask, and generates images with semantically plausible foreground shadows.

For more detailed user guidance and method description, please refer to our [documents].

Requirements

The main branch is built on the Linux system with Python 3.8 and PyTorch 1.10.1. For other dependencies, please refer to [conda_env] and [runtime_dependencies].

Get Started

Create runtime environment

git clone https://github.com/bcmi/libcom.git
cd libcom/requirements
conda env create -f libcom.yaml
conda activate Libcom
pip install -r runtime.txt # -i https://pypi.tuna.tsinghua.edu.cn/simple
# install a specific version of taming-transformers from source code
cd ../libcom/controllable_composition/source/ControlCom/src/taming-transformers
python setup.py install

Installation

pip install libcom

or

python setup.py install

After that, you can verify the installation by running:

cd tests
sh run_all_tests.sh

The visualization results can be found in results folder.

Download pretrained models

During using the toolbox, the pretrained models and related files will be automatically downloaded to the installation directory. Note downloading the pretrained models may take some time when you first call some models, especially ShadowGenerationModel, ControlComModel, and PainterlyHarmonizationModel.

Alternatively, you can download these files from [Modelscope] or [Huggingface] in advance, and move them to the installation directory.

Contributors

• Institution: Brain-like Computing and Machine Intelligence (BCMI) Lab.
• Project Initiator & Team Manager: Li Niu.
• Architect & Lead Developer: Bo Zhang.
• Documentation Manager: Jiacheng Sui.
• Module Developers: Jiacheng Sui, Binjie Gao, Lingxiao Lu, Xinhao Tao, Junyan Cao.

License

This project is released under the Apache 2.0 license.

Bibtex

If you use our toolbox, please cite our survey paper using the following BibTeX [arxiv]:

@article{niu2021making,
  title={Making images real again: A comprehensive survey on deep image composition},
  author={Niu, Li and Cong, Wenyan and Liu, Liu and Hong, Yan and Zhang, Bo and Liang, Jing and Zhang, Liqing},
  journal={arXiv preprint arXiv:2106.14490},
  year={2021}
}