DRT

Implementation of Differentiable Refraction-Tracing for Mesh Reconstruction of Transparent Objects.

Project page | paper | video

We introduce a reconstruction method for solid transparent objects. Using a static background with a coded pattern, we establish a mapping between the camera view rays and locations on the background (Environment Matting). Differentiable tracing of refractive ray paths is then used to directly optimize a 3D mesh approximation(visual hull) of the object, while simultaneously ensuring silhouette consistency and smoothness.

Dependencies

Requirements: (tested on ubuntu 16.04)

• NVIDIA-OptiX-SDK-6.5.0 : Used to find the triangles intersected by each ray path.
• Meshlab: We use meshlabserver to remesh and calculate average per-vertex distance (Hausdorff Distance). Note that meshlab older than 2020.04 may not support explicit remesh operation.(Tested on MeshLabServer2020.04-linux.AppImage)
• Python with pytorch, numpy, trimesh, opencv-python, h5py, tqdm and imageio.
• Ninja build system (needed by pytorch's cpp_extension). Run apt install ninja-build if you use ubuntu.

Captured data

Our example data is captured by camera point grey or cellphone Redmi and released here

Scanned mesh(GT) and visual hull already contained in ./data/

Usage

• Download Optix-SDK, meshlabserver and our captured data.
• Set corresponding paths in config.py. Note that you should use absolute path to Optix.
• Run python optim.py
• Reconstructed mesh will be saved in ./result/ by default.

Citation

Please cite the paper in your publications if it helps your research:

@article{DRT,
title = {Differentiable Refraction-Tracing for Mesh Reconstruction of Transparent Objects},
author = {Jiahui Lyu and Bojian Wu and Dani Lischinski and Daniel Cohen-Or and Hui Huang},
journal = {ACM Transactions on Graphics (Proceedings of SIGGRAPH ASIA 2020)},
volume = {39},
number = {6},
pages = {195:1--195:13},
year = {2020},
}