Skip to content
/ Gen3D Public

[CSUR 2023] A Survey on Deep Generative 3D-aware Image Synthesis

weihaox.github.io/Gen3D

License

MIT license
160 stars 11 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

weihaox/Gen3D

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

13 Commits
CONTRIBUTING.md
CONTRIBUTING.md
 
 
LICENSE
LICENSE
 
 
README.md
README.md
 
 
index.html
index.html
 
 
jemdoc.css
jemdoc.css
 
 

Repository files navigation

Awesome 3D-aware Image Synthesis – Papers, Codes and Datasets

Introduction

This homepage lists some representative papers/codes/datasets all about 3D-aware image synthesis. We aim to constantly update the latest relevant papers and help the community track this topic. Please feel free to join us and contribute to the project. If you have any questions, please feel free to contact Weihao Xia.

Survey paper

Datasets

Summary of popular 3D-aware image synthesis datasets.

Multi-view image collections

The images are rendered or collected according to different experimental settings, such as Synthetic-NeRF dataset, the DTU dataset, and the Tanks and Temples dataset for general purposes, the crowded Phototourism dataset for varying lighting conditions, the Blender Forward Facing (BLEFF) dataset to benchmark camera parameter estimation and novel view synthesis quality, and the San Francisco Alamo Square Dataset for large-scale scenes.

Examples of multi-view image datasets.

dataset published in # scene # samples per scene range (m × m) resolution keyword
DeepVoxels CVPR 2019 4 simple objects 479 / 1,000 \ 512 × 512 synthetic, 360 degree
NeRF Synthetics ECCV 2020 8 complex objects 100 / 200 \ 800 ×800 synthetic, 360 degree
NeRF Captured ECCV 2020 8 complex scenes 20-62 a few 1,008 × 756 real, forward-facing
DTU CVPR 2014 124 scenes 49 or 64 a few to thousand 1,600 × 1,200 often used in few-views
Tanks and Temples CVPR 2015 14 objects and scenes 4,395 - 21,871 dozen to thousand 8-megapixel real, large-scale
Phototourism IJCV 2021 6 landmarks 763-2,000 dozen to thousand 564-1,417 megapixel varying illumination
Alamo Square CVPR 2022 San Francisco 2,818,745 570 × 960 1,200 × 900 real, large-scale

Single-view image collections

Summary of popular single-view image datasets organized by their major categories and sorted by their popularity.

dataset year category # samples resolution keyword
FFHQ CVPR 2019 Human Face 70k 1024 × 1024 single simple-shape
AFHQ CVPR 2020 Cat, Dog, and Wildlife 15k 512 × 512 single simple-shape
CompCars CVPR 2015 Real Car 136K 256 × 256 single simple-shape
CARLA CoRL 2017 Synthetic Car 10k 128 × 128 single simple-shape
CLEVR CVPR 2017 Objects 100k 256 × 256 multiple, simple-shape
LSUN 2015 Bedroom 300K 256 × 256 single, simple-shape
CelebA ICCV 2015 Human Face 200k 178 × 218 single simple-shape
CelebA-HQ ICLR 2018 Human Face 30k 1024 × 1024 single, simple-shape
MetFaces NeurIPS 2020 Art Face 1336 1024 × 1024 single, simple-shape
M-Plants NeurIPS 2022 Variable-Shape 141,824 256 × 256 single, variable-shape
M-Food NeurIPS 2022 Variable-Shape 25,472 256 × 256 single, variable-shape

3D Control of 2D Generative Models

3D Control Latent Directions

  • On the "steerability" of generative adversarial networks.
    Ali Jahanian, Lucy Chai, Phillip Isola.
    ICLR 2020. [PDF] [Project]

  • Unsupervised Discovery of Interpretable Directions in the GAN Latent Space.
    Andrey Voynov, Artem Babenko.
    ICML 2020. [PDF] [Github]

  • Interpreting the Latent Space of GANs for Semantic Face Editing.
    Yujun Shen, Jinjin Gu, Xiaoou Tang, Bolei Zhou.
    CVPR 2020. [PDF] [Project] [Github]

  • GANSpace: Discovering Interpretable GAN Controls.
    Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, Sylvain Paris.
    NeurIPS 2020. [PDF] [Github]

  • SeFa: Closed-Form Factorization of Latent Semantics in GANs.
    Yujun Shen, Bolei Zhou.
    CVPR 2021. [PDF] [Github] [Project]

3D Parameters as Controls

  • StyleRig: Rigging StyleGAN for 3D Control over Portrait Images.
    Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhöfer, Christian Theobalt.
    CVPR 2020 (oral). [PDF] [Project]

  • DiscoFaceGAN: Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learning.
    Yu Deng, Jiaolong Yang, Dong Chen, Fang Wen, Xin Tong.
    CVPR 2020. [PDF] [Github]

  • PIE: Portrait Image Embedding for Semantic Control.
    A. Tewari, M. Elgharib, M. BR, F. Bernard, H-P. Seidel, P. P‌érez, M. Zollhöfer, C.Theobalt.
    SIGGRAPH Asia 2020. [PDF] [Project]

  • CONFIG: Controllable Neural Face Image Generation.
    Marek Kowalski, Stephan J. Garbin, Virginia Estellers, Tadas Baltrušaitis, Matthew Johnson, Jamie Shotton.
    ECCV 2020. [PDF] [Github]

  • GAN-Control: Explicitly Controllable GANs.
    Alon Shoshan, Nadav Bhonker, Igor Kviatkovsky, Gerard Medioni.
    ICCV 2021. [PDF] [Project]

  • 3D-FM GAN: Towards 3D-Controllable Face Manipulation.
    Yuchen Liu, Zhixin Shu, Yijun Li, Zhe Lin, Richard Zhang, and Sun-Yuan Kung.
    ECCV 2022. [PDF] [Project]

3D Prior Knowledge as Constraints

  • Generative Image Modeling using Style and Structure Adversarial Networks.
    Xiaolong Wang, Abhinav Gupta.
    ECCV 2016. [PDF]

  • 3D Shape Induction from 2D Views of Multiple Objects.
    Matheus Gadelha, Subhransu Maji, Rui Wang.
    3DV 2017. [PDF] [Project]

  • Visual Object Networks: Image Generation with Disentangled 3D Representation.
    Jun-Yan Zhu, Zhoutong Zhang, Chengkai Zhang, Jiajun Wu, Antonio Torralba, Joshua B. Tenenbaum, William T. Freeman.
    NeurIPS 2018. [PDF] [Project] [Github]

  • RGBD-GAN: Unsupervised 3D Representation Learning From Natural Image Datasets via RGBD Image Synthesis.
    Atsuhiro Noguchi, Tatsuya Harada.
    ICLR 2020. [PDF] [Github]

  • NGP: Towards a Neural Graphics Pipeline for Controllable Image Generation.
    Xuelin Chen, Daniel Cohen-Or, Baoquan Chen, Niloy J. Mitra.
    Eurographics 2021. [PDF] [Github]

  • Lifting 2D StyleGAN for 3D-Aware Face Generation.
    Yichun Shi, Divyansh Aggarwal, Anil K. Jain.
    CVPR 2021. [PDF]

  • 3D-Aware Indoor Scene Synthesis with Depth Priors.
    Zifan Shi, Yujun Shen, Jiapeng Zhu, Dit-Yan Yeung, Qifeng Chen.
    ECCV 2022 (oral). [PDF] [Project] [Github]

3D Novel View Synthesis from Multiple Views

Neural Scene Representation

  • DeepVoxels: Learning Persistent 3D Feature Embeddings.
    Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, Michael Zollhöfer.
    CVPR 2019 (Oral). [Project] [PDF] [Code]

  • Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations.
    Vincent Sitzmann, Michael Zollhöfer, Gordon Wetzstein.
    NeurIPS 2019 (Oral, Honorable Mention "Outstanding New Directions"). [PDF] [Project] [Github] [Dataset]

  • LLFF: Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines.
    Ben Mildenhall, Pratul Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, Abhishek Kar.
    SIGGRAPH 2019. [PDF] [Project] [Github]

  • Differentiable Volumetric Rendering (DVR): Learning Implicit 3D Representations without 3D Supervision.
    Michael Niemeyer, Lars Mescheder, Michael Oechsle, Andreas Geiger.
    CVPR 2020. [PDF] [Github]

  • NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis.
    Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng.
    ECCV 2020. [PDF] [Project] [Gtihub-Tensorflow] [krrish94-PyTorch] [yenchenlin-PyTorch]

  • Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields.
    Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, Pratul P. Srinivasan.
    ICCV 2021. [PDF] [Project]

  • Light Field Networks (LFNS): Neural Scene Representations with Single-Evaluation Rendering.
    Vincent Sitzmann, Semon Rezchikov, William T. Freeman, Joshua B. Tenenbaum, Fredo Durand.
    NeurIPS 2021. [PDF] [Project]

  • Scene Representation Transformer (SRT): Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations.
    Mehdi S. M. Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lucic, Daniel Duckworth, Alexey Dosovitskiy, Jakob Uszkoreit, Thomas Funkhouser, Andrea Tagliasacchi.
    CVPR 2022. [PDF]

Speed up

From Constrained Environmental Conditions to In-the-wild

Few images

  • GRF: Learning a General Radiance Field for 3D Representation and Rendering.
    Alex Trevithick, Bo Yang.
    ICCV 2021. [PDF]

  • pixelNeRF: Neural Radiance Fields from One or Few Images.
    Alex Yu, Vickie Ye, Matthew Tancik, Angjoo Kanazawa.
    CVPR 2021. [PDF] [Project]

  • IBRNet: Learning Multi-View Image-Based Rendering.
    Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan T. Barron, Ricardo Martin-Brualla, Noah Snavely, Thomas Funkhouser.
    CVPR 2021. [PDF] [Project]

  • MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo.
    Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, Hao Su.
    ICCV 2021. [PDF] [Project] [Github]

  • CodeNeRF: Disentangled Neural Radiance Fields for Object Categories.
    Wonbong Jang, Lourdes Agapito.
    ICCV 2021. [PDF] [Project] [Github]

  • NeRF-VAE: A Geometry Aware 3D Scene Generative Model.
    Adam R. Kosiorek, Heiko Strathmann, Daniel Zoran, Pol Moreno, Rosalia Schneider, Soňa Mokrá, Danilo J. Rezende.
    ICML 2021. [PDF]

Pose-free

Varying appearance

Large-scale scene

  • Grid-guided Neural Radiance Fields for Large Urban Scenes.
    Linning Xu, Yuanbo Xiangli, Sida Peng, Xingang Pan, Nanxuan Zhao, Christian Theobalt, Bo Dai, Dahua Lin.
    CVPR 2023. [PDF] [Project]

  • Shadow Neural Radiance Fields for Multi-view Satellite Photogrammetry.
    Dawa Derksen, Dario Izzo.
    CVPR 2021. [PDF]

  • Block-NeRF: Scalable Large Scene Neural View Synthesis.
    Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P. Srinivasan, Jonathan T. Barron, Henrik Kretzschmar.
    CVPR 2022. [PDF] [Project]

  • Urban Radiance Fields.
    Konstantinos Rematas, Andrew Liu, Pratul P. Srinivasan, Jonathan T. Barron, Andrea Tagliasacchi, Thomas Funkhouser, Vittorio Ferrari.
    CVPR 2022. [PDF] [Project]

  • Mega-NERF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs.
    Haithem Turki, Deva Ramanan, Mahadev Satyanarayanan.
    CVPR 2022. [PDF]

  • BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering.
    Yuanbo Xiangli, Linning Xu, Xingang Pan, Nanxuan Zhao, Anyi Rao, Christian Theobalt, Bo Dai, Dahua Lin.
    ECCV 2022. [PDF] [Project]

  • S3-NeRF: Neural Reflectance Field from Shading and Shadow under a Single Viewpoint.
    Wenqi Yang, Guanying Chen, Chaofeng Chen, Zhenfang Chen, Kwan-Yee K. Wong.
    NeurIPS 2022. [PDF] [Project]

Dynamic scene

3D Generative Models from Single Views

Unconditional 3D Generative Models

Mimic3D: Thriving 3D-Aware GANs via 3D-to-2D Imitation.
Xingyu Chen, Yu Deng, Baoyuan Wang.
arxiv 2022. [PDF] [Project]

  • GRAM-HD: 3D-Consistent Image Generation at High Resolution with Generative Radiance Manifolds.
    Jianfeng Xiang, Jiaolong Yang, Yu Deng, Xin Tong.
    arxiv 2022. [PDF] [Project]

  • CIPS-3D: A 3D-Aware Generator of GANs Based on Conditionally-Independent Pixel Synthesis.
    Peng Zhou, Lingxi Xie, Bingbing Ni, Qi Tian.
    arxiv 2021. [PDF] [Github]

  • EG3D: Efficient Geometry-aware 3D Generative Adversarial Networks.
    Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, Gordon Wetzstein.
    CVPR 2022. [PDF] [Project]

  • StylizedNeRF: Consistent 3D Scene Stylization as Stylized NeRF via 2D-3D Mutual Learning.
    Yi-Hua Huang, Yue He, Yu-Jie Yuan, Yu-Kun Lai, Lin Gao.
    CVPR 2022. [PDF]

  • Multi-View Consistent Generative Adversarial Networks for 3D-aware Image Synthesis.
    Xuanmeng Zhang, Zhedong Zheng, Daiheng Gao, Bang Zhang, Pan Pan, Yi Yang.
    CVPR 2022. [PDF] [Github]

  • Disentangled3D: Learning a 3D Generative Model with Disentangled Geometry and Appearance from Monocular Images.
    Ayush Tewari, Mallikarjun B R, Xingang Pan, Ohad Fried, Maneesh Agrawala, Christian Theobalt.
    CVPR 2022. [PDF] [Project]

  • GIRAFFE HD: A High-Resolution 3D-aware Generative Model.
    Yang Xue, Yuheng Li, Krishna Kumar Singh, Yong Jae Lee.
    CVPR 2022. [PDF]

  • StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation.
    Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, Ira Kemelmacher-Shlizerman.
    CVPR 2022. [PDF] [Project] [Github]

  • FENeRF: Face Editing in Neural Radiance Fields.
    Jingxiang Sun, Xuan Wang, Yong Zhang, Xiaoyu Li, Qi Zhang, Yebin Liu, Jue Wang.
    CVPR 2022. [PDF]

  • LOLNeRF: Learn from One Look.
    Daniel Rebain, Mark Matthews, Kwang Moo Yi, Dmitry Lagun, Andrea Tagliasacchi.
    CVPR 2022. [PDF] [Project]

  • GRAM: Generative Radiance Manifolds for 3D-Aware Image Generation.
    Yu Deng, Jiaolong Yang, Jianfeng Xiang, Xin Tong.
    CVPR 2022. [PDF] [Project] [Github]

  • VolumeGAN: 3D-aware Image Synthesis via Learning Structural and Textural Representations.
    Yinghao Xu, Sida Peng, Ceyuan Yang, Yujun Shen, Bolei Zhou.
    CVPR 2022. [PDF] [Project] [Github]

  • MOST-GAN: 3D Morphable StyleGAN for Disentangled Face Image Manipulation.
    Safa C. Medin, Bernhard Egger, Anoop Cherian, Ye Wang, Joshua B. Tenenbaum, Xiaoming Liu, Tim K. Marks.
    AAAI 2022. [PDF]

  • Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks.
    Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Woo Ha, Jinwoo Shin.
    ICLR 2022. [PDF] [Project] [Github]

  • StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis.
    Jiatao Gu, Lingjie Liu, Peng Wang, Christian Theobalt.
    ICLR 2022. [PDF] [Project]

  • Injecting 3D Perception of Controllable NeRF-GAN into StyleGAN for Editable Portrait Image Synthesis.
    Jeong-gi Kwak, Yuanming Li, Dongsik Yoon, Donghyeon Kim, David Han, Hanseok Ko.
    ECCV 2022. [PDF] [Project] [Github]

  • Generative Multiplane Images: Making a 2D GAN 3D-Aware.
    Xiaoming Zhao, Fangchang Ma, David Güera, Zhile Ren, Alexander G. Schwing, Alex Colburn.
    ECCV 2022. [PDF] [Project] [Github]

  • 3D-FM GAN: Towards 3D-Controllable Face Manipulation.
    Yuchen Liu, Zhixin Shu, Yijun Li, Zhe Lin, Richard Zhang, and Sun-Yuan Kung.
    ECCV 2022. [PDF] [Project]

  • VoxGRAF: Fast 3D-Aware Image Synthesis with Sparse Voxel Grids.
    Katja Schwarz, Axel Sauer, Michael Niemeyer, Yiyi Liao, Andreas Geiger.
    NeurIPS 2022. [PDF] [Github]

  • GeoD: Improving 3D-aware Image Synthesis with A Geometry-aware Discriminator.
    Zifan Shi, Yinghao Xu, Yujun Shen, Deli Zhao, Qifeng Chen, Dit-Yan Yeung.
    NeurIPS 2022. [PDF] [Project]

  • EpiGRAF: Rethinking training of 3D GANs.
    Ivan Skorokhodov, Sergey Tulyakov, Yiqun Wang, Peter Wonka.
    NeurIPS 2022. [PDF] [Project] [Github]

  • VoxGRAF: Fast 3D-Aware Image Synthesis with Sparse Voxel Grids.
    Schwarz, Katja, Sauer, Axel, Niemeyer, Michael, Liao, Yiyi, and Geiger, Andreas.
    NeurIPS 2022. [PDF] [Project]

Conditional 3D Generative Models

3D-Aware Generative Model on ImageNet

  • 3D Generation on ImageNet.
    Ivan Skorokhodov, Aliaksandr Siarohin, Yinghao Xu, Jian Ren, Hsin-Ying Lee, Peter Wonka, Sergey Tulyakov.
    ICLR 2023 (Oral). [PDF] [Project] [Github]

  • VQ3D: Learning a 3D-Aware Generative Model on ImageNet.
    Kyle Sargent, Jing Yu Koh, Han Zhang, Huiwen Chang, Charles Herrmann, Pratul Srinivasan, Jiajun Wu, Deqing Sun.
    arxiv 2022. [PDF] [Project]

3D-aware Diffusion Models

  • 3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models.
    Biao Zhang, Jiapeng Tang, Matthias Niessner, Peter Wonka.
    SIGGRAPH 2023. [PDF] [Project] [Github]

  • NeuralField-LDM: Scene Generation with Hierarchical Latent Diffusion Models.
    Seung Wook Kim, Bradley Brown, Kangxue Yin, Karsten Kreis, Katja Schwarz, Daiqing Li, Robin Rombach, Antonio Torralba, Sanja Fidler.
    CVPR 2023. [PDF]

  • Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion.
    Tengfei Wang, Bo Zhang, Ting Zhang, Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen, Dong Chen, Fang Wen, Qifeng Chen, Baining Guo.
    CVPR 2023. [PDF] [Project]

  • DiffRF: Rendering-guided 3D Radiance Field Diffusion.
    Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulò, Peter Kontschieder, Matthias Nießner.
    CVPR 2023 (Highlight). [PDF] [Project]

  • RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation.
    Titas Anciukevičius, Zexiang Xu, Matthew Fisher, Paul Henderson, Hakan Bilen, Niloy J. Mitra, Paul Guerrero.
    CVPR 2023. [PDF] [Project]

  • SparseFusion: Distilling View-conditioned Diffusion for 3D Reconstruction.
    Zhizhuo Zhou, Shubham Tulsiani.
    CVPR 2023. [PDF] [Project] [Github]

  • HoloDiffusion: Training a 3D Diffusion Model using 2D Images.
    Animesh Karnewar, Andrea Vedaldi, David Novotny, Niloy Mitra.
    CVPR 2023. [PDF] [Project]

  • 3DiM: Novel View Synthesis with Diffusion Models.
    Daniel Watson, William Chan, Ricardo Martin-Brualla, Jonathan Ho, Andrea Tagliasacchi, Mohammad Norouzi.
    ICLR 2023. [PDF] [Project]

Learning a Diffusion Prior for NeRFs.
Guandao Yang, Abhijit Kundu, Leonidas J. Guibas, Jonathan T. Barron, Ben Poole.
arXiv 2023. [PDF]

  • Generative Novel View Synthesis with 3D-Aware Diffusion Models.
    Eric R. Chan, Koki Nagano, Matthew A. Chan, Alexander W. Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, Gordon Wetzstein.
    arXiv 2023. [PDF] [Project] [Github]

  • 3D-aware Image Generation using 2D Diffusion Models.
    Jianfeng Xiang, Jiaolong Yang, Binbin Huang, Xin Tong.
    arXiv 2023. [PDF] [Project] [Github]

  • GAUDI: A Neural Architect for Immersive 3D Scene Generation.
    Miguel Angel Bautista, Pengsheng Guo, Samira Abnar, Walter Talbott, Alexander Toshev, Zhuoyuan Chen, Laurent Dinh, Shuangfei Zhai, Hanlin Goh, Daniel Ulbricht, Afshin Dehghan, Josh Susskind.
    arXiv 2022. [PDF] [Project]

  • LatentSwap3D: Semantic Edits on 3D Image GANs.
    Enis Simsar, Alessio Tonioni, Evin Pınar Örnek, Federico Tombari.
    arXiv 2022. [PDF]

  • 3D-LDM: Neural Implicit 3D Shape Generation with Latent Diffusion Models.
    Gimin Nam, Mariem Khlifi, Andrew Rodriguez, Alberto Tono, Linqi Zhou, Paul Guerrero.
    arXiv 2022. [PDF]

3D-aware Video Synthesis


The following papers are not directly related to 3D-aware image synthesis. But it would be beneficial to pay attention to those works. For example, the inverse rendering papers are not classified as 3D-aware image synthesis methods in this survey as they are not deliberately designed for this purpose. But with the inferred underlying intrinsic components of a scene, photorealistic images can be rendered. 3D shape reconstruction methods model geometry only with no appearance information, meaning them not able to render images with photorealistic textures. But these representations can also be used for the 3D-aware image synthesis task. They can be introduced as the geometric representation along with a textural representation (e.g., Texture Field) for 3D image synthesis.

3D Representations

  • K-Planes: Explicit Radiance Fields in Space, Time, and Appearance.
    Sara Fridovich-Keil, Giacomo Meanti, Frederik Warburg, Benjamin Recht, Angjoo Kanazawa.
    arxiv 2022. [PDF] [Project] [Github]

  • HexPlane: A Fast Representation for Dynamic Scenes.
    Ang Cao, Justin Johnson.
    arxiv 2022. [PDF] [Project] [Github]

  • Neural Volumes: Learning Dynamic Renderable Volumes from Images.
    Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, Yaser Sheikh.
    TOG 2019. [PDF] [Github]

  • DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation.
    eong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, Steven Lovegrove.
    CVPR 2019. [PDF] [Github]

  • Occupancy Networks: Learning 3D Reconstruction in Function Space.
    Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, Andreas Geiger.
    CVPR 2019. [PDF] [Project] [Github]

  • Geometry-Consistent Neural Shape Representation with Implicit Displacement Fields.
    Wang Yifan, Lukas Rahmann, Olga Sorkine-Hornung.
    ICLR 2022. [PDF] [Project] [Github]

  • GIFS: Neural Implicit Function for General Shape Representation.
    Jianglong Ye, Yuntao Chen, Naiyan Wang, Xiaolong Wang.
    CVPR 2022. [PDF] [Project]

Neural Inverse Rendering (Neural De-rendering)

The inverse rendering is to infer underlying intrinsic components of a scene from rendered 2D images. These properties include shape (surface, depth, normal), material (albedo, reflectivity, shininess), and lighting (direction, intensity), which can be further used to render photorealistic images.

Neural Rerendering

  • Neural Rerendering in the Wild.
    Moustafa Meshry, Dan B Goldman, Sameh Khamis, Hugues Hoppe, Rohit Pandey, Noah Snavely, Ricardo Martin-Brualla.
    CVPR 2019. [PDF]

  • Revealing Scenes by Inverting Structure from Motion Reconstructions.
    Francesco Pittaluga, Sanjeev J. Koppal, Sing Bing Kang, Sudipta N. Sinha.
    CVPR 2019. [PDF]

  • Neural Re-Rendering of Humans from a Single Image.
    Kripasindhu Sarkar, Dushyant Mehta, Weipeng Xu, Vladislav Golyanik, Christian Theobalt.
    ECCV 2020. [PDF]

  • Neural Lumigraph Rendering.
    Petr Kellnhofer, Lars Jebe, Andrew Jones, Ryan Spicer, Kari Pulli, Gordon Wetzstein.
    CVPR 2021. [PDF] [Project] [Data]

  • Hybrid Neural Fusion for Full-frame Video Stabilization.
    Yu-Lun Liu, Wei-Sheng Lai, Ming-Hsuan Yang, Yung-Yu Chuang, Jia-Bin Huang.
    ICCV 2021. [PDF] [Github]

visitors
Visitors since Nov. 2022
Page generated 2022-10-25, by jemdoc.

About

[CSUR 2023] A Survey on Deep Generative 3D-aware Image Synthesis

weihaox.github.io/Gen3D

Topics

computer-vision computer-graphics 3d-aware-image-synthesis

Resources

Readme

License

MIT license
Activity

Stars

160 stars

Watchers

12 watching

Forks

11 forks
Report repository

Contributors 2

  •  
  •  

Languages