Image-to-3D Models Survey

Zero-Shot Novel View Synthesis

• 3DiM (ICLR 2023)
  : Diffusion model for 3D novel view synthesis
  $\rarr$ Zero-shot generalization capability가 떨어짐

• NeRDi (CVPR 2023), RealFusion (CVPR 2023), NeuralLift-360 (CVPR 2023)
  : Language-guided prior와 textual inversion을 통한 image-to-3D generation

• Zero-1-to-3
  : Synthetic dataset통한 zero-shot conditioned viewpoint control 학습

Single-View Object Reconstruction

• 'Multi-View Mesh Reconstruction with Neural Deferred Shading' (CVPR 2022)
• DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction (NeurIPS 2019)
• 'Learning a Predictable and Generative Vector Representation for Objects' (ECCV 2016)
• MarrNet: 3D Shape Reconstruction via 2.5D Sketches (NIPS 2017)
• A Point Set Generation Network for 3D Object Reconstruction from a Single Image (CVPR 2022)
• 'Occupancy Networks: Learning 3D Reconstruction in Function Space' (CVPR 2019)
  : Meshes, voxels, point clouds 등의 3D primitives에 대한 직접적인 학습 방법론 제시
  $\rarr$ Generalization capability가 떨어짐

• PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization (ICCV 2019)
• pixelNeRF: Neural Radiance Fields from One or Few Images (CVPR 2021)
• IBRNet: Learning Multi-View Image-Based Rendering (CVPR 2021)
• Single-View View Synthesis with Multiplane Images (CVPR 2020)
• Revealing Occlusions with 4D Neural Fields (CVPR 2022)
  : Local image features로부터 scene reconstruction 진행.
  $\rarr$ Cross-domain generalization cabability를 보이나 close-by view reconstruction으로 제한됨.

• Multiview Compressive Coding for 3D Reconstruction (CVPR 2022)
  : RGB-D view로부터 3D reconstruction 진행

• Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis (ICCV 2021)

일반적으로, 3D dataset이 크게 부족하기에 2D dataset으로 학습된 모델의 prior를 활용하여 view synthesis 혹은 3D reconstruction을 진행하는 추세. 실질적으로 사용 가능한 성능의 모델로 한정지어 보면,

Zero-1-to-3

[Paper Summary]

[Official Live Demo]

1. Zero-1-to-3 모델로 제한된 angle view에 대해 sampling한 뒤 Instant-NGP fitting 후 rendering을 해볼까.
2. 마찬가지로 Zero-1-to-3 모델로 sampling한 뒤 FILM 모델로 frame interpolation을 진행할까.

RealFusion

[Paper Summary]

3. Full 3D-reconstruction을 수행할 것이 아니기에, reconstruction objective만 부여하여 제한된 angle view에 대해서만 reconstruction을 진행한다면 이미지가 얼마나 깨지는가?