Unifying Correspondence, Pose and NeRF for Generalized Pose-Free Novel View Synthesis from Stereo Pairs CVPR-2024 (Highlight)

This is our official implementation of CoPoNeRF CVPR-2024 (Highlight)!

[arXiv] [Project]

by Sunghwan Hong, Jaewoo Jung, Heeseong Shin, Jiaolong Yang, Seungryong Kim, Chong Luo,

Introduction

We delve into the task of generalized pose-free novel view synthesis from stereo pairs, a challenging and pioneering task in 3D vision.

For further details and visualization results, please check out our paper and our project page.

❗️Update: This repository includes refactored codes:

• We retrained the network using this code base.
• If you want the codes and weights for the CVPR version, please email Sunghwan!

Installation

Requirements

• Linux or macOS with Python ≥ 3.8
• PyTorch ≥ 1.13 is recommended and torchvision that matches the PyTorch installation. Install them together at pytorch.org to make sure of this.

An example of installation is shown below:

git clone https://github.com/KU-CVLAB/CoPoNeRF.git
cd CoPoNeRF
conda create -n CoPoNeRF python=3.8
conda activate CoPoNeRF
conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.7 -c pytorch -c nvidia
pip install -r assets/requirements.txt

Data Preparation

Please follow dataset preperation.

Training

python train.py --experiment_name [EXPERIMENT_NAME] 

If you want to specify batch size or number of gpus, include them as arguments. Also, you can freely add the losses, e.g., cycle loss. To train on ACID, you can do it by simply changing the directories and the dataset in train.py.

Evaluation

python test.py --checkpoint_path [CHECKPOINT_PATH]

Pretrained Models

We provide pretrained weights here. These models were trained with 4 A6000.

Acknowledgement

We would like to acknowledge the contributions of public projects, such as Du et al. and UFC, whose code has been utilized in this repository.

Citing CoPoNeRF:

@article{hong2023unifying,
  title={Unifying Correspondence, Pose and NeRF for Pose-Free Novel View Synthesis from Stereo Pairs},
  author={Hong, Sunghwan and Jung, Jaewoo and Shin, Heeseong and Yang, Jiaolong and Kim, Seungryong and Luo, Chong},
  journal={arXiv preprint arXiv:2312.07246},
  year={2023}
}