Multi-View Azimuth Stereo via Tangent Space Consistency

Xu Cao · Hiroaki Santo · Fumio Okura · Yasuyuki Matsushita

CVPR 2023

3D reconstruction only using surface azimuth maps.

Quick Start

Our code was tested on Ubuntu18.04 with Python 3.9, PyTorch 1.12, and Cuda 11.3. Follow these steps to reproduce our environment and results.

git clone https://github.com/xucao-42/mvas.git
cd mvas

conda env create -f environment.yml 
conda activate mvas

mkdir ./data
cd ./code

Train on DiLiGenT-MV data (280 MB)

Download data from Google Drive and extract it under data folder. Run

python exp_runner.py --config configs/diligent_mv.conf

Train on SymPS data (5.7 GB)

Download data from Google Drive and extract it under data folder.

Run

python exp_runner.py --config configs/symps_gargoyle.conf
# or you can try symps_house.conf and symps_moai.conf

Train on PANDORA data (2.7 GB)

Download data from Google Drive and extract it under data folder. Run

python exp_runner.py --config configs/pandora.conf

Results will be saved in results/$obj_name/$exp_time.

Data structure

DiLiGenT-MV

• input_azimuth_maps: These are 16-bit RGBA images where the alpha channel represents the object mask and the RGB channels are identical. Each RGB channel can be converted to azimuth angles within [0, pi] by multiplying it by pi/65535. The azimuth angle is measured clockwise from the x-axis, which points to the right, and is consistent with OpenCV convention (x-axis to the right, y-axis downward). The azimuth maps do not need to be stored in the range [0, 2π], as our method is π-invariant.
• vis_azimuth_maps: These are for visualization purposes only and are not used during training.
• normal_maps: These are the normal maps used to create the input azimuth maps. We applied SDPS-Net independently in each view to obtain the normal maps.
• params.json: This file is from PS-NeRF preprocessing and contains the camera intrinsic parameters, as the normal and azimuth maps are cropped to 400 x 400.
• Calib_Results.mat: This file is from the original DiLiGenT-MV dataset and provides the camera extrinsic information.

SymPS

• input_azimuth_maps: These are 16-bit gray-scale images. The pixel values can be converted to azimuth angles within [0, pi] by multiplying them by pi/65535. The azimuth angle is measured clockwise from the x-axis, which points to the right, and is consistent with OpenCV convention (x-axis to the right, y-axis downward). The azimuth maps do not need to be stored in the range [0, 2π], as our method is π-invariant.
• mask: Binary masks indicating the object silhouettes.
• sparse: This folder contains Colmap-calibrated camera intrinsic and extrinsic information.
• images_SfM: These are images used for structure from motion in Colmap.

PANDORA

• input_azimuth_maps: These are 16-bit RGBA images where the alpha channel represents the object mask and the RGB channels are identical. Each RGB channel can be converted to azimuth angles within [0, pi] by multiplying it by pi/65535. The azimuth angle is measured clockwise from the x-axis, which points to the right, and is consistent with OpenCV convention (x-axis to the right, y-axis downward). The azimuth maps do not need to be stored in the range [0, 2π], as our method is π-invariant. Note that since PANDORA is a polarization image dataset, the azimuth maps have half-pi ambiguity.
• vis_azimuth_maps: These are for visualization purposes only and are not used during training.
• sparse: This folder is from PANDORA and contains Colmap-calibrated camera intrinsic and extrinsic information.
• images: These are for reference purpose and not used in training.

Acknowledgement

Our implementation is built upon IDR , and benefits from PS-NeRF and PANDORA.

Bibtex

@inproceedings{mvas2023cao,
title = {Multi-View Azimuth Stereo via Tangent Space Consistency},
author = {Cao, Xu and Santo, Hiroaki and Okura, Fumio and Matsushita, Yasuyuki},
year = {2023},
booktitle = CVPR,
}