Skip to content

Latest commit

 

History

History
641 lines (544 loc) · 31.3 KB

benchmark.adoc

File metadata and controls

641 lines (544 loc) · 31.3 KB

kapture-localization / Image Retrieval Benchmark from 3DV2020 paper

Table of Contents

Introduction

In this tutorial, we present how to run a majority of the experiments presented in our 3DV2020 paper from scratch. Here, we assume that kapture-localization is already installed. Please refer to the installation tutorial for more details. The fastest way would be using the public docker container.

If this benchmark is used in research, please cite this paper:

@inproceedings{benchmarking_ir3DV2020,
      title={Benchmarking Image Retrieval for Visual Localization},
      author={No{\'e} Pion, Martin Humenberger, Gabriela Csurka, Yohann Cabon, Torsten Sattler},
      year={2020},
      booktitle={International Conference on 3D Vision}
}

@article{humenberger2022investigating,
  title={Investigating the Role of Image Retrieval for Visual Localization},
  author={Humenberger, Martin and Cabon, Yohann and Pion, No{\'e} and Weinzaepfel, Philippe and Lee, Donghwan and Gu{\'e}rin, Nicolas and Sattler, Torsten and Csurka, Gabriela},
  journal={International Journal of Computer Vision},
  year={2022},
  publisher={Springer}
}
Warning
 The image-retrieval benchmark script now runs kapture_pycolmap_localsfm, whereas we used the script kapture_colmap_localize_localsfm in the papers and they are not equivalent. We decided to change it because kapture_colmap_localize_localsfm was too slow to be practical (and it created too many matches files). localsfm results are therefore not comparable. We provide updated numbers as well as the local features we used to produce them (feather2d2, 20k, a slightly altered version of fast r2d2 with dsize=32) here.

Datasets

In order to download the 3 datasets used in the paper, we recommend using the kapture dataset downloader. Note that the dataset downloader does not only provide the raw datasets, it also provides some precomputed data such as the global features used in the paper. In this tutorial, we will store all datasets in a folder called kapture_datasets. This folder will not only contain the raw datasets in kapture format, but also all processed data (features, matches, etc.) and benchmark results.

Aachen Day-Night v1.1

mkdir kapture_datasets
cd kapture_datasets
kapture_download_dataset.py update
kapture_download_dataset.py list # this will list all available datasets

# this will download all available data for Aachen Day-Night v1.1
kapture_download_dataset.py install "Aachen*"
cd Aachen-Day-Night-v1.1
# this will merge query_day and query_night into query_all
kapture_merge.py -v info \
                 -i query_day query_night \
                 -o query_all \
                 --image_transfer link_relative
# this will merge mapping and query in one kapture folder
kapture_merge.py -v info \
                 -i mapping query_all \
                 -o map_plus_query \
                 --image_transfer link_relative

If all Aachen Day-Night v1.1 data was downloaded, the kapture_datasets folder should look like this (not all subfolders are shown here):

kapture_datasets
└─ Aachen-Day-Night-v1.1
    ├─ global_features
    │  ├─ AP-GeM-LM18
    │  ├─ DELG
    │  ├─ densevlad_multi
    │  └─ netvlad_vd16pitts
    │     └─ global_features
    ├─ local_features
    │  └─ feather2d2_dim32_20k
    │     ├─ keypoints
    │     └─ descriptors
    ├─ colmap-sfm
    ├─ map_plus_query
    ├─ query_all
    ├─ readme_kapture.txt
    ├─ LICENSE.txt
    ├─ README_Aachen-Day-Night.md
    ├─ README_Aachen-Day-Night_v1_1.md
    ├─ query_day
    ├─ query_night
    └─ mapping

Baidu-Mall

The images of this dataset are not provided in the dataset downloader, thus, they need to be first downloaded (and extracted) from the original source and stored in the kapture_datasets folder.

Convert the original image folder structure to the one used in the kapture version of the dataset:

cd kapture_datasets/IDL_dataset_cvpr17_3852
for cam in cdm lp mi qf sx wh wj wl xyf; do \
  mkdir -p query_images_undistort/${cam}; \
  mv query_images_undistort/${cam}*.jpg query_images_undistort/${cam}/; \
done

mkdir -p training_images_undistort/00
mkdir -p training_images_undistort/01
mv training_images_undistort/nikon5300a*.jpg training_images_undistort/00/
mv training_images_undistort/nikon5300b*.jpg training_images_undistort/01/

Download the Baidu-Mall kapture dataset (no images, just sensor data) using the downloader:

kapture_download_dataset.py install "baidu*"

Move the images to the kapture folder:

cd kapture_datasets
mkdir -p baidu-mall/mapping_wo_images/sensors/records_data
mv IDL_dataset_cvpr17_3852/training_images_undistort/* baidu-mall/mapping_wo_images/sensors/records_data/
mv baidu-mall/mapping_wo_images baidu-mall/mapping

mkdir -p baidu-mall/query_wo_images/sensors/records_data
mv IDL_dataset_cvpr17_3852/query_images_undistort/* baidu-mall/query_wo_images/sensors/records_data/
mv baidu-mall/query_wo_images baidu-mall/query

cd kapture_datasets/baidu-mall
# this will merge mapping and query in one kapture folder
kapture_merge.py -v info \
                 -i mapping query \
                 -o map_plus_query \
                 --image_transfer link_relative

The kapture_datasets folder should look like this now (not all subfolders are shown here):

kapture_datasets
├─ Aachen-Day-Night-v1.1
└─ baidu-mall
    ├─ global_features
    │  ├─ AP-GeM-LM18
    │  ├─ DELG
    │  ├─ densevlad_multi
    │  └─ netvlad_vd16pitts
    │     └─ global_features
    ├─ local_features
    │  └─ feather2d2_dim32_20k
    │     ├─ keypoints
    │     ├─ descriptors
    │     ├─ NN_no_gv
    │     └─ NN_colmap_gv
    ├─ colmap-sfm
    ├─ map_plus_query
    ├─ query
    ├─ readme_kapture.txt
    ├─ readme.txt
    └─ mapping

Note that we also provide local feature matches for this dataset. This will enable you to even quicker run your own experiments.

RobotCar Seasons

We used the first version of the RobotCar Seasons (v1) dataset for the 3DV2020 paper. Our downloader, however, provides RobotCar Seasons v2, incl. images, separated in its individual locations (same as the original dataset). V1 is provided as all-locations-in-one kapture, but without images. Thus, we have to first download all available RobotCar data using the downloader and then convert/merge the data.

cd kapture_datasets
# this will download all available data for RobotCar Seasons v1 and v2
kapture_download_dataset.py install "RobotCar_Seasons*"

cd RobotCar_Seasons-v2
# this will merge all v2 kapture folders
mapping=`find . -maxdepth 2 -type d -name mapping  -printf '%P '`
query=`find . -maxdepth 2 -type d -name query  -printf '%P '`
kapture_merge.py -v info \
                 -i $mapping $query \
                 -o all \
                 --image_transfer link_relative
cd ../RobotCar_Seasons-v1
mv mapping_wo_images mapping
cd mapping/sensors
ln -s ../../../RobotCar_Seasons-v2/all/sensors/records_data records_data
cd ../..
mv query_wo_images query
cd query/sensors
ln -s ../../../RobotCar_Seasons-v2/all/sensors/records_data records_data
cd ../..

# this will merge mapping and query in one kapture folder
kapture_merge.py -v info \
                 -i mapping query \
                 -o map_plus_query \
                 --image_transfer link_relative

The kapture_datasets folder should look like this (not all subfolders are shown here):

kapture_datasets
├─ Aachen-Day-Night-v1.1
├─ baidu-mall
├─ RobotCar_Seasons-v2
└─ RobotCar_Seasons-v1
    ├─ global_features
    │  ├─ AP-GeM-LM18
    │  ├─ DELG
    │  ├─ densevlad_multi
    │  └─ netvlad_vd16pitts
    │     └─ global_features
    ├─ local_features
    │  └─ feather2d2_dim32_20k
    │     ├─ keypoints
    │     └─ descriptors
    ├─ colmap-sfm
    ├─ map_plus_query
    ├─ query
    ├─ readme_kapture.txt
    ├─ LICENSE.txt
    ├─ README_RobotCar-Seasons.md
    └─ mapping

Gangnam Station B2

cd kapture_datasets/

kapture_download_dataset.py install  "GangnamStation_B2*"
# read license terms and type y [enter] to agree

cd GangnamStation/B2/release

kapture_merge.py -v info \
                 -i test validation  \
                 -o query_all \
                 --image_transfer link_relative
kapture_merge.py -v info \
                 -i mapping query_all \
                 -o map_plus_query \
                 --image_transfer link_relative

cd ../../../..

Finally, the kapture_datasets folder should look like this (not all subfolders are shown here):

kapture_datasets
├─ Aachen-Day-Night-v1.1
├─ baidu-mall
├─ RobotCar_Seasons-v2
├─ RobotCar_Seasons-v1
└─ GangnamStation/B2/release
    ├─ global_features
    │  ├─ AP-GeM-LM18
    │  ├─ r101delg_gldv2clean
    │  ├─ densevlad_multi
    │  └─ openibl_vgg16_netvlad
    │     └─ global_features
    ├─ local_features
    │  └─ feather2d2_dim32_20k
    │     ├─ keypoints
    │     └─ descriptors
    ├─ colmap-sfm
    ├─ map_plus_query
    ├─ test
    ├─ validation
    ├─ query_all
    ├─ mapping
    └─ mapping_lidar_only

Image Features

Global features

As can be seen above, we provide some global features in our dataset downloader and up to this point, they should already be downloaded.

Pairsfile

For HOW and FIRe, we provide precomputed pairs for all the datasets for use with kapture_pipeline_image_retrieval_benchmark_from_pairsfile.py.

# Aachen Day-Night v1.1
mkdir -p Aachen-Day-Night-v1.1/pairsfile/query
wget -O Aachen-Day-Night-v1.1/pairsfile/query/fire_top50.txt http://download.europe.naverlabs.com/kapture/Aachen_Day_Night_1_1_fire_top50_query_pairs.txt
# sha256sum available at http://download.europe.naverlabs.com/kapture/Aachen_Day_Night_1_1_fire_top50_query_pairs.txt.sha256sum
wget -O Aachen-Day-Night-v1.1/pairsfile/query/how_top50.txt http://download.europe.naverlabs.com/kapture/Aachen_Day_Night_1_1_how_top50_query_pairs.txt
# sha256sum available at http://download.europe.naverlabs.com/kapture/Aachen_Day_Night_1_1_how_top50_query_pairs.txt.sha256sum

# Baidu Mall
mkdir -p baidu-mall/pairsfile/query
wget -O baidu-mall/pairsfile/query/fire_top50.txt http://download.europe.naverlabs.com/kapture/baidu_mall_fire_top50_query_pairs.txt
# sha256sum available at http://download.europe.naverlabs.com/kapture/baidu_mall_fire_top50_query_pairs.txt.sha256sum
wget -O baidu-mall/pairsfile/query/how_top50.txt http://download.europe.naverlabs.com/kapture/baidu_mall_how_top50_query_pairs.txt
# sha256sum available at http://download.europe.naverlabs.com/kapture/baidu_mall_how_top50_query_pairs.txt.sha256sum

# RobotCar_Seasons-v1
mkdir -p RobotCar_Seasons-v1/pairsfile/query
wget -O RobotCar_Seasons-v1/pairsfile/query/fire_top50.txt http://download.europe.naverlabs.com/kapture/RobotCar_Seasons-v1_fire_top50_query_pairs.txt
# sha256sum available at http://download.europe.naverlabs.com/kapture/RobotCar_Seasons-v1_fire_top50_query_pairs.txt.sha256sum
wget -O RobotCar_Seasons-v1/pairsfile/query/how_top50.txt http://download.europe.naverlabs.com/kapture/RobotCar_Seasons-v1_how_top50_query_pairs.txt
# sha256sum available at http://download.europe.naverlabs.com/kapture/RobotCar_Seasons-v1_how_top50_query_pairs.txt.sha256sum

# Gangnam Station B2
mkdir -p GangnamStation/B2/release/pairsfile/query
wget -O GangnamStation/B2/release/pairsfile/query/fire_top50.txt http://download.europe.naverlabs.com/kapture/GangnamStation_B2_fire_top50_query_pairs.txt
# sha256sum available at http://download.europe.naverlabs.com/kapture/GangnamStation_B2_fire_top50_query_pairs.txt.sha256sum
wget -O GangnamStation/B2/release/pairsfile/query/how_top50.txt http://download.europe.naverlabs.com/kapture/GangnamStation_B2_how_top50_query_pairstxt
# sha256sum available at http://download.europe.naverlabs.com/kapture/GangnamStation_B2_how_top50_query_pairs.txt.sha256sum

# slicing
for DATASET in Aachen-Day-Night-v1.1 baidu-mall RobotCar_Seasons-v1 GangnamStation/B2/release; do
   for METHOD in how fire; do
      for TOPK in 20 10 5 4 3 2 1; do
         kapture_slice_pairsfile.py -v info \
                                    -i ${DATASET}/pairsfile/query/${METHOD}_top50.txt \
                                    -o ${DATASET}/pairsfile/query/${METHOD}_top${TOPK}.txt \
                                    --topk ${TOPK}
      done
   done
done

Local features

Any local features in kapture format can be used. For this tutorial, we suggest to use our pre-extracted feather2d2 lightweight features. If you followed the instructions above to download and install the datasets, then you should have them in the local_features subfolders.

For baidu-mall, we also provide local feature matches (feather2d2). If you test your own global features or pair files, and some feature matches are missing, the pipeline will automatically compute the missing matches and skip the existing ones.

To reproduce the numbers that we reported in the publications, you would have to use R2D2 (NeurIPS paper) following these instructions:

cd kapture_datasets
git clone https://github.com/naver/r2d2.git
# extract 20k R2D2 features for:
# Aachen Day-Night v1.1
python r2d2/extract_kapture.py --model r2d2/models/r2d2_WASF_N8_big.pt \
                               --kapture-root Aachen-Day-Night-v1.1/map_plus_query \
                               --top-k 20000 \
                               --max-size 9999 \
                               --keypoints-type r2d2_WASF-N8_20k \
                               --descriptors-type r2d2_WASF-N8_20k

# Baidu Mall
python r2d2/extract_kapture.py --model r2d2/models/r2d2_WASF_N8_big.pt \
                               --kapture-root baidu-mall/map_plus_query \
                               --top-k 20000 \
                               --max-size 9999 \
                               --keypoints-type r2d2_WASF-N8_20k \
                               --descriptors-type r2d2_WASF-N8_20k

# RobotCar Seasons
python r2d2/extract_kapture.py --model r2d2/models/r2d2_WASF_N8_big.pt \
                               --kapture-root RobotCar_Seasons-v1/map_plus_query \
                               --top-k 20000 \
                               --max-size 9999 \
                               --keypoints-type r2d2_WASF-N8_20k \
                               --descriptors-type r2d2_WASF-N8_20k

# Gangnam Station B2
python r2d2/extract_kapture.py --model r2d2/models/r2d2_WASF_N8_big.pt \
                               --kapture-root GangnamStation/B2/release/map_plus_query \
                               --top-k 20000 \
                               --max-size 9999 \
                               --keypoints-type r2d2_WASF-N8_20k \
                               --descriptors-type r2d2_WASF-N8_20k

The extracted features can either stay were they are (in folder map_plus_query/reconstruction) or, better, moved to a dedicated location for easy reuse:

mkdir -p Aachen-Day-Night-v1.1/local_features/r2d2_WASF-N8_20k
mv Aachen-Day-Night-v1.1/map_plus_query/reconstruction/keypoints/r2d2_WASF-N8_20k Aachen-Day-Night-v1.1/local_features/r2d2_WASF-N8_20k/keypoints
mv Aachen-Day-Night-v1.1/map_plus_query/reconstruction/descriptors/r2d2_WASF-N8_20k Aachen-Day-Night-v1.1/local_features/r2d2_WASF-N8_20k/descriptors

mkdir -p baidu-mall/local_features/r2d2_WASF-N8_20k
mv baidu-mall/map_plus_query/reconstruction/keypoints/r2d2_WASF-N8_20k baidu-mall/local_features/r2d2_WASF-N8_20k/keypoints
mv baidu-mall/map_plus_query/reconstruction/descriptors/r2d2_WASF-N8_20k baidu-mall/local_features/r2d2_WASF-N8_20k/descriptors

mkdir -p RobotCar_Seasons-v1/local_features/r2d2_WASF-N8_20k
mv RobotCar_Seasons-v1/map_plus_query/reconstruction/keypoints/r2d2_WASF-N8_20k RobotCar_Seasons-v1/local_features/r2d2_WASF-N8_20k/keypoints
mv RobotCar_Seasons-v1/map_plus_query/reconstruction/descriptors/r2d2_WASF-N8_20k RobotCar_Seasons-v1/local_features/r2d2_WASF-N8_20k/descriptors

In the paper, we also used two other local feature types:

  • SIFT: A simple way of using SIFT (as an alternative to R2D2) would be to extract the features using COLMAP and then to import the COLMAP database to kapture using this script.

  • D2-Net: Instructions to extract D2-Net features in kapture format can be found here.

If multiple local feature types are used (e.g. for comparison), we strongly recommend to follow our suggested folder structure. For example:

Aachen-Day-Night-v1.1
└─ local_features
   ├─ SIFT
   ├─ D2-Net
   └─ r2d2_WASF-N8_20k
      ├─ keypoints
      ├─ descriptors
      ├─ NN_no_gv           # raw matches after cross-validation
      └─ NN_colmap_gv       # matches after geometric verification using COLMAP

Each local feature subfolder contains keypoints, descriptors, and matches (see next section). Note that this data can be stored in kapture folders as well (as defined in the specification). However, for easy reuse we recommend to store them somewhere else (as done in this example). The kapture pipelines will generate links to these folders.

Mapping

Note
 If you use feather2d2, you can download (or have already downloaded it if you followed the dataset instructions above) the exact map that we used, and can skip this section
Note
 For other features, you have to compute your own maps (our map cannot be used because there is no guarantee that your local features correspond to our keypoints in the map).

You have to create a map on which you’ll evaluate your global features. Different global features should be evaluated on the same map or the global sfm scores won’t be comparable. In order to have numbers similar to what we reported, we provide the pairsfile we used for the mapping.

# for all kapture_pipeline_mapping.py commands,
# if the COLMAP executable is not available from PATH, the parameter -colmap needs to be set
#   example: -colmap C:/Workspace/dev/colmap/colmap.bat

# Aachen Day-Night v1.1
mkdir -p Aachen-Day-Night-v1.1/pairsfile/mapping
wget -O Aachen-Day-Night-v1.1/pairsfile/mapping/Aachen_Day_Night_1_1_ir_benchmark_pairs.txt http://download.europe.naverlabs.com/kapture/Aachen_Day_Night_1_1_ir_benchmark_pairs.txt
# sha256sum available at http://download.europe.naverlabs.com/kapture/Aachen_Day_Night_1_1_ir_benchmark_pairs.txt.sha256sum
kapture_pipeline_mapping.py -v info \
    -i Aachen-Day-Night-v1.1/mapping \
    -kpt Aachen-Day-Night-v1.1/local_features/r2d2_WASF-N8_20k/keypoints \
    -desc Aachen-Day-Night-v1.1/local_features/r2d2_WASF-N8_20k/descriptors \
    --pairsfile-path Aachen-Day-Night-v1.1/pairsfile/mapping/Aachen_Day_Night_1_1_ir_benchmark_pairs.txt \
    -matches Aachen-Day-Night-v1.1/local_features/r2d2_WASF-N8_20k/NN_no_gv/matches \
    -matches-gv Aachen-Day-Night-v1.1/local_features/r2d2_WASF-N8_20k/NN_colmap_gv/matches \
    --colmap-map Aachen-Day-Night-v1.1/colmap-sfm/r2d2_WASF-N8_20k/ir_benchmark/colmap

# Baidu Mall
mkdir -p baidu-mall/pairsfile/mapping
wget -O baidu-mall/pairsfile/mapping/baidu_mall_ir_benchmark_pairs.txt http://download.europe.naverlabs.com/kapture/baidu_mall_ir_benchmark_pairs.txt
# sha256sum available at http://download.europe.naverlabs.com/kapture/baidu_mall_ir_benchmark_pairs.txt.sha256sum
kapture_pipeline_mapping.py -v info \
    -i baidu-mall/mapping \
    -kpt baidu-mall/local_features/r2d2_WASF-N8_20k/keypoints \
    -desc baidu-mall/local_features/r2d2_WASF-N8_20k/descriptors \
    --pairsfile-path baidu-mall/pairsfile/mapping/baidu_mall_ir_benchmark_pairs.txt \
    -matches baidu-mall/local_features/r2d2_WASF-N8_20k/NN_no_gv/matches \
    -matches-gv baidu-mall/local_features/r2d2_WASF-N8_20k/NN_colmap_gv/matches \
    --colmap-map baidu-mall/colmap-sfm/r2d2_WASF-N8_20k/ir_benchmark/colmap

# RobotCar Seasons
mkdir -p RobotCar_Seasons-v1/pairsfile/mapping
wget -O RobotCar_Seasons-v1/pairsfile/mapping/RobotCar_Seasons_v1_ir_benchmark_pairs.txt http://download.europe.naverlabs.com/kapture/RobotCar_Seasons_v1_ir_benchmark_pairs.txt
# sha256sum available at http://download.europe.naverlabs.com/kapture/RobotCar_Seasons_v1_ir_benchmark_pairs.txt.sha256sum
kapture_pipeline_mapping.py -v info \
    -i RobotCar_Seasons-v1/mapping \
    -kpt RobotCar_Seasons-v1/local_features/r2d2_WASF-N8_20k/keypoints \
    -desc RobotCar_Seasons-v1/local_features/r2d2_WASF-N8_20k/descriptors \
    --pairsfile-path RobotCar_Seasons-v1/pairsfile/mapping/RobotCar_Seasons_v1_ir_benchmark_pairs.txt \
    -matches RobotCar_Seasons-v1/local_features/r2d2_WASF-N8_20k/NN_no_gv/matches \
    -matches-gv RobotCar_Seasons-v1/local_features/r2d2_WASF-N8_20k/NN_colmap_gv/matches \
    --colmap-map RobotCar_Seasons-v1/colmap-sfm/r2d2_WASF-N8_20k/ir_benchmark/colmap

# Gangnam Station B2
mkdir -p GangnamStation/B2/release/pairsfile/mapping
wget -O GangnamStation/B2/release/pairsfile/mapping/GangnamStation_B2_ir_benchmark_pairs.txt http://download.europe.naverlabs.com/kapture/GangnamStation_B2_ir_benchmark_pairs.txt
# sha256sum available at http://download.europe.naverlabs.com/kapture/GangnamStation_B2_ir_benchmark_pairs.txt.sha256sum
kapture_pipeline_mapping.py -v info \
    -i GangnamStation/B2/release/mapping \
    -kpt GangnamStation/B2/release/local_features/r2d2_WASF-N8_20k/keypoints \
    -desc GangnamStation/B2/release/local_features/r2d2_WASF-N8_20k/descriptors \
    --pairsfile-path GangnamStation/B2/release/pairsfile/mapping/GangnamStation_B2_ir_benchmark_pairs.txt \
    -matches GangnamStation/B2/release/local_features/r2d2_WASF-N8_20k/NN_no_gv/matches \
    -matches-gv GangnamStation/B2/release/local_features/r2d2_WASF-N8_20k/NN_colmap_gv/matches \
    --colmap-map GangnamStation/B2/release/colmap-sfm/r2d2_WASF-N8_20k/ir_benchmark/colmap

Benchmark

See our results for feather2d2_dim32_20k and r2d2_WASF-N8_20k in benchmark_results

# here it is assumed that you want to use feather2d2 features
export LFEAT=feather2d2_dim32_20k

cd kapture_datasets
# if the COLMAP executable is not available from PATH, the parameter -colmap needs to be set
#   example: -colmap C:/Workspace/dev/colmap/colmap.bat

# run full benchmark (all 3 tasks) for AP-GeM-LM18_top20 on
# Aachen Day-Night v1.1
kapture_pipeline_image_retrieval_benchmark.py -v info \
      -i Aachen-Day-Night-v1.1/mapping \
      --query Aachen-Day-Night-v1.1/query_all \
      -kpt Aachen-Day-Night-v1.1/local_features/${LFEAT}/keypoints \
      -desc Aachen-Day-Night-v1.1/local_features/${LFEAT}/descriptors \
      -gfeat Aachen-Day-Night-v1.1/global_features/AP-GeM-LM18/global_features \
      -matches Aachen-Day-Night-v1.1/local_features/${LFEAT}/NN_no_gv/matches \
      -matches-gv Aachen-Day-Night-v1.1/local_features/${LFEAT}/NN_colmap_gv/matches \
      --colmap-map Aachen-Day-Night-v1.1/colmap-sfm/${LFEAT}/ir_benchmark/colmap \
      -o Aachen-Day-Night-v1.1/image_retrieval_benchmark/${LFEAT}/ir_benchmark/AP-GeM-LM18_top20 \
      --topk 20 \
      --config 2

# Baidu Mall
kapture_pipeline_image_retrieval_benchmark.py -v info \
      -i baidu-mall/mapping \
      --query baidu-mall/query \
      -kpt baidu-mall/local_features/${LFEAT}/keypoints \
      -desc baidu-mall/local_features/${LFEAT}/descriptors \
      -gfeat baidu-mall/global_features/AP-GeM-LM18/global_features \
      -matches baidu-mall/local_features/${LFEAT}/NN_no_gv/matches \
      -matches-gv baidu-mall/local_features/${LFEAT}/NN_colmap_gv/matches \
      --colmap-map baidu-mall/colmap-sfm/${LFEAT}/ir_benchmark/colmap \
      -o baidu-mall/image_retrieval_benchmark/${LFEAT}/ir_benchmark/AP-GeM-LM18_top20 \
      --topk 20 \
      --config 2

# RobotCar Seasons
kapture_pipeline_image_retrieval_benchmark.py -v info \
      -i RobotCar_Seasons-v1/mapping \
      --query RobotCar_Seasons-v1/query \
      -kpt RobotCar_Seasons-v1/local_features/${LFEAT}/keypoints \
      -desc RobotCar_Seasons-v1/local_features/${LFEAT}/descriptors \
      -gfeat RobotCar_Seasons-v1/global_features/AP-GeM-LM18/global_features \
      -matches RobotCar_Seasons-v1/local_features/${LFEAT}/NN_no_gv/matches \
      -matches-gv RobotCar_Seasons-v1/local_features/${LFEAT}/NN_colmap_gv/matches \
      --colmap-map RobotCar_Seasons-v1/colmap-sfm/${LFEAT}/ir_benchmark/colmap \
      -o RobotCar_Seasons-v1/image_retrieval_benchmark/${LFEAT}/ir_benchmark/AP-GeM-LM18_top20 \
      --topk 20 \
      --config 2 \
      --benchmark-style RobotCar_Seasons # important for RobotCar (needed for visuallocalization.net)

# Gangnam Station B2
kapture_pipeline_image_retrieval_benchmark.py -v info \
      -i GangnamStation/B2/release/mapping \
      --query GangnamStation/B2/release/query_all \
      -kpt GangnamStation/B2/release/local_features/${LFEAT}/keypoints \
      -desc GangnamStation/B2/release/local_features/${LFEAT}/descriptors \
      -gfeat GangnamStation/B2/release/global_features/AP-GeM-LM18/global_features \
      -matches GangnamStation/B2/release/local_features/${LFEAT}/NN_no_gv/matches \
      -matches-gv GangnamStation/B2/release/local_features/${LFEAT}/NN_colmap_gv/matches \
      --colmap-map GangnamStation/B2/release/colmap-sfm/${LFEAT}/ir_benchmark/colmap \
      -o GangnamStation/B2/release/image_retrieval_benchmark/${LFEAT}/ir_benchmark/AP-GeM-LM18_top20 \
      --topk 20 \
      --config 2 \
      --benchmark-style Gangnam_Station # important for Gangnam Station B2 (needed for visuallocalization.net)

The benchmark script will execute the following commands:

  1. kapture_compute_image_pairs.py associates similar images between the mapping and query sets

  2. kapture_merge.py merges the mapping and query sensors into the same folder (necessary to compute matches)

  3. kapture_compute_matches.py computes 2D-2D matches using local features and the list of pairs

  4. kapture_run_colmap_gv.py runs geometric verification on the 2D-2D matches

  5. kapture_colmap_localize.py runs the camera pose estimation (Task 2b: global sfm)

  6. kapture_import_colmap.py imports the COLMAP results into kapture

  7. kapture_export_LTVL2020.py exports the global sfm results to a format compatible with the https://www.visuallocalization.net/ benchmark

  8. kapture_pycolmap_localsfm.py runs the camera pose estimation (Task 2a: local sfm)

  9. kapture_export_LTVL2020.py exports the local sfm results to a format compatible with the https://www.visuallocalization.net/ benchmark

  10. kapture_pose_approximation.py run 3 variants of camera pose approximation (Task 1)

  11. kapture_export_LTVL2020.py exports the three pose approximation results (called 3 times) to a format compatible with the https://www.visuallocalization.net/ benchmark

  12. kapture_evaluate.py if query ground truth is available, this evaluates the localization results

In this script, the --config option will select the parameters passed to the COLMAP image_registrator. The parameters are described in colmap_command.py.

The benchmark script also has a parameter --skip which can be used to skip parts of the benchmark. For example, if you want to evaluate your global features only on global SFM, you could use --skip local_sfm pose_approximation.

The folder image_retrieval_benchmark/feather2d2_dim32_20k/frustum_thresh10_far50/AP-GeM-LM18_top20 contains the pairs file as well as the LTVL-style results and kapture-style eval results for each dataset.

# run full benchmark (all 3 tasks) for fire_top20 on
# Aachen Day-Night v1.1
kapture_pipeline_image_retrieval_benchmark_from_pairsfile.py -v info \
      -i Aachen-Day-Night-v1.1/mapping \
      --query Aachen-Day-Night-v1.1/query_all \
      -kpt Aachen-Day-Night-v1.1/local_features/${LFEAT}/keypoints \
      -desc Aachen-Day-Night-v1.1/local_features/${LFEAT}/descriptors \
      --pairsfile-path Aachen-Day-Night-v1.1/pairsfile/query/fire_top20.txt \
      -matches Aachen-Day-Night-v1.1/local_features/${LFEAT}/NN_no_gv/matches \
      -matches-gv Aachen-Day-Night-v1.1/local_features/${LFEAT}/NN_colmap_gv/matches \
      --colmap-map Aachen-Day-Night-v1.1/colmap-sfm/${LFEAT}/ir_benchmark/colmap \
      -o Aachen-Day-Night-v1.1/image_retrieval_benchmark/${LFEAT}/ir_benchmark/fire_top20 \
      --config 2

# Baidu Mall
kapture_pipeline_image_retrieval_benchmark_from_pairsfile.py -v info \
      -i baidu-mall/mapping \
      --query baidu-mall/query \
      -kpt baidu-mall/local_features/${LFEAT}/keypoints \
      -desc baidu-mall/local_features/${LFEAT}/descriptors \
      --pairsfile-path baidu-mall/pairsfile/query/fire_top20.txt \
      -matches baidu-mall/local_features/${LFEAT}/NN_no_gv/matches \
      -matches-gv baidu-mall/local_features/${LFEAT}/NN_colmap_gv/matches \
      --colmap-map baidu-mall/colmap-sfm/${LFEAT}/ir_benchmark/colmap \
      -o baidu-mall/image_retrieval_benchmark/${LFEAT}/ir_benchmark/fire_top20 \
      --config 2

# RobotCar Seasons
kapture_pipeline_image_retrieval_benchmark_from_pairsfile.py -v info \
      -i RobotCar_Seasons-v1/mapping \
      --query RobotCar_Seasons-v1/query \
      -kpt RobotCar_Seasons-v1/local_features/${LFEAT}/keypoints \
      -desc RobotCar_Seasons-v1/local_features/${LFEAT}/descriptors \
      --pairsfile-path RobotCar_Seasons-v1/pairsfile/query/fire_top20.txt \
      -matches RobotCar_Seasons-v1/local_features/${LFEAT}/NN_no_gv/matches \
      -matches-gv RobotCar_Seasons-v1/local_features/${LFEAT}/NN_colmap_gv/matches \
      --colmap-map RobotCar_Seasons-v1/colmap-sfm/${LFEAT}/ir_benchmark/colmap \
      -o RobotCar_Seasons-v1/image_retrieval_benchmark/${LFEAT}/ir_benchmark/fire_top20 \
      --config 2 \
      --benchmark-style RobotCar_Seasons # important for RobotCar (needed for visuallocalization.net)

# Gangnam Station B2
kapture_pipeline_image_retrieval_benchmark_from_pairsfile.py -v info \
      -i GangnamStation/B2/release/mapping \
      --query GangnamStation/B2/release/query_all \
      -kpt GangnamStation/B2/release/local_features/${LFEAT}/keypoints \
      -desc GangnamStation/B2/release/local_features/${LFEAT}/descriptors \
      --pairsfile-path GangnamStation/B2/release/pairsfile/query/fire_top20.txt \
      -matches GangnamStation/B2/release/local_features/${LFEAT}/NN_no_gv/matches \
      -matches-gv GangnamStation/B2/release/local_features/${LFEAT}/NN_colmap_gv/matches \
      --colmap-map GangnamStation/B2/release/colmap-sfm/${LFEAT}/ir_benchmark/colmap \
      -o GangnamStation/B2/release/image_retrieval_benchmark/${LFEAT}/ir_benchmark/fire_top20 \
      --config 2 \
      --benchmark-style Gangnam_Station # important for Gangnam Station B2 (needed for visuallocalization.net)

The folder image_retrieval_benchmark/feather2d2_dim32_20k/frustum_thresh10_far50/fire_top20 contains the LTVL-style results and kapture-style eval results for each dataset.