Hubble Violations Through Time (VTT) Visual Place Recognition (VPR)

Work in progress repo repurposing the VPR repo for violations through time. This repo is in active development so some things wont work.

Overview

Installing dependencies

For now I cannot get the pip requirements to work (faiss does not install correctly) so I am using the default conda environment:

• Install conda
• Run conda env create --name vpr --file=.devcontainer/environment.yml
• Then activate the environment: conda activate vpr

Running on our data

Please generate the VTT dataset using the scripts located here: violations through time, and copy the output directory image_dataset into this repo.

Then:

• Generate the dataset in the formats for VTT by running python3 generate_hubble_vpr_dataset.py. This will create a new directory images/HTT_example containing distinct datasets for each violation type comparing a scan with its previous scan from a different batch
• To run on one dataset run hubble_baseline_demo.py with the following prameters python3 hubble_baseline_demo.py --descriptor [Feature descriptor to use] --dataset [Dataset to run on (Leave out images/HTT_example in string)] --output [Optional: output dir (defaults to results)]
• To run on all datasets use the helper bash script supplied: bash run_baseline.bash

Optional filters (WIP)

• --iou [Float]: Minimum IoU between violation bounding boxes. Any below will be disregarded as matches (This does not work very well in practice)
• --distance [Float]: Maximum distance (in km) that a query violation can be from the one being considered as a match. Any further away than this will be disregarded (This works as well as our GPS data allows)
• WIP: --local_descriptor [Option of featire extractor] (Only HDF-DELF implemented at this time): Run a second stage feature extractor on violation bounding boxes. The outputs will then ensure that the global features and local violation features are above a matching threshold to be considered. (This is getting some stage results, potentially as bounding boxes are different sizes but also as distance thresholding can be a little tricky. Left partially implemented for now)
• WIP: --local_descriptor_normalised [Option of featire extractor] (Only HDF-DELF implemented at this time): Same as above but it finds the largest bounding box between each violation being considered for matching and uses this for generating local features. This has only been partially implemented at this point, still needs work

Old readme:

Visual Place Recognition: A Tutorial

Work in progress: This repository provides the example code from our paper "Visual Place Recognition: A Tutorial". The code performs VPR on the GardensPoint day_right--night_right dataset. Output is a plotted pr-curve, matching decisions, two examples for a true-positive and a false-positive matching, and the AUC performance, as shown below.

If you use our work for your academic research, please refer to the following paper:

@article{SchubertRAM2023ICRA2024,
  title={Visual Place Recognition: A Tutorial},
  author={Schubert, Stefan and Neubert, Peer and Garg, Sourav and Milford, Michael and Fischer, Tobias},
  journal={IEEE Robotics \& Automation Magazine},  
  year={2023},
  doi={10.1109/MRA.2023.3310859}
}

How to run the code

Online (Using GitHub Codespaces)

This repository is configured for use with GitHub Codespaces, a service that provides you with a fully-configured Visual Studio Code environment in the cloud, directly from your GitHub repository.

To open this repository in a Codespace:

1. Click on the green "Code" button near the top-right corner of the repository page.
2. In the dropdown, select "Open with Codespaces", and then click on "+ New codespace".
3. Your Codespace will be created and will start automatically. This process may take a few minutes.

Once your Codespace is ready, it will open in a new browser tab. This is a full-featured version of VS Code running in your browser, and it has access to all the files in your repository and all the tools installed in your Docker container.

You can run commands in the terminal, edit files, debug code, commit changes, and do anything else you would normally do in VS Code. When you're done, you can close the browser tab, and your Codespace will automatically stop after a period of inactivity.

Local

python3 demo.py

The GardensPoints Walking dataset will be downloaded automatically. You should get an output similar to this:

python3 demo.py
===== Load dataset
===== Load dataset GardensPoint day_right--night_right
===== Compute local DELF descriptors
===== Compute holistic HDC-DELF descriptors
===== Compute cosine similarities S
===== Match images
===== Evaluation

===== AUC (area under curve): 0.74
===== R@100P (maximum recall at 100% precision): 0.36
===== recall@K (R@K) -- R@1: 0.85 , R@5: 0.925 , R@10: 0.945

Precision-recall curve	Matchings M	Examples for a true positive and a false positive

Requirements

The code was tested with the library versions listed in requirements.txt. Note that Tensorflow or PyTorch is only required if the corresponding image descriptor is used. If you use pip, simply:

pip install -r requirements.txt

You can create a conda environment containing these libraries as follows (or use the provided environment.yml):

mamba create -n vprtutorial python numpy pytorch torchvision natsort tqdm opencv pillow scikit-learn faiss matplotlib-base tensorflow tensorflow-hub tqdm scikit-image patchnetvlad -c conda-forge

List of existing open-source implementations for VPR (work in progress)

Descriptors

Holistic descriptors

AlexNet	code*	paper
AMOSNet	code	paper
DELG	code	paper
DenseVLAD	code	paper
HDC-DELF	code	paper
HybridNet	code	paper
NetVLAD	code	paper
CosPlace	code	paper
EigenPlaces	code	paper

Local descriptors

D2-Net	code	paper
DELF	code	paper
LIFT	code	paper
Patch-NetVLAD	code	paper
R2D2	code	paper
SuperPoint	code	paper

Local descriptor aggregation

DBoW2	code	paper
HDC (Hyperdimensional Computing)	code	paper
iBoW (Incremental Bag-of-Words) / OBIndex2	code	paper
VLAD (Vector of Locally Aggregated Descriptors)	code*	paper

Sequence methods

Delta Descriptors	code	paper
MCN	code	paper
OpenSeqSLAM	code*	paper
OpenSeqSLAM 2.0	code	paper
OPR	code	paper
SeqConv	code	paper
SeqNet	code	paper
SeqVLAD	code	paper
VPR	code	paper

Misc

ICM	code	paper	Graph optimization of the similarity matrix S
SuperGlue	code	paper	Local descriptor matching

*Third party code. Not provided by the authors. Code implements the author's idea or can be used to implement the authors idea.