Awesome

Audio Retrieval with Natural Language Queries

This repository is the implementation of Audio Retrieval with Natural Language Queries and it is based on the Use What You Have: Video retrieval using representations from collaborative experts repo. Datasets used in this paper are AudioCaps, CLOTHO, Activity-Net and QuerYD.

More information can be found at our project page: https://www.robots.ox.ac.uk/~vgg/research/audio-retrieval/

:exclamation: An extension of this work along with the new SoundDescs dataset for audio retrieval can be found here. :exclamation:

Requirements

We used PyTorch 1.7.1., CUDA 10.1, and Python 3.7 to generate results and models. The required libraries for running this code can be found in requirements/requirements.txt.

conda create --name audio-retrieval python=3.7
conda activate audio-retrieval
pip install -r requirements/requirements.txt

To be able to run the code below, features extracted from various datasets need to be downloaded. If there is not enough space in your working location to store some of these features (for AudioCaps the file is 6GB while the others are under 1GB) then you will need to create a folder called data inside this repository which should be a symlink to a folder with enough space. As an example, run the following from the audio-experts code-base.

ln -s <path-where-data-can-be-saved> data

To download features for each dataset, follow the steps here

Evaluating a pretrained model on multiple seeds and reproducing results

To reproduce the results in tables below, multiple models trained with different seeds need to be downloaded and evaluated on the test sets.

The steps needed to reproduce the results are:

1. Select the experiment to be reproduced which is in the form <dataset-name>-<config-file-name>. Tables with experiments names and the corresponding form can be found in misc/exps-names.md.
2. Download the features and splits corresponding to the dataset for which the experiment is run. For example, for AudioCaps run:

# fetch the pretrained experts for AudioCaps 
python3 misc/sync_experts.py --dataset AudioCaps

Additional examples for the datasets used in this paper can be found in misc/exps-names.md.

3. Running the eval.py script.

For example, to reproduce the experiments for AudioCaps with all visual and audio experts, run the following line:

python eval.py --experiment audiocaps-train-full-ce-r2p1d-inst-vggish-vggsound

If the --experiment flag is not provided, the eval.py script will download and evaluate all models on the test set.

Training a new model

Training a new audio-text embedding requires:

1. The pretrained experts for the dataset used for training, which should be located in <root>/data/<dataset-name>/symlinked-feats (this will be done automatically by the utility script, or can be done manually). Examples can be found in misc/exps-names.md.
2. A config.json file. You can define your own, or use one of the provided configs in the configs directory.

Training is then performed with the following command:

python3 train.py --config <path-to-config.json> --device <gpu-id>

where <gpu-id> is the index of the GPU to train on. This option can be ommitted for training on the CPU.

For example, to train a new embedding for the CLOTHO dataset, run the following sequence of commands:

# fetch the pretrained experts for CLOTHO 
python3 misc/sync_experts.py --dataset CLOTHO

# Train the model
python3 train.py --config configs/clotho/train-vggish-vggsound.json --device 0

AudioCaps

These are the retrieval results obtained for the AudioCaps dataset when using only audio experts:

Using only visual experts for AudioCaps:

Visual and audio experts for AudioCaps:

CLOTHO

Pretraining on AudioCaps, finetuning on CLOTHO

Visual centric datasets

More information can be found at our project page: https://www.robots.ox.ac.uk/~vgg/research/audio-retrieval/

This code is based on https://github.com/albanie/collaborative-experts

References

[1] If you find this code useful, please consider citing:

@inproceedings{Oncescu21a,
               author       = "Oncescu, A.-M. and Koepke, A.S. and Henriques, J. and Akata, Z., Albanie, S.",
               title        = "Audio Retrieval with Natural Language Queries",
               booktitle    = "INTERSPEECH",
               year         = "2021"
             }

[2] If you find this code useful, please consider citing:

@inproceedings{Liu2019a,
  author    = {Liu, Y. and Albanie, S. and Nagrani, A. and Zisserman, A.},
  booktitle = {arXiv preprint arxiv:1907.13487},
  title     = {Use What You Have: Video retrieval using representations from collaborative experts},
  date      = {2019},
}