Home

Awesome

No Bells, Just Whistles: Sports Field Registration by Leveraging Geometric Properties

<div align="center">

Scholar Conference arXiv

<p align="center"><img src=figures/FieldReconstruction-1.png width=70% height=70%></p> </div>

Contents

<hr>

Abstract

Broadcast sports field registration is traditionally addressed as a homography estimation task, mapping the visible image area to a planar field model, predominantly focusing on the main camera shot. Addressing the shortcomings of previous approaches, we propose a novel calibration pipeline enabling camera calibration using a 3D soccer field model and extending the process to assess the multiple-view nature of broadcast videos. Our approach begins with a keypoint generation pipeline derived from SoccerNet dataset annotations, leveraging the geometric properties of the court. Subsequently, we execute classical camera calibration through DLT algorithm in a minimalist fashion, without further refinement. Through extensive experimentation on real-world soccer broadcast datasets such as SoccerNet-Calibration, WorldCup 2014 and TS-WorldCup, our method demonstrates superior performance in both multiple- and single-view 3D camera calibration while maintaining competitive results in homography estimation compared to state-of-the-art techniques

<p align="center"><img src=figures/Pipeline_r.png width=90% height=90%></p> <strong>Overview of our proposed framework. Top: Training data generation pipeline. Beginning with SoccerNet annotations, we utilize field line extraction and ellipse fitting to establish a hierarchical structure for computing each set of keypoints. Bottom: The encoder-decoder networks produce heatmaps for keypoints and extremities of soccer field lines to extract their positions in the image space. The obtained keypoint set is augmented with intersections of lines generated by the second model to ensure a sufficient number of points.</strong> <hr>

Weights

Download the keypoints and line detection model weights for both single-view and multiple-view configurations. Our models are pre-trained on the SoccerNet distribution and further finetuned on the WC14 and TSWC datasets for enhanced performance.

Single-View

ModelFinetunedLink
KeypointsXSV_kp
LinesXSV_lines
KeypointsWC14SV_FT_WC14_kp
LinesWC14SV_FT_WC14_lines
KeypointsTSWCSV_FT_TSWC_kp
LinesTSWCSV_FT_TSWC_lines

Multiple-View

ModelFinetunedLink
KeypointsXMV_kp
LinesXMV_lines
<hr>

Inference

On video:

python inference.py --weights_kp "SV_kp" --weights_line "SV_lines" --input_path "examples/iniesta_sample.mp4" --input_type "video" --display

On image:

python inference.py --weights_kp "SV_kp" --weights_line "SV_lines" --input_path "examples/messi_sample.png" --input_type "image" --save_path "examples/messi_results.png"
<hr>

Experiments

Datasets

SoccerNet-Calibration-V3:

To download the 2022 version, switch the downloader's task to "calibration".

from SoccerNet.Downloader import SoccerNetDownloader
mySoccerNetDownloader = SoccerNetDownloader(LocalDirectory="</nfs/data/soccernet>")
mySoccerNetDownloader.downloadDataTask(task="calibration-2023", split=["train","valid","test"])

See https://github.com/MM4SPA/tvcalib to download the camera-type annotations for SoccerNet-Calibration-2022.

WorldCup 2014

mkdir -p datasets/WC-2014/test && cd datasets/WC-2014/test
# Images and provided homography matrices from test split
wget https://nhoma.github.io/data/soccer_data.tar.gz
tar -zxvf soccer_data.tar.gz

See https://github.com/MM4SPA/tvcalib to download the additional segment annotations in SoccerNet-Calibration format.

TS-WorldCup

Download the TS-WorldCup dataset

<hr>

Metrics

Segment Reprojection Error

See https://github.com/SoccerNet/sn-calibration for details on the evaluation metric.

IoU, Projection error and Reprojection error:

For the Homography Estimation evaluation, we adopt the approach outlined at https://github.com/ericsujw/KpSFR. Minor modifications of the used script can be seen in model/metrics.py

<hr>

Evaluation

We provide scripts scripts/ to reproduce the paper's results for the presented approach. Make sure to change the dataset location on the bash scripts, default is set as "datasets/calibration-2023/", and the location of model weights, default is set as weights/MV_kp.

#Multi-view camera parameter estimation for SN-Calib-2023
chmod +x scripts/run_pipeline_sn23.sh
./scripts/run_pipeline_sn23.sh

#Single-view camera parameter estimation for SN-Calib-2022
chmod +x scripts/run_pipeline_sn22.sh
./scripts/run_pipeline_sn22.sh

#WorldCup 2014 as camera parameter estimation task
#Change the dataset location inside "scripts/run_pipeline_sn23.sh" file, i.e. "datasets/WC-2014"
#Change the save directory inside "scripts/run_pipeline_sn23.sh" file to "inference/inference_3D/inference_wc14"
chmod +x scripts/run_pipeline_sn22.sh
./scripts/run_pipeline_sn22.sh

#WorldCup 2014 homography estimation task
chmod +x scripts/run_pipeline_wc14.sh
./scripts/run_pipeline_wc14.sh

#TS-WorldCup homography estimation task
chmod +x scripts/run_pipeline_tswc.sh
./scripts/run_pipeline_tswc.sh

Task results will be printed on screen.

<hr>

Requirements

Conda Environment

conda env create -f NBJWCalib.yml
conda activate NBJWCalib

PIP

python -m venv NBJWCalib
source env/bin/activate
pip install -r requirements.txt
<hr>

Citation

@InProceedings{Gutierrez-Perez_2024_CVPR,
    author    = {Guti\'errez-P\'erez, Marc and Agudo, Antonio},
    title     = {No Bells Just Whistles: Sports Field Registration by Leveraging Geometric Properties},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
    month     = {June},
    year      = {2024},
    pages     = {3325-3334}
}