Awesome
PnLCalib: Sports Field Registration via Points and Lines Optimization
Extension of the previous paper "No Bells, Just Whistles: Sports Field Registration by Leveraging Geometric Properties"
<div align="center">
🚀 Key Features
-
Improved Keypoint and Line Detection
Enhanced models for faster and more accurate detection of keypoints and line extremities. -
Upgraded Camera Calibration Pipeline
Refined pipeline for better initial camera calibration performance. -
PnL Refinement Module
Added a powerful refinement module that leverages detected field lines for non-linear optimization, boosting calibration accuracy.
Contents
<hr>Abstract
Camera calibration in broadcast sports videos presents numerous challenges for accurate sports field registration due to multiple camera angles, varying camera parameters, and frequent occlusions of the field. Traditional search-based methods depend on initial camera pose estimates, which can struggle in non-standard positions and dynamic environments. In response, we propose an optimization-based calibration pipeline that leverages a 3D soccer field model and a predefined set of keypoints to overcome these limitations. Our method also introduces a novel refinement module that improves initial calibration by using detected field lines in a non-linear optimization process. This approach outperforms existing techniques in both multi-view and single-view 3D camera calibration tasks, while maintaining competitive performance in homography estimation. Extensive experimentation on real-world soccer datasets, including SoccerNet-Calibration, WorldCup 2014, and TS-WorldCup, highlights the robustness and accuracy of our method across diverse broadcast scenarios. Our approach offers significant improvements in camera calibration precision and reliability.
<p align="center"><img src=figures/Pipeline_r-1.png width=90% height=90%></p> <strong>Overview of our proposed framework.</strong> Top: Training data generation pipeline. Beginning with SoccerNet annotations, we utilize field line extraction and ellipse fitting to establish a hierarchical structure for computing each set of keypoints. Bottom: Inference stage pipeline. The encoder-decoder networks produce heatmaps for keypoints and extremities of soccer field lines to extract their positions in the image space. The obtained keypoint set is augmented with intersections of lines generated by the second model to ensure a sufficient number of points. After initial calibration, our PnL refinement module is applied to further refine the calibration estimate by jointly using detected points and lines information.Weights
Download the keypoints and line detection model weights for both single-view and multiple-view configurations. Our models are pre-trained on the SoccerNet distribution and further finetuned on the WC14 and TSWC datasets for enhanced performance.
Single-View
| Model | Finetuned | Link |
|---|---|---|
| Keypoints | X | SV_kp |
| Lines | X | SV_lines |
| Keypoints | WC14 | SV_FT_WC14_kp |
| Lines | WC14 | SV_FT_WC14_lines |
| Keypoints | TSWC | SV_FT_TSWC_kp |
| Lines | TSWC | SV_FT_TSWC_lines |
Multiple-View
| Model | Finetuned | Link |
|---|---|---|
| Keypoints | X | MV_kp |
| Lines | X | MV_lines |
Inference
Remove --pnl_refine argument for a base inference.
On video:
python inference.py --weights_kp "SV_kp" --weights_line "SV_lines" --pnl_refine --input_path "examples/iniesta_sample.mp4" --input_type "video" --display
On image:
python inference.py --weights_kp "SV_kp" --weights_line "SV_lines" --pnl_refine --input_path "examples/messi_sample.png" --input_type "image" --save_path "examples/messi_results.png"
Experiments
Datasets
SoccerNet-Calibration-V3:
To download the 2022 version, switch the downloader's task to "calibration".
from SoccerNet.Downloader import SoccerNetDownloader
mySoccerNetDownloader = SoccerNetDownloader(LocalDirectory="</nfs/data/soccernet>")
mySoccerNetDownloader.downloadDataTask(task="calibration-2023", split=["train","valid","test"])
See https://github.com/MM4SPA/tvcalib to download the camera-type annotations for SoccerNet-Calibration-2022.
WorldCup 2014
mkdir -p datasets/WC-2014/test && cd datasets/WC-2014/test
# Images and provided homography matrices from test split
wget https://nhoma.github.io/data/soccer_data.tar.gz
tar -zxvf soccer_data.tar.gz
See https://github.com/MM4SPA/tvcalib to download the additional segment annotations in SoccerNet-Calibration format.
TS-WorldCup
Download the TS-WorldCup dataset
<hr>Metrics
Segment Reprojection Error
See https://github.com/SoccerNet/sn-calibration for details on the evaluation metric.
IoU, Projection error and Reprojection error:
For the Homography Estimation evaluation, we adopt the approach outlined at https://github.com/ericsujw/KpSFR. Minor modifications of the used script can be seen in
model/metrics.py
Evaluation
We provide scripts scripts/ to reproduce the paper's results for the presented approach.
Make sure to change the dataset location on the bash scripts, default is set as "datasets/calibration-2023/", and the location of model weights, default is set as weights/MV_kp.
#Multi-view camera parameter estimation for SN-Calib-2023
chmod +x scripts/run_pipeline_sn23.sh
./scripts/run_pipeline_sn23.sh
#Single-view camera parameter estimation for SN-Calib-2022
chmod +x scripts/run_pipeline_sn22.sh
./scripts/run_pipeline_sn22.sh
#WorldCup 2014 camera parameter estimation task
chmod +x scripts/run_pipeline_wc14_3D.sh
./scripts/run_pipeline_wc14_3D.sh
#WorldCup 2014 homography estimation task
chmod +x scripts/run_pipeline_wc14.sh
./scripts/run_pipeline_wc14.sh
#TS-WorldCup homography estimation task
chmod +x scripts/run_pipeline_tswc.sh
./scripts/run_pipeline_tswc.sh
Task results will be printed on screen.
<hr>Training
Keypoint detection
python train.py --cfg config/hrnetv2_w48.yaml --dataset SoccerNet --root_dir datasets/calibration-2023/ --save_dir weights/MV_kp_exp1
Line detection
python train_l.py --cfg config/hrnetv2_w48_l.yaml --dataset SoccerNet --root_dir datasets/calibration-2023/ --save_dir weights/MV_line_exp1
Options for --dataset argument are: [SoccerNet, WorldCup2014, TSWorldCup]
Additional available arguments for both keypoint and line detection training:
--cuda: CUDA device index (default: 'cuda:0')--batch: Batch size for train / val (default: 2)--num_workers: Number of workers for data loading (default: 4)--num_epochs: Number of training epochs (default: 200)--pretrained: Pretrained weights path--lr0: Initial learning rate (default: 0.001)--patience: Patience parameter for lr scheduler (default: 8)--factor: Reducing factor for lr scheduler (default: 0.5)--wandb_project: Wandb project name
Requirements
Conda Environment
conda env create -f PnLCalib.yml
conda activate PnLCalib
PIP
python -m venv PnLCalib
source env/bin/activate
pip install -r requirements.txt
Citation
@article{gutierrez4998149pnlcalib,
title={Pnlcalib: Sports Field Registration Via Points and Lines Optimization},
author={Guti{\'e}rrez-P{\'e}rez, Marc and Agudo, Antonio},
journal={Available at SSRN 4998149}
}