Awesome

<br>This repository contains the code for the NPSN method applied to the trajectory forecasting models.

🎲 NPSN method 🎲

• Inferences of all stochastic models are biased toward the random sampling, <br> fail to generate a set of realistic paths from finite samples.
• Quasi-Monte Carlo method, ensuring uniform coverage on the sampling space, <br>improves all the multimodal prediction results.
• Non-Probability Sampling Network (NPSN), a very small network that generates purposive sample sequences, <br> can significantly improve both the prediction accuracy (up to 60%) and reliability.
• Simply replacing one line of code on random sampling(click) with our NPSN(here)!

Model Training

Setup

Environment <br>All models were trained and tested on Ubuntu 18.04 with Python 3.7 and PyTorch 1.7.1 with CUDA 10.1.

Dataset <br>Preprocessed ETH and UCY datasets are included in this repository, under ./dataset/. The train/validation/test splits are the same as those fond in Social-GAN.

Baseline models <br>This repository supports three baseline models: Social-STGCNN, SGCN and PECNet. We have included model source codes from their official GitHub in the ./baselines/ folder.

Train NPSN

To train our NPSN on the ETH and UCY datasets at once, we provide a bash script train.sh for a simplified execution.

./train.sh -b <baseline_model>

For example, you can run the following command to train our NPSN with SGCN baseline with the best configuration.

./train.sh -b sgcn

We provide additional arguments for experiments:

./train.sh -t <experiment_tag> -b <baseline_model> -d <space_seperated_dataset_string> -i <space_seperated_gpu_id_string>

# Examples
./train.sh -b sgcn -d "hotel" -i "1"
./train.sh -t exp1 -b stgcnn -d "hotel" -i "1"
./train.sh -t npsn -b pecnet -d "eth hotel univ zara1 zara2" -i "0 0 0 0 0"

If you want to train the model with custom hyper-parameters, use train_npsn.py instead of the script file.

python train_npsn.py --baseline <baseline_model> --tag <experiment_tag> --dataset <dataset_name> \
--num_samples <number_of_samples> --obs_len <observation sequence length> --pred_len <prediction sequence length> \
--batch_size <minibatch_size> --num_epochs <number_of_epochs> --lr <learning_rate> --lr_sh_rate <number_of_steps_to_drop_lr> \
--clip_grad <gradient_clipping> --gpu_num <gpu_id> --use_lrschd 

Reproducibility

All the experiments in this work are performed with seed=0, strictly following PyTorch Reproducibility. <br>We checked that the same results were reproduced on RTX 2080Ti (Ubuntu 18.04, PyTorch 1.7.1, CUDA 10.1) and RTX 3090 (Ubuntu 20.04, PyTorch 1.9.0, CUDA 11.1) environments.

Model Evaluation

Pretrained Models

Baseline models <br>We use model source codes and pretrained weights from their official GitHub. Pretrained baseline models are included in the ./pretrained/ folder.

• Data Split <br>For an apple-to-apple comparison, we used the train-validation-test split of Social-GAN. We used the same data split strategy for all other baseline models.

• Dataloader <br>We used Social-GAN's dataloader instead of their pre-processed pickle file. To work similarly to PECNet's original dataloader, we wrote codes for custom batch-sampler and collate-function.

• Data Types <br>We used torch.FloatTensor instead of torch.DoubleTensor as the data type of the model. We checked that the performance difference between them was negligible.

NPSN method <br>We have included pretrained NPSN models for each baseline model in the ./checkpoints/ folder.

Evaluate NPSN

You can use test_npsn.py to easily run any of the sampling methods on any of the baseline models.

python test_npsn.py --baseline <baseline_model> --tag <experiment_tag> --method <sampling_method> --gpu_num <gpu_id_for_evaluation>

For example, you can replicate the Table 1 results for all datasets for SGCN baseline like this:

python test_npsn.py --baseline sgcn --tag npsn-sgcn --method mc --gpu_num 0
python test_npsn.py --baseline sgcn --tag npsn-sgcn --method qmc --gpu_num 1
python test_npsn.py --baseline sgcn --tag npsn-sgcn --method npsn --gpu_num 2

📖 Citation

If you find this code useful for your research, please cite our trajectory prediction papers :)

💬 LMTrajectory (CVPR'24) 🗨️ | 1️⃣ SingularTrajectory (CVPR'24) 1️⃣ | 🌌 EigenTrajectory (ICCV'23) 🌌 | 🚩 Graph‑TERN (AAAI'23) 🚩 | 🧑‍🤝‍🧑 GP‑Graph (ECCV'22) 🧑‍🤝‍🧑 | 🎲 NPSN (CVPR'22) 🎲 | 🧶 DMRGCN (AAAI'21) 🧶

@inproceedings{bae2022npsn,
  title={Non-Probability Sampling Network for Stochastic Human Trajectory Prediction},
  author={Bae, Inhwan and Park, Jin-Hwi and Jeon, Hae-Gon},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2022}
}

@inproceedings{bae2024lmtrajectory,
  title={Can Language Beat Numerical Regression? Language-Based Multimodal Trajectory Prediction},
  author={Bae, Inhwan and Lee, Junoh and Jeon, Hae-Gon},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2024}
}

@inproceedings{bae2024singulartrajectory,
  title={SingularTrajectory: Universal Trajectory Predictor Using Diffusion Model},
  author={Bae, Inhwan and Park, Young-Jae and Jeon, Hae-Gon},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2024}
}

@inproceedings{bae2023eigentrajectory,
  title={EigenTrajectory: Low-Rank Descriptors for Multi-Modal Trajectory Forecasting},
  author={Bae, Inhwan and Oh, Jean and Jeon, Hae-Gon},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  year={2023}
}

@article{bae2023graphtern,
  title={A Set of Control Points Conditioned Pedestrian Trajectory Prediction},
  author={Bae, Inhwan and Jeon, Hae-Gon},
  journal={Proceedings of the AAAI Conference on Artificial Intelligence},
  year={2023}
}

@inproceedings{bae2022gpgraph,
  title={Learning Pedestrian Group Representations for Multi-modal Trajectory Prediction},
  author={Bae, Inhwan and Park, Jin-Hwi and Jeon, Hae-Gon},
  booktitle={Proceedings of the European Conference on Computer Vision},
  year={2022}
}

@article{bae2021dmrgcn,
  title={Disentangled Multi-Relational Graph Convolutional Network for Pedestrian Trajectory Prediction},
  author={Bae, Inhwan and Jeon, Hae-Gon},
  journal={Proceedings of the AAAI Conference on Artificial Intelligence},
  year={2021}
}

Acknowledgement

Part of our code is borrowed from Social-STGCNN, SGCN and PECNet. We thank the authors for releasing their code and models.