Awesome

Deep Motion Modeling Tracker

We introduce the Deep Motion Modeling Network (DMM-Net) that performs implicit detection and association of the objects in an end-to-end manner. DMM-Net models comprehensive object features over multiple frames and simultaneously infers object motion parameters, categories and visibilities. These outputs are readily used to update the tracklets for efficient MOT. DMM-Net achieves PR-MOTA score of 12.80 @ 120+ fps for jointly performing detection and tracking on the popular UA-DETRAC challenge - orders of magnitude faster than the existing methods with better performance.

• Deep Motion Modeling Tracker
  • Results
  • RoadMap
  • Protocol
  • DMM-Net
  • DMM Tracker
  • Requirement
  • Preparation
  • OMOTD
    • Test
    • Train
  • UA-DETRAC
    • Test
    • Train
  • Citation
  • Acknowledge
  • License

Results

DMM-Net on Omni-MOT dataset

DMM-Net on UA-DETRAC dataset

RoadMap

Protocol

• <img src="https://latex.codecogs.com/svg.latex?N_F,N_C,N_P,N_T"/> respectively denote the number of input frames, object categories (0 for `background'), time-related motion parameters, and anchor tunnels.
• <img src="https://latex.codecogs.com/svg.latex?W,H"/> are the frame width, and frame height.
• <img src="https://latex.codecogs.com/svg.latex?\bf{I}_t"/> denotes the video frame at time <img src="https://latex.codecogs.com/svg.latex?t"/>. Subsequently, a 4-D tensor <img src="https://latex.codecogs.com/svg.latex?\bf{I}_{t_1:t_2:N_F}\in\mathbb{R}^{3\times&space;N_F&space;\times&space;W&space;\times&space;H}"/> denotes <img src="https://latex.codecogs.com/svg.latex?N_F"/> video frames from time <img src="https://latex.codecogs.com/svg.latex?t_1"/> to <img src="https://latex.codecogs.com/svg.latex?t_2-1"/>. For simplicity, we often ignore the subscript ``<img src="https://latex.codecogs.com/svg.latex?:N_F"/>''.
• <img src="https://latex.codecogs.com/svg.latex?\bf{B}_{t_1:t_2:N_F},\bf{C}_{t_1:t_2:N_F},\bf{V}_{t_1:t_2:N_F}"/> respectively denote the ground truth boxes, categories, and visibilities in the selected <img src="https://latex.codecogs.com/svg.latex?N_F"/> video frames from time <img src="https://latex.codecogs.com/svg.latex?t_1"/> to <img src="https://latex.codecogs.com/svg.latex?t_2-1"/>. The text also ignores ``<img src="https://latex.codecogs.com/svg.latex?:N_F"/>'' for these notations.
• <img src="https://latex.codecogs.com/svg.latex?\bf{O}_{M,t_1:t_2:N_F},O_{C,t_1:t_2:N_F},O_{V,t_1:t_2:N_F}"/> denote the estimated motion parameters, categories, and visibilities. With time stamps and frames clear from the context, we simplify these notations as <img src="https://latex.codecogs.com/svg.latex?O_M,O_C,O_V"/>.

DMM-Net

Schematics of end-to-end trainable DMM-Net: <img src="https://latex.codecogs.com/svg.latex?N_F"/> frames and their time stamps <img src="https://latex.codecogs.com/svg.latex?t_1:t_2"/> are input to the network. The frame sequence is first processed with a Feature Extractor comprising 3D ResNet-like convolutional groups. Outputs of selected groups are processed by Motion Subnet, Classifier Subnet, and Visibility Subnet. Each sub-network uses 3D convolutions to learn features that are concatenated and used to predict motion parameters (<img src="https://latex.codecogs.com/svg.latex?O_M\in\mathbb{R}^{N_T\times&space;N_P\times&space;4}"/>), object categories (<img src="https://latex.codecogs.com/svg.latex?O_C\in\mathbb{R}^{N_T\times&space;N_C}"/>), and visibility (<img src="https://latex.codecogs.com/svg.latex?O_V\in\mathbb{R}^{N_F\times&space;N_T\times&space;2}"/>), where <img src="https://latex.codecogs.com/svg.latex?N_T"/> <img src="https://latex.codecogs.com/svg.latex?N_P"/> and <img src="https://latex.codecogs.com/svg.latex?N_C"/> denote the number of anchor tunnels, motion parameters and object categories.

DMM Tracker

We directly deploy the trained network into the DMM Tracker (DMMT), as shown in the following figure. <img src="https://latex.codecogs.com/svg.latex?2N_F"/> frames are processed by the tracker, where the trained DMM-Net selects <img src="https://latex.codecogs.com/svg.latex?N_F"/> frames as its input, and outputs predicted tunnels containing all possible object's motion parameter matrice <img src="https://latex.codecogs.com/svg.latex?(O_M)"/>, category matrice <img src="https://latex.codecogs.com/svg.latex?(O_C)"/> and visibility matrice <img src="https://latex.codecogs.com/svg.latex?(O_V)"/>, which are then filtered by the Tunnel Filter. After that, the track set <img src="https://latex.codecogs.com/svg.latex?\mathcal{T}_{t_i}"/> is updated by associating the filtered tunnels by their IOU with previous track set <img src="https://latex.codecogs.com/svg.latex?\mathcal{T}_{t_{i-1}}"/>.

This tracker can achieve 120+ fps for jointly performing detection and tracking.

Requirement

Besides, install all the python package by following command

cd <project path>
pip install -r requirement.txt

Preparation

• Clone this repository

  git clone <repository url>
  

• Download the pre-trained base net model, and save it into <project path>/weights/resnext-101-64f-kinetics.pth

OMOTD

• Download the training dataset and testing dataset from [baidu] or [dropbox no space availabe :-(]

Test

• Download the network weights file ([dropbox], [baidu])

• Modify the <project path>/config/__init__.py to

  15. # configure_name = 'config_gpu4_ua.json'
  16. # configure_name = 'config_gpu4_ua_test.json'
  17. # configure_name = 'config_gpu4_ua_with_amot.json'
  18. configure_name = 'config_gpu4_amot.json'
  

• Modify the <project path>/config/config_gpu4_amot.json

  1.    {
  2.       "dataset_name": "AMOTD",
  3.       "dataset_path": <your downloaded omotd folder>,
  4.		 "phase": "test",    
      ...
  24.			"test": {
  25.				"resume": <your downloaded weights>,
  26.			    "dataset_type": <you can use "train" or "test">,    
      ...
  30.    			"base_net_weights": null,
  31.    			"log_save_folder": <your log save folder>,
  32.			    "image_save_folder": <your image save folder>,
  33.			    "weights_save_folder": <your weights save folder>, 
  

• Activate your python environment, and run

  cd <project folder>
  python test_tracker_amotd.py
  

Train

• Modify the <project path>/config/__init__.py to

  15. # configure_name = 'config_gpu4_ua.json'
  16. # configure_name = 'config_gpu4_ua_test.json'
  17. # configure_name = 'config_gpu4_ua_with_amot.json'
  18. configure_name = 'config_gpu4_amot.json'
  

• Modify the <project path>/config/config_gpu4_amot.json

  1.	{
  2.		"dataset_name": "AMOTD",
  3.	  	"dataset_path": <your downloaded omotd folder>,
      ...
  4.		"phase": "train",    
      ...
  64. 	"train": {    
  65.    		"resume": null,
  66.    		"batch_size": 8 <you can change it according to your gpu capability>,
      ...
  71.			"log_save_folder": <your log save folder>,
  72.		    "image_save_folder": <your image save folder>,
  73.		    "weights_save_folder": <your weights save folder>,    
      ...    
  

• Activate your python environment, and run

  cd <project folder>
  python train_amot.py
  

UA-DETRAC

• Download the training and testing dataset from [UA-DETRAC Official Site] or [baidu]

Test

• Download the network weights file([dropbox], [baidu])

• Modify the <project path>/config/__init__.py to

  15. configure_name = 'config_gpu4_ua.json'
  16. # configure_name = 'config_gpu4_ua_test.json'
  17. # configure_name = 'config_gpu4_ua_with_amot.json'
  18. # configure_name = 'config_gpu4_amot.json'
  

• Modify the <project path>/config/config_gpu4_ua.json

  1. {
  2.   "dataset_name": "UA-DETRAC",
  3.   "dataset_path": <the UA-DETRAC dataset folder>,
  4.   "phase": "test",
      ...
  36.  "test": {
  37.    "resume": <network weights file>,
  38.    "dataset_type": "test",    
      ...
  42.    "base_net_weights": null,
  43.    "log_save_folder": <your log save folder>,
  44.    "image_save_folder": <your image save folder>,
  45.    "weights_save_folder": <your network weights save folder>,    
  

• Activate your python environment, and run

  cd <project folder>
  python test_tracker_ua.py
  

Train

• Modify the <project path>/config/__init__.py to

  15. configure_name = 'config_gpu4_ua.json'
  16. # configure_name = 'config_gpu4_ua_test.json'
  17. # configure_name = 'config_gpu4_ua_with_amot.json'
  18. # configure_name = 'config_gpu4_amot.json'
  

• Modify the <project path>/config/config_gpu4_ua.json

  1. {
  2.   "dataset_name": "UA-DETRAC",
  3.   "dataset_path": <the UA-DETRAC dataset folder>,
  4.   "phase": "train", 
      ...
  81.		"log_save_folder": <your log save folder>,
  82.     "image_save_folder": <your image save folder>,
  83.     "weights_save_folder": <your network weights save folder>,
  

• activate your python environment, and run

  cd <project folder>
  python train_ua.py
  

Citation

@inproceedings{ShiJie20,
  author = {Shijie Sun, Naveed Aktar, XiangYu Song, Huansheng Song, Ajmal Mian, Mubarak Shah},
  title = {Simultaneous Detection and Tracking with Motion Modelling for Multiple Object Tracking},
  booktitle = {Proceedings of the European conference on computer vision (ECCV)}},
  year = {2020}

Acknowledge

This work is based on the Pytroch and 3D ResNet. It also inspired by SSD and DAN.

License

The methods provided on this page are published under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 License . This means that you must attribute the work in the manner specified by the authors, you may not use this work for commercial purposes and if you alter, transform, or build upon this work, you may distribute the resulting work only under the same license. If you are interested in commercial usage you can contact us for further options.