Awesome
DLC2DLT2DLC
A small set of utilities that allow conversion between the data storage formats of DeepLabCut (DLC) and one of the DLT-based 3D tracking systems: either Ty Hedrick's DigitizingTools in MATLAB, or the Python-based Argus. These functions should allow you to use data previously digitized in a DLT system to create the files needed to train a DLC model, and to import DLC-tracked points back into a DLT 3D calibration to reconstruct 3D points.
While DLC has 3D capabilities (using checkerboards), many field-based setups necessitate the ability to use a wand calibration (easywand or Argus). DLC can be used to auto-track the wand and/or subjects in each video (treated as separate 2D videos), then moved back to DLT for 3d reconstruction.
Multi-camera setups that are not frame-synced (e.g. GoPros) must have offsets calculated (Argus Sync), and adjusted for. These scripts use those offset values.
NOTE: These have been tested on minimal sample data and in a single complete workflow. Be especially careful to make sure your data lines up on the appropriate frame of video, and that frame offsets are working correctly. Please feel free to test, edit, contribute, suggest.
Once they are well tested, I may include these functions directly in Argus, and welcome others to modify these to generate a similar set of MATLAB scripts for DLTdv8.
Getting Started
Download the scripts. Put them somewhere handy. Call them on the command-line. See below.
Usage ouline:
- The videos used for training DeepLabCut must have unique names. If, like me, your DLT videos are all named
cam1.mp4
,cam2.mp4
, etc,renameVids.py
will help give unique names. - If you have data digitized in a DLT program that you want to use as labelled data in DLC:
- Use DLC to add videos to your project and extract frames
- Use
dlt2dlclabels.py
to skip the DLC labeling GUI. This requires DLC version 2.2b8 or higher. - If you have multiple animals visible and are using multianimal DLC, then label each animal in separate DLT data files.
- DLT track names must exactly match bodyparts in DLC
config.yaml
.
- Train DLC, and analyze your videos.
- Use
dlc2dlt.py
to convert DLC coordinates back to DLT style coordinates (this is might be working with multianimal projects...not well tested as of 12 Nov. 2020) - In your DLT program (Argus or DLTdv), either load the videos and new data, or use command line functions in each to perform the 3D reconstruction with your dlt coefficients.
Detailed Usage
In all relevant functions, the flag -flipy
defaults to True
which is necessary if your data are from Argus, or DLTdv versions prior to 8.0. In those packages, the coordinate origin is in the lower left of the video, but the computer vision standard (used by DeepLabCut and by DLTdv8) is for the origin to be in the upper left. If you are using DLTdv8, just add -flipy False
to the call.
dlt2dlclabels
Many of us have lots of data already digitized in a DLT-calibrated environment. Those already digitized points can be used in place of "labeling" video in DeepLabCut. That's what dlt2dlc.py and dlt2dlclabels.py are for. It "essentially" replaces/parallels the labelling function in DLC.
python dlt2dlclabels.py --help
usage: dlt2dlclabels.py [-h] [-config CONFIG] [-xy XY] [-vid VID] [-cnum CNUM]
[-flipy FLIPY] [-offset OFFSET] [-ind IND] [-addbp ADDBP]
convert Argus or DLTdv to DLC labeled frames for training
optional arguments:
-h, --help show this help message and exit
-config CONFIG input path to DLC config file
-xy XY input path to xypts file
-vid VID input path to video file
-cnum CNUM enter 1-indexed camera number for extraction
-flipy FLIPY flip y coordinates - necessary for DLTdv versions 1-7 and
Argus, set to False for DLTdv8, default = True
-offset OFFSET enter offset of chosen camera as integer
-ind IND enter 0-indexed individual number from config file.
xypts.csv must have only one indiv digitized
-addbp if new tracks/bodyparts were digitized in Argus/DLTdv, add
this flag to add those to already labeled data in DLC.
Not including this flag means the DLT data will overwrite any existing DLC labels for that video.
For example,
python dlt2dlclables.py -config /path/to/deeplabcut/project/config.yaml -xy /path/to/xypts/file-xypts.csv -vid /path/to/xypts/cam2.mp4 -cnum 2 -flipy False -offset -21 -ind 1 -addbp
will take the data and video from the second camera (with a -21 frame offset) in a three camera setup, with individual 2 digitized, and save those data in the CollectedData
file of the relevant camera in the DLC project. With the -addbp
flag, it will not overwrite existing data in that file.
dlc2dlt
If you have some beautifully accurate 3d DLT calibrations, but are sick of manually digitizing each video, then train and use DLC to digitize your videos independently. Once they are digitized, dlc2dlt will create the needed files to complete DLT reconstruction. It essentially allows you to use DLC to replace the clicking part of DLT.
python dlc2dlt.py --help
usage: dlc2dlt.py [-h] [-dlctracks DLCTRACKS [DLCTRACKS ...]]
[-newpath NEWPATH] [-flipy FLIPY]
[-offsets OFFSETS [OFFSETS ...]]
convert argus to DLC labeled frames for training
optional arguments:
-h, --help show this help message and exit
-dlctracks DLCTRACKS [DLCTRACKS ...]
input paths of DLC tracked coordinates (h5) in order
used in DLT calibration, each path separated by a
space
-newpath NEWPATH enter a path for saving, will overwrite if it already
exists, should not be in DLC project folder, should
end with filename prefix
-flipy FLIPY flip y coordinates - necessar for Argus and DLTdv
versions 1-7, set to False for DLTdv8
-offsets OFFSETS [OFFSETS ...]
enter offsets as space separated list including first
camera e.g.: -offsets 0 -12 2
-like LIKE set a likelihood threshold so that only good DLC fits
are allowed to pass. 0.9 by default.
for example, if you have a three camera DLT setup, and each camera has been tracked in DLC, and you want only the very best fits...
python dlc2dlt.py -dlctracks /pathtoDLCproectfolder/videos/cam1tracked.h5 /pathtoDLCproectfolder/videos/cam2tracked.h5 /pathtoDLCproectfolder/videos/cam3tracked.h5 -newpath /a/path/to/somewhere/else/trial01DLCtracked -offsets 0 -20 34 -like 0.9999
will make the standard DLT files (4 of them) at new path, named trial01DLCtracked-xypts.csv, trial01DLCtracked-xyzpts.csv, trial01DLCtracked-xyzres.csv, trial01DLCtracked-offsets.csv. The -xyzpts and -xyzres files will be empty, but they must be created to be able to load the data into DLTdv or Argus.
To get xyz pts, load the new -xypts.csv as data in DLTdv or Argus as if you digitized it there, load your DLT coefficients, camera profiles, check the data, and save. Note that both DLTdv and Argus have command-line functions (dlt_reconstruct) to get the 3d points without loading in the GUI.
Authors
- Brandon E. Jackson, Ph.D.
References
If you use this code, for now, please cite the appropriate DeepLabCut and Argus or DLTdv papers.
License
This project is licensed under the GNU Lesser General Public License v3.0; see the LICENSE.md file for details. Note that the software is provided "as is", without warranty of any kind, express or implied. If you use the code, please cite DeepLabCut and either DLTdv or Argus.