Home

Awesome

<div align="center"> <a href="http://camma.u-strasbg.fr/"> <img src="https://github.com/CAMMA-public/rendezvous/raw/main/files/CammaLogo.png" width="18%"> </a> </div> <br/>

PyTorch TensorFlow

Tripnet: Recognition of instrument-tissue interactions in endoscopic videos via action triplets

<i>CI Nwoye, C Gonzalez, T Yu, P Mascagni, D Mutter, J Marescaux, and N Padoy</i>

<img src="https://github.com/CAMMA-public/rendezvous/raw/main/files/examples-1.png" width="100%">

This repository contains the implementation code, inference code, and evaluation scripts.<br />ArXiv paper Journal Publication

Abstract

Recognition of surgical activity is an essential component to develop context-aware decision support for the operating room. In this work, we tackle the recognition of fine-grained activities, modeled as action triplets <instrument, verb, target> representing the tool activity.

To this end, we introduce a new laparoscopic dataset, <i>CholecT40</i>, consisting of 40 videos from the public dataset Cholec80 in which all frames have been annotated using 128 triplet classes.

Furthermore, we present an approach to recognize these triplets directly from the video data. It relies on a module called <i>class activation guide</i>, which uses the instrument activation maps to guide the verb and target recognition. To model the recognition of multiple triplets in the same frame, we also propose a trainable <i>3D interaction space (3Dis)</i>, which captures the associations between the triplet components. Finally, we demonstrate the significance of these contributions via several ablation studies and comparisons to baselines on CholecT40.

<br />

News and Updates

<br />

Model Overview

<img src="files/tripnet.png" width="98%">

The Tripnet model is composed of:

We hope this repo will help researches/engineers in the development of surgical action recognition systems. For algorithm development, we provide training data, baseline models and evaluation methods to make a level playground. For application usage, we also provide a small video demo that takes raw videos as input without any bells and whistles.

<br />

Performance

Results Table

DatasetComponents APAssociation AP
..AP<sub>I</sub>AP<sub>V</sub>AP<sub>T</sub>AP<sub>IV</sub>AP<sub>IT</sub>AP<sub>IVT</sub>
CholecT4089.760.738.335.519.919.0
CholecT4589.959.937.431.827.124.4
CholecT5092.154.533.229.726.420.0
<br />

Installation

Requirements

The model depends on the following libraries:

  1. sklearn
  2. PIL
  3. Python >= 3.5
  4. ivtmetrics
  5. Developer's framework:
    1. For Tensorflow version 1:
      • TF >= 1.10
    2. For Tensorflow version 2:
      • TF >= 2.1
    3. For PyTorch version:
      • Pyorch >= 1.10.1
      • TorchVision >= 0.11
<br />

System Requirements:

The code has been test on Linux operating system. It runs on both CPU and GPU. Equivalence of basic OS commands such as unzip, cd, wget, etc. will be needed to run in Windows or Mac OS.

<br />

Quick Start

<br />

Dataset Zoo

<br />

Data Preparation

<br />

Evaluation Metrics

The ivtmetrics computes AP for triplet recognition. It also support the evaluation of the recognition of the triplet components.

pip install ivtmetrics

or

conda install -c nwoye ivtmetrics

Usage guide is found on pypi.org.

<br />

Running the Model

The code can be run in a trianing mode (-t) or testing mode (-e) or both (-t -e) if you want to evaluate at the end of training :

<br />

Training on CholecT45/CholecT50 Dataset

Simple training on CholecT50 dataset:

python run.py -t  --data_dir="/path/to/dataset" --dataset_variant=cholect50 --version=1

You can include more details such as epoch, batch size, cross-validation and evaluation fold, weight initialization, learning rates for all subtasks, etc.:

python3 run.py -t -e  --data_dir="/path/to/dataset" --dataset_variant=cholect45-crossval --kfold=1 --epochs=180 --batch=64 --version=2 -l 1e-2 1e-3 1e-4 --pretrain_dir='path/to/imagenet/weights'

All the flags can been seen in the run.py file. The experimental setup of the published model is contained in the paper.

<br />

Testing

python3 run.py -e --dataset_variant=cholect45-crossval --kfold 3 --batch 32 --version=1 --test_ckpt="/path/to/model-k3/weights" --data_dir="/path/to/dataset"
<br />

Training on Custom Dataset

Adding custom datasets is quite simple, what you need to do are:

<br />

Model Zoo

PyTorch

NetworkBaseResolutionDatasetData splitLink
TripnetResNet-18LowCholecT50RDVDownload
TripnetResNet-18HighCholecT50RDV[Download]
TripnetResNet-18LowCholecT50ChallengeDownload
TripnetResNet-18LowCholecT50crossval k1Download
TripnetResNet-18LowCholecT50crossval k2Download
TripnetResNet-18LowCholecT50crossval k3Download
TripnetResNet-18LowCholecT50crossval k4Download
TripnetResNet-18LowCholecT50crossval k5Download
TripnetResNet-18LowCholecT45crossval k1Download
TripnetResNet-18LowCholecT45crossval k2Download
TripnetResNet-18LowCholecT45crossval k3Download
TripnetResNet-18LowCholecT45crossval k4Download
TripnetResNet-18LowCholecT45crossval k5Download
<br />

TensorFlow v1

NetworkBaseResolutionDatasetData splitLink
TripnetResNet-18HighCholecT50RDV[Download]
TripnetResNet-18HighCholecT50Challenge[Download]
<br /> <br />

TensorFlow v2

NetworkBaseResolutionDatasetData splitLink
TripnetResNet-18HighCholecT50RDV[Download]
TripnetResNet-18LowCholecT50RDV[Download]
TripnetResNet-18HighCholecT50Challenge[Download]
<br />

Baseline and Ablation Models

TensorFlow v1

ModelAP<sub>i</sub>AP<sub>iv</sub>AP<sub>it</sub>AP<sub>IVT</sub>Link
Naive CNN27.57.56.85.9[Download]
MTL baseline74.614.07.26.4[Download]
Tripnet w/o CAG89.520.612.112.1[Download]
Tripnet w/c untrained 3Dis89.716.77.66.3[Download]

Models are being re-trained and weights are released periodically.

<br /><br />

License

This code, models, and datasets are available for non-commercial scientific research purposes provided by CC BY-NC-SA 4.0 LICENSE attached as LICENSE file. By downloading and using this code you agree to the terms in the LICENSE. Third-party codes are subject to their respective licenses.

<br />

Related Resources

<b> </b> <br />

Citation

If you find this repo useful in your project or research, please consider citing the relevant publications:

@inproceedings{nwoye2020recognition,
   title={Recognition of instrument-tissue interactions in endoscopic videos via action triplets},
   author={Nwoye, Chinedu Innocent and Gonzalez, Cristians and Yu, Tong and Mascagni, Pietro and Mutter, Didier and Marescaux, Jacques and Padoy, Nicolas},
   booktitle={International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI)},
   pages={364--374},
   year={2020},
   organization={Springer}
}

@article{nwoye2021rendezvous,
  title={Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos},
  author={Nwoye, Chinedu Innocent and Yu, Tong and Gonzalez, Cristians and Seeliger, Barbara and Mascagni, Pietro and Mutter, Didier and Marescaux, Jacques and Padoy, Nicolas},
  journal={Medical Image Analysis},
  volume={78},
  pages={102433},
  year={2022}
}

@article{nwoye2022data,
  title={Data Splits and Metrics for Benchmarking Methods on Surgical Action Triplet Datasets},
  author={Nwoye, Chinedu Innocent and Padoy, Nicolas},
  journal={arXiv preprint arXiv:2204.05235},
  year={2022}
}

@article{nwoye2021rendezvous,
  title={Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos},
  author={Nwoye, Chinedu Innocent and Yu, Tong and Gonzalez, Cristians and Seeliger, Barbara and Mascagni, Pietro and Mutter, Didier and Marescaux, Jacques and Padoy, Nicolas},
  journal={Medical Image Analysis},
  volume={78},
  pages={102433},
  year={2022}
}

@article{nwoye2022cholectriplet2021,
  title={CholecTriplet2021: a benchmark challenge for surgical action triplet recognition},
  author={Nwoye, Chinedu Innocent and Alapatt, Deepak and Vardazaryan, Armine ... Gonzalez, Cristians and Padoy, Nicolas},
  journal={arXiv preprint arXiv:2204.04746},
  year={2022}
}

This repo is maintained by CAMMA. Comments and suggestions on models are welcomed. Check this page for updates.