Awesome
<div align="center">OpenDistill3D: Open-World 3D Instance Segmentation with Unified Self-Distillation for Continual Learning and Unknown Class Discovery <p> (ECCV 2024) </p>
</div> <div align="center"> <a href="">Mohamed El Amine Boudjoghra</a><sup>1</sup>, <a href=""> Jean Lahoud</a><sup>1</sup>, <a href="">Hisham Cholakkal</a><sup>1</sup>, <a href="">Rao Muhammad Anwer</a><sup>1,2</sup>, <a href="">Salman Khan</a><sup>1,3</sup>, <a href="">Fahad Khan</a><sup>1,4</sup><sup>1</sup>Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI) <sup>2</sup>Aalto University <sup>3</sup>Australian National University <sup>4</sup>Linköping University
</div> <br>News
<!-- * **25 September 2023**: [Open]() released on arXiv. 📝 -->- 05 July 2024: Code and trained models released. 💻
- 01 July 2024: Paper accepted into <b>ECCV2024</b>. 🥳
Abstract
Open-world 3D instance segmentation is a recently introduced problem with diverse applications, notably in continually learning embodied agents. This task involves segmenting unknown instances and learning new instances when their labels are introduced. However, prior research in the open-world domain has traditionally addressed the two sub-problems, namely continual learning and unknown object identification, separately. This approach has resulted in limited performance on unknown instances and cannot effectively mitigate catastrophic for- getting. Additionally, these methods bypass the utilization of the information stored in the previous version of the continual learning model, instead relying on a dedicated memory to store historical data samples, which inevitably leads to an expansion of the memory budget. In this paper, we argue that continual learning and unknown class identification should be tackled in conjunction. Therefore, we propose a new exemplar- free approach for 3D continual learning and the discovery of unknown classes through self-distillation. Our approach leverages the pseudo-labels generated by the model from the preceding task to improve the unknown predictions during training while simultaneously mitigating catastrophic forgetting. By integrating these pseudo-labels into the continual learning process, we achieve enhanced performance in handling unknown classes. We validate the efficacy of the proposed approach via comprehensive experiments on various splits of the ScanNet200 dataset, showcasing superior performance in continual learning and unknown class retrieval compared to the state-of-the-art.
Figure I: Proposed open-world 3D instance segmentation pipeline.
</div>Installation guide
Kindly check Installation guide on how to setup the Conda environment and how to preprocess ScanNet200 dataset.
Training
<SPLIT_ID> takes the following values A, B, or C.
sh scripts/train/<SPLIT_ID>/opendistill3d.sh
The evaluation results throughout the training will be stored in ./saved/<EXPERIMENT_NAME>/<TASK_ID>/ow_results/<EPOCH>_ow_results.yml
Download trained models and data
<ul> <li>Use <a href="https://mbzuaiac-my.sharepoint.com/:u:/g/personal/mohamed_boudjoghra_mbzuai_ac_ae/EYljqtg_3ShMvV2Dm4uE1l8BofmaXhZWlDHryhUTU6-ocQ?e=SbfVG4">this link</a> for the checkpoints to reproduce our results in Table 1 in the paper.</li> <li>Use <a href="https://mbzuaiac-my.sharepoint.com/:u:/g/personal/mohamed_boudjoghra_mbzuai_ac_ae/EQ6sVhTw8ddNsi2_IwG1U4YBRZzSfuiQeJH1un5zLIu88w?e=XbQrAI">this link</a> for the checkpoints to reproduce our results in Table 7 in the paper.</li> <li>Use <a href="https://mbzuaiac-my.sharepoint.com/:u:/g/personal/mohamed_boudjoghra_mbzuai_ac_ae/EU06GlcxC6JLhTQzdRXIdHYBnLD2NNJXMGKDx6y0Gvy2WQ?e=804ZdD">this link</a> to download the ScanNet200 sampled points used for training models in Table 7 in the paper.</li> </ul>Evaluation
<TASK_ID> takes the following values task1, task2, or task3
sh evaluate.sh <SPLIT_ID> <TASK_ID> <CHECKPOINT_PATH>
BibTeX :pray: