Home

Awesome

Back to the Feature: Classical 3D Features are (Almost) All You Need for 3D Anomaly Detection

Project | Paper <br>

PWC

PWC

PWC

Official PyTorch Implementation for the "Back to the Feature: Classical 3D Features are (Almost) All You Need for 3D Anomaly Detection" paper.


Back to the Feature: Classical 3D Features are (Almost) All You Need for 3D Anomaly Detection<br> Eliahu Horwitz, Yedid Hoshen<br> https://arxiv.org/abs/2203.05550 <br>

Abstract: Despite significant advances in image anomaly detection and segmentation, few methods use 3D information. We utilize a recently introduced 3D anomaly detection dataset to evaluate whether or not using 3D information is a lost opportunity. First, we present a surprising finding: standard color-only methods outperform all current methods that are explicitly designed to exploit 3D information. This is counter-intuitive as even a simple inspection of the dataset shows that color-only methods are insufficient for images containing geometric anomalies. This motivates the question: how can anomaly detection methods effectively use 3D information? We investigate a range of shape representations including hand-crafted and deep-learning-based; we demonstrate that rotation invariance plays the leading role in the performance. We uncover a simple 3D-only method that beats all recent approaches while not using deep learning, external pre-training datasets, or color information. As the 3D-only method cannot detect color and texture anomalies, we combine it with color-based features, significantly outperforming previous state-of-the-art. Our method, dubbed BTF (Back to the Feature) achieves pixel-wise ROCAUC: 99.3% and PRO: 96.4% on MVTec 3D-AD.

Getting Started

Setup

  1. Clone the repo:
git clone https://github.com/eliahuhorwitz/3D-ADS.git
cd 3D-ADS
  1. Create a new environment and install the libraries:
python3.7 -m venv 3d_ads_venv
source 3d_ads_venv/bin/activate
pip install -r requirements.txt
  1. Download and extract the dataset
mkdir datasets && cd datasets
mkdir mvtec3d && cd mvtec3d
wget https://www.mydrive.ch/shares/45920/dd1eb345346df066c63b5c95676b961b/download/428824485-1643285832/mvtec_3d_anomaly_detection.tar.xz
tar -xvf mvtec_3d_anomaly_detection.tar.xz
<br> <br>

Training

We provide the implementations for 7 methods investigated in the paper. These are:

To run all methods on all 10 classes and save the PRO, Image ROCAUC, Pixel ROCAUC results to markdown tables run

python3 main.py

To change which classes are used, see mvtec3d_classes located at data/mvtec3d.py. <br> To change which methods are used, see the PatchCore constructor located at patchcore_runner.py and the METHOD_NAMES variable located at main.py.

Note: The results below are of the raw dataset, see the preprocessing section for the preprocessing code and results (as seen in the paper). Note: The pixel-wise metrics benefit from preprocessing. As such, the unprocessed results are slightly below the ones reported in the paper.

AU PRO Results

MethodBagelCable<br>GlandCarrotCookieDowelFoamPeachPotatoRopeTireMean
RGB iNet0.8980.9480.9270.8720.9270.5550.9020.9310.9030.8990.876
Depth iNet0.7010.5440.7910.8350.5310.1000.8000.5490.8270.1850.586
Raw0.0400.0470.4330.0800.2830.0990.0350.1680.6310.0930.191
HoG0.5180.6090.8570.3420.6670.3400.4760.8930.7000.7390.614
SIFT0.8940.7220.9630.8710.9260.6130.8700.9730.9580.8730.866
FPFH0.9720.8490.9810.9390.9630.6930.9750.9810.9800.9490.928
BTF (Ours)0.9760.9670.9790.9740.9710.8840.9760.9810.9590.9710.964

Image ROCAUC Results

MethodBagelCable<br>GlandCarrotCookieDowelFoamPeachPotatoRopeTireMean
RGB iNet0.8540.8400.8240.6870.9740.7160.7130.5930.9200.7240.785
Depth iNet0.6240.6830.6760.8380.6080.5580.5670.4960.6990.6190.637
Raw0.5780.7320.4440.7980.5790.5370.3470.3060.4390.5170.528
HoG0.5600.6150.6760.4910.5980.4890.5420.5530.6550.4230.560
SIFT0.6960.5530.8240.6960.7950.7730.5730.7460.9360.5530.714
FPFH0.8200.5330.8770.7690.7180.5740.7740.8950.9900.5820.753
BTF (Ours)0.9380.7650.9720.8880.9600.6640.9040.9290.9820.7260.873

Pixel ROCAUC Results

MethodBagelCable<br>GlandCarrotCookieDowelFoamPeachPotatoRopeTireMean
RGB iNet0.9830.9840.9800.9740.9850.8360.9760.9820.9890.9750.966
Depth iNet0.9410.7590.9330.9460.8290.5180.9390.7430.9740.6320.821
Raw0.4040.3060.7720.4570.6410.4780.3540.6020.9050.5580.548
HoG0.7820.8460.9650.6840.8480.7410.7790.9730.9260.9030.845
SIFT0.9740.8620.9930.9520.9800.8620.9550.9960.9930.9710.954
FPFH0.9950.9550.9980.9710.9930.9110.9950.9990.9980.9880.980
BTF (Ours)0.9960.9910.9970.9950.9950.9720.9960.9980.9950.9940.993

<br> <br>

Preprocessing

As mentioned in the paper, the results reported use the preprocessed dataset. <br> While this preprocessing helps in cases where depth images are used, when using the point cloud the results are less pronounced. <br> It may take a few hours to run the preprocessing. Results for the preprocessed dataset are reported below. <br>

To run the preprocessing

python3 utils/preprocessing.py datasets/mvtec3d/

Note: the preprocessing is performed inplace (i.e. overriding the original dataset)

Preprocessed AU PRO Results

MethodBagelCable<br>GlandCarrotCookieDowelFoamPeachPotatoRopeTireMean
RGB iNet0.9020.9480.9290.8730.8910.5700.9030.9330.9090.9050.876
Depth iNet0.7630.6760.8840.8830.8640.3220.8810.8400.8440.6340.759
Raw0.4020.3140.6390.4980.2510.2590.5270.5310.8080.2150.444
HoG0.7120.7610.9320.4870.8330.5200.7430.9490.9160.8580.771
SIFT0.9440.8450.9750.8940.9090.7330.9460.9810.9530.9280.911
FPFH0.9740.8780.9820.9080.8920.7300.9770.9820.9560.9620.924
BTF (Ours)0.9760.9680.9790.9720.9320.8840.9750.9810.9500.9720.959

Preprocessed Image ROCAUC Results

MethodBagelCable<br>GlandCarrotCookieDowelFoamPeachPotatoRopeTireMean
RGB iNet0.8750.8800.7770.7050.9380.7200.7180.6150.8590.6810.777
Depth iNet0.6900.5970.7530.8620.8810.5900.5970.5980.7910.5770.694
Raw0.6270.5070.6000.6540.5730.5240.5320.6120.4120.6780.572
HoG0.4870.5870.6910.5450.6430.5960.5160.5840.5070.4300.559
SIFT0.7220.6400.8920.7620.8290.6780.6230.7540.7670.6030.727
FPFH0.8250.5340.9520.7830.8830.5810.7580.8890.9290.6560.779
BTF (Ours)0.9230.7700.9670.9050.9280.6570.9130.9150.9210.8810.878

Preprocessed Pixel ROCAUC Results

MethodBagelCable<br>GlandCarrotCookieDowelFoamPeachPotatoRopeTireMean
RGB iNet0.9830.9840.980.9740.9730.8510.9760.9830.9870.9770.967
Depth iNet0.9570.9010.9660.9700.9670.7710.9710.9490.9770.8910.932
Raw0.8030.7500.8490.8010.6100.6960.8300.7720.9510.6700.773
HoG0.9110.9330.9850.8230.9360.8620.9230.9870.9800.9550.930
SIFT0.9860.9570.9960.9520.9670.9210.9860.9980.9940.9830.974
FPFH0.9950.9650.9990.9470.9660.9280.9960.9990.9960.9910.978
BTF (Ours)0.9960.9920.9970.9940.9810.9730.9960.9980.9940.9950.992

<br> <br>

Citation

If you find this repository useful for your research, please use the following.

@article{horwitz2022empirical,
  title={An Empirical Investigation of 3D Anomaly Detection and Segmentation},
  author={Horwitz, Eliahu and Hoshen, Yedid},
  journal={arXiv preprint arXiv:2203.05550},
  year={2022}
}

Acknowledgments