Home

Awesome

A Guided-Based Approach for Deepfake Detection: RGB-Depth Integration via Features Fusion

Introduction

Deep fake technology paves the way for a new generation of super realistic artificial content. While this opens the door to extraordinary new applications, the malicious use of deepfakes allows for far more realistic disinformation attacks than ever before. In this paper, we start from the intuition that generating fake content introduces possible inconsistencies in the depth of the generated images. This extra information provides valuable spatial and semantic cues that can reveal inconsistencies facial generative methods introduce. To test this idea, we evaluate different strategies for integrating depth information into an RGB detector. Our Masked Depthfake Network (MDN) method is 3.2% more robust against common adversarial attacks on average than a typical RGB detector. Furthermore, we show how this technique allows the model to learn more discriminative features than RGB alone.

<img src="https://github.com/gleporoni/rgbd-depthfake/blob/5e2b9bab1af4a1d1bc60e4e123f6dd1e062eafac/doc/DepthFake_2-1.png" width="900">

Main Results

Detection capabilities

Quantitative results obtained on deepfake detection task for RAW and C40 dataset settings when trained on Deepfakes (DF), Face2Face (F2F), FaceSwap (FS), NeuralTextures (NT), and all (ALL) forgeries. The best results for each configuration are reported in bold.

ClassRAWC40
RGBMDNRGBMDN
DF96,0096,8688,1591,26
F2F95,3595,8582,5781,82
FS95,3296,2986,1187,17
NT92,0192,6570,7770,50
ALL95,0294,8782,3782,43

Feature analysis

Heatmaps generated by the GradCam algorithm. The CAM RGB shows the obtained heatmaps for the RGB baseline model, while CAM MDN for the proposed method.

<img src="https://github.com/gleporoni/rgbd-depthfake/blob/e0224b6f1fedeb277743276322425dfe477bcde4/doc/depthfake3-1.png" width="300">

Robustness

Accuracy results obtained on deepfake detection task for RAW and C40 dataset settings when Blur (BLR), Noise (NSE), Rescale (RSC), Translation (TRN), and all Combined (CMB) black box attacks are applied. The best results are in bold.

AttackModelRAWC40
DFF2FFSNTALLDFF2FFSNTALL
BLRRGB50,3254,3050,4549,0079,7466,0367,8258,0256,0079,27
MDN50,9870,3850,6250,7379,6175,1774,6071,9249,7079,90
NSERGB85,8088,7393,8376,2192,0687,2081,0585,5762,0081,41
MDN95,6995,4395,6791,6194,6789,7581,6986,8670,6582,00
RSCRGB60,6360,6050,5853,8974,4873,0270,1964,6757,1880,11
MDN61,6275,6450,6260,8078,7678,3774,9676,5950,4680,38
TRNRGB95,8795,1195,1991,6394,8987,6381,2384,8370,1381,07
MDN96,7595,4996,2392,1394,5189,7681,4686,2469,8181,82
CMBRGB50,3155,2250,3349,4777,9558,7366,3255,7550,7179,67
MDN50,2764,6450,6150,4679,8071,9673,2265,9649,5279,80

Run code

Instruction to run the code:

  1. Run the Docker Image.
./run.sh
  1. Edit the config files accordingly to the experiment you want to execution.

  2. Run the train or test.

python src/$select_a_file$.py

Pretrained models available at: weigths