Home

Awesome

<img src='https://s3.amazonaws.com/drivendata-public-assets/logo-white-blue.png' height='70'> <br><br> <img src='https://s3.amazonaws.com/drivendata-public-assets/fb-isc-koala-banner.jpg' height='200' alt='Example of original and manipulated image pair from the Challenge.'>

Image Similarity Challenge

Goal of the Competition

Competitors built models to help detect whether a given query image is derived from any of the images in a large reference set.

Content tracing is a crucial component on all social media platforms today, used for such tasks as flagging misinformation and manipulative advertising, preventing uploads of graphic violence, and enforcing copyright protections. But when dealing with the billions of new images generated every day on sites like Facebook, manual content moderation just doesn't scale. They depend on algorithms to help automatically flag or remove bad content.

This competition allowed participants to test their skills in building a key part of that content tracing system, and in so doing contribute to making social media more trustworthy and safe for the people who use it.

<p align="center"> <img src='https://drivendata-public-assets.s3.amazonaws.com/fb-isc-deudeuche.jpg' height='250' align='center' alt='Example of manipulations of a source image.'> </p> <p class="small" align="center"><i>A reference image is manipulated to produce new images. <br>In this challenge competitors built models to detect whether a given query image is derived from a reference set.</i></p>

There were two tracks to this challenge:

Winning Submissions

See below for links to winning submissions' arXiv papers and code.

As a condition for being awarded a prize, all winning solutions are open source under a permissive open source license approved by the Open Source Initiative. See each individual solution's repository for its license information.

Matching Track

PlaceTeam or UserCodePaperScoreSummary of Model
1VisionForceGitHub repositoryD2LV: A Data-Driven and Local-Verification Approach for Image Copy Detection0.8329A "data-driven and local-verification (D^2LV)" approach using pre-training on a set of basic and advanced image augmentations, and a global-local and local-global matching strategy for testing.
2separateGitHub repository2nd Place Solution to Facebook AI Image Similarity Challenge Matching Track0.8291A Vision Transformer approach that uses concatenated query and reference images to learn the relationship between query and reference images directly.
3imgFpGitHub repository3rd Place: A Global and Local Dual Retrieval Solution to Facebook AI Image Similarity Challenge0.7682A global+local recall approach with EsViT for global recall and SIFT point features for local recall.

Descriptor Track

PlaceTeam or UserCodePaperScoreSummary of Model
1lyakaapGitHub repositoryContrastive Learning with Large Memory Bank and Negative Embedding Subtraction for Accurate Copy Detection0.6354Uses an EfficientNet backbone trained with contrastive loss and cross-batch memory, and a training neighbor subtraction step in post-processing.
2S-squareGitHub repositoryProducing augmentation-invariant embeddings from real-life imagery0.5905Ensembles EfficientNet and NFNet backbones using an ArcFace loss function, and applies a sample normalization step in post-processing.
3VisionForceGitHub repositoryBag of Tricks and A Strong baseline for Image Copy Detection0.5788Uses a pretrained Barlow Twins model, yolov5 model to detect overlays, and a descriptor stretching step in post-processing.

Additional resources

Winners announcement: Meet the Winners of the Image Similarity Challenge

"Getting Started" blog post: Facebook AI Image Similarity Challenge - Getting Started