Awesome

Awesome-Sketch-Based-Applications

A collection of sketch based applications.

Feel free to create a PR or an issue.

Outline

0. Survey
1. Sketch Based Image Synthesis
- 1.1 Automatic Synthesis
- 1.2 Style-conditioned
- 1.3 Text-conditioned
2. Sketch Based Image Editing
3. Sketch Based Image Retrieval (SBIR)
4. Sketch Based 3D Shape Retrieval
5. Sketch Based 3D Shape Modeling
6. Sketch Based Garment Design
7. Sketch Completion
8. Sketch Restoration, Retargeting and Inpainting
9. Sketch and Shading
10. Sketch Enhancement / Beautification
11. Sketch-Guided Object Localization
12. Sketch-Guided Video Synthesis
13. Sketch Recognition
14. Sketch Segmentation and Perceptual Grouping
15. Sketch Representation Learning
16. Sketch and Visual Correspondence
17. Sketch Animation/Inbetweening
18. Sketch and AR/VR
19. Sketch Based Incremental Learning
20. Sketch Quality Measurement
21. Cloud Augmentation with Sketches
22. Sketch and Re-identification
23. Sketch-based Salient Object Detection
24. Study on Strokes

0. Survey

Paper	Source	Code/Project Link
Deep Learning for Free-Hand Sketch: A Survey	TPAMI 2022	[code]

1. Sketch Based Image Synthesis

1.1 Automatic Synthesis

Natural Image or Object

Paper	Source	Code/Project Link
SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis	CVPR 2018	[code]
Image Generation from Sketch Constraint Using Contextual GAN	ECCV 2018	[code] [project]
Multi-Instance Sketch to Image Synthesis With Progressive Generative Adversarial Networks	IEEE Access 2019
Interactive Sketch & Fill: Multiclass Sketch-to-Image Translation	ICCV 2019	[code] [project]
SketchyCOCO: Image Generation from Freehand Scene Sketches	CVPR 2020	[code]
Unsupervised Sketch-to-Photo Synthesis	ECCV 2020	[code] [project]
Sketch Your Own GAN	ICCV 2021	[code] [webpage]
Adversarial Open Domain Adaption for Sketch-to-Photo Synthesis	WACV 2022	[code]
Customizing GAN Using Few-Shot Sketches	ACM MM 2022
DiffSketching: Sketch Control Image Synthesis with Diffusion Models	BMVC 2022	[code]
MaskSketch: Unpaired Structure-guided Masked Image Generation	CVPR 2023	[project] [code]
Picture that Sketch: Photorealistic Image Generation from Abstract Sketches	CVPR 2023	[project] [code]

Human Face / Portrait / Body

Paper	Source	Code/Project Link
LinesToFacePhoto: Face Photo Generation from Lines with Conditional Self-Attention Generative Adversarial Network	ACM MM 2019
DeepFaceDrawing: Deep Generation of Face Images from Sketches	SIGGRAPH 2020	[project] [code]
DeepFacePencil: Creating Face Images from Freehand Sketches	ACM MM 2020	[project] [code]
Controllable Sketch-to-Image Translation for Robust Face Synthesis	TIP 2021
DrawingInStyles: Portrait Image Generation and Editing with Spatially Conditioned StyleGAN	TVCG 2022
DeepPortraitDrawing: Generating Human Body Images from Freehand Sketches	C&G 2023
Semantics-Preserving Sketch Embedding for Face Generation	TMM 2023	[project] [code]
Parsing-Conditioned Anime Translation: A New Dataset and Method	TOG 2023	[code]
Sketch2Human: Deep Human Generation with Disentangled Geometry and Appearance Constraints	TVCG 2024

3D image

Paper	Source	Code/Project Link
3D-aware Conditional Image Synthesis	CVPR 2023	[project] [code]

1.2 Style-conditioned

Paper	Source	Code/Project Link
TextureGAN: Controlling Deep Image Synthesis with Texture Patches	CVPR 2018	[code]
Multimodal Unsupervised Image-to-Image Translation	ECCV 2018	[code]
SketchPatch: Sketch Stylization via Seamless Patch-level Synthesis	SIGGRAPH Asia 2020
Self-Supervised Sketch-to-Image Synthesis	AAAI 2021	[code]
CoGS: Controllable Generation and Search from Sketch and Style	ECCV 2022
Adaptively-Realistic Image Generation from Stroke and Sketch with Diffusion Model	WACV 2023	[project] [code]
DemoCaricature: Democratising Caricature Generation with a Rough Sketch	CVPR 2024	[project] [code]
SMFS-GAN: Style-Guided Multi-class Freehand Sketch-to-Image Synthesis	CGF 2024

1.3 Text-conditioned

Paper	Source	Code/Project Link
Sketch-Guided Text-to-Image Diffusion Models	SIGGRAPH 2023	[project]
Adding Conditional Control to Text-to-Image Diffusion Models	ICCV 2023	[code]
UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild	NeurIPS 2023	[code] [project]
Stable Doodle (ClipDrop)		[demo]
CustomSketching: Sketch Concept Extraction for Sketch-based Image Synthesis and Editing	PG 2024
Block and Detail: Scaffolding Sketch-to-Image Generation	UIST 2024	[webpage] [code]
One-Step Image Translation with Text-to-Image Models	arxiv 24.03	[code]
It's All About Your Sketch: Democratising Sketch Control in Diffusion Models	CVPR 2024	[code]
T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models	AAAI 2024	[code]
Training-Free Sketch-Guided Diffusion with Latent Optimization	arxiv 24.09
Scribble-Guided Diffusion for Training-free Text-to-Image Generation	arxiv 24.09	[code]

2. Sketch Based Image Editing

Arbitrary Image

Paper	Source	Code/Project Link
Free-Form Image Inpainting with Gated Convolution (DeepFill v2)	ICCV 2019	[project] [code]
Fashion Editing with Multi-scale Attention Normalization	1906.00884
DeFLOCNet: Deep Image Editing via Flexible Low-level Controls	CVPR 2021	[code]
DeepSIM: Image Shape Manipulation from a Single Augmented Training Sample	ICCV 2021	[code] [project]
SketchEdit: Mask-Free Local Image Manipulation with Partial Sketches	CVPR 2022	[code] [project]
Draw2Edit: Mask-Free Sketch-Guided Image Manipulation	ACM MM 2023	[code]
Towards Interactive Image Inpainting via Robust Sketch Refinement	TMM 2024	[code] [project]

Human Face / Portrait / Hair

Paper	Source	Code/Project Link
FaceShop: Deep Sketch-based Face Image Editing	SIGGRAPH 2018	[project]
CaricatureShop: Personalized and Photorealistic Caricature Sketching	TVCG 2018
Sparse, Smart Contours to Represent and Edit Images	CVPR 2018	[project]
Example-Guided Style-Consistent Image Synthesis from Semantic Labeling	CVPR 2019	[code]
SC-FEGAN: Face Editing Generative Adversarial Network With User's Sketch and Color	ICCV 2019	[code]
Intuitive, Interactive Beard and Hair Synthesis with Generative Models	CVPR 2020
Deep Plastic Surgery: Robust and Controllable Image Editing with Human-Drawn Sketches	ECCV 2020	[code] [project]
DeepFaceEditing: Deep Face Generation and Editing with Disentangled Geometry and Appearance Control	SIGGRAPH 2021	[code] [project]
SketchHairSalon: Deep Sketch-based Hair Image Synthesis	SIGGRAPH Asia 2021	[project]
Paint2Pix: Interactive Painting based Progressive Image Synthesis and Editing	ECCV 2022	[code] [project]
NeRFFaceEditing: Disentangled Face Editing in Neural Radiance Fields	SIGGRAPH Asia 2022	[project]
SketchFaceNeRF: Sketch-based Facial Generation and Editing in Neural Radiance Fields	SIGGRAPH 2023

Anime Editing

Paper	Source	Code/Project Link
AniFaceDrawing: Anime Portrait Exploration during Your Sketching	SIGGRAPH 2023	[project]

Video Editing

Paper	Source	Code/Project Link
DeepFaceVideoEditing: Sketch-based Deep Editing of Face Videos	SIGGRAPH 2022	[project]

3. Sketch Based Image Retrieval (SBIR)

Object-level

Paper	Source	Code/Project Link
Sketch Me That Shoe	CVPR 2016	[code-caffe] [code-tf] [project]
Deep Multi-task Attribute-driven Ranking for Fine-grained Sketch-based Image Retrieval	BMVC 2016
Deep Sketch Hashing: Fast Free-hand Sketch-Based Image Retrieval	CVPR 2017	[code]
Deep Spatial-Semantic Attention for Fine-Grained Sketch-Based Image Retrieval	ICCV 2017	[project]
Zero-Shot Sketch-Image Hashing	CVPR 2018
SketchMate: Deep Hashing for Million-Scale Human Sketch Retrieval	CVPR 2018
Generative Domain-Migration Hashing for Sketch-to-Image Retrieval	ECCV 2018	[code]
A Zero-Shot Framework for Sketch Based Image Retrieval	ECCV 2018	[code]
Generalising Fine-Grained Sketch-Based Image Retrieval	CVPR 2019
Doodle to Search: Practical Zero-Shot Sketch-Based Image Retrieval	CVPR 2019	[project] [code]
LiveSketch: Query Perturbations for Guided Sketch-Based Visual Search	CVPR 2019
Semantically Tied Paired Cycle Consistency for Zero-Shot Sketch-Based Image Retrieval	CVPR 2019	[code]
Learning Structural Representations via Dynamic Object Landmarks Discovery for Sketch Recognition and Retrieval	TIP 2019
Semantic-Aware Knowledge Preservation for Zero-Shot Sketch-Based Image Retrieval	ICCV 2019
TC-Net for iSBIR: Triplet Classification Network for Instance-level Sketch Based Image Retrieval	ACM MM 2019
Sketch-Based Image Retrieval With Multi-Clustering Re-Ranking	TCSVT 2019
Semi-Heterogeneous Three-Way Joint Embedding Network for Sketch-Based Image Retrieval	TCSVT 2019
Zero-Shot Sketch-Based Image Retrieval via Graph Convolution Network	AAAI 2020
Sketch Less for More: On-the-Fly Fine-Grained Sketch Based Image Retrieval	CVPR 2020	[code]
Solving Mixed-modal Jigsaw Puzzle for Fine-Grained Sketch-Based Image Retrieval	CVPR 2020
Fine-Grained Instance-Level Sketch-Based Image Retrieval	IJCV 2020
StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrieval	CVPR 2021
More Photos are All You Need: Semi-Supervised Learning for Fine-Grained Sketch-Based Image Retrieval	CVPR 2021	[code]
DLA-Net for FG-SBIR: Dynamic Local Aligned Network for Fine-Grained Sketch-Based Image Retrieval	ACM MM 2021
Domain-Aware SE Network for Sketch-based Image Retrieval with Multiplicative Euclidean Margin Softmax	ACM MM 2021	[code]
Relationship-Preserving Knowledge Distillation for Zero-Shot Sketch Based Image Retrieval	ACM MM 2021
Transferable Coupled Network for Zero-Shot Sketch-Based Image Retrieval	TPAMI 2021	[project]
TVT: Three-Way Vision Transformer through Multi-Modal Hypersphere Learning for Zero-Shot Sketch-Based Image Retrieval	AAAI 2022
Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval	CVPR 2022	[code]
Sketch3T: Test-time Training for Zero-Shot SBIR	CVPR 2022
Augmented Multi-Modality Fusion for Generalized Zero-Shot Sketch-based Visual Retrieval	TIP 2022	[code]
Adaptive Fine-Grained Sketch-Based Image Retrieval	ECCV 2022	[code]
Conditional Stroke Recovery for Fine-Grained Sketch-Based Image Retrieval	ECCV 2022	[code]
A Sketch Is Worth a Thousand Words: Image Retrieval with Text and Sketch	ECCV 2022	[code] [project]
Multi-Level Region Matching for Fine-Grained Sketch-Based Image Retrieval	ACM MM 2022	[code]
Prototype-based Selective Knowledge Distillation for Zero-Shot Sketch Based Image Retrieval	ACM MM 2022
DLI-Net: Dual Local Interaction Network for Fine-Grained Sketch-Based Image Retrieval	TCSVT 2022	[code]
Data-Free Sketch-Based Image Retrieval	CVPR 2023	[code]
Zero-Shot Everything Sketch-Based Image Retrieval, and in Explainable Style	CVPR 2023	[code]
Exploiting Unlabelled Photos for Stronger Fine-Grained SBIR	CVPR 2023	[project]
CLIP for All Things Zero-Shot Sketch-Based Image Retrieval, Fine-Grained or Not	CVPR 2023	[project]
Cross-Domain Alignment for Zero-Shot Sketch-Based Image Retrieval	TCSVT 2023
Semi-transductive Learning for Generalized Zero-Shot Sketch-Based Image Retrieval	AAAI 2023
Text-to-Image Diffusion Models are Great Sketch-Photo Matchmakers	CVPR 2024	[project]
You'll Never Walk Alone: A Sketch and Text Duet for Fine-Grained Image Retrieval	CVPR 2024	[project]
How to Handle Sketch-Abstraction in Sketch-Based Image Retrieval?	CVPR 2024	[project]
Asymmetric Mutual Alignment for Unsupervised Zero-Shot Sketch-Based Image Retrieval	AAAI 2024
Composite Sketch+Text Queries for Retrieving Objects with Elusive Names and Complex Interactions	AAAI 2024	[project]
Freeview Sketching: View-Aware Fine-Grained Sketch-Based Image Retrieval	ECCV 2024
Elevating All Zero-Shot Sketch-Based Image Retrieval Through Multimodal Prompt Learning	ECCV 2024	[code]
Zero-Shot Sketch Based Image Retrieval via Modality Capacity Guidance	IJCAI-24	[code]

Scene-level

Paper	Source	Code/Project Link
SceneSketcher: Fine-Grained Image Retrieval with Scene Sketches	ECCV 2020
Partially Does It: Towards Scene-Level FG-SBIR with Partial Input	CVPR 2022
SceneSketcher-v2: Fine-Grained Scene-Level Sketch-Based Image Retrieval using Adaptive GCNs	TIP 2022
FS-COCO: Towards Understanding of Freehand Sketches of Common Objects in Context	ECCV 2022	[code] [Dataset]
Scene-Level Sketch-Based Image Retrieval with Minimal Pairwise Supervision	AAAI 2023
FreestyleRet: Retrieving Images from Style-Diversified Queries	ECCV 2024	[code]

Video Retrieval

Paper	Source	Code/Project Link
Fine-Grained Instance-Level Sketch-Based Video Retrieval	TCSVT 2020
Fine-Grained Video Retrieval with Scene Sketches	TIP 2023	[project]

4. Sketch Based 3D Shape Retrieval

Paper	Source	Code/Project Link
Sketch-based 3D Shape Retrieval using Convolutional Neural Networks	CVPR 2015
Learning Cross-Domain Neural Networks for Sketch-Based 3D Shape Retrieval	AAAI 2016
Deep Correlated Metric Learning for Sketch-based 3D Shape Retrieval	AAAI 2017
Learning Barycentric Representations of 3D Shapes for Sketch-based 3D Shape Retrieval	CVPR 2017
Deep Cross-modality Adaptation via Semantics Preserving Adversarial Learning for Sketch-based 3D Shape Retrieval	ECCV 2018
Unsupervised Learning of 3D Model Reconstruction from Hand-Drawn Sketches	ACMMM 2018
Towards 3D VR-Sketch to 3D Shape Retrieval	3DV 2020	[code] [project]
Uncertainty Learning for Noise Resistant Sketch-Based 3D Shape Retrieval	TIP 2021
Towards Fine-Grained Sketch-Based 3D Shape Retrieval	TIP 2021
Domain Disentangled Generative Adversarial Network for Zero-Shot Sketch-Based 3D Shape Retrieval	AAAI 2022
Retrieval-Specific View Learning for Sketch-to-Shape Retrieval	TMM 2023
Doodle to Object: Practical Zero-Shot Sketch-Based 3D Shape Retrieval	AAAI 2023	[code]
Democratising 2D Sketch to 3D Shape Retrieval Through Pivoting	ICCV 2023

5. Sketch Based 3D Shape Modeling

Free-hand sketch

Paper	Source	Code/Project Link
3D Shape Reconstruction from Sketches via Multi-view Convolutional Networks	3DV 2017	[code] [project]
Shape Synthesis from Sketches via Procedural Models and Convolutional Networks	TVCG 2017
DeepSketch2Face: A Deep Learning Based Sketching System for 3D Face and Caricature Modeling	SIGGRAPH 2017	[project] [code]
BendSketch: Modeling Freeform Surfaces Through 2D Sketching	SIGGRAPH 2017	[project]
Robust Flow-Guided Neural Prediction for Sketch-Based Freeform Surface Modeling	SIGGRAPH Asia 2018	[code] [project]
DeepSketchHair: Deep Sketch-based 3D Hair Modeling	TVCG 2019
Interactive Liquid Splash Modeling by User Sketches	SIGGRAPH Asia 2020
Monster Mash: A Single-View Approach to Casual 3D Modeling and Animation	SIGGRAPH Asia 2020	[project] [code] [demo]
SAniHead: Sketching Animal-like 3D Character Heads Using a View-surface Collaborative Mesh Generative Network	TVCG 2020
Towards Practical Sketch-Based 3D Shape Generation: The Role of Professional Sketches	TCSVT 2020
Sketch2Model: View-Aware 3D Modeling from Single Free-Hand Sketches	CVPR 2021
Sketch2Mesh: Reconstructing and Editing 3D Shapes from Sketches	ICCV 2021
Real-time Skeletonization for Sketch-based Modeling	SMI 2021	[code]
Sketch2Pose: Estimating a 3D Character Pose from a Bitmap Sketch	SIGGRAPH 2022	[project] [code]
Sketch2PQ: Freeform Planar Quadrilateral Mesh Design via a Single Sketch	TVCG 2022
Structure-aware Editable Morphable Model for 3D Facial Detail Animation and Manipulation	ECCV 2022	[code]
SketchSampler: Sketch-based 3D Reconstruction via View-dependent Depth Sampling	ECCV 2022	[code]
Deep Reconstruction of 3D Smoke Densities from Artist Sketches	EG 2022	[project] [code]
A Diffusion-ReFinement Model for Sketch-to-Point Modeling	ACCV 2022	[code]
RaBit: Parametric Modeling of 3D Biped Cartoon Characters with a Topological-consistent Dataset	CVPR 2023	[project]
SketchMetaFace: A Learning-based Sketching Interface for High-fidelity 3D Character Face Modeling	TVCG 2023	[project] [code]
Reality3DSketch: Rapid 3D Modeling of Objects from Single Freehand Sketches	TMM 2023
GA-Sketching: Shape Modeling from Multi-View Sketching with Geometry-Aligned Deep Implicit Functions	PG 2023	[code]
Doodle Your 3D: From Abstract Freehand Sketches to Precise 3D Shapes	CVPR 2024	[project] [code]
SENS: Part-Aware Sketch-based Implicit Neural Shape Modeling	EG 2024	[code] [project]
3D Reconstruction from a Single Sketch via View-dependent Depth Sampling	TPAMI 2024	[code]
S2TD-Face: Reconstruct a Detailed 3D Face with Controllable Texture from a Single Sketch	ACM MM 2024	[code]
Sketch2Vox: Learning 3D Reconstruction from a Single Monocular Sketch	ECCV 2024

Scene Generation

Paper	Source	Code/Project Link
Sketch2Scene: Automatic Generation of Interactive 3D Game Scenes from User's Casual Sketches	arxiv 24.08	[webpage]
GroundUp: Rapid Sketch-Based 3D City Massing	ECCV 2024	[code]
External Knowledge Enhanced 3D Scene Generation from Sketch	ECCV 2024

NeRF

Paper	Source	Code/Project Link
Sketch2NeRF: Multi-view Sketch-guided Text-to-3D Generation	arxiv 24.01
SKED: Sketch-guided Text-based 3D Editing	ICCV 2023	[project] [code]
SketchDream: Sketch-based Text-to-3D Generation and Editing	SIGGRAPH 2024	[project]

CAD sketch

Paper	Source	Code/Project Link
Lifting Freehand Concept Sketches into 3D	SIGGRAPH Asia 2020	[project] [code]
Sketch2CAD: Sequential CAD Modeling by Sketching in Context	SIGGRAPH Asia 2020	[project] [code]
Free2CAD: Parsing Freehand Drawings into CAD Commands	SIGGRAPH 2022	[project] [code]
Symmetry-driven 3D Reconstruction From Concept Sketches	SIGGRAPH 2022	[project]
Piecewise-smooth Surface Fitting Onto Unstructured 3D Sketches	SIGGRAPH 2022	[project]
Reconstruction of Machine-Made Shapes from Bitmap Sketches	SIGGRAPH Asia 2023	[project]
CAD-SIGNet: CAD Language Inference from Point Clouds using Layer-wise Sketch Instance Guided Attention	CVPR 2024
SfmCAD: Unsupervised CAD Reconstruction by Learning Sketch-based Feature Modeling Operations	CVPR 2024	[code]

6. Sketch Based Garment Design

Paper	Source	Code/Project Link
Sketching Folds: Developable Surfaces from Non-Planar Silhouettes	TOG 2015	[project]
FoldSketch: Enriching Garments with Physically Reproducible Folds	SIGGRAPH 2018	[project]
Learning a Shared Shape Space for Multimodal Garment Design	SIGGRAPH Asia 2018	[project]
Garment Ideation: Iterative view-aware sketch-based garment modeling	3DV 2022	[code]
Controllable Visual-Tactile Synthesis	ICCV 2023	[project] [code]
Controllable Garment Image Synthesis Integrated with Frequency Domain Features	PG 2023
Toward Intelligent Interactive Design: A Generation Framework Based on Cross-domain Fashion Elements	ACM MM 2023
FashionDiff: A Controllable Diffusion Model Using Pairwise Fashion Elements for Intelligent Design	ACM MM 2023
TexControl: Sketch-Based Two-Stage Fashion Image Generation Using Diffusion Model	NICOGRAPH 2024

7. Sketch Completion

Sketch completion

Paper	Source	Code/Project Link
SketchGAN: Joint Sketch Completion and Recognition with Generative Adversarial Network	CVPR 2019
SketchHealer: A Graph-to-Sequence Network for Recreating Partial Human Sketches	BMVC 2020	[code]
Sketch-BERT: Learning Sketch Bidirectional Encoder Representation from Transformers by Self-supervised Learning of Sketch Gestalt	CVPR 2020	[code]
Generative Sketch Healing	IJCV 2022

Sketch gap / connectivity detection

Paper	Source	Code/Project Link
Joint Gap Detection and Inpainting of Line Drawings	CVPR 2017	[project] [code]
Detecting Viewer-Perceived Intended Vector Sketch Connectivity	SIGGRAPH 2022	[project]

8. Sketch Restoration, Retargeting and Inpainting

Manga

Paper	Source	Code/Project Link
Exploiting Aliasing for Manga Restoration	CVPR 2021	[webpage] [code]
Seamless Manga Inpainting with Semantics Awareness	SIGGRAPH 2021	[webpage] [code]
Screentone-Preserved Manga Retargeting	arxiv 22.03
Manga Rescreening with Interpretable Screentone Representation	arxiv 23.06

Hand-Drawn Drawings

Paper	Source	Code/Project Link
Restoration of Hand-Drawn Architectural Drawings using Latent Space Mapping with Degradation Generator	CVPR 2023

9. Sketch and Shading

Paper	Source	Code/Project Link
Learning to Shadow Hand-drawn Sketches	CVPR 2020	[project] [code]
SmartShadow: Artistic Shadow Drawing Tool for Line Drawings	ICCV 2021	[project]

10. Sketch Enhancement / Beautification

Sketch Enhancement

Paper	Source	Code/Project Link
SketchMan: Learning to Create Professional Sketches	ACM MM 2020	[code]

Sketch Beautification

Paper	Source	Code/Project Link
Sketch Beautification: Learning Part Beautification and Structure Refinement for Sketches of Man-made Objects	TVCG 2024
PICASSO: A Feed-Forward Framework for Parametric Inference of CAD Sketches via Rendering Self-Supervision	arxiv 24.07

Super Resolution

Paper	Source	Code/Project Link
APISR: Anime Production Inspired Real-World Anime Super-Resolution	CVPR 2024	[code]

11. Sketch-Guided Object Localization

Paper	Source	Code/Project Link
Sketch-Guided Object Localization in Natural Images	ECCV 2020	[project] [code]
Localizing Infinity-shaped fishes: Sketch-guided object localization in the wild	arxiv 21.09	[code]
What Can Human Sketches Do for Object Detection?	CVPR 2023	[project]
Sketch-based Video Object Segmentation: Benchmark and Analysis	BMVC 2023	[code]
Query-guided Attention in Vision Transformers for Localizing Objects Using a Single Sketch	WACV 2024	[project] [code]

12. Sketch-Guided Video Synthesis

Paper	Source	Code/Project Link
Deep Sketch-guided Cartoon Video Inbetweening	TVCG 2021	[code]
ToonCrafter: Generative Cartoon Interpolation	SIGGRAPH Asia 2024	[code] [webpage]
LVCD: Reference-based Lineart Video Colorization with Diffusion Models	SIGGRAPH Asia 2024	[webpage]

13. Sketch Recognition

Paper	Source	Code/Project Link
Sketch-a-Net that Beats Humans	BMVC 2015	[code]
Sketch-a-Net: A Deep Neural Network that Beats Humans	IJCV 2017	[code]
Deep Self-Supervised Representation Learning for Free-Hand Sketch	TCSVT 2020	[code]
S3Net:Graph Representational Network For Sketch Recognition	ICME 2020	[code]
Sketchformer: Transformer-based Representation for Sketched Structure	CVPR 2020	[code]
Sketch-BERT: Learning Sketch Bidirectional Encoder Representation from Transformers by Self-supervised Learning of Sketch Gestalt	CVPR 2020	[code]
Multi-Graph Transformer for Free-Hand Sketch Recognition	TNNLS 2021	[code]
Sketch-R2CNN: An RNN-Rasterization-CNN Architecture for Vector Sketch Recognition	TVCG 2021	[code]
Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting	CVPR 2021	[code]
SketchAA: Abstract Representation for Abstract Sketches	ICCV 2021
SketchLattice: Latticed Representation for Sketch Manipulation	ICCV 2021	[code]
Multi-Stage Spatio-Temporal Networks for Robust Sketch Recognition	TIP 2022

14. Sketch Segmentation and Perceptual Grouping

Semantic / Instance Segmentation

<table> <tr> <td>Type</td> <td>Paper</td> <td>Source</td> <td>Code/Project Link</td> </tr> <tr> <td rowspan=4">Pixelwise</td> <td> <a href="https://ieeexplore.ieee.org/abstract/document/8784880">SPFusionNet: Sketch Segmentation Using Multi-modal Data Fusion</a> </td> <td> ICME 2019 </td> <td> </td> </tr> <tr> <td> <a href="http://openaccess.thecvf.com/content_ECCV_2018/papers/Changqing_Zou_SketchyScene_Richly-Annotated_Scene_ECCV_2018_paper.pdf">SketchyScene: Richly-Annotated Scene Sketches</a> (scene-level) </td> <td> ECCV 2018 </td> <td> <a href="https://github.com/SketchyScene/SketchyScene">[code]</a> </td> </tr> <tr> <td> <a href="https://ieeexplore.ieee.org/abstract/document/9686584">Exploring Local Detail Perception for Scene Sketch Semantic Segmentation</a> (scene-level) </td> <td> TIP 2022 </td> <td> </td> </tr> <tr> <td> <a href="https://arxiv.org/abs/2312.12463">Open Vocabulary Semantic Scene Sketch Understanding</a> (scene-level) </td> <td> CVPR 2024 </td> <td> <a href="https://github.com/AhmedBourouis/Scene-Sketch-Segmentation">[code]</a> <a href="https://ahmedbourouis.github.io/Scene_Sketch_Segmentation/">[project]</a> </td> </tr> <tr> <td rowspan="11">Stroke-level</td> <td> <a href="https://www.microsoft.com/en-us/research/wp-content/uploads/2012/01/16-2012-eccv-sketch-segmenation.pdf">Free Hand-Drawn Sketch Segmentation</a> </td> <td> ECCV 2012 </td> <td> </td> </tr> <tr> <td> <a href="https://scholars.cityu.edu.hk/files/15839162/9_5915645_TOG_Data_driven_postprint.pdf">Data-driven Segmentation and Labeling of Freehand Sketches</a> </td> <td> SIGGRAPH Asia 2014 </td> <td> <a href="http://sweb.cityu.edu.hk/hongbofu/projects/SketchSegmentationLabeling_SA14/src_global_interpretation.zip">[code]</a> <a href="http://sweb.cityu.edu.hk/hongbofu/projects/SketchSegmentationLabeling_SA14">[project]</a> <a href="http://sweb.cityu.edu.hk/hongbofu/projects/SketchSegmentationLabeling_SA14/Sketch_dataset.zip">[dataset]</a> </td> </tr> <tr> <td> <a href="http://homes.esat.kuleuven.be/~konijn/publications/2016/a151-schneider.pdf">Example-Based Sketch Segmentation and Labeling Using CRFs</a> </td> <td> TOG 2016 </td> <td> </td> </tr> <tr> <td> <a href="https://ieeexplore.ieee.org/abstract/document/8766108">SketchSegNet+: An End-to-End Learning of RNN for Multi-Class Sketch Semantic Segmentation</a> </td> <td> IEEE Access 2019 </td> <td> </td> </tr> <tr> <td> <a href="http://sweb.cityu.edu.hk/hongbofu/doc/fast_sketch_segmentation_CGA2019.pdf">Fast Sketch Segmentation and Labeling With Deep Learning</a> </td> <td> CGA 2019 </td> <td> </td> </tr> <tr> <td> <a href="https://arxiv.org/pdf/1901.03427.pdf">Stroke-based sketched symbol reconstruction and segmentation</a> </td> <td> CGA 2020 </td> <td> </td> </tr> <tr> <td> <a href="http://sweb.cityu.edu.hk/hongbofu/doc/SketchGNN_TOG21.pdf">SketchGNN: Semantic Sketch Segmentation with Graph Neural Networks</a> </td> <td> TOG 2021 </td> <td> <a href="https://github.com/sYeaLumin/SketchGNN">[code]</a> </td> </tr> <tr> <td> <a href="https://arxiv.org/abs/2112.10838">One Sketch for All: One-Shot Personalized Sketch Segmentation</a> </td> <td> TIP 2022 </td> <td> </td> </tr> <tr> <td> <a href="https://ieeexplore.ieee.org/abstract/document/10214525">Sketch-Segformer: Transformer-Based Segmentation for Figurative and Creative Sketches</a> </td> <td> TIP 2023 </td> <td> <a href="https://github.com/PRIS-CV/Sketch-SF">[code]</a> </td> </tr> <tr> <td> <a href="https://ieeexplore.ieee.org/abstract/document/10471272">CreativeSeg: Semantic Segmentation of Creative Sketches</a> </td> <td> TIP 2024 </td> <td> <a href="https://github.com/PRIS-CV/Sketch-CS">[code]</a> </td> </tr> <tr> <td> <a href="https://openaccess.thecvf.com/content/CVPR2024/papers/Wang_ContextSeg_Sketch_Semantic_Segmentation_by_Querying_the_Context_with_Attention_CVPR_2024_paper.pdf">ContextSeg: Sketch Semantic Segmentation by Querying the Context with Attention </a> </td> <td> CVPR 2024 </td> <td> <a href="https://enigma-li.github.io/projects/contextSeg/contextSeg.html">[webpage]</a> </td> </tr> </table>

Perceptual Grouping

FloorPlan

<table> <tr> <td>Type</td> <td>Paper</td> <td>Source</td> <td>Code/Project Link</td> </tr> <tr> <td rowspan="6">Vector</td> <td> <a href="https://proceedings.neurips.cc/paper/2021/hash/cdf1035c34ec380218a8cc9a43d438f9-Abstract.html">Recognizing Vector Graphics without Rasterization</a> </td> <td> NeurIPS 2021 </td> <td> <a href="https://github.com/microsoft/YOLaT-VectorGraphicsRecognition">[code]</a> </td> </tr> <tr> <td> <a href="https://openaccess.thecvf.com/content/ICCV2021/papers/Fan_FloorPlanCAD_A_Large-Scale_CAD_Drawing_Dataset_for_Panoptic_Symbol_Spotting_ICCV_2021_paper.pdf">FloorPlanCAD: A Large-Scale CAD Drawing Dataset for Panoptic Symbol Spotting</a> </td> <td> ICCV 2021 </td> <td> <a href="https://floorplancad.github.io/">[project]</a> </td> </tr> <tr> <td> <a href="https://arxiv.org/abs/2201.00625">GAT-CADNet: Graph Attention Network for Panoptic Symbol Spotting in CAD Drawings</a> </td> <td> CVPR 2022 </td> <td> </td> </tr> <tr> <td> <a href="https://openaccess.thecvf.com/content/CVPR2022/papers/Fan_CADTransformer_Panoptic_Symbol_Spotting_Transformer_for_CAD_Drawings_CVPR_2022_paper.pdf">CADTransformer: Panoptic Symbol Spotting Transformer for CAD Drawings</a> </td> <td> CVPR 2022 </td> <td> <a href="https://github.com/VITA-Group/CADTransformer">[code]</a> </td> </tr> <tr> <td> <a href="https://openaccess.thecvf.com/content/CVPR2023/papers/Yang_VectorFloorSeg_Two-Stream_Graph_Attention_Network_for_Vectorized_Roughcast_Floorplan_Segmentation_CVPR_2023_paper.pdf">VectorFloorSeg: Two-Stream Graph Attention Network for Vectorized Roughcast Floorplan Segmentation</a> </td> <td> CVPR 2023 </td> <td> <a href="https://github.com/DrZiji/VecFloorSeg">[code]</a> </td> </tr> <tr> <td> <a href="https://ieeexplore.ieee.org/abstract/document/10508965">Hierarchical Recognizing Vector Graphics and A New Chart-based Vector Graphics Dataset</a> </td> <td> TPAMI 2024 </td> <td> <a href="https://github.com/microsoft/YOLaT-VectorGraphicsRecognition">[code]</a> </td> </tr> </table>

15. Sketch Representation Learning

Stroke order importance/saliency, sketch abstraction

Paper	Source	Code/Project Link	Data Manner	Related Applications
Learning Deep Sketch Abstraction	CVPR 2018		Vector/stroke-level	FG-SBIR
Goal-Driven Sequential Data Abstraction	ICCV 2019		Vector/stroke-level	Sketch recognition
Pixelor: a competitive sketching AI agent. So you think you can sketch?	SIGGRAPH Asia 2020	[Project] [Code]	Vector/stroke-level	Sketch synthesis and recognition
SketchAA: Abstract Representation for Abstract Sketches	ICCV 2021		Vector/stroke-level	Sketch recognition, FB-SBIR, sketch healing
CLIPasso: Semantically-Aware Object Sketching	SIGGRAPH 2022	[project] [code]	Vector/stroke-level
Abstracting Sketches through Simple Primitives	ECCV 2022	[code]	Vector/stroke-level	Sketch recognition, FG-SBIR
Learning Geometry-aware Representations by Sketching	CVPR 2023	[code]	Vector/stroke-level	object attribute classification, domain transfer, stroke-based generation, FG-SBIR
SketchXAI: A First Look at Explainability for Human Sketches	CVPR 2023	[project]	Vector/stroke-level	sketch recognition
Prediction with Visual Evidence: Sketch Classification Explanation via Stroke-Level Attributions	TIP 2023		Vector/stroke-level
What Sketch Explainability Really Means for Downstream Tasks	CVPR 2024		Vector/stroke-level
Do Generalised Classifiers really work on Human Drawn Sketches?	ECCV 2024		Pixel-level

Conventional Representation Learning

Paper	Source	Code/Project Link	Data Manner	Related Applications
A Neural Representation of Sketch Drawings (Sketch-RNN)	ICLR 2018	[code]	Vector/stroke-level	Reconstruction and interpolation
SketchHealer: A Graph-to-Sequence Network for Recreating Partial Human Sketches	BMVC 2020	[code]	Vector/stroke-level	Sketch recognition, retrieval, completion and analogy
Sketchformer: Transformer-based Representation for Sketched Structure	CVPR 2020	[code]	Vector/stroke-level	Sketch classification, SBIR, reconstruction and interpolation
CoSE: Compositional Stroke Embeddings	NeurIPS 2020	[Code]	Vector/stroke-level	Auto-completing diagrams
SketchLattice: Latticed Representation for Sketch Manipulation	ICCV 2021	[code]	Lattice graph	Sketch healing and image-to-sketch synthesis
SketchODE: Learning neural sketch representation in continuous time	ICLR 2022	[Project]	Vector/Stroke-level	Continuous-time representation, reconstruction & interpolation
Linking Sketch Patches by Learning Synonymous Proximity for Graphic Sketch Representation	AAAI 2023	[code]	Vector/Stroke-level	Sketch synthesis and sketch healing
SketchKnitter: Vectorized Sketch Generation with Diffusion Models	ICLR 2023	[code]	Vector/Stroke-level	vectorized sketch generation
ChiroDiff: Modelling chirographic data with Diffusion Models	ICLR 2023	[Project]	Vector/Stroke-level	vectorization, de-noising/healing, abstraction
Enhance Sketch Recognition's Explainability via Semantic Component-Level Parsing	AAAI 2024	[code]	Vector/Stroke-level	recognition and segmentation
Modelling complex vector drawings with stroke-clouds	ICLR 2024	[code]	Vector/Stroke-level	reconstruction and generation
Sketchformer++: A Hierarchical Transformer Architecture for Vector Sketch Representation	CVM 2024		Vector/Stroke-level	reconstruction, recognition and semantic segmentation

Paper	Source	Code/Project Link	Data Manner	Related Applications
A Learned Representation for Scalable Vector Graphics	ICCV 2019	[code]	SVG	Font design
DeepSVG: A Hierarchical Generative Network for Vector Graphics Animation	NeurIPS 2020	[Code] [Project]	SVG	Vector Graphics Animation, reconstruction and interpolation
SVGformer: Representation Learning for Continuous Vector Graphics using Transformers	CVPR 2023		SVG	Reconstruction, classification, interpolation, retrieval
StrokeNUWA: Tokenizing Strokes for Vector Graphic Synthesis	arxiv 24.01		SVG	generation

Self-supervised or few/zero-shot learning

Paper	Source	Code/Project Link	Data Manner	Related Applications
Sketch Fewer to Recognize More by Learning a Co-Regularized Sparse Representation	TCSVT 2019			few-shot classification
On Learning Semantic Representations for Large-Scale Abstract Sketches	TCSVT 2020	[Code]	Bitmap and Vector	Retrieval and recognition
Deep Self-Supervised Representation Learning for Free-Hand Sketch	TCSVT 2020	[Code]	Vector/stroke-level	Retrieval and recognition
Sketch-BERT: Learning Sketch Bidirectional Encoder Representation from Transformers by Self-supervised Learning of Sketch Gestalt	CVPR 2020	[Code]	Vector/stroke-level	Sketch recognition, retrieval, and gestalt
Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting	CVPR 2021	[Code]	Both Vector and Raster-level	Recognition
Self-Supervised Learning for Enhancing Spatial Awareness in Free-Hand Sketches	IJCAI 2024	[Code]	Vector-level	Synthesis

Scene-level

Paper	Source	Code/Project Link
SceneTrilogy: On Human Scene-Sketch and its Complementarity with Photo and Text	CVPR 2023

Implicit Neural Representations

Paper	Source	Code/Project Link
SketchINR: A First Look into Sketches as Implicit Neural Representations	CVPR 2024

16. Sketch and Visual Correspondence

Datasets

Name	Paper	Source	Code/Project Link
CreativeFlow+	Creative Flow+ Dataset	CVPR 2019	[code]
ATD-12K	Deep animation video interpolation in the wild	CVPR 2021	[code]
AnimeRun	AnimeRun: 2D Animation Correspondence from Open Source 3D Movies	NeurIPS 2022	[code]
PaintBucket-Character	Learning Inclusion Matching for Animation Paint Bucket Colorization	CVPR 2024	[code]

Methods

Matching Type	Paper	Source	Code/Project Link
Region	Globally optimal toon tracking	SIGGRAPH 2016	[project]
Region	The Animation Transformer: Visual Correspondence via Segment Matching	ICCV 2021
Region	Learning Inclusion Matching for Animation Paint Bucket Colorization	CVPR 2024	[code] [project]

Point	SketchDesc: Learning Local Sketch Descriptors for Multi-view Correspondence	TCSVT 2020
Point	SketchZooms: Deep Multi-view Descriptors for Matching Line Drawings	CGF 2021	[project]

Stroke	Differential Operators on Sketches via Alpha Contours	SIGGRAPH 2023	[code]

17. Sketch Animation/Inbetweening

Please refer to repository MarkMoHR/Awesome-2D-Animation for details.

18. Sketch and AR/VR

Paper	Source	Code/Project Link
SweepCanvas: Sketch-based 3D Prototyping on an RGB-D Image	UIST 2017	[video]
Model-Guided 3D Sketching	TVCG 2018	[video]
Mobi3DSketch: 3D Sketching in Mobile AR	CHI 2019	[video]
Interactive Liquid Splash Modeling by User Sketches	SIGGRAPH Asia 2020	[video]
Towards 3D VR-Sketch to 3D Shape Retrieval	3DV 2020	[code] [project]
3D Curve Creation on and around Physical Objects with Mobile AR	TVCG 2021	[video]
HandPainter - 3D Sketching in VR with Hand-based Physical Proxy	CHI 2021	[video]
Fine-Grained VR Sketching: Dataset and Insights	3DV 2021	[code]
Structure-Aware 3D VR Sketch to 3D Shape Retrieval	3DV 2022	[code]
GestureSurface: VR Sketching through Assembling Scaffold Surface with Non-Dominant Hand	TVCG 2023
3D VR Sketch Guided 3D Shape Prototyping and Exploration	ICCV 2023	[code]

19. Sketch Based Incremental Learning

Paper	Source	Code/Project Link
Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches	CVPR 2022	[code]

20. Sketch Quality Measurement

Paper	Source	Code/Project Link
Annotation-Free Human Sketch Quality Assessment	IJCV 2024	[code]
Finding Badly Drawn Bunnies	CVPR 2022	[code]

21. Cloud Augmentation with Sketches

Paper	Source	Code/Project Link
Cloud2Sketch: Augmenting Clouds with Imaginary Sketches	ACM MM 2022	[project]

22. Sketch and Re-identification

Paper	Source	Code/Project Link
Cross-Compatible Embedding and Semantic Consistent Feature Construction for Sketch Re-identification	ACM MM 2022	[code]
SketchTrans: Disentangled Prototype Learning with Transformer for Sketch-Photo Recognition	TPAMI 2023	[code]
Differentiable Auxiliary Learning for Sketch Re-Identification	AAAI 2024

23. Sketch-based Salient Object Detection

Paper	Source	Code/Project Link
Sketch2Saliency: Learning to Detect Salient Objects from Human Drawings	CVPR 2023

24. Study on Strokes

Paper	Source	Code/Project Link
Neural Brushstroke Engine: Learning a Latent Style Space of Interactive Drawing Tools	SIGGRAPH Asia 2022	[project]
Ciallo: GPU-Accelerated Rendering of Vector Brush Strokes	SIGGRAPH 2024	[code]