Awesome
Awesome-Sketch-Based-Applications
A collection of sketch based applications.
Feel free to create a PR or an issue.
Outline
- 0. Survey
- 1. Sketch Based Image Synthesis
- 2. Sketch Based Image Editing
- 3. Sketch Based Image Retrieval (SBIR)
- 4. Sketch Based 3D Shape Retrieval
- 5. Sketch Based 3D Shape Modeling
- 6. Sketch Based Garment Design
- 7. Sketch Completion
- 8. Sketch Restoration, Retargeting and Inpainting
- 9. Sketch and Shading
- 10. Sketch Enhancement / Beautification
- 11. Sketch-Guided Object Localization
- 12. Sketch-Guided Video Synthesis
- 13. Sketch Recognition
- 14. Sketch Segmentation and Perceptual Grouping
- 15. Sketch Representation Learning
- 16. Sketch and Visual Correspondence
- 17. Sketch Animation/Inbetweening
- 18. Sketch and AR/VR
- 19. Sketch Based Incremental Learning
- 20. Sketch Quality Measurement
- 21. Cloud Augmentation with Sketches
- 22. Sketch and Re-identification
- 23. Sketch-based Salient Object Detection
- 24. Study on Strokes
0. Survey
Paper | Source | Code/Project Link |
---|---|---|
Deep Learning for Free-Hand Sketch: A Survey | TPAMI 2022 | [code] |
1. Sketch Based Image Synthesis
1.1 Automatic Synthesis
- Natural Image or Object
- Human Face / Portrait / Body
- 3D image
Paper | Source | Code/Project Link |
---|---|---|
3D-aware Conditional Image Synthesis | CVPR 2023 | [project] [code] |
1.2 Style-conditioned
1.3 Text-conditioned
2. Sketch Based Image Editing
- Arbitrary Image
- Human Face / Portrait / Hair
- Anime Editing
Paper | Source | Code/Project Link |
---|---|---|
AniFaceDrawing: Anime Portrait Exploration during Your Sketching | SIGGRAPH 2023 | [project] |
- Video Editing
Paper | Source | Code/Project Link |
---|---|---|
DeepFaceVideoEditing: Sketch-based Deep Editing of Face Videos | SIGGRAPH 2022 | [project] |
3. Sketch Based Image Retrieval (SBIR)
- Object-level
- Scene-level
- Video Retrieval
Paper | Source | Code/Project Link |
---|---|---|
Fine-Grained Instance-Level Sketch-Based Video Retrieval | TCSVT 2020 | |
Fine-Grained Video Retrieval with Scene Sketches | TIP 2023 | [project] |
4. Sketch Based 3D Shape Retrieval
5. Sketch Based 3D Shape Modeling
- Free-hand sketch
- Scene Generation
Paper | Source | Code/Project Link |
---|---|---|
Sketch2Scene: Automatic Generation of Interactive 3D Game Scenes from User's Casual Sketches | arxiv 24.08 | [webpage] |
GroundUp: Rapid Sketch-Based 3D City Massing | ECCV 2024 | [code] |
External Knowledge Enhanced 3D Scene Generation from Sketch | ECCV 2024 |
- NeRF
Paper | Source | Code/Project Link |
---|---|---|
Sketch2NeRF: Multi-view Sketch-guided Text-to-3D Generation | arxiv 24.01 | |
SKED: Sketch-guided Text-based 3D Editing | ICCV 2023 | [project] [code] |
SketchDream: Sketch-based Text-to-3D Generation and Editing | SIGGRAPH 2024 | [project] |
- CAD sketch
6. Sketch Based Garment Design
7. Sketch Completion
- Sketch completion
- Sketch gap / connectivity detection
Paper | Source | Code/Project Link |
---|---|---|
Joint Gap Detection and Inpainting of Line Drawings | CVPR 2017 | [project] [code] |
Detecting Viewer-Perceived Intended Vector Sketch Connectivity | SIGGRAPH 2022 | [project] |
8. Sketch Restoration, Retargeting and Inpainting
- Manga
Paper | Source | Code/Project Link |
---|---|---|
Exploiting Aliasing for Manga Restoration | CVPR 2021 | [webpage] [code] |
Seamless Manga Inpainting with Semantics Awareness | SIGGRAPH 2021 | [webpage] [code] |
Screentone-Preserved Manga Retargeting | arxiv 22.03 | |
Manga Rescreening with Interpretable Screentone Representation | arxiv 23.06 |
- Hand-Drawn Drawings
Paper | Source | Code/Project Link |
---|---|---|
Restoration of Hand-Drawn Architectural Drawings using Latent Space Mapping with Degradation Generator | CVPR 2023 |
9. Sketch and Shading
Paper | Source | Code/Project Link |
---|---|---|
Learning to Shadow Hand-drawn Sketches | CVPR 2020 | [project] [code] |
SmartShadow: Artistic Shadow Drawing Tool for Line Drawings | ICCV 2021 | [project] |
10. Sketch Enhancement / Beautification
- Sketch Enhancement
Paper | Source | Code/Project Link |
---|---|---|
SketchMan: Learning to Create Professional Sketches | ACM MM 2020 | [code] |
- Sketch Beautification
- Super Resolution
Paper | Source | Code/Project Link |
---|---|---|
APISR: Anime Production Inspired Real-World Anime Super-Resolution | CVPR 2024 | [code] |
11. Sketch-Guided Object Localization
12. Sketch-Guided Video Synthesis
Paper | Source | Code/Project Link |
---|---|---|
Deep Sketch-guided Cartoon Video Inbetweening | TVCG 2021 | [code] |
ToonCrafter: Generative Cartoon Interpolation | SIGGRAPH Asia 2024 | [code] [webpage] |
LVCD: Reference-based Lineart Video Colorization with Diffusion Models | SIGGRAPH Asia 2024 | [webpage] |
13. Sketch Recognition
14. Sketch Segmentation and Perceptual Grouping
- Semantic / Instance Segmentation
- Perceptual Grouping
- FloorPlan
15. Sketch Representation Learning
- Stroke order importance/saliency, sketch abstraction
- Conventional Representation Learning
Paper | Source | Code/Project Link | Data Manner | Related Applications |
---|---|---|---|---|
A Neural Representation of Sketch Drawings (Sketch-RNN) | ICLR 2018 | [code] | Vector/stroke-level | Reconstruction and interpolation |
SketchHealer: A Graph-to-Sequence Network for Recreating Partial Human Sketches | BMVC 2020 | [code] | Vector/stroke-level | Sketch recognition, retrieval, completion and analogy |
Sketchformer: Transformer-based Representation for Sketched Structure | CVPR 2020 | [code] | Vector/stroke-level | Sketch classification, SBIR, reconstruction and interpolation |
CoSE: Compositional Stroke Embeddings | NeurIPS 2020 | [Code] | Vector/stroke-level | Auto-completing diagrams |
SketchLattice: Latticed Representation for Sketch Manipulation | ICCV 2021 | [code] | Lattice graph | Sketch healing and image-to-sketch synthesis |
SketchODE: Learning neural sketch representation in continuous time | ICLR 2022 | [Project] | Vector/Stroke-level | Continuous-time representation, reconstruction & interpolation |
Linking Sketch Patches by Learning Synonymous Proximity for Graphic Sketch Representation | AAAI 2023 | [code] | Vector/Stroke-level | Sketch synthesis and sketch healing |
SketchKnitter: Vectorized Sketch Generation with Diffusion Models | ICLR 2023 | [code] | Vector/Stroke-level | vectorized sketch generation |
ChiroDiff: Modelling chirographic data with Diffusion Models | ICLR 2023 | [Project] | Vector/Stroke-level | vectorization, de-noising/healing, abstraction |
Enhance Sketch Recognition's Explainability via Semantic Component-Level Parsing | AAAI 2024 | [code] | Vector/Stroke-level | recognition and segmentation |
Modelling complex vector drawings with stroke-clouds | ICLR 2024 | [code] | Vector/Stroke-level | reconstruction and generation |
Sketchformer++: A Hierarchical Transformer Architecture for Vector Sketch Representation | CVM 2024 | Vector/Stroke-level | reconstruction, recognition and semantic segmentation |
Paper | Source | Code/Project Link | Data Manner | Related Applications |
---|---|---|---|---|
A Learned Representation for Scalable Vector Graphics | ICCV 2019 | [code] | SVG | Font design |
DeepSVG: A Hierarchical Generative Network for Vector Graphics Animation | NeurIPS 2020 | [Code] [Project] | SVG | Vector Graphics Animation, reconstruction and interpolation |
SVGformer: Representation Learning for Continuous Vector Graphics using Transformers | CVPR 2023 | SVG | Reconstruction, classification, interpolation, retrieval | |
StrokeNUWA: Tokenizing Strokes for Vector Graphic Synthesis | arxiv 24.01 | SVG | generation |
- Self-supervised or few/zero-shot learning
Paper | Source | Code/Project Link | Data Manner | Related Applications |
---|---|---|---|---|
Sketch Fewer to Recognize More by Learning a Co-Regularized Sparse Representation | TCSVT 2019 | few-shot classification | ||
On Learning Semantic Representations for Large-Scale Abstract Sketches | TCSVT 2020 | [Code] | Bitmap and Vector | Retrieval and recognition |
Deep Self-Supervised Representation Learning for Free-Hand Sketch | TCSVT 2020 | [Code] | Vector/stroke-level | Retrieval and recognition |
Sketch-BERT: Learning Sketch Bidirectional Encoder Representation from Transformers by Self-supervised Learning of Sketch Gestalt | CVPR 2020 | [Code] | Vector/stroke-level | Sketch recognition, retrieval, and gestalt |
Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting | CVPR 2021 | [Code] | Both Vector and Raster-level | Recognition |
Self-Supervised Learning for Enhancing Spatial Awareness in Free-Hand Sketches | IJCAI 2024 | [Code] | Vector-level | Synthesis |
- Scene-level
Paper | Source | Code/Project Link |
---|---|---|
SceneTrilogy: On Human Scene-Sketch and its Complementarity with Photo and Text | CVPR 2023 |
- Implicit Neural Representations
Paper | Source | Code/Project Link |
---|---|---|
SketchINR: A First Look into Sketches as Implicit Neural Representations | CVPR 2024 |
16. Sketch and Visual Correspondence
- Datasets
Name | Paper | Source | Code/Project Link |
---|---|---|---|
CreativeFlow+ | Creative Flow+ Dataset | CVPR 2019 | [code] |
ATD-12K | Deep animation video interpolation in the wild | CVPR 2021 | [code] |
AnimeRun | AnimeRun: 2D Animation Correspondence from Open Source 3D Movies | NeurIPS 2022 | [code] |
PaintBucket-Character | Learning Inclusion Matching for Animation Paint Bucket Colorization | CVPR 2024 | [code] |
- Methods
Matching Type | Paper | Source | Code/Project Link |
---|---|---|---|
Region | Globally optimal toon tracking | SIGGRAPH 2016 | [project] |
Region | The Animation Transformer: Visual Correspondence via Segment Matching | ICCV 2021 | |
Region | Learning Inclusion Matching for Animation Paint Bucket Colorization | CVPR 2024 | [code] [project] |
Point | SketchDesc: Learning Local Sketch Descriptors for Multi-view Correspondence | TCSVT 2020 | |
Point | SketchZooms: Deep Multi-view Descriptors for Matching Line Drawings | CGF 2021 | [project] |
Stroke | Differential Operators on Sketches via Alpha Contours | SIGGRAPH 2023 | [code] |
17. Sketch Animation/Inbetweening
Please refer to repository MarkMoHR/Awesome-2D-Animation for details.
18. Sketch and AR/VR
19. Sketch Based Incremental Learning
Paper | Source | Code/Project Link |
---|---|---|
Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches | CVPR 2022 | [code] |
20. Sketch Quality Measurement
Paper | Source | Code/Project Link |
---|---|---|
Annotation-Free Human Sketch Quality Assessment | IJCV 2024 | [code] |
Finding Badly Drawn Bunnies | CVPR 2022 | [code] |
21. Cloud Augmentation with Sketches
Paper | Source | Code/Project Link |
---|---|---|
Cloud2Sketch: Augmenting Clouds with Imaginary Sketches | ACM MM 2022 | [project] |
22. Sketch and Re-identification
23. Sketch-based Salient Object Detection
Paper | Source | Code/Project Link |
---|---|---|
Sketch2Saliency: Learning to Detect Salient Objects from Human Drawings | CVPR 2023 |
24. Study on Strokes
Paper | Source | Code/Project Link |
---|---|---|
Neural Brushstroke Engine: Learning a Latent Style Space of Interactive Drawing Tools | SIGGRAPH Asia 2022 | [project] |
Ciallo: GPU-Accelerated Rendering of Vector Brush Strokes | SIGGRAPH 2024 | [code] |