Awesome
Sketch-Based Deep Learning Papers
A resource repository for sketch based deep learning papers and work.
Table of Contents
Dataset
Blending Texture Features from Multiple Reference Images for Style Transfer (SIGGRAPH Asia Tech Briefs 2016) [Paper] Nico Open data [Link]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/nicoopen.png" height="250"> </p>The Sketchy Database: Learning to Retrieve Badly Drawn Bunnies (TOG, SIGGRAPH 2016) [Link]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/sketchydatabase.png" height="200"> </p>Quick, Draw! Dataset (2017) [Link]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/quickdraw.png" height="200"> </p>SketchyScene: Richly-Annotated Scene Sketches (ECCV 2018) [Link]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/sketchyscene.png" height="200"> </p>Universal Sketch Perceptual Grouping (ECCV 2018) [Paper]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/spg.png" height="150"> </p>OpenSketch: A Richly-Annotated Dataset of Product Design Sketches (TOG, SIGGRAPH Asia 2019) [Link]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/opensketch.png" height="250"> </p>EmoG: Supporting the Sketching of Emotional Expressions for Storyboarding (CHI on Human Factors in Computing Systems, 2020) [Link]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/facex.png" height="250"> </p>Building a Manga Dataset "Manga109" with Annotations for Multimedia Applications (IEEE MultiMedia, 2020) [Paper] [Link]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/manga109.png" height="250"> </p>SPARE3D: A Dataset for SPAtial REasoning on Three-View Line Drawings (CVPR, 2020) [Paper]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/3dlinedrawing.png" height="250"> </p>Colorization
Paintschainer (2017) [Link]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/paintchainer.png" height="250"> </p>Outline colorization through tandem adversarial networks (2017) [Paper]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/outline.png" height="150"> </p>Comicolorization: semi-automatic manga colorization (SIGGRAPH Aisa 2017 Tech Briefs) [Paper]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/comicolor.png" height="250"> </p>Two-stage sketch colorization (TOG, SIGGRAPH Asia 2018) [Paper]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/twostagecolo.png" height="200"> </p>User-Guided Deep Anime Line Art Colorization with Conditional Adversarial Networks (ACM Multimedia, 2018) [Paper]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/colorMM18.png" height="300"> </p>Tag2pix: Line art colorization using text tag with secat and changing loss (ICCV 2019) [Paper]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/tag2pix.png" height="200"> </p>Automatic Colorization of Anime Sketch Image [Link]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/autocolo.png" height="250"> </p>Language-based Colorization of Scene Sketches (TOG, SIGGRAPH Asia 2019) [Link]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/languagecolo.png" height="250"> </p>Deep Line Art Video Colorization with a Few References (2020) [Paper]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/videocolo.png" height="250"> </p>Reference-Based Sketch Image Colorization using Augmented-Self Reference and Dense Semantic Correspondence (CVPR, 2020) [Paper]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/refercolo.png" height="250"> </p>Stylization
Learning to Simplify: Fully Convolutional Networks for Rough Sketch Cleanup (TOG, SIGGRAPH 2016) [Link]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/sketchsimp.png" height="200"> </p>Deep extraction of manga structural lines (TOG, SIGGRAPH 2017) [Link]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/deepextract.png" height="250"> </p>A Neural Representation of Sketch Drawings (ICLR 2018) [Paper]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/sketchrnn.png" height="250"> </p>Mastering sketching: adversarial augmentation for struc- tured prediction (TOG, SIGGRAPH 2018) [Link]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/mastersketch.png" height="200"> </p>Real-Time Data-Driven Interactive Rough Sketch Inking (TOG, SIGGRAPH 2018) [Link]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/smartinker.png" height="250"> </p>Im2pencil: Controllable pencil illustra- tion from photographs (CVPR 2019) [Paper]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/im2pencil.png" height="250"> </p>Adversarial Colorization Of Icons Based On Structure And Color Conditions (2019) [Paper]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/iconcolor.png" height="250"> </p>Learning to Shadow Hand-drawn Sketches (CVPR 2020) [Link]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/shadesketch.png" height="250"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/shadesketch.gif" height="250"> </p>Neural Contours: Learning to Draw Lines from 3D Shapes (CVPR 2020) [Paper]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/neuralcontour.png" height="250"> </p>Sketch-BERT: Learning Sketch Bidirectional Encoder Representation from Transformers by Self-supervised Learning of Sketch Gestalt (CVPR, 2020) [Paper]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/sketchbert.png" height="250"> </p>A Benchmark for Rough Sketch Cleanup (SIGGRAPH Asia 2020) [Link]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/benchmark4sketch.png" height="250"> </p>Creative Sketch Generation (ICLR 2021) [Link]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/doodlergan.png" height="150"> </p>General Virtual Sketching Framework for Vector Line Art (SIGGRAPH 2021) [Link]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/linevector.png" height="200"> </p>Sketch to 3D Representation
Sketch to Normal Maps. Sketch to 3D Model.
Normal Maps
Deep Normal Estimation for Automatic Shading of Hand-Drawn Characters (ECCV Workshop 2018) [Link]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/deepnormal.png" height="150"> </p>Interactive Sketch-Based Normal Map Generation with Deep Neural Networks (I3D 2018) [Paper]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/sketch2normal.png" height="150"> </p>Novel View
Lifting Freehand Concept Sketches into 3D (SIGGRAPH Asia, 2020) [Link]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/freehand.png" height="250"> </p>3D Model
BendSketch: Modeling Freeform Surfaces Through 2D Sketching (TOG, SIGGRAPH 2017) [Paper]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/bendsketch.png" height="250"> </p>DeepSketch2Face: A Deep Learning Based Sketching System for 3D Face and Caricature Modeling (TOG, SIGGRAPH 2017) [Paper]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/sketch2face.png" height="100"> </p>3D Shape Reconstruction from Sketches via Multi-view Convolutional Networks (3DV 2017) [Paper]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/3drecons3dv.png" height="200"> </p>3D Sketching using Multi-View Deep Volumetric Prediction (I3D 2018) [Paper]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/3dsketch.png" height="200"> </p>Contour-based 3D Modeling through Joint Embedding of Shapes and Contours (I3D 2020) [Paper]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/contour3dmodel.png" height="250"> </p>Realistic
SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis (CVPR 2018) [Paper]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/sketchygan.png" height="150"> </p>Sketch-R2CNN: An Attentive Network for Vector Sketch Recognition (2018) [Paper]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/sketchr2cnn.png" height="100"> </p>Learning to Sketch with Shortcut Cycle Consistency (CVPR 2018) [Paper]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/sketch2img.png" height="150"> </p>Sketch-a-Classifier: Sketch-based Photo Classifier Generation (CVPR 2018) [Paper]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/sketchclassifier.png" height="150"> </p>Learning Deep Sketch Abstraction (CVPR 2018) [Paper]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/sketchabs.png" height="200"> </p>SketchMate: Deep Hashing for Million-Scale Human Sketch Retrieval (CVPR 2018) [Paper]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/sketchmate.png" height="150"> </p>Interactive Sketch & Fill: Multiclass Sketch-to-Image Translation (ICCV 2019) [Link]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/cupcake.gif" height="200"> </p>Deep Zero-Shot Learning for Scene Sketch (ICIP 2019) [Paper]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/0shotsketch.png" height="150"> </p>Doodle to Search: Practical Zero-Shot Sketch-based Image Retrieval (CVPR 2019) [Paper]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/0shotir.png" height="200"> </p>Generalising Fine-Grained Sketch-Based Image Retrieval (CVPR 2019) [Paper]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/finegrainedsketchretrieval.png" height="250"> </p>SketchyCOCO: Image Generation from Freehand Scene Sketches (CVPR, 2020) [Paper]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/sketchycoco.png" height="150"> </p>BézierSketch: A generative model for scalable vector sketches (ECCV, 2020) [Paper]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/beziersketch.png" height="200"> </p>Pixelor: A Competitive Sketching AI Agent. So you think you can beat me? (SIGGRAPH Asia, 2020) [Link]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/sketch_agent.png" height="200"> </p>Others
Researchers & Labs
(Random order)
Dr. Aaron Hertzmann's Blog [Link]
-
Why Edge Detection Doesn’t Explain Line Drawing [Link]
-
Apparent Ridges, Edges, and Perception of Line Drawing [Link]
-
Why Does Line Drawing Work? A Realism Hypothesis [Link]
SketchX Lab [Link]
Dr. Changqing Zou [Link]
Dr. Yulia Gryaditskaya [Link]
Misc
A Survey on Sketch Based Content Creation: from the Desktop to Virtual and Augmented Reality (Eurographics, 2020) [Paper]
Why Do Line Drawings Work? A Realism Hypothesis (Perception 2020) [Paper]
SYNLA Dataset (Synthetic Line Art Dataset) [Link]
<p align="center"> <img src="https://github.com/qyzdao/Sketch-based-DL/blob/master/images/synladataset.png" height="200"> </p>Disclaimer
This is a personal collection. No warranty. Working in progress, pull request welcome. :heart: