Awesome

Awesome Multi-Task Learning

This page contains a list of papers on multi-task learning for computer vision. Please create a pull request if you wish to add anything. If you are interested, consider reading our recent survey paper.

Multi-Task Learning for Dense Prediction Tasks: A Survey

Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai and Luc Van Gool.

Workshop

:loudspeaker: :loudspeaker: :loudspeaker: We organized a workshop on multi-task learning at ICCV 2021 (Link).

• Jan 13: The recordings of our invited talks are now available on Youtube.

Table of Contents:

• Survey papers
• Datasets
• Architectures
  • Encoder-based
  • Decoder-based
  • Other
• Neural Architecture Search
• Optimization strategies
• Transfer learning

<a name="survey"></a>

Survey papers

• <a name="vandenhende2020revisiting"></a> Vandenhende, S., Georgoulis, S., Van Gansbeke, W., Proesmans, M., Dai, D., & Van Gool, L. Multi-Task Learning for Dense Prediction Tasks: A Survey, T-PAMI, 2020. [PyTorch]

• <a name="ruder2017survey"></a> Ruder, S. An overview of multi-task learning in deep neural networks, ArXiv, 2017.

• <a name="zhang2017survey"></a> Zhang, Y. A survey on multi-task learning, ArXiv, 2017.

• <a name="gong2019comparison"></a> Gong, T., Lee, T., Stephenson, C., Renduchintala, V., Padhy, S., Ndirango, A., ... & Elibol, O. H. A comparison of loss weighting strategies for multi task learning in deep neural networks, IEEE Access, 2019.

<a name="datasets"></a>

Datasets

The following datasets have been regularly used in the context of multi-task learning:

• NYUDv2
• Cityscapes
• PASCAL
• Taskonomy
• KITTI
• SUN RGB-D
• BDD100K

<a name="architectures"></a>

Architectures

<a name="encoder"></a>

Encoder-based architectures

• <a name="misra2016cross"></a> Misra, I., Shrivastava, A., Gupta, A., & Hebert, M. Cross-stitch networks for multi-task learning, CVPR, 2016. [PyTorch]

• <a name="gao2019nddr"></a> Gao, Y., Ma, J., Zhao, M., Liu, W., & Yuille, A. L. Nddr-cnn: Layerwise feature fusing in multi-task cnns by neural discriminative dimensionality reduction, CVPR, 2019. [Tensorflow] [PyTorch]

• <a name="liu2019mtan"></a> Liu, S., Johns, E., & Davison, A. J. End-to-end multi-task learning with attention, CVPR, 2019. [PyTorch]

<a name="decoder"></a>

Decoder-based architectures

• <a name="bilen2016multi"></a> Bilen, H., Vedaldi, A. Integrated perception with recurrent multi-task neural networks, NeurIPS, 2016.

• <a name="xu2018pad"></a> Xu, D., Ouyang, W., Wang, X., & Sebe, N. Pad-net: Multi-tasks guided prediction-and-distillation network for simultaneous depth estimation and scene parsing, CVPR, 2018.

• <a name="zhang2018jtrl"></a> Zhang, Z., Cui, Z., Xu, C., Jie, Z., Li, X., & Yang, J. Joint task-recursive learning for semantic segmentation and depth estimation, ECCV, 2018.

• <a name="ruder2019sluice"></a> Ruder, S., Bingel, J., Augenstein, I., & Søgaard, A. Latent multi-task architecture learning, AAAI, 2019.

• <a name="zhang2019papnet"></a> Zhang, Z., Cui, Z., Xu, C., Yan, Y., Sebe, N., & Yang, J. Pattern-affinitive propagation across depth, surface normal and semantic segmentation, CVPR, 2019.

• <a name="zhou2020structure"></a> Zhou, L., Cui, Z., Xu, C., Zhang, Z., Wang, C., Zhang, T., & Yang, J. Pattern-Structure Diffusion for Multi-Task Learning, CVPR, 2020.

• <a name="vandenhende2020mti"></a> Vandenhende, S., Georgoulis, S., & Van Gool, L. MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning, ECCV, 2020. [PyTorch]

<a name="otherarchitectures"></a>

Other

• <a name="yang2016deep"></a> Yang, Y., & Hospedales, T. Deep multi-task representation learning: A tensor factorisation approach, ICLR, 2017.

• <a name="kokkinos2017uber"></a> Kokkinos, Iasonas. Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory, CVPR, 2017.

• <a name="rebuffi2017learning"></a> Rebuffi, S. A., Bilen, H., & Vedaldi, A. Learning multiple visual domains with residual adapters, NIPS, 2017.

• <a name="long2017multilinear"></a> Long, M., Cao, Z., Wang, J., & Philip, S. Y. Learning multiple tasks with multilinear relationship networks, NIPS, 2017.

• <a name="meyerson2017beyond"></a> Meyerson, E., & Miikkulainen, R. Beyond shared hierarchies: Deep multitask learning through soft layer ordering, ICLR, 2018.

• <a name="rosenbaum2017routing"></a> Rosenbaum, C., Klinger, T., & Riemer, M. Routing networks: Adaptive selection of non-linear functions for multi-task learning, ICLR, 2018.

• <a name="mallya2018piggy"></a> Mallya, A., Davis, D., & Lazebnik, S. Piggyback: Adapting a single network to multiple tasks by learning to mask weights, ECCV, 2018.

• <a name="rebuffi2018efficient"></a> Rebuffi, S. A., Bilen, H., & Vedaldi, A. Efficient parametrization of multi-domain deep neural networks, CVPR, 2018.

• <a name="maninis2019astmt"></a> Maninis, K. K., Radosavovic, I., & Kokkinos, I. Attentive single-tasking of multiple tasks, CVPR, 2019. [PyTorch]

• <a name="kanakis2020reparameterizing"></a> Kanakis, M., Bruggemann, D., Saha, S., Georgoulis, S., Obukhov, A., & Van Gool, L. Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference, ECCV, 2020.

• <a name="wang2020multi"></a> Wang, Q., Ke, J., Greaves, J., Chu, G., Bender, G., Sbaiz, L., Go, A., Howard, A., Yang, F., Yang, M.H. & Gilbert, J. Multi-path Neural Networks for On-device Multi-domain Visual Classification, WACV, 2021.

• <a name="bruggemann2021exploring"></a> Bruggemann, D., Kanakis, M., Obukhov, A., Georgoulis, S., & Van Gool, L. Exploring Relational Context for Multi-Task Dense Prediction, ArXiv, 2021.

• <a name="li2021universal"></a> Li, W. H., Liu, X., & Bilen, H. Universal Representation Learning from Multiple Domains for Few-shot Classification, ICCV, 2021.

• <a name="li2021learning"></a> Li, W. H., Liu, X., & Bilen, H. Learning Multiple Dense Prediction Tasks from Partially Annotated Data, ArXiv, 2021.

<a name="nas"></a>

Neural Architecture Search

• <a name="lu2017fully"></a> Lu, Y., Kumar, A., Zhai, S., Cheng, Y., Javidi, T., & Feris, R. Fully-adaptive feature sharing in multi-task networks with applications in person attribute classification, CVPR, 2017.

• <a name="bragman2019stochastic"></a> Bragman, F. J., Tanno, R., Ourselin, S., Alexander, D. C., & Cardoso, J. Stochastic filter groups for multi-task cnns: Learning specialist and generalist convolution kernels, ICCV, 2019.

• <a name="newell2019feature"></a> Newell, A., Jiang, L., Wang, C., Li, L. J., & Deng, J. Feature partitioning for efficient multi-task architectures, ArXiv, 2019.

• <a name="guo2020learning"></a> Guo, P., Lee, C. Y., & Ulbricht, D. Learning to Branch for Multi-Task Learning, ICML, 2020.

• <a name="standley2019tasks"></a> Standley, T., Zamir, A. R., Chen, D., Guibas, L., Malik, J., & Savarese, S. Which Tasks Should Be Learned Together in Multi-task Learning?, ICML, 2020.

• <a name="vandenhende2019branched"></a> Vandenhende, S., Georgoulis, S., De Brabandere, B., & Van Gool, L. Branched multi-task networks: deciding what layers to share, BMVC, 2020.

• <a name="bruggeman2020auomated"></a> Bruggemann, D., Kanakis, M., Georgoulis, S., & Van Gool, L. Automated Search for Resource-Efficient Branched Multi-Task Networks, BMVC, 2020.

• <a name="sun2019adashare"></a> Sun, X., Panda, R., & Feris, R. AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning, NIPS, 2020.

<a name="optimization"></a>

Optimization strategies

• <a name="kendall2018uncertainty"></a> Kendall, A., Gal, Y., & Cipolla, R. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics, CVPR, 2018.

• <a name="zhao2018modulation"></a> Zhao, X., Li, H., Shen, X., Liang, X., & Wu, Y. A modulation module for multi-task learning with applications in image retrieval, ECCV, 2018.

• <a name="chen2018gradnorm"></a> Chen, Z., Badrinarayanan, V., Lee, C. Y., & Rabinovich, A. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks, ICML, 2018.

• <a name="sener2018mgda"></a> Sener, O., & Koltun, V. Multi-task learning as multi-objective optimization, NIPS, 2018. [PyTorch]

• <a name="liu2017adversarial"></a> Liu, P., Qiu, X., & Huang, X. Adversarial multi-task learning for text classification, ACL, 2018.

• <a name="guo2018dynamic"></a> Guo, M., Haque, A., Huang, D. A., Yeung, S., & Fei-Fei, L. Dynamic task prioritization for multitask learning, ECCV, 2018.

• <a name="lin2019pareto"></a> Lin, X., Zhen, H. L., Li, Z., Zhang, Q. F., & Kwong, S. Pareto multi-task learning, NIPS, 2019.

• <a name="suteu2019orthogonal"></a> Suteu, M., & Guo, Y. Regularizing Deep Multi-Task Networks using Orthogonal Gradients, ArXiv, 2019.

• <a name="yu2020surgery"></a> Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., & Finn, C. Gradient surgery for multi-task learning, NIPS, 2020. [Tensorflow]

• <a name="chen2020sign"></a> Chen, Z., Ngiam, J., Huang, Y., Luong, T., Kretzschmar, H., Chai, Y., & Anguelov, D. Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout, NIPS, 2020.

• <a name="li2020knowledge"></a> Li, W. H., & Bilen, H. Knowledge Distillation for Multi-task Learning, ECCV-Workshop, 2020. [PyTorch]

• <a name="borse2021inverseform"></a> Borse, S., Wang, Y., Zhang, Y., & Porikli, F. InverseForm: A Loss Function for Structured Boundary-Aware Segmentation, CVPR 2021.

• <a name="vasu2021instance"></a> Vasu P., Saxena S., Tuzel O. Instance-Level Task Parameters: A Robust Multi-task Weighting Framework, ArXiv, 2021.

<a name="transfer"></a>

Transfer learning & Domain Adaptation

• <a name="cui2018large"></a> Cui, Y., Song, Y., Sun, C., Howard, A., & Belongie, S. Large scale fine-grained categorization and domain-specific transfer learning, CVPR, 2018.

• <a name="zamir2018taskonomy"></a> Zamir, A. R., Sax, A., Shen, W., Guibas, L. J., Malik, J., & Savarese, S. Taskonomy: Disentangling task transfer learning, CVPR, 2018. [PyTorch]

• <a name="achille2019task2vec"></a> Achille, A., Lam, M., Tewari, R., Ravichandran, A., Maji, S., Fowlkes, C. C., ... & Perona, P. Task2vec: Task embedding for meta-learning, ICCV, 2019. [PyTorch]

• <a name="dwivedi2019rsa"></a> Dwivedi, K., & Roig, G. Representation similarity analysis for efficient task taxonomy & transfer learning, CVPR, 2019. [PyTorch]

• <a name="saha2021learning"></a> Saha, S., Obukhov, A., Paudel, D. P., Kanakis, M., Chen, Y., Georgoulis, S., & Van Gool, L. Learning to Relate Depth and Semantics for Unsupervised Domain Adaptation, CVPR, 2021.

<a name="robustness"></a>

Robustness

• <a name="maomultitask2020"></a> Mao, C., Gupta, A., Nitin, V., Ray, B., Song, S., Yang, J., & Vondrick, C. Multitask Learning Strengthens Adversarial Robustness, ECCV, 2020. [PyTorch]

• <a name="zamirrobust2020"></a> Zamir, A. R., Sax, A., Cheerla, N., Suri, R., Cao, Z., Malik, J., & Guibas, L. J. Robust Learning Through Cross-Task Consistency, CVPR, 2020.

• <a name="georgescu2020anomaly"></a> Georgescu, M. I., Barbalau, A., Ionescu, R. T., Khan, F. S., Popescu, M., & Shah, M. Anomaly Detection in Video via Self-Supervised and Multi-Task Learning, CVPR, 2021.

<a name="other"></a>

Other

• <a name="steerable"></a> Eftekhar, A.,Sax, A., Malik, J., Zamir, A. Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets From 3D Scans, ICCV, 2021.