A Comparison of Loss Weighting Strategies for Multi task Learning in Deep Neural Networks

IEEE Access

Oguz H. Elibol¹  Cory Stephenson¹  Ting Gong¹  Gokce Keskin¹  Suchismita Padhy¹  Tyler Lee¹  Venkata Renduchintala¹  Anthony Ndirango¹ 

[1] Intel AI Lab, Santa Clara, CA, USA;

关键词: Dynamic weighting average;    multi-MNIST;    multi-objective optimization;    multi-task learning;    uncertainty weighting;   

DOI  :  10.1109/ACCESS.2019.2943604

来源: DOAJ

【 摘 要 】

With the success of deep learning in a wide variety of areas, many deep multi-task learning (MTL) models have been proposed claiming improvements in performance obtained by sharing the learned structure across several related tasks. However, the dynamics of multi-task learning in deep neural networks is still not well understood at either the theoretical or experimental level. In particular, the usefulness of different task pairs is not known a priori. Practically, this means that properly combining the losses of different tasks becomes a critical issue in multi-task learning, as different methods may yield different results. In this paper, we benchmarked different multi-task learning approaches using shared trunk with task specific branches architecture across three different MTL datasets. For the first dataset, i.e. Multi-MNIST (Modified National Institute of Standards and Technology database), we thoroughly tested several weighting strategies, including simply adding task-specific cost functions together, dynamic weight average (DWA) and uncertainty weighting methods each with various amounts of training data per-task. We find that multi-task learning typically does not improve performance for a user-defined combination of tasks. Further experiments evaluated on diverse tasks and network architectures on various datasets suggested that multi-task learning requires careful selection of both task pairs and weighting strategies to equal or exceed the performance of single task learning.

【 授权许可】

Unknown