Poster

Learning to Branch for Multi-Task Learning

Pengsheng Guo · Chen-Yu Lee · Daniel Ulbricht

Keywords: [ Architectures ] [ Transfer and Multitask Learning ] [ Meta-learning and Automated ML ] [ Transfer, Multitask and Meta-learning ]

[ Abstract ] [ Join Zoom
Please do not share or post zoom links

Abstract:

Training multiple tasks jointly in one deep network yields reduced latency during inference and better performance over the single-task counterpart by sharing certain layers of a network. However, over-sharing a network could erroneously enforce over-generalization, causing negative knowledge transfer across tasks. Prior works rely on human intuition or pre-computed task relatedness scores for ad hoc branching structures. They provide sub-optimal end results and often require huge efforts for the trial-and-error process.

In this work, we present an automated multi-task learning algorithm that learns where to share or branch within a network, designing an effective network topology that is directly optimized for multiple objectives across tasks. Specifically, we propose a novel tree-structured design space that casts a tree branching operation as a gumbel-softmax sampling procedure. This enables differentiable network splitting that is end-to-end trainable. We validate the proposed method on controlled synthetic data, CelebA, and Taskonomy.

Chat is not available.