Timezone: »

 
Oral
Learn-to-Share: A Hardware-friendly Transfer Learning Framework Exploiting Computation and Parameter Sharing
Cheng Fu · Hanxian Huang · Xinyun Chen · Yuandong Tian · Jishen Zhao

Thu Jul 22 06:00 PM -- 06:20 PM (PDT) @
in Algorithms 4 »

Task-specific fine-tuning on pre-trained transformers has achieved performance breakthroughs in multiple NLP tasks. Yet, as both computation and parameter size grows linearly with the number of sub-tasks, it is increasingly difficult to adopt such methods to the real world due to unrealistic memory and computation overhead on computing devices. Previous works on fine-tuning focus on reducing the growing parameter size to save storage cost by parameter sharing. However, compared to storage, the constraint of computation is a more critical issue with the fine-tuning models in modern computing environments. In this work, we propose LeTS, a framework that leverages both computation and parameter sharing across multiple tasks. Compared to traditional fine-tuning, LeTS proposes a novel neural architecture that contains a fixed pre-trained transformer model, plus learnable additive components for sub-tasks. The learnable components reuse the intermediate activations in the fixed pre-trained model, decoupling computation dependency. Differentiable neural architecture search is used to determine a task-specific computation sharing scheme, and a novel early stage pruning is applied to additive components for sparsity to achieve parameter sharing. Extensive experiments show that with 1.4% of extra parameters per task, LeTS reduces the computation by 49.5% on GLUE benchmarks with only 0.2% accuracy loss compared to full fine-tuning.

Keywords: Natural Language Processing

Author Information

Cheng Fu (University of California, San Diego)
Hanxian Huang (UC San Diego)
Xinyun Chen (UC Berkeley)
Yuandong Tian (Facebook AI Research)
Jishen Zhao (University of California, San Diego)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors