Timezone: »

 
Poster
Data Efficient Neural Scaling Law via Model Reusing
Peihao Wang · Rameswar Panda · Zhangyang “Atlas” Wang

Thu Jul 27 01:30 PM -- 03:00 PM (PDT) @ Exhibit Hall 1 #735
in Poster Session 5 »

The number of parameters in large transformers has been observed to grow exponentially. Despite notable performance improvements, concerns have been raised that such a growing model size will run out of data in the near future. As manifested in the neural scaling law, modern learning backbones are not data-efficient. To maintain the utility of the model capacity, training data should be increased proportionally. In this paper, we study the neural scaling law under the previously overlooked data scarcity regime, focusing on the more challenging situation where we need to train a gigantic model with a disproportionately limited supply of available training data. We find that the existing power laws underestimate the data inefficiency of large transformers. Their performance will drop significantly if the training set is insufficient. Fortunately, we discover another blessing - such a data-inefficient scaling law can be restored through a model reusing approach that warm-starts the training of a large model by initializing it using smaller models. Our empirical study shows that model reusing can effectively reproduce the power law under the data scarcity regime. When progressively applying model reusing to expand the model size, we also observe consistent performance improvement in large transformers. We release our code at: https://github.com/VITA-Group/Data-Efficient-Scaling.

[ PDF]  [ Poster] 

Author Information

Peihao Wang (The University of Texas at Austin)
Rameswar Panda (MIT-IBM Watson AI Lab)
Zhangyang “Atlas” Wang (University of Texas at Austin)

More from the Same Authors