Learn-to-learn on Arbitrary Textual Conditioning: A Hypernetwork-Driven Meta-gated LLM

Conventional LLMs may suffer from corpus heterogeneity and subtle changes in conditions. While finetuning can create the catastrophe forgetting issue, applications of meta-learning on LLMs are also limited due to their complexity and scalability. In this paper, we activate the meta-signal of $\beta$ within the SwiGLU blocks, resulting in a meta-gating mechanism that adaptively adjusts the nonlinearity of FFN. A hypernetwork is employed to dynamically produce $\beta$ under textual conditions, providing meta-controllability over LLMs. By testing on different condition types such as task, domain, persona, and style, our method outperforms finetuning and meta-learning baselines, and can generalize reasonably on unseen tasks, condition types, or instructions. Our codes are in https://github.com/AaronJi/MeGan.

(1) Current large language models (LLMs) often fail when the situation they encounter in the real world looks very different from the data they were trained on. (2) To solve this, we turned one fixed part of the model into a flexible component, and added a small helper network that reads the changing context (such as the user’s style, topic, or emotion) and adjusts the model accordingly. (3) As a result, our model can adapt itself on the fly to new or rare scenarios, leading to better performance on tasks and datasets it has never seen during training.