Universal Algorithm-Implicit Learning

Current meta-learning methods are constrained to narrow task distributions with fixed feature and label spaces, limiting applicability. Moreover, the current meta-learning literature uses key terms like “universal” and “general-purpose” inconsistently and lacks precise definitions, hindering comparability. We introduce a theoretical framework for meta-learning which formally defines practical universality and introduces a distinction between algorithm-explicit and algorithm-implicit learning, providing a principled vocabulary for reasoning about universal meta-learning methods. Guided by this framework, we present TAIL, a transformer-based algorithm-implicit meta-learner that functions across tasks with varying domains, modalities, and label configurations. TAIL features three innovations over prior transformer-based meta-learners: random projections for cross-modal feature encoding, random injection label embeddings that extrapolate to larger label spaces, and efficient inline query processing. TAIL achieves state-of-the-art performance on standard few-shot benchmarks while generalizing to unseen domains. Unlike other meta-learning methods, it also generalizes to unseen modalities, solving text and audio classification tasks despite training exclusively on images, handles tasks with up to 20× more classes than seen during training, and provides orders-of-magnitude computational savings over prior transformer-based approaches.

Most AI systems need thousands of examples to learn a new task, but in many real-world settings, such as medical diagnosis of rare diseases, only a handful of labeled examples are available. Meta-learning methods aim to solve this by training systems that can learn new tasks from just a few examples. However, existing approaches only work when the new task closely resembles what they were trained on: a system trained on photographs cannot handle text or audio, and one trained to distinguish five classes fails when faced with 50. We introduce TAIL, a meta-learning method that overcomes these limitations and is designed to function across tasks with varying data types, domains, and numbers of classes. TAIL learns a general-purpose learning strategy from a large collection of diverse image tasks and can then apply this strategy to completely new settings, including different data types it has never seen before. Remarkably, TAIL solves text and audio classification tasks despite being trained exclusively on images. It also scales to tasks with far more categories than seen during training, handling up to 20 times the number of classes, all without any architectural changes or retraining. Alongside the method, we introduce a formal framework that clarifies what "universal" meta-learning means and why certain approaches generalize more broadly than others. These results suggest that transferable learning abilities can emerge through large-scale meta-training, that go beyond the training domain, potentially enabling the usage of AI systems in data-scarce settings across science, medicine, and industry.