Learning to Extrapolate to New Tasks: A Relational Approach to Task Extrapolation

Modern learning systems excel at interpolation but struggle to generalize to unseen tasks outside the training distribution's support. This failure occurs even in simple settings, such as handling task parameters beyond the training range, and persists despite advances in foundation models. To this end, we develop the Relational Task Extrapolator (RTE), an algorithm designed to enable systematic extrapolation to novel tasks. The key observation is that extrapolation is inherently relational: extrapolating to unseen tasks requires learning how tasks transform into one another. If a model learns the transformation between tasks A and B during training, it can apply that same transformation to relate known tasks to unseen ones at test time. RTE operationalizes this idea by decomposing each target task into a known anchor task and a transformation linking the anchor and target. It then learns a relational operator, mapping an anchor–transformation pair to predictions for the target task. We instantiate RTE across multiple task extrapolation regimes in function prediction, e.g. where target tasks use out-of-range parameters (parameter extrapolation), has greater compositional depth (length extrapolation), and/or recombine function primitives in unseen ways (compositional extrapolation). We further extend RTE to sequence prediction, integrating it into fine-tuning algorithms for foundation models. Across empirical studies, we find that RTE substantially outperforms existing approaches on extrapolation to novel, unseen tasks.

This paper is about helping machine learning systems handle unseen new tasks that differ from the tasks they were trained on. Modern systems can be very good at recognizing familiar patterns, but they often fail when an unseen new task asks them to go beyond that familiar range, such as using larger numbers, following longer patterns, or combining known pieces in new ways. To this end, we develop the Relational Task Extrapolator (RTE). The key idea is to take a relational approach to solving unseen new tasks. If the system learns how one task changes into another, it can reuse that kind of change to reason about a new task. RTE does this by linking a new task to a familiar one and using the difference between them to make predictions. Across several studies, including tests with large pre-trained models, RTE improves performance on unseen novel tasks outside the training range.