Unifying Stacking and Cascading for Efficient Ensemble Inference

We introduce LazyStack, a method for efficient model ensemble inference. The core idea is intuitive: after each model executes, we check whether accumulated evidence is sufficient to exit confidently. Sometimes one model suffices; other times we aggregate predictions from several models via trained meta-learners before reaching confidence. Two insights make this work. First, most inputs follow only 3 to 8 execution trajectories. This reduces the training problem from exponential to linear: we learn aggregators only for these common paths, not all possible model combinations. Second, we formulate trajectory selection as an MDP and use value iteration to compute the optimal routing policy, which reveals counterintuitive model orderings. On intrusion detection, starting with a moderately expensive model outperforms starting with the cheapest, because its higher confidence enables earlier overall exit. Across vision, text, tabular, and LLM tasks, we achieve up to 38x speedup at 97\%+ accuracy retention compared to a complete ensemble. The result: ensemble-quality predictions at cascade-level cost. Code and a project page are available at https://ashwincolaco.github.io/lazystack.

Modern AI systems often combine several models to get better answers, like asking for a second and third opinion before making a decision. But this is slow and expensive: every model gets asked, even on easy questions that one model alone could have handled. A common shortcut is to ask the cheapest model first and stop if it sounds confident, but a single model can be confidently wrong, so we lose the benefit of cross checking. We built a method called LazyStack. From past examples it learns a few useful orders in which to ask its models. After every model, it does not just look at that single answer: it pools together every answer it has seen so far, just as asking three experts together is more reliable than asking any one of them. The decision to stop is made on this pooled view, not on any one model alone. Easy questions are resolved after one or two models combine into a confident verdict; hard questions take a few more, but rarely all of them. We tested LazyStack on eight tasks, including recognising images, classifying news articles, detecting attacks on computer networks, and choosing which large AI assistant to use for a question. Because each stop decision came from the combined view rather than from any single model, LazyStack matched the accuracy of asking every model but was 2 to 38 times faster, typically using only 2 or 3 models per question. Surprisingly, it often learned that starting with a slightly more expensive model was better than the cheapest: that model's more confident first answer pulled the combined view above the stopping point sooner, saving more time than it cost.