Consistency Deep Equilibrium Models

Deep Equilibrium Models (DEQs) have emerged as a powerful paradigm in deep learning, offering the ability to model infinite-depth networks with constant memory usage. However, DEQs incur significant inference latency due to the iterative nature of fixed-point solvers. In this work, we introduce the Consistency Deep Equilibrium Model (C-DEQ), a novel framework that leverages consistency distillation to accelerate DEQ inference. We cast the DEQ iterative inference process as evolution along a fixed ODE trajectory toward the equilibrium. Along this trajectory, we train C-DEQs to consistently map intermediate states directly to the fixed point, enabling few-step inference while preserving the performance of the teacher DEQ. At the same time, it facilitates multi-step evaluation to flexibly trade computation for performance gains. Extensive experiments across various domain tasks demonstrate that C-DEQs achieve consistent 2-20$\times$ accuracy improvements over implicit DEQs under the same few-step inference budget. Our code is available at \url{https://github.com/landrarwolf/CDEQ}.

Many modern AI models make predictions by passing information through many layers. Deep equilibrium models take a different approach: they repeatedly refine an internal representation until it settles into a stable answer. This design can be memory-efficient and powerful, but it often makes prediction slow because the model must repeat this refinement many times. In this paper, we introduce Consistency Deep Equilibrium Models, or C-DEQ, a method that teaches such models to reach their final stable answer much faster. Instead of running many repeated refinement steps, C-DEQ learns from the trajectory of a trained teacher model and predicts the final answer from any intermediate states in only a few steps. It can also use more steps when higher accuracy is needed. We test C-DEQ on language modeling, image classification, and graph learning tasks. Across these settings, C-DEQ greatly improves few-step prediction quality and reduces inference time, making deep equilibrium models more practical for real-world deployment.