Diffusion models have achieved impressive results in generating content across domains like images, videos, text, and audio. However, their sampling speed is a practical challenge due to repeated evaluation of score estimation networks during inference. To address this, we propose a novel framework that optimizes compute allocation for score estimation, reducing overall sampling time. Our key insight is that the computation required for score estimation varies at different time steps. Based on this observation, we introduce an early-exiting scheme that selectively skips the subset of parameters in the score estimation network during the inference, guided by a time-dependent exit schedule. We apply this technique to image synthesis with diffusion models and demonstrate significantly improved sampling throughput without compromising image quality. Moreover, our approach seamlessly integrates with various types of solvers for faster sampling, leveraging their compatibility to enhance overall efficiency.