Deep learning-based image restoration methods generally struggle with faithfully preserving the structures of the original image. In this work, we propose a two-pass Residual-Conditioned Optimal Transport (RCOT) approach, which incorporates the degradation-specific information by introducing the degradation domain gap (estimated by transport residual) from an optimal transport (OT) perspective. Specifically, we design a frequency sparsity-aware OT (FSOT) criterion that exploits the frequency sparsity in the degradation domain gap. By duality, the OT problem with FSOT criterion is equivalent to a minimax problem for seeking the OT map. We design the RCOT map as a two-pass map, in which the second pass generates the refined image conditioned on the intermediate degradation-specific residual embedding obtained from the first pass via the transport residual condition module. Experiments show that RCOT achieves state-of-the-art performance for multiple types of image degradation, well preserving structures in the restored images.