Multi-fidelity surrogate modeling aims to describe complex systems governed by partial differential equations with few high-fidelity data points and abundant low-fidelity data points. Recent works leverage deep neural networks and few-shot transfer learning to achieve good results on several high-dimensional surrogate modeling problems. However, these works treat "multi-fidelity" as "multi-resolution" where low-fidelity simulations are computed using the same algorithm as high-fidelity simulations but with coarser grids. In real practice, low-fidelity simulations are often computed by approximating hard-to-compute terms and neglecting physics that are difficult to model. The features learned from low-fidelity data are not useful for predicting phenomena caused by those ignored physics. During fine-tuning, new features that the model learns for these regions will be inaccurate and can corrupt the pre-trained features. This can create unnecessary uncertainty for the predictions of regions that are less dependent on ignored physics. To overcome this problem, we propose a multi-step transfer learning method that, in each step, adaptively relaxes the constraint on model weights and collects regional pseudo-high-fidelity data to enlarge the training set. Our experiments on modeling wind farm flow fields show that our method significantly outperforms vanilla transfer learning methods.