Interpreting scattered acoustic and electromagnetic wave patterns is a computational task that enables remote imaging in a number of important applications, including medical imaging, geophysical exploration, sonar and radar detection, and nondestructive testing of materials. However, accurately and stably recovering an inhomogeneous medium from far-field scattered wave measurements is a computationally difficult problem, due to the nonlinear and non-local nature of the forward scattering process. We design aneural network, called Multi-Frequency Inverse Scattering Network with Refinement (MFISNet-Refinement), and a training method to approximate the inverse map from far-field scattered wave measurements at multiple frequencies. Our method is inspired by the recursive linearization method — a commonly used technique for stably inverting scattered wavefield data — that progressively refines the estimate with higher frequency content. MFISNet-Refinement outperforms existing methods in regimes with high-contrast, heterogeneous large objects, and inhomogeneous unknown backgrounds.