Recent years have witnessed the development of robust training to defend against the vulnerability of RL policies. Existing threat models impose static constraints on perturbations at each timestep and overlook the temporal influence of past perturbations on the current ones, despite its crucial consideration in many real-world scenarios. We formally introduce temporally-coupled attacks to account for the temporal coupling between perturbations at consecutive time steps, presenting a novel challenge for existing robust RL methods. To tackle this challenge, we propose GRAD, a novel game-theoretic response approach that treats the temporally-coupled robust RL problem as a partially-observable two-player game. By finding an approximate equilibrium in our approach, GRAD ensures the agent's robustness against the learned adversary. Empirical experiments on a variety of continuous control tasks demonstrate that our proposed approach exhibits significant robustness advantages compared to baselines against both standard and temporally-coupled attacks, in both the state and action spaces.