Federated learning is a distributed optimization paradigm that allows training machine learning models across decentralized devices while keeping the data localized. The standard optimization method in FL, FedAvg, suffers from client drift, which can hamper performance and increase communication costs over centralized methods. Previous works proposed various strategies to mitigate drift. However, for all these methods, the best-known theoretical communication complexities do not improve over those of centralized gradient-based methods. In this work, we propose a novel framework FedRed that utilizes doubly regularized drift correction to address the communication bottleneck and allows the choice of arbitrary local solvers on the devices. We show that FedRed can exploit Hessian similarity and reduce communication when minimizing continuously differentiable functions. Moreover, when choosing the standard gradient descent as a local solver, the resulting algorithm consistently outperforms the centralized gradient descent regarding the communication-computation trade-off for the minimization of smooth functions.