In this paper, we present new high-probability PAC-Bayes bounds for losses with a bounded range. First, we develop a strengthened versionof Catoni’s bound that holds simultaneously for all parameter values. This leads to new fast rate and mixed rate bounds that are interpretable and tighter than previous bounds in the literature. All the presented bounds are easily extended to any-time valid thanks to the recent developments from Jang et al. (2023).