Due to the high cost in annotation, existing Image Quality Assessment (IQA) datasets are relatively small in scale. Consequently, achieving robust generalization remains a challenging task for prevalent deep learning-based IQA methods. In this paper, we propose a novel end-to-end blind IQA method called Causal-IQA that addresses this issue. Specifically, we first analyze the causal mechanisms in IQA tasks and construct a causal graph to understand the interplay and confounding effects between distortion types, image contents, and subjective human ratings. Then, through shifting the focus from correlations to causality, Causal-IQA aims to improve the estimation accuracy of image quality scores by mitigating the confounding effects using a causality-based optimization strategy. This optimization strategy is implemented on the sample subsets constructed by a Counterfactual Division process based on the Backdoor Criterion. Extensive experiments demonstrate the effectiveness and superiority of Causal-IQA, including the enhanced generalization capacity, interpretability, and adaptability.