Deep reinforcement learning has shown potential for traffic signal control. However, the lack of explainability has limited its use in real-world conditions. In this work, we present a Deep Q-learning approach, with the SHAP framework, able to explain its policy. Our approach can explain the impact of features on each action, which promotes the understanding of how the agent behaves in the face of different traffic conditions. Furthermore, our approach improved travel time, waiting time, and speed by 21.49%, 27.97%, 20.87%, compared to fixed-time traffic signal controllers.