In high-stakes areas such as healthcare, it is interesting to ask counterfactual questions: what if some executed treatments had been performed earlier/later or changed to other types? Answering such questions can help us debug the observational treatment policies and further improve the treatment strategy. Existing methods mainly focus on generating the whole counterfactual trajectory, which provides overwhelming information and lacks specific feedback on improving certain actions. In this paper, we propose a counterfactual treatment optimization framework where we optimize specific treatment actions by sampling counterfactual symptom rollouts and meanwhile satisfying medical rule constraints. Our method can not only help people debug their specific treatments but also has strong robustness when training data are limited.