The objective of drug discovery is to identify effective chemical compounds that possess specific pharmaceutical properties toward a binding target. The existing large language models (LLMS) can achieve high token matching scores in terms oflikelihood for molecule generation. However, relying solely on LLMS decoding often results in the generation of invalid molecules due to a single misused token, or suboptimal molecules due to unbalanced exploration and exploitation as a consequence of the prior experience of LLMS. We propose ERP, Entropy-Reinforced Planning for Transformer Decoding, which utilizes an entropy-reinforced planning algorithm to enhance the transformer decoding process and strike a balance between exploitation and exploration. It aims to achieve improvements in multiple properties compared to direct sampling from Transformer. We evaluated ERP on the SARS-CoV-2virus (3CLPro) and human cancer cell target protein (RTCB) benchmarks and demonstrated ERP consistently outperforms the current state-of-the-art algorithm and competing baselines in both benchmarks. Moreover, it is capable of making significant improvement across Transformer models trained with different objectives.