Most deep learning models on drug-target affinity (DTA) prediction are black box hence are difficult to interpret and verify its result, and thus risking acceptance. Explanation is necessary to allow the DTA model more trustworthy. The interaction between sub-structure of two inputs, drug functional groups and protein residues, is an important factor in the DTA model prediction. Explanation based on substructure interaction allows domain experts to verify the binding mechanism used by DTA model in its prediction. We propose a multi-agent reinforcement learning framework, Multi-Agent Counterfactual Drug-target binding Affinity (MACDA), to generate counterfactual explanations for the drug-protein complex. Our proposed framework provides human-interpretable counterfactual instances while optimizing both the input drug and target for counterfactual generation at the same time. MACDA also explains the substructure interaction between inputs in the DTA model prediction.