Explainable AI (XAI) techniques have becomepopular for multiple use-cases in the past fewyears. Here we consider its use in studying modelpredictions to gather additional training data. Weargue that this is equivalent to Active Learning,where the query strategy involves a human-in-the-loop. We provide a mathematical approximationfor the role of the human, and present a generalformalization of the end-to-end workflow. Thisenables us to rigorously compare this use withstandard Active Learning algorithms, while al-lowing for extensions to the workflow. An addedbenefit is that their utility can be assessed viasimulation instead of conducting expensive user-studies. We also present some initial promisingresults.