Mistakes as Epistemic Signatures: An Efficiency-Modulated Cumulative Error Framework for Comparison and Diagnosis of AI Errors

AI-human performance comparison using accuracy metrics alone could give an incomplete, possibly false, impression of processing or knowledge similarity between agents. A more reliable approach may be systematically studying these agents' errors to infer more about their underlying competencies and to what extent they overlap or differ. Here too, using only coarse statistical measures like error consistency could mask informative nuances in the data that could be signatures of processing strategy, and hence also relevant in tailoring interventions for error reduction. This work explores the possibility of a broad domain-agnostic framework or typology of errors applicable for any kind of artificial or biological agent, seeking to deploy it as a window into the epistemic nature of errors, and by extension the agents who commit them. We also argue that focus on error comparison alone could give a partial picture, and encourage a broader perspective on an agent's behavior to better capture the epistemology of learning systems.