Timezone: »

 
Counterfactual Metrics for Auditing Black-Box Recommender Systems for Ethical Concerns
Nil-Jana Akpinar · Liu Leqi · Dylan Hadfield-Menell · Zachary Lipton

@
in Responsible Decision Making in Dynamic Environments »

Recommender systems can shape peoples' online experience in powerful ways which makes close scrutiny of ethical implications imperative. Most existing work in this area attempts to measure induced harm exclusively based on observed recommendations under a set policy. This neglects potential dependencies on other quantities and can lead to misleading conclusions about the behavior of the algorithm. Instead, we propose counterfactual metrics for auditing recommender systems for ethical concerns. By asking how recommendations would change if users behaved differently or if the training data was different, we are able to isolate the effects of the recommendation algorithm from components like user preference and information. We discuss the ethical context of the suggested metrics and propose directions for future work.

Author Information

Nil-Jana Akpinar (Carnegie Mellon University)
Liu Leqi (Carnegie Mellon University)
Dylan Hadfield-Menell (Massachusetts Institute of Technology)
Zachary Lipton (Carnegie Mellon University)

More from the Same Authors