Timezone: »

 
Poster
Datamodels: Understanding Predictions with Data and Data with Predictions
Andrew Ilyas · Sung Min (Sam) Park · Logan Engstrom · Guillaume Leclerc · Aleksander Madry

Wed Jul 20 03:30 PM -- 05:30 PM (PDT) @ Hall E #526
in Poster Session 2 »
We present a conceptual framework, \emph{datamodeling}, for analyzing the behavior of a model class in terms of the training data. For any fixed ``target'' example $x$, training set $S$, and learning algorithm, a {\em datamodel} is a parameterized function $2^S \to \mathbb{R}$ that for any subset of $S' \subset S$---using only information about which examples of $S$ are contained in $S'$---predicts the outcome of training a model on $S'$ and evaluating on $x$. Despite the complexity of the underlying process being approximated (e.g. end-to-end training and evaluation of deep neural networks), we show that even simple {\em linear} datamodels successfully predict model outputs. We then demonstrate that datamodels give rise to a variety of applications, such as:accurately predicting the effect of dataset counterfactuals; identifying brittle predictions; finding semantically similar examples; quantifying train-test leakage; and embedding data into a well-behaved and feature-rich representation space.
Keywords: DL: Everything Else · DL: Robustness · MISC: General Machine Learning Techniques

Author Information

Andrew Ilyas (Massachusetts Institute of Technology)
Sung Min (Sam) Park (MIT)
Logan Engstrom (MIT)
Guillaume Leclerc (MIT)
Aleksander Madry (MIT)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors