Timezone: »
There has been growing interest in rectifying deep neural network vulnerabilities. Challenges arise when models receive samples drawn from outside the training distribution. For example, a neural network tasked with classifying handwritten digits may assign high confidence predictions to cat images. Anomalies are frequently encountered when deploying ML models in the real world. Well-calibrated predictive uncertainty estimates are indispensable for many machine learning applications, such as self-driving cars and medical diagnosis systems. Generalization to unseen and worst-case inputs is also essential for robustness to distributional shift. In order to have ML models reliably predict in open environment, we must deepen technical understanding in the following areas: (1) learning algorithms that are robust to changes in input data distribution (e.g., detect out-of-distribution examples); (2) mechanisms to estimate and calibrate confidence produced by neural networks and (3) methods to improve robustness to adversarial and common corruptions, and (4) key applications for uncertainty such as in artificial intelligence (e.g., computer vision, robotics, self-driving cars, medical imaging) as well as broader machine learning tasks.
This workshop will bring together researchers and practitioners from the machine learning communities, and highlight recent work that contribute to address these challenges. Our agenda will feature contributed papers with invited speakers. Through the workshop we hope to help identify fundamentally important directions on robust and reliable deep learning, and foster future collaborations.
|
Fri 8:30 a.m. - 8:40 a.m.
|
Welcome
|
Sharon Yixuan Li 🔗 |
|
Fri 8:40 a.m. - 9:30 a.m.
|
Spotlight
|
Tyler Scott · Kiran Koshy · Jonathan Aigrain · Rene Bidart · Priyadarshini Panda · Dian Ang Yap · Yaniv Yacoby · Raphael Gontijo Lopes · Alberto Marchisio · Erik Englesson · Wanqian Yang · Moritz Graule · Yi Sun · Daniel Kang · Mike Dusenberry · Min Du · Hartmut Maennel · Kunal Menda · Vineet Edupuganti · Luke Metz · David Stutz · Vignesh Srinivasan · Timo Sämann · Vineeth N Balasubramanian · Sina Mohseni · Rob Cornish · Judith Butepage · Zhangyang Wang · Bai Li · Bo Han · Honglin Li · Maksym Andriushchenko · Lukas Ruff · Meet P. Vadera · Yaniv Ovadia · Sunil Thulasidasan · Disi Ji · Gang Niu · Saeed Mahloujifar · Aviral Kumar · SANGHYUK CHUN · Dong Yin · Joyce Xu Xu · Hugo Gomes · Raanan Rohekar
|
|
Fri 9:30 a.m. - 10:00 a.m.
|
Keynote by Max Welling: A Nonparametric Bayesian Approach to Deep Learning (without GPs)
(
Keynote
)
We present a new family of exchangeable stochastic processes suitable for deep learning. Our nonparametric Bayesian method models distributions over functions by learning a graph of dependencies on top of latent representations of the points in the given dataset. In doing so, they define a Bayesian model without explicitly positing a prior distribution over latent global parameters; they instead adopt priors over the relational structure of the given dataset, a task that is much simpler. We show how we can learn such models from data, demonstrate that they are scalable to large datasets through mini-batch optimization and describe how we can make predictions for new points via their posterior predictive distribution. We experimentally evaluate FNPs on the tasks of toy regression and image classification and show that, when compared to baselines that employ global latent parameters, they offer both competitive predictions as well as more robust uncertainty estimates. |
Max Welling 🔗 |
|
Fri 10:00 a.m. - 11:00 a.m.
|
Poster Session 1 (all papers)
(
Poster
)
|
🔗 |
|
Fri 11:00 a.m. - 11:30 a.m.
|
Keynote by Kilian Weinberger: On Calibration and Fairness
(
Keynote
)
We investigate calibration for deep learning algorithms in classification and regression settings. Although we show that typically deep networks tend to be highly mis-calibrated, we demonstrate that this is easy to fix - either to obtain more trustworthy confidence estimates or to detect outliers in the data. Finally, we relate calibration with the recently raised tension between minimizing error disparity across different population groups while maintaining calibrated probability estimates. We show that calibration is compatible only with a single error constraint (i.e. equal false-negatives rates across groups), and show that any algorithm that satisfies this relaxation is no better than randomizing a percentage of predictions for an existing classifier. These unsettling findings, which extend and generalize existing results, are empirically confirmed on several datasets. |
Kilian Weinberger 🔗 |
|
Fri 11:30 a.m. - 11:40 a.m.
|
Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem
(
Contributed talk
)
Classifiers used in the wild, in particular for safety-critical systems, should know when they don’t know, in particular make low confidence predictions far away from the train- ing data. We show that ReLU type neural networks fail in this regard as they produce almost always high confidence predictions far away from the training data. For bounded domains we propose a new robust optimization technique similar to adversarial training which enforces low confidence pre- dictions far away from the training data. We show that this technique is surprisingly effective in reducing the confidence of predictions far away from the training data while maintaining high confidence predictions and test error on the original classification task com- pared to standard training. This is a short version of the corresponding CVPR paper. |
Maksym Andriushchenko 🔗 |
|
Fri 11:40 a.m. - 11:50 a.m.
|
Detecting Extrapolation with Influence Functions
(
Contributed talk
)
In this work, we explore principled methods for extrapolation detection. We define extrapolation as occurring when a model’s conclusion at a test point is underdetermined by the training data. Our metrics for detecting extrapolation are based on influence functions, inspired by the intuition that a point requires extrapolation if its inclusion in the training set would significantly change the model’s learned parameters. We provide interpretations of our methods in terms of the eigendecomposition of the Hessian. We present experimental evidence that our method is capable of identifying extrapolation to out-of-distribution points. |
David Madras 🔗 |
|
Fri 11:50 a.m. - 12:00 p.m.
|
How Can We Be So Dense? The Robustness of Highly Sparse Representations
(
Contributed talk
)
Neural networks can be highly sensitive to noise and perturbations. In this paper we suggest that high dimensional sparse representations can lead to increased robustness to noise and interference. A key intuition we develop is that the ratio of the match volume around a sparse vector divided by the total representational space decreases exponentially with dimensionality, leading to highly robust matching with low interference from other patterns. We analyze efficient sparse networks containing both sparse weights and sparse activations. Simulations on MNIST, the Google Speech Command Dataset, and CIFAR-10 show that such networks demonstrate improved robustness to random noise compared to dense networks, while maintaining competitive accuracy. We propose that sparsity should be a core design constraint for creating highly robust networks. |
Subutai Ahmad 🔗 |
|
Fri 12:00 p.m. - 12:30 p.m.
|
Keynote by Suchi Saria: Safety Challenges with Black-Box Predictors and Novel Learning Approaches for Failure Proofing
(
Keynote
)
|
Suchi Saria 🔗 |
|
Fri 2:00 p.m. - 2:10 p.m.
|
Subspace Inference for Bayesian Deep Learning
(
Contributed talk
)
Bayesian inference was once a gold standard for learning with neural networks, providing accurate full predictive distributions and well calibrated uncertainty. However, scaling Bayesian inference techniques to deep neural networks is challenging due to the high dimensionality of the parameter space. In this pa- per, we construct low-dimensional subspaces of parameter space that contain diverse sets of models, such as the first principal components of the stochastic gradient descent (SGD) trajectory. In these subspaces, we are able to apply elliptical slice sampling and variational inference, which struggle in the full parameter space. We show that Bayesian model averaging over the induced posterior in these subspaces produces high accurate predictions and well-calibrated predictive uncertainty for both regression and image classification. |
Polina Kirichenko · Pavel Izmailov · Andrew Wilson 🔗 |
|
Fri 2:10 p.m. - 2:20 p.m.
|
Quality of Uncertainty Quantification for Bayesian Neural Network Inference
(
Contributed talk
)
Bayesian Neural Networks (BNNs) place priors over the parameters in a neural network. Inference in BNNs, however, is difficult; all inference methods for BNNs are approximate. In this work, we empirically compare the quality of predictive uncertainty estimates for 10 common inference methods on both regression and classification tasks. Our experiments demonstrate that commonly used metrics (e.g. test log-likelihood) can be misleading. Our experiments also indicate that inference innovations designed to capture structure in the posterior do not necessarily produce high quality posterior approximations. |
Jiayu Yao 🔗 |
|
Fri 2:20 p.m. - 2:30 p.m.
|
'In-Between' Uncertainty in Bayesian Neural Networks
(
Contributed talk
)
We describe a limitation in the expressiveness of the predictive uncertainty estimate given by mean- field variational inference (MFVI), a popular approximate inference method for Bayesian neural networks. In particular, MFVI fails to give calibrated uncertainty estimates in between separated regions of observations. This can lead to catastrophically overconfident predictions when testing on out-of-distribution data. Avoiding such over-confidence is critical for active learning, Bayesian optimisation and out-of-distribution robustness. We instead find that a classical technique, the linearised Laplace approximation, can handle ‘in- between’ uncertainty much better for small network architectures. |
Yue Kwang Foong 🔗 |
|
Fri 2:30 p.m. - 3:00 p.m.
|
Keynote by Dawn Song: Adversarial Machine Learning: Challenges, Lessons, and Future Directions
(
Keynote
)
|
Dawn Song 🔗 |
|
Fri 3:30 p.m. - 4:00 p.m.
|
Keynote by Terrance Boult: The Deep Unknown: on Open-set and Adversarial Examples in Deep Learning
(
Keynote
)
The first part of the talk will explore issues with deep networks dealing with "unknowns" inputs, and the general problems of open-set recognition in deep networks. We review the core of open-set recognition theory and its application in our first attempt at open-set deep networks, "OpenMax" We discuss is successes and limitations and why classic "open-set" approaches don't really solve the problem of deep unknowns. We then present our recent work from NIPS2018, on a new model we call the ObjectoSphere. Using ObjectoSphere loss begins to address the learning of deep features that can handle unknown inputs. We present examples of its use first on simple datasets sets (MNIST/CFAR) and then onto unpublished work applying it to the real-world problem of open-set face recognition. We discuss of the relationship between open set recognition theory and adversarial image generation, showing how our deep-feature adversarial approach, called LOTS can attack the first OpenMax solution, as well as successfully attack even open-set face recognition systems. We end with a discussion of how open set theory can be applied to improve network robustness. |
Terrance Boult 🔗 |
|
Fri 4:00 p.m. - 5:00 p.m.
|
Panel Discussion (moderator: Tom Dietterich)
(
Panel
)
|
Max Welling · Kilian Weinberger · Terrance Boult · Dawn Song · Thomas Dietterich 🔗 |
|
Fri 5:00 p.m. - 6:00 p.m.
|
Poster Session 2 (all papers)
(
Poster
)
|
🔗 |
Author Information
Sharon Yixuan Li (Facebook AI)
Sharon Y. Li is currently a postdoc researcher in the Computer Science department at Stanford, working with Chris Ré. She will be joining the Computer Sciences Department at University of Wisconsin Madison as an assistant professor, starting in Fall 2020. Previously, she completed her PhD from Cornell University in 2017, where she was advised by John E. Hopcroft. Her thesis committee members are Kilian Q. Weinberger and Thorsten Joachims. She has spent time at Google AI twice as an intern, and Facebook AI as a Research Scientist. She was named Forbes 30 Under 30 in Science in 2020. Her principal research interests are in the algorithmic foundations of deep learning and its applications. Her time in both academia and industry has shaped my view and approach in research. She is particularly excited about developing open-world machine learning methods that can reduce human supervision during training, and enhance reliability during deployment.
Dan Hendrycks (UC Berkeley)
Thomas Dietterich (Oregon State University)
Balaji Lakshminarayanan (Google DeepMind)
Justin Gilmer (Google Brain)
More from the Same Authors
-
2021 : A simple fix to Mahalanobis distance for improving near-OOD detection »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Precise characterization of the prior predictive distribution of deep ReLU networks »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effect »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Repulsive Deep Ensembles are Bayesian »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Calibrated Out-of-Distribution Detection with Conformal P-values »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Are Bayesian neural networks intrinsically good at out-of-distribution detection? »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Provably Robust Detection of Out-of-distribution Data (almost) for free »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Out-of-Distribution Dynamics Detection: RL-Relevant Benchmarks and Results »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Rethinking Assumptions in Deep Anomaly Detection »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Multiple Moment Matching Inference: A Flexible Approximate Inference Algorithm »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : PAC Prediction Sets Under Covariate Shift »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Correct-N-Contrast: a Contrastive Approach for Improving Robustness to Spurious Correlations »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Do We Really Need to Learn Representations from In-domain Data for Outlier Detection? »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : DATE: Detecting Anomalies in Text via Self-Supervision of Transformers »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantification »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Implicit Ensemble Training for Efficient and Robust Multiagent Reinforcement Learning »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Failures of Uncertainty Estimation on Out-Of-Distribution Samples: Experimental Results from Medical Applications Lead to Theoretical Insights »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : On Out-of-distribution Detection with Energy-Based Models »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Deterministic Neural Networks with Inductive Biases Capture Epistemic and Aleatoric Uncertainty »
Andreas Kirsch · Balaji Lakshminarayanan · Jasper Snoek -
2021 : Transfer and Marginalize: Explaining Away Label Noise with Privileged Information »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Meta-Calibration: Meta-Learning of Model Calibration Using Differentiable Expected Calibration Error »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Inferring Black Hole Properties from Astronomical Multivariate Time Series with Bayesian Attentive Neural Processes »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Towards improving robustness of compressed CNNs »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : SAND-mask: An Enhanced Gradient Masking Strategy for Invariant Prediction in Domain Generalization »
Soroosh Shahtalebi · Jasper Snoek · Balaji Lakshminarayanan -
2021 : Efficient Gaussian Neural Processes for Regression »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Simple, Attack-Agnostic Defense Against Targeted Training Set Attacks Using Cosine Similarity »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Safety & Exploration: A Comparative Study of Uses of Uncertainty in Reinforcement Learning »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Rethinking Function-Space Variational Inference in Bayesian Neural Networks »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Understanding the Under-Coverage Bias in Uncertainty Estimation »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : BETH Dataset: Real Cybersecurity Data for Anomaly Detection Research »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Mean Embeddings with Test-Time Data Augmentation for Ensembling of Representations »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Deep Ensemble Uncertainty Fails as Network Width Increases: Why, and How to Fix It »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Exact and Efficient Adversarial Robustness with Decomposable Neural Networks »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Consistency Regularization for Training Confidence-Calibrated Classifiers »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Diverse and Amortised Counterfactual Explanations for Uncertainty Estimates »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Quantization of Bayesian neural networks and its effect on quality of uncertainty »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Class-Distribution-Aware Calibration for Long-Tailed Visual Recognition »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Bayesian Neural Networks with Soft Evidence »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Anomaly Detection for Event Data with Temporal Point Processes »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Batch Inverse-Variance Weighting: Deep Heteroscedastic Regression »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : An Empirical Study of Invariant Risk Minimization on Deep Models »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : A Bayesian Approach to Invariant Deep Neural Networks »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Practical posterior Laplace approximation with optimization-driven second moment estimation »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Variational Generative Flows for Reconstruction Uncertainty Estimation »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Improving the Accuracy-Robustness Trade-Off for Dual-Domain Adversarial Training »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Consistency Regularization Can Improve Robustness to Label Noise »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Neural Variational Gradient Descent »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Evaluating the Use of Reconstruction Error for Novelty Localization »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Accuracy on the Line: on the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : The Hidden Uncertainty in a Neural Network’s Activations »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : On the Calibration of Deterministic Epistemic Uncertainty »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Objective Robustness in Deep Reinforcement Learning »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Epistemic Uncertainty in Learning Chaotic Dynamical Systems »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Towards Stochastic Neural Networks via Inductive Wasserstein Embeddings »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Distribution-free uncertainty quantification for classification under label shift »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : How does a Neural Network's Architecture Impact its Robustness to Noisy Labels? »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Top-label calibration »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Learning to Align the Support of Distributions »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Beyond First-Order Uncertainty Estimation with Evidential Models for Open-World Recognition »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Revisiting Out-of-Distribution Detection: A Simple Baseline is Surprisingly Effective »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Contrastive Predictive Coding for Anomaly Detection and Segmentation »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Multi-headed Neural Ensemble Search »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : A variational approximate posterior for the deep Wishart process »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : What Are Effective Labels for Augmented Data? Improving Calibration and Robustness with AutoLabel »
Yao Qin · Jasper Snoek · Balaji Lakshminarayanan -
2021 : On Stein Variational Neural Network Ensembles »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Uncertainty-Aware Boosted Ensembling in Multi-Modal Settings »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : RouBL: A computationally cheap way to go beyond mean-field variational inference »
Sahar Karimi · Balaji Lakshminarayanan · Jasper Snoek -
2021 : No True State-of-the-Art? OOD Detection Methods are Inconsistent across Datasets »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Out-of-Distribution Generalization with Deep Equilibrium Models »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Mixture Proportion Estimation and PU Learning: A Modern Approach »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : On The Dark Side Of Calibration For Modern Neural Networks »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Domain Adaptation with Factorizable Joint Shift »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Scaling Laws for the Out-of-Distribution Generalization of Image Classifiers »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Learning Invariant Weights in Neural Networks »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Relational Deep Reinforcement Learning and Latent Goals for Following Instructions in Temporal Logic »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : On the Effectiveness of Mode Exploration in Bayesian Model Averaging for Neural Networks »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Training-Free Uncertainty Estimation for Dense Regression: Sensitivity as a Surrogate »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Detecting OODs as datapoints with High Uncertainty »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Multi-task Transformation Learning for Robust Out-of-Distribution Detection »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Directly Training Joint Energy-Based Models for Conditional Synthesis and Calibrated Prediction of Multi-Attribute Data »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Deep Learning with Quantified Uncertainty for Free Electron Laser Scientific Facilities »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : On the reversed bias-variance tradeoff in deep ensembles »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Robust Generalization of Quadratic Neural Networks via Function Identification »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Exploring Corruption Robustness: Inductive Biases in Vision Transformers and MLP-Mixers »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Deep Random Projection Outlyingness for Unsupervised Anomaly Detection »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Deep Deterministic Uncertainty for Semantic Segmentation »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Identifying Invariant and Sparse Predictors in High-dimensional Data »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : On Misclassification-Aware Smoothing for Robustness and Uncertainty Calibration »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : On Pitfalls in OoD Detection: Entropy Considered Harmful »
Andreas Kirsch · Jasper Snoek · Balaji Lakshminarayanan -
2021 : PnPOOD : Out-Of-Distribution Detection for Text Classification via Plug andPlay Data Augmentation »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Augmented Invariant Regularization »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Model-Based Robust Deep Learning: Generalizing to Natural, Out-of-Distribution Data »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Improved Adversarial Robustness via Uncertainty Targeted Attacks »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Notes on the Behavior of MC Dropout »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Distribution-free Risk-controlling Prediction Sets »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Stochastic Bouncy Particle Sampler for Bayesian Neural Networks »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Novelty detection using ensembles with regularized disagreement »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : A Tale Of Two Long Tails »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Defending against Adversarial Patches with Robust Self-Attention »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Intrinsic uncertainties and where to find them »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Dataset to Dataspace: A Topological-Framework to Improve Analysis of Machine Learning Model Performance »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Analyzing And Improving Neural Networks By Generating Semantic Counterexamples Through Differentiable Rendering »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Thinkback: Task-Specific Out-of-Distribution Detection »
Jasper Snoek · Balaji Lakshminarayanan -
2021 : Relating Adversarially Robust Generalization to Flat Minima »
Balaji Lakshminarayanan · Jasper Snoek -
2021 : Deep Quantile Aggregation »
Balaji Lakshminarayanan · Jasper Snoek -
2022 : Plex: Towards Reliability using Pretrained Large Model Extensions »
Dustin Tran · Andreas Kirsch · Balaji Lakshminarayanan · Huiyi Hu · Du Phan · D. Sculley · Jasper Snoek · Jeremiah Liu · Jie Ren · Joost van Amersfoort · Kehang Han · E. Kelly Buchanan · Kevin Murphy · Mark Collier · Mike Dusenberry · Neil Band · Nithum Thain · Rodolphe Jenatton · Tim G. J Rudner · Yarin Gal · Zachary Nado · Zelda Mariet · Zi Wang · Zoubin Ghahramani -
2022 : Plex: Towards Reliability using Pretrained Large Model Extensions »
Dustin Tran · Andreas Kirsch · Balaji Lakshminarayanan · Huiyi Hu · Du Phan · D. Sculley · Jasper Snoek · Jeremiah Liu · JIE REN · Joost van Amersfoort · Kehang Han · Estefany Kelly Buchanan · Kevin Murphy · Mark Collier · Michael Dusenberry · Neil Band · Nithum Thain · Rodolphe Jenatton · Tim G. J Rudner · Yarin Gal · Zachary Nado · Zelda Mariet · Zi Wang · Zoubin Ghahramani -
2023 : Building One-class Detector for Anything: Open-vocabulary Zero-shot OOD Detection Using Text-image Models »
Yunhao Ge · Jie Ren · Jiaping Zhao · Kaifeng Chen · Andrew Gallagher · Laurent Itti · Balaji Lakshminarayanan -
2023 : Morse Neural Networks for Uncertainty Quantification »
Benoit Dherin · Huiyi Hu · JIE REN · Michael Dusenberry · Balaji Lakshminarayanan -
2023 Poster: Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the Machiavelli Benchmark »
Alexander Pan · Jun Shern Chan · Andy Zou · Nathaniel Li · Steven Basart · Thomas Woodside · Hanlin Zhang · Scott Emmons · Dan Hendrycks -
2023 Poster: Scaling Vision Transformers to 22 Billion Parameters »
Mostafa Dehghani · Josip Djolonga · Basil Mustafa · Piotr Padlewski · Jonathan Heek · Justin Gilmer · Andreas Steiner · Mathilde Caron · Robert Geirhos · Ibrahim Alabdulmohsin · Rodolphe Jenatton · Lucas Beyer · Michael Tschannen · Anurag Arnab · Xiao Wang · Carlos Riquelme · Matthias Minderer · Joan Puigcerver · Utku Evci · Manoj Kumar · Sjoerd van Steenkiste · Gamaleldin Elsayed · Aravindh Mahendran · Fisher Yu · Avital Oliver · Fantine Huot · Jasmijn Bastings · Mark Collier · Alexey Gritsenko · Vighnesh N Birodkar · Cristina Vasconcelos · Yi Tay · Thomas Mensink · Alexander Kolesnikov · Filip Pavetic · Dustin Tran · Thomas Kipf · Mario Lucic · Xiaohua Zhai · Daniel Keysers · Jeremiah Harmsen · Neil Houlsby -
2023 Poster: A Simple Zero-shot Prompt Weighting Technique to Improve Prompt Ensembling in Text-Image Models »
James Allingham · JIE REN · Michael Dusenberry · Xiuye Gu · Yin Cui · Dustin Tran · Jeremiah Liu · Balaji Lakshminarayanan -
2023 Oral: Scaling Vision Transformers to 22 Billion Parameters »
Mostafa Dehghani · Josip Djolonga · Basil Mustafa · Piotr Padlewski · Jonathan Heek · Justin Gilmer · Andreas Steiner · Mathilde Caron · Robert Geirhos · Ibrahim Alabdulmohsin · Rodolphe Jenatton · Lucas Beyer · Michael Tschannen · Anurag Arnab · Xiao Wang · Carlos Riquelme · Matthias Minderer · Joan Puigcerver · Utku Evci · Manoj Kumar · Sjoerd van Steenkiste · Gamaleldin Elsayed · Aravindh Mahendran · Fisher Yu · Avital Oliver · Fantine Huot · Jasmijn Bastings · Mark Collier · Alexey Gritsenko · Vighnesh N Birodkar · Cristina Vasconcelos · Yi Tay · Thomas Mensink · Alexander Kolesnikov · Filip Pavetic · Dustin Tran · Thomas Kipf · Mario Lucic · Xiaohua Zhai · Daniel Keysers · Jeremiah Harmsen · Neil Houlsby -
2023 Oral: Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the Machiavelli Benchmark »
Alexander Pan · Jun Shern Chan · Andy Zou · Nathaniel Li · Steven Basart · Thomas Woodside · Hanlin Zhang · Scott Emmons · Dan Hendrycks -
2022 : Plex: Towards Reliability using Pretrained Large Model Extensions »
Dustin Tran · Andreas Kirsch · Balaji Lakshminarayanan · Huiyi Hu · Du Phan · D. Sculley · Jasper Snoek · Jeremiah Liu · JIE REN · Joost van Amersfoort · Kehang Han · Estefany Kelly Buchanan · Kevin Murphy · Mark Collier · Michael Dusenberry · Neil Band · Nithum Thain · Rodolphe Jenatton · Tim G. J Rudner · Yarin Gal · Zachary Nado · Zelda Mariet · Zi Wang · Zoubin Ghahramani -
2022 Poster: Scaling Out-of-Distribution Detection for Real-World Settings »
Dan Hendrycks · Steven Basart · Mantas Mazeika · Andy Zou · joseph kwon · Mohammadreza Mostajabi · Jacob Steinhardt · Dawn Song -
2022 Spotlight: Scaling Out-of-Distribution Detection for Real-World Settings »
Dan Hendrycks · Steven Basart · Mantas Mazeika · Andy Zou · joseph kwon · Mohammadreza Mostajabi · Jacob Steinhardt · Dawn Song -
2021 Workshop: A Blessing in Disguise: The Prospects and Perils of Adversarial Machine Learning »
Hang Su · Yinpeng Dong · Tianyu Pang · Eric Wong · Zico Kolter · Shuo Feng · Bo Li · Henry Liu · Dan Hendrycks · Francesco Croce · Leslie Rice · Tian Tian -
2021 : Live Panel Discussion »
Thomas Dietterich · Chelsea Finn · Kamalika Chaudhuri · Yarin Gal · Uri Shalit -
2021 : RL Foundation Panel »
Matthew Botvinick · Thomas Dietterich · Leslie Kaelbling · John Langford · Warrren B Powell · Csaba Szepesvari · Lihong Li · Yuxi Li -
2021 Workshop: Uncertainty and Robustness in Deep Learning »
Balaji Lakshminarayanan · Dan Hendrycks · Sharon Li · Jasper Snoek · Silvia Chiappa · Sebastian Nowozin · Thomas Dietterich -
2021 : Welcome »
Balaji Lakshminarayanan -
2020 : Keynote #5 Justin Gilmer »
Justin Gilmer -
2020 Workshop: Uncertainty and Robustness in Deep Learning Workshop (UDL) »
Sharon Yixuan Li · Balaji Lakshminarayanan · Dan Hendrycks · Thomas Dietterich · Jasper Snoek -
2020 : Opening Remarks »
Sharon Yixuan Li -
2020 Poster: Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors »
Mike Dusenberry · Ghassen Jerfel · Yeming Wen · Yian Ma · Jasper Snoek · Katherine Heller · Balaji Lakshminarayanan · Dustin Tran -
2019 : Panel Discussion (moderator: Tom Dietterich) »
Max Welling · Kilian Weinberger · Terrance Boult · Dawn Song · Thomas Dietterich -
2019 : Welcome »
Sharon Yixuan Li -
2019 Poster: Learning from Delayed Outcomes via Proxies with Applications to Recommender Systems »
Timothy Mann · Sven Gowal · Andras Gyorgy · Huiyi Hu · Ray Jiang · Balaji Lakshminarayanan · Prav Srinivasan -
2019 Oral: Learning from Delayed Outcomes via Proxies with Applications to Recommender Systems »
Timothy Mann · Sven Gowal · Andras Gyorgy · Huiyi Hu · Ray Jiang · Balaji Lakshminarayanan · Prav Srinivasan -
2019 Oral: Hybrid Models with Deep and Invertible Features »
Eric Nalisnick · Akihiro Matsukawa · Yee-Whye Teh · Dilan Gorur · Balaji Lakshminarayanan -
2019 Poster: Adversarial Examples Are a Natural Consequence of Test Error in Noise »
Justin Gilmer · Nicolas Ford · Nicholas Carlini · Ekin Dogus Cubuk -
2019 Poster: Hybrid Models with Deep and Invertible Features »
Eric Nalisnick · Akihiro Matsukawa · Yee-Whye Teh · Dilan Gorur · Balaji Lakshminarayanan -
2019 Oral: Adversarial Examples Are a Natural Consequence of Test Error in Noise »
Justin Gilmer · Nicolas Ford · Nicholas Carlini · Ekin Dogus Cubuk -
2019 Poster: Using Pre-Training Can Improve Model Robustness and Uncertainty »
Dan Hendrycks · Kimin Lee · Mantas Mazeika -
2019 Oral: Using Pre-Training Can Improve Model Robustness and Uncertainty »
Dan Hendrycks · Kimin Lee · Mantas Mazeika -
2018 Poster: Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV) »
Been Kim · Martin Wattenberg · Justin Gilmer · Carrie Cai · James Wexler · Fernanda Viégas · Rory sayres -
2018 Oral: Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV) »
Been Kim · Martin Wattenberg · Justin Gilmer · Carrie Cai · James Wexler · Fernanda Viégas · Rory sayres -
2018 Poster: Discovering and Removing Exogenous State Variables and Rewards for Reinforcement Learning »
Thomas Dietterich · George Trimponias · Zhitang Chen -
2018 Poster: Open Category Detection with PAC Guarantees »
Si Liu · Risheek Garrepalli · Thomas Dietterich · Alan Fern · Dan Hendrycks -
2018 Oral: Discovering and Removing Exogenous State Variables and Rewards for Reinforcement Learning »
Thomas Dietterich · George Trimponias · Zhitang Chen -
2018 Oral: Open Category Detection with PAC Guarantees »
Si Liu · Risheek Garrepalli · Thomas Dietterich · Alan Fern · Dan Hendrycks -
2017 Workshop: Implicit Generative Models »
Rajesh Ranganath · Ian Goodfellow · Dustin Tran · David Blei · Balaji Lakshminarayanan · Shakir Mohamed -
2017 Poster: Neural Message Passing for Quantum Chemistry »
Justin Gilmer · Samuel Schoenholz · Patrick F Riley · Oriol Vinyals · George Dahl -
2017 Talk: Neural Message Passing for Quantum Chemistry »
Justin Gilmer · Samuel Schoenholz · Patrick F Riley · Oriol Vinyals · George Dahl -
2017 Poster: Input Switched Affine Networks: An RNN Architecture Designed for Interpretability »
Jakob Foerster · Justin Gilmer · Jan Chorowski · Jascha Sohl-Dickstein · David Sussillo -
2017 Talk: Input Switched Affine Networks: An RNN Architecture Designed for Interpretability »
Jakob Foerster · Justin Gilmer · Jan Chorowski · Jascha Sohl-Dickstein · David Sussillo