Joint Model and Data Sparsification via the Marginal Likelihood

Sparse recovery in linear systems underpins applications from signal processing to high-dimensional regression. Sparse Bayesian Learning, grounded in the principle of automatic relevance determination (ARD), offers a practical Bayesian mechanism for feature sparsity via marginal likelihood optimization. Yet, its reliance on a homoscedastic noise model renders it sensitive to data contaminations such as outliers or misspecified noise, harming model fit and predictions. Instead, we propose jointly learning individual feature and sample relevancies, enabling simultaneous model and data sparsification via a single Bayesian objective. This symmetric pruning of model and data offers a natural extension that preserves conjugacy, admits closed-form updates for standard optimization procedures, and aligns with perspectives from robust regression and influence functions. Empirical results across diverse regression tasks affirm that a joint ARD approach consistently yields both sparse and robust prediction models.

Many prediction models try to find the few inputs that really matter and ignore the rest. Sparse Bayesian learning is one way to do this automatically. However, it usually treats every training example as equally trustworthy, which can be a problem when some data points are outliers, corrupted, or unusually noisy. We extend this method so that it learns two things at once: which inputs are useful and which training examples are reliable. Data points that look suspicious are automatically given less influence during training. This lets the model do feature selection and data cleaning within the same Bayesian framework, while keeping the computations efficient. Across several regression tasks, this produces models that are simpler and more robust to noisy data. It also helps flag potentially unreliable samples, giving users a clearer view of both what the model relies on and which data points may be problematic.