Automatic Unsupervised Ensemble Outlier Model Selection

Unsupervised outlier detection is attractive because it eliminates the need for labeled data. Moreover, forming multi-model ensembles can improve detection robustness. However, composing an ensemble without labeled data is challenging. Naively composed ensembles can suffer from ensemble saturation, where redundant or unreliable detection models degrade performance and incur unnecessary computation. We propose MetaEns, an automatic unsupervised framework for selecting ensembles of outlier detection models. Using labeled meta-datasets, MetaEns learns a model that predicts marginal ensemble gains, estimating the expected improvement from adding a candidate model to a partially constructed ensemble. At test time, this learned signal is combined with a submodular-inspired proxy objective that enforces diminishing returns through diversity-aware discounting and family-level risk regularization, thereby enabling reedy sequential selection with adaptive early stopping. As a result, MetaEns constructs compact, high-quality ensembles without access to ground-truth labels. Experiments on 39 real-world datasets show that MetaEns consistently outperforms state-of-the-art unsupervised selectors and ensemble baselines, achieving higher average precision while using fewer models.

Many real-world systems, such as fraud detection, medical screening, and infrastructure monitoring, need to identify unusual cases without having labeled examples to guide them. In these settings, using several detectors together can improve reliability, but it is not obvious which detectors should be combined, or how many should be used. Adding too many can introduce redundant or low-quality detectors, which may hurt performance and waste computation. We propose MetaEns, a method that learns from past labeled datasets how much benefit a detector is likely to add to an existing ensemble. For a new unlabeled dataset, MetaEns builds the ensemble step by step, choosing detectors that are expected to contribute new useful information and stopping automatically when further additions are unlikely to help. Across 39 benchmark datasets, MetaEns consistently outperforms existing unsupervised selection methods while using only a small number of detectors on average. This makes anomaly detection more accurate, more efficient, and more practical in situations where labeled data is unavailable.