Diversity-Aware Recursive Feature Multiple Kernel Learning

\textit{\underline{M}ultiple \underline{k}ernel \underline{l}earning}~(MKL) combines several base kernels in the spirit of ensemble learning, yet existing methods rarely model kernel diversity---a known cornerstone of ensembles---and most traditional kernels weight all features uniformly, ignoring feature-level discriminability. We address both gaps with DARFMMKL: a data-driven kernel family (Recursive Feature Machine kernels) that learns feature importance directly from data, paired with a kernel selection method that jointly optimizes diversity and quality. The resulting NP-hard binary quadratic program is reformulated via Glover linearization and continuous relaxation into a linear program, and accelerated by Nystr\"om sketching, yielding a selector whose cost is decoupled from the sample size. We provide a covering-number generalization bound that explicitly relates kernel diversity to estimation error. Experiments on 12 benchmark datasets show that DARFMMKL consistently outperforms 9 state-of-the-art MKL methods.

Many machine-learning methods compare examples using a "similarity measure," and a powerful idea is to combine several such measures instead of relying on one, much like consulting a panel of advisors rather than a single expert. Two issues limit existing approaches: the similarity measures usually treat every feature of the data as equally important, even when many features are irrelevant or noisy; and the methods often combine measures that are redundant and all capture the same thing. A good panel should be both knowledgeable and diverse. We propose DARFMMKL, which addresses both issues. First, it builds similarity measures that learn from the data which features actually matter, so informative features count for more. Second, it selects a set of measures that are at once high-quality and diverse, complementing rather than echoing one another, by casting the choice as an optimization problem that we solve efficiently and scale to large datasets. We also prove mathematically that encouraging diversity improves the method's ability to generalize to unseen data. On twelve standard datasets, DARFMMKL consistently outperforms nine state-of-the-art competitors.