Joker: Joint Optimization Framework for Lightweight Kernel Machines

Kernel methods are powerful tools for nonlinear learning with well-established theory. The scalability issue has been their long-standing challenge. Despite the existing success, there are two limitations in large-scale kernel methods: (i) The memory overhead is too high for users to afford; (ii) existing efforts mainly focus on kernel ridge regression (KRR), while other models lack study. In this paper, we propose Joker, a joint optimization framework for diverse kernel models, including KRR, logistic regression, and support vector machines. We design a dual block coordinate descent method with trust region (DBCD-TR) and adopt kernel approximation with randomized features, leading to low memory costs and high efficiency in large-scale learning. Experiments show that Joker saves up to 90% memory but achieves comparable training time and performance (or even better) than the state-of-the-art methods.

This work studied the kernel method, a powerful tool for machine learning.However, this tool usually requires top-tier computers in the big data era.Therefore, it is unaffordable for most users.On the other hand, the application of kernel methods is limited to a small number of machine learning models.To address these problems, we proposed a new strategy named Joker to improve the process of kernel methods.In short, Joker divides a big task into several small ones and handles each small task iteratively.Then, we can process each task with little effort and greatly reduce the burden on the computer.In practice, Joker can handle a large dataset (with 5 million images) in about 2 hours using a consumer GPU.To summarize, Joker lowers the requirement of computer hardware, making kernel methods easy to use for the public.