In this talk we propose the DeepXML framework for deep extreme multi-label learning and apply it to short-text document classification. We demonstrate that DeepXML can: (a) be used to analyze seemingly disparate deep extreme classifiers; (b) can lead to improvements in leading algorithms such as XML-CNN & MACH when they are recast in the proposed framework; and (c) can lead to a novel algorithm called Astec which can be up to 12% more accurate and up to 40x faster to train than the state-of-the-art for short text document classification. Finally, we show that when flighted on Bing, Astec can be used for personalized search, ads and recommendation for billions of users. Astec can handle billions of events per day, can process more than a hundred thousand events per second and leads to a significant improvement in key metrics as compared to state-of-the-art methods in production.

In this talk, I will present several simple ideas that were proposed a long time ago to deal with extremely large output spaces in the language modeling. These include various types of hierarchical softmax, and other approaches that decompose the labels into smaller parts such as sub-word language modeling.

We consider multi-label classification where the goal is to annotate each data point with the most relevant subset of labels from an extremely large label set. Efficient annotation can be achieved with balanced tree predictors, i.e. trees with logarithmic-depth in the label complexity, whose leaves correspond to labels. Designing prediction mechanism with such trees for real data applications is non-trivial as it needs to accommodate sending examples to multiple leaves while at the same time sustain high prediction accuracy. In this paper we develop the LdSM algorithm for the construction and training of multi-label decision trees, where in every node of the tree we optimize a novel objective function that favors balanced splits, maintains high class purity of children nodes, and allows sending examples to multiple directions but with a penalty that prevents tree over-growth. Each node of the tree is trained once the previous node is completed leading to a streaming approach for training. We analyze the proposed objective theoretically and show that minimizing it leads to pure and balanced data splits. Furthermore, we show a boosting theorem that captures its connection to the multi-label classification error. Experimental results on benchmark data sets demonstrate that our approach achieves high prediction accuracy and low prediction time and position LdSM as a competitive tool among existing state-of-the-art approaches.

This talk is about a new learned dynamic memory controller for organizing prior experiences in a way that is empirically useful for a number of downstream tasks. The controller supports logarithmic time operations and can thus be integrated into existing statistical learning algorithms as an augmented memory unit without substantially increasing training and inference computation. It also supports optional reward reinforcement, which brings a steady improvement empirically. The controller operates as a reduction to online classification, allowing it to benefit from advances in representation or architecture. This is joint work with Wen Sun, Hal Daume, John Langford, and Paul Mineiro (published at ICML-2019).

Use the below Link and Password to joining the Zoom poster sessions. Please do not share the below links/password on any chat or public forum or with any person unregistered for ICML.

Password: xc2020p

Zoom Links:

Poster 1 - Unbiased Estimates of Decomposable Losses for Extreme Classification With Missing Labels

[ protected link dropped ]

Poster 2 - Online probabilistic label trees

[ protected link dropped ]

Poster 3 - Visualizing Classification Structure in Large-Scale Classifiers

[ protected link dropped ]

Poster 4 - Generalizing across (in)visible spectrum

[ protected link dropped ]

Poster 5 - Extreme Regression for Ranking & Recommendation

[ protected link dropped ]

Poster 6 - Heteroskedastic and Imbalanced Deep Learning with Adaptive Regularization

[ protected link dropped ]

Many challenging problems in modern applications amount to finding relevant results from an enormous output space of potential candidates, for example, finding the best matching product from a large catalog or suggesting related search phrases on a search engine. The size of the output space for these problems can be in the millions to billions. Moreover, observational or training data is often limited for many of the so-called “long-tail” of items in the output space. Given the inherent paucity of training data for most of the items in the output space, developing machine learned models that perform well for spaces of this size is challenging. Fortunately, items in the output space are often correlated thereby presenting an opportunity to alleviate the data sparsity issue. In this talk, I will first discuss the challenges in modern multi-output prediction, including missing values, features associated with outputs, absence of negative examples, and the need to scale up to enormous data sets. Bilinear methods, such as Inductive Matrix Completion~(IMC), enable us to handle missing values and output features in practice, while coming with theoretical guarantees. Nonlinear methods such as nonlinear IMC and DSSM (Deep Semantic Similarity Model) enable more powerful models that are used in practice in real-life applications. However, inference in these models scales linearly with the size of the output space. In order to scale up, I will present the Prediction for Enormous and Correlated Output Spaces (PECOS) framework, that performs prediction in three phases: (i) in the first phase, the output space is organized using a semantic indexing scheme, (ii) in the second phase, the indexing is used to narrow down the output space by orders of magnitude using a machine learned matching scheme, and (iii) in the third phase, the matched items are ranked by a final ranking scheme. The versatility and modularity of PECOS allows for easy plug-and-play of various choices for the indexing, matching, and ranking phases, and it is possible to ensemble various models, each arising from a particular choice for the three phases.

We create a computationally tractable algorithm for contextual bandit learning with one-dimensional continuous actions with unknown structure on the loss functions. In a nutshell, our algorithm, Continuous Action Tree with Smoothing (CATS), reduces continuous-action contextual bandit learning to cost-sensitive extreme multiclass classification, where each class corresponds to a discretized action. We show that CATS admits an online implementation that has low training and test time complexities per example, and enjoys statistical consistency guarantees under certain realizability assumptions. We also verify the efficiency and efficacy of CATS through large-scale experiments.

Use the below Link and Password to joining the Zoom poster sessions. Please do not share the below links/password on any chat or public forum or with any person unregistered for ICML.

Password: xc2020p

Zoom Links:

Poster 1 - Unbiased Estimates of Decomposable Losses for Extreme Classification With Missing Labels

[ protected link dropped ]

Poster 2 - Online probabilistic label trees

[ protected link dropped ]

Poster 3 - Visualizing Classification Structure in Large-Scale Classifiers

[ protected link dropped ]

Poster 4 - Generalizing across (in)visible spectrum

[ protected link dropped ]

Poster 5 - Extreme Regression for Ranking & Recommendation

[ protected link dropped ]

Poster 6 - Heteroskedastic and Imbalanced Deep Learning with Adaptive Regularization

[ protected link dropped ]

Developing algorithms that are able to generalize to a novel task given only a few labeled examples represents a fundamental challenge in closing the gap between machine- and human-level performance. The core of human cognition lies in the structured, reusable concepts that help us to rapidly adapt to new tasks and provide reasoning behind our decisions. However, existing meta-learning methods learn complex representations across prior labeled tasks without imposing any structure on the learned representations. In this talk I will discuss how meta-learning methods can improve generalization ability by learning to learn along human-interpretable concept dimensions. Instead of learning a joint unstructured metric space. We learn mappings of high-level concepts into semi-structured metric spaces, and effectively combine the outputs of independent concept learners. Experiments on diverse domains, including a benchmark image classification dataset and a novel single-cell dataset from a biological domain show significant gains over strong meta-learning baselines.