Timezone: »

Stochastic Flows and Geometric Optimization on the Orthogonal Group
Krzysztof Choromanski · David Cheikhi · Jared Quincy Davis · Valerii Likhosherstov · Achille Nazaret · Achraf Bahamou · Xingyou Song · Mrugank Akarte · Jack Parker-Holder · Jacob Bergquist · Yuan Gao · Aldo Pacchiano · Tamas Sarlos · Adrian Weller · Vikas Sindhwani

Thu Jul 16 06:00 AM -- 06:45 AM & Thu Jul 16 05:00 PM -- 05:45 PM (PDT) @ Virtual #None

We present a new class of stochastic, geometrically-driven optimization algorithms on the orthogonal group O(d) and naturally reductive homogeneous manifolds obtained from the action of the rotation group SO(d). We theoretically and experimentally demonstrate that our methods can be applied in various fields of machine learning including deep, convolutional and recurrent neural networks, reinforcement learning, normalizing flows and metric learning. We show an intriguing connection between efficient stochastic optimization on the orthogonal group and graph theory (e.g. matching problem, partition functions over graphs, graph-coloring). We leverage the theory of Lie groups and provide theoretical results for the designed class of algorithms. We demonstrate broad applicability of our methods by showing strong performance on the seemingly unrelated tasks of learning world models to obtain stable policies for the most difficult Humanoid agent from OpenAI Gym and improving convolutional neural networks.

Author Information

Krzysztof Choromanski (Google Brain Robotics)
David Cheikhi (Columbia University)
Jared Quincy Davis (DeepMind & Stanford University)
Valerii Likhosherstov (University of Cambridge)
Achille Nazaret (Columbia University)
Achraf Bahamou (Columbia University)
Xingyou Song (Google Brain)
Mrugank Akarte (Columbia University)
Jack Parker-Holder (University of Oxford)
Jacob Bergquist (Columbia University)
Yuan Gao (Columbia University)

Yuan studied Applied Mathematics and Statistics at National University of Singapore. He continued to pursue a PhD in Operations Research at Columbia University. He works on numerical optimization in machine learning and game theory.

Aldo Pacchiano (UC Berkeley)
Tamas Sarlos (Google)
Adrian Weller (University of Cambridge, Alan Turing Institute)

Adrian Weller is a Senior Research Fellow in the Machine Learning Group at the University of Cambridge, a Faculty Fellow at the Alan Turing Institute for data science and an Executive Fellow at the Leverhulme Centre for the Future of Intelligence (CFI). He is very interested in all aspects of artificial intelligence, its commercial applications and how it may be used to benefit society. At the CFI, he leads their project on Trust and Transparency. Previously, Adrian held senior roles in finance. He received a PhD in computer science from Columbia University, and an undergraduate degree in mathematics from Trinity College, Cambridge.

Vikas Sindhwani (Google)

More from the Same Authors