Table of Links
Abstract and 1. Introduction
-
Some recent trends in theoretical ML
2.1 Deep Learning via continuous-time controlled dynamical system
2.2 Probabilistic modeling and inference in DL
2.3 Deep Learning in non-Euclidean spaces
2.4 Physics Informed ML
-
Kuramoto model
3.1 Kuramoto models from the geometric point of view
3.2 Hyperbolic geometry of Kuramoto ensembles
3.3 Kuramoto models with several globally coupled sub-ensembles
-
Kuramoto models on higher-dimensional manifolds
4.1 Non-Abelian Kuramoto models on Lie groups
4.2 Kuramoto models on spheres
4.3 Kuramoto models on spheres with several globally coupled sub-ensembles
4.4 Kuramoto models as gradient flows
4.5 Consensus algorithms on other manifolds
-
Directional statistics and swarms on manifolds for probabilistic modeling and inference on Riemannian manifolds
5.1 Statistical models over circles and tori
5.2 Statistical models over spheres
5.3 Statistical models over hyperbolic spaces
5.4 Statistical models over orthogonal groups, Grassmannians, homogeneous spaces
-
Swarms on manifolds for DL
6.1 Training swarms on manifolds for supervised ML
6.2 Swarms on manifolds and directional statistics in RL
6.3 Swarms on manifolds and directional statistics for unsupervised ML
6.4 Statistical models for the latent space
6.5 Kuramoto models for learning (coupled) actions of Lie groups
6.6 Grassmannian shallow and deep learning
6.7 Ensembles of coupled oscillators in ML: Beyond Kuramoto models
-
Examples
7.1 Wahba’s problem
7.2 Linked robot’s arm (planar rotations)
7.3 Linked robot’s arm (spatial rotations)
7.4 Embedding multilayer complex networks (Learning coupled actions of Lorentz groups)
-
Conclusion and References
ABSTRACT
We propose the idea of using Kuramoto models (including their higher-dimensional generalizations) for machine learning over non-Euclidean data sets. These models are systems of matrix ODE’s describing collective motions (swarming dynamics) of abstract particles (generalized oscillators) on spheres, homogeneous spaces and Lie groups. Such models have been extensively studied from the beginning of XXI century both in statistical physics and control theory. They provide a suitable framework for encoding maps between various manifolds and are capable of learning over spherical and hyperbolic geometries. In addition, they can learn coupled actions of transformation groups (such as special orthogonal, unitary and Lorentz groups). Furthermore, we overview families of probability distributions that provide appropriate statistical models for probabilistic modeling and inference in Geometric Deep Learning. We argue in favor of using statistical models which arise in different Kuramoto models in the continuum limit of particles. The most convenient families of probability distributions are those which are invariant with respect to actions of certain symmetry groups.
1 Introduction
Machine Learning (ML) is, to a great extent, a science of inferring models and patterns from data. From that point of view, its core objective consists in learning optimal (according to a certain criterion) mappings between spaces. For several decades these mappings have been dominantly encoded using artificial neural networks with different topologies [1]. The spaces have almost always been assumed Euclidean or equipped with some flat metric. The data have been represented by points in Euclidean spaces or in finite sets.
An enormous progress in ML and Data Science in XXI century led to the growing understanding that a great deal (possibly, majority) of data sets have inherent non-Euclidean geometries. This fact has been mostly neglected in ML until very recently. Only the last decade brought systematic research efforts focused on geometric-sensitive architectures of neural networks (NN’s).
In parallel, traditional ways of designing artificial NN’s are being reexamined and enriched by new ideas. Diversity of applications and conceptual complexity of ML problems motivated investigations of new architectures. Over the centuries mathematicians elaborated various ways of encoding maps between Euclidean spaces or Riemannian manifolds. The corresponding theories have been established before the advent of ML, and now provide a solid theoretical background for its future developments. Following an explosive expansion of ML applications and practices, there is a huge backlog of theoretical work to be done. Mathematical foundations of ML are being actively reconsidered and expanded. Certain fields of mathematics that have been almost invisible in ML until very recently are now actively exploited with a great potential for future applications. The examples include Riemannian Geometry, Game Theory and Lie Group Theory – to name just a few.
Systematic approaches in ML must be based on well established theories and well understood models. The choice of adequate models and appropriate data representations appears to be the key issue. An appropriate choice greatly reduces the dimension (number of parameters), increases the efficiency of algorithms and, equally important, improves their transparency.
The main goal of the present paper is to point out a broad class of models which constitute a powerful theoretical framework for encoding geometric data. These models describing collective motions of interacting particles have been studied in Science for almost half of a century from various points of view. In physics of complex systems they are known as Kuramoto models [2] (including generalizations to higher-dimensional manifolds [3, 4]) and Viscek models [5]. In systems theory they are said to be (anti-)consensus algorithms on manifolds [6, 7]. Finally, in Engineering they are sometimes referred to as swarms on manifolds [8, 9]. All these models fit into the unifying mathematical framework that we refer to as systems of geometric Riccati ODE’s, as will be explained in sections 3 and 4.
Our exposition will be focused on the following questions:
-
Which kinds of mappings can be encoded by collective motions of Kuramoto oscillators/swarms on manifolds?
-
Which symmetries/patterns can be learned using these dynamics?
-
Which statistical models are associated with these dynamics and how can they be used in statistical ML over manifolds?
-
Which problems can be efficiently solved using such models?
-
How these models can be trained?
Our proposal on using swarms/Kuramoto oscillators in ML is inspired by some recent developments in theoretical ML which will be mentioned in Section 2. Section 3 is devoted to classical Kuramoto models (i.e. models describing collective motions of the classical phase oscillators) and their potential applications to learning coupled actions of transformation groups, as well as data on circles, tori and hyperbolic multi-discs. Section 4 contains an overview of (generalized) Kuramoto models that describe collective motions on spheres, Lie groups and other manifolds. In Section 5 we present families of probability measures over Riemannian manifolds which provide appropriate statistical models for probabilistic ML algorithms over non-Euclidean data sets. Some of these families are generated by the corresponding swarming dynamics. Connections with directional statistics will be particularly emphasized. In Section 6 we clarify how swarms can be used for supervised, unsupervised and reinforcement learning over Riemannian manifolds. In Section 7 we analyze some illustrative geometric ML problems in low dimensions, thus supporting our main points. Finally, Section 8 contains some concluding remarks and an outlook for the future research efforts.
:::info
This paper is available on arxiv under CC by 4.0 Deed (Attribution 4.0 International) license.
:::
:::info
Author:
(1) Vladimir Jacimovic, Faculty of Natural Sciences and Mathematics, University of Montenegro Cetinjski put bb., 81000 Podgorica Montenegro ([email protected]).
:::
