A Lie Group Approach to Riemannian Batch Normalization | HackerNoon

Table of Links

Abstract and 1 Introduction

2 Preliminaries

3. Revisiting Normalization

3.1 Revisiting Euclidean Normalization

3.2 Revisiting Existing RBN

4 Riemannian Normalization on Lie Groups

5 LieBN on the Lie Groups of SPD Manifolds and 5.1 Deformed Lie Groups of SPD Manifolds

5.2 LieBN on SPD Manifolds

6 Experiments

6.1 Experimental Results

7 Conclusions, Acknowledgments, and References

APPENDIX CONTENTS

A Notations

B Basic layes in SPDnet and TSMNet

C Statistical Results of Scaling in the LieBN

D LieBN as a Natural Generalization of Euclidean BN

E Domain-specific Momentum LieBN for EEG Classification

F Backpropagation of Matrix Functions

G Additional Details and Experiments of LieBN on SPD manifolds

H Preliminary Experiments on Rotation Matrices

I Proofs of the Lemmas and Theories in the Main Paper

ABSTRACT

Manifold-valued measurements exist in numerous applications within computer vision and machine learning. Recent studies have extended Deep Neural Networks (DNNs) to manifolds, and concomitantly, normalization techniques have also been adapted to several manifolds, referred to as Riemannian normalization. Nonetheless, most of the existing Riemannian normalization methods have been derived in an ad hoc manner and only apply to specific manifolds. This paper establishes a unified framework for Riemannian Batch Normalization (RBN) techniques on Lie groups.

Our framework offers the theoretical guarantee of controlling both the Riemannian mean and variance. Empirically, we focus on Symmetric Positive Definite (SPD) manifolds, which possess three distinct types of Lie group structures. Using the deformation concept, we generalize the existing Lie groups on SPD manifolds into three families of parameterized Lie groups. Specific normalization layers induced by these Lie groups are then proposed for SPD neural networks. We demonstrate the effectiveness of our approach through three sets of experiments: radar recognition, human action recognition, and electroencephalography (EEG) classification. The code is available at https://github.com/GitZH-Chen/LieBN.git.

1 INTRODUCTION

Over the past decade or so, Deep Neural Networks (DNNs) have achieved remarkable progress across various scientific fields (Hochreiter & Schmidhuber, 1997; Krizhevsky et al., 2012; He et al., 2016; Vaswani et al., 2017). Conventionally, DNNs have been developed with the underlying assumption of the Euclidean geometry inherent to input data. Nonetheless, there exists a plethora of applications wherein the latent spaces are defined by non-Euclidean structures such as manifolds (Bronstein et al., 2017).

To address this issue, researchers have attempted to extend various types of DNNs to manifolds based on the theories of Riemannian geometry (Huang & Van Gool, 2017; Huang et al., 2017; 2018; Ganea et al., 2018; Chakraborty et al., 2018; Brooks et al., 2019a;e;c;d; Brooks, 2020; Brooks et al., 2020; Chen et al., 2020; Chakraborty et al., 2020; Chakraborty, 2020; Chen et al., 2021; Wang et al., 2022b;a; Nguyen, 2022a;b; Nguyen & Yang, 2023; Chen et al., 2023c;e;b; Wang et al., 2024).

Motivated by the great success of normalization techniques within DNNs (Ioffe & Szegedy, 2015; Ba et al., 2016; Ulyanov et al., 2016; Wu & He, 2018; Chen et al., 2023a), researchers have sought to devise normalization layers tailored for manifold-valued data. Brooks et al. (2019b) introduced Riemannian Batch Normalization (RBN) specifically designed for SPD manifolds, with the ability to regulate the Riemannian mean. This approach was further refined in Kobler et al. (2022b) to extend the control over the Riemannian variance. However, the above methods are constrained within the affine-invariant metric (AIM) on SPD manifolds, limiting their applicability and generality.

On the other hand, Chakraborty (2020) proposed two distinct Riemannian normalization frameworks, one tailored for Riemannian homogeneous spaces and the other catering to matrix Lie groups. Nonetheless, the normalization designed for Riemannian homogeneous spaces cannot regulate mean nor variance, while the normalization approach for matrix Lie groups is confined to a specific type of distance (Chakraborty, 2020, Sec. 3.2). A principled Riemannian normalization framework capable of controlling both Riemannian mean and variance remains unexplored.

Given that Batch Normalization (BN) (Ioffe & Szegedy, 2015) serves as the foundational prototype for various types of normalization, our paper only concentrates on RBN currently and can be readily extended to other normalization techniques. As several manifold-valued measurements form Lie groups, including SPD manifolds, Special Orthogonal (SO) groups, and Special Euclidean (SE) groups, we further direct our attention towards Lie groups. We propose a general framework for RBN over Lie groups, referred to as LieBN, and validate our approach in normalizing both the Riemannian mean and variance.

On the empirical side, we focus on SPD manifolds, where three distinct types of Lie groups have been recognized in the literature. We generalize these existing Lie groups into parameterized forms through the deformation concept. Then, we showcase our LieBN framework on SPD manifolds under these Lie groups and propose specific normalization layers. Extensive experiments conducted on widely-used SPD benchmarks demonstrate the effectiveness of our framework. We highlight that our work is entirely theoretically different from Brooks et al. (2019b); Kobler et al. (2022a); Lou et al. (2020), and more general than Chakraborty (2020). The previous RBN methods are either designed for a specific manifold or metric (Brooks et al., 2019b; Kobler et al., 2022a; Chakraborty, 2020), or fail to control mean and variance (Lou et al., 2020), while our LieBN guarantees the normalization of mean and variance on general Lie groups.

In summary, our main contributions are as follows: (a) a general Lie group batch normalization framework with controllable first- and second-order statistical moments, and (b) the specific construction of our LieBN layers on SPD manifolds based on three deformed Lie groups and their application on SPD neural networks. Due to page limites, all the proofs are placed in App. I.