Shallow neural networks as Wasserstein gradient flows

Motivation

Artificial neural networks (ANNs) consist of layers of artificial "neurons" which take in information from the previous layer and output information to neurons in the next layer. Gradient descent is a common method for updating the weights of each neuron based on training data. While in practice every layer of a neural network has only finitely many neurons, it is beneficial to consider a neural network layer with infinitely many neurons, for the sake of developing a theory that explains how ANNs work. In particular, from this viewpoint the process of updating the neuron weights for a shallow neural network can be described by a Wasserstein gradient flow.

Single Layer Neural Networks

{See also Mathematics of Artificial Neural Networks}

Discrete Formulation

Let us introduce the mathematical framework and notation for a neural network with a single hidden layer. Let $D\subset \mathbb {R} ^{d}$ be open . The set $D$ represents the space of inputs into the network. There is some unknown function $f:D\rightarrow \mathbb {R}$ which we would like to approximate. Let $N\in \mathbb {N}$ be the number of neurons in the hidden layer. Define

$F_{N}:D\times \Omega ^{N}\rightarrow \mathbb {R} ^{k}$

be given by

$F_{N}(x,\omega _{1},\dots ,\omega _{N},\theta _{1},\dots ,\theta _{N})={\frac {1}{N}}\sum _{i=1}^{N}\omega _{i}h(\theta _{i},x)$

where $h$ is a fixed activation function and $\Omega ^{N}$ is a space of possible parameters $(\omega ,\theta )=(\omega _{1},\dots ,\omega _{N},\theta _{1},\dots ,\theta _{N})$. The goal is to use training data to repeatedly update the weights $\omega _{i}$ and $\theta _{i}$ based on how close $f_{N,\omega ,\theta }:=F_{N}(\cdot ,\omega _{1},\dots ,\omega _{N},\theta _{1},\dots ,\theta _{N})$ is to the function $f$. More concretely, we want to find $\omega ,\theta$ that minimizes the loss function:

$l(f,f_{N,\omega ,\theta }):={\frac {1}{2}}\int _{D}|f(x)-f_{N,\omega ,\theta }(x)|^{2}dx$

A standard way to choose and update the weights is to start with a random choice of weights ${\bar {\omega }},{\bar {\theta }}$ and perform gradient descent on these parameters. Unfortunately, this problem is non-convex, so the minimizer may not be achieved. It turns out in practice, neural networks are surprisingly good at finding the minimizer. A nicer minimization problem that may provide insight into how neural networks work is a neural network model with infinitely many neurons.

Continuous Formulation

For the continuous formulation (i.e. when $N=\infty$), we rephrase the above mathematical framework. In this case, it no longer makes sense to look for weights $\omega ,\theta$ that minimize the loss function. We instead look for a probability measure $\mu \in {\mathcal {P}}(\Omega )$ such that

$f_{\mu }(x):=\int _{\Omega }\Phi (\xi ,x)d\mu (\xi )$

minimizes the loss function:

$F(\mu ):={\frac {1}{2}}\int _{D}(f-f_{\mu })^{2}dx$.

Here $\Phi (\xi ,x)$ is an activation function with parameter $\xi =(\omega ,\theta )\in \Omega$.

Note that by restricting choices of $\mu$ to probability measures of the form $\mu _{N}={\frac {1}{N}}\sum _{i=1}^{N}\delta _{\xi _{i}}$, the above minimization problem generalizes to case with finitely many neurons as well.

To avoid overfitting the network to the training data, a potential term is added the loss function. For the remainder of this article, we define the loss function $F$ to be:

$F(\mu ):={\frac {1}{2}}\int _{D}(f-(\int _{\Omega }\Phi (\xi ,x)d\mu (\xi )))^{2}dx+\int _{\Omega }V(\xi )d\mu (\xi )$

for a convex potential function $V:\Omega \rightarrow \mathbb {R}$. Often we choose $V(\xi )={\frac {\lambda }{2}}|\xi |^{2}$. In fact, $F$ is convex (along linear interpolations), in contrast to the minimization function in the finite neuron case.

Gradient Flow

When $X\subseteq \mathbb {R} ^{n}$, the gradient flow of a differentiable function $f:X\rightarrow \mathbb {R}$ starting at a point $x_{0}$ is a curve $x(t):[0,T)\rightarrow X$ satisfying the differential equation

${\frac {d}{dt}}x(t)=-\nabla f(x(t)),x(0)=x_{0}$.

where $\nabla f$ is the gradient of f.

Crucially, the gradient flow heads in the direction that decreases the value of $f$ the fastest. We would like to use this nice property of gradient flow in our setting with the functional $F$. However, it is not immediately straightforward how to do this, since $F$ is defined on the space of probability measures, rather than on $\mathbb {R} ^{n}$, so the usual gradient is not defined. Before we generalize the notion of gradient flow, recall that $\nabla f$ is the unique function such that

$\langle \nabla f,v\rangle ={\rm {{d}f(v)}}$.

$\nabla f(x(t))=\nabla f\cdot {\dot {x}}(t)$.

Recall that for a vector field $V,{\rm {{div}V=-\nabla \cdot V}}$.

Motivated by this and using the notion of a subdifferential we can define the gradient flow in a Hilbert space ${\mathcal {H}}$ of a convex and lower semi-continuous functional $\Phi$. An absolutely continuous curve $x:[0,\infty )\rightarrow {\mathcal {H}}$ is a gradient flow for $\Phi$ starting at $x_{0}\in {\mathcal {H}}$ if

$x(0)=x_{0},$ and ${\dot {x}}(t)\in -\partial \Phi (x(t))$ for almost every $t>0$

where $\partial \Phi$ is the subdifferential of $\Phi$.

In order to define the gradient in ${\mathcal {P}}(\Omega )$, we also need to utilize the Riemannian structure of the space of probability measures.

Wasserstein Distance

Let us first define the (pth) Wasserstein distance between two probability measures. This can be defined for probability measures in any separable metric space. Let $\mu ,\nu \in {\mathcal {P}}(\Omega )$ and let $\Gamma (\mu ,\nu )$ denote the space of transport plans from $\mu$ to $\nu$. Define the pth Wasserstein distance from $\mu$ to $\nu$ to be:

$W_{p}(\mu ,\nu ):=\min\{\int _{\Omega \times \Omega }d(\xi _{1},\xi _{2})^{p}d\gamma (\xi _{1},\xi _{2}):\gamma \in \Gamma (\mu ,\nu )\}$

where $d(\xi _{1},\xi _{2})$ is the distance between $\xi _{1}$ and $\xi _{2}$ in the metric space $\Omega \times \Omega$. In this context, $\Omega \times \Omega$ is just a subset of $\mathbb {R} ^{d}$ for some $d$ with the euclidean metric. Often $p=2$.

First Variation / Functional Derivative

The first variation ( or functional derivative) of a function from ${\mathcal {P}}(\Omega )$ to $\mathbb {R}$ measures

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \frac{\delta F}{\delta \mu}(\mu_*) = \int_D \Phi( \cdot, x) \big{[}\int_\Omega \Phi(\overline{\xi},x)d \mu_*(\overline{xi}) - f(x) \big{]}dx + V }

Wasserstein Gradient Flow

$\partial _{t}\mu _{t}={\rm {{div}(\mu _{t}\nabla {\frac {\delta F}{\delta \mu }}(\mu _{t}))}}$