2 layer neural networks as Wasserstein gradient flows: Difference between revisions
Paigehillen (talk | contribs) No edit summary |
Paigehillen (talk | contribs) No edit summary |
||
| Line 1: | Line 1: | ||
<ref name="Figalli" /> | <ref name="Figalli" /> | ||
[https://en.wikipedia.org/wiki/Artificial_neural_network#:~:text=Artificial%20neural%20networks%20(ANNs)%2C,neurons%20in%20a%20biological%20brain. Artificial neural networks] (ANNs) consist of layers of artificial "neurons" which take in information from the previous layer and output information to the next layer. Gradient descent is a common method for updating the weights of each neuron based on training data. While in practice every layer of a neural network has only finitely many neurons, it is beneficial to consider a neural network layer with infinitely many neurons, for the sake of developing a theory that explains how ANNs work. In particular, from this viewpoint the process of updating the neuron weights for a shallow neural network can be described by a Wasserstein gradient flow. | [https://en.wikipedia.org/wiki/Artificial_neural_network#:~:text=Artificial%20neural%20networks%20(ANNs)%2C,neurons%20in%20a%20biological%20brain. Artificial neural networks] (ANNs) consist of layers of artificial "neurons" which take in information from the previous layer and output information to neurons in the next layer. Gradient descent is a common method for updating the weights of each neuron based on training data. While in practice every layer of a neural network has only finitely many neurons, it is beneficial to consider a neural network layer with infinitely many neurons, for the sake of developing a theory that explains how ANNs work. In particular, from this viewpoint the process of updating the neuron weights for a shallow neural network can be described by a Wasserstein gradient flow. | ||
==Motivation== | ==Motivation== | ||
| Line 7: | Line 7: | ||
==Shallow Neural Networks== | ==Shallow Neural Networks== | ||
Let us introduce the mathematical framework and notation for a neural network with a single hidden layer. Let <math> D \subset \mathbb{R}^d <\math> be open . The set D represents the space of inputs into the NN. | |||
Revision as of 03:29, 10 February 2022
Artificial neural networks (ANNs) consist of layers of artificial "neurons" which take in information from the previous layer and output information to neurons in the next layer. Gradient descent is a common method for updating the weights of each neuron based on training data. While in practice every layer of a neural network has only finitely many neurons, it is beneficial to consider a neural network layer with infinitely many neurons, for the sake of developing a theory that explains how ANNs work. In particular, from this viewpoint the process of updating the neuron weights for a shallow neural network can be described by a Wasserstein gradient flow.
Motivation
Shallow Neural Networks
Let us introduce the mathematical framework and notation for a neural network with a single hidden layer. Let <math> D \subset \mathbb{R}^d <\math> be open . The set D represents the space of inputs into the NN.