Open AccessPosted Content
Sharp bounds for the number of regions of maxout networks and vertices of Minkowski sums
TLDR
For networks with a single layer of maxout units, the linear regions correspond to the upper vertices of a Minkowski sum of polytopes as discussed by the authors, and they also obtain asymptotically sharp upper bounds for networks with multiple layers.Abstract:
We present results on the number of linear regions of the functions that can be represented by artificial feedforward neural networks with maxout units. A rank-k maxout unit is a function computing the maximum of $k$ linear functions. For networks with a single layer of maxout units, the linear regions correspond to the upper vertices of a Minkowski sum of polytopes. We obtain face counting formulas in terms of the intersection posets of tropical hypersurfaces or the number of upper faces of partial Minkowski sums, along with explicit sharp upper bounds for the number of regions for any input dimension, any number of units, and any ranks, in the cases with and without biases. Based on these results we also obtain asymptotically sharp upper bounds for networks with multiple layers.read more
Citations
More filters
Posted Content
Towards Lower Bounds on the Depth of ReLU Neural Networks
TL;DR: In this article, the authors investigated whether the class of exactly representable functions strictly increases by adding more layers (with no restrictions on size), and they provided a mathematical counterbalance to the universal approximation theorems which suggest that a single hidden layer is sufficient for learning tasks.
Posted Content
On the Expected Complexity of Maxout Networks
Hanna Tseran,Guido Montúfar +1 more
TL;DR: In this paper, the authors show that the complexity of deep ReLU networks is often far from the theoretical maximum and investigate different parameter initialization procedures to increase the speed of convergence in training.
References
More filters
Proceedings ArticleDOI
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
TL;DR: In this paper, a Parametric Rectified Linear Unit (PReLU) was proposed to improve model fitting with nearly zero extra computational cost and little overfitting risk, which achieved a 4.94% top-5 test error on ImageNet 2012 classification dataset.
Proceedings Article
Deep Sparse Rectifier Neural Networks
TL;DR: This paper shows that rectifying neurons are an even better model of biological neurons and yield equal or better performance than hyperbolic tangent networks in spite of the hard non-linearity and non-dierentiabil ity.
Book
Lectures on Polytopes
TL;DR: In this article, the authors present a rich collection of material on the modern theory of convex polytopes, with an emphasis on the methods that yield the results (Fourier-Motzkin elimination, Schlegel diagrams, shellability, Gale transforms, and oriented matroids).
MonographDOI
Tame Topology and O-minimal Structures
TL;DR: In this article, the authors give a self-contained treatment of the theory of o-minimal structures from a geometric and topological viewpoint, assuming only rudimentary algebra and analysis, and cover the monotonicity theorem, cell decomposition, and the Euler characteristic in the ominimal setting and show how these notions are easier to handle than in ordinary topology.