Depth with nonlinearity creates no bad local minima in ResNets.
Kenji Kawaguchi,Yoshua Bengio +1 more
TLDR
In this article, the authors prove that depth with nonlinearity creates no bad local minima in a type of arbitrarily deep ResNets with arbitrary nonlinear activation functions, in the sense that the values of all local minimima are no worse than the global minimum value of corresponding classical machine-learning models, and are guaranteed to further improve via residual representations.About:
This article is published in Neural Networks.The article was published on 2019-10-01 and is currently open access. It has received 52 citations till now. The article focuses on the topics: Maxima and minima.read more
Citations
More filters
Journal ArticleDOI
Deep Network Approximation Characterized by Number of Neurons
TL;DR: The approximation power of deep feed-forward neural networks is quantitatively characterizes in terms of the number of neurons, i.e., the product of the network width and depth, to provide a general guide for selecting the width and the depth of ReLU FNNs to approximate continuous functions especially in parallel computing.
Journal ArticleDOI
Why ResNet Works? Residuals Generalize
TL;DR: In this paper, the authors studied the influence of residual connections on the hypothesis complexity of the neural network in terms of the covering number of its hypothesis space and obtained a margin-based multiclass generalization bound for ResNet.
Journal ArticleDOI
Nonlinear approximation via compositions.
TL;DR: It is shown that dictionaries consisting of wide FNNs with a few hidden layers are more attractive in terms of computational efficiency than dictionaries with narrow and very deep Fnns for approximating Hölder continuous functions if the number of computer cores is larger than N in parallel computing.
Journal ArticleDOI
Review: Deep Learning in Electron Microscopy
TL;DR: In this paper, a review of deep learning in electron microscopy is presented, with a focus on hardware and software needed to get started with deep learning and interface with electron microscopes.
Journal ArticleDOI
A review on computer vision systems in monitoring of poultry: A welfare perspective
Cedric Okinda,Innocent Nyalala,Tchalla Korohou,Celestine Okinda,Jintao Wang,Tracy Achieng,Patrick Wamalwa,Tai Mang,Mingxia Shen +8 more
TL;DR: This review summarizes the current advances in poultry monitoring techniques based on computer vision systems, i.e., conventional machine learning- based and deep learning-based systems.
References
More filters
Proceedings ArticleDOI
Deep Residual Learning for Image Recognition
TL;DR: In this article, the authors proposed a residual learning framework to ease the training of networks that are substantially deeper than those used previously, which won the 1st place on the ILSVRC 2015 classification task.
Book ChapterDOI
Identity Mappings in Deep Residual Networks
TL;DR: In this paper, the forward and backward signals can be directly propagated from one block to any other block, when using identity mappings as the skip connections and after-addition activation.
Proceedings ArticleDOI
Aggregated Residual Transformations for Deep Neural Networks
TL;DR: ResNeXt as discussed by the authors is a simple, highly modularized network architecture for image classification, which is constructed by repeating a building block that aggregates a set of transformations with the same topology.
Proceedings ArticleDOI
Accurate Image Super-Resolution Using Very Deep Convolutional Networks
TL;DR: In this article, a very deep convolutional network inspired by VGG-net was used for image superresolution, which achieved state-of-the-art performance in accuracy.