We propose a deep convolutional neural network architecture codenamed Inception that achieves the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition Challenge 2014 (ILSVRC14). The main hallmark of this architecture is the improved utilization of the computing resources inside the network. By a carefully crafted design, we increased the depth and width of the network while keeping the computational budget constant. To optimize quality, the architectural decisions were based on the Hebbian principle and the intuition of multi-scale processing. One particular incarnation used in our submission for ILSVRC14 is called GoogLeNet, a 22 layers deep network, the quality of which is assessed in the context of classification and detection.

/pdf/going-deeper-with-convolutions-1yobw2o2ds.pdf

Going deeper with convolutions

Many problems of recent interest in statistics and machine learning can be posed in the framework of convex optimization. Due to the explosion in size and complexity of modern datasets, it is increasingly important to be able to solve problems with a very large number of features or training examples. As a result, both the decentralized collection or storage of these datasets as well as accompanying distributed solution methods are either necessary or at least highly desirable. In this review, we argue that the alternating direction method of multipliers is well suited to distributed convex optimization, and in particular to large-scale problems arising in statistics, machine learning, and related areas. The method was developed in the 1970s, with roots in the 1950s, and is equivalent or closely related to many other algorithms, such as dual decomposition, the method of multipliers, Douglas–Rachford splitting, Spingarn's method of partial inverses, Dykstra's alternating projections, Bregman iterative algorithms for l1 problems, proximal methods, and others. After briefly surveying the theory and history of the algorithm, we discuss applications to a wide variety of statistical and machine learning problems of recent interest, including the lasso, sparse logistic regression, basis pursuit, covariance selection, support vector machines, and many others. We also discuss general distributed optimization, extensions to the nonconvex setting, and efficient implementation, including some details on distributed MPI and Hadoop MapReduce implementations.

/pdf/distributed-optimization-and-statistical-learning-via-the-8s9gtddw0d.pdf

Distributed Optimization and Statistical Learning Via the Alternating Direction Method of Multipliers

http://folk.uib.no/eha070/mat262/papers/Rudin%20Osher%20Fatemi.pdf

Nonlinear total variation based noise removal algorithms

This paper considers the model problem of reconstructing an object from incomplete frequency samples. Consider a discrete-time signal f/spl isin/C/sup N/ and a randomly chosen set of frequencies /spl Omega/. Is it possible to reconstruct f from the partial knowledge of its Fourier coefficients on the set /spl Omega/? A typical result of this paper is as follows. Suppose that f is a superposition of |T| spikes f(t)=/spl sigma//sub /spl tau//spl isin/T/f(/spl tau/)/spl delta/(t-/spl tau/) obeying |T|/spl les/C/sub M//spl middot/(log N)/sup -1/ /spl middot/ |/spl Omega/| for some constant C/sub M/>0. We do not know the locations of the spikes nor their amplitudes. Then with probability at least 1-O(N/sup -M/), f can be reconstructed exactly as the solution to the /spl lscr//sub 1/ minimization problem. In short, exact recovery may be obtained by solving a convex optimization problem. We give numerical values for C/sub M/ which depend on the desired probability of success. Our result may be interpreted as a novel kind of nonlinear sampling theorem. In effect, it says that any signal made out of |T| spikes may be recovered by convex programming from almost every set of frequencies of size O(|T|/spl middot/logN). Moreover, this is nearly optimal in the sense that any method succeeding with probability 1-O(N/sup -M/) would in general require a number of frequency samples at least proportional to |T|/spl middot/logN. The methodology extends to a variety of other situations and higher dimensions. For example, we show how one can reconstruct a piecewise constant (one- or two-dimensional) object from incomplete frequency samples - provided that the number of jumps (discontinuities) obeys the condition above - by minimizing other convex functionals such as the total variation of f.

/pdf/robust-uncertainty-principles-exact-signal-reconstruction-32lo4m4643.pdf

Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information

We propose a new model for active contours to detect objects in a given image, based on techniques of curve evolution, Mumford-Shah (1989) functional for segmentation and level sets. Our model can detect objects whose boundaries are not necessarily defined by the gradient. We minimize an energy which can be seen as a particular case of the minimal partition problem. In the level set formulation, the problem becomes a "mean-curvature flow"-like evolving the active contour, which will stop on the desired boundary. However, the stopping term does not depend on the gradient of the image, as in the classical active contour models, but is instead related to a particular segmentation of the image. We give a numerical algorithm using finite differences. Finally, we present various experimental results and in particular some examples for which the classical snakes methods based on the gradient are not applicable. Also, the initial curve can be anywhere in the image, and interior contours are automatically detected.

/pdf/active-contours-without-edges-2tt3qz9lq0.pdf

Active contours without edges

Shock filters for image enhancement are developed. The filters use new nonlinear time dependent partial differential equations and their discretizations. The evolution of the initial image $u_0 (x,y)$ as $t \to \infty $ into a steady state solution $u_\infty (x,y)$ through $u(x,y,t)$, $t > 0$, is the filtering process. The partial differential equations have solutions which satisfy a maximum principle. Moreover the total variation of the solution for any fixed $t > 0$ is the same as that of the initial data. The processed image is piecewise smooth, nonoscillatory, and the jumps occur across zeros of an elliptic operator (edge detector). The algorithm is relatively fast and easy to program.

/pdf/feature-oriented-image-enhancement-using-shock-filters-1k5679rokr.pdf

Feature-oriented image enhancement using shock filters

A new total variation based approach was developed by Rudin, Osher and Fatemi (see Physica D., vol.60, p.259, 1992) to overcome the basic limitations of all smooth regularization algorithms. The TV-based technique use the L/sup 1/ norm of the magnitude of a gradient, thus making discontinuous and nonsmooth solutions possible. In TV image restoration, the solution is obtained by solving a time-dependent, nonlinear PDE on a manifold that satisfies the degradation constraints. In practical applications, one assumes a space-varying blurring kernel and signal-dependent (e.g. multiplicative) noise. The evolution part of the TV-based PDE turned out to be related to the curve shortening equation, but scaled by an inverse. >

Total variation based image restoration with free local constraints

In [447, 449, 450], a constrained optimization type of numerical algorithm for restoring blurry, noisy images was developed and successfully tested. In this paper we present both theoretical and experimental justification for the method. Our main theoretical results involve constrained nonlinear partial differential equations. Our main experimental results involve blurry images which have been further corrupted with multiplicative noise. As in additive noise case of [447, 450] our numerical algorithm is simple to implement and is nonoscillatory (minimal ringing) and noninvasive (recovers sharp edges).

Multiplicative Denoising and Deblurring: Theory and Algorithms

Two new filters for image enhancement are developed, extending the early work of the authors. One filter uses a new nonlinear time dependent partial differential equation and its discretization, the second uses a discretization which constrains the backwards heat equation and keeps it variation bounded. The evolution of the initial image as t increases through U(x,y,t) is the filtering process. The processed image is piecewise smooth, nonoscillatory and apparently an accurate reconstruction. The algorithms are fast and easy to program.

Leonid I. Rudin

Papers

Nonlinear total variation based noise removal algorithms

Feature-oriented image enhancement using shock filters

Total variation based image restoration with free local constraints

Multiplicative Denoising and Deblurring: Theory and Algorithms

Shocks and other nonlinear filtering applied to image processing