Showing papers on "Piecewise linear function published in 2017"

Deep Learning with Low Precision by Half-Wave Gaussian Quantization

Abstract: The problem of quantizing the activations of a deep neural network is considered. An examination of the popular binary quantization approach shows that this consists of approximating a classical non-linearity, the hyperbolic tangent, by two functions: a piecewise constant sign function, which is used in feedforward network computations, and a piecewise linear hard tanh function, used in the backpropagation step during network learning. The problem of approximating the widely used ReLU non-linearity is then considered. An half-wave Gaussian quantizer (HWGQ) is proposed for forward approximation and shown to have efficient implementation, by exploiting the statistics of of network activations and batch normalization operations. To overcome the problem of gradient mismatch, due to the use of different forward and backward approximations, several piece-wise backward approximators are then investigated. The implementation of the resulting quantized network, denoted as HWGQ-Net, is shown to achieve much closer performance to full precision networks, such as AlexNet, ResNet, GoogLeNet and VGG-Net, than previously available low-precision networks, with 1-bit binary weights and 2-bit quantized activations.

Semiparametric methods for estimation of a nonlinear exposure-outcome relationship using instrumental variables with application to Mendelian randomization.

Abstract: Mendelian randomization, the use of genetic variants as instrumental variables (IV), can test for and estimate the causal effect of an exposure on an outcome. Most IV methods assume that the function relating the exposure to the expected value of the outcome (the exposure-outcome relationship) is linear. However, in practice, this assumption may not hold. Indeed, often the primary question of interest is to assess the shape of this relationship. We present two novel IV methods for investigating the shape of the exposure-outcome relationship: a fractional polynomial method and a piecewise linear method. We divide the population into strata using the exposure distribution, and estimate a causal effect, referred to as a localized average causal effect (LACE), in each stratum of population. The fractional polynomial method performs metaregression on these LACE estimates. The piecewise linear method estimates a continuous piecewise linear function, the gradient of which is the LACE estimate in each stratum. Both methods were demonstrated in a simulation study to estimate the true exposure-outcome relationship well, particularly when the relationship was a fractional polynomial (for the fractional polynomial method) or was piecewise linear (for the piecewise linear method). The methods were used to investigate the shape of relationship of body mass index with systolic blood pressure and diastolic blood pressure.

Macroscopic urban dynamics: Analytical and numerical comparisons of existing models

Abstract: In this paper we compare a single reservoir model and a trip-based model under piecewise linear MFD and a piecewise constant demand. These assumptions allow to establish the exact solution of the accumulation-based model, and continuous approximations of the trip-based model at any order using Taylor series.

Practical Gauss-Newton Optimisation for Deep Learning

Abstract: We present an efficient block-diagonal approximation to the Gauss-Newton matrix for feedforward neural networks. Our resulting algorithm is competitive against state-of-the-art first-order optimisation methods, with sometimes significant improvement in optimisation performance. Unlike first-order methods, for which hyperparameter tuning of the optimisation parameters is often a laborious process, our approach can provide good performance even when used with default settings. A side result of our work is that for piecewise linear transfer functions, the network objective function can have no differentiable local maxima, which may partially explain why such transfer functions facilitate effective optimisation.

Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks

Abstract: We prove new upper and lower bounds on the VC-dimension of deep neural networks with the ReLU activation function. These bounds are tight for almost the entire range of parameters. Letting $W$ be the number of weights and $L$ be the number of layers, we prove that the VC-dimension is $O(W L \log(W))$, and provide examples with VC-dimension $\Omega( W L \log(W/L) )$. This improves both the previously known upper bounds and lower bounds. In terms of the number $U$ of non-linear units, we prove a tight bound $\Theta(W U)$ on the VC-dimension. All of these bounds generalize to arbitrary piecewise linear activation functions, and also hold for the pseudodimensions of these function classes. Combined with previous results, this gives an intriguing range of dependencies of the VC-dimension on depth for networks with different non-linearities: there is no dependence for piecewise-constant, linear dependence for piecewise-linear, and no more than quadratic dependence for general piecewise-polynomial.

Feature Selection with Annealing for Computer Vision and Big Data Learning

Abstract: Many computer vision and medical imaging problems are faced with learning from large-scale datasets, with millions of observations and features. In this paper we propose a novel efficient learning scheme that tightens a sparsity constraint by gradually removing variables based on a criterion and a schedule. The attractive fact that the problem size keeps dropping throughout the iterations makes it particularly suitable for big data learning. Our approach applies generically to the optimization of any differentiable loss function, and finds applications in regression, classification and ranking. The resultant algorithms build variable screening into estimation and are extremely simple to implement. We provide theoretical guarantees of convergence and selection consistency. In addition, one dimensional piecewise linear response functions are used to account for nonlinearity and a second order prior is imposed on these functions to avoid overfitting. Experiments on real and synthetic data show that the proposed method compares very well with other state of the art methods in regression, classification and ranking while being computationally very efficient and scalable.

Deep Lattice Networks and Partial Monotonic Functions

Abstract: We propose learning deep models that are monotonic with respect to a user-specified set of inputs by alternating layers of linear embeddings, ensembles of lattices, and calibrators (piecewise linear functions), with appropriate constraints for monotonicity, and jointly training the resulting network. We implement the layers and projections with new computational graph nodes in TensorFlow and use the Adam optimizer and batched stochastic gradients. Experiments on benchmark and real-world datasets show that six-layer monotonic deep lattice networks achieve state-of-the art performance for classification and regression with monotonicity guarantees.

A Convex Primal Formulation for Convex Hull Pricing

Abstract: In certain electricity markets, because of nonconvexities that arise from their operating characteristics, generators that follow the independent system operator's (ISO's) decisions may fail to recover their cost through sales of energy at locational marginal prices. The ISO makes discriminatory side payments to incentivize the compliance of generators. Convex hull pricing is a uniform pricing scheme that minimizes these side payments. The Lagrangian dual problem of the unit commitment problem has been solved in the dual space to determine convex hull prices. However, this approach is computationally expensive. We propose a polynomially solvable primal formulation for the Lagrangian dual problem. This formulation explicitly describes for each generating unit the convex hull of its feasible set and the convex envelope of its cost function. We cast our formulation as a second-order cone program when the cost functions are quadratic, and a linear program when the cost functions are piecewise linear. A 96-period 76-unit transmission-constrained example is solved in less than 15 s on a personal computer.

Anchored Regression Networks Applied to Age Estimation and Super Resolution

Abstract: We propose the Anchored Regression Network (ARN), a nonlinear regression network which can be seamlessly integrated into various networks or can be used stand-alone when the features have already been fixed. Our ARN is a smoothed relaxation of a piecewise linear regressor through the combination of multiple linear regressors over soft assignments to anchor points. When the anchor points are fixed the optimal ARN regressors can be obtained with a closed form global solution, otherwise ARN admits end-toend learning with standard gradient based methods. We demonstrate the power of the ARN by applying it to two very diverse and challenging tasks: age prediction from face images and image super-resolution. In both cases, ARNs yield strong results.

Deep Learning with Low Precision by Half-wave Gaussian Quantization

Abstract: The problem of quantizing the activations of a deep neural network is considered. An examination of the popular binary quantization approach shows that this consists of approximating a classical non-linearity, the hyperbolic tangent, by two functions: a piecewise constant sign function, which is used in feedforward network computations, and a piecewise linear hard tanh function, used in the backpropagation step during network learning. The problem of approximating the ReLU non-linearity, widely used in the recent deep learning literature, is then considered. An half-wave Gaussian quantizer (HWGQ) is proposed for forward approximation and shown to have efficient implementation, by exploiting the statistics of of network activations and batch normalization operations commonly used in the literature. To overcome the problem of gradient mismatch, due to the use of different forward and backward approximations, several piece-wise backward approximators are then investigated. The implementation of the resulting quantized network, denoted as HWGQ-Net, is shown to achieve much closer performance to full precision networks, such as AlexNet, ResNet, GoogLeNet and VGG-Net, than previously available low-precision networks, with 1-bit binary weights and 2-bit quantized activations.

Finite-time H∞ control of periodic piecewise linear systems

Abstract: In this paper, the finite-time stability, stabilisation, L2-gain and H∞ control problems for a class of continuous-time periodic piecewise linear systems are addressed. By employing a time-varying control scheme in which the time interval of each subsystem constitutes a number of basic time segments, the finite-time controllers can be developed with periodically time-varying control gains. Based on a piecewise time-varying Lyapunov-like function, a sufficient condition of finite-time stability and the relevant time-varying controller are proposed. Considering the finite-time boundedness of the closed-loop periodic system, the L2-gain criterion with continuous time-varying Lyapunov-like matrix function is studied. A finite-time H∞ controller is proposed based on the L2-gain analysis. Finally, numerical simulations are presented to illustrate the effectiveness of the proposed criteria.

A cut discontinuous Galerkin method for the Laplace-Beltrami operator

Abstract: We develop a discontinuous cut finite element method for the Laplace–Beltrami operator on a hypersurface embedded in R. The method is constructed by using a discontinuous piecewise linear finite el ...

A Unit Commitment Algorithm and a Compact MILP Model for Short-Term Hydro-Power Generation Scheduling

Abstract: This paper presents a unit commitment algorithm that defines each unit discharge given the water head, the total plant downstream flow, the variable discharge upper limit, the unit efficiency curves, and the restricted operating zones in order to maximize power efficiency. This algorithm is part of the preprocessing phase that is intended to approximate a hydro-power production function that represents individualized unit decisions. A compact mixed-integer linear programming formulation, with fewer integer variables, based on an equivalent unit model and a piecewise linear generation function, is proposed. The unit commitment is integrated without increasing the model size and complexity due to the preprocessing phase. Moreover, the optimal aggregate decision is automatically converted to unit decisions by the proposed algorithm. The coordination with mid/long-term planning is performed by taking into account the power demand allocated to the hydro-power plants. Numerical tests on Brazilian hydro-power plants demonstrate that the proposed formulation has lower computational cost than unit individualized models considering a given accuracy level for the generation function approximation.

Jacobi Fiber Surfaces for Bivariate Reeb Space Computation

Abstract: This paper presents an efficient algorithm for the computation of the Reeb space of an input bivariate piecewise linear scalar function f defined on a tetrahedral mesh. By extending and generalizing algorithmic concepts from the univariate case to the bivariate one, we report the first practical, output-sensitive algorithm for the exact computation of such a Reeb space. The algorithm starts by identifying the Jacobi set of f , the bivariate analogs of critical points in the univariate case. Next, the Reeb space is computed by segmenting the input mesh along the new notion of Jacobi Fiber Surfaces, the bivariate analog of critical contours in the univariate case. We additionally present a simplification heuristic that enables the progressive coarsening of the Reeb space. Our algorithm is simple to implement and most of its computations can be trivially parallelized. We report performance numbers demonstrating orders of magnitude speedups over previous approaches, enabling for the first time the tractable computation of bivariate Reeb spaces in practice. Moreover, unlike range-based quantization approaches (such as the Joint Contour Net), our algorithm is parameter-free. We demonstrate the utility of our approach by using the Reeb space as a semi-automatic segmentation tool for bivariate data. In particular, we introduce continuous scatterplot peeling, a technique which enables the reduction of the cluttering in the continuous scatterplot, by interactively selecting the features of the Reeb space to project. We provide a VTK-based C++ implementation of our algorithm that can be used for reproduction purposes or for the development of new Reeb space based visualization techniques.

Switching Frequency Regulation in Sliding Mode Control by a Hysteresis Band Controller

Abstract: Fixing the switching frequency is a key issue in sliding mode control implementations. This paper presents a hysteresis band controller capable of setting a constant value for the steady-state switching frequency of a sliding mode controller in regulation and tracking tasks. The proposed architecture relies on a piecewise linear modeling of the switching function behavior within the hysteresis band, and consists of a discrete-time integral-type controller that modifies the amplitude of the hysteresis band of the comparator in accordance with the error between the desired and the actually measured switching period. For tracking purposes, an additional feedforward action is introduced to compensate the time variation of the switching function derivatives at either sides of the switching hyperplane in the steady state. Stability proofs are provided, and a design criterion for the control parameters to guarantee closed-loop stability is subsequently derived. Numerical simulations and experimental results validate the proposal.

A comparison of piecewise linear programming formulations for stochastic disassembly line balancing

Abstract: Recently, several mathematical programming formulations and solution approaches have been developed for the stochastic disassembly line balancing problem (DLBP). This paper aims at finding optimal ...

Delay Scheduled Impulsive Control for Networked Control Systems

Abstract: This paper presents a new design approach for networked control systems under the integral quadratic constraint (IQC) framework. Two types of network induced time-varying delays, that is, measurement delay and actuation delay, are considered. A novel delay scheduled impulsive (DSI) controller is proposed, which utilizes both plant state and the IQC dynamic state, as well as the real-time network-induced delay information for gain scheduling feedback control. Robust ${\mathcal L}_{2}$ stability analysis of the resulting impulsive closed-loop system is performed using dynamic IQCs combined with a clock-dependent storage function. Based on the analysis results, the synthesis conditions for the proposed DSI controller are established as a finite number of linear matrix inequalities by specifying a piecewise linear storage function, which can be solved effectively via convex optimization. Finally, an application to a dc motor system demonstrates the effectiveness and advantages of the proposed design approach.

Theoretical properties of the global optimizer of two-layer Neural Network

Abstract: In this paper, we study the problem of optimizing a two-layer artificial neural network that best fits a training dataset. We look at this problem in the setting where the number of parameters is greater than the number of sampled points. We show that for a wide class of differentiable activation functions (this class involves "almost" all functions which are not piecewise linear), we have that first-order optimal solutions satisfy global optimality provided the hidden layer is non-singular. Our results are easily extended to hidden layers given by a flat matrix from that of a square matrix. Results are applicable even if network has more than one hidden layer provided all hidden layers satisfy non-singularity, all activations are from the given "good" class of differentiable functions and optimization is only with respect to the last hidden layer. We also study the smoothness properties of the objective function and show that it is actually Lipschitz smooth, i.e., its gradients do not change sharply. We use smoothness properties to guarantee asymptotic convergence of O(1/number of iterations) to a first-order optimal solution. We also show that our algorithm will maintain non-singularity of hidden layer for any finite number of iterations.

Pointwise frequency responses framework for stability analysis in periodically time-varying systems

Abstract: This paper explicates a pointwise frequency-domain approach for stability analysis in periodically time-varying continuous systems, by employing piecewise linear time-invariant PLTI models defined via piecewise-constant approximation and their frequency responses. The PLTI models are piecewise LTI state-space expressions, which provide theoretical and numerical conveniences in the frequency-domain analysis and synthesis. More precisely, stability, controllability and positive realness of periodically time-varying continuous systems are examined by means of PLTI models; then their pointwise frequency responses PFR are connected to stability analysis. Finally, Nyquist-like and circle-like criteria are claimed in terms of PFR's for asymptotic stability, finite-gain Lp-stability and uniformly boundedness, respectively, in linear feedbacks and nonlinear Lure connections. The suggested stability conditions have explicit and direct matrix expressions, where neither Floquet factorisations of transition matrices nor open-loop unstable poles are involved, and their implementation can be graphical and numerical. Illustrative studies are sketched to show applications of the main results.

Nonconvex piecewise linear functions: Advanced formulations and simple modeling tools

Abstract: We present novel mixed-integer programming (MIP) formulations for optimization over nonconvex piecewise linear functions. We exploit recent advances in the systematic construction of MIP formulations to derive new formulations for univariate functions using a geometric approach, and for bivariate functions using a combinatorial approach. All formulations are strong, small (so-called logarithmic formulations), and have other desirable computational properties. We present extensive experiments in which they exhibit substantial computational performance improvements over existing approaches. To accompany these advanced formulations, we present PiecewiseLinearOpt, an extension of the JuMP modeling language in Julia that implements our models (alongside other formulations from the literature) through a high-level interface, hiding the complexity of the formulations from the end-user.