Showing papers on "Iterative method published in 2017"

Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization

Abstract: Gatys et al. recently introduced a neural algorithm that renders a content image in the style of another image, achieving so-called style transfer. However, their framework requires a slow iterative optimization process, which limits its practical application. Fast approximations with feed-forward neural networks have been proposed to speed up neural style transfer. Unfortunately, the speed improvement comes at a cost: the network is usually tied to a fixed set of styles and cannot adapt to arbitrary new styles. In this paper, we present a simple yet effective approach that for the first time enables arbitrary style transfer in real-time. At the heart of our method is a novel adaptive instance normalization (AdaIN) layer that aligns the mean and variance of the content features with those of the style features. Our method achieves speed comparable to the fastest existing approach, without the restriction to a pre-defined set of styles. In addition, our approach allows flexible user controls such as content-style trade-off, style interpolation, color & spatial controls, all using a single feed-forward neural network.

Channel Pruning for Accelerating Very Deep Neural Networks

Abstract: In this paper, we introduce a new channel pruning method to accelerate very deep convolutional neural networks. Given a trained CNN model, we propose an iterative two-step algorithm to effectively prune each layer, by a LASSO regression based channel selection and least square reconstruction. We further generalize this algorithm to multi-layer and multi-branch cases. Our method reduces the accumulated error and enhance the compatibility with various architectures. Our pruned VGG-16 achieves the state-of-the-art results by 5× speed-up along with only 0.3% increase of error. More importantly, our method is able to accelerate modern networks like ResNet, Xception and suffers only 1.4%, 1.0% accuracy loss under 2× speedup respectively, which is significant.

Deep Convolutional Neural Network for Inverse Problems in Imaging

Abstract: In this paper, we propose a novel deep convolutional neural network (CNN)-based algorithm for solving ill-posed inverse problems. Regularized iterative algorithms have emerged as the standard approach to ill-posed inverse problems in the past few decades. These methods produce excellent results, but can be challenging to deploy in practice due to factors including the high computational cost of the forward and adjoint operators and the difficulty of hyperparameter selection. The starting point of this paper is the observation that unrolled iterative methods have the form of a CNN (filtering followed by pointwise non-linearity) when the normal operator ( $H^{*}H$ , where $H^{*}$ is the adjoint of the forward imaging operator, $H$ ) of the forward model is a convolution. Based on this observation, we propose using direct inversion followed by a CNN to solve normal-convolutional inverse problems. The direct inversion encapsulates the physical model of the system, but leads to artifacts when the problem is ill posed; the CNN combines multiresolution decomposition and residual learning in order to learn to remove these artifacts while preserving image structure. We demonstrate the performance of the proposed network in sparse-view reconstruction (down to 50 views) on parallel beam X-ray computed tomography in synthetic phantoms as well as in real experimental sinograms. The proposed network outperforms total variation-regularized iterative reconstruction for the more realistic phantoms and requires less than a second to reconstruct a $512\times 512$ image on the GPU.

Solving ill-posed inverse problems using iterative deep neural networks

Abstract: We propose a partially learned approach for the solution of ill-posed inverse problems with not necessarily linear forward operators. The method builds on ideas from classical regularisation theory ...

One Network to Solve Them All — Solving Linear Inverse Problems Using Deep Projection Models

Abstract: While deep learning methods have achieved state-of-theart performance in many challenging inverse problems like image inpainting and super-resolution, they invariably involve problem-specific training of the networks. Under this approach, each inverse problem requires its own dedicated network. In scenarios where we need to solve a wide variety of problems, e.g., on a mobile camera, it is inefficient and expensive to use these problem-specific networks. On the other hand, traditional methods using analytic signal priors can be used to solve any linear inverse problem; this often comes with a performance that is worse than learning-based methods. In this work, we provide a middle ground between the two kinds of methods — we propose a general framework to train a single deep neural network that solves arbitrary linear inverse problems. We achieve this by training a network that acts as a quasi-projection operator for the set of natural images and show that any linear inverse problem involving natural images can be solved using iterative methods. We empirically show that the proposed framework demonstrates superior performance over traditional methods using wavelet sparsity prior while achieving performance comparable to specially-trained networks on tasks including compressive sensing and pixel-wise inpainting.

Superpixels and Polygons Using Simple Non-iterative Clustering

Abstract: We present an improved version of the Simple Linear Iterative Clustering (SLIC) superpixel segmentation. Unlike SLIC, our algorithm is non-iterative, enforces connectivity from the start, requires lesser memory, and is faster. Relying on the superpixel boundaries obtained using our algorithm, we also present a polygonal partitioning algorithm. We demonstrate that our superpixels as well as the polygonal partitioning are superior to the respective state-of-the-art algorithms on quantitative benchmarks.

Joint User Scheduling and Power Allocation Optimization for Energy-Efficient NOMA Systems With Imperfect CSI

Abstract: Non-orthogonal multiple access (NOMA) exploits successive interference cancellation technique at the receivers to improve the spectral efficiency. By using this technique, multiple users can be multiplexed on the same subchannel to achieve high sum rate. Most previous research works on NOMA systems assume perfect channel state information (CSI). However, in this paper, we investigate energy efficiency improvement for a downlink NOMA single-cell network by considering imperfect CSI. The energy efficient resource scheduling problem is formulated as a non-convex optimization problem with the constraints of outage probability limit, the maximum power of the system, the minimum user data rate, and the maximum number of multiplexed users sharing the same subchannel. Different from previous works, the maximum number of multiplexed users can be greater than two, and the imperfect CSI is first studied for resource allocation in NOMA. To efficiently solve this problem, the probabilistic mixed problem is first transformed into a non-probabilistic problem. An iterative algorithm for user scheduling and power allocation is proposed to maximize the system energy efficiency. The optimal user scheduling based on exhaustive search serves as a system performance benchmark, but it has high computational complexity. To balance the system performance and the computational complexity, a new suboptimal user scheduling scheme is proposed to schedule users on different subchannels. Based on the user scheduling scheme, the optimal power allocation expression is derived by the Lagrange approach. By transforming the fractional-form problem into an equivalent subtractive-form optimization problem, an iterative power allocation algorithm is proposed to maximize the system energy efficiency. Simulation results demonstrate that the proposed user scheduling algorithm closely attains the optimal performance.

Joint Beamforming and Power-Splitting Control in Downlink Cooperative SWIPT NOMA Systems

Abstract: This paper investigates the application of simultaneous wireless information and power transfer (SWIPT) to cooperative non-orthogonal multiple access (NOMA). A new cooperative multiple-input single-output (MISO) SWIPT NOMA protocol is proposed, where a user with a strong channel condition acts as an energy-harvesting (EH) relay by adopting power splitting (PS) scheme to help a user with a poor channel condition. By jointly optimizing the PS ratio and the beamforming vectors, we aim at maximizing the data rate of the “strong user” while satisfying the QoS requirement of the “weak user”. To resolve the formulated nonconvex problem, the semidefinite relaxation (SDR) technique is applied to reformulate the original problem, by proving the rank-one optimality. And then an iterative algorithm based on successive convex approximation (SCA) is proposed for complexity reduction, which can at least attain its stationary point efficiently. In view of the potential application scenarios, e.g., Internet of Things (IoT), the single-input single-output (SISO) case is also studied. The formulated problem is proved to be strictly unimodal with respect to the PS ratio. Hence, a golden section search (GSS) based algorithm with closed-form solution at each step is proposed to find the unique global optimal solution. It is worth pointing out that the SCA method can also converge to the optimal solution in SISO cases. In the numerical simulation, the proposed algorithm is numerically shown to converge within a few iterations, and the SWIPT-aided NOMA protocol outperforms the existing transmission protocols.

Extended dynamic mode decomposition with dictionary learning: A data-driven adaptive spectral decomposition of the Koopman operator

Abstract: Numerical approximation methods for the Koopman operator have advanced considerably in the last few years. In particular, data-driven approaches such as dynamic mode decomposition (DMD)51 and its generalization, the extended-DMD (EDMD), are becoming increasingly popular in practical applications. The EDMD improves upon the classical DMD by the inclusion of a flexible choice of dictionary of observables which spans a finite dimensional subspace on which the Koopman operator can be approximated. This enhances the accuracy of the solution reconstruction and broadens the applicability of the Koopman formalism. Although the convergence of the EDMD has been established, applying the method in practice requires a careful choice of the observables to improve convergence with just a finite number of terms. This is especially difficult for high dimensional and highly nonlinear systems. In this paper, we employ ideas from machine learning to improve upon the EDMD method. We develop an iterative approximation algorithm which couples the EDMD with a trainable dictionary represented by an artificial neural network. Using the Duffing oscillator and the Kuramoto Sivashinsky partical differential equation as examples, we show that our algorithm can effectively and efficiently adapt the trainable dictionary to the problem at hand to achieve good reconstruction accuracy without the need to choose a fixed dictionary a priori. Furthermore, to obtain a given accuracy, we require fewer dictionary terms than EDMD with fixed dictionaries. This alleviates an important shortcoming of the EDMD algorithm and enhances the applicability of the Koopman framework to practical problems.

Denoising of Hyperspectral Images Using Nonconvex Low Rank Matrix Approximation

Abstract: Hyperspectral image (HSI) denoising is challenging not only because of the difficulty in preserving both spectral and spatial structures simultaneously, but also due to the requirement of removing various noises, which are often mixed together. In this paper, we present a nonconvex low rank matrix approximation (NonLRMA) model and the corresponding HSI denoising method by reformulating the approximation problem using nonconvex regularizer instead of the traditional nuclear norm, resulting in a tighter approximation of the original sparsity-regularised rank function. NonLRMA aims to decompose the degraded HSI, represented in the form of a matrix, into a low rank component and a sparse term with a more robust and less biased formulation. In addition, we develop an iterative algorithm based on the augmented Lagrangian multipliers method and derive the closed-form solution of the resulting subproblems benefiting from the special property of the nonconvex surrogate function. We prove that our iterative optimization converges easily. Extensive experiments on both simulated and real HSIs indicate that our approach can not only suppress noise in both severely and slightly noised bands but also preserve large-scale image structures and small-scale details well. Comparisons against state-of-the-art LRMA-based HSI denoising approaches show our superior performance.

Unrolled Optimization with Deep Priors

Abstract: A broad class of problems at the core of computational imaging, sensing, and low-level computer vision reduces to the inverse problem of extracting latent images that follow a prior distribution, from measurements taken under a known physical image formation model Traditionally, hand-crafted priors along with iterative optimization methods have been used to solve such problems In this paper we present unrolled optimization with deep priors, a principled framework for infusing knowledge of the image formation into deep networks that solve inverse problems in imaging, inspired by classical iterative methods We show that instances of the framework outperform the state-of-the-art by a substantial margin for a wide variety of imaging problems, such as denoising, deblurring, and compressed sensing magnetic resonance imaging (MRI) Moreover, we conduct experiments that explain how the framework is best used and why it outperforms previous methods

Primal-Dual Plug-and-Play Image Restoration

Abstract: We propose a new plug-and-play image restoration method based on primal-dual splitting Existing plug-and-play image restoration methods interpret any off-the-shelf Gaussian denoiser as one step of the so-called alternating direction method of multipliers (ADMM) This makes it possible to exploit the power of such a highly-customized Gaussian denoising method for general image restoration tasks in a plug-and-play fashion However, the ADMM-based plug-and-play approach (ADMMPnP) has several limitations: 1) it often requires a problem-specific iterative method in solving a subproblem, which results in a computationally expensive inner loop; and 2) it is specialized to handle the formulation of a regularization (plug-and-play) term plus a data-fidelity term, so that it does not allow to impose hard constraints useful for image restoration Our approach resolves these issues by leveraging the nature of primal-dual splitting, yielding a very flexible plug-and-play image restoration method Experimental results demonstrate that the proposed method is much more efficient than ADMMPnP with an inner loop, whereas it keeps the same efficiency as ADMMPnP in the case where the subproblem of ADMMPnP can be solved efficiently

Decomposition based least squares iterative identification algorithm for multivariate pseudo-linear ARMA systems using the data filtering

Abstract: This paper develops a decomposition based least squares iterative identification algorithm for multivariate pseudo-linear autoregressive moving average systems using the data filtering. The key is to apply the data filtering technique to transform the original system to a hierarchical identification model, and to decompose this model into three subsystems and to identify each subsystem, respectively. Compared with the least squares based iterative algorithm, the proposed algorithm requires less computational efforts. The simulation results show that the proposed algorithms can work well.

Energy-Efficient Power Allocation in Millimeter Wave Massive MIMO With Non-Orthogonal Multiple Access

Abstract: In this letter, we investigate the energy efficiency (EE) problem in a millimeter wave massive MIMO system with non-orthogonal multiple access (NOMA). Multiple two-user clusters are formulated according to their channel correlation and gain difference. Following this, we propose a hybrid analog/digital precoding scheme for the low radio frequency chains structure at the base station. On this basis, we formulate a power allocation problem aiming to maximize the EE under users’ quality of service requirements and per-cluster power constraint. An iterative algorithm is proposed to obtain the optimal power allocation. Simulation results show that the proposed NOMA scheme achieves superior EE performance than conventional OMA scheme.

Super-Resolution Delay-Doppler Estimation for OFDM Passive Radar

Abstract: In this paper, we consider the problem of joint delay-Doppler estimation of moving targets in a passive radar that makes use of orthogonal frequency-division multiplexing communication signals. A compressed sensing algorithm is proposed to achieve supper resolution and better accuracy, using both the atomic norm and the $\ell _1$-norm. The atomic norm is used to manifest the signal sparsity in the continuous domain. Unlike previous works that assume the demodulation to be error free, we explicitly introduce the demodulation error signal whose sparsity is imposed by the $\ell _1$-norm. On this basis, the delays and Doppler frequencies are estimated by solving a semidefinite program (SDP) which is convex. We also develop an iterative method for solving this SDP via the alternating direction method of multipliers where each iteration involves closed-form computation. Simulation results are presented to illustrate the high performance of the proposed algorithm.

Iterative Constrained Weighted Least Squares Source Localization Using TDOA and FDOA Measurements

Abstract: This paper investigates the constrained weighted least squares (CWLS) source localization problem by using time difference of arrival and frequency difference of arrival measurements. The problem can be formulated as a quadratic programming with two indefinite quadratic equality constraints, which is nonconvex and NP-hard. Moreover, the weighting matrix is coupled with the unknown source position and velocity. We propose an iterative CWLS method that can efficiently solve this problem. It iteratively performs a linearization procedure on the quadratic equality constraints to obtain an approximate programming with linear constraints, which can be analytically solved, and the weighting matrix is updated in each iteration. Theoretical analysis reveals that the proposed method, if converges, can lead to the global optimal solution of the formulated problem that reaches the CRLB accuracy under mild assumptions on the measurement noises. The Monte Carlo simulation results indicate that the percentage of convergence within $\mathbf 20$ iterations is more than $\mathbf 96\%$ , and the localization accuracy is significantly improved over the previous methods with less computation time requirement. Moreover, it is found from simulations that the iterative CWLS method retains acceptable performance even under the ill-conditioned situation when the sensor geometry is not desirable.

A unified approach to error bounds for structured convex optimization problems

Abstract: Error bounds, which refer to inequalities that bound the distance of vectors in a test set to a given set by a residual function, have proven to be extremely useful in analyzing the convergence rates of a host of iterative methods for solving optimization problems. In this paper, we present a new framework for establishing error bounds for a class of structured convex optimization problems, in which the objective function is the sum of a smooth convex function and a general closed proper convex function. Such a class encapsulates not only fairly general constrained minimization problems but also various regularized loss minimization formulations in machine learning, signal processing, and statistics. Using our framework, we show that a number of existing error bound results can be recovered in a unified and transparent manner. To further demonstrate the power of our framework, we apply it to a class of nuclear-norm regularized loss minimization problems and establish a new error bound for this class under a strict complementarity-type regularity condition. We then complement this result by constructing an example to show that the said error bound could fail to hold without the regularity condition. We believe that our approach will find further applications in the study of error bounds for structured convex optimization problems.

A new directional stability transformation method of chaos control for first order reliability analysis

Abstract: The HL-RF iterative algorithm of the first order reliability method (FORM) is popularly applied to evaluate reliability index in structural reliability analysis and reliability-based design optimization. However, it sometimes suffers from non-convergence problems, such as bifurcation, periodic oscillation, and chaos for nonlinear limit state functions. This paper derives the formulation of the Lyapunov exponents for the HL-RF iterative algorithm in order to identify these complicated numerical instability phenomena of discrete chaotic dynamic systems. Moreover, the essential cause of low efficiency for the stability transform method (STM) of convergence control of FORM is revealed. Then, a novel method, directional stability transformation method (DSTM), is proposed to reduce the number of function evaluations of original STM as a chaos feedback control approach. The efficiency and convergence of different reliability evaluation methods, including the HL-RF algorithm, STM and DSTM, are analyzed and compared by several numerical examples. It is indicated that the proposed DSTM method is versatile, efficient and robust, and the bifurcation, periodic oscillation, and chaos of FORM is controlled effectively.

An eigenmodel for iterative line planning, timetabling and vehicle scheduling in public transportation

Abstract: Planning a public transportation system is a multi-objective problem which includes among others line planning, timetabling, and vehicle scheduling. For each of these planning stages, models are known and advanced solution techniques exist. Some of the models focus on costs, others on passengers’ convenience. Setting up a transportation system is usually done by optimizing each of these stages sequentially. In this paper we argue that instead of optimizing each single step further and further it would be more beneficial to consider the whole process in an integrated way. To this end, we develop and discuss a generic, bi-objective model for integrating line planning, timetabling, and vehicle scheduling. We furthermore propose an eigenmodel which we apply for these three planning stages and show how it can be used for the design of iterative algorithms as heuristics for the integrated problem. The convergence of the resulting iterative approaches is analyzed from a theoretical point of view. Moreover, we propose an agenda for further research in this field.

Iterative Adaptive Dynamic Programming for Solving Unknown Nonlinear Zero-Sum Game Based on Online Data

Abstract: $H_\infty $ control is a powerful method to solve the disturbance attenuation problems that occur in some control systems. The design of such controllers relies on solving the zero-sum game (ZSG). But in practical applications, the exact dynamics is mostly unknown. Identification of dynamics also produces errors that are detrimental to the control performance. To overcome this problem, an iterative adaptive dynamic programming algorithm is proposed in this paper to solve the continuous-time, unknown nonlinear ZSG with only online data. A model-free approach to the Hamilton–Jacobi–Isaacs equation is developed based on the policy iteration method. Control and disturbance policies and value are approximated by neural networks (NNs) under the critic–actor–disturber structure. The NN weights are solved by the least-squares method. According to the theoretical analysis, our algorithm is equivalent to a Gauss–Newton method solving an optimization problem, and it converges uniformly to the optimal solution. The online data can also be used repeatedly, which is highly efficient. Simulation results demonstrate its feasibility to solve the unknown nonlinear ZSG. When compared with other algorithms, it saves a significant amount of online measurement time.

Iterative machine teaching

Abstract: In this paper, we consider the problem of machine teaching, the inverse problem of machine learning. Different from traditional machine teaching which views the learners as batch algorithms, we study a new paradigm where the learner uses an iterative algorithm and a teacher can feed examples sequentially and intelligently based on the current performance of the learner. We show that the teaching complexity in the iterative case is very different from that in the batch case. Instead of constructing a minimal training set for learners, our iterative machine teaching focuses on achieving fast convergence in the learner model. Depending on the level of information the teacher has from the learner model, we design teaching algorithms which can provably reduce the number of teaching examples and achieve faster convergence than learning without teachers. We also validate our theoretical findings with extensive experiments on different data distribution and real image datasets.

Parameter estimation for control systems based on impulse responses

Abstract: The impulse signal is an instant change signal in very short time. It is widely used in signal processing, electronic technique, communication and system identification. This paper considers the parameter estimation problems for dynamical systems by means of the impulse response measurement data. Since the cost function is highly nonlinear, the nonlinear optimization methods are adopted to derive the parameter estimation algorithms to enhance the estimation accuracy. By using the iterative scheme, the Newton iterative algorithm and the gradient iterative algorithm are proposed for estimating the parameters of dynamical systems. Also, a damping factor is introduced to improve the algorithm stability. Finally, using simulation examples, this paper analyzes and compares the merit and weakness of the proposed algorithms.

A new technology for solving diffusion and heat equations

Abstract: In this paper, a new technology combing the variational iterative method and an integral transform similar to Sumudu transform is proposed for the first time for solutions of diffusion and heat equations. The method is accurate and efficient in development of approximate solutions for the partial differential equations.

A Non-Greedy Algorithm for L1-Norm LDA

Abstract: Recently, L1-norm-based discriminant subspace learning has attracted much more attention in dimensionality reduction and machine learning However, most existing approaches solve the column vectors of the optimal projection matrix one by one with greedy strategy Thus, the obtained optimal projection matrix does not necessarily best optimize the corresponding trace ratio objective function, which is the essential criterion function for general supervised dimensionality reduction In this paper, we propose a non-greedy iterative algorithm to solve the trace ratio form of L1-norm-based linear discriminant analysis We analyze the convergence of our proposed algorithm in detail Extensive experiments on five popular image databases illustrate that our proposed algorithm can maximize the objective function value and is superior to most existing L1-LDA algorithms

Iterative Reconstruction of Memory Kernels

Abstract: In recent years, it has become increasingly popular to construct coarse-grained models with non-Markovian dynamics to account for an incomplete separation of time scales One challenge of a systematic coarse-graining procedure is the extraction of the dynamical properties, namely, the memory kernel, from equilibrium all-atom simulations In this article, we propose an iterative method for memory reconstruction from dynamical correlation functions Compared to previously proposed noniterative techniques, it ensures by construction that the target correlation functions of the original fine-grained systems are reproduced accurately by the coarse-grained system, regardless of time step and discretization effects Furthermore, we also propose a new numerical integrator for generalized Langevin equations that is significantly more accurate than the more commonly used generalization of the velocity Verlet integrator We demonstrate the performance of the above-described methods using the example of backflow-indu

Multiple-Frequency DBIM-TwIST Algorithm for Microwave Breast Imaging

Abstract: A novel distorted Born iterative method (DBIM) algorithm is proposed for microwave breast imaging based on the two-step iterative shrinkage/thresholding method. We show that this implementation is more flexible and robust than using traditional Krylov subspace methods such as the CGLS as solvers of the ill-posed linear problem. This paper presents several strategies to increase the algorithm’s robustness: a hybrid multifrequency approach to achieve an optimal tradeoff between imaging accuracy and reconstruction stability; a new approach to estimate the average breast tissues properties, based on sampling along their range of possible values and running a few DBIM iterations to find the minimum error; and finally, a new regularization strategy for the DBIM method based on the $L^{1}$ norm and the Pareto curve. We present reconstruction examples which illustrate the benefits of these optimization strategies, which have resulted in a DBIM algorithm that outperforms our previous implementations for microwave breast imaging.

Fast Iterative Method with a Second-Order Implicit Difference Scheme for Time-Space Fractional Convection–Diffusion Equation

Abstract: In this paper we intend to establish fast numerical approaches to solve a class of initial-boundary problem of time-space fractional convection–diffusion equations. We present a new unconditionally stable implicit difference method, which is derived from the weighted and shifted Grunwald formula, and converges with the second-order accuracy in both time and space variables. Then, we show that the discretizations lead to Toeplitz-like systems of linear equations that can be efficiently solved by Krylov subspace solvers with suitable circulant preconditioners. Each time level of these methods reduces the memory requirement of the proposed implicit difference scheme from $${\mathcal {O}}(N^2)$$ to $${\mathcal {O}}(N)$$ and the computational complexity from $${\mathcal {O}}(N^3)$$ to $${\mathcal {O}}(N\log N)$$ in each iterative step, where N is the number of grid nodes. Extensive numerical examples are reported to support our theoretical findings and show the utility of these methods over traditional direct solvers of the implicit difference method, in terms of computational cost and memory requirements.