Probability Distributions.- Linear Models for Regression.- Linear Models for Classification.- Neural Networks.- Kernel Methods.- Sparse Kernel Machines.- Graphical Models.- Mixture Models and EM.- Approximate Inference.- Sampling Methods.- Continuous Latent Variables.- Sequential Data.- Combining Models.

Pattern Recognition and Machine Learning

Matrix Factorization Techniques for Recommender Systems

IEEE transactions on systems, man and cybernetics. Part B, Cybernetics

Wireless Body Sensor Nodes (WBSN) are frequently used for real time IoT-based health monitoring of patients outside the hospital environment. These WBSNs involve bio-sensors to capture signals from a patient’s body and wireless transmitters to transmit the collected signals to a server located in private/public cloud in real time. These WBSNs include hardware for processing of signals before being transmitted to the cloud. Simultaneous occurrence of all these processes inside energy constrained WBSNs results in considerable amount of power consumption, thus limiting their operational lifetime. Due to the inherent error-resilience in signal processing algorithms, most of these data reaching the servers are redundant in nature and hence of not much clinical importance. Transmission and storage of these excess data result in inefficient usages of transmission bandwidth and storage capabilities. In this paper, we develop a real time encoding scheme that performs iterative thresholding and approximation of wavelet coefficients for sparse encoding of bio-signals (ECG signals), thereby reducing the energy and bandwidth consumption of the WBSN. The encoding scheme compresses bio-signals (ECG signals), while still maintaining the clinically important features. We optimize various process parameters to model a low power hardware prototype for the implementation of our algorithm on a real time microcontroller based IoT platform that operates as an end-to-end WBSN system in real time. Experimental results show a system-level energy improvement of 96% with a negligible impact on signal quality (2%).

Energy-Efficient IoT-Health Monitoring System using Approximate Computing

Suppose an $n \times d$ design matrix in a linear regression problem is given, but the response for each point is hidden unless explicitly requested. The goal is to sample only a small number $k \ll n$ of the responses, and then produce a weight vector whose sum of squares loss over all points is at most $1+\epsilon$ times the minimum. When $k$ is very small (e.g., $k=d$), jointly sampling diverse subsets of points is crucial. One such method called volume sampling has a unique and desirable property that the weight vector it produces is an unbiased estimate of the optimum. It is therefore natural to ask if this method offers the optimal unbiased estimate in terms of the number of responses $k$ needed to achieve a $1+\epsilon$ loss approximation. 
Surprisingly we show that volume sampling can have poor behavior when we require a very accurate approximation -- indeed worse than some i.i.d. sampling techniques whose estimates are biased, such as leverage score sampling. We then develop a new rescaled variant of volume sampling that produces an unbiased estimate which avoids this bad behavior and has at least as good a tail bound as leverage score sampling: sample size $k=O(d\log d + d/\epsilon)$ suffices to guarantee total loss at most $1+\epsilon$ times the minimum with high probability. Thus, we improve on the best previously known sample size for an unbiased estimator, $k=O(d^2/\epsilon)$. 
Our rescaling procedure leads to a new efficient algorithm for volume sampling which is based on a determinantal rejection sampling technique with potentially broader applications to determinantal point processes. Other contributions include introducing the combinatorics needed for rescaled volume sampling and developing tail bounds for sums of dependent random matrices which arise in the process.

Leveraged volume sampling for linear regression.

In this paper, we introduce a novel and robust approach to quantized matrix completion. First, we propose a rank minimization problem with constraints induced by quantization bounds. Next, we form an unconstrained optimization problem by regularizing the rank function with Huber loss. Huber loss is leveraged to control the violation from quantization bounds due to two properties: first, it is differentiable; and second, it is less sensitive to outliers than the quadratic loss. A smooth rank approximation is utilized to endorse lower rank on the genuine data matrix. Thus, an unconstrained optimization problem with differentiable objective function is obtained allowing us to advantage from gradient descent technique. Novel and firm theoretical analysis of the problem model and convergence of our algorithm to the global solution are provided. Another contribution of this letter is that our method does not require projections or initial rank estimation, unlike the state-of-the-art. In the Numerical Experiments section, the noticeable outperformance of our proposed method in learning accuracy and computational complexity compared to those of the state-of-the-art literature methods is illustrated as the main contribution.

A Novel Approach to Quantized Matrix Completion Using Huber Loss Measure

Adaptive thresholding methods have proved to yield a high signal-to-noise ratio (SNR) and fast convergence in sparse signal recovery. The robustness of a class of iterative sparse recovery algorithms, such as the iterative method with adaptive thresholding, has been found to outperform the state-of-art methods in respect of reconstruction quality, convergence speed, and sensitivity to noise. In this study, the authors introduce a new method for compressed sensing, using the sensing matrix and measurements. In our method, they iteratively threshold the signal and project the thresholded signal onto the translated null space of the sensing matrix. The threshold level is assigned adaptively. The results of the simulations reveal that the authors’ proposed method outperforms other methods in the signal reconstruction (in terms of the SNR). This performance advantage is noticeable when the number of available measurements approaches twice the sparsity number.

https://ietresearch.onlinelibrary.wiley.com/doi/epdf/10.1049/iet-spr.2016.0626

Iterative null space projection method with adaptive thresholding in sparse signal recovery

Finding a small subset of data whose linear combination spans other data points, also called column subset selection problem (CSSP), is an important open problem in computer science with many applications in computer vision and deep learning. There are some studies that solve CSSP in a polynomial time complexity w.r.t. the size of the original dataset. A simple and efficient selection algorithm with a linear complexity order, referred to as spectrum pursuit (SP), is proposed that pursuits spectral components of the dataset using available sample points. The proposed non-greedy algorithm aims to iteratively find K data samples whose span is close to that of the first K spectral components of entire data. SP has no parameter to be fine tuned and this desirable property makes it problem-independent. The simplicity of SP enables us to extend the underlying linear model to more complex models such as nonlinear manifolds and graph-based models. The nonlinear extension of SP is introduced as kernel-SP (KSP). The superiority of the proposed algorithms is demonstrated in a wide range of applications.

https://openaccess.thecvf.com/content_CVPR_2020/papers/Joneidi_Select_to_Better_Learn_Fast_and_Accurate_Deep_Learning_Using_CVPR_2020_paper.pdf

Select to Better Learn: Fast and Accurate Deep Learning Using Data Selection From Nonlinear Manifolds

Adaptive thresholding methods have proved to yield high SNRs and fast convergence in finding the solution to the Compressed Sensing (CS) problems. Recently, it was observed that the robustness of a class of iterative sparse recovery algorithms such as Iterative Method with Adaptive Thresholding (IMAT) has outperformed the well-known LASSO algorithm in terms of reconstruction quality, convergence speed, and the sensitivity to the noise. In this paper, we introduce a new method towards solving the CS problem. The logic of this method is based on iterative projections of the thresholded signal onto the null-space of the sensing matrix. The thresholding is carried out by recovering the support of the desired signal by projection on thresholding subspaces. The simulations reveal that the proposed method has the capability of yielding noticeable output SNR values with about as many samples as twice the sparsity number, while other methods fail to recover the signals when approaching the algebraic bound for the number of samples required. The computational complexity of our method is also comparable to other methods as observed in the simulations. We have also extended our Algorithm to Matrix Completion (MC) scenarios and compared its efficiency to other well-reputed approaches for MC in the literature.

Iterative Null-space Projection Method with Adaptive Thresholding in Sparse Signal Recovery and Matrix Completion.

Recovering low-rank and sparse components from missing observations is an essential problem in various fields. In this paper, we have proposed a method to address the missing low-rank and sparse decomposition problem. We have used the smoothed nuclear norm and the $L_{1}$ norm to impose the low-rankness and sparsity constraints on the components, respectively. Furthermore, we have suggested a linear modeling for the corrupted observations. The problem has been solved with the aid of alternating minimization. Moreover, some simplifications have been applied to the relations to reduce the computational complexity, which makes the algorithm suitable for large-scale problems. To evaluate the proposed method, different simulation scenarios have been devised. The superiority of the suggested scheme over its counterparts has been confirmed on both the recovery accuracy and the convergence speed in various applications.

Ashkan Esmaeili

Papers

A Novel Approach to Quantized Matrix Completion Using Huber Loss Measure

Iterative null space projection method with adaptive thresholding in sparse signal recovery

Select to Better Learn: Fast and Accurate Deep Learning Using Data Selection From Nonlinear Manifolds

Iterative Null-space Projection Method with Adaptive Thresholding in Sparse Signal Recovery and Matrix Completion.

Missing Low-Rank and Sparse Decomposition Based on Smoothed Nuclear Norm