Most of the signal processing that we will study in this course involves local operations on a signal, namely transforming the signal by applying linear combinations of values in the neighborhood of each sample point. You are familiar with such operations from Calculus, namely, taking derivatives and you are also familiar with this from optics namely blurring a signal. We will be looking at sampled signals only. Let's start with a few basic examples. Local difference Suppose we have a 1D image and we take the local difference of intensities, DI(x) = 1 2 (I(x + 1) − I(x − 1)) which give a discrete approximation to a partial derivative. (We compute this for each x in the image.) What is the effect of such a transformation? One key idea is that such a derivative would be useful for marking positions where the intensity changes. Such a change is called an edge. It is important to detect edges in images because they often mark locations at which object properties change. These can include changes in illumination along a surface due to a shadow boundary, or a material (pigment) change, or a change in depth as when one object ends and another begins. The computational problem of finding intensity edges in images is called edge detection. We could look for positions at which DI(x) has a large negative or positive value. Large positive values indicate an edge that goes from low to high intensity, and large negative values indicate an edge that goes from high to low intensity. Example Suppose the image consists of a single (slightly sloped) edge:

http://ieeexplore.ieee.org/iel5/5/31307/01456462.pdf

Linear systems

The construction of a meaningful graph topology plays a crucial role in the effective representation, processing, analysis and visualization of structured data. When a natural choice of the graph is not readily available from the data sets, it is thus desirable to infer or learn a graph topology from the data. In this tutorial overview, we survey solutions to the problem of graph learning, including classical viewpoints from statistics and physics, and more recent approaches that adopt a graph signal processing (GSP) perspective. We further emphasize the conceptual similarities and differences between classical and GSP-based graph inference methods, and highlight the potential advantage of the latter in a number of theoretical and practical scenarios. We conclude with several open issues and challenges that are keys to the design of future signal processing and machine learning algorithms for learning graphs from data.

Learning graphs from data: A signal representation perspective

The construction of a meaningful graph topology plays a crucial role in the effective representation, processing, analysis, and visualization of structured data. When a natural choice of the graph is not readily available from the data sets, it is thus desirable to infer or learn a graph topology from the data. In this article, we survey solutions to the problem of graph learning, including classical viewpoints from statistics and physics, and more recent approaches that adopt a graph signal processing (GSP) perspective. We further emphasize the conceptual similarities and differences between classical and GSP-based graph-inference methods and highlight the potential advantage of the latter in a number of theoretical and practical scenarios. We conclude with several open issues and challenges that are keys to the design of future signal processing and machine-learning algorithms for learning graphs from data.

Learning Graphs From Data: A Signal Representation Perspective

Understand the benefits of robust statistics for signal processing with this authoritative yet accessible text. The first ever book on the subject, it provides a comprehensive overview of the field, moving from fundamental theory through to important new results and recent advances. Topics covered include advanced robust methods for complex-valued data, robust covariance estimation, penalized regression models, dependent data, robust bootstrap, and tensors. Robustness issues are illustrated throughout using real-world examples and key algorithms are included in a MATLAB Robust Signal Processing Toolbox accompanying the book online, allowing the methods discussed to be easily applied and adapted to multiple practical situations. This unique resource provides a powerful tool for researchers and practitioners working in the field of signal processing.

Robust Statistics for Signal Processing

Federated Learning is a machine learning scheme in which a shared prediction model can be collaboratively learned by a number of distributed nodes using their locally stored data. It can provide better data privacy because training data are not transmitted to a central server. Federated learning is well suited for edge computing applications and can leverage the the computation power of edge servers and the data collected on widely dispersed edge devices. To build such an edge federated learning system, we need to tackle a number of technical challenges. In this survey, we provide a new perspective on the applications, development tools, communication efficiency, security & privacy, migration and scheduling in edge federated learning.

A survey of federated learning for edge computing: Research problems and solutions

The problem of simultaneously learning several related tasks has received considerable attention in several domains, especially in machine learning, with the so-called multitask learning (MTL) problem, or learning to learn problem [1], [2]. MTL is an approach to inductive transfer learning (using what is learned for one problem to assist with another problem), and it helps improve generalization performance relative to learning each task separately by using the domain information contained in the training signals of related tasks as an inductive bias. Several strategies have been derived within this community under the assumption that all data are available beforehand at a fusion center.

https://hal.archives-ouvertes.fr/hal-03347202/document

Multitask Learning Over Graphs: An Approach for Distributed, Streaming Machine Learning

Driven by the need to solve increasingly complex optimization problems in signal processing and machine learning, there has been increasing interest in understanding the behavior of gradient-descent algorithms in non-convex environments. Most available works on distributed non-convex optimization problems focus on the deterministic setting where exact gradients are available at each agent. In this work and its Part II, we consider stochastic cost functions, where exact gradients are replaced by stochastic approximations and the resulting gradient noise persistently seeps into the dynamics of the algorithm. We establish that the diffusion learning strategy continues to yield meaningful estimates non-convex scenarios in the sense that the iterates by the individual agents will cluster in a small region around the network centroid. We use this insight to motivate a short-term model for network evolution over a finite-horizon. In Part II [2] of this work, we leverage this model to establish descent of the diffusion strategy through saddle points in O(1/$\mu$) steps and the return of approximately second-order stationary points in a polynomial number of iterations.

Distributed Learning in Non-Convex Environments -- Part I: Agreement at a Linear Rate

The diffusion strategy for distributed learning from streaming data employs local stochastic gradient updates along with exchange of iterates over neighborhoods. In Part I [2] of this work we established that agents cluster around a network centroid and proceeded to study the dynamics of this point. We established expected descent in non-convex environments in the large-gradient regime and introduced a short-term model to examine the dynamics over finite-time horizons. Using this model, we establish in this work that the diffusion strategy is able to escape from strict saddle-points in O(1/$\mu$) iterations; it is also able to return approximately second-order stationary points in a polynomial number of iterations. Relative to prior works on the polynomial escape from saddle-points, most of which focus on centralized perturbed or stochastic gradient descent, our approach requires less restrictive conditions on the gradient noise process.

Distributed Learning in Non-Convex Environments -- Part II: Polynomial Escape from Saddle-Points

Graphs provide a powerful framework to represent high-dimensional but structured data, and to make inferences about relationships between subsets of the data. In this work we consider graph signals that evolve dynamically according to a heat diffusion process and are subject to persistent perturbations. We develop an online algorithm that is able to learn the underlying graph structure from observations of the signal evolution. The algorithm is adaptive in nature and in particular able to respond to changes in the graph structure and the perturbation statistics.

Online graph learning from sequential data

In empirical risk optimization, it has been observed that stochastic gradient implementations that rely on random reshuffling of the data achieve better performance than implementations that rely on sampling the data uniformly. Recent works have pursued justifications for this behavior by examining the convergence rate of the learning process under diminishing step sizes. This work focuses on the constant step-size case and strongly convex loss functions. In this case, convergence is guaranteed to a small neighborhood of the optimizer albeit at a linear rate. The analysis establishes analytically that random reshuffling outperforms uniform sampling by showing explicitly that iterates approach a smaller neighborhood of size $O(\mu ^2)$ around the minimizer rather than $O(\mu)$ . Furthermore, we derive an analytical expression for the steady-state mean-square-error performance of the algorithm, which helps clarify in greater detail, the differences between sampling with and without replacement. We also explain the periodic behavior that is observed in random reshuffling implementations.

Stefan Vlaski

Papers

Multitask Learning Over Graphs: An Approach for Distributed, Streaming Machine Learning

Distributed Learning in Non-Convex Environments -- Part I: Agreement at a Linear Rate

Distributed Learning in Non-Convex Environments -- Part II: Polynomial Escape from Saddle-Points

Online graph learning from sequential data

Stochastic Learning Under Random Reshuffling With Constant Step-Sizes