# Online Non-linear Topology Identification from Graph-connected Time Series

05 Jun 2021-

TL;DR: In this article, the authors proposed an online kernel-based algorithm for topology estimation of non-linear vector autoregressive time series by solving a sparse online optimization framework using the composite objective mirror descent method.

Abstract: Estimating the unknown causal dependencies among graph-connected time series plays an important role in many applications, such as sensor network analysis, signal processing over cyber-physical systems, and finance engineering. Inference of such causal dependencies, often know as topology identification, is not well studied for non-linear non-stationary systems, and most of the existing methods are batch-based which are not capable of handling streaming sensor signals. In this paper, we propose an online kernel-based algorithm for topology estimation of non-linear vector autoregressive time series by solving a sparse online optimization framework using the composite objective mirror descent method. Experiments conducted on real and synthetic data sets show that the proposed algorithm outperforms the state-of-the-art methods for topology estimation.

##### Citations

More filters

••

TL;DR: In this paper , an algorithmic framework is proposed to learn time-varying graphs from online data, where different functions regulate different desiderata, e.g., data fidelity, sparsity or smoothness.

Abstract: This work proposes an algorithmic framework to learn time-varying graphs from online data. The generality offered by the framework renders it model-independent, i.e., it can be theoretically analyzed in its abstract formulation and then instantiated under a variety of model-dependent graph learning problems. This is possible by phrasing (time-varying) graph learning as a composite optimization problem, where different functions regulate different desiderata, e.g., data fidelity, sparsity or smoothness. Instrumental for the findings is recognizing that the dependence of the majority (if not all) data-driven graph learning algorithms on the data is exerted through the empirical covariance matrix, representing a sufficient statistic for the estimation problem. Its user-defined recursive update enables the framework to work in non-stationary environments, while iterative algorithms building on novel time-varying optimization tools explicitly take into account the temporal dynamics, speeding up convergence and implicitly including a temporal-regularization of the solution. We specialize the framework to three well-known graph learning models, namely, the Gaussian graphical model (GGM), the structural equation model (SEM), and the smoothness-based model (SBM), where we also introduce ad-hoc vectorization schemes for structured matrices (symmetric, hollows, etc.) which are crucial to perform correct gradient computations, other than enabling to work in low-dimensional vector spaces and hence easing storage requirements. After discussing the theoretical guarantees of the proposed framework, we corroborate it with extensive numerical tests in synthetic and real data.

9 citations

•

08 Dec 2020

TL;DR: This article transforms the given LDM into an LDM with hidden nodes, where the hidden nodes are characterized using maximal cliques in the correlation graph and all the nodes are excited by uncorrelated noise.

Abstract: In this article, we address the problem of reconstructing the topology of a linear dynamic model (LDM), where the nodes are excited with spatially correlated exogenous noise sources, from its time series data. We transform the given LDM into an LDM with hidden nodes, where the hidden nodes are characterized using maximal cliques in the correlation graph and all the nodes are excited by uncorrelated noise. Then, using the sparse $+$ low-rank matrix decomposition of the imaginary part of the inverse power spectral density matrix--computed solely from the time series observation of the true LDM--, we reconstruct the topology along with the correlation graph.

7 citations

••

TL;DR: In this article , a kernel-based algorithm for online graph topology estimation is proposed, which performs a Fourier-based random feature approximation to tackle the curse of dimensionality associated with kernel representations.

Abstract: Online topology estimation of graph-connected time series is challenging in practice, particularly because the dependencies between the time series in many real-world scenarios are nonlinear. To address this challenge, we introduce a novel kernel-based algorithm for online graph topology estimation. Our proposed algorithm also performs a Fourier-based random feature approximation to tackle the curse of dimensionality associated with kernel representations. Exploiting the fact that real-world networks often exhibit sparse topologies, we propose a group-Lasso based optimization framework, which is solved using an iterative composite objective mirror descent method, yielding an online algorithm with fixed computational complexity per iteration. We provide theoretical guarantees for our algorithm and prove that it can achieve sublinear dynamic regret under certain reasonable assumptions. In experiments conducted on both real and synthetic data, our method outperforms existing state-of-the-art competitors.

3 citations

••

TL;DR: An online algorithm for missing data imputation for networks with signals defined on the edges is presented in this article , which exploits the causal dependencies and the flow conservation, respectively via (i) a sparse line graph identification strategy based on a group-Lasso and (ii) a Kalman filtering-based signal reconstruction strategy developed using simplicial complex (SC) formulation.

Abstract: An online algorithm for missing data imputation for networks with signals defined on the edges is presented. Leveraging the prior knowledge intrinsic to real-world networks, we propose a bi-level optimization scheme that exploits the causal dependencies and the flow conservation, respectively via (i) a sparse line graph identification strategy based on a group-Lasso and (ii) a Kalman filtering-based signal reconstruction strategy developed using simplicial complex (SC) formulation. The advantages of this first SC-based attempt for time-varying signal imputation have been demonstrated through numerical experiments using EPANET models of both synthetic and real water distribution networks.

2 citations

••

29 Aug 2022

TL;DR: A group lasso-based optimization framework for topology identification is proposed, which is solved online using alternating minimization techniques and can be illustrated using several numerical experiments conducted using both synthetic and real data.

Abstract: Extracting causal graph structures from multivariate time series, termed topology identification, is a fundamental problem in network science with several important applications. Topology identification is a challenging problem in real-world sensor networks, especially when the available time series are partially observed due to faulty communication links or sensor failures. The problem becomes even more challenging when the sensor dependencies are nonlinear and nonstationary. This paper proposes a kernel-based online framework using random feature approximation to jointly estimate nonlinear causal dependencies and missing data from partial observations of streaming graph-connected time series. Exploiting the fact that real-world networks often exhibit sparse topologies, we propose a group lasso-based optimization framework for topology identification, which is solved online using alternating minimization techniques. The ability of the algorithm is illustrated using several numerical experiments conducted using both synthetic and real data.

1 citations

##### References

More filters

••

01 Dec 2001TL;DR: Learning with Kernels provides an introduction to SVMs and related kernel methods that provide all of the concepts necessary to enable a reader equipped with some basic mathematical knowledge to enter the world of machine learning using theoretically well-founded yet easy-to-use kernel algorithms.

Abstract: From the Publisher:
In the 1990s, a new type of learning algorithm was developed, based on results from statistical learning theory: the Support Vector Machine (SVM). This gave rise to a new class of theoretically elegant learning machines that use a central concept of SVMs-kernels--for a number of learning tasks. Kernel machines provide a modular framework that can be adapted to different tasks and domains by the choice of the kernel function and the base algorithm. They are replacing neural networks in a variety of fields, including engineering, information retrieval, and bioinformatics.
Learning with Kernels provides an introduction to SVMs and related kernel methods. Although the book begins with the basics, it also includes the latest research. It provides all of the concepts necessary to enable a reader equipped with some basic mathematical knowledge to enter the world of machine learning using theoretically well-founded yet easy-to-use kernel algorithms and to understand and apply the powerful algorithms that have been developed over the last few years.

7,880 citations

•

01 Mar 1990

TL;DR: In this paper, a theory and practice for the estimation of functions from noisy data on functionals is developed, where convergence properties, data based smoothing parameter selection, confidence intervals, and numerical methods are established which are appropriate to a number of problems within this framework.

Abstract: This book serves well as an introduction into the more theoretical aspects of the use of spline models. It develops a theory and practice for the estimation of functions from noisy data on functionals. The simplest example is the estimation of a smooth curve, given noisy observations on a finite number of its values. Convergence properties, data based smoothing parameter selection, confidence intervals, and numerical methods are established which are appropriate to a number of problems within this framework. Methods for including side conditions and other prior information in solving ill posed inverse problems are provided. Data which involves samples of random variables with Gaussian, Poisson, binomial, and other distributions are treated in a unified optimization context. Experimental design questions, i.e., which functionals should be observed, are studied in a general context. Extensions to distributed parameter system identification problems are made by considering implicitly defined functionals.

6,120 citations

••

16 Jul 2001

TL;DR: The result shows that a wide range of problems have optimal solutions that live in the finite dimensional span of the training examples mapped into feature space, thus enabling us to carry out kernel algorithms independent of the (potentially infinite) dimensionality of the feature space.

Abstract: Wahba's classical representer theorem states that the solutions of certain risk minimization problems involving an empirical risk term and a quadratic regularizer can be written as expansions in terms of the training examples. We generalize the theorem to a larger class of regularizers and empirical risk terms, and give a self-contained proof utilizing the feature space associated with a kernel. The result shows that a wide range of problems have optimal solutions that live in the finite dimensional span of the training examples mapped into feature space, thus enabling us to carry out kernel algorithms independent of the (potentially infinite) dimensionality of the feature space.

1,707 citations

••

[...]

TL;DR: A regularized model for linear regression with ℓ1 andℓ2 penalties is introduced and it is shown that it has the desired effect of group-wise and within group sparsity.

Abstract: For high-dimensional supervised learning problems, often using problem-specific assumptions can lead to greater accuracy. For problems with grouped covariates, which are believed to have sparse effects both on a group and within group level, we introduce a regularized model for linear regression with l1 and l2 penalties. We discuss the sparsity and other regularization properties of the optimal fit for this model, and show that it has the desired effect of group-wise and within group sparsity. We propose an algorithm to fit the model via accelerated generalized gradient descent, and extend this model and algorithm to convex loss functions. We also demonstrate the efficacy of our model and the efficiency of our algorithm on simulated data. This article has online supplementary material.

1,233 citations

[...]

01 Dec 2006

TL;DR: Conditions on the features of a continuous kernel are investigated so that it may approximate an arbitrary continuous target function uniformly on any compact subset of the input space.

Abstract: In this paper we investigate conditions on the features of a continuous kernel so that it may approximate an arbitrary continuous target function uniformly on any compact subset of the input space. A number of concrete examples are given of kernels with this universal approximating property.

509 citations