Matrix Differential Calculus with Applications in Statistics and Econometrics

Newness and distinctiveness is claimed in the features of ornamentation as shown inside the broken line circle in the accompanying representation.

Display screen or portion thereof with graphical user interface

Deep learning approaches to anomaly detection have recently improved the state of the art in detection performance on complex datasets such as large collections of images or text. These results have sparked a renewed interest in the anomaly detection problem and led to the introduction of a great variety of new methods. With the emergence of numerous such methods, including approaches based on generative models, one-class classification, and reconstruction, there is a growing need to bring methods of this field into a systematic and unified perspective. In this review we aim to identify the common underlying principles as well as the assumptions that are often made implicitly by various methods. In particular, we draw connections between classic 'shallow' and novel deep approaches and show how this relation might cross-fertilize or extend both directions. We further provide an empirical assessment of major existing methods that is enriched by the use of recent explainability techniques, and present specific worked-through examples together with practical advice. Finally, we outline critical open challenges and identify specific paths for future research in anomaly detection.

A Unifying Review of Deep and Shallow Anomaly Detection

Deep learning approaches to anomaly detection (AD) have recently improved the state of the art in detection performance on complex data sets, such as large collections of images or text. These results have sparked a renewed interest in the AD problem and led to the introduction of a great variety of new methods. With the emergence of numerous such methods, including approaches based on generative models, one-class classification, and reconstruction, there is a growing need to bring methods of this field into a systematic and unified perspective. In this review, we aim to identify the common underlying principles and the assumptions that are often made implicitly by various methods. In particular, we draw connections between classic “shallow” and novel deep approaches and show how this relation might cross-fertilize or extend both directions. We further provide an empirical assessment of major existing methods that are enriched by the use of recent explainability techniques and present specific worked-through examples together with practical advice. Finally, we outline critical open challenges and identify specific paths for future research in AD.

/pdf/a-unifying-review-of-deep-and-shallow-anomaly-detection-61ph1bv9hs.pdf

The theory and applications of integral equations is an very important subject within applied mathematics. Integral equations are used as mathematical models for many and varied physical situations but the integral equations also occur as reformulations of other mathematical problems.

https://link.springer.com/content/pdf/10.1007%2F978-94-011-5338-6_4.pdf

Numerical Treatment of the Integral Equations

Support vector data description (SVDD) is a popular technique for detecting anomalies. The SVDD classifier partitions the whole space into an inlier region, which consists of the region near the training data, and an outlier region, which consists of points away from the training data. The computation of the SVDD classifier requires a kernel function, and the Gaussian kernel is a common choice for the kernel function. The Gaussian kernel has a bandwidth parameter, whose value is important for good results. A small bandwidth leads to overfitting, and the resulting SVDD classifier overestimates the number of anomalies. A large bandwidth leads to underfitting, and the classifier fails to detect many anomalies. In this paper we present a new automatic, unsupervised method for selecting the Gaussian kernel bandwidth. The selected value can be computed quickly, and it is competitive with existing bandwidth selection methods.

The Mean and Median Criteria for Kernel Bandwidth Selection for Support Vector Data Description

Support vector data description (SVDD) is a machine learning technique that is used for single-class classification and outlier detection. The idea of SVDD is to find a set of support vectors that defines a boundary around data. When dealing with online or large data, existing batch SVDD methods have to be rerun in each iteration. We propose an incremental learning algorithm for SVDD that uses the Gaussian kernel. This algorithm builds on the observation that all support vectors on the boundary have the same distance to the center of sphere in a higher-dimensional feature space as mapped by the Gaussian kernel function. Each iteration involves only the existing support vectors and the new data point. Moreover, the algorithm is based solely on matrix manipulations; the support vectors and their corresponding Lagrange multiplier αi’s are automatically selected and determined in each iteration. It can be seen that the complexity of our algorithm in each iteration is only O(k2), where k is the number of support vectors. Experimental results on some real data sets indicate that FISVDD demonstrates significant gains in efficiency with almost no loss in either outlier detection accuracy or objective function value.

/pdf/fast-incremental-svdd-learning-algorithm-with-the-gaussian-30vpk6mv2c.pdf

Fast Incremental SVDD Learning Algorithm with the Gaussian Kernel

Support vector data description (SVDD) is a machine learning technique that is used for single-class classification and outlier detection. The idea of SVDD is to find a set of support vectors that defines a boundary around data. When dealing with online or large data, existing batch SVDD methods have to be rerun in each iteration. We propose an incremental learning algorithm for SVDD that uses the Gaussian kernel. This algorithm builds on the observation that all support vectors on the boundary have the same distance to the center of sphere in a higher-dimensional feature space as mapped by the Gaussian kernel function. Each iteration involves only the existing support vectors and the new data point. Moreover, the algorithm is based solely on matrix manipulations; the support vectors and their corresponding Lagrange multiplier $\alpha_i$'s are automatically selected and determined in each iteration. It can be seen that the complexity of our algorithm in each iteration is only $O(k^2)$, where $k$ is the number of support vectors. Experimental results on some real data sets indicate that FISVDD demonstrates significant gains in efficiency with almost no loss in either outlier detection accuracy or objective function value.

Local outlier factor (LOF) is a state-of-the-art unsupervised machine learning model for anomaly detection in the Internet of Things (IoT). In recent years, there have been many important applications of LOF in large IoT systems, such as wireless transmission device system in high-speed trains, smart city vehicular social networks, industrial conditional monitoring systems, and network infrastructures for cyber security. LOF aims to forecast the imminent faults of an appliance in the IoT system, whose predictive performance greatly depends on the selection of its hyperparameters. The major challenge of hyperparameter tuning for unsupervised machine learning models including LOF in the context of IoT systems is that the incoming data may contain new types of anomaly that are unseen in the training data. In this paper, we propose a novel, heuristic methodology to tune the hyperparameters in LOF that accounts for this challenge. A tuned LOF model that uses the proposed method shows good predictive performance in both simulations and real data applications.

Automatic Hyperparameter Tuning Method for Local Outlier Factor, with Applications to Anomaly Detection

Support Vector Data Description (SVDD) is a machine learning technique used for single class classification and outlier detection. SVDD formulation with kernel function provides a flexible boundary around data. The value of kernel function parameters affects the nature of data boundary. For example, it is observed that with Gaussian kernel, as the value of kernel bandwidth is lowered, the data boundary changes from spherical to wiggly. The spherical data boundary leads to underfitting and extremely wiggly data boundary leads to overfitting. In this paper we propose an empirical criteria to obtain a good value of Gaussian kernel bandwidth which provides a smooth boundary capturing the essential visual features of the data.

Deovrat Kakde

Papers

The Mean and Median Criteria for Kernel Bandwidth Selection for Support Vector Data Description

Fast Incremental SVDD Learning Algorithm with the Gaussian Kernel

Fast Incremental SVDD Learning Algorithm with the Gaussian Kernel

Automatic Hyperparameter Tuning Method for Local Outlier Factor, with Applications to Anomaly Detection

Peak criterion for choosing Gaussian kernel bandwidth in Support Vector Data Description