Statistical Foundations of Data ScienceNew Perspectives and Challenges in Econophysics and SociophysicsStatistical Learning with SparsityMathematical Foundations of Infinite-Dimensional Statistical ModelsStatistics and Analysis of ShapesMultivariate StatisticsHigh-Dimensional Covariance EstimationPrinciples and Methods for Data ScienceHigh-Dimensional StatisticsHigh-Dimensional ProbabilityHandbook of Financial Econometrics and StatisticsStatistics for High-Dimensional DataRegularization in High-dimensional StatisticsAnalysis of Multivariate and High-Dimensional DataProbability and ComputingLarge Sample Covariance Matrices and High-Dimensional Data AnalysisHigh-Dimensional Data Analysis in Cancer ResearchHandbook of Big Data AnalyticsIntroduction to Clustering Large and High-Dimensional DataAnalyzing High-Dimensional Gene Expression and DNA Methylation Data with RSufficient Dimension ReductionHandbook of Big DataHandbook of Mixture AnalysisHigh Dimensional Probability VIStatistical Analysis for High-Dimensional DataHigh-Dimensional ProbabilitySpectral Analysis of Large Dimensional Random MatricesBig Data AnalyticsIntroduction to High-Dimensional StatisticsGeometric Structure of High-Dimensional Data and Dimensionality ReductionApplied Biclustering Methods for Big and High-Dimensional Data Using RBig and Complex Data AnalysisHigh-dimensional Data AnalysisFunctional Statistics and Related FieldsHandbook of Data VisualizationNonlinear Dimensionality ReductionModern Statistics for Modern BiologySparse Modeling for Image and Vision ProcessingModeling and Stochastic Learning for Forecasting in High DimensionsContributions to Fault Detection and Diagnosis with High-dimensional Data

Statistics for High-Dimensional Data: Methods, Theory and Applications

In recent times, deep artificial neural networks have achieved many successes in pattern recognition. Part of this success can be attributed to the reliance on big data to increase generalization. However, in the field of time series recognition, many datasets are often very small. One method of addressing this problem is through the use of data augmentation. In this paper, we survey data augmentation techniques for time series and their application to time series classification with neural networks. We propose a taxonomy and outline the four families in time series data augmentation, including transformation-based methods, pattern mixing, generative models, and decomposition methods. Furthermore, we empirically evaluate 12 time series data augmentation methods on 128 time series classification datasets with six different types of neural networks. Through the results, we are able to analyze the characteristics, advantages and disadvantages, and recommendations of each data augmentation method. This survey aims to help in the selection of time series data augmentation for neural network applications.

/pdf/an-empirical-survey-of-data-augmentation-for-time-series-5bdy19uq2z.pdf

An empirical survey of data augmentation for time series classification with neural networks.

Parkinson's disease (PD) is a neurodegenerative disease characterized by a progressive loss of dopaminergic neurons from the nigrostriatal pathway, formation of Lewy bodies, and microgliosis. During the past decades multiple cellular pathways have been associated with PD pathology (i.e., oxidative stress, endosomal-lysosomal dysfunction, endoplasmic reticulum stress, and immune response), yet disease-modifying treatments are not available. We have recently used genetic data from familial and sporadic cases in an unbiased approach to build a molecular landscape for PD, revealing lipids as central players in this disease. Here we extensively review the current knowledge concerning the involvement of various subclasses of fatty acyls, glycerolipids, glycerophospholipids, sphingolipids, sterols, and lipoproteins in PD pathogenesis. Our review corroborates a central role for most lipid classes, but the available information is fragmented, not always reproducible, and sometimes differs by sex, age or PD etiology of the patients. This hinders drawing firm conclusions about causal or associative effects of dietary lipids or defects in specific steps of lipid metabolism in PD. Future technological advances in lipidomics and additional systematic studies on lipid species from PD patient material may improve this situation and lead to a better appreciation of the significance of lipids for this devastating disease.

The Role of Lipids in Parkinson’s Disease

Transformers have achieved superior performances in many tasks in natural language processing and computer vision, which also triggered great interest in the time series community. Among multiple advantages of Transformers, the ability to capture long-range dependencies and interactions is especially attractive for time series modeling, leading to exciting progress in various time series applications. In this paper, we systematically review Transformer schemes for time series modeling by highlighting their strengths as well as limitations. In particular, we examine the development of time series Transformers in two perspectives. From the perspective of network structure, we summarize the adaptations and modifications that have been made to Transformers in order to accommodate the challenges in time series analysis. From the perspective of applications, we categorize time series Transformers based on common tasks including forecasting, anomaly detection, and classification. Empirically, we perform robust analysis, model size analysis, and seasonal-trend decomposition analysis to study how Transformers perform in time series. Finally, we discuss and suggest future directions to provide useful research guidance. To the best of our knowledge, this paper is the first work to comprehensively and systematically summarize the recent advances of Transformers for modeling time series data. We hope this survey will ignite further research interests in time series Transformers.

Transformers in Time Series: A Survey

Diagnosis of Parkinson's disease (PD) is commonly based on medical observations and assessment of clinical signs, including the characterization of a variety of motor symptoms. However, traditional diagnostic approaches may suffer from subjectivity as they rely on the evaluation of movements that are sometimes subtle to human eyes and therefore difficult to classify, leading to possible misclassification. In the meantime, early non-motor symptoms of PD may be mild and can be caused by many other conditions. Therefore, these symptoms are often overlooked, making diagnosis of PD at an early stage challenging. To address these difficulties and to refine the diagnosis and assessment procedures of PD, machine learning methods have been implemented for the classification of PD and healthy controls or patients with similar clinical presentations (e.g., movement disorders or other Parkinsonian syndromes). To provide a comprehensive overview of data modalities and machine learning methods that have been used in the diagnosis and differential diagnosis of PD, in this study, we conducted a literature review of studies published until February 14, 2020, using the PubMed and IEEE Xplore databases. A total of 209 studies were included, extracted for relevant information and presented in this review, with an investigation of their aims, sources of data, types of data, machine learning methods and associated outcomes. These studies demonstrate a high potential for adaptation of machine learning methods and novel biomarkers in clinical decision making, leading to increasingly systematic, informed diagnosis of PD.

/pdf/machine-learning-for-the-diagnosis-of-parkinson-s-disease-a-gfmgb6ry9x.pdf

Machine Learning for the Diagnosis of Parkinson's Disease: A Review of Literature.

Deep learning performs remarkably well on many time series analysis tasks recently. The superior performance of deep neural networks relies heavily on a large number of training data to avoid overfitting. However, the labeled data of many real-world time series applications may be limited such as classification in medical time series and anomaly detection in AIOps. As an effective way to enhance the size and quality of the training data, data augmentation is crucial to the successful application of deep learning models on time series data. In this paper, we systematically review different data augmentation methods for time series. We propose a taxonomy for the reviewed methods, and then provide a structured review for these methods by highlighting their strengths and limitations. We also empirically compare different data augmentation methods for different tasks including time series anomaly detection, classification, and forecasting. Finally, we discuss and highlight five future directions to provide useful research guidance.

/pdf/time-series-data-augmentation-for-deep-learning-a-survey-8b9ejcq10z.pdf

Time Series Data Augmentation for Deep Learning: A Survey

Although Transformer-based methods have significantly improved state-of-the-art results for long-term series forecasting, they are not only computationally expensive but more importantly, are unable to capture the global view of time series (e.g. overall trend). To address these problems, we propose to combine Transformer with the seasonal-trend decomposition method, in which the decomposition method captures the global profile of time series while Transformers capture more detailed structures. To further enhance the performance of Transformer for long-term prediction, we exploit the fact that most time series tend to have a sparse representation in well-known basis such as Fourier transform, and develop a frequency enhanced Transformer. Besides being more effective, the proposed method, termed as Frequency Enhanced Decomposed Transformer ({\bf FEDformer}), is more efficient than standard Transformer with a linear complexity to the sequence length. Our empirical studies with six benchmark datasets show that compared with state-of-the-art methods, FEDformer can reduce prediction error by $14.8\%$ and $22.6\%$ for multivariate and univariate time series, respectively. Code is publicly available at https://github.com/MAZiqing/FEDformer.

FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting

/pdf/time-series-data-augmentation-for-deep-learning-a-survey-26td23zmn5.pdf

Time Series Data Augmentation for Deep Learning: A Survey.

Convolutional Neural Networks (CNNs) have dominated computer vision for years, due to its ability in capturing locality and translation invariance. Recently, many vision transformer architectures have been proposed and they show promising performance. A key component in vision transformers is the fully-connected self-attention which is more powerful than CNNs in modelling long range dependencies. However, since the current dense self-attention uses all image patches (tokens) to compute attention matrix, it may neglect locality of images patches and involve noisy tokens (e.g., clutter background and occlusion), leading to a slow training process and potentially degradation of performance. To address these problems, we propose a sparse attention scheme, dubbed k-NN attention, for boosting vision transformers. Specifically, instead of involving all the tokens for attention matrix calculation, we only select the top-k similar tokens from the keys for each query to compute the attention map. The proposed k-NN attention naturally inherits the local bias of CNNs without introducing convolutional operations, as nearby tokens tend to be more similar than others. In addition, the k-NN attention allows for the exploration of long range correlation and at the same time filter out irrelevant tokens by choosing the most similar tokens from the entire image. Despite its simplicity, we verify, both theoretically and empirically, that $k$-NN attention is powerful in distilling noise from input tokens and in speeding up training. Extensive experiments are conducted by using ten different vision transformer architectures to verify that the proposed k-NN attention can work with any existing transformer architectures to improve its prediction performance.

KVT: k-NN Attention for Boosting Vision Transformers.

In this paper, we propose a Minimax Concave Penalized Multi-Armed Bandit (MCP-Bandit) algorithm for a decision-maker facing highdimensional data with latent sparse structure in an online learning and decision-making process. We demonstrate that the MCP-Bandit algorithm asymptotically achieves the optimal cumulative regret in the sample size T , O(log T ), and further attains a tighter bound in both the covariates dimension d and the number of significant covariates s, O(s(s + log d)). In addition, we develop a linear approximation method, the 2step Weighted Lasso procedure, to identify the MCP estimator for the MCP-Bandit algorithm under non-i.i.d. samples. Using this procedure, the MCP estimator matches the oracle estimator with high probability. Finally, we present two experiments to benchmark our proposed the MCPBandit algorithm to other bandit algorithms. Both experiments demonstrate that the MCP-Bandit algorithm performs favorably over other benchmark algorithms, especially when there is a high level of data sparsity or when the sample size is not too small.

/pdf/minimax-concave-penalized-multi-armed-bandit-model-with-high-es1bv283zl.pdf

Xue Wang

Papers

Time Series Data Augmentation for Deep Learning: A Survey

FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting

Time Series Data Augmentation for Deep Learning: A Survey.

KVT: k-NN Attention for Boosting Vision Transformers.

Minimax concave penalized multi-armed bandit model with high-dimensional convariates