In this paper, we propose a new subsequence matching method, Dual Match, which exploits duality in constructing windows and significantly improves performance. Dual Match divides data sequences into disjoint windows and the query sequence into sliding windows, and thus, is a dual approach of the one by Faloutsos et al. (FRM in short), which divides data sequences into sliding windows and the query sequence into disjoint windows. We formally prove that our dual approach is correct, i.e., it incurs no false dismissal. We also prove that, given the minimum query length, there is a maximum bound of the window size to guarantee correctness ofDual Match and discuss the effect of the window size on performance. FRM causes a lot of false alarms (i.e., candidates that do not qualify) by storing minimum bounding rectangles rather than individual points representing windows to avoid excessive storage space required for the index. DualMatch solves this problem by directly storing points, but without incurring excessive storage overhead. Experimental results show that, in most cases, Dual Match provides large improvement in both false alarms and performance over FRM, given the same amount of storage space. In particular, for low selectivities (less than 10 -4 ), Dual Match significantly improves performance up to 430-fold. On the other hand, for high selectivities (more than 10 -2 ), it shows a very minor degradation (less than 29%). For selectivities in between (10- 4 ∼10 -2 ), Dual Match shows performance slightly better than that of FRM. Dual Match is also 4.10∼25.6 times faster than FRM in building indexes of approximately the same size. Overall, these results indicate that our approach provides a new paradigm in subsequence matching that improves performance significantly in large database applications.

Duality-based subsequence matching in time-series databases

Database management is currently an inevitable part of information technology. It requires data to be obtained in a fast, reliable, and consistent manner. Indexes are defined as an optional structure to improve Select statement performance. On the other hand, they cause additional demands, if data are changed, whereas such change must be reflected in the index, as well. This paper aims to discuss the auto-indexing methods provided by DBS Oracle, highlighting their limitations. We propose own techniques to remove the impact of peaks caused by adding new queries to the system, to which no suitable index is present. It is based on the block data analysis in the pointer layer.

Analysis of current trends in relational database indexing

We introduce VOCALExplore, a system designed to support users in building domain-specific models over video datasets. VOCALExplore supports interactive labeling sessions and trains models using user-supplied labels. VOCALExplore maximizes model quality by automatically deciding how to select samples based on observed skew in the collected labels. It also selects the optimal video representations to use when training models by casting feature selection as a rising bandit problem. Finally, VOCALExplore implements optimizations to achieve low latency without sacrificing model performance. We demonstrate that VOCALExplore achieves close to the best possible model quality given candidate acquisition functions and feature extractors, and it does so with low visible latency (~1 second per iteration) and no expensive preprocessing.

VOCALExplore: Pay-as-You-Go Video Data Exploration and Model Building

The widespread popularity of the Internet of human life has been accompanied by a significant increase in the cost of protecting private data from malicious attacks. Researchers have proposed many deep learning-based intrusion detection methods. However, traditional methods rely on a large number of unpolluted data to learn benign data distributions, while nonidentical distributions will affect the performance in distinguishing normal and abnormal data. To address this problem, this article proposes an intrusion detection method feedback semi-supervised learning with meta-gradient for intrusion detection (FSMG) based on feedback deep semi-supervised learning. FSMG constructs a lightweight evaluation network with slight data augmentation and nonprocessing on the same input, respectively, and uses a small amount of labeled data to infer nonidentically distributed data flows hidden in the training dataset. Then, FSMG converts malicious flows into useful information, continuously track and update the model, reducing data labeling errors. Furthermore, for the updating of gradients in the model, a bilevel nested optimization is used to ensure the model converges within <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$O({\mathrm{C}}/{\sqrt{T}})$</tex-math></inline-formula> . Unlike other semisupervised algorithms, FSMG uses labeled data and nonidentically distributed unlabeled data proportionally to construct the training dataset, achieving higher classification accuracy, and better robustness even with an 80% polluted rate. 

A Feedback Semi-Supervised Learning With Meta-Gradient for Intrusion Detection

The widespread popularity of the Internet of human life has been accompanied by a significant increase in the cost of protecting private data from malicious attacks. Researchers have proposed many deep learning-based intrusion detection methods. However, traditional methods rely on a large number of unpolluted data to learn benign data distributions, while nonidentical distributions will affect the performance in distinguishing normal and abnormal data. To address this problem, this article proposes an intrusion detection method feedback semi-supervised learning with meta-gradient for intrusion detection (FSMG) based on feedback deep semi-supervised learning. FSMG constructs a lightweight evaluation network with slight data augmentation and nonprocessing on the same input, respectively, and uses a small amount of labeled data to infer nonidentically distributed data flows hidden in the training dataset. Then, FSMG converts malicious flows into useful information, continuously track and update the model, reducing data labeling errors. Furthermore, for the updating of gradients in the model, a bilevel nested optimization is used to ensure the model converges within $O({\mathrm{C}}/{\sqrt{T}})$. Unlike other semisupervised algorithms, FSMG uses labeled data and nonidentically distributed unlabeled data proportionally to construct the training dataset, achieving higher classification accuracy, and better robustness even with an 80% polluted rate.

Scalable subsequence matching is critical for supporting analytics on big time series from mining, prediction to hypothesis testing. However, state-of-the-art subsequence matching techniques do not scale well to TB-scale datasets. Not only does index construction become prohibitively expensive, but also the query response time deteriorates quickly as the length of the query subsequence exceeds several 100s of data points. Although Locality Sensitive Hashing (LSH) has emerged as a promising solution for indexing long time series, it relies on expensive hash functions that perform multiple passes over the data and thus is impractical for big time series. In this work, we propose a lightweight distributed indexing framework, called ChainLink, that supports approximate kNN queries over TB-scale time series data. As a foundation of ChainLink, we design a novel hashing technique, called Single Pass Signature (SPS), that successfully tackles the above problem. In particular, we prove theoretically and demonstrate experimentally that the similarity proximity of the indexed subsequences is preserved by our proposed single-pass SPS scheme. Leveraging this SPS innovation, Chainlink then adopts a three-step approach for scalable index building: (1) in-place data re-organization within each partition to enable efficient record-level random access to all subsequences, (2) parallel building of hash-based local indices on top of the re-organized data using our SPS scheme for efficient search within each partition, and (3) efficient aggregation of the local indices to construct a centralized yet highly compact global index for effective pruning of irrelevant partitions during query processing. ChainLink achieves the above three steps in one single map-reduce process. Our experimental evaluation shows that ChainLink indices are compact at less than 2% of dataset size while state-of-the-art index sizes tend to be almost the same size as the dataset. Better still, ChainLink is up to 2 orders of magnitude faster in its index construction time compared to state-of-the-art techniques, while improving both the final query response time by up to 10 fold and the result accuracy by 15%.

ChainLink: Indexing Big Time Series Data For Long Subsequence Matching

Deep Learning techniques have been widely used in detecting anomalies from complex data. Most of these techniques are either unsupervised or semi-supervised because of a lack of a large number of labeled anomalies. However, they typically rely on a clean training data not polluted by anomalies to learn the distribution of the normal data. Otherwise, the learned distribution tends to be distorted and hence ineffective in distinguishing between normal and abnormal data. To solve this problem, we propose a novel approach called ELITE that uses a small number of labeled examples to infer the anomalies hidden in the training samples. It then turns these anomalies into useful signals that help to better detect anomalies from user data. Unlike the classical semi-supervised classification strategy which uses labeled examples as training data, ELITE uses them as validation set. It leverages the gradient of the validation loss to predict if one training sample is abnormal. The intuition is that correctly identifying the hidden anomalies could produce a better deep anomaly model with reduced validation loss. Our experiments on public benchmark datasets show that ELITE achieves up to 30% improvement in ROC AUC comparing to the state-of-the-art, yet robust to polluted training data.

/pdf/elite-robust-deep-anomaly-detection-with-meta-gradient-v9mge2w6g7.pdf

ELITE: Robust Deep Anomaly Detection with Meta Gradient

Similarity search is the basis for many data analytics techniques, including k-nearest neighbor classification and outlier detection. Similarity search over large data sets relies on i) a distance metric learned from input examples and ii) an index to speed up search based on the learned distance metric. In interactive systems, input to guide the learning of the distance metric may be provided over time. As this new input changes the learned distance metric, a naive approach would adopt the costly process of re-indexing all items after each metric change. In this paper, we propose the first solution, called OASIS, to instantaneously adapt the index to conform to a changing distance metric without this prohibitive re-indexing process. To achieve this, we prove that locality-sensitive hashing (LSH) provides an invariance property, meaning that an LSH index built on the original distance metric is equally effective at supporting similarity search using an updated distance metric as long as the transform matrix learned for the new distance metric satisfies certain properties. This observation allows OASIS to avoid recomputing the index from scratch in most cases. Further, for the rare cases when an adaption of the LSH index is shown to be necessary, we design an efficient incremental LSH update strategy that re-hashes only a small subset of the items in the index. In addition, we develop an efficient distance metric learning strategy that incrementally learns the new metric as inputs are received. Our experimental study using real world public datasets confirms the effectiveness of OASIS at improving the accuracy of various similarity search-based data analytics tasks by instantaneously adapting the distance metric and its associated index in tandem, while achieving an up to 3 orders of magnitude speedup over the state-of-art techniques.

https://dl.acm.org/doi/pdf/10.1145/3318464.3380601

Continuously Adaptive Similarity Search

/pdf/lancet-labeling-complex-data-at-scale-4hn4ykg4wr.pdf

LANCET: labeling complex data at scale

Anomaly detection is a critical task in applications like preventing financial fraud, system malfunctions, and cybersecurity attacks. While previous research has offered a plethora of anomaly detection algorithms, effective anomaly detection remains challenging for users due to the tedious manual tuning process. Currently, model developers must determine which of these numerous algorithms is best suited for their particular domain and then must tune many parameters by hand to make the chosen algorithm perform well. This demonstration showcases AutoOD, the first unsupervised self-tuning anomaly detection system which frees users from this tedious manual tuning process. AutoOD outperforms the best un-supervised anomaly detection methods it deploys, with its performance similar to those of supervised anomaly classification models, yet without requiring ground truth labels. Our easy-to-use visual interface allows users to gain insights into AutoOD's self-tuning process and explore the underlying patterns within their datasets.

Huayi Zhang

Papers

ChainLink: Indexing Big Time Series Data For Long Subsequence Matching

ELITE: Robust Deep Anomaly Detection with Meta Gradient

Continuously Adaptive Similarity Search

LANCET: labeling complex data at scale

A demonstration of AutoOD