A review on time series data mining

It has long been known that Dynamic Time Warping (DTW) is superior to Euclidean distance for classification and clustering of time series. However, until lately, most research has utilized Euclidean distance because it is more efficiently calculated. A recently introduced technique that greatly mitigates DTWs demanding CPU time has sparked a flurry of research activity. However, the technique and its many extensions still only allow DTW to be applied to moderately large datasets. In addition, almost all of the research on DTW has focused exclusively on speeding up its calculation; there has been little work done on improving its accuracy. In this work, we target the accuracy aspect of DTW performance and introduce a new framework that learns arbitrary constraints on the warping path of the DTW calculation. Apart from improving the accuracy of classification, our technique as a side effect speeds up DTW by a wide margin as well. We show the utility of our approach on datasets from diverse domains and demonstrate significant gains in accuracy and efficiency. E u clid ean D istan c e D yn am ic T im e W arp in g D is tan ce Figure 1: Note that while the two time series have an overall similar shape, they are not aligned in the time axis. Euclidean distance, which assumes the i point in one sequence is aligned with the i point in the other, will produce a pessimistic dissimilarity measure. The non-linear Dynamic Time Warped alignment allows a more intuitive distance measure to be calculated.

/pdf/making-time-series-classification-more-accurate-using-4pd6knwgvc.pdf

Making Time-Series Classification More Accurate Using Learned Constraints.

Market Models provides an authoritative and up–to–date treatment of the use of market data to develop models for financial analysis. Written by a leading figure in the field of financial data analysis, this book is the first of its kind to address the vital techniques required for model selection and development. Model developers are faced with many decisions, about the pricing, the data, the statistical methodology and the calibration and testing of the model prior to implementation. It is important to make the right choices and Carol Alexander s clear exposition provides valuable insights at every stage. In each of the 13 Chapters, Market Models presents real world illustrations to motivate theoretical developments. The accompanying CD contains spreadsheets with data and programs; this enables you to Market models: a guide to financial data analysis Market models: A guide to financial data analysis. Home > Research > Our research > STUDY WITH US ABOUT US NEWS & EVENTS INTERNATIONAL RESEARCH HOME implement and adapt many of the examples. The pricing of options using normal mixture density functions to model returns; the use of Monte Carlo simulation to calculate the VaR of an options portfolio; modifying the covariance VaR to allow for fat–tailed P&L distributions; the calculation of implied, EWMA and historic volatilities; GARCH volatility term structure forecasting; principal components analysis; and many more are all included. Carol Alexander brings many new insights to the pricing and hedging of options with her understanding of volatility and correlation, and the uncertainty which surrounds these key determinants of option portfolio risk. Modelling the market risk of portfolios is covered where the main focus is on a linear algebraic approach; the covariance matrix and principal component analysis are developed as key tools for the analysis of financial systems. The traditional time series econometric approach is also explained with coverage ranging from the application cointegration to long–short equity hedge funds, to high–frequency data prediction using neural networks and nearest neighbour algorithms. Throughout this text the emphasis is on understanding concepts and implementing solutions. It has been designed to be accessible to a very wide audience: the coverage is comprehensive and complete and the technical appendix makes the book largely self–contained. Market Models: A Guide to Financial Data Analysis is the ideal reference for all those involved in market risk measurement, quantitative trading and investment analysis. Item Type: Book Schools and Departments: School of Business, Management and Economics > Business and Management Subjects: H Social Sciences > HG Finance Depositing User: Carol Alexander Date Deposited: 27 Sep 2012 12:36 Last Modified: 27 Sep 2012 12:36 URI: http://sro.sussex.ac.uk/id/eprint/40646

Market Models: A Guide to Financial Data Analysis

We develop a warped correlation finder to identify correlated user accounts in social media websites such as Twitter. The key observation is that humans cannot be highly synchronous for a long duration, thus, highly synchronous user accounts are most likely bots. Existing bot detection methods are mostly supervised, which requires a large amount of labeled data to train, and do not consider cross-user features. In contrast, our bot detection system works on activity correlation without requiring labeled data. We develop a novel lag-sensitive hashing technique to cluster user accounts into correlated sets in near real-time. Our method, named DeBot, detects thousands of bots per day with a 94% precision and generates reports online everyday. In September 2016, DeBot has accumulated about 544,868 unique bots in the previous one year. We compare our detection technique with per-user techniques and with Twitter's suspension system. We observe that some bots can avoid Twitter's suspension mechanism and remain active for months, and, more alarmingly, we show that DeBot detects bots at a rate higher than the rate Twitter is suspending them.

DeBot: Twitter Bot Detection via Warped Correlation

Advances in hardware technology have lead to an ability to collect data with the use of a variety of sensor technologies. In particular sensor notes have become cheaper and more efficient, and have even been integrated into day-to-day devices of use, such as mobile phones.This has lead to a much larger scale of applicability and mining of sensor data sets. The human-centric aspect of sensor data hascreated tremendous opportunities in integrating social aspects of sensor data collection into the mining process. Managing and Mining Sensor Data is a contributed volume by prominent leaders in this field, targeting advanced-level students in computer science as a secondary text book or reference. Practitioners and researchers working in this field will also find this book useful.

Managing and Mining Sensor Data

Data arriving in time order (a data stream) arises in fields including physics, finance, medicine, and music, to name a few. Often the data comes from sensors (in physics and medicine for example) whose data rates continue to improve dramatically as sensor technology improves. Further, the number of sensors is increasing, so correlating data between sensors becomes ever more critical in order to distill knowlege from the data. In many applications such as finance, recent correlations are of far more interest than long-term correlation, so correlation over sliding windows (windowed correlation) is the desired operation. Fast response is desirable in many applications (e.g., to aim a telescope at an activity of interest or to perform a stock trade). These three factors -- data size, windowed correlation, and fast response -- motivate this work.Previous work [10, 14] showed how to compute Pearson correlation using Fast Fourier Transforms and Wavelet transforms, but such techniques don't work for time series in which the energy is spread over many frequency components, thus resembling white noise. For such "uncooperative" time series, this paper shows how to combine several simple techniques -- sketches (random projections), convolution, structured random vectors, grid structures, and combinatorial design -- to achieve high performance windowed Pearson correlation over a variety of data sets.

/pdf/fast-window-correlations-over-uncooperative-time-series-51zu64yx2g.pdf

Fast window correlations over uncooperative time series

Financial time series streams are watched closely by millions of traders. What exactly do they look for and how can we help them do it faster? Physicists study the time series emerging from their sensors. The same question holds for them. Musicians produce time series. Consumers may want to compare them. This tutorial presents techniques and case studies for four problems:1. Finding sliding window correlations in financial, physical, and other applications.2. Discovering bursts in large sensor data of gamma rays.3. Matching hums to recorded music, even when people don't hum well.4. Maintaining and manipulating time-ordered data in a database setting.This tutorial draws mostly from the book High Performance Discovery in Time Series: techniques and case studies, Springer-Verlag 2004. You can find the power point slides for this tutorial at http://cs.nyu.edu/cs/faculty/shasha/papers/sigmod04.ppt.The tutorial is aimed at researchers in streams, data mining, and scientific computing. Its applications should interest anyone who works with scientists or financial "quants." The emphasis will be on recent results and open problems. This is a ripe area for further advance.

Fast algorithms for time series with applications to finance, physics, music, biology, and other suspects

You have a tune lingering in your head for many days, but you don’t know where you heard this tune or which song it is from. This demo will show a Query by Humming system that will tell you the name of that song. Most of the research in pre-existing query by humming systems uses pitch contour to match similar melodies (for example [1]). The user’s humming is transcribed to a sequence of discrete notes and the contour information is extracted from the notes. This contour information is represented by a few letters. For example, (“U”, “D”, “S”) represents that a note is above, below or the same as the previous one. The tunes in the databases are also represented by contour information. The edit distance can be use to measure the similarity between two melodies. Unfortunately, it is very hard to segment a user’s humming into discrete notes. Some recent work proposes to match the query directly from audio based on dynamic time warping to match the hum-query with the melodies in the music databases. But this quality improvement comes at a price because a brute-force search using DTW is very slow. The database community has been researching problems in similarity query for time series databases for many years. The techniques developed in the area might shed light on the query by humming problem. In this demo, we treat both the melodies in the music databases and the user humming input as time series. Such an approach allows us to integrate many database indexing techniques into a query by humming system, improving the quality of such system over the traditional (contour) string databases approach. We design special searching techniques that are invariant to shifting, time scaling and local time warping. This makes the system robust and allows more flexible user humming input.

Query by humming: in action with its technology revealed

A time series (or equivalently a data stream) consists of data arriving in time order. Single or multiple data streams arise in fields including physics, finance, medicine, and music, to name a few. Often the data comes from sensors (in physics and medicine for example) whose data rates continue to improve dramatically as sensor technology improves and as the number of sensors increases. So fast algorithms become ever more critical in order to distill knowledge from the data. This paper presents our recent work regarding the incremental computation of various primitives: windowed correlation, matching pursuit, sparse null space discovery and elastic burst detection. The incremental idea reflects the fact that recent data is more important than older data. Our StatStream system contains an implementation of these algorithms, permitting us to do empirical studies on both simulated and real data.

/pdf/incremental-methods-for-simple-problems-in-time-series-41qham1mer.pdf

Incremental methods for simple problems in time series: algorithms and experiments

Data arriving in time order (a data stream) arises in fields ranging from physics to finance to medicine to music, to name a few. Often the data comes from sensors (in physics and medicine for example) whose data rates continue to improve dramatically as sensor technology improves. Furthermore, the number of sensors is increasing, so analyzing data between sensors becomes ever more critical in order to distill knowledge from the data. Fast response is desirable in many applications (e.g. to aim a telescope at an activity of interest or to perform a stock trade). In applications such as finance, recent information, e.g. correlation, is of far more interest than older information, so analysis over sliding windows is a desired operation. 
These three factors---huge data size, fast response, and windowed computation---motivated this work. Our intent is to build a foundational library of primitives to perform online or near online statistical analysis, e.g. windowed correlation, incremental matching pursuit, burst detection, on thousands or even millions of time series. Beside the algorithms, we also propose the concept of "uncooperative" time series, whose power spectra are spread over all frequencies with any regularity. 
Previous work showed how to do windowed correlation with Fast Fourier Transforms and Wavelet Transforms, but such techniques don't work for uncooperative time series. This thesis will show how to use sketches (random projections) in a way that combines several simple techniques---sketches, convolution, structured random vectors, grid structures, combinatorial design, and bootstrapping---to achieve high performance, windowed correlation over a variety of data sets. Experiments confirm the asymptotic analysis. To conduct matching pursuit (MP) over time series windows, an incremental scheme is designed to reduce the computational effort. Our empirical study demonstrates a substantial improvement in speed. 
In previous work, Zhu and Shasha introduced an efficient algorithm to monitor bursts within windows of multiple sizes. We implemented it in a physical system by overcoming several practical challenges. Experimental results support the authors' linear running time analysis.

/pdf/high-performance-algorithms-for-multiple-streaming-time-4bh88nynm5.pdf

Xiaojian Zhao

Papers

Fast window correlations over uncooperative time series

Fast algorithms for time series with applications to finance, physics, music, biology, and other suspects

Query by humming: in action with its technology revealed

Incremental methods for simple problems in time series: algorithms and experiments

High performance algorithms for multiple streaming time series