The discovery of complex patterns such as clusters, outliers, and associations from huge volumes of streaming data has been recognized as critical for many domains. However, pattern detection with sliding window semantics, as required by applications ranging from stock market analysis to moving object tracking remains largely unexplored. Applying static pattern detection algorithms from scratch to every window is prohibitively expensive due to their high algorithmic complexity. This work tackles this problem by developing the first solution for incremental detection of neighbor-based patterns specific to sliding window scenarios. The specific pattern types covered in this work include density-based clusters and distance-based outliers. Incremental pattern computation in highly dynamic streaming environments is challenging, because purging a large amount of to-be-expired data from previously formed patterns may cause complex pattern changes including migration, splitting, merging and termination of these patterns. Previous incremental neighbor-based pattern detection algorithms, which were typically not designed to handle sliding windows, such as incremental DBSCAN, are not able to solve this problem efficiently in terms of both CPU and memory consumption. To overcome this, we exploit the "predictability" property of sliding windows to elegantly discount the effect of expiring objects on the remaining pattern structures. Our solution achieves minimal CPU utilization, while still keeping the memory utilization linear in the number of objects in the window. Our comprehensive experimental study, using both synthetic as well as real data from domains of stock trades and moving object monitoring, demonstrates superiority of our proposed strategies over alternate methods in both CPU and memory utilization.

/pdf/neighbor-based-pattern-detection-for-windows-over-streaming-70nz793xt2.pdf

Neighbor-based pattern detection for windows over streaming data

Recently, significant progress has been made in the development of physics-based, knowledge-aided (KA) signal processing strategies supported by improvements in real-time embedded computing architectures. These developments provide designers of advanced sensor systems an unprecedented degree of flexibility when implementing next generation adaptive sensor systems. In the case of radar, this has been manifested in the first ever, real-time, KA space-time adaptive processing (KA-STAP) system for advanced clutter/interference suppression. This paper provides exemplars of real-world effects giving rise to the need for "intelligent" adaptation schemes and overviews the KA approach to sensor signal processing in some detail. Moreover, we survey a collection of papers describing recent KA sensor research that follow in this issue

Knowledge-aided signal processing: a new paradigm for radar and other advanced sensors

This letter presents a novel approach for extracting moving targets in a multichannel wide-area surveillance radar system. In the algorithm, after proper preprocessing and matrix combination, the combined matrix of radar echoes can be regarded as the superposition of three matrices, namely, a low-rank matrix of ground clutter, a sparse matrix of moving targets, and an entry-wise matrix of noise component. Then, the recently proposed relaxed version of principal component pursuit is used to realize ground clutter (low-rank matrix) and moving target (sparse matrix) separation under the influence of entry-wise noise. Both simulation and real data processing results are provided to demonstrate the effectiveness of the proposed method. In addition, the results show the advantage of the proposed method in a nonhomogeneous environment when compared with a reduced-dimension space-time adaptive processing method.

Ground Moving Target Extraction in a Multichannel Wide-Area Surveillance SAR/GMTI System via the Relaxed PCP

The detection of abnormal moving objects over high-volume trajectory streams is critical for real time applications ranging from military surveillance to transportation management. Yet this problem remains largely unexplored. In this work, we first propose classes of novel trajectory outlier definitions that model the anomalous behavior of moving objects for a large range of real time applications. Our theoretical analysis and empirical study on the Beijing Taxi and GMTI (Ground Moving Target Indicator) datasets demonstrate its effectiveness in capturing abnormal moving objects. Furthermore we propose a general strategy for efficiently detecting the new outlier classes. It features three fundamental optimization principles designed to minimize the detection costs. Our comprehensive experimental studies demonstrate that our proposed strategy drives the detection costs 100-fold down into practical realm for applications producing high volume trajectory streams to utilize.

Detecting moving object outliers in massive-scale trajectory streams

Out-of-sequence measurements (OOSMS) arise in a multi-sensor central-tracking system due to communication network delays and varying preprocessing times at the sensor platforms. During the last few years a great deal of research has focussed attention on the OOSM filtering problem. However, research in the multi-sensor multi-target OOSM tracking involving data association, filtering, and hypothesis management is still lacking. Some previous efforts have used buffering and measurement reprocessing to handle the OOSMs. In this paper, we present single-model multiple-lag OOSM algorithms for data association, likelihood computation, and hypothesis management for a dwell-based multi-sensor multi-target multi-hypothesis tracking (MHT) system that handles missed detections and clutter. We present numerical results using simulated multi-sensor ground moving target indicator (GMTI) radar measurements.

/pdf/multi-sensor-multi-target-tracking-using-out-of-sequence-5fe237wziu.pdf

Multi-sensor multi-target tracking using out-of-sequence measurements

The concept of airborne surveillance of enemy ground forces with a Ground Moving Target Indicator [now called GMTI] radar capable of detecting moving ground vehicles and helicopters was proposed in 1968 and resulted in a DoD program to realize its potential. This article traces the program history starting with the Army's Stand Off Target Acquisition System [SOTAS] as it evolved through the Small Aerostat Surveillance System [SASS] and the Assault Breaker/Pave Mover programs into the currently fielded Joint Surveillance and Target Attack System [JointSTARS], which, in prototype form, more than proved its worth in the 1991 Gulf War. New developments and trends in GMTI radars are also discussed together with other potential platforms.

JointSTARS and GMTI: past, present and future

It has been brought to the attention of the authors that Fig. 9 in the above-named article [ibid., 35, 2 (Apr. 1999), 748?761] was incorrect. They had intended to show the Hughes Pave Mover Antenna System but inadvertently showed the Grumman Antenna System. The Hughes Antenna System is shown here.

J.N. Entzminger

Papers

JointSTARS and GMTI: past, present and future

Errata: incorrect figure in "JointStars and GMTI: Past, present and future"