The past decade has witnessed a rapid proliferation of video cameras in all walks of life and has resulted in a tremendous explosion of video content. Several applications such as content-based video annotation and retrieval, highlight extraction and video summarization require recognition of the activities occurring in the video. The analysis of human activities in videos is an area with increasingly important consequences from security and surveillance to entertainment and personal archiving. Several challenges at various levels of processing-robustness against errors in low-level processing, view and rate-invariant representations at midlevel processing and semantic representation of human activities at higher level processing-make this problem hard to solve. In this review paper, we present a comprehensive survey of efforts in the past couple of decades to address the problems of representation, recognition, and learning of human activities from video and related applications. We discuss the problem at two major levels of complexity: 1) "actions" and 2) "activities." "Actions" are characterized by simple motion patterns typically executed by a single human. "Activities" are more complex and involve coordinated actions among a small number of humans. We will discuss several approaches and classify them according to their ability to handle varying degrees of complexity as interpreted above. We begin with a discussion of approaches to model the simplest of action classes known as atomic or primitive actions that do not require sophisticated dynamical modeling. Then, methods to model actions with more complex dynamics are discussed. The discussion then leads naturally to methods for higher level representation of complex activities.

/pdf/machine-recognition-of-human-activities-a-survey-4jrl67n0jj.pdf

Machine Recognition of Human Activities: A Survey

Modal analysis is used extensively for understanding the dynamic behavior of structures. However, a major concern for structural dynamicists is that its validity is limited to linear structures. New developments have been proposed in order to examine nonlinear systems, among which the theory based on nonlinear normal modes is indubitably the most appealing. In this paper, a different approach is adopted, and proper orthogonal decomposition is considered. The modes extracted from the decomposition may serve two purposes, namely order reduction by projecting high-dimensional data into a lower-dimensional space and feature extraction by revealing relevant but unexpected structure hidden in the data. The utility of the method for dynamic characterization and order reduction of linear and nonlinear mechanical systems is demonstrated in this study.

The Method of Proper Orthogonal Decomposition for Dynamical Characterization and Order Reduction of Mechanical Systems: An Overview

In this paper, we present two novel algorithms to realize a finite dimensional, linear time-invariant state-space model from input-output data. The algorithms have a number of common features. They are classified as one of the subspace model identification schemes, in that a major part of the identification problem consists of calculating specially structured subspaces of spaces defined by the input-output data. This structure is then exploited in the calculation of a realization. Another common feature is their algorithmic organization: an RQ factorization followed by a singular value decomposition and the solution of an overdetermined set (or sets) of equations. The schemes assume that the underlying system has an output-error structure and that a measurable input sequence is available. The latter characteristic indicates that both schemes are versions of the MIMO Output-Error State Space model identification (MOESP) approach. The first algorithm is denoted in particular as the (elementary MOESP scheme)...

Subspace model identification-Part 1. The output-error state-space model identification class of algorithms

System theoretic approaches to action recognition model the dynamics of a scene with linear dynamical systems (LDSs) and perform classification using metrics on the space of LDSs, e.g. Binet-Cauchy kernels. However, such approaches are only applicable to time series data living in a Euclidean space, e.g. joint trajectories extracted from motion capture data or feature point trajectories extracted from video. Much of the success of recent object recognition techniques relies on the use of more complex feature descriptors, such as SIFT descriptors or HOG descriptors, which are essentially histograms. Since histograms live in a non-Euclidean space, we can no longer model their temporal evolution with LDSs, nor can we classify them using a metric for LDSs. In this paper, we propose to represent each frame of a video using a histogram of oriented optical flow (HOOF) and to recognize human actions by classifying HOOF time-series. For this purpose, we propose a generalization of the Binet-Cauchy kernels to nonlinear dynamical systems (NLDS) whose output lives in a non-Euclidean space, e.g. the space of histograms. This can be achieved by using kernels defined on the original non-Euclidean space, leading to a well-defined metric for NLDSs. We use these kernels for the classification of actions in video sequences using (HOOF) as the output of the NLDS. We evaluate our approach to recognition of human actions in several scenarios and achieve encouraging results.

/pdf/histograms-of-oriented-optical-flow-and-binet-cauchy-kernels-w8oz0byf0m.pdf

Histograms of oriented optical flow and Binet-Cauchy kernels on nonlinear dynamical systems for the recognition of human actions

In this paper, we examine image and video-based recognition applications where the underlying models have a special structure-the linear subspace structure. We discuss how commonly used parametric models for videos and image sets can be described using the unified framework of Grassmann and Stiefel manifolds. We first show that the parameters of linear dynamic models are finite-dimensional linear subspaces of appropriate dimensions. Unordered image sets as samples from a finite-dimensional linear subspace naturally fall under this framework. We show that an inference over subspaces can be naturally cast as an inference problem on the Grassmann manifold. To perform recognition using subspace-based models, we need tools from the Riemannian geometry of the Grassmann manifold. This involves a study of the geometric properties of the space, appropriate definitions of Riemannian metrics, and definition of geodesics. Further, we derive statistical modeling of inter and intraclass variations that respect the geometry of the space. We apply techniques such as intrinsic and extrinsic statistics to enable maximum-likelihood classification. We also provide algorithms for unsupervised clustering derived from the geometry of the manifold. Finally, we demonstrate the improved performance of these methods in a wide variety of vision applications such as activity recognition, video-based face recognition, object recognition from image sets, and activity-based video clustering.

/pdf/statistical-computations-on-grassmann-and-stiefel-manifolds-19mlvvn7t1.pdf

Statistical Computations on Grassmann and Stiefel Manifolds for Image and Video-Based Recognition

Subspace angles between ARMA models

In this paper we define a notion of subspace angles between two ARMA models as the principal angles between the column spaces generated by the observability matrices of the two models extended with the observability matrices of the inverse models. We show that these angles can be calculated via the observability Gramians of concatenations of the two given models and their inverses. Furthermore, we give their relation to different distance measures for ARMA models. Most attention is paid to a recently defined metric for ARMA models that is based on the cepstra of the models involved.

Subspace angles and distances between ARMA models

In this paper a methodology to cluster time series based on measurement data is described. In particular, we propose a distance for stochastic models based on the concept of sub- space angles within a model and between two models. This distance is used to obtain a clustering over the set of time series. We show how it is related to the mutual information of the past and the future output processes, and to a previously defined cepstral distance. Finally, the methodol- ogy is applied to the clustering of time series of power consumption within the Belgian electricity grid.

/pdf/clustering-time-series-subspace-identification-and-cepstral-5x7lny2e7q.pdf

Clustering Time Series, Subspace Identification and Cepstral Distances

Principal Angles in System Theory, Information Theory and Signal Processing

We give an alternative derivation for the expected value and variance of the number of molecules in a first order reversible reaction. Our derivation is based on the master equation and the corresponding generating function. Moreover, we show that from the generating function, the time dependent distribution function for the number of molecules can be obtained when all parameters, i.e., the initial amount of molecules and the reaction rates, are given.

Katrien De Cock

Papers

Subspace angles between ARMA models

Subspace angles and distances between ARMA models

Clustering Time Series, Subspace Identification and Cepstral Distances

Principal Angles in System Theory, Information Theory and Signal Processing

Comment on "Stiffness in stochastic chemically reacting systems: the implicit tau-leaping method" [J. Chem. Phys. 119, 12784 (2003)].