We focus on modeling human activities comprising multiple actions in a completely unsupervised setting. Our model learns the high-level action co-occurrence and temporal relations between the actions in the activity video. We consider the video as a sequence of short-term action clips, called action-words, and an activity is about a set of action-topics indicating which actions are present in the video. Then we propose a new probabilistic model relating the action-words and the action-topics. It allows us to model long-range action relations that commonly exist in the complex activity, which is challenging to capture in the previous works. We apply our model to unsupervised action segmentation and recognition, and also to a novel application that detects forgotten actions, which we call action patching. For evaluation, we also contribute a new challenging RGB-D activity video dataset recorded by the new Kinect v2, which contains several human daily activities as compositions of multiple actions interacted with different objects. The extensive experiments show the effectiveness of our model.

/pdf/watch-n-patch-unsupervised-understanding-of-actions-and-4hyqna9byw.pdf

Watch-n-patch: Unsupervised understanding of actions and relations

In this paper, we present an unsupervised method for mining activities in videos. From unlabeled video sequences of a scene, our method can automatically recover what are the recurrent temporal activity patterns (or motifs) and when they occur. Using non parametric Bayesian methods, we are able to automatically find both the underlying number of motifs and the number of motif occurrences in each document. The model's robustness is first validated on synthetic data. It is then applied on a large set of video data from state-of-the-art papers. We show that it can effectively recover temporal activities with high semantics for humans and strong temporal information. The model is also used for prediction where it is shown to be as efficient as other approaches. Although illustrated on video sequences, this model can be directly applied to various kinds of time series where multiple activities occur simultaneously.

/pdf/extracting-and-locating-temporal-motifs-in-video-scenes-2qs0rig4be.pdf

Extracting and locating temporal motifs in video scenes using a hierarchical non parametric Bayesian model

This paper introduces a novel probabilistic activity modeling approach that mines recurrent sequential patterns called motifs from documents given as word 
 $$\times $$
  time count matrices (e.g., videos). In this model, documents are represented as a mixture of sequential activity patterns (our motifs) where the mixing weights are defined by the motif starting time occurrences. The novelties are multi fold. First, unlike previous approaches where topics modeled only the co-occurrence of words at a given time instant, our motifs model the co-occurrence and temporal order in which the words occur within a temporal window. Second, unlike traditional Dynamic Bayesian networks (DBN), our model accounts for the important case where activities occur concurrently in the video (but not necessarily in synchrony), i.e., the advent of activity motifs can overlap. The learning of the motifs in these difficult situations is made possible thanks to the introduction of latent variables representing the activity starting times, enabling us to implicitly align the occurrences of the same pattern during the joint inference of the motifs and their starting times. As a third novelty, we propose a general method that favors the recovery of sparse distributions, a highly desirable property in many topic model applications, by adding simple regularization constraints on the searched distributions to the data likelihood optimization criteria. We substantiate our claims with experiments on synthetic data to demonstrate the algorithm behavior, and on four video datasets with significant variations in their activity content obtained from static cameras. We observe that using low-level motion features from videos, our algorithm is able to capture sequential patterns that implicitly represent typical trajectories of scene objects.

/pdf/a-sequential-topic-model-for-mining-recurrent-activities-2f8fpk2xbq.pdf

A Sequential Topic Model for Mining Recurrent Activities from Long Term Video Logs

In this paper, we present a new model for unsupervised discovery of recurrent temporal patterns (or motifs) in time series (or documents). The model is designed to handle the difficult case of multivariate time series obtained from a mixture of activities, that is, our observations are caused by the superposition of multiple phenomena occurring concurrently and with no synchronization. The model uses nonparametric Bayesian methods to describe both the motifs and their occurrences in documents. We derive an inference scheme to automatically and simultaneously recover the recurrent motifs (both their characteristics and number) and their occurrence instants in each document. The model is widely applicable and is illustrated on datasets coming from multiple modalities, mainly videos from static cameras and audio localization data. The rich semantic interpretation that the model offers can be leveraged in tasks such as event counting or for scene analysis. The approach is also used as a mean of doing soft camera calibration in a camera network. A thorough study of the model parameters is provided and a cross-platform implementation of the inference algorithm will be made publicly available.

/pdf/temporal-analysis-of-motif-mixtures-using-dirichlet-23galv4ks3.pdf

Temporal Analysis of Motif Mixtures Using Dirichlet Processes

This paper introduces a novel probabilistic activity modeling approach that mines recurrent sequential patterns from documents given as word-time occurrences. In this model, documents are represented as a mixture of sequential activity motifs (or topics) and their starting occurrences. The novelties are threefold. First, unlike previous approaches where topics only modeled the co-occurrence of words at a given time instant, our topics model the co-occurrence and temporal order in which the words occur within a temporal window. Second, our model accounts for the important case where activities occur concurrently in the document. And third, our method explicitly models with latent variables the starting time of the activities within the documents, enabling to implicitly align the occurrences of the same pattern during the joint inference of the temporal topics and their starting times. The model and its robustness to the presence of noise have been validated on synthetic data. Its effectiveness is also illustrated in video activity analysis from low-level motion features, where the discovered topics capture frequent patterns that implicitly represent typical trajectories of scene objects.

/pdf/probabilistic-latent-sequential-motifs-discovering-temporal-47ol3nfzjv.pdf

Probabilistic Latent Sequential Motifs: Discovering temporal activity patterns in video scenes

In this paper we address the problem of time based activity inference in unsupervised manner for an area under surveillance. We use a Latent Dirichlet Allocation based model that captures the activities and how they change over time. We use agglomerative clustering on optical flow vectors to code direction and spatial information. In this model each activity is associated with not only a mixture distribution over these cluster occurrences but also on the distribution over timestamps of their occurrences. Our method thus helps in determining the prominence and the correlation of activities over a period of time.

/pdf/time-based-activity-inference-using-latent-dirichlet-4q42acs5y5.pdf

Time based Activity Inference using Latent Dirichlet Allocation

This paper addresses the problem of discovering activities and their temporal significance in surveillance videos in an unsupervised manner. We propose a generative model that can jointly capture the activities and their behaviour over time. We use multinomial distribution over local motion features to model activities and a mixture distribution over their time stamps to capture the multi-modal temporal distribution of these activities. We give a Gibbs sampling algorithm to infer the parameters of the model. We demonstrate the effectiveness of our approach on real life surveillance feed of outdoor scenes.

Unsupervised Discovery of Activities and Their Temporal Behaviour

In this paper, we address the problem of discovering activities and their temporal significance in an area under surveillance. Discovering activities along with its expectation of occurrence at a particular time plays an important role in many surveillance applications. We propose an unsupervised model, called Time pLSA model, that extends the probabilistic Latent Semantic Analysis (pLSA) model to jointly capture the activities and their behaviour over time. We use adaptive background subtraction to detect spatio-temporal patches, which are used as feature representation for activity patterns. Each of these patches are associated with the time slot in which they occur. Multinomial distributions are used to model both activities as distribution over spatio-temporal patches and time significance as distribution over the time-line. We demonstrate the effectiveness of our approach on a real life surveillance feed of an outdoor scene.

Discovering Activities and Their Temporal Significance

Surveillance videos of public places often consist of group activities composed from multiple co-occurring individual activities. However, latent topic models, such as Latent Dirichlet Allocation (LDA), which have been successfully used to discover individual activities, do not discover group activities. In this paper we propose a method to discover group activities along with individual activities. We use a two layer latent structure where a latent variable is used to discover correlation of individual activities as a group activity using multinomial distribution. Each individual activity is in turn represented as a distribution over local visual features. We use a Gibbs sampling-based algorithm to jointly infer the individual and group activities. Our method can summarize not only the individual activities but also the common group activities in a video. We demonstrate the strength of our method by discovering activities and the salient correlation amongst them in real life videos of crowded public places.

Tanveer A. Faruquie

Papers

Time based Activity Inference using Latent Dirichlet Allocation

Unsupervised Discovery of Activities and Their Temporal Behaviour

Discovering Activities and Their Temporal Significance

Latent topic model-based group activity discovery