Visual surveillance in dynamic scenes, especially for humans and vehicles, is currently one of the most active research topics in computer vision. It has a wide spectrum of promising applications, including access control in special areas, human identification at a distance, crowd flux statistics and congestion analysis, detection of anomalous behaviors, and interactive surveillance using multiple cameras, etc. In general, the processing framework of visual surveillance in dynamic scenes includes the following stages: modeling of environments, detection of motion, classification of moving objects, tracking, understanding and description of behaviors, human identification, and fusion of data from multiple cameras. We review recent developments and general strategies of all these stages. Finally, we analyze possible research directions, e.g., occlusion handling, a combination of twoand three-dimensional tracking, a combination of motion analysis and biometrics, anomaly detection and behavior prediction, content-based retrieval of surveillance videos, behavior understanding and natural language description, fusion of information from multiple sensors, and remote surveillance.

/pdf/a-survey-on-visual-surveillance-of-object-motion-and-302zsaqo09.pdf

A survey on visual surveillance of object motion and behaviors

Our experience of the world is multimodal - we see objects, hear sounds, feel texture, smell odors, and taste flavors Modality refers to the way in which something happens or is experienced and a research problem is characterized as multimodal when it includes multiple such modalities In order for Artificial Intelligence to make progress in understanding the world around us, it needs to be able to interpret such multimodal signals together Multimodal machine learning aims to build models that can process and relate information from multiple modalities It is a vibrant multi-disciplinary field of increasing importance and with extraordinary potential Instead of focusing on specific multimodal applications, this paper surveys the recent advances in multimodal machine learning itself and presents them in a common taxonomy We go beyond the typical early and late fusion categorization and identify broader challenges that are faced by multimodal machine learning, namely: representation, translation, alignment, fusion, and co-learning This new taxonomy will enable researchers to better understand the state of the field and identify directions for future research

Multimodal Machine Learning: A Survey and Taxonomy

Recent progress in using recurrent neural networks (RNNs) for image description has motivated the exploration of their application for video description. However, while images are static, working with videos requires modeling their dynamic temporal structure and then properly integrating that information into a natural language description model. In this context, we propose an approach that successfully takes into account both the local and global temporal structure of videos to produce descriptions. First, our approach incorporates a spatial temporal 3-D convolutional neural network (3-D CNN) representation of the short temporal dynamics. The 3-D CNN representation is trained on video action recognition tasks, so as to produce a representation that is tuned to human motion and behavior. Second we propose a temporal attention mechanism that allows to go beyond local temporal modeling and learns to automatically select the most relevant temporal segments given the text-generating RNN. Our approach exceeds the current state-of-art for both BLEU and METEOR metrics on the Youtube2Text dataset. We also present results on a new, larger and more challenging dataset of paired video and natural language descriptions.

/pdf/describing-videos-by-exploiting-temporal-structure-4og87ailep.pdf

Describing Videos by Exploiting Temporal Structure

Recent developments in human motion analysis

https://hal.inria.fr/inria-00459653/document

A survey of vision-based methods for action representation, segmentation and recognition

We propose a method for describing human activities from video images based on concept hierarchies of actions. Major difficulty in transforming video images into textual descriptions is how to bridge a semantic gap between them, which is also known as inverse Hollywood problem. In general, the concepts of events or actions of human can be classified by semantic primitives. By associating these concepts with the semantic features extracted from video images, appropriate syntactic components such as verbs, objects, etc. are determined and then translated into natural language sentences. We also demonstrate the performance of the proposed method by several experiments.

/pdf/natural-language-description-of-human-activities-from-video-3w2udgafo5.pdf

Natural Language Description of Human Activities from Video Images Based on Concept Hierarchy of Actions

In visual surveillance applications, it is becoming popular to perceive video images and to interpret them using natural language concepts. We propose an approach to generating a natural language description of human behavior appearing in real video images. First, a head region of a human, on behalf of the whole body, is extracted from each frame. Using a model based method, three dimensional pose and position of the head are estimated. Next, the trajectory of these parameters is divided into segments of monotonous motions. For each segment, we evaluate conceptual features such as degree of change of pose and position and that of relative distance to some objects in the surroundings, and so on. By calculating the product of these feature values, a most suitable verb is selected and other syntactic elements are supplied. Finally natural language text is generated using a technique of machine translation.

Generating natural language description of human behavior from video images

We propose a method for segmentation of written regions on a blackboard in the lecture room using a video image. We first detect the static edges of which locations on the image are stationary. Next, we extract several rectangular regions in which these static edges are located densely. Finally, by use of fuzzy rules, the extracted rectangles are merges as contextual regions, where letters and figures in each contextual region explain each context. We apply our method to automatic production of lecture video, and archives segmentation of written regions on the blackboard in lecture rooms.

/pdf/blackboard-segmentation-using-video-image-of-lecture-and-its-3pvjsu0kwy.pdf

Blackboard segmentation using video image of lecture and its applications

Recently, attention is paid to monitoring systems that watch behavior of a solitude person in home as the population of solitude seniors has increased. This paper proposes a monitoring system focused on behavioral patterns of a solitude senior. In general, the number of behavioral patterns are limited in the case of aged person in a room. It is possible for a system like a HMM to learn the frequently appeared behavioral patterns. By utilizing HMMs to deal with behavioral patterns, the system can detect irregular pattern by the degree of likelihood in the case when non-daily behavioral patterns appeared. Such a case could be an irregular state of the living life of the person. Soon after, the system reports an alarm of occurrence of the irregular state according to demand.

Detection of a Solitude Senior's Irregular States Based on Learning and Recognizing of Behavioral Patterns

It is generally considered that human behavior includes both regularities and habits. In this paper, the regularities and habits of behavior are called the behavioral pattern, and we wish to learn and recognize them. The conventional approaches considered behavioral patterns but used only infrared sensors or information about whether electrical appliances were on or off. Thus, it was difficult to recognize in detail how a person was performing motions in the room. In order to realize a procedure for the detailed recognition of motion in ordinary environments, on the other hand, a large number of models must be prepared beforehand. To deal with this problem, this paper proposes the following technique. Motions conducted in the learning period are automatically classified and individual models are constructed. Then, motions can be recognized in detail without preparing a large number of models, and behavioral patterns can be recognized by considering the sequence of motions. In experiments, human motions and behavioral patterns in an indoor environment were learned and recognized, and the effectiveness of the method was demonstrated. © 2005 Wiley Periodicals, Inc. Syst Comp Jpn, 36(13): 45–56, 2005; Published online in Wiley InterScience (). DOI 10.1002sscj.20293

/pdf/learning-and-recognizing-behavioral-patterns-using-position-4kcp18sfag.pdf

Kunio Fukunaga

Papers

Natural Language Description of Human Activities from Video Images Based on Concept Hierarchy of Actions

Generating natural language description of human behavior from video images

Blackboard segmentation using video image of lecture and its applications

Detection of a Solitude Senior's Irregular States Based on Learning and Recognizing of Behavioral Patterns

Learning and recognizing behavioral patterns using position and posture of human body and its application to detection of irregular states