This paper presents a multimodal emotion recognition system, which is based on the analysis of audio and visual cues. From the audio channel, Mel-Frequency Cepstral Coefficients, Filter Bank Energies and prosodic features are extracted. For the visual part, two strategies are considered. First, facial landmarks’ geometric relations, i.e., distances and angles, are computed. Second, we summarize each emotional video into a reduced set of key-frames, which are taught to visually discriminate between the emotions. In order to do so, a convolutional neural network is applied to key-frames summarizing videos. Finally, confidence outputs of all the classifiers from all the modalities are used to define a new feature space to be learned for final emotion label prediction, in a late fusion/stacking fashion. The experiments conducted on the SAVEE, eNTERFACE’05, and RML databases show significant performance improvements by our proposed system in comparison to current alternatives, defining the current state-of-the-art in all three databases.

Audio-Visual Emotion Recognition in Video Clips

An efficient feature detection algorithm and image classification is a very crucial task in computer vision system. There are various state-of-the-art feature detectors and descriptors available for an object recognition task. In this paper, the authors have compared the performance of Shi-Tomasi corner detector with SIFT and SURF feature descriptors and evaluate the performance of Shi-Tomasi in combination with SIFT and SURF feature descriptors. To make the computations faster, authors have reduced the size of features computed in all cases by applying locality preserving projection methodology. Features extracted using these algorithms are further classified with various classifiers like K-NN, decision tree and random forest. For experimental work, a public dataset, namely Caltech-101 image dataset, is considered in this paper. This dataset comprises of 101 object classes. These classes have further contained many images. Using a combination of Shi-Tomasi, SIFT and SURF features, the authors have achieved a recognition accuracy of 85.9%, 80.8% and 74.8% with random forest, decision tree and K-NN classifier, respectively. In this paper, the authors have also computed true positive rate, false positive rate and area under curve in all cases. Finally, the authors have applied the adaptive boosting methodology to improve the recognition accuracy. Authors have reported improved recognition accuracy of 86.4% using adaptive boosting with random forest classifier and a combination of Shi-Tomasi, SIFT and SURF features.

An efficient technique for object recognition using Shi-Tomasi corner detection algorithm

This dataset supports the article entitled "Energy-Efficient Run-time Mapping and Thread Partitioning of Concurrent OpenCL Applications on CPU-GPU MPSoCs" accepted for publication in ACM Transactions on Embedded Computing Systems, Special Issue on ACM/IEEE International Conference on Hardware/Software Codesign and System Synthesis, October 2017.

Dataset supporting the article entitled "Energy-Efficient Run-time Mapping and Thread Partitioning of Concurrent OpenCL Applications on CPU-GPU MPSoCs"

Internet of Things (IoT) is playing an increasingly important role in Intelligent Transportation Systems (ITS) for real-time sensing and communication. In ITS, vehicle types, volume and speeds provide important information for road traffic management. However, the present methods for on-road traffic monitoring are lacking in providing cost-effective means to meet the demands. In this paper, we propose MagMonitor, a novel method for on-road traffic surveillance through a single small and easy-to-install magnetic sensor. The developed magnetic sensor system is wireless-connected, cost-effective, and environmental-friendly. First, a magnetic model of a moving vehicle is presented. The model employs multiple magnetic dipoles for modelling moving vehicle and varies depending on the on-road vehicle types. Through modelling of local magnetic field perturbations caused by moving vehicles, we extract the characteristics of magnetic waveforms for vehicle identification and speed estimation. The proposed model and estimation technique are validated with real field experimental data. Furthermore, we analyze and compare the performance of the proposed estimation technique with other speed estimation algorithms, which shows the superior accuracy of the proposed technique.

MagMonitor: Vehicle Speed Estimation and Vehicle Classification Through A Magnetic Sensor

https://hal-mines-paristech.archives-ouvertes.fr/hal-01689709/document

Harris corner detection on a NUMA manycore

Fast and accurate detection of vehicles on road traffic scenes captured by traffic surveillance cameras, is essential for large-scale deployment of automated traffic surveillance systems. The state-of-the-art techniques typically employ background modeling for low-complexity foreground detection. However, this is a challenging problem as these methods need to be robust to varying road scene conditions (such as illumination changes, camera jitter, stationary vehicles, and heavy traffic) leading to huge computation cost. In this paper, we propose a highly accurate yet low-complexity foreground (i.e., vehicle) detection technique, which can effectively deal with the varying road scene conditions, and generate accurate pixel-level foreground masks in real-time. We propose a novel robust block-based feature suitable for modeling road background and detecting vehicles as foreground, and employ Bayesian probabilistic modeling on these features. The experimental evaluations on widely used traffic datasets demonstrate that the proposed method can achieve comparable accuracy to the existing state-of-the-art techniques but at a much higher processing frame rate (40x speedup over PAWCS). The real-time performance of the proposed system has also been demonstrated by implementing it on a low-cost embedded platform, Odroid XU-4, that still achieves a frame rate of over 80 frames/s, thereby enabling the real-time detection of foreground objects in road scenes.

Rapid and Robust Background Modeling Technique for Low-Cost Road Traffic Surveillance Systems

Widely-used corner detectors such as Shi-Tomasi and Harris necessitate the selection of a threshold parameter manually in order to identify good quality corners. The recent attempts based on trial-and-error methods for threshold setting are time-consuming, making them unsuitable for low-cost and embedded video processing applications. In this paper we propose a novel automated thresholding technique for Shi-Tomasi and Harris corner detectors based on an iterative pruning strategy. The proposed pruning strategy involves the rapid extraction of potential corner regions and their evaluation for detecting corners. This pruning strategy is applied iteratively until the required number of corners is identified without necessitating the selection of the threshold parameter. As the complex corner measure computations of the Shi-Tomasi and Harris detectors are only applied to very small regions selected by the proposed pruning method, significant savings in computation is also achieved. In addition, the pruning strategy is computationally simpler, making it suitable for deployment in low cost and embedded applications. Our evaluations on the NiOS-II embedded platform show that the proposed automated thresholding technique is able to achieve an average speedup of 67% in Shi-Tomasi and 51% in Harris, with almost no loss in accuracy. The proposed method to identify corners without the manual selection of a threshold parameter makes it ideal for corner detection on a wide range of imagery where the quantity and quality of corners is not known a priori such as in video processing applications.

Automated thresholding for low-complexity corner detection

In this paper, we present a novel and computationally efficient pruning technique to speed up the Shi-Tomasi and Harris corner detectors. The proposed technique quickly prunes non-corners and selects a small corner candidate set by approximating the complex corner measure of Shi-Tomasi and Harris. The actual corner measure is then applied only to the reduced candidate set. Experimental results on the NiOS-II platform show that the proposed technique achieves an average execution time savings of 90% for Shi-Tomasi and 70% for Harris detectors for 500 corners with no loss in accuracy.

Low-complexity pruning for accelerating corner detection

Low-complexity corner detection is essential for many real-time computer vision applications that need to be executed on low-cost/low-power embedded platforms such as robots. The widely used Shi---Tomasi and Harris corner detectors become prohibitive in such platforms due to their high computational complexity, which is attributed to the need to apply a complex corner measure on the entire image. In this paper, we introduce a novel and computationally efficient technique to accelerate the Shi---Tomasi and Harris corner detectors. The proposed technique consists of two steps. In the first step, the complex corner measure is replaced with simple approximations to quickly prune away non-corners. In the second step, the complex corner measure is applied to a small corner candidate set obtained after pruning. Evaluations using standard image benchmarks show that the proposed pruning technique achieves up to 75 % speedup on the Nios-II platform, while yielding corners with comparable or better accuracy than the conventional Shi---Tomasi and Harris detectors.

Enhanced low-complexity pruning for corner detection

The Kanade-Lucas-Tomasi tracking KLT algorithm is widely used for local tracking of features. As it employs a translation model to find the feature tracks, KLT is not robust in the presence of distortions around the feature resulting in high inaccuracies in the tracks. In this paper we show that the window size in KLT must vary to adapt to the presence of distortions around each feature point in order to increase the number of useful tracks and minimize noisy ones. We propose an adaptive window size strategy for KLT that uses the KLT iterations as an indicator of the quality of the tracks to determine near-optimal window sizes, thereby significantly improving its robustness to distortions. Our evaluations with a well-known tracking dataset show that the proposed adaptive strategy outperforms the conventional fixed-window KLT in terms of robustness. In addition, compared to the well-known affine KLT, our method achieves comparable robustness at an average runtime speedup of 7x.

Nirmala Ramakrishnan

Papers

Rapid and Robust Background Modeling Technique for Low-Cost Road Traffic Surveillance Systems

Automated thresholding for low-complexity corner detection

Low-complexity pruning for accelerating corner detection

Enhanced low-complexity pruning for corner detection

Adaptive Window Strategy for High-Speed and Robust KLT Feature Tracker