Convolutional neural networks have enabled accurate image super-resolution in real-time. However, recent attempts to benefit from temporal correlations in video super-resolution have been limited to naive or inefficient architectures. In this paper, we introduce spatio-temporal sub-pixel convolution networks that effectively exploit temporal redundancies and improve reconstruction accuracy while maintaining real-time speed. Specifically, we discuss the use of early fusion, slow fusion and 3D convolutions for the joint processing of multiple consecutive video frames. We also propose a novel joint motion compensation and video super-resolution algorithm that is orders of magnitude more efficient than competing methods, relying on a fast multi-resolution spatial transformer module that is end-to-end trainable. These contributions provide both higher accuracy and temporally more consistent videos, which we confirm qualitatively and quantitatively. Relative to single-frame models, spatio-temporal networks can either reduce the computational cost by 30% whilst maintaining the same quality or provide a 0.2dB gain for a similar computational cost. Results on publicly available datasets demonstrate that the proposed algorithms surpass current state-of-the-art performance in both accuracy and efficiency.

/pdf/real-time-video-super-resolution-with-spatio-temporal-1qw8m7lifm.pdf

Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation

The invention discloses an improved air handwriting action recognition method of a convolution neural network. The method extracts features by using CNN of a multi-time series and a partial weight sharing technology, and solves the shortcomings of the traditional method that features need to be designed manually and the like. The method mainly includes the following contents: 1, preprocessing thesensor data and extracting the motion segments; 2, extracting motion features by using the improved CNN, improvements include the introduction of multiple time series as input, In the convolution layer, partial weight sharing technique is used to solve the problem of decreasing accuracy caused by the weakening of local differences caused by traditional feature extraction, which makes the featuresmore representative. Combining with partial weight sharing in the pooling layer and back propagation, corresponding improvements are made, so that the network can be correctly trained. The invention improves the recognition accuracy of the air handwriting action through the improved CNN, can be used in a pattern recognition system, and is used as a technical support for the recognition of the handaction based on the intelligent wristband.

An improved CNN-based aerial handwritten motion recognition

The invention discloses a human-human interaction behavior identification method based on a lightweight convolutional neural network. The implementation steps are as follows: (1) inputting human skeleton data to be identified; (2) constructing a two-person interaction behavior image set; (3) constructing a lightweight convolutional neural network; (4) training the lightweight convolutional neuralnetwork; and (5) testing the lightweight convolutional neural network. According to the method, the two-person interaction behavior image set is constructed, the light-weight convolutional neural network is constructed, the problems that in the prior art, only one person behavior is recognized, and the parameter quantity of a recognition model is large in the prior art are solved, and accurate recognition of the human-person interaction behavior can be achieved with extremely small parameter quantity.

human-human interaction behavior identification method based on a lightweight convolutional neural network

The invention provides a neural network circuit and a self-circulation multi-stage iteration method thereof. The neural network circuit comprises an image reduction unit, a sensitive area image data reading unit, a multi-channel selection unit, a configurable neural network circuit unit, a neural network structure configuration unit, a sensitive area coordinate information storage unit and a coordinate restoration unit, wherein firstly the image reduction unit reduces an original image, a sensitive object area in the image is segmented by the configurable neural network circuit unit, the coordinate restoration unit restores the sensitive object area into the original image, and then the sensitive area image data reading unit performs local reading; and then the read data is sent to the configurable neural network circuit unit for classification and recognition. The recognition process is performed by adopting a self-circulation multi-stage iteration mode, and a continuously refined recognition result is obtained through going deep for multiple turns.

Neural network circuit and self-circulation multi-stage iteration method thereof

Automatically training an actor upon the occurrence of a physical condition with respect to that actor. Upon detecting that the actor has the physical condition (e.g., is engaging in or is about to engage in a physical activity), the system determines that training is to be provided for that activity. Upon determining that training is to be provided, the system automatically dispatches training. For instance, the system might cause a human or robot to be dispatched to the actor to show the actor how to perform the activity. Alternatively or instead, a representation of a signal segment may be dispatched to the actor. The representation providing the training to the actor may include a similar target of work to what the actor is presently targeting by the activity. The representation may also include a representation of a person that engaged in the activity properly previously.

Automated activity-time training

Training data from multiple types of sensors and captured in previous capture sessions can be fused within a physics-based tracking framework to train motion priors using different deep learning techniques, such as convolutional neural networks (CNN) and Recurrent Temporal Restricted Boltzmann Machines (RTRBMs). In embodiments employing one or more CNNs, two streams of filters can be used. In those embodiments, one stream of the filters can be used to learn the temporal information and the other stream of the filters can be used to learn spatial information. In embodiments employing one or more RTRBMs, all visible nodes of the RTRBMs can be clamped with values obtained from the training data or data synthesized from the training data. In cases where sensor data is unavailable, the input nodes may be unclamped and the one or more RTRBMs can generate the missing sensor data.

Deep-learning motion priors for full-body performance capture in real-time

Presented herein are systems and methods configured to generate virtual entities representing real-world users. In some implementations, the systems and/or methods are configured to capture user appearance information with imaging devices and sensors, determines correspondence values conveying correspondences between the appearance of the user's body or user's head and individual ones of default body models and/or default head models, modifies a set of values defining a base body model and/or base head model based on determined correspondence values and sets of base values defining the default body models and/or default head models. The base body model and/or base head model may be modified to model the appearance of the body and/or head of the user.

Ivan Huerta Casado

Papers

Deep-learning motion priors for full-body performance capture in real-time

Single shot capture to animated vr avatar