In this work, we develop a new motion-compe (MC) interpolation algorithm to enhance the temporal resolution of video sequences. First, we propose the bilateral motion estimation scheme to obtain the motion field of an interpolated frame without yielding the hole and overlapping problems. Then, we partition a frame into several object regions by clustering motion vectors. We apply the variable-size block MC (VS-BMC) algorithm to object boundaries in order to reconstruct edge information with a higher quality. Finally, we use the adaptive overlapped block MC (OBMC), which adjusts the coefficients of overlapped windows based on the reliabilities of neighboring motion vectors. The adaptive OBMC (AOBMC) can overcome the limitations of the conventional OBMC, such as over-smoothing and poor de-blocking. Experimental results show that the proposed algorithm provides a better image quality than conventional methods both objectively and subjectively

Motion-Compensated Frame Interpolation Using Bilateral Motion Estimation and Adaptive Overlapped Block Motion Compensation

We propose a new frame rate up-conversion algorithm for high quality video. In the proposed scheme, bi-directional motion estimation (ME) is performed to construct the motion vector (MV) field for the frame to be interpolated. Unlike conventional motion-compensated interpolation (MCI) algorithms, the proposed technique does not produce any overlapped pixel and hole region in the interpolated frame, and thus can utilize the overlapped block motion compensation technique to reduce the blocking artifacts. The proposed algorithm is very simple to implement on consumer products when compared to conventional MCI methods. Computer simulation shows a high visual performance of the proposed frame rate up-conversion algorithm.

New frame rate up-conversion using bi-directional motion estimation

In a method of estimating motion, at least two motion parameter sets are generated (PE1-PEn) from input video data (n, n-1), a motion parameter set being a set of parameters describing motion in an image, by means of which motion parameter set motion vectors can be calculated. One motion parameter set indicates a zero velocity for all image parts in an image, and each motion parameter set has corresponding local match errors. Output motion data are determined from the input video data (n, n-1) in dependence on the at least two motion parameter sets, wherein the importance of each motion parameter set in calculating the output motion data depends on the motion parameter sets' local match errors.

Motion estimation and motion-compensated interpolation

Motion estimation (ME) and motion compensation (MC) have been widely used for classical video frame interpolation systems over the past decades. Recently, a number of data-driven frame interpolation methods based on convolutional neural networks have been proposed. However, existing learning based methods typically estimate either flow or compensation kernels, thereby limiting performance on both computational efficiency and interpolation accuracy. In this work, we propose a motion estimation and compensation driven neural network for video frame interpolation. A novel adaptive warping layer is developed to integrate both optical flow and interpolation kernels to synthesize target frame pixels. This layer is fully differentiable such that both the flow and kernel estimation networks can be optimized jointly. The proposed model benefits from the advantages of motion estimation and compensation methods without using hand-crafted features. Compared to existing methods, our approach is computationally efficient and able to generate more visually appealing results. Furthermore, the proposed MEMC-Net can be seamlessly adapted to several video enhancement tasks, e.g., super-resolution, denoising, and deblocking. Extensive quantitative and qualitative evaluations demonstrate that the proposed method performs favorably against the state-of-the-art video frame interpolation and enhancement algorithms on a wide range of datasets.

MEMC-Net: Motion Estimation and Motion Compensation Driven Neural Network for Video Interpolation and Enhancement

Motion estimation (ME) and motion compensation (MC) have been widely used for classical video frame interpolation systems over the past decades. Recently, a number of data-driven frame interpolation methods based on convolutional neural networks have been proposed. However, existing learning based methods typically estimate either flow or compensation kernels, thereby limiting performance on both computational efficiency and interpolation accuracy. In this work, we propose a motion estimation and compensation driven neural network for video frame interpolation. A novel adaptive warping layer is developed to integrate both optical flow and interpolation kernels to synthesize target frame pixels. This layer is fully differentiable such that both the flow and kernel estimation networks can be optimized jointly. The proposed model benefits from the advantages of motion estimation and compensation methods without using hand-crafted features. Compared to existing methods, our approach is computationally efficient and able to generate more visually appealing results. Furthermore, the proposed MEMC-Net architecture can be seamlessly adapted to several video enhancement tasks, e.g., super-resolution, denoising, and deblocking. Extensive quantitative and qualitative evaluations demonstrate that the proposed method performs favorably against the state-of-the-art video frame interpolation and enhancement algorithms on a wide range of datasets.

A new recursive block-matching motion estimation algorithm with only eight candidate vectors per block is presented. A fast convergence and a high accuracy, also in the vicinity of discontinuities in the velocity plane, was realized with such new techniques as bidirectional convergence and convergence accelerators. A new search strategy, asynchronous cyclic search, which allows a highly efficient implementation, is presented. A new block erosion postprocessing proposal further effectively eliminates block structures from the generated vector field. Measured with criteria relevant for the field rate conversion application, the new motion estimator is shown to have a superior performance over alternative algorithms, while its complexity is significantly less. >

True-motion estimation with 3-D recursive search block matching

An IC for 100-Hz television has been realized implementing motion estimation and compensation algorithms for high-quality upconversion and a judder-free motion portrayal of movie material. The four embedded video signal processors on this IC provide a processing power of 10 giga-operation per second (GOPS). Their architecture was generated from an algorithm description using behavioral synthesis. The required 25-Gb/s memory bandwidth was realized by embedding 24 single/dual port SRAM/DRAM instances. The test approach includes full scan, boundary scan, functional testing, built-in-self-test, and IDDq-test. The so-called "Natural Motion" feature implemented by this IC was demonstrated at the IFA'95 and received the European Video Innovation Award 95-96.

A video signal processor for motion-compensated field-rate upconversion in consumer television

A novel wireless multi sensor bracelet has been developed. The design choices of the bracelet - based on insights obtained with a predecessor sensor bracelet -, as well as the rationale for the choice of sensors, are presented. The hardware and software architecture are described. An example of obtained sensor data is shown. The limited battery life of the performance optimized product software fell short of the one week design target. A power optimization of the software has been made, which meets the battery life design target. It is based on current consumption measurements, and optimized sensor timing. The tradeoffs between high performance - short battery life, and low performance - long battery life are analyzed. The learnings from recent field studies on work-related stress and affective health are discussed.

/pdf/wireless-multi-sensor-bracelet-with-discreet-feedback-p93vlpnm8m.pdf

Wireless multi sensor bracelet with discreet feedback

Recent motion estimation algorithms have reached a quality level that allows an improved motion portrayal for field rate conversion systems As, in practice, still situations may occur in which motion estimation fails, a strategy for graceful degradation is required, to prevent the possible artifacts resulting from the processing from outweighing its advantages In this paper both a global fall back detection and processing mode is introduced, and a novel method applying ordered statistical filtering in the up-convertor that realizes a graceful degradation for local errors in the estimated motion vector field

Graceful Degradation in Motion-Compensated Field-Rate Conversion

Small wearable devices measuring vital-signs and connecting Direct-to-Cloud through the cellular network are a game changer for healthcare, enabling better outcomes for patients, better patient experience, better staff experience, and lower cost of care. These devices leverage mMTC technologies such as LTE-M to provide hassle-free, out-of-the-box, and ubiquitous connectivity. However, a lack of cellular coverage may inhibit the uptake of this new class of devices. This paper presents results of preliminary tests indicating that “100%” LTE-M coverage cannot be achieved in hospitals, but that MNO-approved solutions may help alleviate this cost-effectively. Furthermore, experiments on a live commercial network show that the combination of NB-IoT, as alternate RAT, with a very lean-and-mean, yet non-cloud-native, protocol gains 10.5 dB in link-budget of which 6.5 dB can be attributed to the RAT itself.

H. Huijgen

Papers

True-motion estimation with 3-D recursive search block matching

A video signal processor for motion-compensated field-rate upconversion in consumer television

Wireless multi sensor bracelet with discreet feedback

Graceful Degradation in Motion-Compensated Field-Rate Conversion

Addressing coverage concerns for Direct-to-Cloud wearables