Recursive Video Matting and Denoising
23 Aug 2010-pp 4524-4527
TL;DR: A video matting method with simultaneous noise reduction based on the Unscented Kalman filter (UKF) that incorporates spatio-temporal information from the current and previous frame during estimation of the alpha matte as well as the foreground.
Abstract: In this paper, we propose a video matting method with simultaneous noise reduction based on the Unscented Kalman filter (UKF). This recursive approach extracts the alpha mattes and denoised foregrounds from noisy videos, in a unified framework. No assumptions are made about the type of motion of the camera or of the foreground object in the video. Moreover, user-specified trimaps are required only once every ten frames. In order to accurately extract information at the borders between the foreground and the background, we include a discontinuity-adaptive Markov random field (MRF) prior. It incorporates spatio-temporal information from the current and previous frame during estimation of the alpha matte as well as the foreground. Results are given on videos with real film-grain noise.
References
More filters
28 Jul 1997
TL;DR: It is argued that the ease of implementation and more accurate estimation features of the new filter recommend its use over the EKF in virtually all applications.
Abstract: The Kalman Filter (KF) is one of the most widely used methods for tracking and estimation due to its simplicity, optimality, tractability and robustness. However, the application of the KF to nonlinear systems can be difficult. The most common approach is to use the Extended Kalman Filter (EKF) which simply linearizes all nonlinear models so that the traditional linear Kalman filter can be applied. Although the EKF (in its many forms) is a widely used filtering strategy, over thirty years of experience with it has led to a general consensus within the tracking and control community that it is difficult to implement, difficult to tune, and only reliable for systems which are almost linear on the time scale of the update intervals. In this paper a new linear estimator is developed and demonstrated. Using the principle that a set of discretely sampled points can be used to parameterize mean and covariance, the estimator yields performance equivalent to the KF for linear systems yet generalizes elegantly to nonlinear systems without the linearization steps required by the EKF. We show analytically that the expected performance of the new approach is superior to that of the EKF and, in fact, is directly comparable to that of the second order Gauss filter. The method is not restricted to assuming that the distributions of noise sources are Gaussian. We argue that the ease of implementation and more accurate estimation features of the new filter recommend its use over the EKF in virtually all applications.
5,314 citations
"Recursive Video Matting and Denoisi..." refers background in this paper
...Since (3) is non-linear, we resort to the UKF, which is based on the principle of unscented transformation (UT), to calculate moments of nonlinearly transformed random variable (see [9] and [14] for details)....
[...]
...where M = 4 is the dimension of augmented mean; λ = u(2)(M +κ)−M and κ = 2 are the scaling parameters, while u = 1 is the sigma point spread [9]....
[...]
...where α̂− is the predicted mean of α, ŷα is the weighted observation mean, and y is the given observed image (refer [9] for expressions of weights)....
[...]
11 May 2004
TL;DR: By proving that this scheme implements a coarse-to-fine warping strategy, this work gives a theoretical foundation for warping which has been used on a mainly experimental basis so far and demonstrates its excellent robustness under noise.
Abstract: We study an energy functional for computing optical flow that combines three assumptions: a brightness constancy assumption, a gradient constancy assumption, and a discontinuity-preserving spatio-temporal smoothness constraint. In order to allow for large displacements, linearisations in the two data terms are strictly avoided. We present a consistent numerical scheme based on two nested fixed point iterations. By proving that this scheme implements a coarse-to-fine warping strategy, we give a theoretical foundation for warping which has been used on a mainly experimental basis so far. Our evaluation demonstrates that the novel method gives significantly smaller angular errors than previous techniques for optical flow estimation. We show that it is fairly insensitive to parameter variations, and we demonstrate its excellent robustness under noise.
2,902 citations
"Recursive Video Matting and Denoisi..." refers methods in this paper
...An accurate optical flow method presented in [6] (which is robust to reasonable levels of noise) is utilized for motion estimation, using the implementation given in [1]....
[...]
01 Jul 2005
TL;DR: This work presents an interactive video cutout system that allows users to quickly extract foreground objects from video sequences for use in a variety of applications including compositing onto new backgrounds and NPR cartoon style rendering.
Abstract: We present an interactive system for efficiently extracting foreground objects from a video. We extend previous min-cut based image segmentation techniques to the domain of video with four new contributions. We provide a novel painting-based user interface that allows users to easily indicate the foreground object across space and time. We introduce a hierarchical mean-shift preprocess in order to minimize the number of nodes that min-cut must operate on. Within the min-cut we also define new local cost functions to augment the global costs defined in earlier work. Finally, we extend 2D alpha matting methods designed for images to work with 3D video volumes. We demonstrate that our matting approach preserves smoothness across both space and time. Our interactive video cutout system allows users to quickly extract foreground objects from video sequences for use in a variety of applications including compositing onto new backgrounds and NPR cartoon style rendering.
407 citations
"Recursive Video Matting and Denoisi..." refers methods in this paper
...An interactive cutout method that utilizes spatio-temporal coherence is described in [15], but it is not precise in cases of fine details or significant camera translation....
[...]
01 Jul 2002
TL;DR: A Bayesian matting technique uses the flowed trimaps to yield high-quality mattes of moving foreground elements with complex boundaries filmed by a moving camera, and a novel technique for smoke matte extraction is also demonstrated.
Abstract: This paper describes a new framework for video matting, the process of pulling a high-quality alpha matte and foreground from a video sequence. The framework builds upon techniques in natural image matting, optical flow computation, and background estimation. User interaction is comprised of garbage matte specification if background estimation is needed, and hand-drawn keyframe segmentations into "foreground," "background" and "unknown". The segmentations, called trimaps, are interpolated across the video volume using forward and backward optical flow. Competing flow estimates are combined based on information about where flow is likely to be accurate. A Bayesian matting technique uses the flowed trimaps to yield high-quality mattes of moving foreground elements with complex boundaries filmed by a moving camera. A novel technique for smoke matte extraction is also demonstrated.
388 citations
"Recursive Video Matting and Denoisi..." refers background or methods or result in this paper
...where c denotes color plane and B is an arbitrary background [7]....
[...]
...The problem of video matting was first described in [7], which used a Bayesian technique to solve for α, F and B....
[...]
..., new features appear in future frames), we can combine both forward and backward flow, as described in [7]....
[...]
...Figure 2(a) (top row) shows an image taken from the video provided by [7], while the bottom row shows an image from a 1960’s video (with real film-grain) obtained from archives [2]....
[...]
...We have not compared with other still image matting methods, since only the Bayesian matting approach has been extended to video [7]....
[...]
13 Oct 2003
TL;DR: This work has developed a novel foreground-background segmentation algorithm that explicitly accounts for the nonstationary nature and clutter-like appearance of many dynamic textures.
Abstract: The algorithm presented aims to segment the foreground objects in video (e.g., people) given time-varying, textured backgrounds. Examples of time-varying backgrounds include waves on water, clouds moving, trees waving in the wind, automobile traffic, moving crowds, escalators, etc. We have developed a novel foreground-background segmentation algorithm that explicitly accounts for the nonstationary nature and clutter-like appearance of many dynamic textures. The dynamic texture is modeled by an autoregressive moving average model (ARMA). A robust Kalman filter algorithm iteratively estimates the intrinsic appearance of the dynamic texture, as well as the regions of the foreground objects. Preliminary experiments with this method have demonstrated promising results.
350 citations
"Recursive Video Matting and Denoisi..." refers methods in this paper
...Segmentation of objects from dynamic backgrounds using Kalman filtering is proposed in [16]....
[...]