Time-varying, smooth trajectory estimation is of great interest to the vision community for accurate and well behaving 3D systems. In this paper, we propose a novel principal component local regression filter acting directly on the Riemannian manifold of unit dual quaternions DH1. We use a numerically stable Lie algebra of the dual quaternions together with exp and log operators to locally linearize the 6D pose space. Unlike state of the art path smoothing methods which either operate on SO (3) of rotation matrices or the hypersphere H1 of quaternions, we treat the orientation and translation jointly on the dual quaternion quadric in the 7-dimensional real projective space RP7. We provide an outlier-robust IRLS algorithm for generic pose filtering exploiting this manifold structure. Besides our theoretical analysis, our experiments on synthetic and real data show the practical advantages of the manifold aware filtering on pose tracking and smoothing.

/pdf/camera-pose-filtering-with-local-regression-geodesics-on-the-1b9y0zq9p7.pdf

Camera Pose Filtering with Local Regression Geodesics on the Riemannian Manifold of Dual Quaternions

Recently, video frame interpolation using a combination of frame- and event-based cameras has surpassed traditional image-based methods both in terms of performance and memory efficiency. However, current methods still suffer from (i) brittle image-level fusion of complementary interpolation results, that fails in the presence of artifacts in the fused image, (ii) potentially temporally inconsistent and inefficient motion estimation procedures, that run for every inserted frame and (iii) low contrast regions that do not trigger events, and thus cause events-only motion estimation to generate artifacts. Moreover, previous methods were only tested on datasets consisting of planar and far-away scenes, which do not capture the full complexity of the real world. In this work, we address the above problems by introducing multi-scale feature-level fusion and computing one-shot non-linear inter-frame motion-which can be efficiently sampled for image warping-from events and images. We also collect the first large-scale events and frames dataset consisting of more than 100 challenging scenes with depth variations, captured with a new experimental setup based on a beamsplitter. We show that our method improves the reconstruction quality by up to 0.2 dB in terms of PSNR and up to 15% in LPIPS score.

Time Lens++: Event-based Frame Interpolation with Parametric Nonlinear Flow and Multi-scale Fusion

A wide range of system models in modern robotics and avionics applications admit natural symmetries. Such systems are termed equivariant and the structure provided by the symmetry is a powerful tool in the design of observers. Significant progress has been made in the last ten years in the design of filters and observers for attitude and pose estimation, tracking of homographies, and velocity aided attitude estimation, by exploiting their inherent Lie-group state-space structure. However, little work has been done for systems on homogeneous spaces, that is systems on manifolds on which a Lie-group acts rather than systems on the Lie-group itself. Recent research in robotic vision has discovered symmetries and equivariant structure on homogeneous spaces for a host of problems including the key problems of visual odometry and visual simultaneous localisation and mapping. These discoveries motivate a deeper look at the structure of equivariant systems on homogeneous spaces. This paper provides a comprehensive development of the foundation theory required to undertake observer and filter design for such systems.

Equivariant Systems Theory and Observer Design

Event cameras are ideally suited to capture HDR visual information without blur but perform poorly on static or slowly changing scenes. Conversely, conventional image sensors measure absolute intensity of slowly changing scenes effectively but do poorly on high dynamic range or quickly changing scenes. In this paper, we present an event-based video reconstruction pipeline for High Dynamic Range (HDR) scenarios. The proposed algorithm includes a frame augmentation pre-processing step that deblurs and temporally interpolates frame data using events. The augmented frame and event data are then fused using a novel asynchronous Kalman filter under a unifying uncertainty model for both sensors. Our experimental results are evaluated on both publicly available datasets with challenging lighting conditions and fast motions and our new dataset with HDR reference. The proposed algorithm outperforms state-of-the-art methods in both absolute intensity error (48% reduction) and image similarity indexes (average 11% improvement).

An Asynchronous Kalman Filter for Hybrid Event Cameras.

Time-varying, smooth trajectory estimation is of great interest to the vision community for accurate and well behaving 3D systems. In this paper, we propose a novel principal component local regression filter acting directly on the Riemannian manifold of unit dual quaternions $\mathbb{D} \mathbb{H}_1$. We use a numerically stable Lie algebra of the dual quaternions together with $\exp$ and $\log$ operators to locally linearize the 6D pose space. Unlike state of the art path smoothing methods which either operate on $SO\left(3\right)$ of rotation matrices or the hypersphere $\mathbb{H}_1$ of quaternions, we treat the orientation and translation jointly on the dual quaternion quadric in the 7-dimensional real projective space $\mathbb{R}\mathbb{P}^7$. We provide an outlier-robust IRLS algorithm for generic pose filtering exploiting this manifold structure. Besides our theoretical analysis, our experiments on synthetic and real data show the practical advantages of the manifold aware filtering on pose tracking and smoothing.

This paper addresses the problem of estimating the attitude and angular velocity of a rigid object by exploiting its second order kinematic model. The approach is particularly useful in cases where angular velocity measurements are not available and the attitude and angular velocity of an object need to be estimated from accelerometers and magnetometers. We propose a novel sensor modality that uses multiple accelerometers to measure the angular acceleration of an object as well as using magnetometers to measure partial attitude. We extend the approach of equivariant observer design to second order attitude kinematics by demonstrating that the special Euclidean group acts as a symmetry group on the system considered. The observer design is based on the lifted kinematics and we prove almost global asymptotic stability and local uniform exponential stability of the estimation error. The performance of the observer is demonstrated in simulation.

Attitude Observation for Second Order Attitude Kinematics

This paper presents the equivariant systems theory and observer design for second order kinematic systems on matrix Lie groups. The state of a second order kinematic system on a matrix Lie group is naturally posed on the tangent bundle of the group with the inputs lying in the tangent of the tangent bundle known as the double tangent bundle. We provide a simple parameterization of both the tangent bundle state space and the input space (the fiber space of the double tangent bundle) and then introduce a semi-direct product group and group actions onto both the state and input spaces. We show that with the proposed group actions the second order kinematics are equivariant. An equivariant lift of the kinematics onto the symmetry group is defined and used to design a nonlinear observer on the lifted state space using nonlinear constructive design techniques. A simple hovercraft simulation verifies the performance of our observer.

/pdf/equivariant-systems-theory-and-observer-design-for-second-yhlrfhi019.pdf

Equivariant Systems Theory and Observer Design for Second Order Kinematic Systems on Matrix Lie Groups

Event cameras output asynchronous events to represent intensity changes with a high temporal resolution, even under extreme lighting conditions. Currently, most of the existing works use a single contrast threshold to estimate the intensity change of all pixels. However, complex circuit bias and manufacturing imperfections cause biased pixels and mismatch contrast threshold among pixels, which may lead to undesirable outputs. In this paper, we propose a new event camera model and two calibration approaches which cover event-only cameras and hybrid image-event cameras. When intensity images are simultaneously provided along with events, we also propose an efficient online method to calibrate event cameras that adapts to time-varying event rates. We demonstrate the advantages of our proposed methods compared to the state-of-the-art on several different event camera datasets.

Event Camera Calibration of Per-pixel Biased Contrast Threshold.

Event cameras are bio-inspired sensors that capture per-pixel asynchronous intensity change rather than the synchronous absolute intensity frames captured by a classical camera sensor. Such cameras are ideal for robotics applications since they have high temporal resolution, high dynamic range and low latency. However, due to their high temporal resolution, event cameras are particularly sensitive to flicker such as from fluorescent or LED lights. During every cycle from bright to dark, pixels that image a flickering light source generate many events that provide little or no useful information for a robot, swamping the useful data in the scene. In this paper, we propose a novel linear filter to preprocess event data to remove unwanted flicker events from an event stream. The proposed algorithm achieves over 4.6 times relative improvement in the signal-to-noise ratio when compared to the raw event stream due to the effective removal of flicker from fluorescent lighting. Thus, it is ideally suited to robotics applications that operate in indoor settings or scenes illuminated by flickering light sources. Code, Datasets and Video: https://github.com/ziweiWWANG/EFR

Yonhon Ng

Papers

An Asynchronous Kalman Filter for Hybrid Event Cameras.

Attitude Observation for Second Order Attitude Kinematics

Equivariant Systems Theory and Observer Design for Second Order Kinematic Systems on Matrix Lie Groups

Event Camera Calibration of Per-pixel Biased Contrast Threshold.

A Linear Comb Filter for Event Flicker Removal