Kinect range sensing

In recent years, the videogame industry has been characterized by a great boost in gesture recognition and motion tracking, following the increasing request of creating immersive game experiences. The Microsoft Kinect sensor allows acquiring RGB, IR and depth images with a high frame rate. Because of the complementary nature of the information provided, it has proved an attractive resource for researchers with very different backgrounds. In summer 2014, Microsoft launched a new generation of Kinect on the market, based on time-of-flight technology. This paper proposes a calibration of Kinect for Xbox One imaging sensors, focusing on the depth camera. The mathematical model that describes the error committed by the sensor as a function of the distance between the sensor itself and the object has been estimated. All the analyses presented here have been conducted for both generations of Kinect, in order to quantify the improvements that characterize every single imaging sensor. Experimental results show that the quality of the delivered model improved applying the proposed calibration procedure, which is applicable to both point clouds and the mesh model created with the Microsoft Fusion Libraries.

/pdf/calibration-of-kinect-for-xbox-one-and-comparison-between-19n1br77ez.pdf

Calibration of Kinect for Xbox One and Comparison between the Two Generations of Microsoft Sensors

Time-of-flight (TOF) cameras are sensors that can measure the depths of scene points, by illuminating the scene with a controlled laser or LED source and then analyzing the reflected light. In this paper, we will first describe the underlying measurement principles of time-of-flight cameras, including: (1) pulsed-light cameras, which measure directly the time taken for a light pulse to travel from the device to the object and back again, and (2) continuous-wave-modulated light cameras, which measure the phase difference between the emitted and received signals, and hence obtain the travel time indirectly. We review the main existing designs, including prototypes as well as commercially available devices. We also review the relevant camera calibration principles, and how they are applied to TOF devices. Finally, we discuss the benefits and challenges of combined TOF and color camera systems.

https://hal.inria.fr/hal-01325045/document

An overview of depth cameras and range scanners based on time-of-flight technologies

An electronic device coupleable to a display screen includes a camera system that acquires optical data of a user comfortably gesturing in a user-customizable interaction zone having a z 0 plane, while controlling operation of the device. Subtle gestures include hand movements commenced in a dynamically resizable and relocatable interaction zone. Preferably (x,y,z) locations in the interaction zone are mapped to two-dimensional display screen locations. Detected user hand movements can signal the device that an interaction is occurring in gesture mode. Device response includes presenting GUI on the display screen, creating user feedback including haptic feedback. User three-dimensional interaction can manipulate displayed virtual objects, including releasing such objects. User hand gesture trajectory clues enable the device to anticipate probable user intent and to appropriately update display screen renderings.

Method and system enabling natural user interface gestures with an electronic system

A $252 \times 144$ single-photon avalanche diode (SPAD) pixel sensor, called Ocelot, is reported for light detection and ranging (LiDAR). The sensor, fabricated in the 180-nm CMOS technology, features 1728 12-bit time-to-digital converters (TDCs) with 48.8-ps resolution (LSB). Each 126 pixels in a half-column are connected to six TDCs through a collision detection bus, which enables effective sharing of resources, and consequently a fill factor of 28% with a pixel pitch of 28.5 $\mu \text{m}$ . The column-parallel TDCs, based on dual-clock architecture, exhibit a DNL of +0.48/−0.48 LSB and an INL of +0.89/−1.67 LSB; they are dynamically reallocated in a scalable daisy chain approach that enables a maximum of five photon detections per illumination cycle per half-column. The sensor can operate in time-correlated single-photon counting (TCSPC) and single-photon counting (SPC) modes, while peak detection (PD) and partial histogramming (PH) are included in the operation of the sensor. The PD and PH modes are enabled by the first implementation of integrated histogramming for a full array via 3.32-Mb SRAM-based PH readout (PHR) scheme providing a 14.9-to-1 compression. Telemetry measurements up to 50 m achieve an accuracy of 8.8 cm and worst-case precision of 1.4 mm ( $\sigma $ ). A flash LiDAR using direct time of flight (dTOF) and based on Ocelot is demonstrated, achieving depth imaging at short distances with a frame rate of 30 frames/s, employing an ultra-low power laser with an average power of 2 mW and peak power of 0.5 W.

A 30-frames/s, $252\times144$ SPAD Flash LiDAR With 1728 Dual-Clock 48.8-ps TDCs, and Pixel-Wise Integrated Histogramming

We introduce a 512 × 424 time-of-flight (TOF) depth image sensor designed in a TSMC 0.13 μm LP 1P5M CMOS process, suitable for use in Microsoft Kinect for XBOX ONE. The 10 μm × 10 μm pixel incorporates a TOF detector that operates using the quantum efficiency modulation (QEM) technique at high modulation frequencies of up to 130 MHz, achieves a modulation contrast of 67% at 50 MHz and a responsivity of 0.14 A/W at 860 nm. The TOF sensor includes a 2 GS/s 10 bit signal path, which is used for the high ADC bandwidth requirements of the system that requires many ADC conversions per frame. The chip also comprises a clock generation circuit featuring a programmable phase and frequency clock generator with 312.5-ps phase step resolution derived from a 1.6 GHz oscillator. An integrated shutter engine and a programmable digital micro-sequencer allows an extremely flexible multi-gain/multi-shutter and multi-frequency/multi-phase operation. All chip data is transferred using two 4-lane MIPI D-PHY interfaces with a total of 8 Gb/s input/output bandwidth. The reported experimental results demonstrate a wide depth range of operation (0.8–4.2 m), small accuracy error ( $ 1%), very low depth uncertainty ( $ 0.5% of actual distance), and very high dynamic range ( $>$ 64 dB).

A 0.13 μm CMOS System-on-Chip for a 512 × 424 Time-of-Flight Image Sensor With Multi-Frequency Photo-Demodulation up to 130 MHz and 2 GS/s ADC

The quest for accurate, high-resolution, low-power-consumption, and small-footprint 3D depth cameras has driven a rapid improvement in Continuous-Wave (CW) Time-of-Flight (ToF) technology. Commercially available 3D image acquisition techniques include Stereo Vision, Structured Light, and ToF. CW ToF imaging systems offer excellent mechanical robustness, no baseline requirement, high effective depth image resolution, low computational cost, and simultaneous IR ambient light invariant intensity capture (Active Brightness). In a CW ToF camera, light from an amplitude modulated light source is backscattered by objects in the camera's field of view, and the phase delay of the amplitude envelope is measured between the emitted and reflected light. This phase difference is translated into a distance value for each pixel in the imaging array.

IMpixel 65nm BSI 320MHz demodulated TOF Image sensor with 3μm global shutter pixels and analog binning

Interest in 3D depth cameras has been piqued by the release of the Kinect motion sensor for the Xbox 360 gaming console. This paper presents the pixel and 2GS/s signal paths in a state-of-the-art Time-of-Flight (ToF) sensor suitable for use in the latest Kinect sensor for Xbox One. ToF cameras determine the distance to objects by measuring the round trip travel time of an amplitude-modulated light from the source to the target and back to the camera at each pixel. ToF technology provides an accurate high pixel resolution, low motion blur, wide field of view (FoV), high dynamic range depth image as well as an ambient light invariant brightness image (active IR) that meets the highest quality requirements for 3D motion detection.

7.6 A 512×424 CMOS 3D Time-of-Flight image sensor with multi-frequency photo-demodulation up to 130MHz and 2GS/s ADC

A phase-based TOF system preferably generates an optical waveform with fast rise and fall times, to enhance modulation contrast, notwithstanding there will be many high order harmonics. The system is preferably operated with an odd number of phases, to reduce system bias error due to the higher order harmonics, while maintaining good modulation contrast, without unduly increasing system memory requirements. Preferably the system can dynamically calibrate (and compensate for) higher order harmonics in the TOF generated optical energy waveform, over time and temperature. Within the optical energy transmission channel, or within the optical energy detection channel, detection amplifier gain may be modified, and/or detector signal integration time may be varied, and/or digital values may be employed to implement calibration and error reduction The resultant TOF system can operate with improved phase-vs-distance characteristics, with reduced calibration requirements.

Method and system for multi-phase dynamic calibration of three-dimensional (3d) sensors in a time-of-flight system

Phase-based TOF systems operate with reduced depth error due to motion blur, and/or spatial blur, and/or pixel offset by intelligently determining how best to combine pixel values, and how best to compensate for individual pixel offsets. Such determination(s) may be carried out on a per pixel basis, dynamically, in real-time during TOF operation, or on archived TOF data. Offsets for individual pixels may be dynamically calculated and subtracted from the values acquired by those pixels Individual pixel offsets may be calculated for example by combining data acquired by the same pixel at two acquisitions, 180° out of phase with respect to each other. Calculated offsets may be averaged, or on a per pixel basis, and if target object motion is detected, one or more offset calculations can be discarded rather than averaged to reduce motion blur. Offsets acquired a priori during a TOF system calibration procedure may be used.

Travis Perry

Papers

A 0.13 μm CMOS System-on-Chip for a 512 × 424 Time-of-Flight Image Sensor With Multi-Frequency Photo-Demodulation up to 130 MHz and 2 GS/s ADC

IMpixel 65nm BSI 320MHz demodulated TOF Image sensor with 3μm global shutter pixels and analog binning

7.6 A 512×424 CMOS 3D Time-of-Flight image sensor with multi-frequency photo-demodulation up to 130MHz and 2GS/s ADC

Method and system for multi-phase dynamic calibration of three-dimensional (3d) sensors in a time-of-flight system

Method and system to maximize space-time resolution in a time-of-flight (TOF) system