Since their inception in the 1930–1960s, the research disciplines of computational imaging and machine learning have followed parallel tracks and, during the last two decades, experienced explosive growth drawing on similar progress in mathematical optimization and computing hardware. While these developments have always been to the benefit of image interpretation and machine vision, only recently has it become evident that machine learning architectures, and deep neural networks in particular, can be effective for computational image formation, aside from interpretation. The deep learning approach has proven to be especially attractive when the measurement is noisy and the measurement operator ill posed or uncertain. Examples reviewed here are: super-resolution; lensless retrieval of phase and complex amplitude from intensity; photon-limited scenes, including ghost imaging; and imaging through scatter. In this paper, we cast these works in a common framework. We relate the deep-learning-inspired solutions to the original computational imaging formulation and use the relationship to derive design insights, principles, and caveats of more general applicability. We also explore how the machine learning process is aided by the physics of imaging when ill posedness and uncertainties become particularly severe. It is hoped that the present unifying exposition will stimulate further progress in this promising field of research.

On the use of deep learning for computational imaging

How to image objects that are hidden from a camera's view is a problem of fundamental importance to many fields of research, with applications in robotic vision, defence, remote sensing, medical imaging and autonomous vehicles. Non-line-of-sight (NLOS) imaging at macroscopic scales has been demonstrated by scanning a visible surface with a pulsed laser and a time-resolved detector. Whereas light detection and ranging (LIDAR) systems use such measurements to recover the shape of visible objects from direct reflections, NLOS imaging reconstructs the shape and albedo of hidden objects from multiply scattered light. Despite recent advances, NLOS imaging has remained impractical owing to the prohibitive memory and processing requirements of existing reconstruction algorithms, and the extremely weak signal of multiply scattered light. Here we show that a confocal scanning procedure can address these challenges by facilitating the derivation of the light-cone transform to solve the NLOS reconstruction problem. This method requires much smaller computational and memory resources than previous reconstruction methods do and images hidden objects at unprecedented resolution. Confocal scanning also provides a sizeable increase in signal and range when imaging retroreflective objects. We quantify the resolution bounds of NLOS imaging, demonstrate its potential for real-time tracking and derive efficient algorithms that incorporate image priors and a physically accurate noise model. Additionally, we describe successful outdoor experiments of NLOS imaging under indirect sunlight.

Confocal non-line-of-sight imaging based on the light-cone transform.

Scientists demonstrate an experimental method that allows them to locate and track moving targets that are hidden from the direct line of sight, for example, by a wall or an obstacle, with only a few seconds acquisition time and centimetre precision. The ability to detect motion and track a moving object hidden around a corner or behind a wall provides a crucial advantage when physically going around the obstacle is impossible or dangerous. Previous methods have demonstrated that it is possible to reconstruct the shape of an object hidden from view. However, these methods do not enable the tracking of movement in real time. We demonstrate a compact non-line-of-sight laser ranging technology that relies on the ability to send light around an obstacle using a scattering floor and then detect the return signal from a hidden object within only a few seconds of acquisition time. By detecting this signal with a single-photon avalanche diode (SPAD) camera, we follow the movement of an object located a metre away from the camera with centimetre precision. We discuss the possibility of applying this technology to a variety of real-life situations in the near future.

/pdf/detection-and-tracking-of-moving-objects-hidden-from-view-2rorllupvx.pdf

Detection and tracking of moving objects hidden from view

Non-line-of-sight imaging allows objects to be observed when partially
 or fully occluded from direct view, by analysing indirect diffuse reflections off a
 secondary relay surface. Despite many potential
 applications1–9, existing methods lack practical usability because of
 limitations including the assumption of single scattering only, ideal diffuse
 reflectance and lack of occlusions within the hidden scene. By
 contrast, line-of-sight imaging systems do not impose any assumptions about the
 imaged scene, despite relying on the mathematically simple processes of linear
 diffractive wave propagation. Here we show that the problem of non-line-of-sight
 imaging can also be formulated as one of diffractive wave propagation, by
 introducing a virtual wave field that we term the phasor field. Non-line-of-sight
 scenes can be imaged from raw time-of-flight data by applying the mathematical
 operators that model wave propagation in a conventional line-of-sight imaging
 system. Our method yields a new class of imaging algorithms that mimic the
 capabilities of line-of-sight cameras. To demonstrate our technique, we derive three
 imaging algorithms, modelled after three different line-of-sight systems. These
 algorithms rely on solving a wave diffraction integral, namely the
 Rayleigh–Sommerfeld diffraction integral. Fast solutions to Rayleigh–Sommerfeld
 diffraction and its approximations are readily available, benefiting our method. We
 demonstrate non-line-of-sight imaging of complex scenes with strong multiple
 scattering and ambient light, arbitrary materials, large depth range and occlusions.
 Our method handles these challenging cases without explicitly inverting a
 light-transport model. We believe that our approach will help to unlock the
 potential of non-line-of-sight imaging and promote the development of relevant
 applications not restricted to laboratory conditions. Algorithms based on diffractive wave propagation of light offer
 effective imaging of complex scenes hidden from direct view.

/pdf/non-line-of-sight-imaging-using-phasor-field-virtual-wave-34644gw4lb.pdf

Non-line-of-sight imaging using phasor-field virtual wave optics

Computational imaging combines measurement and computational methods with the aim of forming images even when the measurement conditions are weak, few in number, or highly indirect. The recent surge in quantum-inspired imaging sensors, together with a new wave of algorithms allowing on-chip, scalable and robust data processing, has induced an increase of activity with notable results in the domain of low-light flux imaging and sensing. We provide an overview of the major challenges encountered in low-illumination (e.g., ultrafast) imaging and how these problems have recently been addressed for imaging applications in extreme conditions. These methods provide examples of the future imaging solutions to be developed, for which the best results are expected to arise from an efficient codesign of the sensors and data analysis tools.

/pdf/quantum-inspired-computational-imaging-zf1sjl66cs.pdf

Quantum-inspired computational imaging

By using time-of-flight information encoded in multiply scattered light, it is possible to reconstruct images of objects hidden from the camera’s direct line of sight. Here, we present a non-line-of-sight imaging system that uses a single-pixel, single-photon avalanche diode (SPAD) to collect time-of-flight information. Compared to earlier systems, this modification provides significant improvements in terms of power requirements, form factor, cost, and reconstruction time, while maintaining a comparable time resolution. The potential for further size and cost reduction of this technology make this system a good base for developing a practical system that can be used in real world applications.

Jessica Zeman

Papers

Non-line-of-sight imaging using a time-gated single photon avalanche diode.