Abstract With the advances in scientific foundations and technological implementations, optical metrology has become versatile problem-solving backbones in manufacturing, fundamental research, and engineering applications, such as quality control, nondestructive testing, experimental mechanics, and biomedicine. In recent years, deep learning, a subfield of machine learning, is emerging as a powerful tool to address problems by learning from data, largely driven by the availability of massive datasets, enhanced computational power, fast data storage, and novel training algorithms for the deep neural network. It is currently promoting increased interests and gaining extensive attention for its utilization in the field of optical metrology. Unlike the traditional “physics-based” approach, deep-learning-enabled optical metrology is a kind of “data-driven” approach, which has already provided numerous alternative solutions to many challenging problems in this field with better performances. In this review, we present an overview of the current status and the latest progress of deep-learning technologies in the field of optical metrology. We first briefly introduce both traditional image-processing algorithms in optical metrology and the basic concepts of deep learning, followed by a comprehensive review of its applications in various optical metrology tasks, such as fringe denoising, phase retrieval, phase unwrapping, subset correlation, and error compensation. The open challenges faced by the current deep-learning approach in optical metrology are then discussed. Finally, the directions for future research are outlined. 

/pdf/deep-learning-in-optical-metrology-a-review-2ggtgtrp.pdf

Deep learning in optical metrology: a review

Deep learning in the context of nano-photonics is mostly discussed in terms of its potential for inverse design of photonic devices or nano-structures. Many of the recent works on machine-learning inverse design are highly specific, and the drawbacks of the respective approaches are often not immediately clear. In this review we want therefore to provide a critical review on the capabilities of deep learning for inverse design and the progress which has been made so far. We classify the different deep-learning-based inverse design approaches at a higher level as well as by the context of their respective applications and critically discuss their strengths and weaknesses. While a significant part of the community’s attention lies on nano-photonic inverse design, deep learning has evolved as a tool for a large variety of applications. The second part of the review will focus therefore on machine learning research in nano-photonics “beyond inverse design.” This spans from physics-informed neural networks for tremendous acceleration of photonics simulations, over sparse data reconstruction, imaging and “knowledge discovery” to experimental applications.

/pdf/deep-learning-in-nano-photonics-inverse-design-and-beyond-4lpbfo8urm.pdf

Deep learning in nano-photonics: inverse design and beyond

/pdf/deep-learning-in-optical-metrology-a-review-1etdbnzu.pdf

Deep learning in the context of nano-photonics is mostly discussed in terms of its potential for inverse design of photonic devices or nanostructures. Many of the recent works on machine-learning inverse design are highly specific, and the drawbacks of the respective approaches are often not immediately clear. In this review we want therefore to provide a critical review on the capabilities of deep learning for inverse design and the progress which has been made so far. We classify the different deep learning-based inverse design approaches at a higher level as well as by the context of their respective applications and critically discuss their strengths and weaknesses. While a significant part of the community's attention lies on nano-photonic inverse design, deep learning has evolved as a tool for a large variety of applications. The second part of the review will focus therefore on machine learning research in nano-photonics "beyond inverse design". This spans from physics informed neural networks for tremendous acceleration of photonics simulations, over sparse data reconstruction, imaging and "knowledge discovery" to experimental applications.

Light sheet fluorescence microscopy (LSFM) uses a thin sheet of light to excite only fluorophores within the focal volume. Light sheet microscopes (LSMs) have a true optical sectioning capability and, hence, provide axial resolution, restrict photobleaching and phototoxicity to a fraction of the sample and use cameras to record tens to thousands of images per second. LSMs are used for in-depth analyses of large, optically cleared samples and long-term three-dimensional (3D) observations of live biological specimens at high spatio-temporal resolution. The independently operated illumination and detection trains and the canonical implementations, selective/single plane illumination microscope (SPIM) and digital scanned laser microscope (DSLM), are the basis for many LSM designs. In this Primer, we discuss various applications of LSFM for imaging multicellular specimens, developing vertebrate and invertebrate embryos, brain and heart function, 3D cell culture models, single cells, tissue sections, plants, organismic interaction and entire cleared brains. Further, we describe the combination of LSFM with other imaging approaches to allow for super-resolution or increased penetration depth and the use of sophisticated spatio-temporal manipulations to allow for observations along multiple directions. Finally, we anticipate developments of the field in the near future. Light sheet fluorescence microscopy (LSFM) is a technique that uses a thin sheet of light for illumination, allowing optical sectioning of the sample. In this Primer, Stelzer et al. outline the fundamental concepts behind LSFM, discuss the different experimental set-ups for light sheet microscopes and detail steps for processing LSFM images. The Primer also describes the range of applications for this technique across the biological sciences and concludes by discussing advances for enhancing imaging depth and resolution.

Light sheet fluorescence microscopy

We present a new class of wavefront sensors by extending their design space based on machine learning. This approach simplifies both the optical hardware and image processing in wavefront sensing. We experimentally demonstrated a variety of image-based wavefront sensing architectures that can directly estimate Zernike coefficients of aberrated wavefronts from a single intensity image by using a convolutional neural network. We also demonstrated that the proposed deep learning wavefront sensor can be trained to estimate wavefront aberrations stimulated by a point source and even extended sources.

Deep learning wavefront sensing

In this paper, we analyze a machine-learning-based non-iterative phase retrieval method. Phase retrieval and its applications have been attractive research topics in optics and photonics, for example, in biomedical imaging, astronomical imaging, and so on. Most conventional phase retrieval methods have used iterative processes to recover phase information; however, the calculation speed and convergence with these methods are serious issues in real-time monitoring applications. Machine-learning-based methods are promising for addressing these issues. Here, we numerically compare conventional methods and a machine-learning-based method in which a convolutional neural network is employed. Simulations with several conditions show that the machine-learning-based method realizes fast and robust phase recovery compared with the conventional methods. We also numerically demonstrate machine-learning-based phase retrieval from noisy measurements with a noisy training data set for improving the noise robustness. The machine-learning-based approach used in this study may increase the impact of phase retrieval, which is useful in various fields, where phase retrieval has been used as a fundamental tool.

/pdf/analysis-of-non-iterative-phase-retrieval-based-on-machine-trhvnj4f1i.pdf

Analysis of non-iterative phase retrieval based on machine learning

In this paper, we present a noniterative method for 3D computer-generated holography based on deep learning. A convolutional neural network is adapted for directly generating a hologram to reproduce a 3D intensity pattern in a given class. We experimentally demonstrated the proposed method with optical reproductions of multiple layers based on phase-only Fourier holography. Our method is noniterative, but it achieves a reproduction quality comparable with that of iterative methods for a given class.

Three-dimensional deeply generated holography [Invited]

We present a method for single-shot spectrally resolved imaging through scattering media by using the spectral memory effect of speckles. In our method, a single speckle pattern from a multi-colored object is captured through scattering media with a monochrome image sensor. The color object is recovered by correlation of the captured speckle and a three-dimensional phase retrieval process. The proposed method was experimentally demonstrated by using point sources with different emission spectra located between diffusers. This study paves the way for non-invasive and low-cost spectral imaging through scattering media.

Spectral speckle-correlation imaging.

In this Letter, we present wave propagation models of spatially partially coherent (or spatially incoherent) light to compress the computational load of forward and back propagations in inverse problems. In our model, partially coherent light is approximated as a set of random or plane wavefronts passing through spatial bandpass filters, which corresponds to an illumination pupil, and each wave coherently propagates onto a sensor plane through object space. We show that our models reduce the number of coherent propagations in inverse problems, which are essential in optical control and sensing, such as computer-generated holography (CGH) and quantitative phase imaging. We verify the proposed models by numerical and experimental demonstrations of CGH incorporating spatially partially coherent light. 

Yohei Nishizaki

Papers

Deep learning wavefront sensing

Analysis of non-iterative phase retrieval based on machine learning

Three-dimensional deeply generated holography [Invited]

Spectral speckle-correlation imaging.

Compressive propagation with coherence