Unconventional, special-purpose machines may aid in accelerating the solution of some of the hardest problems in computing, such as large-scale combinatorial optimizations, by exploiting different operating mechanisms than those of standard digital computers. We present a scalable optical processor with electronic feedback that can be realized at large scale with room-temperature technology. Our prototype machine is able to find exact solutions of, or sample good approximate solutions to, a variety of hard instances of Ising problems with up to 100 spins and 10,000 spin-spin connections.

/pdf/a-fully-programmable-100-spin-coherent-ising-machine-with-inaxlu4ff3.pdf

A fully programmable 100-spin coherent Ising machine with all-to-all connections

There is an ever-growing demand for artificial intelligence. Optical processors, which compute with photons instead of electrons, can fundamentally accelerate the development of artificial intelligence by offering substantially improved computing performance. There has been long-term interest in optically constructing the most widely used artificial-intelligence architecture, that is, artificial neural networks, to achieve brain-inspired information processing at the speed of light. However, owing to restrictions in design flexibility and the accumulation of system errors, existing processor architectures are not reconfigurable and have limited model complexity and experimental performance. Here, we propose the reconfigurable diffractive processing unit, an optoelectronic fused computing architecture based on the diffraction of light, which can support different neural networks and achieve a high model complexity with millions of neurons. Along with the developed adaptive training approach to circumvent system errors, we achieved excellent experimental accuracies for high-speed image and video recognition over benchmark datasets and a computing performance superior to that of cutting-edge electronic computing platforms. Linear diffractive structures are by themselves passive systems but researchers here exploit the non-linearity of a photodetector to realize a reconfigurable diffractive ‘processing’ unit. High-speed image and video recognition is demonstrated.

Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit

Despite their widespread use for performing advanced computational tasks, digital signal processors suffer from several restrictions, including low speed, high power consumption and complexity, caused by costly analogue-to-digital converters. For this reason, there has recently been a surge of interest in performing wave-based analogue computations that avoid analogue-to-digital conversion and allow massively parallel operation. In particular, novel schemes for wave-based analogue computing have been proposed based on artificially engineered photonic structures, that is, metamaterials. Such kinds of computing systems, referred to as computational metamaterials, can be as fast as the speed of light and as small as its wavelength, yet, impart complex mathematical operations on an incoming wave packet or even provide solutions to integro-differential equations. These much-sought features promise to enable a new generation of ultra-fast, compact and efficient processing and computing hardware based on light-wave propagation. In this Review, we discuss recent advances in the field of computational metamaterials, surveying the state-of-the-art metastructures proposed to perform analogue computation. We further describe some of the most exciting applications suggested for these computing systems, including image processing, edge detection, equation solving and machine learning. Finally, we provide an outlook for the possible directions and the key problems for future research. Metamaterials provide a platform to leverage optical signals for performing specific-purpose computational tasks with ultra-fast speeds. This Review surveys the basic principles, recent advances and promising future directions for wave-based-metamaterial analogue computing systems.

Analogue computing with metamaterials

The development of artificial intelligence is typically focused on computer algorithms and integrated circuits. Recently, all-optical diffractive deep neural networks have been created that are based on passive structures and can perform complicated functions designed by computer-based neural networks. However, once a passive diffractive deep neural network architecture is fabricated, its function is fixed. Here we report a programmable diffractive deep neural network that is based on a multi-layer digital-coding metasurface array. Each meta-atom on the metasurfaces is integrated with two amplifier chips and acts an active artificial neuron, providing a dynamic modulation range of 35 dB (from −22 dB to 13 dB). We show that the system, which we term a programmable artificial intelligence machine, can handle various deep learning tasks for wave sensing, including image classification, mobile communication coding–decoding and real-time multi-beam focusing. We also develop a reinforcement learning algorithm for on-site learning and a discrete optimization algorithm for digital coding. Using a multi-layer metasurface array in which each meta-atom of the metasurface acts as an active artificial neuron, a programmable diffractive deep neural network can be created that directly processes electromagnetic waves in free space for wave sensing and wireless communications. 

/pdf/a-programmable-diffractive-deep-neural-network-based-on-a-1dp8nhri.pdf

A programmable diffractive deep neural network based on a digital-coding metasurface array

Matrix computation, as a fundamental building block of information processing in science and technology, contributes most of the computational overheads in modern signal processing and artificial intelligence algorithms. Photonic accelerators are designed to accelerate specific categories of computing in the optical domain, especially matrix multiplication, to address the growing demand for computing resources and capacity. Photonic matrix multiplication has much potential to expand the domain of telecommunication, and artificial intelligence benefiting from its superior performance. Recent research in photonic matrix multiplication has flourished and may provide opportunities to develop applications that are unachievable at present by conventional electronic processors. In this review, we first introduce the methods of photonic matrix multiplication, mainly including the plane light conversion method, Mach-Zehnder interferometer method and wavelength division multiplexing method. We also summarize the developmental milestones of photonic matrix multiplication and the related applications. Then, we review their detailed advances in applications to optical signal processing and artificial neural networks in recent years. Finally, we comment on the challenges and perspectives of photonic matrix multiplication and photonic acceleration. 

/pdf/photonic-matrix-multiplication-lights-up-photonic-3ufkf760.pdf

Photonic matrix multiplication lights up photonic accelerator and beyond

In this Letter we propose the Fourier-space diffractive deep neural network (F-D^{2}NN) for all-optical image processing that performs advanced computer vision tasks at the speed of light. The F-D^{2}NN is achieved by placing the extremely compact diffractive modulation layers at the Fourier plane or both Fourier and imaging planes of an optical system, where the optical nonlinearity is introduced from ferroelectric thin films. We demonstrated that F-D^{2}NN can be trained with deep learning algorithms for all-optical saliency detection and high-accuracy object classification.

Fourier-space Diffractive Deep Neural Network.

Training an artificial neural network with backpropagation algorithms to perform advanced machine learning tasks requires an extensive computational process. This paper proposes to implement the backpropagation algorithm optically for in situ training of both linear and nonlinear diffractive optical neural networks, which enables the acceleration of training speed and improvement in energy efficiency on core computing modules. We demonstrate that the gradient of a loss function with respect to the weights of diffractive layers can be accurately calculated by measuring the forward and backward propagated optical fields based on light reciprocity and phase conjunction principles. The diffractive modulation weights are updated by programming a high-speed spatial light modulator to minimize the error between prediction and target output and perform inference tasks at the speed of light. We numerically validate the effectiveness of our approach on simulated networks for various applications. The proposed in situ optical learning architecture achieves accuracy comparable to in silico training with an electronic computer on the tasks of object classification and matrix-vector multiplication, which further allows the diffractive optical neural network to adapt to system imperfections. Also, the self-adaptive property of our approach facilitates the novel application of the network for all-optical imaging through scattering media. The proposed approach paves the way for robust implementation of large-scale diffractive neural networks to perform distinctive tasks all-optically.

https://www.researching.cn/ArticlePdf/m00072/2020/8/6/06000940.pdf

In situ optical backpropagation training of diffractive optical neural networks

Face restoration from low resolution and noise is important for applications of face analysis recognition. However, most existing face restoration models omit the multiple scale issues in the face restoration problem, which is still not well solved in the research area. In this paper, we propose a sequential gating ensemble network (SGEN) for a multiscale noise robust face restoration issue. To endow the network with multiscale representation ability, we first employ the principle of ensemble learning for SGEN network architecture design. The SGEN aggregates multilevel base-encoders and base-decoders into the network, which enables the network to contain multiple scales of receptive field. Instead of combining these base-en/decoders directly with nonsequential operations, the SGEN takes base-en/decoders from different levels as sequential data. Specifically, it is visualized that SGEN learns to sequentially extract high-level information from base-encoders in a bottom-up manner and restore low-level information from base-decoders in a top-down manner. Besides, we propose realizing bottom-up and top-down information combination and selection with a sequential gating unit (SGU). The SGU sequentially takes information from two different levels as inputs and decides the output based on one active input. Experimental results on the benchmark dataset demonstrate that our SGEN is more effective at multiscale human face restoration with more image details and less noise than state-of-the-art image restoration models. Further utilizing an adversarial training scheme, SGEN also produces more visually preferred results than other models under subjective evaluation.

Sequential Gating Ensemble Network for Noise Robust Multiscale Face Restoration

Abstract Endowed with the superior computing speed and energy efficiency, optical neural networks (ONNs) have attracted ever-growing attention in recent years. Existing optical computing architectures are mainly single-channel due to the lack of advanced optical connection and interaction operators, solving simple tasks such as hand-written digit classification, saliency detection, etc. The limited computing capacity and scalability of single-channel ONNs restrict the optical implementation of advanced machine vision. Herein, we develop Monet: a multichannel optical neural network architecture for a universal multiple-input multiple-channel optical computing based on a novel projection-interference-prediction framework where the inter- and intra- channel connections are mapped to optical interference and diffraction. In our Monet, optical interference patterns are generated by projecting and interfering the multichannel inputs in a shared domain. These patterns encoding the correspondences together with feature embeddings are iteratively produced through the projection-interference process to predict the final output optically. For the first time, Monet validates that multichannel processing properties can be optically implemented with high-efficiency, enabling real-world intelligent multichannel-processing tasks solved via optical computing, including 3D/motion detections. Extensive experiments on different scenarios demonstrate the effectiveness of Monet in handling advanced machine vision tasks with comparative accuracy as the electronic counterparts yet achieving a ten-fold improvement in computing efficiency. For intelligent computing, the trends of dealing with real-world advanced tasks are irreversible. Breaking the capacity and scalability limitations of single-channel ONN and further exploring the multichannel processing potential of wave optics, we anticipate that the proposed technique will accelerate the development of more powerful optical AI as critical support for modern advanced machine vision. 

/pdf/a-multichannel-optical-computing-architecture-for-advanced-2h3mwebs.pdf

Tiankuang Zhou

Papers

Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit

Fourier-space Diffractive Deep Neural Network.

In situ optical backpropagation training of diffractive optical neural networks

Sequential Gating Ensemble Network for Noise Robust Multiscale Face Restoration

A multichannel optical computing architecture for advanced machine vision