Replacing electrons with photons is a compelling route towards light-speed, highly parallel, and low-power artificial intelligence computing. Recently, all-optical diffractive neural deep neural networks have been demonstrated. However, the existing architectures often comprise bulky components and, most critically, they cannot mimic the human brain for multitasking. Here, we demonstrate a multi-skilled diffractive neural network based on a metasurface device, which can perform on-chip multi-channel sensing and multitasking at the speed of light in the visible. The metasurface is integrated with a complementary metal oxide semiconductor imaging sensor. Polarization multiplexing scheme of the subwavelength nanostructures are applied to construct a multi-channel classifier framework for simultaneous recognition of digital and fashionable items. The areal density of the artificial neurons can reach up to 6.25x106/mm2 multiplied by the number of channels. Our platform provides an integrated solution with all-optical on-chip sensing and computing for applications in machine vision, autonomous driving, and precision medicine.

Metasurface-Enabled On-Chip Multiplexed Diffractive Neural Networks in the Visible

Replacing electrons with photons is a compelling route toward high-speed, massively parallel, and low-power artificial intelligence computing. Recently, diffractive networks composed of phase surfaces were trained to perform machine learning tasks through linear optical transformations. However, the existing architectures often comprise bulky components and, most critically, they cannot mimic the human brain for multitasking. Here, we demonstrate a multi-skilled diffractive neural network based on a metasurface device, which can perform on-chip multi-channel sensing and multitasking in the visible. The polarization multiplexing scheme of the subwavelength nanostructures is applied to construct a multi-channel classifier framework for simultaneous recognition of digital and fashionable items. The areal density of the artificial neurons can reach up to 6.25 × 106 mm-2 multiplied by the number of channels. The metasurface is integrated with the mature complementary metal-oxide semiconductor imaging sensor, providing a chip-scale architecture to process information directly at physical layers for energy-efficient and ultra-fast image processing in machine vision, autonomous driving, and precision medicine. 

/pdf/metasurface-enabled-on-chip-multiplexed-diffractive-neural-1iszp31z.pdf

Metasurface-enabled on-chip multiplexed diffractive neural networks in the visible

Abstract The exponential growth of information stored in data centers and computational power required for various data-intensive applications, such as deep learning and AI, call for new strategies to improve or move beyond the traditional von Neumann architecture. Recent achievements in information storage and computation in the optical domain, enabling energy-efficient, fast, and high-bandwidth data processing, show great potential for photonics to overcome the von Neumann bottleneck and reduce the energy wasted to Joule heating. Optically readable memories are fundamental in this process, and while light-based storage has traditionally (and commercially) employed free-space optics, recent developments in photonic integrated circuits (PICs) and optical nano-materials have opened the doors to new opportunities on-chip. Photonic memories have yet to rival their electronic digital counterparts in storage density; however, their inherent analog nature and ultrahigh bandwidth make them ideal for unconventional computing strategies. Here, we review emerging nanophotonic devices that possess memory capabilities by elaborating on their tunable mechanisms and evaluating them in terms of scalability and device performance. Moreover, we discuss the progress on large-scale architectures for photonic memory arrays and optical computing primarily based on memory performance.

/pdf/photonic-computational-memories-tunable-nanophotonics-for-2a9830sv.pdf

Photonic (computational) memories: tunable nanophotonics for data storage and computing

Abstract Research on optical computing has recently attracted significant attention due to the transformative advances in machine learning. Among different approaches, diffractive optical networks composed of spatially-engineered transmissive surfaces have been demonstrated for all-optical statistical inference and performing arbitrary linear transformations using passive, free-space optical layers. Here, we introduce a polarization-multiplexed diffractive processor to all-optically perform multiple, arbitrarily-selected linear transformations through a single diffractive network trained using deep learning. In this framework, an array of pre-selected linear polarizers is positioned between trainable transmissive diffractive materials that are isotropic, and different target linear transformations (complex-valued) are uniquely assigned to different combinations of input/output polarization states. The transmission layers of this polarization-multiplexed diffractive network are trained and optimized via deep learning and error-backpropagation by using thousands of examples of the input/output fields corresponding to each one of the complex-valued linear transformations assigned to different input/output polarization combinations. Our results and analysis reveal that a single diffractive network can successfully approximate and all-optically implement a group of arbitrarily-selected target transformations with a negligible error when the number of trainable diffractive features/neurons ( N ) approaches $$N_pN_iN_o$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:msub> <mml:mrow> <mml:mi>N</mml:mi> </mml:mrow> <mml:mrow> <mml:mi>p</mml:mi> </mml:mrow> </mml:msub> <mml:msub> <mml:mrow> <mml:mi>N</mml:mi> </mml:mrow> <mml:mrow> <mml:mi>i</mml:mi> </mml:mrow> </mml:msub> <mml:msub> <mml:mrow> <mml:mi>N</mml:mi> </mml:mrow> <mml:mrow> <mml:mi>o</mml:mi> </mml:mrow> </mml:msub> </mml:mrow> </mml:math> , where N i and N o represent the number of pixels at the input and output fields-of-view, respectively, and N p refers to the number of unique linear transformations assigned to different input/output polarization combinations. This polarization-multiplexed all-optical diffractive processor can find various applications in optical computing and polarization-based machine vision tasks. 

/pdf/polarization-multiplexed-diffractive-computing-all-optical-2dy91de1.pdf

Polarization multiplexed diffractive computing: all-optical implementation of a group of linear transformations through a polarization-encoded diffractive network

/pdf/polarization-multiplexed-diffractive-computing-all-optical-1abexssf.pdf

Matrix computation, as a fundamental building block of information processing in science and technology, contributes most of the computational overheads in modern signal processing and artificial intelligence algorithms. Photonic accelerators are designed to accelerate specific categories of computing in the optical domain, especially matrix multiplication, to address the growing demand for computing resources and capacity. Photonic matrix multiplication has much potential to expand the domain of telecommunication, and artificial intelligence benefiting from its superior performance. Recent research in photonic matrix multiplication has flourished and may provide opportunities to develop applications that are unachievable at present by conventional electronic processors. In this review, we first introduce the methods of photonic matrix multiplication, mainly including the plane light conversion method, Mach-Zehnder interferometer method and wavelength division multiplexing method. We also summarize the developmental milestones of photonic matrix multiplication and the related applications. Then, we review their detailed advances in applications to optical signal processing and artificial neural networks in recent years. Finally, we comment on the challenges and perspectives of photonic matrix multiplication and photonic acceleration. 

/pdf/photonic-matrix-multiplication-lights-up-photonic-fmtloxf9.pdf

Zhichao Ruan

Papers

Photonic matrix multiplication lights up photonic accelerator and beyond