The separation of the data capture and analysis in modern vision systems has led to a massive amount of data transfer between the end devices and cloud computers, resulting in long latency, slow response, and high power consumption. Efficient hardware architectures are under focused development to enable Artificial Intelligence (AI) at the resource-limited end sensing devices. One of the most promising solutions is to enable Processing-in-Pixel (PIP) scheme. However, the conventional schemes suffer from the low fill-factor issue. This paper proposes a PIP based CMOS sensor architecture, which allows convolution operation before the column readout circuit to significantly improve the image reading speed with much lower power consumption. The simulation results show that the proposed architecture could support the computing efficiency up to 11.65 TOPS/W at the 8-bit weight configuration, which is three times as high as the conventional schemes. The transistors required for each pixel are only 2.5T, significantly improving the fill-factor.

A Reconfigurable Convolution-in-Pixel CMOS Image Sensor Architecture.

The separation of the data capture and analysis in modern vision systems has led to a massive amount of data transfer between the end devices and cloud computers, resulting in long latency, slow response, and high power consumption. Efficient hardware architectures are under focused development to enable Artificial Intelligence (AI) at the resource-limited sensing devices. One of the most promising solutions is to enable Processing-in-Pixel (PIP) scheme. However, the conventional schemes suffer from the low fill-factor issue. This paper proposes a PIP based Complementary Metal-Oxide-Semiconductor (CMOS) sensor architecture, which allows convolution operation before the column readout circuit to significantly reduce the overall power consumption while improving the resource utilization of the succeeding deep learning accelerator. The simulation results show that the proposed architecture could support the computing efficiency up to 3.37 TOPS/W at the 8-bit weight configuration, which is four times as high as the conventional schemes after normalization. The transistors required for each pixel are only 3.5T, significantly improving the fill-factor. 

A Reconfigurable Convolution-in-Pixel CMOS Image Sensor Architecture

Significant growth in the development and deployment of artificial intelligence (AI) is being witnessed. Driven by the great versatility of emerging computer science and material science, various AI sensors provide cost-effective approaches for a wide range of monitoring applications toward the realization of smart homes and personal healthcare. Advanced AI sensors have multiple sensors capable of detecting multidimensional information and human-brain-like computation device for data processing. Herein, this review outlines the recent advances in the development of AI sensors. This review first introduces the materials, fabrication methods, and algorithms of current AI sensors and their applications, i.e., complementary metal oxide semiconductor image sensors for computer vision, microelectromechanical systems, microphone sensors for voice recognition, and wearable sensors for gesture recognition. Then, the recent advances in AI wearables sensors and self-powered sensor systems are highlighted. Next, the current developments of neuromorphic computing systems, multimodality, and digital twins are reviewed. Last, a perspective on future directions for further research development is also provided. In summary, the trend of advanced AI sensors is the complementary between edge computing and cloud computing, which will show great potential in the applications of smart buildings, individual healthcare, the Internet of things, etc. 

https://onlinelibrary.wiley.com/doi/pdfdirect/10.1002/adsr.202200072

Recent Advances in Artificial Intelligence Sensors

An adaptive readout scheme for low-power CMOS image sensors is proposed in this paper. In the first step, the entire pixel array is partitioned into square blocks. Subsequently, the intensity range of each block is divided into a fix number of bins which are adapted to the intensity range of the block. For this purpose, the maximum and minimum intensity values in each block are obtained using winner take all circuit.The proposed scheme is implemented and simulated on standard test images. The observed peak signal-to-noise ratio for the Cameraman and Thermal image is 40.37 dB and 41.16 dB, respectively for 8×8 block size and 16 bins, with 54% saving in the total power compared to the conventional readout. An additional power saving can be observed by reducing the total number of threshold levels.

CMOS image sensor with adaptive readout scheme for low power applications

CMOS Image Sensors (CIS) are fundamental to emerging visual computing applications. While conventional CIS are purely imaging devices for capturing images, increasingly CIS integrate processing capabilities such as Deep Neural Network (DNN). Computational CIS expand the architecture design space, but to date no comprehensive energy model exists. This paper proposes CamJ, a detailed energy modeling framework that provides a component-level energy breakdown for computational CIS and is validated against nine recent CIS chips. We use CamJ to demonstrate three use-cases that explore architectural trade-offs including computing in vs. off CIS, 2D vs. 3D-stacked CIS design, and analog vs. digital processing inside CIS. The code of CamJ is available at: https://github.com/horizon-research/CamJ.

CAMJ: Enabling System-Level Energy Modeling and Architectural Exploration for In-Sensor Visual Computing

A compressive acquisition technique for on-array image compression is proposed in this paper. It capitalizes on representation ability of accumulated spatial gradients of the acquired scene. The local variations inferred from strength of the accumulated gradients are used as cues to vary number of samples read through the image sensor readout. Such sampling enables the reconstruction using traditional interpolation techniques with desired quality. The proposed method is first verified using MATLAB simulations, where on an average, a compression of 87% is achieved, for a threshold of 40 intensity levels. The images are reconstructed using nearest neighbour interpolation (NNI) method which results in a mean peak signal to noise ratio (PSNR) value of 29.09 dB. The reconstructed images are further enhanced using deep convolutional neural network, which improves the PSNR to 32.46 dB. The biggest advantage of the proposed technique is low-complex hardware design. As a proof of concept, a hardware implementation of the technique is performed using discrete components. Pixel intensity values of standard images are converted into analog voltages using a data acquisition system and mapped in the input voltage range of 1.5 V −5.5 V. For a threshold of 3.8 V, the compression of 81% - 83% is observed for the considered images. The proposed technique is simple and effective, and is suitable for low-power complementary metal oxide semiconductor (CMOS) image sensors.

K. M. Amogh

Papers

On-Array Compressive Acquisition in CMOS Image Sensors Using Accumulated Spatial Gradients