The organization of behavior: A neuropsychological theory

In this manuscript, new quadrant-based search algorithm with zero motion prejudgment is proposed for motion estimation (ME) in HEVC (High Efficiency Video Coding) standard. The HEVC standard is used to obtain efficient output with low motion estimation time. The proposed quadrant-based search algorithm is a fast block matching algorithm that obtain better block matching amid the current block and reference block. The zero motion prejudgment (ZMP) method is used to find the block, whether it is motion or static and it is used for decreasing the computational complexity (CC) in the proposed quadrant-based search algorithm. The proposed quadrant-based search algorithm with ZMP technique for motion estimation in HEVC is implemented on the FPGA hardware platform. The entire architecture is executed in Verilog HDL with Virtex-5 technology and integrated with Xilinx ISE Design Suite 14.5. The results are integrated into the CIF (352 × 288 pixels) and HD (1280 × 720 pixels) video input sequence. The evaluation metrics like PSNR, Motion estimation time, sum of absolute difference (SAD) value are analyzed with existing method like hexagon, adaptive root pattern algorithm, and diamond search algorithm. Then the hardware parameters like power consumption and maximum operating frequency are measured. The hardware utilization is reduced and the power consumption of the proposed model is diminished to 0.143 W. The maximal operating frequency of the proposed model is 440.470 MHz. The experimental outcomes demonstrate that the proposed motion evaluation approach in HEVC is more effective than existing algorithms.

Correction to: An Efficient VLSI Architecture for Fast Motion Estimation Exploiting Zero Motion Prejudgment Technique and a New Quadrant-Based Search Algorithm in HEVC

Analog hardware accelerators, which perform computation within a dense memory array, have the potential to overcome the major bottlenecks faced by digital hardware for data-heavy workloads such as deep learning. Exploiting the intrinsic computational advantages of memory arrays, however, has proven to be challenging principally due to the overhead imposed by the peripheral circuitry and due to the non-ideal properties of memory devices that play the role of the synapse. We review the existing implementations of these accelerators for deep supervised learning, organizing our discussion around the different levels of the accelerator design hierarchy, with an emphasis on circuits and architecture. We explore and consolidate the various approaches that have been proposed to address the critical challenges faced by analog accelerators, for both neural network inference and training, and highlight the key design trade-offs underlying these techniques.

Analog architectures for neural network acceleration based on non-volatile memory

In this article, we review the recent progress of ferroelectric field-effect transistors (FeFETs) based on ferroelectric hafnium oxide (HfO2), ten years after the first report on such a device. With a focus on the use of FeFET for nonvolatile memory application, we discuss its basic operation principles, switching mechanisms, device types, material properties and array structures. Key device performance metrics such as cycling endurance, retention, memory window, multi-level operation and scaling capability are analyzed. We also briefly survey recent developments in alternative applications for FeFETs including neuromorphic and in-memory computing as well as radiofrequency devices.

Ferroelectric field-effect transistors based on HfO2: a review.

We review camera architecture in the age of artificial intelligence. Modern cameras use physical components and software to capture, compress and display image data. Over the past 5 years, deep learning solutions have become superior to traditional algorithms for each of these functions. Deep learning enables 10-100x reduction in electrical sensor power per pixel, 10x improvement in depth of field and dynamic range and 10-100x improvement in image pixel count. Deep learning enables multiframe and multiaperture solutions that fundamentally shift the goals of physical camera design. Here we review the state of the art of deep learning in camera operations and consider the impact of AI on the physical design of cameras.

Smart Cameras

This paper presents a ferroelectric FET (FeFET)-based processing-in-memory (PIM) architecture to accelerate the inference of deep neural networks (DNNs). We propose a digital in-memory vector-matrix multiplication (VMM) engine design utilizing the FeFET crossbar to enable bit-parallel computation and eliminate analog-to-digital conversion in prior mixed-signal PIM designs. A dedicated hierarchical network-on-chip (H-NoC) is developed for input broadcasting and on-the-fly partial results processing, reducing the data transmission volume and latency. Simulations in 28-nm CMOS technology show $115\times $ and $6.3\times $ higher computing efficiency (GOPs/W) over desktop GPU (Nvidia GTX 1080Ti) and resistive random access memory (ReRAM)-based design, respectively.

A Ferroelectric FET-Based Processing-in-Memory Architecture for DNN Acceleration

Spatio-temporal dynamics of physical processes are generally modeled using partial differential equations (PDEs). Though the core dynamics follows some principles of physics, real-world physical processes are often driven by unknown external sources. In such cases, developing a purely analytical model becomes very difficult and data-driven modeling can be of assistance. In this paper, we present a hybrid framework combining physics-based numerical models with deep learning for source identification and forecasting of spatio-temporal dynamical systems with unobservable time-varying external sources. We formulate our model PhICNet as a convolutional recurrent neural network (RNN) which is end-to-end trainable for spatio-temporal evolution prediction of dynamical systems and learns the source behavior as an internal state of the RNN. Experimental results show that the proposed model can forecast the dynamics for a relatively long time and identify the sources as well.

Physics-Incorporated Convolutional Recurrent Neural Networks for Source Identification and Forecasting of Dynamical Systems

With recent advances in sensor technology, multispectral systems are becoming increasingly attractive for intelligence, surveillance, and reconnaissance applications. Fusing information from multiple imaging modalities is a major task for such systems. Combining feature maps obtained from multiple deep neural network pipelines demonstrates promising performance for object detection and tracking. However, feature fusion using multiple deep networks is computationally intensive and therefore not suitable for resource-constrained IoT edge devices. In this paper, we propose a novel method to fuse the input space to enable processing of multispectral data via a single deep network. We use task-driven feedback as a reward signal for our reinforcement learning-based multispectral input fusion. Proposed approach not only improves tracking accuracy but also maximizes modality-specific information as intended by the user.

Multispectral Information Fusion With Reinforcement Learning for Object Tracking in IoT Edge Devices

Round-the-clock surveillance requires robust object detection and tracking independent of lighting conditions. Fusing information from visual-infrared object detection network pair at feature level or decision level shows promising accuracy. However, such fused object detection network is not suitable for edge devices with limited processing power and memory. In this paper, we propose a technique to control spatial modality using feedback from the object detection network and create a mixed-modality image by eliminating the redundancy between visual and infrared information. Mixed-modality image enables object tracking with a single deep neural network as opposed to the decision- level fusion with two separate networks for visual image and infrared image. Proposed approach achieves at least 8% better object tracking accuracy than decision-level fusion while operating at 2X frame-rate and consuming 50% less energy.

Adaptive Control of Camera Modality with Deep Neural Network-Based Feedback for Efficient Object Tracking

Today’s cameras are designed to approximate what they observe in a manner that preserves entropy. However, time critical autonomous applications such as autonomous driving, surveillance and defense systems require task critical information at the highest quality. With rapid advances in frame rates and resolutions, observing scenes at the highest quality raises concerns for the transmission bandwidth. In this paper, we introduce a new paradigm of smart camera that captures only task-critical information at the highest quality. Embedded deep neural network (DNN) algorithms within the camera enhance quality of information through real-time control of sensor parameters. We show the hardware feasibility of this camera by demonstrating a 3D-stacked architecture with a Digital Pixel Sensor (DPS). We demonstrate a number of high-level vision applications that benefit through this task-guided control including object detection, object tracking and activity recognition. Finally, we present the unique challenges faced created as a result of feedback and show how software/hardware innovations can be used to mitigate them.

Priyabrata Saha

Papers

A Ferroelectric FET-Based Processing-in-Memory Architecture for DNN Acceleration

Physics-Incorporated Convolutional Recurrent Neural Networks for Source Identification and Forecasting of Dynamical Systems

Multispectral Information Fusion With Reinforcement Learning for Object Tracking in IoT Edge Devices

Adaptive Control of Camera Modality with Deep Neural Network-Based Feedback for Efficient Object Tracking

CAMEL: An Adaptive Camera With Embedded Machine Learning-Based Sensor Parameter Control