Developing high performance embedded vision applications requires balancing run-time performance with energy constraints. Given the mix of hardware accelerators that exist for embedded computer vision (e.g. multi-core CPUs, GPUs, and FPGAs), and their associated vendor optimized vision libraries, it becomes a challenge for developers to navigate this fragmented solution space. To aid with determining which embedded platform is most suitable for their application, we conduct a comprehensive benchmark of the run-time performance and energy efficiency of a wide range of vision kernels. We discuss rationales for why a given underlying hardware architecture innately performs well or poorly based on the characteristics of a range of vision kernel categories. Specifically, our study is performed for three commonly used HW accelerators for embedded vision applications: ARM57 CPU, Jetson TX2 GPU and ZCU102 FPGA, using their vendor optimized vision libraries: OpenCV, VisionWorks and xfOpenCV. Our results show that the GPU achieves an energy/frame reduction ratio of 1.1–3.2× compared to the others for simple kernels. While for more complicated kernels and complete vision pipelines, the FPGA outperforms the others with energy/frame reduction ratios of 1.2–22.3×. It is also observed that the FPGA performs increasingly better as a vision application's pipeline complexity grows.

https://lib.dr.iastate.edu/cgi/viewcontent.cgi?article=1219&context=ece_pubs

Comparing Energy Efficiency of CPU, GPU and FPGA Implementations for Vision Kernels

Vehicle detection and classification has been an area of application of image processing and machine learning which is being researched extensively in accordance with its importance due to increasing number of vehicles, traffic rule defaulters and accidents. This paper aims to review various methodologies used and how it has evolved to give better results in the past years, closely moving towards usage of machine learning. This has resulted in advancing the problem statement towards helmet detection followed by number plate detection of defaulters. Object detection and Text recognition that are available in various frameworks offer built-in models which are easy to use or offer easy methods to build and train customized models.

A review on various methodologies used for vehicle classification, helmet detection and number plate recognition

https://lib.dr.iastate.edu/cgi/viewcontent.cgi?article=1264&context=ece_pubs

Benchmarking vision kernels and neural network inference accelerators on embedded platforms

Developing high performance embedded vision applications requires balancing run-time performance with energy constraints. Given the mix of hardware accelerators that exist for embedded computer vision (e.g. multi-core CPUs, GPUs, and FPGAs), and their associated vendor optimized vision libraries, it becomes a challenge for developers to navigate this fragmented solution space. To aid with determining which embedded platform is most suitable for their application, we conduct a comprehensive benchmark of the run-time performance and energy efficiency of a wide range of vision kernels. We discuss rationales for why a given underlying hardware architecture innately performs well or poorly based on the characteristics of a range of vision kernel categories. Specifically, our study is performed for three commonly used HW accelerators for embedded vision applications: ARM57 CPU, Jetson TX2 GPU and ZCU102 FPGA, using their vendor optimized vision libraries: OpenCV, VisionWorks and xfOpenCV. Our results show that the GPU achieves an energy/frame reduction ratio of 1.1-3.2x compared to the others for simple kernels. While for more complicated kernels and complete vision pipelines, the FPGA outperforms the others with energy/frame reduction ratios of 1.2-22.3x. It is also observed that the FPGA performs increasingly better as a vision application's pipeline complexity grows.

Mobile robotics applications process large amounts of data in real-time and require compute platforms that provide high performance and energy-efficiency. FPGAs are well-suited for many of these applications, but there is a reluctance in the robotics community to use hardware acceleration due to increased design complexity and a lack of consistent programming models across the software/hardware boundary. In this paper we present ReconROS, a framework that integrates the widely-used robot operating system (ROS) with ReconOS, which features multithreaded programming of hardware and software threads for reconfigurable computers. This unique combination gives ROS2 developers the flexibility to transparently accelerate parts of their robotics applications in hardware. We elaborate on the architecture and the design flow for ReconROS and report on a set of experiments that underline the feasibility and flexibility of our approach.

ReconROS: Flexible Hardware Acceleration for ROS2 Applications

The rapidly growing applications based on morphological operations in image processing and computer vision make efficient implementations of these key blocks an important topic of research. Nevertheless, a detailed comparison of the energy efficiency and performance of these implementations that covers all available major hardware platforms is still missing. In this paper we evaluate the performance and power consumption of the most efficient available morphological image processing algorithms for CPU, GPU, and FPGA platforms in detail. In addition, we study the suitability of available morphological library units for high-level synthesis and compare the results with an optimized hand-coded FPGA implementation. We demonstrate that even high-end GPUs cannot achieve the throughputs of modern CPUs and FPGAs by far. Our experimental results show that an FPGA implementation is 8–10 times more energy efficient for this application, being comparable in speed to CPUs for large kernels.

A quantitative cross-architecture study of morphological image processing on CPUs, GPUs, and FPGAs

For many years, morphological operations have been widely used as image processing techniques for various applications. However, the rapidly growing computational complexity due to processing of large structuring elements for high resolution images, challenges the limits of conventional processing platforms. Nowadays, hybrid field programmable gate arrays (FPGAs) have shown a promising approach for such high performance applications. These devices have compelling advantages over traditional platforms due to their low power consumption and high-customization. However, the efficiency comes at the cost of loss in flexibility and adaptability compared to software solutions. Moreover, morphological operations require significant memory resources. In this paper, we present a dedicated IP block, configurable for different morphological operations, size and origin of structuring elements. This IP core is optimized for low power, small resource consumption and high throughput for the application in an industry 4.0 environment. Furthermore, the image size and radiometric resolutions can be configured to meet the requirements of flexibility. Our new architecture achieves a high throughput comparable to the state-of-the-art, while providing a high flexibility.

A Reconfigurable Accelerator for Morphological Operations

In recent years, connected component analysis (CCA) has become one of the vital image/video processing algorithms due to its wide-range applicability in the field of computer vision. Numerous applications such as pattern recognition, object detection and image segmentation involve connected component analysis. In the context of camera-based inspection systems, CCA plays an important role for quality assurance. State-of-the-art hardware architectures offer high performance implementations of CCA using field programmable gate arrays (FPGAs). However, due to their high memory-demand, most of these implementations inhibit a large resource utilization. In this paper, we propose a hybrid software-hardware architecture of CCA for an industrial application using Xilinx Zynq-7000 All Programmable System on Chip (SoC). By offloading the most resource consuming part of the algorithm to the embedded CPU, we achieved high performance, while reducing the required resources on the FPGA. Our proposed architecture saves more than 30% of on-chip memory (Block RAMs) compared to state-of-the-art hardware architectures without affecting the throughput. Furthermore, due to the embedded CPU, our system provides a versatile and highly flexible feature extraction at run-time without the necessity to reconfigure the FPGA.

Martin Klein

Papers

A quantitative cross-architecture study of morphological image processing on CPUs, GPUs, and FPGAs

A Reconfigurable Accelerator for Morphological Operations

An advanced embedded architecture for connected component analysis in industrial applications