As a bio-inspired and emerging sensor, an event-based neuromorphic vision sensor has a different working principle compared to the standard frame-based cameras, which leads to promising properties of low energy consumption, low latency, high dynamic range (HDR), and high temporal resolution It poses a paradigm shift to sense and perceive the environment by capturing local pixel-level light intensity changes and producing asynchronous event streams Advanced technologies for the visual sensing system of autonomous vehicles from standard computer vision to event-based neuromorphic vision have been developed In this tutorial-like article, a comprehensive review of the emerging technology is given First, the course of the development of the neuromorphic vision sensor that is derived from the understanding of biological retina is introduced The signal processing techniques for event noise processing and event data representation are then discussed Next, the signal processing algorithms and applications for event-based neuromorphic vision in autonomous driving and various assistance systems are reviewed Finally, challenges and future research directions are pointed out It is expected that this article will serve as a starting point for new researchers and engineers in the autonomous driving field and provide a bird's-eye view to both neuromorphic vision and autonomous driving research communities

Event-Based Neuromorphic Vision for Autonomous Driving: A Paradigm Shift for Bio-Inspired Visual Sensing and Perception

A critical challenge in neuromorphic computing is to present computationally efficient algorithms of learning. When implementing gradient-based learning, error information must be routed through the network, such that each neuron knows its contribution to output, and thus how to adjust its weight. This is known as the credit assignment problem. Exactly implementing a solution like backpropagation involves weight sharing, which requires additional bandwidth and computations in a neuromorphic system. Instead, models of learning from neuroscience can provide inspiration for how to communicate error information efficiently, without weight sharing. Here we present a novel dendritic event-based processing (DEP) algorithm, using a two-compartment leaky integrate-and-fire neuron with partially segregated dendrites that effectively solves the credit assignment problem. In order to optimize the proposed algorithm, a dynamic fixed-point representation method and piecewise linear approximation approach are presented, while the synaptic events are binarized during learning. The presented optimization makes the proposed DEP algorithm very suitable for implementation in digital or mixed-signal neuromorphic hardware. The experimental results show that spiking representations can rapidly learn, achieving high performance by using the proposed DEP algorithm. We find the learning capability is affected by the degree of dendritic segregation, and the form of synaptic feedback connections. This study provides a bridge between the biological learning and neuromorphic learning, and is meaningful for the real-time applications in the field of artificial intelligence.

/pdf/efficient-spike-driven-learning-with-dendritic-event-based-3w52pztv0c.pdf

Efficient Spike-Driven Learning With Dendritic Event-Based Processing.

The role of shared autonomous vehicle systems in delivering smart urban mobility: A systematic review of the literature

Objective: To realize the full coverage, full spectrum, and full application, of 6G networks, the channel measurement, channel characteristics, and channel research of the 6G-oriented full-spectrum full-scene wireless network are explored. Methods: At present, constructing an information network covering three dimensions of sea, land and air is one of the research directions of 6G technology development. Meanwhile, the full coverage of three dimensions of communication requires multiple channels including multiple frequency bands and multiple scenarios, mainly including terahertz, light band, satellite, unmanned aerial vehicle, ocean, high-speed rail, and vehicle-to-vehicle. For different channels, different parameters are analyzed and targeted discussed according to channel characteristics. Also, there are many other problems of wireless communication channel that need to be solved. Here, only the above problems are analyzed in detail. Because of the current development of the Internet of Vehicles, the interconnection of connected vehicles is analyzed, and this scenario is also the most widely common scenario in daily life. Results: For the above-mentioned communication channels, the relevant measurement and modeling results of the 6G channel are shown; besides, the characteristics, time, and space correlation functions are obtained for each channel. Conclusion: The existing research results of 6G-oriented wireless channel measurement and modeling are analyzed; the research methods are summarized; new ideas are proposed to provide researchers with important references.

6G-Enabled Network in Box for Internet of Connected Vehicles

Human motion intention recognition is a key to achieve perfect human-machine coordination and wearing comfort of wearable robots. Surface electromyography (sEMG), as a bioelectrical signal, generates prior to the corresponding motion and reflects the human motion intention directly. Thus, a better human-machine interaction can be achieved by using sEMG based motion intention recognition. In this paper, we review and discuss the state of the art of the sEMG based motion intention recognition that is mainly used in detail. According to the method adopted, motion intention recognition is divided into two groups: sEMG-driven musculoskeletal (MS) model based motion intention recognition and machine learning (ML) model based motion intention recognition. The specific models and recognition effects of each study are analyzed and systematically compared. Finally, a discussion of the existing problems in the current studies, major advances, and future challenges is presented.

/pdf/semg-based-human-motion-intention-recognition-1ax1nhxmdf.pdf

sEMG Based Human Motion Intention Recognition

Neuromorphic vision sensor is a new passive sensing modality and a frameless sensor with a number of advantages over traditional cameras. Instead of wastefully sending entire images at fixed frame rate, neuromorphic vision sensor only transmits the local pixel-level changes caused by the movement in a scene at the time they occur. This results in advantageous characteristics, in terms of low energy consumption, high dynamic range, sparse event stream, and low response latency, which can be very useful in intelligent perception systems for modern intelligent transportation system (ITS) that requires efficient wireless data communication and low power embedded computing resources. In this paper, we propose the first neuromorphic vision based multivehicle detection and tracking system in ITS. The performance of the system is evaluated with a dataset recorded by a neuromorphic vision sensor mounted on a highway bridge. We performed a preliminary multivehicle tracking-by-clustering study using three classical clustering approaches and four tracking approaches. Our experiment results indicate that, by making full use of the low latency and sparse event stream, we could easily integrate an online tracking-by-clustering system running at a high frame rate, which far exceeds the real-time capabilities of traditional frame-based cameras. If the accuracy is prioritized, the tracking task can also be performed robustly at a relatively high rate with different combinations of algorithms. We also provide our dataset and evaluation approaches serving as the first neuromorphic benchmark in ITS and hopefully can motivate further research on neuromorphic vision sensors for ITS solutions.

/pdf/neuromorphic-vision-based-multivehicle-detection-and-3jtixoenp9.pdf

Neuromorphic Vision Based Multivehicle Detection and Tracking for Intelligent Transportation System

Neuromorphic vision sensors are bio-inspired cameras that naturally capture the dynamics of a scene with ultra-low latency, filtering out redundant information with low power consumption. Few works are addressing the object detection with this sensor. In this work, we propose to develop pedestrian detectors that unlock the potential of the event data by leveraging multi-cue information and different fusion strategies. To make the best out of the event data, we introduce three different event-stream encoding methods based on Frequency, Surface of Active Event (SAE) and Leaky Integrate-and-Fire (LIF). We further integrate them into the state-of-the-art neural network architectures with two fusion approaches: the channel-level fusion of the raw feature space and decision-level fusion with the probability assignments. We present a qualitative and quantitative explanation why different encoding methods are chosen to evaluate the pedestrian detection and which method performs the best. We demonstrate the advantages of the decision-level fusion via leveraging multi-cue event information and show that our approach performs well on a self-annotated event-based pedestrian dataset with 8,736 event frames. This work paves the way of more fascinating perception applications with neuromorphic vision sensors.

/pdf/multi-cue-event-information-fusion-for-pedestrian-detection-4x2yivez0g.pdf

Multi-Cue Event Information Fusion for Pedestrian Detection With Neuromorphic Vision Sensors.

We demonstrate an extended Kalman filtering-enhanced linewidth measurement in short-delay self-heterodyne interferometry (SDSHI). We found that a modified SDSHI trace closely resembles a biased cosine wave, which would enable convenient linewidth estimation by its uniform envelope contrast without any correction factor. Experimentally, we adopted this approach for kHz laser linewidth measurement, taking advantages of extended Kalman filtering (EKF) to adaptively track the cosine wave. Apart from the measurement noise suppression, this approach could use as many data points as possible in the noisy trace to make a linewidth estimation at each tracked data point, from which we can deduce valuable statistical parameters such as the mean and standard deviation. This approach involves no more equipment than conventional SDSHI and sophisticated EKF so that it can be easily implemented. Therefore, we believe it will find wide applications in ultra-narrow laser linewidth measurement.

Xingbo Liu

Papers

Neuromorphic Vision Based Multivehicle Detection and Tracking for Intelligent Transportation System

Multi-Cue Event Information Fusion for Pedestrian Detection With Neuromorphic Vision Sensors.

Kalman filtering-enhanced short-delay self-heterodyne interferometry for linewidth measurement.