As a bio-inspired and emerging sensor, an event-based neuromorphic vision sensor has a different working principle compared to the standard frame-based cameras, which leads to promising properties of low energy consumption, low latency, high dynamic range (HDR), and high temporal resolution It poses a paradigm shift to sense and perceive the environment by capturing local pixel-level light intensity changes and producing asynchronous event streams Advanced technologies for the visual sensing system of autonomous vehicles from standard computer vision to event-based neuromorphic vision have been developed In this tutorial-like article, a comprehensive review of the emerging technology is given First, the course of the development of the neuromorphic vision sensor that is derived from the understanding of biological retina is introduced The signal processing techniques for event noise processing and event data representation are then discussed Next, the signal processing algorithms and applications for event-based neuromorphic vision in autonomous driving and various assistance systems are reviewed Finally, challenges and future research directions are pointed out It is expected that this article will serve as a starting point for new researchers and engineers in the autonomous driving field and provide a bird's-eye view to both neuromorphic vision and autonomous driving research communities

Event-Based Neuromorphic Vision for Autonomous Driving: A Paradigm Shift for Bio-Inspired Visual Sensing and Perception

The role of shared autonomous vehicle systems in delivering smart urban mobility: A systematic review of the literature

Objective: To realize the full coverage, full spectrum, and full application, of 6G networks, the channel measurement, channel characteristics, and channel research of the 6G-oriented full-spectrum full-scene wireless network are explored. Methods: At present, constructing an information network covering three dimensions of sea, land and air is one of the research directions of 6G technology development. Meanwhile, the full coverage of three dimensions of communication requires multiple channels including multiple frequency bands and multiple scenarios, mainly including terahertz, light band, satellite, unmanned aerial vehicle, ocean, high-speed rail, and vehicle-to-vehicle. For different channels, different parameters are analyzed and targeted discussed according to channel characteristics. Also, there are many other problems of wireless communication channel that need to be solved. Here, only the above problems are analyzed in detail. Because of the current development of the Internet of Vehicles, the interconnection of connected vehicles is analyzed, and this scenario is also the most widely common scenario in daily life. Results: For the above-mentioned communication channels, the relevant measurement and modeling results of the 6G channel are shown; besides, the characteristics, time, and space correlation functions are obtained for each channel. Conclusion: The existing research results of 6G-oriented wireless channel measurement and modeling are analyzed; the research methods are summarized; new ideas are proposed to provide researchers with important references.

6G-Enabled Network in Box for Internet of Connected Vehicles

Hand gesture recognition is an important way for human machine interaction, and it is widely used in many areas, such as health care, smart home, virtual reality as well as other areas. While many valuable efforts have been made, it still lacks efficient ways to capture fine grain hand gesture as well as track data of long distance dependency in complex gesture. In this paper, we firstly design a novel data glove with two arm rings and a specially integrated three-dimensional flex sensor to capture fine grain motion from full arm and all knuckles. Secondly, an improved deep feature fusion network is proposed to detect long distance dependency in complex hand gestures. In order to track detailed motion features, a convolutional neural network based feature fusion strategy is given to fuse data from multi-sensors by extracting both shallow and deep features. Moreover, a residual module is introduced to avoid over fitting and gradient vanishing during deepening the neural network. Thirdly, a long short term memory (LSTM) model with fused feature vector as input is introduced to classify complex hand motions into corresponding categories. Results of comprehensive experiments demonstrate that our work performs better than related algorithms, especially in America sign language (with the precise of 99.93%) and Chinese sign language (with the precise of 96.1%).

Hand Gesture Recognition Using Deep Feature Fusion Network Based on Wearable Sensors

In this paper, we introduce a framework for dynamic gesture recognition with background suppression operating on the output of a moving event-based camera. The system is developed to operate in real-time using only the computational capabilities of a mobile phone. It introduces a new development around the concept of time-surfaces. It also presents a novel event-based methodology to dynamically remove backgrounds that uses the high temporal resolution properties of event-based cameras. To our knowledge, this is the first Android event-based framework for vision-based recognition of \textit{dynamic} gestures running on a smartphone without off-board processing. We assess the performances by considering several scenarios in both indoors and outdoors, for static and dynamic conditions, in uncontrolled lighting conditions. We also introduce a new event-based dataset for gesture recognition with static and dynamic backgrounds (made publicly available). The set of gestures has been selected following a clinical trial to allow human-machine interaction for the visually impaired and older adults. We finally report comparisons with prior work that addressed event-based gesture recognition reporting comparable results, without the use of advanced classification techniques nor power greedy hardware.

/pdf/event-based-gesture-recognition-with-dynamic-background-2k1rcwfr6t.pdf

Event-Based Gesture Recognition With Dynamic Background Suppression Using Smartphone Computational Capabilities.

Neuromorphic vision sensor is a new passive sensing modality and a frameless sensor with a number of advantages over traditional cameras. Instead of wastefully sending entire images at fixed frame rate, neuromorphic vision sensor only transmits the local pixel-level changes caused by the movement in a scene at the time they occur. This results in advantageous characteristics, in terms of low energy consumption, high dynamic range, sparse event stream, and low response latency, which can be very useful in intelligent perception systems for modern intelligent transportation system (ITS) that requires efficient wireless data communication and low power embedded computing resources. In this paper, we propose the first neuromorphic vision based multivehicle detection and tracking system in ITS. The performance of the system is evaluated with a dataset recorded by a neuromorphic vision sensor mounted on a highway bridge. We performed a preliminary multivehicle tracking-by-clustering study using three classical clustering approaches and four tracking approaches. Our experiment results indicate that, by making full use of the low latency and sparse event stream, we could easily integrate an online tracking-by-clustering system running at a high frame rate, which far exceeds the real-time capabilities of traditional frame-based cameras. If the accuracy is prioritized, the tracking task can also be performed robustly at a relatively high rate with different combinations of algorithms. We also provide our dataset and evaluation approaches serving as the first neuromorphic benchmark in ITS and hopefully can motivate further research on neuromorphic vision sensors for ITS solutions.

/pdf/neuromorphic-vision-based-multivehicle-detection-and-3jtixoenp9.pdf

Neuromorphic Vision Based Multivehicle Detection and Tracking for Intelligent Transportation System

Neuromorphic vision sensors such as the Dynamic and Active-pixel Vision Sensor (DAVIS) using silicon retina are inspired by biological vision, they generate streams of asynchronous events to indicate local log-intensity brightness changes. Their properties of high temporal resolution, low-bandwidth, lightweight computation, and low-latency make them a good fit for many applications of motion perception in the intelligent vehicle. However, as a younger and smaller research field compared to classical computer vision, neuromorphic vision is rarely connected with the intelligent vehicle. For this purpose, we present three novel datasets recorded with DAVIS sensors and depth sensor for the distracted driving research and focus on driver drowsiness detection, driver gaze-zone recognition, and driver hand-gesture recognition. To facilitate the comparison with classical computer vision, we record the RGB, depth and infrared data with a depth sensor simultaneously. The total volume of this dataset has 27360 samples. To unlock the potential of neuromorphic vision on the intelligent vehicle, we utilize three popular event-encoding methods to convert asynchronous event slices to event-frames and adapt state-of-the-art convolutional architectures to extensively evaluate their performances on this dataset. Together with qualitative and quantitative results, this work provides a new database and baseline evaluations named NeuroIV in cross-cutting areas of neuromorphic vision and intelligent vehicle.

NeuroIV: Neuromorphic Vision Meets Intelligent Vehicle Towards Safe Driving With a New Database and Baseline Evaluations

A neuromorphic vision sensor is a novel passive sensing modality and frameless sensors with several advantages over conventional cameras. Frame-based cameras have an average frame-rate of 30 fps, causing motion blur when capturing fast motion, e.g. hand gesture. Rather than wastefully sending entire images at a fixed frame rate, neuromorphic vision sensors only transmit the local pixel-level changes induced by the movement in a scene when they occur. This leads to advantageous characteristics, including low energy consumption, high dynamic range, a sparse event stream, and low response latency. In this study, a novel representation learning method was proposed: Fixed Length Gists Representation (FLGR) learning for event-based gesture recognition. Previous methods accumulate events into video frames in a temporal duration (e.g. 30 ms) to make the accumulated image-level representation. However, the accumulated-frame-based representation waives the friendly event-driven paradigm of neuromorphic vision sensors. New representation is urgently needed to fill the gap in non-accumulated-frame-based representation and exploit the further capabilities of neuromorphic vision. The proposed FLGR is a sequence learned from mixture density autoencoder and preserves nature of event-based data better. FLGR has a data format of fixed length, and it is easy to feed to sequence classifier. Moreover, an RNN-HMM hybrid was proposed to address the continuous gesture recognition problem. A recurrent neural network (RNN) was applied for FLGR sequence classification while a hidden Markov model(HMM) is employed for localizing the candidate gesture and improving the result in a continuous sequence. A neuromorphic continuous hand gestures dataset (Neuro ConGD Dataset) was developed with 17 hand gestures classes for the community of the neuromorphic research. Hopefully, FLGR can inspire the study on the event-based highly efficient, high-speed, and high-dynamic-range sequence classification tasks.

/pdf/flgr-fixed-length-gists-representation-learning-for-rnn-hmm-4hdjlyl6ar.pdf

Jieneng Chen

Papers

Neuromorphic Vision Based Multivehicle Detection and Tracking for Intelligent Transportation System

NeuroIV: Neuromorphic Vision Meets Intelligent Vehicle Towards Safe Driving With a New Database and Baseline Evaluations

FLGR: Fixed Length Gists Representation Learning for RNN-HMM Hybrid-Based Neuromorphic Continuous Gesture Recognition.