Radar, as one of the sensors for human activity recognition (HAR), has unique characteristics such as privacy protection and contactless sensing. Radar-based HAR has been applied in many fields such as human–computer interaction, smart surveillance and health assessment. Conventional machine learning approaches rely on heuristic hand-crafted feature extraction, and their generalization capability is limited. Additionally, extracting features manually is time–consuming and inefficient. Deep learning acts as a hierarchical approach to learn high-level features automatically and has achieved superior performance for HAR. This paper surveys deep learning based HAR in radar from three aspects: deep learning techniques, radar systems, and deep learning for radar-based HAR. Especially, we elaborate deep learning approaches designed for activity recognition in radar according to the dimension of radar returns (i.e., 1D, 2D and 3D echoes). Due to the difference of echo forms, corresponding deep learning approaches are different to fully exploit motion information. Experimental results have demonstrated the feasibility of applying deep learning for radar-based HAR in 1D, 2D and 3D echoes. Finally, we address some current research considerations and future opportunities.

https://www.mdpi.com/2072-4292/11/9/1068/pdf

A Survey of Deep Learning-Based Human Activity Recognition in Radar

Ultra-wideband (UWB) radar plays an important role in search and rescue at disaster relief sites. Identifying vital signs and locating buried survivors are two important research contents in this field. In general, it is hard to identify a human's vital signs (breathing and heartbeat) in complex environments due to the low signal-to-noise ratio of the vital sign in radar signals. In this paper, advanced signal-processing approaches are used to identify and to extract human vital signs in complex environments. First, we apply Curvelet transform to remove the source-receiver direct coupling wave and background clutters. Next, singular value decomposition is used to de-noise in the life signals. Finally, the results are presented based on FFT and Hilbert-Huang transform to separate and to extract human vital sign frequencies, as well as the micro-Doppler shift characteristics. The proposed processing approach is first tested by a set of synthetic data generated by FDTD simulation for UWB radar detection of two trapped victims under debris at an earthquake site of collapsed buildings. Then, it is validated by laboratory experiments data. The results demonstrate that the combination of UWB radar as the hardware and advanced signal-processing algorithms as the software has potential for efficient vital sign detection and location in search and rescue for trapped victims in complex environment.

Advanced Signal Processing for Vital Sign Extraction With Applications in UWB Radar Detection of Trapped Victims in Complex Environments

The ability to identify human movements can serve as an important tool in many different applications such as surveillance, military combat situations, search and rescue operations and patient monitoring in hospitals. This information can provide soldiers, security personnel and search and rescue workers with critical knowledge that can be used to potentially save lives and/or avoid dangerous situations. Most research involving human activity recognition employs the short-time Fourier transform (STFT) as a method of analysing human micro-Doppler signatures. However, the STFT has time-frequency resolution limitations and Fourier transform-based methods are not well-suited for use with non-stationary and non-linear signals. The authors approach uses the empirical mode decomposition to produce a unique feature vector from the human micro-Doppler signals following which a support vector machine is used to classify human motions. This study presents simulations of simple human motions, which are subsequently validated using experimental data obtained from both an S-band radar and a W-band millimetre wave (mm-wave) radar. Very good classification accuracies are obtained at distances of up to 90 m between the human and the radar.

Classification of human motions using empirical mode decomposition of human micro-Doppler signatures

This paper presents a framework based on multi-layer bi-LSTM network (bidirectional Long Short-Term Memory) for multimodal sensor fusion to sense and classify daily activities’ patterns and high-risk events such as falls. The data collected in this work are continuous activity streams from FMCW radar and three wearable inertial sensors on the wrist, waist, and ankle. Each activity has a variable duration in the data stream so that the transitions between activities can happen at random times within the stream, without resorting to conventional fixed-duration snapshots. The proposed bi-LSTM implements soft feature fusion between wearable sensors and radar data, as well as two robust hard-fusion methods using the confusion matrices of both sensors. A novel hybrid fusion scheme is then proposed to combine soft and hard fusion to push the classification performances to approximately 96% accuracy in identifying continuous activities and fall events. These fusion schemes implemented with the proposed bi-LSTM network are compared with conventional sliding window approach, and all are validated with realistic “leaving one participant out” (L1PO) method (i.e. testing subjects unknown to the classifier). The developed hybrid-fusion approach is capable of stabilizing the classification performance among different participants in terms of reducing accuracy variance of up to 18.1% and increasing minimum, worst-case accuracy up to 16.2%.

/pdf/bi-lstm-network-for-multimodal-continuous-human-activity-1wmvb2lzkf.pdf

Bi-LSTM Network for Multimodal Continuous Human Activity Recognition and Fall Detection

Micro-Doppler signals refer to Doppler scattering returns produced by the motions of the target other than gross translation. The small micro-motions of a subject, and even just parts of a subject, can be observed through the micro-Doppler signature it creates in response to an active emitter such as a radar, laser, and even acoustic emitters. These micro-Doppler signatures are produced by the kinematic properties of the subject's motion and can be used to extract the salient features of the subject's motion, and often, identify the subject. The rapidly declining cost of micro-Doppler-capable active sensors like radar with their dramatically improving capabilities, provide significant motivation in developing micro-Doppler techniques that can improve the exploitation of these sensors. Micro-Doppler techniques aim at extracting the micro-motion of the subject that may be unique to a particular subject class or activity in order to distinguish probable false alarms from real detections, as well as to increase the value of the information extracted from the sensor. The source of micro-motion depends on the subject and can be a rotating propeller on a fixed-wing aircraft, the multiple spinning rotor blades of a helicopter, or an unmanned aerial vehicle (UAV); the vibrations of an engine shaking a vehicle; an antenna rotating on a ship; the flapping wings of birds; the swinging arms and legs of a walking person; and many other sources. Confuser detections, such as birds for UAVs or animals for humans, can be interpreted as false alarms for a sensor system, so using the available micro-Doppler returns for classification can significantly reduce the sensor false alarm rate, thereby improving the utility of the sensor system. This study reviews the current research in micro-Doppler based on subject type, sensor capabilities, as well as environmental effects, and then proposes future research areas for micro-Doppler.

https://ietresearch.onlinelibrary.wiley.com/doi/pdf/10.1049/iet-rsn.2015.0118

Review of micro-Doppler signatures

We have developed an ultrawideband (UWB) random noise radar for through-wall surveillance applications. The operating frequency is in the ultrahigh frequency range, and the entire system is built around the concept of software defined radio. The radar receiver performance is statistically evaluated using both simulation studies and actual measurement results. We also discuss the phenomena of interference level and radar cross section (RCS) of the human target using the receiver operating characteristics (ROC). Experimental results presented show that random noise radars are useful for detecting and tracking humans obscured by building walls.

Ultrawideband Random Noise Radar Design for Through-Wall Surveillance

We present the design and implementation of a portable, digital, real-time random noise radar system operating in the ultrahigh frequency range for through-the-wall detection and imaging. Noise radar technology is combined with modern digital signal processing approaches to architect a system to covertly perform range imaging of obscured stationary and moving targets as well as to detect the presence of humans via micro-Doppler detection combined with empirical mode decomposition. We model the propagation and sampling nonidealities in the system and propose techniques to overcome the effect of these nonidealities. Experimental results demonstrate the system's capability to image target scenes and characterize human activity from different stand-off distances.

A Portable Real-Time Digital Noise Radar System for Through-the-Wall Imaging

Various parts of the human body have different movements when a person is performing different physical activities. There is a need to remotely detect and recognise specific human activities for applications involving anti-terrorism, hostage liberation and search-and-rescue. An ultrawideband noise radar system has been built to detect signatures of human activities. The Hilbert-Huang transform, which is used to extract the human-induced Doppler frequency shift from the received signal, facilitates the identification and recognition of various human activities. A non-stationary Doppler model is developed to describe various aspects of human motion, which compares well with measured data. The theoretical analysis and experimental results are discussed.

Hilbert-Huang transform analysis of human activities using through-wall noise and noise-like radar

Various parts of the human body have different movements when a person is performing different physical activities. There is a need to remotely detect human heartbeat and breathing for applications involving anti-terrorism and search-and-rescue. Ultrawideband noise radar systems are attractive because they are covert and immune from interference. The conventional time-frequency analyses of human activity are not generally applicable to nonlinear and nonstationary signals. If one can decompose the noisy baseband reflected signal and extract only the human-induced Doppler from it, the identification of various human activities becomes easier. We propose a nonstationary model to describe human motion and apply the Hilbert-Huang transform (HHT), which is adaptive to nonlinear and nonstationary signals, in order to analyze frequency characteristics of the baseband signal. When used with noise-like radar data, it is useful covertly identify specific human movement.

Hilbert-Huang Transform (HHT) Analysis of Human Activities Using Through-Wall Noise Radar

In this work, a portable and real-time through-wall random noise radar is presented. The system is inherently covert and is immune from interference and jamming since a random noise waveform is used. By combining mature cross-correlation and SAR algorithm with FPGA-based receiver technology, the system is able to perform real-time range detection as well as imaging of targets, including human beings, behind a wall.

Chieh-Ping Lai

Papers

Ultrawideband Random Noise Radar Design for Through-Wall Surveillance

A Portable Real-Time Digital Noise Radar System for Through-the-Wall Imaging

Hilbert-Huang transform analysis of human activities using through-wall noise and noise-like radar

Hilbert-Huang Transform (HHT) Analysis of Human Activities Using Through-Wall Noise Radar

Through wall ranging and imaging using UWB random noise waveform: System design considerations and preliminary experimental results