WiFi-enabled Device-free Gesture Recognition for Smart Home Automation
12 Jun 2018-pp 476-481
TL;DR: FreeGesture is proposed, a device-free gesture recognition scheme that can automatically identify common gestures via deep learning using only commodity WiFi-enabled IoT devices and achieves a 95.8% gesture recognition accuracy.
Abstract: Gesture recognition is playing a vital role in human-computer interaction and smart home automation. Conventional gesture recognition systems require either dedicated extra infrastructure to be deployed or users to carry wearable and mobile devices, which is high-cost and intrusive for large-scale implementation. In this paper, we propose FreeGesture, a device-free gesture recognition scheme that can automatically identify common gestures via deep learning using only commodity WiFi-enabled IoT devices. A novel OpenWrt-based IoT platform is developed so that the fine-grained Channel State Information (CSI) measurements can be obtained directly from IoT devices. Since these measurements are time-series data, we consider them as continuous RF images and construct CSI frames with both amplitudes and phase differences as features. We design a dedicated convolutional neural network (CNN) to uncover the discriminative local features in these CSI frames and construct a robust classifier for gesture recognition. All the parameters in CNN are automatically fine-tuned end-to-end. Experiments are conducted in a typical office and the results validate that FreeGesture achieves a 95.8% gesture recognition accuracy.
Citations
More filters
TL;DR: This paper analyzes the key components and core characteristics of the system architecture of human behavior recognition using CSI and elaborates the typical behavior recognition applications from five aspects, including experimental equipment, experimental scenario, behavior, classifier, and system performance.
Abstract: Recently, device-free human behavior recognition has become a hot research topic and has achieved significant progress in the field of ubiquitous computing. Among various implementation, behavior recognition based on WiFi CSI (channel state information) has drawn wide attention due to its major advantages. This paper investigates more than 100 latest CSI based behavior recognition applications within the last 6 years and presents a comprehensive survey from every aspect of human behavior recognition. Firstly, this paper reviews general behavior recognition applications using the WiFi signal and presents the basic concept of CSI and the fundamental principle of CSI-based behavior recognition. This paper analyzes the key components and core characteristics of the system architecture of human behavior recognition using CSI. Afterward, we divide the sensing procedures into many steps and summarize the typical studies from these steps, including base signal selection, signal preprocessing, and identification approaches. Next, based on the recognition technique, we classify the applications into three groups, including pattern-based, model-based, and deep learning-based approach. In every group, we categorize the state-of-the-art applications into three groups, including coarse-grained specific behavior recognition, fine-grained specific behavior recognition, and activity inference. It elaborates the typical behavior recognition applications from five aspects, including experimental equipment, experimental scenario, behavior, classifier, and system performance. Then, this paper presents comprehensive discussions about representative applications from the implementation view and outlines the major consideration when developing a recognition system. Finally, this article concludes by analyzing the open issues of CSI-based behavior recognition applications and pointing out future research directions.
95 citations
Cites background from "WiFi-enabled Device-free Gesture Re..."
...falling detection [143], [144], syncope detection [145], hand gesture recognition [8], [146], [147], [149], sign language recognition [150], gait and walking direction recognition [151], [152], human detection [153], [154], crowd counting [156]–[158], user authentication [161], and respiration...
[...]
...ing [4], behavior understanding [5], user profile construction [6], activity inference [7], smart home control [8], human localization [9], human position tracking [10], [11], and occupancy detection [12], etc....
[...]
...ior recognition [138]–[142], falling detection [143], [144], syncope detection [145], hand gesture recognition [8], [146]–[149], sign language recognition [150], gait and walking direction recognition [151], [152], human presence detection [153]–[155], crowd counting [156]–[159], user authentication [160]–[163], and respiration monitoring [22]....
[...]
...Development of a universal framework to tackle this problem seems to be a potential approach [8], [22], [138], [141], [147], [150], [161], [187], [188]....
[...]
...[79], WiCatch [80], iGest [81], DeNum [146], FreeGesture [8], DeepNum [147], Widar3....
[...]
TL;DR: A WiFi-based human activity recognition system that synthesizes variant activities data through eight channel state information (CSI) transformation methods to mitigate the impact of activity inconsistency and subject-specific issues is proposed and a novel deep-learning model is designed that caters to the small-size WiFi activity data.
Abstract: Recent research has devoted significant efforts on the utilization of WiFi signals to recognize various human activities. An individual’s limb motions in the WiFi coverage area could interfere with wireless signal propagation, that manifested as unique patterns for activity recognition. Existing approaches though yielding reasonable performance in certain cases, are ignorant of two major challenges. The performed activities of the individual normally have inconsistent speed in different situations and time. Besides that the wireless signal reflected by human bodies normally carries substantial information that is specific to that subject. The activity recognition model trained on a certain individual may not work well when being applied to predict another individual’s activities. Since only recording activities of limited subjects in a certain speed and scale, recent works commonly have a moderate amount of activity data for training the recognition model. The small-size data could often incur the overfitting issue that negative affect the traditional classification model. To address these challenges, we propose a WiFi-based human activity recognition system that synthesizes variant activities data through eight channel state information (CSI) transformation methods to mitigate the impact of activity inconsistency and subject-specific issues, and also design a novel deep-learning model that caters to the small-size WiFi activity data. We conduct extensive experiments and show synthetic data improve performance by up to 34.6% and our system achieves around 90% of accuracy with well robustness in adapting to small-size CSI data.
87 citations
Cites background from "WiFi-enabled Device-free Gesture Re..."
...a typical deep-learning model to extract representative feature in CSI data and has been widely adopted in other WiFibased activity recognition works [35], [37], [52]–[54]....
[...]
TL;DR: A new approach that uses deep transfer learning techniques to recognize gestures based on the channel state information (CSI) extracted from WiFi signals and demonstrates that the proposed method outperformed other state-of-the-art WiFi-based gesture recognition methods.
Abstract: Different gestures have different action speeds, directions, and trajectories that can cause distinctive effects on the propagation of WiFi signals. In this paper, we present a new approach that uses deep transfer learning techniques to recognize gestures based on the channel state information (CSI) extracted from WiFi signals. Firstly, the CSI streams of gestures are captured and the gesture segments are extracted based on the CSI amplitude changes, and then the WiFi-based gesture recognition problem is innovatively converted to an image classification problem by expressing CSI streams as an image matrix. After that, two deep transfer learning methods are applied to recognize gestures using high-level features extracted by deep convolutional neural network (CNN) and fine-tuned CNN models. We evaluated our method using a collected dataset with 12 gestures in two environments, and the experimental results demonstrated that the proposed method outperformed other state-of-the-art WiFi-based gesture recognition methods.
22 citations
TL;DR: This study provides essential data for all researchers who want to apply deep learning for smart homes, identifies the main trends, and can help to guide design and evaluation decisions for particular smart home services.
Abstract: In recent years, research on convolutional neural networks (CNN) and recurrent neural networks (RNN) in deep learning has been actively conducted. In order to provide more personalized and advanced functions in smart home services, studies on deep learning applications are becoming more frequent, and deep learning is acknowledged as an efficient method for recognizing the voices and activities of users. In this context, this study aims to systematically review the smart home studies that apply CNN and RNN/LSTM as their main solution. Of the 632 studies retrieved from the Web of Science, Scopus, IEEE Explore, and PubMed databases, 43 studies were selected and analyzed according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) methodology. In this paper, we examine which smart home applications CNN and RNN/LSTM are applied to and compare how they were implemented and evaluated. The selected studies dealt with a total of 15 application areas for smart homes, where activity recognition was covered the most. This study provides essential data for all researchers who want to apply deep learning for smart homes, identifies the main trends, and can help to guide design and evaluation decisions for particular smart home services.
13 citations
22 Aug 2022
TL;DR: A WiFi-based IoT-enabled human pose estimation scheme for metaverse avatar simulation, namely MetaFi, which is enforced to learn the annotations from the accurate computer vision model, thus achieving cross- modal supervision.
Abstract: Avatar refers to a representative of a physical user in the virtual world that can engage in different activities and interact with other objects in metaverse. Simulating the avatar requires accurate human pose estimation. Though camera-based solutions yield remarkable performance, they encounter the privacy issue and degraded performance caused by varying illumination, especially in smart home. In this paper, we propose a WiFi-based IoT-enabled human pose estimation scheme for metaverse avatar simulation, namely MetaFi. Specifically, a deep neural network is designed with customized convolutional layers and residual blocks to map the channel state information to human pose landmarks. It is enforced to learn the annotations from the accurate computer vision model, thus achieving cross-modal supervision. WiFi is ubiquitous and robust to illumination, making it a feasible solution for avatar applications at smart home. The experiments are conducted in the real world, and the results show that the MetaFi achieves very high performance with a PCK@50 of 95.23%.
11 citations
References
More filters
TL;DR: This work takes an object recognition approach, designing an intermediate body parts representation that maps the difficult pose estimation problem into a simpler per-pixel classification problem, and generates confidence-scored 3D proposals of several body joints by reprojecting the classification result and finding local modes.
Abstract: We propose a new method to quickly and accurately predict human pose---the 3D positions of body joints---from a single depth image, without depending on information from preceding frames. Our approach is strongly rooted in current object recognition strategies. By designing an intermediate representation in terms of body parts, the difficult pose estimation problem is transformed into a simpler per-pixel classification problem, for which efficient machine learning techniques exist. By using computer graphics to synthesize a very large dataset of training image pairs, one can train a classifier that estimates body part labels from test images invariant to pose, body shape, clothing, and other irrelevances. Finally, we generate confidence-scored 3D proposals of several body joints by reprojecting the classification result and finding local modes.The system runs in under 5ms on the Xbox 360. Our evaluation shows high accuracy on both synthetic and real test sets, and investigates the effect of several training parameters. We achieve state-of-the-art accuracy in our comparison with related work and demonstrate improved generalization over exact whole-skeleton nearest neighbor matching.
3,034 citations
22 Jan 2011
TL;DR: The measurement setup comprises the customized versions of Intel's close-source firmware and open-source iwlwifi wireless driver, userspace tools to enable these measurements, access point functionality for controlling both ends of the link, and Matlab scripts for data analysis.
Abstract: We are pleased to announce the release of a tool that records detailed measurements of the wireless channel along with received 802.11 packet traces. It runs on a commodity 802.11n NIC, and records Channel State Information (CSI) based on the 802.11 standard. Unlike Receive Signal Strength Indicator (RSSI) values, which merely capture the total power received at the listener, the CSI contains information about the channel between sender and receiver at the level of individual data subcarriers, for each pair of transmit and receive antennas.Our toolkit uses the Intel WiFi Link 5300 wireless NIC with 3 antennas. It works on up-to-date Linux operating systems: in our testbed we use Ubuntu 10.04 LTS with the 2.6.36 kernel. The measurement setup comprises our customized versions of Intel's close-source firmware and open-source iwlwifi wireless driver, userspace tools to enable these measurements, access point functionality for controlling both ends of the link, and Matlab (or Octave) scripts for data analysis. We are releasing the binary of the modified firmware, and the source code to all the other components.
1,354 citations
"WiFi-enabled Device-free Gesture Re..." refers methods in this paper
...Conventional CSI-based sensing systems adopt the Intel 5300 NIC tool [22] to extract the CSI data from PCs or laptops with modified WiFi NIC cards....
[...]
30 Sep 2013
TL;DR: WiSee is presented, a novel gesture recognition system that leverages wireless signals (e.g., Wi-Fi) to enable whole-home sensing and recognition of human gestures and achieves this goal without requiring instrumentation of the human body with sensing devices.
Abstract: This paper presents WiSee, a novel gesture recognition system that leverages wireless signals (e.g., Wi-Fi) to enable whole-home sensing and recognition of human gestures. Since wireless signals do not require line-of-sight and can traverse through walls, WiSee can enable whole-home gesture recognition using few wireless sources. Further, it achieves this goal without requiring instrumentation of the human body with sensing devices. We implement a proof-of-concept prototype of WiSee using USRP-N210s and evaluate it in both an office environment and a two- bedroom apartment. Our results show that WiSee can identify and classify a set of nine gestures with an average accuracy of 94%.
1,045 citations
"WiFi-enabled Device-free Gesture Re..." refers background in this paper
...RF-based Systems such as WiSee [3] and WiTrack [4] are able to identify common gestures with high accuracy using software defined radios (SDRs) platforms....
[...]
07 Sep 2015
TL;DR: CARM is a CSI based human Activity Recognition and Monitoring system that quantitatively builds the correlation between CSI value dynamics and a specific human activity and recognizes a given activity by matching it to the best-fit profile.
Abstract: Some pioneer WiFi signal based human activity recognition systems have been proposed. Their key limitation lies in the lack of a model that can quantitatively correlate CSI dynamics and human activities. In this paper, we propose CARM, a CSI based human Activity Recognition and Monitoring system. CARM has two theoretical underpinnings: a CSI-speed model, which quantifies the correlation between CSI value dynamics and human movement speeds, and a CSI-activity model, which quantifies the correlation between the movement speeds of different human body parts and a specific human activity. By these two models, we quantitatively build the correlation between CSI value dynamics and a specific human activity. CARM uses this correlation as the profiling mechanism and recognizes a given activity by matching it to the best-fit profile. We implemented CARM using commercial WiFi devices and evaluated it in several different environments. Our results show that CARM achieves an average accuracy of greater than 96%.
861 citations
"WiFi-enabled Device-free Gesture Re..." refers background in this paper
...[14], human activity recognition [15], [16], and even human identification [17], have been realized using CSI in a devicefree and privacy-preserving manner....
[...]
02 Apr 2014
TL;DR: WiTrack bridges a gap between RF-based localization systems which locate a user through walls and occlusions, and human-computer interaction systems like Kinect, which can track a user without instrumenting her body, but require the user to stay within the direct line of sight of the device.
Abstract: This paper introduces WiTrack, a system that tracks the 3D motion of a user from the radio signals reflected off her body. It works even if the person is occluded from the WiTrack device or in a different room. WiTrack does not require the user to carry any wireless device, yet its accuracy exceeds current RF localization systems, which require the user to hold a transceiver. Empirical measurements with a WiTrack prototype show that, on average, it localizes the center of a human body to within a median of 10 to 13 cm in the x and y dimensions, and 21 cm in the z dimension. It also provides coarse tracking of body parts, identifying the direction of a pointing hand with a median of 11.2°. WiTrack bridges a gap between RF-based localization systems which locate a user through walls and occlusions, and human-computer interaction systems like Kinect, which can track a user without instrumenting her body, but require the user to stay within the direct line of sight of the device.
661 citations
"WiFi-enabled Device-free Gesture Re..." refers background in this paper
...RF-based Systems such as WiSee [3] and WiTrack [4] are able to identify common gestures with high accuracy using software defined radios (SDRs) platforms....
[...]