Providing accurate and opportune information on people's activities and behaviors is one of the most important tasks in pervasive computing. Innumerable applications can be visualized, for instance, in medical, security, entertainment, and tactical scenarios. Despite human activity recognition (HAR) being an active field for more than a decade, there are still key aspects that, if addressed, would constitute a significant turn in the way people interact with mobile devices. This paper surveys the state of the art in HAR based on wearable sensors. A general architecture is first presented along with a description of the main components of any HAR system. We also propose a two-level taxonomy in accordance to the learning approach (either supervised or semi-supervised) and the response time (either offline or online). Then, the principal issues and challenges are discussed, as well as the main solutions to each one of them. Twenty eight systems are qualitatively evaluated in terms of recognition performance, energy consumption, obtrusiveness, and flexibility, among others. Finally, we present some open problems and ideas that, due to their high relevance, should be addressed in future research.

A Survey on Human Activity Recognition using Wearable Sensors

Objective: To demonstrate optical coherence tomography for high-resolution, noninvasive imaging of the human retina. Optical coherence tomography is a new imaging technique analogous to ultrasound B scan that can provide cross-sectional images of the retina with micrometer-scale resolution. Design: Survey optical coherence tomographic examination of the retina, including the macula and optic nerve head in normal human subjects. Settings Research laboratory. Participants: Convenience sample of normal human subjects. Main Outcome Measures: Correlation of optical coherence retinal tomographs with known normal retinal anatomy. Results: Optical coherence tomographs can discriminate the cross-sectional morphologic features of the fovea and optic disc, the layered structure of the retina, and normal anatomic variations in retinal and retinal nerve fiber layer thicknesses with 10- μm depth resolution. Conclusion: Optical coherence tomography is a potentially useful technique for high depth resolution, cross-sectional examination of the fundus.

Optical Coherence Tomography of the Human Retina

We deployed 72 sensors of 10 modalities in 15 wireless and wired networked sensor systems in the environment, in objects, and on the body to create a sensor-rich environment for the machine recognition of human activities. We acquired data from 12 subjects performing morning activities, yielding over 25 hours of sensor data. We report the number of activity occurrences observed during post-processing, and estimate that over 13000 and 14000 object and environment interactions occurred. We describe the networked sensor setup and the methodology for data acquisition, synchronization and curation. We report on the challenges and outline lessons learned and best practice for similar large scale deployments of heterogeneous networked sensor systems. We evaluate data acquisition quality for on-body and object integrated wireless sensors; there is less than 2.5% packet loss after tuning. We outline our use of the dataset to develop new sensor network self-organization principles and machine learning techniques for activity recognition in opportunistic sensor configurations. Eventually this dataset will be made public.

/pdf/collecting-complex-activity-datasets-in-highly-rich-3fujzavxun.pdf

Collecting complex activity datasets in highly rich networked sensor environments

Applying new sensing technology to healthcare maybe part of a solution to the financial and demographic crisis facing global healthcare systems. Researchers applying new approaches to noninvasive patient monitoring and diagnostics are assisted by the features of Sensing Health with Intelligence, Modularity, Mobility and Experimental Reusability (SHIMMER™), a flexible sensing platform. Integrated peripherals, open software, modular expansion, specific power management hardware, and a library of applications supported with platform validation provide SHIMMER with advantages over many other medical research platforms.

SHIMMER™ – A Wireless Sensor Platform for Noninvasive Biomedical Research

This paper presents a literature review on the use of depth for hand tracking and gesture recognition. The survey examines 37 papers describing depth-based gesture recognition systems in terms of (1) the hand localization and gesture classification methods developed and used, (2) the applications where gesture recognition has been tested, and (3) the effects of the low-cost Kinect and OpenNI software libraries on gesture recognition research. The survey is organized around a novel model of the hand gesture recognition process. In the reviewed literature, 13 methods were found for hand localization and 11 were found for gesture classification. 24 of the papers included real-world applications to test a gesture recognition system, but only 8 application categories were found (and three applications accounted for 18 of the papers). The papers that use the Kinect and the OpenNI libraries for hand tracking tend to focus more on applications than on localization and classification methods, and show that the OpenNI hand tracking method is good enough for the applications tested thus far. However, the limitations of the Kinect and other depth sensors for gesture recognition have yet to be tested in challenging applications and environments.

Hand gesture recognition with depth images: A review

We investigate the potential of the Kinect depth-mapping camera for sign language recognition and verification for educational games for deaf children. We compare a prototype Kinect-based system to our current CopyCat system which uses colored gloves and embedded accelerometers to track children's hand movements. If successful, a Kinect-based approach could improve interactivity, user comfort, system robustness, system sustainability, cost, and ease of deployment. We collected a total of 1000 American Sign Language (ASL) phrases across both systems. On adult data, the Kinect system resulted in 51.5% and 76.12% sentence verification rates when the users were seated and standing respectively. These rates are comparable to the 74.82% verification rate when using the current(seated) CopyCat system. While the Kinect computer vision system requires more tuning for seated use, the results suggest that the Kinect may be a viable option for sign verification.

American sign language recognition with the kinect

Previous research has established the connection between the way in which children interact with objects and the potential early identification of children with autism. Those findings motivate our own work to develop "smart toys," objects embedded with wireless sensors that are safe and enjoyable for very small children, that allow detailed interaction data to be easily recorded. These sensor-enabled toys provide opportunities for autism research by reducing the effort required to collect and analyze a child's interactions with objects. In the future, such toys may be a useful part of clinical and in-home assessment tools. In this paper, we discuss the design of a collection of smart toys that can be used to automatically characterize the way in which a child is playing. We use statistical models to provide objective, quantitative measures of object play interactions. We also developed a tool to view rich forms of annotated play data for later analysis. We report the results of recognition experiments on more than fifty play sessions conducted with adults and children as well as discuss the opportunities for using this approach to support video annotation and other applications.

https://static.springer.com/sgw/documents/1389130/application/pdf/10.1007_s00779-011-0386-0_PUC_16%2C2.pdf

Monitoring children's developmental progress using augmented toys and activity recognition

We describe the activity recognition component of the Soldier Assist System (SAS), which was built to meet the goals of DARPA’s Advanced Soldier Sensor Information System and Technology (ASSIST) program. As a whole, SAS provides an integrated solution that includes on-body data capture, automatic recognition of soldier activity, and a multimedia interface that combines data search and exploration. The recognition component analyzes readings from six on-body accelerometers to identify activity. The activities are modeled by boosted 1D classifiers, which allows efficient selection of the most useful features within the learning algorithm. We present empirical results based on data collected at Georgia Tech and at the Army’s Aberdeen Proving Grounds during official testing by a DARPA appointed NIST evaluation team. Our approach achieves 78.7% for continuous event recognition and 70.3% frame level accuracy. The accuracy increases to 90.3% and 90.3% respectively when considering only the modeled activities. In addition to standard error metrics, we discuss error division diagrams (EDDs) for several Aberdeen data sequences to provide a rich visual representation of the performance of our system.

Recognizing Soldier Activities in the Field

The galvanic skin response (GSR), also known as elec- trodermal response, measures changes in electrical resistance across two regions of the skin. Galvanic skin response can measure arousal levels in children with autism; however, the GSR signal may be overwhelmed by the vigorous movements of the children. This paper introduces ActionGSR, a wireless sensor capable of measuring both GSR and acceleration simultaneously in an attempt to dis ambiguate valid GSR signals from motion artifacts.

ActionGSR: A Combination Galvanic Skin Response-Accelerometer for Physiological Measurements in Active Environments

We perform real-time American Sign Language (ASL) phrase verification for an educational game, CopyCat, which is designed to improve deaf children's signing skills. Taking advantage of context information in the game we verify a phrase, using Hidden Markov Models (HMMs), by applying a rejection threshold on the probability of the observed sequence for each sign in the phrase. We tested this approach using 1204 signed phrase samples from 11 deaf children playing the game during the phase two deployment of CopyCat. The CopyCat data set is particularly challenging because sign samples are collected during live game play and contain many variations in signing and disfluencies. We achieved a phrase verification accuracy of 83% compared to 90% real-time performance by a sign linguist. We report on the techniques required to reach this level of performance.

Peter Presti

Papers

American sign language recognition with the kinect

Monitoring children's developmental progress using augmented toys and activity recognition

Recognizing Soldier Activities in the Field

ActionGSR: A Combination Galvanic Skin Response-Accelerometer for Physiological Measurements in Active Environments

American Sign Language Phrase Verification in an Educational Game for Deaf Children