scispace - formally typeset
Search or ask a question
Proceedings ArticleDOI

Exploring audio and kinetic sensing on earable devices

10 Jun 2018-pp 5-10
TL;DR: This paper prototyped earbud devices with a 6-axis inertial measurement unit and a microphone to demonstrate that earable devices have a superior signal-to-noise ratio under the influence of motion artefacts and are less susceptible to acoustic environment noise.
Abstract: In this paper, we explore audio and kinetic sensing on earable devices with the commercial on-the-shelf form factor. For the study, we prototyped earbud devices with a 6-axis inertial measurement unit and a microphone. We systematically investigate the differential characteristics of the audio and inertial signals to assess their feasibility in human activity recognition. Our results demonstrate that earable devices have a superior signal-to-noise ratio under the influence of motion artefacts and are less susceptible to acoustic environment noise. We then present a set of activity primitives and corresponding signal processing pipelines to showcase the capabilities of earbud devices in converting accelerometer, gyroscope, and audio signals into the targeted human activities with a mean accuracy reaching up to 88% in varying environmental conditions.
Citations
More filters
Journal ArticleDOI
TL;DR: A review of the different areas of the recent machine learning research for healthcare wearable devices is presented, and different challenges facing machine learning applications on wearable devices are discussed.
Abstract: Using artificial intelligence and machine learning techniques in healthcare applications has been actively researched over the last few years. It holds promising opportunities as it is used to track human activities and vital signs using wearable devices and assist in diseases' diagnosis, and it can play a great role in elderly care and patient's health monitoring and diagnostics. With the great technological advances in medical sensors and miniaturization of electronic chips in the recent five years, more applications are being researched and developed for wearable devices. Despite the remarkable growth of using smart watches and other wearable devices, a few of these massive research efforts for machine learning applications have found their way to market. In this study, a review of the different areas of the recent machine learning research for healthcare wearable devices is presented. Different challenges facing machine learning applications on wearable devices are discussed. Potential solutions from the literature are presented, and areas open for improvement and further research are highlighted.

37 citations

Proceedings ArticleDOI
09 Sep 2019
TL;DR: This work evaluates the performance of eSense, a representative earable device, to track head rotations and investigates the interference generated by a magnetometer in an earable to understand the barriers to its use in these types of devices.
Abstract: Head tracking is a fundamental component in visual attention detection, which, in turn, can improve the state of the art of hearing aid devices. A multitude of wearable devices for the ear (so called earables) exist. Current devices lack a magnetometer which, as we will show, represents a big challenge when one tries to use them for accurate head tracking.In this work we evaluate the performance of eSense, a representative earable device, to track head rotations. By leveraging two different streams (one per earbud) of inertial data (from the accelerometer and the gyroscope), we achieve an accuracy up to a few degrees. We further investigate the interference generated by a magnetometer in an earable to understand the barriers to its use in these types of devices.

32 citations


Cites background or methods from "Exploring audio and kinetic sensing..."

  • ...The eSense platform [7, 12] consists in a pair of true wireless earbuds which have been augmented with kinetic, audio and proximity sensing options....

    [...]

  • ...focus on the evaluation of the eSense platform [7, 12] in track-...

    [...]

Proceedings ArticleDOI
04 Nov 2018
TL;DR: The eSense platform, the data exploration tool with the open APIs for the real-time visualisation of multi-modal sensory data, and its manifestation in a 360° workplace well-being application are demonstrated.
Abstract: We present eSense - an open and multi-sensory in-ear wearable platform for personal-scale behaviour analytics. eSense is a true wireless stereo (TWS) earbud and supports dual-mode Bluetooth and Bluetooth Low Energy. It is also augmented with a 6-axis in-ertial measurement unit and a microphone. We demonstrate the eSense platform, the data exploration tool with the open APIs for the real-time visualisation of multi-modal sensory data, and its manifestation in a 360° workplace well-being application.

26 citations


Cites background from "Exploring audio and kinetic sensing..."

  • ...The experimental results demonstrate that eSense can reach up to 88% detection accuracy for the targeted human activities [1, 4]....

    [...]

Proceedings ArticleDOI
16 Nov 2020
TL;DR: This work proposes a new modality of acoustic motion tracking using earphones, which mitigates the constraints associated with traditional smartphone-based tracking and shows earphone-based motion tracking can achieve a great flexibility and a high accuracy at the same time.
Abstract: Acoustic motion tracking is an exciting new research area with promising progress in the last few years. Due to the inherent low propagation speed in the air, acoustic signals have the unique advantage of fine sensing granularity compared to RF signals. Speakers and microphones nowadays are pervasively available in devices surrounding us, such as smartphones and voice-controlled smart speakers. Though promising, one fundamental issue hindering the adoption of acoustic-based motion tracking is that the positions of microphones and speakers inside a device are fixed, which greatly limits the flexibility of acoustic motion tracking. In this work, we propose a new modality of acoustic motion tracking using earphones. Earphone-based tracking mitigates the constraints associated with traditional smartphone-based tracking. With novel designs and comprehensive experiments, we show earphone-based motion tracking can achieve a great flexibility and a high accuracy at the same time. We believe this is an important step towards "earable" sensing.

22 citations


Cites background from "Exploring audio and kinetic sensing..."

  • ...Another recent work [35] can recognize human activities such as nodding, shaking, walking, stepping up, speaking and so on, using earable devices with a 6-axis inertial measurement unit and a microphone....

    [...]

Proceedings ArticleDOI
21 Apr 2020
TL;DR: The consequences of acoustic transparency are explored, both on the perception of virtual audio content, given the presence of a real-world auditory backdrop, and more broadly in facilitating a wearable, personal, private, always-available soundspace.
Abstract: Auditory headsets capable of actively or passively intermixing both real and virtual sounds are in-part acoustically transparent. This paper explores the consequences of acoustic transparency, both on the perception of virtual audio content, given the presence of a real-world auditory backdrop, and more broadly in facilitating a wearable, personal, private, always-available soundspace. We experimentally compare passive acoustically transparent, and active noise cancelling, orientation-tracked auditory headsets across a range of content types, both indoors and outdoors for validity. Our results show differences in terms of presence, realness and externalization for select content types. Via interviews and a survey, we discuss attitudes toward acoustic transparency (e.g. being perceived as safer), the potential shifts in audio usage that might be precipitated by adoption, and reflect on how such headsets and experiences fit within the area of Mixed Reality.

20 citations


Cites background from "Exploring audio and kinetic sensing..."

  • ...[72] Chulhong Min, Akhil Mathur, and Fahim Kawsar....

    [...]

References
More filters
Proceedings ArticleDOI
01 Nov 2015
TL;DR: It is indicated that on-device sensor and sensor handling heterogeneities impair HAR performances significantly and a novel clustering-based mitigation technique suitable for large-scale deployment of HAR is proposed, where heterogeneity of devices and their usage scenarios are intrinsic.
Abstract: The widespread presence of motion sensors on users' personal mobile devices has spawned a growing research interest in human activity recognition (HAR). However, when deployed at a large-scale, e.g., on multiple devices, the performance of a HAR system is often significantly lower than in reported research results. This is due to variations in training and test device hardware and their operating system characteristics among others. In this paper, we systematically investigate sensor-, device- and workload-specific heterogeneities using 36 smartphones and smartwatches, consisting of 13 different device models from four manufacturers. Furthermore, we conduct experiments with nine users and investigate popular feature representation and classification techniques in HAR research. Our results indicate that on-device sensor and sensor handling heterogeneities impair HAR performances significantly. Moreover, the impairments vary significantly across devices and depends on the type of recognition technique used. We systematically evaluate the effect of mobile sensing heterogeneities on HAR and propose a novel clustering-based mitigation technique suitable for large-scale deployment of HAR, where heterogeneity of devices and their usage scenarios are intrinsic.

561 citations

Journal ArticleDOI
01 Oct 2010
TL;DR: This article presents a survey of the techniques for extracting specific activity information from raw accelerometer data, and presents experimental results to compare and evaluate the accuracy of the various techniques using real data sets collected from daily activities.
Abstract: The ubiquity of communication devices such as smartphones has led to the emergence of context-aware services that are able to respond to specific user activities or contexts. These services allow communication providers to develop new, added-value services for a wide range of applications such as social networking, elderly care and near-emergency early warning systems. At the core of these services is the ability to detect specific physical settings or the context a user is in, using either internal or external sensors. For example, using built-in accelerometers, it is possible to determine whether a user is walking or running at a specific time of day. By correlating this knowledge with GPS data, it is possible to provide specific information services to users with similar daily routines. This article presents a survey of the techniques for extracting this activity information from raw accelerometer data. The techniques that can be implemented in mobile devices range from classical signal processing techniques such as FFT to contemporary string-based methods. We present experimental results to compare and evaluate the accuracy of the various techniques using real data sets collected from daily activities.

534 citations


"Exploring audio and kinetic sensing..." refers methods in this paper

  • ...Here, we used the features reported in [4]....

    [...]

Proceedings Article
09 Jul 2016
TL;DR: This paper rigorously explore deep, convolutional, and recurrent approaches across three representative datasets that contain movement data captured with wearable sensors, and describes how to train recurrent approaches in this setting, introduces a novel regularisation approach, and illustrates how they outperform the state-of-the-art on a large benchmark dataset.
Abstract: Human activity recognition (HAR) in ubiquitous computing is beginning to adopt deep learning to substitute for well-established analysis techniques that rely on hand-crafted feature extraction and classification methods. However, from these isolated applications of custom deep architectures it is difficult to gain an overview of their suitability for problems ranging from the recognition of manipulative gestures to the segmentation and identification of physical activities like running or ascending stairs. In this paper we rigorously explore deep, convolutional, and recurrent approaches across three representative datasets that contain movement data captured with wearable sensors. We describe how to train recurrent approaches in this setting, introduce a novel regularisation approach, and illustrate how they outperform the state-of-the-art on a large benchmark dataset. We investigate the suitability of each model for HAR, across thousands of recognition experiments with randomly sampled model configurations, explore the impact of hyperparameters using the fANOVA framework, and provide guidelines for the practitioner who wants to apply deep learning in their problem setting.

463 citations


"Exploring audio and kinetic sensing..." refers background in this paper

  • ...[5] proposed several deep learning models to detect physical activities using wearables....

    [...]

Proceedings ArticleDOI
07 Sep 2015
TL;DR: This work describes the implementation and evaluation of an approach for inferring eating moments based on 3-axis accelerometry collected with a popular off-the-shelf smartwatch, with applicability in areas ranging from health research and food journaling.
Abstract: Recognizing when eating activities take place is one of the key challenges in automated food intake monitoring. Despite progress over the years, most proposed approaches have been largely impractical for everyday usage, requiring multiple on-body sensors or specialized devices such as neck collars for swallow detection. In this paper, we describe the implementation and evaluation of an approach for inferring eating moments based on 3-axis accelerometry collected with a popular off-the-shelf smartwatch. Trained with data collected in a semi-controlled laboratory setting with 20 subjects, our system recognized eating moments in two free-living condition studies (7 participants, 1 day; 1 participant, 31 days), with F-scores of 76.1% (66.7% Precision, 88.8% Recall), and 71.3% (65.2% Precision, 78.6% Recall). This work represents a contribution towards the implementation of a practical, automated system for everyday food intake monitoring, with applicability in areas ranging from health research and food journaling.

284 citations

Book ChapterDOI
11 Sep 2005
TL;DR: It is demonstrated that sound from the user's mouth can be used to detect that he/she is eating and how different kinds of food can be recognized by analyzing chewing sounds.
Abstract: The paper reports the results of the first stage of our work on an automatic dietary monitoring system. The work is part of a large European project on using ubiquitous systems to support healthy lifestyle and cardiovascular disease prevention. We demonstrate that sound from the user's mouth can be used to detect that he/she is eating. The paper also shows how different kinds of food can be recognized by analyzing chewing sounds. The sounds are acquired with a microphone located inside the ear canal. This is an unobtrusive location widely accepted in other applications (hearing aids, headsets). To validate our method we present experimental results containing 3500 seconds of chewing data from four subjects on four different food types typically found in a meal. Up to 99% accuracy is achieved on eating recognition and between 80% to 100% on food type classification.

240 citations


"Exploring audio and kinetic sensing..." refers methods in this paper

  • ...Acoustic sensing approaches have been used to infer human activities and states such as eating [1] and coughing [7]....

    [...]

  • ...[1] placed a microphone inside the ear canal to detect eating activities and to classify them into four food types....

    [...]

Trending Questions (2)
How do earable devices compare to other ADHD treatments?

The given text does not provide any information about how earable devices compare to other ADHD treatments.

Are noise headphones good?

Our results demonstrate that earable devices have a superior signal-to-noise ratio under the influence of motion artefacts and are less susceptible to acoustic environment noise.