scispace - formally typeset
Search or ask a question
Proceedings ArticleDOI

Consumer Wearables and Affective Computing for Wellbeing Support

TL;DR: There is no versatile device suitable for all purposes in the field of wellbeing support, and the WellAff system able to recognize affective states for wellbeing support is proposed.
Abstract: Wearables equipped with pervasive sensors enable us to monitor physiological and behavioral signals in our everyday life. We propose the WellAff system able to recognize affective states for wellbeing support. It also includes health care scenarios, in particular patients with chronic kidney disease (CKD) suffering from bipolar disorders. For the need of a large-scale field study, we revised over 50 off-the-shelf devices in terms of usefulness for emotion, stress, meditation, sleep, and physical activity recognition and analysis. Their usability directly comes from the types of sensors they possess as well as the quality and availability of raw signals. We found there is no versatile device suitable for all purposes. Using Empatica E4 and Samsung Galaxy Watch, we have recorded physiological signals from 11 participants over many weeks. The gathered data enabled us to train a classifier that accurately recognizes strong affective states.
Citations
More filters
Journal ArticleDOI
16 Nov 2020-Sensors
TL;DR: The experimental results showed that the CNN-based architectures might be more suitable than LSTM-based architecture for affect recognition from physiological sensors, and the performance of the models depends on the intensity of the physiological response induced by the affective stimuli.
Abstract: To further extend the applicability of wearable sensors in various domains such as mobile health systems and the automotive industry, new methods for accurately extracting subtle physiological information from these wearable sensors are required. However, the extraction of valuable information from physiological signals is still challenging—smartphones can count steps and compute heart rate, but they cannot recognize emotions and related affective states. This study analyzes the possibility of using end-to-end multimodal deep learning (DL) methods for affect recognition. Ten end-to-end DL architectures are compared on four different datasets with diverse raw physiological signals used for affect recognition, including emotional and stress states. The DL architectures specialized for time-series classification were enhanced to simultaneously facilitate learning from multiple sensors, each having their own sampling frequency. To enable fair comparison among the different DL architectures, Bayesian optimization was used for hyperparameter tuning. The experimental results showed that the performance of the models depends on the intensity of the physiological response induced by the affective stimuli, i.e., the DL models recognize stress induced by the Trier Social Stress Test more successfully than they recognize emotional changes induced by watching affective content, e.g., funny videos. Additionally, the results showed that the CNN-based architectures might be more suitable than LSTM-based architectures for affect recognition from physiological sensors.

26 citations


Cites background from "Consumer Wearables and Affective Co..."

  • ...These physiological changes can be captured by sensors embedded into wearable devices that can measure [3]:...

    [...]

Journal ArticleDOI
TL;DR: In this paper , the Emognition dataset is dedicated to testing methods for emotion recognition (ER) from physiological responses and facial expressions from short film clips eliciting nine discrete emotions: amusement, awe, enthusiasm, liking, surprise, anger, disgust, fear, and sadness.
Abstract: The Emognition dataset is dedicated to testing methods for emotion recognition (ER) from physiological responses and facial expressions. We collected data from 43 participants who watched short film clips eliciting nine discrete emotions: amusement, awe, enthusiasm, liking, surprise, anger, disgust, fear, and sadness. Three wearables were used to record physiological data: EEG, BVP (2x), HR, EDA, SKT, ACC (3x), and GYRO (2x); in parallel with the upper-body videos. After each film clip, participants completed two types of self-reports: (1) related to nine discrete emotions and (2) three affective dimensions: valence, arousal, and motivation. The obtained data facilitates various ER approaches, e.g., multimodal ER, EEG- vs. cardiovascular-based ER, discrete to dimensional representation transitions. The technical validation indicated that watching film clips elicited the targeted emotions. It also supported signals' high quality.

13 citations

Journal ArticleDOI
TL;DR: In this paper , the Emognition dataset is dedicated to testing methods for emotion recognition (ER) from physiological responses and facial expressions from short film clips eliciting nine discrete emotions: amusement, awe, enthusiasm, liking, surprise, anger, disgust, fear, and sadness.
Abstract: The Emognition dataset is dedicated to testing methods for emotion recognition (ER) from physiological responses and facial expressions. We collected data from 43 participants who watched short film clips eliciting nine discrete emotions: amusement, awe, enthusiasm, liking, surprise, anger, disgust, fear, and sadness. Three wearables were used to record physiological data: EEG, BVP (2x), HR, EDA, SKT, ACC (3x), and GYRO (2x); in parallel with the upper-body videos. After each film clip, participants completed two types of self-reports: (1) related to nine discrete emotions and (2) three affective dimensions: valence, arousal, and motivation. The obtained data facilitates various ER approaches, e.g., multimodal ER, EEG- vs. cardiovascular-based ER, discrete to dimensional representation transitions. The technical validation indicated that watching film clips elicited the targeted emotions. It also supported signals' high quality.

8 citations

Journal ArticleDOI
TL;DR: In this article, the authors evaluated the feasibility of detecting stress using deep learning, a subfield of machine learning, on a small data set consisting of electrodermal activity, skin temperature, and heart rate measurements, in combination with selfreported anxiety and stress.
Abstract: Workplace-related stressors, economic strain, and lack of access to educational and basic needs have exacerbated feelings of stress in the United States. Ongoing stress can result in an increased risk of cardiovascular, musculoskeletal, and mental health disorders. Similarly, workplace stress can translate to a decrease in employee productivity and higher costs associated with employee absenteeism in an organization. Detecting stress and the events that correlate with stress during a workday is the first step to addressing its negative effects on health and wellbeing. Although there are a variety of techniques for stress detection using physiological signals, there is still limited research on the ability of behavioral measures to improve the performance of stress detection algorithms. In this study, we evaluated the feasibility of detecting stress using deep learning, a subfield of machine learning, on a small data set consisting of electrodermal activity, skin temperature, and heart rate measurements, in combination with self-reported anxiety and stress. The model was able to detect stress periods with 96% accuracy when using the combined wearable device and survey data, compared to the wearable device dataset alone (88% accuracy). Creating multi-dimensional datasets that include both wearable device data and ratings of perceived stress could help correlate stress-inducing events with feelings of stress at the individual level and help reduce intra-individual variabilities due to the subjective nature of the stress response.

8 citations

Journal ArticleDOI
21 Mar 2022
TL;DR: This work aims to explore the cold start problem, where no data from the target subjects (users) are available at the beginning of the experiment to train the reasoning model, and investigates the potential of per-group personalization and the amount of data needed to perform it.
Abstract: Emotion recognition in real life from physiological signals provided by wrist worn devices still remains a great challenge especially due to difficulties with gathering annotated emotional events. For that purpose, we suggest building pre-trained machine learning models capable of detecting intense emotional states. This work aims to explore the cold start problem, where no data from the target subjects (users) are available at the beginning of the experiment to train the reasoning model. To address this issue, we investigate the potential of per-group personalization and the amount of data needed to perform it. Our results on real-life data indicate that even a week’s worth of personalized data improves the model performance.

3 citations

References
More filters
Journal ArticleDOI
TL;DR: In this article, a method of over-sampling the minority class involves creating synthetic minority class examples, which is evaluated using the area under the Receiver Operating Characteristic curve (AUC) and the ROC convex hull strategy.
Abstract: An approach to the construction of classifiers from imbalanced datasets is described. A dataset is imbalanced if the classification categories are not approximately equally represented. Often real-world data sets are predominately composed of "normal" examples with only a small percentage of "abnormal" or "interesting" examples. It is also the case that the cost of misclassifying an abnormal (interesting) example as a normal example is often much higher than the cost of the reverse error. Under-sampling of the majority (normal) class has been proposed as a good means of increasing the sensitivity of a classifier to the minority class. This paper shows that a combination of our method of oversampling the minority (abnormal)cla ss and under-sampling the majority (normal) class can achieve better classifier performance (in ROC space)tha n only under-sampling the majority class. This paper also shows that a combination of our method of over-sampling the minority class and under-sampling the majority class can achieve better classifier performance (in ROC space)t han varying the loss ratios in Ripper or class priors in Naive Bayes. Our method of over-sampling the minority class involves creating synthetic minority class examples. Experiments are performed using C4.5, Ripper and a Naive Bayes classifier. The method is evaluated using the area under the Receiver Operating Characteristic curve (AUC)and the ROC convex hull strategy.

17,313 citations

Journal ArticleDOI
TL;DR: In this article, a method of over-sampling the minority class involves creating synthetic minority class examples, which is evaluated using the area under the Receiver Operating Characteristic curve (AUC) and the ROC convex hull strategy.
Abstract: An approach to the construction of classifiers from imbalanced datasets is described. A dataset is imbalanced if the classification categories are not approximately equally represented. Often real-world data sets are predominately composed of "normal" examples with only a small percentage of "abnormal" or "interesting" examples. It is also the case that the cost of misclassifying an abnormal (interesting) example as a normal example is often much higher than the cost of the reverse error. Under-sampling of the majority (normal) class has been proposed as a good means of increasing the sensitivity of a classifier to the minority class. This paper shows that a combination of our method of over-sampling the minority (abnormal) class and under-sampling the majority (normal) class can achieve better classifier performance (in ROC space) than only under-sampling the majority class. This paper also shows that a combination of our method of over-sampling the minority class and under-sampling the majority class can achieve better classifier performance (in ROC space) than varying the loss ratios in Ripper or class priors in Naive Bayes. Our method of over-sampling the minority class involves creating synthetic minority class examples. Experiments are performed using C4.5, Ripper and a Naive Bayes classifier. The method is evaluated using the area under the Receiver Operating Characteristic curve (AUC) and the ROC convex hull strategy.

11,512 citations

Journal ArticleDOI
TL;DR: Ecological momentary assessment holds unique promise to advance the science and practice of clinical psychology by shedding light on the dynamics of behavior in real-world settings.
Abstract: Assessment in clinical psychology typically relies on global retrospective self-reports collected at research or clinic visits, which are limited by recall bias and are not well suited to address how behavior changes over time and across contexts. Ecological momentary assessment (EMA) involves repeated sampling of subjects’ current behaviors and experiences in real time, in subjects’ natural environments. EMA aims to minimize recall bias, maximize ecological validity, and allow study of microprocesses that influence behavior in real-world contexts. EMA studies assess particular events in subjects’ lives or assess subjects at periodic intervals, often by random time sampling, using technologies ranging from written diaries and telephones to electronic diaries and physiological sensors. We discuss the rationale for EMA, EMA designs, methodological and practical issues, and comparisons of EMA and recall data. EMA holds unique promise to advance the science and practice of clinical psychology by shedding ligh...

4,286 citations


"Consumer Wearables and Affective Co..." refers background or methods in this paper

  • ...Therefore, we apply in the WellAff system the concept of Ecological Momentary Assessment (EMA) [43], which is in line with Descriptive Experience Sampling (DES)....

    [...]

  • ...EMA is particularly suited for studying substance use – in our case, strong affect events....

    [...]

  • ...If they agree, the phone application provides the EMA-type questionnaire [43] - a list of affective states along with the open text option....

    [...]

  • ...EMA, in turn, is a data collection method in real-time and in real-world settings to avoid retrospective biases and gather ecologically valid data....

    [...]

Proceedings ArticleDOI
02 Oct 2018
TL;DR: This work introduces WESAD, a new publicly available dataset for wearable stress and affect detection that bridges the gap between previous lab studies on stress and emotions, by containing three different affective states (neutral, stress, amusement).
Abstract: Affect recognition aims to detect a person's affective state based on observables, with the goal to e.g. improve human-computer interaction. Long-term stress is known to have severe implications on wellbeing, which call for continuous and automated stress monitoring systems. However, the affective computing community lacks commonly used standard datasets for wearable stress detection which a) provide multimodal high-quality data, and b) include multiple affective states. Therefore, we introduce WESAD, a new publicly available dataset for wearable stress and affect detection. This multimodal dataset features physiological and motion data, recorded from both a wrist- and a chest-worn device, of 15 subjects during a lab study. The following sensor modalities are included: blood volume pulse, electrocardiogram, electrodermal activity, electromyogram, respiration, body temperature, and three-axis acceleration. Moreover, the dataset bridges the gap between previous lab studies on stress and emotions, by containing three different affective states (neutral, stress, amusement). In addition, self-reports of the subjects, which were obtained using several established questionnaires, are contained in the dataset. Furthermore, a benchmark is created on the dataset, using well-known features and standard machine learning methods. Considering the three-class classification problem ( baseline vs. stress vs. amusement ), we achieved classification accuracies of up to 80%,. In the binary case ( stress vs. non-stress ), accuracies of up to 93%, were reached. Finally, we provide a detailed analysis and comparison of the two device locations ( chest vs. wrist ) as well as the different sensor modalities.

486 citations


"Consumer Wearables and Affective Co..." refers methods in this paper

  • ...Data from SpO2, ECG, RSP, and MIC allow us to diagnose the sleep apnea [15, 28, 33]....

    [...]

  • ...They can also provide other data derived from the monitored signals: HR - heart rate (extracted either from BVP/PPI or ECG/RRI), STP - number of steps, RSP - respiration rate, EDR - RSP from ECG, CAL - calories burned, SP - sleep phases....

    [...]

  • ...Those signals can be supplemented with ACC, GYRO SKT, RSP [2, 23, 27, 39, 40], as well as with UV, GPS, and MIC data [25]....

    [...]

  • ...Bio-signals related to stress include EEG, ECG, BVP, EDA, SKT, and RSP [9, 18]....

    [...]

Journal ArticleDOI
10 Jun 2014-Sensors
TL;DR: This paper explores how these various motion sensors behave in different situations in the activity recognition process, and shows that they are each capable of taking the lead roles individually, depending on the type of activity being recognized, the body position, the used data features and the classification method employed.
Abstract: For physical activity recognition, smartphone sensors, such as an accelerometer and a gyroscope, are being utilized in many research studies. So far, particularly, the accelerometer has been extensively studied. In a few recent studies, a combination of a gyroscope, a magnetometer (in a supporting role) and an accelerometer (in a lead role) has been used with the aim to improve the recognition performance. How and when are various motion sensors, which are available on a smartphone, best used for better recognition performance, either individually or in combination? This is yet to be explored. In order to investigate this question, in this paper, we explore how these various motion sensors behave in different situations in the activity recognition process. For this purpose, we designed a data collection experiment where ten participants performed seven different activities carrying smart phones at different positions. Based on the analysis of this data set, we show that these sensors, except the magnetometer, are each capable of taking the lead roles individually, depending on the type of activity being recognized, the body position, the used data features and the classification method employed (personalized or generalized). We also show that their combination only improves the overall recognition performance when their individual performances are not very high, so that there is room for performance improvement. We have made our data set and our data collection application publicly available, thereby making our experiments reproducible.

426 citations