Remote measurements of the cardiac pulse can provide comfortable physiological assessment without electrodes. However, attempts so far are non-automated, susceptible to motion artifacts and typically expensive. In this paper, we introduce a new methodology that overcomes these problems. This novel approach can be applied to color video recordings of the human face and is based on automatic face tracking along with blind source separation of the color channels into independent components. Using Bland-Altman and correlation analysis, we compared the cardiac pulse rate extracted from videos recorded by a basic webcam to an FDA-approved finger blood volume pulse (BVP) sensor and achieved high accuracy and correlation even in the presence of movement artifacts. Furthermore, we applied this technique to perform heart rate measurements from three participants simultaneously. This is the first demonstration of a low-cost accurate video-based method for contact-free heart rate measurements that is automated, motion-tolerant and capable of performing concomitant measurements on more than one person at a time.

/pdf/non-contact-automated-cardiac-pulse-measurements-using-video-557l0m5qu6.pdf

Non-contact, automated cardiac pulse measurements using video imaging and blind source separation.

We present a simple, low-cost method for measuring multiple physiological parameters using a basic webcam. By applying independent component analysis on the color channels in video recordings, we extracted the blood volume pulse from the facial regions. Heart rate (HR), respiratory rate, and HR variability (HRV, an index for cardiac autonomic activity) were subsequently quantified and compared to corresponding measurements using Food and Drug Administration-approved sensors. High degrees of agreement were achieved between the measurements across all physiological parameters. This technology has significant potential for advancing personal health care and telemedicine.

/pdf/advancements-in-noncontact-multiparameter-physiological-3t98r5y6lj.pdf

Advancements in Noncontact, Multiparameter Physiological Measurements Using a Webcam

Respiration rate is an important indicator of a person's health, and thus it is monitored when performing clinical evaluations. There are different approaches for respiration monitoring, but generally they can be classed as contact or noncontact. For contact methods, the sensing device (or part of the instrument containing it) is attached to the subject's body. For noncontact approaches the monitoring is performed by an instrument that does not make any contact with the subject. In this article a review of respiration monitoring approaches (both contact and noncontact) is provided. Concerns related to the patient's recording comfort, recording hygiene, and the accuracy of respiration rate monitoring have resulted in the development of a number of noncontact respiration monitoring approaches. A description of thermal imaging based and vision based noncontact respiration monitoring approaches we are currently developing is provided.

/pdf/respiration-rate-monitoring-methods-a-review-1ykeblev32.pdf

Respiration Rate Monitoring Methods: A Review

Facial expressions are an important way through which humans interact socially. Building a system capable of automatically recognizing facial expressions from images and video has been an intense field of study in recent years. Interpreting such expressions remains challenging and much research is needed about the way they relate to human affect. This paper presents a general overview of automatic RGB, 3D, thermal and multimodal facial expression analysis. We define a new taxonomy for the field, encompassing all steps from face detection to facial expression recognition, and describe and classify the state of the art methods accordingly. We also present the important datasets and the bench-marking of most influential methods. We conclude with a general discussion about trends, important questions and future lines of research.

Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications

Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-related Applications

Interacting with human physiology

This paper unveils a thermal imaging methodology to recover the breathing waveform from the subject's nostrils. The resulting functionality is equivalent to that of a thermistor, but it is materialized in a contact-free manner. First, the nostril region is segmented and tracked over time through a network of cooperating probabilistic trackers. The mean thermal signal of the nostril region carries the breathing information. This information is extracted through wavelet analysis. The method has been tested on 20 healthy individuals. The breathing waveforms determined via the imaging computation were compared with the corresponding ones extracted from thermistors. The high degree of agreement between the two measurement methods confirms the validity of the proposed approach and opens the way for clinical applications. Furthermore, thermal imaging can be potentially used as an investigative tool to understand breathing physiology in ways not possible with contact sensors.

Thermistor at a Distance: Unobtrusive Measurement of Breathing

We introduce a novel methodology to characterize breathing patterns based on thermal infrared imaging. We have retrofitted a Mid-Wave Infra-Red (MWIR) imaging system with a narrow band-pass filter in the CO(2) absorption band (4130 - 4427 nm). We use this system to record the radiation information from within the breathing flow region. Based on this information we compute the mean dynamic thermal signal of breath. The breath signal is quasi-periodic due to the interleaving of high and low intensities corresponding to expirations and inspirations respectively. We sample the signal at a constant rate and then filter the high frequency noise due to tracking instability. We detect the breathing cycles through zero cross thresholding, which is insensitive to noise around the zero line. We normalize the breathing cycles and align them at the transition point from inhalation to exhalation. Then, we compute the mean breathing cycle. We use the first eight (8) harmonic components of the mean cycle to characterize the breathing pattern. The harmonic analysis highlights the intra-individual similarity of breathing patterns. Our method opens the way for desktop, unobtrusive monitoring of human respiration and may find widespread applications in clinical studies of chronic ailments. It also brings up the intriguing possibility of using breathing patterns as a novel biometric.

Analysis of Breathing Air Flow Patterns in Thermal Imaging

THE DIAGNOSIS OF SLEEP APNEA TYPICALLY INVOLVES AN OVERNIGHT POLYSOMNOGRAM WITH CONTINUOUS MONITORING OF SEVERAL PHYSIOLOGIC parameters and surrogate measures of airflow (nasal pressure [Pn], oronasal thermistor, expired CO2 waveform [PECO2]) using contact sensors.1–5 A subject can come in contact with at least 20 such sensors during a study. These sensors and the wires can influence not only the usual sleep architecture, but also the posture of the patient during the study night.6–8 Contact sensors like nasal prongs and oronasal thermistors are routinely used as surrogate measures of airflow because they are less obtrusive than the reference-standard airflow sensor, namely, the pneumotachometer.1,9,10 However, the presence of the surrogate airflow sensors and the associated wires in the proximity of the subject's oronasal area can still cause discomfort to the patient during sleep. Contact sensors also have the potential to misalign during the study, requiring repositioning at the bedside by a technician, to obtain good-quality data.

We have demonstrated that measurement of breathing rates is feasible at a distance using thermal infrared imaging (TIRI).11 The method is based on second-order statistics of the presence or absence of the hot expiratory plume in the vicinity of the nostrils. We have also studied a more advanced method that is based on wavelet analysis of the thermal imaging signal on the nostrils themselves.12,13 When analyzing the thermal imaging signal, we extracted the full breathing waveform (rate and amplitude). Consistent segmentation of the nostril area and facial tissue-tracking algorithms14 enabled sustained monitoring of breathing, resulting in functionality equivalent to that of a thermistor, delivered in a contact-free manner. In 5-minute recordings of 20 awake healthy individuals, agreement between breathing waveforms from the virtual (imaging) and contact thermistors was found to be greater than 90%.

The hardware and software system that was developed to implement the latest method is called automatic thermal monitoring system (ATHEMOS). The centerpiece of the system is a midwave infrared camera (FLIR® SC-6000), supported by a black body for calibration and a pan-tilt mechanism for positioning. The system is controlled and the acquired imaging signals are processed by real-time custom software developed in our labs.

ATHEMOS and the imaging method that we reported on in a previous publication12 constitute the TIRI portion of the present study. The objective in this current study was to determine the efficacy of TIRI to detect apnea and hypopnea by using conventional surrogate airflow sensors as comparison standards during polysomnography. Thus, our hypothesis was that there would be a very high degree of chance-corrected agreement (κ ≥ 0.6) between TIRI and each of the conventional flow sensors, such as thermistor, Pn, and PECO2 in the detection of apnea and hypopnea. Some of the data from this manuscript has been presented at scientific meetings and published in the form of abstracts.15,16

/pdf/thermal-infrared-imaging-a-novel-method-to-monitor-airflow-v0mpwu4uql.pdf

Thermal infrared imaging: a novel method to monitor airflow during polysomnography.

In this paper, we propose a novel tracker to capture the human breathing signal through an infrared imaging method. Human facial physiology information is used to select salient thermal features on the human face as good features to track. The major component of the tracker is mean shift localization (MSL)-based particle filtering. A special measurement model is designed for particle filtering so that the tracker can handle significant head movement and object occlusion. The breathing signal is achieved based on tracking results. The experiments show that the tracker is robust and stable and the recovered breathing signal is clear enough for breathing functionality computation.

Jin Fei

Papers

Interacting with human physiology

Thermistor at a Distance: Unobtrusive Measurement of Breathing

Analysis of Breathing Air Flow Patterns in Thermal Imaging

Thermal infrared imaging: a novel method to monitor airflow during polysomnography.

Tracking human breath in infrared imaging