Using video data of neonates admitted to the neonatal intensive care unit (NICU) we developed and compared the performance of various techniques for noncontact respiration rate (RR) estimation. Data were collected from an overhead colour and depth (RGB-D) camera, while gold standard physiologic data were captured from the hospital's patient monitor. We developed a deep learning algorithm for automatic detection of the face and chest area of the neonate. We then use this algorithm to identify time periods with low patient motion and to locate regions of interest for RR estimation. We produce a respiration signal by quantifying the chest movement using the raw RGB video, motion-magnified RGB video, and depth video. We compare this to a respiration signal derived from the changes in the green channel of the face. We were able to estimate RR from motion-magnified video and depth video, achieving a mean absolute error of less than 3.5 BPM for 69% and 67% of the time for each stream, respectively. We achieve this result without the need for skin segmentation and can apply our technique to fully clothed neonatal patients. We show that similar performance can be achieved using the depth and colour stream using this technique.

Noncontact Neonatal Respiration Rate Estimation Using Machine Vision

Video-based motion analysis recently appeared to be a promising approach in neonatal intensive care units for monitoring the state of preterm newborns since it is contact-less and noninvasive. However it is important to remove periods when the newborn is absent or an adult is present from the analysis. In this paper, we propose a method for automatic detection of preterm newborn presence in incubator and open bed. We learn a specific model for each bed type as the camera placement differs a lot and the encountered situations are different between both. We break the problem down into two binary classifications based on deep transfer learning that are fused afterwards: newborn presence detection on the one hand and adult presence detection on the other hand. Moreover, we adopt a strategy of decision intervals fusion in order to take advantage of temporal consistency. We test three deep neural network that were pre-trained on ImageNet: VGG16, MobileNetV2 and InceptionV3. Two classifiers are compared: support vector machine and a small neural network. Our experiments are conducted on a database of 120 newborns. The whole method is evaluated on a subset of 25 newborns including 66 days of video recordings. In incubator, we reach a balanced accuracy of 86%. In open bed, the performance is lower because of a much wider variety of situations whereas less data are available.

/pdf/preterm-newborn-presence-detection-in-incubator-and-open-bed-3sz901th3s.pdf

Preterm Newborn Presence Detection in Incubator and Open Bed Using Deep Transfer Learning

The development of non-contact patient monitoring applications for the neonatal intensive care unit (NICU) is an active research area, particularly in facial video analysis. Recent studies have used facial video data to estimate vital signs, assess pain from facial expression, differentiate sleep-wake status, detect jaundice, and in face recognition. These applications depend on an accurate definition of the patient’s face as a region of interest (ROI). Most studies have required manual ROI definition, while others have leveraged automated face detectors developed for adult patients, without systematic validation for the neonatal population. To overcome these issues, this paper first evaluates the state-of-the-art in face detection in the NICU setting. Finding that such methods often fail in complex NICU environments, we demonstrate how fine-tuning can increase neonatal face detector robustness, resulting in our NICUface models. A large and diverse neonatal dataset was gathered from actual patients admitted to the NICU across three studies and gold standard face annotations were completed. In comparison to state-of-the-art face detectors, our NICUface models address NICU-specific challenges such as ongoing clinical intervention, phototherapy lighting, occlusions from hospital equipment, etc. These analyses culminate in the creation of robust NICUface detectors with improvements on our most challenging neonatal dataset of +36.14, +35.86, and +32.19 in AP30, AP50, and mAP respectively, relative to state-of-the-art CE-CLM, MTCNN, img2pose, RetinaFace, and YOLO5Face models. Face orientation estimation is also addressed, leading to an accuracy of 99.45%. Fine-tuned NICUface models, gold-standard face annotation data, and the face orientation estimation method are also released here.

https://ieeexplore.ieee.org/ielx7/6287639/9791217/09791241.pdf

NICUface: Robust Neonatal Face Detection in Complex NICU Scenes

The development of non-contact patient monitoring applications for the neonatal intensive care unit (NICU) is an active research area, particularly in facial video analysis. Recent studies have used facial video data to estimate vital signs, assess pain from facial expression, differentiate sleep-wake status, detect jaundice, and in face recognition. These applications depend on an accurate definition of the patient&#x2019;s face as a region of interest (ROI). Most studies have required manual ROI definition, while others have leveraged automated face detectors developed for adult patients, without systematic validation for the neonatal population. To overcome these issues, this paper first evaluates the state-of-the-art in face detection in the NICU setting. Finding that such methods often fail in complex NICU environments, we demonstrate how fine-tuning can increase neonatal face detector robustness, resulting in our NICUface models. A large and diverse neonatal dataset was gathered from actual patients admitted to the NICU across three studies and gold standard face annotations were completed. In comparison to state-of-the-art face detectors, our NICUface models address NICU-specific challenges such as ongoing clinical intervention, phototherapy lighting, occlusions from hospital equipment, etc. These analyses culminate in the creation of robust NICUface detectors with improvements on our most challenging neonatal dataset of +36.14, +35.86, and +32.19 in AP30, AP50, and mAP respectively, relative to state-of-the-art CE-CLM, MTCNN, img2pose, RetinaFace, and YOLO5Face models. Face orientation estimation is also addressed, leading to an accuracy of 99.45%. Fine-tuned NICUface models, gold-standard face annotation data, and the face orientation estimation method are also released here. 

/pdf/nicuface-robust-neonatal-face-detection-in-complex-nicu-1278yf0i.pdf

We have collected a multi-modal neonatal patient dataset suitable for development of noncontact continuous monitoring techniques. Data was simultaneously collected from a RGB-D video camera placed above the patient and a pressure sensitive mat (PSM) beneath the patient. This paper explores the use of various transforms to achieve registration between the video image plane and the PSM, with the ultimate goal of fusing PSM and video modalities of our patient dataset. A series of experiments were conducted to evaluate transforms requiring different numbers of registration landmarks. The expected error in determining landmark locations in both video and PSM is characterized, including the impact of camera offset, registration instrument angle, the degree of collinearity of landmarks, the spacing between landmarks and the use of “secondary” landmarks estimated from patient anatomy. A landmark spacing greater than 450 cm2 is recommended since it achieves an error of less than 3 cm when aligning points between video and PSM planes. For a top-down camera view, a similarity transform is recommended while for an angled camera view, a projective transform is recommended.

Fusing Pressure-Sensitive Mat Data with Video through Multi-Modal Registration

Noncontact video-based patient monitoring promises several advantages over wearable sensors, particularly for patients in the NICU who have fragile skin. However, such approaches often require definition of a region-of-interest (ROI), such as the patient’s forehead. For example, a number of neonatal monitoring studies have estimated heart rate and respiration from video by first manually cropping the face of the patient before performing analyses within that region. Relying on a static ROI can fail due to patient motion or during clinical interventions, thereby demanding additional manual ROI selection over the course of the monitoring period. Widely used face detection algorithms tend to fail in a neonatal context. We therefore propose a semi-automated method where the ROI is automatically and repeatedly reinitialized to ensure robustness of the ROI for continuous monitoring. Factors such as the displacement of the patient and the change in patient poses are addressed using multiple computer vision techniques before selecting a comprehensive method for ROI tracking. Results were obtained from three patients admitted at the NICU using 20-minute videos including periods of rest, motion, and occlusion events. Compared to a static ROI, the proposed method achieves significantly improved tracking of the patient’s face, as demonstrated by an area under the curve > 0.63 across all patients.

Neonatal Face Tracking for Non-Contact Continuous Patient Monitoring

Newborns admitted to the neonatal intensive care unit (NICU) require a high level of care due to their precarious condition. Nurses typically monitor their vital signs continuously using wearable sensors such as electrocardiogram (ECG) electrodes placed on their chest and a pulse oximeter on a limb. When the patient moves, this can cause motion artifacts on one or more physiologic signals, potentially resulting in a false alarm on the patient monitor. We therefore propose a motion detection algorithm to mitigate these alarms by providing context. Using a camera positioned above the crib or overhead warming bed, we recorded videos from six patients and annotated all patient movements. These data were used to train and evaluate two different approaches for non-contact motion detection. Results were stronger for the optical flow technique than for the long short-term memory network approach. This represents a challenging problem due to variable lighting, patient clothing and bed coverings, and the complex clinical environment in the NICU.

Video-Based Neonatal Motion Detection

Patient movements can cause motion artifacts on physiological signals and can result in false alarms in a continuous patient care environment. This paper explores the use of data from a pressure sensitive mat (PSM), placed below neonates in the neonatal intensive care unit (NICU), to detect patient movement. The centre of pressure (COP) is tracked over time using a sliding window. Windows exhibiting large deviations in the COP are indicative of patient motion. Local averaging and window boundary suppression leads to improved movement detection accuracy. Using data from five patients, optimal parameter values were determined using a grid search method. After averaging the best performing parameters, a window size of six seconds was found to be optimal across patients, resulting in an area under the ROC curve of 0.909. Detection accuracy is maintained when evaluated on a patient not used to optimize algorithm parameters, with an accuracy of 93.4%. It is hoped that the movement detection algorithm developed in this work will be useful for gating motion-artifact related false alarms from neonatal patient monitors.

Samreen Aziz

Papers

Neonatal Face Tracking for Non-Contact Continuous Patient Monitoring

Video-Based Neonatal Motion Detection

Detection of Neonatal Patient Motion Using a Pressure-Sensitive Mat