Feasibility of contactless pulse rate monitoring of neonates using google glass

In neonatal intensive care units in hospitals, vital signs of neonates are monitored continuously using wired sensors. However, these wired sensors introduce skin irritations, pain, discomfort and sleep disruptions for the neonates. State of the art camera-based vital sign algorithms are becoming popular as a solution to these issues. However, there are limited investigations into the feasibility of monitoring the neonates in a clinical setting with these algorithms. Also, recent emergence of a wide variety of wearable head-mounted devices, like Google Glass, enable vital sign monitoring to be ubiquitous. Again, feasibility of the use of such a device for vital sign monitoring is unknown. This paper investigates both the feasibility of using a camera-based algorithm for pulse rate monitoring of neonates in a clinical setting and the feasibility of using Google Glass for such pulse rate monitoring. The results of our research show under what conditions the monitoring of the pulse rate of neonates would be reliable and highlights the challenging conditions. Also, they give insights into the applicability of a Google Glass prototype for pulse rate monitoring and it’s current limitations.


INTRODUCTION
In neonatal intensive care units (NICU) in hospitals, the vital signs of pre-term infants as young as 24 weeks are monitored continuously for early medical diagnosis and treatment.These vital signs (including pulse rate, respiration rate and blood oxygen saturation) are monitored by multiple monitoring sensors that are attached to the neonates.The placement of these sensors and the presence of all the wires connecting the sensors lead to discomfort and even painful .stimuli when the adhesive sensors are removed.These can further lead to skin irritations and sleep disruption for the neonates.It has been proven that sensory exposure in the NICU environment causes adverse neurodevelopmental issues for the neonates [1].Contactless camera-based algorithms [2] are becoming popular as a solution to this intrusive monitoring challenge.However, the investigations into the feasibility of contactless monitoring neonates in a clinical setting are limited.
Also, there is a recent emergence of a wide variety of wearable head-mounted devices like Google Glass and Microsoft HoloLens.These devices trigger an interesting use-case for such vital sign monitoring.However, the feasibility of monitoring vital signs using such a wearable head-mounted device is unknown.
To address these challenges, this paper investigates the feasibility of: (1) monitoring the pulse rate of neonates in a clinical setting with a camera-based algorithm and (2) using Google Glass with such an algorithm.
The following are the key contributions of this paper: • A clinical study to record scenes with challenging conditions for camera-based pulse rate monitoring in the NICU.• A demonstrative prototype on the Google Glass platform with a camera-based algorithm for pulse rate monitoring of neonates.• An evaluation of the video scenes obtained during the clinical study to show the conditions for reliable contactless pulse rate monitoring.
The remainder of this paper is organized as follows: Section 2 describes current trends of camera-based pulse rate monitoring in neonatal clinical settings and vital sign monitoring with Google Glass.Section 3 introduces the clinical study.Section 4 gives the details of the demonstrative prototype on the Google Glass platform.Section 5 provides the results of the evaluation.Section 6 concludes the paper by giving insights about camera-based pulse rate monitoring.

CURRENT TRENDS
Recent trends [3][4][5][6][7] in validating camera-based pulse rate monitoring methods in neonatal clinical settings are highlighted in Table 1.The first medical study of camera-based pulse rate monitoring for neonates was reported by Scalise et al. [4].However, they recorded the infants only in supine positions together with a non-ambient green light source (green channel shows the strongest pulse signal [8]).The first to include ambient light conditions in a medical study was Aarts et al [4].They explored different scenes with challenging conditions for 19 infants in durations of 1 to 5 mins.Though Google Glass has been used in several healthcare use-cases, the only use-case for physiological monitoring has been reported by BioGlass [9].This BioGlass concept only measures the wearer's pulse rate.In contrast, the wearer (e.g., medical doctor) of our prototype can remotely determine the pulse rate of observed patient.

CLINICAL STUDY
The clinical study to monitor infants was conducted in the NICU at Maxima Medical Center (MMC), Veldhoven.The approval of the medical ethical research committee 1 at MMC and the informed parental consent of the neonates were obtained prior to the recordings in the clinical study.

Recording Setup
The recording setup of the clinical study is shown in The clinical study also includes the monitoring of camerabased respiration rates.The cameras were connected to a recording laptop (HP EliteBook 8570w) with an external data storage (WD My Book Studio II).In addition, the reference pulse rate data was recorded from the Philips Intellivue MX800 Health Monitor through a USB serial connection to the recording laptop.For medical safety standards, this connection and the electrical lines were galvanically isolated.

Protocol
For each video recording of a scene, the following sequence was followed.
1. Synchronize the time of the Intellivue MX800 Health Monitor with the recording laptop.2. Calibrate the iris, zoom and focus.3. Apply "automatic white balance" and "automatic exposure" on a facial region manually selected before each recording.
During the actual recording, they are disabled.4. Record the illumination level using a lightmeter (Voltcraft MS-1300).5. Record the skin tone using the Felix von Luschan skin chart.6. Record both the video and the reference data.
To capture the different situational conditions in the NICU, the following types of scenes are recorded.

Dataset
Table 2 highlights the details of the infants recorded.The infants had gestational ages of 32±4 weeks and postnatal ages of 31±23 days.One recording was during a phototherapy session.This is used to treat neonatal jaundice.

PROTOTYPE
Considering the realistic challenge of body motion for contactless pulse rate monitoring, we implement a motion robust remote photoplethysmographic (rPPG) algorithm proposed by Wang et al. [2], which achieves the state-of-the-art performance in motion robustness.Essentially, it exploits the spatial redundancy of a camera sensor to create a statistical pulse-signal that is immune to motion noise.

Algorithm
As illustrated in Figure 2, the motion robust rPPG algorithm consists of three steps: (1) Pixel-to-pixel pulse extraction.It uses an online object tracker to track the region of interest (e.g., the neonate's face) and dense optical flow to align the skin-pixels between adjacent frames.From the temporally aligned skin-pixel pairs, the chrominance-based rPPG algorithm (CHROM) [10] is applied to extract pulse intervals from RGB values of skin-pixels in parallel, i.e., each skin-pixel is considered as an independent rPPG-sensor;  In order to derive a robust pulse-signal from multiple signal-traces, the algorithm uses an adaptive band-pass filtering to strengthen the pulse frequency and uses principal component analysis to select the signal trace with maximal variance/energy.Consequently, it generates a clean pulse-signal that is resistant to motion noise.The pulse signal is then transformed to the frequency domain using FFT (8s window) to compute the pulse rate.Similarly, the photoplethysmographic signal from the pulse oximeter is transformed to the frequency domain using FFT (8s window) to compute the reference pulse rate for comparison.The detailed background on the algorithm is found in [2].

Google Glass Platform
The Google Glass platform is a wireless head-mounted device.This platform is equipped with a touch pad, a display, a camera, a speaker, and a battery.The prototype is based on the Google Glass Developer Explorer version 1 running Android Kitkat 4.4.2.

Technical Challenges
A custom Android application based on the above algorithm was developed.This custom application consisted of two layers: Java and native C. The Java layer was only used for user interaction while the native C layer was required for frame capturing (at 15 fps) and image processing (under 15 fps).Figure 3a shows a demonstrative screen shot of the custom application running on this prototype.The major technical challenges of this prototype were:  • Camera access -To meet the stable frame-rate of 15 fps and to disable the camera's internal algorithms used for manipulating raw data, the camera access was enabled through native C.
However, not all of the camera's internal algorithms could be disabled, due to a lack of an open camera API.
• Real-time performance -To meet the real-time performance of 15 fps for image processing, the frame size, the facial region of interest and the number of pixels tracked were all tuned.Furthermore, several functions (e.g., fast fourier transform and color conversion) were parallelized using the ARM NEON instructions.
• Prototype development -The lack of documentation of the camera API made native C (on the ARM processor) development challenging.The other GPU, DSP and camera processors were also not easily accessible, to get further performance.
In addition, a physical limitation of the Google Glass device was observed.There was significant heat dissipation with high power consumption when running this high performance (15 fps) camera application.See Figure 3b for a heat map of the Google Glass after running the application, contrasted against a hand This is because of the heavy workload in the calls to the camera service API, as characterized here [11].This excessive heating has two impacts: (1) health impact -skin damage due to excessive heat (Erythema ab igne disease) is proven in the case of laptops [12], and (2) performance impact -due to heat, the processor frequency gets throttled down.Therefore, the application cannot meet the performance required and cannot extract the pulse rate accurately.The prototype is currently limited to an operation period around 1 min before requiring a cooling down period.

RESULTS
In this section, we present the results of comparing the pulse rate obtained from the Google Glass prototype against the reference pulse rate data.In addition, we present an evaluation of the recorded video scenes with challenging conditions using the clinical study in Section 3. Figure 4a shows the pulse rate, of a neonate, from the camera-based algorithm in the prototype against the reference pulse rate data recorded from the Philips Intellivue MX800 Health Monitor.It shows that the pulse rate from the camera PPG does follow the PPG pulse rate, but with some variance.The variance is due to the lack of access to raw pixels.This is because of the inaccessible internal camera algorithms (highlighted earlier) processing these pixels.Due to the limited operational time of the Google Glass prototype, further evaluation could only be done on the multiple video scenes recorded using the clinical study.The same algorithm in the prototype was used to extract the camerabased pulse rate from the video recordings and they were compared against the reference pulse rate from oximeter sensors.The accuracy and reliability metrics for comparison were the Bland-Altman test (we only report the standard deviation (STD) of the difference between the camera pulse rate and the reference pulse rate) and the intraclass correlation coefficient (ICC), respectively [13].Table 3 shows the metric results for all the video data.As observed, the algorithm performs well with a lower STD and a higher ICC under ambient daylight conditions that have high LUX levels (see example wave sequence in Figure 4b).In the other cases, low LUX levels together with the type of light, affected both the agreement and the reliability of the vital sign extraction.In incandescent light, there is a lower intensity of the blue and green colors, which causes a color imbalance.During phototherapy, there is only a high intensity of blue color, which causes overexposure and a color imbalance as there is hardly any red or green color.
Therefore, it was necessary to investigate the limit of operation for the prototype.For that, we define a metric called the Mean Color (MC), where . For all the above scenes in Table 3, the trade-offs between the Mean Color (MC) and the ICC are plotted (see Figure 5).The MC has been normalized for the different exposure times and the gain correction values, in different scenes.A ICC value of 0.70 is observed when the MC is higher than 71 (an ICC value of 0.70 is the typically accepted value to allow an interchange of an existing measurement method to a new measurement method [13]).This MC value can be used to notify the user that the environmental conditions must be changed to increase the pulse rate reliability.

CONCLUSIONS
In this paper, we investigated the feasibility of: (1) monitoring the pulse rate of neonates in a clinical setting with a camera-based algorithm and (2) using Google Glass with such a camera-based pulse rate monitoring algorithm.The investigation showed that the Google Glass concept is feasible for wearable pulse rate monitoring, and that the al-gorithm performs well under limited conditions; further research is required to improve the robustness against illumination.Also, the existing hardware has to be replaced to achieve better performance and lower heat dissipation.If the raw video data is processed on state-of-the-art hardware, we foresee no fundamental problems with real-time performance and heat dissipation for wearable pulse rate monitoring devices.We envision that such pulse rate monitoring wearable devices can not only be used for neonatal care in hospitals, but also by paramedics on-the-go and for at-home monitoring of all types of patients.
Figure 1.There were two cameras (uEye, IDS imaging, UI-2220SE with 768x576 resolution, 20 fps) with lenses (Tamron 12VM412ASIR with manual iris, zoom and focus) observing the infant in the incubator bed.The first camera observes the facial region of the infant to observe color change for pulse rate extraction.The second camera observes the motion of the chest region for respiration rate extraction.

Figure 1 :
Figure 1: Recording Setup *The pulse oximetry probe and ECG electrode connections are not shown for simplicity. 1 The medical ethical research committee at Maxima Medical Center has reviewed the research proposal and considered that the rules laid down in the Medical Research involving Human Subjects Act (also known by its Dutch abbreviation WMO), do not apply to this research proposal.

Figure 2 :
Figure 2: Framework of motion robust rPPGSpatial pruning.After the temporal normalization, all the pulse intervals are identical.The motion-induced outliers that spread along the motion direction in the color space are pruned spatially; (3) Temporal filtering.The remaining pulse intervals (e.g., inliers) are concatenated into longterm signal-traces.In order to derive a robust pulse-signal from multiple signal-traces, the algorithm uses an adaptive band-pass filtering to strengthen the pulse frequency and uses principal component analysis to select the signal trace with maximal variance/energy.Consequently, it generates a clean pulse-signal that is resistant to motion noise.The pulse signal is then transformed to the frequency domain using FFT (8s window) to compute the pulse rate.Similarly, the photoplethysmographic signal from the pulse oximeter is transformed to the frequency domain using FFT (8s window) to compute the reference pulse rate for comparison.The detailed background on the algorithm is found in[2].
(a) Screenshot of what the doctor sees through Google Glass (b) Heatmap of Google Glass

Figure 4 :
Figure 4: Comparison of pulse rate with the reference pulse rate from the pulse oximeter sensorFigure4ashows the pulse rate, of a neonate, from the camera-based algorithm in the prototype against the reference pulse rate data recorded from the Philips Intellivue MX800 Health Monitor.It shows that the pulse rate from the camera PPG does follow the PPG pulse rate, but with some variance.The variance is due to the lack of access to raw pixels.This is because of the inaccessible internal camera algorithms (highlighted earlier) processing these pixels.Due to the limited operational time of the Google Glass prototype, further evaluation could only be done on the multiple video scenes recorded using the clinical study.The same algorithm in the prototype was used to extract the camerabased pulse rate from the video recordings and they were compared against the reference pulse rate from oximeter sensors.The accuracy and reliability metrics for comparison were the Bland-Altman test (we only report the standard deviation (STD) of the difference between the camera pulse rate and the reference pulse rate) and the intraclass correlation coefficient (ICC), respectively[13].Table3shows the metric results for all the video data.

Table 1 :
Comparison of neonatal clinical studies for camera-based pulse rate monitoring

•
Lighting -ambient daylight, ambient incandescent light and ambient low light • Coverage -with or without blankets • Camera view -top or side view • Skin tone -different skin tones • Gestational age -different ages from 24 weeks onward

Table 2 :
List of recorded infants * With photo-therapy ** With Google Glass # Von Luschan's chromatic scale

Table 3 :
Results for pulse rate extraction