Siamese-rPPG Network:

Remote Photoplethysmography Signal

Estimation from Face Videos
Yun-Yun Tsou, Yi-An Lee, Chiou-Ting Hsu and Shang-Hung Chang
National Tsing Hua University
Linkou Chang Gung Memorial Hospital
Heart rate estimation
•With specific contact devices
• Electrocardiography (ECG) and Photoplenthesmography
• Disadvantages
• Require specific devices and professional attention
• Hardly extend to mass monitoring for a large group of subjects
Heart rate estimation
•Contactless video-based methods
• Remote photoplethysmography (rPPG)
• Analyze blood volume changes in optical information

PPG signal
70 bpm
ECG signal
Heart rate estimation
•Contactless video-based methods
• Traditional method
• Vulnerable to environmental interference and subjects’ motion
• Performance heavily depends on the quality of videos
• Learning-based method
• Require image pre-processing step
• Impractical for real-world applications
Introduction
•Goal:
• Estimate rPPG signals directly from input video sequences
• Siamese-rPPG network
• Simultaneously learn rPPG signals from two face regions
• 3D convolutional layers
• Characterize the spatial and temporal information of the videos
 Related work: EVM-CNN
• EVM-CNN: Real-Time Contactless Heart Rate Estimation From Facial Video [2]

[2] Ying Qiu, Yang Liu, Juan Arteaga-Falconi, Haiwei Dong, and Abdulmotaleb E. Saddik. 2019. EVM-CNN: Real-Time Contactless Heart Rate Estimation From Facial Video. IEEE Transactions on Multimedia21
, 7 (July 2019), 1778–1787
Related work: HR-CNN
• Visual Heart Rate Estimation with Convolutional Neural Network [1]

[1] Radim Spetlík, Vojtêch Franc, Jan Čech, and Jiří Matas 2018. Visual Heart Rate Estimation with Convolutional Neural Network. In Proceedings of British Machine Vision Conference
Our approach
•Siamese-rPPG network
• Face detector
• Locate two regions of interest (ROIs)
• 3D convolutional layers
• Directly process the located videos
• Two weight sharing networks
• Mutually learn the characteristic information from two ROIs
Our approach
•Siamese-rPPG network
• Weight-sharing network
• Heterogeneous features
• Each branch learn the characteristic features of its corresponding region
• Homogeneous features
• Capture the homogeneity from two different facial regions
Our approach
•Siamese-rPPG network
• Loss function
• Pearson correlation

• Negative Pearson loss

Dataset
•COHFACE dataset
• 160 heavily compressed videos from 40 subjects
• Training set contains 24 subjects and the testing set contains 16 subjects

•UBFC-RPPG dataset
• 42 videos from 42 subjects
• The training set contains 28 subjects and testing set contains 14 subjects

•PURE dataset
• 60 videos from 10 subjects
• The training set contains 7 subjects and testing set contains 3 subjects

11
Experiment
•Evaluation measurement
• Pearson correlation coefficient (R) • Root mean square error (RMSE)

• Mean absolute error (MAE) • Precision at 2.5 or 5 bpm

12
Experiment
•Ablation study on COHFACE dataset

13
Experiment
•Results for concatenation and addition

(a) Concatenation (b) Element-wise addition

14
Experiment
•Comparison on COHFACE dataset

(a) Controlled scenario

(b) Natural scenario

15
Experiment
•Comparison on UBFC-RPPG dataset

16
Experiment
•Comparison on PURE dataset
Experiment
•Comparison of cross-dataset estimation
Conclusion

•Propose Siamese-rPPG network to estimate rPPG from facial videos

• 3D convolutional layers
• Model the spatial and temporal characteristics of rPPG signals
• Weight-sharing mechanism
• Learn robust, distinctive, and complementary features from multiple facial regions
• Outperform all the existing methods

19