A PROJECT REPORT

ON
HUMAN PULSE DETECTION BASED ON EULERIAN VIDEO
MAGNIFICATION ON ARM CORTEX-A PROCESSORS
A Thesis report submitted in partial fulfillment
of the requirement for the award of the degree in

MASTER OF TECHNOLOGY
In
EMBEDDED SYSTEMS
SUBMITTED BY

N NARAYANA MURTY
(12MN1D5503)
Under the esteemed guidance of

Mr. A Venkateswara Rao,

M.Tech.

Associate professor, ECE

DEPARTMENT OF ELECTRONICS AND COMMUNICATION ENGINEERING

SRI SIVANI INSTITUTE OF TECHNOLOGY

(Affiliated to JNTU, Kakinada)
Chilakapalem Jn, Etcherla (M),
Srikakulam (Dist)

SRI SIVANI INSTITUTE OF TECHNOLOGY

(Approved by AICTE & Affiliated to JNT University Kakinada)
Chilakapalem Jn., Etcherla (MD) -532402 Srikakulam.Dist

DEPARTMENT OF ELECTRONICS COMMUNICATION AND

ENGINEERING

CERTIFICATE
This is to certify that the project report entitled human pulse
detection based on Eulerian video magnification on arm cortex-a processors is a bonafide
work done by

N Narayana Murty (12MN1D5503) submitted in partial fulfillment of the

requirements for the award of the degree of MASTER OF TECHNOLOGY in EMBEDDED

SYSTEMS (ECE) of Sri Sivani Institute of Technology, to the Jawaharlal Nehru
Technological University (JNTU), Kakinada, is a record of original work carried out by him
under my guidance and supervision. The results of investigation enclosed in this report have
been verified and found satisfactory. The results embodied in this project report have not been
submitted to any other University or Institute for the award of any other degree or diploma.

Project Guide

Head of the Department

Mr. A Venkateswara Rao, M.Tech.

Dept. of ECE

Associate professor, ECE

External Examiner

ACKNOWLEDGEMENT
It is indeed with a great sense of pleasure and immense sense of gratitude that I
acknowledge the help of each individuals for their cooperation in bringing the successful
completion of project work.
I feel elated in manifesting my sense of gratitude to my guide and Head of the
Department , Mr. A Venkateswara Rao, M.Tech. ,for his valuable guidance. He has been a
constant source of inspiration for me and I am very sincerely thankful to his support and
invaluable advice.
I am highly indebted to Principal Dr. G Ramesh Babu ,M.Tech Ph.D for the facilities
provided to accomplish this project.
I am extremely great full to my Departmental staff members, lab technicians and non
teaching staff members for their extreme help throughout my project.
Finally I express my heart full thanks to all of my friends who helped me in successful
completion of this project.

N Narayana Murty
(12MN1D5503)

DECLARATION
I hereby declare that this project entitled human pulse detection based on Eulerian
video magnification on arm cortex-a processors is an original work done by me and
submitted towards partial fulfillment for the award of the Master of Technology in Embedded
systems and has not been submitted to any other University or institute for the award of any
degree.

N Narayana Murty,
(12MN1D5503)

Contents
Abstract
1. Introduction .

1.1. Context

1.2. Motivation

1.3. Objectives

1.4. Contributions

2. State of Art

2.1. Photo-plethysmography

2.2. Signal post-processing

2.2.1. Independent Component Analysis

2.2.2 Eulerian Video Magnification

2.2.2.1. Spatial filtering

2.2.2.2. Temporal filtering

10

2.2.2.3 Emphasize color variations for human pulse

11

2.2.3 Detrending
2.3 Heart rate estimation .

12
12

2.3.1 Power spectrum

13

2.3.2 Pulse wave detection

13

2.4 Technologies

14

2.4.1 ARM Cortex A Processor with NEON

14

2.4.2 Embedded Linux

15

2.4.3 Qt Frame Work

15

2.4.4 OpenCV Computer Vision Library

15

2.5 Summary

3. Hardware

16

17

3.1 S5PV210 Processor

18

3.2 Development Board & Target System

19

3.2 Host System

21

3.4 Summery

21

4. Software and device development Process .

22

4.1 Host environment

23

4.1.1 ARM DS-5

24

4.1.1.1 Project manager

25

4.1.1.2 Editors

25

4.1.1.3 DS-5 Debugger

26

4.1.1.4. Advanced Session Control & System Views

26

4.1.1.5. Non-Intrusive Trace Analysis

26

4.1.1.6. Linux Awareness

27

4.1.1.7. File Transfer to Target

27

4.1.2. Qt Creator

27

4.1.2.1. Editors

27

4.1.2.2. Debuggers

28

4.1.3. GCC cross-compiler Tool chain

28

4.1.4. NFS Server

30

4.1.5.TFTP Server on Host

30

4.1.6. Minicom

30

4.3 Target Environment

30

4.3 Summary

31

5. Device Software &Pulse vital signs monitoring application

5.1 Device Software

32
33

5.1.1 Boot loader for x210ii

33

5.1.2. Linux Kernel

34

5.1.3. Device driver

34

5.1.4. File system

34

5.1.5. Target Libraries

35

5.2 Pulse Detector vital signs monitoring application

35

5.2.1 Problem description

35

5.2.2 Linux based implementation of Eulerian Video Magnification

35

5.2.3 Vital signs monitoring

36

5.3 Implementation details

37

5.3.1 Overview

37

5.3.2 Eulerian Video Magnification implementations

38

5.3.2.1 EvmGdownIdeal

39

5.3.2.2 EvmGdownIIR

40

5.3.2.3 EvmLpyrIIR

40

5.4 Performance optimized version EvmGdownIIR

42

5.4.1. Face detection

43

5.4.2. Signal validation

44

5.4.3. Heart rate estimation

45

5.4.4. OpenCV

45

5.5. Pulse Detector

46

5.6. Summary

47

6 . Result and Conclusion

49

6.1 Heart rate estimation comparison

50

6.2. Future work

50

6.3. Conclusions

51

6.3 Summary

53

References

54

Abstract of Project Work

Today world is progressing in all fields of engineering. The devices become
smart enough to take oral orders. This is achieved by the embedded computing become more
powerful. The olden days work stations performance can be achieved on today's smart phone
or hand-held battery driven devices.
More and more complex algorithms are even being implemented on embedded
devices due to increase in CPU power of computation being used in today's embedded world.
The human pulse detection based on Eulerian Video Magnification on arm
cortex-a processors is bit complex algorithm presented at SIGGRAPH 2012 conference,
which reviles the subtle changes in video by magnifying the video changes by colour or
motion. How ever the projected algorithm is computationally more complex calculations are
to be implemented. This is possible only on high end system like x86_64 servers. The
applications in medical field, security field are very huge.
So we optimized the the Eulerian video magnification Process and implemented
it on Embedded Linux ported ARM Cortex-A8 platform. The implementation was possible
because of the NEON SIMD engine, this is more powerful and competes with the x86 counter
parts on hand-held embedded systems. The final device is named as Pulse Detector , uses
the algorithms and captures the video frames via webcam attached to device and estimates the
heart-rate more accurately and displays graph and pulse rate on the interface application.

List of Figures
2.1 : Overview of Eulerian Video Magnification.

2.2 : Examples of temporal filters. (source: [WRS + 12])

10

2.3 : Colour changes of face using Eulerian Video Magnification (Source: [WRS 12])

11

2.4 : Original RR series and fitted trends (above), and detrended without magnitude
distortion(below).(source:[TRaK02])

12

3.1 : Samsung S5PV210 MCU(Micro controller unit)

19

3.2 : X210ii_B development board

21

4.1 : Host and target system setup sketch.

25

4.2 : DS-5 IDE showing development of application

26

4.3 : QtCreator IDE x210ii application connection setup.

29

5.1: Overview of the implemented algorithm to obtain the heart rate of a person from a
webcam or video using the Eulerian Video Magnification method.

37

5.2: Overview of the Eulerian Video Magnification method steps.

39

5.3 : shows the gaussian & laplacian Pyramid algorithm

41

5.4 : Overview of the interaction between the library and application

47

5.5 : Snap shot of Pulse Detector application

48

List of tables
Table 5.1 : Table of target file system (rootfs) directories

35

Chapter 1
Introduction

Chapter 1
Introduction

This chapter introduces this dissertation, by first presenting its context,

motivation, and projects objectives, on sections 1.1, 1.2, and 1.3, respectively. Finally, section
1.4 describes the document outline.

1.1 Context
Eulerian Video Magnification is a method, recently presented at SIGGRAPH 1
2012, capable of revealing temporal variations in videos that are impossible to see with the
naked eye. Using this method, it is possible to visualize the flow of blood as it fills the face
[WRS + 12]. And to assess the heart rate in a contact-free way using a camera [WRS + 12,
PMP10, PMP11]. The main field of this research work is image processing and computer
vision, whose main purpose is to translate dimensional data from the real world in the form of
images into numerical or symbolical information. Other fields include medical applications,
software development for handheld devices, digital signal processing.
This research work was developed at Khyatha Abhizit Pharma and Healthcare
Systems , visakhapatnam. An organization working in reasearch and development of medical
devices. From their work and inspiration I attempted to work on portable device to implement
this eulerian video magnification algorithm for hart rate estimation similar to ECG.
aims on the creation of scientific knowledge capable of generating added value to
its clients and partners, exploring technology innovations oriented towards economic
growth, the social well-being and the improvement of the quality of life of its end users.

1.2 Motivation
There has been some successful effort on the assessment of vital signs, such as,
heart rate, and breathing rate, in a noninvasive way (contact-free) using a web camera [WRS
+ 12, PMP10, PMP11], and even a smartphone [Tec13, Phi13]. Other similar products, which
require specialist hardware and are thus expensive, include laser Doppler [UT93], microwave

Doppler radar [Gre97], and thermal imaging [GSMP07]. Since it is a cheaper method of
assessing vital signs in a contact-free way than the above products, this research work has
potential for advancing fields, such as, tele- medicine, personal health-care, and ambient
assisting living.
Despite the existence of very similar products by Philips [Phi13] and ViTrox
Technologies [Tec13] to the one proposed on this research work, none of these implement the
Eulerian Video Magnification method. Due to being recently proposed, the Eulerian Video
Magnification method implementation has not been tested in hand held devices and smartphones yet.

1.3 Objectives
In this work, an linux application for monitoring vital signs based on the
Eulerian Video Magnification method which works ARM Cortex A8 (any cortex A series)
processors will be developed, which should include the following features:

heart rate detection and assessment based on the Eulerian Video Magnification
method;

display real-time changes, such as, the magnified blood flow, obtained from the
Eulerian Video Magnification method;

deal with artifacts motion, due to, person and/or smartphone movement.
In order to accomplish that, the feasibility of the Eulerian Video Magnification

method on ARM Cortex A series processors has to be tested. It should be noted that a
straightforward implementation of the Eulerian Video Magnification method is not possible,
due algorithm's computational complexity, that of low powered processors. First, the Eulerian
Video Magnification method provides motion magnification along with color magnification
which will introduce several problems with artifacts motion. Second, the requirement of
implementing a real-time hand-held device application will create performance issues which
will have to be addressed and trade-offs will have to be considered.
The application performance should then be evaluated through tests with
several individuals and the assessed heart rate compared to the ones detected by another
application.

1.4 Contributions
The work of this thesis contributes to the current state of the art by testing the
feasibility of implementing a performance optimized version of the Eulerian Video
Magnification method for color amplification in order to be capable of being used on a handheld at a reasonable frame rate.
In addition, it also tests the usage of such method in the assessment of a
persons heart rate using the web-camera. This shows that the method is ready to be used on
mobile/handheld devices and that it can start being used in other kinds of applications.

Chapter 2
State of the Art

Chapter 2
State of the art
This chapter focus on the heart rate estimation from a persons face captured
through a simple webcam. Section 2.1 describes the concept that explains how the cardiac
pulse is detected from a persons face in a remote, contact-free way. Post-processing methods,
which may be applied to the retrieved signal, are detailed on section 2.2. In order to estimate
the heart rate, some techniques are also detailed on section 2.3. Finally, section 2.4 reviews
the main technologies and tools used throughout this work.

2.1 Photo-plethysmography
The photoplethysmographic (PPG) waveform, also known as the pulse
oximeter waveform, is one of the most commonly displayed clinical waveforms. First
described in the 1930s, the technology behind the waveform is simple. The waveform, as
displayed on the modern pulse oximeter, is an amplified and highly filtered measurement of
light absorption by the local tissue over time. It is optimized by medical device manufacturers
to accentuate its pulsatile components. Physiologically, it is the result of a complex, and not
well understood, interaction between the cardiovascular, respiratory, and autonomic systems.
All modern pulse oximeters extract and display the heart rate and oxygen saturation derived
from the PPG measurements at multiple wavelengths. As is, the PPG is an excellent
monitor for cardiac arrhythmia, particularly when used in conjunction with the
electrocardiogram (ECG). With slight modifications in the display of the PPG (either to a strip
chart recorder or slowed down on the monitor screen), the PPG can be used to measure the
ventilator-induced modulations which have been associated with hypovolemia. Research
efforts are under way to analyze the PPG using improved digital signal processing methods to
develop new physiologic parameters. It is hoped that when these new physiologic parameters
are combined with a more modern understanding of cardiovascular physiology (functional
hemodynamics) the potential utility of the PPG will be expanded. The clinical researcher's
objective is the use of the PPG to guide early goal-directed therapeutic interventions (fluid,
vasopressors, and inotropes), in effect to extract from the simple PPG the information and
therapeutic guidance that was previously only obtainable from an arterial pressure line and the
pulmonary artery catheter.

Photo-plethysmography (PPG) is the concept of measuring volumetric changes

of an organ optically. Its most established use is in pulse oximeters. PPG is based on the
principle that blood absorbs more light than surrounding tissue thus variations on blood
volume affect light reflectance [VSN08]. The use of dedicated light sources and infra-red
wavelengths, and contact probes has been the norm [UT93, Gre97, GSMP07]. However,
recently, remote, non-contact PPG imaging has been explored [WMVdS05, HZCS08].
The method used on the article [VSN08] captures the pixel values (red, green,
and blue channels) of the facial area of a previously recorded video where volunteers were
asked to minimize movements. The pixel values within a region of interest (ROI) were then
averaged for each frame.
This spatial averaging was found to significantly increase signal-to-noise ratio.
The heart rate estimation was then calculated by applying the Fast Fourier transform and the
power spectrum as explained on section 2.3.1. The authors of [VSN08] demonstrate that the
green channel features a stronger cardiac signal as compared to the red and blue channels.
This is a strong evidence that the signal is due to variations in the blood volume, because
hemoglobin absorbs green light better than red and blue light.

2.2 Signal post-processing

After obtaining the raw pixel values (red, green, and blue channels), a
combination of the following methods may be used to extract and improve the reflected
plethysmography signal. However, each method introduces complexity and expensive
computation.
2.2.1 Independent Component Analysis
Independent Component Analysis is a special case of blind source separation
and is a technique for uncovering independent signals from a set of observations that are
composed of linear mixtures of the underlying sources [Com94].
In this case, the underlying source signal of interest is the cardiac pulse that
propagates throughout the body, which modifies the path length of the incident ambient light
due to volumetric changes in the facial blood vessels during the cardiac cycle, such that
subsequent changes in amount of reflected light indicate the timing of cardiovascular events.
By recording a video of the facial region, the red, green, and blue (RGB) color
sensors pick up a mixture of the reflected plethysmographic signal along with other sources of
fluctuations in light due to artifacts. Each color sensor records a mixture of the original source
7

signals with slightly different weights. These observed signals from the red, green and blue
color sensors are denoted by x1 (t), x2 (t) and x3 (t) respectively, which are amplitudes of the
recorded signals at time point t. In conventional Independent Component Analysis model the
number of recoverable sources cannot exceed the number of observations, thus three
underlying source signals were assumed, represented by s1 (t), s2 (t) and s3 (t). The
Independent Component Analysis model assumes that the observed signals are linear mixtures
3

of the sources, i.e. xi(t) =

a ij s j (t)

for each i = 1, 2, 3. This can be represented compactly

j=1

by the mixing equation

x(t) = As(t) ..(2.1)
where the column vectors x(t) = [x 1 (t), x 2 (t), x 3 (t)] T , s(t) = [s1 (t), s2 (t),
s3 (t)] T and the square 3 3 matrix A contains the mixture coefficients a ij . The aim of
Independent Component Analysis model is to find a separating or demixing matrix W that is
an approximation of the inverse of the original mixing matrix A whose output
s^ (t)=W x(t) ..(2.2)
is an estimate of the vector s(t) containing the underlying source signals. To
uncover the independent sources, W must maximize the non-Gaussianity of each source. In
practice, iterative methods are used to maximize or minimize a given cost function that
measures non Gaussianity [PMP10, PMP11].
2.2.2 Eulerian Video Magnification
In contrast to the Independent Component Analysis that focuses extracting
single number, The Eulerian Video Magnification uses localized spatial pooling and temporal
filtering to extract the signal corresponding to cardiac pulse.

Figure 2.1: Overview of Eulerian Video Magnification.

This allows for amplification and visualization of heart rate signal at each
location of face. This creates potential for monitoring and diagnostic applications to medicine,
i.e. the asymmetry in facial blood flow can be a symptom of arterial problem.
Besides color amplification, the Eulerian Video Magnification method is also
able to reveal low-amplitude motion which may hard or impossible to naked eye to see.
Previous attempts to unveil imperceptible motions in videos have been made such as, [LFT +
05] which follows a Lagrangian perspective, as in fluid dynamics where the trajectory of
particle is tracked over time. By relying on accurate motion estimation and additional
techniques to produce good quality synthesis, such as, motion segmentation and image inpainting, the algorithm complexity and computation is expensive and difficult.
In contrast, we are inspired by the Eulerian perspective, where properties of a
voxel of fluid, such as pressure and velocity, evolve over time. In our case, we study and
amplify the variation of pixel values over time, in a spatially-multiscale manner. In our
Eulerian approach to motion magnification, we do not explicitly estimate motion, but rather
exaggerate motion by amplifying temporal color changes at fixed positions. We rely on the
same differential approximations that form the basis of optical flow algorithms.
This method we approach, illustrated in figure , combines spatial and temporal
processing to emphasize subtle temporal changes in video. First the video sequence is
decomposed into different spatial frequency bands. Because they exhibit different signal-tonoise ratios, they may be magnified differently. In general case, the full Laplacian pyramid
may be computed. Then , temporal processing is performed on each spatial band. The
temporal processing is uniform for all spatial bands, and for all pixels within each band. After
that, the extracted bandpass signal is magnified by a factor of , which can be specified by the
user, and may be attenuated automatically. Finally, the magnified signal is added to the
original image and the spatial pyramid collapsed to obtain the final output.

2.2.2.1 Spatial filtering

As mention before, the work of [WRS + 12] computes the full Laplacian
pyramid [BA83] as a general case for spatial filtering. Each layer of the pyramid may be
magnified differently because it may exhibit different signal-to-noise ratios, or contain spatial
frequencies for which the linear approximation used in motion magnification does not hold
[WRS + 12, Section 3]. Spatial filtering may also be used to significantly increase signal-tonoise ratio, as previously mention on section 2.1 and demonstrated on the work of [VSN08]
9

and [WRS + 12]. Subtle signals, such as, a persons heart rate from a video of its face, may be
enhanced this way. For this purpose the work of [WRS + 12] computes a layer of the
Gaussian pyramid which may be obtained by successively scaling down the image by
calculating the Gaussian average for each pixel.
However, for the signal of interest to be revealed, the spatial filter applied must
be large enough. Section 5 of [WRS + 12] provides an equation to estimate the size for a
spatial filter needed to reveal a signal at a certain noise power level:
S ( )=S (r )= 02=k

2
r

where S( ) represents the signal over spatial frequencies, and since the
wavelength, , cutoff of a spatial filter is proportional to its radius, r, the signal may be
represented as S(r). The noise power, 2 , can be estimated using to the technique of
[LFSK06]. Finally, because the filtered noise power level, 02 , is inversely proportional to r
2 , it is possible to solve the equation for r, where k is a constant that depends on the shape of
the low pass filter.
2.2.2.2 Temporal filtering
Temporal filtering is used to extract the motions or signals to be amplified.
Thus, the filter choice is application dependent. For motion magnification, a broad bandpass
filter, such as, the butterworth filter, is preferred. A narrow bandpass filter produces a more
noise-free result for color amplification of blood flow. An ideal bandpass filter is used on
[WRS + 12] due to its sharp cutoff frequencies. Alternatively, for a real-time implementation
low-order IIR filters can be useful for both: color amplification and motion magnification.
These filters are illustrated on 2.2.

figure 2.2 :Examples of temporal filters. (source: [WRS + 12])

10

2.2.2.3 Emphasize color variations for human pulse

The extraction of a persons cardiac pulse using the Eulerian Video
Magnification method was demonstrated in [WRS + 12]. It was also presented hard that or
using the right configuration can be help extract the desired video signal.
There are four steps to take when processing a video using Eulerian Video
Magnification method.
1. Select a temporal bandpass filter.
2. Select an amplification factor, ;
3. select a spatial frequency cutoff (specified by spatial wavelengh c ) beyond
which an attenuated version of is used;
4. select the form of the attenuation for either force to zero for all < c ,
or linearly scale down to zero.
For human pulse color variation, two temporal filters used, first selecting
frequencies within 0.4-4Hz, corresponding to 24-240 beats per minute (bpm), then a narrow
band of 0.83 1Hz (50-60 bpm) may be used, if the extraction of the pulse rate was
successful.
To emphasize the color change as much as possible, a large amplification
factor, 100, and spatial frequency cutoff, c 1000, is applied formulations With an
attenuation of to for spatial wavelengths below c .The resulting output can be seen in figure

figure 2.3: Colour changes of face using Eulerian Video Magnification (Source: [WRS 12])

11

2.2.3 Detrending
Detrending is a method of removing very large ultralow-frequency trends
motion an input signal without any magnitude distortion, acting as an high-pass filter. The
main advantage of the method presented on the work of [TRaK02], compared to methods
presented in [LOCS95] and [PB90], is its simplicity.
The method consists of separating the input signal, z, into two components, as
z = z

stat

+ z

trend

, where z

stat

is the nearly stationary component, and z trend is the low

frequency aperiodic trend component.

Figure 2.4 : Original RR series and fitted trends (above), and detrended without magnitude
distortion(below).(source:[TRaK02])
An estimation of the nearly stationary component, Z ^stat , can be obtained
using the equation below. The detailed derivation of equation can be found in [TRaK02].
Z ^stat=(I (I + 2 DT2 D 2))Z
Where I is the identity matrix, D2 is the discrete approximation of the second
order, and is the regularization parameter.
Figure : presents an example of what this method is able to achieve. The
examples, taken from the work of [TRaK02], uses real RR series and effect of the method on
time and frequency domain analysis of heart rate variability is demonstrated not to lose any
useful information.

2.3 Heart rate estimation

In order to convert the extracted plethysmographic signal into the number of
beats per minute (bpm), further processing must be done. Below two methods capable of
achieving this goal are highlighted.
12

2.3.1 Power spectrum

Fourier transform is a mathematical transform capable of converting a function
of time, f(t), into a new function representing the frequency domain of the original function.
To calculate the power spectrum, the resulting function from the Fourier transform is then
multiplied by itself.
Since the values are captured from video, sequence of frames are the function
of time is actually discrete, with frequency rate equal to the video frame rate, FPS. The index,
i, corresponding to the maximum of the power spectrum can then be converted intoa
frequency value, F, using the equation:
F=

i FPS
2N

where N is the size of the signal extracted. F can then be multiplied by 60 to

convert it to beats per minute, and have an estimation of the heart rate from the extracted
signal.

2.3.2 Pulse wave detection

In [NI10], an automated algorithm for fast pulse wave detection is presented.
The algorithm is capable of obtaining an estimative of the heart rate from PPG signal, as an
alternative to the power spectrum described above. Moreover, it also introduces validation to
the waveform detection by verifying its shape and timing. Below a simplified description of
the algorithm is presented. A more detailed description can be found in [NI10].
1) Identification of possible peaks and foots of individual pulses
(a) Maximum (MAX)
The signal is divided into consecutive 200ms time intervals and for every
segment the absolute maximum is determined. Some of these maximums are
rejected: if they fall below a predetermined amplitude threshold; or if the
distance between two maximums is less than or equal to 200ms, then the lower
maximum is rejected.
(b) Minimum (MIN)
The absolute minimum is determined between every two adjacent maximums.
A minimum is rejected, if it is above a predetermined amplitude threshold.
When a minimum is rejected, the lower-amplitude maximum of the two
13

maximum adjacent to the rejected minimum is discarded too.

2) Examination and verification of the rising edges
(a) Validation of a single rising edge
If a rising edge is rejected, its maximum and minimum are rejected. A rising
edge is rejected, if its amplitude (AMPL = MAX MIN) is lower than
amplitude threshold; or its duration is lower than a threshold that depends on
the sampling rate; or its amplitude does not increase smoothly.
(b) Estimation of the similarity of a rising edge to preceding and following rising
edges accepted as valid
Two rising edges are considered similar, if the amplitude of the loweramplitude rising edge is greater than 50% of the amplitude of the higheramplitude rising edge; and if the maximum of the lower-amplitude rising edge
is between 60% of the maximum of the higher-amplitude rising edge; and if
the minimum of the lower-amplitude rising edge is between 60% of the
minimum of the higher-amplitude rising edge; and if the duration of the shorter
rising edge is greater then 33% of the duration of the longer rising edge. The
valid rising edges are then categorized according to its characteristics for the
following step. The categorization description is suppressed for brevity and can
be found at [NI10].
(c) Verification of the current rising edge
The rising edges categorized on the previous step are considered valid edges of
a pulse wave if they fulfill at least one of the decision rules presented on
[NI10] and suppressed for brevity.
The validation process described here is important for discarding signals which
are not representative of pulse waves. Providing a way of calculating the heart rate estimation
only on valid pulse signals.

2.4 Technologies
Below two of the main technologies that will be used during this research work
are shortly described.
2.4.1 ARM Cortex A Processor with NEON
The ARM Cortex Application series processors are target to high end
14

computing applications on risk processors with low power range. Comparative to all ARM
processors Cortex A series processors are having on chip co-processors such as math coprocessor for floting point computation, NEON (SIMD) engine for single instruction multiple
data in parallel. The NEON is used for audio video data and codecs processing and DSP based
algorithms implementation. So we selected ARM Cortex A8 Processor for our device
development.
2.4.2 Embedded Linux
The most popular OS in the embedded world and most flexible OS for
development. The source code is freely available and many more tools such as gcc C and
C++ compiler are available in GPL V3 licensing. The Linux Porting to ARM Cortex A
Processor is a bit challenging and can be done . The libraries glibc and other are freely
available on the internet.
2.4.3 Qt Frame Work
This is a GUI toolkit for developing an embedded device application with qws.
Qws works as run time graphics engine where as qt libraries support the appliction in both
development and run time. Qt Frame work is totally written in C++ with built in event
management system. The event management system is most important in GUI based
application development and this Frame work is also available in LGPL license, So device
development is very easy on it.
2.4.4 OpenCV Computer Vision Library
OpenCV is a library of programming functions mainly aimed at real-time
image processing. To support these, it also includes a statistical machine learning library.
Moreover, it is a cross-platform and open source library that is free to use and modify under
the BSD license.
OpenCV was built to provide a common infrastructure for computer vision
applications and to accelerate the use of machine perception in the commercial products.
OpenCV is written in C/C++ There are binding for other languages, such as, Python, Java,
and even Android. However, Java and Android implementation is recent and lacks features
and stability.

15

2.5 Summary
This chapter starts by describing the concept behind the extraction of cardiac
pulse from a persons face captured through a simple video or webcam. It then presents
several possible post-processing methods for improving the extraction of the actual pulse
signal. These methods include:

Independent Component Analysis, a method capable of uncovering independent

signals from a set of observations that are composed of linear mixtures of the
underlying sources.

Eulerian Video Magnification, a method inspired by the Eulerian perspective that

exaggerates color variations by analyzing how each pixel value changes over time.

Detrend, a method which removes small trends from an input signal without distorting
its amplitude.
Then algorithms for obtaining the actual beats per minute of the heart rate from

the signal are described:

Power spectrum, a set of equations capable of finding the frequency of a signal using
the Fourier transform.

Pulse wave detection, an algorithm for detecting and validating rising edges from a
pulse signal.

Finally, important technologies for the work are described:

ARM Cortex A processor with NEON engine.

Linux operating system, primarily designed for touchscreen embedded devices

Qt Frame Work as a GUI toolkit for embedded devices based on Linux.

OpenCV, a Computer Vision library of programming functions mainly aimed at realtime image processing.

16

Chapter 3
The Hardware

17

Chapter 3
The Hardware
Although the application is targeted to hand-held devices, due to complexity of
application and algorithms to be implemented we need to select suitable architecture ARM
Cortex A8 with NEON processors based board. We selected a Samsung S5PV210 SOM
(System on Module) which implements ARM Cortex A8 processor with NEON with 1 GHz
speed.

3.1 S5PV210 Processor

The Samsung S5PV210 Processor is implementation of ARM Cortex-A8 core
and other following features are available:
the high performance Cortex-A8 MCU
45-nanometer (nm) Low Power fabrication process and intricate low power
architectures
1GHz clock speed
supports up to 1GBytes DDR2
0.65mm pitch, 17X17mm2 FBGA package
ARM CORTEXTM-A8 based
32KB data and instruction caches, 512KB L2 cache
64-bit multi buses
powerful built-in POWERVR SGX 3D graphics engine
supports MPEG4, H.263, H.264 1080P@30fps encode/decode,and
MPEG4 1080P@30fps decode
up to 80Mpixels/s JPEG decoding
HDMI, TV OUT
supports LCD/camera with MIPI interface

18

Figure 3.1 : Samsung S5PV210 MCU(Micro controller unit)

3.2 Development Board & Target System
Target and development board is x210ii. Wave Share Electronic manufacture's
x210ii the Cortex-A8 development board is procured with with Samsung S5PV210 processor
also provides the necessary buses and interfaces to interact with host system such as RS232
serial communication ports and Ethernet module for easy and fast application development.
The x210ii board also provides USB interface for connectivity of external devices, LCD touch
screen display and also many more following features.
Core:ARM Cortex-A8 S5PV210
Operating Frequency:1GHz;
Memory:512MB /1G DDR2 onboard
Flash:4GB iand flash onboard
24-bit RGB interface
Four USB HOST interface, supports connecting multi USB devices at the same
time
USB OTG interface
Two RS232 interface

19

 SD card interface
Four LEDs
Reset button
Software on/off button
Hardware on/off button
Onboard external speaker
MIC input connector
Earphone output connector
Boot configuration switch
Back light multi-level adjustment
Features HDMI interface
Supports four-lines resistive touch control
Supports capacitive touch control
USB WIFI
G-sensor
Supports peripheral expansion using various interfaces, such as SPI, I2C, UART
etc.
Supports MPEG2/MPEG4, H.263, H.264 encode/decode, MPEG4 1080P@30fps
decode
2D, 3D high-performance graphics accelerator
Real time clock
DM9000CEP Ethernet
Keypad interface
Camera interface
GPS interface
GPRS interface
Supports external USB 3G module
Supports USB keyboard/mouse

20

Figure 3.2 : X210ii_B development board

3.3 Host System

The host system is simple dell inspiron 1506 series laptop. Which includes
Intel i3 processor and 4GB DDR3 RAM, USB connectivity only. But for hardware and Linux
kernel debugging we need to connect the development board through serial port RS232. We
used QinHeng Electronics HL-340 USB-Serial adapter for converting USB to serial
protocol convertion.
The host system uses Fedora 22 Linux OS for rich facilities development of
embedded application. And online connectivity through WiFi and Ethernet. Best file system
for management of our project during development and deployment.

3.5 Summery
In this chapter we discussed the hardware requirements of

device and

development board.

The S5PV210 Processor features

target board x210ii features for our requirement.

Host system with fedora 22 linux as operating system.

21

Chapter 4
Software and device development Process

22

Chapter 4
Software and device development Process
4.1 Host environment
The host system is general purpose Fedora 22 Linux running on Intel I3
processor Dell laptop. Its features are briefly discussed in previous chapter. For our embedded
device development we use two different environments. One is tools running on host i.e
native tools and second one is cross-platform development tools. Native tools such as
program editors and connecting & communication tools , terminal for command line
execution. Cross platform tools such as compiler tools, cross-platform libraries. In this chapter
we are going to give brief description on development environment.
The perception that there are no good embedded Linux tools is not entirely
true. First, a few commercial tools are available that are worth considering. However, contrary
to the traditional embedded systems development world, commercial tools aren't the only
option. A lot can be done with relatively little effort using freely available open-source tools.
In fact, I would argue that the do-it-yourself development environment can be far superior to
any commercial offering, as it will allow you the level of customization, flexibility, and agility
that none of the off-the-shelf products can match.
Embedded Linux integrated development environment (IDE) software suites
are usually available from the same companies that sell embedded Linux. Wind River,
MontaVista, TimeSys, LynuxWorks, and a dozen other vendors come to mind. Although these
companies will try to sell you both the operating system and the accompanying IDE, this IDE
may or may not be tied to that particular distribution. Most of these tools are Eclipse-based
and offer similar functionality, which at closer look turns out to be no more than a clumsy
editor, compiler wrapper, and debugger. The only exception I'm aware of is Wind River
Workbench, which is actually a commercial grade product , not surprising considering Wind
River's experience with its Tornado IDE for VxWorks.
The major problem with off-the-shelf IDE suites isn't the software itself, but
rather the nature of embedded systems development. To unleash the full power of an IDE, you
must run an agent on the target embedded platform, as illustrated in Figure . This agent,
however, may not be available if you're working on a customer system, or you may not have
23

enough time to integrate it if you're doing a relatively short-term project. However, this agent
typically fails to run because it relies on some kernel functionality that may not be available,
as all embedded platforms are different and embedded systems programmers love to tweak
system internals, often breaking some functionality the IDE agent relies on.

Figure 4.1 : Host and target system setup sketch.

We used two IDE systems namely ARM DS5, QT Creator. There is good reason
for using two IDEs (integrated Development Environment), because the DS-5 is targeted for
ARM integrated development IDE, which is used for linux kernel and libraries porting and
debugging on board. Where as QT is only used to develop the GUI application only, because
it has Qt libraries integrated and easy to use with it.
4.1.1 ARM DS-5
The ARM DS-5 (Development Studio -5) is a professional software
development solution for Linux-based and baremetal

embedded systems, covering all the

stages in development, from boot code and kernel porting to application debug.
DS-5 includes an Eclipse-based IDE, compilation tools, a fully featured
graphical debugger, simulation models of complete ARM processor-based devices and several
Linux example projects. DS-5 includes the following component tools:
Eclipse-based IDE combines software development with the compilation
technology of the DS-5 tools.

24

 DS-5 Compilation Tools.

DS-5 Debugger, together with a supported debug target, enables debugging of
application programs and complete control over the flow of program execution to
quickly isolate and correct errors.
Eclipse for DS-5 is an Integrated Development Environment (IDE) that
combines the Eclipse IDE from the Eclipse Foundation with the compilation and debug
technology of the ARM tools. Some third-party compilers are compatible with DS-5. For
example, the GNU Compiler tools enable you to compile bare-metal, Linux kernel and Linux
applications for ARM targets.

Figure 4.2 : DS-5 IDE showing development of application

4.1.1.1 Project manager:
The project manager enables you to perform various project tasks such as
adding or removing files and dependencies to projects, importing, exporting, or creating
projects, and managing build options.
4.1.1.2 Editors:
Editors enables you read, write, or modify C/C++ or ARM assembly language
source files. Perspectives and views Perspectives provide customized views, menus, and
toolbars to suit a particular type of environment. DS-5 uses the C/C++ and DS-5 Debug
perspectives

25

4.1.1.3 DS-5 Debugger:

DS-5 Debugger is a graphical debugger supporting end-to-end software
development on ARM processor-based targets and Fixed Virtual Platform (FVP) targets. It
makes it easy to debug Linux and bare-metal applications with comprehensive and intuitive
views, including synchronized source and disassembly, call stack, memory, registers,
expressions, variables, threads, breakpoints, and trace.
Using the Debug Control view, you can single-step through applications at
source-level or instruction level and see the other views update as the code is executed.
Setting breakpoints or watch points can assist you by stopping the application and enabling
you to explore the behavior of the application. We can also use the Trace view on some targets
to trace function executions in your application with a time line showing the sequence of
events. Professional End-to-End Debugger
Single intuitive user interface for all software layers, from bootloader to native
application debug
Debugs code generated by ARMCC and GCC compilers
Professionally supported solution, bringing reliability and responsiveness for
software development
One license for all supported ARM application cores

4.1.1.4. Advanced Session Control & System Views

Control multiple simultaneous debug sessions, to one or more targets, from a
single debugger environment
Run and stop mode debugging seamlessly supported
Full system visibility: memory, CPU registers, peripheral registers, frame buffer
(screen), and more

4.1.1.5. Non-Intrusive Trace Analysis

Instruction (ETM) and instruction flow (PTM) tracing with source code
synchronization
Instruction and function views for trace log analysis
Up to 4 GB off-chip trace buffer with DSTREAM
26

4.1.1.6. Linux Awareness

Linux kernel and user space context awareness, including process and threads
Modules view, listing dynamically loaded modules and shared libraries
4.1.1.7. File Transfer to Target
DS-5 includes a Remote System Explorer (RSE) perspective for easy transfer of
applications and libraries to the Linux file system on the target.
FTP connection to the target to explore its file system, create new folders, and drag
& drop files from the host machine
Open files on the targets file system by double-clicking on them in the FTP view.
Edit them within Eclipse and save them directly to the targets file system
Shell and terminal windows enable running Linux commands on the target system
without a monitor and keyboard
Display a list of processes running on the target

4.1.2. Qt Creator:
Qt Creator is a cross-platform C++, JavaScript and QML integrated
development environment which is part of the SDK for the Qt GUI Application development
framework. It includes a visual debugger and an integrated GUI layout and forms designer.
The editor's features include syntax highlighting and auto completion, but purposely not tabs
(although plug-ins are available). Qt Creator uses the C++ compiler from the GNU Compiler
Collection on Linux and FreeBSD.Qt Creator includes a project manager that can use a
variety of project formats such as .pro, CMake, Auto-tools and others. A project file can
contain information such as what files are included into the project, custom build steps and
settings for running the applications.
4.1.2.1. Editors:
Qt Creator includes a code editor and integrates Qt Designer for designing and
building graphical user interfaces (GUIs) from Qt widgets. The code editor in Qt Creator
supports syntax highlighting for various languages. In addition to that, the code editor can
parse code in C++ and QML languages and as a result code completion, context-sensitive
help, semantic navigation are provided.

27

Qt Designer is a tool for designing and building graphical user interfaces

(GUIs) from Qt widgets. It is possible to compose and customize the widgets or dialogs and
test them using different styles and resolutions directly in the editor. Widgets and forms
created with Qt Designer are integrated with programmed code, using the Qt signals and slots
mechanism.
4.1.2.2. Debuggers:
Qt Creator does not include a debugger for native code. It provides a debugger
plugin that acts as an interface between the Qt Creator core and external native debuggers to
debug the C++ language. Qt Creator displays the raw information provided by the native
debuggers in a simplified manner. Debuggers supported are:
GNU Symbolic Debugger (GDB)
LLVM debugger (LLDB)

Figure 4.3 : QtCreator IDE x210ii application connection setup.

4.1.3. GCC cross-compiler Tool chain:

ARM GNU/Linux tool chain is the version with the support for the latest
ARM architecture. Mainline gcc also has stable ARM support. Enhancements are made in the
Codesourcery version first, and are then pushed back to mainline.

28

As of August 2009, 2009q1-203 version is recommended. If you look for

native tool chain, you can also install a native tool-chain from Angstrom. There's also an armnone-eabi download from 2014 arm-2014.05-28-arm-none-eabi-i686-pc-linux-gnu and arm2014.05-29-arm-none-linux-gnueabi-i686-pc-linux-gnu. The Linaro Toolchains offers only
prebuilt arm-linux-gnueabihf (hardfloat compilers) gcc-linaro-arm-linux-gnueabihf-4.82014.04_linux.
As per the recommendations of x210ii manufacture arm-none-linux-gnueabigcc 4.4.1 (Sourcery G++ Lite 2009q3-67) is built and installed on host system. Tha tool-chain
consists the following :
arm-none-linux-gnueabi-ar
arm-none-linux-gnueabi-as
arm-none-linux-gnueabi-c++
arm-none-linux-gnueabi-c++filt
arm-none-linux-gnueabi-cpp
arm-none-linux-gnueabi-elfedit
arm-none-linux-gnueabi-g++
arm-none-linux-gnueabi-gcc
arm-none-linux-gnueabi-gdb
arm-none-linux-gnueabi-gprof
arm-none-linux-gnueabi-ld
arm-none-linux-gnueabi-nm
arm-none-linux-gnueabi-objcopy
arm-none-linux-gnueabi-objdump
arm-none-linux-gnueabi-ranlib
arm-none-linux-gnueabi-readelf
arm-none-linux-gnueabi-size

29

 arm-none-linux-gnueabi-strip
And cross-platform libraries such as glibc, qt libraries, zlib and opencv
libraries.
4.1.4. NFS Server:
The NFS server provides the target system mountable root file system tools.
The target system root filesystem is exported on NFS server via TCP/IP the file system is
mounted on target linux. So that the IDE's can interact with the development application
which is running on target board.
4.1.5.TFTP Server on Host :
Most target boards use the Trivial File Transfer Protocol (TFTP) to download
executable files to the board. This requires that a TFTP server be available on the system on
which you are hosting the Linux development board. On most modern Linux systems, the
TFTP server is installed as a part of a network-capable system installation, but it is usually
deactivated. This document explains how to activate the TFTP server on your Linux system
and how to copy the kernel into an area from which the TFTP server can deliver the kernel to
the target.
4.1.6. Minicom
A terminal emulator program to interact with the target over the serial port is
minicom on host system. This enables the host system to interact in command mode similar to
tty terminal on Linux system. Serial port is so simple and easy to implement emulator is easy.
So it embedded system developers best tool.

4.2 Target Environment:

The target environment on x210ii board is intended to run the application on
Linux OS, and also during development we need to run the debugging agent such as gdb
(GNU Debugger) on target Linux. So the board is provided with X loader a kind of
application which loads boot-loader in to memory form flash memory.
The boot loader U-Boot (Universal Bootloader) source code is provided by
manufacturer, So it is compiled and installed on x210ii board through serial port. The u-boot
boot up the kernel from TFTP server and then mounts the target file system from NFS server.
30

So that for application development the target file system is avalible on host system,
modification can be done as on native file system.

4.3 Summary
In this chapter we discussed briefly about the development environment,
which includes the following :
Host desktop environment
GNU Cross-platform tool-chain
ARM DS5 development studio v5 for target Linux development
Qt Creator for qt application development.
NFS and TFTP servers for target connectivity with host.

31

Chapter 5
Device Software & Pulse vital signs monitoring application

32

Chapter 5
Device Software & Pulse vital signs monitoring application

Our pulse detector device software is broadly divided in to two parts such as
device software and pulse monitoring application Pulse Detector. The device software
includes boot loaders, Linux kernel, device drivers, application libraries, management tools
and file system . On other hand application software implements the Eulerian video
magnification algorithm and others, video capturing and processing methods, finally heartrate extraction and display.

5.1 Device Software:

The bare metal target device requires the device software such as Linux os
should be booted on. Booting and running of Linux on target system requires boot loader,
kernel and root file system. Buildroot, an open-source, simple and fast embedded Linux build
system implemented on Host system made it easy to compile kernel , busybox, libraries and
other tools.
Buildroot is a set of Makefiles and patches that makes it easy to generate a
complete embedded Linux system. Buildroot can generate any or all of a cross-compilation
toolchain, a root filesystem, a kernel image and a bootloader image. Buildroot is useful
mainly for people working with small or embedded systems, using various CPU architectures
(x86, ARM, MIPS, PowerPC, etc.) : it automates the building process of your embedded
system and eases the cross-compilation process.
5.1.1 Boot loader for x210ii:
The vendor supplied source code for two varieties of bootloaders such as U-Boot
which is know as universal bootloader and the other is xboot. XBOOT is a custom designed
bootloader intended for booting of multiple Operating Systems. So the device pre loaded with
xboot, the main disadvantage with the xboot provided by vendor is not supporting NFS based
target mounting and debugging.
Now we removed the xboot from device and burnt the U-Boot into device for
booting further kernel and file system. The U-boot supports both NFS and flash based file
systems for production use.
33

5.1.2. Linux Kernel:

The main part of any device operating system is kernel. Kernel selection is
mainly depends on hardware architecture, memory footprint ,type of application and
availability of support either community or commercial.
Because vendor specification kernel 2.6.34 latter versions works fine with the
target board x210ii. So we select the kernel 2.6.35.7 tar.xz file was downloaded into buildroot
directory and configured for cross-compilation arm-none-linux-gnueabi tool chain provided
by the host system. The compiled kernel bz image is ' zImage ' . The kernel zImage is preconfigured for inclusion of initrd image which is popularly known as initial RAM filesystem
image. The kernel image is renamed as zImage-qt for convenience.

5.1.3. Device driver:

As we are using the Logitech webcam pro 6000 webcam for our application,
the driver available in kernel is not working properly. The vendor is stop the support of driver
on Linux. The device details are reverse engineered and uvcvideo driver is adjusted for
smooth running of device driver. The device driver working on USB connection. The driver
made working with the help of old 'libwebcam' library available on the net, which was
developed by the vendor Logitech.

5.1.4. File system :

The target file system is first built on host system ext3 file system. The main root
file system as follows
5.1 Table of target file system (rootfs) directories
Directory

usage

/bin

Essential user command binaries (busy box )

/dev

Devices and other special files , device nodes for user space access

/etc

System configuration files, including start-up files

/home

User home directories

/lib

Essential libraries, such as the C library, and kernel modules

/media

Mount points for removable media

34

/mnt

Mount points for temporarily mounted file systems

/proc

Virtual file system for kernel and process information

/root

Root users home directory

/sbin

Essential system administration binaries

/sys

Virtual filesystem for system information and control (buses, devices,etc)

/tmp

Temporary files

/usr

Secondary hierarchy containing most applications and documents useful to

most users including QT libraries

/var

Variable data stored by daemons and utilities

The directories are created on target directory on the host system for NFS server use.
The directory is made as ex3 flash image using cpio archive and dd command named as
rootfs.ext3 . Which is further used to fuse into inand flash for device production use.
5.1.5 Target Libraries :
Essentially the root file system is loaded with necessary for OS functioning
libc, libgcc, libm, libnss, pthread libs,lib-rt and libutils . The usr directory is equipped with
ncursers, openssl , qt core and other qt libraries, qwt for graph plots, OpenCV certain modules
such as core, imageproc, objectdetect for application use. In addition to the Libraries the the
target file system is also equipped with start-up scripts for application binaries and LED
display, mouse and other services.

5.2 Pulse vital signs monitoring application:

5.2.1 Problem description
This section describes the main objective of the work which consists of an
implementation a video magnification method based on the Eulerian perspective capable of
running on a hand-held device. Then, section 5.2.2 provides a description of a simple
application of the Eulerian Video Magnification method.
5.2.2 Linux based implementation of Eulerian Video Magnification
As stated on the previous chapters, the Eulerian Video Magnification method is
capable of magnifying small motion and amplifying color variation which may be invisible to
the naked eye. Examples of the method application include: estimation of a persons heart rate

35

from the variation of its faces color; respiratory rate from a persons chest movements; and
even, detect asymmetry in facial blood flow, which may be a symptom of arterial problems.
The benefits of the Eulerian perspective is its low requirements for
computational resources and algorithm complexity, in comparison to other attempts which
rely on accurate motion estimation [LTF + 05]. However, the existing limits of computational
power on mobile devices may not allow the Eulerian Video Magnification method to execute
in real-time.
The main projects goal is to develop a lightweight, real-time Eulerian Video
Magnification based method capable of executing on a mobile device, which will require
performance optimizations and trade-offs will have to taken into account.

Figure 5.1: Overview of the implemented algorithm to obtain the heart rate of a person from a
webcam or video using the Eulerian Video Magnification method.

5.2.3 Vital signs monitoring

As an objective to demonstrate that the Eulerian Video Magnification-based
method developed is working as expected, the creation of an Android application which
estimates a persons heart rate in real-time using the devices camera was pursued.
This goal requires comprehension of the photo-plethysmography concept,
extraction of a frequency from a signal, and recognition / validation of a signal as a cardiac
pulse. The application will then need to be tested in order to verify its estimations. The test
will be achieved by comparing results from a ECG monitor and other existing application
[Tec13] which use a different method to estimate a persons heart rate.

36

5.3 Implementation details

Section 3.2.1 provides an overview of the overall algorithm implemented.
Then, various implementations of the Eulerian Video Magnification method are described on
section 3.2.2. The face detection, signal validations, and heart rate estimation are also detailed
on sections 3.2.3, 3.2.4, and 3.2.5, respectively. Finally, section 3.2.6 details the interactions
between the Linux platform and the implemented library.
5.2.1 Overview
In order to create an Linux application capable of estimating a persons heart
rate, a desktop test application was first developed because of its faster implementation speed,
and easier testing. The overall algorithm was divided into several steps, illustrated in figure
3.1, which, later, was extracted into a library, named Pulse Detector , to be integrated into an
Linux application, also named Pulsedector. The language used to implement the desktop
application and library was C / C++. In addition, for the image processing operations, the
computer vision library, OpenCV, was used. The OpenCV is a rich library in case of desktop
application. Its performance is very high with x86_64 architecture. But unfortunately on
ARM Cortex-A8 its performance is very poor and consuming more than 80% of CPU because
the OpenCV library is not optimized yet for NEON SIMD instructions.
A short description of the overall algorithms steps on figure 3.1 and
application work flow is as follows:
1. Original frame: read frame from devices webcam;
2. Face detector: detect faces in current frame and match with previously
detected faces in order to track multiple faces. Each face information is then fed into the
following steps:
(a)

Eulerian Video Magnification, magnify detected faces rectangle;

(b)

Signal noise, verify if the signal extracted from the current face is too

noisy. If so, that faces signal is reset and marked as not valid;
(c)

trending & Normalization, if the current faces signal is not too noisy,

then detrend and normalize the signal in order to facilitate further operations with the signal.
(d)

Validate signal, the faces signal is then validated by verifying its shape
37

and timing, in a similar but simpler manner as found in [NI10] paper at end of thesis.
(e)

If the signal is given as invalid, it is kept as valid for a couple of time,

because the validation algorithm may miss some peaks;

(f)

Calculate beats per minute, if the current faces signal is valid, it is then

used to estimate the persons heart rate;

3. Processed frame: is displayed on screen
4. Graph : pulse rate graph is displayed on screen.
On the next sections, there are more detailed information and descriptions of
the algorithms steps.
5.3.2 Eulerian Video Magnification implementations
This section presents the details of several different implementations of the
Eulerian Video Magnification method. The first implementations, described on sections
3.2.2.1, 3.2.2.2 and 3.2.2.3, were developed in C++ to facilitate the integration into the Qt
Frame work based application to run on embedded Linux. However, the OpenCV was still in
its early stages which ended up creating difficulties for the development and integration of
application. Thus, the final implementation, on section 3.2.2.4, was implemented in C/C++
and increases the application performance.

Figure 5.2: Overview of the Eulerian Video Magnification method steps.

The purpose of implementing multiple variants of the method was to study how
the method worked and select which spatial and temporal filters would better fit the
application goal: amplify color variation in real-time. Figure 5.2 shows generic steps of the
method which will be detailed on each of the following sections.
38

The final step, add to original frame, however, remains the same in all
implementations. Which is when the magnified values are added back to the original frame in
order to obtain the processed frame.
5.3.2.1 EvmGdownIdeal
This was the first implementation, thus, its goal was to understand how the
method worked, and match the implementation provided, in MATLAB, by [WRS + 12]. In
addition, real-time support was implemented by using a sliding window of 30 frames.
Resize down
This step applies a spatial filter by calculating a level of the Gaussian pyramid.
This is achieved by looping to the desired level where the input to the next loop is the result
from the previous loop, starting with the original frame. A Gaussian pyramid level is
calculated by, first, convolving the input frame with the kernel, K:

1
4
1
K=
6
256
4
1

4
16
24
16
4

6
24
36
24
6

4
16
24
16
4

1
4
6
4
1

and then, down sampling the frame by rejecting even rows and columns.
Temporal filter
An ideal bandpass filter was used to remove any amplification of undesired
frequency from the color variation of each pixel. To construct this ideal filter, the Fourier
transform was calculated for each pixel over the sliding window of 30 frames. Then,
frequencies below 45 and above 240 where set to zero, and the frame was rebuilt using the
inverse Fourier transform.
Amplification
In this step, the result of the temporal filter is multiplied by an value, which
results in the magnification of the color variation selected by the temporal filter.
Resize up
This step performs the inverse operation of the resize down step, where it
upsamples the frame by inserting even rows and columns with zeros, and then, convolves the

39

input frame with the same kernel multiplied by 4. However, when the original frame is not
multiple of two, an additional resize operation as to be done in order for the upsampled frame
to match the original frames size.
5.3.2.2 EvmGdownIIR
This implementation is very similar to the one above, but uses a different
temporal filter which does not require a sliding window of frames to support real-time results.
The filter used was an IIR bandpass filter, which was constructed from the subtraction of two
first-order lowpass IIR filters. Each lowpass filter is computed as follows:
L n = L n1 (1 ) + M
where M is the current frame, L is the lowpass filter accumulator for each
frame, and is the cutoff frequency percentage. The IIR temporal bandpass filter
demonstrated similar results to the ideal temporal filter used on the first implementation,
without the need for persisting a sliding window of frames, which simplifies the solution and
reduces the computational power required by the device.
5.3.2.3 EvmLpyrIIR
Using the same IIR temporal filter as above, this implementation uses a
different spatial filter, which, instead of, computing a level of the Gaussian pyramid, it
constructs the full Laplacian pyramid and then applies the temporal filter to each of its bands
and each band is amplified differently.

Figure 5.3 : shows the gaussian & laplacian Pyramid algorithm

40

Resize down
Figure 5.3 shows the steps to decompose and reconstruct an image for the
purpose of building a Laplacian pyramid. The original image must be decomposed into two
images, blurred and fine, by applying any type of spatial lowpass filter and scaling the image
down or up by 2. In this case, a Gaussian filter was applied as described on steps resize down
and resize up of the first implementation. Further levels of the pyramid can be computed by
decomposing the blurred image in the same manner.
Temporal filter
The temporal filter used is the IIR bandpass filter, as described above for the
previous implementation, only this time it is applied to each level of the pyramid.
Amplification
The amplification method in this implementation is more complex than the one
previously used. It is based on the implementation provided by [WRS + 12]. It uses a different
value for each band of spatial frequencies, which corresponds to the Laplacian pyramid
levels.
The magnification value, , follows the equation:
(1+) (t)< ...(5.3)
8
where (t) represents the displacement function and the spatial wavelength.
Further details about this equation may be found on [WRS + 12, Section 3.2].
Resize up
This step reconstructs the original image by iteratively reconstructing each
blurred image until the now processed original image is reached.
This implementation demonstrated that by constructing a Laplacian pyramid
for the spatial filter finer motion detail would be revealed, whereas the color variation, the
property to be analyzed, was less evident.

41

5.4 Performance optimized version EvmGdownIIR

Because the OpenCV was not working with the desired performance at the
time, the python bindings on desktop implementations were not executing fast enough for
real-time processing. Thus, a C/C++ implementation of the method that demonstrated better
color variation was developed.
The OpenCV is used to detect the face at once in the begening of detection or
start of application reading and then the detected frame is kept as reference frame coordinates. In addition, after studying the performance of the application, detailed on section
5.2.1, the resize operations proved to be compuationally expensive. Therefore, the resize
down and resize up steps were modified to faster resize operations that did not alter the
resulting image.
Resize down
This step was changed to a single resize operation using the OpenCV
interpolation method named area, which produces a similar result to the one provided by
using a Gaussian filter and down sampling the image. However, instead of, iteratively down
sampling the frame multiple times, it is now a single resize to a predefined size.
Gaussian separable kernel based pyramid algorithm is used to resize down the
sample frame. The filter deigned for this is implemented in NEON intrinsics and inline
assembly code along with C/C++ coding. The separable kernel is then implemented as
convolution iteration over the frame sample.
This implementation has two parts one is vertical pass and other is horizontal
pass. The filter in vertical pass stores the result in temporary buffer, the horizontal pass over
temporary buffer results the blurred image with down sampling by 2. the down sampled signal
is sent to validation.
Resize up
This step was also modified to a single resize operation using the linear
interpolation method, which produces a similar result to the one used previously where the
image was up sampled iteratively using a Gaussian filter.
The same convolution filter is implemented with NEON intrinsic inline
assembly code but the final amplified frame totally not added to original frame in order to cut

42

down the computation cycles. Instead we added only one mean pixel value to the signal
resulted signal is send to further process.
5.4.1 Face detection
The face detection step uses the OpenCV object detector initially proposed on
[VJ01] and improved on [LM02]. Which has been previously trained to detect faces.
Because object detectors are computationally expensive, in order to improve
performance, a minimum size for the face detector was set to 40% of the frame width and
height. In addition, since it was expected for the person to remain still during the reading, the
face detector was set to execute only once a second.
Face tracking is simply done by matching the previous and newly detected
faces to the closest one.
If there are less newly detected faces than the previously detected ones, then match
each new face to the nearest older face. Any older face that is not matched with a
newly detected face is marked for deletion. However, it is only deleted if it fails to
find a match on the next time the face detector executes. This measure allows the
face detector to miss a detection of a face one time.
Otherwise, if the number of newly detected faces is equal or more than the older
faces, then match each older face to the nearest newly detected face. Any newly
detected face that is not matched with an older face is then marked as a new face.
Another performance boost was achieved by only magnifying each face
rectangle instead of the whole frame. Because of this, the face rectangle must remain as still
as possible, in order to introduce as few artifacts and noise to the Eulerian Video
Magnification method. To achieve this, the position and size of the faces rectangle that is fed
into the Eulerian Video Magnification method is interpolated between the previous and newly
detected faces, if the distance between the two, d, is less than one third of the previous faces
rectangle width, w:
d<

w
..(5.4)
3

And, the interpolation percentage, r, used is the ratio between these values:
r=3

d
...(5.5)
w
43

5.4.2.Signal validation
The signal validation has two phases. First, the raw signal, obtained by
averaging the mean value of the green channel of a rectangles face, is checked for noise, on
step signal noise in figure 5.1.
Then, on step validate signal, the shape and timing of the detrended and
normalized signal is verified.
The raw signal noise is simply verified if the signal standard deviation is
higher than 50% of the , amplification factor.
Then, each of the following operations are applied to the raw signal:
1. detrend, as in [TRaK02], also shortly described on section 2.2.3;
2. normalization, the normalized signal, S' , is obtained from the detrended
signal,

S , by subtracting the mean of the signal, S, and dividing it by the signal standard

deviation, :
SI =

( S S )
.(5.5)

3. mean filter, is done by convolving the signal 3 times with the kernel, K:

1
1
1
K=
1
(55)
1
1

1
1
1
1
4

1
1
1
1
1

1
1
1
1
1

1
1
1
1
1

Finally, the shape and timing of signal is validated by a simplification of the

fast pulse wave detection algorithm [NI10]. It identifies possible signal peaks by dividing the
signal into consecutive 400ms time intervals and finding the absolute maximum. Some of
these maximums are rejected: if they are a boundary value of that segment; or if they fall
below a threshold, which is 60% of the mean of the amplitude of the possible identified peaks
so far; or if the distance between two maximums is less than or equal to 200ms, then the lower
maximum is rejected.

44

The signal is said to be valid if:

it includes at least two peaks;

the peak count is between the valid human range, between 40 and 240
bpm;

the peaks amplitude standard deviation is less than 0.5;

the standard deviation of the time interval between peaks is less than 0.5;

To prevent invalidating the signal if a peak is missed or miscounted, the signal is

kept as valid for the next 15 frames.
5.4.3 Heart rate estimation
Even though the simplification of the fast pulse wave detection algorithm
presented on the previous section was capable of counting pulse wave peaks, it would
frequently miscount at least by one peak. Thus, since the period analyzed was short, only 5
seconds at 15 frames per second, a miscounted peak would introduce a large error to the final
value.
In order to obtain the value of beats per minute from the pulse signal, the
method presented on section 2.3.1 was used every time the signal was marked as valid. To
prevent big fluctuations, the value was averaged over the values obtained from the power
spectrum method every 1 second. This work fine on both Desktop as well as on device.
5.4.4 OpenCV
The OpenCV library is installed separately, which is built on host system . The
installation is done only core, imageproc and object detect modules appropriate for that
devices Pulse Detector application because on embedded system memory is constly.

45

Because OpenCV for ARM was still in its early stage, it did not support a
couple of features which had to be implemented, such as, switching cameras at runtime,
portrait mode, stretching frame to container, removing the alpha channel from the frame
without introducing more operations. So restrict the application only for detection of face
reference co-ordinates on the original frame for the beginning of the process capture start and
timely manner.

Receive
Frame

Process&
Draw

Draw face rectangle,

Pulse signal & BPM

Qt frame work

Pulse Library
algorthm

Face rec

Pulse Sig

BPM

Linux native Gcc based libs

Figure 5.4 : Overview of the interaction between the library and application

5.5 Pulse Detector

The user interface of the implemented on Qt Frame work for application,
Pulse Detector , is shown on figure 5.5.
When the application Start button is clicked, the camera starts processing
images and requests the user to place its face in front of the camera. If a face is detected the
user is instructed to remain still in a bright place. Finally, if a cardiac pulse is detected, the
heart rate estimated and the signal are shown on the top of the screen, while, the persons face
magnification is also shown.

46

The buttons exists on the right side starts the process mode. The record mode
averages all the heart rate values estimated in that time period. Pressing the button again
finishes the record mode and displays the average of the beats per minute. This mode was
used for the procedure described on section 4.2.

Figure 5.5 : Snap shot of Pulse Detector application

5.6 Summary
This chapter starts by describing the main goal and motivation for developing a
lightweight, real time Eulerian Video Magnification-based method for the ARM Embedded
Linux platform. Then, an overview of the implemented algorithms steps is described. The
algorithm begins by detecting a persons face and magnifying it using the Eulerian Video
Magnification method. Then, it extracts a possible pulse signal by averaging the green
channel of a rectangles face, which is validated, processed by detrending and normalization,
and validated again, by verifying its shape and timing. Finally, the heart rate is estimated
using the power spectrum technique to obtain a signals frequency.
The chapter then details various implementations of the Eulerian Video
Magnification method which were implemented:
EvmGdownIdeal, applies a spatial filter by computing a level of the Gaussian
47

pyramid, and uses the Fourier transform to implement a temporal ideal bandpass filter;
EvmGdownIIR, applies the same spatial filter as the previous, but uses a
temporal IIR bandpass filter, which was constructed from the subtraction of two first-order
lowpass IIR filters, which are more suitable for a real-time implementation;
EvmLpyrIIR, applies the same temporal filter as above, but computes a full
Laplacian pyramid for the spatial filter, where each level is amplified differently;
Performance optimized EvmGdownIIR, is implemented in C/C++, while the
others were implemented in QT C++, and the spatial filter is modified to single resize
operations but the result is similar to computing a level of the Gaussian pyramid.
Face detection uses the OpenCV object detector module. In order to increase
performance, the face detector only executes every one second. Moreover, only the faces
rectangle is magnified, instead of the whole frame.
Two phases for validating the extracted signal exists. First, the raw signal
standard deviation is checked against a threshold. Then, the raw signal is detrended and
normalized, and the processed signal shape and timing is verified by detecting its peaks.
Finally, the resulting Pulse Detector application user interface is described.

48

Chapter 6
Results and Conclusion

49

Chapter 6
Results and Conclusions

6.1 Heart rate estimation comparison

To validate the heart rate values obtained by the implemented Pulse Detector
application a test was run and readings obtained to a total of 4 participants. The procedure for
the test was as follows:
1. use the implemented application, named Pulse Detector, to obtain an
estimation of the persons heart rate, while at the same time obtain a
reading from digital stethoscope , which also produces an average of the
heart rate at the end, which would take approximately 60 seconds;
2. then, in order to compare the results to another application, step one was
repeated using the ECG application.
3. finally, each person was asked to do some physical exercise so their heart
rate would in crease, then step one was repeated.
4. The implemented algorithm has worse estimations with higher heart rates.
However, by picking only heart rates lower than 95 bpm .

6.2 Future work

The reasons for the different results between the ECG and Pulse Detector
application may be due to several factors. These may include:
in order to increase the implemented application frames per second, face
detection was set to execute only once a second and its minimum size was
large, introducing some instability which led to a persons face not to be
detected from time to time, thus affecting the implemented application
performance.

50

 also because any small movement or lighting change, due to people passing
nearby or participants moving slightly, would be amplified and greatly affect
the final result.
Improvements to the algorithm in order to identify and reject the values
resulting from the situation stated above could be made in a future work.
It should be noted that the main goal of this dissertation was testing the
feasibility of implementing an Eulerian Video Magnification-based method on
a mobile device with the Linux platform. The creation of an Embedded
application to monitor a persons heart rate was a simple example of the
application of the performance optimized Eulerian Video Magnification
method developed. Hence, the effort taken on the algorithm performance was
higher than the validation of the heart rate estimation algorithm.
The algorithms implemented all are not SIMD optimized, So we need to
optimize the algorithms to NEON SIMD in order to improve the performance on ARM Linux
application. Mainly in order to remove artifacts of head motion OpenCV are similar library
which is optimized to NEON should be choose for Embedded application performance
improvement.
Last but not least we use RGB colour space for the processing , but as per
SIGGRAPH 2012 best results can be obtained using YIQ colour space. The YIQ colour space
provides the luma and chrominance components separately. The implementation Eulerian
Video Magnification in YIQ colour space requires more floting point operations, so it may be
a bit difficult on low powered devices without special hardware.
The Pulse Detector application is only implemented one EVMdown level, in
oreder to improve the end results we need to alter the application to implement multilevel
estimation of signal.

6.2 Conclusions
This final chapter presents a review of the relevant information obtained from
this work and an exposition of further work and research.

51

Section 6.1 gives an overall description of the work done, from the
performance improvements of the Eulerian Video Magnification method, to the creation of the
Embedded Linux application capable of estimating a persons heart rate using the devices
camera.
Finally, section 6.2 exposes future work that could follow the development of
an Embedded Linux based implementation of the Eulerian Video Magnification on ARM
Cortex A processor with NEON SIMD architecture.
Objective satisfaction
The Projects main goal is to implement the EMBEDDED LINUX on ARM
Cortex-A8 processor with NEON SIMD engine is achieved. Linux file system on mmc card is
implemented successfully. Finally the platform for running the Eulerian Video Magnification
algorithm based application Pulse Detector is up and running on the device.
The main goal of this work was providing an Eulerian Video Magnificationbased method capable of running on an ARM Cortex A based Linux device. To achieve that,
various real-time implementations of the Eulerian Video Magnification method were
developed with the aid of the image processing library OpenCV. However, these were not
efficient enough to execute on a smartphone in real-time.
Hence, a performance profiler was integrated into a desktop application, in
order to increase the performance of the application and the Eulerian Video Magnification
method, which was occupying 28% of the total CPU cycles used by the application.
This Eulerian Video Magnification method implementation was using a
temporal bandpass filter composed by subtracting two first-order IIR lowpass filters, which is
more convenient for a real-time implementation than an ideal temporal filter implemented by
applying the Fourier transform to each pixel for a video segment.
The main performance boost was accomplished by replacing the multiple
operations of Gaussian blurring and downsampling, suggested by [WRS + 12], by a single
resize operation using own developed functions for NEON.
application, named Pulse, was implemented which was capable of estimating a
persons heart rate by capturing and analyzing that persons PPG signal. Since the
implemented algorithm was developed in the programming language C/C++ for performance
reasons., the integration into the Linux platform was done through the QT frame work.
52

The application workflow started by grabbing an image from the devices

camera. A persons face was detected using the OpenCV object detect module which was
previously trained to detect human faces. A region of interest (ROI) of the persons face
would then be fed into the implemented Eulerian Video Magnification method to amplify
color variations. The average of the ROI green channel was computed, in order to increase the
signal-to-noise ratio, and stored. Along the time, these stored values represent a PPG signal of
the underlying blood flow variations. The signal is further processed using the detrend method
to remove trends from the signal without magnitude distortion. It is then validated as a cardiac
pulse signal by detecting its peaks in order to verify its shape and timing. Finally, the heart
rate estimation is computed by identifying the frequency with the higher power spectrum of
the signal.

6.3 Summary
In this chapter, the performance optimizations of the algorithm and application
along with its metrics are presented. From a basic, real-time implementation of the Eulerian
Video Magnification method to an optimized version capable of executing on an Linux
device at a reasonable rate, approximately 15 frames per second, with a performance
improvement of 22%.
In addition, the heart rate estimations obtained using the implemented
Embedded application, Pulse Detector , were compared to readings from a ECG and another
Android application

53

References
1.

[ISSN No: 2348-4845] Human Pulse Detection Based on Eulerian Video

Magnification on Arm Cortex A Processors on Linux Platform.

2. [WRS + 12] Hao-Yu Wu, Michael Rubinstein, Eugene Shih, John Guttag, Frdo
Durand, and William T. Freeman. Eulerian video magnification for revealing subtle
changes in the world. ACM Trans. Graph. (Proceedings SIGGRAPH 2012), 31(4),
2012.
3. Mohamed A. Elgharib, Mohamed Hefeeda, Frdo Durand, William T. Freeman
Video Magnification in Presence of Large Motions IEEE Conf. on Computer Vision
and Pattern Recognition (CVPR), 2015
4. Frdo Durand, William T. Freeman, Michael Rubinstein A World of Movement
Scientific American, Volume 312, Number 1, January 2015. An expository article
describing several of the motion magnification techniques and applications we have
worked on.
5. Neal Wadhwa, Michael Rubinstein, Frdo Durand, William T. Freeman Riesz
Pyramids for Fast Phase-Based Video Magnification Computational Photography
(ICCP), 2014 IEEE International Conference on
6. Neal Wadhwa, Michael Rubinstein, Frdo Durand, William T. Freeman Phase-based
Video Motion Processing ACM Transactions on Graphics, Volume 32, Number 4
(Proc. SIGGRAPH), 2013.
7. Michael Rubinstein, Neal Wadhwa, Frdo Durand, William T. Freeman Revealing
Invisible Changes In The World Science Vol. 339 No. 6119, Feb 1 2013
8. Hao-Yu Wu, Michael Rubinstein, Eugene Shih, John Guttag, Frdo Durand, William
T. Freeman Eulerian Video Magnification for Revealing Subtle Changes in the
World ACM Transactions on Graphics, Volume 31, Number 4 (Proc. SIGGRAPH),
2012
9. OpenCV Developers Team: Itseez. About | opencv. http://opencv.org/about. html,
January 2013.
54

10. Wim Verkruysse, Lars O Svaasand, and J Stuart Nelson. Remote plethysmographic
imaging using ambient light. Optics express, 16(26):2143421445, 2008.
11. Guha Balakrishnan, Frdo Durand, John Guttag Detecting Pulse from Head Motions
in Video Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference
12. Optimizing Code for ARM Cortex-A8 with NEON SIMD

from

https:

//pixhawk.ethz.ch /omap/optimization/arm_cortex_a8
13. A case study: optimizing GCC on ARM for performance of libevas rasterization
library Dmitry Melnik 1 , Andrey Belevantsev 1 , Dmitry Plotnikov 1 , and Semun
Lee 2.
14. Optimization of Multimedia Codecs using ARM NEON ,Incube Solutions Pvt. Ltd
15. [And13]

Andrew.

High

performance

c++

profiling

andrew.

http://floodyberry.wordpress.com/2009/10/07/high-performance-cplusplus-profiling/,
March 2013.
16. [GSMP07] M. Garbey, N. Sun, A. Merla, and I. Pavlidis. Contact-free measurement of
cardiac
17. pulse based on the analysis of thermal imagery. Biomedical Engineering, IEEE
18. [LTF + 05] C. Liu, A. Torralba, W.T. Freeman, F. Durand, and E.H. Adelson. Motion
magnification. In ACM Transactions on Graphics (TOG), volume 24, pages 519526.
ACM, 2005.
19. [TRaK02] Mika P Tarvainen, Perttu O Ranta-aho, and Pasi A Karjalainen. An
advanced detrending method with application to hrv analysis. Biomedical
Engineering, IEEE Transactions on, 49(2):172175, 2002.
Web URLs:
1. for U-Boot http://www.denx.de/wiki/U-Boot
2. for kernel http://kernel.org/
3. for busybox www.busybox.net/
4. for ARM http://www.arm.com/products/processors/technologies/neon.php
55