ENHANCING EMOTION RECOGNITION FROM

ECG SIGNALS USING SUPERVISED

DIMENSIONALITY REDUCTION
HANY FERDINANDO, TAPIO SEPPNEN, ESKO ALASAARELA
 OUTLINE

BACKGROUND AND SOME INTRODUCTIONS

METHODS
RESULTS
DISCUSSIONS
CONCLUSIONS
ACKNOWLEDGMENT
 BACKGROUND

SINCE PUBLISHED IN 2012, THE MAHNOB-HCI DATABASE PROVIDED BASELINES FOR EMOTION
RECOGNITION FROM EEG, EYE GAZE, ALL PERIPHERAL PHYSIOLOGICAL SIGNALS, FACE AND
BODY EXPRESSION AND SOME COMBINATIONS OF THEM.
FERDINANDO ET AL. PROVIDED BASELINE FROM ECG SIGNAL ONLY, 43% AND 48% FOR
VALENCE AND AROUSAL RESPECTIVELY, WITH FEATURES BASED ON HRV ANALYSIS.
HRV ANALYSIS REQUIRES AT LEAST 5 MINUTES ECG SIGNAL, EVEN HOURS IN ORDER TO GET
RELIABLE ANALYSIS. UNFORTUNATELY, THE ECGS FROM THE MAHNOB ARE BETWEEN 35 AND
117 SECONDS.
 BACKGROUND

FERDINANDO ET AL. APPLIED BIVARIATE EMPIRICAL MODE DECOMPOSITION (BEMD) TO

ANALYZE ECG SIGNALS IN ORDER TO GET INTRINSIC MODE FUNCTIONS WHICH ENDED UP
THE STATISTICAL DISTRIBUTION OF DOMINANT FREQUENCIES AS FEATURES. IT IMPROVED THE
ACCURACY FROM 43% TO 56% AND FROM 48% TO 60% FOR VALENCE AND AROUSAL
RESPECTIVELY.
IS THERE ANY OTHER METHOD TO IMPROVE THE PERFORMANCE OF THE EMOTION
RECOGNITION USING ECG SIGNAL FROM THE MAHNOB-HCI DATABASE?
THIS PRESENTATION IS ABOUT ENHANCING ITS PERFORMANCE USING SUPERVISED
DIMENSIONALITY REDUCTION.
 THE DATABASE

THE MAHNOB-HCI DATABASE WAS PUBLISHED IN 2012.

IT CONTAINS MANY SYNCHRONIZED SIGNALS FOR MULTIMODAL EMOTION RECOGNITION.
EEG
PERIPHERAL PHYSIOLOGICAL SIGNALS: ECG, GSR, TEMPERATURE, RESPIRATION
FACE AND BODY VIDEO
EYE GAZING
SPEECH

THE MEASUREMENT INVOLVED 27 SUBJECTS (11 MALES AND 16 FEMALES).

SOLEYMANI, M., LICHTENAUER, J., PUN, T. & PANTIC, M., 2012. A MULTIMODAL DATABASE FOR AFFECT RECOGNITION AND
IMPLICIT TAGGING. IEEE TRANSACTIONS ON AFFECTIVE COMPUTING, 3(1), PP. 1-14.
 DIMENSIONALITY REDUCTION (1)

DIMENSIONALITY REDUCTION (DR) IS A VITAL PRE-PROCESSING PHASE IN HIGH-DIMENSIONAL

DATA ANALYSIS TO HANDLE THE CURSE OF DIMENSIONALITY.
THE MAIN IDEA IS TO REDUCE THE DIMENSIONALITY OF THE FEATURES WITHOUT LOSING THE
IMPORTANT CHARACTERISTIC.
THE DR CAN BE USED TO VISUALIZE THE DATA AS HUMAN VISION IS LIMITED TO 3D.
FOR KNN, THE DR IS USEFUL IN DATA COMPRESSION PRIOR STORING ALL TRAINING DATA.
 DIMENSIONALITY REDUCTION (2)

ACCORDING TO THE LINEARITY OF THE METHOD:

LINEAR DR, E.G. PRINCIPAL COMPONENTS ANALYSIS (PCA)
NON-LINEAR DR, E.G. SAMMONS MAPPING

ACCORDING TO FEEDBACK OF THE METHOD:

UNSUPERVISED DR, E.G. PRINCIPAL COMPONENTS ANALYSIS (PCA)
SUPERVISED DR, E.G. NEIGHBORHOOD COMPONENTS ANALYSIS (NCA)

ALL CALCULATIONS WERE BASED ON THE DRTOOLBOX FOR MATLAB BY VAN DER MAATEN TO GET
PROJECTION MATRIX A, SUCH THAT THE CLASSIFIER PERFORMS WELL IN THE TRANSFORMED SPACE.
 DIMENSIONALITY REDUCTION (3)

IN THE DRTOOLBOX, THE SUPERVISED DR INCLUDES

LINEAR DISCRIMINANT ANALYSIS (LDA)
GENERALIZED DISCRIMINANT ANALYSIS (GDA)
NEIGHBOURHOOD COMPONENTS ANALYSIS (NCA)
MAXIMALLY COLLAPSING METRIC LEARNING (MCML)
LARGE MARGIN NEAREST NEIGHBOR (LMNN)
GDA WAS EXCLUDED FROM THE EXPERIMENT BECAUSE IT DID NOT PRODUCE MATRIX A AS THIS
MATRIX WAS USED TO TRANSFORMED THE FEATURES DURING VALIDATION SAMPLES.
LMNN WAS EXCLUDED BECAUSE IT DID NOT OFFER DIMENSIONALITY REDUCTION BUT THE
PROJECTION MATRIX MIGHT IMPROVE THE PERFORMANCE OF THE SYSTEM.
 DATA AND PRE-PROCESSING METHODS

DATA WAS DOWNLOADED FROM SELECTION OF EMOTION ELICITATION FROM THE

MAHNOB-HCI DATABASE:
IT CONTAINS 513 SAMPLES FROM 26 SUBJECTS.
DATA FROM SESSION 2508 WAS DISCARDED BECAUSE THE ECG SIGNAL WAS CORRUPTED.

PRE-PROCESSING INCLUDES BASELINE WANDERING AND POWER LINE INTERFERENCE

REMOVALS.
THE ECG SIGNALS WERE FROM STIMULATED MEASUREMENT ONLY.
BLOCK DIAGRAM OF THE SYSTEM
 FEATURE EXTRACTION (1)

IT USED THE BIVARIATE EMPIRICAL MODE DECOMPOSITION (BEMD) WHICH REQUIRED A

COMPLEX VALUE SIGNAL.
THE ORIGINAL SIGNAL SERVED AS REAL PART WHILE THE SYNTHETIC SIGNAL AS IMAGINARY
PART
SYNTHETIC SIGNAL IS SYNCHRONIZED TO THE ORIGINAL ONE BASED ON THE R-WAVE EVENT
SYNTHETIC SIGNAL IS GENERATED USING A MODEL BY MCSHARRY ET AL.
SINCE THE BEMD IS SENSITIVE TO THE SIGNAL LENGTH AND THE ECG SIGNALS DO NOT HAVE
THE SAME LENGTH, THE SIGNAL IS SEGMENTED FOR 5 SECONDS AND THE BEMD WORKS ON
THE 5 SECONDS ECG SIGNAL TO GET 5-6 INTRINSIC MODE FUNCTIONS (IMFS) PLUS RESIDUE.
 FEATURE EXTRACTION (2)

WE ARE INTERESTED TO GET DOMINANT FREQUENCIES FROM THE FIRST THREE IMFS AS IN
FERDINANDO ET AL.*
THE DOMINANT FREQUENCIES IS ESTIMATED USING SPECTROGRAM ANALYSIS WITH WINDOW SIZE
AND OVERLAP PARAMETERS.
1ST IMF CONTAINS 3 DOMINANT FREQUENCIES
2ND IMF CONTAINS 2 DOMINANT FREQUENCIES
3RD IMF CONTAINS 1 DOMINANT FREQUENCY

THE SIX DOMINANT FREQUENCIES FROM THE SAME ECG SIGNAL ARE JOINED TO REPRESENT THE
DOMINANT FREQUENCIES OF THAT ECG SIGNAL.
* Ferdinando, H., Seppnen, T. & Alasaarela, E., 2016. Comparing Features from ECG Pattern and HRV Analysis
for Emotion Recognition System. Chiang Mai, Thailand, The annual IEEE International Conference on
Computational Intelligence in Bioinformatics and Computational Biology (CIBCB 2016)
 FEATURE EXTRACTION (3)

FEATURES ARE THE STATISTICAL DISTRIBUTION (MEAN, STANDARD DEVIATION, MEDIAN, Q1, Q3,
IQR, SKEWNESS, KURTOSIS, PERCENTILE 2.5, PERCENTILE 10, PERCENTILE 90, PERCENTILE 97.5,
MAXIMUM, AND MINIMUM; 14 ITEMS) OF 6 DOMINANT FREQUENCIES AND THEIR FIRST
DIFFERENCE.
TOTAL FEATURES ARE 168 (84 FROM DOMINANT FREQUENCIES AND 84 FROM THE FIRST
DIFFERENCE OF DOMINANT FREQUENCIES).
SEQUENTIAL FORWARD-FLOATING SEARCH METHOD IS APPLIED TO SELECT HIGH
DISCRIMINANT FEATURES. IT VARIED FROM 2 TO 23, DEPENDS ON THE PARAMETERS OF THE
SPECTROGRAMS AND EMOTION LABEL (VALENCE AND AROUSAL).
 DIMENSIONALITY REDUCTION

INITIAL PROJECTION MATRIX A WAS GENERATED BY RANDOM NUMBER SUCH THAT THERE IS NO
GUARANTEE TO GET THE OPTIMUM RESULT WITH ONE PASS.
THE ALGORITHM IN DRTOOLBOX WAS MODIFIED SUCH THAT THE PROCESS IS STOPPED WHEN
THERE IS NO IMPROVEMENT IN CERTAIN ITERATIONS.
THE DR WAS ONLY APPLIED WHEN THE NUMBER OF SELECTED FEATURES IS GREATER THAN THE
TARGET DIMENSIONALITY TO TRAINING SAMPLES ONLY AND THE MATRIX A WAS SAVED FOR
FURTHER PROCESS.
VALIDATION SAMPLES WERE TRANSFORMED TO NEW SPACE BY MULTIPLYING THEM WITH THE
SAVED MATRIX A.
 CLASSIFIER AND VALIDATION METHOD

KNN WAS CHOSEN TO SOLVE 3-CLASS CLASSIFICATION PROBLEM FOR VALENCE AND
AROUSAL.
VALIDATION METHODS:
SUBJECT-DEPENDENT VALIDATION: 20% OF THE SAMPLES WERE HELD OUT FOR VALIDATION WHILE
THE REST ARE SUBJECT FOR 10-FOLD CROSS VALIDATION.
SUBJECT-INDEPENDENT VALIDATION: SAMPLES FROM CERTAIN SUBJECT WERE EXCLUDED FOR
VALIDATION WHILE BUILDING THE MODEL WITH THE REST OF THE SAMPLES. THIS PROCESS WAS
REPEATED FOR ALL SUBJECTS. REPORTED ACCURACY WAS THE AVERAGE FOR ALL PERFORMANCE.
 RESULTS FOR LDA (SUBJECT-DEPENDENT VALIDATION)
Dimensionality Valence Arousal
2D 55.1 7.4 59.9 6.8
LDA IS LIMITED BY THE RULE THAT THE LARGEST DIMENSIONALITY TARGET CANNOT BE LARGER
THAN THE NUMBER OF CLASSES.
THE NUMBER OF CLASSES IS 3: LOW, MEDIUM, AND HIGH.
THE RESULTS WERE VERY CLOSE THE ONES WITHOUT DR: 56% AND 60% FOR VALENCE AND
AROUSAL RESPECTIVELY.
THE IMPROVEMENTS WERE NOT SIGNIFICANT ALTHOUGH IT OCCUPIED LESS STORAGE SPACE.
RESULTS FOR NCA (SUBJECT-DEPENDENT VALIDATION)
Dimensionality Valence Arousal
2D 61.3 7.2 65.6 6.2
3D 57.0 8.0 66.0 8.1
4D 65.3 6.5 60.1 7.7
5D 64.5 6.7 61.0 8.1
6D 53.2 7.6 61.5 7.5
7D 60.4 6.6 61.2 7.2
The results looked promising since the best performances were
higher than the previous results.
RESULTS FOR MCML (SUBJECT-DEPENDENT VALIDATION)
Dimensionality Valence Arousal
2D 54.5 7.9 60.5 7.5
3D 54.6 7.4 48.9 7.3
4D 41.8 6.9 49.3 7.2
5D 41.9 7.2 49.3 7.1
6D 42.1 7.6 49.2 7.0
7D 43.5 7.3 48.4 8.9
The results showed no significant improvement compare to the one
without DR, i.e. 56% and 60% for valence and arousal respectively
 SUBJECT-DEPENDENT VALIDATION SIDE-BY-SIDE

VALENCE (%) AROUSAL (%)

BASELINE 55.8 59.7
LDA 54.2 7.4 59.8 7.3
NCA 64.1 7.4 (4D) 66.1 7.4 (3D)
MCML 53.6 7.3 (3D) 59.5 7.1 (2D)

AMONG THE BEST PERFORMANCE FOR ALL DR

METHODS, TRUE PERFORMANCES WERE
ESTIMATED BASED ON THE LAW OF LARGE
NUMBERS WITH 1000 ITERATIONS
RESULTS FOR LDA (SUBJECT-INDEPENDENT VALIDATION)
Dimensionality Valence Arousal
2D 56.5 10.7 60.6 9.1
THE RESULTS WERE CLOSE THE ONES IN SUBJECT-DEPENDENT VALIDATION, I.E. 59% FOR BOTH
VALENCE AND AROUSAL.
THE VARIANCES WERE HIGH AS EXPECTED IN THIS VALIDATION METHOD.
RESULTS FOR NCA (SUBJECT-INDEPENDENT VALIDATION)
Dimensionality Valence Arousal
2D 61.7 14.1 69.6 12.4
3D 59.4 11.6 51.1 9.5
4D 44.0 12.0 53.3 11.0
5D 40.1 12.0 47.3 11.9
6D 40.0 13.0 51.5 8.6
7D 38.7 11.1 45.7 12.3
Result from 2D of arousal showed significant improvement
RESULTS FOR MCML (SUBJECT-INDEPENDENT VALIDATION)
Dimensionality Valence Arousal
2D 55.9 9.3 61.7 12.3
3D 56.3 12.1 50.2 9.8
4D 41.9 10.6 50.2 10.0
5D 38.8 10.6 50.5 10.4
6D 39.3 11.0 50.3 10.5
7D 39.1 10.8 48.4 8.9
 SUBJECT-INDEPENDENT VALIDATION SIDE-BY-SIDE

VALENCE (%) AROUSAL (%)

BASELINE 59.2 58.7
LDA 56.5 10.7 60.6 9.1
NCA 61.7 14.1 (2D) 69.6 12.4 (2D)
MCML 55.9 9.3 (2D) 61.7 12.3 (2D)

THERE WAS NO SIGNIFICANT IMPROVEMENT

FOR VALENCE BUT THE AROUSAL HAD
SIGNIFICANT IMPROVEMENT.
 SUMMARY OF THE EXPERIMENTS
Subject-dependent validation
Baseline LDA NCA MCML
Valence (55.8%) 54.2 7.4 64.1 7.4 (4D) 53.6 7.3 (3D)
Arousal (59.7%) 59.8 7.3 66.1 7.4 (3D) 59.5 7.1 (2D)

Subject-independent validation
Baseline LDA NCA MCML
Valence (59.2%) 56.5 10.7 61.7 14.1 (2D) 55.9 9.3 (2D)
Arousal (58.7%) 60.6 9.1 69.6 12.4 (2D) 61.7 12.3 (2D)
SUMMARY OF THE EXPERIMENTS
 DISCUSSIONS (1)

THE LDA IS NOT RECOMMENDED FOR THIS RESEARCH AS THE METHOD IS LIMITED BY THE
NUMBER OF THE CLASS. HOWEVER, THE LDA OFFERS BETTER INITIAL VALUE FOR PROJECTION
MATRIX A.
LOOKING INTO DETAIL OF THE RESULTS, THE MCML MOSTLY FAILED TO FIND THE OPTIMUM
PROJECTION MATRIX A.
THE NCA WORKED WELL FOR THIS APPLICATION. THE SUBJECT-DEPENDENT VALIDATION
METHOD SHOWED SIGNIFICANT IMPROVEMENT COMPARE TO THE BASELINE, INCREASED 8%
AND 6% FOR VALENCE AND AROUSAL RESPECTIVELY.
 DISCUSSIONS (2)

THE ALGORITHM WAS MODIFIED SUCH THAT IT SEARCH THE OPTIMUM PROJECTION MATRIX
USING ITERATION AS THERE WAS NO GUARANTEE TO GET THE OPTIMUM ONE WITHIN SINGLE
PROCESS.
THE MATRIX A SHOULD BE INITIALIZED SUCH THAT RANDOM NUMBER GENERATOR IS AVOIDED.
THE ALGORITHM ALSO USED SIMPLE STOPPING CRITERION. MORE COMPLEX ALGORITHM
COMBINED WITH THE MATRIX A INITIALIZATION METHOD MIGHT IMPROVE THE PERFORMANCE
OF THE NCA.
 CONCLUSSIONS

AMONG THE SUPERVISED DR, THE NCA COULD IMPROVE THE PERFORMANCE FOR BOTH
VALIDATION METHODS.
THE PERFORMANCE FOR SUBJECT-DEPENDENT VALIDATION IMPROVED FROM 55.8% TO 64.1%
AND FROM 59.7% TO 66.1% FOR VALENCE AND AROUSAL RESPECTIVELY.
THE PERFORMANCE FOR SUBJECT-INDEPENDENT VALIDATION IMPROVED FOR AROUSAL ONLY
FROM 58.7% TO 69.6%. THERE WAS NO SIGNIFICANT IMPROVEMENT FOR VALENCE.
THE GENERALISABILITY OF THESE RESULTS IS SUBJECT TO CERTAIN LIMITATIONS, FOR EXAMPLE,
THE STOPPING CRITERION IS TOO SIMPLE.
 ACKNOWLEDGEMENTS

THIS RESEARCH WAS SUPPORTED BY THE DIRECTORATE GENERAL OF

HIGHER EDUCATION, MINISTRY OF HIGHER EDUCATION AND
RESEARCH, REPUBLIC OF INDONESIA AND THE FINNISH CULTURAL
FOUNDATION NORTH OSTROBOTHNIA REGIONAL FUND.
Thank you!