Session S1F

Cardiac Pulse Detection in BCG Signals Implemented on a Regular Classroom Chair Integrated to an Emotional and Learning Model for Personalization of Learning Resources
Carlos Ramirez, Carlos Concha and Benjamin Valdes
Tecnolgico de Monterrey, cramireg@itesm.mx, a00882046@itesm.mx, a00882900@itesm.mx Abstract - Emotions are related with learning processes, and physiological signals can be used to detect them. In this work, it is presented a model for the dynamic personalization of a learning environment. In the model, a specific combination of emotions and cognition processes are connected and integrated with the concept of flow, since it may provide a way to improve learning abilities. Physiological signals can be used to relate temporal emotions and subjects learning processes; the cardiac pulse is a reliable signal that carries useful information about the subjects emotional condition, which is detected using a classroom chair adapted with non invasive Electro-Mechanical Film (EMFi) sensors and an acquisition system that generates a ballistocardiogram (BCG), which is analyzed by a developed algorithm to obtain cardiac pulse statistics. A study including different data obtained from the chairs sensors was carried out and interpreted in terms of emotions, which together with a cognitive model is used for the personalization of content in a learning scenario. Such model establishes a relation between the learners knowledge, the content of a course or a learning activity and the emotional condition of each student. Index Terms Cardiac Pulse Detection, EMFi Sensors, Emotions in Learning, Personalization of Learning Resources. INTRODUCTION As technology advances and the human-computer interaction (HCI) increases, new education models and different approximations towards real learning are becoming more frequent. A very particular matter, which has been given special attention in the last decades is the emotionality directly related with learning processes. Although different theories have tried to model, conceptualize and define the learning process, one thing is common to every one of them: the increase of knowledge in the Long Term Memory (LTM). More modern theories relate the cognitive processes that manifest during learning experiences with specific emotions. In the world of affective learning, term first used by Picard [1], the emotions have a key-role directly related to knowledge gaining or, new concept register. Moreover, the processing of physiological signals that correspond with emotional changes is important for the design of new systems involving HCI. Some of the variables of interest for designing such systems are the cardiac pulse, blood pressure, skin conductance, respiration and body temperature as well [2], [3]. In the present work, some learning models are presented, along with their relationship with specific emotions and the design of a prototype for the acquisition of physiological signals, specifically, cardiac pulse. This acquisition system is been implemented through sensors attached to a regular classroom chair. Another important aspect of the interaction system is the ability to distinguish and select the academic information that is most important according to the student profile. LEARNING MODELS I. Basic Cognitive Model The most basic cognitive model, presented by Newell [4] includes acquisition, memory and retrieval processes, illustrated in Figure 1. It is a very simple model, though very representative of the basic processes that any human being experiences when learning. The acquisition of learning is given when some concept is successfully registered in the LTM, so it can be accessed at any time at any moment in the future. However, this model doesnt define the relationship of the cognitive processes with emotions.

FIGURE 1 BASIC COGNITIVE MODEL APPROACH [4].

978-1-4244-6262-9/10/$26.00 2010 IEEE October 27 - 30, 2010, Washington, DC 40th ASEE/IEEE Frontiers in Education Conference S1F-1

Sessi S1F ion

II. II Kort, Reilly and Picard Mo odel As A Picard has i introduced the affective learn ning concept, she has h proposed a learning model which is b based on anot ther im mportant conc cept called flo [5]. Flow i defined as the ow is mental state in which one loses the pass of time due to m n l s in ntense concent tration, and gen nerally the info formation is bet tter processed achi p ieving better results. The relation betwe een flow and emoti f ions can be vis sualized in Fig gure 2. Accord ding to the difficulty or challenge level (y-axis) of the material or o y, l concept and co c onsidering the student skill ( (x-axis), flow c can be b present if bo levels are balanced. How oth b wever, some ot ther emotions can b present if eit e be ther challenge or skill levels (or both) are not ba b alanced. For ex xample, if the challenge leve is el high and the s h skill level is lo then the p ow proposed emot tion would be anxi w iety, as the student doesnt understand the s material being presented. On the other hand if the challen m d, nge level is low a and the skill level is high, the student w l will experience bore e edom. differen affect axes which consi nt ider different ranges of emotion (Figure 4). ns

FIGURE 3 KORT, REILLY & PICARD FOUR-QUADRANT MODEL RELATING EMOTIONS G

AND LEARNING G

FIGURE 4 SEVE ERAL EMOTION AX PROPOSED FOR THE 4-QUADRANT MODEL. XES R T

III.

Other Theori ies

FIG GURE 2 FLOW AND REL LATED EMOTION NS

The T flow state has been men ntioned by Stee in his book A els k Learning Zone of Ones Ow [6], where he proposes t L e wn that schools should pay more atten s ntion in provid ding students w with challenging si c ituations that require the application of different skill d ls, as well as presenting material a g and experiences tha help loose imagination th lead to proper e at i hat activity involv a vement. If all the latter are present in the e classroom, the learning exper c rience is optimu um. According to F A Figure 3, Kort, et. al [7] prop pose a 4-quadr rant model, in which each quadran represents a different stage in m h nt e th learning pro he ocess. When on complete cy (including all ne ycle g four quadrants) is achieved, flow state is pr f ) f resent and hen nce, learning. In th horizontal axis, called affect axis the he emotions can b classified as negative or positive and in the e be s vertical axis, c v called learning axis, the ga g aining, or not, of learning is considered. In this model, emotions l n like satisfaction, c s curiosity, deception, frustr ration, hope or confusion are treated. Mo c oreover, the model propo oses 978-1-4244-6262-9/10/$26.00 2010 IEEE 9 0 E Oc ctober 27 - 30, 2010, Washin , ngton, DC 40th ASE EE/IEEE Fron ntiers in Educ cation Confere ence S1F-2

Several theories and w work from dif fferent authors that try to explain the relationship between learning and specific n emotion are presente in Table I. T control-va ns ed The alue theory, presente by Pekrun [8], explains the relation be ed etween the experien nced emotion and academ achieveme of the ns mic ent students The theory defines specifi emotions ac s. ic ccording to certain domain (class or subje ect, e.g., m math). The classific cation criterion is given by th valence of th emotion n he he (positiv negative) a the activat ve and tion level, whic depends ch on the value, or imp portance that t subject gi the ives to the situation Experimen involving German stud n. nts dents were carried out in order to evaluate the p proposed emoti ions [9]. Chaffar and Frasson proposed a m r model based o Gagns on theory a relate emotions with ever cognitive pr and ry rocess. The model i represented in Figure 5, w is where the basic processes c can be seen. Gagns theory [10] is a model based on expecte responses th begins with an external sti ed hat imulus and ends w with an answer or response from the su r, e, ubject. The respons can be see as an attitu or behavi change se en ude ior towards knowledge ga s aining. The rel lated emotions with each s cognitiv process are explained in T ve Table II.

Session S1F
TABLE I DIFFERENT THEORIES RELATING EMOTIONS AND LEARNING Author(s) [7] [11] [13] [14] Work/Theory 4 quadrant affective model Constructivist Learning Framework (AutoTutor [12]) AutoTutor Posture Analysis Control-Value Theory Emotions relevant for learning Curiosity, confusion, frustration, hopefulness Boredom, Confusion, Flow Boredom, Engagement (flow), Confusion and Frustration High, medium, low interest level, taking a break Activity Emotions Enjoyment Anger Frustration Boredom Positive Affect Negative Affect Joy Feeling worried Contentment Tensed At ease Irritated Secure Displeased

EMOTIONS AND PHYSIOLOGICAL SIGNALS The relation between emotions and learning is an important aspect for establishing an individualized learning architecture. Every person learns in a different way and hence, the information needed to improve or increase a skill varies according with the emotional state. In recent years, non-invasive sensing technologies for HCI systems have been of particular interest since in real interaction scenarios, the emotions can be very spontaneous or could be nonauthentic due to invading devices attached to the body. According to the learning model proposed by Chaffar and Frasson [10], which was explained in the previous section, the different cognitive processes and their relation with emotions were taken in order to define the way in which the emotions can be detected. As a resume, Table III presents those variables and signals related to emotions and later two types of non-invasive technology adopted for this particular project are explained.
TABLE III EMOTIONS AND CORRESPONDING SENSORS Variables /Sensing Cognitive Process Emotions Technology Cardiac pulse1 / EMFi sensor Curiosity (+) Attention Posture analysis / Force sensors Facial recognition / Camera Concept construction Boredom (-) Posture analysis / idealization Force sensors Facial recognition / Happiness (+) Camera Memorization - retrieval Satisfaction (+) Cardiac pulse1 / EMFi sensor Posture analysis / Curiosity (+) force sensors Objective evaluation Satisfaction (+) Facial recognition / response Deception (-) Camera
1

[8]

[15]

Self-Regulation Model

FIGURE 5 LEARNING MODEL PROPOSED BY CHAFFAR AND FRASSON [10].

Only for a short period of time

TABLE II RELATIONSHIP BETWEEN EMOTIONS AND LEARNING PROCESSES IN CHAFFAR AND FRASSONS MODEL Learning Process Related Emotions Negative Emotions and Joy affect directly; try to avoid Attention: According with students them. goals and objectives. Curiosity must be induced to increase attention. The emotional content of the presented material must Acquisition: Information processing correspond with students learned capabilities. emotional state. Remembering pleasant or unpleasant material Retrieving: Remembering and depends on students fetching of information in LTM. emotional state. Induce positive emotions, Organizational Response: Subjects e.g., joy. external action.

IV.

EMFi Sensor

There is a direct relationship between a physiological change e.g. cardiac rhythm or respiration, and emotions. As a first approximation y as a part of a prototype, a system involving a sensor attached to a regular classroom chair is proposed. Due that it is necessary to use non-invasive technology; the sensor is in contact with the subject while sitting on the chair. The sensor is made with thin electro-mechanical film layers and has a voltage output, depending on the distance between the layers, that is proportional to the force applied [16]. This type of sensor has been used in previous work for acquiring a signal known as ballistocardiogram (BCG) which carries cardiac information [17], [18], [19], [20]. The cardiac pulse is taken from the blood pressure of the femoral artery when the subject is sitting on the sensor (chair). It has been demonstrated through BCG signal acquisition that emotional changes can be detected [21], [22]. The detection

978-1-4244-6262-9/10/$26.00 2010 IEEE October 27 - 30, 2010, Washington, DC 40th ASEE/IEEE Frontiers in Education Conference S1F-3

Sessi S1F ion

system for the cardiac rhyt s e thm is illustra ated in Figure 6, where the diff w ferent electron filtering st nic tages are show wn. Filters are im F mplemented to have a more reliable noi e isereduced signal. r previou work in this field [14] wh commerci systems us s hen ial speciall designed fo posture an ly for nalysis are use In the ed. present work, howev Flexiforce piezoresistive sensors ver, are considered [23]. T idea is to b The build a sensor matrix ( ~ 64 sens sors) in order to obtain a g good resolution in signal n acquisit tion. This typ of devices varies their resistance pe accordin to the force that is being applied to th ng g hem and if connect to an oper ted rational amplif fier, voltage ch hanges can be obse erved. The Flexiforce is very thin thus pro y oviding the necessa manageabi ary ility for the p present application. The complet system in te ncluding both EMFi and Flexiforce sensors is shown in Fi igure 8. As me entioned at the beginning, the chai is part of an advanced class ir sroom project p proposal.

FIG GURE 6 SENSOR, ACQUISITION SYSTEM AND FILTERING STAGE FOR EMFI SENSO D ES OR TACHED TO REGUL LAR CLASSROOM C CHAIR. ATT

A comparison between a conventional e c electrocardiogr ram (ECG), a BCG and an oxyme signal (blood pressure tak eter ken from the tip of the finger) is shown in Fig f f s gure 7. As can be seen, the signal acquired with the EMFi sen s l h nsor (Figure 7-B) is consistent an has low no nd oise, thus providing very use eful in nformation as the ECG sho own in Figure 7-A. The ma ajor disadvantage o EMFi senso is the high sensitiveness to d of ors h s sudden abrupt changes in movement. How s m wever, the pul lses can c be detected when the movement is ke minimal. T d m ept The oxymeter signa can be seen in Figure 7-C and shows the o al n C pulses correspo p onding to the same time fra ame as the ot ther tw signals, wh is of 10 se wo hich econds.
FIGURE 8 SYST WITH BOTH SE TEM ENSORS.

PERSONALIZ LEARNING ARCHITECTU ZED G URE Until th point, expl his lanations have been present for the e ted ways in which the em n motional condit tion impacts th learning he process by altering th learners ca he apacity to conc centrate as well as his motivation to do so. Yet, the relationship between n cognitio and emotion on nality must be addressed; tho two can ose be brid dged though le earning activities. The content of an activity and the way it is presented a y t affect the emot tionality of the lear rner. For instan the level o difficulty of a task and nce, of f the style of presentati of its conte do have an impact in ion ent n the em motional condi ition [6], the erefore in the learning e process. The stu udent requires previous kno s owledge to co omprehend new ide and learn. A eas Altering the or rder of learning activities g is a wa to induce an adequate emotional con ay ndition for learning g. The ma goal is to cr ain reate a positive emotional sta through e ate persona alization to bri the learner to the desire learning ing r ed conditio In order to personalize, a on. adaptation is re equired; an adaptati requires a object to ad ion an dapt, a criterion for the adaptati ion and a sou urce of inform mation. Each e-learning technolo ogy resource has different properties an can be nd

FIG GURE 7 CARDIAC SIGNALS FOR A NORMAL SUBJECT IN A 10 SE FRAME A) ECG, B) S S EC , BCG AND C) SIGN FROM OXYMET B NAL TER

V. V Force Senso ors Another impor A rtant aspect considered in this work is the systems abilit to distingu s ty uish between different sitt ting positions and relate them to emotions. There has be p een

978-1-4244-6262-9/10/$26.00 2010 IEEE 9 0 E Oc ctober 27 - 30, 2010, Washin , ngton, DC 40th ASE EE/IEEE Fron ntiers in Educ cation Confere ence S1F-4

Session S1F
adapted in different ways. A list of learning technology adaptations is presented in [24]. In our project, the adaptations are carried out to the learning flows and to the styles of presentation of learning objects; the criterion is based on the knowledge that the learner lacks and the style of learning; the source of information is the learners profile. To create a student profile, learning must first be understood from a computational approach. Learning encompasses the acquisition of concepts, detailed explanations about the cognitive processes involved can be found in the works of Anderson [25], Newell [26], Ramirez [27] and Wang [28]. The specific computer learning processes used for this project are described in [29]. The learning flow and the criteria of adaptation can be found in existing specifications and learning management systems or learning design methodologies, the student knowledge profile is a different issue. To represent the students knowledge in a comprehensive, tractable and structured fashion, a model for knowledge representation called Memory Map MM [28] was developed. As its name states, MM is a formal representation of what a student knows, including concepts and skills. The concepts and skills are obtained from the learning objectives of the activities. For each particular student, a direct mapping can be established between the students concepts and skills, and the objectives of each activity of a learning flow. Such mapping lets us know what the student knows, and what the learning activity is supposed to teach him. Therefore, we can also know what previous concepts or skills are required for the learning activity. This adaptation could be performed at design time or at execution time. At design time a whole course can be personalized according to each students specific needs. In execution time, the sequence of activities in the learning flow is constantly adapted in reaction to the MM and the emotional conditions that occur during a learning session; here is where most advantage can be obtained. The real time emotional condition feedback and learning flow adaptation enables the mapping between emotions and content. This mapping can be done by keeping the record of emotional feedback and the current activity of the learning flow. Through the study of such relations, negative emotional conditions can be predicted and avoided through design time adaptations. Examples of design time adaptations in common student cases are as follows: a) If a student is unable to understand the content of an activity because it is too advanced, it is probable that the student will experience frustration and anxiety. In this case, a learning activity can be introduced in the learning flow to develop the required skills before entering the more advanced activity. b) If the student is presented with concepts already known, the student would probably experience apathy or boredom. In this case the learning activity can be skipped or eliminated from the sequence of learning activities. Additionally, in both cases the style of presentation of the activity can be changed in order to match the students learning style. On the other hand, real time adaptation through emotional feedback can be used to check if a previous design time adaptation on learning flow structure is adequate. For example, if an emotional condition of frustration is detected in a student while performing an activity, the students MM can be checked to verify that the student indeed has the required skills for the activity, or if the style of presentation is the best one; in case the student is unable to understand the content of the activity, a system notification for assistance can be emitted. VI. An Advanced Learning Environment The emotional detection technology and the personalized learning architecture presented in this paper are part of an Advanced Learning Environment (ALE) designed under a constructivist paradigm. The goal of ALE is to enhance learning experience through a wide variety of adaptations in physical, emotional and cognitive contexts. ALE is a model for a state of the art adaptive classroom, its main components are a students profile containing a record of emotional responses, a MM, interests, learning style and project evidence encased in an IMS E-portfolio; a Leaning Management System capable of performing real time modifications of the Learning Flow [30]; Non-invasive emotion detection devices, such as the workbench presented in this paper; and the algorithms that coordinate all of the components to perform advanced adaptations such as the one described above. CONCLUSION AND FUTURE WORK A model that connects a specific combination of emotions with cognition process through the concept of flow was presented. The model can be used for personalization in a learning environment using the heart beat rate and other physiological signals to determine the emotions. A classroom chair was equipped with non invasive technology to detect a students emotional conditions through his cardiac pulse. The cardiac pulse is detected by an EMFi sensor and an acquisition system that generates a ballistocardiogram (BCG) which was used to obtain statistics for a cardiac pulse study. Experiments were carried out including BCG information along with regular ECG and oxymeter signals taken from the subjects fingertip, as well; results showed a good correlation between the three signals. The study of the cardiac pulse along with the flow concept presented earlier showed how to detect emotions such as curiosity, boredom, happiness, satisfaction and deception; and their corresponding cognitive processes such as attention, concept construction and idealization, memorization retrieval and objective evaluation. It was also shown how the chair, through its emotion cognitive process mapping algorithms, can be used as the source of information for adaptations in an advanced learning environment, relating the students emotions and knowledge to improve the quality of his learning process.

978-1-4244-6262-9/10/$26.00 2010 IEEE October 27 - 30, 2010, Washington, DC 40th ASEE/IEEE Frontiers in Education Conference S1F-5

Session S1F
Besides the cardiac pulse study a force-sensor matrix --still in development-- for detecting body postures related also to emotional conditions was also partially implemented and not yet tested. Complete implementation and testing results will be published in future papers. The implantation and testing of such adaptations are part of future work. REFERENCES
[1] Picard, Rosalind W. Affective Computing. s.l. : MIT Press, 2000. [2] Nakasone, A., Prendinger, H. and Ishizuka, M. Emotion Recognition from Electromyography and Skin Conductance. Tokyo, Japan : s.n., 2005. The 5th International Workshop on Biosignal Interpretation. pp. 219-222. [3] Wagner, J., Kim, J. and Andr, E. From Physiological Signals to Emotions: Implementing and Comparing Selected Methods For Feature Extraction and Classification. 2005. IEEE International Conference on Multimedia & Expo. [4] Newell, A. Unified Theories of Cognition. Cambridge, Massachussetts : Harvard University Press, 1990. [5] Cskszentmihlyi, Mihly. Flow: The Psychology of Optimal Experience. New York : Harper and Row, 1990. [6] Steels, L. An Architecture of Flow. [book auth.] M. Tokoro and L. Steels. A Learning Zone of One's Own. Amsterdam : IOS Press, 2004, pp. 137-150. [7] Kort, B., Reilly, R. and Picard, R.W. An Affective Model of Interplay between Emotions and Learning. Madison, WI : IEEE, 2001. Conference on Advanced Learning Technologies. pp. 43-46. [8] Pekrun, R., Frenzel, A. and Goetz, T. The Control-Value Theory of Achievement Emotions: An Integrative Approach to Emotion in Education. [book auth.] M. Tokoro and L. Steels. Emotion in Education. San Diego : Academic Press, 2007, pp. 13-36. [9] Kleine, M., et al. The structure of students' emotions experienced during a mathematical achievement test. 2005, ZDM. [10] Chaffar, S. and Frasson, C. The Emotional Conditions of Learning. 2005, American Association for Artificial Intelligence. [11] Craig, S. D., et al. Affect and learning: an exploratory look into the role of affect in learning with AutoTutor. October 2004, Journal of Educational Media. [12] Graesser, A. C., et al. Detection of Emotions During Learning with AutoTutor. Vancouver, Canada. 2006. Proceedings of the 28th Annual Meeting of the Cognitive Science Society. pp. 285-290. [13] D'Mello, S., et al. AutoTutor Detects and Responds to Learners Affective and Cognitive States. Montreal, Canada. 2008. Workshop on Emotional and Cognitive Issues at the International Conference of Intelligent Tutoring Systems. [14] Mota, Selene and Picard, Rosalind. Automated Posture Analysis for Detecting Learner's Interest Level. 2003, Computer Vision and Patten Recognition Workshop. [15] Boekaerts, M. and Corno, L. Self-Regulation in the Classroom: A Perspective on Assessment and Intervention. University of Leiden, The Netherlands : Applied Psychology: An Internatinal Review, 2005. [16] Lekkala, J., Salpavaara, T. and Krki, S. EMFI - Vesatile Material for Monitoring of Human Functions. Rio de Janeiro : s.n., 2006. XVIII Imeko World Congress. [17] Alamets, J., et al. The Potential of EMFi Sensors in Heart Activity Monitoring. Berlin, Germany. 2004. 2nd OpenECG Workshop "Integration of the ECG into the EHR & Interoperability of ECG Device Systems". [18] Koivistoinen, T., Junnila, S. and Vrri, A. An EMFi-film Sensor based Ballistocardiographic Chair: Performance and Cycle Extraction Method. 2005. IEEE Workshop on Signal Processing Systems Design and Implementation. pp. 373-377. [19] Koivistoinen, T., et al. A New Method for Measuring the Ballistocardiogram using EMFi Sensors in a normal chair. San Francisco, CA. 2004. Proceedings of the 26th Annual International Conference of the IEEE EMBS. [20] Krki, S., Kriinen, M. and Lekkala, J. Measurement of heart sounds with EMFi transducer. Lyon, France. 2007. Proceedings of the 29th Annual International Conference of the IEEE EMBS. [21] Anttonen, J. Using the EMFi chair to measure the user's emotionrelated heart rate responses. Department of Computer Sciences, University of Tampere. 2005. Master's Thesis. [22] Vyzas, E. Recognition of Emotional and Cognitive States Using Pshysiological Data. Department of Mechanical Engineering, Massachusetts Institute of Technology. Massachusetts : s.n., 1999. Bachelor Thesis. [23] Kamiya, K., et al. Sitting Posture Analysis by Pressure Sensors. 2008. ICPR08. pp. 1-4. [24] Torres, J., Dodero, J. M., Ramirez, C., Valdes, B., & Lugo, A. (2009). Adaptive Learning Scenarios Based on Student Profile. In U. A. de Yucatn, Recursos Digitales para el Aprendizaje (pp. 348-349). [25] Anderson, J. R., Bothell, D., Byrne, M. D., Douglass, S., Lebiere C., Qin, Y., et al. (2004). An Integrated Theory of the Mind. Psychological Review, 111, (4), 1036-1060. Retrieved from http://actr.psy.cmu.edu/publications/pubinfo.php?id=526. [26] Newell, A. (1994). Unified Theories of Cognition. MA: Harvard University Press. [27] Ramirez, C., & Cooley, R. (1997). A Theory of the Acquisition of Episodic Memory. In Proceeding of the ECML-97: Case-Based Reasoning Workshop. Prague, 1997a, p48-55. Czech Republic.: Springer-Verlag. [28] Wang, Y. (2007). The Theoretical Framework and Cognitive Process of Learning. In D. Zhang, Y. Wang, & W. Kinsner, Proc. 6th IEEE Int. Conf. on Cognitive Informatics (ICCI'07) (pp. 470-479). [29] Ramirez, C., & Valdes, B. (2008). Web Education Standards and Specifications, Educational Theories and Models of Instruction. Quertaro. [30] Torres, J., Jurez, E., Dodero, J. M., & Aedo, I. (2009). Advanced Transactional Models for Complex Learning Processes. In Recursos Digitales para el Aprendizaje. Merida: U A. de Yucatan..

978-1-4244-6262-9/10/$26.00 2010 IEEE October 27 - 30, 2010, Washington, DC 40th ASEE/IEEE Frontiers in Education Conference S1F-6