THE HUMAN FACE

Measurement and Meaning

 THE HUMAN FACE
Measurement and Meaning

edited by

Mary Katsikitis

Australian Psychological Society

Melbourne, Victoria

SPRINGER SCIENCE+BUSINESS MEDIA, LLC

Library of Congress Cataloging-in-Publication Data

The human face: measurement and meaning I edited by Mary Katsikitis.

p. cm.
Includes bibliographical refercnccs and index.
ISBN 978-1-4613-5376-8 ISBN 978-1-4615-1063-5 (eBook)
DOI 10.1007/978-1-4615-1063-5
1. Face. 1. Katsikitis, Mary.

QM535 .11862002
612'.92--dc21
2002073007

Copyright © 2003 by Springer Science+Business Media New York

Originally published by Kluwer Academic Publishers in 2003
Softcover reprint of the hardcover 1st edition 2003
AII rights reservcd. No part ofthis work may be reproduced, stored in a retrieval system,
or transmitted in any form or by any means, electronic, mechanical, photocopying,
microfilming, record ing, or otherwise, without the written pcrmission from thc Puhlishcr,
with the exception of any material supplied specifically for the purpose of being entered and
executed on a computer system, for exclusive use by the purchaser ofthe work.

Permission for books puhlished in Europe: permissions@wkapnl

Permission for books published in the United States of America: permissions,(qwkap.com

Printed an acid-free paper.

For George, Lauren and Sarah
 TABLE OF CONTENTS

List of Contributors IX

Foreword Xl

Acknowledgements Xlll

Section A
Culture, Evolution, History and Medicine

Chapter 1: Foreign faces: A Voyage to the Land of

EEPICA 1
Robert J Barrett and Mary Katsikitis

Chapter 2: Human Face in Biological Anthropology:

Craniometry, Evolution and Forensic Identification 29
Maciej Henneberg, Ellie Simpson and Carl Stephan

Chapter 3: The Face in Medicine and Psychology:

A Conceptual History 49
German E. Berrios

Chapter 4: Neurology of Human Facial Expression 63

Jason Warren and Philip Thompson

Section B 97
Measurement and Meaning

Chapter 5: FACEM: The Facial Expression

Measurement System 99
Mary Katsikitis
Chapter 6: Facial Affect Recognition Deficits in Schizophrenia:
A Case for Applying Facial Measurement Techniques 119
Patrick Johnston and Vaughan Carr

Chapter 7: Context-Analysis of Facial-Affective Behavior

in Clinical Populations 131
Jdrg Merten

Chapter 8: Causes and Reasons in Failures to

Perceive Fearful Faces 149
Monique de Bonis

Chapter 9: Recognition of Emotion Specific Populations:

Compensation, Deficit or Sepcific (Dis)Abilities? 169
Harald Wallbott

Chapter 10: Facial Expression and the Self-Report

of Pain by Children 189
Belinda Goodenough, Carl L. Von Baeyer,
& G. David Champion

Chapter 11: What Facial Activity Can and Cannot

Tell Us about Emotions 215
Arvid Kappas

Chapter 12: Facial Expressions as Indicators of

"Functional" and "Dysfunctional" Emotional Processes 235
Susanne Kaiser

Chapter 13: Getting to Know your Patient: How Facial

Expression Can Help Reveal True Emotion 255
Mark G. Frank

Index 285

Vlll
 Contributors

Robert J. Barrett, University of Adelaide

German Berrios, University of Cambridge
Vaughan Carr, University of Newcastle
G. David Champion, Sydney Children's Hospital
Monique de Bonis, Universite Paris XI
Mark G. Frank, Rutgers University
Belinda Goodenough, Sydney Children's Hospital
Maciej Henneberg, University of Adelaide
Patrick Johnston, University of Newcastle
Susanne Kaiser, University of Geneva
Arvid Kappas, University of Hull
Mary Katsikitis, University of Adelaide
Jorg Merten, Universitat des Saarlandes
Ellie Simpson, University of Adelaide
Carl Stephan, University of Adelaide
Philip Thompson, University of Adelaide
Carl L. von 8aeyer, University of Saskatchewan
Harald Wallbott, University of Salzburg
Jason Warren, University of Adelaide
 Foreword

James W. Pennebaker

The University of Texas at Austin

At first glance, a scientific book on the face doesn't make a

great deal of sense. After all, the face is not a biological unit that falls
into a specific medical specialty. By the same token, it is not part of a
clear functional system that falls within the purview of a specific
subdiscipline of psychology, philosophy, sociology, or any other
traditional area. It seems that the only organizing principle of the face
is that all humans have one and that it is central to the experience of
being human.
As a social stimulus, the face can signal emotions, personality,
sex, physical and mental health, social status, age, and aspects of our
thoughts, intentions, and our inner selves. At various points in our
lives, we spend a tremendous amount of time and money for cosmetics,
cleansers, medicines, and, occasionally, surgery to enhance our face. In
the same way that a normative, symmetrical face can attract praise and
even adoration, damage to the face through birth defects, disease, or
injury is almost always stigmatizing. Our faces, then, are social
advertisements for who we are.
Just as our faces are proclaiming who we are, all of us are avid
consumers of other people's faces. Most pictures on television or in
print are of faces. When a crime has been committed, the sketch artist
is asked to draw the perpetrator's face (not his or her legs). While
sitting alone in a cafe, we often idly watch the sea of faces going by
and try to imagine the lives of the people behind the faces. Early
developmental work suggests that newborns are early connoisseurs of
faces. Intriguing clinical evidence suggests that the ability to recognize
faces and facial expressions is dependent on highly specific brain
activity and can be disrupted by mental disorders such as
schizophrenia.
My own appreciation of the importance of the face was spurred
by a pilot experiment that my students and I conducted several years
ago. We asked a group of students to put on masks while they talked to
one another. No one knew what their own masked looked like. In each
group, all but one person put on masks that were rather bland and were
not associated with any facial expression. One person, however, was
given a mask that looked angry and threatening. The group interaction
lasted about 20 minutes. During that time, the people with plain masks
gradually distanced themselves from the person with the angry mask.
When the angry-masked person spoke, the others didn't look at him or
her; they also didn't engage in a real conversation with the angry-
masked individual. At the end of the interaction, questionnaires
indicated that the person with the angry mask felt, well, angry. The
angry-masked person said that others didn't treat himlher with respect;
that they didn't listen to hislher comments. The mask - like a real face
- served as a powerful social cue that caused others to behave in ways
to assure that the mask-wearer would feel consistent with his or her
"face." Our faces help to construct our social worlds which, in turn,
construct us.
To understand and appreciate the many dimensions of faces, it
is critical to draw on the collective knowledge of maverick researchers
in philosophy, anthropology, history, medicine, and psychology. This
collection of readings is the first to bring together such a broad base of
thinkers to discuss the face. This is what a cross-disciplinary forum
should be. A multi-disciplinary book on the face may not make a great
deal of intuitive sense but a book such as this is long overdue.

xii
 Acknowledgements

My sincerest thanks and heartfelt appreciation extends to my fellow

contributors who have provided scholarly and thoughtful chapters
and also participated in the review process for earlier drafts of this
volume. I appreciate the support given to me by Christiane Roll and
Mary Panarelli from Kluwer Academic Publishers and to Lara Duff
and Aspa Pappas for their assistance with word processing and with
the formatting required for each ofthe chapters. Finally, I know I
speak for all the contributors when I say thank you to the
anonymous reviewers whose comments were well received and
improved the thematic content of the book.
Section A: Culture, Evolution, History and Medicine
 ROBERT J BARRETT AND MARY KATSIKITIS

CHAPTER 1

FOREIGN FACES: A VOYAGE TO THE LAND OF EEPICA

EVOLUTION AND EMPRICISM: A PSYCHOLOGIST TRAVELS TO

MICRONESIA

In 1967 the psychologist Paul Ekman, of the Langley Porter Neuropsychiatric

Institute in San Francisco, visited the South Eastern Highlands of New Guinea to
study the facial expression of emotion and its recognition among the Fore. He was
interested in the Fore because they were a preliterate, geographically remote people.
They had had very little contact with the Western world, not two decades before
having been an isolated, Neolithic, material culture (Ekman & Friesen, 1971, p. 125)
although Gajdusek's investigations into the degenerative disorder, Kuru, in this
group had brought them to international focus (Gajdusek, 1963; Sorenson &
Gajdusek, 1966).
With him, Ekman took a postulate. There were six primary emotions (happiness,
sadness, anger, fear, disgust, surprise), and each, he hypothesized, was associated
with a distinct facial expression, universally recognizable, and produced by specific
muscular movements. He also took with him 30 photographs culled from a larger
collection of over 3000. To him, they best exemplified the primary emotions, and he
had validated this across a range of different cultures, though all of them literate.
The pictures were shown to a sample of Fore people, asking them to select one of a
list of words, spoken aloud, that best fitted the emotion depicted. The list comprised
words in the Fore language, each of which was a translation of the English word for
one of the primary emotions. The choice of words was inconsistent, agreement was
weak, perhaps because the relevant English words did not translate into single Fore
words, perhaps because the Fore could not remember the list. The enterprise
provided little support for a universal theory of emotions (Ekman, Sorenson &
Friesen, 1969).
Within a year, Ekman returned to New Guinea with two new experimental
paradigms in hand and a different set of photographs, now 40. His aim was to
minimize the above problems with language and memory because he wished to study
emotional expression in vitro rather than in vivo. This time, he put three
photographs, each of a different emotion, in front of the Fore experimental subjects.
They were read a brief scenario that pertained to just one of the three emotions. For
example, the scenario for sadness was, "His (her) child (mother) has died, and he
(she) feels very sad" (Ekman & Friesen, 1971, p. 126). When asked to choose which
picture corresponded to the scenario, Fore subjects were accurately able to select the
correct one, with levels of agreement that matched those of people from literate
cultures. The experiment was conducted among the most isolated of the isolated,
just 3% of the Fore (189 adults and 130 children). Given that these were Fore who
had never seen a movie, did not speak English or Pidgin, and had never lived or
worked with Caucasians, the results confirmed his original postulate. For the return
journey, Ekman arranged for a separate group of Fore to be read these scenarios.
They were asked to mimic the emotion that fitted each one. Their expressions were
videotaped and later shown to US college students who accurately judged the
intended emotion, providing reciprocal support for the universality of emotion.
The findings of these remarkable experiments were published in two articles, one
in Science (Ekman, Sorenson & Friesen, 1969), the other in Journal of Personality
and Social Psychology (Ekman & Friesen, 1971). It was these articles that provided
the basis for the development of a 20 th century research tradition, still active today,
into the universal elements of facial expression of emotion. What is striking about
Ekman's original reports is how succinct they were, three pages and six pages in
length respectively. Their style paralleled an experimental paradigm that was visual
rather than verbal, minimalist rather than discursive.

CONTEXT AND INTEPRETATION : ACULTURAL ANTHROPOLOGIST IN

THE SOUTH PACIFIC

In 1977 Catherine Lutz, from the State University of New York at Binghamton, New
York, travelled to a low lying coral atoll, some half a square mile in all, situated
amidst the West Caroline Islands of Micronesia just to the north east of New Guinea.
There she stayed with and studied among the 430 people who lived there, the Ifaluk.
Her account describes the waning American influence as she flew to Honolulu,

2
Guam, and then Yap, and on by boat to the atoll. On arrival, she was immediately
welcomed and cared for by one of the clan leaders who went by the title of "Chief of
the Foreigners". Unmarried, female, with no family, no children, and no taro
gardens of her own, she was identified as a "needy" guest requiring special care. But
she was expected, in return, to behave as befits a young Ifaluk woman: for example,
to grate coconuts daily, to move to another part of the house with the other women if
male guests came to consult with the clan elder, and to ask permission before
travelling to the village or giving presents. As a social anthropologist, however, she
did have a lighter workload than other women, for they acknowledged that her
constant note-taking and interviewing was indeed work.
With her, Lutz carried the spirit of American feminism of the 1970s. In fact she
was first interested in the Ifaluk because inheritance among them was reckoned by
matrilineal principles and women there played a substantial economic role. She was
seeking an Eden of gender equality and non violence, a sentiment she recognized as
hopelessly romantic when confronted by constant expressions of anger, and a
womenfolk who bent over nearly double in subordination when walking past a group
of seated men.
Her purpose was to study emotion and in this regard, the Ifaluk were celebrated
within the anthropological literature for their emotional song-poems (Burrows,
1963). Her mode of tackling the study of emotions was typically ethnographic. She
engaged in a long-term relationship with those she studied, and searched for
emotional expressions in vivo, not in vitro, by learning the language, and then
listening, listening, listening to the play of human feelings as people interacted with
one another. She aimed to grasp an Ifaluk ethno-psychology of emotion
experientially.
Lutz's work appeared in a 273 page book, Unnatural Emotions: Everyday
Sentiments on a Micronesian Atoll & Their Challenge to Western Theory published
by the University of Chicago Press in 1988. It is difficult to summarize her findings
because they are not circumscribed experimental results; in fact her findings are the
book in its entirety. Suffice it to say that the book has to do with emotions as
socially embedded, culturally constituted within a complex web of local meanings,
contextually experienced, emergent in language (in the form of words and
expressions that cluster around indigenous concepts, or "ethnotheoretical ideas"),
and immanent in everyday interaction, where they comprise a set of practices as
much as they do feelings. Fago is a prime example. With no obvious Western
equivalent ([ago may loosely be translated as a complex of compassion, love, and
sadness), Lutz was able to understand what it meant to feel/ago only by putting her

3
own preconceptions about emotions to one side, and gradually building up
experience of different social contexts in which fago was evident: feeling goose
bumps when listening to a nostalgic love song on the radio; seeing a man farewelling
his son who was about to sail away from the atoll; thinking about the plight of a
legless man who had no family to care for him; hearing from across the lagoon, the
singing of a young fisherman, and so on. What is so striking about Lutz's report is
the rich texture of social relationships and meanings that emerge in her writing: this
is an extended essay on the relationship between language, culture and emotion: an
exploration, not an experiment. Her ethnography locates emotions within discourse
and interaction rather than on people's faces-nowhere is there mention of an lfaluk
facial expression-and in this sense its primary medium is verbal rather than visual.

TWO CAMPS: EEP AND ICA

Ekman and Lutz may be regarded as the most eminent contemporary spokespersons
of two camps that exist within the academic field of studies into culture and
emotions. At times they may appear quite distinct, and at loggerheads with each
other, but other times the distinctions between them become blurred. For these
camps are not diametrically opposed-they share common ground that will be
explored later in this chapter. There are critical points of contact between them that
permit the emergence of hybrid scholars working from an intermediate position that
has affinities with both camps. In the first instance however, and for want of better
labels, we might characterize one camp as occupied by Empiricist Experimental
Psychologists (the EEP camp), and the second as inhabited by Interpretivist Cultural
Anthropologists ( the ICA camp). It is the aim of this chapter to examine these two
scholarly camps from a traditional sociology of knowledge perspective, asking who
are their respective adherents, what are their ideologies, their institutional
affiliations, their modes of discourse? As if we were foreigners, we ask what
theories do they bring to the field, what questions do they ask, and what is their
modus operandi when they try to answer them? Most importantly, how do these
background features influence the way these authorities constitute the objects of their
study-humans in emotional intercourse?
In this chapter we describe in some detail what we regard as paradigmatic
research, our aim being to scrutinize a few exemplars in the field-travel with them
as it were-to find out exactly what they were up to in foreign parts. The chapter is
not to be a comprehensive review of the study of culture and emotional expression.

4
This has been thoroughly covered by Ekman (1973), and a succinct yet inclusive
precis of the field appears in the chapter by Frank in this volume.
As a preliminary remark, we offer the observation that the literature on culture
and emotional expression has a repetitive quality. The same hypotheses are
advanced over and over again-universality, cultural specificity, evolutionary
determinism, social constructionism, and so on-hypotheses that have changed very
little from the mid nineteenth century when they were first put forward. It is
remarkable how much effort is expended in perpetually reproving these already well-
proven ideas. The entire field of enquiry appears to be suspended within old
paradigms, we argue, and it is the intention of this chapter to suggest possible ways
forward-new questions that might break the impasse. Specifically, we suggest that
while engaging in the study of culture and emotion, adherents of both camps have
been more or less blind to the Western cultural presuppositions they bring to their
studies. By means of a critical reflexive analysis, we hope to advance the field by
proposing methods of researching culture and emotion that are not encumbered by
the tacit cultural assumptions of the researchers.
The origins of EEPs and the ICAs may be found at opposite cornerstones of the
foundations of Western academia, which have provided the two camps with different
theoretical underpinnings, different professional affiliations, and different personal
identifications. EEPs are experimental psychologists. One might find them in
numbers at the American Psychological Association meetings. They are in dialogue
with evolutionary biologists. They mix with ethologists: Lorenz is held in highest
respect. Products of their generation, they came to prominence in the early 1970s as
part of a broader bio-science movement within psychology that was beginning to
move away from psychoanalysis and learning theory by turning back to the brain.
ICAs are cultural anthropologists, to be found at the "triple A" meetings. Their
fellow travelers are sociolinguists, qualitative sociologists, and social historians.
Children of their generation, too, they began to carry out ethnographic studies of
culture and emotion at a time when anthropology took the so-called "interpretive"
turn, moving away from French structuralism and British structural functionalism,
from belief and behaviour toward a meaning and experience (Good, 1994, p.174).
Rarely would you find EEPs and ICAs talking to each other (pace the
International Society for Research on Emotions, and other rare exceptions that prove
this rule). This is not surprising because they are busy heading off in opposite
directions, the psychologists toward bioscience and the brain, the anthropologists
toward hermeneutics and the mind. If they ever had been in bed together during
WWII, which is doubtful, then they got out on opposite sides in the post-war era and

5
have never reconciled. The study of culture and emotion thus straddles a classical
body / mind divide: it is fundamentally structured as a polemic.
EEPs tend to do neat experiments, executed within weeks to months on a fly-in-
fly-out schedule (quick and dirty) that aim to produce "facts"--quantified
observations measured from an objective distance. The tempo of their projects is
reflected in the style of their research pUblications-the terse scientific article
appearing in a peer reviewed international journal with a high impact factor, such as
the Journal of Personality and Social Psychology. leAs aspire to ethnographic
projects that are at once more languid and laborious, stretching on from months to
years on a live-in basis (slow and dirty). Much of what they learn is by immersion-
osmosis. They eventually come home with intimate qualitative understandings,
reported with longiloquence in books, usually carried by academic publishing houses
like the University of Chicago Press.
It is only partly in cheek that we refer to Darwin as the patriarch of the EEPs
because in truth the pivotal contributions from this camp have come from men: Bell
(1844), Duchenne (1862), Darwin (1872), Landis (1921), Woodworth (1938),
Schlosberg (1941), Tomkins (1962), Ekman, Sorenson and Friesen (1969), Izard
(1971) and Russell (1980). Goodenough (1932) and Arnold (1960) are the two
important exceptions.
Prominent scholars in the leA camp, by contrast, have been mainly women, the
most notable being Lutz (1988) and Rosaldo (1980). Perhaps the most significant
exception is Schieffelin (1976). In addition, one thinks of three great contributors to
the anthropological literature, Birdwhistell (1963), LaBarre (1947) and Leach (1972)
as counterexamples, all relativists with a focus on culture. Yet when one recognizes
that, for all three, the primary datum was visual and observational not contextual
(they were principally concerned with kinesics, gesture and non-verbal
communication), and that their methodology was comparative not interpretive
(culture became a sort of variable), one recognizes that they bear a closer family
resemblance to the EEP genealogy than to the leAs. Previously debates within this
field of research have been construed as battles between universalists and cultural
relativists (Matsumoto, 1991, p. 129), which is true enough at one level, yet these
disagreements are only slight perturbations on the surface that cloud our view of
much deeper rifts and incompatibilities that are embedded in gender differences
within the sciences and the emotions. In relation to the study of emotional
expression, the reasons for these gender differences can be tracked down, at least
partly, into the cultural history of the psychological and social sciences, since the
seminal psychological studies of emotional expression were carried out during an

6
epoch when it was men who did science. Standing behind a number of EEP studies
one can discern a familiar archetype of imperial science, the male scientist-explorer
who draws his data from colonized peoples. It is only the emergence of post World
War II feminism within Western academic institutions that created the conditions of
possibility for ICA studies. But irrespective of epoch and political inspiration, is it
not more than a little stereotypically masculine, we ask, at least in Western cultural
terms, to isolate, separate, and objectify emotions into visually observable, non-
verbal things that are located in individuals and measurable on their surface? And is
it not just as stereotypically feminine to integrate emotions into felt and verbally
expressed qualities that are rooted deep within people but at the same time, permeate
relationships between people and are immanent in context? 1
Thus far, Ekman has served as the exemplar of the EEP approach and Lutz has
epitomized the ICA approach. Other examples, equally representative, could have
been chosen from either side. Among the EEPs we might have started with Charles
Darwin: the patriarch, the paragon. In 1867 Darwin set out to test his evolutionary
theory of the emotions by means of a 16-item questionnaire he designed to tap the
observations of individuals in far flung parts of the globe on the natives of their
region. He posted these questions out across the world, from North America to
Tierra del Fuego, from the great civilizations of China and India to tribal Africa and
tropical Borneo. A sense of the project can be gained from the questions themselves:
"(\.) Is astonishment expressed by the eyes and mouth being opened wide, and by
the eyebrows being raised? ... (9.) Is contempt expressed by a slight protrusion of
the lips and by turning up the nose, and with a slight expiration? ... (I\.) Is extreme
fear expressed in the same general manner as with Europeans?" (Darwin, 1872/1999,
pp. 22-23). He received responses from 36 observers, among them missionaries and
catechists, Protectors of Aborigines, botanists and naturalists, station owners (notably
those on property in remote regions), a teacher, a police magistrate, and a judge.
Only one reply came from a native, Christian Gaika, brother of Chief Sandilli,
although another report contained detailed observations made by one Captain Speedy
on his charge, the son of King Theodore of Abyssinia. In short, Darwin's responses
came chiefly from that myriad of minor administrators beavering away at the frontier
of Empire. Notable exceptions were Professor and Mrs. Gray whose observations
were made while sailing up the Nile, and Mrs. Barber, who observed the Kafirs and
Fingoes. Perhaps the most distinguished respondent was Brooke, of the celebrated
family of "White Rajahs" of Sarawak. From Darwin's account, it is difficult to be
sure who provided the response. Was it the first White Rajah, James Brooke,
himself a firm friend of Wallace, but who, by the time of Darwin's survey, was close

7
to the end of his life and already back in England? Or did it come from the heir, his
nephew Charles, who had already been acting Rajah for five years, though not yet
formally invested with that title (Runciman, 1960, p. 159)? Either would have been
admirably suited to comment on expressions of astonishment, extreme fear and
disdain among the Dayaks, James having led, in league with the British navy, the
infamous sea massacre at Beting Marau Point that resulted in the slaughter of 500
Dayaks, and Charles having personally led a series of punitive expeditions against
these people, beginning in 1856 with a raid that first plundered 25 longhouses and
then burned them to the ground (Pringle, 1970, p. 103).2
On the basis of the data provided by these kind collaborators, Darwin was able to
conclude that "all the chief expressions exhibited by man are the same throughout
the world," and this, to him, argued "in favour of the several races being descended
from a single parent-stock, which must have been almost completely human in
structure, and to a large extent in mind" (Darwin 1872/1999, p. 355).
Other than Darwin, there are several who might equally well epitomize the EEP
approach to the study of facial emotions. In a contemporary vein, we could have
mentioned Carroll Izard, whose collaborative research sites spanned Europe,
America, Africa and the Orient. Using somewhat less bloodthirsty research assistants
(he was careful to use psychologists who were native to each culture, and to conduct
the research in the local language), Izard nevertheless drew largely on the original
nine Darwinian universal categories of emotion to demonstrate a high level of cross-
cultural agreement in the judgement of facial emotions, further evidence supporting
the hypothesis that "the expression and recognition of the fundamental emotions are
determ ined by evolutionary-genetic processes" (Izard, 1971, p. 266).
In the Lutz camp there is an equally rich tradition of research. Perhaps the most
noteworthy, because it is so enticingly exotic, is the work of Michelle Rosaldo,
whose monograph, "Knowledge and Passion: l/ongot Notions of Self and Social
Life" (1980), is a study of culturally constituted emotions that pertain to
headhunting-emotions that are so different that they seem to lie altogether outside
any Darwinian or Ekmanian categorization of emotions. What makes these I1ongot
passions seem counterintuitive is that emotions like anger, grief and fear are, to the
Western mind and heart, strongly associated with negative experience, yet to the
I1ongot, they are endowed with positive cultural signification.
The I1ongot live in the Northern Luzon region of the Philippines, some 150 miles
from Manila. A small group, only 3,500 strong, they were renowned for their
headhunting. The Rosaldos (Michelle was accompanied by her husband Renato)
based themselves at Kakidugen, chosen as a "traditional settlement" far removed

8
socially and geographically from the influence of Christian miSSIOns that had
remained intact in the face of the Japanese occupation and the return of the American
forces. Rosaldo herself was attracted to the locale because it was so exotic, lush and
"picturesque," and to these "people of the forest" with their "simple" social structure,
because of the celebrated diaries of the anthropologist William Jones, who had
himself been killed in 1909 by his I1ongot porters.
At first the Rosaldos could not understand "how people whom we liked and
admired ... could be killers" (p. 137) and "why I1ongots find deep psychological
satisfaction in killing, in slashing victims and in severing and tossing to the ground a
human head" (p. 138). She expected to find explanations of cosmic or religious
dimensions that matched the extraordinary nature of this violence. Instead she found
that I1ongot spoke about the subject in a "bland" way. She had expected to find that
the severed heads were invested with powerful ritual significance, yet found that they
were just tossed away and discarded like empty coca-cola cans. Headhunting and
the emotion surrounding it was so mundane and taken-for-granted that I1ongot
people could not see why it required explanation-indeed they were perplexed as to
why she was even interested in the topic.
The answer to this puzzle, for Rosaldo (1980), lay in understanding how I1ongot
bachelors had hearts consumed by youthful figel. At first she translated figel to
herself as "anger," but gradually she came to realize that this I1ongot emotional
concept did not equate, in a one-to-one correspondence, with the English language
concept of anger. So she added "passion" and "energy" to the translation (p. 45).
Increasingly, as she recorded more and more instances of figel, and further
developed her linguistic analysis of the concept, she came to realize that it lay at the
conceptual centrepoint of a web of semantic connections that not only ramified
broadly throughout I1ongot social and emotional life and the I1ongot natural world,
but also penetrated deeply into the basic origins of I1ongot persons. She came to
recognize the extent to which, for the I1ongot, it was an emotion both singular and
pervasive. Ligel suggested "potency, energy, intensity, the irritating heat of chili
peppers, the rush of rapids, or the force of wind ... irritation ... violent
action ... readiness to be 'different' or take offense." Ligel is "quick-
moving ... youthful, active ... and 'tied up tight' or 'strong'." It "generates a 'redness'
in the self... and is associated with a sense of 'focus' that encourages industry and
success." "'Concentrated' in the form of sperm," it produces babies. Ligel arises in
response to "insults, slights, and other intimations of inequality." It is "born of
'envy' " and importantly it is a "natural response" to the vicissitudes of life. In
particular figel is the "pain that strikes a youth who has not killed" -an" 'angry'

9
musing" in the face of loss, or not getting ones desires fulfilled, where desire is
defined largely as a desperate craving for things other people have that you do not.
So fundamental is this emotion that "without figet to move our hearts ... there would
be no human life" (pp. 45-47 passim).
Once having comprehended figet, in its various moods and instances, the
headtaking made sense, for it was figet that had to be divested, purged, transcended
in the passage from youth to adulthood. The act of severing a human head from a
body and tossing it into the air released liget and enabled the youth to assume adult
status, central to which was the acquisition of beya (knowledge), the calm, mature
wisdom of Hongot elders-the counterpoint, if you like, to figet. The act of taking
and tossing a head (or heads) would fill the young man's heart with potency and
pride at having shed figet and gained beya. Thus the killing emotion was a positive
feeling of resolution, achievement, and readiness for adulthood.
How might Ekman and his fellow intellectual travellers have approached figet.
Perhaps it would be interpreted as a higher-order composite of two or more
fundamental emotions, say a "blend" of anger and sadness (Ekman & Friesen, 1975,
p. 122). Liget might be understood in terms of culturally learned "display rules" that
serve to augment or diminish the expression of its fundamental component emotions,
shaped and channeling them, revealing or masking them, and combining them in
varying proportions (Ekman, Sorenson & Friesen, 1969). Ilongot culture would also
be seen to provide the headhunting context that triggered a display of figet. If
examined experimentally for the purpose of judging or rating of emotions, however,
the distinctive features of the basic, universal emotions that together comprise liget,
would be observable in the face of the Ilongot youth-until he tossed the head.
But Rosaldo makes it clear that to the Ilongot, liget is about as fundamental and
basic as you can get-emotion-wise. And it is an emotional atom. Like Darwin's
"fear," for example, which is not a mixture of more fundamental emotional particles,
so figet is irreducible to other more basic elements. Moreover, just as Darwin
proposed in relation to his list of nine emotions, liget to the Ilongot is natural, innate,
inherited, and fundamental to the human condition.
In other words, had Darwin, Ekman and Izard belonged to successive generations
of an endogamous Ilongot kindred, we might have seen, by now, a number of cross-
cultural studies of liget and its corresponding facial features, not only in societies
where it was manifested in full and pure form, but also in social groupings where it
was blended with beya, masked by means of cultural display rules, or not triggered at
all (no headhunting). This comical "thought experiment" serves, by its very
absurdity, to highlight the Eurocentric assumptions that underpin contemporary

10
academic discourse on emotional expression. It makes the point that what is a salient
emotion in one culture, is not necessarily a salient emotion in another. And who is to
say which version is correct?3 Only twenty-five years ago, in the full flight of
modernism, it would have been obvious that the Euro-American version was the
more objective, valid and universal. In the contemporary cultural context of post-
modem perspectivism, things seem a little less certain.

FUNDAMENTALL Y DIFFERENT THEORIES OF EMOTION AND CULTURE

On reading the EEP literature, the reader is immediately struck by its well articulated
"theory of emotions." It is a theory that is "up front" both in terms of its
explicitness, and in the sense that it temporally and logically precedes experimental
research, as one would expect, because the purpose of research is to test such a
theory. To the EEP researcher, emotions are basic things.4 He does not primarily
investigate the nuances of human emotional life-petulant spite, for example, wistful
nostalgia, heartfelt remorse or lukewarm affection-but works on fundamental,
unambiguous emotions, between four and nine of them. The vector of emotional
force, for this researcher, is an arrow that arises inside the biological core of the
human organism and points outward to the face. That is to say, emotions are
neurobiologically hard-wired, innate, and common to all humans. For Darwin
(187211999) expressions of emotion in infants and in the insane were critical test
cases of the fundamental human condition. This universal quality was thought by
some to extend beyond the human species at least as far as our cousin primates,
perhaps further to other animals and their passions. Most importantly, emotions are
a product of evolution in so far as they are adaptive (in, for example, communication
between mother and infant) and gradually acquired.
The reader is also struck by the fact that this general theory is split into two mid-
range theories to do with the expression of emotion, and the recognition of emotion
respectively, and within each of these lie specific hypotheses addressing particular
elements of emotional processing. This tendency to theorize and atomize,
hypothesize and analyze, by breaking emotions down into smaller and smaller
components so each can be studied separately, is characteristic of the EEP approach,
and reminiscent of the classificatory impulse of its progenitor, natural science.
Theories of expression are predicated on the a priori distinction between
involuntary and voluntary facial expressions (or spontaneous versus controlled), and
are concerned initially and predominantly with the former. They rest on two
additional a priori dichotomies: inner versus outer, and mind versus body (or will

11
versus nervous system). All these, however, are recognizable cultural dichotomies
which spring from mainstream traditions of Western thought in general, and Western
philosophy and science in particular. On the basis of such assumptive foundations, a
one-to-one correspondence is postulated between outward involuntary expressions
and inner states of mind. According to his principle of "serviceable associated
habits," for example, Darwin (1872/1999) proposed that:
Certain complex actions are of direct or indirect service under certain states of the mind,
in order to relieve or gratify certain sensations, desires, etc.; and whenever the same
state of mind is induced, however feebly, there is a tendency through the force of habit
and association, for the same movements to be performed, though they may not then be
of the least use (p. 34).

Opposite states of mind, ipso facto, produce opposite facial movements. And
causal vectors that point from inner to outer may also operate in reverse for, in an
early statement of the facial feedback hypothesis (Tomkins, 1962; Izard, 1971;
Gellhorn, 1964), Darwin (1872/1998) proposed that the "free expression by outward
signs of an emotion intensifies it" (p.359). These one-to-one relationships between
experience and expression are particularized to finer and finer levels of detail, as if a
microtome is being applied to the feelings. For example Ekman's analysis of
component units (or muscular action units) suggests that distinct muscular
movements are associated with discrete emotions (Ekman & Friesen, 1971).
From an EEP perspective, theories of the recognition of emotion (or, in the
terminology of the literature, its appraisal, or its judgment) focus on the perception
of visible movement of facial muscles. Duchenne de Boulogne (1862) observed that:
to express and to monitor the signs of facial expression seem to me to be inseparable
abilities that man must possess at birth. Education and civilization only develop or
modify them (p. 29).

There was some uncertainty in Darwin's mind as to whether the recognition of

emotions was innate or learned. First, he studied Duchenne's renowned photographs
(in which Duchenne had applied an electric current to the anaesthetized face of a
psychiatric patient to obtain facial expressive movements of the muscles). Next
Darwin "made odd noises and strange grimaces, and tried to look savage" (Darwin
1872/1999, p. 353) in front of his four month old son, who apparently took this in
good humour. Although one might think that this argues against innate recognition,
at least of anger, it was interpreted by Darwin as the correct recognition of "good
jokes." Finally, at six months, when the child's nurse was persuaded to pretended to
cry in front of the boy, something the little boy had never been exposed to before, he
instantly recognized this emotion by assuming a melancholy expression, evidence

12
according to Darwin for innate recognition of the facial expression of emotions. His
tentative conclusion was:
as most of the movements of expression must have been gradually acquired, afterwards becoming
instinctive, there seems to be some degree of a priori probability that their recognition would
likewise have become instinctive (p. 353).

EEP research in this century has continued to explore the innate nature of
emotional recognition, at least as far as this applies to the basic emotions, by means
of cross-cultural judgment studies demonstrating how certain unique spatial
configurations of facial musculature are instantly and accurately recognized
irrespective of cultural, linguistic, or geographic background (Ekman, Sorenson &
Friesen, 1969; Izard, 1971, Tomkins, 1962). In the final analysis, the critical test
case involved an experiment among the Grand Valley Dani of the Central Highlands
of New Guinea carried out by an anthropologist-psychologist team, Karl and Eleanor
Heider. It was a test case because the Dani were pre literate (they could not have read
descriptions or viewed photographs of emotional expressions, and did not even have
words for all the basic emotions), visually isolated (they could not have seen
expressions on the faces of foreign peoples), and had only given up using stone axes
some 10 years previously (they were paragons of primitiveness). The Heiders'
conclusion was all the more robust because it ran against their own expectations: "A
preliterate, visually isolated people interpret almost all of the basic facial expressions
of emotion in the same way as do members of literate cultures" (Ekman 1973, p.
214).
Compared to the way they think about emotions, EEPs have a relatively under-
theorized concept of culture (which, for clarity, we refer to as EEP-cultureV It is
accorded the status of a "factor" or a "variable" that is then appropriated to the task
of testing of the universality hypothesis. It is of little value to such research if
culture does not vary, but this raises immense methodological problems. First, it is
necessary to postulate what is essentially unverifiable-that variation occurs between
cultures not within cultures. This leads to a static, armchair-stereotypic concept of
culture that overemphasizes both intracultural homogeneity and intercultural
heterogeneity. The EEP assumption that people within a culture are all much the
same but they differ markedly from people in other cultures cannot be tested.
Problems attend the measurement of such a vague indefinable as culture because it
has to do with meaning and interpretation rather than quantity and measurement. In
the end it is just not possible to do so: the EEP solution is to find a proxy for culture
that has obvious specificity and variation, and this is geographical location. Thus,
when Ekman and Friesen (Ekman, 1973) show pictures to college students in Japan,

13
USA, Brazil, Chile, and Argentina, or when Izard (1971) shows pictures to North
American, English, German, Swedish, French, Swiss, Greek, Japanese, and African
people, they convey an aura of cultural variation without specifying this variation.
The dimension that is being varied does not appear, strictly speaking, to be language,
because they include Spanish speaking people of Chile and Argentina, or English
speaking people of North America and England as separate cells in the research
design. Nor does it appear to be culture, as they include college students in all these
sites, who, it could be argued, conform to similar organizational, educational and
class-based cultural norms that pervade institutions of privilege and learning in all
these countries.
Where geographically based differences in expression and recognition do
emerge, these emotional variants are construed as derivative in order to preserve the
concept of basic, universal, invariant, emotions. They become secondary
modulations, amplifications, or distortions of primary emotions that have been
fashioned or sculpted by culture. The paradigmatic experiment was devised by
Ekman (1973) who showed a strong correlation between the facial reactions of
Americans and Japanese to watching neutral films and stressful films when alone,
but when watching these films in the presence of another person, Japanese subjects
tended to mask their emotional expression with polite smiles, especially in response
to the stressful film. Ekman postulated the presence of the aforementioned "display
rules," the learned rules that mould pure forms into their distinctive culture-specific
shape, in this instance, the display rule of inscrutability. This is an elegant
contemporary demonstration of Darwin's (1872/1999) idea that certain muscles and
their movements can be "repressed through the will" (p. 34). The concept of
"display rules" was applied not only to the expression of emotions but also their
recognition by Matsumoto and Ekman (1989) when they argued that "people modify
their perceptions of emotion in the same manner that they learn rules for modifying
expressions" (Matsumoto, 1991, p. 130).
Another derivative form is the so-called "blend," analogous to the idea of an
orange emotion that enables us to preserve the concept of primary red emotions and
yellow emotions. In sum, the place allocated to culture in EEP research design as a
dependent variable parallels its place in EEP theory as a modulator or blender of
universal basic emotions. In a broader sense, EEP science represents a contemporary
working out of a cascade of dichotomous motifs that pervaded nineteenth century
European intellectual culture6 according to which:

14
 nature is to culture (or

nurture);

as body is to mind;

as archaic is to recent;

as primary is to secondary;

as phenomenal is to epiphenomenal;

as primitive is to civilized;

as lower is to higher
(basic,fundamental) (superstrucuture,
superficial);
as simple or pure is to complex or

mixed;

as involuntary or is to voluntary and

automatic controlled;

as expressive are to suppressive;

as positive is to negative

as paleocortex is to neocortex.

as mid and hindbrain are to forebrain.

15
 IfEEP research were to be portrayed graphically as in Figure I with the X axis
representing degrees of difference in the expression and recognition of emotions and
the Y axis representing increasing degrees of EEP-cultural difference, then the
universal emotions could be depicted as a straight horizontal line, while EEP-cultural
blends and modulations would be plotted as an oblique line ascending from left to
right.

Display rules and blends

Influence of
culture

Universally recognised
expressions

Difference in the expression and recognition of emotion

Figure I. The influence of culture according to EEP theory.

In the service of demonstrating universality it becomes necessary for EEP

research to eliminate as many EEP-culture elements as possible. In these
experiments, EEP culture becomes a confounding rather than a dependent variable.
For example, Ekman (1973) avoided using the judgment of observers as a means of
selecting photographed facial expressions for his cross-cultural studies. Even when
such photographs were agreed by observers in one culture to represent a particular
emotion, it might have been the case, argued Ekman, that they were concurring on
matters of distinctive blends or expressions emblematic of their culture, which
observers from other cultures (where the blends and the emblematic expressions
perhaps differ) could disagree over (Ekman, 1973, pp. 204 - 205). Instead, Ekman

16
used a limited number of photographs from a large bank, personally selected for the
extent to which they fitted his proposed basic emotions of happiness, sadness, anger,
fear, disgust and surprise, and the muscular movements he postulated to be
associated with these six emotions.
Not only are aspects of EEP-culture held stable, but elements that are central to
an leA definition of culture are eliminated. When Izard (1971) found a level of
agreement between Africans and Japanese that was significantly lower than expected
this was treated as a methodological problem to do with language. What lies for the
leAs at the core of culture is methodological noise to the EEPs that muddies his
experiment. As related at the outset of this chapter, when Ekman tentatively
disproved his universality hypothesis by demonstrating a level of agreement among
people from Borneo and New Guinea that varied from low to absent, he identified
language as the culprit and redesigned the experimental paradigm to eliminate its
confounding effects (Ekman, Sorenson & Friesen, 1969).
The criticism could be leveled that EEP research merely demonstrates that when
you show images of faces in different countries (not necessarily different cultures but
certainly different places) in such a way as to eliminate any possible cultural
influence, you can demonstrate universal emotions that are recognized independent
of culture. That is to say, eliminate culture and you will not find cultural effects-
the results are inescapably locked into the research design. The leA critic might
imagine a Z axis on the above graph, on which it becomes possible to plot increasing
degrees of leA cultural difference. She would immediately object that no values
relating to leA culture have been entered on that axis-the graph remains two
dimensional-and the experimental paradigm has failed in a profound way to test the
link between culture and emotion. A more sympathetic interpretation coming from
the EEP side would be that it is possible to hypothesize the existence of a limited
number of fundamental emotions derived from a theory of facial emotional
expression and have observers, located all over the world, agree with your judgment
on these few emotions.
In contrast to the EEP literature, it is possible to wade through many pages of
leA publications without finding a programmatic definition of a theory of emotions.
The reader must pick it up here and there-it emerges from the overall gestalt of the
text, as befits the ethnographic modus operandi, whereby research findings emerge
from (rather than being stated at the outset and then tested by) the research
experience. Whereas EEPs dissect down to a molecular anatomy of emotions, leAs
integrate. Lumpers rather than splitters, they have no separate sub-theories, for
example, of emotional expression as opposed to the recognition of emotions. In

17
general terms, they locate emotions in activities that people do together. In the
diffuse, all encompassing ICA concept of culture, emotions are emergent features of
social interactions and cultural contexts; they arise out of communication, dialogue,
inter-subjective interpretation, mutual understanding (or misunderstanding); they are
located in shared language, in discourse; they are jointly achieved, mutually
accomplished, they are socially constructed and culturally mediated. If an ICA could
be persuaded to draw emotional vectors, she would come up with a crazed diagram
of arrows going back and forth, cross-cutting this way and that in order to represent
multiplex, multi-layered interactions between individuals who live entangled in
groups that share common cultural elements. Whereas EEPs take emotions from the
inner neurobiological core out as far as the human face, ICAs find emotions at the
human interface. And while EEPs are concerned with correspondence between the
inner mental state and outer facial expression, ICAs are concerned with reciprocity
between individuals, whereby one man's anger entails another man's fear, or
"someone's experience of grief and frustration (tang) creates
compassion/love/sadness ([ago) in others" (Lutz, 1988, p. 82). Expression and
recognition, for the ICA, are thus bound up in the same interchange, and both are
socially learned more than innately expressed-an achievement not an endowment.
As we have seen, EEPs are primarily concerned with involuntary behaviour.
ICAs have a far more voluntaristic notion of emotional interplay . This is not to say
that they are just concerned with consciously displayed, simulated or counterfeit
emotions, though the enactment of emotion is an area of interest for anthropologists
interested in ritual drama. But ICAs start from the premise that people are "social
actors" who are engaged in the social world, to use the language of symbolic
interactionist theory (Rose, 1962). Even where emotional interchange lies outside
conscious awareness, it is because it is unnoticed-taken-for-granted-rather than
involuntary and innate. It has meaning and purpose. 7 In short, emotions "are
socially located and have a social aim" (Schieffelin, 1985, p.180).
The distinction between ICA-culture and EEP-culture parallels the distinction
between the ways these two camps approach the emotions. We have argued that
EEP-culture is a global, comparative, pared-down construct that has appointed
geographic locale as its deputy. Difference is located at the inter-cultural level
(between countries), sameness at the intra-cultural level (individual cultures are
homogenous). ICA-culture is a construct with a more specific indigenous focus that
celebrates difference. ICA cultures differ from one another, but there are also
differences at the intra-cultural level (cultures are heterogeneous). Since individual
variation is its focus, an intermediate ICA construct is the "culturally constituted self,

18
positioned at the nexus of personal and social world" (Lutz & White, 1986, p. 417).
Where does sameness fit into the ICA-culture equation? It is usually located at a
pan-cultural level. ICA-culture theory postulates universal human processes (social
interaction, interpretation, language, socialization). Whereas EEPs give precedence
to the fundamental biological nature of humans, lCAs assert that humans are
fundamentally social creatures. Given that human universals such as language and
8
meaning are so central to an lCA vision of culture, the core EEP question of whether
there are universal emotions that are invariant across cultures is reframed by ICAs
into a group of altogether different questions: is it possible to translate from one
language or cultural group to another the emotional discourse of a people, their
modes of interpreting emotion, the semantics and the pragmatics of emotions, their
embeddeness in the social structure, their location in the moral order, and their
origins in the history of a people?

THE HYBRIDS: MEAD AND MATSUMOTO

Having advanced the essentialist claim that the study of culture and emotion
comprises two separate worlds of scholarship that ask different questions and answer
them in different academic languages, let us begin to undermine this argument.
Certainly it is true that there are different paradigms, but these might also be seen as
two platforms on which respective protagonists can stand and remonstrate with each
other. Much of this can be characterized as quarrelsome misunderstanding, but
nonetheless the very existence of a dialogue opens space for rapprochement, and the
articulation of intermediate, hybrid positions.
Margaret Mead might be taken as representative of an lCA who very early in the
piece sought to bridge the divide. She was well placed to do so because her original
work, though essentially anthropological, bears a curious resemblance to the work of
EEP scientists. Like many before and since, Mead together with Bateson sought out
the most isolated location to conduct her study, which she found in the remote
village of Bayung Gede, accessible only along a dirt path high in the mountains of
Bali. Here she found simplicity itself. Compared to the plains, there was "little stress
or strain" at this altitude (Bateson & Mead, 1942, p. 10). The people lived in a
"simple" casteless society and spoke among themselves in a "clipped" tongue-not
the exaggerated, complex language necessary to communicate with visitors from the
coast.
But although the work was ethnographic in spirit, it fell short of the ethnographic
ideal. For example, neither spoke the language well. Like Ekman, Mead and

19
Bateson required an interpreter. Yet the interpreter, Made Kaler, could only translate
Balinese into a sort of Dutch English which left them with the further problem of
converting this into American English (Jensen & Suriyani, 1992). As can be seen in
Balinese Character: A Photographic Analysis (Bateson & Mead, 1942), the analysis
was, like Ekman's, heavily reliant on the interpretation of photographic images,
although in this instance it was the researchers who made the judgments about these
images rather than experimental subjects. The study concluded that the folk of
Bayung Gede were an emotionally withdrawn people among whom anger was never
expressed outwardly, especially in public. Cockfights, traditional dance, and
ceremony provided culturally sanctioned outlets for emotional release, though even
here feelings were directly expressed before and after such events but concealed at
the moment of ritual climax. The authors described the Balinese character as one
curiously cut off from interpersonal conflict, leading to a "life centred in one's own
body into which all emotion long ago vanished" (Bateson & Mead, 1942, p. 48).
Their principal conclusion was that this suppression of emotional display was the
product of socialization. It might have been written by Ekman himself, so
sympathetic was it to his concept of primary emotions that are modulated by culture
through social learning.
It is not surprising therefore that Mead (1975) produced perhaps the earliest
prospectus for reconciliation that made a stand against discipline-centric approaches
to the study of culture and emotion. It is true that she still took issue with Ekman.
She quibbled about the ethological connotations of the phrase "display rules," and
argued that his reliance on posed emotions proved little about the recognition of
emotion in differing cultural contexts but a lot about the universal recognition of
simulated emotions. Despite these reservations, Mead essentially conceded that
biology must be taken seriously, and advocated a common middle ground of
culturally patterned innate emotional expressions, the ground that Ekman already
occupied.
At the other end of this debate, and from the other side of the fence, Matsumoto
(1991) is perhaps the only EEP investigator who has sued for peace by conceding
that culture must be taken seriously. He was concerned that culture is accorded so
little theoretical attention in cross-cultural psychology that it is mainly
conceptualized in terms of cliches and stereotypes. As an antidote, Matsumoto
sought to develop an intermediate theory linking culture and emotion. He
operationalized a definition of culture by bringing Hofstede's (1980) distinction
between individualistic and collective cultures, into conjunction with Tajfel's (1982)
distinction between in-group and out-group, to produce a four-cell table of

20
possibilities for the evaluation of cultures. Following Triandis and his colleagues
(1988), he argued that collective cultures foster more cohesion than individualistic
cultures among in-group members, but less within out-group members. "It thus
follows that collective cultures will foster emotional displays of their members that
maintain and facilitate group cohesion, harmony, and cooperation to a greater degree
than individualistic cultures" (Matsumoto, 1991, p. 132). Matsumoto's work goes
well beyond Mead's call for reconciliation, because he has begun to put in place a
detailed experimental design that, if feasible (and the question of how to measure
individualism, collectivisim, and the collective ratio of in-group to out-group is still
in the air), will enable this field to push ahead.

DEEPER AFFINITIES: THE QUEST FOR THE MAGIC MIRROR

Beneath the hybrids and their white flags lie deeper, unexpected affinities between
the EEPs and the ICAs. To the proverbial Martian they appear more similar than
different. From the upper atmosphere the Martian looks down and sees members of
both camps travelling in pursuit of pre-literate, traditional and simple societies,
located in exotic, picturesque environments, as far from Western academia as
possible: geographically, altitudinally, culturally, and ethnically. Is it not ironic, asks
the Martian, that Ekman and Lutz, for all their disagreements, ended up in the same
Micronesian corner of the globe-not a stone's throw from the Southern Highlands
of New Guinea to the West Carolines? Secondly, the Martian assumed that this was
a scientific endeavour (be it psychological or social science), and is surprised,
therefore, to find that neither camp publishes evidence that refutes their theory; nor,
in fact, do they design research that might run this risk (Ekman came closest).
Finally, the Martian makes the same observation that has been noted earlier in this
chapter, that the field of culture and emotions has a recurring quality; the chief
investigators seem to the Martian to be suspended in space, as it were, perpetually
performing variations on the same experiment, and publishing variations on the same
results, until eternity.
Coming back to earth, we suggest that these observations provide a clue to the
impasse within this field of enquiry. We propose the field has become entrapped,
like much cross-cultural research, within "primitivism." In their analysis of similar
problems within the psychiatric cross-cultural literature, Lucas and Barrett (J 995)
define primitivism as "a body of ideas, images and vocabularies about cultural
others. While ostensibly about peoples from elsewhere, it is more fundamentally

21
concerned with the way the West understands itself in contradistinction to these
others" (p. 289).
We suggest therefore, that it is healthy to be skeptical of any form of research
activity that takes scientists on a romantic quest off the beaten track to study isolated
people because it may carry the implication, explicitly or implicitly, in part or in full,
that compared to us civilized sophisticates, they are basic, primitive folk; ancient,
stone-age peoples closely connected to the parent stock; natives who live in simple
societies that are close to nature because they are nested in exotic forest surrounds.
Such research often hints or makes glaringly obvious that whereas we are modern,
they are traditional. That these societies are referred to as "pre-literate" rather than
"non-literate" or "oral" implicitly places them earlier in some imaginary cultural
time, as if they represent an era through which civilized people passed long ago. The
term "Neolithic" serves the same purpose. These opinions are predicated on a post-
colonial political order (it would be hard to talk about these people in such a way if
they were politically and economically dominant), from which science would be well
advised to extract itself if it does not want to attract the label of pseudo-science. For
a start, the "natives" might find such value judgments offensive, as Jensen and
Suriyani (1992) have shown in some detail in relation to Mead's work. But the
greater moral outrage from a scientific point of view is that these ideas that silently
frame research design are never tested. Though the various authors provide us with
assurances, we have yet to see evidence for, or measures of, the lack of stress in the
heights of Bayung Gede for example, the simple social structure of the I1ongot, or
the Neolithic material culture of the Fore. These are self-evident to the respective
authors, yet not to the scientific skeptic.
The chief purpose of such far-flung geographical research is to study the "Other":
someone who is as different from me as is humanely possible. Hopper (in press) has
observed in relation to cross-cultural psychiatric research that" 'culture' has been a
mock elegant way of referring to 'there' as opposed to 'here' ," to which we would
add 'them' as opposed to 'us'. If we are so complex, and they are so different, then
they represent, for the psychologist or anthropologist, an elegant test case-the
simple, distilled essence of humanity. The problem with this exercise is that
primitivism is essentially reflexive, as is suggested by the above definition: it is a
way of coming to know about the self by looking at the Other. The primitive
becomes a mirror that the Westerner holds up to gaze at him or herself. One might
argue that the research reviewed in this chapter is ostensibly a project to look at the
relationship between culture and emotions, but unwittingly, unintentionally, an
unconscious programme of self definition in contradistinction to the imagined Other.

22
It is no wonder therefore, that EEP and lCA research repeatedly reconfirms its own
presuppositions.
What probably mesmerizes the EEP and lCA researcher alike, however, is what
he or she does not appear to notice, which is that these Others from far off places
represent two opposite values at one and the same time. They are Paradoxical
Others who are simultaneously Different and Identical (PODI).9 On the one hand
they symbolize the essence of difference (unlike us, they live in the forest, they do
not read or write, etc.) and, in the same breath, they symbolize the essence of
similarity (like us, they are quintessentially human, perhaps even more so).10 When
we look into an ordinary mirror, it is easy to forget that the image we see is laterally
reversed. Analogously, when science gazes into the PODl mirror looking for
differences it sees similarities in the reflection; when it looks into the mirror for
similarities, it sees difference in the reflection. In other words, EEP researchers look
into the PODI mirror for difference (civilized people versus simple tribesmen) and
they see sameness (universal emotions). lCA researchers look into the POD I mirror
for sameness (universal principles of social interaction) and they see difference
(exotic emotions such as liget andfago). We suggest that the results of these many
experiments and field trips were prefigured in this deep paradox that silently
structures the psychological and social sciences when they become entangled with
primitivism.

NA VIGA TING OUT OF THE SHOALS

We further suggest that the first step in solving these problems is for these scientists
to critically analyze their own cultural presuppositions, to which we would see this
chapter as a preliminary contribution. That is, we are arguing that the logical
beginning to the study of culture and emotion is the study of the culture of the people
who study culture and emotion. Unless this fiercely reflexive definition of culture is
adopted, the study of culture and emotion is predicated on the erroneous assumption
that only the subjects of an experiment, or participants in ethnographic field-work,
have culture, whereas the observer does not, a condescending definition of culture if
there ever was one. Hence this chapter is not concerned in the first instance with
emotions in the Fore, Ifaluk, Iiongot, or Grand Valley Dani, it is a voyage through
the dangerous shoals near the shores of the land of EEPICA, where, like Gulliver, we
encounter the strange and curious customs of the inhabitants (Swift, 1984). One
group cannot stop showing photographs, the others keep writing things down all the

23
time, the first group worships a deity called Evolution, whilst European Romanticism
is the others' god.
The second step we propose is to construct the comparative exercise more
carefully than has been done hitherto. Most cross-cultural studies have proceeded to
the study of universal emotions with unseemly haste by identifying basic emotions in
Western cultural settings then transporting them to other cultures in such a way as to
prove their universality. This practice occurs in many areas of psychological and
social science, which apply Western "emic" categories across the developing world
in such a way as to demonstrate they are, indeed, "etic" categories. It rests on a
familiar encompassing process of Western intellectual hegemony, in which our
"emic" becomes our and their "etic," but their "emic" remains a local, indigenous
variant or blend.
An alternative is to begin with the idea of salient or dominant emotions that was
suggested in the earlier discussion of Rosaldo's work. It would appear from our
limited reading of the literature that in each culture studied so far, there is a concept
of salient emotions. I I To the Western person, anger, fear, and several others stand out
(Ekman's six, perhaps). To the Ilongot youth, liget and fago are the most salient.
For the time being this is a legitimate strategy because, until we are revisited by the
proverbial Martian, who might give us a culture-free or etic way of defining
dominant emotions, we are left with emic definitions as a starting point.
Starting in this way raises interesting and potentially fruitful questions
concerning how a people identify which emotions are salient and which are not. Are
there common criteria across different cultural contexts? Comparing the literature on
Ilongot and Western culture, we tentatively suggest that "natural" is a quality which
people attach to an emotion they regard as salient. "Inborn," "original," and
"essentially human" are further candidate criteria. Such emotions are also seen as
"atomistic," unable to be broken down into component emotions. Careful
comparative research would be required to establish in a more systematic way what
these criteria might be.
After identifying salient emotions in several cultures, and the criteria by which
they are judged to be salient, the next step would be to identify common dimensions
of these emotions. Again, on the basis of our reading of the literature, we tentatively
suggest that there are at least 4 commonly recurring dimensions:
1. the interpretive dimension;
2. the bodily (including facial) dimension;
3. the subjective (experiential) dimension;
4. the inter-subjective (social dimension).

24
 Pursuing this course might lead to novel paradigms for the comparative study of
emotions, in which multi-centre research programmes might seek to identify cross-
cultural similarities and differences, both at the level of salient emotions, or their
dimensions. Such a strategy would have the advantage of stepping around the
intellectual debris that has cluttered this field, leaving behind fervent beliefs in the
primacy of biology or community, hollow arguments about the respective
contributions of nature or nurture, and avoiding altogether those gratuitous
designations of people as traditional or modern, simple or complex, primitive or
sophisticated. A core group of salient emotions might be identified in all of the
cultures under study, with a peripheral group that differs from one culture to another.
Core factors might emerge from such research, such as "ease of recognizability"
within the interpretive dimension.
Such strategies, we hope, may enable research to break away from old paradigms
toward integrative methodologies that instead of asking whether, ask how, culture
and biology, both separately or together, influence the expression of emotion.

AFFILIA nONS

Department of Psychiatry
University of Adelaide
Australia
Email: robert.barrett@adelaide.edu.au
Email: m.katsikitis@psychsociety.com.au

NOTES
I. Compare with studies of Western gender stereotypes in relation to emotion which suggest that
femininity is equated with the capacity to experience, express, empathize with, and communicate
emotions, in contrast to the masculine archetype of "restrictive emotionality" that involves control and
suppression of feelings (Jansz, 2000). These contrasts rest on an underlying cultural dichotomy in which
rationality, control, cognition and order is associated with males, as opposed to the irrational,
uncontrolled, bodily passion that is associated with females (Crawford et al., 1992, p.16).
2. The same point applies to the "Protector of Aborigines." With few exceptions (Arkley, 2000) the
incumbents exerted absolute, oppressive control over the lives of indigenous Australians who were legally
designated as non-persons, while at the same time failing to protect them from the atrocities of the frontier
wars waged across the nineteenth century (Reynolds, 1987). Darwin's survey, in fact, took place in the
aftermath of the systematic massacres and exterminations in Victoria (Clark, 1995), from where his
responses were mainly derived.

25
3. This "thought experiment" follows the musings ofObeyesekere's (1985), in which he imagines what
the world of official psychiatric disorders would be like had Asian "semen-loss" rather than Euro-
American "anxiety" been adopted as a basic, universal, diagnostic category, anxiety disorder under these
circumstances becoming a sort of "culture bound" variant of the more fundamental condition of semen
loss.
4. cfKappas, this volume, for a discussion of what he calls the "basic emotions" perspective which, of the
three perspectives he identifies, most closely approximates that of the EEPs.
5. So profound is the difference between the EEP and ICA concepts of culture that we have adopted, for
the purpose of this chapter, the awkward but necessary convention of designating them "EEP-culture" and
"ICA-culture. "
6. The work of Hugh lings Jackson best exemplifies these dichotomies (see Young, 1993).
7. Pertinent here is the distinction between expression "given off'and expressions "given"originally made
by Goffman (1959, p.4). The former are involuntary, the latter are more strategic, and their compass
ranges from the unselfconscious interactions of the person immersed in social interaction, to the deliberate
imitation or simulation of the dramatic performer. It is this range that lies within the province of the
ICAs.
8. See Barrett & Lucas (1993) for a typical ICA analysis of emotions that is predicated
on an understanding of language depth.
9. For an extensive discussion of this conundrum see Taussig (1993).
10. c.f. Lutz (1988, p. II).
II. Even the term "basic" might be avoided because it rests on a spatial metaphor within the English
language, as Lakoff and Johnson (1980) demonstrated, that might begin to lock us into other basic
qualities (body, original, etc.)

REFERENCES
Arklcy, L. (2000). The hated protector. The story of Charles Wightman Sievwright, protector of
aborigines 1839-42. QLD: Orbit Press.
Arnold, M. (1960). Emotion and personality. Vol. I. Psychological aspects. Vo!. 2. Neurological and
physiological aspects. New York: Columbia University Press.
Barrett, R. J. & Lucas, R. H. (1993). The skulls are cold, the house is hot: Interpreting depths of meaning
in Iban therapy. Man, 28, 573-596.
Bateson, G., and Mead, M. (1942). Balinese character, a photographic analysis. New York: The New
York Academy of Sciences.
Bell, C. (1844). The anatomy and philosophy of expression as connected with the fine arts. London, John
Murray.
Birdwhistell, R.L. (l963). The kinesic level in the investigation of emotions. In P.I1. Knapp (Ed.),
Expression of the Emotions in Man (pp. 123-139). New York: International Universities Press.
Burrows, E.G. (l963). Flower in my ear: Arts and ethos oflfaluk atoll. Seattle, University of Washington
Press.
Clark, I. (1995) Scars in the landscape: A register of massacre sites in western Victoria, 1803-1859.
Australian Institute of Aboriginal and Torres Strait Islander Studies Report Series. Canberra:
Aboriginal Studies Press.
Crawford, J., Kippax, S., Onyx, J., Gault, U., and Benton, P. (l992).Emotion and gender: Constructing
meaning from memory. London: Sage.

26
Darwin, C. (1872). The expression of the emotions in man and animals. London: John Murray.
Darwin, C. (1999). The expression of the emotions in man and animals. Harper Collins: UK.
Duchenne, B. (1862). Mecanisme de la physionomie humaine; ou, analyse electrophysiologique de
I 'expression des passions. Paris: Balliere.
Ekman. P. (Ed). (1973). Darwin andfacial expression: A century of research in review. Academic Press:
New York.
Ekman, P., and Friesen, W.V. (1971). Constants across cultures in the face and emotion. Journal of
Personality and Social Psychology, 17, 124-129.
Ekman, P., and Friesen, W.V. (1975). Unmasking theface. New Jersey, Prentice-Hall Inc.
Ekman, P., Sorenson, E.R., and Friesen, W.V. (1969}.Pan-cultural elements in facial displays of emotion.
SCience, 164, 86-88.
Frank, M. (2002). Getting to know your patient: How facial expression reveals emotion. In MKatsikitis
(Ed.). The Human Face: Measurement and Meaning (pp. 255-283). Dordrecht: Kluwer Academic
Publishers.
Gajdusek, D.C. (1963). Kuru epidemiological patro/jrom the New Guinea highlands to Papua: August 21,
1957 to November /0, 1957. BethesdaMd: Study of Child Growth and Development and Disease
Patterns in Primitive Cultures. National Institute of Neurological Diseases and Stroke, National
Institute of Health.
Gellhorn, E. (1964). Motion and emotion: The role of proprioception in the physiology and pathology of
the emotions. Psychological Review, 71, 457-472.
Goffman, E. (1959). The presentation of self in everyday life. Garden City, New York: Doubleday,
Anchor Books.
Good, B.J. (1994). Medicine, rationality, and experience: An anthropological perspective. Cambridge,
Cambridge University Press.
Goodenough, F.L. (1932). Expression of the emotions in a blind-deaf child. Journal of Abnormal and
Social Psychology, 27, 328-333.
Hofstede, G. (1980). Culture's consequences. Beverley Hills, Sage.
Hopper, K. (in press). Interrogating "culture" in the WHO international studies of schizophrenia. In
Jenkins, J., & Barrett, R. (Eds.), The Edge of Experience: Culture, Subjectivity, and Schizophrenia.
New York, Cambridge University Press.
Izard, C.E. (1971). Theface of emotion. New York: Appleton Century Crofts.
Jansz, J. (2000). Masculine identity and restrictive emotionality. In A.H Fischer (Ed.), Gender and
Emotion: Social Psychological Perspectives (pp. 166-186). Cambridge University Press: Cambridge.
Jensen, G. D., & Suryarni, L. K. (1992). The Balinese people. A reinvestigation of character. New York:
Oxford University Press.
Kappas, A. (2002). What facial activity can and cannot tell us about emotions. In MKatsikitis (Ed.), The
human face: Measurement and meaning (pp. 215-234). Dordrecht: Kluwer Academic Publishers.
LaBarre W. (1947). The cultural bias of emotions and gestures. Journal of Personality, 16,49-68.
LakoffG., & Johnson, M. (1980). Metaphors we live by. Chicago: University of Chicago Press.
Landis, C. (1924). Studies of emotion reactions: I. A preliminary study of facial expression. Journal of
Experimental Psychology, 4,447-498.
Leach, E. (1972). The influence of cultural context on nonverbal communication in man. In R.Hinde
(Ed.), Nonverbal Communication (pp 315-347). Cambridge: Cambridge University Press.
Lucas, R. H. & Barrett, R. J. (1995). Interpreting culture and psychopathology: Primitivist themes in
cross-cultural debate. Culture, Medicine and Psychiatry, 19,287-326.
Lutz, C (1988). Unnatural emotions: Everyday sentiments on a Micronesian atoll & their challenge to
western theory. Chicago: University of Chicago Press.

27
Lutz, C., & White, G.M. (1986). The anthropology of emotions. Annual Review of Anthropology, 15, 405-
436.
Matsumoto, D. (1991). Cultural influences on facial expressions of emotion,Southern Communication
Journal, 56, 128-\37.
Matsumoto, D., and Ekman, P. (1989). American-Japanese cultural differences in intensity ratings of
facial expressions of emotion. Motivation and Emotion, 13, 143-157.
Mead, M. (1975). Review of Darwin and facial expression. Journal of Communication, 25, 209-213.
Obeyesekere, G. (1985). Buddhism, depression, and the work of culture in Sri Lanka. In A.Kleinman, &
B. Good (Eds.), Culture and Depression (pp. 134-152). Berkeley: University of California.
Pringle, R. (1970). Rajahs and rebels: The iban of Sarawak under Brooke rule, 1841-1941. London:
Macmillan.
Reynolds, H. (1987). Frontier: Reportsfrom the edge of white selllement. Sydney: Allen and Unwin.
Rosaldo, M.Z. (1980). Knowledge and passion: Illongot notions of self and social life. Cambridge,
Cambridge University Press.
Rose, A. M. (1962) A systematic summary of symbolic interaction theory. InA.M.Rose (Ed.), Human
Behavior and Social Processes: An interactionist Approach (pp. 3-19) London: Houghton Mifflin Co.
Runciman, S. (1960). The white rajahs: A history of Sarawakfrom 1841-1946. Cambridge: Cambridge
University Press.
Russell, J.A. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39,
1161-1178.
Schieffelin, E. (1976). The sorrow of the lonely and the burning of the dancers. New York: St. Martin's
Press.
Schieffelin, E. (1985). Anger, grief, and shamc: Toward aKaluli cthnopsychology. In G.M. White & J.
Kirkpatrick (Eds.), Person, Self and Experience: Exploring PacificEthnopsychologies (pp. 168-182).
Bcrkeley and Los Angeles: University of California Press.
Schlosberg, H. (1941). A scale for the judgement of facial expressions. Journal of Experimental
Psychology, 29,497-510.
Sorenson, E.R., and Gajdusek, D.C. (1966). The study of child behavior and development in primitive
cultures. A research archive for ethnopediatric film investigations of styles in the patterning of the
nervous system. Pediatrics, 37 (I, Pt. 2).
Swift, J. (1984). Gulliver's travels. London, Hcrbert.
Tajfel, H. (1982). Social psychology of intergroup relations. Annual Review of Psychology, 40, 45-81.
Taussig, M. (1993). Mimesis and alterity: A particular history of the senses. New York, Routledge.
Tomkins, S.S. (1962). Affect, imagery, consciousness. Vol. 1. The positive affects. New York, Springcr.
Triandis, ltC., Botempo, R., Villareal, MJ., Asai, M., and Lucca, N. (1988). Individualism and
collectivism: Cross-cultural perspectives on self-ingroup relationships. Journal of Personality Q/!d
Social Psychology, 4, 323-338.
Woodworth, R.S. (1938). Experimental psychology. New York, Holt.
Young, A. (1993). Editorial: W. H. R. Rivers and the anthropology of psychiatry.Social Science and
Medicine, 36, iii-vii.

28
 MACIEl HENNEBERG, ELLIE SIMPSON AND
CARL STEPHAN

CHAPTER 2

HUMAN FACE IN BIOLOGICAL ANTHROPOLOGY:

CRANIOMETRY, EVOLUTION AND FORENSIC
IDENTIFICATION

The face is the body part that epitomises a human person. The face is what is
required for identification of individuals in passports, on driver's licences and
other documents. It can even be argued that the human face is a cultural
construct that cannot be studied without taking into account cultural values
(Berrios, 2002, this volume). And yet the human face is an anatomical entity
that arose through biological processes during the course of human evolution and
its structure is regulated by the same embryological, anatomical and
physiological mechanisms that form all other parts of the body. Thus the face
can be studied by means of natural sciences or more specifically by biological
anthropology, which can be defined as the scientific study of human biological
characters.
The Age of Enlightenment beginning in the 18 th century brought interest in
objective descriptions of the world, this included the human face . In the
Netherlands, Peter Camper introduced the first system of measurements
describing numerically variation of human faces, followed soon thereafter by the
formal system of craniometry established by Johann F Blumenbach. This system
has been largely used by physical anthropologists of the 19th century and was
entrenched in the 20 th century by Rudolf Martin (Martin, 1913) and W.W.
Howells (Howells, 1973). It provides a standardised set of diameters and angles
based on craniometric points which can be measured reliably by anyone familiar
with the system. The system is now universally accepted by physical
(biological) anthropologists. It is also used, with modifications dictated by
clinical needs, by orthodontists and other medical specialists. This ensures strict
comparability of data collected by various anthropologists working in various
countries and in various academic systems. The craniometric system is also
applicable to the fossils providing a record of human ancestry. In this way a
large, uniform, quantitative database describing the variability of human faces
 across geographical space and through evolutionary and historical time has been
provided by numerous craniometric publications. Not all characteristics of the
human face can be described by simple metrics, so a series of standardised
categorical scales describing shapes of the entire face and its elements has been
created within the broader range of descriptive scales. Basic craniometric
dimensions and some cephaloscopic scales are illustrated in this chapter.
Although great apes are our closest relatives in the animal world, the human
face differs considerably from that of an ape. The most striking difference lies in
the reduction of the masticatory apparatus that in humans fits almost entirely
under the anterior part of the braincase instead of protruding forward. The nasal
cavity of humans must warm and humidity the air going into the lungs. The
amount of air required by humans is similar to that of apes, and thus the size of
the nasal cavity has to be substantial. With the reduction of jaws around and
below the nose, the nasal area became prominent and protruding. Position and
size of the eyes is, in general, the same in apes and humans. Significant
reduction of the temporalis muscle and of the supraorbital torus distributing bite
forces to the forehead, produced a steep and rounded forehead above the eyes
that is uniquely human. A small face tucked under a braincase that is rounded
and devoid of thick muscle cover is the hallmark of humanity. Such an
anatomical arrangement is characteristic for foetal and early postnatal stages of
development in all primates, hence it is no wonder that young monkeys and apes
resemble humans much more than adult forms do.
These differences in comparative anatomy of human and ape faces arose in
the process of hominid evolution over the last approx. 5 Ma (million years) ago.
The evolution of the human face from its ape like ancestral form was a gradual
process regulated by changing diets and abilities to process food extra-orally
(outside of the oral cavity) and by the development of the uniquely human ability
to produce spoken language. Our knowledge of dietary change is better than that
of the evolution of speech. The remains of the oldest member of the human
family (Hominidae), Ardipithecus ramidus who lived more than 5 Ma ago in
Ethiopia contain fragmentary jaws with significantly reduced canine and human-
like molars. These characteristics are also displayed by the jaws of the early
australopithecines Australopithecus anamensis (about 4 Ma ago) and A.
afarensis (about 3 Ma ago) who lived in East Africa. Their dental arcades are
large with nearly parallel rows of molars. In the South African A. africanus
(about 2.5 Ma ago) molar teeth become somewhat reduced producing shorter,
less forward protruding jaws which creates a more vertical, and hence human-
like face (Figure I). At the same time, however, hominid forms evolved in both
South and East Africa which had grossly enlarged buccal teeth (molars and
premolars) but rather small, reduced canines and incisors. These are variously
called "robust australopithecines" (A . aethiopicus, A. robustus, A. boisei) or
included into a separate genus Paranthropus. Their molars have especially thick
enamel. The large jaws of these robust forms produce a "dished in" appearance

30
of the face, thick supraorbital ridges and often, sagittal crests where large
temporal muscles meet in the middle of the skull vault. The enlargement of the
posterior dentition has been variously interpreted as adaptation to chewing large
quantities of hard seeds (Jolly 1970) or eating tubers dug out from the ground
(Brain, 1981). Robust forms disappear from the fossil record at about 1.5 Ma
ago.
At the time when robust australopithecines lived, in the same regions, and
sometimes at the same sites in Africa, hominid fossils are found with jaws
reduced in size, especially in relation to the rest of the body and the braincase.
The earliest of these appear just before 2 Ma and are considered to be members
of our own genus Homo belonging to variously defined and named species of H.
ergaster, H. rudolfensis, H. habilis and H. erectus. In all these forms the basic
human pattern of a small face attached to a relatively large braincase is evident,
though to much lesser degree than in present-day humans. Ample archaeological
evidence shows that early Homo used sharp stone weapons and tools to obtain
the meat of large animals. Burnt bones dating back to at least 1.5 Ma were
identified in Swartkrans cave in South Africa (Brain & Sillen, 1988) while
remnants of fireplaces are even earlier in East Africa (Bellomo, 1994).

Figure 1. Theface of the Taung child; Australopithecus africanus, who lived some 2.5 Ma
(million years) ago and died at the age of 5-7 years. (Our photograph of a research-
quality cast.)

The study of the shape of post-cranial skeleton indicates a fully erect position of
the body with small trunk and long limbs, essentially the same as that of modern
humans. The small trunk indicates reduction in the size of the contents of the
abdominal cavity - reduction of the gut caused by consumption of higher quality

31
 foods (mostly meat and fat), which were partially processed before eating
(chopped and cooked).
By 1.8 Ma early humans were living in Europe (Spain, Georgia), and Asia
(Indonesia, China, Pakistan). To acquire meat, to produce well-designed
weapons and tools and to survive in a variety of environments, early humans had
to have complex social organisation that necessitated efficient inter-individual
communication. Although there is no direct fossil evidence for this, spoken
language most probably appeared in these early times. There has been
considerable controversy about the time that spoken language originated in
human evolution.
Speech is produced from soft tissue structures of the larynx that are not
preserved in the fossil record. Any interpretations about the anatomy of the
larynx in fossils are based on extrapolations of the relationship of these
structures to hard tissues such as the cranial base, palate and hyoid bone. The
contribution of the hyoid bone to speech is made even more complicated by the
fact that this bone does not articulate with other bones, but is suspended above
the larynx by soft tissues. In addition, this bone is rarely preserved in fossil
remains. Relationships of various parts of the vocal apparatus are determined by
studying the supra-laryngeal tract and associated structures in extant forms of
humans and apes.
There is a significant difference between apes and humans in the anatomy of
structures related to production of speech. Vocalisation, especially production of
vowels, requires a large resonant space above the glottis of the larynx where
vocal chords' vibrations produce sound waves. In apes, the larynx is located high
in the throat while the pharyngeal cavity linking the larynx with the posterior
part of the nasal cavity is small. In humans, due to the greater flexion of the
cranial base, and shortening of the facial skeleton, the larynx is located lower in
the throat while the height of the pharynx is increased. This allows for good
resonance of vowels. The reduction of the jaw size with accompanying
diminution of the muscles of mastication, produced in humans a versatile
organisation of the mouth and tongue that allows for precise articulation of
sounds. We pay the price, though, since with our larynx located lower at the
base of the tongue we are unable to drink and breathe at the same time, as is
possible with the superiorly located larynx of the apes. Newborn humans have a
highly positioned larynx. This allows them to drink through two "gutters"
surrounding the larynx at the same time as breathing through the aditus (inlet) to
the larynx. With high position of the larynx the inlet opens above the two
conduits for food and liquids so that liquids cannot enter into the airway. As the
child grows the larynx slowly descends thus enabling production of the sounds
required for speech.
Early investigations showed that laryngeal anatomy in modern, adult humans
differs from that in infant humans, chimpanzees and Neanderthal skulls, and it is
suggested that the modifications of the adult vocal tract allow the production of

32
 the vowel sounds [a], [i] and [u]. Infant humans and chimpanzees are not
capable of producing these sounds, and their vowel repertoire consists of the
sounds [e] and [I] and [U]. The latter sounds can be produced by different
combinations of the vocal tract, whereas the former sounds are limited in their
sound by the supra-laryngeal structures, making production only possible in
modern human adults (Lieberman, Crelin, & Klatt, 1972). The structures that
allow the production of the sounds [a], [i] and [u] include a flexed cranial base,
relatively shorter distance between the palate and basion (anterior border of the
foramen magnum), and the hyoid and larynx positioned low in the neck
(Lieberman & Crelin, 1971). This hypothesis supports the notion that speech
(and therefore language) was not possible until modern Homo sapiens (Laitman
1985; Lieberman & Crelin, 1971; Lieberman, Crelin, & Klatt, 1972). However,
a number of other researchers suggest that the relationships between cranial base
flexion, horizontal length of the pharynx and location of the larynx are more
closely related to swallowing and breathing than they are to vocalisation. These
researchers questioned the validity of the Neanderthal and modern human
comparisons in the studies conducted by Lieberman, Crelin and colleagues, as it
appears that the reconstruction of the laryngeal structures in Neanderthals was
incorrect, and that the Neanderthal vocal tract may not differ from that of
modern humans(Carlisle & Siegel, 1974; Falk 1975; Houghton 1993). In
support of this, Arensburg and colleagues (1990) suggest that the anatomy of the
mandible and the hyoid bone is more important in reconstructing laryngeal
anatomy than the flexion of the cranial base. These researchers conclude that a
Neanderthal fossil of about 60,000 years ago from the Middle Palaeolithic
period, the Kebara hominid, had cervical and laryngeal anatomy that resembled
that of modern humans, and was as capable of speech as modern humans
(Arensburg et a!., 1990).
At present, the debates of when speech (and therefore language) became
possible have not been resolved. Research continues into the definition of
language (and other symbolic forms of communication), the role of speech in
language, and the relationship between anatomical structures and the capacity for
speech.
Reduced dentition and face, fully erect body with reduced trunk size resulting
in the brain size/body size ratio falling into the range characteristic for modern
humans (although at its lower end) and the ability to exploit a variety of
environments by the use of stone-based technologies and fire, led some authors
to believe that these early humans living at about 1.5 Ma should be included into
our own species (H. sapiens) rather than being relegated to H. erectus and their
later forms to H. heidelbergensis and H. neanderthalensis (Henneberg, 1990;
Wolpoff, 1996). Irrespective of taxonomic debates, in the last 1.5 Ma the human

33
 Early Australopithecine
3.5 Ma (A. q{arensis)

Classic Australopithecine
2.5 Ma (A. qfricanus)

I.OM3
('·-- ---~\~l
0J~ f(
Early Homo

(H. ereclIIs)

'- f'~~.>
~
Modern Homo
o Ma (If. sapiens)

Figure 2. Lateral views a/skulls a/basic hominid/arms illustrate the relationship

between the size and shape o/the/acial structures and the braincase. Note reduction in
the protrusion o/the/ace and its overall height. Reconstructions based on Aiello and
Dean (1990). Drawings not to scale, time scale approximate.

face was undergoing further reduction in the masticatory apparatus and the size
of dentition and this trend continued even in historical times (Brace, Smith, &
Hunt, 1991; Frayer, 1978), but without any dramatic restructuring. Changes in

34
 ~:§--'.:
4'

-
\ ..
"'"
'--'
,

/'~'~ \

Figure 3. Variation of male facial features in one European population - Island of Crete,
(These are standardised tracings from photographs, tracings by E Simpson from the
photographs published by Poulianos, 1971),

the size and shape of the face during the hominid evolution are illustrated in
Figure 2.
In addition to variation in facial morphology seen between anatomically
modern humans and fossil hominids are variations within modern Homo sapiens.
Slight modifications in the structural elements of the face (bones, muscles,
cartilage, adipose tissue) allow individual features or facial morphology to be
superimposed over the general, modern human face pattern. These variations in
facial morphology arise through differential growth, and create an individual

35
face that allows us to distinguish one person from another. These variations are
controlled by a number of factors, including genetic heritage (why we resemble
our relatives) and the climate or environment in which we live. Influence of sex
hormones makes us look more male or female. The combination of all of these
influences produces slight modifications in size or shape of different parts of the
face, which result in the development of a unique, recognisable visage. Even
within the same population, variation of facial features is considerable (Figure
3).

34
Nasal width
(mm)
32
0
0

,
0
30
o 0 0
o X 0 0

28 OX 0 + X+
0
X OXX -+X
26
0
X X
.+X
0 +X++ +
+ +

0
24 0 X XX ++ + +
X X +X
22 +

20+---~--~--------~------~------~
110 120 130 140 150
Bizygomatic diameter (mm)

Figure 4. Distribution 0/ individual nasal widths and bizygomatic diameters from skulls

(0). The averages/or each sample are plotted (. Norwegian, =

0/20 Norwegian males (+), 20 Ugandan males (X) and 20 Australian aboriginal males
Ugandan and. =
Australian aboriginal). Note that differences between averages are less than differences
.=
between many individuals, and that there is substantial overlap 0/ individual
distributions. (Data/rom Schreiner, 1939; Gorny, 1957; Milicerowa, 1955).

The appearance of a person is not constant during life, but shows gradual
changes with age. These changes include the growth of the face, the changes
seen in a child with the loss of primary teeth and the emergence of the permanent
dentition, the maturation of the adult face after puberty, the loss of teeth and the
formation of lines and wrinkles in the skin that characterise old age. During

36
childhood growth, different parts of the face grow at different rates. The bones
around the cranial vault (surrounding the brain) do not grow much after about 7
years, whereas the bones of the facial skeleton and the jaws initially grow
slower, but for a longer time. While most bones of the face stop growing around
puberty or early adulthood, soft tissues such as the cartilages of the nose and ear
will keep growing throughout life. The time at which bones stop growing
depends on various factors, such as a person's genetic background, nutritional
status, and the age at which they undergo puberty. Puberty is an important time
in development, and signifies a time when most skeletal growth ceases.

A
B

Figure 5. Some measurements of the face: A). Bilateral dimensions: eu-eu - maximum
breadth of the braincase, ek-ek - external interorbital width, width - bony nose width, al-
ai- soft nose width, ekm-ekm - width of the upper jaw, go-go - mandible width, zy-zy-
maximum face width. B). Vertical dimensions: n-ns - nose height, n-pr - upper face
height, n-gn - total face height, go-gn - mandibular body length. Points: eu - euryon -
most lateral point on the braincase, ek - ectokanthion - the lateral-most point on the orbit,
al - alare - most lateral point on the wing of the nose, ekm - ektomolare - a point at the
alveolar margin ofthe lateral root of the second molar, go - gonion - mandibular angle,
zy - zygion - most lateral point on the zygomatic arch. n - nasion - at the root of the nose,
ns - nasospinale - at the base of the anterior nasal spine, pr - prosthion - at the upper
border of the enamel of the central incisors, gn - gnathion - lower-most point on the chin.
There is noformal point to measure width of the bony opening of the nose. (After
Malinowski & Bozilow, 1997, modified).

37
 It is commonly known that most girls begin and end puberty at younger ages
than boys. The effect this has on their growth creates a face that is, on average,
much "younger" in appearance when compared to boys. Comparisons between
female and male skulls show that males grow for a longer time (later puberty)
and hence have not only larger skulls, but also more sloping foreheads and more
prominent areas for muscle attachment (especially the muscles which balance the
head on the neck). Differences are also seen in the relative size of the bony
orbits and the shape of the mandible. In addition to variation in facial
morphology during development and between males and females, differences
have also been observed between more or less closely related people. Since the
human species maintained its genetic cohesion over hundreds of thousands of
years by constant gene flow between continents and regions (Hawks & Wolpoff,
2001), a large portion of variation in anatomical details of the face is shared by
members of various popUlations, and it is only a few minor characteristics that
distinguish people whose ancestors lived on different continents. This principle
is illustrated by comparing the amount of variation in the width of the face or
width of the nose that is due to differences between populations with the amount
of variation between individuals within each of the populations (Figure. 4).
Facial dimensions of people from any country and from any time period can
be compared uniformly. Biological anthropologists have devised over the years
a quantitative system of describing human faces by a number of standardised
dimensions and indices derived from these dimensions. Figure 5 illustrates
craniometric points used to characterise human faces. The basic facial
dimensions are: face height measured from the point "nasion" (root of the nose)
to the point "gnathion" under the tip of the chin, bizygomatic (maximum) width
of the face measured between the two most lateral points on zygomatic arches,
nose height measured from nasion to "subnasale" (base of the nose), nose width
between the most lateral points on the wings (ala) of the nasal cartilages (Figure
6), inter-pupillary distance, width of the mouth and lip thickness (Figure 7).
Many other dimensions are used for specialised studies. To express shape rather
than size, quotient indices are used. These indices express one facial dimension
as a percentage of the other. For example, the face index is the face height
(nasion-gnathion) as a percentage of face width (bizygomatic diameter). This
index classifies faces as follows:

Males Females
Very wide x-78.9 x-76.9
Wide 79.0-83.9 77.0-80.9
Medium width 84.0-87.9 81.0-84.9
Narrow 88.0-92.9 85.0-89.9
Very narrow 93.0-x 90.0-x

38
The nasal index is the width of the nose as a percentage of its height (nasion-
subnasale): It classifies noses as follows (males and females jointly):

Very narrow x-54.9

Narrow 55.0-69.9
Medium 70.0-84.9
Wide 85.0-99.9
Very wide 100.0-x

Further details on facial characteristics can be found in Farkas (l994a).

Averages of the major facial dimensions are presented in Table I, together with
some basic quotient indices (Morris, 1992).
Geographic, inter-populational variability of human facial forms results from
two processes: Darwinian adaptation to local conditions and migrations. The
most obvious adaptive characters of the human face include the width of the nose
which is greater in warm climates and the size of jaws and dentition which
decreases with the sophistication of food preparation techniques. The micro-
evolutionary trends are those of general reduction in the size of the human skull
which includes the decreasing robusticity of the facial skeleton.

Dimension Europeans South African Khoi-San

Blacks (Bushmen)
male female male female male female

Minimum fore- 92-101 86-98 93-103 89-97 90-100 88-97

head width (ft-ft)

Maximum face 123- 117- 125- 114- 119- 113-

width (zy-zy) 132 127 135 125 128 124

Upper face height 68-76 64-73 64-73 60-68 56-65 53-64

(n-pr)

Nasal breadth 22-25 21-25 26-31 25-29 24-29 23-28

Table 1: Major dimensions of the human face in three groups, data from Morris (1992)
recalculated into ranges. Ranges are those of me an +/- standard deviation, this includes
68% of individuals in a given population. Note that most ranges overlap.

39
 Figure 6: Variation in theform of inferior aspect of the human nose and the height of
human lips. Top row: forms possibly adapted to breathing cold, dry air; bottom row:
forms possibly adapted to warm, humid climates. (After Malinowski & Bozi/ow, 1997,
modified).

Figure 7: Variation in the height of human lips. (After Malinowski and Bozilow, 1997,
modified)

Some pressures of natural selection were relaxed by the increasing role of

technological and cultural adaptations. This produced increased intra-
populational variability of facial characteristics. Although averages of metric
characters and average whole face images differ significantly between
populations, an individual face can be placed into a particular population with

40
 varying degrees of probability only because a large inter-individual variation
exists in each population. Ranges of this variation overlap between populations.
Contrary to the popular belief that faces of members of various human "races"
differ predictably, the majority of the variation of facial metric characters occurs
among members of same populations, while sexual dimorphism and population
differences each are responsible for minor, though significant portions of the
total facial variation. We have conducted an analysis of the facial soft tissue
depths of a number of white Australian cadavers, and compared these results
with those of published facial soft tissue depths of people from other
populations. These have been presented as tables of average soft tissue depths,
and include white Americans (Rhine & Moore, 1982, in Prag & Neave, 1997),
black Americans (Rhine & Campbell, 1980), Germans (Helmer, 1984), several
groups in Eastern Europe/Asia (Korean, Buryat, Kazakhistan, Uzbekhistan,
Armenian, Russian, Lithuanian, Bashkirs) (Lebedinskaya, Balueva, &
Veselovskaya, 1993), Japanese (Suzuki, 1948), and African Zulus (Aulsebrook,
Becker, & Iscan, 1996). Each new study presents the data as being specific for
that population, on the basis of significant differences being identified between
the average values of that population compared to the average values of
previously published studies. When these data, including our own white
Australian measurements, were compared, we found that the average differences
seen between these populations (inter-population variation) were of little
significance when intra-population variation was considered. Only five soft
tissue depths were found to have more than 25% variation between populations.
These were the upper and lower lips (28% and 43% respectively), pogonion,
(depth at the soft tissue point of the chin, 27%), gonion, (at the angle of the
mandible, 26%), and a point on the maxilla about 15 mm lateral to the nasal ala
(32%). Differences between males and females were slight, with the greatest
differences seen in the upper lip (8%), nasion (6%) and metopion, (at the centre
of the forehead, 6%). All variables showed substantial individual variation, all
having at least 50% of the variation attributable to individual variation, and more
than two thirds (12 out of 17) having more than 70% individual variation.
Our findings contribute to the existing literature recognising the substantial
contribution of individual variation to the total observed variability among
humans. These studies have observed that individual variation contributes 50%
to cranial capacity and body weight variation among populations, while variation
between populations contributes only 25%, the rest being sexual dimorphism
(Henneberg, 1992). The conventional system gives preference to the relatively
small contributions of inter-sexual and inter-populational differences. Our study
shows that there are situations where the classification of a person into a "race"
may be of some value, for example in forensic science for narrowing down the
list of possible victims. However, it also highlights the arbitrary nature of the
"racial" classification system, and that division of human groups based on
physical characteristics has no biological meaning (see Figure 4).

41
 Since the human face is prominent in interpersonal communication, it is no
surprise that it appears to be the most distinguishing physical feature of a
person's identity (Burton et al., 1999). For this reason physical anthropologists
are often called upon to help in forensic identification. The human face
advertises a person's sex, age, emotional state, some genes, previous
environmental exposure and health. In Western societies the face is also one of
the few anatomical parts of the body that generally remains naked (apart from
the hands) and therefore, readily signals many honest cues (excepting make-up).
This allows people to reliably and quickly identify each other.
There are many scenarios in which the comparison of facial appearances can
aid in forensic identification. A witness may make facial comparisons from a
line up of individuals to his/her memory, to determine if he/she can recognise the
offender. Biological anthropologists are often called upon to attempt to
determine the facial appearance of an individual who is missing, a suspect in a
crime, or dead.
Physical anthropologists may be involved in the comparison of facial images
when surveillance cameras record the facial appearance of a criminal at a crime
scene. A facial image of the suspect can be compared to the surveillance image
to determine if the two faces may be of the same individual. Indices and
morphological assessments are used for comparisons to eliminate any differences
in image size. This method is often limited, since true full face and profile views
are rarely available for comparison. The morphological assessments may also be
rather subjective, reducing the reliability of the method. Factors such as lighting,
emotional expression, camera lens and distance to subject may also vary between
photographs, altering the appearance of the face and subsequent reliability of the
comparisons (Iscan, 1993; Yoshino et aI., 2000).
When an individual has been missing for a prolonged period of time, police
departments may also use the help of a biological anthropologist (usually in
conjunction with an artist) to age progress an individual's face, usually from
photographs. This gives police officers and the public a new image of the person
that hopefully is a more truthful or recognisable representation than the original
pictures. Age progressions are often done on missing children but may also be
done on adults. Adult progressions are technically simpler since the progression
relies mainly upon aging processes alone. In contrast, child progressions are
more complicated since they require anticipation of growth processes of the face
as well as ageing processes (Taylor, 200 I). Average craniometric measures are
used in the process to produce a face of the required chronological age (Farkas,
1994b). It is also essential that many photographs of the individual are reviewed
prior to beginning the progression as well as gathering information on the
individual's life history (occupation, environmental exposure, medical conditions
and genetic background) (Taylor 200 I). Often family photographs are also used
for comparisons of children whose faces may be progressed with certain

42
similarities of family members whilst still retaining 'their individual identity'
(Clark, 1994; Taylor, 2001).

Figure 8. An example of video superimposition of a skull over a face. In this example, it is

not likely that the skull belongs to the individual shown since the skull does not fit in the
correct anatomical location beneath the face.

Since facial appearance is dependent upon both soft and hard tissues of the head,
forensic anthropologists may also use the relationship between the soft and hard
facial tissues to aid in identification. This can take various forms, such as video
superimposition or facial approximation (modelling of a face on a dry skull).
When a skull is suspected to belong to a particular individual it can be
compared to a suspect's face by video superimposition. This technique involves
mixing a video image of a photograph of the suspected individual with the video
image of the skull (Figure 8). When the skull is aligned to the same position as
the head of the photographed suspect it can be determined if the skull fits into the
correct anatomical position beneath the face of the suspect. This is done by
varying the opacity of the skull image over the photograph or vice versa. If the
skull fits correctly then a possible identification has been made. This method is
useful to eliminate people to whom the skull does not belong. However, the
reliability of superimposition may also be affected by the difference between the
initial photography conditions and the video camera conditions (Glassman
2001).

43
 Figure 9. An example offorensic facial approximation using a three dimensional clay
sculpting technique. Half of the face has been removed to display the muscular framework
beneath the cutaneous and subcutaneous tissue. These muscles have been built according
to skull morphology and the wood dowels. indicating the average soft tissue depths. have
heen used as a guide for determining facial soft tissue depth.

Biological anthropologists may also be involved in facial approximation, the

process of building faces from dry skulls (Figure 9). The aim of this procedure
is to approximate the facial appearance of an individual that can then be
recognised as the person to whom the skull belongs. Variations of the three-
dimensional clay sculpting technique are commonly used methods of facial
approximation (Gatliff, 1984; Prag & Neave 1997; Stephan & Henneberg, 200 I;
Taylor, 200 I; Taylor & Angel, 1998). This method is also used to determine the
appearance of ancient humans and hominids, and is often used in forensics when
other methods of identification have not been successful ego dental identification,
DNA identification. Forensic facial approximations are used to provide law
enforcement agencies with tentative identifications of individuals who are
recognised from the face built on the skull. These leads can then be investigated
to determine if the skull belongs to one of the tentatively identified persons
(Taylor, 2001; Tyrell et aI., 1997; Vanezis & Vanezis, 2000).
Since it appears that some soft tissue features have little association with the
skull, for example head/facial hair, freckles, wrinkles, it is highly improbable
that facial approximations will generate an exact likeness of the person to whom
the skull belongs. Although facial approximations appear to display similarities
to the individual to whom the skull belongs, they appear to be infrequently
recognised correctly when recognition is based on the face alone (Stephan and

44
 Henneberg 200 1). In addition it has been proposed that contextual information
may be the primary factor responsible for the identification of facial
approximations in forensic scenarios (Haglund & Reay, 1991).
The aetiology of misidentifications from facial approximations may be due to
either ambiguities in the facial approximation method or in the perception of the
facial approximation. It is presently unknown what effect errors in a facial
approximation have on the perception and "recognisability" of it. It also appears
that anatomically correct facial approximations may be misidentified as a result
of the viewer's perception. Facial features such as hair and skin pigmentation
(for example, freckles, birth marks) may not be represented on a facial
approximation as they cannot be accurately determined from the skull. As a
result the individual may not be recognised despite an accurate representation of
the rest of the face.
Many of the methods mentioned above involve the interpretation of the soft
tissue from the skull, currently relatively little is known of the soft tissue
relationship between these two. Consequently the respective methods above are
rather unreliable. The struggle to improve these methods does not only require
anthropometric examination of the actual relationships between the soft and hard
tissues, but also scientific contributions from other fields of science that include
facial emotion, facial recognition, facial perception, and facial evolution. The
combination of these specialties should generate a deeper understanding of all
the mechanisms involved in determining a personal identification from the face.
We have attempted to illustrate above some of the areas in which biological
anthropology contributes to our understanding of human faces and applications
of some of this knowledge to forensic science. All in all, faces are complex and
highly dynamic structures. While it readily comes to mind that faces not only
change from second to second with our expression, but also from year to year
with age or new hair styles, it is, however, the gradual change of faces, brought
about by many millennia of natural and sexual selection, that is responsible for it
all. Despite our unique and individualised faces we all have shared similar
evolutionary processes that have shaped our faces, on the front of our heads, with
a regular set, pattern and function of features. If we are to understand the how
and why of human faces we must understand their evolution. This is one of the
ultimate and continuing aims of biological anthropology of the face.

45
 AFFILIA TION

Department of Anatomical Sciences

The University of Adelaide
Australia
Email: m.henneberg@adelaide.edu.au

ACKNOWLEDGMENT

Special thanks to Jane Taylor (Forensic Odontology Unit, The University of

Adelaide) for use of video superimposition equipment; and Joseph D'Arienzo
and Peter Knowles for construction of the prosthetic eyes used in the facial
approximation.

REFERENCES
Aiello, L., & Dean, C. (1990). An introduction to human evolutionary anatomy. London: Academic
Press.
Arensburg, 13, Schepartz, L.A., Tillier, A.M., Vandermeersch, B., & Rak, Y. (1990). A reappraisal of
the anatomical basis for speech in middle palaeolithic hominids. American Journal of Physical
Anthropology, 83, 137-146.
Aulsebrook, W. A, Becker, P.1., & Iscan, M.Y. (1996). Facial soH tissue thickness in the adult male
Zulu. Forensic Science International, 79,83-102.
Bellomo, R. V. (1994). Methods of determining early hominid behavioral activities associated with
the controlled use of fire at FxJj20 Main, Koobi Fora, Kenya. Journal of Human Evolution, 27,
173-195.
Ikrrios, G. E. (2002). The face in medicine and psychology: A conceptual history. In M. Katsikitis
(Ed), The human face: Measurement and meaning (pp. 49-62). Dordrecht: Kluwcr Academic
Publishers.
Brace, C. L., Smith, S.L., & Hunt, K.D. (1991). What big teeth you had grandma! Human tooth size
past and prescnt. In, edited by M.A. Kelly & C.S. Larsen (Eds), Advances in Dental
Anthropology (pp. 33-57). New York: Wiley-Liss.
I3rain, C. K. (1981). The hunters or the hunted: An introduction to African cave taphonomy. Chicago:
University of Chicago Press.
Brain, C. K, & Sillen, A. (1988). Evidence from the Swartkrans eave for the earliest lise of fire.
Nature, 336,464-466.
Burton, A. M., Wilson, S., Cowan, M., & Bruce, V. (1999). Face recognition in poor-quality video:
Evidence from security surveillance. Psychological Science, 108,243-248.
Carlisle, R. c., & Siegel, M.1. (1974). Some problems in the interpretation of Neanderthal speech
capabilities: A reply to Lieberman. American Anthropologist, 76,319-322.
Clark, B. (1994). Method used in "aging" the photographic records of missing children. In L. G.
Farkas (Ed), Anthropometry of the head andface (pp. 184-186). New York: Raven Press.
Falk, D. (1975). Comparative anatomy of the larynx in man and the chimpanzee: Implications for
language in Neanderthal. American Journal of Physical Anthropology, 43, 123-132.
Farkas,. L. G. (I994a). Anthropometry of the head andface in medicine. New York: Raven Press.
Farkas, L. G. (1994b). Anthropometric guidelines in photographic reconstruction of a missing child's
faee. In L.G. Farkas (Ed), Anthropometry of the head andface. (pp. 182-183). New York: Raven
Press.

46
Frayer, 1. D. (1978). Evolution of the dentition in upper paleolithic and mesolithic Europe.
Publications in Anthropology. Lawrence: University of Kansas.
Gatliff, B. (1984). Facial sculpture on the skull for identification. American Journal of Forensic
Medicine and Pathology, 5, 327-332.
Glassman, M. (2001). Methods of superimposition. In K. Taylor (Ed), Forensic art and illustration
(pp. 477-498). Florida: CRC Press.
G6rny, S. (1957). Crania Africana: Uganda. Wrociaw: Polska Akademia Nauk.
Haglund, W.D., & Reay, D.T. (1991). Use of facial approximation techniques in identification of
Green River serial murder victims. The American Journal of Forensic Medicine and Pathology
12, 132-142.
Hawks, J D, & Wolpoff, M. (2001). The four faces of Eve: Hypothesis, compatibility, and human
origins. Quaternary International, 75,41-50.
Helmer, R. (1984). Schadelidentijizierung durch elektronische bildmischung. Heidelberg:
Kriminalistic Verlag, GMBH.
Henneberg, M. (1990). Brain size/body weight variability in Homo sapiens: Consequences for
interpreting hominid evolution. Homo, 39, 21-130.
Henneberg, M. (1992). Continuing human evolution: Bodies, brains and the role of variability.
Transactions of the Royal Society of South Africa, 48, 159-182.
Houghton, P. (1993). Neandertal supralaryngeal vocal tract. American Journal of Physical
Anthropology, 90, 19-146.
Howells, W W. 1973. Cranial variation in man. Vol. 67, Papers of the Peabody Museum.
Cambridge: Harvard University.
Iscan, M Y. (1993). Craniofacial image analysis and reconstruction. In M. Y. Iscan & R. P. Helmer
(Eds), Forensic analysis of the skull. (pp. 1-9). New York: Wiley-Liss.
Jolly, C. J. (1970). The seed eaters: A new model of hominid differentiation based on baboon
analogy. Man, 5, 5-26.
Laitman, J. T. (1985). Evolution of the hominid upper respiratory tract: The fossil evidence. In P. V.
Tobias (Ed), Hominid Evolution: Past, Present and Future (pp. 281-286). New York: Alan R
Liss.
Lebedinskaya, G V, Balueva, T.S., & Veselovskaya, E.V. (1993). Principles offacial reconstruction.
In M. Y. Iscan & R. P. Helmer (Eds), Forensic analysis of the skull (pp. 183-198). New York:
Wiley-Liss.
Lieberman, P, & Crelin, E.S. (1971). On the speech of Neanderthal man. Linguistic Inquiry, 2,203-
222.
Lieberman, P., Crelin, E.S., & Klatt, D.H. (1972). Phonetic ability and related anatomy of the
newborn and adult human, Neanderthal man, and the chimpanzee. American Anthropologist, 74,
287-307.
Malinowski, A., & Bozilow, W. (1997) Podstawy antropometrii: Metody, techniki, normy. Warsawa:
Wydawnictwo Naukowe PWN.
Martin, R. (1913). Lehrbuch der anthropologie. Stuttgart: Fisher.
Milicerowa, A. G. (1955). Crania Australica. Wroclaw: Polska Akademia Nauk.
Morris, A. G. (1992). The skeletons of contact. Johannesburg: Witwatersrand University Press.
Poulianos, A. (1971). 1 katagogi ton Kriton. Athens: Anthropologiki Etaireas.
Prag, J, & Neave, R. (1997). Making faces: Using forensic and archaeological evidence. London:
British Museum Press.
Rhine, J. S, & Campbell, H.R. (1980). Thickness of facial tissue in American blacks. Journal of
Forensic SCiences, 25,847-858.
Schriener, K. E. (1939). Crania Norvegica. Oslo: H Aschehoug & Co.
Stephan, C N, & Henneberg, M. (2001). Building faces from dry skulls: Are they recognized above
chance rates? Journal of Forensic Sciences, 46,8-16.
Suzuki, K. (1948). On the thickness of the soft parts of the Japanese face. Journal of the
Anthropological Society of Nippon, 60, 7-11.
Taylor, K. (2001. Forensic art and illustration. Florida: CRC Press.

47
Taylor, R G., & Angel, C. (1998). Facial reconstruction and approximation. In J. G. Clement & D. L.
Ranson (Eds), Craniofacial Identification in Forensic Medicine (pp. 177-185). London: Arnold.
Tyrell, J., Evison, M., Chamberlain, A., & Green, M. (1997). Forensic three-dimensional facial
reconstruction: historical review and contemporary developments. Journal of Forensic Science,
42,649-661.
Vanezis, M., & Vanezis, P. (2000). Cranio-facial reconstruction in forensic identification--historical
development and a review of current practice. Medicine, Science and the Law, 40, 197-205.
Wolpoff, M. (1996). Human evolution. New York: McGraw-Hili.
Yoshino, M., Matsuda, M., Kubota, S., lmaizumi, K., & Miyasaka, S. (2000). Computer-assisted
facial image identification system using a 3-D physiognomic range finder. Forensic Science
International, 109,225-237.

48
 GERMAN E BERRIOS

CHAPTER 3

THE FACE IN MEDICINE AND PSYCHOLOGY:

A CONCEPTUAL HISTORY

THE FACE IN MEDICINE & PSYCHOLOGY: A CONCEPTUAL HISTORY

Confirmation of current 'truths' is not the task of history.l The historian cannot
accept all that is officially predicated of the object of inquiry, including that it is
a 'natural kind'.2 This applies to the concept of 'human face,.3 According to the
OED (1992) the face is defined as
The front part of the head, from the forehead to the chin; the visage, countenance in
man .. (In Anat. sometimes with narrowed sense, as excluding the forehead: see
quot. 1831)

So, is the forehead part of the face or or not? Who is to decide? Neither
'operational definitions' ,4 nor further empirical research can 'resolve' this
problem (see Henneberg et aI., this volume). Indeed, it is likely that the stability
and validity of the construct 'face' is parasitical upon the 'ontology' of its
components, namely, forehead, nose, eyes, cheeks, mouth, etc. (for their history
see Lange, 1942);5 and its endurance is based on the usefulness of its social6 ,
economic and religious? role. The same historical approach should be taken of
the 'semaphore model' ,8 i.e. the view that the face is a natural device for the
signaling of information. This is because explanations by 'historical design',
whether divine or by evolution, have exiguous explanatory force (Davies, 200 1).
In general, the claim that all facial gesturing is always 'informational' can be
either foundational or empirical. Foundational claims do not need empirical
confirmation but are open to challenge and negotiation. Empirical claims are
sturdier but need justification. For example, the foundational claim is based on a
broad definition of 'signaling information' (i.e. that all facial gestures are
expressive and hence intentional). This clashes with the observation that there
are facial movements that do not seem to be expressing or signalling any
information to others.
 The hypothesis that the face is an artefact or higher level construct invites
broader questions including those pertaining to its endurance as a concept and to
whether the face can be studied independently from the social mind (Forgas,
Williams, & Wheeler, 200 I), cultural networks, the rest of the human body, and
the 'faces' of relevant others (i.e. those enveloped in the same cognitive or
emotional episode). Given its condition as a construct, the issue here is whether
the meaning of 'face' requires a dialogical or 'polyphonic' context. 9 For
example, it is possible to imagine a group emotion that in each member of the
group is represented by a different facial gesture. In such situation, the unit of
analysis would be the group as the face of each individual and would only
express (as in orchestral music) part of the emotional narrative.1O Other thought
experiments could be easily construed to show that the view that the face is a
'fixed' object, sufficiently explained in terms of 'design' (whether divine or
evolutionary) is heuristically limited.
In this regard, the historian could go as far as suggesting that the only way to
do justice to the concept of 'face' would be to create a hermeneutic science
which would equally borrow from biology, theology, poetry, history, philosophy,
portraiture, and aesthetics; and given that in Europe the concept of 'face'
developed pari passu with others related to the formation of individualism as a
political philosophy (e.g. 'person', 'self, 'soul', 'character',ll 'expression',
'likeness',12 'portrait'I3 and caricature, (Wechsler, 1982) the study of the face
should be linked up with their history.
Based on a detail analysis of the history of the construction of the concept of
face, various hypotheses about its function could be tested:
a) Is the face a 'corollary projection system' for humans to rehearse past
and future emotions and gestures? Coded in an idiosyncratic shorthand,
such gestures would not be destined at all for public consumption. 14

b) Is the face an instrument to generate componential gestures, parts of

collective expressions or emotional choirs? These polyphonic messages
would be a form of expression of families or tightly knitted cultural
groups and used for higher-level communication. Comprehension of
this function would once again be well beyond the semaphore model.

c) Is the face a semaphore that is a biological system designed to express

information (whether spontaneous or not; or overtly or covertly) about
some aspect of the psyche of the possessor?

In the ideal world, the above hypotheses should all be researched upon with the
same interest. The fact that the semaphore model has predominated needs
therefore to be explained in some detail. 15

THE FACE AND EMOTIONS IN HISTORY

Whatever its predominance in current research, it is clear that the history of the
concept of the human face is broader and older than that of the 'semaphore'

50
model. The latter, however, can already be found in Babylonian culture where
the belief was present that the structure of the face reflected or 'correlated' with
notions equivalent to character, temperament, intelligence, personality, emotions;
in summary, that the face could provide private information. Since those days
the logical frame on which the semaphore model is based has not changed much
in that it remains based on the 'correlation' between two components, one of
which is used to make 'predictions' about the other (ManeUi, 1993). A similar
structure sustained the use if animal allegories. For example, the belief that the
rostral features of a lion or eagle related to their (mythical) valour and capacity to
soar high, respectively, was used to predict that human beings who 'looked like'
a lion or an eagle possessed those very personality features. Indeed, this
structure can be found in Aristotle's Physiognomica (see below), in Lavater
(1789), and in Darwin (1904) who believed that stereotyped facial emotion tells
more about the species than the individual.
The current concept of emotion (Gardiner, Metcalf, & Beebe-Center, 1937;
McCosh, 1880) developed during the late 18th century when aspects of the old
notions offeelings (Ziegler, 1893) and passion (Brunschwig & Nussbaum, 1993;
Levi, 1964; Virey, 1819)16 were combined with ongoing fibre and vibration
physiology, the 'sensible body' and Lockean associationism (Vila, 1998).
During the 19th century, this narrow concept was made into a natural kind, given
a brain address, and considered as the substratum of obscure selection forces
(Braund & Gill, 1997; James, 1997; Levi, 1964; Meyer, 2000; Sorabji, 2000). It
was this new concept that Darwin (Ekman, 1998), Duchenne (1862) and later in
the century Lange and James (Gardiner et ai, 1937) incorporated into their work ..
The old principle relating the soul to the prosopon was thus naturalized and the
belief started that such a relationship survived on account of some evolutionary
advantage (Fridlund, 1994). This process of universalization also caused the loss
of insights into the social (Despret, 1999) and national (Wierzbicka, 1999)
aspects of some emotions.
The way in which the concept of emotion has actually developed during the
last two centuries shows that it is nonsense to talk about 'Descartes' error',
overemphasize his dualism, and anachronistically accuse him of ignoring a
concept of emotion (Damasio, 1994) which was only constructed 150 years later.
In fact, the concept of 'passion' was central to Descartes' view of man
(Descartes, 1989; Kenny, 1963; Meyer, 2000; Naya, 1998). Originally
epistemological (Foster, 1991) rather than physiological, Descartes' dualism was
overemphasized by 18th century writers making materialist claims about the
brain and mind (soul) in order to protect themselves from religious persecution.

PROSOPON AND THE BIBLE

The Biblical notion of 1tP---O~01tOU is the proto-concept of the face. Originally

meaning "that which struck the eye, that which one looks at", prosopon also
named general appearance, whole man, semblant, demeanour, death-mask,
actor's mask, face, the top surface of an object, the surface facing the observer,
etc. Ab initio, prosopon referred to the image of God but subsequently it also
referred to godly demeanour, particularly to that aspect of god that engaged

51
humankind.. In the Old Testament, 'looking at the face of God' means also
f
'bein in the presence of god'. Looking at God brought a reward to the pure in
soul l but caused the death of the impure. IS To avoid the latter, some religions
prohibited the construction of images of God. 19 According to all meanings, the
prosopon or face is an end in itself(Brown, 1975), not a means for anything (e.g.
a semaphore).

PHYSIOGNOMY IN THE ARISTOTELIAN CORPUS

Already present in Mesopotamian culture, the discipline of 'physiognomy'

shared the same logic and semantic structures as the Babylonian divinatory
practices (Manetti, 1993). Two millennia later, the author of Physiognomonika20
examined the data on which the physiognomic signs were based:
movements, gestures of the body, colour, characteristic facial expression, the
growth of the hair, the smoothness of the skin, the voice, conditions of the flesh, the
parts of the body, and the built of the body as a whole.(p. 1239).

and identified the three inferential methods used by these practices: a) features
regarded as specific to certain animals, so that
if a man resembles such and such a genus in body he will resemble
it also in soul (p. 1236)
b) features observed in other races of men; and c) changes observed in the
individual himself:
facial expressions [could be] observed to accompany different conditions of man
such as anger, fear, erotic excitement and all the other passions (p. 1236).

To show their weakness, the author of Physiognomonika provides

counterexamples against all three forms of inference but then writes:
soul and body, as it seems to me, are affected sympathetically by one another: on
the one hand, an alteration of the state of the soul produces an alteration in the form
of the body, and contrariwise, an alteration in bodily form produces an alteration
ion the state of the soul (p. I242). (my italics) 21

About six hundred years later, this interactional assumption was satirized by
Juvenal(1991)
Men's faces are not to be trusted; does not every street abound in gloomy-visaged
debauchees? And do you rebuke foul practices, when you are yourself the most
notorious delving-ground among Socratic reprobates? A hairy body, and arms stiff
with bristles, give promise of a manly soul: but sleek are your buttocks when the
grinning doctor cuts into the swollen piles .. Men of your kidney talk little; they
glory in taciturnity, and cut their hair shorter than their eyebrows .. Peribomius
himself is more open and more honest; his face, his walk, betray his distemper, and
I charge Destiny with his failings (Satire 2)11

PHYSIOGNOMY, PATHOGNOMY AND FACE RECOGNITION

The difference between physiognomy and pathognomy (the sciences of the face
and of the expression of passions or emotions, respectively) was based on an

52
ancient distinction between enduring and transient states of the body (Lemoine,
1865); it lives in the trait/state and anatomy/physiology dichotomies of today.
Physiognomy concerned the correlation between static features and mental traits:
The art of jUdging character and disposition from the features of the face or the
form and lineaments of the body generally (OED, 1992).

The view that the face reflects the soul is ancient too:

Wisdom brightens a man's face and changes its hard appearance (Ecclesiastes, 8).

Pathognomy deals with the dynamic aspects of the face:

The knowledge or study of the passions or emotions, or of the signs or expressions
of them (OED, 1992).

Called 'mimic' after 1800, pathognomy was absorbed into the discipline that
studied the 'expression of emotions in man and animals' (Cuyer, 1906; Darwin,
1904; Dromard, 1909; Mantegazza, 1890; Schack, 1887) and now is known as
the science of 'gestures' .. Perhaps because it dealt with 'indelible' traits, i.e. the
character of man, throughout history physiognomy has been considered as more
important than pathognomy (Cortes, 1741; Piderit, 1867; Sarr6, 1942; Schack,
1887). Be that as it may, physiognomy itself is at the moment going through a
bad patch and has been dismissed, together with phrenology, as a 'pseudo-
science' .23
The semaphore model requires that the signallee is able to decipher the
message. The science of face recognition has grown out of the belief that
evolution has also provided the signallee with the appropriate receptors (Bruce &
Young, 1998; Young, 1998). Gestures are believed to enhance the signal/noise
ratio of the message. This has encouraged the reappearance of pathognomy
under a different name (Allport & Vernon, 1933; BUhler, 1968; Ekman, Friesen
& Ellsworth, 1972; Nespoulous, Perron, & Lecours, 1986; Russell & Fernandez-
Dols, 1997; Wolff, 1951).

THE FACE AND PORTRAITURE

Egyptian pharaohs sought to strengthen their power by erecting representations

of themselves everywhere; in the form of portraits and sculptures. Their
prosopon thus signified power and authority (Brilliant, 1997). Egyptian artists
were faced with the problem of 'appropriate' representation: i.e. what were they
to portrait? The real individual (warts and all)? An idealized image or
prototypical representation of power and position? The early Egyptian dynasties
chose idealized images. Represented in profile with no person or object
obscuring their image, all Pharaohs looked the same. Official representations
contrast with the realistic portraits seen on their sarcophagi. This means that
Egyptian artists did possess adequate representational skills.
'Appropriate representation' posed a similar problem during the Renaissance
(Courtine & Haroche, 1988) when the marked increase in travel and cultural
exchange (e.g. arranged marriages) created the need for keepsake portraits
(Ciappelli & Rubin, 2000; Pigeaud, 1995). In this regard, Woodall (1997) has

53
rightly stated
"Because of these crucial functions, portraiture had to be theorized as unmediated
realism (p .. 3). (my italics).

'Unmediated realism' concerns the issue of likeness, its definition and criteria,
and its relationship to any present or future art or science of the face. How is
'likeness' to be represented and defined? Should the yardstick be 'realistic'
likeness as apparently achieved by a photographic mug shot? Or is it the case
that such exact likeness is impossible for even the action of the photographic
camera is biased in subtle ways? (Hamilton & Hargreaves, 200 I). In addition to
technical elements such as light, focus, etc. is it not the case that the good
photographer will 'add' his own perceptions of the subject's personality, etc.?
This impossibility to achieve 'objectivity' in the representation of a face should
be considered as relevant to the 'sciences of the face'. For one wou Id have
thought that the face scientist is confronted by similar questions i.e. what
components of the face need more attention, enhancement, etc? Which are more
representative of the individual and his emotions? Are there national faces (say
English, as argued by Piper (1957)? Which variables are to be privileged here as
important albeit hidden correlates for a scientific study of the face?

THE FACE AND THE SELF

The concept of psychological self as a 'private space' is a contribution of the

Lutheran Reformation of the early 16th century (Berrios, 2002). Little work has
been done on the impact that the private self had on the tenets sustaining the
semaphore function of the face. For if according to the new Lutheran notion
man is to become the manager of his private images and emotions, then, will this
not affect the 'spontaneity' of the signalling function on which the semaphore
model relies? Would this mediation by the self actually vitiate the information
signalled by the face? Could the old tenet of the semaphore model (present in
Darwin and Galton) hold, i.e. that the will of the 'person' or 'self be bypassed
by the strength of the correlation between face and emotions?
The ancient view that the face 'represents something' which cannot be
hidden and is persistently identifiable beautifully reappeared in the 'mug shot'
method developed by Bertillon. 24 Adopted in 1890 by the French gendarmerie,
and later by all police forces in the world, the mug shots combine a frontal and
profile pictures of the face of the alleged crim inal with a written 'description'.
Although the view by Lombroso (1895) and Galton (1883) that these pictures
represent a 'criminal class' is no longer officially accepted, the fact remains that
they are used as databases by the police. 25 In other words, it cannot be argued
that such databases are the same as a collection of pictures of people randomly
selected from the community. It is also the case that the 'mug shot method' does
more than collecting information. It is a ritual of degradation and social control.
At the beginning of the 20th century, the denial by Behaviourism of the
existence of a mediating or interfering self and/or an internal world made life
easy for all concerned. Problems started again when subjectivity returned with a
vengeance after the Second World War and philosophers like Ryle (1949) and
Wittgenstein (Saunders & Henze, 1967) argued that the existence of private inner

54
spaces and communication codes used only by the one individual were a logical
impossibility.

THE FACE AND DISEASE

The ancient medical notion of 'facies' relates to both physiognomy and
pathognomy and is another expression of the semaphore model. 'Facies' is
defined as the structural and gestural changes characteristic of a specific disease
or of the stage of a disease. For centuries physicians were taught to recognize
the facies cancerosa, cholerica, hysterica, leontina, leprosa, rubra, tetanica,
tubercularis, and uterine, each characteristic of a specific disease (Power &
Sedwick, 1882). An example of a 'generic' facies was the so-called
Hippocratica, cadaverica or tortualis, that which named the facial expression that
is assumed immediately before death. The concept of facies was based less on a
theory of signs than on physiognomic principles. In other words, the issue was
not that the face showed an actual sign of the disease (e.g. that the facies of
jaundice was predictably yellow) but that it showed gestures and features
putatively relating to the theoretical account of the disease (e.g. the 'green' face
of young ladies with chlorosis, a disorder which has disappeared from medical
nosology after the First World War).
Partly as a reflection of the work of Lavater (1789) and Spurzheim (1815),
and of the development of a new philosophy of signs and symptoms, there was at
the beginning of the 19th century an important debate on meaning and
mechanisms of the 'facies'; and a number of medical dissertations were defended
(mainly) in French and German Universities. Reviewing the clinical value of
these works, Widal (1877) wrote:
If as Virey has said, the healthy man is all in his face, the one diseased is all in his
facies (p.132).

Widal (1877) went on to explain why the face was a semaphore:

nature provided man with a set of organs which can signal the various components
of expression such as the vascular system which deals with colour and temperature
and the nervous system that deals with muscle changes .. In the olden days people
simply guessed the meaning of the facial changes but during the 19th century this
study had became part of semiology, and with the help of the laboratory, chemistry,
measurements and other scientific accoutrements, it was a veritable science (p.
133).

As mentioned above, the new philosophy of signs required by the clinico-

pathological model of disease was an important factor in the revision of ideas
about the face during the tum of the 18th century. Such revisions included not
only phenomenic but also anatomical and physiological studies. An example is
the remarkable medical thesis by Francoise Cabuchet (1801), dedicated to
Bichat, and as its title stated, a veritable "Essai sur I'expression de la face dans
I'etat de sante et de maladie". The material collected in this thesis was to be
used by Landre-Beauvais (1813) in his classical book on the new signs and
symptoms of medicine, and constituted the central reference for his classical
entry on the face in the Panckoucke dictionary (Landre-Beauvais, 1819).
Cabouchet (1801) divided up his thesis on the face into an anatomical and

55
physiological section including the effect of age, race and temperament; a section
on the expression of the face in states of health both habitual and under the effect
of the passions; and a third section dedicated to the face in disease including
changes in physical appearance (colour, size, shape) and the use of the facies in
prognosis. Cabouchet only refers to Camper's work on beaut/6 but Landre-
Beauvais (\ 819) makes use of Camper's angle to differentiate faces according to
race and age.
Current clinicians only occasionally will refer to the 'facies' of Down's
syndrome, Williams's syndrome, anorexia nervosa, dystrophia myotonica,
Parkinson's disease, etc. It can be suggested in this regard that 'facies' is but a
final common pathway for the expression of genetically controlled traits, and
primary and secondary signs and symptoms (Gilman, 1988). During the middle
the 19th century, interest in the concept of facies and of facial studies in general
was encouraged by the development of degeneration theory (Morel, 1857).
Often affecting the face, the 'stigmata of degeneration' were to become an
important area of research, particularly in relation to mental handicap and
disorder (Pick, 1898; Talbot, 1898). The sensitivity and specificity of the
various types of facies described in medicine have never been studied.

FACE AND MENTAL ILLNESS

Medical practitioners have also used the face and its expressions in the diagnosis
of mental disorders (Gilman, 1982). Encouraged by changes in portraiture styles
and the development of daguerreotype, the practice returned during the 19th
century.27 Inspired by Lavater and others, Pinel, Esquirol and Georget became
interested in physiognomy and in the portraiture of madness. For example, Pinel
(\ 809) included lithographs in his treatise of insanity; Esquirol (1838) recruited
Georges-Franyoise Gabriel (Adhemar, 1961 in Morel & Quetel, 1984) and
Ambroise Tardieu (Morel & Quete1, 1984); and Georget sought the help of
Theodore Gericault (Cadinouche, 1929)?S A similar trend was detectable in
Germany, for example, in the work of Wilhelm Kaulbach (Hofmann, 1977 in
Morel & Quetel, 1984).
Although putatively portraying specific mental disorders (mania,
melancholia, etc.) all that Gabriel, Tardieu and Gericault can depict is faces and
bodies expressing anger, sadness or impassiveness. In fact, were it not for the
straitjackets and exaggerations of expression, nothing in their pictures can
differentiate anger in the mad and anger in the normal. For example, what
information on 'gambling' is there in Gericault's 'woman with gambling
mania'? Is it her facies or bodily attitude that is pathognomonic of such a
'disorder'? In portraits of subjects with 'furious mania', is there anything else
other than exaggerated anger being depicted? Why are such persons being called
manic? Is their fury different from that expressed by a furious non-mad person?
The answer is that then (as now) much of what is said about the 'informational
value' of the images of the face in psychiatry is just in the eye of the beholder. It
is the result of a straight phenomenon of projection or completion of what is in
the canvas by a mental construct.. Influenced by physiognomy and pathognomy,
19th century alienists assumed that 'realistic' representations of the insane would

56
 actually contain overt or even hidden information about madness. Similar
assumptions and expectations can be seen in the modern equivalent of
neuroimaging where regions of interest, hot spots, etc. are gradually becoming
images representing or hiding information on 'correlations' between two proxy
variables-one representing a neural event and the other a subjective one
(Logothetis, Pauls, Augath, Trinath, & Oelterman, 2001). Indeed, neuroimagers
soon become unaware of the phenomenon of 'completion' and of how much of
what they claim to see is in the eye of the beholder.
By the 1850s, lithographs were replaced by photographs (Clarke, 1997), and
soon enough sepia pictures of faces of the mad became de rigueur in textbooks
of psychiatry. Early photographic techniques required that subjects remained
motionless for a while. This explains why patients with depressive stupor,
catatonia, or hysterical catalepsy feature in textbooks far more often than those
with agitated depression or mania. This may have led some (wrongly) to infer
that catatonia or catalepsy was more common in the past. It could be speculated
that the representational bias caused by the limitation in photographic technology
may have also caused the slants in research interest that can be noticed in
European psychiatry between the 1880s and the First World War.
As with neuroimaging nowadays, journals specialized in carrying pictures of
the face, body and brains of the insane appeared during the late 19th century and
were called Iconographies, the most famous being the one published at la
Salpetriere in Paris. Then (as now) it was fervently believed that the information
conveyed by those pictures (correlating something and madness) constituted a
major advance in the science of the brain ..
By the same token, around the same time it became almost a medico-legal
obligation for European mental asylums to keep mug shots of the face of the
patient in his/her casenotes; and add later post-mortem images .. Indeed this
practice only stopped just before the Second World War. Little work has ever
been done to find out what scientific and clinical information was ever provided
to alienists by this ritualized representation of the mad. It can only be concluded
that these images, together with verbal descriptions of the behaviour of the mad,
post-mortem studies of their brains (up to the 1880s deformed by rotting as
preservation techniques were not yet being used), asylum architecture, etc. are
just part of a complex representational language of madness which is still extant
(under different guises) during the 21 st century (Berrios, 1996).

CONCLUSIONS
The face is not a 'natural kind' but a 'cultural construct' and hence its study
requires a discipline that may borrow equally from biology, theology, poetry,
history, philosophy, portraiture, and aesthetics. The first task of this new
hermeneutic discipline must be the development of a conceptual apparatus that
regains and updates the old ancient proto-notion of prosopon. According to the
pro sop on the face was not a fixed concept nor a means to an end. It could not
thus have been created by God or nature to be just a semaphore. Physiognomy,
pathognomy, and the new sciences of the face remain wedded to the 'semaphore
model' and this has made them the poorer. An alliance with history and other

57
human and social sciences might provide these disciplines with fresher insights.
Central to all their endeavours should be the view that the face is a construct.

AFFILIA nON

Consultant Neuropsychiatrist
Department 0/ Psychiatry, University o/Cambridge,
Addenbrooke's Hospital, Box 189,
Hills Ret, Cambridge, CB2 2QQ;
Tel: 01223336965; Fax 01223 336968;
Email: gebll@cam.ac.uk

NOTES

lin this regard, all scientific claims and science itself must considered as social practices.
Bewitchment by the present transforms history in sycophancy.
2Plants, boulders, and horses are 'natural kinds'. The origin of such objects is considered as unrelated
to human activity; they were on earth before humankind appeared and hopefully will outlive it.
'Natural kinds' are contrasted with artefacts, i.e. with man-made things or constructs. Artefacts
can be solid (e.g. a chair) or abstract (e.g. a virtue) (Hale, 1987). Because they are historical and
contextual, the meaning and autonomy of artefacts is particularly vulnerable to challenge. For
many objects, however, it remains unclear whether they are natural kinds or constructs, and there
is no doubt that hybrids abound. This is particularly the case in regards to the mind and body of
man. Fashions such as the 'naturalization of the mind' movement (Churchland, 1995; Dretske.
1995; Kornblith, 1994) endeavour to transform constructs into natural kinds. It is also incumbent
upon history to scrutinize such activities.
'Concepts can be blurred, vague or 'under-specified' due to grammatical, syntactic or definitional
reasons. Definitions of the face include variable boundaries and components. Underspecification
is well discussed by Deemter & Peters, 1996.
·An 'operational definition' defines an entity in terms of the laboratory operations or procedures
required to illustrate its 'existence' Bridgman (1927) stated: "if we have more than one set of
operations, we have more than one concept, and strictly there should be a separate name to
correspond to each different set of operations" (p I 0). Developed out the verification ism of the
Vienna Circle, this theory of meaning does not work in physics let alone in psychology.
~ The view that 'nature' has defined the human face cannot not be sustained for no where in nature it
is written that the neck, upper part of the chest, forehead, etc. should or should not be part of the
face. To say that nowadays everyone agrees what should be included in the face does not make it
a 'natural kind' but reflects the fact that agreement has been reached. For an excellent treatment
of the face as a cultural construct see Hartley (200 I).
('It can of course be argued that 'social' concepts such as social interaction, recognition, etc. can be
can naturalized, i.e. subject to evolutionary explanation (e.g. Barkow, et ai, 1992). It is not unfair
to say that whilst the application of evolutionary theory to biological structures has been rich in
facts, the opposite seems to be the case in regards to behaviour in general and mental illness in
particular.
7In the Old Testament, the commonest usage of 'face' (prosopon: 'that what strikes the eye")
concerns the dialogue between God and man (p585, Brown, 1975)
K A semaphore is "an apparatus for making signals, consisting of an upright post with one or more

arms moving in a vertical plane" (OED, 1992).

"The terms polyphonic and polyphony ('many voiced') refer to a method according to which musical
texture is formed by the interweaving of melodic lines, different rhythms and different melodies
and in which the lines nonetheless sound together harmonically. Bakhtin (1984) wrote: "It is
quite possible to imagine and postulate a unified truth that requires a plurality of
consciousnesses, one that cannot in principle be fitted into the bounds of a single consciousness,

58
 one that is, so to speak, by its very nature full of event potential and is born at a point of contact
among various consciousnesses."
IOThis scenario is just one step further from the 'situationist view' according to which gestures are
part of a social act (see Femandez-Dols, 1999).
liOn the history of the interaction between character personality, constitution and temperament see
Berrios (1996); Ricord (2000); Ribery (1902).
12The definition of true 'likeness' in portraiture and photography depends upon the resolution of two
preliminary problems: the meaning of object similarity and its perceptual ascertainment; in other
words, what does it mean for two objects (or images) to look 'the same'? Who is the best judge
of this sameness? The OED (1992) defines likeness as: "The quality or fact of being like;
resemblance, similarity; an instance of this ... The representation of an object; a copy,
counterpart, image, portrait." By Iike ness it is meant both the abstract quality that an object gains
when it is mapped in a particular relationship to another and the object itself. A picture (or
likeness) of a face is 'like' the real face for it is: a) mappable onto it or b) because there will be
consensual agreement between observers. The former (definable as 'objective') may be
implemented by a machine; the latter (definable as 'subjective'), remains the gold standard for
the ascertainment oflikeness.
11 In terms of representation and portraiture, it would seem that Holbein and others in his time upheld
a Platonic view. For example, in his portraits of Henry VIII the painter not only imitated what he
saw (to create a realistic portrait) but ' completed' the picture by adding from a Platonic ideal.
One could ask whether this 'completion' element is also active in the science of the face? Do
scientists believe that what is seen is not sufficient and that special techniques must be developed
to get at hidden information carried by the face? During the 19th century, Bell (1844) saw a
connection between the anatomy and philosophy of expression and the fine arts.
I'This refers to the view that much ofthe psychological activity that takes place in the human self has
little to do with communication. It is hypothesized here that human beings spend a great deal of
their time rehearsing old experiences and testing future actions. All this non-communicative
activity involves a great deal of role playing, ideas, emotions and inner speech. According to the
'corollary projection system' hypothesis, this activity may be reflected in the face which would
act as an screen for inner projections. Much of these 'gestures' would thus be incomplete or
manque and plainly unintelligible and would not correlated with anything specific.
I'Such predominance could be explained by the claim that semaphore view is more scientific or
'truthful' or that it is more ' instrumental', more able to ' generate wealth', or that it is useful to
methods of social control.
l"As it had been when the concept was born in the thick of Greek thought, the heart provided an
important address to the passions until the 17th century (Erickson, 1997).
17E.g. "Restore us, 0 God; make your face shine upon us, that we may be saved" (Psalms, 80).
10 "God himself or his glory which could not be seen by Moses (or by any man) or he would die" (Ex
33:20).
1"ln the Koran (Prophets) this prohibition is clear: "[21.57] And, by Allah! I will certainly do
something against your idols after you go away, turning back. [21.58] So he broke them into
pieces, except the chief of them, that haply they may return to it."
2('In earlier times, this treatise was attributed to Aristotle. In proposing that body and soul influence
each other (i.e. an interaction), its unknown writer goes beyond the correlational structure of
physiognomy.
21The author does not say how this might strengthen the old inferential processes. Later day
physiognomists never took up the interactional assumption contained in Physiognomonika and to
this day have limited themselves to follow an unidirectional model of causation.
22Examples can be multiplied. For an analysis of the Hellenistic period see: Caro-Baroja, 1988. There
is also an interesting dissociation between face and soul (and its sins) in the famous Greek
palindrome "wash my transgressions, not only my face" which appears as a legend in many
church fonts (p796, Brewer, 1993).
21The epithet 'pseudo-science' is one of the rhetorical devices used by modern science. By
exaggerating the badness and falsehood of older disciplines it enhances, by contrast, its own
purity and truthfulness. A moment's thought, however, will show that the old assumptions of
physiognomy and pathognomy remain alive in the new sciences of the face and that what has
changed is the type of technical instrumentation and numerical management in use. There is, of
course, nothing wrong with the fact that old and new narratives are continuous.

59
24Alphonse Bertillon (1853-1914) is regarded as the 'inventor' of anthropometric portraiture. After a
chequered career which took him even to Scotland to teach German, in 1880 Bertillon settled
down as an employee of the Prefecture de Police in Paris. Never a '!lic', his main contributions
remain the invention of the "portrait parle"; and after 1902, the introduction of finger printing. A
child of 19th century French positivism, Bertillon believed that nature could be fully captured
and/or represented. On occasions, this lead to unpalatable conclusions: for example, he wrote a
condemnatory report of Captain Dreyfus based on the latter' hand-writing! He also sought to
correlate penile size with intensity of homosexual behaviour and clitoral size with tendency to
prostitution (Guerrin & Roux, 2000). This confusing correlations with causes & reasons is still
present in current research into testosterone in homosexual behaviour.
25Given that such databases are not 'purged' according to trial results, the point is that it is
theoretically possible that they might contain very few 'criminals' or none.
2"Pierre Camper (1722-1789) was a Dutch savant and politician who inter alia wrote a thesis on the
way to differentiate faces of men from different countries and ages. 'Camper's angle' was for
decades central to the work of physical anthropologists and was used to even to draw
conclusions on levels of intelligence (e.g. the nearer the angle was to 90 degrees the brighter the
person was) (Montanier, 1870).
27Representation of the insane can certainly be found before the 19th century such as the work of
Bosch, Brueghel, Weydmans, DUrher, and later Hogarth. As Morel & Quetel (1984) have
noticed, however, these are representations of 'metaphorical madness' rather than of real
patients. For 19th century views see Morrison (1838).
2HGericault was a Romantic French painter (1791-1824) whose work, mainly related to military and
political motifs, shows an energetic and vivacious brush work and much introspective insight
into emotional expression and movement. He was an important influence at the beginning of the
19th century. During a visit to England, Gericault fell from a horse and died without completing
the Georget assignment.

REFERENCES
Allport, G.W., & Vernon, P.E. (1933) Studies in expressive movement. New York: McMillan.
Aristotle. (1984). The complete works. The revised oxford translation. Edited by Jonathan Barnes. 2
vols. Princeton: Princeton University Press.
Bakhtin, M. (1984) Problems of Dostoevsky's poetics. Ed. and trans. Caryl Emerson. Minneapolis:
University of Minnesota Press.
Barkow, J.H., Cosmides, L. & Tooby, J. (\ 992). The adapted mind. Oxford: Oxford University Press.
Bell, e. (1844). The anatomy and philosophy of expression as connected with the fine arts. Third
Edition. London: John Murray.
Berrios G.E. (1996) The history of mental symptoms. Descriptive psychopathology since the 19th
century. Cambridge: Cambridge University Press.
Berrios, G.E., & Markova, I.S. (2000). The self and psychiatry: A conceptual history. In A. David., &
T. Kircher (Eds.), The self and schizophrenia. Cambridge: Cambridge University Press.
Braund, S.M., & Gill, e. (2000). The passions in Roman thought and literature. Cambridge:
Cambridge University Press.
Brewer's Dictionary (1993). The Wordsworth dictionary of phrase & fable. London: Wordsworth
Editions.
Bridgman, P.W. (1927). The logic of modern physics. New York: The McMillan Company.
Brilliant, R. (1997). Portraiture. London: Reaktion Books.
Brown, e. (1975). The new international dictionary of new testament theology. 4 Vols. Exeter: The
Paternoster Press.
Bruc,e V., & Young, A. (1998). In the eye of the beholder. The science offace recognition. Oxford:
Oxford University Press.
Brunschwig, J., & Nussbaum, M.e. (1993). Passions and perceptions. Cambridge: Cambridge
University Press.
BUhler, K. (1968). Ausdruckstheorie. Das system an der geschichte aufgezeigt. Stuttgart: Fischer.
Cabuchet, F. (180 I). Essai sur I'expression de la face dans I'etat de sante et de maladie. Paris:
Brosson.
Cadinouche, H. (1929). La medecine dans I'oeuvre de Gericault. these de medecine, Paris:
Caro-Baroja, J. (1988). Historia de lafisiognomica. Madrid: Edicioneslstmo.

60
Churchland, P.M. (1995). The enigma of reason, the seat of the soul. Cambridge: MIT Press.
Ciappelli, G., & Rubin, P.L. (2000). Art, memory, and family in Renaissance Florence. Cambridge:
Cambridge University Press.
Clark, G. (1997). The photograph. Oxford: Oxford University Press.
Cortes, G. (1741). Fisionomia y varios secretos de naturaleza. Barcelona: Giralt.
Courtine, J-J., & Haroche ,C. (1988). Histoire du visage XVIe-debut XIXe siecle. Paris, Rivages.
Cuyer, E. (1906). La mimica. Translated by A. Miquis. Madrid: Jorro.
Damasio, A.R. (1994). Descartes' error. New York: Putnam.
Darwin, C. (1904). The expression of the emotions in man and animals. London: John Murray.
Davies, P.S. (2001). Norms of nature. Cambridge:The MIT Press.
Deemter K. van. & Peters, S. (1996). Semantic ambiguity and underspecification. Stanford: CSLI
Publications.
Descartes, R. (1989). The passions of the soul. Indianapolis: Hackett.
Despret, V. (1999), Ces emotions qui nousfabriquent. Ethnopsychologie de ['authentici/e. Paris: Les
Empecheurs de Penser en Rond.
Dretske, F. (1995). Naturalizing the mind. Cambridge: MIT Press.
Duchenne, G.B. (1862). Mecanisme de la physionomie humaine ou analyse e!ectro-physio!ogique de
['expression des passions applicable a la pratique des arts plastiques. Paris: Renouard.
Dromard, G. (1909). La mimique chez les alienes. Paris: Alcan.
Ekman, P. (1998). Preface to the third edition. In C. Darwin. The expression of the emotions in man
and animals( pp xiii-xxxvi). London: Harper Collins.
Ekman, P., Friesen, W.V., & Ellsworth ,Po (1972). Emotion and the human face. New York:
Pergamon Press.
Erickson, R.A. (1997). The Language of the heart 1600-1750. Philadelphia: University of
Pennsylvania Press.
Fernandez-Dols, J.M. (1999). Facial expression and emotion: A situationist view. In R. Philippot.,
R.S. Feldman, & EJ. Coats (Eds), The social context of non-verbal behaviour (pp242-261).
Cambridge: Cambridge University Press
Forgas, J.P., Williams, K.D., & Wheeler, L. (Eds). (2001). The social mind. Cognitive and
motivational aspects of interpersonal behaviour. Cambridge: Cambridge University Press.
Foster, 1. (1991). Theimmaterial self. London: Routledge.
Fridlund, AJ. (1994). Human facial expression. An evolutionary view. San Diago, Academic Press.
Hamilton, P. & Hargreaves, R. (2001). The beautiful and the damned. The creation of identity in 19th
century photography. London: Lund Humphries.
Hartley, L. (2001). Physiognomy and the meaning of expression in nineteenth century culture.
Cambridge: Cambridge University Press.
Henneberg, M., Simpson, E., & Stephan, C. (2002). Human face in biological anthropology:
Craniometry, evolution, and forensic identification. In M. Katsikitis (Ed), The human face:
Measurement and Meaning.(pp. 29-48). Dordrecht: Kluwer Academic Publishers.
Galton, F. (1883). Inquiries into human faculty and its development. London, McMillan.
Gardiner, H.M., Metcalf, R.C., & Beebe-Center, 1.G. (1937). Feeling and emotion. A history of
theories. New York: American Book Company.
Gilman, S.L. (1982). Seeing the insane. New York: John Wiley.
Gilman, S.L. (1988). Disease and representation. Images of illness from madness to AIDS. Ithaca:
Cornell University Press.
Guerrin, M. & Roux E. de. (2000). Alphonse Bertillon, inventeur du portrait anthropometrique. Le
Monde, 3rd February 2000, p27.
Hale, B. (1987). Abstract objects. Oxford: Blackwell.
James, S. (1997). Passion and action. The emotions in seventeenth century philosophy. Oxford:
Oxford, University Press.
Juvenal, (1991). Satires. With an English translation by G.G. Ramsay. Cambridge: 'tarvard
University Press.
Kenny, A. (1963). Action, emotion and will. London: Routledge & Kegan Paul.
Kornblith, H. (1994). Naturalizing epistemology. Cambridge: MIT Press.
Landre-Beauvais, AJ. (1813). Semeiotique, ou traite des signes des maladies. Second Edition. Paris:
Brosson.
Landre-Beauvais, AJ. (1819). Face. In Adelon (no initial) (Ed), Dictionaire des sciences medicales.
(pp. 367-384). Paris: Panckoucke ..
Lange,. F. (1942). EI Lenguage del rostro. Translated by F. Fernandez, Barcelona: Miracle.

61
Lavater, J.e. (1789). Essays on physiognomy for the promotion of the knowledge and the love of
mankind. 3 Vols, Translated by T Holcroft. London: Robinson.
Lemoine, A. (1865). De la physionomie et de la parole. Paris: Bailliere.
Levi, A. (1964). French moralists. The theory of the passions 1585-1649. Oxford: Clarendon Press.
Logothetis N.K., Pauls J., Augath M., Trinath, T., & Oelterman, A. (2001) Neurophysiological
investigation of the basis of the fMRI signal. Nature, 412,: 150-157.
Lombroso, C. (1895). L 'homme criminel. atlas. Paris: Alcan.
McCosh, 1. (1880). The emotions. New York: Charles Scribner's Sons.
Manetti, G. (1993). Theories of the sign in classical antiquity. Bloomington: Indiana University
press.
Mantegazza, P. (1890). Physiognomy and expression. London: Walter Scott.
Meyer, M. (2000). Philosophy and the passions. Pennsylvania: The Pennsylvania State University
Press.
Montanier, H. (1870). Pierre Camper. In A. Dechambre & L. Lereboullet (Eds), Dictionnaire
encyc\opedique des sciences medicales (pp. 81-84). Paris: Masson.
Morel, B.A.(l857). Traite des degenerescences physiques intellectuelles et morales de /'espece
humaine. Paris: Bailliere.
Morel, P. & Quetel, e. (1984). Reflexions sur les representations iconographiques de I'aliene au
XIXeme siec\e. Art et Folie (Nantes), 6, 155-171.
Morrison, A. (1838). The physiognomy of mental illness. London: Odell.
Naya, E. (1998). Rabelais. Une anthropologie humaniste de passions. Paris: Presses Universitaires de
France.
Nespoulous, J-L, Perron, P., & Lecours, A.R. (1986) The biological foundations of gestures. Motor
and semiotic aspects. London: Lawrence Erlbaum.
Pick, D. (1989). Faces of degeneration. Cambridge: Cambridge University Press.
Piderit, T. (1867) Wissenschaftliches system der mimik und physiognomik klingenberg, Detmold.
Pigeaud, J. (1995). L 'art et Ie vivant. Paris: Gallimard.
Piper, D. (1957). The Englishface. London: Thames and Hudson.
Plessner, H. (1941). Lachen und weinen. Berne: Francke.
Power, H. & Sedwick, L.W. (1882) The new Sydenham society's lexicon afmedicine and the allied
sciences. 5 Vols. London: The New Sydenham Society.
Ribery, e. (1902). Essai de classification naturelle des caracteres. Paris: A\can.
Ricord, M. (2000), Les caracteres de la bruyere ou les exercices de I 'esprit. Paris: Presses
Universitaires de France.
Russell, J.A., & Femandez-Dols, J.M. (1997). The psychology of facial expression. Cambridge:
Cambridge University Press.
Ryle, G. (1949). The concept of mind. London: Hutchinson.
Sarr6, R. (1942). Fisiognomica y patognomica. Introduction to Lange F. EI lenguaje del rostro.
Translated by F. Fernandez. Barcelona, Luis Miracle.
Saunders, J.T. & Henze, D.F. (1967). The private-language problem. New York: Random House.
Schack, S. (1887) .La physionomie chez l'homme et chez les animaux. Paris: Bailliere.
Sorabji, R (2000). Emotions and peace of mind. From Stoic ggitation to christian temptation. Oxford:
Oxford University Press.
Spurzheim, J.G. (1815). Physiognomical system of Gall and Spurzheim. London: Baldwin, Cradock
& Joy.
Talbot, E.S. (1898). Degeneracy, its causes, signs and results. London: Walter Scott.
Vila, A.C. (1998). Enlightenment and pathology. Baltimore: Johns Hopkins University Press.
Virey, (no initial) (1819). Passion. In Adelon (no initial) (Ed), Dictionaire des sciences medicales
(pp. 411-490). Paris: Panckoucke.
Wechsler, J. (1982). A human comedy. Physiognomy and caricature in 19th century Paris. London:
Thames and Hudson.
Widal, V. (1877). Facies. In A. Dechambre & L. Lereboullet L. (Eds), Dictionnaire encyclopedique
des sciences medicales (pp. \32-148). Paris, Masson.
Wierzbicka, A. (1999). Emotions across languages and cultures. Cambridge: Cambridge University
Press.
Wolff, C. (1948) .. Psychology of gesture. London: Methuen.
Woodall, J. (1997). Introduction. Facing the subject. In J. Woodall (Ed), Portraiture. Facing the
subject (pp. 1-25). Manchester: Manchester University Press.
Young, A.W. (1998). Face and mind. Oxford: Oxford University Press.

62
 JASON WARREN AND PHILIP THOMPSON

CHAPTER 4

NEUROLOGY OF HUMAN FACIAL EXPRESSION

There's no art / To find the mind's construction in the face

Macbeth I. iv. 11

INTRODUCTION

The age-old human preoccupation with the face and its magical properties is evident
in the masks and totems of preliterate cultures. Tales of mistaken identity and
transformation are common to all mythologies and embodied in such fearsome
creatures as Medusa and the Doppelganger. The scientific study of human facial
expression, or metoposcopy, is at least as old as Hippocrates, who recorded the facial
features which portend death in the Prognostics. Books devoted to the significance
of human physiognomy were popular in the Middle Ages. The anatomical drawings
of Leonardo and Vesalius faithfully and beautifully depict the facial muscles and
skeleton. Renaissance artists understood that the essence of a facial expression is
conveyed by a few critical physiognomic points. The technique of sfumato,
employed to soften the corners of mouth and eyes, is illustrated most famously in the
ambiguous, enigmatic smile of the Mona Lisa. The Flemish school in the sixteenth
century excelled in the representation of both normal and abnormal facial
expressions, most notably Brueghel, who rendered orofacial chorea and dystonia in
his paintings (Marsden, 1976).
In the early nineteenth century, Charles Bell (1844), also an accomplished artist,
attempted to explain facial expression in physiological terms. In comparative
anatomical studies of man and other animals, he identified those muscles subserving
the uniquely human faculty of emotional communication. He believed that these
purely expressive muscles were supplied by a distinctive system of nerves, "the
office of which is to influence the muscles in Respiration, in Speech, and in
Expression" (Cule, 1993). Duchenne de Boulogne was also fascinated by the "laws
which govern the expressions of the human face", which he believed could be
deduced by studying muscular action. To this end, he attempted to reproduce
various expressions using galvanic stimulation of facial muscles (Duchenne de
Boulogne, 1862). He identified 33 expressions, including the famous 'Duchenne
smile', and devised a nomenclature of the facial muscles, published in his
Mecanisme de la Physionomie Humaine of 1862. Duchenne proposed there is a
specific facial muscle for the expression of each emotion. He also observed that the
illusion ofa change in the general expression of the whole face could be produced by
an isolated contraction of one brow, and speculated that this conferred the
physiological advantage of economy of nervous pathways. Duchenne's ideas were
subsequently adopted by Charles Darwin in The Expression of the Emotions in Man
and Animals (Darwin, 1872). In response to Darwin, Israel Waynbaum put forward
a vascular reafference theory of emotional expression, in which the facial muscles
were conceived to act as ligatures regulating cerebral blood flow (and thus subjective
feeling) by their action on facial blood flow (Zajonc, 1985).
The systematic study of the nervous system in humans and animals in the early
twentieth century lead to recognition and classification of neurological diseases, and
a new appreciation of the function of different parts of the brain. These discoveries
also applied to facial expression and the neurological substrates of normal and
pathological emotional expression. The elements of the motor hierarchy responsible
for facial motor patterns were uncovered. These included the corticobulbar tracts
and brainstem (Wilson, 1924), mesencephalon (Meige, 1910), hypothalamus
(Cushing, 1912; Martin, 1950), diencephalon (Davison & Kelman (1939) and
cerebral cortex (Penfield & Jasper, 1954). It is now recognised that all levels of the
motor hierarchy (cortex, basal ganglia, limbic structures and brainstem) contribute to
facial motor patterns and expression.
The mimetic muscles, eyelids and their brainstem controls form a logical starting
point for the present review. The anatomy and physiology of facial movement,
along with disorders of each level of the facial motor hierarchy will be outlined.
This will be followed by a survey of the neurological basis for the emotional content
of facial expression. Finally, the recognition of the facial expressions of self and
others will be examined. This integral determinant of facial motor patterns requires
additional neural processes, flexible enough to permit recognition of facial identity
and affect under widely varying conditions. Accordingly, the mechanisms of face
recognition and other aspects of the hemispheric and higher cortical organisation of

64
face processing in health and as revealed by selective cortical lesions will be
considered in the second part of the review.

FACIAL MOTOR CONTROL IN HEALTH AND DISEASE

Neurology Of The Mimetic Muscles And The Facial Nuclei

The Mimetic Muscles

The 17 paired mimetic (Greek mimesis, imitation) muscles of facial expression are
innervated by the facial nerve and differ from other skeletal muscles in several
respects. They have a distinct embryological origin from the second branchial arch,
corresponding to the gill arches of vertebrate fish, and retain some autonomic
connections via the facial nerve (Rinn, 1984), emphasising their heightened potential
to respond to emotional states. In the course of development they form investments
in the moveable facial skin and fascia, without bony attachments. They are believed
to lack muscle spindles (Shahani, 1970), which may reflect the absence of external
loads, and implies an absence of stretch reflexes.

The Facial Nuclei

The facial nucleus, the motor nucleus of the facial nerve, is located in the caudal
pons in humans, and as in other mammals, has a topographical representation of the
muscles it supplies (Courville, 1966; Rinn, 1984: Welt & Abbs, 1990). The nucleus
is organised into longitudinal motor neuronal columns which innervate particular
muscles (Welt & Abbs, 1990). The ventrolateral part probably supplies mainly the
muscles of the lower face (used in smiling), and the dorsolateral portion the muscles
of the upper face (used in frowning and closing the eyes), whereas the smaller,
medially distributed groups supply the stapedius, external auricular muscles and
platysma.

Peripheral Neuromuscular Facial Weakness

Paralysis of facial muscles due to peripheral neuromuscular disease impairs both
voluntary and emotional facial movement. Bilateral facial weakness (facial diplegia)
produces an impassive facial expression with an inability to smile and absence of the
automatic facial expressions that accompany speech. Common causes include bulbar

65
anterior horn cell disease or bilateral facial nerve palsies, myasthenia gravis and
myopathies. In addition to the difficulties experienced because of weakness of
voluntary eye and mouth closure, patients with facial diplegia may appear sad and
depressed to others because of reduced spontaneous facial movement. An example
of this is evident in the impaired nonverbal interaction between mothers and children
with Moebius syndrome (congenital aplasia of facial and abducens nuclei) in whom
expressive facial movement is absent (Gillberg & Steffenburg, 1989).

Eyelid neuroanatomy and neurophysiology

Elevation of the upper eyelids is maintained by tonic activity in the fatigue resistant
levator palpebrae muscles, innervated by the oculomotor nerve. Cessation of this
activity causes the lids to close and blinking is caused by abrupt inhibition of the
levator palpebrae superioris muscle, accompanied by contraction of the palpebral
portion of orbicularis oculi. The levator palpebrae motor neurone pool lies in the
central caudal subgroup of the oculomotor nuclear complex and projects to both
muscles. The nucleus is intimately related to the dorsal midbrain vertical gaze
centres (rostral interstitial nucleus of the medial longitudinal fasciculus and
interstitial nucleus of Cajal), facilitating lid-globe coordination. The generator of
levator tone probably lies in the midbrain periaqueductal grey matter and is
modulated by a wide range of limbic and reticular influences. Levator tone is closely
related to the level of alertness. The cortical pathways controlling levator palpebrae
have not been completely defined though eyelid elevation follows stimulation of
frontal, temporal and occipital areas (De Renzi, Gentilini, & Bazoli, 1986:
Schmidtke & BUttner-Ennever, 1992). The right hemisphere may be dominant in lid
control (De Renzi et aI., 1986; Schmidtke & BUttner-Ennever, 1992).

Nuclear and infranuclear ptosis

The most common causes of unilateral ptosis or drooping of the eyelid are lesions of
the oculomotor nerve or sympathetic innervation of the tarsal muscles (Horner's
syndrome). Weakness of the levator palpebrae in myopathies and neuromuscular
junction disorders produce bilateral ptosis. Drooping of the eyelids results in a facial
expression often interpreted as indicating drowsiness, fatigue or somnolence,
frequently to the distress of the sufferer.

66
Supranuclear ptosis
Supranuclear ptosis may result from damage to the descending pathways in the
rostral midbrain which normally maintain levator tone. Caplan (1974) concluded
that destructive vascular hemispheric or corticonuclear pathway lesions may produce
bilateral ptosis, sometimes mainly affecting the contralateral lid.

Apraxias of lid movement

The supranuclear control of the eyelids is poorly understood. The term 'apraxia' is
often applied to these conditions, though this is a misnomer, since the executive
motor system is damaged in most cases. However no completely satisfactory
alternative has emerged. Apraxia of eyelid opening is characterised by transient
inability to initiate eyelid opening in the absence of oculomotor dysfunction or
ocular myopathy (Boghen, 1997). It is frequently accompanied by vigorous frontalis
contraction in an effort to lift the eyelids voluntarily. There may be contraction of
the pretarsal portion of orbicularis oculi, often only detected by electromyography.
The proposed mechanism is involuntary levator inhibition and in the presence of
orbicularis contraction, this represents a loss of reciprocal innervation between
antagonist muscles, a common phenomenon in basal ganglia disease. Apraxia of
eyelid opening is observed in idiopathic Parkinson's disease, Huntington's chorea,
Wilson's disease, multiple system atrophy, Steele-Richardson-Olszewski syndrome
and may occur in isolation (Dewey & Maraganore, 1994). It also may result from
stereotaxic lesions in the region of the red nucleus and prerubral field of Forel
(Nashold & Gills, 1967), frontal and non-dominant parietal lobe lesions. The eye
opening difficulties are distinguished from blepharospasm by the absence of forceful
contraction of orbicularis oculi causing eye closure, though occasionally the two
conditions coexist. The converse, 'apraxia of eyelid closure', is characterised by
inability to keep the eyes closed ('closing impersistence') occurs after focal or
diffuse hemispheric damage (De Renzi et aI., 1986; Schmidtke & Biittner-Ennever,
1992). Unilateral hemispheric lesions producing impersistence of eye closure are
almost always right sided. Reflex blinking, lid-eye coordination and eye closure in
sleep remain undisturbed. Neurogenic lid retraction, perhaps the commonest
disorder of supranuclear lid control (Schmidtke & Biittner-Ennever, 1992), is found
in many basal ganglia diseases. This is often accompanied by a staring facial
expression and infrequent blinking, sometimes referred to as a striatal or 'reptilian
stare'. Lid retraction, often combined with a furrowed brow due to frontalis
overactivity, confers the characteristic 'surprised' stare of Steele-Richardson-
Olszewski syndrome. Lid retraction is a prominent feature of the dorsal midbrain or

67
Parinaud's syndrome (Collier's sign). In hydrocephalus, lid retraction may be
combined with tonic down gaze (the 'setting sun' sign).

Blinking
Blink frequency depends on affective, attentional and cognitive factors, as well as
central dopaminergic tone (Karson, 1988). It decreases in schizophrenic patients
receiving neuroleptic medication, may increase in Huntington's disease and
Tourette's syndrome and varies according to the state of dopaminergic stimulation in
Parkinson's disease (Kimber and Thompson, 2000). Blinking is thus an excellent
illustration of the role of the basal ganglia in 'automatic' facial movements.
The blink reflex is a cutaneous reflex mediated by medium-sized trigeminal
afferents in response to stimulation of the cornea and face (Shahani, 1970).
Descending pathways from basal ganglia and cortex probably modulate the blink
reflex via bulbar interneurones (Berardelli, Rothwell, Day & Marsden, 1985).
Accordingly, the blink reflex and blinking are influenced by states of arousal, sleep
and local brainstem lesions (Berardelli et aI., 1985). Abnormalities of the blink
reflex are found in a variety of conditions including cranial dystonias such as
blepharospasm (Berardelli et aI., 1985), Parkinson's disease and Steele-Richardson-
Olszewski syndrome (Valls-Sole, Valderiola, Tolosa & Marti, 1997). Reflex
blinking and facial contortion are components of the auditory startle response. This
reflex response is generated in the caudal brainstem and propagated via the
reticulospinal tracts (Brown, Rothwell, Thompson, Britton, Day, & Marsden, 1991).

Neurology of automatic facial movements

Automatic facial movements such as blinking, frowning with concentration, brow

movements during speech and the facial contortions of sneezing and yawning are
under partial voluntary control but are not associated with strong emotional arousal.

Associated movements
Brief contractions of mimetic muscles, particularly the upper face and brow, add
semantic emphasis and mirror the pitch and stress contours of vocal intonation
during speech ('punctuation', Rinn, 1984). These movements serve a paralinguistic
function (Ekman, 1979) and "give vividness and energy to our spoken words"
(Darwin, 1872). Indeed, the highly developed expressive capabilities of the human
face mirror the evolution of language; although Lees (\988) called facial expression
'the elder brother of speech', they appear to have evolved in tandem. Facial

68
movements also accompany many tasks requiring concentration or fine motor
control. In some diseases of the motor system, such as athetoid cerebral palsy and
primary dystonia, these 'overflow' facial movements are pronounced with facial
grimacing and tongue protrusion. These automatic, associated movements may
result from activation of supplementary motor, temporal and parietal cortices, which
receive widespread inputs from other cortical areas. These cortical areas project via
the striatum and globus pallidus to the red nuclei, and in turn via crossed rubrobulbar
projections to the dorsomedial and dorsolateral portions of the facial nuclei, which
innervate chiefly the upper face (Courville, 1966).

Sneezing and yawning

Sneezing and yawning are examples of complex motor synergies involving eye
closure, mouth opening, variable limb and truncal movements and interruption of the
breathing pattern. They represent the output of polysynaptic brainstem reflexes
coordinating the activity of multiple brainstem motor nuclei, and are modified by
suprabulbar influences. Sneezing may be triggered by a wide range of stimuli,
including local nasal irritation, chilling, light, sexual excitement, menstruation,
epilepsy, and 'resentment and frustration' (Co, 1979). Yawning has the additional
social connotation of fatigue and boredom and may be 'infectious'. Pathological
yawning has been described with intrinsic lesions of the brain stem including multiple
sclerosis (Postert, Pohlau, Meves, Nastos, & Przuntek, 1995), posterior fossa
tumours, focal cerebral lesions, diffuse encephalopathies, autonomic diencephalic
seizures and as a sequel to encephalitis lethargica (Barbizet, 1958).

Involuntary movements and the face

The face is theatre to the entire gamut of involuntary movements, organised at all
levels of the motor hierarchy, from cortex (mannerisms and stereotypies) to
brainstem (facial myoclonus and myokymia). Involuntary facial movements can
mimic or be misinterpreted as facial expressions, with embarrassment and anxiety for
the sufferer.

Tics
Tics are among the commonest involuntary facial movements and overlap with the
spectrum of idiosyncratic, habitual facial motor gestures or mannerisms evident in
the general popUlation and facial stereotypies observed in neuropsychiatric
conditions such as schizophrenia (Rogers & Hymas, 1988). A colourful panoply of

69
facial tics has been described (Lees, 1988). Simple motor tics are brief stereotyped
movements that can be suppressed by an effort of will. The most frequently
encountered facial tics are blinking, winking, unilateral elevation of the corner of the
mouth and nasolabial fold, flaring of the nostril, elevation of the nose often
accompanied by sniffing, and ear wiggling. Tics are typically preceded by a
premonitory inner tension. This sensation increases if the tics are voluntarily
suppressed and subsides with release of the tic.' Tics may be triggered by a variety of
emotional and physical stimuli (anxiety, anger, fatigue) and are characteristically
ameliorated by intense concentration. In contrast, tics may be most conspicuous
when the patient is quietly relaxing, apparently unobserved. Simple tics appear to be
generated by different neural pathways to those that produce voluntary movement.
Lees (1988) regarded tics, obsessions, coprolalia, echo phenomena and stereotypies
as different manifestations of a fundamental defect in the gating of impulses by the
anterior cingulate cortex. In Tourette's syndrome, positron emission tomography
identified nonspecific activation of a motor loop and disease specific
hypometabolism in subcortical, midbrain and limbic structures (Eidelberg, Moeller,
Antonini, Kazumata, Dhawan, Budman, & Feigin, 1997).

Cranial dystonias
The dystonias are syndromes of sustained muscle contractions causing tWlstmg,
repetitive movements and abnormal postures. Blepharospasm and oromandibular
dystonia are the most common expressions of dystonia affecting the facial and
masticatory muscles. Blepharospasm is characterised by prolonged spasms of the
orbicularis oculi muscles producing eye closure and functional blindness. Most
cases of cranial dystonia are idiopathic, however some have been reported in
association with diseases of the cerebral hemispheres, basal ganglia, and brainstem
or induced by drugs (Jankovic, 1988). In oromandibular dystonia, spasms of the
masticatory and facial muscles produce contortion of the face and involuntary jaw
opening and closure, interfering with chewing and talking. Occasionally, jaw
dystonia is unilateral, causing deviation of the jaw to one side. The combination of
blepharospasm with dystonic spasm of masticatory and facial muscles is referred to
as Meige syndrome. 'Brueghel's syndrome' refers to jaw opening dystonia, after the
Brueghel painting of 1558, De Gaper (TheYawning Man) (Marsden, 1976). On
pathological examination of four cases of primary Meige syndrome (Tolosa,
Kulsevsky, & Fahn, 1988), gliotic changes were found in the striatum, substantia
nigra, midbrain tegmentum, locus coeruleus and periaqueductal grey, implicating
these areas in the pathogenesis of the syndrome, as Meige (1910) originally

70
suggested. Involuntary ear wiggling may represent another rare, focal cranial
dystonia (Caviness, Gabellini, Kneebone, Thompson, Lees, & Marsden, 1994).
Pathophysiological and functional imaging studies have also reaffirmed the organic
nature of the cranial dystonias (Berardelli et aI., 1985; Hutchinson, Nakamura,
Moeller, Antonini, Belakhlef, Dhawan, & Eidelberg, 2000).

Oculogyric crises
Oculogyric crises were first described in post encephalitic parkinsonism by Von
Economo (1931). These comprise conjugate upward, tonic deviations of the eyes
frequently accompanied by tics and obsessive-compulsive behaviours. Oculogyric
crises may also occur as part of an acute dystonic reaction to dopamine antagonist
drugs. They probably result from dysfunction in pathways from substantia nigra to
superior colliculi or the midbrain tegmentum and reticular formation, in the vicinity
of the oculomotor nuclei (Devinsky, 1983).

Orofacial dyskinesia
Orofacial or oro-bucco-lingual dyskinesias present a spectrum of orofacial
movements typically chewing, lip smacking and tongue protrusion. Denervation
supersensitivity of striatal dopamine receptors following chronic dopamine receptor
blockade by neuroleptic drugs has been proposed as the mechanism of tardive
orofacial dyskinesia (Jankovic, 1981). Similar movements may also occur
spontaneously in the elderly, and may be more frequent in the edentulous.

Facial myoclonus
Focal cortical myoclonus and epiJepsia partialis continua commonly affect the face,
and may be restricted to the face or tongue. Anoxia and metabolic encephalopathies
may give rise to myoclonus involving the face (Jankovic, 1981). Reticular reflex
myoclonus and the hyperekplexias produce facial grimacing as part of an
exaggerated startle response, with a characteristic sequence of activation of cranial
musculature, implicating a generator in the caudal brainstem (Brown et aI., 1991),
probably in the vicinity of the pontine tegmental reticular nuclei (Kimber &
Thompson, 1997). The rhythmic movements of palatal myoclonus (also referred to
as palatal tremor) may extend beyond the palate to the face and other muscles
derived from the branchial arches.

71
Tremor
Essential and Parkinsonian tremors frequently affect the face, jaw and head as part of
a generalised tremor. Occasionally, action and task specific tremors may be
restricted to the cranial muscles, only affecting the facial or jaw muscles during
particular tasks such as smiling (Jacome & Yanez, 1987), whistling, playing a wind
instrument or drinking. The rare condition of 'galloping tongue' (Keane, 1984)
consists of episodic wavelike tongue movements with frequency 3 Hz, which spread
from the posterior midline to the sides of the tongue, observed occasionally with
pontine lesions.

Geniospasm (Hereditary Chin Trembling)

Hereditary chin trembling or geniospasm is a rare inherited disorder in which the
sole abnormality is semicontinuous arrhythmic quivering movements of the chin
produced by mentalis contraction (Danek, 1993). Patients with geniospasm
complain that they appear on the verge of tears.

Hemifacial spasm
Hemifacial spasm is common and typically unilateral, consisting of brief, repetitive
'flickering' movements of the orbicularis oculi and transient closure of one eye.
With the passage of time, the movements spread to involve most muscles on one side
of the face, including the perioral muscles and platysma. The postulated cause is
generation of ectopic impulses at the facial nerve root entry zone by compression
and irritation of the nerve by an ectatic vascular loop. The abnormal signals spread
to all facial nerve fibres by ephaptic transmission and may also influence the
excitability of the facial nucleus.

Facial myokymia
Facial myokymia consists of irregular, quivering 'worm like' contractions of the
mimetic muscles produced by repetitive, spontaneous motor unit discharges. Eyelid
myokymia is common with fatigue in healthy individuals. Myokymia of other facial
muscles occurs with pontine lesions (glioma or multiple sclerosis). The mechanism
is thought to be irritation of the facial nerve before it exits the brainstem, or
enhanced excitability of the facial nucleus itself due to alteration of its inputs
(Andermann, Cosgrove, Lloyd-Smith, Gloor, & McNaughton, 1961).

72
Hemimasticatory spasm
Hemimasticatory spasm (Thompson, Obeso, Delgado, Gallego, & Marsden, 1986) is
a rare disorder in which unilateral spasm of the masticatory muscles produces
bruxism, tongue biting and masseter and temporalis hypertrophy. The mechanisms
generating this activity also include the removal of normal reflex inhibition of
masticatory activity after jaw closure, probably due to a lesion of trigeminal
afferents, since the jaw jerk is frequently absent on the side of spasm. The condition
occurs in isolation and in association with facial hemiatrophy (Parry-Romberg
syndrome).

Myotonia
Delayed muscle relaxation due to high frequency muscle fibre discharges in
myotonia may affect the face, jaw and tongue. Eyelid myotonia and delayed eye
opening are particularly characteristic of myotonia congenita. Eyelid myotonia may
be the sole inter ictal sign in hypokalaemic periodic paralysis.

Neuromyotonia
Neuromyotonia, a syndrome of continuous motor unit actIvIty with impaired
relaxation after contraction, due to hyperexcitability of peripheral motor nerves
frequently affects the face. Schwartz-lampeI syndrome is a rare, recessively
inherited dysmorphic condition with skeletal abnormalities and a characteristic facial
appearance (blepharophimosis, dimpled chin and pursed lips). The latter are
produced by neuromyotonia in facial muscles.

Facial synkinesias
Various abnormal stereotyped facial synkinesias may appear after injury or Bell's
palsy, due to aberrant regeneration of facial nerve fibres and reinnervation of
different muscles by the same motor neurone. The precise origin (central or
peripheral) of synkinetic movements is debated (Montserrat & Benito, 1988), and a
combination of aberrant nerve fibre regeneration, ephaptic transmission and central
reorganisation seems plausible. The Marcus Gunn phenomenon or 'jaw winking'
refers to the syndrome of unilateral ptosis with elevation of the eyelid on opening the
mouth, protrusion of the jaw or tongue, clenching the teeth or smiling. It is usually
congenital, sometimes recessively inherited, but may follow anomalous
reinnervation of levator palpebrae after oculomotor, facial or trigeminal nerve injury
(Eve, 1987). The inverse phenomenon of eye closure with jaw opening (Marin-

73
Amat syndrome) may result from a developmental trigeminofacial anastomosis
(Rana and Wadia 1985). Abnormal synkinesis between muscles innervated by facial
and trigeminal nerves has been reported after resection of a trigeminal schwannoma
(Rubin, Matsumoto, Suarez, & Auger, 1999). Palpebromandibular synkinesia
(Pullicino, Jacobs, McCall, Garvey, Ostrow, & Miller, 1994), consisting of
anterolateral jaw movements accompanying blinking, may follow bilateral
hemisphere or upper brain stem lesions, or occur spontaneously. The eyelid and jaw
movements appear to be integrated in the brainstem reticular formation and may
represent a fragment of the facial expression related to sucking or crying, normally
suppressed by corticobulbar pathways.

Other movement disorders affecting the face

Oculomasticatory and oculofacial-skeletal myorhythmia (Hausser-Hauw, Roullet,
Robert, & Marteau, 1988), pathognomonic of Whipple's disease, are rare forms of
segmental rhythmic myoclonus which also appear to arise in a brainstem pattern
generator. Tonic spasms of multiple sclerosis, probably reflecting ephaptic
transmission due to demyelination in the corticobulbar tracts, sometimes affect the
face. Now rare, generalised tetanus produces the characteristic 'risus sardonicus', a
grimace caused by intense spasm of the facial muscles due to disinhibition of
brainstem interneurones. Localised cephalic tetanus in contrast produces segmental
contraction of facial muscles (Park, 1970). Involuntary painful masseter spasms
(trismus) occur with local disease of the jaw or throat, and rarely with brainstem
lesions (Jelasic & Freitag, 1978). Hypocalcaemic tetany produces twitching of facial
muscles in response to percussion of the facial nerve (Chvostek's sign) as a
manifestation of nerve fibre hyperexcitability. Difficulty sustaining tongue
protrusion and repetitive flickering movements give rise to the 'trombone tongue' of
tertiary syphilis and the 'serpentine' or 'jack in a box' tongue of Huntington's
chorea.

Neurology of voluntary facial movement

As first appreciated by Hughlings Jackson, patterns of facial movement rather than

individual muscles are represented by the motor and premotor areas of the cerebral
cortex. The cortical territory devoted to the face, lips and tongue within the human
motor homunculus is disproportionately large, reflecting the complexity of facial
movement (Penfield & Jasper, 1954). The corpus callosum connecting the two
hemispheres allows the transfer of information that enables both sides of the face to

74
act in concert and produce a coherent expression (Gazzaniga, 2000). After
integration of premotor influences from prefrontal and subcortical structures, the
supranuclear corticobulbar pathway for voluntary facial expression passes from area
4 (the primary motor cortex) via the genu of the internal capsule and cerebral
peduncle, decussates just above the level of the ipsilateral facial nucleus, and
terminates on interneurones within the contralateral facial nucleus, which synapse on
facial motor neurones. Facial motor neurones do not undergo reciprocal inhibition
and lack axon collaterals, required for recurrent inhibition suggesting a
proportionately greater degree of cortical (versus segmental) control (Curra,
Romaniello, Berardelli, Cruccu, & Manfredi, 2000). Corticofacial projections to
polysynaptic brainstem pathways may be responsible for bilateral long latency
responses to transcranial magnetic stimulation in human facial muscles (Benecke,
Meyer, Schonle, & Conrad, 1988).

Supranuclear corticobulbar facial paresis

The clinical observation that brow movements and eye closure are generally spared
in supranuclear corticobulbar facial paresis is traditionally explained by bilateral
cortical projections to the dorsal facial nuclei innervating the upper face. Lower
facial weakness varies according to the strength of ipsilateral cortical projections to
the ventral nuclear groups innervating the lower face (Kuypers, 1958). Volitional
facial paresis with preservation of emotional facial movements is well recognised in
corticobulbar lesions at sites from the cortex to the pons. Conversely, emotional
paresis with preservation of voluntary movements may occur with lesions of the
striatum, hypothalamus and thalamus. These observations indicate an anatomical
separation of the pathways mediating voluntary and emotional innervation of the
facial motor nuclei (discussed below).

Buccofacial apraxia
Oro facial and limb praxis probably have different neural substrates (Raade, Rothe, &
Heilman, 1991), though they are often associated (Pramstaller & Marsden, 1996).
Buccofacial apraxia refers to the inability to perform learned facial movements,
especially in pantomime, in the absence of facial weakness, for example, inability to
demonstrate how to use a straw, blow a kiss or whistle (Rinn, 1984). Movements of
the mouth, lower face and tongue are usually emphasised in the clinical literature on
buccofacial apraxia, however, the upper face may also be involved (Bizzozero,
Costano, DellaSala, Papagno, Spinnler, & Venneri, 2000). Acquired buccofacial
apraxia frequently results from lesions of the left frontal operculum, lower precentral

75
 gyrus or insula (Pramstaller & Marsden, 1996). Lesions in this area damage both the
motor area of the dominant left hemisphere and inter-hemispheric connections to the
right hemisphere. There is frequently, though not invariably, an accompanying
Broca's aphasia. Facial movements can no longer be activated from the left
hemisphere and become dependent on the right. In a study of 39 aphasic and non-
aphasic, brain-injured patients, Goodglass and Kaplan (1963) found that left
hemisphere lesions produced greater impairment of gestural movements than right
hemisphere lesions, even in the absence of aphasia. However, no clear distinction
was drawn between emotional and other types of gesture (nor buccofacial versus
other body movements). Orofacial apraxia is also observed with striatal lesions
(Pramstaller & Marsden, 1996; Raade et aI, 1991). There appear to be distinct
pathways for facial movements in response to verbal command (from posterior
language areas via the superior longitudinal fasciculus to association cortex anterior
to the motor face area) and visual input (from parietooccipital areas via the inferior
parietal region to the motor face area) (Pramstaller & Marsden, 1996).
Examples of facial apraxia with right hemisphere lesions (Bizzozero et aI, 2000)
demonstrate that facial praxis and language are not always linked. The respective
roles normally played by the two hemispheres in facial praxis remain poorly defined,
however visuo-spatial planning of complex facial movements may occur in the right
hemisphere. The traditional preeminence of the left hemisphere has also been
challenged by the finding of hypometabolism of the inferior and lateral portions of
both frontal lobes (Tyrell, Kartsounis, Frackowiak, Findley & Rossor, 1991) in PET
studies of patients with focal, progressive cortical degenerations leading to loss of
speech and orofacial apraxia. Functional imaging (Vargha-Khadem, Watkins, Price,
Ashburner, Alcock, Connely, Frackowiak, Friston, Pembrey, Mishkin, Gadian, &
Passingham, 1998) in a family with an inherited disorder of speech articulation and
orofacial praxis indicated dysfunction of left insular, perisylvian, supplementary
motor and premotor cortices, neostriatum, and bilateral structural abnormalities of
the caudate nuclei.

Control of emotional facial expression

The emotional expressions of anger, disgust, happiness, sadness, fear, surprise are
universal among human societies (see Barrett & Katsikitis, this volume; Rinn, 1984).
Genuine emotional expressions can be easily distinguished from the posed ones of
social display. This is exemplified by the 'Duchenne smile', in which contraction of
the outer portion of the orbicularis oculi is "only put in play by the sweet emotions

76
of the soul" (Duchenne de Boulogne, 1862). Similarly, facial expressions resulting
from pathological excitation of the facial musculature, for example risus sardonicus,
are easily distinguishable from normal smiling. These observations suggest that the
facial motor apparatus is normally tuned by emotional content to convey expression.
The delicate shadings of expression which embellish normal human social relations,
such as the laughter of nervousness or embarrassment indicate significant cortical
regulation of emotional motor responses.

Pathways for emotional facial movement

As discussed above, the dissociations of emotional and volitional facial expression
observed after various brain lesions, illustrate different cerebral influences which
descend onto the final common outflow from the facial nucleus to the mimetic
muscles. Isolated voluntary facial paresis following pontine lesions suggests that
volitional and emotional influences converge in the lower pons (Trepel, Weller,
Dichgans, & Petersen, 1996). Trepel et al (1996) proposed a pathway for emotional
facial movement parallel to that for voluntary control, passing from the amygdala
and associated limbic structures via the thalamus, supplementary motor and premotor
cortices and external capsule to the contralateral facial nucleus. Hopf, Muller-Forrell,
and Hopf (1992) documented emotional facial paresis following lesions of the
supplementary motor area, frontal white matter, striatocapsular pathways,
anterolateral and posterior thalamus, insula, operculum and mesial temporal lobe.
Interruption of connections between the thalamus, frontal lobes and mesial temporal
lobes, and lesions of the anterior limb of the internal capsule, therefore appears
critical to the development of isolated emotional facial paresis. Emotional facial
paresis has also been described with lesions of the subthalamus and dorsal midbrain
(Wilson, 1924). The experience of emotion involves a highly distributed network of
brain regions. Cortical visual and auditory association areas project to the rostral
temporal region, which is interconnected with frontal cortex, amygdala,
hippocampus (via multisynaptic arcs), thalamus and hypothalamus. Amygdala
projections are widely distributed with reciprocal connections to cortex,
hippocampus and hypothalamus. The anterior cingulate gyrus, like the amygdala,
has extensive, reciprocal connections with frontal cortex, limbic structures, basal
ganglia and brainstem and is strategically placed to integrate the widespread cortical
influences over the subcortical emotional motor system. These extensive reciprocal
connections provide a mechanism for the conjunction of intellect and affect to
modify facial expression.(Devinsky, Morrell & Vogt, 1995; Nieuwenhuys, 1996).

77
 Anterior cingulate stimulation produces facial stereotypies in man, similar to tics
(Talairach, Bancaud, Geie, Bordas-Ferrer, Bonis, Szikla, & Rusu, 1973). Complex
facial automatisms, such as lip-smacking and grimacing, as well as brief
vocalisations are observed in cingulate seizures (Devinsky et ai, 1995). Orofacial
movements such as lip smacking, chewing and swallowing movements are also
provoked by stimulation of the amygdala in man (Baldwin, Frost, & Wood, 1954).
Fish, Gloor, Quesney, and Olivier (1993) produced a variety of alterations in facial
expression and tone of voice by stimulating temporal lobe structures, especially the
amygdala, during depth electrode recordings in epilepsy surgery. Changes in the
affective experience most frequently accompanied these alterations when the
stimulus produced a spreading after-discharge, leading these authors to propose that
the temporal lobe contains the neural substrate of both affective experience and
expression (see Kappas, this volume). Bejjani, Damier, Arnulf, Thivard, Bonnet,
Dormont, Cornu, Pidoux, Samson and Agid (1999) described a unique patient whose
face 'expressed profound sadness within five seconds' after commencing high
frequency stimulation of the left (but not right) substantia nigra for Parkinson's
disease. The facial expressions were attended by the affective and cognitive
experience of acute major depression.
Akinetic mutism may follow large lesions of the cingulate and adjacent cortical
(supplementary motor) areas in humans (Freemon, 1971), especially when bilateral.
Amygdala lesions reduce emotional expression with a loss of fear and aggression.

Laughter
Arroyo, Lesser, Gordon, Uematsu, Hart, Schwerdt, Andreasson, and Fisher (1993)
emphasise the processing of information from multi modal cortical association areas
and especially, temporal lobe structures, which must take place when humans grasp a
comic situation and the experience the sense of 'mirth', which precedes the
generation of an appropriate motor response such as chuckling or laughter. The
frontal cortex may act to inhibit this response if the social context demands restraint,
vividly illustrated by the compulsive merriment of patients following frontal
lobotomy (Kramer, 1954). A recent fMRI study (Shibata, Zhong, Kwok, Shrier,
Numaguchi, & Wang, 2000) found activation of supplementary motor, right
premotor and ventromedial frontal lobe when normal subjects 'laughed internally' in
response to recorded laughter. Viewing of jokes and cartoons was associated with
bilateral (predominantly left-sided) ventromedial frontal, right amygdala and
supplementary motor activations, which may represent the neuroanatomical
correlates of mirth. Direct stimulation of the supplementary motor area at operation

78
also produced laughter associated with mirth, for which the patient invoked an
appropriate stimulus context, confirming the intimate association between motor,
affective and cognitive dimensions (Fried, Wilson, Macdonald, & Behnke, 1998).
Laughter has also been induced by globus pallidus stimulation (Hassler &Reichert
1961), indicating a role for subcortical structures in generating the emotional motor
programme.

Hemispheric control of emotional facial expressions

The non-dominant right hemisphere is generally ascribed a pre-eminent role in
emotional facial expression. Both positive and negative emotional expressions are
judged more intense on the left side of the face (Campbell, 1978; Dopson, Beckwith,
Tucker, & Bullard-Bates, 1984), perception of emotional faces is superior in the left
visual field (Campbell, 1978) and interpretation of emotional intonation is superior
via the left ear (Safer & Leventhal, 1977). The generation of facial expression is one
manifestation of right hemispheric superiority in processing and expressing emotion,
which also embraces affective prosody and gestures such as the eloquent shrug of the
shoulders or the dejected posture of sadness. Studies from brain-damaged patients
have corroborated this (Langer, Wilson, Pettigrew, & Blonder, 2000). Borod, Koff,
Lorch, and Nicholas (1985) found that patients with right hemisphere
cerebrovascular accidents used facial expression and verbal prosody less frequently
than those with left hemisphere lesions and normal controls. Asymmetry of
emotional smiling has been observed in temporal lobe epilepsy (Remillard,
Anderman, Rhi-Sausi, & Robbins, 1977), and has been used in the lateralisation of
temporal epileptic foci.
Although most authors have accepted the greater intensity of left facial
expression as evidence of the key role of the right hemisphere in the elaboration of
emotional behaviour, other interpretations are possible. Monrad-Krohn (1924)
observed that, despite impaired voluntary facial movement, emotional facial
expressions tended to be exaggerated contralateral to cortical lesions. Dopson et al.
(1984) and Rinn (1984) proposed that left hemisphere superiority for inhibiting
emotion, especially in response to social cues, accounts for reduced right facial
movement.
The most consistent asymmetries of facial expression have been observed for
posed expressions (Ekman, Hager & Friesen, 1981). It is therefore important to
distinguish voluntary, posed expressions from spontaneous, presumably genuine
ones, raising methodological questions as to whether facial expressions evoked under
experimental conditions can ever be truly spontaneous. Patients with buccofacial

79
apraxia generally retain the ability to pose emotional expressions (Rinn, 1984). Rinn
(1984) suggested that posing an emotional expression is a visuo-spatial matching
task and a function of the right hemisphere. According to this interpretation, when
the appropriate motor pattern is constructed, the facial features are matched against a
sensory schema stored in the right hemisphere. This may be viewed in the context of
a right hemisphere affective 'lexicon' of nonverbal expressions, including facial
emotions (Blonder, Bowers & Heilman, 1991), analogous to the verbal lexicon of the
left hemisphere.

Pathological affect

Much physiological interest centres on clinical cases of pathological affect in

humans, in whom emotional displays are liberated by withdrawal of descending
inhibitory controls.

Pseudobulbar palsy
The most frequent cause of pathological affect is a pseudobulbar palsy. The
syndrome results from bilateral upper motor neurone lesions affecting corticobulbar
fibres above or within the pons, pre-motor influences descending in the genu and
anterior limb of the internal capsule (Ross, 1997), or corticohypothalamic tracts
controlling facio-respiratory mechanisms (Ironside, 1956). Bulbar motor function is
impaired with dysarthria, dysphagia and loss of normal spontaneous and volitional
facial movement. The bilateral facial paresis is attended by intermittent involuntary
laughing, crying or mixtures of both, incorporating the usual respiratory, vasomotor,
secretory and vocal accompaniments (Alusi, Colquoun, Sensky, & Bain, 1999).
Paroxysms of crying or laughter may be the only means of communication. In most
cases, the provoking factor appears trivial and the display of affect is
disproportionate or even inappropriate to the stimulus and its context. The
accompanying inner emotional state is neutral or even at odds with the outward
display (Ironside, 1956).

Pathological affect without pseudobulbar palsy

Midline lesions, especially involving the posterior hypothalamus and midbrain, have
frequently been identified as causes of pathological affect (Davison & Kelman,
1939; Ironside, 1956: Martin 1950). Rarely, unilateral lesions, involving internal
capsule (anterior limb and genu), insular cortex, caudate nucleus, putamen, external
capsule, globus pallidus and left thalamus (Poeck, 1985) are responsible, however

80
the presence of pathological changes in the sound hemisphere can seldom be
excluded (Ironside, 1956). In the study of Kim and Choi-Kwon (2000), strokes
affecting the frontal and lenticulocapsular areas, and less frequently thalamus,
midbrain, basis pontis, medial medulla and cerebellum were identified, but there was
no relationship with infarct laterality or size. Pathological affect incongruent with
inner mood has been described in patients with lesions of the right frontal operculum
when combined with a major depressive disorder (Ross, 1997). In contrast, the
patient reported by Swash (1972), who developed released laughter after left
temporal infarction, "always experienced an appropriate alteration of mood during
her involuntary laughter, smiling and chuckling". Bilateral cingulate lesions may
also give rise to pathological laughter (Mendez, Nakawatase & Brown, 1999).
Stern and Brown (1957) reported spasms of pathological laughter in association
with an extensive, metastatic melanoma of the diencephalon and upper brainstem,
and proposed that the expanding tumour first irritated, and subsequently destroyed, a
diencephalo-mesencephalic integrating system for emotional expression. Prepontine
meningiomas occasionally present as pathological laughter or crying (Shafqat,
Elkind, Chiocca, Takeoka, & Koroshetz, 1998), cured by removal of the tumour.
Repeated stimulation of the posterior hypothalamus by swabbing the floor of the
third ventricle caused sudden outbursts of laughter, whistling and jocularity in the
patient reported by Foerster and Gagel (1933). Pathological laughter in association
with a large midbrain arteriovenous malformation was provoked by a variety of non-
humorous stimuli, including fundoscopy and touching the right side of the body
(Alusi et ai, 1999). Interestingly, this patient reported mirth associated with the act
of laughter implying that feedback to higher levels may engender an associated
emotional state (Arroyo et ai, 1993; Zajonc, 1985).
Released involuntary laughter occurs rarely as a prodrome of a cerebral vascular
event as in 'Ie fou rire prodromique' (Fere 1903). Examples have occurred with
bilateral thalamic and internal capsular haemorrhage (Ironside, 1956), rupture of a
large basilar aneurysm deforming the upper brainstem and posterior third ventricle
(Martin, 1950), thalamic, basal ganglia, red nucleus and pontine infarcts (Assai,
Valenza, Landis, & Hornung, 2000; Ertekin, Ekmekci, & Celebisy, 1997; Ironside,
1956; Lamer, 1998). In survivors, a prodromal intense attack of pathological
laughter may give way to a more subdued hyperemotional ism (Ertekin, Ekmekci, &
Celebisy, 1997). Based on a detailed c1inicoanatomical correlation of a patient with
fou rire prodromique secondary to a circumscribed ventral pontine infarction, Assai
et al (2000) proposed that serotonergic nuclei in the caudal pontine raphe nucleus
magnus exert an important modulatory influence over descending emotional

81
pathways, a possible site of action of serotonin reuptake inhibitors in some cases of
pathological affect (Kim & Choi-Kwon, 2000). A further illustration of the complex
brainstem circuitry involved in the generation of laughter, and its susceptibility to an
unexpectedly wide range of influences, is the unique case of gaze-induced laughter
in the presence of a large extramedullary brainstem osteochondroma described by
Leopold (1977).
Crying spells also may herald transient ischaemic attacks (Mendez & Bronstein,
1999). Lamer (1998) proposed the term 'folies larmes prodromiques' for
pathological crying heralding basilar artery occlusion. Pathological crying has also
been described in central pontine myelinolysis (Van Hilten, Buruma, Kessing, &
Vlasveld, 1988).

Gelastic and dacrystic epilepsy

Ictal pathological affect occurs in rare cases of gelastic (,laughing') and dacrystic or
quiritarian (,crying') epilepsy, arising from lesions in the vicinity of the mesial
temporal lobe (Sethi & Rao, 1976), especially the left in gelastic seizures, and
possibly the right in dacrystic seizures (Sackeim, Greenberg, Weiman, Gur,
Hungerbuhler, & Geschwind, 1982). Gelastic seizures are clinically distinguishable
from 'released' laughter by their short duration, paroxysmal quality, associated
amnesia and provoking factors. It appears unlikely that the affective display in
gelastic epilepsy is simply a response to the epileptic aura itself, since the reported
emotional experience in most cases of temporal lobe epilepsy (and in those laughing
fits which are recalled) is unpleasant (Martin, 1950). No strong emotional tone was
reported after a dacrystic seizure throughout which the patient remained conscious
(Davison & Kelman, 1939). The temporal lobe is uniquely placed to integrate
affective and motor phenomena (Nieuwenhuys, 1996) and has been implicated in
most cases of pathological affect accompanying hypothalamic lesions (Sethi & Rao
1976). On the other hand, Gumpert, Hansiota, and Upton (1970) argued that
associated episodes of sleepiness, excitement, goose flesh and temperature changes
indicated a diencephalic or hypothalamic origin. Gelastic seizures may accompany
mesial frontal lesions (Arroyo et al 1993) and their onset in the cingulate has been
verified in EEG studies (Arroyo et ai, 1993). Neutral and sad expressions were the
most frequently observed spontaneous facial expressions at the onset of seizure
activity in the study of Strauss, Wada and Kosaka (1983). Sometimes dissociation
between facial and vocal components of laughing was seen. There was no obvious

82
relationship between site of origin of the ictal discharge and the resulting facial
expression.

Poverty of emotional facial expression

Paralysis of laughter or crying is apparently rare, but appears to have occurred in one
patient with an astrocytoma of the floor of the third ventricle, in whom the capacity
to laugh or weep was lost although the corresponding emotions were present
(Ironside, 1956; Martin, 1950). Cummings, Benson, Houlihan, and Gosenfeld
(1983) described a patient with pseudobulbar palsy, complete mutism and absence of
emotional as well as posed facial expressions, attributed to multiple bilateral
periventricular and capsular lacunes. They proposed a convergence of pyramidal and
limbic connections in the region of the internal capsule and medial forebrain bundle
below the thalamus. Impassivity of emotional facial expression and a lack of
spontaneous facial expression is common in basal ganglia disorders, notably
Parkinson's disease (Jacobs, Shuren, Bowers, & Heilman, 1995), drug induced
parkinsonism, schizophrenia and frontal lobe lesions. As discussed above, paralysis
of facial muscles due to peripheral neuromuscular disease impairs both emotional
and volitional facial expression.

F ACE RECOGNITION IN HEALTH AND DISEASE

Fundamentals offace recognition

Faces are unusually complex visual stimuli, demanding analysis of fine internal
structure and having (if familiar) a large number of unique semantic associations.
Recognition of individual faces normally proceeds independently of size, contrast,
spatial resolution, position on the retina, angle of view, movement, and the emotional
expression. Conversely, emotional expressions are recognised without facial
identification. Some conditions, notably inversion and contrast negativity cause
significant delays in face processing (Perrett, Smith, Potter, Mistlin, Head, Milner &
Jeeves, 1984). Humans show remarkable facility in extracting the key features of
heavily degraded faces (cartoonists' caricatures, the abstract productions of Picasso
and Arcimboldo), which may indicate that faces are stored as loose templates, rather
than rigid representations (Dolan, Fink, Rolls, Booth, Holmes, Krackowiak, &
Friston, 1997). An influential model of face processing, developed by Bruce and
Young (1986), posits parallel, sequential streams of expression analysis, facial

83
speech analysis, matching of unfamiliar faces and recognition of personal identity.
The visual recognition stream converges with data derived from other sensory
modalities at so-called 'personal identity nodes', at which semantic information can
be retrieved and names assigned.

Disorders offace recognition: prosopagnosia and misidentification syndromes

A diverse spectrum of anatomical and pathological lesions lead to impaired face
recognition, or prosopagnosia (Damasio & Damasio, 1986; Evans, Heggs, Antoun,
& Hodges, 1995; Sergent & Signoret, 1992; Young, Newcombe, De Haan, Small, &
Hay, 1993). Conclusions regarding the anatomical substrate of prosopagnosia have
been hindered by the frequent association of other deficits, including agnosias which
obscure distinctions between different classes of objects. The ventromedial
occipitotemporal junction (the fusiform gyrus) of the right hemisphere has been
regarded as crucial, but in some cases this area is spared (Sergent & Signoret, 1992).
It has been argued that damage to both posterior hemispheres is required (Damasio &
Damasio, 1986), although a unilateral right hemisphere lesion may suffice (Evans et
aI, 1995). Young et al (1993) found a three-way dissociation between recognition of
a familiar face, matching of unfamiliar faces, and identification of facial expressions
in a study of ex-servicemen who had sustained penetrating brain injuries. Some
prosopagnosic patients identify jumbled faces more quickly than normal ones
(Perrett, Mistlin, Chitty, Harries, Newcombe & De Haan, 1988), consistent with a
piecemeal feature-by-feature analysis rather than the normal perceptual gestalt.
Prosopagnosia may be associated with the Capgras delusion, the belief that
certain individuals, usually close relatives, have been replaced by impostors. This
delusion may appear in a variety of organic settings (Ellis & Young, 1990; Signer
1994). Signer (1994) found an approximately equal distribution of lesions between
the two hemispheres, with a left temporal and right frontal emphasis, in those where
localised pathology was identified. It seems plausible that at least two alternative
pathways for face recognition exist (Bauer, 1982). The first, a conscious route, from
visual cortex to temporal lobes via the inferior longitudinal fasciculus is implicated
in prosopagnosia, and the second a covert pathway conveys emotional overtones
from visual cortex to limbic system via the inferior parietal lobule. Viewed in this
way, the Capgras delusion is the mirror image of prosopagnosia (Ellis & Young,
1990). In prosopagnosia, recognition of faces is lost but familiarity with the
individual is retained. Conversely, in the Capgras syndrome, faces are recognised
without the associated emotional overtones which convey familiarity, leading to
misidentification. Other misidentification syndromes, such as the delusions of

84
Fregoli, intermetamorphosis and autoscopy, can also be interpreted in terms of the
Bruce and Young model (Ellis & Young, 1990). In the patient of Hudson and Grace
(2000), a misidentification syndrome developed following ischaemic damage to the
right anterior fusiform, middle and inferior temporal gyri. It was postulated that the
fusiform face area was disconnected from visual memories pertaining to the familiar
face in question (the patient's husband), presumably imparting a sense of
'strangeness' necessary for the delusion to evolve. The syndrome displayed by the
patient of Gentileschi, Sperber, and Spinnler (1999), in whom prosopagnosia was
associated with progressive inability to recognise familiar people using alternative
channels (such as voice or gait cadence), while relatively sparing recognition of
personal names and common objects, would also be consistent with the Bruce and
Young model. This patient had a frontotemporal degeneration of uncertain nature
with bitemporal atrophy, more pronounced on the right side.

Disorders ofperception offacial affect

Deficits in recognition of facial affect may follow damage to either hemisphere
(Young et aI, 1993). Blonder et al. (1991) found that patients with right hemisphere
damage are impaired in comprehension of emotional prosody and facial expression
even when verbal descriptors were used, again suggesting destruction of the
emotional images constructed from lexical-semantic information. Such evidence
appears to support the notion of a right hemisphere lexicon of facial emotional
expressions. Interestingly, however, selective impairments of facial expression
processing were found only with left hemisphere lesions in the study of Young et al
(1993), suggesting that this function cannot be restricted to the right hemisphere.
The intimate association of perceptual and motor disorders of facial expression is
illustrated by the effects of basal ganglia disease. Impaired perception of facial affect
has been described in Parkinson's disease (Jacobs et aI, 1995) and Huntington's
chorea (Gray, Young, Curtis & Gibson, Sprengelmeyer, Young, Calder, Karnat,
Lange, Homberg, Perrett, & Rowland, 1996). Patients with strokes are more likely
to have impaired perception of facial emotions if the basal ganglia are affected
(Jacobs et aI, 1995). Formation of mental images and internal representations of the
motor act may require joint participation in a central processor (Jacobs et aI, 1995),
perhaps mediated via reentrant circuits including cortex, basal ganglia and other
subcortical structures.
Individuals with Huntington's disease have particular difficulty in recognising
certain emotions, notably disgust (Sprengelmeyer et aI, 1996), and this impairment
appears to be a selective, early feature exhibited by disease carriers (Gray et aI,

85
1997). Patients with bilateral amygdala damage display an apparently selective
impairment in recognising fear (Tranel & Hyman, 1990), corroborated in functional
imaging studies (Morris, Friston, Buchel, Frith, Young, Calder, & Dolan, 1998).
Such impairments may extend to the experience of the emotions themselves (Young
et aI, 1995; Gray et aI, 1997), suggesting that basic emotions may have dedicated
neural substrates (Morris et aI., 1998; Sprengelmeyer et aI., 1996); however, this
remains controversial (Rapcsak, Galper, Comer, Reminger, Nielsen, Kaszniak,
Verfaellie, Laguna, Labiner, & Cohen, 2000). The facial expression accompanying
disgust recalls the grimace in response to unpleasant smells and tastes, and (as
recognised by Darwin) it is likely to be ancient in evolutionary terms. In addition to
impaired recognition of emotional expressions, patients with bilateral surgical
amygdalotomy display selective deficits in learning new faces and interpreting the
direction of gaze (Young, Aggleton, Hellawell, Johnson, Broks, & Hanley, 1995),
implying a more general inability to read social signals from the face.

Mechanisms offace processing: functional imaging evidence

PET data obtained from normal subjects suggest that the processing of faces is
subserved by an extensive neural network encompassing most of the ventromedial
right hemisphere and the left anterior temporal pole (Gorno- Temperini, Price,
Josephs, Vandenberghe, Cappa, Kapur, Frackowiak, & Tempini, 1998; Sergent &
Signoret, 1992). Within these areas exist substrates for extraction of facial features
(lingual and fusiform gyri) and association with emotional (parahippocampal gyrus)
and biographical (anterior temporal cortex) memories have been identified,
consonant with radiological findings in prosopagnosic patients. This network is
distinct from that usually employed in identifying objects, although discriminations
within certain categories of objects may also recruit the network to a lesser extent
(Gauthier, Skudlarski, Gore, & Anderson, 2000). Gorno Tempini et al (1998) found
PET activations specific for faces, relative to stimuli controlled for visual complexity
(scrambled faces) and to proper names, in the fusiform gyri bilaterally, especially the
right, and the right lingual gyrus. These activations were similar for familiar and
unfamiliar faces. Famous faces (and famous names) activated specific left anterior
and posterior extrasylvian temporal regions relative to unfamiliar faces, attributable
to semantic information processing. A different control for visual complexity, photo
negatives of faces, was employed by George, Dolan, Fink, Baylis, Russell & Driver,
(1999) in a tMRI study designed to identify brain regions responsible for the
identification of particular individuals (rather than merely responding to faces as a
specific class of complex visual stimulus). Bilateral posterior areas in the fusiform

86
gyrus responded more strongly for faces with positive than negative contrast
polarity, and a right-sided anterior fusiform region was selectively activated by
famous (in contrast to unfamiliar) faces. Posterior fusiform areas are also activated in
response to degraded faces (Dolan et aI, 1997). Nakamura, Kawashima, Sato,
Nakamura, Sugiura, Kato, Hatano, Ito, Fukuda, Schormann & Zilles, (2000)
presented PET evidence that feature extraction on faces and other complex stimuli is
performed by the occipital cortices and posterior fusiform gyri bilaterally, whereas
the right inferior temporal and fusiform gyri respond selectively to faces but not to
non-face stimuli, and the right temporal pole is activated by familiar faces.
Methodological differences may account for the apparent inconsistencies between
these and other studies.
PET studies have identified activations specific for facial expressions in the left
amygdala, left insula and left pulvinar (Morris et aI, 1998), right inferior frontal
cortex (Nakamura, Kawashima, Ito, Sugiura, Kato, Nakamura, Hatano, Nagumo,
Kubota, Fukuda, & Kojima, 1999), and both cingulate gyri (Morris et aI, 1998). The
methodological designs of these studies emphasise the different roles played by the
amygdala, in the implicit processing of emotional content (especially fear), versus
the right frontal cortex, in the explicit characterisation of facial emotions. The
degree of limbic engagement may depend on the perceived intensity of the expressed
emotion (Nakamura et aI., 1999). The amygdala may in turn modulate the strength
of extrastriate responses to emotional faces (Morris et aI., 1998). As pointed out by
Morris et al (1998), the precise facial discrimination task may have a large bearing
on the engagement of either hemisphere, and components of neural networks within
each hemisphere.

Special aspects offace processing

Certain perceptual impairments are selective for faces. In the rare condition of
prosometamorphopsia, patients report that all faces appear distorted, 'like Picassos',
with especially prominent eyes. Aetiologies include cerebral lesions, prolonged
visual deprivation and migraine (Ffytche & Howard, 1999). Santhouse, Howard, and
Ffytche (2000) have fitted this and other visual hallucinations involving faces within
a framework corresponding to distinct visual processing streams (ventral temporal
lobe, superior temporal gyrus and visual parietal lobe). Further insights into the
fractionation of various aspects of facial processing have emerged from studies of
the motion-blind patient, L.M. (Campbell, Zihl, Massaro, Munhall, & Cohen, 1997).
Lip-reading improves the understanding of speech even in those with normal
hearing, and facial speech actions have distinctive dynamic characteristics. In order

87
to interpret them, and permit phonetic information to be extracted, the visual system
must be sensitive to the precise order of facial actions. This is a quite different
requirement to that underlying other forms of face perception (and indeed, other
forms of biological motion detection). L.M. was able to recognise static face shapes,
and even other forms of biological motion, without difficulty; however, lip-reading
was severely impaired, as was tracking direction of gaze sequences. Campbell et al.
interpreted this deficit as a disconnection of visual projections to superior temporal
areas by the lesion of movement cortex VS. They concluded that both form and
motion perception are required for lip-reading. Perception of gaze direction and
mouth movements independent of facial identity, crucial in social and emotional
communication, is mediated by areas in the superior temporal sulcus (Hoffman &
Haxby, 2000).
There is some support for a visual attentional subsystem centred on the face in
humans. One patient with a right hemisphere stroke neglected the left side of faces
but not other objects (Young, De Haan, Newcombe & Hay, 1990). Ladavas , Zelone
and Fahn (1998) demonstrated extinction of contralateral tactile stimuli in patients
with right hemisphere lesions by visual stimuli delivered near the ipsilesional side of
the face. This effect, which was not observed for more distant visual stimuli, is
consistent with a face-centred visual peri personal space analogous to that described
for the hand, and may indicate that some neurones within the cortical population
representing the face respond to both tactile and visual stimuli, enabling coordination
of an appropriate motor response (for example, when guiding food into the mouth).

SYNTHESIS

The elaborate neural machinery devoted to perception and production of facial

expressions in humans illustrate the evolutionary advantage that such nonverbal
communication channels confer within the social milieu.
It is apparent that the human face may be divided into neuroanatomically and
functionally distinct, upper and lower zones (Rinn, 1984). This is reflected in the
much greater lateral independence of action of the lower face, and the high degree of
voluntary control of individual perioral (in contrast to brow) muscles. The face is
the principal vehicle for the wide repertoire of human emotional expressions. Such
versatility is made possible by the multiple, hierarchically organised, convergent
inputs from cortical, limbic, and basal ganglia networks upon the final, common
motor outflow to the mimetic muscles (Nieuwenhuys, 1996). The development of
pathological affect with lesions at different levels of the descending motor pathways

88
 illustrates well the hierarchical organisation of facial motor control. Sneezing,
yawning and pathological phenomena such as the 'fou rire prodromique' of Fere
clearly demonstrate that complex motor synergies can be organised at the level of the
brainstem. The continuum of volitional control over facial involuntary movements
(Lees, 1988) illustrates the blending of cortical and subcortical mechanisms in the
production of normal facial expressions. In contrast, autonomous, parallel activity of
these mechanisms is evident in the broad dissociations which characterise the clinical
literature on facial neurology. Possible sites of convergence of the pathways
mediating emotional and voluntary facial expression include the internal capsule,
medial forebrain bundle (Cummings et aI., 1983), and pons (Hopf et aL, 1992;
Trepel et aI., 1996). Key structures which synthesise and integrate the diverse
components (experiential, motor and autonomic) of the normal emotional response
include the anterior cingulate cortex (Arroyo et aI., 1993; Devinsky et aI., 1995),
temporal lobe (Swash, 1972; Sethi & Rao, 1976; Fish et aI., 1993), amygdala (Morris
et aI., 1998), hypothalamus (Ironside, 1956; Martin, 1950) and midbrain (Eidelberg
et aI., 1997). At the level of hemispheric organisation, homologies exist within the
right hemisphere for the receptive, integrative and expressive language functions of
the left (Blonder et aI., 1991), including a 'lexicon' of emotional representations,
which may be translated into facial expressions (Blonder et aI., 1991). As with
language and praxis (Tyrell et aI., 1991), it appears that hemispheric dominance for
facial reception and expressivity is not absolute: certain basic neuropsychological
functions, such as a superiority for holistic, visuo-spatial processing (Rinn, 1984),
probably reside in the right hemisphere, but normal facial processing probably
requires a high order of inter-hemispheric cooperation (Young et aI., 1993).
On the perceptual side, the Bruce and Young model of overt face recognition sits
well with clinical and experimental data in support of a highly fractionated cortical
representation of the human face, though the details are far from worked out.
Studies such as those of Sprengelmeyer et aL (1996), Young et aL (1995) and Jacobs
et aL (1995) show how basal ganglia and limbic structures may cooperate in
modelling the emotional facial expressions of self and others. Again, the mutual
interdependence of parallel circuits is clear: convergent perceptual information about
other faces, modulated by neocortical influences, plays a crucial role in the
generation of one's own facial motor programme. Darwin wrote in 1872 that human
facial expression "deserves still further attention, especially from any able
physiologist." This is even more apposite today, in the light of complementary lines
of evidence drawn from neuroanatomy, electrophysiology, neuropsychology and
functional brain imaging which promise to fit the curiosities and apparent

89
contradictions of clinical practice within the framework of a comprehensive
physiology of human facial expression.

AFFILIATION

University Department of Medicine and Department of Neurology

Royal Adelaide Hospital
South Australia
Australia
Email: philip.thompson@adelaide.edu.au

REFERENCES
Alusi, S.H., Colquoun, I., Sensky T., & Bain, P.G. (1999). Pathologic laughter resulting from a large
midbrain arteriovenous malformation. Movement Disorders, 14, 185-189.
Andermann, F, Cosgrove. J.B.R., Lloyd-Smith, D.L., Gloor. P., & McNaughton, F.L. (1961). Facial
myokymia in multiple sclerosis. Brain, 84, 31-44.
Arroyo, S., Lesser, R.P., Gordon, B., Uematsu, S., Hart, J., Schwerdt, P., Andreasson, K., & Fisher, R.S.
(1993). Mirth, laughter and gelastic seizures. Brain, 116,757-780.
Assai, F., Valenza, N., Landis, T., & Hornung, J.P. (2000). Clinicoanatomical correlates of a fou rire
prodromique in a pontine infarction. Journal of Neurology, Neurosurgery & Psychiatry, 69, 692-
709.
Baldwin, M., Frost, L.L. & Wood, CD. (1954). Investigation of the primate amygdala movements of the
face and jaws. Neurology, 4,586-98.
Barbizet, J. (1958). Yawning. Journal of Neurology, Neurosurgery & Psychiatry, 21, 203-209.
Barrett, RJ ., & Katsikitis, M. (2002). Foreign faces A voyage to the lanbd of eepica. In M Katsikitis
(Ed.), The human face: Measurement and meaning (pp. 1-28). Dordrecht: Kluwer Academic
Publ ishers.
Bauer, R.M . (1982). Visual hypoemotionality as a symptom of visual-limbic disconnection in man.
Archives of Neurology, 39,702-708.
Bcjjani, B.P., Damicr, P., Amulf, I., Thivard, L., Bonnet, A.M., Dormont, D., Cornu, P., Pidoux, B.,
Samson, Y., & Agid, Y. (1999). Transient acute deprcssion induced by high-frequcncy dcep-brain
stimulation. New England Journal of Medicine, 340, 1476-1480.
Bell, C (1844). The anatomy and philosophy of expression as connected with the fine arts. Third
Edition. London: John Murray.
Benecke, R., Meyer, B. U., Sch(jnle, P., & Conrad, B. (1988). Transcranial magnetic stimulation of the
human brain: Responses in muscles supplied by cranial nerves. Experimental Brain Research, 7/,
623-632.
Berardelli, A., Rothwell, lC, Day, BL, & Marsden CD. (1985). Pathophysiology of blepharospasm
and oromandibular dystonia. Brain, 108,593-608.
Bizzozero I, Costano, D., Della Sala, S., Papagno, C, Spinnler, II., & Venneri, A. (2000). Upper and
lower face apraxia: Role of the right hemisphere. Brain. 123,2213-2230.

90
Blonder, L.x., Bowers, D., & Heilman, K.M. (1991). The role of the right hemisphere in emotional
communication. Brain, 114, 1115-1127.
Boghen D. (1997). Apraxia of lid opening: a review. Neurology, 48, 1491-1503.
Borod, J.C., Koff, E., Lorch, M.P., & Nicholas, M. (1985). Channels of emotional expression in patients
with unilateral brain damage. Archives of Neurology, 42,345-348.
Brown, P., Rothwell, J.e., Thompson, p.o., Britton, T.e., Day, B.L., & Marsden, e.D. (1991). The
hyperekplexias and their relationship to the normal startle reflex. Brain, 114, 1903-1928.
Bruce, V, & Young, A. (1986) Understanding face recognition. British Journal of Psychology, 77,305-
27.
Campbell, R. (1978). Asymmetries in interpreting and expressing a posed facial expression. Cortex, 14,
327-342.
Campbell, R., Zihl, J., Massaro, D., Munhall, K, & Cohen, M.M. (1997). Speech-reading in the
akinetopsic patient, L.M. Brain, 120, 1793-1803.
Caplan, LR. (1974). Ptosis. Journal of Neurology, Neurosurgery & Psychiatry, 37, 1-7.
Caviness, J,N., Gabellini, A., Kneebone, e.S., Thompson, P.D., Lees, A.J., & Marsden, e.D. (1994).
Unusual focal dyskinesias: The ears, the shoulders, the back, and the abdomen. Movement
Disorders, 9, 531-538.
Co, S. (1979). Intractable sneezing: case report and literature review. Archives of Neurology, 36, 111-
112.
Courville, J. (1966). The nucleus of the facial nerve: The relation between cellular groups and peripheral
branches of the nerve. Brain Research, 1, 338-54.
Cule, J. (1993). The enigma of facial expression: Medical interest in metoposcopy. Journal of Historical
Medicine Allied Science, 48,302-319.
Cummings, JL., Benson, F., Houlihan, JP., & Gosenfeld, L.F. (1983). Mutism: loss of neocortical and
limbic vocalization. Journal of Nervous Mental Disease, 171,255-259.
Curra, A., Romaniello, A., Berardelli, A., Cruccu, G., & Manfredi, M. (2000). Shortened cortical silent
period in facial muscles of patients with cranial dystonia. Neurology, 54, 130-135.
Cushing, H. (1912). The pituitary body and its disorders. Philadelphia and London.
Damasio, A.R., & Damasio, H. (1986). The anatomical substrate of prosopagnosia. In R. Bruyer (Ed.),
The neuropsychology offace perception and facial expression (pp. 31-38) Hillsdale (NJ): Lawrence
Erlbaum.
Danek, A. (1993). Geniospasm: hereditary chin trembling. Movement Disorders, 8, 335-338.
Darwin, e. (1872). The expression of the emotions in man and animals. London: John Murray.
Davison, e., & Kelman, H. (1939). Pathologic laughing and crying. Archives of Neurology & Psychiatry,
42,595.
De Renzi, E., Gentilini, M. & Bazolli, e. (1986). Eyelid movement disorders and motor impersistence in
acute hemisphere disease. Neurology, 36,414-418.
Devinsky, O. (1983). Neuroanatomy of Gilles de la Tourette's Syndrome: Possible midbrain
involvement. Archives of Neurology, 40,508-514.
Devinsky, 0., Morrell, MJ., & Vogt, B.A. (1995). Contributions of the anterior cingulate cortex to
behaviour. Brain, 118,279-306.
Dewey, R.B., & Maraganore, D.M. (1994). Isolated eyelid-opening apraxia: report of a new levodopa-
responsive syndrome. Neurology, 44, 1752- I 754.

91
Dolan, RJ., Fink, G.R., Rolls, E., Booth, M., Holmes, A., Frackowiak, R.S, & Friston K.J. (1997). How
the brain learns to see objects and faces in an impoverished context. Nature, 389,596-599.
Dopson, w.G., Beckwith, B.E., Tucker, D.M, & Bullard-Bates, P.C. (1984). Asymmetry of facial
expression in spontaneous emotion. Cortex, 20, 243-251.
Duchenne de Boulogne, G.B. (1990). The mechanism of human facial expression. (trans I. RA
Cuthbertson), New York: Cambridge University Press.
Eidelberg, D., Moeller, I.R., Antonini, A., Kazumata, K., Dhawan, V., Budman, e., & Feigin, A. (1997)
The metabolic anatomy of Tourette's Syndrome. Neurology, 48,927-934.
Ekman P. (1979) About brows: emotional & conversational signals. In M. Von Cranach, K. Foppa, W.
Lepenies, & D. Ploog (Eds.). Human ethology (pp. 169-248) Cambridge: Cambridge University
Press.
Ekman, P., Hager, 1.e., & Friesen W. V. (1981). The symmetry of emotional and deliberate facial actions.
Psychophysiology, 18, 101-106.
Ellis, H.D., & Young, A.W. (1990). Accounting for delusional misidentitications. British Journal of
Psychiatry, 157,239-248.
Ertekin, e., Ekmekci, 0., & Celebisy, N. (1997). Le fou rire prodromique. Journal of Neurology, 244,
271-275.
Evans, J.J., Heggs, A.J., Antoun, N., & Hodges, 1.R. (1995). Progressive prosopagnosia associated with
selective right temporal lobe atrophy: A new syndrome. Brain, 118, 1-13.
Eve, F.R. (1987). Pterygoid-levator synkinesis. Journal of Clinical Neuro-Ophthalomology, 7,61-62.
Fere, M.e. (1903). Le fou rire prodromique. Revue Neurologique, 11,353-358.
Ffytche, D.H., & Howard, R.J. (1999). The perceptual consequences of visual loss: 'positive' pathologies
of vision. Brain, 122, 1247-1260.
Fish, DR, Gloor, P., Quesney, FL, & Olivier, A. (1993). Clinical response to electrical brain
stimulation of the temporal and frontal poles in patients with epilepsy. Brain, 116,397-414.
Foerster, 0., & Gagel, 0 (1933). Ein fall von ependymcyste des III . Ventrikels. ein beitrag zur frage der
beziehungen psychischer st(jrungen zum hirnstamm. Zeit gesamt Neurology & Psychiatry, 149, 312-
344.
Freemon, F.R. (197 I) Akinetic mutism and bilateral antcrior cerebral artcry occlusion. Journal of
Neurology, Neurosurgery & Psychiatry, 34, 693-698.
Fried , I., Wilson, e.L., MacDonald, K.A ., & Behnke, E.J. (1998) Electric current stimulates laughter.
Nature, 391,650.
Gauthier, I., Skudlarski, P., Gore, J.e., & Anderson, A.W. (2000). Expcrtise tor cars and birds rccruits
brain areas involved in face recognition. Nature Neuroscience, 3, 191-197.
Gazzaniga, M.S. (2000). Cerebral specialization and interhemispheric communication: Does the corpus
callosum enable the human condition? Brain, 123, 1293-1326.
Gentileschi, V., Sperber, S., & Spinnler, H. (1999). Progressive defective recognition of familiar pcoplc.
Neurocase, 5,407-424.
George, N., Dolan, R.J., Fink, G.R ., Baylis, G.c., Russell, c., & Driver, J (1999). Contrast polarity and
face recognition in the human fusiform gyrus. Nature Neuroscience, 2, 574-80.
Gill berg, c., & Steffenburg, S. (1989). Autistic behavior in Mocbius Syndrome Acta Paediatrica
Scandinavia, 78,314-316.
Goodglass, H., & Kaplan, E. (1963). Disturbance ofgesturc and pantomimc in aphasia. Brain, 86, 703-
720.

92
Gorno-Tempini, M.L., Price, c.J, Josephs, 0., Vandenberghe, R., Cappa, S.F., Kapur, N., Frackowiak,
R.S., & Tempini, M.L (1998). The neural systems sustaining face and proper name processing.
Brain, 121,2103-2118.
Gray, J.M., Young, AW., Curtis, A., & Gibson, D. (1997). Impaired recognition of disgust in
Huntington's disease gene carriers. Brain, 120, 2029-2038 .
Gumpert, J, Hansotia, P., & Upton, A (1970). Gelastic epilepsy. Journal of Neurology, Neurosurgery &
PsychiatryJi,479-483.
Hassler, R., & Reichert, T. (1961). Wirkungen der reizungen und koagulationen bei stereotaktischen
hirnoperationen. Nervenarzt, 32,97-109.
Hausser-Hauw, C., Roullet, E., Robert, R., & Marteau, R. (1988). Oculo-facio-skeletal myorrhythmia as
a cerebral complication of systemic Whipple's disease. Movement Disorders, 3, 179-184.
Hoffman, E. A., & Haxby, J.V. (2000). Distinct representations of eye gaze and identity in the distributed
human neural system for face perception. Nature Neuroscience, 3, 80-84.
Hopf, H.C., Muller-Forrell, W., & Hopf, N.J. (1992). Localization of emotional and volitional facial
paresis. Neurology, 42, 1918-1923.
Hudson, A.J., & Grace, G.M. (2000). Misidentification syndromes related to face specific area in the
fusiform gyrus. Journal of Neurology. Neurosurgery & Psychiatry, 69, 615-618.
Hutchinson, M., Nakamura, T., Moeller, JR ., Antonini, A., Belakhlef, A., Dhawan, V. , & Eidelberg, D.
(2000). The metabolic topography of essential blepharospasm: a focal dystonia with general
implications. Neurology, 55,673-677.
Ironside, R. (1956). Disorders of laughter due to brain lesions. Brain, 79, 589-609.
Jacobs, D.H., Shuren, J., Bowers, D., & Heilman, K.M. (1995). Emotional facial imagery, perception and
expression in Parkinson's disease. Neurology, 45, 1696-1702.
Jacome, DE. , & Yanez, G.F. (1987). Tremors of the smile. Journal of Neurology, Neurosurgery &
Psychiatry, 50, 489-490.
Jankovic, J (1981). Drug-induced and other orofacial-cervical dyskinesias. Annals of Internal Medicine,
94,788-793 .
Jankovic, J . (1988). Etiology and differential diagnosis of blepharospasm and oromandibular dystonia.
Advances in Neurology, 49, 103-116.
Jelasic, F., & Freitag, V. (1978). Inverse activity of masticatory muscles with and without trismus: a
brainstem syndrome. Journal of Neurology, Neurosurgery & Psychiatry, 41,798-804.
Kappas, A (2002). What facial activity can and cannot tell us about emotions. In M. Katsikitis (Ed .), The
human face: Measurement and meaning (pp. 215-234) Dordrecht: Kluwer:Academic Publishers.
Karson, C. N. (1988). Physiology of normal and abnormal bl inking. Advances in Neurology, 49, 25-37.
Keane, J.R. (1984). Galloping tongue: Po~t-traumatic, episodic, rhythmic movements. Neurology, 34,
251-252.
Kim, J.S., & Choi-Kwon, S. (2000). Post-stroke depression and emotional incontinence: Correlation with
lesion location. Neurology, 54, 1805-1810.
Kimber, T.E., & Thompson, P.D. (1997). Symptomatic hyperekplexia occurring as a result of pontine
infarction. Movement Disorders, 12,814-815.
Kimber, T.E. , & Thompson, p.o. (2000). Increased blink rates in advanced Parkinson's disease: A form
of "off" period dystonia. Movement Disorders, 15,982-985.
Kramer, H.C. (1954). Laughing spells in patients after lobotomy. Journal of Nervous and Mental
Diseases, 119, 517-522 .
Kuypers, H.G .JM. (1958). Corticobulbar connexions to the pons and lower brainstem in man An
anatomical study. Brain, 81, 364-389.

93
Ladavas, E., Zeloni, G., & Fame, A. (1998). Visual peri personal space centred on the face in humans.
Brain, 121,2317-2326.
Langer, S.L., Wilson, J.F., Pettigrew, L.C., & Blonder, L.x. (2000). Channel consistency following
unilateral stroke: An examination of patient communications across verbal and nonverbal domains.
Neuropsychologia, 38,337-344.
-Lamer, A.l (1998). Basilar artery occlusion associated with pathological crying: " Folies larmes
prodromiques"? Neurology, 51,916-917.
Lees, AJ . (1988). Facial mannerisms and tics. Advances in Neurology, 49,255-261.
Leopold, N.A. (1977). Gaze-induced laughter. Journal of Neurology, Neurosurgery & Psychiatry, 40,
815-817.
Marsden, C.D. (1976). Blepharospasm-oromandibular dystonia syndrome (Brueghel's syndrome): a
variant of adult onset torsion dystonia? Journal of Neurology, Neurosurgery & Psychiatry, 39, 1204-
1209.
Martin, P. (1950). Fits of laughter (sham mirth) in organic cerebral disease. Brain, 73,453-464.
Meige, H. (1910). Les convulsions de la face, une forme clinique de convulsion faciale, bilaterale et
mediane. Revue Neurologique, 20,437-443.
Mendez, M.F ., & Bronstein, Y.L. (1999). Crying spells as symptoms of a transient ischaemic attack.
Journal of Neurology, Neurosurgery & Psychiatry, 67,255 .
Mendez, M.F., Nakawatase, T.V ., & Brown, C .V. (1999). Involuntary laughter and inappropriate hilarity .
Journal of Neuropsychiatry Clinical Neuroscience, 11,253-258.
Monrad-Krohn, G.H. (1924). On the dissociation of voluntary and emotional innervation in facial paresis
of central origin. Brain, 47, 22-35.
Montserrat, L., & Benito, M. (1988). Facial synkinesis and aberrant regeneration of the facial nerve.
Advances in Neurology, 49, 211-224.
Morris, lS, Friston , K.l , Buchel, C ., Frith, C.D., Young, A.W. , Calder, AJ ., & Dolan, R.l (1998). A
neuromodulatory role for the human amygdala in processing emotional facial expressions. Brain,
121,47-57.
Nakamura, K, Kawashima, R. , Ito, K., Sugiura, M., Kato, T., Nakamura, A, Hatano, K., Nagumo, S.,
Kubota, K., Fukuda, H., & Kojima, S. (1999). Activation of the right inferior frontal cortex during
assessment of facial emotion. Journal of Neurophysiology, 82, 1610-1614.
Nakamura, K., Kawashima, R., Sato, N., Nakamura, A., Sugiura, M., Kato, T., Hatano, K., Ito, K.,
Fukuda, H., Schormann, T. , & Zilles, K. (2000). Functional delineation of the human occipito-
temporal areas related to face and scene processing: A pet study. Brain, 123, 1903-1912.
Nashold , B.S., & Gills, lP. (1967). Ocular signs from brain stimulation and lesions. Archives of
Ophthalmology, 77, 609-618.
Nieuwenhuys, R. (1996). The greater limbic system, the emotional motor system and the brain.
Progressive Brain Research, 107,551-580.
Park, D.M. (1970). Cranial nerve palsies in tetanus : cephalic tetanus. Journal of Neurology,
Neurosurgery & Psychiatry, 33,212-215 .
Penfield, W., & Jasper, H. (1954) Epilepsy and the functional anatomy of the human brain. Boston,
Little Brown.
Perrett, 0.1., Mistlin, AJ., Chitty, AJ., Harries, M., Newcombe, F., & De Haan, E. (1988). Neuronal
mechanisms of face perception and their pathology. In C. Kennard, & F. Clifford Rose (Eds.),
Physiological aspects of clinical neuro-ophthalmology (pp . 137-154). Chapman Hall : London .
Perrett, 0 .1., Smith, P.A., Potter, D.O., Mistlin, A, Head, A.S ., Milner, A.D., & Jeeves, MA (1984).
Neurons responsive to faces in the temporal cortex : Studies of functional organization, sensitivity to
identity, and relation to perception. Human Neurobiology, 3, 197-208.

94
Poeck. K. (1985). Pathological laughter and crying. In J.A.M. Frederiks (Ed.), Handbook of clinical
neurology (pp. 219-225). Elsevier: Amsterdam.
Postert, .T., Pohlau, D., Meves, S., Nastos, I., & Przuntek H. (1996). Pathological yawning as a symptom
of multiple sclerosis. Journal of Neurology, 243, 300-301.
Pramstaller, P.P., & Marsden, C.D. (1996). The basal ganglia and apraxia. Brain, 119, 319-340.
Pullicino, P.M., Jacobs, L., McCall, W.D., Garvey, M., Ostrow, P.T, & Miller, L.L. (1994). Spontaneous
palpebromandibular synkinesia: a localizing clinical sign. Annals of Neurology, 35, 222-228.
Raade, A.S., Rothe, L.J.G., & Heilman, K.M. (1991). The relationship between buccofacial and limb
apraxia. Brain Cognition, 16, 130-146.
Rana, P.V.S., & Wadia, R.S. (1985). The Marin-Amat Syndrome: An unusual facial synkinesia. Journal
of Neurology, Neurosurgery & Psychiatry, 48, 939-941.
Rapcsak, S.z., Galper, S.R., Comer, J.F., Reminger, M.A., Nielsen , L., Kaszniak, A.W., Verfaellie, M.,
Laguna, J.F., Labiner, D.M., & Cohen, RA. (2000). Fear recognition deficits after focal brain
damage: A cautionary note. Neurology, 54,575-581.
Remillard, G., Anderman, F., Rhi-Sausi, A., & Robbins, N.M. (1977). Facial asymmetry in patients with
temporal lobe epilepsy. Neurology, 27,109-114.
Rinn, W.E. (1984). The neuropsychology of facial expression: A review of the neurological and
psychological mechanisms for producing facial expressions. Psychology Bulletin, 95,52-77.
Rogers, D., & Hymas, N. (1988). Sporadic facial stereotypies in patients with schizophrenia and
compulsive disorders. Advances in Neurology, 49, 383-394.
Ross, E.D. (1977). Cortical representation of the emotions. In M.R. Trimble, & J.L. Cummings. (Eds),
Contemporary behavioral neurology (pp. 107-226). Butterworth-Heinemann.
Rubin, D.I" Matsumoto, J.Y. , Suarez, G.A., & Auger, R. G. (1999). Facial trigeminal synkinesis
associated with a trigeminal schwannoma. Neurology, 53, 635-637.
Sackeim, H.A., Greenberg, M.S., Weiman, A.L, Gur, R.C., Hungerbuhler, J.P. , & Geschwind, N. (1982)
Hemispheric asymmetry in the expression of positive and negative emotions: Neurologic evidence.
Archives in Neurology, 39,210-218.
Safer, M.A., & Leventha, H. (1977). Ear differences in evaluating emotional tones of voice and verbal
content. Journal of Experimental Psychology [Human Perception}, 3,75-82.
Santhouse, A.M., Howard, R.J ., & Ffytche, D.H. (2000). Visual hallucinatory syndromes and the
anatomy of the visual brain . Brain, 123,2055-2064.
Schmidtke, K., & BOttner-Ennever, J.A. (1992). Nervous control of eyelid function . Brain, 115,227-247.
Sergent, J., & Signoret, J.L. (1992). Functional and anatomical decomposition of face processing:
evidence from prosopagnosia and pet study of normal subjects. Philosophical Transactions of the
Royal Society of London series B Biological Sciences. 335, 55-62.
Sethi, P.K., & Rao, T.S. (1976). Gelastic, quiritarian and cursive epilepsy: a clinicopathological
appraisal. Journal of Neurology, Neurosurgery & Psychiatry. 39, 823-828.
Shafqat, S., Elkind, M.S.V., Chiocca, E.A., Takeoka, M., & Koroshetz, W..I. (1998). Petroclival
meningioma presenting with pathological laughter. Neurology, 50,1918-1919.
Shahani, B. (1970). The human blink reflex. Journal of Neurology, Neurosurgery & Psychiatry, 33, 792-
800.
Shibata, D.K., Zhong, J., Kwok, E., Shrier, D.A., Numaguchi, Y, & Wang, H.Z. (1994). Finding the
funny bone in the brain: fmri of humor and laughter [abstract] . RSNA Proceedings 2000, C 17, 396 .
Signer, S.F. (1994). Localization and lateralization in the delusion of substitution. Psychopathology, 27,
168-176.

95
Sprengelmeyer, R., Young, A.W., Calder, A.1., Kamat, A., Lange, H., Homberg, V., Perrett, 0.1., &
Rowland, 0 (1996). Loss of disgust: perception of faces and emotions in huntington's disease. Brain,
119,1647-1665.
Stem, W.E., & Brown, W.1. (1957). Pathological laughter. Journal of Neurosurgery, 14, 129-139.
Strauss, E., Wada, .J, & Kosaka, B. (1983). Spontaneous facial expression occurring at the onset of focal
seizure activity. Archives of Neurology, 40,545-547.
Swash, M. (1972). Released involuntary laughter after temporal lobe infarction. Journal of Neurology,
Neurosurgery & Psychiatry, 35, 108-113.
Talairach, .1, Bancaud, J., Geie, S., Bordas-Ferrer, M., Bonis, A., Szikla, G., & Rusu, M. (1973). The
cingulate gyrus and human behaviour. EEG. Clinical Neurophysiology, 34,45-52.
Thompson, P.O., Obeso, J.A., Delgado, G., Gallego, J., & Marsden, C.D. (1986). Focal dystonia of the
jaw and the differential diagnosis of unilateral jaw and masticatory spasm. Journal of Neurology,
Neurosurgery & Psychiatry, 49, 651-656.
Tolosa, E., Kulsevsky, J., & Fahn, S. (1988). Meige Syndrome: primary and secondary limns. Advances
in Neurology, 50,509-515.
Tranel, D., & Hyman, B.T. (1990). Neuropsychological correlates of bilateral amygdala damage.
Archives in Neurology, 47,349-55.
Trepel, M., Weller, M., Dichgans, J., & Petersen D. (1996). Voluntary facial palsy with a pontine lesion.
Journal of Neurology, Neurosurgery & Psychiatry, 59, 531-533.
Tyrell, P.1., Kartsounis, L.D., Frackowiak, R.S.1., Findley, L.J., & Rossor, M.N. (1991). Progressive
impairment of speech output and orofacial dyspraxia associated with Irontal lobe hypometabolism.
Journal of Neurology, Neurosurgery & Psychiatry, 54, 351-357.
Valls-Sole, J., Valderiola, F., Tolosa, E., & Marti, M.1. (1997). Distinctive abnormalities of facial
reflexes in patients with progressive supranuclear palsy. Brain, 120, 1877-1883.
Van Hilten, J.1., Bumma, O.1.S., Kessing, P., & Vlasveld, L.T. (1988). Pathologic crying as a prominent
behavioral manifestation of central pontine myelinolysis. Archives of Neurology, 45,936.
Vargha-Khadem, F., Watkins, K.E., Price, C.1., Ashbumer, J., Alcock, K.1., Connelly, A., Frackowiak,
R.S., Friston, K.J., Pembrey, M.E., Mishkin, M., Gadian, DG, & Passingham, R.E. (1998) Neural
basis of an inherited speech and language disorder. Proceedings of the National Academic of
Science, U.S.A., 95, 12695-12700.
Von Economo, C. (1931) Encephalitis lethargica: Its sequelae and treatment. Ox lord University Press:
London.
Welt, C, & Abbs, HI. (1990). Musculotopic organisation of the facial motor nucleus in macae a
fascicularis: A morphometric and retrograde tracking study with cholera toxin b-hrp. JOl/rnal of
Comprehensive Neurology, 291, 621-636.
Wilson, S.A.K. (1924). Pathological laughing and crying. Journal of Neurology & Psychopathology, 4,
299-333.
Young, A.W., De Haan, E.I-I.F., Newcombe, F., & Hay DC (1990). Facial neglect. Neuropsychology, 28,
391-415.
Young, A.W., Newcombe, F., De Haan, E.H.F., Small, M., & Hay, D.C (1993). Face perception after
brain injury: selective impainnents affecting identity and expression. Brain, 116,941-959.
Young, A.W., Aggleton, J.P., Hellawell, 0.1., Johnson, M., Broks, P, Hanley, .l.R. (1995). Face
processing impairments after amygdalotomy. Brain, 118, 15-24.
Zajonc, R.B. (1985). Emotion and facial efference: A theory reclaimed. Science, 22, 15-21.

96
Section B: Measurement and Meaning
 MARY KA TSIKITIS

CHAPTERS

FACEM: THE FACIAL EXPRESSION MEASUREMENT

SYSTEM

INTRODUCTION

The study of facial expression has been approached from a variety of perspectives,
both theoretical and methodological. The strong foundations laid by the early
pioneers have attracted researchers from a number of disciplines, such as
psychology, physiology and ethology, to name a few. The study of facial expression
has generally been embraced under the more common quest for a theory of emotion
and is considered to be the most immediate and informative component of the
emotional experience. Methods designed to capture and measure this dynamic
phenomenon have greatly improved our understanding of discrete muscle action
associated with the various facial displays.
Reading faces has produced some ambiguous and contradictory findings, and the
interpretation of nonverbal reactions by an untrained observer have been shown to be
quite misleading (Ekman, 1982; Ekman, Friesen & Ellsworth, 1982). Hence, the
development of a model for the quantification of facial expression has as its aim the
rapid and efficient clarification of facial movements. Furthermore, it may elucidate
the relationship between facial activity and emotion provided that such a model can
accommodate the range of information required for the recognition of facial
expression.
This chapter has two aims. Firstly, to describe the development of a computer-
based technique called the Facial Expression Measurement system or FACEM which
has been designed to quantify facial movement. Secondly, the functional utility of
this model will be discussed, insofar as its applicability, both clinically and
empirically, is concerned. It is envisaged that the demonstrated reliability, validity
and speed of data analysis that this system has to offer will assist and encourage
users in future applications to various research enquiries.

THE HISTORY OF FACEM

FACEM was developed in the Department of Psychiatry at the University of

Adelaide.! There was a special interest in quantifying the facial expression of
emotion displayed by patients (in particular, those who denied affect) during a
therapeutic encounter. It was envisaged that a reliable and objective measurement
system could provide specific clues as to the emotional condition of psychiatrically
disturbed patients in general and that furthermore, any change in the facial
expression of the patient during treatment might be used to monitor the efficacy of
certain treatment procedures.
The development of FACEM had as its aim the means to provide the rapid and
efficient clarification of facial movements. Although Ekman and his colleagues had
developed the Facial Action Scoring Technique (FAST: Ekman, Friesen & Tomkins,
1971) and were working on the Facial Action Coding System (F ACS: Ekman &
Friesen, 1978), it was more relevant to the team in Adelaide, to develop a
measurement tool that could be applied clinically. Thus, a more efficient method for
capturing facial expression was needed, that is, one that would not involve the
amount of training and analysis time which was associated with FAST and FACS.
With the advent of low cost videorecorders and the availability of computers, it
seemed more appropriate to computerise the facial measurement process.

THE DEVELOPMENT OF A MICROCOMPUTER-BASED APPROACH TO THE

QUANTIFICATION OF FACIAL EXPRESSION

FACEM is based on a mathematical model of the face with a focus on easily located
facial points or "landmarks" which were the most relevant to the detection of
emotional signals. 2 Employing a dimensional approach to the study of facial
expressions, a mathematical technique was utilised for scaling expressions along a
happiness-sadness axis. It was envisaged that the model would be designed to rate
expressions along any expressive bipolar dimension.

100
 The development of a mathematical model which was capable of scaling a
variety of facial expressions involved the following stepwise process: (a) selection of
a facial source to be used as input data; (b) definition of point locations to identify
facial "landmarks"; and (c) definition of facial "muscloids" which would simulate
facial muscles.

Selection of afacial source:

The first consideration in the construction of a mathematical face model addressed

the issue of a suitable source of facial input. Thornton (1979) sought quantitative
data that were conducive to mathematical manipulation. Electromyographic
techniques were considered, but later discarded due to their intrusive nature.
Photographic displays of facial expressions were chosen as they were easy to obtain
from subjects. This was not without its problems, however. A photographic image
was found to provide more information than the working computer could handle. A
line drawing representation of a photograph of a face, on the other hand, was
preferred as it substantially reduced the data output without sacrificing vital facial
information. In other words, Thornton produced and worked with a line drawing
from every expressive display. Ekman, Friesen and Ellsworth (1972) pointed out the
difficulties associated with any drawing of a facial expression. They explained that
specifically, an artist's representation of a facial expression may involve features or
feature combinations which do not take place. It will be appreciated however, that in
the case of Thornton's line drawing representations, a true reproduction of facial
configuration was achieved from the key landmarks.

Definition ofpoint locations:

The location and definition of facial points were preceded by the selection of the
facial features which contribute to the display of a facial expression. Each feature
was surrounded by point "markers" which had the effect of positioning and shaping
the feature for graphic representation on the line drawing. The eyes, eyebrows and
the mouth were included as the salient facial landmark points. The nose and outline
of the face were also represented with point markers, and added the finishing touch
for an acceptable line drawing. These point locations were considered to describe
accurately the position of the facial features. In this way, a detailed numeric record
of a photograph, consisting of the landmark points, with emphasis on position and
proportion of facial features could be generated.

101
 With the use of a lightpen and a transparency, fifty points were marked on to a
sensitive computer screen from a single source image (i.e. a photograph) by
concentrating on such areas as the mouth, the eyes, the nose, the eyebrows and the
outline of the face. These points were then joined using a sophisticated
mathematical technique called the B-spline method (Yamaguchi, 1978) to form the
basic structure of the face, that is, in line drawing format. The graphic presentation
of the point locations involved the establishment of a general drawing capability.
The B-spline technique produces a curved line drawing of the face, that is,
connecting of the landmark points. The programme that generates the curve is
simple, reasonably fast, and effectively connects a minimum number of facial points
to produce a smooth curved line drawing. Interestingly, Yamaguchi (1978) applied
his B-spline method to a face profile, demonstrating the smooth finish achieved by
the curve fitting technique.

Definition of the "muscloids ":

A third consideration in the development of the model was its capacity to function as
a true simulator of facial activity. In other words, it is possible that a mere graphical
reconstruction of a facial expression can be achieved by schematic faces or
caricatures, both of which are adequate for quantification procedures. The
mathematical model has the capacity to modify and manipulate the location of facial
landmarks on the line drawing. This is achieved by mathematically contracting or
relaxing the facial "muscloids". The facial muscloids are defined so as to represent
only one muscle action as needed, or the activity of a group of muscles in the
execution of a particular facial action . In essence, the modification of the line
drawing involves the manipulation of muscloid activity. With regard to the graphic
display of the face, muscloids are reduced to a straight line connecting two facial
landmarks. For example, one of the muscloids is defined as the movement of the
Risorius and Zygomaticus Major muscles (or the line joining points I and 16 in
Figure I) which have the effect of raising the outer corners of the mouth. The
information required for any given face simulation is constant as the input points are
common for all faces. The digitised facial landmarks do not change and so the only
variability expected is in the shape and proportion of the landmarks. Similarly, the
position of the facial muscles will be common to most faces, with possible variations
in the function of muscles, for example, in stroke or car accident victims. Hence, the
generation and operation of each face is point specific, a function of its common
features.

102
 6 •• .18

*
12
I~
626
5 9 1 7
24 2 8 28

71
66
69 19 20

~
51 I

65
50
79 29

64 14

Figure 1. Facial Points Locations

The model was first applied to generate expressions that fall along the happiness-
sadness dimension. The extremes of the axis were chosen from Ekman and Friesen's
(1975) documented representations of the facial configurations of these bipolar
emotions. The essence of the model was to provide a range of expressions from
happy to sad by manipulating the facial muscIoids of a targeted neutral expression.
Thornton and Pilowsky (1982) presented an example of the mathematical scaling of
facial expressions, using a photograph of a neutral expression as their baseline
measure. By contracting or relaxing the muscIoid groups around the mouth, the
model was capable of generating a diverse set of line drawing representations of
facial expressions. Furthermore, these simulated expressions may be used as cues in
facial recognition studies, or as an assessment of mood status by the patients
themselves, for example, when language may be a problem. Due to the anonymity
and simplicity of the line drawings and the loss of detailed identity information, a
standard set of data points are attributable to all faces.

103
 THE DEVELOPMENT OF FACEM

Thornton's original model was modified and extended with the early findings
reported in Pilowsky, Thornton and Stokes (1985, 1986). The mathematical model
was automated and adapted for use on a PC and FACEM emerged at that point.
Thornton used a lightpen and a transparency to mark the facial points on to a
computer screen. The new system works by encoding facial landmark points using a
graphics tablet. A photograph is placed on the graphics board and a pen is used to
digitise the facial points which are called up by the computer. To speed up the
digitising process, each facial point inputted into the system is followed by its mirror
image. For example, after the point locating the right outer corner of the mouth has
been digitised, the computer calls for the location of the point of the left corner of the
mouth. This saves time digiti sing by facilitating user memory of the location of the
facial points. The coordinate information is relayed to a PC which integrates the
information and produces a line drawing representation of the face.
The location of the facial points on the source image were determined after
extensive review of Ekman and Friesen's (1978) FACS manual and Izard's {I 979)
MAX coding system. The number of digitised points have increased to sixty-two,
with the addition of more detail to the lips and eyes, thus producing a more
aesthetically appealing line drawing (the points at the top of the head are digitised
but have been omitted in the process which connects the points to produce the line
drawing). Figure I displays the location of the facial points on a line drawing
representation of a real face, taken from a photograph of the Ekman and Friesen
(1975) collection. The use of landmark points provides an efficient and economical
method of encoding (for storage or computational processing) of the absolute and
relative positions and proportions of facial features. The source image can be a still
photograph. However, more often the source image is a photographed image of one
frame of videotape. The imaging system used is a thermal printer attached to a
video-monitor. In this way, a hard copy can be obtained from any piece of
videotaped material. This hard copy is then used as a source of input of the point
locations. The verification of point location occurs at the graphics stage when the
line drawing appears on the screen.

QUANTIFICATION OF FACIAL ACTION

Once the facial points have been located and digitised and the curve drawing
programme has connected all the points, the graphic procedure initiates a line
drawing representation of the source image. The anonymous line drawing of the

104
face displays the major facial landmarks, that is, the eyebrows, eyes, nose, mouth, as
well as the facial outline. It is at this point that the quantification of facial
expressions or facial movement is possible, with the establishment of twelve
computer generated facial measures each representing distances between two facial
landmarks. The point definitions have been refined and Euclidean distances between
two points are measured. The twelve computer generated facial measures consist of
eight vertical measures and four horizontal measures. The vertical measures can be
seen in Figure 2 and are described as follows: End-Lip Measure, Mouth-Opening
Measure, Mid-Top-Lip Measure, Mid-Lower-Lip Measure, Top-Lip Thickness
Measure, Lower-Lip Thickness Measure, Eye-Opening Measure and Mid-Eyebrow
Measure. The horizontal measures are Mouth-Width Measure, Top Eyelid/Iris
Intersect Measure, Lower Eyelid/Iris Intersect Measure and Inner-Eyebrow
Separation Measure (see Figure 3).

,9 ,
1:
I I
.-11-1
I ,
I r I ,
I I ' I

~ ~

~6~::
10

,
~
!
,,
,
,,!
, ,
,,
, ,
1_2_1

Figure 2. Vertical Measures Figure 3. Horizontal Measures

105
 STANDARDISATION OF THE FACIAL MEASURES

The distance values associated with each facial measure do not represent distances in
millimetres or centimetres between key points, but are better regarded as numerical
representations of those distances, expressed as a value within a hypothetical range,
reflecting the greatest possible excursion of those points on the face. Since each
facial measure has a unique minimum and maximum value, a scaling procedure is
applied which allows for the comparability of all facial measures making up the
profile of a particular facial expression. This scaling process involves the subtraction
of each of the minimum scores from its associated obtained distance measure,
multiplication by 100, and then division by the predetermined range value associated
with that facial measure. In this way, twelve scaled facial measures are produced.
Each scaled facial measure is then divided by one of two reference measures: the
distance between the outer canthi of the eyes for the horizontal measures and the
length of the nose for the vertical measures. This process compensates for head
movement during videotaping or the size of the source image on the graphics
terminal and allows for the direct comparison of facial expressions between subjects
(see Figure 4). Each measure can thus be defined in terms of the digitised points
which have to be joined in order to obtain a facial measure.

I • Horizontal ~,
I I
I I

-----+-
I I

~
-------t
I I

~LS::-:- ______J Vertical

Figure 4: Reference Measures.

106
 A limitation of the early work was the restricted array of muscle control points
assigned to simulate the activity of real facial muscles. Only three muscloids were
included on the line drawing and graphics procedure such that the expression
management was located and focused entirely around the activity of the mouth.
With the modification of the model, however, the twelve facial measures represent
the facial muscloid activity responsible for that localised movement. Using the
model, the relationship between the facial measures and the various pancultural
expressions can be explored further. In addition, a profile of the twelve measures is
produced and gives some indication of the associative relationship between each of
the measures. A further dynamic profile can be generated, which represents a
description of the activity of a single measure over time, that is, over several frames
of video, or alternately, the measurements obtained of a single muscloid action
during the display of several different expressions.
FACEM differs from such facial measurement systems as FACS, EMF ACS (see
Kaiser, this volume; see Frank, this volume) and facial electromyography in that
facial action is quantified in terms of distances between facial landmarks, rather than
actual muscle action as is characteristic of the latter techniques. Nevertheless,
FACEM findings accord well with those of Ekman and Friesen (1975) with regard to
mouth activity during smiling and with the data provided by Brown and Schwartz
(1980) who use electromyography to highlight the action of the zygomaticus muscle
(represented by the FACEM scores of End-Lip Measure, Mouth-Opening Measure
and Mouth-Width Measure) in the expression of happiness. These concordant
observations contribute to the concurrent validity of the facial measures. FACEM
also has several advantages over the aforementioned tools. Firstly, it is non-intrusive,
in that no sensors are physically attached to the encoder's face. Furthermore,
minimal training time is required in the digitisation of a face (approximately 15
minutes). In addition, the use of distances between facial points leads to a reduction
in the number of facial movements which are analysed hence saving time. Finally,
the line drawings allow the encoder to remain anonymous in terms of identity, age,
gender, race and social class. This has advantages in the experimental and clinical
context and when findings are reported.

RELIABILITY OF THE DIGITI SING PROCESS

The reliability of the digitising procedure was assessed in two ways: firstly, inter-
rater reliability with two independant judges and secondly, test-retest reliability of
the scores of one of the judges at two points in time. With regard to the resultant raw

107
and standardised facial measures, and where a reliability index was deemed
appropriate (i.e. where the distance measurements had sufficiently large enough
values to make reliability comparisons), inter-rater and test-retest performance were
extremely reliable. The correlations between raters ranged from 0.71 to 0.96, with a
median correlation of 0.89. Test-retest correlations ranged from 0.77 to 0.99, with a
median correlation of 0.95. For four of the twelve facial measures, distances
between landmark points were minute, leaving little scope for variation, and thus
reliability estimates were not considered. A final point that should be noted with
regard to the digitising procedure is that this task does not seem to require a great
exercise in judgement. When judges are instructed on the location of the facial
points and the resultant facial measurements to be made, their performance is
extremely reliable.

VALIDATION OF F ACEM

Smiling versus Neutral Expressions:

The focus on the quantification of smiling firstly stems from previous work which
has shown that the smile is the most easily recognised of all the fundamental
expressions (Drag & Shaw, 1967; Ekman, 1978; Thompson & Meltzer, 1964;
Wagner, Macdonald & Manstead, 1986), even in a deteriorated state of reproduction
(Wallbott, 1991). Secondly, in a laboratory setting, the smile is the easiest to evoke
and from an ethical point of view, the most readily justified. Finally, the
mathematical model of the face was originally validated using the smile expression
(Thornton & Pilowsky, 1982).
Two experiments were conducted to examine the relationship between the ratings
made of a set of smiling and neutral expressions and the facial features which
influenced these ratings. In the first study, judges were shown forty real face
photographs of smile and neutral expressions and forty line drawings derived from
these photographs and were asked to rate the degree of smiling behaviour of each
expression. Significant differences were found between the ratings of smile and
neutral expressions. The mode of presentation (i.e. real face versus line drawings)
did not contribute significantly to the ratings. Furthermore, five mouth measures and
one eye measure were found to discriminate significantly between the ratings made
on smile and neutral expressions.

108
 In the second study, subjects were shown eighty line drawing representations
only of smile and neutral expressions and were asked similarly to rate the degree of
smiling behaviour of each expression. The findings revealed that there was a greater
variation in ratings for the development of the smile than there was for the neutral
expression. Encoder differences emerged and gender differences in the decoding
abilities of the raters were also found (Katsikitis, Pilowsky & Innes, 1990, 1997).

Classification of Facial Expressions:

In the second phase of the validation process, a further three studies were carried out
to build on the previous findings and thereby extending the application of F ACEM to
include the quantification of a variety of different facial expressions.
There is general agreement amongst the categorical theorists as to the existence
of a finite set of fundamental facial emotions which are recognised cross-culturally.
The majority of researchers in this area would agree on the universality of the
following 6 expressions i.e. happiness, sadness, surprise, fear, anger and disgust.
This implies that there exist facial configurations which, within each class, resemble
each other closely, and may be readily distinguished from configurations forming
other classes of facial emotions. Faces displaying such emotions should therefore
lend themselves to objective classification and the classes should be characterised by
the key features of each fundamental expression. This section will present three
experiments which have been designed to examine this issue further.
The first study investigated whether a sample of posed facial expressions could
be classified into the discrete fundamental emotions, using F ACEM scores to
characterise the expression, and a numerical taxonomy package to classify them
(Pilowsky & Katsikitis, 1994a). Twenty-three first year drama students from the
Centre For The Performing Arts in Adelaide posed the six fundamental expressions
and these expressions were videorecorded. A neutral expression was also posed.
Facial measures on each of these expressive displays were obtained using F ACEM.
The FACEM scores were then analysed using a numerical taxonomy package called
"Snob". It is called Snob because it makes a distinction between classes. Snob seeks
to establish classes that are the best fit for a particular set of data. Snob provides its
own classification based on the likeness of attributes describing the members of
classes (Wallace, 1986; Wallace & Boulton, 1968). The result was not six classes
into which each of the posed fundamental expressions were allocated. Instead, the
numerical taxonomy procedure yielded five classes with only one being a "pure"

109
 expression class consisting of many of the smiling expressions, and a second was
"pure" only in the sense that faces showing smiles were completely absent.
The two studies that followed were judgement studies, sharing similar
methodology. There was one variation, however. Study two was a judgement study
of the photographs and in Study three, the judges were asked to classify line drawing
representations of the photographs. The aim of these studies was to see whether
judges could classify the same 160 facial expressions used in the numerical
taxonomy study (described above) into the six fundamental categories they were
intended to represent. In both of these studies, the result was that the judges were
unable as a group to categorise the photographs or the line drawings into the seven
mutually exclusive emotional classes (i.e. 6 fundamentals + neutral). In fact, only
happiness and to some extent surprise, were correctly placed. As the emotions were
not classified neatly into the seven well defined categories, the data were subjected to
a multidimensional scaling analysis (MDS) in an attempt to investigate the
underlying structure of these emotional states. The responses from the judges
resulted in a two-dimensional structure in both the photograph and line drawing
stimulus conditions, thus supporting our earlier findings (Katsikitis, 1992; Katsikitis,
Pilowsky & Innes, (990) and that of others (Calder, Young, Perrett, Etcoff &
Rowland, 1996; Etcoff & Magee, (992), that the mode of stimulus presentation, with
reference to line drawings or photographs, produces a similar outcome in terms of
judgement of the facial expressions of emotion. In both of the studies discussed
here, two dimensions emerged where the six fundamental emotions were positioned
along the perimeter of a circular model (Katsikitis,1997).
Thus, the results of this work extend the application of line drawings as a
stimulus source, beyond the smiling and neutral expressions, to the quantification of
other emotions such as anger, disgust, fear and surprise.

Clinical Applications ofFA CEM:

There are many applications to which a model such as FACEM could be applied, in
disciplines such as general medicine, anthropology and psychology. In recent years,
there has been an increase of interest in facial expression in clinical contexts
particularly in psychiatry and neurology (Bellack, Mueser, Wade, Sayers &
Morrison, 1992; Cramer, Bowen & O'Neill, 1992; Pilowsky & Bassett, 1980). Thus,
a natural progression in testing the validity of the model further, is to apply it to a
clinical issue. The application of FACEM has focused largely on 3 clinical areas,

110
that of Parkinson's disease, chronic and acute pain and Major Depression. More
recently, FACEM has been applied to schizophrenia.

PARKINSON'S DISEASE

Parkinson's disease (PD) is a common and disabling disease with major

consequences in terms of individual suffering, family impact and implications for
medical service provision. Although major advances have been made in
understanding the neuropathology and pharmacotherapy of PD, there remain certain
important psychosocial areas that have been relatively neglected in terms of research
and treatment. One of these is the facial immobility or the well known "mask-like
facies", so often cited as a characteristic clinical feature of PD. Using FACEM, three
separate studies have been conducted with patients suffering from PD and the results
of this work will be presented below.
In the first study, the amount of facial activity shown by PD patients during a
smile was compared to the smiling activity of a group of Normals who were matched
for age and sex (Katsikitis & Pilowsky, 1988). All subjects in this experiment were
asked to watch a series of funny cartoons and facial measurements were taken from
their smiles. On FACEM, there were significant differences in the degree of Mouth-
Opening during a smile and also in the frequency of smiling between PD patients
and Controls. PD patients were found not to open their mouths to the same extent as
Controls and produced fewer smiles whilst watching cartoons than Controls.
A follow-up study was carried out to compare facial activity of PD patients, with
Controls and with those patients suffering from Major Depression. There is some
speculation that the masked facies of PD resemble the lack of expression seen in
depressed people. For example, Trethowan (1979) has made an explicit comparison
between the facial expression of PD sufferers and those with depression noting that
in depression, "mobility of facial expression may be greatly reduced .... almost
parkinsonian-like" (p.175). The occurrence of depression in patients with PD has
also been reported (Mindham, 1970; Warburton, 1967). On the basis of clinical
evidence of this sort, the response of both conditions to treatments such as
electroconvulsive therapy (ECT) and tricyclic antidepressants, it has been suggested
that they may share some common neuropathological abnormalities (Serby, 1980).
All subjects were asked to watch the same series of cartoons as used in the
previous experiment and the degree of smiling activity was measured by FACEM.
The findings revealed significant differences between the groups in the degree of
smiling with the Depressed group having higher scores on 3 FACEM measures, that

111
 is, End-Lip Measure, Mid-Top-Lip Measure and Mid-Eyebrow Measure. The
Depressed group had higher scores on these three measures reflecting what is often
referred to colloquially as a long face. PO and Depressed subjects smiled less
frequently than Controls. Furthermore it was found that on a self-report depression
questionnaire, the depressed affect of patients with PO lies somewhere between that
of normals and patients with Major Depression (Katsikitis & Pilowsky, 1991).
Therefore it would not seem justified to equate the depression of parkinsonian
patients with that of clinically depressed patients. This study also showed that
FACEM was a useful tool for the quantification of the psychomotor functioning of
the face in this clinical setting.
Having shown that patients with Parkinson's disease do have difficulties moving
the muscles of the face, a third study was designed to examine the feasibility and
efficacy of implementing a physiotherapeutic regime, aimed at improving facial
muscle mobility in these individuals (Katsikitis & Pilowsky, 1997). FACEM was
used to measure any change in facial activity pre- and post-treatment. The treatment
involved oro-facial physiotherapeutic treatment. Members of the Treatment group
were seen by a physiotherapist for one hour, twice a week, over a period of four
weeks. Oro-facial techniques such as brushing muscles, applying ice to muscles and
general facial exercises were used to stimulate facial muscle activity. The Control
group had no treatment. Members of the Control group were seen at the pre-
treatment time of the Treatment group for baseline FACEM measures and then not
seen again until post-treatment, where FACEM measures were obtained.
The findings revealed several differences between the Treatment and Control
groups. Firstly, members of the Treatment group were found, after treatment, to
open their mouths to a greater extent than those in the Control group. Secondly,
comparison of pre-treatment and post-treatment scores also revealed significant
differences for the Mid-Top-Lip Measure for the Treatment group. No differences in
F ACEM were recorded in the Control group over a 4 week period. A third analysis
on the Treatment group only, was performed. to examine the degree of change in
facial distance scores from pre-treatment to follow-up. Follow-up FACEM scores
were taken a month after the completion of treatment. Results showed that the
differences achieved at post-treatment were maintained at follow-up for the Mouth-
Opening and Mid-Top-Lip Measures. Thus, the approach to quantification that has
been described here has implications for clinicians. FACEM scores may be used in
the planning and monitoring of treatment regimes aimed at improving the patients'
capacity to communicate emotional states J •

112
 PAIN

There has been a recent surge of interest in the facial expressions associated with
pain facilitated by the use of facial measurement systems, in particular, the Facial
Action Coding System developed by Ekman and Friesen (1978) which provides an
objective method of measuring expression directly from the face of an individual.
F ACEM appears to offer a more time-efficient and valid approach to facial
expression quantification. However, in the area of chronic pain, the findings
produced by F ACEM are in accord with others showing that the chronic pain
expression is characterised by a wider, more open mouth, lowered eyebrows and
eyes with a narrow palpebral fissure (Pilowsky & Katsikitis, I 994b).
FACEM has also been applied in a study involving the display of the acute pain
expression. Pregnant women were contacted through antenatal clinics at a large
general teaching hospital in Adelaide. As the women commenced labour and arrived
at the delivery suite, the researcher was telephoned and arrived to set up the camera.
The camera remained on the face of the subject for the duration of first and second
stage labour. Fifteen women agreed to participate in this study and during labour,
they were asked to rate their pain intensity every 15 minutes using a visual analogue
scale (V AS). Midwives, also using the V AS, were asked to rate what they perceived
to be the patient's pain intensity level at the time. Still photographic images were
captured from the videotapes when VAS scores were rated between 3 and 7 (on a 10
point scale) and these ratings were considered to be representative of mild-moderate
pain, and when VAS scores were from 7 to 10, where the pain was considered to be
severe. Therefore, a woman who was videotaped from the beginning of her labour
was likely to score pain levels in both the mild-moderate and severe categories.
FACEM scores from the digitised images showed that Mid-Lower-Lip Measure was
significantly higher during severe pain than during mild-moderate pain levels. There
were changes in the facial measures consistent with pain, that is, lowered eyebrows,
closed eyes and stretched mouth, at relatively mild levels of pain, but only Mid-
Lower-Lip Measure differentiated further with the transition from mild/moderate
pain to severe pain. The results suggest that while subjects demonstrate changes in
facial expression consistent with pain, the changes are on the whole, not significantly
different between the two pain levels studied here. That is, the changes in facial
measures appears to occur at relatively low levels of pain and do not discriminate
further with the transition from mild-moderate to severe pain. Furthermore, as a
group, the midwives tended to be reliable estimators of mild-moderate pain but not
of severe pain (Baker, Katsikitis & Dawson, 200 I).

113
 DEPRESSION

The third clinical area in which FACEM has been applied has been in measuring
depression. One particular study assessed the efficacy of transcranial magnetic
stimulation (TMS) for depression enlisting both F ACEM and standardised
psychometric tests as objective measures of mood status. Nine patients diagnosed
with a major depressive disorder with melancholia received 10-14 treatments with
TMS over a 12-16 day period. Interviews with each patient before the first TMS
session and after the last treatment with TMS were videotaped. Patients were
assessed before and after treatment with the (I) CORE system, that is, a rating of
psychomotor disturbance in depressed patients measuring melancholia; (2)
Montgomery-Asberg Depression Rating Scale and (3) the Zung self rating
depression scale. All subjects made significant improvement, at discharge, as
measured by the improvement in these 3 scores. Twenty random still images were
obtained from each videotape, separately, both at the first TMS treatment and at the
final TMS treatment. These images were analysed using FACEM and Mouth-Width
Measure, Inner-Eyebrow Separation Measure and Mid-Eyebrow Measure were
significantly lower before TMS treatment compared to post-treatment. This suggests
that the action of the corrugator muscle and the zygomaticus muscle are important
features in the nonverbal expression of depression. Furthermore, the efficacy of
TMS as a treatment for depression shows promise and needs further investigation
(Katsikitis, Pridmore & Marzullo, 1999).

SCHIZOPHRENIA

In the latter half of last century, a myriad of research emerged showing that the
performance of individuals diagnosed with schizophrenia, was impaired with regard
to their recognition of facial expression, as compared to normal controls (Dougherty
& Izard, 1974; Leventhal & Scherer, 1987, Mandai, Pandey & Prasad, 1998;
Pilowsky & Bassett, 1980). The debate is still active in the literature however
regarding the nature of this recognition problem, ie is the impaired recognition of
facial emotion the result of a generalised performance deficit? or is it an information
processing error?; or a specific failure in emotion identification?
Most recently, Johnston and Katsikitis (2001) trained a neural network model to
delineate the so-called fundamental facial expressions (happiness, sadness, anger,
disgust, fear and surprise) on the basis of the 12 FACEM distance scores, to a level

114
of classification accuracy comparable to that of healthy subjects. The network was
systematically lesioned and the result was a marked decrease in the model's capacity
to discriminate amongst the emotions. Discrimination of the negative emotions of
fear, disgust, anger and sadness was particularly compromised. Thus, the resultant
performance due to the degrading of the network resembled the outcomes achieved
for individuals with schizophrenia, on such a task. These findings give added
support to the assertion that this group have a more generalised as opposed to
specific deficit in information processing capability, especially if there is a disparity
in task difficulty levels, in this case, positive emotions were recognised more readily
as compared with the negative expressions. This theory remains to be tested in
normal and clinical populations.

CONCLUSION
This chapter describes a facial expression measurement programme called FACEM.
FACEM operates to quantify facial movement, in particular, the quantification of
facial expression and is based on a mathematical model of the face. This chapter has
presented the history of the development of FACEM and the process of validation
that followed. The following characteristics of FACEM have been emphasised.
(1) The model is able to produce facial measures which represent the distance
between selected facial landmarks, thus making possible the investigation of
their relationship to individual facial expressions.
(2) Line drawing representations can be produced from a photograph or still
image of the face, thus rendering the stimulus source anonymous with
regard to age, sex, culture, and social class.
(3) Information from both sides of the face can be obtained independently
(4) The methodology is non-intrusive.
(5) Minimal training time is requires in the digitisation of the face.
(6) The model is capable of tracking the dynamics of any facial measure over
time.
(7) There is high reliability of quantification.
(8) The model has been demonstrated as suitable for use in a clinical context.
This chapter has focused on an integration of the findings produced by utilising
FACEM in a number of different contexts. The implications for the utility of the
model have been discussed and it is clear that this method has considerable potential
for use in many settings. This chapter has listed a few of the possibilities.

115
 NOTES
I I am indebted to Professor Issy Pilowsky, Head of Department of Psychiatry from 1972-1997 who

encouraged and supported my work with FACEM.

2 I would Iike to acknowledge the support of Mark Thornton for his role in the early development of
FACEM.
; A follow-up study (Katsikitis, 2001) has recently been completed and the findings reported at the recent
FACS conference, Innsbruck, Austria, September, 2001.

AFFILIATION

Department of Psychiatry
University of Adelaide
Australia
Email: m.katsikitis@psychsociety.com.au

REFERENCES
Baker, A., Katsikitis, M., & Dawson, D. The faces of childbirth. (submilled).
Bellack, A.S., Mueser, K.T., Wade, 1., Sayers, S., & Morrison, R.L. (1992). The ability of schizophrenics
to perceive and cope with negative affect. The British Journal of Psychiatry, 160, 473-480.
Brown, S.L., & Schwartz, G.E. (1980). Relationships between facial electromyography and subjective
experience during affective imagery. Biological Psychology, 11,49-62.
Cramer, P, Bowen, J., & O'Neill, M. (1992). Schizophrenics and social judgement. Why do
schizophrenics get it wrong? The British Journal of Psychiatry, 160,481-487.
Calder, A.1., Young, A.W., Perrett, 01., Etcoff, N.L., & Rowland, D. (1996). Categorical perception of
morphed facial expressions. Visual Cognition, 3, 81-117.
Dougherty, F.E., & Izard, CE. (1974). Responses of schizophrenics to expressions of the fundamental
emotions. Journal ofClinical Psychology, 30,243-246.
Drag, R.M., & Shaw, M.E. (1967). Factors influencing the communication of emotional intent by facial
expressions. Psychonomic &ience, 8, 137-138.
Ekman, P. (1982). Methods of measuring facial action.(1982). In K.R. Scherer & P. Ekman (Eds),
Handbook of methods in nonverbal behavior research (pp. 45-90). Cambridge: Cambridge:
University Press.
Ekman, P., & Friesen, W. V. (1975). Unmasking the face. New Jersey: Prentice-Hall Inc,
Ekman, P., & Friesen, W.V . (1978). Facial Action Coding System (FACS): A technique for the
measurement offacial action. Palo Alto: Consulting Psychologists Press.
Ekman, P., Friesen, W. V., & Ellsworth, P. (1972). Emotions in the human face. Guidelines for research
and integration offindings. New York: Pergamon Press.
Ekman, P, Friesen, WV., & Ellsworth, P. (1982). Methodological decisions. In P. Ekman (Ed), Emotion
in the human face (pp. 56-97). Cambridge: Cambridge University Press.
Ekman, P, Friesen, W. V., & Tomkins, S.S . (1971). Facial Affect Scoring Technique (FAST): A first
validity study. Semiotica, 3,37-58.
Etcoff, N.L., & Magee, J.J. (1992). Categorical perception offacial expressions. Cognition, 44,227-240.

116
Frank, M. G. (2002). Getting to know your patient: How facial expression can help reveal true emotion.
In M. Katsikitis (Ed), The human face: Measurement and meaning (pp. 255-283). Dordrecht: Kluwer
Academic Publishers.
Izard, C. E. (1979). The Maximally Discriminative Facial Movement Coding System (MAX).
Unpublished Manuscript. Available From [nstructional Resource Center, University of Delaware,
Newark, Delaware.
Johnston, P., & Katsikitis, M. (200 I). A generalised deficit can account for problems in facial emotion
recognition in schizophrenia. Biological Psychology, 58,203-227.
Kaiser, S. (2002). Facial expressions as indicators of "functional" and "dysfunctional" emotional
processes. [n M. Katsikitis (Ed), The human face; Measurement and meaning (pp. 235-253).
Dordrecht: Kluwer Academic Publishers.
Katsikitis, M. (1992). The quantification of facial expression using a mathematical model of the face.
Validation and extension of a microcomputer-based technique. Unpublished dissertation, University
of Adelaide, Adelaide, South Australia.
Katsikitis, M. (1997). The classification of facial expressions of emotion: A mulidimensional-scaling
approach. Perception, 26, 613-626.
Katsikitis, M. (2001). Does oro-facial stimulation improve facial mobility in Parkinson's disease? Paper
presented at the 9· h European Conference on Facial Expression, Measurement and Meaning (FACS).
September, 200 I, InnsbrOck, Austria.
Katsikitis, M., & Pilowsky, I. (1988). A study of facial expression in parkinson's disease using a novel
microcomputer-based method. Journal of Neurology, Neurosurgery and Psychiatry, 51, 362-366,
Katsikitis, M., & Pilowsky, I. (1991). A controlled quantitative study of facial expression in Parkinson's
disease and depression. Journal of Nervous and Mental Disease, 179, 683-688.
Katsikitis, M., & Pilowsky, I. (1997). A controlled study of facial mobility treatment in Parkinson's
disease. Journal of Psychosomatic Research, 40,387-396.
Katsikitis, M., Pilowsky, I., & Innes, J.M. (1990). The quantification of smiling using a microcomputer-
based approach. Journal of Nonverbal Behavior, 14,3-17.
Katsikitis, M., Pilowsky, I., & Innes, J.M. (1997). Encoding and decoding of facial expression. Journal
of General Psychology, 124, 357-370.
Katsikitis, M., Pridmore, S., & Marzullo, M. (1999). The Facial Expression Measurement system in the
assessment of the efficacy of transcranial magnetic stimulation in the treatment of depression.
European Review of Applied Psychology, 49, 123-128.
Leventahl, H., & Scherer, K. (1987). The relationship of emotion to cognition: A functional approach to
a semantic controversy. Cognition and Emotion, 1,2-28.
Mandai, M., Pandey, R., & Prasad, A. (1998). Facial expressions of emotion and schizophrenia- a
review. Schizophrenia Bulletin, 24, 399-412.
Minham, R.I-I.S. (1970). Psychiatric symptoms on parkinsonism. Journal of Neurology, Neurosurgery
and Psychiatry, 33,188-191.
Pilowsky, I., & Bassett, D. (1980). Schizophrenia and response to facial emotions. Comprehensive
Psychiatry, I, 236-244.
Pilowsky, I., & Katsikitis, M. (1994a). The classification of facial emotions: A computer-based
taxonomic approach. Journal of Affective Disorders, 30, 61-71.
Pilowsky, I., & Katsikitis, M. (1994b). The quantification of facial expression in pain and depression: A
preliminary study The Pain Clinic, 7, 179-184.
Pilowsky, I., Thornton, M., & Stokes, B. (1985) A microcomputer-based approach to the quantification
of facial expressions. Australasian Physical and Engineering Sciences in Medicine, 8, 70-75.

117
Pilowsky, I., Thornton, M., & Stokes, B. (1986). Towards the quantification of facial expressions with
the use ofa mathematical model of the face. In H.D. Ellis, M.A. Jeeves, F. Newcombe, & A. Young
(Eds), Aspects offace processing (pp. 340-348). Dordrecht: Martinus NijhoffPublishers.
Serby, M. (1980). Psychiatric issues in Parkinson's disease. Comprehensive Psychiatry, 21, 317-322.
Thompson, D.F., & Meltzer, L. (1964). Communication of emotion intent by facial expression. Journal
ofAbnormal and Social Psychology, 68, 129-135.
Thornton, M. (1979). The mathematical modelling of facial expressions. Unpublished dissertation,
University of Adelaide, Adelaide, South Australia.
Thornton, M., & Pilowsky, I. (1982). Facial expressions can be modelled mathematically. British
Journal of Psychiatry, 140,61-63.
Trethowan, W.H. (1979). Psychiatry. London: Cassell.
Wagner, H.L., MacDonald, C.J., & Manstead, A.S.R. (1986). Communication of individual emotions by
spontaneous facial expressions. Journal of Personality and Social Psychology, 50, 737-743.
Wallace, C.S. (1986). An improved program for classification. Paper Presented At The Austral ian
Computer Society Conference, Canberra, ACT, Australia, 1986.
Wallace, C.S., & Boulton, D.M. (1968). An information measure for classification. Computer Journal,
11,185-194.
Wallbott, H. (1991). The robustness of communication of emotion via facial expression: Emotion
recognition from photographs with deteriorated pictorial qual ity. European Journal of Social
Psychology, 21, 89-98.
Warburton, 1. W. (1967). Depressive symptoms in parkinsonian patients referred for thalatomy. Journal
of Neurology, Neurosurgery and Psychiatry, 30, 268-270.
Yamaguchi, F. (1978). A new curve fitting method using a CRT computer display. Computer Graphics
and Image Processing, 7, 425-437.

118
 PATRICK JOHNSTON AND VAUGHAN CARR

CHAPTER 6

FACIAL AFFECT RECOGNITION DEFICITS IN

SCHIZOPHRENIA: A CASE FOR APPLYING FACIAL
MEASUREMENT TECHNIQUES

INTRODUCTION

Displays of facial emotion play a major role in human social interaction relaying
important information regarding mood, engagement and social intent. This channel
of information exchange has been shown to be susceptible to breakdown in a
number of psychiatric conditions including schizophrenia, depression and anxiety
disorder. In such conditions problems have been demonstrated as occurring both in
generating affect appropriate facial expressions and in correctly recognising the
emotional facial expressions of other individuals. These phenomena have been
widely researched over recent decades for a number of reasons. First, it has been
suggested that they may have important clinical consequences (eg. misreading social
cues may contribute to social avoidance, delusion formation etc.). It is particularly
ironic that emotional expression and recognition problems occur in clinical groups
for whom increased social engagement may be particularly efficacious. Second, if
dissociations exist between patients' performance on facial emotion processing and
other related cognitive functions (ie. recognition of facial identity; categorisation of
other complex visual stimuli etc.), this could offer a valuable window into the
relationship between cognitive and emotional aspects of psychological functioning
(Mandai, Pandey & Prasad, 1998; Leventhal & Scherer, 1987). The scope of
research investigating facial emotion encoding and decoding deficits in any of the
psychiatric conditions mentioned would merit a chapter in their own right; here,
discussion will focus upon facial emotion recognition deficits in schizophrenia and
its theoretical and clinical implications, considering other conditions only in as much
as they can elucidate our interpretations of research findings pertaining to the
deficits observed in schizophrenia patients. A body of literature also exists
regarding impairments experienced by schizophrenia patients in expressing
appropriate facial emotions. For instance, patients display less congruence between
their verbal and facial affective messages than do healthy subjects (Gottheil,
Thornton & Exline, 1976), and show reduced responsivity of facial action units
(Schneider, Heimann, Himer, Huss, Mattes, & Adams, 1990). However, as such
studies are relatively few, and have generally examined a restricted number of
emotional valences, they will not be considered in detail here, as to date there is
insufficient data regarding possible relationships between encoding and decoding
deficits.
Over the last three decades a wealth of data has emerged pertaining to the ability
of patients with schizophrenia to identify emotions. The following review is
restricted to studies where the recognition of facial expression of emotion has been
the primary aim of the experimental method. Overall, these studies have shown that
schizophrenia patients are impaired in this domain. However, debate continues
regarding the scope and nature of this deficit, as well as its interpretation.
Moreover, any discussion of research findings in this area warrants a cautionary
note: integration of the findings is problematic as the differences between studies
have often been greater than their similarities (Kerr & Neale, 1993).
Differences occur in subject and stimulus-related variables, and methodological
approach. Stage of illness, medication status, duration of illness, clinical subtyping,
age and composition of clinical and control groups have all varied. Studies have
also differed in terms of the emotions that have been examined, both in number and
valence. For example, some studies have considered only happy, sad and neutral
(Mandai, Borod, Asthana, Mohanty, Mohanty, & Koff, I 999b); others have
examined seven universally recognised emotions (Bryson, Bell, & LysakerI997);
still others have included less well recognised emotions such as shame, curiosity, etc
(Walker, Marwit, & Emory, 1980). Methodological variations across studies have
spanned a wide spectrum. These include posed versus natural emotional stimuli,
duration of stimulus presentation, still versus motion stimuli, response requirements
(eg, free response, forced-choice, match/non-match), and whether or not a
differential deficit design (Chapman & Chapman, 1978) was used.
Further variations include use of other neuropsychological measures as covariates
and examination of performance differences in relation to emotional valence.
Despite these problems, studies using differential deficit designs and distinct
categories of emotion can give a relatively coherent picture of the deficit. The main
issue is how this deficit may be interpreted.

EARL Y RESEARCH IN EMOTION RECOGNITION

Early studies established that patients with schizophrenia had a deficit in the
recognition of facial emotion (Dougherty, Bartlett & Izard, 1974; Muzekari & Bates,
1977; Shannon, 1970). Although the results were relatively consistent,

120
interpretation is actually difficult. The major criticism has been the failure to adopt
a differential deficit design (Chapman & Chapman, 1978). Because there have been
no comparisons between the tasks used in these studies and others involving similar
or related cognitive processes, it is difficult to draw conclusions regarding the
specificity of the observed deficit (Kerr & Neale, 1993). More recent research has
attempted to address this issue by including other facial and other emotion
processing tasks, and/or additional measures of specific cognitive functions.
Notwithstanding these difficulties, the findings of the early studies cannot be
dismissed. Several have shown impairments in the recognition of particular facial
emotions. For instance, Dougherty, Bartlett and Izard (1974) used forced choice and
free response tasks (emotion test stimuli: joy, anger, sadness, disgust, fear, surprise,
interest and shame) and showed that schizophrenia patients performed less well than
controls for disgust and shame. Schizophrenia patients and healthy controls were
virtually identical in their ability to recognise joy correctly. Muzekari and Bates
(1977) used the same methodology (testing happiness, sadness, anger and fear) and
also found no differences between controls and schizophrenia patients for happiness,
but impaired recognition of all three negative emotions tested in schizophrenia.
Shannon (1970) used the Brief Affect Recognition Test (testing happiness, sadness,
fear, anger, disgust, and surprise) and found that schizophrenia patients performed
worse than depressed patients and healthy controls in recognising fear, anger and
disgust, but not happiness, sadness and surprise. The last of the early studies
(testing happiness, curiosity, surprise, shame, disgust, sadness, fear and anger)
showed that controls were significantly better than schizophrenia patients on the
identification of all emotions (Walker et aI1980).
In these studies the pattern of accuracy, when reported, showed a very similar
profile in patients and controls. That is, both patients and controls showed greatest
accuracy in recognising happiness and least accuracy in the recognition of fear and
disgust (Dougherty et ai, 1974; Walker et ai, 1980). This result has been replicated
in more sophisticated studies (Archer, Hay & Young, 1992; Bryson et al 1997;
Schneider, Gur, Gur, & Shtasel, 1995). Although the rank ordering of accuracy
measures in emotion recognition has differed between studies, possibly reflecting
differences in stimulus materials, in most cases there are strong consistencies
between controls and schizophrenia patients in terms of relative performance to
specific emotions.

DIFFERENTIAL DEFICIT DESIGN STUDIES

Most of this research has used facial recognition as a control task (Addington &
Addington, 1998; Salem, Kring & Kerr, 1996; Novic, Luchins, & Perline, 1984), but
some studies have also used age discrimination based on face stimuli (Gessler,
Cutting, Frith, & Weinman, 1989; Habel, Gur, MandaI, Salloum, Gur, & Schneider,
2000; Schneider et ai, 1995). At least one study has addressed the hypothesis that
the deficit may be an impairment in labelling rather than discrimination (Walker,
Maguire, & Bettes, 1984). Emotion matching or emotion discrimination tasks have
often been used in addition to emotion recognition tasks. Kerr and Neale (1993)
found no evidence for an emotion specific deficit in tests using facial and vocal

121
emotion identity and discrimination, and in tests of facial identity recognition and
speech perception, with schizophrenia patients performing worse than controls on all
tasks. Similarly, when facial identity performance was used as a covariate in the
analysis of emotion recognition no differences emerged between patients and
controls (Novic et ai, 1984). Other studies comparing emotion identification with
other facial recognition and identification tasks (Feinberg, 1986; Salem & Kring,
1996; Walker et ai, 1984) all confirm a pattern of poor performance for
schizophrenia patients across this range of tasks.
Using happy/sad and old/young dichotomous judgements on tasks matched for
discriminability and difficulty, Gessler et al {I 989) found that acute schizophrenia
patients performed worse than controls on both tasks, but both groups displayed a
bias towards 'happy' and 'young' stimuli. Archer, Hay and Young (1992) compared
performance on tasks of emotion recognition, facial recognition, and unfamiliar face
matching (all forced choice paradigms) and found a deficit in schizophrenia patients
on all tasks. This study also compared recognition performance, with schizophrenia
patients showing greater accuracy to happy faces than surprised, angry or disgusted
faces.

ASSOCIA TlONS WITH DIAGNOSTIC AND NEUROCOGNITIVE MEASURES

Relationships between performance on emotion recognition and tests of cognitive

function, and between face processing performance and diagnostic or symptom
measures have been explored. For example, Kee, Kern, and Green (1998) found a
significant correlation between Span of Apprehension and emotion recognition
performance. Addington & Addington (1998) used tests of affect recognition, affect
identification and facial recognition as well as a Continuous Performance Task and a
Digit-Span Task. Schizophrenia patients performed poorly on all of the face tasks,
and there were significant correlations between task performance on affect
recognition and affect identification and both the neurocognitive tests. There was
also a significant correlation between negative symptoms and facial recognition
performance.
Schneider et al (1995) reported negative associations in schizophrenia between
emotion discrimination performance and symptom severity, as well as negative
associations with measures of abstraction, memory, language and spatial skills. In
addition, the schizophrenia patients performed worse than healthy controls on age
discrimination. Heimberg, Gur, Erwin, & Shtasel (1992) also reported an
association between symptom severity and poor emotion discrimination
performance.
Such deficits are not confined to schizophrenia in that some studies report no
differences between schizophrenia patients and neurotic or depressed patients in
performance on emotion discrimination or recognition tasks (Archer et ai, 1992;
Pilowsky & Bassett, 1980; Ferdowsi, 1983). However, these studies do report
schizophrenia patients having a greater severity of deficit than other clinical groups,
the latter also displaying poorer performance than healthy controls on these tasks.
Moreover, there is some evidence that other clinical groups show a pattern of deficit
to specific emotions that is comparable to that observed in schizophrenia patients.

122
For instance, Zuroff and Colussy (1986) reported that both depressed and
schizophrenia patients were impaired relative to healthy subjects in their recognition
of negative, but not positive emotions.
In patients with focal frontal, parietal and temporal lesions, happiness has been
reported to be most accurately recognised and both fear and anger least accurately
recognised (Mandai, Jain, Haque-Nizamie, Weiss, & Schneider, 1999a). In this
study no relationship was found between lesion localisation and performance.
However there was a significant interaction between lesion side and emotional
valence, patients with right hemisphere lesions performing more poorly for negative
emotions. Since this study did not include a healthy control group, it is not clear
whether differences exist between left hemisphere lesioned patients and normal
subjects. However, patients with right hemisphere lesions performed poorly
specifically to negative emotions, irrespective of the localisation of the lesion in that
hemisphere.

THE FUNCTIONAL NEURAL CIRCUITRY OF FACE PROCESSING

In normal subjects several cortical and subcortical regions are involved in the
processing of face stimuli. Selective impairments in a person's ability to recognise
faces (prosopagnosia) have been well documented. Autopsy studies have shown
that individuals who suffer from prosopagnosia tend to have bilateral damage to the
inferior occipitotemporal regions, particularly the fusiform and lingual gyri
(Damasio et ai, 1990). More recent functional neuroimaging and event related
potential (ERP) studies have confirmed that face selective activity occurs in these
regions (Halgren, Dale, Sereno, Tootell, Marinkovic, & Rosen, 1999; Swithenby,
Bailey, Brautigam, Josephs, Jousmaki, & Tesche, 1998; Puce, Allison, Asgari, Gore,
& McCarthy, 1997) with greater activation in the right hemisphere than in the left.
There is a growing consensus that the regions identified represent specialised areas
for face processing, although there is disagreement as to whether this specialisation
is innate or represents an experience based SUb-specialisation within a region which
deals with complex visual stimuli. In addition, selective involvement of other brain
regions, notably the left prefrontal cortex (Tovee, 1998; Phillips, Williams, Senior,
Bullmore, Brammer, Andrew, Williams, & David, 1998) and the amygdala (Morris,
Friston, Buchel, Frith, Young, Calder & Dolan, 1996), has been reported.
Regional brain activation in relation to differing facial expressions has also been
studied. Deficits in the recognition of fearful face stimuli have been observed in
patients with amygdala damage (Young, Aggleton, Hellawell, Johnson, Broks &
Hanley, 1995). An tMRI study (Sprengelmeyer, Rausch, Eysel, & Przuntek, 1998)
found increased activation to fearful faces in the fusiform gyrus and left dorsolateral
frontal cortex, to expressions of disgust in the right putamen and insula cortex, and
to anger in the right fusiform and left medial temporal gyrus. This study also
demonstrated an increased prefrontal response to all three emotional face types
compared to emotionally neutral faces. PET studies have found increased activation
in the left ventral prefrontal cortex and right fusiform gyrus to positive facial
emotions (Dolan, Fletcher, Morris, Kapur, Deakin, & Frith, 1996), and greater
amygdala activation to fearful faces which varied as a function of emotional

123
intensity (Morris et ai, 1996). These results have contributed to the view that
different neural circuits may sub serve the recognition of different facial emotions.
A number of studies have looked at differences in neural activation in
schizophrenia patients during facial affect recognition tasks. An fMRI study,
(Phillips, Williams, Senior, Bullmore, Brammer, Andrew, Williams, & David, 1999)
reported general reductions in activation amongst patients in brain regions
previously associated with facial emotion processing. Non-paranoid patients tended
to mistake expressions of disgust for either fear or anger, and showed greater
amydala activation to expressions of disgust than did paranoid patients. However,
these results were based upon very small numbers of patients in each cohort (ie. 5
paranoid, 5 non-paranoid). An MEG study by Streit, Ioannides, Sinnemann,
Wolwer, Dammers, Zilles, & Gaebel, (2001) revealed that patients generated weaker
activations than controls in inferior prefrontal, temporal, occipital, and inferior
parietal areas at circumscribed latencies. Behavioral performance was associated
with strength of activation in inferior prefrontal areas, the right posterior fusiform
gyrus region, right anterior temporal cortex, and the right inferiorparietal cortex.

INTERPRETATION
On balance, the evidence supports the notion that people with schizophrenia display
a deficit in the recognition of facial emotion. This is partly related to a more general
problem in cognitive functions including categorisation, discrimination and
identification of facial stimuli, and partly to deficits in working memory and
attentional processes. Secondly, relationship of the deficits to illness severity is
supported, particularly the severity of negative symptoms. Moreover, patients with
other psychiatric conditions also display degrees of the same deficit. Finally,
schizophrenia patients have a greater deficit in their ability to identify or
discriminate negative facial emotions (particularly fear, disgust, and anger) while
their ability to identify happy facial stimuli approaches normal levels.
There are two main explanations for the observed deficit in schizophrenia: one is
that people with schizophrenia have a generalised performance, or perceptual deficit
(Archer et ai, 1992); the other, proposes an emotion specific deficit (Borod, Martin,
Alpert, Brozgold, & Welkowitz, 1993). Traditionally, the notion of an emotion
specific deficit referred to a general problem in processing emotional material.
More recently proponents of an emotion specific deficit have focussed upon the
possibility of a "negative emotion specific deficit" (MandaI et ai, 1998; Philips et al
\999) arising through either cognitively mediated avoidance of negative stimuli
leading to a reduced acuity for processing such information (the social-cognitive
view), or through aberrant localised neuronal processing in brain regions specifically
subserving negative emotion recognition. Evidence for a specific negative emotion
deficit in schizophrenia includes: poorer performance in tasks of facial emotion
recognition than in other forms of facial identification, discrimination and matching
tasks; and poorer performance in the recognition of specific negative emotions
compared to positive and neutral emotions (Borod 1993, MandaI et al 1998; Phillips
et ai, 1999). Functional neuroimaging studies show different structures activated in
relation to different facial emotions, suggesting that recognition of different facial

124
emotions is subserved by distinct neural pathways. Moreover, the poor performance
specific to negative emotions by schizophrenia patients has been interpreted to
reflect localised aberrant neural functioning in these pathways (Phillips et ai, 1999).
However, despite the recent speculation regarding the existence of a negative
emotion specific deficit in schizophrenia, such an interpretation is not strongly
supported by the available evidence. Firstly, it is not clear that a differential deficit
has been convincingly demonstrated. This is particularly so in regard to studies that
have shown differential patterns of performance in recognition of different
emotional valences. No serious attempt has been made to ensure equal
discrimination difficulty across different emotional categories, and, in many studies
there is clear evidence of healthy control subjects showing a similar pattern of
relative performance to different stimulus categories as that shown by schizophrenia
patients. Healthy controls display greatest accuracy to happy faces, and relatively
poorer performance to negative emotion faces. Further, in cross-cultural facial
expression recognition studies (ie. Habel et ai, 2000), there is evidence of poorer
performance in negative emotion categorisation where cross-cultural facial emotion
judgements are made. This strongly implies an a priori task difficulty imbalance
which has not to date been taken into account in facial emotion recognition studies.
Secondly, claims of functional neuroimaging data supporting the notion of a
specific negative emotion deficit must be treated with some caution. It is not clear
to what extent research in this area distinguishes between emotions recognised and
emotions evoked in response to stimuli. In fact a number of studies have used
emotional face stimuli with the explicit intention of evoking different emotional
responses in subjects (ie. Lane, Reiman, Bradley, Lang, Ahem, Davidson, &
Schwartz, 1997; Schneider, Grodd, Weiss, Klose, Mayer, Nagele, & Gur, 1997).
These studies have shown activation to different emotions broadly similar to those
seen using facial emotion recognition tasks. In these studies the activation of areas
such as the amygdala, frontal and occipito-temporal gyrus has been interpreted as
indexing emotional responses to the stimuli, rather than recognition processes. Until
there is direct evidence that specific neural activation is directly linked to
recognition processes rather than the experience of emotion, speculation regarding
the meaning of low levels of activation in clinical cohorts should be viewed with
appropriate skepticism.
At present, therefore, evidence favours the notion that the facial expression
recognition deficit observed in schizophrenia represents a more general perceptual
or performance deficit, rather than an emotion specific deficit. However, it remains
possible that methodological problems have marred attempts to show evidence of an
emotion specific deficit. The resolution of this debate is by no means merely
academic, as it may have profound implications for the types of cognitive
rehabilitative techniques that are likely to prove useful.

THE CLINICAL RELEVANCE OF FACIAL EMOTION RECOGNITION

DEFICITS IN SCHIZOPHRENIA
In the case of many of the neurocognitive tasks on which patients with
schizophrenia display performance deficits (ie. Wisconsin Card Sorting, Stroop

125
Task, Continuous Performance Tasks, Trail Making Tasks), the tasks are sufficiently
abstracted that speculation regarding the day to day consequences of such deficits
remains opaque. The case of facial emotion recognition tasks would appear to be
different, in that there is an obvious sense in which the task is likely to possess some
degree of ecological validity. We all make judgements regarding the mood or
emotion states of others based in part upon their facial expressions on a daily basis.
It may be argued that the types of stimuli typically used in the majority of facial
expression recognition paradigms (ie. static facial images of posed emotions) are
unrealistic. However, healthy subjects perform the task generally with a high level
of confidence and accuracy. Moreover, we routinely have a great deal of exposure
to static images of people's faces in newspapers and magazines in our daily lives,
and generally feel no conscious sense of impropriety or falseness in ascribing
emotional states to the individuals depicted. In short, unconscious mechanisms
accept that such a static image provides us with sufficient information to make such
a judgement. It would therefore seem reasonable to assume that the experimentally
observed deficit in schizophrenia patients might translate into specific difficulties or
erroneous judgements in real world situations, (even if these were only to be in
correctly identifying facial emotion in magazine or newspaper pictures).
Intuitively it seems that there is a clear link between facial emotion decoding and
social interaction. Patients with facial immobility problems often report that their
inability to adequately express facial emotions has a detrimental affect on their
social relationships (Katsikitis & Pilowsky, \996). Patients with schizophrenia often
experience problems in social relationships and reduction in their level of social
competence with concomitant social withdrawal. A deficit in facial affect
recognition could be a contributing factor to these phenomena. Misinterpretations of
facial affect could also lead to incorrect evaluations of social cues that may, in turn,
contribute to the development or consolidation of certain delusions as well as foster
negative self-evaluations that contribute to lowering of self-esteem. A corollary of
the foregoing is that psychosocial treatment or rehabilitation that improves facial
affect recognition may have wider positive outcomes, namely improving social
skills, reducing delusional ideation and enhancing self-esteem.
Although cognitive rehabilitation in schizophrenia is now well established, the
literature examining the application of cognitive rehabilitation principles to the
deficit in facial affect recognition in schizophrenia is sparse. There is a limited
literature on cognitive remediation for recognition of faces in dementia, including
alcohol-related dementia and Korsakoff syndrome. However, there is a substantial
body of work reporting the use of 'molar' approaches to social skills training in
schizophrenia that have demonstrated efficacy. In these approaches social scenarios
or cues, including but not limited to facial affect, are presented to the patient for
identification, evaluation and interpretation in context, followed by the generation of
possible responses, one of which may be put into practice. Therapists influence this
chain of events through the use of instruction, rehearsal, social modeling, positive
reinforcement and coaching with the aid of in vivo exercises and homework
assignments. It is not known to what extent objective deficits in facial affect
recognition are reduced, if at all, by these rehabilitation techniques. It is likely that
the therapeutic benefit relates more to the appropriate integration and interpretation

126
of multiple cues in correct context and the rehearsal of appropriate responses that are
then differentially rewarded. In any event, the result is improved social skills,
increased self-esteem and better control of psychotic symptoms, insofar as the latter
are causally related to social situations.
To date there have been very few published studies which have addressed the
possibility of rehabilitative cognitive training in facial emotion recognition in
schizophrenia. One study, (Penn & Combs, 2000), investigated the effects of
monetary reinforcement and promoting facial mimicry on performance in facial
affect identification in patients with schizophrenia. Both of these interventions led
to significant improvement in facial affect identification, with some evidence of
generalisation to facial affect discrimination performance, with improvements being
robust one week following the intervention. Unfortunately the study did not
examine whether the therapeutic effect could be attributed to improved motivation
and attention, or to performance gains due to corrective feedback. Moreover, whilst
Penn and Combs (2000) suggest that of the two interventions applied, the promoting
of facial feedback through mimicry, may have broader benefits in terms of
facilitating social interactions, this possibility was not tested in the study.
Studies looking at potential therapeutic interventions have not greatly elucidated
our understanding of the scope and generality of the facial emotion recognition
deficit observed in schizophrenia, although they do suggest that tailored therapeutic
interventions may be effective. The development of effective interventions is likely
to proceed in tandem with increases in understanding of the causes and mechanisms
of the deficit.

DEFINING A WAY FORWARD: THE IMPORTANCE OF FACIAL

MEASUREMENT

A major methodological impairment in resolving the issue of whether the problems

in facial affect recognition observed in schizophrenia represents part of a more
generalised deficit or a specific negative emotion deficit has been that, to date, there
have been no studies that have applied a differential deficit approach to the
discrimination of different emotions, explicitly attempting to match stimuli for
degree of difficulty across a range of emotions. In fact, such a strategy presupposes
a categorical structure of emotion, whereby each category is equal in terms of its
discriminability and is relatively independent. This is unlikely to be the case, as
both analysis of error patterns and dimensional accounts of emotional structure (ie.
Katsikitis, 1997), imply patterns of correlation of emotional categories. This
suggests that it may not be possible to derive a stimulus set whereby an exemplar of,
say, disgust is reasonably discriminable as such, and equally likely to be mistakenly
identified as any other emotion. It is our view that rather than attempting to do this,
with the risk of producing a stimulus set which lacks all ecological validity, an
alternative and more viable strategy is to address the issue through a methodology
which incorporates rigorous parameterisation and analysis of facial emotion
stimulus materials.
Methods such as the Facial Action Coding Scheme (FACS), (Ekman & Friesen,
1976), and the Facial Measurement System (FACEM), (Pilowsky & Katsikitis,

127
 1994), provide a means for parameterising, measuring and describing emotional face
stimuli. Such methods may be applied across large databases of stimulus materials
to produce a fuller characterisation of the structural variation of faces within and
between different emotional valences. A measure of the validity of these methods
would be their ability to detect patterns of structural variation that were able to
predict, for instance, differential performance accuracy to different emotional
valences in healthy human subjects, or specific patterns of stimulus category
dependent miscategorisation errors.
In a recent study (Johnston, Katsikitis & Lewin, 2001 a) we have shown that
measures derived using the FACEM technique have a strong predictive capacity, in
terms of predicting differential accuracy to different stimulus categories, and error
patterns, in healthy subjects, on a facial emotion stimulus set produced by Pilowsky
& Katsikitis (1994). In another study, (Johnston, Katsikitis & Carr, 200Ib), we have
demonstrated that, for this particular stimulus set, the measures derived using the
FACEM system predict that a generalised perturbation of categorisation
performance should lead to a greater difficulty in classifying negative emotions such
as disgust and anger, than to happy or neutral stimuli. This is of particular interest
when considering the pattern of impairment observed in schizophrenia. In short,
mathematical relationships defining distances between specific (dynamic) facial
features suggest that certain categories of facial emotion are a priori less
discriminable than others on the basis of their structural configuration . Degrading
the information available about the stimulus has a greater adverse affect on the
accuracy for these categories than on others. Mathematically, certain emotional
categories overlap more strongly with other specific emotional categories;
behaviourally, subjects make more reciprocol stimulus miscategorisation errors for
these category pairs.
We propose that a more systematic application of facial measurement and
parameterisation techniques would greatly enhance our ability to interpret findings
in studies exploring facial emotion recognition deficits in schizophrenia. Moreover,
the application of such techniques could benefit research in to problems in encoding
facial affect in schizophrenia. To date, much of the research in this area has relied
solely upon the evaluations of raters. A more objective measurement technique
would do much to facilitate comparison across studies.

AFFILIA TION

Department of Psychiatry
University of Newcastle
Email: pjohnsto@mail.newcastle.edu.au

REFERENCES
Addington, J., & Addington, D. (1998). Facial affect recognition and information processing in
schizophrenia and bipolar disorder. Schizophrenia Research, 32, 171-81.
Archer, J,. Hay, D., & Young, A. (1992). Face processing in psychiatric conditions. British Journal of
Clinical Psychology, 31,45-61.

128
Borod, J., Martin, C., Alpert, M., Brozgold, A., & Welkowitz, J. (1993). Perception of facial emotion in
schizophrenia and right-brain damaged patients. Journal of Nervous & Mental Disease, 181,494-
502.
Bryson, G., Bell, M., & Lysaker, P. (1997). Affect recognition in schizophrenia - A function of global
impairment or a specific cognitive deficit. Psychiatry Research, 71,105-\3.
Chapman, LJ., & Chapman, J.P. (1978). The measurement of differential deficit. Journal of Psychiatric
Research, 14, 303-311.
Damasio, D. (1990). Face agnosia and the neural substrates of memory. Annual Review of Neuroscience,
13,89-109.
Dolan, RJ., Fletcher, P., Morris, J., Kapur, N., Deakin, J.F.W., & Frith, C.D. (1996). Neural activation
during covert processing of positive emotional facial expressions. Neuroimage, 4, 194-200.
Dougherty, F.E., Bartlett, E.S. & Izard, C.E. (1974). Responses of schizophrenics to expressions of the
fundamental emotions. Journal of Clinical Psychology, 30,243-246.
Ekman, P., & Friesen, W. (1976). Pictures of facial affect. Palo Alto, CA: Consulting Psychologists
Press.
Feinberg, T., Rifkin, A., Schaffer, C., & Walker, E. (1986). Facial discrimination and emotional
recognition in schizophrenia and affective disorders. Archives of General Psychiatry, 43,276-279.
Ferdowsi, S. (1983). The processing offacial iriformation in schizophrenia. Unpublished doctoral thesis,
University of London.
Gessler, S., Cutting, J., Frith, C., & Weinman, J. (1989). Schizophrenic inability to judge facial emotion:
A controlled study. British Journal ofClinical Psychology, 28, 19-29.
Gottheil, E., Thornton, c., & Exline, R. (1976). Appropriate and background affect in facial displays of
emotion comparison of schizophrenic and normal males. Archives of General Psychiatry, 33,_565-
568.
Habel, U., Gur, R., MandaI, M., Salloum, J., Gur, R., & Schneider, F. (2000). Emotional processing in
schizophrenia across cultures: standardized measures of discrimination and experience.
Schizophrenia Research, 42, 57-66.
Halgren, E., Dale, A., Sereno, M., Tootell, R., Marinkovic, K., & Rosen, B. (1999). Location of human
face-selective cortex with respect to retinotopic areas. Human Brain Mapping 7, 29-37.
Heimberg, c., Gur, R., Erwin, R., & Shtasel, D. (1992). Facial emotion discrimination: III. Behavioral
findings in schizophrenia. Psychiatry Research, 42,2253-265 .
Johnston, P., Katsikitis, M., & Lewin, TJ. (2001 a). Assessing the validity of the FACEM facial affect
parameterisation scheme through neural network analysis. Under review. Behaviour Research
Methods, 1nstruments and Computers.
Johnston, P., Katsikitis, M., & Carr, V. (200Ib). A generalised deficit can account for problems in facial
emotion recognition in schizophrenia. Biological Psychology, 58,203-227.
Katsikitis, M. (1997). The classification of facial expressions of emotion - A multi-dimensional scaling
approach. Perception, 26, \3-26.
Katsikitis, M., & Pilowsky, I. (1996). A controlled study of facial mobility treatment in Parkinson's
Disease. Journal of Psychosomatic Research, 40,387-96.
Kee, K., Kern, R., & Green, M. (1998). Perception of emotion and neurocognitive functioning in
schizophrenia - what's the link. Psychiatry Research, 8, 57-65.
Kerr, S., & Neale, J. (1993). Emotion perception in schizophrenia: Specific deficit or further evidence of
generalized poor performance? Journal of Abnormal Psychology, 102,312-18.
Lane, R., Reiman, E. , Bradley, M. , Lang, P., Ahern, G., Davidson, R., & Schwartz, G. (1997).
Neuroanatomical correlates of pleasant and unpleasant emotion. Neuropsychologia, 35,1437-1444 .
Leventhal, H., & Scherer, K. (1987). The relationship of emotion to cognition: A functional approach to a
semantic controversy. Cognition & Emotion, 1,3-28.
MandaI, M., Pandey, R., & Prasad, A. (1998). Facial expressions of emotion and schizophrenia - a
review. Schizophrenia Bulletin, 24,399-412.
MandaI, M., Borod, J., Asthana, H., Mohanty, A., Mohanty, S., & Koff, E. (1999a). Effects of lesion
variables and emotion type on the perception of facial emotion. Journal of Nervous & Mental
Disease, 187,603-609.
MandaI, M.K., Jain, A., Haque-Nizamie, S., Weiss, U., & Schneider, F. (I 999b). Generality and
specificity of emotion-recognition deficit in schizophrenic patients with positive and negative
symptoms. Psychiatry Research, 87, 39-46.

129
Morris, 1., Friston, K., Buchel, e., Frith, C., Young, A., Calder, A., & Dolan, R. (1998). A
neuromodulatory role for the human amygdala in processing emotional facial expressions. Brain
121,47-57.
Muzekari, L., & Bates, M. (1977). Judgment of emotion among chronic schizophrenics. Journal of
Clinical Psychology, 33, 662-666.
Novic, J., Luchins, D., & Perline, R. (1984). Facial affect recognition in schizophrenia: Is there a
differential deficit? British Journal of Psychiatry, 144, 533-537.
Penn, D.L., & Combs, D. (2000). Modification of affect perception deficits in schizophrenia.
Schizophrenia Research, 46,217-229.
Phillips, M., Williams, L., Senior, C., Bullmore, E., Brammer, M., Andrew, C., Williams, S., & David, A.
(1999). A differential neural response to threatening and non-threatening negative facial
expressions in paranoid and non-paranoid schizophrenics. Psychiatry Research: Neuroimaging, 92,
11-3J.
Phillips, M., Bullmore, E., Howard, R., Woodruff, P., Wright, \., Simmons, A., Andrew, C., Brammer,
M., & David, A. (1998). Investigation of facial recognition memory and happy and sad facial
expression perception: An fMRI study. Psychiatry Research: Neuroimaging, 83,127-138.
Pilowsky, \., & Basset, D, (1980). Schizophrenia and the response to facial emotions. Comprehensive
Psychiatry, 21, 236-244.
Pilowsky, \., & Katsikitis, M. (1994). The classification of facial emotions- a computer based taxonomic
approach. Journal of Affective Disorders, 30, 61-71.
Puce, A., Allison, T., Asgari, M., Gore, J.e., & McCarthy, G. (1996). Differential sensitivity of the
human visual cortex to faces, letterstrings an textures - a functional magnetic resonance imaging
study. Journal of Neuroscience, 16, 5205-5215.
Salem, J., Kring, A., & Kerr, S. (1996). More evidence for generalized poor performance in facial
emotion perception in schizophrenia. Journal of Abnormal Psychology, 105,480-483.
Schneider, F., Heimann, H., Himer, W., Huss, D., Mattes, R., & Adams, B. (1990). Computer based
analysis of facial action in schizophrenic and depressed patients. European Analysis of Psychiatry
and Clinical Neuroscience, 40, 67-76.
Schneider, F., Gur, R.e., Gur, R.E., & Shtasel, L. (1995). Emotional processing in schizophrenia -
neurobehavioural probes in relation to psychopathology. Schizophrenia Research, /7,67-75.
Schneider, F., Grodd, W., Weiss, U., Klose, U., Mayer, K., Nagele, T., & Gur, R. (1997). Functional MRI
reveals left amygdala activation during emotion. Psychiatry Research: Neuroimaging, 76, 75-82.
Shannon, A. (1970). Differences between schizophrenics and depressives in the recognition of facial
expressions of emotions. Unpublished doctoral thesis, University of California, San Francisco.
Sprengelrneyer, R., Rausch, M., Eysel, U., Przuntek, H. (1998). Neural structures associated with the
recognition of facial expressions of basic emotions. Proceedings of the Royal Society of London -
Series B: Biological Sciences, 265, 1927-31.
Streit, M,. loannides, A., Sinnemann, T., Wolwer, W., Dammers, J., Zilles, K., & Gaebel. (2001).
Disturbed facial affect recognition in patients with schizophrenia associated with hypoactivity in
distributed brain regions: A magnetoencephalographic study. American Journal of Psychiatry, 158,
1429-1436.
Swithenby, S., Bailey, A., Brautigam, S., Josephs, 0., Jousmaki, Y., & Tesche, C. (1998). Neural
processing of human faces - a magnetoencephalographic study. Experimental Brain Research, 118,
501-10.
Tovee, M. (1998). Face processing: Getting by with a little help from its friends. Current Biology, 8,
R317-R320.
Walker, E., Maguire, M., & Bettes, B. (1984). Recognition and identification of facial stimuli by
schizophrenics and patients with affective disorders. British Journal of Clinical Psychology,j3,
37-44.
Walker, E., Marwit, S., & Emory, E. (1980). A cross-sectional study of emotion recognition in
schizophrenics. Journal of Abnormal Psychology, 89,428-36.
Young, A., Aggleton, J., Hellawell, D., Johnson, M., Broks, P., & Hanley, J. (1995). Face processing
impairments after amygdalotomy. Brain, 118, 15-24.
Zuroff, D., & Colussy, S. (1986). Emotion recognition in schizophrenic and depressed inpatients. Journal
of Clinical Psychology, 42, 411-17.

130
 JORGMERTEN

CHAPTER 7

CONTEXT-ANALYSIS OF FACIAL-AFFECTIVE
BEHAVIOR IN CLINICAL POPULATIONS

INTRODUCTION

Several studies investigating the functions of facial-affective behavior in dyadic

interactions demonstrate the importance of facial-affective behavior for the
expression of emotional feelings and for relationship-regulation in social interactions
(Krause, Steimer-Krause, Merten, & Ullrich, 1998). In the realm of mental disorders
and psychotherapeutic processes attention to facial-affective behaviour is a good
starting point for the understanding of psychopathological processes taking place in
social interactions between patients and their interacting partners (Merten, Ullrich,
Anstadt, Krause, & Buchheim, 1996). Most clinical disorders are based on
maladaptive relationship-patterns (Krause & Merten, 1999). Many authors have
investigated this perspective of mental disorder using different theoretical concepts.
Luborsky (1977), for example, derives patients' core conflictual relationship themes
from narrations about relationship-episodes. Structural Analysis of Social Behavior
(SASB; Benjamin, 1974, 1993) is used to formulate cyclic maladaptive patterns
(CMP, Johnson, Popp, Schacht, Mellon, & Strupp, 1989). Horowitz (1997) uses
role-relationship-model-configurations to describe different forms of social behavior
of patients. These authors use mostly narration about behavior and relationship-
episodes to extract maladaptive relationship-patterns. Beside the narration of
relationship-episodes the actualization and interactive implementation of
relationship-patterns in social interactions is not yet well understood. One can
hypothesise that relationship-patterns are actualized by the aid of spontaneous
behavior that occurs involuntarily and is set into motion by conscious or unconscious
expectations of subjects suffering from mental disorders. Similar phenomena have
been observed in experimental social psychology (Rosenthal, 1978) and are
understood as spontaneous communication of interpersonal expectations by Buck
(1993). The analysis of this spontaneous communication system can provide
answers to the following questions posed by Merten and Krause (2002), i.e. "what
makes psychological disturbances so persistent?" and "how is the psychic
disturbance implemented in the intersubjective social and mental field?"
The present contribution describes facial-affective behavior of different subjects
in dyadic interactions and especially demonstrates the usefulness of context-analysis
for the understanding of the different functions of facial-affective behavior. Context
has to be considered because analysis of facial-affective behavior alone does not
suffice to understand its meaning in dyadic interactions. Its functions and meanings
differ depending on different aspects of the context they occur in. Depending on
context-variables - like gender or mental disorders - the relationship can vary, for
example, from a very intimate and confidential one, to a hostile and dominating one.
But immediate context-variables, as for example, the gazing behavior before, during
and after facial-affective displays, have to be taken into account also to understand
the meaning of them. A method of context-analysis and the results obtained using
this method are described in detail below.

METHOD: CONTEXT-ANALYSIS OF FACIAL AFFECTIVE BEHAVIOR

To reveal the meaning of nonverbal, mostly involuntary communication for social

interactions, different situations were videotaped and analyzed. Subjects were
videotaped with two cameras using a "split-screen" technique (see Figure I). Three
different situations are compared a) discussions between healthy subjects b)
discussions between patients and laymen and c) psychotherapies.

132
 Figure J: Example of a video-recording using "split-screen" technique

Situation a) comprised 30 different dyads of healthy subjects (10 male x male, 10

female x female and 10 male x female). In situation b) 20 dyads of healthy subjects
and patients suffering from schizophrenia and 20 dyads of healthy subjects and
patients suffering from psychosomatic disorders were analyzed. All seventy dyads
were videotaped while discussing political issues. The subjects were not acquainted
to each other until they met in the laboratory. Each conversation lasted 20 minutes.
Facial behavior, gazing behavior, verbal and paraverbal behavior were coded. In
situation c) 10 short psychotherapies of different theoretical orientation
(psychodynamic, cognitive-behavioral, client-centered), each lasting 15 sessions
were analyzed. It was hypothesized that maladaptive relationship-patterns are also
repeated in psychotherapeutic treatments and that the way the therapist reacts is
crucial for therapeutic outcome.

Analysis offacial behavior and its meaning

To code facial-affective behavior EMFACS (Emotional Facial Action Coding

System), developed by Friesen & Ekman (1984), was used. Coders observe the
video and pause if they recognize a muscle movement defined in the EMF ACS
Handbook. After having coded facial movements, an interpretation algorithm - also
part of EMF ACS - assigns emotion categories to the different kinds of facial
behavior. However, only a proportion of the codings are interpreted as emotional
according to EMFACS. Nevertheless, this does not mean that these behaviors
necessarily have to be related to emotional experience. Facial-affective behavior is

133
for example, often shown voluntarily and for social reasons (Fridlund, 1991). To
understand the meaning of facial-affective behavior it is best conceived as a sign-
system that not necessarily expresses the emotional state of a person but that can
serve different functions (Merten, 1997). These different meanings or functions of
facial-affective signs have to be disentangled, and this can be done by the analysis of
the contexts they occur in. (For an extensive discussion of the meaning of facial-
affective behavior, especially of its relation to emotional experience (Frank, this
volume; Kappas, 1997; this volume; Merten, 1997).
Applying EMF ACS to the viderecordings yields information about facial-
affective signs displayed by the interacting subjects. Major emotional categories
distinguished are the so-called basic emotions, mixtures of basic emotions and
masking of basic emotions by smiling. Also, smiles indicating felt happiness can be
distinguished from smiles that do not and yet are shown for social reasons. In Figure
2 several examples of facial displays of the basic emotions are given.
The topic of cultural universality of basic emotions displayed by facial-affective
expression is discussed in detail by Ekman (1994), Russell (1994), Izard (1994) and
Barrett and Katsikitis (this volume).

Anger Disgust Contempt Fear Sadness Joy urprise

Figure 2: Examples offacial displays of basic emotions

The facial displays depicted in Figure 2 are posed voluntarily and with high
intensity. Involuntary displays can vary considerably in duration and intensity, for
example, they can be so short that they are only recognized by well trained coders.
Nevertheless, short and weak displays play an important role in the regulation of
relationships, as will be demonstrated below.

134
Dyadic gazing behavior as context-variable

One important context variable of facial-affective behavior is the dyadic gazing

behavior observed while an emotion is displayed. Several studies demonstrate that
not only gazing behavior of the sender of the facial display is of importance but also
that of the recipient (Merten, 1995). To describe dyadic gazing behavior, 4 dyadic
states that are depicted in Figure 3, can be distinguished. Furthermore, the location
of facial-affective signs in different gazing states is important, as will be shown
below. The gazing states in columns 2 and 3 show a distinction between facial signs
expressing an emotional state and facial signs serving a speech-illustrating function
without necessarily reflecting the emotional state of the sender.

Both look ot Looking ot Looking None Looking

looking looking

Figure 3: Facial-affective signs in different dyadic gazing states

With respect to the regulation of relationships, the dyadic gazing state of mutual
gazing is the most important one. Since mutual gazing is a moment of high
ambivalence (Patterson, 1991), it has to be qualified by emotional signals. For this
reason, the probability for simultaneous facial-affective behavior of both interacting
partners is highest, and it indicates the emotional quality of the relationship.

Pattern analysis offacial behavior

A further important context variable is the facial-affective behavior of the interacting

partner occuring simultaneously to, or immediately before or after, a facial display.
These dyadic emotional patterns are indicators of mutual affective exchange
processes that establish the emotional quality of dyadic interactions. In Figure 4
some examples for dyadic emotional patterns are given. In the first row of Figure 4

135
a pattern compnsmg two elements is depicted. This pattern was found in a
psychotherapy. The therapist's expression of surprise was regularly followed by a
contempt expression by the patient. The complexity of this pattern describes
anumber of elements that it comprises. A high complexity of patterns indicates a
high degree of interweavement of the affective processes of the interacting partners.
In the second row of Figure 4 an example of a pattern comprising three elements is
given. The male partner reacts reciprocally to the female's smile, but afterwards he
shows a blend of anger and sadness.

surprise contempt

Duchenne Smile Duchenne Smile Mixed emotion t ~

Figure 4: Two dyadic emotional patterns of different complexity

Since a lot of facial events take place during a conversation, it is difficult to detect
patterns with the naked eye. To detect occurring patterns and to test them against
chance, an algorithm of Magnusson (1996) is used. It takes into account frequencies
of elements of the patterns as well as their temporal distribution across the
interaction periods observed. Co-occurences of behavioral elements are tested
against chance. If they occur more often than expected - given a fixed chance level

136
- they are called T-patterns.

RESULTS: CONTEXT-ANALYSIS OF FACIAL BEHAVIOR IN DYADIC

INTERACTIONS

Applying the methods described above to video recordings of dyadic interactions

yields the following results on the influence of the gender of participants, mental
disorder, "expressed emotion" and psychotherapeutic processes.

Gender differences and topics regulated by nonverbal behavior

Gender of the dyads influences the interaction (see Table I).

female- female male-male

felt happiness 3S.6 (20.S) IS.l (16.0)
unfelt happiness 12.9 (27.6) 32.0 (18.3)
possible anger 2.S (S.9) 6.2 (7.S)

Table I: Differences in facial-affective behavior depending on "dyadic" gender

M(SD): mean (M) and standard deviation (SD) offacial affects per conversation.

The most striking difference is that between the facial display of felt happiness
(genuine expressions of joy) in male-male dyads in comparison to female-female
dyads. Male subjects in male-male dyads show only half of the felt happiness of
female subjects in female-female dyads. This difference is not due to individual
gender but to the gender of both participants ("dyadic" gender). This view is
supported by the following result. When male subjects are in a conversation with
female subjects, the male subjects display felt happiness as often as female subjects
do (Frisch, 1997, Table 2).

137
 female in female-male male in female-male
felt happiness 48.2 (28.1) 40.8 (22.7)
unfelt happiness 14.8 (7.8) 21.8 (15.3)
possible anger 1.3 (2.8) 2.3 (3.4)

Table 2: Adaptation of male subjects infacial affectivity

when interacting with female subjects
M(SD): mean (M) and standard deviation (SD) facial affects per conversation

Topics of relationship-regulation depending on "dyadic "gender

The differences between the dyads presented above can be explained by different
kinds of relationship-themes. In the case of male-male dyads, several results indicate
that the regulation of dominance is a central topic or theme, while in female-female
dyads high amounts of positive interactional involvement are realized (Merten,
1997). The latter one is indicated by a positive evaluation of mutual gazing.
Duration of mutual gazing in female-female dyads is positively correlated with the
emotional experience of joy and felt happiness. In addition, Duchenne smiles during
mutual gazing are observed at a far higher rate than expected by chance. In male-
male dyads, the duration of mutual gaze is not correlated with the experience of
positive emotions, and Duchenne smiles do not occur more often than expected by
chance while gazing at each other. Instead, Duchenne "social" smiles occur more
often than expected by chance. The frequency of these "social" smiles is positively
correlated with the experience of anger, while Duchenne smiles are correlated
negatively with the experience of anger in male-male dyads (Merten, 1997).

Facial-affective behavior and verbal content

Facial-affective behavior not only serves an expressive function or relationship-

regulation, but it is also often used to enrich narratives. In this case, it is even
possible that the emotional experience reported contradicts the emotional quality of
the facial behavior observed. For example, frequency of anger-expressions in all
three types of dyads with healthy subjects is positively and not negatively correlated
with the experience of positive emotions like interest or joy. The more anger
subjects expressed in the context of narratives about politicians, the more joy they
reported .

138
 Speech-illustrating facial-affective display can be distinguished from emotiolial
expressions by the dyadic gazing states they occur in (Merten, 1997). Facial-
affective displays referring to objects mentionned in narratives occur in most cases
while the sender is not looking at the recipient, while the latter one is looking at the
sender (as illustrated in Figure 3, column 3). So if one analyzes facial-affective
displays in dyadic interactions, one has to take into account the dyadic gazing state to
determine the function of a facial-affective display.

Interactive behavior of subjects with severe mental disorders

A very powerful context variable is the fact that one of the interacting subjects
suffers from a mental disorder. Subjects suffering from different mental disorders
can be characterized by specific relationship patterns. In schizophrenic dyads, for
example, regulation of intimacy is crucial. The regulation of intimacy is handicaped
by a conflict between a wish for closeness and avoidance of it (Merten, 2000: Merten
& Krause, 2000).
This handicap is indicated by a reduction of positive facial expressions and an
outstanding negative affect. The reduction also holds true for their healthy
interacting partners. The patients suffering from severe mental disorders like
schizophrenia or severe psychosomatic disorders most frequently display contempt
or disgust instead of joy. These negative affects do not serve speech-illustrating
functions - as demonstrated above for healthy subjects - but are an expression of
deficient self-regulation and relationship-regulation.

Facial schizophrenic schizophrenic male colitis female colitis

behavior outpatient dyads inpatient dyads dyads dyads
Anger .71'
Contempt -.86"
Disgust -.60' -.73' -.65'

Table 3: Patients' emotional experience ofjoy and their facial-affective behavior

Spearman correlations, * p < .05, + P < .10

This interpretation is consistent with the correlations found with emotional

experience and the dyadic gazing context that the negative affect occurs in. As
depicted in Table 3, the frequency of contempt and disgust is negatively correlated
with the emotional experience of joy (Merten, 1997). These negative affects are part

139
of a deficient relationship-regulation and entail negative emotional experience. A
further indication is the way in which these negative affects are placed in the dyadic
gazing context. They mostly occur while the patient is looking at the healthy
interacting partner whereas the latter one is not looking at the patient (see Figure 3,
column 2). The opposite gazing context was found in the case of healthy subjects'
negative emotions.

"Expressed Emotion" and facial-affective behavior

"Expressed Emotion" (EE) is a concept that is used to characterize the emotional

quality of relationships. There are two negative subscales of EE, i.e. hostile behavior
and over-involvment. EE of significant others is a good predictor of relapse of
schizophrenic patients. Parents of adolescent schizophrenic patients can be
subdivided into high and low EE (Hahlweg et aI., 1995). Patients whose parents are
characterized by high EE relapse far more often. A ranking of high or low EE is
gained by rating parents' reports about their behavior when interacting with their
schizophrenic children. A study by Haack-Dees (2000) compared the interactive
behavior of parents and their schizophrenic children with the behavior of parents of
healthy children. In both groups, parents and children had to discuss the children's
professional career. Parents of healthy children showed twice as many facial affects
as parents of children with schizophrenia. If one distinguishes parents with high and
low EE, one finds that parents of healthy children with high EE (criticism) showed
three times more negative affect than high-EE (criticism) parents of schizophrenic
children.
The usefulness of context-analysis shows up in the following result. The
influence of high EE (criticism and over-involvement) and mental disorder on facial-
affective behavior can be observed best in the rate of negative affects during silence,
when no one is speaking. Parents of schizophrenic children with over-involvment
show more than 40% of their negative affect in moments of silence. Thus,
occurrence of facial-affective behavior in specific contexts discriminates between
different groups of EE.

140
 RELATIONSHIP-REGULATION IN PSYCHOTHERAPIES, THERAPEUTIC
CHANGE AND THERAPEUTIC OUTCOME

The therapeutic relationship in contrast to an everyday relationship

If one compares facial-affective behavior in "everyday" situations and

psychotherapeutic situations, one finds the following results: the overall facial
affectivity of patients and therapists is reduced when compared to that of subjects in
"everyday" interactions. Joy and disgust are displayed less by patients and
therapists, while fear and surprise are shown more often by patients in the
psychotherapeutic situation. Therapists also show more surprise than subjects in
"everyday" interactions. Patients and therapists display less contempt than healthy
subjects in "everyday" interactions, but patients show contempt twice as much as
therapists. In comparison to "everyday" interactions, simultaneous Duchenne smiles
(indicative of felt happiness) of both patient and therapist are reduced significantly.
Healthy subjects in "everyday" interactions show three times more simultaneous
Duchenne smiles than patients and therapists.

How is the implementation o/relationship-patterns tied to therapeutic outcome?

The analysis of facial behavior in therapy-sessions yielded the following results:

facial activity in the first session was highly variable across patients and therapists.
Patients displayed facial events in a range from 145 to 641 events per session,
therapists from 48 to 226. In 10 of the 11 therapies, the patients' facial activity was
higher than that of their therapists. Taking into account only primary affects, only 8
patients were more expressive than their therapists. In general, the therapists showed
less idiosyncratic facial behavior, less affective blends and more "pure" primary
affects than the patients.
Facial behavior of the therapists did not exhibit any change that could be related
to differences in their theoretical orientation. The variance between therapists of the
same theoretical orientation was even higher than that found between the groups of
different theoretical orientation. Facial behavior depends more on individual
characteristics and/or dyadic adaptation processes than on theoretical orientation.
In dyadic interactions between two healthy persons, the most frequent facial
affective event was felt happiness. In therapeutic interaction, this occurred only in 6
therapists and 5 patients out of a sample of 11. The others showed mainly contempt
or disgust, and in one dyad the most frequent facial affective event was anger. The

141
"Leitaffekt", which is the predominant affect of a subject, was shown with different
frequencies. One patient displayed 187 facial events interpreted as disgust during the
50 minutes of the first session but only once expressed felt happiness. These affects
can be considered as indicators of interactive and self-regulatory processes and
should correlate with therapeutic outcome. Nevertheless, neither the affective
valence of the "Leitaffekt" of the patient nor its frequency correlated significantly
with one of the outcome measures.
In line with the above-mentioned results on adaptation, therapists who adapted to
the facial affective behavior of the patients should yield poor therapeutic outcome.
Indeed, the relative frequency of the "Leitaffekt" of the therapist correlated
negatively with therapist's outcome rating which was usually given half a year later
(r=-.63, p < .05, N=II). Therapists who displayed high amounts of one single facial
affect during the first session rated therapeutic outcome as "worse" after the 15th
session, and this was irrespective of the type of "Leitaffekt" displayed. This could be
interpreted as a consequence of the implementation of a maladaptive repetitive
pattern which reduces the normal variance of affectivity not only on the side of the
patient but also on the side of the therapist.

Outcome
perspective
Predictor Therapist Patient Combined
%"Leitaffektr " -.63'
Negative affects r .81'
Happy feltp/negative affects r -.64' -.76'

Table 4: Facial-affective behavior and therapeutic outcome

Spearman correlations, * p < .05, + P < .10
%"LeitaffektT": relative frequency of most frequent facial affect ("Leitaffekt")
Happy feltp/negative affectsT Proportion of patient's happy felt cxpression and therapist's
negative affccts

A further indication for the implementation of a maladaptive relationship pattern is

found in the positive correlation between the negative affects anger, contempt and
disgust with the therapist's rating on therapeutic outcome. The more that negative
affects were shown in the first session, the better the therapeutic outcome was.
Therapist's negative affects also correlated with symptom reduction (r=.54, Q=.II,
N= I 0). On the basis of these results the successful therapeutic relationship is not
characterized by positive facial expressions. On the contrary, there has to be a

142
sufficient amount of negative emotion that seems to make it easier to understand the
problems of the patient and to work on these problems. These negative affects are
also necessary to compensate the patient's attempts to behave as if he/she was in an
everyday interaction. This can be shown by the result depicted in Table 4, row 3.
The proportion of felt happiness of patients and negative affect of therapists
correlated with all three outcome measures. In therapies in which the patient
expressed many positive affects and there were only a few negative affects on behalf
of the therapist outcome was not as good. Successful therapists compensated for too
much positive affects with negative affects.

Dyadic emotional patterns and the implementation o/relationship-patterns

The correlations reported above between frequencies of facial affects and therapeutic
outcome do not cover the actual implementation of maladaptive relationship-
patterns. The actual implementation of maladaptive relationship-patterns takes place
on the level of dyadic emotional patterns that describe emotions of patient and
therapist as they occur simultaneously or within a short temporal distance.
Emotional patterns describe individual and dyadic emotional processes as they
appear in facial-affective behavior of patient and therapist. They also contain
information about emotional self-regulatory processes as well as about the quality of
relationship-regulation between patient and therapist. Definitions and properties of
emotional patterns are given in Merten (1996); an algorithm to detect "hidden real
time patterns" was developed by Magnusson (1996). The application of the
algorithm of Magnusson revealed that dyadic patterns of Duchenne smiles (that is,
smiles that appear simultaneously or almost simultaneously in the interaction)
occured in nearly all analyzed therapies and that no pattern of negative affects on
behalf of both participants was found. Therefore, motor mimicry of facial-affective
behavior - as far as it is registered by EMF ACS - only takes place in the case of a
positive emotion when indicated by a Duchenne smile.
In cases where negative affects are part of a dyadic pattern, the negative affect is
compensated by a social smile or a Duchenne smile of the partner. In addition,
several therapy-specific patterns describe core psychodynamic conflicts of the
patient. To give an example, a patient with a conflict with attachment and separation
engaged in a large amount of positive facial-affective behavior with the therapist, but
also displayed a pattern of anger followed by an expression of fear that could be
directly matched to the conflict of the patient. This and other cases are described
elsewhere in detail (Benecke, Krause & Merten, in press; Merten, 2000; Merten et

143
aI., 1996).
Table 5 shows the correlations between characteristics of emotional patterns in
different therapies and therapeutic outcome. One major result is that the frequency
of dyadic emotional patterns correlates negatively with therapeutic outcome in all
three perspectives. Maximum complexity of the patterns -number of elements in a
pattern - also correlates negatively in the same manner.

Outcome
perspective
Predictor Therapist Patient Combined
Maximum complexity of -.69 -.43
patterns
Number of dyadic patterns -.58+ -.81' -.75'
Simultaneous Duchenne Smiles -.63'

Table 5: Correlations between/acial-affective behavior and therapeutic outcome

Complexity of patterns: Number of elements in a pattern
Spearman correlations, * p < .05, + P < .10

A specific subcategory of emotional patterns is that composed of the appearance of a

Duchenne smile in both interacting partners which occurs simultaneously. The
frequency of simultaneous Duchenne smiles correlates significantly with therapeutic
outcome, namely from the perspective of the therapist (Table 5).
In addition, a curvelinear quadratic relation between the frequency of mutual
smiling initiated by the therapist and therapeutic outcome was found. Therapies in
which, not even one incident of positive mutual smiling initiated by the therapist is
observed, were rated on a medium level of outcome. In the therapies in which the
process deteriorated or the patient dropped out, more than four incidents of mutual
smiling initiated by the therapist were found. The therapies with highest outcome lie
in between the two above-described cases.
It can be concluded that the implementation of relationship-patterns is indicated,
in general, by high frequencies and high complexity of dyadic patterns and also by
the presence of too many patterns of positive emotions from both interacting
partners. Furthermore, this kind of implementation is correlated with poor
therapeutic outcome.
If the therapist gets involved in the maladaptive relationship-pattern, and if it is
not resolved during the course of treatment, the pattern will be repeated and further

144
reinforced. This assumption was confirmed by the following results. In therapies
with better outcome conflict-indicators increased to a certain point in treatment and
tended to decrease in later sessions. Indicators for poor outcome were high
complexity in dyadic relationship-patterns and their predominace in high frequencies
in the last session. In these cases, therapists were unable to recognize and/or resolve
the maladaptive relationship-pattern they were involved in.

Facial-affective behavior and verbal content

Benecke (2000) analyzed the verbal context of negative facial-affects of the

therapists and indicated that in the beginning of treatment they are elicited by
behavior of the patient and are not tied to verbal contents. At the end of treatment,
negative facial affects of therapists occur more often simultaneously while speaking
about significant others of the patient than expected by chance.

SUMMARY

A method of context-analysis was described, and a selection of results was

demonstrated. Dyadic gender and different kinds of mental disorders with referring
kinds of conflicts call for different relationship-topics. As a context variable of
facial-affective behavior the dyadic gazing behavior was analyzed. The immediate
context helps to reveal the different function of facial-affective behavior in dyadic
interactions, as there is the speech-illustrating function, the function of emotional
expression and the function of relationship-regulation.
Applying context-analysis to mental disorders, it was shown that relationship-
patterns of patients with mental disorders evolve in social interactions with laymen
as well as with psychotherapists. Patients with severe mental disorders show mostly
one negative affect that is negatively related to the experience of positive emotions.
Furthermore, these negative affects occur in other dyadic gazing states, than they do
in interactions of healthy subjects.
A crucial difference has to be seen in the different kind of therapists to react to
the offers of the patients. While laymen mostly confirm relationship-patterns,
successful therapists react in a different manner, trying to stay aloof.
As a conclusion, facial-affective behavior should be analyzed and interpreted
only when one has considered global aspects of the situation and immediate context
variables like dyadic gazing state, which makes it possible to distinguish different
functions of it.

145
 AFFILIATION

Universitiit des Saarlandes

1m Stadtwald
66041 Saarbriicken
Email: j.merten@rz.uni-sb.de

REFERENCES
Anstadt, T., Merten, J., Ullrich, 8., & Krause, R. (1997). Affective dyadic behavior, core conflictual
relationship themes and treatment outcome. Psychotherapy Research, 7, 397-418.
Barrett, R. J., & Katsikitis, M. (2002). Foreign faces: A voyage to the land of EEPICA. In M .
Katsikitis (Ed.), The human face. Measurement and meaning (pp. 1-28). Dordrecht: Kluwer
Academic Publishers.
Benecke, C. (2000). Mimischer affektausdruck und sprachinhalt im psychotherapeutischen ProzeB.
Dissertation an der philosophischen Fakultl1t der Universitat des Saari andes, Germany .
Benecke, c., Krause, R. & Merten, J. (in press). On the constitution of the intersubjective field.
European Journal of Psychotherapy.
Benjamin, L. S. (1974). Structural analysis of social behavior. Psychological Review, 8 I, 392-425.
Benjamin, L. S. (1993). Interpersonal diagnostics and treatment of personality disorders. New
York: Guilford.
Buck, R. (1993). Interpersonal expectations. Theory, research, and applications. Cambridge: Press
Syndicate of the University of Cambridge.
Ekman, P. (1994). Strong evidence for universals in facial expressions: A reply to Russell's mistaken
critique. Psychological Bulletin, I 15,268-287.
Frank, M. Getting to know your patient: how facial expression can help reveal true emotion. In M .
Katsikitis (Ed.), The human face: Measurement and meaning (pp. 255-283). Dordrecht: Kluwer
Academic Publishers
Fridlund, A. 1. (1991). The sociality of solitary smiles: Eflccts of an implicit audience. Journal of
Personality and Social Psychology, 60, 229-240.
Frisch, I. (1997). Eine frage des geschlechts? Mimischer ausdruck und afJekterleben in gesprachen.
Saarll1ndische Schriftenreihe zur Frauenforschung. St.lngbert: ROhrig .
Haack-Dees, 8. (2000). AfJektives verhalten in eltern-kind-dyaden mit einem schizophrenen
jugendlichen. AfJective behavior in parent-child-dyads with a schizophrenic adolescent.
Dissertation at the University of Saarland .
Hahlweg, K., Dilrr, H., & Milller, U. (1995), Familienbetreung schizophrener patienten. Weinheim:
Beltz Psychologie Verlags Union.
Horowitz (1997). Formulation as a basis for planning psychotherapy treatment . Washington DC,
London: American Psychiatric Press.
Izard, C.E. (1994). Innate and universal facial expression: Evidence from developmental and cross-
cultural research . Psychological Bulletin, 115,288-299.
Johnson, M. E., Popp, c., Schacht, T. E., Mellon, 1. & Strupp, H. H. (1989) Converging evidence for
identification of recurrent relationship themes : Comparison of two methods. Psychiatry, 52 ,
275-288.

146
Kappas, A. (1997). The fascination with faces. Are they windows to our soul. Journal of Nonverbal
Behavior, 21,157-162.
Kappas, A. (2002). What facial activity can and cannot tell us about emotions. In M. Katsikitis (Ed.), The
human face: Measurement and meaning (pp. 215-234). Dordrecht: Kluwer Academic Publishers.
Krause, R. & Merten, J. (1999) Affects, regulation of relationship, transference and countertransference.
International Forum of Psychoanalysis, 8,.103-114.
Krause, R., Steimer-Krause, E., Merten, J., Ullrich, B., (1996). Dyadic interaction regulation, emotion
and psychopathology. In: W.F. Flack & J. Laird (Eds.), Emotions and psychopathology: Theory and
research. (pp. 70-80). Oxford: Oxford University Press.
Luborsky, L. (1977). Measuring a pervasive structure in psychotherapy: The core conflictual relationship
theme method. In N. Freedman & N. Grand (Eds.), Communicative structures and psychic structure
(pp. 367-395). New York: Plenum Press.
Magnusson, M.S. (1996). Hidden real-time patterns in intra- and inter-individual behavior: Description
and detection. European Journal of Psychological Assessment, 12, I 12-123.
Merten, J. (1996). Affekte und die regulation nonverbalen, interaktiven verhaltens. Strukturelle aspekte
mimisch-affektiven verhaltens und die integration von affekten in regu-Iationsmodelle. Bern: Peter
Lang.
Merten, J. (1997). Facial-affective behavior, mutual gaze and emotional experience in dyadic
interactions. Journal of Nonverbal Behavior, 21, 179-20 I.
Merten, J. (2001). Beziehungsregulation in psychotherapien. Maladaptive beziehungsmuster und der
therapeutische ProzeB. Stuttgart: Kohlhammer.
Merten, J. & Krause, R. (in press). What makes good therapists fail? In P. Philippot, E . .1. Coats, & R. S.
Feldman (Eds.), Nonverbal behavior in clinical sellings. Oxford: Oxford University Press.
Merten, J., Ullrich, B., Anstadt, T., Krause, R. & Buchheim, P. (1996). Emotional experiencing and
facial expression in the psychotherapeutic-process and its relation to treatment outcome. A pilot-
study. Psychotherapy Research, 6, 198-212.
Patterson, M., L. (1991). A functional approach to nonverbal exchange. In R. Feldman & B. Rime
(Eds.), Fundamentals of nonverbal behavior (pp. 458-495). New York:Cambridge University Press.
Rosenthal, R. & Rubin, D. B. (1978). Interpersonal expectancy effects: The first 345 studies. The
Behavioral and Brain Sciences, 3, 377-386.
Russell, J.A. (1994). Is there universal recognition of emotion from facial expression? A review of the
cross-cultural studies. Psychological Bulletin, lIS, 102-141.

147
 MONIQUE de BONIS

CHAPTER 8

CAUSES AND REASONS IN FAILURES TO

PERCEIVE FEARFUL FACES

INTRODUCTION

Human and non human primate brains share the ability to decipher emotional
signals in a human face (Sugase, Yamane & Veno, 1999). Other species, such as
birds, whose visual apparatus is highly developed, can learn some of the facial
signals and discriminate positive from negative human expressions (Jitsumori &
Yoshihara, 1997). Furthermore, humans are capable of designing expert-
systems that are more accurate than a human brain in categorising at least the
restricted repertory of basic emotions from the standard Ekman & Freisen (1976)
Facial Action Coding System (Lien, Kanade, & Cohn, 2000; Stewart-Bartlett,
Hager, Ekman, & Sejnowski, 1999). Automatic devices have been developed
that can substitute for the human eye in the assessment of facial expressions of
emotion in clinical settings (Benson, 1999; Katsikitis, this volume; Katsikitis &
Pilowsky, 1988, 1991, 1996).
However, despite the vast body of knowledge that has accumulated since the
pioneering work of Darwin (1882) and Duchenne (1862)
how the human brain discriminates among behaviorally relevant facial expressions
is unknown (Dolan & Morris 2000, p. 226).

In this chapter, the issue of the perception of facial expression of emotion is

addressed from the perspective of failures in recognition. We will focus on a
particular emotional expression i.e. fear. Among the diversity of emotional
expressions, fear offers the largest body of perception research. Fear is also, to
our knowledge, the only emotional expression for which a total blindness (that
is, complete failure of perception) has been observed (Adolphs, Tranel,
Damasio, & Damasio, 1994). We will therefore pay special attention to fear in
the studies reported.
It may be that errors in facial recognition of emotions is a good way, albeit
an indirect one, to understand the perceptual processes involved in the
perception of facial expression of emotion, as it has been for perception in
general (Piaget, 1961). The question of errors, misidentifications and illusions is
not a new one. As early as 1964, Tomkins drew our attention to the fact that
naive subjects
even when they are inaccurate ... are often in substantial systematic agreement in
their errors. These systematic, shared errors we have labelled 'common
confusions' ... to distinguish them from random errors. We called them 'common'
to indicate that these systematic deviations are shared, rather than idiosyncratic
(Tomkins & McCarter, 1964, pp.135-136).

An examination follows of the "true causes" and "reasons" for an expression to

be ignored or mistaken for another expression. Special attention will be paid to
the intrinsic factors related to visual perception and facial feature processing.
Reasons linked to the internal emotional and mental state of the perceiver, and to
contextual factors, will not be discussed since they have been thoroughly
reviewed elsewhere (Ekman & Rosenberg, 1997, Russell, 1980).
Prior to addressing the issue of fear expressions, there is a brief discussion of
current conceptions about the perception of facial expression of emotion (PFE),
that is, the origins of the perception of facial expression. Then,
neuropsychological evidence for total blindness for fear expressions will be
examined, complemented by related brain imaging studies on normal subjects.
Problems for a hypothesis of a unique cause in terms of brain structure
dysfunction is also discussed. In a second section, the specificity of fear
perception is examined within the framework of a part-based model. Within this
framework, two questions are addressed: are there necessary and sufficient
conditions for the perception of fear expressions, and what happens when some
conditions are not fulfilled?

EMERGENCE OF FACIAL RECOGNITION OF EMOTION AS A SPECIFIC

DOMAIN OF INTEREST

In recent decades, recognition of facial expression of emotion has become a

domain of special interest, independent from the field of recognition of facial
identity. Evidence has been reported in support of an independence between
identity and emotion perception. Specifically, it has been shown that patients
diagnosed as prosopagnosics, who cannot recognise familiar faces, are none-the-
less unimpaired in the recognition of facial expression of emotion (Adolphs,

150
Tranel, Damasio & Damasio, 1994; Shuttleworth Jr, Syring & Allen, 1982;
Tiberghien & Clerc, 1986).
The bulk of evidence reported by studies on neurological patients does not
suggest that the process of recognition of identity and facial expression are
fundamentally different. Complementing the twenty years of research on
perception of facial identity and facial expression, Calder, Young, Keane and
Dean (2000) concluded that the processes are largely similar to the extent that
configural information is used in both instances and that in both cases the
inversion of the face contributes to disrupt configura I processing. Findings on
human subjects supported the results of earlier research on monkeys showing
that separated neural substrates encode identity and expression (Hasselmo, Rolls
& Baylis, 1989).
While PFE probably emerged from research on the perception of facial
identity, it is also rooted in the study of the structure of the emotional system.
The study of PFE has consequently inherited the baggage of long lasting debates
both about the nature of facial perception (i.e., whether it is categorical, holistic,
or configural (Farah, Wilson, Drain, & Tanaka, 1998) and about the structure of
emotion (i.e., whether or not it is categorical or dimensional; see Kappas, this
volume; see Kaiser, this volume).

CURRRENT CONCEPTIONS OF PERCEPTION OF FACIAL EMOTIONS

The categorical model a/perception o//acial emotions

According to classical principles of categorical perception, the products of the

perception of facial expression of emotion fit into discrete categories. These
categories are assumed to have clear-cut boundaries, and within each category,
the perceived distances between exemplars are smaller than between exemplars
belonging to distinct categories. Experimental evidence shows, in fact, that the
two requirements: a) abrupt change in the percentage of response when crossing
the category boundaries and b) chance level within-category discrimination, are
fulfilled . Using the ABX experimental paradigm'the categorical hypothesis has
been tested and supported in human adults (Calder, Young, Perrett, Etcoff, &
Rowland, 1996; de Gelder, & Teunisse, 1997; de Gelder, Teunisse, & Benson,
1997; Etcoff & Magee, 1992) and in 7-month old human infants (Kotsoni,
1999). It has also been recently demonstrated that this categorical accuracy
decreases with age (Bruyer & Granato, 1999).
In the categorical perception approach there is no room for specificity of
categories of emotions; all the categories are assumed to be identical in terms of
their internal structures.

151
From categories to processes

Recently, the consensus in categorical perception has been challenged by

Massaro (1998). Massaro claims that facial recognition is a kind of pattern
recognition process in which categories of emotion are represented as prototypes
that "contain a conjunction of various properties called features" (Massaro,
1998, p. 61). The underlying process of recognition of a given expression is
based on an integration of features. A formal and quantitative representation of
such a process, the Fuzzy Logical Model of Perception (FLMP), was used to test
this notion. The model, which has received its validity in the field of speech
perception (with visual and auditory modalities as main features), was applied to
facial expressions with the upper and lower parts of the face as the main features
to be integrated. The model computes the proportion of truth values
corresponding to a stereotypical expression.
Using one computer-generated synthetic face (Baldi), initially created to
model the auditory and spatial perception of processing speech differentiating
"ba" and "da," Ellison and Massaro (1997) studied the expressions of anger and
happiness manipulating the upper and the lower part of the prototypic expression
features (namely the eyebrow and the corners of the mouth). Using a factorial
design with 5 levels of brow deflection crossed with 5 levels of mouth
deflections, participants judged the facial expressions using a forced-choice
format (happy vs. angry) or a 9 point rating scale. In both experiments, the
probability of identifying an angry face was found to vary continuously as the
brow "changes from a fully depressed and flattened position to a fully elevated
and arched position" (p. 220). Analogously, the probability of recognising a
happy face varied with these two features. The main conclusion was that the
subjective judgements of emotional expression were the result of an integration
of both features at the level of decision. Ellison and Massaro (op. cit.)
emphasised that in terms of goodness of fit of the FLMP, neither the categorical
nor the holistic models account well for the data. This is true of the categorical
model because categories are not discrete, and of the holistic model because it
cannot conform to a quantitative formulation.
One of the major differences between the classical categorical view and the
Fuzzy Logic Model of PFE concerns the perceptual units considered to be the
most primitive (in the sense of not further decomposable) in the decoding of
facial expression. According to Etcoff and Magee (1992) the expression itself is
the most primitive unit, and they make the implicit assumption that the process
of identifying an emotion in the face involves the same processes for all
emotions (at least the basic emotions). According to Ellison and Massaro (op.
cit.) the primitive units are not the entire faces but the facial features. They
make the distinction, in particular, between the contribution to perception played

152
by the upper and lower parts of the face, which is why this approach had been
referred to as the part-based model.
The FLMP provides an interesting alternative to the categorical views. Up to
now, this model has only been tested on synthetic faces and on the two
expressions of happiness and anger. Thus it does not provide an analysis of the
processes involved in fear recognition.

IN SEARCH OF NEURAL SUBSTRATES FOR FEAR RECOGNITION

From perceptual processes to failures ofrecognition: Fear as a case in point.

In pioneering research, Adolphs and colleagues (1994) reported a clinical

observation of a female patient with bilateral lesions of the amygdala that made
her unable to recognise fearful faces, regardless of their intensity. Nevertheless,
she demonstrated intact recognition of facial identity. The patient was also
unable to identify fearful features of facial expressions of emotions, or to
produce a drawing of a fearful face (instead she drew a baby), whereas she did
not present any difficulties in the perception of any other expressions of basic
emotions. Finally, the patient had difficulties in imaging fearful facial
expression and in making semantic links within the semantic domain of fear.
Initially, this observation, which has been cited many times since its publication
(Damasio, 1994), was supported by the results of a group study on left and right
lesions, brain-damaged and normal controls, and was replicated in single case
studies by Calder and colleagues (1996) and by Broks, Young, Maratos, Coffey,
Calder, Isaac, Mayes and Hodges (1998). These clinical findings, which suggest
a dissociation between fear versus other emotion recognition, constitute one of
the major starting points for the amygdala hypothesis as a causal factor of
failures in the recognition of fearful faces, despite some non replicated
contradictory findings (Hamann & Adolphs, 1999).

From lesions to normal functioning: Brain Imaging studies ofrecognition offear

expressions in normal subjects.

If the amygdala plays a crucial role in the recognition of fearful expressions,

then it should be possible to show that this neural substrate is differentially
involved in tasks of fear recognition. A convincing demonstration was reported
in a study by Morris, Frith, Perrett, Rowland, Young, Calder and Dolan (J 996),
in which PET scan was used as a tool for measuring the neural activity of
different brain structures. In this study, subjects were presented with fearful and
happy faces taken from the standard picture of facial affect (Ekman & Friesen,

153
1976) while cerebral blood flow was recorded in different areas of the brain.
Two important details of the experiment design must be noted. First, subjects
were instructed to respond in a "yes" or "no" format, indicating whether the face
was that of a female or a male individual, instead of indicating whether the
emotion expressed a particular emotion, as in the usual designs. In addition, to
make the task more credible, the hair was masked. Results showed increased
blood flow in the left amygdala when subjects viewed the fearful as compared
with the happy faces. This effect was shown to increase with the intensity of the
expression (using morphing procedures to exaggerate the emotional expression).
With the exception that only the left amygdala (not the right) was
differentially activated, whereas in the Adolphs et al (1994) studies, left brain
damaged patients were not impaired in facial fear recognition (nor the right, but
only bilateral brain- damaged amygdala patients), the findings of Morris et aI.,
(op. cit.), parallel the brain-damage studies. Thus, both kinds of evidence
reported here seem to support a causal role for a specific brain structure in the
processing of fearful spatial signals in humans. Further, they corroborate the
findings of animal research on a specific brain structure linked to a specific fear
emotion module. 2

Is the one-to-one causal relation between fear recognition and amygdala as

strong as assumed?

Role of the response format

In an in-depth re-analysis of their bilateral lesioned patient, Adolphs, Tranel,
Damasio and Damasio (1995) provided many interesting additional comments.
One of them deserves special attention. That is, the patient, despite her deficit in
the processing of fearful faces, was able to say "what sorts of situations would
evoke fear in people ... how frightened people tend to behave." Furthermore, in
her spontaneous verbalisations she was able to process the lexical units relevant
to the experience of fear. When asked to sort words semantically related to the
fear domains, she performed as well as normal subjects. The question raised by
these observations was: Do the amygdala lesions only impair the perception of
spatial facial signals or do they also impair the whole fear system?
Two kinds of evidence have been reported in favour of a limited impairment
for visual emotional cues. First, it has been shown that bilateral amygdala
damage does not impair recognition of emotional prosody (Adolphs & Tranel,
1999). Second, using normal subjects, in a fMRI experiment, Hariri,
Bookheimer, and Mazziotta (2000) showed that the amygdala is activated when
subjects are asked to decide if a fearful face is the same or different (the forced-
choice response modality does not require a verbal label), but not when subjects

154
are asked to label the emotion expressed in the stimulus (linguistic task). On the
contrary, in the latter case, activation of the amygdala is diminished and
simultaneously activation of neocortical areas is increased.
Interestingly, these findings fit with the conceptualisation of the fear system
in terms of two routes to processing, as suggested by Ledoux's (1996) on rats.
According to this author
we can begin to see the outline of a fear reaction system. It includes parallel
transmission to the amygdala from the sensory thalamus and sensory cortex. The
subcortical pathways provide a crude image of the external world, whereas more
detailed and accurate representations comes from the cortex (p. 165).

Thus, a high and a low road can be followed by a fearful spatial signal. The low
road is more direct and hence faster; it only requires the signal to be deciphered
in terms of its positive or negative valence. The high road is indirect and hence
slower; it requires the signal to be first perceived, then analysed in terms of its
semantic meaning, which can be innate or acquired through learning.
The two-route hypothesis provides support for an explanation in terms of, not
just one, but multiple causes. However, this conceptual hypothesis is too general
to explain why the processing of facial signals of fear, as compared with other
emotional facial signals, would be special with regard to brain structures, a point
that has been and continue to be a source of experimental investigation (see
brain imaging studies on disgust, anger, and sadness by Phillips, Young, Senior,
Brammer, Andrew and Calder (1997) and Blair, Morris, Frith, Perrett, and Dolan
(1999) as examples of research looking for a specific neural network for each
prototypical facial expression).

Fear recognition and the difficulty hypothesis

Among the reasons that have been put forward to explain the specificity of
fearful facial signals, one of them, the "difficulty" hypothesis, deserves special
attention. In an extensive review of the cross-cultural literature, Rapcsak,
Galper, Comer, Remnger, Nielsen, Kaszniak, Verfafelli, Laguna, Labiner, and
Cohen (2000) found strong evidence for a distinctive failure to recognise fear, as
compared to the other emotional facial patterns. The result has also been found
within-culture as we will discuss in greater detail below. In general, happiness
and fear are at the opposite poles of a continuum of recognition complexity. On
the basis of this assumption, Rapscsak and colleagues compared rates of
recognition of facial expressions of emotion, including fear and five other
expressions, in a large sample of brain-damaged patients (some of them with
amygdala lesions) and normal controls. Results showed that both patients and
controls made more errors for fear than for all the other emotions (focal brain
lesioned patients being more impaired whatever the site of the lesions right, left

155
or bilateral). When scores were adjusted for differences in overall recognition
performance, no evidence for a disproportionate impairment for fear was found.
The implications of this research clearly suggest that the strict relationship
between fear recognition and one single neural substrate does not hold. For
example,
Fear recognition deficits in neurologic patients may be attributable to task
difficulty factor rather than damage to putative neural systems dedicated to fear
perception (Rapsack et aI., p. 575).

FROM ANATOMICAL STRUCTURES TO PERCEPTUAL PROCESSES:

WHY FEAR RECOGNITION SHOULD BE MORE DIFFICULT?

The Difficulty hypothesis re-examined through the study of confusions between

emotions.

Early studies and promising explanations

There exists substantial empirical support for Rapscak et aI., (2000) difficulty
hypothesis. Initial support comes from a paper by Tomkins and Mc Carter
(1964). These authors proposed a kind of a "theory of errors" in facial
recognition of emotions. The theory is based on findings of several experiments
in which subjects were asked to identify posed photographs expressing eight
basic emotions (interest, enjoyment, surprise, distress, fear, shame, contempt and
anger plus a neutral expression), and rates of correct recognition were compared.
When the three most well-recognised stimuli are considered (thus excluding
errors that could be explained by a poor stimulus quality), fear, along with
interest, was the worst recognised expression (with a recognition rate of only
58.7 and 59.7 for interest and fear respectively). Such percentages are clearly
just above chance, whereas the rate of recognition of enjoyment was close to
perfect (91.5 %).
Why is fear recognition so poor and why is fear often mistaken for interest
and surprise? According to Tomkins' theory of recognition of facial affect,
facial expressions are associated with innate neural firing patterns. The major
reason for misidentification of fear as interest and surprise is attributed to their
similarities in terms of an innate neural firing gradient. More precisely, the
neural firing is fastest for surprise followed by fear and interest. Hence, along
the firing-time gradient, fear is so close to surprise that the probability of the two
expressions being mistaken is high. While the neural firing hypothesis appeared
quite speculative for many readers, especially at the time it was put forth
(because little was known about sequential analysis of neural activation during

156
face perception), it does not seem totally unfounded in the light of some recent
evidence provided by single neuron recording studies on monkeys by Sugase et
aI., (1999)3.

Difficulty hypothesis, facial features and underlying process of fearful face

perception. Recent research
Another reason that fearful expressions might be more difficult to recognise is
that the fearful facial pattern is more complex than it is for other emotions. In
the research that we present below, we considered two aspects of complexity
thought to be important: the number of relevant features, and the way in which
the relevant features are combined. Several possibilities were envisaged. First,
the presence of only one feature is sufficient for expression recognition such that
when this feature is present, the recognition of the emotion is guaranteed.
Second, it is possible that recognition requires the presence of two features. In
such a case, either the weight of each feature is equivalent (no dominance), or
the weight of the two features is different, with one more dominant than the
other. Obviously in the latter case, the recognition task is more complex because
the computation does not correspond to a simple addition, but to a
multiplication.

Relevant facial features

Accordingly, we designed two experiments to address the relative importance of
two main facial features, the upper and the lower parts of the face, in the
recognition process, and the way in which these two parts are combined in the
perception of facial expressions of fear. Consistent with past and more recent
research, we hypothesised that the two halves of a face are among the most
relevant features for facial recognition of emotions in general, and fear in
particular. Several findings supporting this two-half view, have been reported
by Dunlap (1927) Bassili (1979) and Katsikitis (1997).
Dunlap (1927) argued that, contrary to common-sense belief, the eyes are not
the most important features in the recognition of facial expression of emotion.
Using, for the first time, the experimental paradigm of hand-made chimeric faces
(i.e., faces composed of two parts of a same model expressing distinct emotions),
he showed that for, at least some emotions, the lower part of the face is the
critical feature that provides for correct recognition. Although the Dunlap result
has been questioned in several studies, there is general agreement on the role of
specific parts of the face in the recognition of different expressions.
This issue was also investigated by Bassili (1979). In this study, a
recognition procedure was used in which subjects were presented full, bottom, or
top-half faces expressing six basic emotions (happiness, sadness, fear, surprise
anger or disgust). In case of top and bottom half presentations, the other part

157
was hidden with black makeup. Interestingly, the percentages of recognition for
fear were 95% for the full face, and 89.2% and 78.8% for the bottom and the top
parts of the face, respectively. These results favour the hypothesis that, at least
for fear, the upper part is a dominant feature, although the role of the lower part
is not negligible.
Such a dominance for the upper part of the face for fear was also evidenced
by Katsikitis (1997). Using full face patterns of the so-called basic emotions
(both realistic and line-drawing stimuli), she demonstrated that emotions can be
represented through multidimensional scaling procedures of data processing, in
terms of two dimensions. One was interpretable in terms of the well established
evaluative, or valence, dimension. The other dimension, of perhaps greater
interest here, contrasts upper and lower face dominance. Within this bimodal
representation of the representational space, fear characterised by an upper face
dominance was contrasted to happiness with a lower part dominance. The
second dimension, which has not been observed in previous factorial studies,
appeared to reflect spatial features and likely that it is specific to figural nature
of the stimulus material. While these findings enlighten the nature of the relative
weight of features in the recognition of facial expression, they do not say much
about how subjects compute the fearful value of a face on the basis of these
relevant features.
In order to enhance understanding of such combinatory processes, we used
the experimental paradigm of "chimeric faces." Chimeric or composite faces are
faces in which one part (either the upper or the lower part) is replaced with
another (from a different expression). A new morphing program (Nahas &
Huitric, 1999) was developed to allow independent manipulation of the upper
and the lower parts of the face without causing the resulting face to look
unrealistic or bizarre 4 • From a standard set of prototypical expressions (Ekman
& Friesen, 1976), we were able to make new faces composed of different facial
expressions. This procedure allows a critical test of the relative importance of
the parts in the whole and possible interactions between the parts. An obvious
advantage in comparison to the synthetic face "Baldi" used by Massaro and
Cohen (1990) for a similar purpose, is that the manipulations are done on
realistic expressions. This procedure was used to examine the probability of
recognising fear in facial stimuli when incongruent signals of fear and happiness
are presented (Experiment I) or when facial signals are incomplete (Experiment
2). Examples of the material used in both studies are presented in Figure I.

158
 FROM DOMINANCE TO NECESSITY AND SUFFICIENCY

Facial perception offear as a conjunctive perception process.

Experiment 1
Subjects were presented with prototypical expressions of fear and happiness, and
with computer-generated expressions made of a cross combination of the upper
and the lower part of each emotion (fearful-lower, happy-upper and happy-
lower, fearful-upper). They were asked to identify the emotion expressed by
each face from an extensive list of emotion- words. This list includes not only
words denoting the five basic emotions, plus surprise but also words sampled
from an extended lexicon, taken from a study by Rosenberg and De Boeck,
1997. Data were processed by the hierarchical non linear combination of
features (van Mechelen, De Boeck, & Rosenberg, 1995). Results showed that
for fear to be recognised, both the upper and the lower parts of the face were
necessary and sufficient. Arguments for a nonlinear combination of the facial
features for fear were provided by the goodness of fit of the model with the data
conditions (see Bonis, De Boeck, Perez-Diaz & Nahas, 1999 for a detailed
presentation of the results). The conjunctive nature of the underlying process is
illustrated in Figure 2, in which the rates of recognition for fear and for
happiness are plotted.
As shown in the upper panel there is a striking difference between fear and
happiness recognition. The percentage of recognition of happiness is high when
only the lower part of a happy face is present. On the contrary, neither the lower
or the upper part is sufficient to decide that the fear expression is present.
Inspection of the slopes of the curves confirm that happiness is "a single-feature
emotion," while fear is a "double-feature emotion." This suggests that the
underlying process is of a conjunctive kind for fear but not for happiness. It
remains to be seen if conjunctivity is specific to fear or if it also holds for other
expressions (de Bonis, De Boeck, Perez-Diaz & Nahas, in preparation).

159
 Upper Part
Happy Neutral Fearfu ~--I

...
Q)
3:
o
...J

Figure I. The 9 types offaces displaying fear. happiness and their composites. On the
centre face is neutral. Note that the upper and lower parts are strictly the same in
composites.

160
 First Experiment
100 100
!'O. ~ognftJon
,H_
% RtcogntUon

""
tAH.~u

""
80 80

70 70 71,11

GO .0

50 50
40 40

""
30

20 20

1O 1O

0
FF HF FH HH HH
Second Experime~~o
100 , AKotnKlon
,. AlcognlUon
dHaptllllMn ,/F_
""
90

80 80
7.) ro
60 60

'0 50
40 40

:J() :;0

20 20

1O 16,00 1O

° NN HN NIt HH NN NF FN FF

Figure 2. Percentages of recognition offear versus happiness in prototypical and

chimeric expressions.

Experiment 2
In a second study the experimental paradigm of chimeric faces was extended to
systematic combinations of neutral and fearful parts. The visual material was
composed of 42 photographed faces of six models, three males and 3 females
(see Figure I for illustration). The same experimental design as in Experiment I.
Judgements were obtained from 20 right-handed volunteers (II males & 9
females) for eight types of faces. Subjects, were presented with the 42 faces,
each on one black and white slide, arranged in a random order, and presented on
a screen for I min. The subjects were asked to indicate as many emotions they
perceived in each face on the same extensive list of 19 emotion-words used in
Experiment I. Results are presented in the lower panel of Figure 2. The

161
percentage of recognition of happiness (left panel) clearly shows that even when
the full face is not a full prototypical happy face (because only the happy lower
part is present, the upper part being neutral) the probability of recognising
happiness attains a rate of 100%. Adding the upper happy part can not improve
the recognition rate. This result on happy faces confirms those obtained in
Experiment 1. The pattern of results is quite different for fear (right panel).
First, the overall recognition of fear is not as high as for happiness, as shown by
the slopes of the curves. Second, the rate of recognition of fear, when only the
upper part is present (FN) is quite high, confirming that the upper part is a salient
feature for fear.
Taken together, the results of the first and second studies suggest that the
perception of fear in a face is the result of the combination of the two parts of the
face (conjunctivity of perceived features), in which the upper part is a dominant
feature. When this dominant feature is associated with a lower part expressing a
contrasted emotion (happiness) the process of recognition is interrupted, the fear
response is inhibited. When the upper dominant feature is associated with a non-
expressive lower part, fear recognition nevertheless takes place.
Two alternative interpretations of these results can be proposed. One
interpretation is that the subject is victim of an illusion. Whereas the lower part
has no fearful meaning at all, it is nevertheless judged as fearful. Such an
interpretation is not totally new. Duchenne de Boulogne (1862) reported this
perceptual phenomenon for sad expressions. When he electrically stimulated
only the upper part of the face, leaving the lower part at rest, he reported the
impression that the lower part was congruent with the emotional expression
created by the stimulation of the upper part (See Bonis & Nahas, 2000 for a
detailed illustration).
In addition, the complex interactions between different parts of the face were
also mentioned by Ekman and Freisen (1975) who wrote that
with many facial expressions a change in just one area gives the impression that the
rest of the facial features have changed as well (p.39).

However, the intensity of the biases had never been reported. Thus the
manipulation of one part of a face has an influence on the perception of the other
part. This influence is dependent on the part manipulated. The bias can be
interpreted either in terms of some kind of optico-geometrical illusion or in
terms of a kind of field effect: modification of the whole through the
modifications of the parts. For the illusory perception hypothesis to be retained,
one has to infer an interaction between the information provided by the two parts
of the face, and to quantify this interaction (de Bonis, De Boeck, Perez-Diaz, in
preparation ).
The other interpretation, which is not exclusive from the 'illusory perception'
refers to the decision processes underlying face recognition of emotion.

162
According to this second hypothesis, when incomplete information is given, the
subject is faced with two choices: either to answer 'no' in which case, he risks
missing the important meaning of the stimulus, or to answer 'yes' in which case
he risks being wrong. What is the overall meaning of this response strategy?
One can speculate that, in terms of evolutionary advantage (a recurrent issue in
the field of fear recognition, see Mazurski, Bond & Siddle, 1996; Ohman,
1986,), it is better to be wrong but safe, than to be accurate, but unsafe. It is
likely that the tendency of our subjects to detect an emotional signal, despite
incomplete information, is evidence of the second strategy. However further
research is needed to ascertain whether the processes suggested for fear is also
valid for the recognition of emotions such as anger which shares with fear an
adaptive advantage.

CONCLUSIONS

This chapter has addressed the issue of the perception of facial expression of
emotion from the perspective of failures in recognition of one target emotion,
that of fear . We have reviewed some of the causes of total blindness for fear,
and some of the reasons for partial failures to recognise fear.
With regard to the former issue i.e. total blindness, the neuropsychological
findings reported emphasise the importance of modelling the perceptual
mechanisms involved in facial recognition of emotion. It remains to show
whether total blindness is due to either a global impairment of a putative fear
system, or to a specific deficit of facial feature integration. With regard to the
latter issue i.e. partial failures, evidence has been provided that normal subjects
are far from accurate in fearful face recognition. Two types of errors have been
distinguished. The first is recognising emotions other than fear when
prototypical fearful signals are present, and the second is attributing a fearful
meaning despite incomplete facial signals of fear (illusory perception). The
issue raised by these findings has some implications on facial measurement. If a
human observer, even trained and assisted by a coding manual, as precise as it
can be, cannot escape from such errors because of constraints of his visual and
neural equipment. Thus, automatic devices of facial measurement must be to
promoted at least for complex configurations such as fearful expressions.

AFFILIA TION

Directeur, Centre National de la Recherche Scientifique

Unite de Resonance Magnetique Mediacale (U2R2M),
Universite Paris Xl.
Email: Monique.debonis@kb. u-psud.Jr

163
 ACKNOWLEDGMENTS

I am very grateful to Paul De Boeck, Fernando Perez-Diaz, Monique Nahas and

to Paul Ekman for permission to use and manipulate photographs from the
Ekman and Freisen (1976) Pictures of Facial Affect series. I also thank Patrick
Vanessche for technical assistance.

NOTES
'In the ABX paradigm, the subject is presented a set of three stimuli (photographs). A and Bare
different and X is identical either to A or B. The subject is instructed to indicate to which
stimulus (A or B) X is identical.
2Within the scope ofthis chapter, it is not relevant to discuss the inferences drawn from these
studies about the amygdala functioning within the brain systems (theoretical assumptions about
segregation, integration, neuromodulation ). However, It seems likely that, brain imaging
studies of facial expressions tells us more about the brain functioning than about facial
perception of emotion itself.
) In this study, monkeys were presented with several types of stimuli, including facial expressions of
happiness, surprise and anger displayed by monkeys and humans. During stimulus presentation,
single neuron activity was recorded in the temporal visual cortex. Findings showed a temporal
difference in firing pattern latencies. Early firing neurons appeared to process the more global
information (monkey vs. human face; faces vs. geometric shapes) while later firing neurons
processed fine-grained information about identity and expression. These results suggest a
temporal gradient for recognition of identity, facial expression of emotions, and geometric
shapes.
• It is important to note that this kind of morphing procedure is different from that which has been
used in other current studies in which the resulting morphed fearful-happy does not contain any
features of the original prototypes (see Nahas & Huitric, 1999), thus preventing a systematic
test of the role of the parts in the whole.

REFERENCES
Adolphs, R., Tranel, D., Damasio, H. & Damasio, A. (1994) Impaired recognition of emotion in facial
expressions following bilateral damage to the human amygdala. Nature, 372, 669-672.
Adolphs, R., Tranel, D., Damasio, H. & Damasio, A. (1995) Fear and the human amygdala. The Journal
of Neuroscience, 15, 5879-5891.
Adolphs, R. & Tranel, D. (1999) Intact recognition of emotional prosody following amygdala damage.
Neuropsychologia, 37, 1285-1292.
Bassili, l.N. (1979) Emotion recognition: the role of facial movement and the relative importance ofth
upper and lower areas of the face. Journal of Personality and Social Psychology, 37, 2049-2058.
Benson, P. (1999) A means of measuring facial expressions and a method for predicting emotion
categories in clinical disorders of affect. Journal of Affective Disorders, 55, 179-185.
Blair, R.1 ., Morris, 1.S., Frith, C.D., Perrett, D.1., Dolan, R.1. (1999) Dissociable neural responses to
facial expressions of sadness and anger. Brain, 122, 883-893.
de Bonis, M., De Boeck, P. , Perez-Diaz, F. & Nahas, M. (1999) A two-process theory of facial
perception of emotions. C. R. Academie des Sciences (Life Sciences). 322, 1-7.
de Bonis, M. & Nahas, M. ( in press) The Duchenne's illusion. Proceedings of the XIth Conference of
the International Society for Research on Emotions, 16-20 August (Summary).

164
de Bonis, M., De Boeck, P. , Perez-Diaz, F. & Nahas, M. (in preparation). A part-based model for facial
perception of emotion: Linear combination and dominance
Broks, P., Young, A.W., Maratos, EJ., Coffey, PJ., Calder, A.J., isaac, C.L., Mayes, A.R., Hodges, J.R.
et al. (1998) Face processing impairments after encephalitis: Amygdala damage and recognition of
fear. Neuropsychologia, 36, 59-70.
Bruyer, R. & Granato, P. (1999) Categorical effects in the perception of facial expressions: M.A.R.I.E.
A simple discriminating clinical tool. European Review of Applied Psychology, 49, 2-20.
Calder, AJ., Young, AW., Perrett, D.L., Etcoff, N.L. & Rowland, D. (1996) Categorical perception of
morphed facial expressions. Visual Cognition, 3, 81-117.
Calder, A.J., Young, A.W., Rowland, D., Perrett, D.I., Hodges, J.R. and Etcoff, N. (1996) Facial
emotion recognition after bilateral amygdale damage: Differentially severe impairment of fear.
Cognitive Neuropsychology, 13,699-745.
Calder, A.J., Young, A.W., Keane, J. and Dean, M. (2000) Configurational information in facial
expression perception. Journal of Experimental Psychology, Human Perception and Performance,
26, 527-551.
Damasio, A. (1994) Descartes' error: Emotion, reason and the human brain. New York,
Grosset/Putnam.
Damasio, AR. (1998) Emotion in the perspective of an integrated nervous system. Brain Research
Review, 26, 83-86.
Darwin, Ch. (1872) The expression of emotions in man and animals. Chicago, University of Chicago
Press. (1965)
Dunlap, K. (1927) The role of eye-muscle and mouth-muscles in the expression of the emotions.
Genetic Pychology Monographs, 2, 199-233.
Dolan R.J. & Morris, J.S. (2000) The functional anatomy of innate and acquired fear. Perspective in
Neuroimaging. in: R.D. Lane & L. Nadel (Eds.), Cognitive neuroscience of emotion (pp.225-24I ).
New York, Series in Affective Neuroscience.
Duchenne de Boulogne, G.B. (1862) Mecanisme de la physionomie humaine (Analyse electro-
physiologique de ['expression des passions) Paris, Jules Renouard.
Ekman, P. and Friesen, W.V. (1978) The Facial Action Coding System (F'ACS): A technique for the
measurement offacial action. Palo Alto, CA, Consulting psychologists Press.
Ekman, P. & Rosenberg, E. (1997) What the face reveals. Basic and applied studies of spontaneous
expressions using the Facial Action Coding System (FACS), New York, Oxford University Press.
Ellison, J.W. & Massaro, D.W. (1997) Featural evaluation, integration, and judgment of facial affect.
Journal of Experimental Psychology: Human Perception and Performance. 23, 213-226.
Etcoff, N.L. & Magee, J.J. (1992) Categorical perception offacial expressions. Cognition, 44,227-240.
Farah, J., Wilson, K.D., Drain, M. and Tanaka, J.N. (1998) What is "special" about face perception?
Psychological Review, 105,482-498.
de Gelder, B., Teunisse, J.P., Benson, P.J. (1997) Categorical perception of facial expressions:
Categories and their internal structure. Cognition and Emotion, 11, 1-23.
Hamann, S.B. & Adolphs, R. (1999) Normal recognition of emotional similarity between facial
expressions following bilateral amygdala damage. Neuropsychologia, 37, 1135-41.
Hariri A.R., Bookheimer, S.Y. & Mazziotta, J.C. (2000) Modulating emotional response effects of a
neocortical network on the limbic system. Neuroreport, 11,43-48.
Hasselmo, M.E., Rolls, E.T. & Baylis, G.C. (1989) The role of expression and identity in the face
selective responses of neurons in the temporal visual cortex of the monkey. Behavioural and Brain
Research, 32,208-218.
Jitsumori, M. & Yoshihara, M. (1997) Categorical discrimination of human facial expressions by
pigeons: A test of the linear feature model. Quarterly Journal of Experimental Psychology, 50B,
253-268.
Kaiser, S. (2002). Expressions as indicators of "functional" and "dysfunctional" emotional processes. In
M. Katsikitis (Ed.), The human face: Measurement and meaning (pp. xxx-xxx). Dordrecht: Kluwer
Academic Publishers.

165
Kappas, A. (2002). What facial activity can and cannot tell us about emotion. [n M. Katsikitis (Ed.), The
human face: Measurement and meaning (pp. 215-234). Dordrecht: Kluwer Academic Publishers.
Katsikitis, M. (2002). FACEM: The Facial Expression Measurement system. [n M. Katsikitis (Ed.), The
human face: Measurement and meaning (pp. 99-118). Dordrecht: Kluwer Academic Publishers.
Katsikitis, M (1997) The classification of facial expression of emotion: a multidimensional-scaling
approach. Perception, 26, 613-626.
Katsikitis, M. & Pilowsky, I. (1988) A study of facial expression in Parkinson's disease using a novel
computer-based method. Journal of Neurosurgery and Psychiatry, 51, 362-366.
Katsikitis, M. & Pilowsky, I. (1991) A controlled quantitative study of facial expression in Parkinson's
and depression. Journal of Neurology Neurosurgery and Psychiatry. The Journal of Nervous and
Mental Disease, 179, 683-688.
Katsikitis, M., Pridmore, S. & Marzullo, M. (1999) The facial measurment system in the assessment of
the efficacy of transcranial magnetic stimulation in the treatment of depression. European Review of
Applied Psychology, 49, [23-128.
Kotsoni, E. (\ 999) Categorical perception of facial expressions by 7-Month-old human irifants. A
summary. Unpublished Manuscript, Catholic University of Leuven, Belgium.
Ledoux, J. (1996) The emotional brain; The mysterious underpinnings of emotional life. Simon &
Schuster, New York.
Lien, 1.1.J., Kanade, T. & Cohn, P.J.F. (2000) Detection, tracking, and classification of actions units in
facial expression. Robotics and Autonomous Systems, 31, 131-146.
Massaro, D.W. (\998) Perceiving talking faces: From speech perception to a behavioural principle.
Cambridge, Cambridge University Press.
Massaro, D.W. & Cohen, M.M. (\990) Perception of synthesized audible and visible speech.
Psychological Science, 1, 55-63.
Morris, J.S., Frith, e.D., Perrett, D.I., Rowland, D., Young, A.W., Calder, A.J. & Dolan, R. J. (1996) A
differential neural response in the human amygdala to fearful and happy facial expressions, Nature,
383,812-815.
Nahas, M. & Huitric, H. (1999) Synthesis of faces: A tool for experiments on facial expressions.
EuropeanReview of Applied Psychology, (Special Issue) 49, 141-149.
Ohman, A. (1986) Face the beast and fear the face: Animal and social fears as prototypes for
evolutionary analysis of emotion. Psychophysiology, 23, 123-145.
Philipps, M.L., Young, A.W., Senior, C., Brammer, M., Andrew, C. & al. (\997) A specific neural
substrate for perceiving facial expression of disgust. Nature, 389, 495-498.
Philips, M.L., Williams, L., Senior, C., Bullmore, E.T., Andrew, C., Williams, S.e.R. & David, A.S.
(1999) A differential neural response to threatening and non-threatening negative facial expressions
in paranoids and non paranoid schizophrenics. Psychiatry Research Neuro-Imaging, 92,11-31.
Phillips, M.L., Young, A.W., Senior, C., Brammer, M., Andrew, C., Calder, A.J. & aI., (1997) A
specific neural substrate for perceiving facial expression of disgust. Nature, 389, 495-498.
Piaget, J. (1961) Les mecanismes perceptifs. Paris, PUF.
Pilowsky, I. & Katsikitis, M. (\994) The classification of facial emotions: A computer-based taxonomic
approach. Journal of Affective Disorders, 30, 61-71.
Rapcsak, S.Z., Galper, S.R., Comer, J.F., Remnger, S.L., Nielsen, L. , Kaszniak, A.W., Verfafellie, M.,
Laguna, J.F., Labiner, D.M . & Cohen, R.A. (2000) Fear recognition deficits after focal brain
damage: a cautionary note. Neurology, 54, 575-581.
Rosenberg S. & De Boeck P. (1997) Emotion depicted and experienced: Picasso's Portraiture . In L.
Dorfman, e. Martindale, D.G., Leontiev, V. Cupchick, V. Petrov, & P. Matchotka (Eds.), Emotion,
creativity & arts. (pp. 371-386). Penn: Institute of Art and Culture.
Shuttleworth Jr, E.e., Syring, V. & Allen, N. (1982) Further observations on the nature of
prosopagnosia. Brain and Cognition, 1,307-322.
Stewart-Bartlett, M., Hager, J.e., Ekman, P., Sejnowski, T.J. & aI., (1999) Measuring facial expressions
by computer image analysis. Psychophysiology, 36,253-263.

166
Sugase, Y., Yamane, S. & Veno, S. (1999) Global and fine information coded by single neurons in the
temporal visual cortex. Nature, 400, 869-872.
Tiberghien, G. & Clerc, I. (1986) The cognitive locus of prosopagnosia. In R. Bmyer (Ed.), The
neuropsychology offace perception andfacial expression. Lawrence Erlbaum, Hillsdale, pp. 39-62.
Tomkins, S.S. & MC Carter, R. (1964) What and where are the primary affects? Some evidence for a
theory. Perceptual and Motor Skills, /8, 119-158 (Monograph Supplement, I-VI8).
van Mechelen, I., De Boeck, P. & Rosenberg, S. (1995) The conjunctive model of hierarchical classes.
Psychometrika, 60, 505-521.
Young, A.W., Rowland D., Calder, A.J., Etcoff, N.L., Seth, A. & Perrett, D. (1997) Facial expression
Megamix: tests of dimensional and category accounts of emotion recognition. Cognition, 63, 271-
313.

167
 HARALD G WALLBOTT

CHAPTER 9

RECOGNITION OF EMOTION IN SPECIFIC

POPULATIONS: COMPENSATION, DEFICIT OR
SPECIFIC (DIS)ABILITIES?

INTRODUCTION

Numerous studies are available on differences in nonverbal sensitivity, i.e. the

ability to decode emotions and attitudes from nonverbal expressive behaviour. It
has been shown that persons differ to a large degree in their ability to decode
emotions from facial expressions and other nonverbal channels (Hall, 1978,
1984; Rosenthal, 1979, Rosenthal, Hall, DiMatteo, Rogers, & Archer, 1979;
Russell & Fernandez-Dols, 1997; Wallbott, 1998). In addition, numerous
studies on the topic of nonverbal sensitivity have been conducted in the clinical
area, indicating for instance, deficits of schizophrenic patients in recognising
emotions (Cutting, 1981; Johnston & Carr, this volume; Johnston & Katsikitis,
200 I; MandaI, 1986; Muzekari & Bates, 1977).
Given this multitude of studies, it is surprising that only few studies are
available on nonverbal sensitivity of the pre-lingually hearing impaired or of
blind persons, as well as on other handicapped groups. One might argue that it
is especially important for hearing impaired persons for example, to gather
information of emotional states of their interaction partners from visual cues like
facial expression or body posture, because emotional information from vocal
cues (like voice quality) is less available to them. Vice versa, this is true for
blind persons lacking information on facial expressions and other visual
information on the emotional state of the interaction partner. Demonstrating
deficits in recognition of emotion from nonverbal (visual) expression in hearing-
impaired persons as well as deficits for vocal expression in blind persons would
have important practical consequences (for training programs etc.; cf. Reynolds,
1978), because nonverbal communication of emotion is an important factor in
interaction and communication in general (Wallbott, 1990).
Reynolds (1978), in his review, contrasted the 'compensation' hypothesis
(handicaps in one sensory channel are compensated by increased
ability/sensitivity in other sensory channels), with the 'deficit' hypothesis,
(handicaps in one sensory channel are associated with deficits in other sensory
channels), with regard to perception in general. Studies in the area of tactile and
visual perception seem to support the compensation hypothesis for hearing
impaired persons (Hoeman, 1978; Larr, 1956; Schiff & Dytell, 1972; for studies
with blind persons see Blau, 1964). Other studies in the area of visual
perception support the deficit hypothesis (Myklebust & Brutten, 1953).
It should be mentioned that talking about the compensation hypothesis and
the deficit hypothesis here is not to be understood from the perspective of
psychology of perception, but more from a social-psychological perspective.
Psychology of perception discusses the issue with respect to modularity of
sensory channels or with respect to the interdependence or independence
between sensory channels, as does the neuropsychological literature. If one
considers sensory channels and processing of different sensory information
independent of each other, one would expect neither deficit nor compensation
when one sensory channel is lost (for a discussion of independence or
interdependence of senses from a developmental perspective see Hatwell, 1987;
Streri, 1993). Some authors on the other hand, claim the existence of mandatory
bi-directional links between emotion recognition structures from visual and
vocal information (de Gelder & Vroomen, 2000; de Gelder, Vroomen, &
Poutois, 1999), which would predict deficits when losing one sensory channel.
Literature on sensory impairment, on the other hand, usually starts from a
more common sense understanding of compensation and deficit in arguing that
persons having lost one sensory channel are forced to at least try to compensate
for that loss, which means that such persons would invest more attention in the
senses they have left and thus maybe even 'sharpening' these (=compensation).
We also start from this common sense, or social-psychological understanding
here. Given the importance of nonverbal cues for emotion in interaction and
communication, it is asked whether persons having lost one sensory channel
(sight or hearing) are forced to focus attention on the remaining senses to obtain
information about emotional states of others.
Studies testing the notion of deficit vs. compensation for emotion recognition
of hearing-impaired persons (and also for blind persons) from nonverbal cues,
on the other hand, are relatively rare. Some studies used schematic faces (highly
abstracted line drawings, Schiff, 1973a, 1973b), and indicated more errors in
emotion decoding of deaf compared to normal hearing subjects. Other studies

170
were conducted with more ecologically valid stimulus material, namely films or
video recordings of actors expressing emotions via facial expressions, hand
movements, and body postures (Sugarman, 1969, cited in Schiff & Thayer,
1974). The study by Sugarman indicated that deaf persons were less accurate in
judging the expressed emotions than normal hearing subjects. Finally, emotion
recognition from visual stimuli of hearing impaired persons (pupils \ 0-\5 years)
was tested using the PONS (Profile of Nonverbal Sensitivity) test by Rosenthal
and his colleagues (1979), a test to determine differences in nonverbal
sensitivity. Less impaired persons reached higher scores in nonverbal sensitivity
than highly impaired and deaf persons (degree of hearing impairment was varied
via a median split of the group of hearing impaired subjects). An additional
study by Rosenthal et al. (1979) with deaf students (i.e. older subjects) and
normal hearing subjects, failed to find differences between both groups. The
authors argue that hearing impairment may be related to a retarded development
of social skills in childhood, which may be later fully compensated in early
childhood.
Taken together, the few studies available in the area of emotion decoding
abilities from visual stimuli of hearing impaired or deaf subjects seem to support
the deficit hypothesis, but not the compensation hypothesis ..
In a study conducted some years ago (Wallbott & Seithe, 1993) we tested
whether hearing impaired subjects would differ from normal hearing subjects in
their ability to decode emotions from video stimuli (48 video takes in which two
actors portrayed six different emotions). Studies in tactile and visual perception
lead one to expect deficits, while there is also some evidence for compensation.
Twenty six hearing impaired subjects and 26 matched normal hearing subjects
participated (average age=25.5 years; 9 female, 17 male subjects in each group).
Results indicated that in general, hearing impaired subjects were slightly, though
not significantly, less successful in decoding emotions from the visual stimuli
than the normal hearing subjects. A comparison between highly (loss> 60-90
dBA) and medium (loss about 30-60 dBA) impaired subjects on the other hand
indicated significantly worse emotion decoding, not for the highly impaired, but
only for the medium impaired group. The major results are depicted in Figure I.
One might argue that this somewhat surprising result was due to the fact that
highly impaired subjects are forced to compensate their sensory deficit and
indeed seem to do so, while medium impaired subjects are entangled in a role
conflict, standing somewhere between normal hearing persons and deaf persons,
having retained some of their hearing ability, maybe so much that they never felt
forced to try to compensate. This, in turn, may result in the deficit observed,
because they may consider themselves normal hearing and functioning persons,
but they are not.

171
 Recognition ntle (difference to mean)

o H I
·1

·2

.)

·5

-6
Control Subjects Medium impaired Ss. Highly impaired Ss.
Experimental Groups
(26 impaired and 26 motched control s.)

Figure I: Emotion recognilionfrom visual stimuli and hearing impairment

(Wal/bot/ & Selthe. 1993)

This research lead to a number of studies, whose intention it was to shed

some more light on the compensation vs. deficit issue. It tried to identify specific
groups of persons where these opposing hypotheses might make sense. The
studies to be reported now in some detail resulted from this idea. A contrasting
issue to hearing impaired persons is obviously the situation of blind and visually
impaired persons. Here the compensation vs. deficit issue can be turned around,
this time focusing on the vocal channel as a means of possible compensation or
deficit for emotion recognition. For blind persons a compensation effect was
expected, specifically that the non-availability of visual cues would lead to better
emotion recognition from vocal stimuli.
Another study (or indeed two studies) were conducted not with a clinical
group, but with a group of persons, whose task involves handling and
recognising emotions in a specific way, namely telephone counsellors. This
branch of counselling is restricted to anonymous telephone calls, which of
course implies that the visual channel (facial expression etc.) is not available for
emotion recognition. Thus, this and their abilities for recognising from visual
and vocal stimuli seemed to be of some special interest for the question: Do

172
telephone counsellors compensate the lacking visual channel with respect to
recognition of emotion by developing an improved ability to recognise emotions
from vocal cues alone, i.e. do they "compensate" their technically dictated
'deficit'? As these counsellors interact with their clients solely through the
telephone, one might expect that they have developed specific abilities to detect
emotions from the voice alone.
Finally, dissocial juveniles were studied with respect to emotion recognition.
Some studies indicate that this group has indeed deficits in handling emotions, as
well as in expressing and recognising emotions. Here it was hypothesised that
this group is generally impaired in emotion recognition.

THE EMPIRICAL STUDIES

Blind Persons

While blind persons, at least if one considers the popular literature or anecdotes
are often taken as a prototypical example of compensation, the few empirical
studies available do not support this optimistic idea. Though it was found that
blind people were about equally as good as normal seeing subjects in identifying
everyday sounds and noises (Blau, 1964), some data indicate that they are worse
in identifying vocally encoded emotions (Blau, 1964; Minter, Hobson, & Pring,
1991). Thus, with respect to decoding emotions, the empirical evidence speaks
more for the deficit than for the compensation hypothesis. The study to be
presented here is an attempt to replicate these results with semi-standardised
stimulus materials, already used in a number of studies (Scherer, Banse &
Wallbott, 2001), because the stimulus material used in the previous studies had
been developed rather ad-hoc and often unsystematically.

Materials and Procedure

Four actors (2 male, 2 female) were used. All were professional actors who
regularly participated in radio and television productions. Voice samples were
elicited for the emotions of joy/happiness, sadness, fear, anger, and disgust
(disgust was later dropped because recognition rates turned out to be
unsatisfactory, for details see Scherer et aI., 200 I). These emotions were
selected because many theories of emotion agree that they can be considered as
basic and universal (Ekman, 1982; Izard, 1977).

173
 In research on emotional expression, encoders are often requested to portray
the emotional states on the basis of verbal labels only (e.g, 'Read this text as if
you were angry'). This procedure presents two problems; first that different
encoders may attribute different meanings to such labels, and second, that they
might envisage different situations as elicitors of these emotions. Such problems
can be avoided by using a scenario approach (Rosenthal et aI., 1979; Wallbott &
Scherer, 1986; Williams & Stevens, 1972). In this approach, actors are provided
with situation vignettes or short scenarios describing an emotion-eliciting
situation. They are instructed to imagine these situations and act them out as if
they were experiencing them.
For the stimulus material, which was also used here, scenarios were not
constructed a priori, but instead were selected from actual situation descriptions
collected in a large-scale, cross-cultural study (Scherer, Wallbott, &
Summerfield, 1986). For various methodological reasons the 'standard
sentence' approach was used. To select the most appropriate stimuli, means
were computed across judges in pilot studies (Scherer, Banse, Wallbott &
Goldbeck, 1991; Scherer et aI., 2001) for each emotion scale and each stimulus
separately. Then, the most unequivocal items were selected by comparing the
criterion (i.e. the emotion the actors attempted to encode) with the mean
intensity judgements of the respective emotion.
The 30 takes selected (with 2 practice takes to acquaint subjects with the
procedure) were edited in random order on a tape so that only the phase when
the actors produced the 'utterance' was audible.
To keep subjects' task in the present study easy, especially for blind subjects,
they were provided with a list of emotion terms (joy, fear, anger, sadness,
neutral) and their task was to select the correct term for each stimulus. Each
stimulus was played twice and subjects had as much time as they wanted for
each judgement. For the blind subjects the questionnaire was prepared in the
Braille alphabet.
Thirty blind subjects were recruited via contact with organisations of blind
persons in the area of Salzburg, 14 of were blind from birth and 16 blinded later
on. A matched control group of 30 subjects was collected, parallel with respect
to mean age (40.1 years for blind participants, 39.2 years for controls), gender
distribution (1511 5 in each of the two samples), as well as in education level (no
significant differences in school level and job status).

Results and Discussion

The results paragraph can be kept short here. It will not focus on possible
recognition differences for specific emotions, because no predictions were made,
but instead on the overall recognition rates by comparing the group of blind

174
subjects with the control group of normal seeing subjects. A mean recognition
rate of 25.0 was found for the control group, and a mean of 24.7 for the blinds.
This difference in an independent samples t-test is far from reaching any
significance, though the means might indicate that the blind subjects are slightly
worse. But statistically speaking they are as good as the subjects with normal
sight. This does not entirely promote the compensation hypothesis, but
evidently does not provide any conclusive evidence for the deficit hypothesis.
But might it be that within the group of blind subjects (as in the hearing
impairment study described above) there are differences in favour of one or the
other hypothesis? It might be speculated that persons blind from birth on have a
much greater demand to compensate than subjects only blinded in later years.
Though the means might point that direction (mean=24.2 for birth-blinds and
mean=23.8 for the other blinds), again the results are far from reaching
significance. One has to admit that by splitting a sample of 30 subjects into 2
sub-samples the Ns become rather small, thus additional follow-up studies might
be of interest. Taken together there is no conclusive evidence for deficits of
blind persons in general, and also not within the sample of blind persons, though
there is also no remarkable compensation effects of blinds when asked to judge
vocally transmitted emotions. Blinds are as good as persons with normal sight in
this task.

Telephone Counsellors

Another population was studied, which is of course not a clinical one, but
one, where sensory deficits also play an important role in daily conduct, here not
imposed by non-functioning sensory organs, but by technical restrictions,
namely telephone counsellors. Telephone counselling is a service free of charge,
available in many countries usually run by a church, by non-profit organisations,
or by the state. Telephone numbers are provided by such organisations where
persons in need (i.e. with social problems, psychological problems, suicidal
tendencies etc.) can call in anonymously, and the also anonymous-counsellor on
the other end of the line tries to provide first-hand help, minor therapeutic
intervention, or just gives the person in need the chance to talk to someone about
his/her problem, if only via the telephone. These telephone counsellors for the
most part work on a honorary basis in their free time, and usually an around-the-
day- service is provided. They are specifically trained to conduct these
telephone conversations, thus are 'semi-professionals' and provide 'semi-
clinical' service to those in need.
This implies that telephone counselling is a situation in which persons with
severe problems may call, and the counsellor is confronted with the task of
understanding what the other person's problem is, what his/her emotional state

175
 is (especially important when persons with suicidal tendencies call), and what
can be done to help via the telephone. Thus, for the counsellors it is a very
demanding situation, in which recognising the other's emotional state is an
important part. Contrary to face-to-face- counselling the available information is
restricted by the fact that the telephone counsellor cannot see his/her interaction
partner, i.e. facial expression and other important cues to emotion are lost.
Telephone counsellors' available information about emotional states is limited to
vocal and verbal information. Thus, on the one hand, telephone counsellors do
not have the complete spectrum of emotion cues available (i.e. no visual
channel), on the other hand a correct judgement of the others' state is essential
for their interventions or suggestions, and for that they are restricted to the vocal
(and verbal) channel. Thus, it may be asked whether telephone counsellors are
able to compensate the 'loss' of the visual channel. To our knowledge there are
no empirical studies available testing this issue. One might predict that due to
their specific situation, telephone counsellors are better than face-to-face
counsellors in recognising emotion from vocal cues, because their job forces
them to concentrate on these (i.e. they compensate), while on the other hand they
might be no better (or even worse?) in recognising emotion from visual cues in
comparison to counsellors working in face-to-face situations. An additional
speculation was that - if the general hypothesis is true - the differences between
both groups of counsellors might become even larger, the longer a counsellor,
especially a telephone counsellor, had worked on this job. The longer the
counselling experience the more sensitive telephone counsellors may become to
vocal cues to emotion.
Thus, an interaction with respect to decoding accuracy for emotions was
expected between type of counselling (telephone vs. face-to-face) and type of
information provided (vocal vs. visual).

Materials and Procedure

Two tasks were used. the series of 30 short utterances described above (Scherer
et aI., 1991, 200 I), and a subset from the "Pictures of Facial Affect" (Ekman &
Friesen, 1976). Though used in numerous studies a short description of this
stimulus set might be in order, because it was also used in the third study to be
reported later with dissocial juveniles. This series of slides (for the present
purposes the slides were transferred to print-outs) contains visual stimuli of
encoders encoding seven different emotions Goy, sadness, fear, anger, disgust,
surprise, contempt) according to the criteria by Ekman and Friesen (\ 976). The
stimulus set contains full expressions (full-blown facial expressions of the
respective emotions), partial expressions (reduced facial expressions, but still
unambiguous with respect to the expressed emotions) and 'blends' (mixtures of

176
 two emotions each) enacted by experienced encoders and selected in pilot
studies by the Ekman group. From the pictures of facial affect here only full
expressions were used, resulting in 28 stimuli presented to subjects (four per
emotion). The reason for this was to confront the subjects with about the same
number of stimuli in the vocal and in the visual domain.
Two studies were conducted, similar in design, with the same stimulus
materials and identical instructions. In one study recognition rates across
emotions were compared between a group of telephone counsellors and a group
of face-to-face counsellors (psychologists, social workers etc.), in another study
again a group of telephone counsellors was compared with a control group
working in non-counselling occupations. To recruit telephone counsellors for
participation, telephone counselling organisations from different parts of Austria
(in Austria usually telephone counselling is run by the church) were contacted to
get access to subjects. Participation was on a voluntary basis, instructions tried
to assure subjects that the study was not a 'test' by the organisation to test their
specific abilities in telephone counselling, but an independent and anonymous
study, where no individualised data would be reported to anybody (fear of the
telephone counsellors being 'tested' was rather high in some cases). The pictures
of facial affect were presented as photographs to subjects with no time limit for
judgements, the vocal stimuli in the usual audio tape format with each stimulus
presented twice. Subjects' task was to select the 'correct' emotion from a list of
emotion terms provided for each stimulus. As the main interest was on general
decoding ability, recognition rate was analysed across stimulus emotions, though
further studies might want to address the issue of specific abilities or deficits for
specific emotions (see Study 3) in more detail.
In the first study (N=40) telephone counsellors (all female; in Austria most
telephone counsellors are female!) and N=40 face-to-face counsellors (to match
samples also all were female) were studied. Half of the subjects first saw the
pictures of facial affect, and than heard the vocal stimuli. For the other half of
subjects, the reverse was the case. After completing the tasks the subjects had to
answer some additional questions, e.g. they had to indicate the number of years
they had worked as telephone counsellors or as face-to-face counsellors. This
was supposed to provide at least a rough estimation of the amount of experience
in the job. Unfortunately, in this study, the researcher was not allowed to ask for
age of subjects (the telephone counselling organisations were afraid that asking
for subject's age explicitly would induce the character of an 'examination
situation' especially for older counsellors!). In order to have a rough indication
of age for matching purposes with the group of face-to-face counsellors the
experimenter made a rough guess of subject's age, later to be used to recruit
face-to-face counsellors of about the same age as controls. The control group
was recruited by directly contacting face-to-face counsellors at their working

177
place, with an additional 'snow ball' system, taking care to match both groups
roughly with respect to age and years of experience.
The second study was essentially similar, though this time the control group
did not consist of counsellors, but of persons working in 'ordinary' occupations,
This time it was possible to ask for subject's age, which made matching with
respect to age easier. Furthermore, also male telephone counsellors (a rare
species!) were to be included. Though it was possible to recruit some male
subjects, the gender distribution was so uneven that gender was not used as an
additional factor. Altogether N=39 telephone counsellors (26 female, 13 male)
participated, as well as N=39 control subjects (matched for gender and age).
In addition to the decoding tasks subjects were asked to fill in a measure of
'emotional intelligence' with the three sub-scales of attention, clarity, and repair.
This was included for reasons of interest because the ability to decode others'
emotions is seen as one important component in conceptions of 'emotional
intelligence' (see Wallbott, 2000).

Results and discussion

Results of the first study show no significant differences between the two groups
in their decoding ability for visual stimuli, but significant differences in the
predicted direction for vocal stimuli, i.e. telephone counsellors reached slightly
higher scores compared to face-to-face counsellors (t=2.55, p=0.013; see Figure
2). It seems of additional interest though that within the group of telephone

26 t.______________________________________-.

~.!=,y, ~ ~R4: : : : : ::::::~ :_~-:_~-:_~-:_~-1: d: : : ~~~~~~~::~

voca l visu.oJ
Telephone Coon ..'''''
\'ocal visual
Faee--Io--Face Councclors

vocal visual voea I vi sual

Telephone Councelors on~councelors

Figure 2. Recognition rates for vocal and visual stimuli by telephone counsellors/non
counsellors

178
counsellors the ability to decide vocally transmitted emotions correlated
negatively with the number of years of counselling experience (r=-0.32,
p=0.043). Thus, the general prediction could be confirmed, while the negative
correlations seems counterintuitive. One might speculate that with many years
of telephone counselling, counsellors shut off emotional information as a self-
protection or coping mechanism, that it may have something to do with
counsellors' age, or that this reflects something like a minor bum-out symptom.
The results of the second study basically replicated the main results of the
first one (also see Figure 2). This time statistical significance reached p=0.03,
again in the expected direction for the vocal channel and p=0.25 for the visual
channel. No substantial or significant correlations were found with counselling
experience (thus the finding of the first study was not replicated), and also no
significant correlations were found with emotional intelligence or the three sub-
scales of the emotional intelligence test used.
Taken together, results of both studies with respect to the main hypothesis
are remarkably parallel, indicating that telephone counsellors are indeed slightly,
but significantly, better in recognising emotions from vocal cues compared to
other groups, while they are not better (or may be even worse) in recognising
emotions from visual stimuli. Thus, their sensitivity for emotions is not
generally higher, but their sensitivity to vocal cues is i.e. they indeed
compensate. Of course, other explanations for this finding may come to mind.
It might, for instance be, that persons working as telephone counsellors are more
sensitive to vocal cues from the beginning, and therefore select this type of
counselling, implying that it is not compensation forced by the occupation, but
due to a general disposition. Obviously this cannot be decided with the present
data, but would call for longitudinal studies in this area.

Dissocial Juveniles

Dissocial juveniles do not have (at least most of them do not have) a deficit of
their sensory organs, i.e. they are usually neither deaf nor blind. Thus, one may
ask why such a study was included in the present context. The idea is that this
group might have a general deficit (not linked to sensory deficits) in emotion
processing and in emotion decoding, which might be of clinical interest.
However, studies providing empirical results are rare.
Barahal, Waterman and Mary tin (1981) found that children who had been
mistreated physically in early childhood (which is considered as one predictor of
dissociality) were significantly worse in decoding emotions from vocal stimuli
compared to a control group of normal children. In a similar vein, Camras,
Grow, and Rivordy (1983) found poorer recognition of visual emotional stimuli

179
 (facial expressions) in bodily mistreated children. Attempts at explanation in
both studies mention problems in emotional socialisation, learning the wrong
cues for certain emotions, or missing cues for other emotions which partly
prevent these children learning, for instance, what a positive face looks like, and
thus their traumatic experiences lead them to wrongly interpret facial
expressions as well as vocal cues (cf. in the direction of negative emotions).
One interpretation of the high aggressiveness of dissocial children and juveniles
(certainly not the only one) might indeed be that they tend to interpret each and
every emotional expression as expressing anger, aggression, disgust and the like,
and that they react due to their - of course partly wrong - interpretations. If a
dissocial juvenile considers other persons often wrongly as expressing anger or
aggression, which indeed she or he might have experienced in early childhood to
be true, it may only be only 'logical' to react in an angry and aggressive way
oneself for the sake of self-protection. Thus, dissocial juveniles (as a result of
bodily mistreatment, conflicts in a parent-child interaction, problems in
emotional socialisation) were chosen as a study group here as they might have a
general deficit in recognising emotions, which could be differentiated if one
looks at decoding of different emotions (cf. anger).
It should be noted here that it is difficult to recruit dissocial juveniles who are
willing to participate in a psychological study. Indeed, in the present study,
some of them, when having made contact to the respective homes and
counsellors, turned out to be also dissocial in the context of the planned study.
Some subjects had to be excluded from data analysis because of that reason. It
turned out that most of these subjects could be best motivated to participate by
giving them a packet of cigarettes as a present (some studies demand strong
motivators!). After contacting a number of social-pedagogical homes in the area
of Salzburg where juveniles considered as dissocial lived (diagnosed dissociality
according to DSM-IV as an antisocial personality disorder - the researchers of
the present study had no influence on the diagnosis made - usually because they
had committed criminal acts of a different number and different severity),
subjects to participate were selected with the help of the respective contact
persons and counsellors. Selection criteria amongst others, included whether the
contact persons considered the subjects to be willing and able to participate,
taking into account criteria like their live history, and their subjective impression
in working with them as social workers. Very severe cases were excluded from
the study from the beginning.

Materials and procedure

To keep demands in an experimental situation for this specific group of subjects

and on the other hand motivation to participate as high as possible (besides the

180
promised cigarettes!) it was decided to focus on the recognition of visually
transmitted emotions, i.e. facially expressed emotions. It was considered that
presenting pictures in a judgement task might be more appealing than presenting
vocal stimuli. For the study the "Pictures of Facial Affect" (Ekman & Friesen,
1976) were used. For the present purpose the number of stimuli was reduced to
minimise the demand on subjects (especially the dissocial juveniles). For this
sake, no blends were used, as well as all contempt stimuli, because a number of
studies (Ricci-Bitti, Brighetti, Garotti & Boggi-Cavallo, 1989; Russell & Fehr,
1987) indicate that these are often confused with anger or disgust. This left
N=37 stimuli for the primary emotions of joy, sadness, fear, anger, disgust and
surprise. Subjects' task was to select from a list of these six emotion words for
each stimulus on a seven-point scale (not at all secure - very secure) how secure
they were about their respective judgement. No time limit was introduced for
the judgements ..
Altogether it was possible to collect a full data set for N=25 dissocial
juveniles (selected according to the criteria described above - mean age= 15.5
years, 10 were female, 15 were male). All lived in supervised groups sharing
flats. Admittedly, this sample is not too large, but the results to be described
turned out to be of interest, and the problems of getting access mentioned above
cannot be neglected. A control group (N=26 pupils) was selected from a
commercial business college (business school) according to teacher impressions
that these pupils had never acted dissocial or were in any other way conspicuous
(mean age=16.0 years, 10 were females, 16 were males). It should be mentioned
(because this might influence interpretation of the results found) that both groups
are not entirely parallel. Disssocial juveniles are not only dissocial, but they
also, have problems with their education level in school and have difficulties in
many other respects. Thus, exact matching was not possible.

Results and Discussion

If one looks at mere recognition rates first, there are highly significant
differences between both groups of subjects. Dissocial juveniles show lower
recognition rates for emotional facial expressions in comparison to the control
group (t=8.06, p<O.OO 1). A look at the six different stimulus emotions
separately indicates that these deficit of dissocial juveniles is not due to all
emotions, but at least in terms of statistical significance mainly to the emotions
of sadness (t=3.42, p=O.OOI), fear (t=3.86, p<O.OOI) and surprise (t=2.75,
p=0.008), where this group were significantly worse compared to the control
group (see Figure 3).
Though again sample sizes become rather small, gender of subjects was
taken into account to test whether the general deficits in recognition rate of

181
dissocial juveniles might be related to this factor. A significant interaction
between gender and experimental group indicates that especially male dissocial
juveniles provided the lowest scores of all groups (F=5.06, p=0.034; recognition

0.9

0.8

0.7
~
8 0.6
:I'.
~
0.5
~
~
0.4
:~
8> 0.3
Il
'" 0.2

0.1

0
Joy Surprise Sadness Anger Disgust Fear

D d;ssoc;al

~ normal

Figure 3. Recognition rates of dissocial and normal juveniles for basic emotions

rates: female/dissocial=61 %, female/normal=69% - no significant differences

between the two female groups; male/dissocial=56%, male/normal=72%
difference between the two male groups highly significant).
Furthermore, it seems important to note that dissocial juveniles, though
worse in decoding accuracy compared to the control group, reported being more
secure about their emotion judgements (t=5.07, p<O.OOI). Further, this group
independent of whether their judgements were right or wrong tended to choose
some emotion terms significantly more often as their 'correct' choice in
comparison to the normal juveniles. Specifically they chose much more
frequently the emotions of anger (t=3.19, p=0.002) and of disgust (t=2.77,
p=0.008), while their choice is less often - though not reaching statistical
significance - sadness, or surprise (see Figure 4).
These findings (carefully interpreted with respect to sample problems)
indicate that dissocial juveniles in fact have a severe problem with recognising
emotions from facial expressions. Their overall decoding accuracy is worse than
that of normal juveniles and this is mainly true for male dissocial juveniles.
Furthermore, they are more secure that they are right, though in fact they are
often wrong. Finally, they very often misattribute emotions by choosing anger
or disgust too often, while for instance, sadness was chosen less often.

182
 ...
0.'

0.1

0.6
~
f O.S
.~

t
0.4

0.3
'" 0.2

0.1

S>dn... Surprise Anger F.... Disgust Jo)-

0 di~ial

W1l normal

Figure 4. Perceived emotions by dissocial and normal juveniles

This presents the picture of a male person who is not very good in
recognising others' emotions, but being very sure that he is able to do so, and a
person who sees anger in faces not at all expressing anger. These findings fit to
an astonishing degree with these persons' violence and antisocial behaviour. If I
am sure I am correct in recognising other's emotions and if I recognise anger or
disgust (falsely most of the time!), why should I not react in an aggressive
fashion to protect myself? False and biased recognition of emotion and undue
security in this ability thus might be part of the problem dissocial juveniles have,
or indeed, may be one of the causes for their problems.

CONCLUSIONS AND SOME SUGGESTIONS

The line of reasoning this contribution is centred around is the controversy

between the deficit hypothesis and the compensation hypothesis. Both
hypotheses were originally developed within the context of research on sensory
deficits (hearing loss, blindness). While the first one states that losing one
sensory organ will hamper performance in other senses, the latter states that
losing one sensory organ will be compensated by increased
performance/sensitivity in other senses, i.e. the first one is a more 'pessimistic'
hypothesis, the second a more 'optimistic' one with respect to performance of
persons with sensory deficits. Here, this controversy was transferred to the issue
of emotion recognition. In order to study this issue one should have a systematic
look at deaf or hearing impaired persons on the one hand, and on blind persons

183
 on the other hand. The predictions of the compensation hypothesis are rather
clear-cut for both groups: deaf subjects should be better at recognising emotions
from visual stimuli compared to a control group, and blind subjects should be
better at recognising emotions from vocal stimuli, again in comparison to a
group with unhampered sight. Unfortunately, only few, and sometimes
methodologically problematic studies, are available in this area. .
Our own interest started with a study on hearing-impaired and deaf persons
(Wallbott & Seithe, 1993). The results indicated no deficit, and yet no
substantial compensation for highly impaired or totally deaf persons was noted.
They were as good as normal hearing control subjects in recognising visually
transmitted emotions. Thus, this group should not have a substantial problem
within this area (of course they do have other severe interaction and
communication problems!). With respect to this issue, the problematic group
were persons with medium hearing impairment, because these persons were
significantly worse. They do not only have a deficit in recognising emotions
from vocal cues due to their hearing impairment, but also a deficit in recognising
emotions from visual cues! This was tentatively interpreted as a consequence of
uncertain or ill-defined group identity. Medium impaired subjects do not define
themselves as 'deaf, but closer to the normal-hearing group, and thus probably
feel no need to focus more on or pay more attention to visual cues. Training
programs might be helpful here to work against this deficit by refocussing the
self-definition of persons with medium hearing loss, fostering insight that they
were hampered, and encouraging them to pay more attention to visual
information, maybe with specific training parts on the visual expression of
emotion in facial expression, body posture etc. This might help to work against
the deficit observed in this study.
A study with blind persons is then reported. No deficit for recognition of
emotions from vocal stimuli was found contrary to some previous studies, but
also no compensation. Blind persons were equally good as normal-seeing
persons. This was independent of factors like blindness from birth or blindness
later acquired. Thus, blind persons seem to cope rather well with their sensory
deficit, at least with respect to the task used here. Still, training programs for
blind persons in recognition of vocal emotion cues might be helpful to increase
their abilities here. It might be very different from the quiet laboratory situation
(possibility to listen to each stimulus twice etc.) used here, where subjects could
concentrate solely on the vocal stimuli, in real life with background noise etc.
Thus, generaliseability of the results might be a matter of discussion.
Two studies with telephone counsellors provided identical results: Telephone
counsellors indeed compensated for the lack of the visual channel by reaching
better results on the vocal channel. A conclusive causal explanation is not
possible within the design used here, but one might at least speculate that the

184
non-availability of visual cues forces this specific group of persons to focus
attention much more on the vocal channel to get access to feelings and emotions.
A negative correlation between job experience and decoding ability was not
replicated in the second study and thus was not considered further. It is planned
to follow this line of research in the future and to conduct studies with therapists,
clinical psychologists etc. with a focus on possible relations between job
experience and decoding ability.
This study with dissocial juveniles has been discussed in some detail above.
It provides interesting results, because it indicates a general deficit of this group,
not only with respect to recognition of emotion in general, but also with respect
to recognising specific emotions, over-confidence in their judgements, and a
significant tendency for wrong attributions. The limitations, like the access
problem, the small sample, matching problems with the control group etc. have
already been mentioned. But it seems fascinating that the results found are to a
large degree in line with developmental theories of this personality disorder. In
terms of clinical implications, of course, training programs confronting subjects
with their attribution faults, their over-confidence and training in valid cues for
specific emotions might be thought of. This should be attempted, though work
with this group, also, might be very demanding.

AFFILIA TION

Universitat Salzburg
Institut fur Psychologie
E-mail: Harald.Wallbott@sbg.ac.at

ACKNOWLEDGEMENT

I would like to thank Tanja Lander!, who invested a lot of work in the study with
blind persons, Gisela Erdbauer, who managed to motivate dissocial juveniles to
participate in the study, Simone Vogt and Sonja Miihlbock, who succeeded in
getting access to telephone counsellors as subjects, and finally Werner Seithe,
who - being hearing impaired himself- started my interest in questions of
emotion recognition in groups of impaired persons some years ago, an interest
which is still continuing. I extend my thanks to Sabine Eichbauer, who provided
technical facilities, help and advice in conducting the different judgement
studies, to Hans Lechner for his statistical assistance, and to Hermine Manzel for
preparing the figures.

185
 REFERENCES
Baharal, R., Waterman, J., & Mary tin, H. (1981). The social-cognitive development of abused
children. Journal of Consulting and Clinical Psychology, 49, 508-516.
Blau, S. (\964). An ear for an eye: Sensory compensation and judgement of affect by the blind: In
J.R. Davitz (Ed.). The communication of emotional meaning (pp. 113-~27). New York:
McGraw-HilI.
Camras, L.A., Grow, J.G., & Rivordy, S.C. (1983). Recognition of emotional expression by abused
children. Journal of Clinical Child Psychology, 12,325-328.
Cutting, J.e. (198\). Judgement of emotional expression in schizophrenia. British Journal of
Psychiatry, 139, 1-6.
de-Gelder, 8., & Vroomen, J. (2000). The perception of emotion by ear and by eye. Cognition and
Emotion, 14, 289-311.
de-Gelder, B., & Vroomen, J., & Pourtois, G. (1999). Seeing cries and hearing smiles: Cross modal
perception of emotional expressions. In G. Aschersleben & T. Bachmann (Eds.), Cognitive
contributions to the perception of spatial and temporal events (pp. 425-438). Amsterdam:
North-Holland.
Ekman, P. (1982). Facial expression and facial nerve surgery. In M.D. Graham & W.F. House
(Eds.), Disorders of the facial nerve (pp. 363-368). New York: Raven Press.
Ekman, P., & Friesen, W.V. (1976). Pictures offacial affect. Palo Alto:Consulting Psychologists
Press.
Hall, J.A. (1978). Gender effects in decoding nonverbal behavior. Psychological Bulletin, 85,845-
857.
Hatwell, Y. (1987). Motor and cognitive functions of the hand in infancy and childhood.
International Journal of Behavioral Development, 10, 509-526.
Hoemann, H.W. (1978). Perception of the deaf. In E.e. Carterette & M.P. Friedman
(Eds.),Ilandbook of perception. Vol X Pereceptual ecolof!Y (pp. 43-64). New York: Academic
Press.
Izard, e.E. (l977).lIuman emotions. New York: Plenum.
Johnston, P., & Carr, V. (2002). Facial affect recognition deficits in schizophrenia: A case for
applying facial measurement techniques. In M. Katsikitis (Ed.), The human face: Measurement
and meaning (pp. 121-132). Dordrecht, Kluwer Academic Publishers.
Johnston, P., & Katsikitis, M. (2001). A generalised deficit can account for problems in facial
emotion recognition in schizophrenia. Biological Psychology, 58,203-227.
Larr, A.L. (1956). Perceptual and conceptual abilities of residential school deaf children.
Exceptional Children, 23, 63-66.
Mandai, M.K. (1986). Judgement of facial affect among depressives and schizophrenics. British
Journal of Clinical Psychology, 25, 87-92.
Minter, M.E., Hobson, R.P., & Pring, L. (1991). Recognition of vocally expressed emotion by
congenitally blind children. Journal of Visual Impairment and Blindness, 85,411-415.
Muzekari, L., & Bates, M.E. (1977). Judgement of emotion among chronic schizophrenia. Journal
of Clinical Psychology, 33, 662-666.
Myklebust, H.R., & Brutten, M. (1953). A study of the visual perception of deaf children. Acta Oto-
Laryngologica, Supplement 105.
Reynolds, H.N. (1978). Perceptual effects of deafness. In R.D. Walk & H.L. Pick (Eds.), Perception
and experience (pp. 241-259). New York: Plenum Press.
Ricci-Bitti, P.E., BrigheUi, G., Garotti, P.L., & Boggi-Cavallo, P. (1989). Is contempt expressed
pan-cultural facial movements? In J.P. Forgas & J.M. Innes (cds.), Recent advances in social
psychology: An international perspective (pp. 329-339). Amsterdam: Elsevier.
Rosenthal, R. (1979). Skill in nonverbal communication: Individual differences. Cambridge,
Oelgeschlager, Gunn.

186
Rosenthal, R., Hall, J.A., DiMatteo, M.R., Rogers, P.L., & Archer, D. (1979). Sensitivity to
nonverbal communication: The PONS test. Baltimore: John Hopkins university Press.
Russell, J.A., & Fehr, B. (1987). Relativity in the perception of emotion in facial expressions.
Journal of Experimental psychology: General, 116, 223-237 .
Russell, J.A., & Fernandez-Dols, J.M. (1997). The psychology of facial expression. Cambridge:
Cambridge University Press.
Scherer, K.R., Banse, R., & Wallbott, H.G. (2001). Emotion inferences from vocal expression
correlate across languages and cultures. Journal of Cross-Cultural Psychology, 32, 76-92.
Scherer, K.R., Banse, R., & Wallbott, H.G., & Goldbeck, T. (1991). Vocal cues in emotion
encoding and decoding. Motivation and Emotion, IS, 123-148.
Scherer, K.R., Wallbott, H.G., & Summerfield, A.B. (1986). Experiencing emotion: A cross-
cultural study. Cambridge: Cambridge University Press.
Schiff, W. (1973a). Social event perception and stimulus pooling in deaf and hearing observers.
American Journal of Psychology, 86, 61-78.
Schiff, W. (I 973b). Social perception in deaf and hearing adolescents. Exceptional Children, 39,
289-297.
Schiff, W., & Dytell, R.S . (1972). Deaf and hearing childrens' performance on a tactual perception
battery. Perceptual and Motor Skills, 35, 683-706.
Schiff, W., & Thayer, S. (1974). An eye for an ear? Social perception, nonverbal communication,
and deafness. Rehabilitation Psychology, 21, 50-70.
Streri, A. (1993). Seeing. reaching. touching: The relations between vision and touch in irifancy.
Cambridge : MIT Press.
Sugarman, I.R. (1969). The perception of facial expressions of affect by deaf and non-deaf high
school students. Unpublished doctoral dissertation, Columbia University.
Wallbott, H.G. (1990). Mimik in kontext - Die bedeutung verschiedener informationkomponenten
for das erkennen von emotionen (Facial expression in context - The irifluence of different
components of information on recognising emotions). Gt\ttingen: Hogrefe.
Wallbott, H.G. (1998). Decoding emotions from facial expressions: Recent developments and
findings. In W. Stroebe & M. Hewstone (Eds.), European review of social psychology (pp. 191-
232). Chichester: Wiley.
Wallbott, H.G. (2000). Empath ie. In HI. Otto, H.A. Euler, & H. Mandl (Eds.), Ifandbuch
emotionspsychologie (pp. 370-380). MUnchen: Urban & Schwarzenberg.
Wallbott, H.G., & Seithe, W. (1993). Sensitivity of persons with hearing impairment to visual
emotional expression-Compensation or deficit? European Journal of Social Psychology, 23,
185-193.
Wallbott, H.G., & Scherer, K.R. (1986). How universal and specific is emotional experience?
Evidence from 27 countries on five continents. Social Science Information , 25, 763-796
(reprinted 1988 in K.R. Scherer (Ed.), Facets of emotion (pp. 31-56). Hillsdale: Erlbaum).
Williams, C.E., & Stevens, K.N. (1972). Emotions and speech: Some acoustical correlates . Journal
of the Acoustical Society of America, 52,1238-1250.

187
 BELINDA GOODENOUGH, CARL L. VON BAEYER,
& G. DAVID CHAMPION

CHAPTER 10

FACIAL EXPRESSION AND THE

SELF-REPORT OF PAIN BY CHILDREN

INTRODUCTION

With the exception of rare individuals who are insensitive to pain, either from
congenital or disease-related anomalies of the nervous system (Erdem, Ozcan,
I1guy & Sirin, 2000; Greco, Villa, Fusetti, Orlandi & Pierotti, 2000), the
experience of pain is universal. It cuts across the boundaries of age, race, and
socioeconomic status, and pain ranks as a major symptom prompting individuals
to seek professional health advice or to take medications.
While pain and concomitant suffering may be a universal phenomenon, pain
is an inherently private experience. It is also a multifaceted experience, evoking
and encompassing a range of sensory and emotional responses, including
anxiety, fear, depression and anger. The subjectivity of the experience is a major
culprit in difficulties associated with assessing pain. Measurement problems are
particularly salient in the paediatric arena. Undertreated pain in infants and
children can retard healing, decrease co-operation with medical procedures,
interfere with appetite and sleep, and potentially impede learning and
development (Mitchell, 1999). Thus reliable measurement is a crucial
component for recognising pain and tailoring effective relief. Clinicians need
reliable practical methods for gauging pain-related distress as a prerequisite to
optimal management.
It would be desirable, if it were possible, to base judgments of pain upon
objective physiological standards such as respiration rate and blood pressure.
Such physiological measurements may be possible and informative when caring
for sick babies and children who are already being carefully monitored for signs
of physical distress and recovery. However, it is generally accepted that these
physiological responses may not be practically possible in many medical and
everyday contexts. Furthermore, physiological indices may not adequately
discriminate pain from other types of distress, and may also not be as objective
as assumed, including being influenced by the paediatric setting itself (Wolf,
1999). Therefore, the principal measurement methods available to the clinician
focus on behaviour. Firstly, there are methods for observing behavioural
responses as expressions of pain. Secondly, there are methods emphasising
information that a child, if old enough and willing, can directly communicate
about pain (i.e., self-report). In both of these approaches to understanding pain
communication, the concept of a 'pain face' has occupied a significant role
(Craig, 1992).
A theoretical model of the communication of pain has been developed by
Prkachin & Craig (1995). In this model, the private experience of pain is
encoded (expressed) in verbal and nonverbal behaviour, and may then be
decoded Gudged or assessed) by observers. Both encoding and decoding are
heavily influenced by contextual factors and individual differences, but facial
expression, when accurately measured, offers perhaps the most reliable channel
for communication of pain in many circumstances. The question therefore arises
i.e. is there a facial expression specific for pain?
A significant proportion of the evidence for a 'universal pain face' is rooted
in the application of facial action coding schemes, especially for measuring pain
in infants. The present chapter will broadly review facial action coding schemes
with respect to the clinical use of such scales and possible limitations.

FACIAL ACTION CODING SCHEMES

Figure 1 illustrates a newborn's videotaped reaction to an injection in the thigh

muscle two hours after birth (Grunau & Craig, 1990). One simple measure of
this facial expression could involve a quick clinical judgement for the presence
or absence of a 'negative expression' (Davis & Calhoon, 1989; Watt-Watson &
Donovan, 1992). While this binary judgement is appealingly simple from a
clinical point of view, the reliability and validity of facial expression as
reflecting pain rather than some other state has required a more minute analysis
of facial actions. Originally drawing upon the adult literature researching
emotional expression, several intensive observational methods have been applied
to infants and children. While several facial expression coding schemes will be
referred to, the three major coding schemes are:
(a) Maximally Discriminative Facial Movement Coding System, MAX (Izard,
Huebner, Risser, McGinnes, & Dougherty, 1980). The MAX codes facial
movements according to preconceived categories of emotion (including pain
as an emotional expression). It has been applied with some reliability in

190
 children and infants older than 2 months (Izard, Hembree, Dougherty, &
Spizzirri, 1983, Johnson & Strada 1986)
(b) Facial Action Coding System, FACS (Ekman & Friesen, 1978) codes 46
specific anatomically based action movements in infants, without reference
to a priori templates of emotional reactions. FACS is not limited to coding
pain expression or distress, but codes all possible movements of facial
musculature.
(c) Neonatal Facial Coding System, NFCS (Grunau & Craig, 1987, 1990), was
developed specifically for coding facial activity during painful events in
neonates (under two months of age). Comprising 10 anatomically-defined
facial actions targeting distress (rather than all facial movements), the NFCS
is shorter and hence less complex than FACS.

Figure I. Facial reaction of a new born infant (two hours old) to a

vitamin il'!iection in the thigh. Reproduced with permission from Grunau & Craig (/990)

There remains debate as to whether facial expressions coded in response to

painful stimuli represent non-specific distress or a pain-specific reaction. While
research is continuing in this area, it is no coincidence that the refinement of
facial action coding schemes has occurred in the recent history of pain
assessment in infants. Nursing professionals have indicated a long-standing
acknowledgment of the importance of facial cues in clinical decisions about pain
in children (Pigeon, McGrath, Lawrence & MacMurray, 1989), but
unfortunately, until recently, it was still widely believed that neonates did not
feel or remember pain, or that pain in preverbal children was too difficult to
assess reliably.
Validation of facial action coding schemes has generally focused on infant
facial reaction to procedures assumed to produce short sharp pain to an
observable, time-limited event. These include injections (e.g. vitamin K), blood
sampling via heel lance or venipuncture, and circumcision. Such studies have

191
found some degree of consensus in the features of a spontaneous 'pain face',
and its generalisability across the lifespan, at least for short sharp pain.

The pain face in neonates

From birth, babies have the ability to move facial muscles in complex
configurations that adult observers can label as emotional expressions. The
spontaneous neonatal distress response - including reactions to handling, cold
and hunger - may be reflected by changes in fussing and crying behaviour (e.g.
pitch and duration), by changes in facial expression, and by changes in body
movements (including rigidity and presence of startle reflexes). Whilst these
changes may typically happen simultaneously, it is the facial response in this
behaviour constellation (e.g. Figure I) which has been found to be the most
consistent indicator of pain-related distress in infants (Craig, Whitfield, Grunau,
Linton, & Hadjistavropoulos, 1993; Dale, 1986; Marvin & Pomietto, 199\). For
example, in reaction to a routine immunisation, facial expression showed the
least variability across infants as compared with other body movements, crying,
or changes in heart rate (Johnston & Strada, 1986). Likewise, for infants
undergoing blood-sampling via heelprick, facial activity explained the largest
proportion of variance (71 %) in observer pain ratings (Hadjistavropoulos,
Whitfield, Grunau, Craig, 1997). Facial reaction has also been found to be a
sensitive outcome measure for comparaive evaluation of different clinical
techniques, such as the method of heel-lancing for routine blood collection in
neonates (Grunau & Craig, 1987).
In neonates, there is likely to be a high correlation between features of facial
expression that represent general distress and those that may specifically signal
pain. Notwithstanding these difficulties, as an example of current applications of
facial expression measures to pain-related distress, Table I lists the features of a
'pain face' as defined by the ten descriptive components of the Neonatal Facial
Coding System (Grunau & Craig, 1987, 1990). As illustrated in Figure 1, the
dominant features are located in three main facial areas which are considered to
be largely independent with respect to control of musculature: (1) the brows and
forehead; (2) the eyes, eyelids, and roots of the nose; and (3) the cheeks, mouth,
lower nose, and chin. Thus, as illustrated in Figure 1, the dominant features of a
pain face in infants are a lowered brow, eyes squeezed shut with a brow bulge, a
deepening of the nasolabial furrow, an open stretched mouth, and a taut cupped
(sometimes quivering) tongue (Grunau et aI., 1987, 1990; Johnston et ai, 1993).
There are data to suggest that a protruding tongue may be a useful index of no
pain, and may occur in response to other stimuli such as alcohol swabbing or
application of triple dye to the umbilicus (Grunau & Craig, 1990).

192
 Facial action Component description
Brow bulge Bulging, creasing and vertical furrows above
and between brows, as a result of lowering
and drawing together eyebrows
Eye squeeze Squeezing or bulging eyelids, pronounced
bulging of fatty pads around eyes
Nasolabial furrow Pulling upwards and furrow deepening of the
line beginning adjacent to the nostril wings
which runs down and outward beyond lip
comers
Open lips Any separation of lips
Horizontal mouth stretch Horizontal pull at mouth comers
Vertical mouth stretch Tautness of lip comers (vertical) coupled
with downward pull of jaw.
Lip purse Lips appear as if pronouncing an "00" sound
Taut tongue Raised cupped tongue with sharp tense
edges.
Chin quiver High-frequency up-down motion of lower
jaw
Protruding tongue Tongue visible between lips extending
beyond mouth

Table I. Neonatal Facial Action Coding System (adapted/rom Grunau & Craig,
199(}). These ten/acial actions are scored as present versus absent.

One issue in the application of facial expression measurement in infants is

the role of relative physical maturity. Most healthy full term babies will
experience a limited number of routine and potentialIy painful procedures such
as intramuscular injection of vaccines or vitamins, and heel-pricks for blood-
sampling. Clearly, a group of 'at risk' infants are those who are born
prematurely or are fulI term but sick from birth. These babies are often
subjected to hospital stays involving many more painful procedures. Reliable
pain behaviour measures are needed in this population to determine the efficacy
of various pharmacological and non-pharmacological pain-relieving
interventions, such as rocking or swaddling, and the administration of sucrose or
topical anaesthetic creams (e.g. Fearon, Tranmer, Muir, Hains, & Kisilevsky,
1997; Johnston, Horton, Stevens, & Stremler, 1996; Ramenghi, Levene, Griffith,
& Wood, 1996). It is also probable that, for these special populations,
convergent physiological measures that take advantage of monitoring systems

193
 already in place (e.g. heart rate, oxygen saturation levels) can be used to
supplement behavioural assessments.
An important question, therefore, is whether premature infants show the
same sorts of facial responses to pain as healthy full-term babies. There is
certainly evidence that facial activity can be observed in the viable premature
newborn as young as 25 weeks gestational age during invasive medical
procedures assumed to cause pain° (Craig, Whitfield, Grunau, Linton, &
Hadjistavropoulos, 1993). Overall, research to date has revealed that a similar
pattern of facial expressions seems to apply to all babies in the age bracket of
premature to 4 months (Johnson & Strada, 1986; Tatman & Johnson, 1998).
One caveat to the generalisability of facial reactions to preterm infants is data
showing that prematurity may be sometimes confounded with the number of
painful invasive procedures (Johnston & Stevens, 1996). Using facial activity as
an outcome measure, it is possible therefore that, due to various mechanisms
such as exhaustion over the course of multiple invasive procedures, pre-term
infants may exhibit a less intense behavioural reaction to pain than their healthier
peers (Johnston & Stevens, 1996). Likewise, prematurity and sickness in some
populations may be correlated with deficits in the central nervous system, which
may have implications for relevant nociceptive transmission and hence
behavioural manifestations of pain. While there are complexities of application
and interpretation of facial action coding schemes to sick or premature babies,
the quest continues to find discrete cues to pain in facial expression. For
example, facial brow raising in pre-term babies increases in proportion to the
number of painful invasive procedures that had occurred in the previous 24
hours (Grunau, Holsti, Whitfield, & Ling, 1999).

The pain face in older children

As with adults and neonates, observable behavioural changes occur in children

when pain is present. The exact nature of these behaviours is likely to be pain
specific, as well as varying with the acute or chronic nature of the pain. For
example, after surgery a child may lie very still in bed having learned, or
assumed, that movement will increase pain. A young child with an earache may
tug at the earlobes, whilst a child with abdominal pain may draw up the knees or
hold the abdomen. In response to a needle a child may flinch or may remain
tensely rigid. Given the range of possible observable responses to pain it would
be helpful to have a single behavioural indicator that could be applied across the
full domain of pain behaviours or expressions in young children.
Relative to adults and older children, preschool aged children are considered
most vulnerable to under-treated pain especially after surgery. Under-treatment
may be due to complications in measurement associated with communication

194
difficulties of the young child. To this end, facial reaction is beginning to be
explored as one possible measure in the clinical context. An example is the
Child Facial Coding System, CFCS (Gilbert, Lilley, Craig, McGrath, Court,
Bennett, & Montgomery, 1999) which was developed for use with preschool
aged children (aged 2-5 years) and has also been applied in adolescents
(Oberlander, Gilbert, Chambers, O'Donnell, & Craig, 1999).

Figure 2. Pictorial representation ofthefada/ expression to persistent pain in children

aged 1 to 6 years. (Reproduced with permission from Gilbert, et a/,1999).

The Child Facial Coding System codes 13 facial actions (see Table 2) which
in large part extend the list identified by the Neonatal Facial Coding System (see
Table 1): brow lowering, squint, eye squeeze, nose wrinkler, nasolabial furrow,
cheek raiser, upper lip raiser, lip corner puller, vertical and horizontal mouth
stretches, flared nostril, open lips, and eye blink. All but the last three features
are coded on a 0-2 range of intensity. As with the infant measures, the CFCS
has been applied in a variety of brief procedural pain contexts such as
venipuncture and immunisation injection (Chambers, Cassidy, McGrath, Gilbert,
& Craig, 1996). More recently, the scale has been applied in the post-operative
acute pain context with a view to identifying a 'persistent pain face.' The initial
data have been encouraging with indications of an identifiable pain grimace by
young children after minor surgery, at least in the first hour postoperatively.
The features of this 'persistent pain face' in the young child have been
pictorially represented by the scale's authors in Figure 2 (from Gilbert et ai,
1999). It remains the task of future research to ascertain the specificity of this
pain face for assessing persistent acute pain, and the sensitivity of the
behavioural definitions as outcome measures for detecting changes in relative

195
pain levels at different points in the post-operative experience (including under
conditions of confounding states such as nausea, fatigue and sedation).

Facial action Component description

Brow lower' Eyebrow lowered and eyebrows pulled
together
Eye squeeze' Eyelids tensed and bulging in lower eyelid
Eye Squint' Eye opening narrowed, eyelid tense, bagging
of lower lid
Blink Eyes closed for less than half a second
Nasolabial furrow' Line adjacent to nostril is deepened and pulled
upward
Nose wrinkler' Skin around nose drawn upwards, horizontal
puckers across nose
Flared nostril Nostrils dilated
Cheek raiser' Cheek raised toward eye, bulging under eye
Open lips Lips parted
Upper lip raiser' Centre of lip raised slightly, upper teeth visible
Lip corner puller' Corners of lips elongated, pulled back and
upward
Horizontal mouth Lips, lip corners, surrounding skin stretched
stretch' laterally
Vertical mouth Lips parted, jaw lowered
stretch'

Table 2. Descriptive components of the Child Facial Coding System (Gilbert et ai,
/999) . Asterisk (*) denotes coding for intensity (range: 0-2), rather than presence

196
Limitations of Facial Action Coding Schemes

To date, facial action coding schemes have been most widely applied in acute
pain contexts, particularly in situations where pain is brief, typically sharp, and
where the change in facial expression seems to be an instinctive spontaneous
reaction. There are fewer clinical applications in the persistent acute pain
context or for chronic pain where self-report measures predominate. Scott,
Riggs, Ling, Fitzgerald, Hill, Grunau, Solimano, & Craig, (1994) found that the
Neonatal Facial Coding Scheme was a useful behavioural outcome measure for
the efficacy of morphine in post-operative pain. The NFCS has also been shown
to be sensitive to facial expression changes in infants satisfying stringent criteria
for colic during problem crying prior to feeding (Barr, Rotman, Yaremko, Ledu,
& Francoeur, 1992). More recently, facial action scores using the Child Facial
Coding Scheme have been shown to correlate well with a global measure of
post-operative pain in preschool aged children (Gilbert et aI., 1999). There is,
however, a relative dearth of studies of observable facial reaction in response to
persistent acute pain or chronic pain in children. It has commonly been observed
that the facial actions typical of short, sharp pain, reviewed above, tend to
attenuate in more prolonged pain. Authoritative guidelines and texts have
therefore been cautious in advocating the application of facial action coding
schemes for the routine clinical assessment of persistent pain.
Perhaps a more significant limitation of the practical application of facial
action coding schemes in clinical contexts is the perceived need to train the
observer, or to achieve adequate consensus between several observers. Many
studies have not applied the coding schemes in real time but depended on the
coding of videotape by highly trained observers (e.g. Lindh, Hakansson,
Sandman, & Wiklund, 1997; Gilbert et aI., 1999). Whether such systems can be
practically applied in clinical practice, without the use of videotape beyond a
training phase, has yet to be determined. However, there is some evidence that
non-trained observers can also reliably apply the Neonatal Facial Coding System
in infants (Craig, Grunau, & Aquan-Assess, 1988). It is likely that systems of
facial action coding that lend themselves to checklist format of presence versus
absence of features, rather than intensity coding, will be better embraced by
clinicians.
There is also little validation work on facial expression scales in infants and
children with significant neurological impairment, e.g., cerebral palsy, which
often affects motor control of facial muscles (Hadjistavropoulos, von Baeyer, &
Craig, 2001). Facial action coding schemes have been developed using alert
healthy individuals. In addition to the neurologically impaired, this may also
limit application to populations of very ill or sleepy (e.g., sedated or intubated)
infants and children. There is evidence that the intensity of facial reactions to

197
 pain tends to decrease with the increasing ill-health of an infant (Johnston &
Strada, 1986; Stevens, Johnston & Horton, 1994). Likewise, components of the
face may be less sensitive indicators of pain according to the initial sleep/awake
status of the child. For example, the taut tongue or vertical mouth stretch
(squarish mouth) component reaction has shown the greatest change in an awake
baby to the heel lance, but the least change when an infant was immediately
roused from a sleeping state just prior to a procedure (Johnson & Strada, 1986;
Tatman & Johnson, 1998).
With respect to older children, the utility of facial expression as an indicator
of pain may be increasingly influenced by 'display rules' (Ekman 1997; Zeman
& Garber, 1996). These are cultural conventions about withholding, disguising,
or exaggerating facial expressions. According to several theorists, spontaneous
facial expressions do not convey accurate information about internal states
(including pain) because of socialisation effects in masking natural
expressiveness (e.g. Fridlund, 1991). To date, there has been little study of the
potential impact of display rules on facial expression of pain in children, and
possible interactions with ethnic background, gender, temperament or even pain
type (e.g. a needle versus a migraine). Such display rules may function either to
exaggerate or suppress the facial expression of pain, depending on the culture,
audience, and other features of the social environment (see Champion,
Goodenough, von Baeyer, & Thomas, 1998).
Because of these limitations, facial expression is often formally measured in
clinical contexts as one of a range of behavioural indicators of pain rather than in
isolation. Two examples are the CRIES method for assessing post-operative
pain in infants (Krechel & Bildner, 1995), and the CHEOPS (Children's Hospital
of Eastern Ontario Pain Scale; McGrath, Johnson, Goodman , Schillinger, Dunn,
& Chapman, 1985) for rating post-operative pain in older children. Both the
CRIES and CHEOPS include coding of facial activity as one of several potential
behavioural indicators of pain. The CRIES acronym is as follows: Crying,
Requirement for oxygen supplementation, facial Expression and Sleeplessness.
The CRIES coding for a pain face in the infant is 0 for absent, 1 for present, and
a 2 for grimace plus non-cry vocalisation (e.g., grunt). Thus the CRIES is a
measure that offers clinical simplicity with minimal training, but where the
observer's assessment of a pain face is coded in binary terms (present/absent).
While the pain face in the CRIES system is coded simply, there is clearly scope
to train clinicians in facial observation potentially specific for pain using the
more detailed component analysis offered by such instruments as the Neonatal
Facial Coding Scale.

198
 SELF-REPORT MEASURES BASED ON FACIAL EXPRESSION

Traditionally, self-report measures of pain are the class of instruments which

involve a direct enquiry about some single dimension of pain experience
(typically intensity). Responses are quantified as specific values on a scale
relative to defined anchor points (e.g. no pain - very much pain). Hence 'self-
report' scales have been contrasted with behavioural measures based on
observation (such as Facial Action Coding Schemes), because the latter do not
involve a direct enquiry to the child about pain. As pain is subjective, self-report
measures have been prioritised: clinicians are often exhorted to base clinical
decisions (e.g. analgesia) on what an individual can communicate directly about
pain, rather than on observers' inferences from behaviour (McGrath, Unruh, &
Finley, 1995).
In the paediatric arena, the dichotomy of 'observed behaviour' and 'self-
report' measures has attracted criticism. Firstly, there is the unresolved debate
concerning the definition of 'self-report.' While infants, pre-verbal children, and
older people with communication impairments are unable to express their pain
verbally, they are able to communicate pain through vocal, facial, and motor
behaviour. However, a narrow definition of verbal self-report as a 'gold
standard' in pain measurement would undermine the value (i.e. perceived
reliability) of such forms of pain assessment based on observation (Anand &
Craig, 1996). Secondly, the dichotomy overlooks pain self-reports as behaviours
which are open to the influence of many factors (e.g. willingness to report pain).
The present chapter is not the forum to explore the debates on the relative utility
or primacy of self-report versus behavioural measures of pain in children, and on
the role of facial expression in communication versus expression (e.g. Ekman,
1997; Fridlund, 1991, 1994; Russell, 1995). However, the issue of pain self-
reporting as a behaviour in its own right will be given some discussion when
reviewing the relevant measures based on facial expression.

Examples of Self-report measures based onfacial expression

There are many different sorts of self-report methods developed for use by
children of various ages (for a comprehensive review, see Champion et ai,
1998). The majority focus on helping children estimate pain intensity, and to a
lesser degree, pain affect or unpleasantness (presumed to be correlated with an
emotional response to pain). A significant number of published self-report scales
use a depiction of facial expression, typically in cartoon format (see Figure 3).

199
 (a)

(b)

(d)

Figure 3. Examples a/published self-report measures/or children: (a) Facial

Affective Scale (McGrath, deVeber, Hearn, /985) ; (b) Children's Anxiety and Pain Scale
(Kuttner & Lepage, 1989); (c) Wong-Baker FACESs (Whaley & Wong 1987); (d) Faces
Pain Scale - Revised (Hicks, von Baeyer, Spafford & Goodenough, 2001; adapted/rom
Bieri et aI, 1990)

200
 Facial expression scales range from relatively simple categorical scales to
measures which, although presented categorically, represent a continuum of the
anatomical changes in facial expression when moving from 'no pain' to an
'extreme pain' endpoint. The majority of face scales are presented in horizontal
format, with a left-right reading bias. Despite this bias, cognitive and linguistic
demands are presumed to be minimal for application of face scales, thereby
taking advantage of perceptual processes assumed to be innate or developed
early in life. Hence, unlike scales based on words or relative number concepts,
face scales are thought to be intuitively understood by children as young as three
years (Yaster, Krane, Kaplan, Cote &, Lappe, 1997). Recent data has also
shown that children across the age range of 4 to 17 years typically prefer to use a
facial expression scale to rate pain over other self-report methods (Champion,
Goodenough, Wu, Chua, Taplin, & Ziegler, 2000; Goodenough, Addicoat,
Champion, McInerney, Young, Juniper, & Ziegler, 1997).
Since the mid-1990s, the intuitive appeal of face scales to both clinicans and
children has led to the production of many self-report measures for paediatric
use. Many of these scales, despite their wide availability and distribution, are
not validated for pain measurement in children. The vast majority of facial
expression scales are not based on sound psychometric research and tend to
make several inappropriate assumptions (Champion et al., 1998). These include
the features of the pain face (e.g. typically confounding reactions to sensory
intensity of pain with mood expressions such as smiles and tears) and the
untested psychometric properties of the measure (e.g. providing a non-validated
ratio-numerical scale on which to 'map' a chosen face in order to achieve a pain
score). These issues have clinical relevance and will be re-visited in summary
fashion under the topic of scale application later in this chapter.
Despite inherent problems that have been identified in the research arena,
each of the published scales can be usefully applied in a clinical context. Before
considering the application of face scales in clinical measurement of acute and
chronic pain complaints in children, it is useful to briefly highlight the properties
of several of these scales.

The Oucher Scale (Beyer 1984)

The Oucher Scale comprises 6 photographs of a child's face showing different

expressions of pain, positioned at 20 unit intervals on a vertical 0-100 numerical
scale. Initial research confirmed that children ranked the 6 photographs in the
same sequence as arranged by the authors of the scale, and that the numerical
measure was valid in the post-operative pain context (Aradine, Beyer &
Tompkins, 1988; Beyer & Aradine, 1988). The Oucher is the only published
scale that presents faces in a vertical rather than horizontal configuration.

201
Wong-Baker FACES (Whaley & Wong, 1987)

Illustrated in Figure 3, this scale comprises 6 faces, on a bipolar scale from 0

("happy") to 5 ("sad"). The faces were said to be developed from drawings by
children, although incorporating emotional reactions to pain (smile, tears) rather
than just a grade of sensory intensity of pain. This scale has been well-received
by the nursing community.

Facial Affective Scale. (McGrath, deVeber & Hearn, 1985)

As depicted in Figure 3, this scale comprises 9 faces on a continuum from 0

("happiest feeling possible") to 1 ("saddest feeling possible"). Using standard
psychophysical methods, children aged 5 to 17 years rated the degree of pain-
related affect ("feelings") depicted in each face. The resulting averaged
estimates of pain-related affect constitute the numerical values under each face.
Recent data suggests that, despite the relatively large number of faces
comprising this scale relative to alternatives, the scale does seem to offer a
measure of some affective component of the pain experience that is separate
from pain intensity in children able to separate the sensory and emotional
dimensions of pain (Goodenough, van Dongen, Brouwer, Abu-Saad &
Champion, 1999).

Faces Pain Scale (Bieri et aI., 1990)

The Faces Pain Scale was originally developed as a 7-point measure of pain
intensity, ranging from 0 (no pain) to 6 (most pain possible), and was based on
children's drawings of pain expressions. The scale has recently been adapted to a
6 face measure, the Faces Pain Scale - Revised (as shown in Figure 3) (Hicks,
von Beyer, Spafford, & Goodenough, 2000). This revision preserves the
psychometric properties of the original scale, and makes it possible to record
scores on the O-to-I 0 metric which is the most widely understood and accepted
convention in the scoring of pain (McCaffery & Pasero, 1999; von Baeyer &
Hicks, 2000).

Children's Anxiety and Pain Scale (Kuttner & LePage, /989)

The Children's Anxiety and Pain Scale (CAPS) is the only known dual faces
measure that attempts to separate children's self-assessed ratings of anxiety
(upper scale) and pain intensity (lower scale). There has been relatively little

202
psychometric evaluation of the scale, although recent data suggests that it is
well-received by children and may provide a useful clinical measure for
identifying pain-related anxiety in older children prior to procedures such as
venipuncture (Roschar, Goodenough, Abu-Saad, Kuttner, & Champion, 2000).

Which self-report/ace scale to use?

In keeping with recommendations to assess pain regularly and in age-appropriate
terms (e.g. McGrath et aI., 1995), facial expression scales are now used in many
paediatric health contexts to help children communicate pain to adults. Many of
the essential validation studies on face scales have been completed in clinical
acute pain contexts, especially short sharp pain associated with blood-sampling
or routine immunisation injection (e.g. Goodenough et aI., 1997) and post-
operative pain (e.g. Aradine et aI., 1988). One corollary benefit of these
validation studies is an increased understanding of the nature of acute pain in
children including age effects in pain report. For example, there is a tendency
for younger children to report more pain from needles than older children (e.g.
Fradet, McGrath, Kay, Adams & Luke, 1990; Goodenough, Thomas, Champion,
Perrott, Taplin, von Baeyer & Ziegler, 1999), whereas age effects are markedly
reduced (if apparent at all) in ratings of persistent post-operative pain (eg.
Perrott, Goodenough, Taplin, Champion, & Ziegler, 1999).
Overall, however, the management of paediatric pain has not matured to the
point that a single international standard has emerged for clinicians as the
preferred self-report measure based on facial expression. Rather, a clinician's
choice to use one of the many useful scales, including those depicted in Figure 3,
is still reasonably parochial: each scale tends to be implemented most frequently
by the institution in which the scale was developed, or by the professional
discipline associated with the authors of the scale. For example, the Facial Affect
Scale features most strongly in clinical and research applications connected to its
institution of origin (The University of Western Ontario, Canada); the Wong and
Baker FACES Scale predominates in texts written for a target audience of
Registered Nurses (e.g. Whaley & Wong, 1987; McCaffery & Pasero, 1999);
and the Faces Pain Scale features strongly in clinical research by psychologists.
To a large extent this situation is an historical accident, as the fruits of several
dedicated efforts to develop an age-appropriate self-report facial expression pain
scale were published in the same short timeframe.
It is clearly important that clinicians use a scale with which they feel
comfortable. It is perhaps for this reason that scales that include a 'happy face'
will have enduring appeal, irrespective of data showing that scales with smiley
rather than neutral 'no pain' anchor points may have a significant and
undesirable influence on pain ratings by children, such as significantly inflating

203
pain scores (Chambers & Craig, 1998). The incorporation of smiling faces in a
scale is clearly and significantly related to underlying assumptions concerning
the continuum of pain experience, especially chronic and persistent pain
problems. One assumption concerns whether the antithesis of 'pain present' is
the presence of pleasure or happy mood, rather than simply the absence of pain.
On this point, it is worth noting that there has been no formal study of children's
opinions concerning which internal state may constitute the 'opposite' of pain,
and therefore the implications this may have for defining anchor points on facial
expression scales.
Despite the apparent embarrassment of riches concerning scale choice, some
guidelines for the clinician are possible:
• Be mindful of the purpose of the pain ratings - The question of 'how does
your pain feel? , should not be confounded with the question of 'how do you
feel about your pain?' or more generally, 'how do you feel? . This important
distinction is the rationale for the dual scale pair for anxiety and pain in the
Children's Anxiety and Pain Scale depicted in Figure 3 (Kuttner & LePage,
1989). To measure pain intensity per se, a single scale without smiles and
tears, such as the Faces Pain Scale - Revised (Figure 3), is recommended.
To measure emotions other than pain intensity, such as anxiety and
depression, the Children's Anxiety and Pain Scale or the Facial Affective
Scale may be more suitable (see Figure 3). There is evidence that children's
estimates of pain using the Faces Pain Scale (as a measure of intensity) and
the Facial Affective Scale (as a measure of affect) do reflect different
components of the pain experience to some degree. For example, in a study
that asked children to use both of these scale types to rate venipuncture pain,
the ratings of the unpleasantness of pain (i.e. Facial Affective Scale scores)
correlated significantly better with a convergent non-facial measure of pain
unpleasantness than did those ratings of pain intensity using the Faces Pain
Scale. The reverse result was found (for the same children) when
correlating scores from the two measures with a convergent visual analogue
rating of pain intensity (Goodenough et aI., 1999).
• Be careful in the specification of the anchor points of the scale - It is
important that the child understands which face corresponds with 'no pain',
and that clinicians do not assume that this anchor point is obvious in
meaning and application. Even with apparently simple scales, some 4-6
year olds do not reliably apply the 'no pain' category in self-report
measures, including face scales (e.g. Goodenough et aI., 1997).
Unfortunately, there is as yet no consensus on how the 'extreme pain'
anchor points of facial expression scales should be defined. Variations
include 'most pain possible,' 'most pain imaginable,' 'happiest feeling
possible,' and 'very much pain.' From a clinical viewpoint, it is useful to

204
 define whether the terms of reference are to all pain of this type (e.g.
postoperative) or to all pain in the child's experience of any type, or to all
pain of any category (real or imagined).
• Be aware of concrete self-referential 'face-matching' - Without consistent
careful instruction, some children may seek to match a face with a self-
perceived impression of their own face during pain. It is noteworthy in this
regard that the Facial Affect Scale is perhaps the only published measure
with formal instructions explicitly asking children to choose a face based on
'how you feel deep down inside?' (McGrath, 1990, p.374), rather than how
the child thinks their face appeared to an observer. This issue is clearly
relevant when using scales with an extreme pain face showing tears, but
where the child did not cry or wish to admit to crying in response to pain.

Self-report measures and the chronic pain context

Thus far, this chapter has predominantly focused on facial expression in the
measurement of time-limited acute pain, both brief (e.g. venipuncture) and
persistent (e.g. post-operative). Chronic pain measurement in children and
babies, as with adults, can present specific difficulties for clinicians due to the
vast and often complicated array of factors that may need to be addressed. These
include family relationships, interference of pain with daily functioning, and
concomitant problems such as anxiety, depression and anger.
Whilst the chronic pain context is necessarily complicated, and the outcome
measures often cannot focus on achieving 'no pain', self-report pain measures
based on facial expression have been very helpful in routine assessment and
screening in older children (i.e. at least 8 years of age). When the outcome of
interest is change in relative pain intensity over time, a facial expression scale
can provide a useful means of helping a child to document pain episodes and
perhaps chart trigger factors (e.g. for headache or abdominal pain).
An example of the use of a self-report facial expression measure in the
chronic pain context is shown in Figure 4, using the original version of the Faces
Pain Scale (Bieri, Reeve, Campion, Addicoat, & Ziegler, 1990). This is an
extract of the screening questionnaire developed for use in the chronic pain
service at Sydney Children's Hospital (Goodenough, Crawford, & McInerney,
1998). It is completed separately by both a child patient (minimum age of 8
years) and accompanying parent on initial referral to the clinic. [n the section of
the questionnaire concerned with obtaining a comprehensive description of the
pain problem, the Faces Pain Scale is used as a means of describing the course
of a typical pain episode over a 24 hour period (see Figure 4). Useful
information also can be obtained by directly comparing the child and parent
ratings. In similar fashion, a facial expression scale can be a useful way of

205
charting pain episodes and maximum pain intensity and other possible symptoms
or events over a given time period or treatment phase.
In general, however, there is little research examining measurement issues
and the meaning of pain faces in the chronic paediatric pain context. Many of
the limitations are similar to those described previously for facial action coding
schemes and assessment of persistent acute pain faces.

9:00 p.m.
1~lbedUme)

12:00 midnight
I,,;ddo of tho nlghl)
3:001.m
(befOllilime to wake)

V Goodenough, Crswford, & McInerney, 1998

Figure 4. Excerptfrom the Sydney Children's Hospital Chronic Pain Service Screening
Questionnaire (Goodenough, Crawford, Mcinerney, 1998)

Other issues in applying self-report measures

In addition to the above considerations, some other issues concerning application

of self-report face scales deserve mention, whether the pain of interest is acute or
chronic. There is some advantage in using cartoon faces rather than real
photographs of children, such as those in the Oucher and irrespective of attempts
to make real photographs culturally relevant (e.g. Beyer & Knott, 1998). Firstly,
the representations of pain facial expression are likely to be less ambiguous in a
cartoon. Secondly, cartoons are more flexible as developmentally relevant
representations of the salient features of a pain face. For example, in drawings
of pain expressions, children generally seem to focus on the activity of the

206
mouth and eyes, rather than information related to the nose and naso-Iabial folds
(see Faces Pain Scale - Revised, Figure 3). Thirdly, some children may be less
likely to identify their pain with that depicted in a photograph of a crying child
as shown in the various forms of the Oucher scale and the Wong-Baker FACES
scale (Figure 3), especially if they themselves did not cry or wish to be
associated with the image of a young crying child.
When considering the facial expression of pain as an acceptable display of
emotion (see limitations of facial action coding schemes section earlier in
chapter), a self-report measure necessarily constitutes a rudimentary 'public
display' involving a child and the person administering the scale. Thus, self-
report ratings can be influenced by who is asking the child about pain - a
parent, a peer, a medical or nursing professional (Zeman & Garber, 1996). In
this sense, it is important to appreciate whether a child interprets a facial scale as
reflecting affect or intensity, and to make clear that the facial expressions in the
scale are not intended to mirror the child's perception of their face as it may
acceptably appear to an observer.
Finally, to what degree will a child's self-report agree with an assessment
based on the child's behaviour? The relationship between children's pain reports
and observers' estimates is exceedingly complex. Many variables can come into
play: e.g. the type of pain, the temperament and coping style of the child, the
context in which the child is assessed, and who is the observer (Champion et aI.,
1998). While the data are sparse, recent studies where both children and
observers use a measure incorporating facial expression have provided useful
insight into the scope of the problem. For example, in a study using the Faces
Pain Scale in the context of routine immunisation in 4 to 6 year olds, disparities
were generally in the direction of an observer underestimating pain relative to
the child's report, even when the observers were experienced paediatric nurses
confident in the accuracy of their estimates (e.g. Goodenough et aI., 1997).
When asked to apply a variety of behaviour cues, both nurses and parents have
reported relying on changes in the child's facial expression in needle pain in 4 to
15 year olds (Goodenough et aI., 1997, 1999). In these studies parents who
reported weighting behaviours other than facial expression (e.g. vocal or motor),
gave pain estimates which correlated relatively more poorly with the child's self
report (Goodenough et ai, 1999). Overall, the level of agreement between child
and observer is an area requiring further research in the clinical arena, especially
as much of the available data is from the context of short sharp pain involving
needles. Until this research is completed, the general rule of thumb has been to
respect, if not prioritise, self-report and to assume that invasive procedures are
painful until proven otherwise.

207
 A 'UNIVERSAL FACIAL EXPRESSION' FOR PAIN?

"Whitstthus screaming their eyes are firmly closed, so that the skin round them is
wrinkled, and the forehead contracted into a frown. The mouth is widely opened
with the lips retracted in a peculiar manner, which causes it to assume a squarish
form; the gums or teeth being more or less exposed" (Darwin, 1872)

Darwin was perhaps the first to formally publish observations on the relative
universality of human facial expression in response to distress, including pain.
Since the early writings of Darwin, studies have shown that, in adults, the most
characteristic response to induced pain is described by four facial actions: brow
lowering, tightening and closing of eyelids, and nose wrinkling I upper lip
raising (Craig, Prkachin, & Grunau, 1992; Prkachin, 1992). Essentially the same
facial actions have been observed in clinical pain, acute and chronic pain
contexts, and psychogenic pain (LeResche, 1982; Prkachin & Mercer, 1989).
The present overview of behavioural methods of paediatric pain
measurement, whether by observation or self-report, suggests that these same
component facial features in the adult response to pain are identifiable in babies
as young as 25 weeks gestation. Also the same feature set (at least the activity
specific to eyes and mouth) is spontaneously drawn in cartoon depictions of pain
intensity by school-aged children that have formed the basis for development of
some self-report scales based on facial expression of pain intensity (e.g. Faces
Pain Scale, Bieri et aI., 1990). From a lifespan perspective, neonate's expression
of pain is similar to that of adults (Craig et aI., 1992). Research investigating
developmental changes in facial expression during painful procedures has
identified a fairly consistent involuntary pain grimace, with only subtle changes
occurring around the mouth and eyes during the first two years of infancy
(Lilley, et aI., 1997; Johnson et aI., 1993)
The relative universality of the pain facial expression may be a helpful
concept given that health care professionals are often ethnocentric with respect
to interpreting patients' behaviours (Martinelli, 1987; Twycross, 1998). To this
end, some researchers have been examining the diagnostic value of pain facial
expression for disease status, such as myocardial infarction in adults (Dalton,
Brown, Carlson, McNutt, & Greer, 1999). However, many questions concerning
the clinical value of facial measures in paediatric pain assessment remain
unanswered.
One issue is whether a universal pain face may be identified in cognitively
impaired individuals, or generalisable to other groups such as the congenitally
blind (e.g. Galati, Guida, Miceli, & Sini, 1999; Hadjistavropoulos et aI., 2001).
This is a new area of study and current evidence is equivocal. The Neonatal
Facial Coding System has been shown to be sensitive to changes in facial
expression during venipuncture in children with moderate to severe

208
developmental delay, including Down Syndrome (Benini, Carli, Agosto, Drigo,
Gobbber, Lago, & Zacchello, 1997). Yet a recent study of adolescents with
cerebral palsy failed to discriminate facial responses to pain during mock and
real needle injections. (Oberlander et aI., 1999). While such data with paediatric
populations awaits replication in younger age groups and extension to rule out
potential measurement problems, the adult literature is also equivocal. Studies
with cognitively impaired adults have observed significant increases in facial
activity in response to time-limited acute pain (e.g. Hadjistavropoulos, Craig,
Martin, Hadjistavropoulos, & McMurtry, 1997; LaChapelle, Hadjistravropoulos,
& Craig, 1999), while other research has identified idiosyncratic facial
expressions in response to pain, and idiosyncratic failure to show signs of pain
when expected (Collignon, Giusiano, Porsmoguer, Jimeno, & Combe, 1995).
There has been little development and application of self-report facial
expression to cognitively impaired children. Generally, based on survey data,
health care professionals have demonstrated a reluctance to rely on self-report
instruments that are perceived to depend on skills of the cognitively impaired
child to any degree (e.g. Fanurik, Koh, Schmitz, Harrison, Roberson, &
Killebrew, 1999). To date, studies have focused on the elderly with cognitive
deficits. It would seem that the ends of the lifespan may share communication
dilemmas, and self-report scales developed for use by children have also been
useful in older populations, such as the Faces Pain Scale with frail elderly (e.g.
Herr, Mobily, Kohovt, & Wagenaar, 1998)

CONCLUSIONS

The study of human facial expressions can contribute to human welfare.

Recently developing scientific knowledge of 'pain faces' is leading to new ways
to assess pain in infants and children who cannot communicate their pain
verbally. While there is still work to be done in assessing the clinical relevance
of definitions of 'pain faces' from those expressions associated with general
distress, the most important developments fall in two categories.
Firstly, detailed observation of facial expressions in infants and children who
are in pain has led to valid and reliable measures of pain. Features that are most
recognizable as communicating pain in infants include the following: lowered
and/or bulging brow, eyes squeezed shut, deepening of the nasolabial furrow
(the crease running from the side of the nose to the corner of the mouth), and an
open, laterally stretched mouth. Many of these features can be seen in older
children and adults. However, their specificity is somewhat limited to short,
sharp pain and such expressions generally attenuate in longer-lasting pain,
though progress is being made on description of the facial features characteristic
of post-operative pain. The Neonatal Facial Coding System and the Children's

209
Facial Coding Scale are available for research on these indicators of pain. These
instruments are helping to identify effective pain-relieving interventions
(pharmacological, physical, and psychological).
Secondly, the use of schematic faces scales, emphasizing clear depiction of
the main universal features of pain expression, can facilitate self-report of pain in
children whose ability to estimate pain verbally, is limited. Separate validated
measures are available to measure pain intensity (e.g., the Faces Pain Scale -
Revised) and to assess the affect or distress associated with pain (e.g., the Facial
Affective Scale).
Research is underway in numerous centres to adapt the above approaches for
people of differing age, culture, and intellectual, sensory and motor ability.
Several challenges for researchers have been highlighted, but the contribution of
knowledge of the human face to the relief of pain is already undisputed.

AFFILIA TION

Belinda Goodenough and G David Champion

Sydney Children's Hospital
Randwick, NSW, Australia
Email: b.goodenough@unsw.edu.au

Carl L von Baeyer

Departments of Psychology and Paediatrics
University ofSaskatchewan
Saskatoon, Canada

ACKNOWLEDGEMENT

We thank the following individuals for constructive input: Liz Rocha (University
of Saskatchewan, Saskatoon, Canada), Kellie Hadden (University of
Saskatchewan, Saskatoon, Canada), and Marita Mcinerney (Paediatric Pain
Management, Sydney Children's Hospital, Randwick). The preparation of this
chapter was supported by generous private donations during fundraising for the
Paediatric Pain Medicine Unit organised by Gay Windeyer, Meri Lane, Clare
Dixon, Helen Wright, Julie Josue, Sue James, Tina Cole, Judy Bull, Tommy
Champion, and their families.

210
 REFERENCES
Anand, KJ.S., & Craig, K.D. (1996). New perspective on the definition of pain. Pain, 67, 3-6.
Aradine, e.R., Beyer, J.E., & Tompkins, J.M. (1988). Children's pain perceptions before and after
analgesia: a study of instrument construct validity and related issues of The Oucher. Journal of
Pediatric Nursing, 3, 11-23.
Barr, R.G., Rotman, A, Yaremko, J., Leduc, D., & Francoeur, T.E. (1992). The crying of infants
with colic: a controlled empirical description. Pediatrics, 90, 14-21.
Benini, F., Carli, G., Agosto, C., Drigo, P., Gobber, D., Lago, P., & Zacchello, F. (1997).
Evaluating pain induced by venipuncture in patients with moderate or severe developmental
delay. Paper presented at the Fourth International Symposium on Pediatric Pain, July, 1997,
Helsinki, Finland.
Beyer, J.E. (1984). The Oucher: A user's manual and technical report. Evanston, IL: Hospital Play
Equipment.
Beyer, J.E., & Aradine, C.R. (1988). Convergent and discriminant validity ofa self-report measure
of pain intensity for children. Children's Health Care, 16,274-282.
Beyer, J.E., & Knott, C.B. (1998). Construct validity estimation for the African-American and
Hispanic versions of the Oucher Scale. Journal of Pediatric Nursing: Nursing Care of Children
and Families, 13,20-31.
Bieri, D., Reeve, R., Champion, GD., Addicoat, L., & Ziegler, J.B. (1990). The Faces Pain Scale for
self-assessment of the severity of pain experienced by children: development, initial validation, and
preliminary investigation for ratio scale properties. Pain, 4, 139-150.
Chambers, C.T., Cassidy, K.L., McGrath, PJ., Gilbert, e.A., & Craig, K.D. (1996). CFCS: Revised
Manual - 1996. IWK Grace Health Centre: Dalhousie University & The University of British
Columbia.
Chambers, C.T., Craig, K.D. (1998). An intrusive impact of anchors in children's faces pain scales.
Pain, 78,27-37.
Champion, G.D., Goodenough, B., von Baeyer, C.L., & Thomas, W. (I998). Self-report measures
of pain in children. In PJ. McGrath, & G.A. Finley (Eds.), Measurement of pain in irifants and
children. Progress in pain research & management (pp. 123-160). Seattle, WA: International
Association for the Study of Pain Press.
Champion, G.D., Goodenough, B., Wu, E., Chua, K., Taplin, J.E., & Ziegler, J.B. (2000).
Children's self-report of pain intensity using six different scale types. Paper presented at the 6'h
International Symposium on Pediatric Pain, June 2000, London.
Collignon, P., Giusiano, B., Porsmoguer, E., Jimeno, M. E., & Combe, J. C. (1995). Difficultes du
diagnostic de la douleur chez I'enfant polyhandicape. Annales de Pediatrie, 42, 123-126.
Craig, K.D. (1992). The facial expression of pain: Better than a thousand words? American Pain
Society Journal, I, 153-162.
Craig, K.D., Grunau, R.v.E., & Aquan-Assee, J. (I988). Judgment of pain in newborns: facial
activity and cry as determinants. Canadian Journal of Behavioural Science, 20, 442-451.
Craig K.D., Prkachin, K.M., & Grunau, R.V.E. (2001). The facial expression of pain. In D.e. Turk
& R. Melzack (Eds.), Handbook ofpain assessment (pp. 153-169). New York: Guilford.
Craig, K.D., Whitfield, M.F., Grunau, R.Y.E., & Linton, J., & Hadjistavropoulos, H. (1993). Pain in
the pre-term neonate: Behavioural and physiological indices. Pain, 52,287-299.
Dale, J.e. (1986). A multidimensional study of infants' responses to painful stimuli. Pediatric
Nursing, 112,27-31.
Dalton, J.A., Brown, L., Carlson, J., McNutt, R., & Greer, S. (1999). An evaluation of facial
expression displayed by patients with chest pain. Heart Lung; National Library of Medicine,
MDX Health Digest, 28,168-74
Darwin, C. (1872). The expression of the emotions in man and animals. University of Chicago
Press, Chicago.

211
Davis, D.H., & Calhoon, M. (1989). Do preterm infants show behavioral responses to painful
procedures? In S.G. Funk, F.M. Tornquist, M.T. Champagne, L.A. Copp, & R.A. Wiese (Eds.),
Key aspects of comfort: Management of pain, fatigue and nausea (pp. 35-45). New York:
Springer.
Ekman, P. (1997). Should we call it expression or communication? Innovation, 10,333.
Ekman, P., & Friesen W.V. (1978). The Facial Action Coding System (FACS). Consulting
Psychologists Press, Palo Alto, CA.
Erdem, T.L., Ozcan, I., I1guy, D., & Sirin, S. (2000). Hereditary sensory and autonomic neuropathy:
review and a case report with dental implications. Journal of Oral Rehabilitation, 27, 180-183.
Fanurik, D., Koh, J.L., Schmitz, M.L., Harrison, R.D., Roberson, P.K., & Killebrew, P. (1999). Pain
assessment and treatment in children with cognitive impairment: a survey of nurses' and
physicians' beliefs. Clinical Journal of Pain, 15, 304-12
Fearon, I., Tranmer, J., Muir, D. W., Hains, S.M., & Kisilevsky, B.S. (1997). Swaddling after heel
lance: age-specific effects on behavioral recovery in preterm infants. Journal of Developmental
& Behavioral Pediatrics, 18,222-232.
Fridlund, A. J. (1991). Evolution and facial action in reflex, social motive, and paralanguage.
Biological Psychology, 32,3-100.
Fridlund, A. J. (1994). Humanfacial expression: An evolutionary view. San Diego: Academic Press.
Galati, D., Guida S., Micheli R., & Sini B. (1999). Facial expression of emotion in congenitally
blind children: Coding and judging the facial expressions of a subject group between the third
and the fifth year of life. Ricerche di Psicologia, 22,45-79.
Gilbert, C.A., Lilley, C.M., Craig, K.D., McGrath, P.J., Court, c.A., Bennett, S.M., & Montgomery,
C.J. (1999). Post-operative pain expression in preschool children: validation of the Child Facial
Action Coding System. Clinical Journal of Pain, 15, 192-200.
Goodenough, B., Addicoat, L., Champion, G.D., McInerney, M., Young, B., Juniper, K., & Ziegler,
J.B. (1997). Pain in 4- to 6-year old children receiving intramuscular injections: a comparison
of the Faces Pain Scale with other self-report and behavioural measures. Clinical Journal Clf
Pain, 13,60-73.
Goodenough, B., Crawford, M., & McInerney, M. (1998). Sydney Children's Hospital Chronic
Pain Service Screening Questionnaire. Sydney Children's Hospital, Randwick, Australia.
Goodenough, B., Thomas, W., Champion, G.D., Perrott, D., Taplin, J.E, von Baeyer, C.L., &
Ziegler, J.B. (1999). Unravelling age effects and sex differences in needle pain: ratings of
sensory intensity and unpleasantness of venipuncture pain by children and their parents. Pain,
80, 179-190.
Goodenough, B., van Dongen, K., Brouwer, N., Abu-Saad, H.H., & Champion, G.D. (1999).
Comparison of the Face Pain Scale and the Facial Affective Scale for children's self-report of
pain during blood sampling procedures. European Journal of Pain, 3, 301-315
Greco, A., Villa, R., Fusetti, L., Orlandi, R., & Pierotti, M.A. (2000). The Gly571Arg mutation,
associated with the autonomic and sensory disorder congenital insensitivity to pain with
anhidrosis, causes the inactivation of the NTRK IInerve growth factor receptor. Journal of
Cellular Physiology, 182,127-33.
Grunau, R.E., Holsti, L., Whitfield, M.F., & Ling, E. (1999). Arc twitches. startles and body
movements pain indicators in extremely low birth weight infants? The Clinical Journal of Pain,
16,37-45.
Grunau, R.V.E., & Craig, K.D. (1987). Pain expression in neonates: facial action and cry. Pain, 28.
395-410.
Grunau, R.V.E., & Craig, K.D. (1990). Facial activity as a measure of neonatal pain expression. In
D.C. Tyler & E.J. Krane (Eds.), Advances in Pain Research and Therapy: Vol 15, Pediatric
Pain (pp. 147-156). Raven Press, New York.
Hadjistavropolos, T., Craig, K.D., Martin, N., Hadjistavropoulos, H., & McMurtry, B. (1997).
Toward a research outcome measure of pain in frail elderly in chronic care. The Pain Clinic,
10, 71-79.

212
Hadjistavropoulos, H.D, Whitfield, M.F., Grunau, R.E., Craig, K.D. (1997). Judging pain in infants:
behavioural, contextual, and developmental determinants. Pain, 73,319-24.
Hadjistavropoulos, T., von 8aeyer, C.L., & Craig, K.D. (2001). Pain assessment in persons with
limited ability to communicate. In D.C. Turk & R. Melzack (Eds.), Handbook oj Pain
Assessment. (pp. \34-149). New York: Guilford.
Herr, K,. Mobily, P., Kohovt, F., & Wagenaar, D. (1998). Evaluation of the Faces Pain Scale for use
with the elderly. The ClinicalJournal ojPain, /4, 1-29.
Hicks, C., von 8aeyer, C.L., Spafford, P., & Goodenough, 8. (2001). Faces Pain Scale - Revised:
Adapting the Faces Pain Scale in the movement toward a common metric in pediatric pain
measurement. Pain, 93, 173-183.
Izard, C.E., Hembree, E.A., Dougherty, L.M., & Spizzirri, C.c. (1983). Changes in facial
expression of 2- to 19-month old infants following acute pain. Developmental Psychology, 19,
418-26.
Izard, C.E., Huebner, R.R., Risser, D., McGinnes, G.c., & Dougherty, L.M. (1980). The young
infant's ability to produce discrete emotional expressions. Developmental Psychology, 16, 132-
140.
Johnston, C.C., & Strada, M.E. (1986). Acute pain response in infants: a multidimensional
description. Pain, 24,373-382.
Johnston, C.C., Horton, L.1., Stevens, 8.1., & Stremler, R.L. (1997). Effectiveness of oral sucrose
and simulated rocking on pain response in preterm neonates. Pain, 72,193-199.
Johnston, C.C., & Stevens, 8.1. (1996). Experience in a neonatal intensive care unit affects pain
response. Pediatrics, 98,925-30.
Krechel, S.W., & 8i1dner, J. (1995). CRIES: a new neonatal post-operative pain measurement score.
Initial testing of validity and reliability. Paediatric Anaesthesia, 5, 53-61.
Kuttner, L., & LePage, T. (1989). Faces Scales for the assessment of pediatric pain: A critical
review. Canadian Journal oj Behavioural Science, 21, 191-209.
LaChapelle, K. L., Hadjistavropoulos, T., & Craig, K. D. (1999). Pain measurement in individuals
with intellectual disabilities. Clinical Journal oj Pain, 15, 13-23.
LeResche, L. (1982). Facial expressions of pain and emotions in chronic TMD patients. Pain, 35,
71-78.
Lilley, C.M., Craig, K.D., & Grunau, R.Y.E. (1997). The expression of pain in infants and toddlers:
developmental changes in facial action. Pain, 72, 161-170.
Lindh, Y., Hakansson, S., Sandman, P.O., & Wiklund, U. (1997). Assessment of acute pain in
preterm infants by evaluation of facial expression and frequency domain analysis of heart rate
variability. Early Humam Development, 48, \31-42.
Martinelli, A.M. (1987). Pain and ethnicity: how people of different cultures experience pain.
Association ojOperating Room Nurses (AORN) Journal, 46,273-281.
Marvin, J.A., & Pomietto, M. (1991). Pain assessment in infants (0-12 months) using the Neonatal
Facial Action Coding System. Journal oj Pain and Symptom Management, 6, 193.
McCaffery, M., & Pasero, C. (1999). Pain: Clinical manual. St. Louis: Mosby, Inc.
McGrath, P.A. (1990. Pain in children: Nature, assessment, and treatment. New York: Guilford
Press.
McGrath, P.A., deYeber, L.L., & Hearn, M.1. (1985). Multidimensional pain assessment in
children. In H.L. Fields, R. Dubner, & F. Cervero, (Eds.), Proceedings of the Fourth World
Congress on Pain. Advances in Pain Research and Therapy New York: Raven Press.
McGrath, P.1., Johnson, G., Goodman, J.T., Schillinger, J., Dunn, J., & Chapman, J. (1985). A
behavioral scale for rating post-operative pain in children. In H.L. Fields, R. Dubner, & F.
Cervero, (Eds.), Advances in Pain Research and Therapy (pp. 395-401). New York: Raven
Press.
McGrath, P.1., Unruh, A.M., & Finley, G.A. (1995). Pain measurement in children. In G.A. Finley
& P.1. McGartyh (Eds.), Pain: Clinical updates (pp. 1-4). Seattle: IASP Press.

213
Mitchell, P. (1999). Understanding a young child's pain. Lancet, 354, 1708.
Oberlander, T.F., Gilbert, C.A., Chambers, C.T., O'Donnell, M.E., & Craig, K.D. (1999).
Biobehavioral responses to acute pain in adolescents with a significant neurological
impairment. The ClinicaLJournal of Pain, 15,201-209.
Perrott, D., Goodenough, B., Taplin, J.E., Champion, G.D., & Ziegler, J.B. (1999). Children's self
report of post-operative pain intensity and pain-related affect: A preliminary comparison of
four different measures. Paper presented at the 20th Annual Scientific Meeting of the
Australian Pain Society, April 1999, Fremantle, Australia.
Pigeon, H.M., McGrath, P.1., Lawrence, J., & MacMurray, S.B. (1989). Nurses' perceptions of pain
in the neonatal intensive care unit. Journal of Pain and Symptom Management, 4, 179-183.
Prkachin, K.M. (1992). The consistency of facial expressions of pain: a comparison across
modalities. Pain, 51,297-306.
Prkachin, K. & Craig, K.D. (1995). Expressing pain: the communication and interpretation of facial
pain signals. Journal of Nonverbal Behavior, 19, 191-205.
Prkachin, K.M., & Mercer, S.R. (1989). Pain expression in patients with shoulder pathology:
validity, properties, and relationship to sickness impact. Pain, 39,257 - 265.
Ramenghi, L.A., Levene, M.I., Griffith, G.C., Wood, C.M. (1996). Reduction of pain response in
premature infants using intraoral sucrose. Archives of Disease in Childhood: Fetal & Neonatal
Edition, 74, 126-128.
Roschar, F., Goodenough, 8., Abu-Saad, H.H., Kuttner, L., & Champion, G.D. (2000). Children's
report of anxiety and pain during venipuncture: convergence between ratings on the Children's
Anxiety and Pain Scale and an analogue measure. Paper presented at the Fifth International
Symposium on Paediatric Pain, June, 2000, London.
Russell, J.A. (1995). Facial expressions of emotion: What lies beyond minimal universality?
Psychological Bulletin, lIB, 379-391.
Scott, C.S., Riggs, K.W., Ling, E.W., Fitzgerald, C.F., Hill, M.L., Grunau, R.Y.E., Solimano, A., &
Craig, K.D. (1999). Morphine pharmacokinetics and pain assessment in premature newborns.
Journal of Pediatrics, 135,423-429.
Stevens, B.1., Johnston, C.C., & Horton, L. (1994). Factors that influence the behavioral responses
of premature infants. Pain, 59, 101-109.
Strack, F., Martin, L.L., & Stepper, S. (1988). Inhibiting and facilitating conditions of the human
smile: A non-obtrusive test of the facial feedback hypothesis. Journal of Personality and Social
Psychology, 54,768-777.
Tatman, A., & Johnson. P. (1998). Pain assessment in the pre-verbal child. In A. Twycross, A.
Moriarty, & T. Betts (Eds.), Paediatric Pain Management: A multidisciplinary approach (pp.
77-94). Radcliffe Medical Press: Oxon.
Twycross, A. (1998) Perceptions about paediatric pain. In A. Twycross, A. Moriarty, & T. Betts
(Eds.), Paediatric Pain Management: A multidisciplinary approach (pp. 1-24). Radcliffe
Medical Press: Oxon.
von Baeyer, C.L., & Hicks, C.L. (2000). Support for a common metric for pediatric pain intensity
scales. Pain Research and Management, 4, 157-160.
Watt-Watson, J.H., & Donovan, M.1. (1992). Pain management: Nursing perspective. Mosby Year
Book: St Louis, USA.
Whaley, L., & Wong, D.L. (1987). Nursing care of infants and children. St Louis: CY Mosby.
Wolf, A.R., (1999). Pain, nociception and the developing infant. Paediatric Anaesthesia, 9, 7-17.
Yaster, M., Krane, E.1., Kaplan, R.F., Cote, C.1., & Lappe, D.G. (1997). Pediatric Pain
Management and Sedation Handbook. Mosby Year Book: Missouri.
Zeman, 1., & Garber, J. (1996). Display rules for anger, sadness and pain: it depends on who is
watching. Child Development, 67,957-973.

214
 ARVID KAPPAS

CHAPTER 11

WHAT FACIAL ACTIVITY CAN AND CANNOT

TELL US ABOUT EMOTIONS

The belief that facial activity is linked to emotional states has a long history in
science and in popular belief. In particular, the 1872 publication of Charles
Darwin's book The expression of the emotions in man and animals (187211998)
has had a major impact on the role of emotional expressions in modem emotion
research and has largely reinforced the commonly held belief that there is a link
between emotions and expressive behavior. Milestone research, particularly by
Paul Ekman and his colleagues, has cemented the common notion that faces
express emotions. However, all of the involved researchers, including Darwin
and Ekman, have usually maintained a view that is far more differentiated than
the distilled version that has resulted from repeated summarizing and
synthesizing in the secondary and tertiary literature. In fact, no current
researcher holds that all emotions are always reflected in facial or other
nonverbal activity. Inversely, almost all theorists will concur that often times
facial expressions are not associated with a concordant affective state, or for that
matter, any affective state at all.
This chapter will take stock of the current controversies in the debate
regarding the relationship of facial activity and affective state. Specifically, I
argue that there are strict limits to inferences regarding underlying affective
states based on measurements of facial activity. The arguments raised here
should be of particular interest to those who intend to apply findings regarding
the relationship of emotion and facial actions, such as in clinical settings for
initial diagnosis or accompanying the therapeutic process. In fact, any context in
which attempts are made to interpret facial activity as a means of determining
underlying affective states (or action tendencies, motivations, attitudes, or other
states of mind) is affected by oversimplifications of the empirical data available
to us today. Yet, regardless of the criticism offered here, I do not reject the
potential usefulness of measuring facial activity for basic research and in applied
contexts. To my mind there is no doubt that one of the origins of the production,
and of the perception of facial actions is genetically determined and that, at
times, there are indeed links between facial actions and the aforementioned
states of mind. Instead, it will be argued that those interested in using measures
of facial actions as tools to augment, or even replace, measures of subjective
experience, should be aware of issues that have been identified and partially
solved in connected domains, such as in psychophysiology.

THE PROBLEM OF HETEROGENEITY OF AFFECTIVE STATES AND

LEVELS OF DESCRIPTION

Any discussion of the relationship between emotion and facial expression is

complicated by the state of affairs in emotion theory. There are three major
schools of thought regarding the sets of emotional states that are supposedly
represented in facial activity (see Russell & Fernandez-Dols, 1997). The basic
emotions view assumes that there is a small and limited set of emotions that are
characterized by signature expressions (e.g., Ekman, 1982; 1992a; 1992b; 1994a;
I 994b; 1995; Izard, 1971, 1997). In contrast, the appraisal view assumes that
emotional states are the consequence of affective information processing and that
there are molecular expressive concomitants of this information processing or its
outcomes. Hence, some facial patterns are the sum or product of a pattern of
appraisal outcomes that in tum also relate to specific emotion labels (e.g.,
Scherer, 1992, 1993; Smith & Kirby, 2001; Smith & Scott, 1997; see also Kaiser
and Wehrle, 2001 and Kaiser, this volume). Finally, the dimensional view holds
that emotional states are fundamentally differentiated on a small number of
dimensions, such as valence and activation and that facial activity is linked to
these dimensions (e.g., Russell, 1997).

Basic emotions view

Common wisdom seems to be closest to the basic emotions view: When people
are happy they smile, and when they are angry they frown. In fact, much of the
research conducted in the last 30 years on the relationship of emotions and facial
actions has focused on the basic emotions view, probably because the lion share
of studies has been strongly influenced by Paul Ekman, Carrol Izard, and their
colleagues, who are proponents of this notion.
The burden of proof for any researcher in the area of facial actions is not a
light one. I will show that it is not sufficient to show specific instances in which
self-report of emotion and "prototypical expressions" are concordant to infer
affective states from facial behavior. In fact, any demonstration of the non-
exclusiveness of this relationship is sufficient to cast doubt on the immediate
diagnostic value of facial actions as indicators of affective state. Consider, for
example, a study by Jose Miguel Fernandez-Dols and his colleagues (Fernandez-
Dols, Sanchez, Carrera, & Ruiz-Belda, 1997). Participants in their study were
left alone in a room waiting for the experimenter, while the actual experiment

216
had surreptitiously already begun. A TV monitor in the room started to show a
segment of a horror movie and participants' facial reactions were being filmed
without their knowledge while they were watching the movie. The experimenter
then returned to the room and obtained the participants' self-reports. There was
almost no coherence between the patterns of facial activity as assessed with
Ekman and Friesen's Facial Action Coding System (FACS; Ekman & Friesen,
1978) and self-reported emotion. This finding was particularly significant as
participants were alone in the room and did not know that they were being
observed. These conditions should have minimized the effect of display rules or
experimental demand characteristics (Cacioppo, Tassinary, & Fridlund 1990;
Ekman & Friesen, 1969). In this sense, the study is a "clean" test of the
assumption of coherence as posited by the basic emotions view. However,
details of the study could be questioned or criticized, such as potential problems
with the exact moment selected for analysis and the way self report was collected
etc. A suspicious reader might argue that the authors of this study clearly are not
proponents of the basic emotions view and might have overstated the
implications of their data to make a particular argument. In fact, other authors
have shown situations under which coherence of self-report and facial action
appears to have been much higher. Erika Rosenberg and Paul Ekman have made
the strongest case in a quite elegant study (Rosenberg & Ekman, 1994). Yet,
even here the match of self-reported emotional state and predicted pattern of
facial actions was far from perfect. For the two films used in the study, only one
yielded a level of coherence between self-report and predicted pattern of facial
action shown significantly above chance level over the duration of the film, and
here the coherence was only at a median quotient of .50 compared to a chance
level of .30. This implies that the assumption that any given occurrence of facial
expression Fx would imply the presence of emotion Ex would fail many times.
Inversely, the absence of expression Fx does not always imply the absence of
state Ex. This underlines another problem - knowing the probability of the
presence of a particular emotional state, given the presence of a particular
expressions, does not allow us to assume that this probability is the same for the
inversion, where we assess what the probability of the occurrence of a particular
expression is, given the presence of a particular state. This can be expressed
more formally as:

p( emotionAl expressionA) t- p(expressionAl emotionA)

I would like to illustrate this issue with a hypothetical example: Assume that a
researcher would find in a well-controlled study that all instances of disgust were
accompanied by a particular sneer. In this case p(sneer I disgust) = Ii. However,
there would also be instances when the same sneer could be found when no
experience of disgust was reported. Hence, the presence of the sneer could not
conclusively indicate an underlying emotional state of disgust. The asymmetry

217
 between the two relations causes problems to the diagnostic usefulness of the
expression (the sneer) to infer the state (disgust) despite the fact that all instances
of disgust were accompanied by a sneer! However, if participants would not
even show a sneer in each and every instance they would feel disgust, i.e.,
p(sneerldisgust) < I, the situation would be even more complicated, or if you
will, worse. I will discuss and illustrate these challenges to statistical (and
logical) inference in more detail later in this chapter.
Let me be clear. There are very specific and restrained contexts where
coherence could be much higher. For example, in the above mentioned study,
Rosenberg and Ekman (1994), using a between-subjects criterion, analyzed
coherence for a specific moment in two films and found much higher coherence
there. However, the crux of the matter is that it is not sufficient to show that
there are isolated instances under which coherence can be shown. If facial
activity is to be used in an applied context, such as clinical diagnosis, we do not
know what the exact internal and external conditions are - the context is much
less controlled than in the laboratory. Hence, we will not know which moment
to pick.
If we wanted to assess the emotional state of a patient, or a child at play, we
do not have a particular stimulus, such as a film, that we can rate over and over
again, as we can in the laboratory. Instead we have to try to infer underlying
affective states based on observable behavior using some sort of boot-strapping
technique. Perhaps there are means to identify specific contexts or conditions
that increase the likelihood of coherence, such as at high intensities of underlying
affective state (see also Rosenberg, 1997; Ruch 1995). However, given that
subjective experience is usually the variable to be predicted in real-world
applications, self-report cannot be the criterion for the identification of instances
of "peak intensity" of the underlying affective state. Clearly, facial activity itself
is not a good indicator for such "peak moments". That is, it cannot assumed that
the moment characterized by the highest level of facial activity is also the "most
genuine" moment. The argument has been made that there are specific patterns,
such as specific smiles (Ekman & Friesen, 1982; Frank, Ekman, & Friesen,
1993; Frank this volume) that are considered true indicators of enjoyment.
However, even in these and other studies there has not been the perfect match
between expression and state that would qualify for complete coherence. So in
summary, there seems to be moderate coherence between specific self-reported
emotional states and facial activity. These findings are interesting and
correspond to what the large majority of researchers in the field would expect
today (but see Fridlund, 1994). However, even proponents of the basic emotions
view have not been able to demonstrate high coherence of subjective experience
and overtly shown facial activity beyond specific episodes that are only a few
seconds in duration.

218
Appraisal view
There are fewer studies on facial activity that have been conducted in the
framework of appraisal theory. Appraisals are information processing steps that
relate a situation or event to their personal implication, such as their implicit
meaning for the goals and needs of the individual (Scherer, Schorr, & Johnstone,
2001). Most of these processes occur outside of awareness and are thought of as
being direct, immediate, and intuitive (Arnold, 1960). Proponents of appraisal
theory assume that the outcomes of the appraisals are associated with changes in
activity in various central and peripheral systems, including the face. Rather than
assuming that facial actions are produced in patterns because a specific affect-
program has been triggered (a la Tomkins, see Ekman, 1995), it is argued that
there are infinite possible combinations of patterns. However, we as observers,
do not perceive that there is random activation, but that there is some internal
coherence between components of facial actions in many circumstances. In fact,
similar patterns are perceived by us as belonging to the same emotion or emotion
family.
To illustrate the rationale of the appraisal view we will consider the
activation of Corrugator Supercilii, a muscle in the brow region that is involved
in the act of frowning. Predating modern appraisal theory, Darwin maintained
that the perception of an obstacle would be accompanied by activation of the
brow muscle that produces the frown. This assumption has been restated by
proponents of the appraisal view, for whom the perception of an obstacle, that is
something that hinders us reaching a goal, is a central dimension of appraisal
(e.g., Scherer, 2001; Smith, 1989). Given that there are several negative
emotional states that share the perception of an obstacle, several patterns of
facial action to be observed will share the frown. In other words, the appraisal
view makes predictions concerning the occurrence of specific facial actions
linked to appraisal outcomes, rather than patterns of facial actions linked to
specific emotions (such as anger, fear, or sadness).
However, appraisal researchers have a handicap that adherents to the basic
emotions view do not have. Specifically, according to most versions of appraisal
theory, it is argued that we do not have conscious access to the appraisal process
as such. This is in contrast to current notions of emotional states outside of
appraisal theory - we assume that someone who is angry, knows she is angry,
someone who is sad, knows that he is sad. Thus, if we want to establish whether
a specific "emotional expression" is linked to a particular state, all we seem to
need to do then is to ask the person how she feels and to measure her facial
actions concurrently. However, this cannot work readily within the appraisal
framework. In fact, from the Zajonc-Lazarus debate in the early 1980s on to the
most recent versions of so-called mUltiple level processing models of appraisal
(Leventhal & Scherer, 1987; Scherer, 2001; Smith, Griner, Kirby, & Scott, 1996;
Van Reekum & Scherer, 1997) it has been maintained that much of the appraisal

219
process is automatic and outside of conscious awareness (Kappas, 2001). The
implication is that it is excruciatingly difficult to validate whether a specific
appraisal, for example that a person evaluating her coping potential as high, is
associated with a specific change in facial actions, such as tight lips (FACS
action unit 23; see Scherer, 2001). We cannot trust self-report of participants,
even if they themselves are rather sure 2 • One possible way of coping with this
problem is to create situations that are judged by a large proportion of
participants in a particular way. For example Kappas and Pecchinenda (Kappas,
1995; Kappas & Pecchinenda, 1999) created a video game that resembles the
classic Pacman game (for similar approaches see Banse, Etter, Van Reekum, &
Scherer, 1996; Kaiser, this volume; Kaiser & Wehrle, 2001). Here the
participants' goal is explicitly defined by the constraints of the game. For
example, the participants' task is to navigate through a maze, collecting as many
points as possible and to avoid "monsters" that can take points away. In such a
context it is possible to introduce events that are plausibly perceived or evaluated
in a particular way because the degrees of freedom of real-life situations are
reduced to the parameters of the game.
In one of our studies3 32 participants (19 women; mean age 22 years) played
a series of fourteen games with a duration of 60 seconds each. Games varied in
difficulty. Specifically, the player would be either faster, slower, or have the
same speed as the monsters and the exact ratios of player to monster speed were
pretested in several pilot studies. Participants navigated through the maze using
a joystick while facial electromyographic activity (EMG) at the brows
(Corrugator Superci/ii) and the cheek (Zygomaticus Major), as well as heart rate
and skin conductance were recorded. The first eight trials served to familiarize
the participants with the game. In the ninth game, starting 19 seconds from the
beginning of the game, the response mode of the joystick was inverted for three
seconds. For this short period the player symbol would move right if the joystick
was moved left, left if the joystick was moved right, up if the joystick was
moved towards the player, and down if the joystick was moved away from the
player. This manipulation occurred only once over the course of the whole
experiment. In objective terms, the inversion of the joystick response created an
obstacle while the participants were engaged in their play and an increase in
activation at the Corrugator Supercilii site was expected. It was assumed also
that the single occurrence of the inversion or "malfunction" would not be
perceived as very unusual after the total sequence of 14 games. A three-second
window, also starting 19 seconds from the beginning of trial 10 served as the
comparison condition. In fact, there was no difference in self-report between the
two trials and participants did not report, in the post-experimental interview,
having noticed the "malfunction". However, the activation of Corrugator
Supercilii in the critical time window was significantly higher than that in the
control condition. The fact that Corrugator Supercilii activity was higher, as
predicted, makes it plausible that the obstacle was perceived albeit potentially

220
outside of participants' awareness. This finding requires replication and the fact
that in this particular experiment the comparison condition was not
counterbalanced over trials nine and ten is a methodological concern. However,
the study illustrates nicely a paradigm in which a specific link between an
appraisal outcome and a single facial action is tested. In this case, the results
seemed to support the hypothesis. However, what if we had not found the
coherence between appraisal outcome and facial action? Had there not been the
predicted change, it would have been very difficult to know whether the
manipulation was not perceived at all, or whether there is in fact no clear
relationship between a specific appraisal and a specific muscle action.
Depending on the research question, this poses a problem. To demonstrate the
usefulness of a paradigm, such as the video game, for the manipulation of
appraisals in real time, it is sufficient to show positive results of a few individual
studies. However, if facial actions were to be used to diagnose ongoing
appraisals it would be critical to see whether there were failures to show the
predicted relationships. Indeed, we have begun investigating conditions, such as
the social context in which the games are played, that modulate the occurrence of
predicted coherence between facial actions and appraisals (e.g. Kappas &
Pecchinenda, 1996).
One of the characteristics of the predictions derived from the basic emotions
view is, as alluded to above, that particular patterns of facial activation are
associated with specific "basic" emotions. If someone is angry, she shows an
angry face - if she shows an angry face, she must be angry. Within the appraisal
tradition the predictions appear fuzzier. For example, both Smith and Scott
(1997) and Scherer (2001) predict that the same facial actions could be linked to
more than one appraisal dimension and that there are different possible facial
actions for the same appraisal dimension. For example, a frown might indicate
the perception of an obstacle or the expenditure of effort (Smith & Scott, 1997).
Obviously, there is nothing a-priori wrong with the validity of these predictions
- the problem is the inversion of the predicted appraisal-expression link. If she
perceives an obstacle she frowns, but if she frowns ... does she perceive an
obstacle or is she simply engaged in effortful behavior? Cacioppo, Tassinary,
and Berntson (2000) refer to this type of relationship between a physical
response and a psychological process as a many-to-many relation. Following
their arguments regarding drawing inferences in such circumstances, it is
essentially impossible to use specific instances of facial actions to reliably
diagnose the occurrence of a specific appraisal or to detect a specific appraisal
outcome. So is there any use in measuring facial activity from an appraisal point
of view? Of course, we can establish links between facial actions and appraisals
that appear reliable when analyzed at a group level. Results of this type can be
important in answering basic questions regarding the appraisal process, but they
do cast a shadow of doubt on the diagnostic value of facial actions within the
appraisal framework at the current time. In other words, the predictions of the

221
basic emotions view are usually so clear, that if they were to be shown to hold,
we could assume that facial activation can serve to identify underlying affective
state. In comparison, it is clear that many predictions of the appraisal view are
not specific enough. Based on this reasoning it appears that we can be pretty
sure, even before many of the necessary studies have actually been conducted,
that there are conceptual problems with using individual facial actions for an
analysis of underlying affective state to infer individual appraisals.

Dimensions view
The dimensions view to facial actions is more similar to the appraisal view than
to the basic emotions view. The assumption here is that the basic dimensions of
an underlying emotional state, particularly valence (positive vs. negative) and
activation are reflected in facial behavior. Keltner and Ekman (2000) argue that
most theorists subscribing to the dimensions view believe that emotions are
socially learned and culturally variable, while those subscribing to the basic
emotions view ("discrete-emotion theorists") take an evolutionary approach.
However, prominent modern proponents of the dimensional analysis of
emotional states seem rather Darwinian and/or biologically oriented and argue
that the predominant valence dimension is related to approach/avoidance
processing and has concrete counter parts in the way our brains are organized
(see Bradley, 2000; Cacioppo, Berntson, Larsen, Poehlmann, & Ito, 2000;
Davidson, 1999).
Whilst there are many studies that focus on whether the perception of facial
expressions can be described better by dimensional processing than specific
emotion categories (see Russell, 1997) there are fewer studies that focus on the
dimensional encoding of affective states in the face. The former group of studies
treats faces as stimuli in judgment studies. The latter group of studies is
concerned with specific facial changes in people whose affective state is
described in terms of two or three affective dimensions (valence, activation,
power). It is important here to distinguish between experiments that were from
the outset designed to measure specific facial correlates of affective dimensions,
or experiments that were conceived of in a discrete-emotions framework and
that, due to their design, failed to test the original hypotheses and simply did not
allow inferences going beyond dimensional differences. For example, it is not
untypical for studies in the field of emotion to compare a positive state to a
negative state, e.g., by showing participants funny and disgusting films. Because
the valence dimension is very pervasive, the argument can be made that
differences found in facial behaviors, and also in physiological activation in
experimental approaches that include only single instances of positive and
negative states, are not due to the specific discrete affective states elicited by the
stimuli, but instead simply to changes towards a pleasant or an unpleasant state.
For example, Winton (1986) criticized years of research on the facial feedback

222
hypothesis, because all relevant studies did not allow distinguishing dimensional
feedback effects from discrete emotional state effects.
To illustrate the problem of confounding dimensional and discrete effects, a
different example will be used here. Assume that you were interested in the
effect of bitter vs. sweet tastes on facial activity. You could film participants
while they tasted sweet and bitter food and measure their facial actions. The
problem is that the differences you would find, might be due to the pleasant
nature of the sweet taste compared to the unpleasant nature of the bitter taste.
The only way to discount the dimensional explanation would be to test different
negative tastes against each other for example bitter and sour food. Now let us
invert the original question - assume that you wanted to test the effect of
pleasant and unpleasant tastes on facial activity. You choose an example of each
class of stimuli, such as a bitter and a sweet liquid. How would you know that
differences in facial reactions to the liquids would be due to the valence
dimension and not to the specific tastes? The core problem in these examples is
the multiple determination of facial actions and the problems this poses for
statistical (or logical) inference. The next section will deal with these issues in a
more systematic way.
Obviously, there is a reason why dimensional organization of emotional
concepts, states, or reactions tends to show up in the results of many studies.
Even if there were discrete emotions as postulated by some of the above-
mentioned theorists, or even if emotional states were caused by appraisals and
the observable reactions would be the sum of a series of molecular reactions,
there would be a dimensional organization, simply because discrete emotions
themselves, or appraisal patterns are related to the two or three dimensions
typically postulated, particularly valence. As Figure 1 suggests, we could
interpret all three approaches, as being true at different levels of observation, or
description. The relationship between discrete emotions and these dimensions
has been shown multiple times. The relationship of appraisal dimensions to
basic dimensions and to discrete emotions is a bit more complicated but highly
plausible (Ortony, Clore, & Collins, 1988). For example, there is a clear relation
between the valence dimension as proposed by Arnold (\ 960) and the basic
valence dimension found in all dimensional approaches. In fact, in almost all
appraisal theories there is a dimension that captures congruence with one's goals
or needs (Scherer, Schorr, & Johnstone, 200 I). In some cases, the complexity of
a theory makes this relationship appear somewhat more complicated. For
example, Scherer (2001) makes a distinction between the intrinsic pleasantness
of a stimulus and congruence with one's goals (goal conduciveness in his
terminology). Here it is possible that a pleasant stimulus is counter to one's
goals or an unpleasant stimulus can be consistent with one's goals. Obviously,
this type of ambiguity is difficult to map on a single valence dimension.
Furthermore, there is the possibility of appraisals occurring in parallel at
different levels, contributing to the possibility of having ambivalent appraisal

223
outcomes (e.g., Scherer, 2001). However, if one would take a large number of
situations that were appraised there would be a strong concordance between
representations of valence at a dimensions level and the appraisal of goal
congruence at an appraisal level of description. Similarly there is a close link
between appraisals of coping potential and the power dimension. The activation
dimension is a bit more complicated because on the one hand there seems to be
some linkage to the importance of the event (goal relevance), on the other hand,
the dimensions view seems to fold aspects of the evaluation of a stimulus and of
the reaction to the stimulus together. In any case, this chapter is not the place to
go into too much detail on how these levels relate to each other, but it is clear
that it would be a mistake to simply assume that there are three groups of
researchers with wildly incompatible views as to what an emotion is. Clearly,
there is "truth" to all three levels of description, but we need more theoretical
work to clarify how the three relate to each other. Obviously, this seeming
confusion renders the discussion on the usefulness of facial expressions in
applied contexts difficult.
This discussion has intended to show is that the basic emotions view is most
useful in the context of diagnosing emotions from facial actions. The predictions
are specific and are usually mutually exclusive. This also means that the
predictions derived from this view are the easiest to test empirically and, if
necessary, reject. On the other end of the spectrum of specificity of predictions
are dimensional approaches where essentially much of the research comes down
to seeing whether positive and negative affective states differ with regard to
specific changes in facial activity. However, even if the activation of
Corrugator Supercilii is sensitive to the valence of a person's affective state
(e.g., Cacioppo, Petty, Losch, & Kim, 1986), there is the problem that
Corrugator Supercilii movements are involved in so many affective and non-
affective processes (e.g., frowning induced by difficulties in seeing a stimulus as
a means of focusing the eyes) that inferences in very specific instances are
impossible. I have found repeatedly in my own research that the activation of
Zygomaticus Major, the cheek muscle involved in smiling, is usually greater
immediately following unpleasant events than pleasant events in the context of
playing different video games (Kappas & Pecchinenda, 1998). Yet, there are
many studies that attempt to measure affective state using facial
electromyographic recordings at the sites of these two muscles. Of course, this is
not useless, despite the criticism of ambiguity raised here. 1am not arguing that
no inferences can be drawn from changes in the activation of a muscle at a
particular muscle site (or a full-blown pattern of changes at different sites) - the
issue is which inferences can be drawn. In the next section, the problem of
inference in the light of the multiple determination of facial actions will be
discussed.

224
Figure 1. The three different levels 0/ affective organization are not exclusive but they are
related. The basic emotions view, proposing discrete emotional states, allows the most
specific predictions regarding the relationship o//acial activity and state - in contrast,
the dimensions view is the least specific 4

225
PROBLEMS OF INFERENCE DUE TO THE MULTIPLE DETERMINATION
OF FACIAL ACTIONS

As we have seen in the previous section, there are many instances in which there
is no coherence or only limited coherence between specific patterns of facial
activation and underlying affective states. This is not surprising given that facial
behavior as well as other nonverbal behaviors are not only assumed to be
influenced by emotion but also by a host of other factors, including motivations,
social context and cultural conventions (e.g., "display rules", Ekman, 1982;
Ekman & Friesen, 1969; see also Philippot, Feldman, & Coats, 1999, Kappas,
1997). In his M notebook Darwin reflected:
Seeing how ancient these expressions are, it is no wonder that they are so difficult
to conceal. A man !insulted! may forgive his enemy & not wish to strike him, but
he will find it far more difficult to look tranquil. - He may despise a man & say
nothing, but without a most distinct will, he will find it hard to keep his lip from
stiffening over his canine teeth. (M93, in Gruber & Barnett, 1974, p. 283)

It is clear from this quote that Darwin considered that strong emotional states are
usually accompanied by specific facial actions. However, he believed also that
the strong link between expression and underlying psychological state, whether
one wants to call it emotion, action tendency, or social motivation, can be broken
by will. In fact, there is no research that could show contexts in which facial
actions are caused by emotion unconditionally (Kappas, Sherer, & Theriault,
2000). The influence of cultural conventions and social rules on conscious and
voluntary use and modulation of emotional displays was as obvious to Darwin as
it was to his contemporary, Wundt (Kappas, Hess, & Scherer, 1991). While
some expressions are more difficult to produce voluntarily, such as the
"Duchenne smile" (which involves the concurrent actions of the Zygomaticus
Major muscle that pulls the corners of the lip back and up and the Orbicularis
Oculi muscle that is responsible for the crow's feet wrinkles around the eye),
they can be produced with practice by many, ifnot most people.
In addition to involuntary and voluntary influences on the so-called
emotional facial expressions (Warren & Thompson, this volume), there are other
non-affective processes that influence facial behaviors, such as mastication,
speech, or respiration. I have suggested elsewhere the need to consider these
influences as parallel and concurrent source streams that are combined through
coupling, decoupling, and inhibition of decoupling to produce facial output
(Kappas, 1999). Specifically, I believe that contexts can differ in the constraints
they put on the coupling of multiple source streams. This implies that the impact
of voluntary control, or speech, or other influences, can be dampened in certain
occasions - it is theoretically possible that at a microscopic level there might
always be effects of underlying psychological states present, but they are
embedded in the signal in much the same way as physiological changes, such as
changes in heart rate have multiple and interacting determinants.

226
 If we acknowledge that facial activity can be mUltiply determined, it follows
logically that we cannot draw an inference from facial activity on the underlying
psychological state in any specific case. It would only be possible to deduce
from facial behavior the emotional state of an expresser, if these states would
produce unique signatures that could not be produced or masked by the other
factors. However, there do not seem to be any such signatures. To evaluate the
relevant empirical evidence, we have to be clear in distinguishing two underlying
assumptions/hypotheses: I) There is a set of emotional states that can be
differentiated and that all of them, or some possess unique signatures; 2) We can
exclude for each or some of the signatures whether they can be produced by any
of the other factors.
A review of the literature on facial expression reveals that there seems indeed
to be a link between emotion and facial activity (Frank, this volume), but that
inversely there are few if any unique signatures that would allow to use patterns
of facial activation to infer the presence of a specific emotion (Ekman, 1982;
Russell & Fernandez-Dols, 1997).
To summarize the key arguments presented in this chapter, it does not suffice
to show that there are instances in which state X is linked to facial action Y if
there are instances in which either X is present and Y is not, or the inverse. Such
a statement could be misconstrued as denying the important influence of
psychological states on expressions - but this is not the point of contention. The
problem is a logical one of drawing inferences from the presence or absence of a
given facial action (Cacioppo & Tassinary, 1990; Cacioppo, Tassinary, &
Berntson, 2000, Kappas & Hess, 1995). Fever can have many different causes
and no one would diagnose the flu, simply and exclusively based on the presence
of fever. Similarly, we cannot use the presence of a smile, even a Duchenne
smile, to deduce that someone is happy.
The research of Rainer Krause and his colleagues is very informative in this
context (e.g., Krause, Steimer-Krause, Merten, & Burkhard, 1998; Steimer-
Krause, Krause, & Wagner, 1997). In different studies these researchers
measured facial activity in dyadic interactions, typically a discussion on some
political topic. One of the recurrent findings is that "I do not show what I feel,
but what you feel". Is this really surprising? If someone told me that her bicycle
was stolen, I might show an expression of sadness or anger - not necessarily
because there are specific (display) rules that make me show this particular
expression 5 , even if! was not saddened or angered by these news. Instead, there
is an aspect of role taking and reinforcing the relationship within the dyad. From
a functional point of view this is a very important element of underlining the
condition of the current interaction contract. "I care about you. I understand
your pain." In this sense, an analysis of interaction patterns might have
diagnostic value and be more informative than assuming to show unequivocally
that at a particular point in time someone felt a particular emotion. Of course,
many researchers try to discard these social influences on facial expression by

227
trying to create a situation where a participant is alone, facing a stimulus, such as
a slide, or a video. In this rationale, the exclusion of the interaction supposedly
allows one to identify the push of emotion on facial expressions while avoiding
the pull of social context (Kappas, Hess, & Scherer, 1991). However, recent
studies have shown conclusively, that social context can be completely
imaginary and implicit in influencing facial behavior. For example it is
sufficient to believe that your friend is watching the same funny video as you in
a different room to make you smile more than if you thought she did not
(Fridlund, 1991; Hess, Banse, & Kappas, 1995; Jakobs, Manstead, & Fischer,
1999a; 1999b). One conclusion from these findings is that a) trying to eliminate
social context by isolating participants might not work after all and b) maybe it is
misplaced even trying to eliminate social context, if the application of my
research is to take place in a social context itself (e.g., clinical settings, school
settings, evaluation settings, etc.; see also Kappas & Descoteaux in press).
Given that there are these multiple determinants of facial actions we cannot
a-priori identify which (combination of) determinant(s) caused a particular
expression. The problem underlying inference is then a practical one (not a
theoretical one as the confusion regarding the emotion concept itself, as
discussed earlier in this chapter). Cacioppo and his colleagues (Cacioppo,
Tassinary & Berntson, 2000; Cacioppo & Tassinary, 1990; see also Kappas &
Hess, 1995 for an application of this logic in the context of oral communication
of emotion) present a taxonomy of possible relations between elements in the
psychological (If') and the physiological (<1» domains. A one-la-one relationship
consists of an exclusive and unique relationship between a psychological state
and a physiological phenomenon. Only in this case can we conclude from the
presence of the one on the presence of the other, that is the inference works in
both directions. There are numerous variations of these relationships possible,
such as one-la-many, many-la-one, many-la-many, and the absence of
relationship - the null relation. As mentioned before, the relationship between
facial actions and their multiple determinants can be described as being of the
many-la-many type. Here the presence of a set of specfic affective states (If' =
discrete emotions, dimensional shifts, appraisal outcomes) would be indicated by
a set of partially shared facial actions (<1». The interesting aspect of the approach
of Cacioppo and his colleagues is that the classification of a causal relationship
can be taken as an initial step in an incremental research strategy. Specifically,
one-to-many and many-to-many can be simplified by reducing mUltiple
physiological changes to a derived subset <1>'. Such patterns or syndromes may
then be used to uniquely differentiate or discriminate between psychological
states. For example, instead of a single occurrence of an isolated facial action
one would assess (I) its duration, (2) its dynamic development over time, (3) its
co-occurrence with other facial actions, (4) the sequential occurrence of mUltiple
instances of the same action, or (5) sequential occurrences of specific actions, (6)
co-occurrence with other non-facial events, or any combination thereof.

228
 Attempts have been made to develop such patterns to identify markers of
specific states, e.g., in the case of enjoyment, the duration of smiles, concurrent
activation of Zygomaticus Major and Orbicularis Oculi (the Duchenne smile, or
enjoyment smile), synchronization of the activation of both muscles so that they
reach the point of highest activation at the same time, symmetry of Zygomaticus
Major activation, and smooth changes in activation of the involved muscles (e.g.,
Ekman & Friesen, 1982; Frank, Ekman, & Friesen, 1993). Keltner (1997)
showed patterning across channels for embarrassment by adding gaze and head
movements to the smiles. Similarly, there have been attempts involving facial
EMG to use the topographical patterning of the EMG signal (Cacioppo,
Marshall-Gooddell, & Dorfman, 1983) to distinguish spontaneous and posed
smiles (Hess, Kappas, McHugo, Kleck, & Lanzetta, 1989) and actions linked to
appraisals in a video game (Kappas, 1995). However, while all of these attempts
showed a better classification of the state under investigation than the simple
presence of a single action, none have shown the specificity that would be
required to use the syndrome as a unique marker. Yet, these approaches are
promising and merit further research.

Where to go from here?

What then are we to make of studies that try to measure emotions using facial
actions, whether they originate in a developmental, a clinical, or other applied
context? We should take these studies with a grain of sal{ The current chapter
outlines a series of reasons why we should not rely on facial actions as reliable
indicators of emotional state. Obviously, there is no point in arguing whether we
will ever be able to identify unique markers for emotional states, be they discrete
emotions, affective states of positive or negative valence, or for appraisal
outcomes. However, that does not mean that there is no place for an analysis of
nonverbal behavior in applied contexts.
Perhaps, it would be useful to establish probabilistic models that would allow
us to estimate the likelihood of the presence of a particular state based the
presence of a certain action . Together with the convergent evidence of multi-
modal measurements we might be on safer ground than simply insisting, against
our better knowledge, that there is a fixed link between facial expressions and
emotions. It is likely that the context of the situation will have to be taken into
account. It does not matter whether we conceive of the influence of context as
mediated by display rules, a modulation of the inhibition and disinhibition of
mUltiple source streams of facial efferences (Kappas, 1999), or in modulating
social motivations as proposed in the context of behavioral ecology (Fridlund,
1991; 1994). In all cases, we need sound theory guiding empirical research that
can establish the probability of state X given condition Y, where Y is a function
of psychological state and context.

229
 Ideally, we would aim at the establishment of sensitivity, specificity, positive
predictive value, and negative predictive value of facial actions for the
classification of transient affective states (see Iacono, 1991 for a discussion of
the evaluation of psychophysiological assessment techniques in the context of
psychopathology). If facial actions are to be used as a diagnostic tool we have to
apply the same criteria as we would for any mode of evaluation. We should not
be distracted by our commonsense understanding of the relationship of facial
expressions and emotion. We should also not be exclusively focused on the
momentary relationship of affective state and facial actions. Instead, we might
use deviations from typical distributions of the occurrence of specific actions
over time (such as during a one-hour session) and on process variables, such as
the presence, absence, or degree of mirroring behavior, synchronicity, time lag,
and the internal relationship of different nonverbal channels. Finally, we have to
be aware that in interactions there is more at play then a simple chain of cause
and effect units. Due to the multiple feedback processes, be they intrapersonal or
interpersonal, the facial activity of an individual is part of a complex dynamic
system that might be better described as chaotic 7 (see Kappas & Descoteaux, in
press; Messinger, Fogel, & Dickson, 1997). This does not mean that facial
actions are random, but that, under certain circumstances we cannot predict their
occurrence. On the other hand, we might be able to specify systems in a way
that there are particular conditions under which we can predict causal links.
However, we are far from having accumulated the necessary empirical data or
from having formulated theoretical models that are sufficiently specific to be
tested. We should take this state of affairs not as a discouragement, but rather as
a challenge for future research. This research cannot rely solely on laboratory
environments, but requires a close collaboration of those interested in
fundamental questions and of those who want to apply facial behavior in their
applied context. Otherwise, we are doomed to knowing better what facial
expressions cannot tell us then what they can tell us.

AFFILIA TION

Department of Psychology
University of Hull
Email: A.Kappas@hull.ac. uk

NOTES
I This statement is read as "the probability of the presence of a sneer. given the presence of a state of

disgust is I".
2 The issue of self-report in appraisal studies has been an issue for debate. Selt~report, particularly
retrospective questionnaires have been widely used in appraisal research. However, as Kappas and
Pecchinenda (1999) and Kappas (200 I) have pointed out, there is a contradiction in the assumption

230
 that much of appraisal is outside of awareness and the choice of self-report as a dependent measure
(see also Parkinson, 1997).
3 The data was collected in the context of a research course by Yannick Dion and Karine Poitras at
Universite Laval in 1995 and has not been previously published.
4 Note that most modem appraisal theorists propose more than three dimensions of appraisal. This
figure is not intended to suggest that there are only three dimensions to appraisal and that they map
directly to those of the dimensions view. However, these three appraisal dimensions are arguably the
most important ones. The spatial position of the discrete emotion labels is not a real projection of the
dimensions of either of the other two levels of analysis of affective states. One of the anonymous
reviewers pointed out that one could add an additional dimension in the comparison of the three
views that represents "specificity of situational context" ranging from low (Ekman) to high (Russell).
This is certainly an interesting idea and merits further thought.
5 I am not implying that display rules would require conscious awareness regarding which rule would
have to be applied when. I do assume that there are learned conventions that regulate, in certain
situations, the intensity or the quality of the expression we show on our face. The term "rule" as such
is more of a metaphor as it might simply refer to learned contingencies, rather than a "rule book"
(Kappas, 200 I). The point I wanted to raise here is that there is no content prescription attached to
this moment. If there would be a rule it would be "if you care for this person or she is of strategic
value for you try to echo the content of her feelings in your expressions". This social motivation
could again be rephrased simply as a series of learned contingencies. Hence, I do not want the
reader to focus too much on the term "rules" in one way or another but instead on the interactive
aspect offacial display regulation that the example tries to illustrate.
(, One of the anonymous reviewers pointed out that it is true that facial activation does not map self-
reports very well, but that there are also problems with the reliability of self-reports. This is certainly
true and it is only because of the limited space and the specific focus on facial activity that this issue
is not explored further here. I believe that self-reports are not simple read-outs but constructions and
they are influenced by various factors, many of them social. In my own theoretical approach to
emotions, the Dynamic Appraisal Theory of Emotion (Kappas, 200 I) I argue that facial actions,
physiological changes, and self-report should be seen as output systems that are driven by appraisals
and dynamically modulated by a variety of influences ranging from biological to social. There is no
obvious gold standard to diagnose the presence of emotions.
7 The interest in complex feedback systems, nonlinear dynamics, and selt~organizing properties has
recently increased and there are various approaches to the organization of emotions themselves that
are relevant here (e.g., Lewis & Granic, 1999).

REFERENCES
Arnold, M. B. (1960). Emotion and personality. New York: Columbia University Press.
Banse, R., Etter, A., Van Reekum, C. & Scherer, K.R. (1996). Psychophysiological responses to emotion-
antecedent appraisal of critical events in a computer game. Presented at the 36110 Annual Meeting of
the Society for Psychophysiological Research, Vancouver, Canada, 16-20 October 1996.
Bradley, M. M. (2000). Emotion and motivation. In J. T. Cacioppo, L. G. Tassinary, & G. G. Berntson
(Eds.), Handbook ofpsychophysiology (pp. 602-642). Cambridge: Cambridge University Press.
Cacioppo, J. T., Berntson, G. G., Larsen, J. T., Poehlmann, K. M., & Ito, T. A. (2000). The
psychophysiology of emotion. In M. Lewis & J. M. Haviland-Jones (Eds.), Handbook of emotions
(pp. 173-191). New York: Guilford.
Cacioppo, J. T., Marshall-Gooddell, 8., & Dorfman, D. D. (1983). Skeletal muscular patteming:
Topographical analysis of the integrated electromyogram. Psychophysiology, 20,269-283.
Cacioppo, J. T., Petty, R. E., Losch, M. E., & Kim, H. S. (1986). Electromyographic activity over facial
muscle regions can discriminate the valence and intensity of affective reactions. Journal of
Personality and Social Psychology, 50, 260-268.

231
Cacioppo, J. T., & Tassinary, L. G. (1990). Psychophysiology and psychophysiological inference. In J. T.
Cacioppo & L. G. Tassinary (Eds.), Principles of psychophysiology, (pp. 3-33). Cambridge:
Cambridge University Press.
Cacioppo, 1.T., Tassinary, L.G., & Berntson, G.G. (2000). Psychophysiological science. In 1. T,
Cacioppo, L.G. Tassinary, & G.G. Berntson (Eds.), Handbook of psychophysiology (pp. 3-23).
Cambridge: Cambridge University Press.
Cacioppo, J. T., Tassinary, L. G., & Fridlund, A. J. (1990). The skeletomotor system. In J. T. Cacioppo &
L. G. Tassinary (Eds.), Principles of psychophysiology: Physical, social, and inferential elements
(pp. 325-384). New York: Cambridge University Press.
Darwin, C. (1872). The expression of the emotions in man and animals. London: John Murray.
Darwin, C. (1998). The expression of the emotions in man and animals: Definitive edition (Edited by Paul
Ekman). Oxford: Oxford University Press, first edition published 1872.
Davidson, R. 1. (1999). Neuropsychological perspectives on affective styles and their cognitive
consequences. In T. Dalgleish & M. Power (Eds.), Handbook of cognition and emotion (pp. 103-
123). Chichester: Wiley.
Ekman, P. (1982). Emotion in the humanface. Cambridge: Cambridge University Press.
Ekman, P. (1992a). Are there basic emotions? Psychological Review, 99,550-553.
Ekman, P. (1992b). An argument for basic emotions. Cognition and Emotion, 6, 169-200.
Ekman, P. (I 994a). All emotions are basic. In P. Ekman & R. J. Davidson (Eds.), The nature of emotion:
Fundamental questions (pp. 15-19). New York: Oxford University Press.
Ekman, P. (I 994b). Strong evidence for universals in facial expressions: A reply to Russell's mistaken
critique. Psychological Bulletin, 115,268-287.
Ekman, P. (1995). Silvan Tomkins and facial expression. In E. V. Demos (Ed.), Exploring affect: The
selected writings ofSilvan S Tomkins (pp. 209-214). Cambridge: Cambridge University Press.
Ekman, P., & Friesen, W. V. (1969). The repertoire of nonverbal behavior: Categories, origins, usage, and
coding. Semiotica, 1,49-98.
Ekman, P., & Friesen, W. V. (1978). The Facial Action Coding System: A technique for the measurement
o/facial movement. Palo Alto: Consulting Psychologists Press.
Ekman, P., & Friesen, W. V. (1982). Felt, false, and miserable smiles. Journal of Nonverbal Behavior, 6,
238-252.
Fernandez-Dols, J. M., Sanchez, F., Carrera, P., & Ruiz-Belda, M. A. (1997). Are spontaneous
expressions and emotions linked? An experimental test of coherence. Journal of Nonverbal Behavior,
31,163-177.
Frank, M.G. (2000). Getting to know your patient: how facial expression can help reveal true emotion. In
M. Katsikitis (Ed.), The human face: Measurement and meaning (pp. 255-283). Dordrecht: Kluwer
Academic Publishers.
Frank, M. G., Ekman, P., & Friesen, W. V. (1993). Behavioral markers and recognizability of the smile of
enjoyment. Journal of Personality and Social Psychology, 64, 83-93.
Fridlund, A. 1. (1991). The sociality of solitary smiling: Potentiation by an implicit audience. Journal of
Personality and Social Psychology, 60,229-240.
Fridlund, A. (1994). Humanfacial expression: An evolutionary view. San Diego: Academic Press.
Gruber, H.E., & Bamett, H.P. (1974). Darwin on man: A psychological study of scientific creativity. New
York: Dutton.
Hess, U., Banse, R., & Kappas, A. (1995). The intensity of facial expression is determined by underlying
affective state and social situation. Journal of Personality and Social Psychology, 69, 280-288.
Hess, U., Kappas, A., McHugo, G. J., Kleck, R. E., & Lanzetta, J. T. (1989). An analysis of the encoding
and decoding of spontaneous and posed smiles: The use of facial electromyography. Journal (jf
Nonverbal Behavior, 13, 121-137.
Iacono, W. G. (1991). Psychophysiological assessment of psychopathology. Psychological Assessment: A
Journal of Consulting and Clinical Psychology, 3,309-320.
Izard, C. E. (1971). Theface of emotion. New York, NY: Appleton-Century-Crofts.
Izard, C. E. (1997). Emotions and facial expressions: A perspective from Differential Emotions Theory.
In J. A. Russell & J. M. Fernandez-Dols (Eds.), The psychology of facial expression (pp. 57-77).
Cambridge: Cambridge University Press.

232
Jakobs, E., Manstead, A. S. R., & Fischer, A. H. (1999a). Social motives and emotional feelings as
determinants of facial displays: The case of smiling. Personality and Social Psychology Bulletin, 25,
424-435.
Jakobs, E., Manstead, A. S. R., & Fischer, A. H. (\999b). Social motives, emotional feelings, and
smiling. Cognition and Emotion, 13,321-345.
Kaiser, S. (2002). Facial expressions as indicators of "functional" and "dysfunctional" emotional
processes. In M. Katsikitis, (Ed.), The human face: Measurement and meaning (pp. 235-253).
Dordrecht: Kluwer Academic Publishers.
Kaiser, S., & Wehrle, T. (200\). Facial expressions as indicator of appraisal processes. In K.R. Scherer,
A. Schorr, & T. Johnstone (Eds.), Appraisal processes in emotion: Theory, methods, research (pp.
285-300). New York: Oxford University Press.
Kappas, A. (1995). Event-related facial activity: In pursuit of componential approaches to the expression
of emotional reactions. Psychophysiology, 32, S6.
Kappas, A. (1997). Looking for a DATE: Towards a dynamic appraisal theory of emotion. Paper
presented at the Fifth Geneva Emotion Week, April 19-23, 1997, Geneva, Switzerland.
Kappas, A. (1999). Reconceptualizing the irifluence of social context on facial displays: Towards a new
view of display rules. Paper presented at the Eighth European Conference on Facial Expression -
Measurement and Meaning, September 27-30, 1999, SaarbrUcken, Germany.
Kappas, A. (2001). A metaphor is a metaphor is a metaphor: Exorcising the homunculus from appraisal
theory. In K. R. Scherer, A. Schorr, & T. Johnstone (Eds.), Appraisal processes in emotion: Theory,
methods, research (pp. 157-172). New York, N.Y.: Oxford University Press.
Kappas, A., Bherer, F., & Theriault, M. (2000). Inhibiting facial expressions: Limitations to the voluntary
control offacial expressions of emotion. Motivation and Emotion, 24,259-270.
Kappas, A., & Descoteaux, J. (in press). Of butterflies and roaring thunder: Nonverbal communication in
interaction and regulation of emotion. In P. Philippot, E. J. Coats, & R. S. Feldman (Eds.), Nonverbal
behavior in clinical settings. New York: Oxford University Press.
Kappas, A., & Hess, U. (1995). Nonverbal aspects of oral communication. In U. M. Quasthoff (Ed.),
Aspects of oral communication (pp. 169-180). Berlin: DeGruyter.
Kappas, A., Hess, U., & Scherer, K. R. (1991). Voice and emotion. In R. S. Feldman & B. Rime (Eds.),
Fundamentals of nonverbal behavior (pp. 200-238). Cambridge: Cambridge University Press.
Kappas, A., & Pecchinenda, A. (1996). The effect of sociality on event-related facial activity in the
context of a video game. Psychophysiology, 33, S48.
Kappas, A., & Peechinenda, A. (1998). Zygomaticus Major activity is not a selective indicator if positive
affective state in ongoing interactive tasks. Psychophysiology, 35, S44.
Kappas, A., & Pecchinenda, A. (1999). Don't wait for the monsters to get you: A video game task to
manipulate appraisals in real time. Cognition and Emotion, 13, 119-124.
Keltner, D. (1997). Signs of appeasement: Evidence for the distinct displays of embarrassment,
amusement, and shame. In P. Ekman & E. Rosenberg (Eds.), What the face reveals: Basic and
applied studies of spontaneous expression using the Facial Action Coding System (FACS) (pp. 133-
157). New York: Oxford University Press.
Keltner, D., & Ekman, P. (2000). Facial expression of emotion. In M. Lewis & J. M. Haviland-Jones
(Eds.), Handbook of emotions (pp. 236-249). New York: Guilford.
Krause, R., Steimer-Krause, E., Merten, 1., & Burkhard, U. (1998). Dyadic interaction regulation,
emotion, and psychopathology. In W. F. J. Flaek & J. D. Laird (Eds.), Emotions in psychopathology:
Theory and research. Series in affective science (pp. 70-80). New York: Oxford University Press.
Leventhal, H., & Scherer, K. (1987). The relationship of emotion to cognition: A functional approach to a
semantic controversy. Cognition and Emotion, I, 3-28.
Lewis, M. D., & Granic, I. (1999). Self-organization of cognition-emotion interactions. In T. Dalgleish &
M. J. Power (Eds.), Handbook of cognition and emotion (pp. 683-701). Chichester: Wiley.
Messinger, D. S., Fogel, A., & Dickson, K. L. (1997). A dynamic systems approach to infant facial
action. In J. A. Russell & J-M. Femfmdez-Dols (Eds.), The psychology offacial expression (pp. 205-
226). Cambridge: Cambridge University Press.
Ortony, A., Clore, G. L., & Collins, A. (1988). The cognitive structure of emotions. Cambridge:
Cambridge University Press.

233
Parkinson, B. (1997). Untangling the appraisal-emotion connection. Personality and Social Psychology
Review, 1,62-79.
Philippot, P., Feldman, R. S., & Coats, E. 1. (1999). The social context of nonverbal behavior.
Cambridge: Cambridge University Press.
Rosenberg, E. L. (1997). Emotions as unified responses. In P. Ekman & E. L. Rosenberg (Eds.), What the
face reveals (pp. 86-88). Oxford: Oxford University Press.
Rosenberg, E. L., & Ekman, P. (1994). Coherence between expressive and experiential systems in
emotion. Cognition arid Emotion, 8,201-229.
Ruch, W. (1995). Will the real relationship between facial expression and affective experience please
stand up: The case of exhilaration. Cognition and Emotion, 9,33-58.
Russell, J. A. (1997). Reading emotions from and into faces: Resurrecting a dimensional-contextual
perspective. In J. A. Russell & J. M. Fernandez-Dols (Eds.), The psychology offacial expression (pp.
295-320). Cambridge: Cambridge University Press.
Russell, J. A., & Fernandez-Dols, J. M. (1997). The psychology of facial expression. Cambridge:
Cambridge University Press.
Scherer, K. R. (1992). What does a facial expression express? In K. T. Strongman (Ed.), International
review of studies on emotion (pp. \39-165). New York: Wiley.
Scherer, K. R. (1993). Neuroscience projections to current debates in emotion psychology. Cognition and
Emotion, 7, 1-41.
Scherer, K.R. (2001). Appraisal considered as a process of multilevel sequential checking. In K. R.
Scherer, A. Schorr, & T. Johnstone (Eds.), Appraisal processes in emotion: Theory, methods,
research (pp. 92-120). New York: Oxford University Press.
Scherer, K.R., Schorr, A. & Johnstone, T. (2001) Appraisal processes in emotion: Theory, methods,
research. New York: Oxford University Press.
Smith, C. A. (1989). Dimensions of appraisal and physiological response in emotion. Journal of
Personality and Social Psychology, 56,339-353.
Smith, C. A., Griner, L. A., Kirby, L. D., & Scott, H. S. (1996). Toward a process model of appraisal in
emotion. In N. H. Frijda (Ed.), Proceedings of the Ninth Conference of the International Society for
Research on Emotions, Toronto, Canada (pp. 101-105). Toronto, Ontario: International Society for
Research on Emotions.
Smith, C.A. & Kirby, L.D. (2001). Towards delivering on the promise of appraisal theory. In
K.R.Scherer, A. Schorr, & TJohnstone (Eds.), Appraisal processes in emotion: Theory, methods,
research (pp. 121-\38). New York: Oxford University Press.
Smith, C. A., & Scott, H. S. (1997). A componential approach to the meaning of facial expressions. In J.
A. Russell & J-M. Fernandez-Dols (Eds.), New directions in the study offacial expression (pp. 229-
254). Cambridge: Cambridge University Press.
Steimer-Krause, E., Krause, R., & Wagner, G. (1997). Interaction regulations used by schizophrenic and
psychosomatic patients: Studies on facial behavior in dyadic interactions. In P. Ekman & E. L.
Rosenberg (Eds.), What the face reveals: Basic and applied studies of spontaneous expression using
the Facial Action Coding System (FACS) (pp. 361-385). New York: Oxford University Press.
Van Reekum, C.M., & Scherer, K.R. (1997). Levels of processing for emotion-antecedent appraisal. In G.
Matthews (Ed.), Cognitive science perspectives on personality and emotion (pp. 259-300).
Amsterdam: Elsevier.
Warren, J., & Thompson, P. (2000). Neurology of human facial expression. In M. Katsikitis (Ed.), The
human face. Measurement and meaning (pp. 63-96). Dordrecht: Kluwer Academic Publishers.
Winton, W. M. (1986). The role of facial response in self-reports of emotion: A critique of Laird.
Journal o/Personality and Social Psychology, 50, 808-812.

234
 SUSANNE KAISER

CHAPTER 12

FACIAL EXPRESSIONS AS INDICATORS OF

"FUNCTIONAL" AND "DYSFUNCTIONAL"
EMOTIONAL PROCESSES

INTRODUCTION

The function of emotions can be described with respect to two interrelated

regulatory processes in which emotions are involved: the intra-individual
regulation of thoughts and behaviour and the inter-individual regulation in social
interactions. Major issues concerning the intra-individual process are the
relation between emotion and cognition, particularly emotion-antecedent
appraisal, and the interactions between different emotion components, in
particular the role of subjective experience or feeling state. The inter-individual
process can be described in terms of how emotions and emotion expression
regulate interpersonal interaction and vice versa, i.e. how emotions are regulated
during interactions. Facial expressions can be seen as an interface between these
two regulatory processes or systems. They are observable reflections of the
internal regulation process and simultaneously vehicles for the regulation of the
interaction.
Starting from a brief description of three theoretical approaches, only two of
these that have led to explicit predictions for emotion-specific facial expression
patterns will be discussed in more detail. Further, this discussion focuses on the
problems confronted when studying facial expressions in spontaneous emotional
interactions. Finally, an empirical paradigm for analysing and synthesizing
emotional facial expressions will be presented followed by a discussion of some
results with respect to current emotion theory and possible implications for
clinical research and treatment.

235
 FACIAL EXPRESSION AND EMOTION: THREE THEORETICAL
APPROACHES

There is a long tradition in emotion psychology of examining facial expressions

as an observable indicator of unobservable emotional processes. Among
emotion theories proposing implicit or explicit predictions for emotion-specific
facial expression patterns, three positions can be distinguished. The first
approach follows the tradition of discrete emotion theories and is represented by
Ekman, Izard, and their respective collaborators (e.g. Ekman, 1992; Izard, 1991).
The second approach has been suggested in the context of appraisal theories of
emotion (e.g. Roseman, Wiest & Swartz, 1994; Scherer, 1984, 1992; Smith &
Ellsworth, 1985; Wehrle, Kaiser, Schmidt & Scherer, 2000; Kaiser & Wehrle,
2001). Third, dimensional emotion theories, following the tradition of Osgood
(I966) and Schlosberg (1952), treat emotions as fundamentally similar in most
respects, differing only in terms of one or more dimensions, the most common
ones being arousal and pleasantness (e.g. Russell, 1997). A more
comprehensive comparison of these theories and their respective strengths and
limitations for the interpretation of facial expressions in applied contexts can be
found in Kappas (this volume). This chapter concentrates on the differences
between the discrete emotion view and the appraisal view.

Discrete emotion theory andfacial expression: Basic emotions

Discrete emotion theorists have studied facial expression as the "via regia" to the
understanding of emotions for many years. Most of their research concerning
the universality of the so-called basic emotions is based on studies about facial
expressions. These theories claim that there are only a limited number of
fundamental or basic emotions and that for each of them there exists a
prototypical, innate, and universal expression pattern. In this tradition, a process
of blending or mixing the basic expression patterns explains the variability of
emotion expressions, commonly observed. According to Ekman (1997), an
interpretation of facial expressions must rely on the postulated configurations
and not on single facial actions.
There is considerable evidence (reviewed in Ekman, Friesen, & Ellsworth,
1982) indicating distinct prototypical facial signals that can be reliably
recognized across a variety of cultures as corresponding to the emotions of
happiness, sadness, surprise, disgust, anger, and fear. These patterns have been
found in studies using photographs of posed facial expressions. However, these
findings have not enabled researchers to interpret facial expressions as
unambiguous indicators of emotions in spontaneous interactions (for reviews see
Philippot, Feldman, & Coats, 1999; Russell & Fernandez-Dols, 1997). The task
of analysing the ongoing facial behaviour in dynamically changing emotional

236
episodes is obviously more complex than linking a static emotional expression
to a verbal label.

Componential appraisal theory andfacial expression

Appraisal theorists following a componential approach, as proposed by
Ellsworth and Smith (1985), Frijda (1986), Roseman et al. (1994), or Scherer
(1984), share the assumption that a) emotions are elicited by a cognitive
evaluation (appraisal) of antecedent situations and events and that b) the
patterning of the reactions in the different response components (physiology,
expression, action tendencies, feeling) is directly shaped by the outcome of this
evaluation process. For these appraisal theorists, the complexity and variability
of different emotional feelings can be explained without resorting to a notion of
basic emotions. Scherer argues that there are a large number of highly
differentiated emotional states, of which the current emotion labels capture only
clusters or central tendencies of regularly recurring ones, referred to as modal
emotions (Scherer, 1994). In line with this reasoning, facial expressions are
analysed as indicators of appraisal processes in addition to or as an alternative to
verbal reports. Facial expressions are not seen as the "readout" of motor
programs but as indicators of mental states and evaluation processes. In contrast
to discrete emotion theorists, they claim that single components of facial patterns
do have a meaning and that this meaning can be explained as manifestations of
specific appraisal outcomes.
Several appraisal theorists have made concrete suggestions concerning
possible links between specific appraisal dimensions and specific facial actions
(Frijda & Tcherkassof, 1997; Kaiser & Scherer, 1998; Kaiser & Wehrle, 200 I;
Smith & Scott, 1997; Wehrle et aI., 2000). An appraisal-based approach for
analysing and synthesizing facial expressions that is based on Scherer's
component process model of emotion is presented here (Scherer, 1984). Using
the most recent version of FACS (Ekman & Friesen, 1978), Wehrle, Kaiser,
Schmidt, and Scherer (2000) have extended and refined Scherer's original
predictions linking facial actions to the postulated appraisal checks (Scherer,
1992). The component process model posits relatively few basic criteria and
assumes sequential processing of these criteria in the appraisal process. The
major "stimulus evaluation checks" (SECs) can be categorized into five major
classes: 1) the relevance of objects or events, including the novelty or familiarity
of an event, its intrinsic pleasantness, and its significance for the individual's
needs or goals, 2) the implications of the event for the individual's needs or
goals, 3) the individual's ability to influence or cope with the consequences of
the event, including the evaluation of who caused the event (agency), and 4) the
compatibility of the event with social or personal standards, norms, or values.
As an example, Figure 1 shows the postulated facial expressions (action units)
representing the appraisal profile for hot anger.

237
 ',OVCllY I1tGH ,OAL OBSTRUCTIVE

1+2+5 4+7 4 + 7 + 10 + 17 + 24 AllleftAUs

Figure I. Predictions for the appraisal patterns and the related facial actions
(synthesized in terms of action unit combinations) for hot anger as published in Wehrle et
al. (2000). From left to right, the pictures illustrate the sequential accumulation of
appraisal-specific faCial action unit combinations resulting in ajinal pattern, as
postulated by Scherer (I 992).

FACIAL EXPRESSIONS IN SOCIAL INTERACTIONS

In spontaneous interactions, facial behaviour accompanies emotional episodes as

they unfold, and changes in facial configurations can occur very rapidly. Many
of the facial patterns that occur during an interaction are not 'true', spontaneous
expressions of internal emotional processes. Individuals often use expressive
behaviours more or less consciously in order to achieve a social goal, for
example to obtain attention or support. Here, the subjective feeling of the person
and his or her facial expression may not necessarily correspond (see also
Kappas, this volume). A lack of correspondence between feeling and expression
can also be the result of expression-management processes serving self-
presentation (Goffman, 1959) and expression-control processes demanded by
socio-cultural norms, such as "display rules" (Ekman & Friesen, 1969; Gross,
1998; Gross & Levenson, 1997).
Expression control is not only important in terms of an inter-individual
regulation but also as an intra-psychic coping strategy for handling
overpowering affects. There appear to be rules for expression control in all
cultures, not only to reduce interactive conflicts but also as an implicit tool for
teaching children how to control "affective bursts" (e.g. Malatesta & Haviland,
1985). Caused by these different control and regulation processes, facial
indicators of emotional processes are often very subtle (e.g. the role of "micro
momentary expressions" and "nonverbal leakage" as discussed by Ekman and
Friesen, 1975; Izard, 1994).
Furthermore, facial expressions also have non-emotional, communicative
functions (Ekman, 1979; Ekman & Friesen, 1969; Fridlund, 1994; Manstead,
Fischer, & Jakobs, 1999; Russell & Fermindez-Do1s, 1997). A smile or a frown,
for instance, can have different meanings. It can be a speech-regulation signal

238
 (e.g. a back-channel signal), a speech-related signal (illustrator), a means for
signalling relationship (e.g. when a couple is discussing a controversial topic, a
smile can indicate that although they disagree on the topic there is no "danger"
for the relationship), an indicator for cognitive processes (e.g. frowning while
concentrating and attending to a problem, or when a difficulty is encountered in
a task), or an indicator for an emotion (affect display). Affect displays that
occur during an interaction can refer to the interaction partner (e.g. becoming
angry with the other) but it can also refer to other persons or themes the
interaction partners are talking about (e.g. sharing the anger about something;
see also Merten, this volume).
Given the multi-functionality of facial behaviour and the swiftness of
expression changes, research paradigms for studying emotions and facial
expressions should fulfil the following requirements:
1. Approaches to measure facial expressions objectively and on a micro-analytic
level are indispensable. Anatomically based coding systems like the Facial
Action Coding System (FACS; Ekman & Friesen, 1978) and other measurement
systems described in this volume (see Katsikitis, this volume) lend themselves to
this purpose. Only if coding is independent of prior assumptions about
prototypical emotion expressions we can test and compare different hypotheses
about linking facial expression to emotions.
2. The current, concrete meaning of a facial expression can only be determined
within the whole temporal and situational context. In everyday interactions, we
know the context and we can use all information that is available to interpret the
facial expression of another person. Therefore, facial expressions and emotions
should be studied in an interactive context.
Coding procedures that fulfil both requirements will be referred to here as
situated coding procedures I. Situated coding procedures are anatomically based,
objective procedures that are part of a given experimental setting. Besides facial
expression data, the emotion eliciting experimental context as well as additional
behavioural records (e.g. gestures, verbal report) is systematically coded and this
data is used for the interpretation of the facial behaviour (for more details see
Kaiser & Wehrle, 2000, and next section). This is in line with Kappas' (this
volume) conclusions, stressing the utility of multi-modal measurements and the
important role of the situational context.

FACIAL EXPRESSIONS AND EMOTION IN HUMAN-COMPUTER

INTERACTIONS

In spontaneous interactions, facial behaviour accompanies emotional episodes as

they unfold, and changes in facial configurations can occur very rapidly. In the
preceding section, methodological and theoretical problems were considered
when studying the process of emotional interactions and its reflection in facial
activity in interactive settings.

239
 This chapter highlights the use of human-computer interactions and
interactive computer games in order to study the dynamics and the interactive
nature of emotional episodes. The experimental setting allows the study of the
dynamics of emotional episodes in a more interactive manner than is usually
done in classical experimental settings 2 . One reason why traditional emotion
induction techniques often trigger only rather weak and not clearly defined
emotional states might be that participants are not really involved and/or that
they have to verbalize their emotional state instead of reacting to an emotion
eliciting event or situation. This is diametrically opposed to one of the most
adaptive functions of human emotionality, i.e., to guide a person's behaviour in
situations that are important for his or her goals and needs and that require
immediate cognitive and behavioural responses. In real life situations, it is often
not possible to develop "the perfect solution" to a problem. One cannot consider
every aspect of imaginable decisions or reactions, e.g. all possible short and long
term consequences. Yet, in many cases, emotions help to find good solutions.
This is referred to here as emotional problem solving (Kaiser & Wehrle, 1996).
Emotional problem solving is not a yes or no decision making but a process that
unfolds in emotional episodes. Decisions are adapted and changed according to
the dynamic changes in the external environment and according to changes
caused by internal processes, concerning for instance memory, motives, and
values. Following Lazarus' original conception, an emotional episode can be
described as a process of primary appraisal (the subjectively estimated
significance of an event for one's well being), secondary appraisal (the
subjectively estimated ability to cope with the consequences of an event),
coping, and reappraisal in a transactional interaction (Lazarus, 1966).
As suggested by Leventhal and Scherer (1987), these appraisal processes can
occur at different levels of processing and are often very fast and automatic.
Appraisal processes occurring on the sensory-motor or schematic level are rarely
or only with great difficulty accessible through verbalization. One reason for
analysing facial expressions in emotional interactions is the hope that these
processes might be accessible or indicated by facial expressions. Another reason
for analysing facial expressions in experimental emotion research is that these
are naturally accompanying an emotional episode, whereas asking participants
about their feeling interrupts and changes the ongoing process.

Experimental setup, data collection, data analysis, and modelling

The Geneva Appraisal Manipulation Environment (GAME; Wehrle 1996/2000)

is a tool for generating experimental computer games that translate
psychological postulates into specific micro-world scenarios (for details about
theoretical and technical embedding of GAME see Kaiser & Wehrle, 1996,
2001; Kaiser, Wehrle & Schmidt, \998). GAME allows automatic data
registration of the dynamic game progress and the participants' actions and

240
automatic questionnaires. For example, participants' evaluations of specific,
theoretically designed, emotion eliciting situations are assessed by means of
pop-up screens (which appear after the completion of each game level)
corresponding to 18 questions referring to Scherer's appraisal dimensions
(SEes; see also Scherer, 1984). While playing the experimental game,
participants are videotaped and these tape recordings allow an automatic
analysis of the participant's facial behaviour with the Facial Expression Analysis
Tool (FEAT; Wehrle, 199211996; Kaiser & Wehrle, 1992). These facial data
can be automatically matched with the corresponding game data (using the
vertical time code as a reference for both kinds of data). In this way, the
computer game provides a relatively small but complete context for the
interpretation of the internal emotional and cognitive regulatory processes. This
is an important point, because what the subject feels and what a certain facial
expression means is often quite context-specific.
The behaviour of the "virtual" interaction partners in the game is to a large
degree determined by the experimenter when defining the experimental game
parameters. This approach allows the researcher to create theoretically based
scenarios for specific emotions and to confront all participants with the same
events. However, those events are embedded within a larger context of not
completely determined individual paths, since the participants' behaviour and
their decisions do influence and change the constellation of the ongoing game to
a certain degree.
For example, although at some point all participants are confronted with a
level that is much faster than all levels before (increased speed episode), the
subjective appraisal of their potential to cope with this situation will determine
the emotion of the subject. One aspect of the individual coping potential is the
number of adequate tools at hand. This number depends partly on the
participants' game history, e.g. how many tools they have collected and how
many tools they kept. Since all the information necessary to reconstruct the
complete situational context is registered, each individual path can be followed.
Thus, a more systematic analysis can be carried out on the sources for
differences in individual reactions to a specific event in terms of behaviour
(attack versus avoidance), appraisal (pleasant versus unpleasant, conducive
versus obstruct), and reported feeling Goy, worry, etc).

241
 Experimental Setup Data Collection Data Analysis Modeling

GAME ( "neial Expression) FEAT

II FACE
II GATF:

CQuestionnaire nata J !DF:A

I I i\:\l\1E

C Gllme Behavior)
I TRACE Ic Appraisal Theory )

Figure 2. The components of our experimental setup (GAME), including computerized

tools for automatic data collection and data analysis, as well as tools for synthetic
modelling and simulations.

Figure 2 shows the experimental set-up (GAME) as well as the computerized

tools for the automatic data collection and data analysis:
The Facial Expression Analysis Tool (FEAT, Wehrle, 1992/1996)) is a
connectionist expert system that uses fuzzy rules, acquired from a FACS
expert, to automatically measure facial expressions. This FACS expertise is
automatically transformed into a network structure. The resulting network is
then able to do the classification task, using FACS as the coding language
(for more details see Kaiser & Wehrle, 1992, 200 I). With FEAT we can
precisely analyse the dynamics of facial behaviour, including intensity (on 5
levels from a to e) and asymmetries 3 •
The Interactive Data Elicitation and Analysis tool (IDEA, Wehrle, 1996)
provides the possibility to analyse multimedia behavioural records of an
experimental session and to add new data to the behavioural records.
Behavioural records include all data registered during an experimental
session, like videotaped data and data automatically registered in an
experiment protocol. Topological Reconstruction and Computational
Evaluation of Situations or TRACE (Wehrle, 1999) can automatically re-
construct and analyse a played game in terms of topological measures (e.g.
tactical constellations), critical episodes, urgency, goal multiplicity, etc.
The ensemble of these components constitutes a situated coding procedure as
defined above. In this case, the FEAT coding procedure is an integral part of the
experimental setting (GAME). The interpretation of the automatically coded
facial behaviour is based on the simultaneously registered and automatically
analysed contextual and behavioural records, using IDEA and/or TRACE.

242
 As can be seen in Figure 2, we complement our experimental studies with
computer simulation and synthetic modelling:
The Facial Action Composing Environment (FACE, Wehrle 199511999) is
a tool for creating animated 3D facial expressions in real-time, including
head- and eye- movements. The contours of the face are represented with
splines, as are the prominent features of the face such as eyebrows and lips,
but also wrinkles and furrows (see Figures I and 4). The repertoire of
facial expressions for the animation is defined on the basis of FACS (for
more details see Kaiser & Wehrle, 2001; Wehrle et aI., 2000).
The Geneva Appraisal Theory Environment (GATE, Wehrle, 1995) is a
tool that allows the simulation of different appraisal theories as black box
models. This tool enables one to refine or change theoretical propositions
incrementally, and it provides immediate feedback on the outcomes (for
more details see Wehrle & Scherer, 1996,2001). It has been conceived in
such a way that not only verbal labels, i.e. decisions on categorical
classifications, represent possible outcomes but also nonverbal response
modalities such as facial expression (with the possibility for extensions into
vocal expression and physiological patterning).
The Autonomous Agent Modelling Environment (AAME, Wehrle, 1993) is
a simulation environment for process modelling. The objective is to
explore psychological and Cognitive Science theories of situated,
autonomous agents and the dynamics of system- environment interactions
(for more details see Wehrle, I 994a, I 994b; Wehrle, 2001; Wehrle &
Kaiser, 2001).
One motive for complementing the analysis of spontaneous facial
expressions with synthetic approaches results from the fact that there are only a
few action units that occur frequently in spontaneous social interactions (among
the most frequents are illustrators like AU 1 + AU2, and social smiles AU 12).
Other action units, especially those expressing negative emotions, like fear,
anger or sadness, occur quite rarely (for more details see e.g. Ellgring, 1989;
Merten, 1997; Steimer-Krause, Krause, & Wagner, 1989). Given the
importance of interpreting these rarely occurring action units in the context of
emotional interactions and also emotional disorders, judgment studies using
synthetic stimuli can help us to assess how rarely occurring action units are
perceived. Another problem we confront when interpreting single action units is
caused by the fact that the meaning of a single action unit seems to depend on
(and/or can be influenced by) other co-occurring, preceding or subsequent action
units and on the temporal unfolding (onset, apex, offset). Since FACE enables
us to systematically manipulate static and dynamic configurations of facial
actions we can methodologically study these complex interactions.
Furthermore, synthetic approaches have the advantage that the synthetic
stimuli can be created on the basis of theoretical models. This allows one to
compare and test even subtle differences between different theoretical models

243
with respect to the dynamics and the variability of facial expressions. First
results from the studies on facial expression synthesis using FACE suggest that
participants perceive the synthetic images and animations in a similar fashion to
photographs of real facial expressions. In addition, using FACE for studying the
effect of static versus dynamic presentation of facial expressions, it was shown
that dynamic presentation increases overall recognition accuracy and reduces
confusion (for more details see Wehrle et aI., 2000).

Results illustrating the relation between facial expression and appraisal

Only present examples of current research will be presented in this chapter to

illustrate the views on facial expression described here (more details are
published in Kaiser & Wehrle, 1996,2001; Kaiser, Wehrle, & Edwards, 1994;
Kaiser, Wehrle, & Schmidt, 1998; Wehrle & Kaiser, 2000; Wehrle & Scherer,
2001). The results show that the theoretically designed emotion eliciting
situations differed from one another in the obtained appraisal profiles. Also, the
participants' sUbjective appraisals matched closely the objective characteristics
of the game situations.
With respect to possible interpretations of the meaning and function of the
facial expressions shown during the emotion eliciting episodes, two theoretical
positions were tested, i.e. a basic-emotion approach and an appraisal-based
approach as discussed previously. The analyses showed that the prototypical
patterns described by Ekman and Friesen (1978) in the Emotion Prediction
Tables of the FACS manual occur quite rarely. This result is supported by
studies analysing spontaneous facial expressions in everyday life interactions
(e.g. Ellgring, 1989, Banninger-Huber & Widmer, 1996; Philippot, Feldman, &
Coats, 1999; Russell & Fermlndez-Dols, 1997). With respect to the predictive
power of the two tested theoretical positions, no significant differences were
found. Still, the results indicate that the appraisal approach does better than the
prototypic pattern approach in explaining the occurrence of action units. The
occurrence of AU5 (upper eyelid raiser), for example, cannot be predicted on the
basis of a basic emotion model. However, with the appraisal-based model tested
here, AU5 can be predicted by an appraisal factor called "predictability",
combining novelty, suddenness, outcome probability, and expectation (for more
details see Kaiser, Wehrle & Schmidt, 1998; Schmidt, 1998). Obviously,
predictions of current appraisal models concerning the link between appraisal
outcome and facial expressions only consider spontaneous facial behaviour that
is determined by push factors. As mentioned before, such raw emotional
expressions occur only rarely in emotional interactions thus models and methods
have been developed to take pull factors and control processes into account (see
also Kappas, this volume). Furthermore, the existing predictions are still limited
to linking appraisal profiles to specific emotional states in terms of a verbal
label. In reality, emotion is a process with constantly varying inputs and

244
 reactions and this is equally true for emotional facial expressions. Given the
process character of emotional expressions, what is needed is a more precise
specification of pertinent details and the elaboration of process models in a strict
sense. Process models have to specify the quantitative aspects and the dynamics
of the generally postulated continuous process of primary and secondary
appraisal, coping, and reappraisal.
Detailed analyses of the subjective appraisal profiles allow one to
systematically determine why some participants report other emotions than the
target emotion, which has been reported by the majority of the participants. For
example, when analysing the above-mentioned increased speed episode, it was
found that 73% of the participants reported relief or happiness when AMIGO (an
animated agent in the game, which is generally supporting and helping the
player) reduced speed again. However, 27% of the participants reported anger
or disappointment. The analysis of the appraisal profiles showed that those
participants who reacted with a negative emotion to the reduction had evaluated
the increased speed as being pleasant (versus unpleasant) and as being less
relevant for their goals than participants that reacted with relief. Furthermore,
the automatic data protocol allowed one to predict participants' emotional
reaction on the basis of how successfully they coped with the high-speed
episode.
The evaluation of situational appraisal profiles allows one to differentiate
between different types of identically labelled emotions. For example, at least
three types of anger can be distinguished consistently: (I) being angry as a
reaction to an unfair event but without blaming anybody, (2) being angry and
blaming somebody else for having caused the event on purpose, and (3) being
angry and blaming the other as well as oneself.
Figure 3 shows an example of anger type I. In this situation, the participant
reacts to the above mentioned speed reduction and she reports "anger". As can
be seen in the appraisal profile, the participant evaluates the situation as very
sudden, very new, very unpleasant, and as not at all expected. Additionally, she
thinks that although the situation was difficult to control - she had enough power
to handle it, that she can easily adjust to its consequences, and that her behaviour
was adequate (referred to as Self in Figure 3c). She does not blame anybody or
circumstances, however she evaluates the situation as being very "unfair"
(referred to as Norm (-) in Figure 3c).
As can bee seen in Figure 3, she reacts by raising her eyebrows (AU 1 +
AU2), only. Within an emotion-based approach, this action unit combination
could not be linked to anger. When we look at the dynamics of the facial action,
we see that a) the change from AU4 (she frowns while reading AMIGO's
message) and the innervation of AUI and AU2 occurs within two frames (0.08
seconds).
The duration of AU 1 and AU2, however, is rather long, i.e. 2.04 seconds.
The mean duration of AU I and AU2 in these studies is 1.04 seconds. Whereas

245
the results show that "short" innervation of AUt and AU2 are linked to the
appraisal dimension of unexpectedness, in the case of the example shown in

042 : 3 & :00 ..

042 : 3 '1 :00

0'1.2 :38 : 00 ..
,I ,
9 1. 1. 2 1.2 1S 9
AU's: 1. 4 26 6 11 20 1.4 ., 1.5 1.2 10
o 2 4 4 1.5 17 U

Cognitive Approl .. 1 of Subjecl21n Situation: Speed Reduction

Reported Emotion: Anger

Figure 3a-c. A sequence offacial reactions that occurs in a situation in which AMIGO
reduces speed (for more details see text). Part a) of the Figure shows still pictures of the
subject's face. Part b) shows the results of the automatic FEAT coding, and part c) shows
the participant's evaluation of this situation in terms ofScherer 's appraisal dimensions
(SEes; on Likert scales from I to 5).

Figure 3b. The distribution ofAction Units over a period of 4 seconds. Along the x-axis
there is the repertoire ofAction Units that are included in the knowledge base of the net.
Similar to the tracing of an electroencephalograph, the intensity ofAction Units can be
seen in the horizontal width of the bars. One can see the onset and offiet of an Action
Unit as well as the duration of the apex.

246
 Figure 3, the holding of the expression can be interpreted as an indicator of
appraising the situation as unfair. This interpretation is supported by the fact,
that naive judges do recognize her nonverbal reaction as expressing anger4.
Although preliminary and speculative, the results show that we need to know
more about the dynamics of facial expression. Focussing on the micro analytical
level of the FEAT coding and the simultaneously registered context information
provides the means for obtaining the quantitative data that is required for the
systematic study of the exact timing and the temporal structure of appraisal and
reappraisal processes.

Facial expression and appraisal: Individual differences and affect disturbances

An important advantage of an appraisal-based approach is that it allows one to

study individual appraisal tendencies and action preferences (coping styles) that
become evident over different situations and which can be seen as indicators of a
more or less stable personal style. For example, some participants tend to make
internal causal attributions (agency self) even in situations that are objectively
not controllable. Furthermore, one can differentiate the participants' coping
strategies according to their rigidity versus flexibility, their impulsivity versus
reflexivity, their complexity, and their situational adaptivness. Therefore, one
might envisage using the experimental environment for diagnostic purposes.
In specifying distinctive cognitive patterns for different emotions,
componential appraisal theories can be used to help make diagnoses, identify
specific etiologies, and guide interventions designed to prevent or treat
emotional dysfunction. As an example, the analysis of the empirically found
situational appraisal profiles led to a detailed specification of an appraisal profile
for shame. Shame can be characterized by a high level of self-agency combined
with a low level of self-intent ("I am clearly responsible for the situation but I
did not intend it at all.") and a high level of control combined with a low level of
power ("Everybody can handle the situation, but not me."). Obviously, people
who exhibit a strong bias to show this appraisal profile, independently of
situational characteristics, are not only vulnerable to experiencing shame but
also to applying maladaptive and self-handicapping coping strategies (more
details on shame-related syndromes and the relation between shame and
depression can be found in Kaufman, 1989; Lewis 1979). The tendency to
appraise different situations in a rigid and inappropriate way might contribute to
the development and/or the maintenance of affective disturbances. Inappropriate
situation appraisal, in particular with respect to causal attribution, more
specifically overestimation of self-agency, has been at the core of cognitive
theories of depression etiology (Abramson, Seligman, & Teasdale, 1978;
Teasdale & Barnard, 1993). Other examples for maladaptive appraisal biases
found in clinical disorders include abnormal affect produced by gross
misjudgements of the relevance of an object for reaching one's goals, or by

247
gross underestimates of one's coping potential (Kaiser & Scherer, 1998;
Roseman & Kaiser, 2001).
Kaiser and Scherer (1998) have argued the utility of applying models of
normal emotions to a variety of affect disturbances and maladaptive stress
responses. Affective disorders can be conceptualised as malfunctions of intra-
individual and inter-individual regulation. The former involves inadequate
appraisal of situations and events as well as dissociation of the relationships
between the different emotion components; the latter comprises inappropriate
expression of affect and major deficiencies in the interpersonal adjustment of
affect in social interaction. We have formulated concrete predictions concerning
disorder specific appraisal biases and deviations from normal emotions
including specific facial expression deviations, which can be used to study the
underlying - mostly unconscious - processes. Generally, this approach allows
us to generate a large number of theoretical predictions that are concrete and
testable and that can be based solidly on empirical work. With FACE, we can
animate the postulated expression patterns and test their validity in judgment
studies asking therapists to decode the expressions in terms of clinical
syndromes and appraisal biases (see Figure 4 for some examples of the
postulated expressions, synthesized with FACE).

Excessively positive Underestimation of Underestimation of Excessive concern

pleasantness and power control about adequacy of
conduciveness power
evaluation
AUS, AU6, AUI2, AUI, AU2, AU5, AUI, AU4, AUI5, AUI, AU2, AU4,
AU25 AU20, AU26 AU25 , AU41 AU5, AU20, AU23

Figure 4. Predictions/or the appraisal biases and the related/acial actions (synthesized
in terms 0/ action unit combinations) lor euphoria, helplessness, hopelessness and
anxiety disorder as published in Kaiser & Scherer (1998).

By associating single Action Units not to basic emotions but to appraisal

dimensions, we can address important empirical findings that, at first sight, seem
counter-intuitive. For example, in Ellgring's (1989) study on endogenous
depressed and neurotic depressed patients, an unexpected variety of facial
patterns was found for the depressed patients compared to rather homogeneous
patterns shown by the controls. Concerning the emotional quality of these

248
expressions, Ellgring states "the content of these affect displays can be
interpreted as pointing to the emotions of fear [A U20] and anger [A U24]. There
were no clear indications for sadness or grief." (Ellgring, 1989, p.80; action units
in square brackets inserted by the present author). Using a componential
approach, another explanation is possible. Rather than arguing that AU20 is an
indicator of fear, it could be interpreted as a sign of appraising one's coping
ability as rather limited, specifically as low power. Similarly, AU24 can be seen
as a sign of appraising an event or a situation as unpleasant. Obviously, both of
these appraisal patterns can be easily related to depression. Thus, the use of a
componential approach may provide diagnostically useful information on both
appraisal and interpersonal action tendencies.

OUTLOOK: POTENTIAL IMPLICATIONS FOR CLINICAL RESEARCH

AND TREATMENT

Roseman and Kaiser (2001) have pointed to another potential advantage of

componential appraisal theory in diagnostic and therapeutic work. The multi-
level approach of componential appraisal theories can help to describe the
cognitions underlying both the source and surface affects in dynamically
interrelated emotion states, and precisely describe the cognitive reinterpretations
involved in alternative emotion-regulation strategies or defence mechanisms.
Multi-level stimulus processing is increasingly used in discussions of cognition
and emotion interaction, especially in the context of memory and affect
disturbance (e.g. Johnson, 1994; Power & Dalgleish, 1997; Teasdale & Barnard,
1993). Multi-level approaches may provide an explanation for how
simultaneous or conflicting emotions (with different degrees of awareness)
could occur. They may also explain why knowledge and feeling can sometimes
be dissociated or discrepant. Facial expressions might be indicative for the
existence of low-level appraisals that contradict high-level conscious appraisal
outcomes. The theoretical predictions resulting from the componential appraisal
approach as suggested in Kaiser and Scherer (1998) might guide clinicians who
aim at looking beyond the verbally reported emotions. However, such an
endeavour can only be successful when therapeutic interactions are analysed on
a micro analytic level. Given the high cost of research using micro-analytic
techniques in interactive settings, particularly with respect to the time needed to
objectively measure and analyse facial behaviour, only a few such studies have
been conducted up to now (examples of such interactive studies in the domain of
clinical disorders can be found in Banninger-Huber, 1997; Heller & Haynal,
1997; Katsikitis & Pilowsky, 1996; Merten, this volume). A common result
from these studies is that micro momentary expressions and subtle control
indicators (like asymmetric, tense, or masking smiles) occur rather often. Znoj
(2000) found that such control indicators can be used for predicting "successful
versus unsuccessful therapeutic outcome" and Heller (personal communication,

249
January 21, 2000) found that clients at risk for re-attempting to suicide showed
more control indicators than non-attempters during an interview that took place
immediately after their first unsuccessful attempt (on average within 3 days).
In general, untrained observers are rather bad in detecting micro momentary
expressions and control indicators. Commenting on the results of a deception
study, Ekman, Friesen and O'Sullivan (1997) report
observers who were shown the videotapes used in this experiment did no better
than chance in distinguishing the honest from deceptive faces. Did these observers
not know what cues to look for, or might the behavioural differences that
distinguish felt happy from masking expression be too subtle to see without slow
motion observation? We believe that the smiles we distinguished are visible.
Although we knew what to look for, we found no difficulty in spotting them at real
time. (p.211).

Being able to correctly decode facial expressions is important for persons

working in the social areas (as part of emotional intelligence). Here, a tool that
allows us to test such capacities might be of interest in several domains
(management, human resources, etc.). This is also true for clinical work and
treatment. Since FACE allows the dynamic animation of low intense, fast, and
rapidly changing facial expressions, including asymmetries (using the same
coding extension as FEAT), it is possible to create stimuli sets that could be used
as a diagnostic tool for testing decoding capacities in clinical popUlations and as
a training tool for improving decoding skills in therapists.

NOTES

I We use the term situated as Suchman has defined it in plans and situated actions (1987). It is
related to but not identical with the term siluationist as used by Femandez-Dols (1999).
2 One traditional approach to study emotions is by asking people to remember as vividly as possible
a situation where they experienced a certain emotion. Another approach tries to evoke emotions
with the aid of selected video sequences that are judged as emotionally arousing.
3 Like FACS, FEAT codes unilateral movements (a left unilateral AU12 at level (j is coded as AU12
[J:), in addition it codes asymmetric movements (e.g. AU12 that is at level a on the left side and
on level (j on the right is coded as AU 12 ad).
4 Probably, judges do not only use the facial expression for their interpretation but also the head
movement, prosody, and vocal expression. The participant murmurs something like "surc?"

AFFILIA TION
Department of Psychology
University of Geneva
Email: Susanne.Kaiser@pse.unige.ch

250
 REFERENCES
Abramson, L. Y., Seligman, M. E. P., & Teasdale, J. (1978). Leamed helplessness in humans:
Critique and reformulation. Journal of Abnormal Psychology, 87, 49-74.
Banninger-Huber, E. (1997). Prototypical affective microsequences in psychotherapeutic interaction.
In P. Ekman & E. L. Rosenberg (Eds.), What the face reveals (pp. 414-433). Oxford: Oxford
University Press.
Banninger-Huber, E., & Widmer, C. (1996). A new model of the elicitation, phenomenology, and
function of emotions. In N. H. Frijda (Ed.), Proceedings of the IXth Coriference of the
International Society for Research on Emotions (pp. 251-255). Toronto: ISRE Publications.
Darwin, C. (1965). The expression of the emotions in man and animals. Chicago: University of
Chicago Press. (Original work published 1876 London: Murray)
Ekman, P. (1992). Facial expressions of emotion: New findings, new questions. Psychological
Science, 3, 34-38.
Ekman, P., & Friesen, W. V. (1969). The repertoire of nonverbal behavior: Categories, origins,
usage, and coding. Semiotica, I, 49-98.
Ekman, P. & Friesen, W. V. (1975). Unmasking theface. Englewood Cliffs: Prentice Hall.
Ekman, P., Friesen, W.V. (1978). The Facial Action Coding System: A technique for the
measurement offacial movement. Palo Alto: Consulting Psychologists Press.
Ekman, P., & Rosenberg, E. L. (1997). What the face reveals. Oxford: Oxford University Press.
Ekman, P., Friesen, W. V., & Ellsworth, P. (1982). Research foundations. In P. Ekman (Ed.),
Emotion in the humanface (pp. 1-143). New York: Cambridge University Press.
Ekman, P., Friesen, W. V., & O'Sullivan, M. (1997). Smiles when lying. In P. Ekman & E. L.
Rosenberg (Eds.), What the face reveals (pp. 201-216). Oxford: Oxford University Press.
Ellgring, H. (1989). Nonverbal communication in depression. Cambridge: Cambridge University
Press.
Fermlndez-Dols, J. M. (1999). Facial expression an emotion: A situationist view. In P. Philippot, R.
S. Feldman, and E. J. Coats (Eds.), The social context of nonverbal behavior (pp. 242-261). New
York: Cambridge University Press.
Fridlund, A. 1. (1994). Humanfacial expression: An evolutionary view. San Diego: Academic Press.
Frijda, N. H. (1986). The emotions. Cambridge and New York: Cambridge University Press.
Frijda, N. H., & Tcherkassof, A. (1997). Facial expressions as modes of action readiness. In J. A.
Russell & J. M. Fernlmdez-Dols (Eds.), The psychology of facial expression (pp. 78-102).
Cambridge: Cambridge University Press.
Izard, C. E. (1991). The psychology of emotions, New York: Plenum Press.
Izard, C. E. (1994). What I should remember from my study of emotions, if 1 were to return to
teaching and practicing psychotherapy. In N. H. Frijda (Ed.), Proceedings of the 8th Conference
of the International Society for Research on Emotions (pp. 149-153). Storrs, CT: International
Society for Research on Emotions.
Gosselin, P., Kirouac, G., & Don:, F. Y. (1995). Components and recognition offacial expression in
the communication of emotion by actors. Journal of Personality and Social Psychology, 68, 1-
14.
Heller, M. & Haynal, V. (1997). Depression and suicide faces. In P. Ekman & E. L. Rosenberg
(Eds.), What theface reveals (pp. 398-413). Oxford: Oxford University Press.
Johnson, M. (1994). Emotion and the multiple entry memory system. In N. H. Frijda (Ed.),
Proceedings of the 8th Conference of the International Society for Research on Emotions (pp.
88-92). Storrs, CT: International Society for Research on Emotions.
Kaiser, S., & Scherer, K. R. (\998). Models of 'normal' emotions applied to facial and vocal
expressions in clinical disorders. In W. F. Flack, Jr. & 1. D. Laird (Eds.), Emotions in
psychopathology (p. 81-98). New York: Oxford University Press.
Kaiser, S. & Wehrle, T. (1992). Automated coding of facial behavior in human-computer
interactions with FACS. Journal of Nonverbal Behavior, 16,67-83.

251
Kaiser, S. & Wehrle, T. (1996). Situated emotional problem solving in interactive computer games.
In N. H. Frijda (Ed.), Proceedings of the IXth Conference of the International Society for
Research on Emotions (pp. 276-280). Toronto: ISRE Publications.
Kaiser, S., & Wehrle, T. (2000). Ausdruckspsychologische Methoden. In J. H. Otto, H. A. Euler, &
H. Mandl (Hrsg.), Handbuch emotionspsychologie (pp. 419-428). Weinheim: Beltz, Psychologie
Verlags Union .
Kaiser, S. & Wehrle, T. (2001). Facial expressions as indicators of appraisal processes. In K. R.
Scherer, A. Schorr, & T. Johnstone (Eds.), Appraisal theories of emotions: Theories, methods,
research (pp. 285-300). New York: Oxford University Press.
Kaiser, S., Wehrle, T., & Schmidt, S. (1998). Emotional episodes, facial expression, and reported
feelings in human-computer interactions. In A. H. Fischer (Ed.), Proceedings of the Xth
Conference of the International Society for Research on Emotions (pp. 82-86). WOrzburg: ISRE
Publications.
Kappas, A. (2002). What facial activity can and cannot tell us about emotion. In M. Katsikitis (Ed.),
The human face: Measurement and meaning (pp. 215-234). Dordrecht, Kluwer Academic
Publishers.
Katsikitis, M. (2002). FACEM: The facial expression measurement system. In M. Katsikitis (Ed.),
The human face: Measurement and meaning (pp. 99-118). Dordrecht, Kluwer Academic
Publishers.
Katsikitis, M., & Pilowsky, I. (1996). A controlled study of facial mobility treatment in Parkinson's
disease, Journal of Psychosomatic Research, 40,387-396.
Kaufman, G. (1989). The psychology of shame: Theory and treatment of shame-based syndromes,
New York: Springer.
Lazarus, R. S. (1966). Psychological stress and the coping process. New York: McGraw Hill.
Leventhal, H., & Scherer, K. (1987). The relationship of emotion to cognition: A functional
approach to a semantic controversy. Cognition and Emotion, I, 3-28.
Lewis, H. B. (1979). Shame in depression and hysteria. In C. E. Izard (Ed.), Emotions in personality
and psychopathology (pp. 399-414). New York: Plenum.
Malatesta, C. Z., Haviland, J. M. (1982). Learning display rules: The socialization of emotional
expression in infancy. Child Development, 53,991-1003.
Manstead, A. S. R., Fischer, A. H., & Jakobs, E. (1999). The social and emotional functions of facial
displays. In P. Philippot, R. S. Feldman, & E. 1. Coats (Eds .), The social context of nonverbal
behavior (pp. 287-316). New York: Cambridge University Press.
Merten, 1. (1997). Facial-affective behavior, mutual gaze and emotional experience in dyadic
interactions. Journal of Nonverbal Behavior, 21, 179-20 I.
Merten, 1. (2002). Context-analysis of facial-affective behaviour in clinical populations. In M.
Katsikitis (Ed.), The human face: Measurement and meaning (pp. 131-147). Dordrecht, Kluwer
Academic Publishers.
Osgood, C. E. (1966). Dimensionality of the semantic space for communication via facial
expressions. Scandinavian Journal of Psychology, 7, 1-30.
Philippot, P., Feldman, R. S., & Coats, E. J. (1999) (Eds.). The social context of nonverbal behavior.
New York: Cambridge University Press.
Power, M., & Dalgleish, T. (1997). Cognition and emotion: From order to disorder. Hove:
Psychology Press.
Roseman, I. J. & Kaiser, S. (200 I) . Applications of appraisal theory to understanding and treating
emotion pathology. In K. R. Scherer, A. Schorr, & T. Johnstone (Eds.), Appraisal theories of
emotions: Theories, methods, research (pp. 249-270). New York: Oxford University Press.
Roseman, I. J., Wiest, C., & Swartz, T. S. (1994). Phenomenology, behaviors, and goals differentiate
discrete emotions. Journal of Personality and Social Psychology, 67, 206-221.
Russell, 1. A. (1997). Reading emotions from and into faces: Resurrecting a dimensional-contextual
perspective. In J. A. Russell & J. M. Fernandez-Dols (Eds.), The psychology offacial expression
(pp. 295-320). Cambridge: Cambridge University Press.
Russell, J. A., & Fernandez-Dols, J. M. (1997) . The psychology of facial expression. Cambridge:
Cambridge University Press.

252
Scherer, K. R. (1984). On the nature and function of emotion: A component process approach. In K.
R. Scherer & P. Ekman (Eds.), Approaches to emotion (pp. 293-318). Hillsdale, N. 1.: Lawrence
Erlbaum.
Scherer, K. R. (1992). What does facial expression express? In K. Strongman (Eds.), International
Review of Studies on Emotion (pp. 139-165). Chichester: Wiley.
Scherer, K.R. (1994). Toward a concept of "modal emotions". In P. Ekman & R. J. Davidson (Eds.),
The nature of emotion: Fundamental questions (pp. 25-31). New York: Oxford University Press.
Schlosberg, (1952). The description of facial expressions in terms of two dimensions. Journal of
Experimental Psychology, 44,229-237.
Schmidt, S. (1998). Les expressions faciales emotionnelles dans Ie cadre d'unjeu d'ordinatuer: rejlet
de processus d'evaluation cognitive ou d'emotions de base? Unpublished doctoral Dissertation,
University of Geneva, Switzerland.
Smith, C.A., & Ellsworth, P.C. (1985). Patterns of cognitive appraisal in emotion. Journal of
Personality and Social Psychology, 48,813-838.
Smith, C. A., & Scott, H. S. (1997). A componential approach to the meaning of facial expression. In
J. A. Russell & J. M. Fernandez-Dols (Eds.), The psychology offacial expression (pp. 229-254).
Cambridge: Cambridge University Press.
Steimer-Krause, E., Krause, R. & Wagner, G. (1989). Interaction regulations used by schizophrenic
and psychosomatic patients. Studies on facial behavior in dyadic interactions. Psychiatry:
Interpersonal and Biological Processes, 53,209-228.
Suchman, L. A. (1987). Plans and situated actions. The problem of human-machine communication.
Cambridge: Cambridge University Press.
Teasdale, 1. D., & Barnard, P. J. (1993). Affect, cognition, and change: Remodelling depressive
thought. Hillsdale, N. J.: Lawrence Erlbaum.
Wehrle, T. (1992/1996). The Facial Expression Analysis Tool (FEA7). [Unpublished computer
software]. University of Geneva, Switzerland.
Wehrle, T. (1993). The Autonomous Agent Modeling Environment (AAME). [Unpublished computer
software]. University of Geneva, Switzerland.
Wehrle, T. (I 994a). Eine Methode zur pychologischen Modellierung und Simulation von autonomen
Agenten. (A method for the psychological modeling and simulation of autonomous agents).
Unpublished doctoral dissertation, University of ZOrich.
Wehrle, T. (1994b). New Fungus Eater experiments. In P. Gaussier & J. D. Nicoud (Eds.), From
perception to action. Los Alamitos: IEEE Computer Society Press.
Wehrle, T. (1995/1999). The Facial Action Composing Environment (FACE). [Unpublished
computer software]. University of Geneva, Switzerland.
Wehrle, T. (1995/2000). The Geneva Appraisal Theory Environment (GATE). [Unpublished
computer software]. University of Geneva, Switzerland.
Wehrle, T. (1996). The Geneva Appraisal Manipulation Environment (GAME). [Unpublished
computer software]. University of Geneva, Switzerland.
Wehrle, T. (1996). The Interactive Data Elicitation and Analysis tool (IDEA). [Unpublished
computer software]. University of Geneva, Switzerland.
Wehrle, T. (1999). Topological Reconstruction And Computational Evaluation of Situations
(TRACE). [Unpublished computer software]. University of Geneva, Switzerland.
Wehrle, T. (200 I). The grounding problem of modeling emotions in adaptive artifacts. In P. Petta &
D. Canamero (Eds.), Grounding emotions in adaptive systems [Special issue]. Cybernetics and
Systems: An International Journal, 32, 561-580.
Wehrle, T. & Kaiser, S. (2000). Emotion and facial expression. In A. Paiva (Ed.), Affect in
Interactions: Towards a new generation of inteifaces (pp. 49-63). Heidelberg: Springer.
Wehrle, T., Kaiser, S., Schmidt, S. & Scherer, K. R (2000). Studying the dynamics of emotional
expression via synthesized facial muscle movements. Journal of Personality and Social
Psychology, 78, 105-119.
Znoj, H. J. (2000). Perspectives of emotion regulation in clinical psychology. Invited talk at the
University of Geneva, Switzerland, January 26.

253
 MARKG FRANK

CHAPTER 13

GETTING TO KNOW YOUR PATIENT: HOW FACIAL

EXPRESSION CAN HELP REVEAL TRUE EMOTION.

INTRODUCTION

The moment the patient/client walks in the door a wide variety of information about
this patient becomes available to the clinician or Health Care Provider (from here
abbreviated to HCP). Not only is written information on the charts or notes about
past history available to the HCP, but there is visual information in the clothing of
the patient that communicates socio-economic status, marital status, and self-image;
visual information in the body that communicates the patient's sex, ethnicity, and
physical strength; visual information in the body movements that communicates
fatigue, enthusiasm, anxiety, extroversion or introversion, dominance or
submissiveness; auditory information in the voice that communicates ethnicity,
fatigue, anxiety, dominance or submissiveness, and current emotional state, and
olfactory information that communicates self-image and hygiene (reviewed by
Knapp & Hall, 1997). These are by no means the only things communicated through
these channels. Regardless, a problem with all these nonverbal clues is that they are
only probabilistically related to what they predict. For example, although
impoverished patients may own more worn clothes, affluent patients may also own
worn clothes. Furthermore, it is possible for thin patients to be physically strong; the
lack of a wedding band does not necessarily mean a patient is not married, and so
forth. Although this information can be helpful to the HCP understanding his or her
patient, it is not accurate enough to make diagnostic judgments.
Yet there is one type of nonverbal information available that evidence suggests is
accurate enough to make such diagnostic judgments - and this is in the area of facial
expression of emotion. There is evidence that some facial expressions, particularly
those tied to emotional states, are not only recognized at rates greater than chance,
but are universal across culture and learning (Ekman, 1994; Izard, 1994). If this is
so, it may be able to help HCP distinguish the true emotional states of their
patients/clients. For example, when a patient smiles, is this because he or she is
really experiencing the emotion of happiness, or is he or she feigning happiness?
When a patient shows a micromomentary facial expression of contempt when he or
she discusses his or her partner, is this expression something to which the HCP
should attend when dispensing advice? The available evidence suggests that these
sorts of judgments can be made, and that they can be made accurately. The key to
making these accurate judgments might depend upon recognizing which facial
expressions are universal, and tied to the human emotional system, and which are
culturally dependent and not indicators of internal emotional state.

WHAT DO FACIAL EXPRESSIONS CONVEY?

Although the musculature of the face allows it to move in 44 distinct movements, not
all possible combinations of these movements occur in everyday life (Ekman &
Friesen, 1978). When discussing "facial expressions", researchers tend to reserve
that term for those recurring configurations of facial muscle movements that
communicate some thought, emotion, or behavior (e.g., Fridlund, 1994). However,
not all recurring facial muscle configurations express specific messages; for
example, some facial muscle actions that might accompany spoken words - such as
raising one's eyebrows in order to emphasize a particular word - may modify that
communication, but the eyebrow movement itself is not a stand-alone message
independent of the speech (e.g., Birdwhistell, 1970; Blurton-Jones, 1967; Ekman,
1979).

Expressing Thoughts

The face is capable of expressing various thoughts. For example, one can convey
sophisticated skepticism by raising the outer corner of one eyebrow (lateral portion
of the frontalis muscle; Ekman, 1979). One can express sympathy by pulling one's
eyebrows up in the middle (Bavelas, et ai, 1986). One can convey kidding by
winking (Efron, 1941). One can greet others by flashing both eyebrows upward

256
(Eibl-Eibesfeldt, (970). One can express uncertainty by lowering both eyebrows
(corrugator muscle; Ekman, (979). Researchers agree for the most part that patterns
of facial configuration that express various thoughts are learned like language, are
culturally specific, and rely on context for proper interpretation (e.g., Ekman, 1989;
Fridlund, 1994; Ortony & Turner, 1990). Therefore, the same lowered eyebrow
expression caused by the corrugator muscle might send a different message in
different cultures - in North America it might mean that one is "uncertain", whereas
in Borneo it might mean that one is negating what was just spoken (Darwin, (872).
Even within culture, the lowered eyebrow expression might mean effort, confusion,
concern, doubt, or anger - depending upon the context (Ekman, 1979).

Expressing Feelings
The face is also capable of expressing emotions - and what makes the facial
expression of emotions different from the facial expression of thoughts is that there
is evidence that these emotions are expressed and interpreted the same across all
cultures (e.g., Ekman, 1994; Izard, 1994). This "universal" production and
perception across cultures suggests that those facial expressions of emotions might
be genetically determined, rather than socially learned, and that they might be linked
to internal emotional states. Thus, if facial expressions of some emotions are
genetically determined, and if they are linked to the emotional state of the person,
they might be a critical source of information for the HCP about the status of the
HCP's relationship with the patient, the patient's relationship to others, and a way to
monitor the emotional progress of the patient during the course of treatment. For
example, humans express the emotion of happiness by smiling. The smile involves
the zygomatic major muscle (which raises the lip corners upward), along with the
orbicularis oculi muscle (which causes the "crow's feet" appearance around the
outside of the eye; Ekman & Friesen, 1982). Humans express sadness or distress by
frowning. The frown may include the triangularis muscle (which pulls down the lip
corners) and the medial portion of the frontalis muscle (which raises the inner
corners of the eyebrows; Ekman & Friesen, 1975). Other emotions that seem to have
specific facial expressions include anger, disgust, fear, and surprise, and to a lesser
extent contempt, embarrassment, interest, pain, and shame (e.g. Ekman, 1984; 1992;
Ekman et. al. 1987; Izard, 1971; Izard & Haynes, 1986; Keltner, 1995; Prkachin,
1992). However, this and other claims for universality of facial expressions of
emotion are not without controversy (e.g., Russell, 1994). This controversy is better
understood by a brief look at the history of research on facial expressions.

257
 FACIAL EXPRESSION RESEARCH.

Brief History

The systematic study of facial expressions began with the publication of Darwin's
book The expression of the emotions in man and animals (187211998). In this book,
Darwin proposed that humans across all cultures have distinct facial expressions for
particular emotions, and that these expressions are produced involuntarily as a result
of that emotion (through what he called "nerve force"). Darwin defined emotions as
behavioral and physiological reactions that have assisted humans and animals
survive the various life challenges they faced throughout their evolutionary history.
For example, the fear reaction assisted humans and animals escape danger, the anger
reaction assisted humans and animals to fight rivals, and so forth. Those who
possessed these emotional reactions were more likely to live to reproductive age, and
therefore pass their genes on to the next generation. Darwin based his ideas on his
own observations of expressions, and informant observations of facial expressions in
different countries.
What Darwin argued (and elaborated later by others; e.g., Ekman, 1992; Izard,
1990; Plutchik, 1962; Tomkins, 1962, 1963) is that social animals, such as humans,
must communicate these emotions to others in the group because emotions express
imminent behavior, such as striking out in anger, fleeing in fear, avoiding spoiled
food in disgust, approachability in happiness, and other action tendencies (e.g.,
Frijda, 1986). The expression of imminent behavior allows humans to coordinate
their behaviors in ways to reduce conflict, avoid danger, and increase harmony. For
example, the emotion of anger produces a facial expression of anger - and this facial
expression becomes the visual signal of this intention to strike. Others in the social
group who see the anger expression can avoid this person - which prevents a fight
and the potential fatal or debilitating injuries therein '.
However, much of the empirical work that followed Darwin's book failed to
support his notion that there were particular facial expressions for particular
emotions. For example, when subjects were startled by firecrackers, embarrassed, or
disgusted by having to decapitate a live rat, their most common facial expression
made across all these situations was a smile - and they smiled although they were not
experiencing a positive emotion (Landis, 1924). Likewise, when infants were
subjected to different emotional stimuli, no clear pattern of facial expressions
emerged (Sherman, 1927). Observations of people outside North America by social
scientists cast further doubt on Darwin's ideas. For example, the smile was observed

258
 as an expression of uncertainty in Africans (LeBarre, 1947), but as an expression of
sadness in Japanese women (Klineberg, 1940). This conclusion was not without
contention; researchers noted that much of the disagreement on which facial
expressions represented which emotion may have been due to researchers classifying
each emotion word, regardless of how synonymous, as a unique emotion (i.e., rage,
anger, and fury were characterized as different emotions, when one could argue they
are synonyms for anger). When' these synonymous terms were collapsed, six
emotion clusters emerged - and when these clusters were employed much of this
disagreement vanished (Woodworth, 1938). Regardless, influential reviews at the
time concluded that facial expressions did not provide accurate information as to
emotional state, and that all facial expressions including facial expressions of
emotion - were culturally relative, socially learned, and that there were no universals
(Bruner & Taguiri, 1954).
Despite the negative findings on universality of facial expressions of emotion,
two theorists latched onto the findings of Woodworth (1938) and others and noted
that the six emotion clusters were similar to the evolutionary-based emotions that
Darwin predicted (Plutchik, 1962; Tomkins, 1962, 1963). These researchers found
evidence consistent with universality when they showed posed photographs of facial
expressions of the six clusters of emotion terms described above to observers, who
showed strong agreement as to which expressions represented which emotions
(Tomkins & McCarter, 1964). Other researchers, using a similar paradigm, began to
find evidence for universality by showing photographs of people posing facial
expressions to cultures in Europe, South American, Africa, and Asia, and reporting
that all these groups agreed on which facial expression was anger, which was
disgust, fear, happy, etc. (Ekman & Friesen, 1971; Izard, 1971). This finding did not
sit well with the proponents of the social learning/cultural relativism perspective,
who counter-argued that the populations upon which this evidence for universality
was based were mostly educated, and thus could have learned from media or other
forms of communication which expressions represented which emotions (e.g.,
Birdwhistell, 1970; see Mead, 1975, as well). To counter that argument, researchers
conducted similar emotion judgment studies with visually isolated peoples whom
had limited contact with Westerners - e.g., the Sadong of Borneo, and Fore in New
Guinea - who therefore could not have learned these expressions from media. These
researchers found for the most part the same pattern of universal production and
recognition of facial expressions of emotion as in the Westernized peoples (e.g.,
Ekman, Sorenson, & Friesen, 1969). Parallel evidence for universality came from
observations of children who were born blind and deaf. These children could not

259
 have seen these facial expressions to learn how to express them; in fact, children who
are only visually impaired need to be trained to adequately pose these facial
expressions (Webb, 1974). Yet when visually and auditory impaired children
experienced spontaneous emotions, they showed similar expressions of emotion as
their sighted counterparts (Eibl-Eibesfeldt, 1973). Follow up research using
variations of the basic paradigm where facial expressions are shown to observers
who rate the emotion shown - e.g., using free recall when examining photographs,
rather than asking observers to choose from a limited number of emotion terms, or
examining different cultural groups - found similar patterns throughout the 1970s
and 1980s, all suggesting universality (e.g., Boucher & Carlson, 1980; Ekman, et.
aI., 1987; Haidt & Keltner, 1999; Izard, 1977; Matsumoto & Ekman, 1989).
However, the proponents of the universalist position were still stuck with the
findings that people of different cultures sometimes showed different expressions for
a given emotion than North Americans. Ekman proposed that the reason this
happened was that different cultures learned different rules to regulate their
expression of emotion - what he called "display rules" (Ekman, 1972). For example,
Japanese culture has a display rule that they do not express emotions such as anger
or disgust to higher status people, whereas North American culture does not have
such a rule. When Japanese and Americans were secretly observed while watching a
gory film, both cultural groups showed facial expressions of disgust. When Japanese
and Americans were shown this film in the presence of a high status person, the
Japanese smiled whereas the Americans still showed facial expressions of disgust -
even though both groups still experienced disgust (Friesen, 1972, cited in Ekman,
1972). Ekman (1972) proposed that these display rules can apply within cultures as
well, as in North America, where boys are typically taught to suppress sadness, and
girls are taught to suppress anger. This concept of display rules seemed to account
for why researchers had observed other cultures smiling to such seemingly different
events as the death of a loved one, confusion, uncertainty, startle, sexual excitement,
disgust, and so on. In all these experiments, the subjects knew they were being
observed. This may have activated their cultural display rules, which in turn caused
them to mask their emotions with a smile (Ekman, Friesen, & Ellsworth, 1972;
Ekman, Friesen, & O'Sullivan, 1988). Based on these findings and other findings,
Ekman (1977) proposed his neurocultural theory of emotions. This theory argued
that certain basic human emotions had specific physiological patterns, that these
emotions produced particular facial expressions that were universal across all
cultures, but that the ultimate facial expression of these emotions was modified,
exacerbated, suppressed, or masked by social learning processes dependent upon

260
cultural or local customs. However, Ekman argued that if the emotional experience
is of sufficient intensity, the facial expression for that emotion would "leak" through
and be visible despite efforts by the person to control or hide them (e.g., Ekman,
1992).

Challenges to Universality

By the early 1990s, a consensus seemingly emerged in the experimental psychology

literature that Darwin was correct after all, and that some facial expressions of
emotion were universal (Matsumoto, 1990). This was not a peaceful consensus;
based on the observations of other cultures, social scientists who placed the
uniqueness of culture at the forefront of any understanding of emotion were not
convinced of universality (e.g., Lutz & White, 1986; Barrett & Katsikitis, this
volume). Moreover, two new challenges to universality were issued by experimental
psychology itself in the early 1990s. One challenge to universality attacked the
methods and concepts used to document universality (Ortony & Turner, 1990;
Russell, 1994). These researchers proposed that not only were there some
conceptual problems in what was meant by the term "basic emotions" and
universality, but there were also many small methodological problems in the
research designs that found evidence for universality. These problems, such as using
biased response forms, using prescreened and posed facial expression photos, and
within-subject judgment designs, when added together, may have collaborated to
bias observers into artificially agreeing on which facial expression represented which
emotion (Russell, 1994). These potentially fatal problems drove researchers to
address them in a series of replications. These replications with improved methods
produced findings consistent with the previous findings on universality (e.g., Frank
& Stennett, 2001; Haidt & Keltner, 1999; Rosenberg & Ekman, 1994).
A second challenge suggested that all facial expressions were simply
communicative gestures; that is, they are not the result of internal emotional states,
but only the result of the social motives of the person within a particular context (the
"behavioral ecology" view; Fridlund, 1994). The behavioral ecology view found
evidence that the amount of smiling, as defined by zygomatic major action, was
related not to a subject's self-report of positive emotion, but was related to the
physical or psychological presence of others (Fridlund, 1991). Critics of the
behavioral ecology view countered that not all smiles are the same (Ekman, 1989).
They too found that when smiling was defined as only zygomatic major action, there
was no relationship between self-report of positive emotion and smiling. However,

261
when they distinguished smiles that feature both zygomatic major action and
orbicularis oculi action (causing the "crow's feet" appearance) - called an enjoyment
smile (Ekman, 1989) - they found that only the number of enjoyment smiles, but not
total number of all smiles, was related to the self report of positive emotional
experience (Ekman, Davidson, & Friesen, 1990). They also found that this
enjoyment smile also features a different pattern of brain activity than other smiles
(Davidson, et. aI, 1990; Ekman & Davidson, 1993), and that these smiles could be
distinguished from each other with the naked eye (Frank, Ekman, & Friesen, 1993).
These findings armed proponents of the universality of facial expressions of emotion
to suggest that the reason other researchers found no relationship between smiling
and positive emotion (e.g., Fridlund, 1991; Klineberg, 1940; Kraut & Johnston,
1979; Landis, 1924) was that they failed to distinguish between enjoyment smiles
and other smiles. Had they done so, they might have found a link between
enjoyment smiles and positive emotion (reviewed by Frank & Ekman, 1993).

Emotion links to Physiology

Thus, by the year 2000, it seems that through all the turbulence Darwin may have
been basically correct all along; there does seem to be some facial expressions of
emotion - specifically anger, disgust, fear, happy, sad, surprise, and to a lesser degree
contempt, embarrassment, interest, pain, and shame - that are universal across all
cultures. Researchers have since documented a relationship between these
expressions and the physiology of the emotional response. For example, they found
a significant relationship between self-report of various emotions and the expressions
of these emotions (Ekman, Friesen, & Ancoli, 1980). Later work found evidence
that anger, disgust, fear, happy, sad, and surprise each have unique physiological
profiles (Ekman, Levenson, & Friesen, 1983); moreover, when subjects are asked to
pose facial expressions of anger, disgust, fear, happiness, sadness, and surprise, they
found the same patterns of emotion-specific physiology (Levenson, Ekman, &
Friesen, 1990). They found this same link between facial expressions and emotion-
specific physiology in different subject groups as well, such as the elderly
(Levenson, et. aI., 1991) and a matrilineal, Muslim, non-western culture (Levenson,
Ekman, Heider, & Friesen, 1992). Research has also found that specific emotions
have specific Central Nervous System (CNS) patterns of hemispheric brain
activation, as measured by the electroencephalogram (EEG). In particular,
experimentally aroused, approach-related positive emotion has been associated with
relatively greater left-sided anterior activation, whereas withdrawal-related negative

262
emotion has been associated with relatively greater right-sided anterior activation
(Davidson, 1984; 1992; Davidson & Tomarken, 1989).
Specifically focusing on positive emotions, research has shown that enjoyment
smiles are linked to the CNS physiology of approach-related positive emotion,
whereas other smiles are not. Thus, when subjects' EEG is monitored as they watch
emotion inducing films, it is only the enjoyment smiles - but not other smiles - that
not only correlate with their self-reports of enjoyment, but produce a relatively
greater left hemisphere brain activation (Davidson, et. aI, 1990; Ekman et aI, 1990).
This same pattern of CNS activity for enjoyment smiles was found in 10 month old
infants when they showed more enjoyment smiles in response to the approach of
their mothers than to the approach of a stranger (Fox & Davidson, 1988). Although
these studies relied upon spontaneous expression of smiling, one study manipulated
the types of smiles by asking subjects to voluntarily generate smiles with and without
the orbicularis oculi component, and this study found the same pattern of EEG
activity as seen in the spontaneous enjoyment smile research (Ekman & Davidson,
1993).

Neuroanatomical Evidence

If one is trying to make this biological/evolutionary argument for the production and
perception of facial expressions of emotion, one's case is strengthened if one can
show specific areas in the brain that account for the production and perception of
such expressions. First, why would humans employ facial expressions as a mode of
communication, when language would seem to be more efficient? It appears that
these facial expressions of emotion are best seen as vestiges of an entirely nonverbal
human communication system that must have existed in extinct hominids such as
Neanderthal, because only modern humans have the throat structures necessary to
produce articulate speech (Lieberman, 1992). A further clue to this prehistoric
human communication comes from the genetically closest living relatives of
humans, the chimpanzees, who have a repertoire of facial expressions of emotion
that parallel, but are not identical to, human facial expressions of emotion
(Chevalier-Skolnikoff, 1973). Thus, current human facial expression of emotion in
all likelihood represents more the communication methods of the past genetic history
of the human species, rather than its present conditions (Too bey & Cosmides, 1990).
In other words, although language permits more efficient and effective
communication for modern humans, humans still have the phylogenetically older,
more nonverbal modes of communication available to them.

263
 Neuroanatomical Production of Facial Expressions
The idea that the face can express deliberate, learned expressions, as well as more
involuntary, unbidden facial expressions of emotion is supported by the architectural
neuroanatomy of the face. There appear to be two distinct neural pathways that
mediate facial expressions, each one originating in a different area of the brain; one
pathway for voluntary, willful facial actions, and a second for involuntary, emotional
facial actions (Meihlke, 1973; Myers, 1976; Tschaissny, 1953). The voluntary facial
movements originate in the brain's cortex - specifically the cortical motor strips. The
right strip controls the left side of the face and body, and the left strip controls the
right side of the face and body. These strips control most deliberate movements
including hand and leg movements. These impulses to move deliberately the face
arrive at the face via the pyramidal motor system. Involuntary facial movements
originate in the subcortical areas within the brain. These areas are associated with
the production of emotion. These impulses arrive at the face via the extrapyramidal
motor system. This dual pathway hypothesis is supported by clinical reports of
patients with lesions on one cortical motor strip, who when asked to pose a smile
cannot deliberately move the side of their face contralateral to the lesioned side.
These patients thus show an asymmetrical smile. However, when these same
patients are induced to experience positive emotion, such as when they are told a
joke they find humorous, they produce a symmetrical smile (Brodal, 1981).
Likewise, patients with lesions of the subcortical areas of the brain such as the basal
ganglia have difficulty showing spontaneous, emotional facial expressions; however,
these patients are able to move their facial muscles on command (Karnosh, 1945).
These facial action observations are so reliable that they serve as diagnostic criteria
for pyramidal and extrapyramidal lesions (OeMyer, 1980). Current research is
attempting to isolate further areas of production of various facial expressions (e.g.,
Anderson & Phelps, 2000).
Not only do voluntary and involuntary facial actions differ by neural pathway,
but the actions mediated by these pathways manifest themselves differently. In a
normal person, voluntary pyramidal motor system based movements are limited
solely by individual effort. A person can consciously move a facial muscle quickly
or slowly, and hold that action for a brief or long period of time, depending upon the
dictates of the circumstance and individual endurance. However, extrapyramidal
motor system based facial actions are characterized by synchronized, smooth,
symmetrical, consistent, and reflex-like or ballistic-like actions on the part of the
component facial muscles (reviewed by Rinn, 1984: Warren & Thompson, this

264
volume). Relatively speaking, these actions appear to be less under the deliberate
control of the individual.

Neuroanatomical Perception of Facial Expressions

Likewise, researchers have identified various areas of the brain involved in the
perception of specific facial expressions of emotion using functional Magnetic
Resonance Imaging (fMRI) techniques. When researchers show a photo of a facial
expression of emotion (typically the same photos used in the universality studies
described earlier), they have found that anger seems to activate the right orbitofrontal
cortex and anterior cingulate cortex (Blair, Morris, Firth, Perret, & Dolan, 1999).
Disgust seems to activate the anterior insula and limbic cortico-striatal-thalamic area
(Phillips, Young, Senior, Brammer, Andrew, Calder, Bullmore, Perrett, Rowland,
Williams, Gray, & David, 1997). Fear seems to activate the left amygdala (Morris,
Frith, Perrett, Rowland, Young, Calder, & Dolan, 1996; Whalen, Rauch, Etcoff,
McInerny, Lee, & Jenke, 1998). Happiness seems to activate the left side of the
lateral frontal, mid-frontal, anterior temporal, and central anterior scalp regions
(Davidson, et. aI., 1990; Ekman & Davidson, 1993). Sadness seems to activate the
left amygdala and right temporal lobe (Blair, et. aI., 1999). What this means is that
humans appear to be biologically wired to perceive these specific emotion faces, and
it seems reasonable to conclude that these expressions must have held some
significance to our species.

FACIAL EXPRESSION AND HEALTH CARE

Once these universals in facial expression were established, they have proven their
utility as markers of various states of social functioning - markers that might be
critical for a HCP to note. For example, the presence of enjoyment smiles, but not
other smiles, on the part of a person who has survived the death of their romantic
partner predicts successful coping with that traumatic loss (Bonnano & Keltner,
1997). Clinically depressed patients show fewer facial expressions in general, and
enjoyment smiles in particular (e.g., Berenbaum & Oltmanns, 1992; Katsikitis &
Pilowsky, 1991; Katsikitis, Pridmore & Marzullo, 1999). In fact, evidence shows
that the increased proportion of enjoyment smiles compared to nonenjoyment smiles
can foretell clinically depressed patients' successful response to therapy (Ekman,
Matsumoto, & Friesen, 1997). Moreover, patients with schizophrenia tend to show
different, and sometimes fewer or more disorganized facial expressions than normal
patients (Krause, et. aI., 1989) when experiencing an emotion. These patients seem

265
to feel emotion, as measured by galvanic skin conductance measures, but they do not
express these emotions (Kring & Neale, 1996). At risk aggressive adolescents
respond with more facial expressions of anger, and less facial expressions of
embarrassment, to the same stimuli than less aggressive adolescents (Keltner, Moffit,
& Strouthamer-Loeber, 1995). Patients with myocardial ischemia, with Type A
personalities, tend to show more elements of an anger facial expression than others
(Rosenberg, Ekman, & Blumenthal, 1998). Mothers show different sorts of smiles to
their difficult compared to their non-difficult children (Bugental, 1986). People who
score high in psychopathology show physiological responses to facial expressions of
anger, but not sadness, suggesting they do not feel sympathy (Patrick, 1994).
Moreover, facial expressions of emotion are useful indicators of relationship status.
The expression of disgust or contempt, but not anger, predicts marital divorce
(Gottman, 1994). Finally, deception research has begun to note that these unbidden
facial expressions of emotion can occur for very brief flashes, called
"microexpressions", that under certain circumstances microexpressions of fear or
disgust can betray deception (Ekman, Friesen, Scherer, & O'Sullivan, 1991; Frank &
Ekman, 1997). Each of these examples above suggests that a perceptive Hep might
make use of the patient's expressions to become a more effective Hep.

Can we succes,~fu/ly train?

It is not clear how well Heps can and do recognize these various emotions in the real
world. We can only presume their skill is similar to those subject groups tested in
laboratory work, which means they can accurately identify these expressions at rates
of over 80% (Ekman, et. aI., 1987). This would seem to represent the higher end of
the accuracy scale, because most of this work involved judging these expressions
under ideal conditions, i.e., judging photographs of posed facial expressions under
no time pressure. Yet in the real world of the Hep, a dynamic facial action
competes for the attention of the Hep with a variety of other information in the
voice, body, and words - all of which would push to reduce judgment accuracy. For
example, when naIve observers try to judge the true emotion of subjects who are
instructed to feign pleasure as they experience strong disgust while viewing a gory
film, observer accuracy drops to under 60% (Ekman & Friesen, 1974). Note that
observers in this task were only asked to distinguish between two emotions, and not
among 7, and yet there was a precipitous drop in accuracy compared to judging the
still photos out of context.

266
 Conversely, if one is feeling a strong emotion that one is not motivated to hide,
typically most of the nonverbal communication channels will generate similar
messages about the emotion state (e.g., Rosenthal, 1979). Thus, a happy person will
show a smile, have a happy lilt in their voice, a bounce in their step, and so forth .
This redundancy of information may serve to drive up accuracy.
These results suggest that high accuracy when judging emotion is possible. The
question is how well we can train HCPs to improve their abilities to recognize these
expressions, particularly under conditions where some patients will be motivated to
conceal, underreport, or misrepresent their feelings. What we do know is that most
HCPs receive little training in nonverbal communication. For example, researchers
suggest that Primary care physicians underdiagnose emotional disorders because of a
failure to detect various behavioral clues associated with such disorders (Ormel &
Tiemens, 1995). Others have suggested that the field of psychotherapy may be
actively opposed to observing and/or interpreting nonverbal behaviors within a
clinical context (Kinseth, 1989). Finally, more experienced psychotherapists were
seen as agreeing more with expert judgments of a clinical patient than were
beginning students, suggesting an element of experience allows increased accuracy
of emotional recognition (Machado, Beutler, & Greenberg, 1999).
There is indirect evidence that people can be trained to improve their abilities to
recognize other forms of nonverbal communication, such as deception (Frank &
Feeley, under review) . Although deception in the real world is more likely to
involve strong emotions (Frank & Ekman, 1997), most of the work looking at
training people to improve their abilities to recognize deception has employed low
stake lies, where it seems that not many strong emotions were elicited on the part of
the liars. Moreover, the training sessions tended to be brief (under an hour), and the
judgment of deception is always probabilistic, in that there is no definitive nonverbal
sign of deception (Frank & Feeley, under review) . Yet even under these poor
training conditions there seems to be a modest increase in deception detection
abilities. This bodes well for future attempts to train HCPs to improve their abilities
to recognize subtle facial signs of emotion. This is because unlike deception
research, where the presence of an emotion expression is not a guarantee of a lie or
truth, the presence or absence of the emotion expression is what is being judged. A
training technique that allows a HCP to practice with feedback, over many days,
should produce a sizable improvement in their abilities to recognize subtle signs of
emotions. But this would need to be determined empirically.

267
Implications

This work has clear implications for the HCP. It suggests that HCPs should learn
how to visually distinguish various emotions through the facial expression in order to
make a more accurate assessment of their patients' current emotional states, or track
their treatment progress. However, in order to accomplish either goal, a HCP must
first know the specific appearance changes in these different facial expressions. The
most accurate way to do this is to learn the Facial Action Coding System (F ACS;
Ekman & Friesen, 1978)2. FACS is a comprehensive scoring system for all facial
muscle actions, and not just those presumed to be related to emotion. What FACS
has shown is that there seem to be basic elements for "core" facial expressions of
each emotion (Ekman, 1993). The exact combinations of muscle movements that
compose these facial expressions is by no means settled, but the following are the
elements found in the facial expressions used in the universality research. The
specific muscle will be mentioned when the movement can be attributed to more
than one muscle by an untrained observer.
Anger: Eyebrows down (corrugator), upper eyelids up, lower eyelids up
(orbicularis oculi, pars medialis), and lips rolled in or tight (orbicularis oris,
pars medialis).
Contempt: One lip corner pulled in (buccinator) and upward slightly
(zygomatic major; this is the only asymmetrical expression).
Disgust: Nose wrinkle, and upper lip raise (levator labii superioris).
Fear: eyebrows up and together (frontalis medialis and lateral is, and
corrugator), upper eyelids up, mouth stretched (risorius).
Sad: Eyebrows up in the middle (frontalis medialis), eyes crinkled
(orbicularis oculi, pars lateralis and medialis), lip corners down
(triangularis), sometimes chin boss up (mentalis).
Surprise: Eyebrows up (frontalis medialis and lateral is), mouth open.
~: Lip corners up (zygomatic major), and eyes crinkled (orbicularis
oculi, pars lateral is) .
Examples of anger, contempt, disgust, fear, sadness, and surprise are shown in
Figures 1-6 below. The expression of happiness will be used to illustrate how to
distinguish an emotional expression from another, and thus will be discussed shortly.

268
Figure I. Anger © 1996 Rutgers/ Alan Goldsmith

Figure 2. Contempt (c) 1996 Rutgers/ Alan Goldsmith

269
Figure 3. Disgust © /996 Rutgers/ Alan Goldsmith

Figure 4. Fear © /996 Rutgers/ Alan Goldsmith

270
Figure 5. Sad (1:J 1996 Rutgers/ Alan Goldsmith

Figure 6. Surprise «:" 1996 Rutgers/ Alan Goldsmith

271
Example: How to Recognize the Enjoyment Smile

It might be informative to think through how some training on improving the ability
to recognize subtle signs of emotion might go. It makes sense to start with the facial
expression of happiness, because there has been more work on the enjoyment smile
than any other facial expression. As shown in Figure 7 below, an enjoyment smile
consists of zygomatic major and orbicularis oculi, pars lateral is, the latter being the
key feature that distinguishes enjoyment smiles from other smiles. This orbicularis
oculi action creates movement of the skin above and below the eye and causes it to
be pulled in towards the eyeball. This action makes for the following changes in
appearance: a). the cheeks are pulled up; b). the skin below the eye may bag or
bulge; c). the lower eyelid moves up; d). crow's feet wrinkles may appear at the outer
corner of the eye socket; e). the skin above the eye is pulled slightly down and
inwards; and f). the eyebrows move down very slightly. A nonenjoyment smile, in
contrast, features the same movement of the lip corners as the enjoyment smile, but
does not involve the changes due to the muscles around the eyes (Ekman & Friesen,
1978). Compare Figure 7 with Figure 8 below.

272
Figure 7. Enjoyment smile (Happiness) © 1996 Rutgers/ Alan Goldsmith

Figure 8 . Nonenjoyment smile ~) 1996 Rutgers/ Alan Goldsmith

273
 There is one complication - when there is a very big or broad smile without
orbicularis oculi action, the strong contraction of the zygomatic major will produce
many of the changes produced by the action of the orbicularis oculi which are listed
above, making it seem as if orbicularis oculi was present. If it is a big smile, clues ~
through Q., above will be present regardless whether orbicularis oculi is present. In a
broad or large smile one must look just for clues ~ and f; these clues appear only
with orbicularis oculi action. Thus, whatever clues are more diagnostic depends upon
whether it is a slight or a broad smile.
A HCP is warned that the main disadvantage of relying upon orbicularis oculi
action with zygomatic major as the sole criterion for positive emotion is that up to
20% of the population can consciously contract their lateral portions of orbicularis
oculi muscles, and thus are capable of producing a false enjoyment smile (Ekman,
Roper, & Hager, 1980). We also note that smiles without orbicularis oculi action do
not necessarily connote non-positive emotions; it may be the case that zygomatic
major action without orbicularis oculi may signal low or weak levels of enjoyment
(Ruch, 1993). Further research will help elucidate this possibility.
Another disadvantage of relying upon F ACS is that to be sufficiently competent
in FACS for all facial muscle movements requires approximately 100 hours of study
(Ekman & Friesen, 1978). This may not be time effective for a HCP. We have
some recent data that suggests that training people to distinguish among facial
expressions of emotion can be done in under one day, and probably even quicker
than that (Frank & Ekman, in prep). We have trained using videotaped
microexpressions of emotion (micro because they last for 1115'h of a second) and we
can produce a 10% rise in accuracy in a fairly brief period of time. The ability to
recognize these microexpressions has been shown to correlate with the ability to
detect deception in Americans, Australians, and British (Ekman & O'Sullivan, 1991;
Frank & Ekman, 1997; Frank, 2002; Frank & Ekman, in prep).
To date no research has examined whether a HCP can spot these different smiles
in a face to face, real time encounter. However, one study has approximated these
conditions by showing groups of observers videotaped clips of women who
spontaneously showed smiles with and without orbicularis oculi action (but of equal
zygomatic major intensity). Both types of smiles occurred in response to films
designed to elicit positive emotion. This study found that observers were able to
distinguish between smiles that had orbicularis oculi action, and smiles that did not,
at significantly better than chance guessing rates (Frank, et. ai, 1993). Interestingly,
these observers were most accurate making this distinction when they judged smiles

274
of low zygomatic major intensity because it seems that lowered zygomatic major
activity increased the salience of orbicularis oculi action.
The presence or absence of orbicularis oculi action in concert with zygomatic
major also caused observers to have different impressions of the same person. In
this study observers were shown the same videotapes as in the aforementioned
experiment, and then asked simply to rate the personality characteristics of each
stimulus subject. The results showed that each stimulus subject was rated as acting
and feeling significantly more pleasant, outgoing, likable, sincere, honest, genuine,
and so forth - in other words, generally rated more positive - when they showed a
smile with orbicularis oculi action compared to when they showed a smile of the
same zygomatic major intensity, but without this action (Frank, et. aI., 1993).
Taken together, these studies suggest that it is possible to identify enjoyment
smiles in real time. The utility of making the distinction between smiles with and
without orbicularis oculi action in diagnosing, monitoring, and tracking the effects of
treatment for various clinical disorders has already been demonstrated in the studies
cited earlier. It seems reasonable to speculate that by identifying the key facial
muscles involved in the expressions of anger, disgust, fear, sadness, and surprise,
describing their appearance as was done above, we can train HCPs to improve their
accuracy at judging these expressions.

Complicating factors

We can only help a HCP by sensitizing them to the most reliable indicators of a
patient's emotional state. However, it is often not this simple. There are contextual
clues, vocal clues, and so forth that will add a richness to the subtleties of a patient's
emotional reaction. Merten (this volume) recognizes this and discusses in more
detail some procedures for assessing a patient's emotional state within a context, by
examining the direction of gaze and content of speech. However, there is still a
significant body of research suggesting that facial expression alone is a good
indicator of emotional state (see review by Ekman, 1993). Studies that did not find
such a relationship used different methods to examine the relationship. For example,
Merten (1997) examined the relationship between enjoyment smiles and self report
of positive emotion and found no relationship. However, this study examined the
number of smiles over 20 minutes of behavior, where a variety of recency and
possibly primacy effects may have biased the emotional self-report. Studies that did
find a relationship between self-report and expression tend to use tighter time
controls; that is, they examine the relationship between expression and self-report

275
 over one or two minute periods (e.g, Ekman, et ai, 1990). Thus, emotional
information expressed through the face can be an important source of information.
Kappas (this volume) raises a logical argument as to why researchers are not at
the point at which they can confidently infer anything from a particular facial
expression, because one can fake an emotional expression that is not felt, or squelch
an emotional expression felt, amongst other alternatives. Given all the possible
combinations of facial expressions and internal states, he argues that it is difficult to
fully support the view that certain facial expressions are manifestations of internal
emotional states. This is a serious challenge to the work presented earlier. However,
this strong argument does not make the data showing that facial expressions of
emotion are strong predictors of recovery from depression and death of a loved one,
child abuse, marital discord, juvenile delinquency, psychopathology, schizophrenia,
deception, Type A personality, and so forth (see above). It also does not change the
findings that researchers have found correspondence between facial expressions of
emotion and self-report, ANS, and eNS responses. It also does not change the
findings that facial expressions of emotion activate specific areas of the brain.
Kappas' does raise a good point about how strong this relationship might be, and his
recommendation that people not see a simple one-to-one correspondence between a
facial expression of emotion and an internal emotional state, as human beings are
capable of managing their expressive behavior. So some caution should be
exercised.
Thus I argue it is still useful to know when a patient is experiencing an emotion,
and the facial expression is one of the most fruitful avenues. However, a Hep must
still interpret the cause of that emotion accurately. For example, a Hep notes an
expression of anger when a patient is discussing his medication. What does that
mean? Is the patient angry at having to take the medication? Is he or she angry at the
side effects? Is he or she angry at the event or illness that caused them to have to take
the medicine in the first place? Is he or she angry at the cost of the medication, or
the insensitive way in which the chemist/pharmacist treated them when having the
prescription filled? Or is he or she angry at the amount of time they spent in the
waiting room prior to meeting with the Hep? The Hep's judgment as to the trigger
of the anger would have very different implications for the treatment program.
Another complication involves the motivation for the patient to mask his or her
emotional reactions. It may prove to be more instructive to train Hep's on the
muscles involved in masking emotions. A nonenjoyment smile in conjunction with
other muscles implicated in sadness, anger, or disgust would be a start. Other
muscles may involve the masseter muscle (that clenches the jaw), which can be

276
employed to attempt control over the lower half of the face (e.g., Ekman & Friesen,
1975). One can often see this muscle flex when a patient grinds his or her teeth. At
this point, there is little research on the muscles involved in masking or controlling
expressions.

Future uses

One fascinating implication of this smile research on clinical practice is that it

appears that one can produce the physiology of positive emotions by asking subjects
to pose a smile with orbicularis oculi action. One of the difficult aspects of emotions
is that they are unbidden and difficult to control. Many affective disorders are
characterized by this problem - unrelenting sadness in depression, or fear in
generalized anxiety disorder. One can imagine the utility of a simple, drug-free
technique that would enable a clinician to assist patients to regulate their moods
more effectively - for example, to help stabilize a depressed patient over the typical
2-3 week period when they are being titrated for a tricyclic antidepressant or
fluoxetine.
The instructions for making the orbicularis oculi action of an enjoyment smile are
simple: instruct patients to "raise your cheeks", and then "part your lips and let your
lip corners come up" (Ekman & Davidson, 1993). It may be easier if a HCP first
asks his or her patients to squint as hard as they can, or tighten their eyes closed as
hard as they can. They should be told that they are using the correct muscle when
they do that; now they should be instructed to contract that same muscle but less
strongly, so their eyes stay open. Then, add in a smile. The patient should hold the
pose for at least 20 seconds. It is important that the HCP coaching the patient insure
that the smile is posed properly; FACS trained individuals are best able to do this,
because they will be more familiar with the criteria for orbicularis oculi pars lateral is
action described earlier.
Be warned that there has been no research on this technique as a therapeutic tool.
Researchers do not know how long lasting these effects are. They do not know
whether they will even work on a clinical population. They do not know if they will
work with patients on medication. They do not know if this technique may
exacerbate their emotional problem by causing mood swings. They do not know
whether this technique can overcome the powerful emotions felt by groups such as
depressives or the anxious. For example, orbicularis oculi has been implicated in
crying, and a HCP that manipulates this muscle might generate more sadness rather
than happiness. However, researchers do know that the patient would have to pose

277
the enjoyment smile accurately. They do know that not all patients can deliberately
contract their orbicularis oculi, pars lateralis. They do know that it appears as if this
technique would treat only the symptom, and not the problem that may underlie or
cause the negative emotions in the first place. Thus, much research remains to be
done, both inside and outside the clinic, before hailing this as an effective therapeutic
technique.
Advances in technology will aid HCPs by allow.ing them to quickly, validly, and
reliably observe facial expressions without distracting them from their interview.
For example, computer based analysis programs have been developed to assess
various facial actions associated with psychopathology (Johnston & Katsikitis, 200 I;
Katsikitis, Pridmore, & Marzullo, 1999; Katsikitis, this volume). Future work will
refine these techniques, using dynamic images of facial expressions, culminating in
programs that will be able to analyze these expressions in real-time to assess the
specific emotional state of the patient. Thus, in the future we will be able to assess
specific facial muscle movements at a much faster, and more reliable way, without
causing awareness on the part of the person being analyzed. This will have the effect
of making facial expression more accessible to more HCPs, which can only help the
delivery of these techniques to the clinic, or even provide a quicker answer to the
question as to whether this is a fruitful avenue of pursuit.

CONCLUSION

It seems apparent that HCPs should attend more to facial clues. There is clear
evidence that there is information in the facial expression that can be used to gauge
the emotional state of the patient. It is also clear that these facial expressions have
been linked to a variety of health and social outcomes. It is also clear that this
information is underused by HCPs, given the paucity of work looking at HCP
abilities to recognize emotions in their patients. But the author is optimistic that
effective training of HCPs on how to spot these subtle signs of emotion can be
delivered. An in depth comprehension of these facial expressions can not only assist
in the rapport building process, but also boost the accuracy of diagnoses, assist in
monitoring of treatment, aid in learning how to ask sensitive questions in a way that
reduces a patient's desire to deceive, and so forth. In other words, focusing on what
we can know and be trained about facial expression of emotion, we can employ it to
help HCPs 'get to know' their patients in a way that allows more effective delivery
of health care.

278
 AFFILIATION

Department o/Communication
Rutgers-The State University 0/ New Jersey
Email: mgfrank@Scils.rutgers.edu

NOTES.
I. One could argue that these expressions of emotion would deprive an individual of a competitive
advantage like the element of surprise prior to an attack (Fridlund, 1994). However, if one thinks in
terms of the survival of the group, and that most of the members of the group share genetic material
(Diamond, 1998), then it seems apparent that the most effective way to insure the transmission of those
genes to the next generation would be cooperation and reduced conflict within the group.
2. Information on FACS can be obtained by writing Paul Ekman, Human Interaction Lab,
University of California, 401 Parnassus Ave., San Francisco, CA 94143.

REFERENCES.
Anderson, A. K., & Phelps, E. A. (2000). Expression without recognition: Contributions of the human
amygdala to emotional communication. Psychological Science, II, 106-111.
Barrett, R., & Katsikitis, M. Foreign faces: A journey to the land of EEPICA. In M. Katsikitis (Ed.), The
human face: Measurement and meaning (pp. 1-28). Dordrecht: Kluwer Academic Publishers.
Bavelas, J. B. , Black, A., Lemery, C. R., & Mullett, J. (1986) "I show how you feel": Motor mimicry as a
communicative act. Journal of Personality and Social Psychology, 50, 322-329.
Berenbaum, H., & Oltmanns, T F. (1992) Emotional experience and expression in schizophrenia and
depression. Journal of Abnormal Psychology. 101,37-44.
Rirdwhistell, R. L. (1970). Kinesics and context. Philadelphia: University of Pennsylvania Press.
Blair, R. J. R., Morris, l S., Frith, C. c., Perrett, D.I., & Dolan, R. J. (1999) Dissociable neural responses to
facial expressions of sadness and anger. Brain, 122, 883-893.
Blurton-Jones, N. G., (1967). An ethological study of some aspects of social behavior of children in nursery
school. In D. Morris (Ed.), Primate ethology. London: Weidenfcld and Nicolson.
Bonanno, G .A., & Keltner, D. (1997) Facial expressions of emotion and the course of conjugal
bereavement. Journal of A bnormal Psychology, 106. 126- I 37.
Boucher, J. D., & Carlson, G. E. (1980). Recognition of facial expression in three cultures. JournalofCros.\"-
Cultural Psychology, II, 263-280.
Brodal, A. (1981) Neurological anatomy: In relation to clinical medicine. New York Oxford University
Press.
Bruner, J. S., & Tagiuri, R (1954). The perception of people. In G. Lindzey (Ed.), Handbook of Social
Psychology. Reading, MA: Addison-Wesley, 634-654.
Bugental, D. B. (1986). Unmasking the "polite smile": Situational and personal detenninants of managed
affect in adult child interaction. Personality and Social Psychology Bulletin, 12, 7-16.
Calder, AJ., Young, A.W., Rowland, D, Perrett, D.I., Hodges, lR., & Etcoft~ N.t. (1996) Facial emotion
recognition after bilateral amygdala damage: Differentially severe impairment of fear. Cognitive
Neuropsychology, 13,699-745.

279
Chevalier-Skolnikoff. S. (1973). Facial expression of emotion in non-human primates. In P. Ekman (Ed .),
Darwin andfacial expression: A century of research in review. New York: Academic Press.
Darwin, C. (1872/1998). The expression of the emotions in man and animals. New York: Oxford. (3,d Ed ., w/
comments by P. Ekman).
Davidson, R.J. (1984). Affect, cognition and hemispheric specialization. In C. E. Izard, 1. Kagan, & R.
Zajonc (Eds.), Emotion, cognition, and behavior (pp . 320-365). New York: Cambridge University Press.
Davidson, RJ. (1992). Emotion and affective style: Hemispheric substrates. Psychological Science, 3,39-43.
Davidson, R. 1., Ekman, P., Saron, c., Senulius, J., & Friesen, W. V. (1990). Approach-withdrawal and
cerebral asymmetry: Emotional expression and brain physiology I. Journal of Personality and Social
Psychology, 58, 330-341.
Davidson, R.J., & Tomarken, AJ. (1989). Laterality and emotion: An electrophysiological approach. In F
Boller & J Grafman (Eds.), Handbook of neuropsychology (pp. 419-44 I). Amsterdam: Elsevier.
DeMyer ,W. (1980). Technique of the neurological examination. New York : McGraw-Hil I.
Diamond, J. (1998). Guns, germs, & steel. London: Vintage.
Efron, D. (1941). Gesture and environment. New York: King's Crown.
Eibl-Eibesfeldt, I. (1970). Ethology, the biology of behavior. New York: Holt, Rinehart, & Winston.
Eibl-Eibesfeldt, I. (1972). Similarities and differences between cultures in expressive movements. In R. A.
Hinde (Ed.), Nonverbal communication (pp . 297-311). Cambridge: Cambridge University Press.
Eibl-Eibesfeldt, I. (1973). The expressive behavior of the deaf-and-blind born. In M. von Cranach, & I. Vine
(Eds.), Social communication and movement (pp. 163-194). San Diego: Academic Press.
Ekman, P. (1972). Universal and cultural differences in facial expressions of emotion. In 1. K. Cole (Ed .),
Nebraska symposium on motivation 1971 (pp. 207-283) Lincoln, NE: University of Nebraska Press.
Ekman, P. (1977). Biological and cultural contributions to body and facial movement. In J. Blacking (Ed .),
Anthropology of the body (pp . 34-84). London : Academic Press.
Ekman, P. (1979). About brows: Emotional and conversational signals. In M. von Cranach, K. Foppa, W.
Lepenes, & D. Ploog (Eds.), Human ethology .(pp. 169-248). Cambridge: Cambridge University Press.
Ekman, P. (1989). The argument and evidence about universals in facial expressions of emotion. In H.
Wagner & A. Manstead (Eds.), Handbook of psychophysiology: The biological psychology of the
emotions and social processes (pp . 143-164). London : John Wiley.
Ekman, P. (1992). Telling lies. New York: Norton.
Ekman, P.(1993). Facial expression and emotion. American Psychologist, 48, 384-392 .
Ekman, P. (1994). Strong evidence for universals in facial expressions: A reply to Russell's mistaken
critique. Psychological Bulletin, 115, 268-287.
Ekman, P., & Davidson, R. 1. (1993) Voluntary smil ing changes regional brain activity. Psychological
Science, 4, 342-345.
Ekman, P., Davidson, R. 1., & Friesen, W. V. (1990). The Duchenne smile: Emotional expression and brain
physiology II. Journal of Personality and Social Psychology, 58, 342-353.
Ekman, P., & Friesen, W. V. (1971) Constants across cultures in the face and emotion. Journal of
Personality and Social Psychology, 17, 124-129.
Ekman, P. , & Friesen, W. V. (1974). Detecting deception from the body or face. Journal of Personality and
Social Psychology, 29, 288-298.
Ekman, P., & Friesen, W. V. (1975). Unmasking thejace . Englewood ClifTs, NJ: Prentice-Hall .
Ekman, P., & Friesen, W. V (1978). The facial action coding system. Palo Alto: Consulting Psychologi sts
Press.
Ekman, P., & Friesen, W V (1982). Felt, false, and miserable smiles. Journal of Nonverbal Behavior, 6,
238-252.

280
Ekman, P., Friesen, W.V., & Ancoli, S. (1980). Facial signs of emotional experience. Journal of Personality
and Social Psychology, 39, 1125-1134.
Ekman, P., Friesen, W. V., & Ellsworth, P. C. (1972). Emotion in the human face. Cambridge: Cambridge
University Press.
Ekman, P., Friesen, W. V., O'Sullivan, M., Chan, A., Diacoyanni-Tarlatzis, I., Heider, K., Krause, R.,
LeCompte, W. A., Pitcairn, T., Ricci-Bitti, P. E., Scherer, K., Tomita, M., & Tzavaras, A. (1987).
Universals and cultural differences in the judgments of facial expressions of emotion. Journal of
Personality and Social Psychology, 53, 712-717.
Ekman, P., Matsumoto, D., & Friesen, W.V. (1997). Facial expression in affective disorders. In P. Ekman &
E.L. Rosenberg (Eds.). Whatthe face reveals (pp. 331-341). New York: Oxford University Press.
Ekman, P., & O'Sullivan, M. (1991). Who can catch a liar? American Psychologist, 46, 913-920.
Ekman, P., Roper, G., & Hager, 1. (1980). Deliberate facial movement. Child Development, 51, 886-891.
Ekman, P., Sorenson, E. R., & Friesen, W. V. (1969). Pan-cultural elements in facial displays of emotions.
Science, 164, 86-88.
Fox, N.A., & Davidson, R.J. (1988). Patterns of brain electrical activity during facial signs of emotion in 10-
month old infants. Developmental Psychology, 24, 230-236.
Frank, M. G. (2002). Detecting deception and emotion by Australians and Americans. Manuscript under
review.
Frank, M. G., & Ekman, P. (1993). Not all smiles are created equal: The differences between enjoyment and
nonenjoyment smiles. Humor: The International Journal of Research in Humor, 6, 9-26.
Frank, M. G., & Ekman, P. (1997). The ability to detect deceit generalizes across different types of high stake
lies. Journal of Personality and Social Psychology, 72, 1429-1439.
Frank, M. G., & Ekman, P. (in prep). Microexpressions and detecting deception by British police.
Manuscript in preparation.
Frank, M.G., Ekman, P., & Friesen, W. V. (1993). Behavioral markers and recognizability of the smile of
enjoyment. Journal of Personality and Social Psychology, 64, 83-93.
Frank, M.G., & Feeley, T. H. (under review). To catch a liar: Challenges for research in lie detection
training.
Frank, M. G., & Stennett, 1. (200 I) The forced choice paradigm and the perception offacial expressions of
emotion. Journal of Personality and Social Psychology, in press.
Fridlund, A. 1. (1991). Sociality of solitary smiling: Potentiation by an implicit audience. Journal of
Personality and Social Psychology, 60, 229-240.
Fridlund, A. J. (1994). Humanfacial expression: An evolutionary view. San Diego: Academic Press.
Friesen, W. V. (1972). Cultural differences infadal expressions in a social situation: An experimental test of
the concept of display rules. Unpublished doctoral dissertation, University of California, San Francisco.
Frijda, N. (1986). The emotions. Cambridge: Cambridge University Press.
Gottman,1. (1994). Why marriages succeed or fail. New York: Fireside.
Haidt, J., & Keltner, D. (1999). Culture and facial expression: Open-ended methods find more expressions
and a gradient of recognition. Cognition & Emotion, 13, 225-266.
Izard, C. E. (1971). Theface of emotion. New York: Appleton-Century Crotts.
Izard, C. E. (1977). Human emotions. New York: Plenum Press.
Izard, C. E. (1994). Innate and universal facial expressions: Evidence from developmental and cross-cultural
research. Psychological Bulletin, 115, 288-299.
Izard, C. E., & Haynes, 0 M. (1988). On the form and universality of the contempt expression: A challenge
to Ekman and Friesen's claim of discovery. Motivation and Emotion, 12, 1-16.
Karnosh, L. J. (1945). Amimia or emotional paralysis of the face. Diseases of the Nervous System, 6, 106-
108.

281
Katsikitis, M. (2002). FACEM: The Facial Expression Measurement system. In M. Katsikitis (Ed.), The
human face: Measurement and meaning (pp. 99-118). Dordrecht: Kluwer Academic Publishers.
Katsikitis, M., & Pilowsky, I. (1991). A controlled quantitative study of facial expression in Parkinson's
disease and depression. Journal of Nervous and Mental Disease, 179, 683-688.
Katsikitis, M., Pridmore, S., & Marzullo, M. (1999). The Facial Expression Measurement system in the
assessment of the efficacy of transcranial magnetic stimulation in the treatment of depression. European
Review ofApplied Psychology, 49, 123-128.
Keltner, D. (1995). The signs of appeasement: Evidence for the distinct displays of embarrassment,
amusement, and shame. Journal of Personality and Social Psychology, 68, 441-454.
Keltner, D., Moffitt, T., & Stouthamer-Loeber, M. (1995). Facial expressions of emotion and
psychopathology in adolescent boys. Journal of Abnormal Psychology, 104,644-652.
Kinseth, L. M. (1989). Nonverbal training for psychotherapy: Overcoming barriers to clinical work with
client nonverbal behavior. Clinical Supervisor, 7, 5-25.
Klineberg, O. (1940). Social psychology. New York: Henry Holt
Knapp, M.L., & Hall, lA. (1997). Nonverbal communication in human interaction. New York: Harcourt-
Brace.
Krause, R., Steimer, E, Sanger-Alt, C , & Wagner, G. (1989). Facial expressions of schizophrenic patients
and their interaction partners. Psychiatry, 52, 1-12.
Kraut, R. E, & Johnston, R. E.(1979). Social and emotional messages of smiling: An ethological approach.
Journal of Personality and Social Psychology, 37, 1539-1553.
Kring, A. M., & Neale, l M. (1996). Do schizophrenics show a disjunctive relationship among expressive,
experiential, and psychophysiological components of emotion? Journal of Abnormal Psychology 105,
249-257.
Landis, C (1924). Studies of emotional reactions : II. General behavior and facial expression. Journal of
Comparative Psychology, 4, 447-509.
LeBarre, W. (1947). The cultural basis of emotions and gestures. Journal of Personality. 16. 49-68.
Levenson, R. W., Carstensen, L.L., Friesen, W.V., & Ekman, P. (1991). Emotion, physiology, and expression
in old age. Psychology and Aging. 6,28-35.
Levenson, R .W , Ekman, P., & Friesen, W V (1990). Voluntary facial action generates emotion-specific
autonomic nervous system activity. Psychophysiology, 27, 363-384.
Levenson, R. W.. Ekman, P , Heider, K., & Friesen, W V .(1992) Emotion and autonomic nervous system
activity in the Minangkabau of West Sumatra. Journal of Personality and Social Psychology. 62. 972-
988.
Liebennan, P. (1992). On Neanderthal speech and Neanderthal extinction. Current Anthropology, 33. 409-
410.
Lutz, C, & White, G. M. (1986). The anthropology of the emotions. Annual Review of Anthropology. 15,
405-436.
Machado, P P ., Beutler, 1.. E, & Greenberg, L S. (1999). Emotion recognition in psychotherapy : Impact of
therapist level of experience and emotional awareness . Journal of Clinical Psychology, 55,39-57.
Matsumoto, D.(1990). Cultural similarities and differences in display rules. Motivation and Emotion. 14,
195-214 .
Matsumoto, D. , & Ekman, P. (1989). American-Japanese cultural ditlcrences in intensity ratings of facial
expressions of emotion. Motivation and Emotion, 13, 143-157.
Mead, M. (1975). Review of Darwin and facial expression. Journal of Communication. 25. 209-213.
Mehrabian, A. (1972). Nonverbal communication. Chicago: Ald ine/Atherton.
Meihlke, A. (1973). Surgery of the facial nerve. Philadelphia: Saunders .

282
Merten, J. (2002). Context-analysis offacial-affective behavior in clinical populations. [n M. Katsikitis (Ed.),
The human/ace: Measurement and meaning (pp. 131-147). Dordrecht: Kluwer Academic Publishers.
Merten, J. (1997). Facial affective behavior, mutual gaze, and emotional experience in dyadic interactions.
Journal 0/ Nonverbal Behavior, 21, 179-20 I.
Morris, J.S., Frith, C.D., Perrett, 0.1., Rowland, D., Young, A.W., Calder, AJ., & Dolan, RJ. (1996). A
differential neural response in the human amygdala to fearful and happy facial expressions. Nature, 383,
812-815.
Myers, R. E. (1976). Comparative neurology of vocalization and speech: Proofofa dichotomy. Annual
Review o/the New York Academy o/Sciences, 280, 745-757.
Ormel, J., & Tiemens, 8. (1995). Recognition and treatment of mental illness in primary care: Towards a
better understanding of a multifaceted problem. General Hospital Psychiatry, 17, 160-164.
Ortony, A., & Turner, T. J. (1990). What's basic about basic emotions? Psychological Review, 74, 315-341 .
Patrick, C. (1994). Emotion and psychopathy: Startling new insights. Psychophysiology, 31, 319-330.
Phillips, M.L., Young, A.W., Senior, c., Brammer, M., Andrew, c., Calder, AJ., Bullmore, E.T., Perrett,
0.1., Rowland, D., Williams, S.C.R., Gray, JA, & David, AS. (1997). A specific neural substrate for
perceiving facial expressions of disgust. Nature, 389,495-498.
Plutchik, R (1962). The emotions: Facts, theories, and a new model. New York: Random House.
Prkachin, K. M. (1992). The consistency of facial expressions of pain: A comparison across modalities. Pain,
51, 297-306.
Rinn, W. E. (1984). The neuropsychology of facial expression: A review of the neurological and
psychological mechanisms for producing facial expressions. Psychological Bulletin, 95, 52-77.
Rosenberg, E.L., & Ekman, P. (1994). Coherence between expressive and experiential systems in emotion.
Cognition and Emotion, 8, 201-229.
Rosenberg, E. L., Ekman, P, & Blumenthal, J.A (1998) Facial expression and the affective component of
cynical hostility in male coronary heart disease patients. Health Psychology, /7, 376-380.
Ruch, W. (1993). Exhilaration and humor. In M. Lewis & J.M. Haviland (Eds.), The handbook o/emotion
(pp. 605-616). New York: Guilford Publications.
Russell, J. A (1994). [s there universal recognition of emotion from facial expression? A review of cross-
cultural studies. Psychological Bulletin, 115, 102-141
Tomkins, S. S. (1962). Affect, imagery, consciousness (Vol. I, The positive affects). New York: Springer.
Tomkins, S. S. (1963). Affect, imagery, consciousness (Vol. 2, The negative affects). New York: Springer.
Tomkins, S. S.,& McCarter, R. (1964). What and where are the primary affects? Some evidence for a theory.
Perceptual and motor skills, 18, 119-158.
Toobey, J., & Cosmides, L. (1990). The past explains the present: Emotion adaptations and the structure of
the ancestral environment. Ethology and Sociobiology, 11, 375-424.
Tschiassny, K. (1953). Eight syndromes offacial paralysis and their significance in locating the lesion.
Annual Review 0/ Otology.. Rhinology, and Laryngology, 62, 677-691.
Webb, N. C. (1974). The use of myoelectric feedback in teaching facial expression to the blind . American
Foundation/or the Blind Research Bulletin, 27, 231-262.
Warren, J., & Thompson, P. (2002). Neurology of human facial expression. In M. Katsikitis (Ed.), The
human/ace: Measurement and meaning (pp. 63-96). Dordrecht: Kluwer Academic Publishers.
Whalen, PJ., Rauch, S.L., Etcoff, N.L., Mclnerpey, S.c., Lee, M.B., & Jenike, MA (1998) . Masked
presentations of emotional facial expressions modulate amygdala activity without cxplicit knowledge.
The Journal 0/ Neuroscience, 18,411-418 .
Woodworth, R.S. (1938). Experimental psychology. New York: Henry Holt.

283
SUBJECT INDEX

accuracy, 176, 182 behaviour, 5, 18

action, 64, 72, 83, 90, 99, 102, 107,
114,116
action, 9, 12
activation, 216, 219, 220-231
actors, 171, 173, 174
adaptive, 240, 253
adolescent, 140, 146
aduh,30-36,46,47
affect, 119-130, 139-145
affective, 215-234
age, 171-181, 189-194,201-212
aggression, 179
alcohol, 192
amygdala, 153-166
anatomy, 17,26,30,32,46,47,65,
92-98
angry, 152,216,219,221,
antecedent, 235, 237
anthropology, 29, 45, 110
antisocial, 180, 183
anxiety, 119, 189,202-205,214
appraisal, 12,216-234
approach, 222, 228, 231- 234
arousal, 236
asymmetries, 250
attention, 170, 177, 184, 185
attributions, 247
auditory, 152
avoidance, 222
basic, 134, 149-159, 173, 182,215-
232
biological, 11, 19,29,41- 45,50,58,
61-65
blend, 10, 14,24
blind, 169-186
blink reflex, 68, 98
blood pressure, 189
body, 169, 171, 184, 192,212
brain, 149, 150- 155, 163- 166
brows, 192, 193
categorical, 201
causal, 184
cheek,49,220,224
child, 31, 32, 36,42,46
chimeric, 157, 158, 161
chin, 192
classification, 64, 109, 115, 117, 118
clusters, 237
cogn~ion,235,249,252,253
componential, 237, 247, 249, 253
computer, 99-105, 117, 118,240,
243, 251-253
configurations, 13,236-243
conflict, 171
conscious, 219,226,231
construct, 49,50,56,57,58,65
contempt, 136-142, 156
contractions, 69, 71, 73
cross-cultural, 21, 173, 187
crow's feet, 226
crying, 192, 197,205,207,211
deaf, 170, 171, 179, 183-187
decode, 169, 171, 177
deficit hypothesis, 170, 171, 174, 183
deficit, 120- 130
dementia, 126
depression, 119, 189,204,205,247-
252
design, 14, 17,21,22
developmental, 170, 185, 229
diagnostic, 216, 218, 221, 227, 230
differences, 6, 14, 18,23,25,28
dimensional, 17, 100, 110
dimensions, 30, 37-39, 216, 222-225,
231
discrete, 222-231, 236, 237, 252
disgust, I, 17, 77, 87,95,98, 139,
141,142,217,230,236
display rules, 198,217,226,229,
231,233
display, 10, 14,20,77,82,83,87
distance, 106, 108, 112, 114, 115
distress, 156, 189, 191, 192,208,209,
210
dominant, 22, 24, 157, 158, 161
Duchenn~ 136, 138, 141, 143, 144,
226-229
duration, 192
dyadic, 131, 132, 135-139, 141-147
dyads, 133-139, 146
ecology, 229
EMFACS, 133, 134, 143
emotion specific, 121, 124, 125
emotional expression, 2- 6, II, 14,
17,64,79,81- 85, 93,119
emotional intelligence, 177, 179
encoders, 173, 176
286
emotional intelligence, 177, 179
enjoyment, 156,218,229,232
ethnographic, 3, 5, 6, 17, 19, 23
ethology, 99
evolution, 29-35, 45-49,53,61,69
evolutionary, 5-8, 45-51, 65, 90, 222
experimental, 2, 20,81, 120,239-247
expressive, 64-69, 91, 138, 141,238
eyebrows, 243, 245
eyelids, 192, 193, 208
eyes,49,63-71,89,193,207-209
face scales, 201, 203, 204, 206
facial electromyographic, 220, 224
facial emotion, 119-130
facial nerve, 65, 66, 73-75, 93, 96
FACS, 127, 191,212,237,239,242-
244,250,251
female, 171-177, 181, 182
feminism, 3, 7
forced-choice, 152, 154
forehead, 49, 58, 65, 192, 208
forensic, 41-48
frown, 216, 219, 221
frowning, 66, 69
fundamental, 8-11,15-19,26,108-
116,236
game, 220, 229-233, 240-245
gazing, 132-139, 145
gender, 174, 177, 181, 198
grimace, 195, 198,208
head,243,250
healthy, 132, 138-145
hearing impaired, 169-172, 183, 185
heart rate, 220, 226
hemispheres, 71, 75, 77, 85, 86
hermeneutic, 50, 57
history, 49, 50, 53, 57-61
human, 29-47, 50-59, 63-69, 75-98
hostile, 132, 140, 150-153, 184
identification, 29, 42- 48
identity, 42, 65,85,99,119,122,
153, 165, 184
images, 17, 20, 21
impaired, 169-172, 184
infants, 189-199,209-214
innate, 10-13,20,156, 165
intensity, 134, 174, 195-214,246
interaction, 227, 238-241, 248- 251
interest, 138, 150, 156, 158
interpersonal, 230, 235, 248
intrapersonal, 230
involuntary, II, 18, 26, 131, 208
joy, 137, 138, 139,241
judgment, 12-17, 116, 180-190
juveniles, 172, 176, 179-185
Korsakoff syndrome, 126
laboratory, 218, 230
language, 1- 9,14-19,26,103
laughter, 78-84, 92-98
learned, 10, 12, 14, 18
line drawing, 10 I-II 0
lip, 38, 41
longitudinal, 179
madness, 56, 57, 60, 61
Major Depression, III, 112
maladaptive, 131, 133, 142-144
287
maladaptive, 247, 248
male, 133-139, 171-183
management, 189,203,211,214
markers, 229, 232
masking, 134
mastication, 226
mathematical, 100-108, 115-118
MAX, 190
measurement, 13, 100, 113-116
medicine, 55, 56, 62, 110
mental, 131-139, 140, 145
methodological, 81, 89, 120-127,
221,239
micro momentary, 238, 249, 250
mimetic, 65, 69, 73, 78, 90
mimicry, 127, 143
morphology, 35, 38, 44
mouth, 63-76,101-108,192-198,
207-209
muscle, 107, 114, 133,219- 226
muscular, I, 12, 17,64
musculature, 13, 191, 192
narration, 13 I
nasolabial, 192, 195, 209
nature, 9, 13, 15, 19, 22, 25
nervous system, 189, 194
neural network, 155
neurological, 197, 214
neurology, 90, 97, 98, 110
neuropsychological, 120, 150, 162,
170
neutral, 103, 108, 109, 110
NFCS, 191, 197
non-intrusive, 107, 115
non-pharmacological, 193
nose, 49, 192-196,207-209
numerical taxonomy, 109, 110 psychodynamic, 133, 143
psychophysiology, 216, 231, 232
observable, 218, 223 psychosomatic, 133, 139
observation, 190, 198-199,208-209 psychotherapeutic, 131, 133, 137,
141,147
paediatric, 189, 190, 199,201-209, puberty, 36, 38
214
pain, Ill, 113, 117, 189-214 quantification, 99, 102-118
parents, 140 quantitative, 152, 165
Parkinson's disease, 67, 84-87, 95, questionnaire, 205
111-118
pathological, 64, 71, 78, 81- 85, 90, race, 189
96,98 reappraisal, 240, 245, 247
patient, 136, 140-146 recognition, 1, 8, 11- 20
perception, 149-171, 186, 215-222 relationship, 131-147
perceptual, 86, 87, 89, 91, 94 relaxation, 74
pharmacological, 193,210 religious, 49, 51
photographs, 1,12-16, 23, 108,118, respiration, 189,226
156,163,177,201,206,236,244 rigidity, 192
physiognomy, 52-59, 62
physiological, 29, 51, 55, 63, 81, 189, scenario, 173
193, 211, 222-234, 243 schematic, 170, 240
physiotherapeutic, 112 schizophrenia, 70, 84, 97, 111-140
pleasant, 222-224, 241, 245 self-regulation, 139
population, 35-40 self-report, 190, 197, 199,200-220,
posed,20, 77, 81, 84,93 231,234
positive, 138-145 semaphore, 49-59, 65
pre literate, 63 sender, 135, 139
premature, 194, 214 sensory, 170-179, 183, 184
primary, 141 shame, 156
primitive, 152 similarity, 23
prosopagnosia, 86, 123 skeleton, 31, 32, 37, 39
prosopon, 51, 53, 57, 58,65 skin conductance, 220
prototypical, 155-163, 173,236,239, sneer, 217, 230
244 social, 131-146,221-234
protruding tongue, 192 socioeconomic status, 189
psychiatry, 110 species, 31, 33, 38

288
speech, 30,47,66,77,99, 135, 139, universality, 2, 5,13,16,17,24,109
145,226 unpleasant, 222, 223, 224
split-screen, 132, 133
spontaneous, II, 131, 154, 165, 235- valence, 120, 123,216,222-231
244 verbal, 173, 175, 190, 199,214,237-
startle, 192 244
stimuli, 222, 223 video, 171
stimulus, 171-187 visual analogue, 204
subjective, 235, 238, 241-245 visual, 77-98,119,123,130, 149-
154, 161-187
temporal, 239, 243, 247 vocal, 169-187,243, 250, 251
therapist, 133-144 volitional, 71, 78, 81, 84, 90, 95,
TMS, 114 voluntary, 11, 15, 177,226,233
Tourette's syndrome, 68, 71
training, 100, 107, 115 Western, 1,3,5,7,8,12,21,24,25
worry, 241
universal, 2-26, 77,173,187-190,
208,236 zygomatic, 37, 38
universalists, 6

289