2011 1

31

English Education in China

No. 1, 2011
Serial No. 31

1 2

1. 1000292. 510275

H030

Gunther Kress Theo van Leeuwen

systemic functional semiotics

systemic
functional-multimodal discourse analysis Sigrid Norris

multimodal interactional analysis

1
Halliday 1994Halliday &
Matthiessen 2004Martin 1992Halliday 1978Hodge & Kress 1988
van Leeuwen 2005asemiotic modeKress & van Leeuwen
200120

Halliday 19782
Halliday 1978

Halliday 1978183
realization

stratification
metafunction
1

2011 1
31

English Education in China

No. 1, 2011
Serial No. 31

Matthiessen 20073-4

Halliday & Hasan 198517

Kress & van Leeuwen 2006OToole 1994van Leeuwen 1999Martinec
2000
Halliday 1978Halliday &
Hasan 1985Martin 1992

2
OToole1994
Kress & van Leeuwen19962006OToole1994
work
episode
figure

member199437
OToole
1994OHalloran 19992005Stenglin 2004

Kress & van Leeuwen

2006van Leeuwen 1999Knox
2007

Kress & van Leeuwen 200641-44 2003

meaning as choice
Halliday 1994
200556
1

Kress & van Leeuwen2006

topology

continuum

coding orientation

/
/
/
Kress & van Leeuwen19962006

Halliday 1994

vector
narrative representation
conceptual representation

//
2

2011 1
31

English Education in China

No. 1, 2011
Serial No. 31

Kress & van

Leeuwen 200646
/
/
Kress & van Leeuwen 200646

1 (Kress & van Leeuwen 2006149)

Bateman 20081

Lemke1998

Lemke 1998Royce 19982002OHalloran 1999Martinec

& Salway 2005Martin 20012004Kress & van Leeuwen
19962006Martin 20012004
Halliday & Hasan
1976
intersemiotic complementarity
Royce 19982002 Halliday1994
Barthes1977

Martinec & Salway 2005

OHalloran 2008Martin2001
2004
Kress & van Leeuwen2006
Martin20012004
3

2011 1
31

English Education in China

No. 1, 2011
Serial No. 31

Kress & van Leeuwen2006

Halliday 1994
naturalize

resemiotizationIedema
2003Martinec & van Leeuwen 2007
Iedema2003

Martinec & van Leeuwen2007

node
link

1
mediated discourse
Scollon 19982001Scollon & Scollon 2004Norris 20022004Norris & Jones 2005

nexus analysis
social action
Scollon & Scollon 200413
multimodal interactional analysisNorris 200220042007
Norris200410

real-time interaction
language
in use
human interactionmediated
action
2
modal density
foreground-background continuum
proxemics

gesture

head movement

posture
hand movement,layoutprint

higher-level action
lower-level action

foregrounded higher-level action

social actorawareness or attention
background(mid-ground)

2011 1
31

English Education in China

No. 1, 2011
Serial No. 31

Norris2004

Sandra Anna Anna

Anna
Norris2004

Norris2004

Gu 2006

200622

20073

saturated experience
Gu 20082009
principle of multiordinalityGu 2006:129
200624

social action over space and timeGu 2006133

/social encounter/social situation
Gu 2006131//
Gu 2006134

social situation(social actor)

/2006138

framing function
enabling functionactivity type
culturally recognized activity

task
episode
Gu 2006142

sociopsychological layer

individual behaviour layer

5

2011 1
31

English Education in China

No. 1, 2011
Serial No. 31

act
prosodic unit

2002490

Gu 2006148

content layer
medium layer

Gu
2006

Fiske 1990

habitus
Bourdieu 1977

Norris 20042007
situatedness
Gu 20022006

Kress & van Leeuwen 2001

OToole 1994Kress & van Leeuwen 19962006

Kress & van Leeuwen 2002van Leeuwen 2005b
OHalloran 19992005van Leeuwen 1999
6

2011 1
31

English Education in China

No. 1, 2011
Serial No. 31

Martinec 2000Hood 2007Stenglin 2004Martin &

Stenglin 2007Lemke 2002Djonov 2007Knox 2007
Baldry 2004Baldry & Thibault 2006Thibault 2000

Norris2007

Hispanic/Latino

Gu 2006

meaning potential

Norris2004

Norris 2004Gu 2006

Gu 2006

Thibault 2000Baldry 2004

7

2011 1
31

English Education in China

No. 1, 2011
Serial No. 31

Baldry & Thibault 20062007)

[1]. Baldry, A. P. Phase and transition, type and instance: Patterns in media texts as seen through a
multimodal concordancer [A]. In K. L. OHalloran (ed.). Multimodal Discourse Analysis [C]. London:
Continuum, 2004. 83-108.
[2]. Baldry, A. P. & P. J. Thibault. Multimodal Transcription and Text Analysis [M]. London: Equinox,
2006.
[3]. Baldry, A. P. & P. J. Thibault. Multimodal corpus linguistics (A). In G. Thompson & S. Junston (eds.).
System and Corpus: Exploring Connections [C]. London: Equinox, 2007. 164-183.
[4]. Barthes, R. Image, Music, Text [M]. London: Fontana, 1977.
[5]. Bateman, J. Multimodality and Genre: A Foundation for the Systematic Analysis of Multimodal
Documents [M]. Basingstoke: Palgrave Macmillan, 2008.
[6]. Bourdieu, P. Outline of a Theory of Practice [M]. Cambridge: Cambridge University Press, 1977.
[7]. Djonov, E. Website hierarchy and the interaction between content organization, webpage and
navigation design: A systemic functional hypermedia discourse analysis perspective [J]. Information
Design Journal, 2007, (15): 144162.
[8]. Fiske J. Introduction to Communication Studies (2nd Edition) [M]. London: Routledge, 1990.
[9]. Gu, Y. Towards an understanding of workplace discourse [A]. In C. Candlin (ed.). Research and
Practice in Professional Discourse [C]. Hong Kong: City University of Hong Kong Press, 2002.
137-185.
[10]. Gu, Y. Multimodal text analysis: A corpus linguistic approach to situated discourse [J]. Text & Talk,
2006, (26): 127167.
[11]. Gu, Y. Come to grips with video data [R]. Workshop on multimodal data-mining, Hong Kong
Polytechnic University, 8 December 2008.
[12]. Gu, Y. From real-life situated discourse to video-stream data-mining: An argument for
agent-oriented modeling for multimodal corpus compilation [J]. International Journal of Corpus
Linguistics, 2009, (14): 433-466.
[13]. Halliday, M.A.K. Language as Social Semiotic: The Social Interpretation of Language and Meaning
[M]. London: Arnold, 1978.
[14]. Halliday, M.A.K. An Introduction to Functional Grammar (2nd Edition) [M]. London: Arnold, 1994.
[15]. Halliday, M. A. K. & R. Hasan. Cohesion in English [M]. London: Longman, 1976.
[16]. Halliday, M.A.K. & R. Hasan. Language, Context and Text: Aspects of Language in a
Social-semiotic Perspective [M]. Geelong, Vic.: Deakin University Press, 1985.
[17]. Halliday, M.A.K. & C.M.I.M, Matthiessen. An Introduction to Functional Grammar (3rd Edition) [M].
London: Arnold, 2004.
[18]. Hodge, R. & G. Kress. Social Semiotics [M]. Cambridge: Polity Press, 1988.
[19]. Hood, S. Embodying language: Bringing gesture in from the paralinguistic cold [R]. Paper presented
at Semiotic margins: Reclaiming Meaning, the University of Sydney, 2007.
[20]. Iedema, R. Multimodality, resemiotization: Extending the analysis of discourse as multi-semiotic
practice [J]. Visual Communication, 2003, (2): 29-57.
[21]. Knox, J. Visual-verbal communication on online newspaper home pages [J]. Visual Communication,
2007, (6): 1953.
8

2011 1
31

English Education in China

No. 1, 2011
Serial No. 31

[22]. Kress, G. & T. van Leeuwen. Reading Images: The Grammar of Visual Design [M]. London:
Routledge, 1996.
[23]. Kress, G. & T. van Leeuwen. Multimodal Discourse: The Modes and Media of Contemporary
Communication [M]. London: Arnold, 2001.
[24]. Kress, G. & T. van Leeuwen. Color as a semiotic mode: Notes for a grammar of color [J] Visual
Communication, 2002, (1): 343-368.
[25]. Kress, G. & T. van Leeuwen. Reading Images: The Grammar of Visual Design (2nd Edition) [M].
London: Routledge, 2006.
[26]. Lemke, J.L. Multiplying meaning: Visual and verbal semiotics in scientific text [A]. In J.R. Martin & R.
Veel (eds.). Reading Science: Critical and Functional Perspectives on Discourses of Science [C].
London: Routledge, 1998. 87-113.
[27]. Lemke, J. L. Multimedia semiotics: Genres for science education and scientific literacy [A]. In M.J.
Schleppegrell & M.C. Colombi (eds.). Developing Advanced Literacy in First and Second
Languages: Meaning with Power [C]. London: Lawrence Erlbaum Associates, 2002. 21-44.
[28]. Martin, J. R. Fair trade: Negotiating meaning in multimodal texts [A]. In P. Coppock (ed.). The
Semiotics of Writing: Transdisciplinary Perspectives on the Technology of Writing [C]. Bloomington:
Indiana University Press, 2001. 311-338.
[29]. Martin, J.R. English Text: System and Structure [M]. Amsterdam: Benjamins, 1992.
[30]. Martin, J. R. Sense and sensibility: Texturing evaluation [A]. In J. Foley (ed.). New Perspectives on
Education and Discourse [C]. London: Continuum, 2004. 270-304.
[31]. Martin, J.R. Intermodal reconciliation: Mates in arms [A]. In L. Unsworth (ed.). New Literacies and
the English Curriculum: Multimodal Perspectives [C]. London: Continuum, 2008. 112-148.
[32]. Martin, J. R. & M. Stenglin. Materialising reconciliation: Negotiating difference in a post-colonial
exhibition [A]. In T. Royce & W. Bowcher (eds.). New Directions in the Analysis of Multimodal
Discourse [C]. Mahwah, New Jersey: Lawrence Erlbaum Associates, 2007. 215-238.
[33]. Martinec, R. Cohesion in action [J]. Semiotica, 1998, (120): 161-180.
[34]. Martinec, R. Types of process in action [J]. Semiotica 2000, (130): 243-268.
[35]. Martinec, R. & A. Salway. A system for ImageText relations in new (and old) media [J]. Visual
Communication, 2005, (4): 337371.
[36]. Martinec, R. & T. van Leeuwen. The Language of New Media Design: Theory and Practice [M].
London: Routledge, 2007.
[37]. Matthiessen, C. M. I. M. The multimodal page: A systemic functional exploration [A]. In T. Royce &
W. Bowcher (eds.). New Directions in the Analysis of Multimodal Discourse [C]. Mahwah, New
Jersey: Lawrence Erlbaum Associates, 2007. 1-62.
[38]. Norris, S. A Theoretical Framework for Multimodal Discourse Analysis Presented via the Analysis of
Identity Construction of two Women Living in Germany [D]. Ph.D Dissertation. Washington D.C.:
Georgetown University, 2002.
[39]. Norris, S. Multimodal discourse analysis: A conceptual framework [A]. In P. LeVine & R. Scollon
(eds.). Discourse and Technology: Multimodal Discourse Analysis [C]. Washington, D.C.:
Georgetown University Press, 2004. 101115.
[40]. Norris, S. The micropolitics of personal national and ethnicity identity [J]. Discourse and Society
2007, (18): 653-674.
[41]. Norris, S. & R. H. Jones. (eds.). Discourse in Action: Introducing Mediated Discourse Analysis [C].
London: Routledge, 2005.
9

2011 1
31

English Education in China

No. 1, 2011
Serial No. 31

[42]. Norris, S. Analyzing Multimodal Interaction: A Methodological Framework [M]. London: Routledge,
2004.
[43]. OHalloran. K.L. Towards a systemic-functional analysis of multi-semiotic mathematics text [J].
Semiotica 1999, (124): 1-29.
[44]. OHalloran, K.L. Mathematical Discourse: Language, Symbolism and Visual Image [M]. London:
Continuum, 2005.
[45]. OHalloran, K. L. Systemic functional-multimodal discourse analysis (SF-MDA): Constructing
ideational meaning using language and visual imagery [J]. Visual Communication, 2008, (7):
443-475.
[46]. OToole, M. The Language of Displayed Art [M]. London: Leicester University Press, 1994.
[47]. Royce, T. Synergy on the page: Exploring intersemiotic complementarity in page-based multimodal
text [A]. In N. Yamaguchi & W. Bowche (eds.). Japan Association Systemic Functional Linguistics
Occasional Papers [C]. 1998, (1): 2549.
[48]. Royce, T. Multimodality in the TESOL classroom: Exploring visual-verbal synergy [J]. TESOL
Quarterly

2002, (36): 191-205.

[49]. Scollon, R. Mediated Discourse as Social Interaction [M]. London: Longman, 1998.
[50]. Scollon, R. Mediated Discourse: The Nexus of Practices [M]. London: Routledge, 2001.
[51]. Scollon, R. & R. Scollon. Nexus Analysis: Discourse and the Emerging Internet [M]. London:
Routledge, 2004.
[52]. Stenglin, M. Packaging Curiosities: Towards a Grammar of Three-dimensional Space [D]. Ph.D
Dissertation. Sydney: The University of Sydney, 2004.
[53]. Thibault, P. J. The multimodal transcription of a television advertisement: Theory and practice. In A.
Baldry (ed.). Multimodality and Multimediality in the Distance Learning Age [C]. Compobasso:
Palladino Editore, 2000. 311-385.
[54]. Thibault, P.J. Multimodality and the school science textbook [A]. In C. Torsello, G. Brunetti & N.
Penello (eds.). Corpora Testuali per Ricerca, Traduzione e Apprendimento Linguistico. Studi
Linguistici Applicati [C]. Padova: Unipress, 2001. 293-335.
[55]. van Leeuwen, T. Speech, Music, Sound [M]. London: Macmillan, 1999.
[56]. van Leeuwen, T. Introducing Social Semiotics [M]. London: Routledge, 2005a.
[57]. van Leeuwen, T. Towards a semiotics of typography [J]. Information Design Journal, 2005b, (14):
139-155.
[58]. . [J]. 199933-14.
[59]. . [A]. 21
[C]2002.
[60]. . AML [A].
/ CODATA [C]. 2006.
[61]. . [J]. 200723-12.
[62]. . [J]. 200711-10.
[63]. . [M]. 2005.
[64]. . [J]. 200351-8.

10

2011 1
31

English Education in China

No. 1, 2011
Serial No. 31

A Critical Survey of Three Approaches to Multimodal

Discourse
ZHANG Zuocheng1, CHEN Yumin2
(1. School of International Studies, University of International Business and Economics, Beijing 100029,
China; 2. School of Chinese as a Second Language, SUN YAT-SEN University, Guangzhou 510275,
China)

Abstract: Multimodal discourse analysis has become an important current research topic in discourse
analysis. There have emerged three theoretical approaches to multimodal discourse, namely the
systemic functional-multimodal discourse analysis, the multimodal interactional analysis, and the corpus
linguistic approach to situated discourse. The paper is intended to make a critical survey of the three
approaches with the focus on their respective theoretical bases and analytical methods. The insights they
offer into the features of human interaction and its research are discussed. The paper also explores the
relevance of the three approaches to the study of multimodal situated discourse with reference to the
research content, analytical categories and techniques.

Key words: multimodal discourse; systemic functional-multimodal discourse analysis; multimodal

interactional analysis; corpus linguistic approach to situated discourse

2010-11-29

11