Jewitt, Carey

Introduction: What is Multimodality?

Jewitt, Carey, (2009) Introduction: What is Multimodality?. FROM: Jewitt, Carey, The Routledge handbook of multimodal analysis . pp.14-27. London: Routledge. [14]

Staff and students of the University of Leeds are reminded that copyright subsists in this extract and the work from which it was taken. This Digital Copy has been made under the terms of a CLA licence which allows you to: access and download a copy. print out a copy. Please note that this material is for use ONLY by students registered on the course of study as stated in the section below. All other staff and students are only entitled to browse the material and should not download and/or print out a copy. This Digital Copy and any digital or printed copy supplied to or made by you under the terms of this Licence are for use in connection with this Course of Study. You may retain such copies after the end of the course, but strictly for your own personal use. All copies (including electronic copies) shall include this Copyright Notice and shall be destroyed and/or deleted if and when required by the University of Leeds. Except as provided for by copyright law, no further copying, storage or distribution (including by e-mail) is permitted without the consent of the copyright holder. The author (which term includes artists and other visual creators) has moral rights in the work and neither staff nor students may cause, or permit, the distortion, mutilation or other modification of the work, or any other derogatory treatment of it, which would be prejudicial to the honour or reputation of the author. This is a digital version of copyright material made under licence from the rightsholder, and its accuracy cannot be guaranteed. Please refer to the original published edition.

Licensed for use for the course: "COMM5160M - Critical Studies in Visual Communication" Digitisation authorised by Janet Jurica ISN: 9780415434379

AN INTRODUCTION TO MULTIMODALITY

As the chapters in this volume de~10nstrate, multimodal research provides tools for analysing and describing the full repertoire of mea~ng-making resources which people use to communicate and represent (e.g. students and teachers ~ring to the classroom) and how these are 'organized' to make meaning. From a multimodal perspective, language is therefore only ever one mode nestled among

An introduction to multimodality
Carey /ewitt

a multimodal ensemble of modes. Ot~ers have looked before at 'non-verbal' modes; however, multimodality differs in that it does not t~ke language as its starting point or as providing a prototypical model of all modes of communication (i.e. where language provides the conceptual tools, framework and processes for understanding all forms of communication). That is, as Scollon and Scollon argue in Chapter 12, multimodality offers a new perspective and methods for understanding language and 'is not a simple rephrasing of nonverbal communication'. The second assumption central to multimodal research is that each mode in a multimodal ensemble is understood as realizing different communicative work. Multimodality assumes that all modes have, like language, been shaped through their cultural, historical and social uses to realize social functions. Multimodality takes all communicational acts to be constituted of and through the social. Image and other non-linguistic modes take on specific roles in a specific context and moment in time. These roles are not fixed but articulated and situated. For example, a key finding of Kress et al.'s (2001) in-depth study of London school science classrooms was

This chapter introduces

the theoretical

assumptions

of multimodality,

describes its scope, and which inform the

that how teachers chose and used different modes of representation led to radically different constructions of the scientific and natural world in the classroom. For example, representation of a cell in the science classroom as an image or through writing, in colour or black and white, or as a 3D model or an animated sequence on a CD-ROM or webpage makes available and foregrounds different aspects of the concept of a cell. Each of these representational forms makes different demands on the learner. There was also evidence that different modes have differential potential effects for learning, the shaping oflearner identities and how learners create read-

elaborates on five core concepts for multimodality: ing potential or metafunction, handbook chapters. and intersemiotic

mode, materiality, modal affordance, meanrelationships

or intermodal

Theoretical assumptions that underpin multimQdality

Multimodality describes approaches that understand cornmunication and representation to be

ing pathways through texts. The choice of mode, then, is a central aspect of the epistemological shaping of knowledge and ideological design.What can be done and thought with image, writing, or through action differs in ways that are significant for learning. In this regard, the longstanding focus on language as the principal, if not sole medium of instruction, can at best offer a very partial view of the work of communicating in the classroom and beyond. In the examples discussed in this volume the resources of gesture (for instance the spatial extent of a gesture; the intonational range of voice; the direction and length of a gaze) are all understood as part of the resources for making meaning. This connects with the third assumption underpinning multimodality - that people orchestrate meaning through their selection and configuration of modes. Thus the interaction between modes is significant for meaning making. While multimodal itselfnew,Ventola communication is not in and of

more than about language, and which attend to the full range of communicational forms people use - image, gesture, gaze, posture, and so on - and the relationships between them. Four interconnected theoretical assumptions underpin multimodality as it is broadly conceived. These are briefly introduced and discussed below. The first assumption underlying multimodality is that language is part of a multimodal ensemble. This weaves across all the chapters in me volume and different perspectives on multimodality. Language is widely taken to be the most significant mode of communication, this is particularly so in contexts oflearning and teaching. Multimodality, however, proceeds on the assumption that representation and communication always draw on a multiplicity of modes, all of which have the potential to contribute equally to meaning.The basic assumption that runs through multimodality is that meanings are made, distributed, received, interpreted and remade in interpretation communicative modes - not just through language - whether asserts:'''all interactions are multimodal" and multimodality"steps always plays the central role in interaction, without denying that through many representational and as speech or as writing. As Norris away from the notion that language it often does'" (2004a: 3).

et al. (2004: 1) have suggested that:

The various possibilities of combining communication modes in the 'new' media, like the computer and the Internet, have forced scholars to think about the particular characteristics of these modes and the way they semiotically modern discourse worlds. function and combine in the

Gaze, gesture and posture for instance tend to be considered a support to speech; reinforcing or otherwise modifYing speech but not providing communication in its own right; and image is often thought to be in a supportive relation to writing. Multimodal research across a range of social settings casts doubt on this assumption. In the classroom or the workplace, for instance, it is entirely common practice for people to use demonstrations and gestures (e.g. Chapters 6, 16 and 19), images and models (e.g. Chapters 4, 19 and 20) to explain phenomena, or to set and engage in tasks that require a response using visual or other non-linguistic 14 means.

The meanings in any mode are always interwoven with the meanings made with those of all other modes co-present and 'co-operating' in the communicative event.The interaction between modes is itself a part of the production of meaning - a point this chapter returns to. Finally, multimodality is built on the assumption that the meanings of signs fashioned from multimodal semiotic resources are, like speech, social. That is, they are shaped by the norms and rules operating at the moment of sign-making, influenced by the motivations and interests of a 15


CAREY JEWITI AN INTRODUCTION TO MULTI MODALITY

sign-maker in a specific social context. That is, sign-makers select, adapt and refashion meanings through the process of reading/interpretation of the sign. These effect and shape the sign that IS made. Before introducing the central concepts of multimodality, the next section sets out to indicate something of the scope of multimodality and, where appropriate, chapters in this volume are mtroduced to point the reader in the direction of a fuller discussion.

Chapters

18-22, which form Part IV of this handbook,

provide case study analysis of modes, objects as modes is described by

and are briefly 'introduced here. The dimensi&ps of tables as products and three-dimensional

Bjorkvall in Cha~ 18, who draws out the relation between practical function and symbolic meaning potential o~such objects. Jaworski and Thurlow in Chapter 19 investigate how gaze and gesture and the camera as an object are central modes in tourists' social interaction with and construction of place and identities. The use of the modal resources of writing and image is discussed by Mavers in Chapter 20 in her analysis of children's multimodal mapping as one instance

The scope of multi modality

This section gives a sense of what multimodal research can do and what it is good for, rather than provIdmg an exhaustive account of the literature on multimodal research. It indicates the potential of multimodal research to describe semiotic resources for meaning-making and intersemlOUc relauons, to develop research tools, and its successful application across a range of topICSor contexts mcludmg technology-mediated interaction, questions of knowledge, pedagogic pracUces and hteracy~ as well as the production of identity. This brief introduction is supported by the analysIs and dlscusslOn across the chapters in this volume, which include key references and suggested reading to support readers who want to investigate a specific aspect of multimodality in more depth.

of image in the multimodal ensemble. The communicative functions of space are the focus of Stenglin's case study in Chapter 21 on the analysis of a gallery exhibition. Finally, in Chapter 22, West offers an analysis of interaction movement, positioning tional setting. Some multimodal as a combination of speech, gesture (including gaze, body in the room, use of artefacts, etc.) and music within the specific educacalled system networks to map the meta-

studies use a style of diagramming

functional meaning potentials of modes (e.g. Kress and Van Leeuwen, 1996; Martinec, 2000;Van Leeuwen, 1999). Several authors in this book make explicit use of the metafunctions and system networks (seeVan Leeuwen, Chapter 5; Bjorkvall, Chapter 18; and Stenglin, Chapter 21), although how the metafunctions feature within multimodal analysis varies. Within systemic functional approaches to multimodality the metafunctions and the development of corresponding system networks are used to map the potentials for making meaning as a system. While within the social semiotic approach to multimodality, the metafunctions are generally used as a conceptual tool to

Describing semiotic resources

Multimodality can be used to build inventories of the semiotic resources (that is, the actions,

describe and explore the semiotic resources and meaning meaning rather than to map the system itself.

potential

that people use to make

matenals and artefacts people communicate with) that modes make available to people in parucular places and times (what is meant by mode is discussed in detail later in this chapter and by Gunther Kress in Chapter 4 and Theo Van Leeuwen in Chapter 5). The work by O'Toole (1994) and Kress and Van Leeuwen (1996) has contributed to mapping the semiotic resources of art and visual images. Detailed studies have also been undertaken to describe the semiotic resources, organizing principles, and cultural references of many other modes. These include the semiotic modes of colour (Kress and Van Leeuwen, 2002), gesture and movement (Kress

Investigating intersemiotic relations

The relationships across and between modes.in multimodal texts and interaction are a central area of multimodal research. Substantial theoretical descriptions of the dynamics of interaction between image and language have been offered, for example, by the early work of Kress and Van Leeuwen (1996), Lemke's work on science textbooks (1998b) and more recent work by Bezemer and Kress (2008). Bezemer and Kress examine contemporary curriculum materials to investigate the learning gains and losses of different multimodal ensembles. This work draws on a corpus of learning resources for secondary schools in science, mathematics and English from the 1930s, the 1980s and the first decade of the twenty-first century, as well as digitally represented and online learning resources from the year 2000 onward. The project provides a social semiotic account of the changes to the design oflearning resources and of their epistemological and sociallpedagogic significance and shows that image and layout are increasingly meshed in the construction of content. O'Halloran has also examined intersemiotic relations between language, images and mathematical symbolism in mathematics texts (2005) and mathematics classrooms (2000). Research on the multimodal resources of digital screen-based texts also supports this finding: in complex multimodal texts, the boundaries between modes blur and mesh in new configurations. These impact on the construction of knowledge and identities and learner positions (Leander and Frank, 2006;Jewitt, 2006). Research by Martinec and Salway (2005) examines the relationship between image and text. They draw on systemic functional grammar to classifY the relations between image and text in terms of the dependency relations between image and word (equal or unequal) and the nature of the semantic linkage between elements. They classifY these relations as either elaboration, extension, or enhancement. Elaboration refers to a relation where one clause restates or clarifies 17

et al., 2001, 2004; Martinec, 2000), gaze (Lancaster, 2001; Bezemer, 2008), voice and music (Van
Leeuwen, 1999; West, 2007), and space (O'Toole, 2004; Stenglin, 2009a; Van Leeuwen, 2005a) to name a few. Numerous studies have been conducted that set out to understand how semiotic resources are used to articulate discourses across a variety of contexts. The semiotic resources, functions and systems of muluple modes have been described with respect to buildings and rooms, for instance how the design of a child's bedroom connects with notions of gender identity (Kress and Van Leeuwen, 2001), how framing - disconnection and connection of elements - is realized through the use of spaual rhyme or contrast, segregation or separation of elements in school and office buildings (Van Leeuwen, 2005a), the analysis of ranks and metafunctional systems in the Sydney Opera House to understand its meanings (O'Toole, 2004), as well as the design and experience of museum exhibitions (e.g. Ravelli, 2005; Diamantopoulou, 2008; Stenglin, this volume). Mulumodal studIes have focused on how the resources of image and writing as well as other modes are organized on the page or screen of textbooks, websites and other digital learning resources (e.g: Unsworth, 2001; Kress and Bezemer, 2008; R.Jones, 2005, and this volume; Guo, 2004; LIm Fel, 2(04) as well as films, adverts and other new media texts (e.g. O'Halloran, 2004; Burn and Parker, 2003; Baldry, 2004) and other multimodal texts. As this brief review suggests, the scope of resources that multimodal research engages with is wide-ranging. 16

AN INTRODUCTION TO MULTIMODALITY CAREY JEWITI

another, in extension one clause adds information to another, and in enhancement one clause provides information such as how, when, where or why in relation to the other clause. Martinec and Salway have identified examples of these three kinds of linkages between images and text segments and offer a semantic system of image-text relations. Martinec and Van Leeuwen's (2008) study of intersemiotic relations in new media texts suggests that the relations between image and text are potentially remade through their reconfiguration in digital media, although these relations are not (yet) fully established or stable. How people draw on different modes when they communicate has also been the subject of studies within multimodality. Bourne and Jewitt (2003), for example, analysed how a teacher's orchestration of modes in a classroom drew together the semiotic and social resources of texts and students to create a gendered debate. Shifting the focus from teachers to students, Flewitt's (2006) multimodal study of preschool classroom interaction demonstrates the strong link between the communicative demands of a context (in this case home and preschool playgroup) and the modes in use. Focusing on all modes of communication (speech, gesture, movement, gaze and so on) she is able to scrutinize young children's multifunctional uses of different modalities as 'intentional, socially organized activity in the construction of meaning' and to argue against 'pathologizing the absence of talk' (Flewitt, 2006: 47). These studies show the potential of multimodal research to offer a different account of classroom communication by locating the analysis of speech in the broader context of children's total multimodal resources. The theorization of intersemiotic resources is a key aspect of multi modality. One associated concept is multimodal cohesion (Van Leeuwen, 2005a); this explores how different semiotic resources are integrated to form multimodal texts and communicative events via rhythm, composition, information linking, and dialogue. Many of the chapters in this volume are concerned with the interaction of semiotic modes. Norris in Chapter 6 introduces two concepts for thinking about these relationships, namely

and Knobel, 2003; Sefton-Green

and Sinker, 2000). Recent

studies suggest that conventional

'Wint

literacy pedagogy proceeds independently of the everyday multimodal social and communicative worlds of many children (e.g. Marsh, 2006). Marsh's studies (2006), for example, lllvestigate young children's (age 2.5 to 4 years) mediascapes to identifY the complex multimodal communicative practices that they are engaged with in the home. Her focus IS on understanding the functions that these digital media expressions have in maintaining the social relations of the family, accessing knowledge, self-expression, and the development ofhteracy skIlls. She documents how migrant students reappropriate and use media designs in creative play, famIly Me and home/school transitions. The visual character of writing comes to the fore on screen, for instance, to function as objects of literacy in fundamentally different ways than it does on the page Qewitt, 2002, 2005).Jewitt's (2002) case study on the transformation from printed novel to novel as CD-ROM, for example, suggests that the visual character of writing on screen, combllled WIth the domlllance of image, serves to restructure seen across many contexts produce information texts, and fragment forms of writing. A modulanzatIon that can be as information is reorganized across the screen, and elsewhere, to chunks regardless of media and mode. Another potentIal

in bite-size

resource of digital technologies is the mode of hypertext, which embeds wntlllg, Image (and other modes) into web-like patterns and layers of information and genres that make mearungmaking a process of navigation and choice, and create new resources (and demands) on meanlllgmaking (Luke, 2003; Lemke, 2002b; Zammit and Callow, 1998; Zammit and Downes, 2002). Technology and its impact on multimodal representation and interaction is a theme that runs across this handbook, and which is foregrounded in several of its chapters. In Chapter 7, for example, O'Halloran examines the impact of the printing press and com-

modal density and modal intensity. Unsworth and Cleirigh discuss image-text
11 and critically engage with early work emphasizing dances of image and language and propose a semiotic framework ergistic nature of image-language interaction in meaning

relations in Chapter the distinctively different semiotic affordesigned to address the synconstruction. Approaching

puter technology on the semiotic landscape of mathematics. Rodney Jones in Chapter 8 mvestigates technology as a site of display and its impact on multImodal commumcatlOn. In partICular he explores how technology has affected the ways people use their bodies as a commumcatIve resource in social interaction through the analysis of how gay men use an lllternet websIte to create displays of their bodies. Kevin Leander and Lalitha Vasudevan approach the theme of culture and multimodality through the lens of mobile and video technology III Chapter 9. In Chapter 10, Lemke explores issues of how technologies produce meaning effects across multiple times cales, using the illustrative example of the computer franchises, which are typically experienced or months. and VIdeo games and transmedIa over times as long as 50 hours over penods of weeks

llltersemiotic relations from a different starting point in Chapter 7, O'Halloran takes a historical view of the semiotic landscape in mathematics, with a particular focus on mathematical symbolic notation.

Multimodality and technology Knowledge, pedagogy and literacy

The multimodal facilities of digital technologies enable image, sound and movement to enter When first made by the New London Group (1996) (Courtney Cazden,James Gee, Gunther Kress, Allan Luke and others) the call to understand knowledge and pedagogy as multimodal was radical. A key design element of a future pedagogy was heralded as 'designs for other modes of meaning' (New London Group, 1996). In part this call was a response to the social and cultural reshaping of the com~unicational landscape (related to globalization, new technologIes, and new demands for work). The conclusion that reading this 'new' multimedia, multlmodallandscape for its linguistic meanings alone is not enough was inevitable. This work spawned a strand of educational research and pedagogic models within which multimodality (broadly conceIved) is a key factor (Cope and Kalantzis, 2000). . . A special issue of LinJ(uistics and Education on multimodality was one of the first pubhcatIons to provide tools for educational researchers wanting to undertake multullodal research (Lemke, 1998c). From early 2000 there has been an explosion of interest in multimodahty wIthm research 19 18 the communicational landscape in new and significant ways. The facilities and features of multimodal configurations can impact on design and text production and interpretative practices. Multimodal research has been conducted on the technologization of practices and communication and interaction more generally (e.g. Marsh, 2005;Alvermann, 2001; Leander, 2007; Unsworth et al., 2005; Lankshear and Knobel, 2003; Cope and Kalantzis, 2000). Much of this work explores and theorizes the nature of image and writing relations in narratives, relationships between book- and computer-based versions of texts, and the role of on-line communities of various kinds in the critique, as well as the interpretation and generation of new forms of multimodal and digital narratives and literacy practices.This work often describes new forms of literacy in an attempt to remap the territory of communication in a contemporary context and the kinds of practices that help move across it, such as blogging, and culture jamming (Lankshear

CAREY JEWITI

AN INTRODUCTION TO MULTIMODALITY

and this perspective has been actively taken up by educational researchers leading to substantial work that looks at multimodal meaning-making across a wide range of sites. Multimodal research has been undertaken in preschool and early-years contexts, with a focus on multimodal meaning-making practices (e.g. Bearne and Kress, 2001; Kress, 1997a; Kenner, 2004; Mavers, 2003; Pahl, 1999; Lancaster, 2001; Marsh, 2006). Science education has proven to be a productive site for multImodal investigations into the construction of knowledge across a range of resources (e.g. Kress et al., 2001; Reiss et al., 2007), as has mathematics education (O'Halloran, 2000, 2005), musIC education (West, 2007) and school English and media education (Goodwyn, 2000; Kress et al., 2005; Burn and Durran, 2007). Multimodal studies in higher education have also examined pedagogic strategies and learning (Archer, 2008). Taken as a whole, these multimodal studies show that significant pedagogic work is realized through a range of modes. The Multimodal Production of School English project (see Kress

artifacts, visual representation

and so on. In the Olifantsvlei

fresh stories project, Stein (2003)

undertook a literacy project over six months with teachers and students of grades one and two at a Johannesburg primary school which serves children of unemployed and migrant families living in informal settlements. She worked with multimodalliteracy practices and pedagogy through a systematic use of different semiotic modes in order to develop forms of learnlllg beyond language. The project explored the relations between creatIVIty, multImodal pedagogy: representation and learning. Student case studies involved observation and llltervIews, students use of two-dimensional drawings, writing, three-dimensional figures, spoken dIalogues, multImodal play and performance to create narratives of identity and culture. The focus was on the representation of doll and child figures and their symbolic meanings. Stein descnbes the chIldren's transformation and recontextualization of culturally and historically situated practices of these representations. Stein argues that multimodal pedagogy enables the assertion of student identity, cultural practices and community to enter the school context in ways that are SIgnIficant for literacy and teaching. Identity is another theme that weaves its way across several chapters in this book, some more strongly than others. For instance, how young men interact with mobile technologies in the performitivity of their identities is a theme in Chapter 9 by Leander andVausdevan. Lemke, m Chapter 10, argues that how people experience and make sense of multimedia is strongly related to theIr Identities, values and desiresTaking a different starting point for thinking about identity, Bjorkvall touches on the table as a marker of identity in Chapter 18, while Jaworski and Thurlow explore gesture and gaze as part of the identity practices of tourists in Chapter 19. The following section discusses five core concepts for multimodality: , ' mode, matenalIty, modal

et al., 2004) showed that this holds true even in a curriculum context such as English where talk and writing dominate the classroom. It studies the complex ways in which image, gesture, gaze,
mteractlOn WIth objects, body posture, writing and speech interact in the classroom production of school subject knowledge. It highlighted how students and teachers co-produce notions of ability, resistance and identity in the classroom through their multimodal interaction. The project also analysed how classroom displays, space, furniture and artefacts were designed to realize versions of English. This research showed that the work of interpreting school English is beyond language and requires the ability to make sense of a range of modes and the relationships between them. It also highlighted engaged with in the classroom. the complex multimodal identity work that students are

Gunther Kress, in Chapter 4, explores how modes shape knowledge and practices and the role oflayout in the social and ontological consequences of the designed multimodal classification of information. The need to rethink what it means to learn and to be literate is a thread that runs through much multimodal research. This raises numerous research questions in relation to learning, including how representations impact on thinking and learning, as well as what literacy is and could be in a multimodal and multilingual communicational landscape, and how to study It. These issues are picked up across the handbook, and are central to Chapter 14 by Street, Pahl and Rowsell who explore the potential multimodality. of merging of New Literacy Studies with

affordance, meaning potential or metafunction, and intersemiotic or intermodal relatiOnShIps. The nuances and differences in emphasis between the approaches to multimodality are discussed in Chapter import or attention 2, but it is worth noting here that these concepts are not given the same within each approach.

Core concepts for multi modal analysis

The following core concepts are in a state of change and fluidity, and are continuously taken up and shaped in different ways by different approaches to multimodal research. The descnptions that follow are therefore intended to provide the reader with useful (worklllg) defillltIons.

Identity practices
How identities are articulated through multimodal means is an area that has attracted some attention within multimodal research. Norris (2004a) has explored this by examining how people shIft modes and modal awareness to manage multiple identity roles in interaction (e.g. a mother engaglllg with their child, in the context of another interaction). Bezemer and Kress (2008) explore how pupil identities are negotiated in the classroom through the use of gaze, and onentatlOn to texts and teachers. Rodney Jones (2005, and in this volume) explores how sexuality identities are performed and managed through the display and withholding of representatIons. of the body. Much of the work on what might broadly be called literacy practices from a multllllOdal perspective is also concerned with the production of identities (e.g. Pahl and Pollard, 2008; Stein, 2003, 2008; Stein and Mamabolo, 2005). For example, Stein (2008) explores how students 1ll South African educational contexts express complex narratives of identities and culture through multimodal texts, highlighting the links between representational means, the production of identities, and social justice. Her work explores how multimodal pedagogy can reconnect linguistically disenfranchised learners - through the use of performance, semiotic 20

Mode
Within social semiotics, a mode, its organizing principles and resources, is understood as an outcome of the cultural shaping of a material. The resources come to display regularities through the ways in which people use them. In other words in a specific context (time and place) modes are shaped by the daily social interaction of people. It is these that multimodal analysts call modes. Kress (Chapter 4 this volume) sets out the central issues of the category of mode, as seen from a social semiotic multimodal perspective.The chapter includes discussion of the 'logics' of modes; issues of mode, representation and 'knowledge; sign and mode, including the notion of modal choice and epistemological commitment; as well as the notion of sign, genre and discourse. O'Halloran and O'Toole have a slightly different conception of'semiotic resource', and 'mode' does not feature in the same way as for Kress and Van Leeuwen. From this perspective (and following from Halliday (1978), LanguaJ(e as Social Semiotic) language, mathematical sY~llbolism and images, for example, are understood as semiotic resources rather than modes. SemlOtIc 21

CAREY JEWITT

AN INTRODUCTION TO MULTIMODALITY

resources consist of systems of meaning

that realize different functions, and therefore, meaning

muscles we use to make facial expressions and gestures - or technologically

- for example,

becomes a matter of chOIce from the systems of meaning from different semiotic resources, and how these choices integrate in multimodal phenomena. From this starting point, language can be realized through written text (a visual mode) and spoken language (an oral mode). As O'Halloran argues m Chapter 7, the meanings of modern mathematical symbolism, for example, change when developed m the wntten mode as opposed to when it is realized in the oral mode. Multimodal interactional analysis focuses on action and therefore does not have the same focus on mode, although all actions are understood as 'mediated by the systems of representatIOn that they draw o~' (Norris, 2004a: 12-13). However, the focus is on the situated interplay between modes at a gIven moment in social interaction (see Chapter 6). The regular pattern of using a set of resources has traditionally been called' grammar'. Machin has questIOned the possibility and usefulness of applying a linguistic term such as 'grammar' to Image and other modes (Chapter 13 this volume, and Machin, 2007). However, a general prinCIple IS that m order for something to 'be a mode' there needs to be a shared cultural sense of a set of resources and how these can be organized to realize meaning. A number of detailed studies on specific modes have helped to describe these semiotic resources, material affordances organizing principles, and cultural references (e.g. the work on images by O'Toole (1994) and Kress and Van Leeuwen (1996) set out to map the semiotic resources of image). The chapters WIthm thIS volume examine a variety of modal resources and their use, including space, gesture, gaze, body posture and movement, sound, voice and music, image, mathematical symbolism, wntten and spoken language, and three-dimensional objects such as tables, as well as examining how these modal choices and configurations vary across a range of social settings and media. The purpose of multimodal investigations is to understand the principles of use and modal resources available in a multimodal representation (a multimodal text) or the situated communicative moment rather than to seek to establish a universal inventory for a mode. What is considered a mode and interaction between modes is inextricably shaped and construed by social, cultural and histoncal factors. These factors influence and mediate how modes are taken up and used for meaningmakmg and the productIOn of modal conventions. People draw upon the available modal resources to make meaning in specific contexts. Consequently; any given mode is contingent upon fluid and dynamIC resources of meaning, rather than static skill replication and use. It is in this way that modes are constantly transformed by their users in response to the communicative needs of communities institutions and societies: new modes are created, and existing modes are transformed. ' The influences of timescales, technologies and sites of display, history and culture are central to the idea of mode; these issues inform many of the chapters, in particular see O'Halloran (Chapter 7). The resources of modes and the functions they are used to realize are also shaped m Important ways by changing facilities of technologies, itself a part of the social landscape (see Jones, Chapter 8; Leander andVausdevan, Chapter 9; Lemke, Chapter 10).

with pen and ink, or computer hardware and software - together with the ways in which these resources can be organized. Semiotic resources have a meaning potential, based on their past uses, and a set of affordances based on their possible uses, and these will be actualized in concrete social contexts where their use is subject to some form of semiotic regime. (Van Leeuwen, 2005: 285) As indicated earlier, O'Toole (1994) and O'Halloran (2005) view semiotic resources (language,

image, etc.) as systems of meaning that people have at their disposal. People make choices across different semiotic resources simultaneously and these combine in multimodal phenomena. The emphasis on rules within social semiotics is on rules as socially made and changeable through social interaction (Van Leeuwen, 2005a). This stands in contrast to traditional semiotic understanding of rules (or codes) as fixed and resistant to modification (Saussure, 1974;

Barthes, 1964). Traditional semiotics sees language and other semiotic systems as a code or sets of rules for connecting signs and meanings. This means that once two or more people have understood the same code they can connect the same meanings to the same sounds or graphic patterns and understand each other. This suggests that the semiotic system is simply 'there'. It can be used but it cannot be changed in any way. The sign is viewed as a pre-existing conjunction of a signifier and signified, an element in a code, to be understood and used. This view of semiotic systems places people in a passive role to the production of meaning and establishes language and other semiotic systems as entirely stable. The concept of semiotic resource offers a different starting point for thinking . about semI-

otic systems and the role of the sign-maker in the process of making meaning. In this perspective, signs are a product of a social process of sign-making. A person (sign-maker) 'chooses' a semiotic resource from an available system of resources. They bring together a semIOtIc resource (a signifier) with the meaning (the signified) that they want to express. In other words, people express meanings through their selection from the semiotic resources that are available to them in a particular moment: meaning is choice from a system. But thIS chOIce is always socially located and regulated, both with respect to what resources are made available to whom, and the discourses that regulate and shape how modes are used by people. There are various kinds of normative discourses for how we use semiotic resources - sometimes more, sometimes less binding, and of different kinds, but nonetheless they do provide 'rules' for their use. Discourses of gender, social class, race, generation, institutional norms and other articulations of power shape and regulate people's use of semiotic resources. These are not 'codes' in the sense that they cannot be changed and that they 'are simply there' - but they are social rules (Van Leeuwen, 2005a). In addition to mapping the semiotic resources available to people to make meaning with, multimodality can also contribute to the development of new ways of using semiotic resources. Focusing (through historical analysis) on how semiotic resources come to be as they are, multimodality can ask why they are as they are. This is a powerful way of enabling people to see how a 'reality' comes to be represented and offering the potential to imagine it differently and to

Semiotic resource
Semiotic resource is central to multimodality, although it features slightly differently within partICular approaches. Kress and Van Leeuwen suggest semiotic resource can be thought of as the connectIOn between representational resources and what people do with them. Van Leeuwen describes semiotic resource as follows:

redesign it. This shifts the primary focus from descriptive accounts to connect more explicitly with
macro social, political and cultural concerns within design.

Metafunctions
Semiotic resources are the actions, materials and artifacts we use for communicative purposes, whether produced physiologically - for example, with our vocal apparatus, the The turn to social explanations within linguistics, for instance in the work of McDerrmot, Hymes and Halliday to name a few, shifted attention onto how language is shaped by how 23

22

CAREY JEWITT

AN INTRODUCTION TO MULTIMODALITY

people use it - the social functions

that the resources of language are put to. This turn to the of the social

social provides the context for the linguist Michael Halliday's (1978) theorization functions oflanguage as realizing three metc!functions.

It thus follows that image, in the form of graphic marks on a two-dimensional surface (i.e. its material) as well as how it has been socially shaped through its use, combine to offer different meaning potentials for the expression and representation of meaning than speech III the form of sounds over time and space. The particular semiotic resources of a mode (or the particular systems of semiotic resources, from the perspective of O'Toole and O'Halloran) have come to be shaped through these dIfferent histories and so has people's use of them. Each mode (as it is realized III a particular sOClal context) possesses a specific logic and provides different communicational and representational potentials, referred to as modal logic. The sounds of speech, for instance, usually happen across time, and this sequence in time shapes what can be done with (speech) sounds. The lOgIC of sequence in time is unavoidable for speech: one sound is uttered after another, one word after another, one syntactic and textual element after another. This sequence becomes an affordance or meaning potential: it produces the possibilities for putting things first or last, or somewhere else in a sequence. The mode of speech is therefore strongly governed by the logIC of time ..In contrast, (still) images are more strongly governed by the logic of space and sImultaneIty. LIke all governing principles they do not hold in all contexts and are realized through the. complex interaction of the social as material and vice versa - in this sense the matenal constitutes the social and vice versa. The connection between culture and nature, meaning and materiality is the focus of Chapter 5 by Theo Van Leeuwen who argues for 'their inextricable semiotics of voice quality. The repeated use of modes to represent connection' through an account of the . . . meanlllgs m speClfic

People, Halliday theorized, construct representations of'what goes on in the world' and their experience of the world through the ideational resources of a mode. (Ideational meaning is also referred to as presentational meaning, and sometimes called experiential meaning or logical meanlllg.) In language this may be achieved in a number of ways including the words chosen to represent people, places, and things in the world; or the creation of different kinds of relationships between these 'participants' by positioning them as active, passive, or reactive. Social relations between the person who makes a sign, the person who engages with it, and the thing that IS represented are also realized in a sign. These relations are constituted and enacted by the mterpersonal resources of a mode (sometimes referred to as 'orientational' meaning). These meanlllgS need to be organized into texts and this draws on the textual resources of a mode (sometimes called organizationalllleaning). Language provides many different resources for realizing the coheslOn and structure of a text. Halliday's work explores how these three kinds of meaning potentials are 'held by' the grammar and elements oflanguage. Multimodal approaches take up the concept of the metafunctions and apply them to all modal (semiotic) resources (that is, the metafunctions are viewed as a higher order of meaning rather than speClfi~ to language).Another way of thinking about the metafunctions is as meaning potentIal, that IS, what can be meant' or 'what can be done' with a particular set of modal (semiotic) resources.Jewltt (2006), for example, employed the metafunctions to analyse how technologymedIated learnlllg lllfluences the shapes of curriculum knowledge in the classroom with respect to Image, gaze, gesture, writing, speech and so on. The ideational metafunction enabled questions to be asked about how the multimodal computer applications in use presented 'the world': what was included and excluded and how what was displayed shaped curriculum knowledge. USlllg the lllterpersonal metafunction made it possible to explore how learners were positioned to knowledge through the design of the multimodal applications they used. The textual metafunction provided a tool with which to get at how the arrangement of multimodal elements on screen organized the text.

and communicate

particular

social contexts has resulted in the specialization of modes. From this perspective not every mode is equally 'useable' for a particular task. Some things can be signified in an image as well as through talk, while some others can only be realized in an image and others only m talk. This introduces the importance of the concept of intersemiotic relations, also referred to as intermodal relations, or multimodal ensembles.

Intersemiotic Modal affordance, meaning potential and materiality

The term affordances is contested and continuously debated within multimodal research. It has particular emphasis and currency in social semiotic approaches to multimodality. It originated III the work on cognitive perception of Gibson (1977). It was later taken up by Norman in relation to design (1988, 1990). Norman's view of affordance considers the material and social aspects o~ design. Van Leeuwen uses affordance following Gibson and uses the term 'meaning potentIal to refer to the matenal and the cultural aspects of modes. Taking a slightly different emphaSIS, modal qffordance is used by Kress (1993) to refer to what it is possible to express and represent eaSIly with a mode. He positions affordance as a complex concept connected to both the material and the cultural, and the social historical use of a mode (each of which is intimately connected), whIle Jewltt argues that neither Gibson nor Norman's notion of affordance can adequately acknowledge how tools (conceptual and material objects) are shaped by people's use of them III specific social situations 0ewitt, 2008a). In other words, the affordance in Kress's terms, or the meaning potential of a mode for Van Leeuwen, is shaped by how a mode has been used, what it has been repeatedly used to mean and do, and the social conventions that inform its use in context. Where a mode' comes from' its history of cultural work, its provenance, becomes a part of its alfordance or meaning potentiaZ: 24

relationships

Modal affordance or meaning potential raises the question of what image is 'best' for and what words and other modes and their arrangements are 'best' for in a particular context (Lanham, 2001):When several modes are involved in a communicative event (e.g. a text, a website, a spoken interchange) all of the modes combine to represent a message's meaning (e.g. Kress et al., 2001,2004). The meaning of any message is, however, distributed across all of these modes and not necessarily evenly. The different aspects of meaning are carried in different ways by each of the modes in the ensemble. Anyone mode in that ensemble is carrying a part of the message only: each mode is therefore partial in relation to the whole of the meaning, and speech and writing are no exception 0ewitt and Kress, 2003a). Multimodal research attends to the mterplay between modes to look at the specific work of each mode and how each mode mteracts with and contributes to the others in the multimodal ensemble. At times the meanmg reahzed by two modes can be 'aligned', at other times they may be complementary and at other times each mode may be used to refer to distinct aspects of meaning and be contradIctory, or m tension (Lemke, 1998b).As Lemke has stated (2002: 303): No [written] text is an image. No text or visual representation means in all and only the same ways that text can mean. It is this essential incommensurability that enables genume new meanings to be made from the combinations of modalities.

25

AN INTRODUCTION TO MULTIMODALITY CAREY JEWITI

The relationships between modes as they are orchestrated in interactions

(and texts) may real-

Multimodal Multimodal

analysis is an intensive

research process both in relation

to time and labour.

ize tensions between the aspects of meaning in a text. This kind of tension can itself be meaningful and a means for encouraging reflection and critique.The structure of a text and hyperlinks realize connections and disconnections between screens. These contribute to the expansion of meaning relations between elements. The question of what to attend to, what to 'make meaningful' is a significant aspect of the work of making meaning. In other words, the task of what to attend to and to select as salient to the task at hand is amplified by a multimodal focus. As mentioned earlier, the question of how modes interact is dealt with by authors throughand Clerigh out this volume, in particular by Norris (Chapter 6),Jones (Chapter 8), Unsworth (Chapter 11), and Jaworski and Thurlow (Chapter 19).

research can be applied to take a detailed look at 'big' issues and questIOns through

specific instances. Nonetheless the scale of multimodal research can restrict the potenual of multlmodality to comment beyond the specific to the general. The development of mulumodal corpora may help to overcome some of these limitations, as might the potenual to combme multimodal analysis with quantitative analysis m lllnovaUve ways. One purpose of this handbook is to engage with these limitations, . . to clanfY what mulu-

modality can and cannot do, and to explore the ways in which multimodality can be brought into productive conversation with other theoretical approaches to further develop thIS approach and its methodological tools (see Part III).

The concepts introduced above are common across multimodal research in this volume.There are, however, important nuances and differences in emphasis with the field of multimodality and these are the focus of the next chapter.

Summary
This chapter has provided an introduction to the theoretical assumptions that underpin multimodal research. In response to whether multimodality can truly be consIdered a theory or whether it is more appropriate to view it as a method, this chapter has suggested that the comparison with Green and Bloome's (1997) commentary on ethnography, as theory, as perspective or as method is useful and that these different degrees of engagement WIth multllllodahty hel~ to make sens'e of what is seen to count as multimodal. The chapter has provided a set of working definitions for the key concepts that inform approaches to mulumodahty, elaboratlllg on the origins and development of these concepts over the past decade and has begun to explore some of the nuances and differences across approaches most strongly assOCIated with mulumodality: social semiotics, discourse analysis, and interactional analysis. Having sketched the terrain of multimodality as a research field, the next chapter focuses in on these three approaches to examine the variation between this volume. these three approaches, each of which informs the work III

The limitations of multi modal research

Although multimodal ered and understood. research has much to offer, it also has limitations that need to be consid-

A criticism sometimes made of multimodality is that it can seem rather impressionistic in its analysis. How do you know that this gesture means this or this image means that? In part this is an issue of the linguistic heritage of multimodality, that is, how do you get from linguistics to all modes. In part it is the view of semiotic resources as contextual, fluid and flexible - which makes the task of building 'stable analytical inventories' of multimodal semiotic resources complex. It is perhaps useful to note that this problem exists for speech or writing. The principles for establishing the 'security' of a meaning or a category are the same for multimodality as for linguistics (or philosophy or fine art). It is resolved by linking the meanings people make (whatever the mode) to context and social function. Increasingly, multimodal research looks across a range of data (combining textual/video analysis with interviews for example) and towards participant involvement to explore analytical meanings as one response to this potential problem. Linked with the above problem of interpretation is the criticism that multimodality is a kind of linguistic imperialism' that imports and imposes linguistic terms on everything. But these critics overlook the fact that much of the work on multimodality has its origins in a particular strand of linguistics: namely, the social semiotic theory of communication first proposed by HallIday (1978). The social component of this perspective on language sets it apart from narrower concerns with syntactic structures, language and mind and language universals that have long dominated the discipline. This view of communication can be applied (in different ways) to all modes, to gesture and image no less and no more than to speech and writing. Description is theoretically grounded and contributes to theory building. There is a need to ask questions of and through detailed description. For instance, to ask what kind of discourses are being articulated in a context and why, and what is the social function of the representations being described. For example, to ask how the multimodal design of the English classroom shapes what school English is, what texts are included in English and how does this shape what it might mean to be a student in that classroom, and so on. This analytical focus is important to show how discourses are articulated in a context so that they can be made explicit, shared or challenged and redesigned. Multimodal research is problematic if it offers endless detailed descriptions and fails to make clear the broad questions it seeks to answer.

27

26