Dorothy Chun

UC Santa Barbara
 Why do it?
1. To be able to say with greater
certainty, beyond anecdotal
evidence, that your efforts are
having an effect (hopefully
positive!).
2. To conduct formative and
summative evaluation.
 Today’spresentation is framed in
terms of my experience with 2
research projects:
◦ ICE (UCSB-Uni Kassel, Germany)
based on Cultura
◦ Chin 332 project w/ Stephen & David
 Bothbegan w/o research
questions.
 1stround: ICE added to curriculum
so only online questionnaires and
forums
 In writing a chapter for Teaching
with Technology we discovered
that students were missing a lot
of subtle differences
 E-mail exchanges vs. forums
 2nd round: different partner
group with unequal numbers
 Small forum groups with 6/group
 Students reported back to the
class: mismatch between what
they said in forums and in class
 Analysis of classroom discourse
using Byram’s ICC disappointing
 3rdround: different format (ABC)
due to partner’s request
 Teacher-chosen topics did not
work as well as student-chosen
topics
 1. What would you like your
students to learn from the cafés?
 2. What kinds of evidence would
show that they have learned it?
 3. How can you find or obtain
this kind of evidence?
 What are the 3-4 most important
things you want your students to
learn from this exchange?
 Take a moment to write them
down.
 How will you define success?
 Build a community
◦ Intercultural exchange (including
business culture)
◦ Linguistic community
 Affective: Sense of solidarity
 Linguistic: Cohesion in the
discourse
 Expanded vocabulary
 Grammar practice
 Additional chances to use the L2
 Language variation
 Authentic language in the real
world, beyond the textbooks
 How others think
 How others act/behave in social
contexts
 How things work in daily life in
the other’s culture
 Openness, awareness, curiosity
about other culture
 Examine cultural assumptions
about the other culture
 Local adaptation of the other
culture

 Reflecton your own culture

 Grapple with identity issues
 Qualitative vs. Quantitative
 Qualitative:
◦ Often, but not always longitudinal,
descriptive, e.g., pedagogical best
practices
 Quantitative:
◦ Often, but not always cross-
sectional, “treatment” studies
 Teacher journals: record
successes as well as frustrations
(e.g., use the facilitator forums as
data)
 Questionnaires: ask students what
they thought they learned, how
they liked the project
 Data analysis: examine actual
“data” produced by the students,
e.g., in the forums
 Learning outcomes: linguistic
progress, intercultural
competence
 Be specific about the outcomes
you are most interested in (and do
not try to solve the world’s
problems!).
 Different methods are needed for
different goals/questions.
 Collect data systematically.
 Collectas much demographic
data as possible:
◦ Age
◦ Gender
◦ First language (L1)
◦ L2 exposure
◦ Extent of language use (L1 and L2)
◦ Contexts of language use (family,
school, work)
 Think about all of the other
variables that could affect
learning outcomes (level, prior
knowledge, motivation, time on
task, etc.).
 Try to find a way to ensure that
these other variables are kept the
same or are noted.
 It’s better to have more data than
you think you need than less (save
everything!).
 Combine qualitative + quantitative:
Study learning outcomes in addition
to students’ perceptions about what
they thought they learned or their
attitudes towards the cafés.
 During the project, by keeping
good records, including recording
communication between both
sides, you can identify what works
and what doesn’t.
 You can then change/improve
your strategies
 Experiment with sequencing
 Experiment with prompts and
instructions
 Compare online vs. offline tasks
 Compare use of different media
 Compare use of different online
tools
 For what specific purpose are
each of the following used for?
 Is the use of the medium effective
for your goal?
◦ Text
◦ Graphics
◦ Video
◦ Audio
 For what specific purpose are
each of the following used for?
 Is the use of the tool effective for
your goal?
◦ BRIX activities
◦ Wikis
◦ Skype, videoconferencing
 Which types of teacher behavior
are successful?
 Where (in which activities), when (at
what point in the activity), how often
(whenever there is a “lull”?) should
teacher participate?
 Look at forums to see whether:
◦ Students address each other
◦ Students address the teacher
◦ Students respond to others’ postings
(cohesion)
◦ Students use different types of
speech acts (state opinions, inquire,
apologize, request clarification,
discuss)
 Look at forums to see if/whether:
◦ Students only state their opinions
◦ Students only talk about L2 and C2
◦ Students reflect on their L1 and C1
◦ Students express awareness of
similarities and differences between
C1 and C2
◦ Students express understanding of
C2
 Look at data to see how
◦ Students demonstrate understanding
of the other culture
◦ Students’ attitudes, perceptions,
interactions have changed over the
course of the exchange (note that it’s
not always easy to know whether
anything has changed)
 Look at data to see
◦ How knowledge is co-constructed
and what it looks like
◦ What causes interaction
◦ Whether students speculate and
hypothesize (rather than just make
statements or ask questions)
 Action research can be done in
the classroom and with the online
cafés.
 You do not have to do a rigorous
experiment with treatment groups
(and statistical analyses).
 You do need to have specific
research questions (targeted
outcomes).
 You do want to try and control for
as many other contributing
variables as possible.
 Doing action research can help
both in formative evaluation
(seeing what works and what
doesn’t during the project) and in
summative evaluation (seeing
whether the original learning
goals have been met at the end of
the project).
 Have fun with your impressive and
exciting projects!

 Mahalo nui loa