December 2017
Volume 1, Issue 2
Editorial Notes
Welcome to the second edition of The CALT Report. In
this issue, we feature an article about implementing
a teacher-based assessment system in a general
English program in Australia, and we learn from the
experiences of a teacher tasked with creating a new
program-wide assessment instrument. Our member
spotlight shines on Emma Tudor in San Francisco,
and NB quotes provide "food for thought" about
assessment. In addition, all our photos were taken at
the 2017 CALT Conference, November 3-4 at the
University of Arizona, Tucson.
CALT Conference co-chairs Mariana Menchola-Blanco and Eddy
White get ready for the Welcome Ceremony. (Eddy shows off his leg
brace after recent knee surgery.)
1
The CALT Group is a professional
community  of language teachers,
assessment coordinators, and other
educators. It is a community of
practice for anyone interested in
effective classroom assessment in
ESL/EFL programs. Visit our website
at http://groupspaces.com/CALT. 
We hope you were able to attend one of our online
workshops this past year. If not, look for more
sessions in 2018. We hold two types of 60-minute
meetings:
     Talkshops are unstructured Q&A discussions
about anything related to classroom assessment. 
     Workshops are more structured sessions about a
particular area of classroom assessment, e.g.,
assessing speaking, and typically involve a
background reading to help focus the discussion.
Attendees can bring any questions or issues related
to the topic to the session for discussion.
Please feel free to send your feedback on the
newsletter or any of the online workshops to
CALT@groupspaces.com . Email us if you would like
to make a contribution to The CALT Report, as well.
 Feature Article

Implementing and learning

from teacher-based assessment
in a General English course
BY KYLE SMITH AND LAUREN FAULL-ORTIZ
Kyle is the Assistant Programs Manager and Lauren is the Coordinator of the General English program
at Queensland University of Technology International College in Brisbane, Australia.
INTRODUCTION
"Classroom assessment is an ongoing process through
which teachers and students interact to promote
greater learning. The assessment process involves using
a range of strategies to make decisions regarding
instruction and gathering information about student
performance or behavior in order to diagnose students’
problems, monitor their progress, or give feedback for
improvement" (Butler & McMunn 2006).
The above definition of ‘classroom assessment’ was
included in the very first CALT Report. It is, however,
somewhat problematic: on the basis of this definition, it
is difficult to distinguish ‘classroom assessment’ from
similar terms, including ‘formative assessment’ and
‘assessment for learning’. In a 2009 paper about
‘teacher-based assessment’ (TBA), Davison and Leung
claim that all these various terms are “used
interchangeably to refer to the same practices and
procedures” and...
Certainly, this type of assessment practice is something
that we have sought to introduce to the General English
(GE) program at the Queensland University of Technology
International College (QUTIC). In this article, we will
discuss Davison and Leung’s (2009) model of TBA, what it
looks like in practice here at QUTIC, and what we have
learned in 2017 as we have developed and implemented a
TBA system. 
BACKGROUND
For many students in QUTIC’s GE program, they are
taking the first step in their pathway through our English
for Academic Purposes (EAP) program into undergraduate
and postgraduate courses. Pathway students typically
enroll in two five-week sessions of GE, ahead of an EAP
Entry Test; at the end of each session, teachers make a
summative judgment as to whether their students will
progress to the next five-week stage of the program or
repeat the current one.

"...tend to be used to signify a more teacher-mediated, Prior to 2017, to assist teachers in making these
context-based, classroom-embedded assessment progression recommendations, GE students took eight
practice, explicitly or implicitly defined in opposition to standardized tests – two for every macro-skill – every five
traditional externally set and assessed large scale formal weeks. The reading and listening tests were adapted from
examinations used primarily for selection and/or commercially available materials by a team of four
accountability purposes" (p. 395). ongoing language educators, who also developed the
writing and speaking tests from scratch. More broadly,
As Davison and Leung (2009) conceive of it, TBA has this assessment system had essentially the same goal as
several important characteristics:
1. It happens in classrooms, as opposed to testing                                                                            continued on p. 3
     centers or halls;
2. Students’ own teachers are directly involved in:
     a. Planning, designing and modifying assessment
          tasks and activities,
     b. Giving immediate and constructive feedback, and
     c. Making judgments about their students’
         performance and work;
3. Students are more actively involved, perhaps through
     self- and peer-assessment; and
4. Teachers’ judgments are based on a range of samples
    of student work, collected over a period of time.

2
"Implementing and learning from TBA" continued...
large-scale English language proficiency testing: maximizing
fairness through reliability, understood in psychometric
terms. To achieve this goal, student and teacher attention
was focused “on highly discrete and defined tasks and …
standardized procedures and standardized outcomes”
(Murphy, 1994, p. 194).
In the case of reading and listening tests, these tasks
consisted of selected response and short gap-fill items.
Results were reported to students as a percentage for each
test; sometimes students also received feedback on how
they performed on the different task types, e.g., ”You got all
of the multiple-choice questions correct but had difficulty
with gap fill so work on your vocabulary and spelling.”
In the case of writing tests, students completed one timed,
impromptu task consisting of either one or two paragraphs,
depending on their level; at Pre-Intermediate and
Intermediate levels, they were essentially writing mini-
essays. Students received an overall grade for each test of
either Satisfactory or Needs work and feedback in several
different forms, including a checklist and error correction
codes, all created internally. Speaking tests often consisted
of highly-structured and well-rehearsed dialogues or
monologues; the grading and feedback was very similar to
the writing tests. These tasks generally led to “simplistic
judgments” of students’ capabilities (Hamp-Lyons, 1994, p.
52) and made it quite difficult for teachers confidently to
“evaluate higher-order composing skills including creativity,
problem solving, and metacognition which are badly needed
in students’ future study and professional careers” (Ibid, p.
86).  Given that these are also increasingly important as
students progress through the GE program, onto the EAP
program and into university entry programs and beyond,
their under-representation in the assessment system was a
significant drawback. 
Jarred Brinkman from Iowa State University presents at
CALT 2017.
3
Additionally, the work produced by students in response to
the standardized tasks seemed to us to demonstrate a very
narrow dynamic range . This is an audio production concept,
which very simply, refers to the difference in, say, a 
recording of a vocal performance between the loudest and
quietest notes; to standardize the listening experience 
across different playback devices (e.g., car stereos,
smartphones, etc.), this difference is typically reduced by
lowering the volume of the loud parts and raising the volume
of the quiet parts. We’ve applied the concept to language
assessment to refer to the qualitative differences (in relation
to criteria such as overall effectiveness, use of grammar and
vocabulary, etc.) between stronger and weaker responses by
students to test tasks, both written and spoken. We noticed
that the GE writing and speaking test tasks were significantly
reducing the dynamic range of responses–students of higher
ability were producing mediocre responses which were hard
to distinguish from those produced by students of lower
ability–making it harder for teachers to make confident
progression recommendations. 
Feedback to aid the learning and teaching processes had not
been given much attention during the development of the
assessment system; the emphasis was much more on the
summative function than the formative. Consequently, the
system did not facilitate direct or unambiguous
communication with students about their learning; rather, it
used “a form of code that only the most effective learners can
decode” (Boud, 2000, p. 155). This code took several different
forms (i.e., percentages, Satisfactory, Needs Work, error
correction codes) but these seemed likely to “act as a barrier
to student understanding” because, when using such a code,
“teachers encode their comments into a symbol whose
meaning is not shared only to have students attempt to
decode this symbol to gain feedback” (Ibid, p. 157).  
                                                                               continued on p. 4
Jacqueline Church and Cyndriel Miemban from
Northern Arizona University prepare for their talk.
"Implementing and learning from TBA" continued...
Implementing TBA AT QUTIC
Starting in March 2017, the current GE team, consisting
of the authors and English language teachers, took several
important steps away from this system towards TBA. We
organised a workshop with the teachers and discussed with
them Wiggins’s belief that “all assessment is subjective; the
task is to make the judgment defensible and credible”
(1994, p. 136). We informed them that, in order to achieve a
more defensible and credible assessment system, we
would:
     • Reduce the number of tests per sessions to just one per
       macro-skill
     • Use ‘assessment task cycles’ to ‘fill the gap’ created by
       the missing tests (Figure 1).  
ood fr Thought
Figure 1
4
In designing this ‘task cycle’, we had three goals:
    1. To improve the formative assessment in terms of both
         quantity and quality,
    2. To indicate to teachers and students the importance
        of giving, receiving and acting on feedback, and
    3. To help teachers gather a richer and more trustworthy
         representation of students’ capabilities and
         achievement to support summative judgments.
In Figure 1, the arrows labelled ‘Collect’ indicate stages
where the teacher could gather evidence of students’
capabilities, usually in the form of scanned documents or
audio recordings saved to a shared drive.
                                                                              continued on p. 9
 CALT Best Practices
This section presents recommended assessment practices
and principles that teachers should incorporate in the
courses they teach to maximize and measure student learning,
as well as to develop their own assessment literacy.
(Compiled by Eddy White, PhD) 

High-Quality Assessment
High-quality assessment practices are those that provide results verifying and promoting targeted student learning.
There are a number of fundamental aspects of such high-quality practices.
----------------------------------------------------------------------------------------------------------------------------------------------------------
                                                     High Quality Assessment Practices (Cheng & Fox, 2017)
----------------------------------------------------------------------------------------------------------------------------------------------------------
1. Alignment: The degree of agreement among curriculum, instruction, standards, and assessments (tests). In order to
achieve alignment, we need to select appropriate assessment methods which reflect or represent clear and appropriate
learning outcomes or goals.
----------------------------------------------------------------------------------------------------------------------------------------------------------
2. Validity: The appropriateness of inferences, uses, and consequences that result from assessment. This means that a
high-quality assessment process (i.e., the gathering, interpreting, and using of the information elicited) is sound,
trustworthy and legitimate based on the assessment results.
----------------------------------------------------------------------------------------------------------------------------------------------------------
3. Reliability: The consistency, stability, and dependability of the assessment results are related to reliability. This
quality criteria guards against the various errors of our assessments (e.g. inconsistency in marking student errors).
----------------------------------------------------------------------------------------------------------------------------------------------------------
4. Fairness: This is achieved when students are provided with an equal opportunity to demonstrate achievement and
assessment yields scores that are comparably valid. It requires transparency, in that all students know the learning
targets, criteria for success, and on what and how they will be assessed. Fairness also means students are given equal
opportunities to learn, and assessment tasks and procedures avoid stereotyping and bias.
----------------------------------------------------------------------------------------------------------------------------------------------------------
5. Consequences: This term is associated with the results of the use or misuse of assessment results. The term
washback-the influence of testing on teaching and learning- is now commonly used. Assessment can motivate (and
when it is of low quality, potentially demotivate) students to learn. The student-teacher relationship is influenced by
the nature of assessment.
----------------------------------------------------------------------------------------------------------------------------------------------------------
6. Practicality and Efficiency: Considerations are given to the information that is gathered by assessment. A teacher’s
life is extremely busy, and this influences the choice of assessment events, tools and processes. Are the resources,
effort and time required for assessment worth the investment? 
----------------------------------------------------------------------------------------------------------------------------------------------------------
Our classroom assessment principles and practices should incorporate these fundamental aspects of high-quality
assessment (EW). 

Best practices in this edition of the newsletter come from the recent book, Assessment in the Language Classroom
(Cheng & Fox, 2017, pp.11-12). 

5
Teacher as Learner BY MICHAEL SCHWARTZ
Considering the teacher also as a learner in the classroom (and St. Cloud University, MN
workplace), in this section we ask a member to describe a memorable

assessment experience and the learning that occurred.

   Authenticity, Integration, and Assessment: the format was too difficult or too easy? What

                    Lessons Learned would it ultimately tell us about our students’

listening and speaking skills? More specifically,

Norm referenced, criterion referenced, what would it tell us about their academic

summative, formative, discrete point, global, oral/aural skills? What impact would the test

top-down, bottom up, integrated, authentic, have on the students’ identities, motivation, and

teacher-made, standardized, multiple choice, fill investment?

in the blank, cloze, short answer, etc…. Am I

doing the right thing? These terms, their I decided to design an “integrative” test in which

applications, implementation, and most students listened to a talk from the popular TED

importantly, implications for students’ futures Talks website. Students were asked to listen to

were all swirling around in my head and turning the talk, take notes, and then answer several

in stomach as I was writing the mid-term short-answer or multiple-choice test items.

programmatic assessment instrument for the EAP Students were only allowed to listen once, since

Listening and Speaking bridge classes at the that is their reality in their other university

university where I work.  classes. The results were, as you might predict,

awful. To be fair, I decided to curve the test by

I have been working in adult ESL for over 25 30 points, allowing students who scored 40%,

years and have written unit tests and final exams 50%, and 60% to receive a passing grade. Of

for specific classes, proctored standardized tests, course, this meant that some students earned

served as a holistic grader for writing and over 100% but that was okay.

speaking assessments, and even directed

calibration trainings. However, not until I was In reflection, I learned several important things.

asked to take on the responsibilities of director First and foremost, I learned that you must try,

for an established English for Academic but that in trying you must also be open to

Purposes program, where teachers and staff had learning from your mistakes. My first attempt at

a better understanding of the semester processes program-wide assessment was a disaster, but

than I, have I been given the responsibility of with adjustments, the final exam will be a better

creating an assessment instrument intended to be instrument for measuring language development.

used for measuring student progress in the Second, I learned that while integrative

first seven weeks of the semester. I felt the assessment instruments are still the most

weight of the institution, the students’ futures, authentic, there are some realities to authenticity

the teacher’s expectations, and more on my that cannot be replicated within the context of

shoulders. The pressure had become nearly

debilitating. What if the instrument I created was                                           continued on p. 7

ineffective? What if

6
"Teacher as Learner" continued...

summative assessment. In this instance, I was perspective it is a crap-shoot in which they have

reminded that students had not been prepared for no control. Third, I know that I have a lot to

the Ted Talks topic in the same way that students learn about assessment and have compiled a

enrolled in Biology 101, for example, had been. holiday reading list. While it is important to be

After all, the students in Bio 101, have had half concerned about and to understand the terms and

of a semester’s worth of biology content processes identified at the beginning, ultimately,

presented to them. Thus, in this sense, an if you are fair to your students, then their

integrative skills assessment instrument is not identities, motivations, and investments will

authentic at all. Rather, from the students’ remain intact and they, too, will learn.

Left: Nasrin Nazemi from the University of

Washington presents at CALT 2017 in
Tucson.

Below Left: Justin Jernigan, Ph.D., from

Georgia Gwinnett College and Yingling Liu,
Ph.D., a visiting scholar at the University of
Arizona, give their conference presentation.

-----------------------------------------------
THE CALT REPORT CONTRIBUTORS
 DECEMBER 2017
Editor
Eddy White
Production & Design
Holly Wehmeyer
Contributors
Isabella Anikst
Jarred Brinkman
Jacqueline Church
Lauren Faull-Ortiz
Justin Jernigan
Yingliang Liu
Mariana Menchola-Blanco
Cyndriel Miemban
Nasrin Nazemi
Michael Schwartz
Kyle Smith
Michael Thomas
Emma Tudor
---------------------------------------------------
7
 Member Spotlight

Q. In your context, with your students, what assessment-

Emma Tudor related issues do you and your colleagues have?

A. Consistency of grading, calculating grade averages,

students comprehension of the grading system, and

Emma is the communication of that.

Director of

Academic Q. What specific assessment challenges do you or your

Management for colleagues face in the classroom?

A. A lot of our assessment is task based or communication

EF International
focused, which is quite subjective. It can, therefore, vary
Campuses in the
from class to class and teacher to teacher. While rubrics are
U.S. and Canada
provided, they can be interpreted differently. 

Q. Describe your current position and teaching situation.

Q. What have you learned about assessment in the past
A. I am currently the Director of Academic Management for
year that you have tried to incorporate into your teaching
EF International Campuses in the USA and Canada,
practices?
responsible for academics in 14 language schools.
A. The importance of transparency and communication about

assessment with students is vital. They need to know what

Q. What assessment responsibilities do you have?
they are being graded on, how that is weighted, and how they
A. I work with the global central team on developing
can improve upon that. I have encouraged giving students a
assessment policies and processes which are implemented
copy of the assessment criteria policy and talking through it
across all our worldwide schools.
with them in class, giving 1-to-1 feedback to students about

their grades and assessment results, and giving them advice

Q. Does your program have an Assessment Committee?
on how they can improve. Mentoring and providing feedback
A. Essentially, yes. It’s part of the central academic team
is an essential part of the assessment process, I believe.    
but we do not work solely on assessment.  

Q. What is an area of assessment that you want to learn

Q. If yes, how does the Assessment Committee function?
more about?
A. We have monthly online meetings and annual in persona
A. How we can be more consistent and accurate with
meetings where we review assessment policies and
formative assessment?
processes. We take feedback from the school staff, teachers,

and administrators, as well as students, and look at feedback

Q. When you want to forget about assessment, teaching,
and assessment results to inform decisions.
and work, what do you like to do to relax?  

A. Travelling the world and walking my Boston Terrier. 

8
"Implementing and learning from TBA" continued...
At this workshop in March, we also initiated a curriculum
development process comprising the following steps:
------------------------------------------------------------------------------
Step            Activity
------------------------------------------------------------------------------
   1                Teachers work in groups to design one
                     assessment task cycle (typically involving
                     integration of at least two macro-skills) for each
                     five-week stage of the course. The GE
                     Coordinator also designs task cycles.
------------------------------------------------------------------------------
   2                Completed task cycles are collected by the GE
                     Coordinator and shared with the entire GE
                     teaching team.
------------------------------------------------------------------------------
   3               New task cycles are implemented in classrooms
                    by teachers.
------------------------------------------------------------------------------
   4               Students’ spoken and written responses are
                    analysed for key performance features.
------------------------------------------------------------------------------
   5               Insights into the tasks, the teaching, and the
                    learners gained through these analyses lead to
                    further changes to assessment task design and
                    are shared with the teaching team.  students’ work. The tasks involve more authentic,
------------------------------------------------------------------------------
This process has been repeated seven times between April
and December 2017, with each iteration taking about five
weeks. The overall result has been to align the assessment
system more with the four key features of Davison and
Leung’s (2009) TBA model listed above. We have also learnt
a great deal about task design, our teachers, and also the
learners and their capabilities.
LESSONS LEARNED
Active involvement of students
TBA has yielded valuable insights for all of us about
involving students as active participants in their own
learning. We have learned that students can edit their work
effectively, making numerous grammar corrections with a
high degree of efficacy when converting their work to a
different mode (for instance transcribing an audio file or
typing written work) and with little or no intervention from
their teacher or just by referring back to a stimulus text.
Also, we have seen that students can justify changes made
to their work, demonstrating higher-order processing skills
(e.g., when asked to use the comment tool in Word to
defend changes to a draft, student comments focused on
clarity of message, the need for further explanation or
clarification, emphasis, missing key information, and
misquoted or incorrect details).
9
Dynamic range
One clear trend that has emerged from the collected data is
the distinction between strong, average, and weak student
performances. The integration of skills in the task cycles
and also the requirement for students to act on feedback as
they refine their initial responses helps to highlight students
who are confident responding to the challenges of the task
(and, by extension, more likely to cope well in the next stage
of their academic pathway); those with ability but who are
underperforming; and those with a potential need for
further support or intervention. Overall, we have found that
TBA creates more favorable conditions for students to
demonstrate their language abilities than the timed
impromptu independent writing tasks and, in this sense, is
more fair to students, although it is important to
acknowledge that, from a psychometric perspective,
reliability is somewhat reduced.
Teachers’ judgments
As samples of student work are collected throughout the
course, they reflect a wide variety of tasks, topics, and
approaches, and also highlight students’ abilities to act on
feedback and, in this way, our TBA system supports more
credible and defensible summative judgments about
spontaneous language and behaviors, and they are more
closely aligned with the practices and values that teachers
wish to assess. Also, through this collection process, we
have effectively compiled individual student portfolios
which have given us a clearer picture as to how different
individuals progress through our course and also some of
the risk factors which can help identify students who are not
making progress.
                                                                               continued on p. 10
Isabella Anikst and Michael Thomas from the UCLA Extension
listen to Eddy make an important point.
"Implementing and learning from TBA" continued...
TEACHER LEARNING / DEVELOPMENT
By involving teachers in all stages of the TBA decisions —
planning, designing, implementing, giving feedback, and
making judgment s— we feel that we have initiated a
teacher learning project which has been both challenging
and rewarding for all concerned. The collected evidence has
created a record of the GE teachers’ skills, approaches,
strengths, and learning, and indicated teacher learning in
the following areas:
     • Increased confidence in designing assessment tasks and
       ‘owning’ assessment decisions,
     • Feedback targeting practical steps to future
       improvement, and
     • Critical analysis of student work for evidence of
       comprehension of reading and listening texts.
Additionally, teacher energy has been redirected from
repetitively marking and preparing students for “highly
discrete and defined tasks” (Murphy, 1994, p. 194) which
generate ambiguous, coded information, to more authentic
tasks that integrate both skills and materials and provide
the basis for meaningful discussions with students
regarding what they can do next to further develop.      Classroom Assessment: Understanding and Using
CONCLUSION
To sum up, we have seen that an assessment system with
the TBA features described by Davison and Leung (2009) can
be successful in our context (i.e., an Australian university
language centre GE program). Not only that, we have
repeatedly seen teachers and students rising to the
challenge and producing work which is delightful,
surprising, and inspiring. Our misgivings about the previous
assessment system and our belief in TBA as a better
alternative have been, we feel, validated many times over
this year.
At the end of 2017, however, our assessment system is a
hybrid of TBA and the three standardized tests which occur
every five-week session. In 2018, our focus will be on
resolving the contradictions and tensions inherent in such a
hybrid, most likely by designing new tests which align more
closely with the TBA tasks. 
10
If you would like to keep up to date with these efforts, or
find out more about our activities this year, please feel free
to email us at k15.smith@qut.edu.au or
lauren.faullortiz@qut.edu.au.
NOTE
1. For more on this, see Fulcher, et al. (2011, p. 23), who
describe an approach to assessment based on ‘thick
description’ of student performance and “an analysis of how
people use language in actual communicative contexts”
which is more flexible than a traditional psychometric
approach and does “not assume a linear, unidimensional,
reified view of how second language learners
communicate”; and “pragmatic, focusing as [it does] upon
observable action and performance, while attempting to
relate actual performance to communicative competence.”
REFERENCES
Boud, D. (2000). Sustainable assessment: Rethinking
    assessment for the learning society. Studies in Continuing
    Education, 22(2), 151-167.
Butler, S. M., & McMunn, N. D. (2006). A Teacher’s Guide to
    Assessment to Improve Student Learning. Jossey-Bass.
Davison, C. & Leung, C. (2009). Current issues in language
    teacher-based assessment. TESOL Quarterly, 43(3), 393-
    415.
Fulcher, G., Davidson, F. & Kemp, J. (2011). ‘Effective rating
    scale development for speaking tests: Performance
    decision trees’. Language Testing, 28(1), 5-29.
Hamp-Lyons, L. (1994). Interweaving assessment and
    instruction in college ESL writing classes. College ESL,
    4(1), 43-55.
Murphy, S. (1994). Portfolios and curriculum reform:
    Patterns in practice. Assessing Writing, 1(2), 175-206.
Wiggins, G. (1994). The constant danger of sacrificing
    validity to reliability: Making writing assessment serve
    writers. Assessing Writing, 1(1), 129-139.