LING 906: Language

Testing & Evaluation

Assessing language
proficiency: concepts and
issues
A/P Mehdi Riazi
Macquarie University
 Review of last week’s
lecture
Type, uses, purposes, and functions of language
tests
– Screening or selection
– Diagnosis
– Placement
– Etc.
 Two major purposes

Students’ Students’
mastery of overall (global)
specific language ability
objectives

Achievement Proficiency
 Achievement Tests

Teacher-made Standardised
 Achievement Tests

Teacher-made Standardised

The National Assessment Program – Literacy

and Numeracy (NAPLAN) commenced in
Australian schools. Every year, all students in
Years 3, 5, 7 and 9 are assessed on the same
days using national tests in Reading, Writing,
Language Conventions (Spelling, Grammar
and Punctuation) and Numeracy
 Descriptions of the 10 Writing
criteria (standards)

Persuasive genre
Audience The writer’s capacity to orient,
engage and persuade the reader
Text structure The organisation of the structural
components of a persuasive text
(introduction, body and
conclusion) into an appropriate
and effective text structure
Ideas The selection, relevance and
elaboration of ideas for a
persuasive argument
Persuasive devices The use of a range of persuasive
devices to enhance the writer’s
position and persuade the reader
Vocabulary The range and precision of
contextually appropriate language
choices
Cohesion The control of multiple threads and
relationships across the text,
achieved through the use
of grammatical elements (referring
words, text connectives,
conjunctions) and lexical elements
(substitutions, repetitions, word
associations)

Paragraphing The segmenting of text into

paragraphs that assists the reader
to follow the line of argument

Sentence structure The production of grammatically

correct, structurally sound and
meaningful sentences

Punctuation The use of correct and appropriate

punctuation to aid the reading of
the text

Spelling The accuracy of spelling and the

difficulty of the words used
Our focus in these two
sessions
 Languageproficiency tests
–The concept
– Models
– Examples
 Approaches to prof. test
construction
1. Target Language Use (TLU) tasks
(what candidates are required to do in
the TLU and then creating the test)
2. Using proficiency frameworks (e.g.,
CEFR: hierarchically organised
description of language ability)
3. Construct-driven (defining or
modeling language proficiency and then
creating the test accordingly)
Task-based assessment
 TB test development procedure

• Define TLU domain

• Select tasks from TLU domain to
use as a basis for assessment
tasks
• Develop appropriate tasks for
assessment
The Occupational English
Test (OET)
 OET is a language test for
health practitioners who have qualified
to practise in one country and wish to
gain provisional registration to practise
their profession in another which has
an English-speaking context.
 The Test assesses English language
proficiency as it is used in medical and
health professions.
 Approaches to prof. test
construction
1. Target Language Use (TLU) tasks
(what candidates are required to do in
the TLU and then creating the test)
2. Using proficiency frameworks (e.g.,
CEFR: hierarchically organised
description of language ability)
3. Construct-driven (defining or
modeling language proficiency and then
creating the test accordingly)
Common European
Framework (CEF)
Level Definition
C2
C1 Proficient user
B2
B1 Independent user

A2
A1 Basic user
Construct-driven approach

 Defining and modeling the

construct of language proficiency

 Operationalisation of the model

through test design, test
specifications, and scoring rubrics
The Concept of Language
Proficiency
 Proficiency is a “construct” or “trait”
like “intelligence” or “motivation”

 Constructs or traits are invisible

 How to assess the constructs?

The process involved

Construct
Definition
(modeling)

Operationalisation
(Test/assessment
)
Constructs or Traits

Wechsler Adult
Definition
Intelligence Intelligence
(Model)
Scale

Scores (Operational
Trait or construct definition of intelligence)
The construct and its
operationalisation

Definition IELTS or TOEFL

Proficiency (Model) test

Scores (Operational
Trait or construct definition of proficiency)
The concept of construct
validity
 Validity: a) to what extent a test
tests what it claims to test, b) to what
extent our interpretation of the test
score reflects the test’s content
 Construct validity: to what extent
the test represents the underlying
construct
How is Proficiency
defined?
Different scholars and
test specialists have
defined “language
proficiency” differently
Alternative Terms
 Proficiency
 Competence
 Communicative Competence
 Knowledge of Language
 Language Ability
 Communicative Language Ability
(CLA)
Why do we need a
model?
 Tests of proficiency need to
be based on a sound and
acceptable underlying model
or theory of language
proficiency. Otherwise,
“construct validity” of the test
would be under question.
Models of L2 Proficiency
1. Skills and components model (Lado,
1960; Carroll, 1968)
2. Unitary competence hypothesis (Oller,
1976)
3. BICS/CALP (Cummins, 1979; 1983)
4. Communicative competence (Canale and
Swain, 1980; Canale, 1983)
5. Communicative language ability
(Bachman, 1990; Bachman & Palmer, 1996)
Skills & Component
Model
Underlying theories:

1. Structural Linguistics

2. Behavioral Psychology
Lado (1961)
Proficiency
Skills Components

Listening Grammar
Speaking Vocabulary
Reading Phonology
Writing
Features of tests in
skills/component model
 Discrete-point, Multiple-choice items
 Different sections on different skills
and components, but mostly
Listening, Reading, Gr., & Voc.
 Major emphasis on “linguistic
competence” (knowledge of lang)
Features of tests in
skills/component model
 Recognition items & tests
 Usage rather than use

 A good model for curriculum and

testing
 Difficult to make; easy to score

 Objective scoring
 Definition First generation
Proficiency (Model) of TOEFL test
Skills/Comp
onents
Scores (Operational
Trait or construct definition of proficiency)
First generation of TOEFL

 Skills
– Listening
– Reading
 Components
– Phonology (segmental & suprasegmental)
– Grammar (rules of language)
– Vocabulary (knowledge of words)
Oller (1976)
 Based on principal components
analysis of scores obtained from
numerous tests (vocabulary,
grammar, phonology, reading, etc.),
Oller (1976) defined proficiency in
terms of one general factor (g-
factor)
Unitary Comp. Hypothesis

 Thisrepresentation by
Oller is known as the
unitary competence
hypothesis (UCH)
Tests based on UCH
 Integrative tests including:
1. Dictation
2. Cloze-test
3. Composition
4. Oral interview
 Criticisms on UCH
 Conceptually: Derivation of one factor
is the result of the nature of the data
used (conscious & explicit features of
language & not functional &
sociolinguistics aspects)
 Methodologically: PCA is
inappropriate. It overestimates the sig. of
the first derived factor.
Criticisms on UCH

Oller (1983) himself:

 One-factor representation of lang
proficiency is implausible and

 that a general factor can be

divided into relatively more
analytic components