Developing the APTIS Testing

System
Professor Barry OSullivan
Senior Advisor, English Language Assessment

Overview
Developing theories
Localisation
Computerisation
Conclusions

conceptualising the test

DEVELOPING THEORIES

System Design

Thinking about Test Performance

Task
Performance
Parameters

Social Factors

Target
Language

Task Language

Personal
Characteristics

PERFORMANCE

Cognitive
Characteristics

Then Theres Scoring

Accuracy
Selection &
Training

Consistency
Answer Key or
Rating Scale

Machine or
Human

Interpretation

Validation Theory

Decisions, Decisions

Why are we
testing?

How Many Decisions?

Social Factors
Task
Performance
Parameters

Interlocutor [who;
number]
Relative Age, gender
etc.
Familiarity & Status
Etc.

Time & Support

Knowing Assessment
Criteria
Physical Conditions, etc.

Personal
Characteristics
Physical
Psychological
Experiential

Target
Language

Cognitive
Characteristics

Vocabulary &
Grammar
Function, Mode etc.

Maturity
Ability
Etc.

Task Language
Vocabulary &
Grammar
Genre, Mode, etc.

TEST
SCORE

Scoring System
Raters & Rating
Scales & Keys
Analysis & Grading
Etc.

Why all the Fuss?

Curriculum
explicit
implicit

Decisions
Delivery

Assessment

room design
materials
teachers

judgemental
developmental

theories of language progression

FOCUS ON READING

The Reading Model

How it Works
C

Inter Text
[full text level]
Inter Text
[paragraph level]

Text [external]
Text [within]

Paragraph
Word/Sentence

Introducing
CORE
[Grammar & Vocabulary]

Writing
Reading
Speaking
Listening

the british council approach

LOCALISATION

Theory to Practice Test Design

Appropriate
Content

Test
Taker

Appropriate
Topic
Appropriate
Task

Test
Task

Appropriate
Language
Appropriate
Scoring

LOCALISATION

Scoring
System

Defining Localisation
The process of ensuring that all construct-irrelevant
variance related to the test taking population is
controlled for in the design and delivery of a test

Testing a well defined population within a well defined context

in order to make decisions that will apply only to that context

Localisation Case Study

Curriculum

Delivery

Assessment

Case Study - Approach

Context

Localisation

Population 15 & 17 yrs

Secondary School
Mixed ability, Gender
Various L1 [multilingual]
Similar Culture
Similar Curriculum
Similar Delivery

Language Reviewed
[inc. instructions]
Reading Topics
Listening Topics
Writing Topics/Tasks
Speaking Topics
Speaking Images

the growing importance of the machine

COMPUTERISATION

The Computer & Test Practice

Research
Language
Data Analysis
Task Performance
Rating Performance

Test Scoring
Rater Training
Test Break Up
Scoring
Analysing/Awarding
Score Reporting

Test Taking
Knowledge
Receptive Skills
Productive Skills

System Management
Back Office
Reporting to Clients

Item Development
Specification
Writing
Trialling
Banking
Archiving
Test Delivery
Computer
Tablet
Other

The Computer & Test Practice

Item Development
Specification
Writing
Trialling
Banking
Archiving

whats the point?

CONCLUSIONS

The Message?
Every decision impacts on ALL stakeholders, though
especially on the test taker
Without a clearly defined ABILITY MODEL the test is
suspect
Without a clearly defined VALIDATION MODEL the test
is suspect
It is close to impossible to deliver any test without
technology

Professor Barry OSullivan

barry.osullivan@britishcouncil.org