Advancements in Large Scale Assessments Heiko Rolke

Advancements in Large-scale
Assessment
Heiko Rlke & Krisztina Toth
DIPF - Deutsches Institut fr Internationale
Pdagogische Forschung
Frankfurt am Main
TAO Days | 10.09.2012
Seite 1
Overview
Trends in LSA
Closer Look at Trends
Example: Making Use of Data
Challenges in LSA
TAO Days| 10.09.2012
Seite 2
Trends
Main trend: CBA
Sub-trends/effects of CBA:
CAT
towards formative assessment
Complex items
Simulation, interconnection
Interweaving
different tests, questionnaires, etc.
Big data
TAO Days | 10.09.2012
Seite 3
Trend: CBA
Computer-based Assessment has to serve a purpose
Costs?
Time!
Validity!
TAO Days | 10.09.2012
Seite 4
CAT
Also not practiced for its own sake
Not as important in pure summative assessment
But:
Strict time limits (e.g. PIAAC)
Growing demand for formative aspects in summative
assessment
TAO Days | 10.09.2012
Seite 5
Complex Items
Closer at reality
Simulation
Most important:
Ability to assess new domains
TAO Days | 10.09.2012
Seite 6
Examples of Complex Items
Simulation of an Email / Web Scenario

The test person receives an email and should book
cinema tickets online.
Dynamic Model (MicroDYN)

The test person should explore and master a dynamic
system with input (exogenous) variables influencing
output (endogenous) variables.
Automaton with Finite State Machine (MicroFIN)

The test person should interact with a mobile phone and set time to
summertime.
Interweaving of Tests and

Questionnaires
Framing
test only if preconditions are fulfilled
Double-check
findings of questionnaire
TAO Days | 10.09.2012
Seite 8
Big Data
Log data, not only results
Find out what is going on
Make use of data
E.g. partial scoring
-> Examples from

PIAAC PS-TRE
TAO Days | 10.09.2012
Seite 9
Simulation-based assessment
Modern assessments:
real life situations: e.g. web environments
complexity of instrument various ways
Monitoring students actions - log files
Analysis of test-taking paths is a relatively new field in education
(reason: printed vs. computer-based data collection)
Process data requires to consider and evaluate methods
Berlin, 10.9.2012 | H. Rlke & K. Tth | TAO Days 2012 | Advancements in Large-Scale Assessments
10
Aim
to investigate how log data can be integrated into the process of
educational assessment and evaluation to support researchers
and practitioners in making use of the data assembled in
computer-based test delivery
11
Aim
to investigate how log data can be integrated into the process of
educational assessment and evaluation to support researchers
and practitioners in making use of the data assembled in
computer-based test delivery
12
Hypertext Item
13
14
Example log
829709 15
{"sender":"15","type":"loading","data":""}
833463 15
{"sender":"15","type":"loaded","data":""}
848153 15
{"sender":"15","type":"user_interaction","":"<?xml version=\"1.0\">
\u000d\u000a<cbaloggingmodel:EmbeddedLinkLogEntry
xmlns:cbaloggingmodel=\"http://www.softcon.de/cba/cbaloggingmodel\" id=\b576759f1:-7fed\" sourcePageId=\"Item15_linklist\"
targetPageId=\"Item15_website1\"
textFieldId=\"cbaTextField_71"/>\u000d\u000a"}
849094 15
{"sender":"15","type":"variable_change","data":{"name":
"snapshot_url,"value":"http://localhost:8101/cba-runtime/itemjsessionid=
LB1?custom_servicehandler=downloadService&file=
C:\\...snapshot5537437125227002523.xml"}}
855852 15
{"sender":"15","type":"user_interaction","data":"<?xmlversion=
\"1.0\encoding=\"UTF8\"?>\u000d\u000a
<cbaloggingmodel:ButtonLogEntryxmlns:cbaloggingmodel=\"http://www.soft
con.de/cba/cbaloggingmodel\id=\"cbaBackButton_14_13454937565947\"/>
\u000d\u000a}":"15","type":"unloaded","data":""}
15
Promising methods
Statistics and visualisation
Clustering
Classification
16
Process Measures
1.
2.
3.
4.
5.
6.
7.
Number of page visits

Number of different page visits
Visit of relevant page
Time spent on the relevant page
Ratio of time spent on the relevant page
Ratio of time spent on the opening screen
Completion time
17
Statistics
Process measures
Mean
Number of page visits

Number of different page visits
Time spent on the relevant page
Ratio of time spent on the relevant page
Ratio of time spent on the opening screen
Completion time
5.18
2.38
10.87
.14
.64
67.74
Standard
Deviation
3.13
1.38
9.77
.11
.18
27.49
18
Distribution of different page

visits
19
Distribution of different page

visits
20
Visualisation
Berlin, 10.9.2012 | H. Rlke & K. Tth | TAO Days 2012 | Advancements in

21
Large-Scale Assessments
Visualisation
Berlin, 10.9.2012 | H. Rlke & K. Tth | TAO Days 2012 | Advancements in

22
Large-Scale Assessments
Cluster Analysis
Features
Cluster 1
Cluster 2
Cluster 3
Cluster 4
23%
16%
34%
27%
Nr. of page visits

Nr. of different page visits
Completion time
Relevant page visited (Y/N)
Ratio of time on relevant page
Ratio of time on starting page
Distribution of sequences (%)
Ratio of correct responses
23
Cluster Analysis
Features
Cluster 1
Cluster 2
Cluster 3
Cluster 4
23%
16%
34%
27%
25.0%
1.7%
92.6%
84.4%
Nr. of page visits

Completion time
24
Cluster Analysis
Features
Cluster 1
Cluster 2
Nr. of page visits
.72
5.41
.36
2.54
28.38
62.08
No
No
.93
.60
23%
16%
25.0%
1.7%
Completion time
Cluster 3
Cluster 4
25
Cluster Analysis
Features
Cluster 1
Cluster 3
Cluster 4
Nr. of page visits
4.73
9.57
2.29
4.53
Completion time
59.60
91.60
Yes
Yes
.20
.17
.62
.45
34%
27%
92.6%
84.4%
Cluster 2
26
Cluster Analysis
Features
Cluster 1
Cluster 2
Cluster 3
Nr. of page visits
5.41
4.73
2.54
2.29
Completion time
62.08
59.60
No
Yes
.20
.60
.62
16%
34%
1.7%
92.6%
Cluster 4
27
Cluster Analysis
Features
Cluster 1
Cluster 2
Cluster 3
Cluster 4
Nr. of page visits
.72
5.41
4.73
9.57
.36
2.54
2.29
4.53
28.38
62.08
59.60
91.60
No
No
Yes
Yes
.20
.17
.93
.60
.62
.45
23%
16%
34%
27%
25.0%
1.7%
92.6%
84.4%
Completion time
28
Summary and Conclusions
29
Summary and Conclusions

Features
Cluster 1
Cluster 2
Cluster 3
Cluster 4
Nr. of page visits
.72
5.41
4.73
9.57
.36
2.54
2.29
4.53
28.38
62.08
59.60
91.60
No
No
Yes
Yes
.20
.17
.93
.60
.62
.45
23%
16%
34%
27%
25.0%
1.7%
92.6%
84.4%
Completion time
30
Future work
Pilot study sample size
Validation
Other types of items require new process measures
31
32
Future work
Pilot study sample size
Validation
Other types of items require new process measures
We have a lot of work to do:
Software developer
Test developer
Psychometricans
33
(Selected) Challenges
Authoring and Management
Delivery
Re-use and exchange
TAO Days | 10.09.2012
Seite 34
Delivery
Re-use and exchange
TAO Days | 10.09.2012
Seite 35
Delivery
Re-use and exchange
TAO Days | 10.09.2012
Seite 36
Delivery
Re-use and exchange
TAO Days | 10.09.2012
Seite 37

Advancements in Large Scale Assessments Heiko Rolke

Încărcat de

Informații document

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Advancements in Large Scale Assessments Heiko Rolke

Încărcat de

Drepturi de autor:

Formate disponibile

Advancements in Large-scale

TAO Days| 10.09.2012

TAO Days | 10.09.2012

TAO Days | 10.09.2012

TAO Days | 10.09.2012

Examples of Complex Items

Simulation of an Email / Web Scenario

Dynamic Model (MicroDYN)

Automaton with Finite State Machine (MicroFIN)

Interweaving of Tests and

TAO Days | 10.09.2012

-> Examples from

TAO Days | 10.09.2012

Number of page visits

Number of page visits

Distribution of different page

Distribution of different page

Berlin, 10.9.2012 | H. Rlke & K. Tth | TAO Days 2012 | Advancements in

Berlin, 10.9.2012 | H. Rlke & K. Tth | TAO Days 2012 | Advancements in

Nr. of page visits

Ratio of correct responses

Nr. of page visits

Ratio of correct responses

Nr. of page visits

Nr. of different page visits

Ratio of time on relevant page

Ratio of time on starting page

Distribution of sequences (%)

Ratio of correct responses

Nr. of page visits

Nr. of different page visits

Relevant page visited (Y/N)

Ratio of time on relevant page

Ratio of time on starting page

Distribution of sequences (%)

Ratio of correct responses

Nr. of page visits

Nr. of different page visits

Ratio of time on relevant page

Ratio of time on starting page

Distribution of sequences (%)

Ratio of correct responses

Relevant page visited (Y/N)

Nr. of page visits

Nr. of different page visits

Ratio of time on relevant page

Ratio of time on starting page

Distribution of sequences (%)

Ratio of correct responses

Summary and Conclusions

Summary and Conclusions

Nr. of page visits

Nr. of different page visits

Ratio of time on relevant page

Ratio of time on starting page

Distribution of sequences (%)

Ratio of correct responses

TAO Days | 10.09.2012

TAO Days | 10.09.2012

TAO Days | 10.09.2012

TAO Days | 10.09.2012

S-ar putea să vă placă și