E

Educational Research and Innovation
Measuring Innovation
in Education
A NEW PERSPECTIVE
Centre for Educational Research and Innovation

Measuring Innovation
in Education
A NEW PERSPECTIVE
Stphan Vincent-Lancrin, Kiira Krkkinen,

Sebastian Pfotenhauer, Adele Atkinson, Gwnal Jacotin
and Michele Rimini
This work is published under the responsibility of the Secretary-General of the OECD. The
opinions expressed and arguments employed herein do not necessarily reflect the official
views of OECD member countries.
This document and any map included herein are without prejudice to the status of or
sovereignty over any territory, to the delimitation of international frontiers and boundaries
and to the name of any territory, city or area.
Please cite this publication as:

OECD (2014), Measuring Innovation in Education: A New Perspective, Educational Research and
Innovation, OECD Publishing.
http://dx.doi.org/10.1787/9789264215696-en
ISBN 978-92-64-21568-9 (print)

ISBN 978-92-64-21569-6 (PDF)
Series: Educational Research and Innovation

ISSN 2076-9660 (print)
ISSN 2076-9679 (online)
The statistical data for Israel are supplied by and under the responsibility of the relevant Israeli authorities. The use
of such data by the OECD is without prejudice to the status of the Golan Heights, East Jerusalem and Israeli
settlements in the West Bank under the terms of international law.
Photo credits: Cover Tycho Brahe, "Armill Zodicales", in Astronomi Instaurat Mecanica, Wandsbek 1598.
Original manuscript owned by The Royal Library of Copenhagen, Denmark
Corrigenda to OECD publications may be found on line at: www.oecd.org/about/publishing/corrigenda.htm.

OECD 2014
You can copy, download or print OECD content for your own use, and you can include excerpts from OECD publications, databases and
multimedia products in your own documents, presentations, blogs, websites and teaching materials, provided that suitable
acknowledgment of the source and copyright owner is given. All requests for public or commercial use and translation rights should be
submitted to rights@oecd.org. Requests for permission to photocopy portions of this material for public or commercial use shall be
addressed directly to the Copyright Clearance Center (CCC) at info@copyright.com or the Centre franais dexploitation du droit de copie
(CFC) at contact@cfcopies.com.
Foreword
Innovation drives improvement, either incrementally by advancing existing processes or more

radically by introducing new practices. Improving peoples life and education requires to better
document and understand what is (and will) change in education, a central mission of the OECD
Centre for Educational Research and Innovation.
While we have made tremendous progress over the past two decades in developing international
measures and indicators in education, measuring innovation in the education sector has long been
elusive. Measuring innovation in education, a first international compendium of measures of innovation
in education, makes a start and also opens perspectives for improved measures of innovation in the
education sector.
A commonly held belief is that education is not as innovative as other sectors of human activity.
One interesting result of the book is that this may be a myth. Another important feature of the book
is that it shows not only the volume of changes which countries have introduced in their primary
and secondary education systems over the past decade, but also what these changes or innovations
have been. This makes it a useful resource for policy makers wishing to explore to what extent their
intended policy reforms have had the expected impact on school and teacher practices.
Without observation, simulation and measurement, advances in knowledge and practice are
limited. Before becoming a pedagogic device, the armillary sphere on the cover of the book was
designed and improved to take new measures and make simulations that led to a dramatic shift
in understanding the complex movements of the universe. Now that we know that innovation in
education can be measured, and that having such measures can improve education policy and
practice, we should weigh the associated costs and benefits of the exercise. Measurement is indeed
a key element of any educational innovation strategy.
Andreas Schleicher
Director for Education and Skills
MEASURING INNOVATION IN EDUCATION : A NEW PERSPECTIVE OECD 2014 3

Acknowledgements
This book was authored by Stphan Vincent-Lancrin (OECD), Kiira Krkkinen (OECD), Sebastian
Pfotenhauer (Massachussets Institute of Technology), Adele Atkinson (OECD), Gwnal Jacotin
(OECD) and Michele Rimini (OECD), with the help of Michael Beeman during his internship at the
OECD. The book is an output of the Innovation Strategy for Education and Training by the OECD
Centre for Educational Research and Innovation (CERI), a project led and conceptualised by Stphan
Vincent-Lancrin (OECD).
Vincent-Lancrin, Krkkinen and Pfotenhauer designed the initial overall concept of the
book, and Vincent-Lancrin, Atkinson and Rimini subsequently designed the composite innovation
indices and Part 3. Vincent-Lancrin led the process and drafted the Overview; Krkkinen drafted
most of Part I with Rimini; Rimini, Atkinson and Karkkinen drafted Part II; Atkinson and Rimini
drafted Part III and Annexes; Jacotin did the statistical work and prepared the graphs. At different
stages of the process, Francesco Avvisati, Beat Bilbao-Osario, Alfonso Echazarra, Carlos Gonzalez-
Sancho, Francisco Martinez Toro and Richard Scott made helpful comments as members of the
CERI Innovation Strategy team. Dirk Van Damme, Head of the Innovation and Measuring Progress
division at the OECD Directorate for Education and Skills, supported the project from the beginning
and is thankfully acknowledged for his continued encouragements.
Within the OECD, colleagues from the Directorate for Science, Technology and Industry (STI),
notably Alessandra Colecchia, Fernando Galindo-Rueda and Vladimir Lopez-Bassols, gave helpful
feedback on a previous version of the report. Fred Gault (Merit University), Nathalie Greenan (Centre
dEtude de lEmploi) and Edward Lorenz (University of Nice) are also gratefully acknowledged
for enriching discussions with Vincent-Lancrin on innovation in the public sector and on the
measurement of innovation.
Colleagues from the Publications and Communication Directorate made helpful comments on
graph design and titles and are thankfully acknowledged. Lynda Hawe and Anne-Lise Prigent were
in charge of the publication process. RosZ did the layout of the book.
Last but not least, members of the CERI governing board are thanked for their comments and
input at different phases of the project. Rien Rouw (the Netherlands) made helpful comments
on a previous version of the draft. During the initial development phases of the project, several
experts and country representatives participated in two expert meetings held in Paris to discuss
different avenues to develop new instruments for the measurement of innovation in education.
Three countries (Denmark, France and the Netherlands) accepted to pilot a survey instrument. Even
though this initial project had to be abandoned, it was instrumental for the progress of the project.
It is indeed as the development of this project faced difficulties that the idea of a compendium
of measures of innovation in education based on existing data emerged. These three countries
and their representatives are thankfully acknowledged for supporting the project, notably Jrn
Skovsgaard (Denmark), Claude Sauvageot (France), and Hans Burgman (the Netherlands).
The book is dedicated to the memory of Claude Sauvageot.
4 MEASURING INNOVATION IN EDUCATION : A NEW PERSPECTIVE OECD 2014

Table of contents
Foreword...................................................................................................................................................3
Acknowledgements.................................................................................................................................4
Executive summary...............................................................................................................................15
Overview Why and how to measure innovation in education........................................................19
Objectives of the book............................................................................................................................................................ 20
Why measure innovation in education: context and rationales..................................................................................... 20
How innovation in education is measured in this publication........................................................................................ 22
Better understanding the role of innovation in education............................................................................................... 26
Summary of the main findings............................................................................................................................................. 27
How to improve the measurement of innovation in education ...................................................................................... 32
References................................................................................................................................................................................ 39
Readers guide........................................................................................................................................43
Part I
COMPARING INNOVATION IN EDUCATION WITH OTHER SECTORS
Chapter 1 Highly innovative workplaces in education and other sectors...................................51

Innovation occurrence and types within the education sector....................................................................................... 52
Innovation occurrence within the education sector compared with other sectors...................................................... 55
Innovation types within the education sector compared with other sectors................................................................ 58
Innovation within the education sector by level of education......................................................................................... 61
Chapter 2 Employee participation in innovation in education and other sectors.......................65

Innovation occurrence and types within the education sector ...................................................................................... 66
Chapter 3 Speed of adoption of innovation in education and other sectors................................79

Level of Innovation within the education sector............................................................................................................... 80
Lead innovation within the education sector compared with other sectors................................................................. 83
Chapter 4 Highly innovative jobs........................................................................................................91

Innovation occurrence and types within the education sector....................................................................................... 92
Innovation within the education sector by level of education....................................................................................... 101
Part II
INNOVATION AS CHANGE IN CLASSROOMS AND SCHOOLS
Chapter 5 Innovation in teaching style............................................................................................107

Use of front-of-class teaching............................................................................................................................................. 108
Independent work................................................................................................................................................................ 114
Summary................................................................................................................................................................................ 117

TABLE OF CONTENTS
Chapter 6 Innovation in instructional practices ............................................................................119

Relating lessons to real-life................................................................................................................................................. 120
Data and text interpretation............................................................................................................................................... 125
Reasoning............................................................................................................................................................................... 127
Summary .............................................................................................................................................................................. 131
Chapter 7 Innovation in class organisation.....................................................................................135

Group work............................................................................................................................................................................ 136
In-class ability grouping...................................................................................................................................................... 139
Self-directed work ................................................................................................................................................................ 142
Individualisation................................................................................................................................................................... 147
Summary................................................................................................................................................................................ 150
Chapter 8 Innovation in the use of textbooks in classrooms........................................................155

Use of textbooks as primary resources ............................................................................................................................. 156
Use of textbooks as supplementary resources ............................................................................................................... 160
Summary................................................................................................................................................................................ 163
Chapter 9 Innovation in the methods of assessment used in classrooms.................................167

Test based assessment ........................................................................................................................................................ 168
Non-test based assessments............................................................................................................................................... 170
Summary................................................................................................................................................................................ 173
Chapter 10 Innovation in the availability of computers and the internet in the classroom ......175
Computer availability .......................................................................................................................................................... 176
Network accessibility ......................................................................................................................................................... 179
Summary................................................................................................................................................................................ 182
Chapter 11 Innovation in the use of computers in the classroom...............................................185

Use of computers for practising skills and procedures................................................................................................... 186
Use of computers for information research ..................................................................................................................... 189
Use of computers for data analysis and scientific experimentation ........................................................................... 192
Use of computers for reading and writing ....................................................................................................................... 195
Summary................................................................................................................................................................................ 197
Chapter 12 Innovation in the provision of special education in schools....................................203

Remedial education ............................................................................................................................................................. 204
Enrichment education ......................................................................................................................................................... 208
Summary................................................................................................................................................................................ 212
Chapter 13 Innovation in the extent of teacher collaboration in schools...................................215

Instructional collaboration.................................................................................................................................................. 216
Classroom observations....................................................................................................................................................... 221
Summary................................................................................................................................................................................ 224
Chapter 14 Innovation in feedback mechanisms in schools.........................................................227

Progress benchmarking and monitoring .......................................................................................................................... 228
Curriculum and instruction development........................................................................................................................ 231
Summary................................................................................................................................................................................ 233
Chapter 15 Innovation in evaluation and hiring in schools..........................................................237

Use of data for teacher and principal evaluation............................................................................................................. 238
External evaluation............................................................................................................................................................... 241
Peer evaluation...................................................................................................................................................................... 244
Incentive use for recruitment and retention of teachers................................................................................................ 247
Summary................................................................................................................................................................................ 250

TABLE OF CONTENTS
Chapter 16 Innovation in schools external relations....................................................................255

Informing parents................................................................................................................................................................. 256
Parental involvement .......................................................................................................................................................... 260
Public relations...................................................................................................................................................................... 263
Summary................................................................................................................................................................................ 266
Chapter 17 Composite indices of innovation in classrooms and schools...................................271

Composite innovation indices............................................................................................................................................ 272
Part III
CHANGE IN EDUCATIONAL OUTCOMES
Chapter 18 Innovation and educational outcomes........................................................................279

Composite innovation indices and learning outcomes................................................................................................... 280
Composite innovation indices, equality and equity........................................................................................................ 290
Composite innovation indices and changes in education expenditure....................................................................... 298
Composite innovation indices and satisfaction............................................................................................................... 301
Annex A Data sources and methods................................................................................................309

Data sources: overview........................................................................................................................................................ 310
Coverage of the statistics..................................................................................................................................................... 310
Country coverage.................................................................................................................................................................. 311
Sample sizes.......................................................................................................................................................................... 311
Year coverage......................................................................................................................................................................... 317
Calculation of cross-country means and totals................................................................................................................ 317
Calculation of effect sizes.................................................................................................................................................... 317
Further resources.................................................................................................................................................................. 318
Annex B Composite innovation indices...........................................................................................319

Creating the indices.............................................................................................................................................................. 320
Missing values....................................................................................................................................................................... 321
Criteria for including questions in the indices................................................................................................................. 322
Developing and reporting the indices................................................................................................................................ 324
Cautions................................................................................................................................................................................. 324
Countries top 5 areas of classroom-level innovations..................................................................327

Countries top 5 areas of school-level innovations.........................................................................329
Country index......................................................................................................................................331

TABLE OF CONTENTS
Figures
Figure1.1 Education professionals in highly innovative workplaces regarding at least one type of innovation, by country.............. 53
Figure1.2 Education professionals in highly innovative workplaces across three types of innovation, by country.............................. 53
Figure1.3 Education professionals in highly innovative workplaces, by innovation type and country..................................................... 54
Figure1.4 Professionals in highly innovative workplaces regarding at least one type of innovation, by sector .................................... 56
Figure1.5 Professionals in highly innovative workplaces regarding at least one type of innovation, by sector and country ............. 56
Figure1.6 Professionals in highly innovative workplaces across three types of innovation, by sector ..................................................... 57
Figure1.7 Professionals in highly innovative workplaces across three types of innovation, by sector and country............................. 57
Figure1.8 Professionals in highly innovative workplaces, by sector and innovation type............................................................................ 59
Figure1.9 Professionals in highly innovative workplaces regarding knowledge or methods innovation, by sector and country ..... 59
Figure1.10 Professionals in highly innovative workplaces regarding product or service innovation, by sector and country............. 60
Figure1.11 Professionals in highly innovative workplaces regarding technology, tools or instruments innovation, by sector and country... 60
Figure1.12 Education professionals in highly innovative workplaces, by innovation type and education level.................................... 62
Figure1.13 Professionals in highly innovative workplaces across three types of innovation, by sector and education level ............. 63
Figure1.14 Professionals in highly innovative workplaces, by innovation type, sector and education level .......................................... 63
Figure2.1 Participation of education professionals in the introduction of at least one type of innovation, by country....................... 67
Figure2.2 Participation of education professionals in the introduction of three types of innovation, by country................................. 67
Figure2.3 Participation of education professionals in innovation, by innovation type and country.......................................................... 68
Figure2.4 Professionals participating in innovation, by sector............................................................................................................................ 70
Figure2.5 Professionals participating in innovation, by sector and country................................................................................................... 70
Figure2.6 Professionals participating across three types of innovation, by sector........................................................................................ 71
Figure2.7 Professionals participating across three types of innovation, by sector and country.................................................................. 71
Figure2.8 Professionals participating in innovation, by sector and innovation type..................................................................................... 73
Figure2.9 Professionals participating in knowledge or methods innovation, by sector and country........................................................ 73
Figure2.10 Professionals participating in product or service innovation, by sector and country .............................................................. 74
Figure2.11 Professionals participating in technology, tools or instruments innovation, by sector and country ................................... 74
Figure2.12 Professionals participating in innovation, by innovation type and education level ................................................................. 76
Figure2.13 Professionals participating in innovation across three types of innovation, by sector and education level and sectors of the economy.... 77
Figure2.14 Professionals participating in innovation, by innovation type, sector and education level..................................................... 77
Figure3.1 Education professionals in workplaces at least sometimes quick to adopt innovation, by country....................................... 81
Figure3.2 Education professionals in highly innovative workplaces at least sometimes quick to adopt innovation, by country..... 81
Figure3.3 Education professionals in workplaces mostly quick to adopt innovation, by country ............................................................. 82
Figure3.4 Education professionals in highly innovative workplaces mostly quick to adopt innovation, by country............................ 82
Figure3.5 Professionals in workplaces at least sometimes quick to adopt innovation, by sector .............................................................. 84
Figure3.6 Professionals in highly innovative workplaces at least sometimes quick to adopt innovation, by sector ........................... 84
Figure3.7 Professionals in workplaces mostly quick to adopt innovation, by sector .................................................................................... 85
Figure3.8 Professionals in highly innovative workplaces mostly quick to adopt innovation, by sector ................................................... 85
Figure3.9 Professionals in workplaces at least sometimes quick to adopt innovation, by sector and country ..................................... 86
Figure3.10 Professionals in workplaces mostly quick to adopt innovation, by sector and country .......................................................... 86
Figure3.11 Professionals in highly innovative workplaces at least sometimes quick to adopt innovation, by sector and country .. 87
Figure3.12 Professionals in highly innovative workplaces mostly quick to adopt innovation, by sector and country ........................ 87
Figure3.13 Professionals in workplaces quick to adopt innovation, by sector and education level........................................................... 89
Figure3.14 Professionals in highly innovative workplaces quick to adopt innovation, by sector and education level ........................ 90
Figure4.1 Education professionals in highly innovative jobs regarding at least one type of innovation, by country............................. 93
Figure4.2 Education professionals in highly innovative jobs in education across three types of innovation, by country................... 93
Figure4.3 Education professionals in highly innovative jobs in education, by innovation type and country......................................... 94

TABLE OF CONTENTS
Figure4.4 Professionals in highly innovative jobs, by sector................................................................................................................................ 96

Figure4.5 Professionals in highly innovative jobs, by sector and country........................................................................................................ 96
Figure4.6 Professionals in highly innovative jobs across three types of innovation, by sector................................................................... 97
Figure4.7 Professionals in highly innovative jobs across three types of innovation, by sector and country........................................... 97
Figure4.8 Professionals in highly innovative jobs, by sector and innovation type.......................................................................................... 99
Figure4.9 Professionals in highly innovative jobs regarding knowledge or methods innovation, by sector and country.................... 99
Figure4.10 Professionals in highly innovative jobs regarding product or service innovation, by sector and country......................... 100
Figure4.11 Professionals in highly innovative jobs regarding technology, tools or instruments innovation, by sector and country... 100
Figure4.12 Education professionals in highly innovative jobs, by innovation type and education level................................................. 102
Figure4.13 Professionals in highly innovative jobs across three types of innovation, by sector and education level ........................ 103
Figure4.14 Professionals in highly innovative jobs, by innovation type, sector and education level ...................................................... 103
Figure5.1 Time spent on lecture-style presentations in 8th grade maths, according to teachers............................................................... 109
Figure5.2 Time spent on lecture-style presentations in 8th grade maths, according to students.............................................................. 109
Figure5.3 Time spent on lecture-style presentations in 8th grade science, according to teachers............................................................. 110
Figure5.4 Time spent on lecture-style presentations in 8th grade science, according to students............................................................. 110
Figure5.5 Watching the teacher demonstrate an experiment or investigation in 8th grade science, according to students ............. 111
Figure5.6 Time spent on lecture-style presentations in 4th grade maths, according to teachers............................................................... 112
Figure5.7 Watching the teacher demonstrate an experiment or investigation in 4th grade science, according to students............... 112
Figure5.8 Reading aloud to 4th grade students, according to teachers............................................................................................................. 113
Figure5.9 Time spent on independent work in 8th grade maths, according to teachers.............................................................................. 115
Figure5.10 Time spent on independent work in 8th grade science, according to teachers.......................................................................... 115
Figure5.11 Time spent on independent work in 4th grade maths, according to teachers............................................................................ 116
Figure5.12 Students reading silently alone in 4th grade, according to teachers............................................................................................. 116
Figure5.13 Change in time spent on lecture-style presentations in 8th grade, according to teachers...................................................... 117
Figure5.14 Change in watching the teacher demonstrate an experiment or investigation in at least half the lessons, according to students...... 117
Figure6.1 Relating 8th grade maths learning to students daily life, according to teachers......................................................................... 121
Figure6.2 Relating 8th grade maths learning to students daily life, according to students......................................................................... 121
Figure6.3 Relating 8th grade science learning to students daily life, according to teachers........................................................................ 122
Figure6.4 Relating 8th grade science learning to students daily life, according to students ...................................................................... 122
Figure6.5 Relating 4th grade maths learning to students daily life, according to teachers......................................................................... 123
Figure6.6 Relating 4th grade science learning to students daily life, according to students....................................................................... 123
Figure6.7 Relating 4th grade reading to students own experience, according to teachers.......................................................................... 124
Figure6.8 Interpreting data in tables, figures or graphs in 8th grade maths, according to students.......................................................... 126
Figure6.9 Students making generalisations and drawing inferences from a text in 4th grade reading, according to teachers.......... 126
Figure6.10 Students explaining answers during 8th grade maths lessons, according to teachers............................................................ 128
Figure6.11 Students explaining answers during 8th grade maths lessons, according to students............................................................ 128
Figure6.12 Students explaining what they are studying during 8th grade science lessons, according to teachers............................... 129
Figure6.13 Students observing and describing natural phenomena during 8th grade science lessons, according to teachers.......... 129
Figure6.14 Students explaining answers during 4th grade maths lessons, according to teachers............................................................ 130
Figure6.15 Students explaining what they are studying during 4th grade science lessons, according to teachers............................... 130
Figure6.16 Change in students relating what they learn to their daily lives in 8th grade............................................................................ 131
Figure6.17 Change in students relating what they learn to their daily lives in 4th grade ........................................................................... 131
Figure6.18 Change in students reasoning in 8th grade ....................................................................................................................................... 132
Figure6.19 Change in students reasoning in 4th grade ....................................................................................................................................... 132
Figure7.1 8th grade maths students working in small groups, according to students ................................................................................. 137
Figure7.2 8th grade science students working in small groups, according to students................................................................................ 137
Figure7.3 4th grade students reading aloud in small groups or pairs, according to teachers ..................................................................... 138
Figure7.4 4th grade science students working in small groups, according to students................................................................................ 138

TABLE OF CONTENTS
Figure7.5 Within class, same-ability grouping policy in secondary education ............................................................................................. 140
Figure7.6 4th grade students in same-ability groups for reading, according to teachers ............................................................................. 140
Figure7.7 4th grade students in mixed-ability groups for reading, according to teachers ........................................................................... 141
Figure7.8 8th grade maths students asked to decide on procedures for solving complex problems, according to teachers ............. 143
Figure7.9 8th grade maths students asked to decide on procedures for solving complex problems, according to students............. 143
Figure7.10 8th grade science students asked to design or plan experiments, according to teachers ....................................................... 144
Figure7.11 8th grade science students asked to design or plan experiments, according to students....................................................... 144
Figure7.12 4th grade science students asked to design or plan experiments, according to teachers ....................................................... 145
Figure7.13 4th grade science students asked to design or plan experiments, according to students....................................................... 145
Figure7.14 4th grade students reading books that they choose themselves, according to teachers ......................................................... 146
Figure7.15 4th grade students reading books that they choose themselves, according to students.......................................................... 146
Figure7.16 Teachers meeting individual students needs in secondary education...................................................................................... 148
Figure7.17 Student perception of teacher willingness to help them individually in secondary education............................................ 148
Figure7.18 4th grade students receiving individualised instruction in reading, according to teachers .................................................... 149
Figure7.19 Change in 8th grade students working in small groups, according to students ........................................................................ 150
Figure7.20 Change in 4th grade students working in small groups, according to students ........................................................................ 150
Figure7.21 Change in 8th grade students being asked to decide and design procedures............................................................................. 151
Figure7.22 Change in 4th grade students being asked to decide and design procedures............................................................................. 151
Figure8.1 Textbooks as a primary resource for 8th grade mathematics instruction...................................................................................... 157
Figure8.2 Textbooks as a primary resource for 8th grade science instruction................................................................................................. 157
Figure8.3 Textbooks as a primary resource for 4th grade mathematics instruction...................................................................................... 158
Figure8.4 Textbooks as a primary resource for 4th grade science instruction................................................................................................. 158
Figure8.5 Textbooks as a primary resource for 4th grade reading instruction................................................................................................ 159
Figure8.6 Textbooks as a supplementary resource for 8th grade mathematics instruction........................................................................ 161
Figure8.7 Textbooks as a supplementary resource for 8th grade science instruction................................................................................... 161
Figure8.8 Textbooks as a supplementary resource for 4th grade mathematics instruction........................................................................ 162
Figure8.9 Textbooks as a supplementary resource for 4th grade science instruction................................................................................... 162
Figure8.10 Change in the use of textbooks as a primary resource for 8th grade instruction ...................................................................... 163
Figure8.11 Change in the use of textbooks as a primary resource for 4h grade instruction...................................................................... 163
Figure8.12 Change in the use of textbooks as a supplementary resource for 8th grade instruction......................................................... 164
Figure8.13 Change in the use of textbooks as a supplementary resource for 4th grade instruction......................................................... 164
Figure9.1 Use of standardised tests as an assessment method in classrooms............................................................................................. 169
Figure9.2 Use of teacher-developed tests as an assessment method.............................................................................................................. 169
Figure9.3 Use of teachers judgmental ratings as an assessment method.................................................................................................... 171
Figure9.4 Use of students portfolio as an assessment method....................................................................................................................... 171
Figure9.5 Use of students assignments/projects/homework as an assessment method ......................................................................... 172
Figure9.6 Change in the use of test-based assessment methods in classrooms.......................................................................................... 173
Figure9.7 Change in the use of non-test-based assessment methods in classrooms................................................................................. 173
Figure10.1 Computer availability in 8th grade maths lessons ............................................................................................................................ 177
Figure10.2 Computer availability in 8th grade science classrooms.................................................................................................................... 177
Figure10.3 Computer availability in 4th grade maths classrooms..................................................................................................................... 178
Figure10.4 Computer availability in 4th grade science classrooms.................................................................................................................... 178
Figure10.5 Internet availability in the 8th grade maths classrooms.................................................................................................................. 180
Figure10.6 Internet availability in the 8th grade science classrooms................................................................................................................ 180
Figure10.7 Internet availability in the 4th grade maths classrooms.................................................................................................................. 181
Figure10.8 Internet availability in the 4th grade science classrooms................................................................................................................ 181
Figure10.9 Change in computer availability in 8th grade classrooms .............................................................................................................. 182
Figure10.10 Change in computer availability in 4th grade classrooms............................................................................................................. 182

TABLE OF CONTENTS
Figure10.11 Change in Internet availability in 8th grade classrooms................................................................................................................ 183

Figure10.12 Change in Internet availability in 4th grade classrooms................................................................................................................ 183
Figure11.1 Computer use to practice skills and procedures in 8th grade maths ........................................................................................... 187
Figure11.2 Computer use to practice skills and procedures in 8th grade science ......................................................................................... 187
Figure11.3 Computer use to practice skills and procedures in 4th grade maths ........................................................................................... 188
Figure11.4 Computer use to practice skills and procedures in 4h grade science ......................................................................................... 188
Figure11.5 Computer use to look up ideas and information in 8th grade maths .......................................................................................... 190
Figure11.6 Computer use to look up ideas and information in 8th grade science ........................................................................................ 190
Figure11.7 Computer use to look up ideas and information in 4th grade maths .......................................................................................... 191
Figure11.8 Computer use to look up ideas and information in 4th grade science ........................................................................................ 191
Figure11.9 Computer use to analyse data in 8th grade maths ........................................................................................................................... 193
Figure11.10 Computer use to analyse data in 8th grade science ....................................................................................................................... 193
Figure11.11 Computer use to conduct scientific experiments in 8th grade science ..................................................................................... 194
Figure11.12 Computer use to study natural phenomena through simulations in 4th grade science ....................................................... 194
Figure11.13 Computer use to read text in 4th grade ............................................................................................................................................. 196
Figure11.14 Computer use to write text in 4th grade ........................................................................................................................................... 196
Figure11.15 Change in the use of computers to practice skills and procedures in 8th grade ..................................................................... 197
Figure11.16 Change in the use of computers to practice skills and procedures in 4th grade ..................................................................... 197
Figure11.17 Change in the use of computers to look up ideas and information in 8th grade .................................................................... 198
Figure11.18 Change in the use of computers to look up ideas and information in 4th grade .................................................................... 198
Figure11.19 Change in the use of computers to analyse data in 8th grade ..................................................................................................... 198
Figure11.20 Change in the use of computers to study and conduct scientific experiments .................................................................... 199
Figure11.21 Change in the use of computers to read or write text in 4th grade ............................................................................................ 199
Figure12.1 Remedial mathematics provision in 8th grade .................................................................................................................................. 205
Figure12.2 Remedial science provision in 8th grade ............................................................................................................................................. 205
Figure12.3 Remedial mathematics provision in 4th grade .................................................................................................................................. 206
Figure12.4 Remedial science provision in 4th grade ............................................................................................................................................. 206
Figure12.5 Availability of specialists to work with 4th grade students with reading difficulties................................................................ 207
Figure12.6 Enrichment mathematics provision in 8th grade.............................................................................................................................. 209
Figure12.7 Enrichment science provision in 8th grade......................................................................................................................................... 209
Figure12.8 Enrichment mathematics provision in 4th grade.............................................................................................................................. 210
Figure12.9 Enrichment science provision in 4th grade......................................................................................................................................... 210
Figure12.10 Provision of informal initiatives to encourage 4th grade students to read................................................................................ 211
Figure12.11 Change in the provision of remedial education in 8th grade ....................................................................................................... 212
Figure12.12 Change in the provision of remedial education in 4th grade ....................................................................................................... 212
Figure12.13 Change in the provision of enrichment education in 8th grade ................................................................................................. 213
Figure12.14 Change in the provision of enrichment education in 4th grade .................................................................................................. 213
Figure12.15 Change in the provision of special education in 4th grade reading............................................................................................. 213
Figure13.1 Peer discussions amongst 8th grade mathematics teachers........................................................................................................... 217
Figure13.2 Peer discussions amongst 8th grade science teachers ..................................................................................................................... 217
Figure13.3 Peer discussions amongst 4th grade teachers ................................................................................................................................... 218
Figure13.4 Teacher collaboration in preparing instructional materials in 8th grade mathematics ........................................................... 219
Figure13.5 Teacher collaboration in preparing instructional materials in 8th grade science ...................................................................... 219
Figure13.6 Teacher collaboration in preparing instructional materials in 4th grade .................................................................................... 220
Figure13.7 Classroom observations in 8th grade mathematics.......................................................................................................................... 222
Figure13.8 Classroom observations in 8th grade science..................................................................................................................................... 222
Figure13.9 Classroom observations in 4th grade.................................................................................................................................................... 223
Figure13.10 Change in the frequency of teachers peer discussions............................................................................................................... 224

TABLE OF CONTENTS
Figure13.11 Change in the frequency of teacher collaboration in preparing instructional materials .................................................... 224
Figure13.12 Change in the frequency of classroom observations ................................................................................................................... 225
Figure14.1 Use of student assessments for district or national benchmarking............................................................................................ 229
Figure14.2 Use of student assessments for between-school comparisons ................................................................................................... 229
Figure14.3 Use of student assessments for monitoring progress over time ................................................................................................. 230
Figure14.4 Administrative tracking of school achievement data ..................................................................................................................... 230
Figure14.5 Use of student assessments for instructional or curriculum improvement ............................................................................ 232
Figure14.6 Principals time devoted to curriculum and pedagogy development at the 4th grade.............................................................. 232
Figure14.7 Change in the use of student assessments for external comparison ........................................................................................ 233
Figure14.8 Change in the use of student assessments to monitor progress and to improve curriculum in secondary education..... 233
Figure15.1 Use of assessment data to make judgments about teachers effectiveness ............................................................................. 239
Figure15.2 Use of achievement data for evaluating teachers performance ................................................................................................. 239
Figure15.3 Use of achievement data for the evaluation of principals performance ................................................................................... 240
Figure15.4 Evaluation of teachers practices by inspector observation in 8th grade mathematics .......................................................... 242
Figure15.5 Evaluation of teachers practices by inspectors observation in 8th grade science .................................................................. 242
Figure15.6 Evaluation of teachers practices by inspectors observation in 4th grade ................................................................................. 243
Figure15.7 Peer review evaluation of teachers practices in 8th grade mathematics .................................................................................... 245
Figure15.8 Peer review evaluation of teachers practices in 8th grade science ............................................................................................... 245
Figure15.9 Peer review evaluation of teachers practices in 4th grade .............................................................................................................. 246
Figure15.10 Use of incentives for teacher recruitment and retention in 8th grade mathematics.............................................................. 248
Figure15.11 Use of incentives for teacher recruitment and retention in 8th grade science......................................................................... 248
Figure15.12 Use of incentives for teacher recruitment and retention in 4th grade ....................................................................................... 249
Figure15.13 Change in the use of achievement and assessment data for school evaluation.................................................................... 250
Figure15.14 Change in the evaluation of teachers practices by inspectors observation .......................................................................... 250
Figure15.15 Change in peer review evaluation of teachers practices ........................................................................................................... 251
Figure15.16 Change in the use of incentives for teachers recruitment and retention .............................................................................. 251
Figure16.1 Parental perceptions of receiving regular information about their childs progress................................................................ 257
Figure16.2 Use of assessment data to inform parents about their childs progress..................................................................................... 257
Figure16.3 Information provided to parents about their childs performance relative to other students in the same school......... 258
Figure16.4 Information provided to parents about their childs performance relative to national or regional benchmarks ............ 258
Figure16.5 Information provided to parents about student group performance relative to students in the same grade in other schools ... 259
Figure16.6 Schools asking parents of 8th grade students to serve on committees ....................................................................................... 261
Figure16.7 Schools asking parents of 4th grade students to serve on committees ....................................................................................... 261
Figure16.8 Schools asking parents of 8th grade students to volunteer for projects ...................................................................................... 262
Figure16.9 Schools asking parents of 4th grade students to volunteer for projects ...................................................................................... 262
Figure16.10 Publicly posting school achievement data ...................................................................................................................................... 264
Figure16.11 Public relations and fundraising by school principals................................................................................................................... 264
Figure16.12 Time devoted to community/parent relations by 4th grade principals....................................................................................... 265
Figure16.13 Change in provision of comparative information on students performance......................................................................... 266
Figure16.14 Change in school involvement by parents of 8th grade students ............................................................................................... 266
Figure16.15 Change in school involvement by parents of 4th grade students................................................................................................ 267
Figure16.16 Change in public relation practices of schools .............................................................................................................................. 267
Figure17.1 Overall composite education innovation index, 2000-2011........................................................................................................... 273
Figure17.2 School and classroom level composite innovation indices, 2000-2011 ...................................................................................... 273
Figure17.3 Subject-level composite innovation indices for maths and science education, 2000-2011.................................................... 273
Figure17.4 Composite innovation indices for 8th grade and 4th grade, 2000-2011.......................................................................................... 274
Figure18.1 Overall education innovation and 8th grade mathematics outcomes.......................................................................................... 281
Figure18.2 School innovation and 8th grade mathematics outcomes.............................................................................................................. 281

TABLE OF CONTENTS
Figure18.3 Classroom innovation and 8th grade mathematics outcomes....................................................................................................... 282

Figure18.4 Innovation in mathematics and 8th grade mathematics outcomes............................................................................................. 284
Figure18.5 Innovation at 8th grade and 8th grade mathematics outcomes...................................................................................................... 284
Figure18.6 Innovation at 4th grade and 4th grade mathematics outcomes..................................................................................................... 284
Figure18.7 Overall education innovation and 8th grade mathematics outcome trends............................................................................... 285
Figure18.8 School innovation and 8th grade mathematics outcome trends................................................................................................... 286
Figure18.9 Classroom innovation and 8th grade mathematics outcome trends ........................................................................................... 286
Figure18.10 Overall education innovation and 8th grade science outcome trends........................................................................................ 288
Figure18.11 School innovation and 8th grade science outcome trends............................................................................................................ 288
Figure18.12 Classroom innovation and 8th grade science outcome trends..................................................................................................... 289
Figure18.13 Overall education innovation and students performance variation in reading.................................................................... 291
Figure18.14 Schools innovation and students performance variation in reading ...................................................................................... 291
Figure18.15 Classroom innovation and students performance variation in reading ................................................................................ 292
Figure18.16 Innovation at 8th grade and students performance variation in reading ................................................................................ 292
Figure18.17 Overall education innovation and equity of learning opportunities......................................................................................... 294
Figure18.18 School innovation and equity of learning opportunities.............................................................................................................. 294
Figure18.19 Classroom innovation and equity of learning opportunities .................................................................................................... 295
Figure18.20 Overall education innovation and variation in equity between schools.................................................................................. 297
Figure18.21 Overall education innovation and variation in equity within schools ................................................................................ 297
Figure18.22 Overall education innovation and trend in expenditure per student ...................................................................................... 299
Figure18.23 School innovation and trend in expenditure per student ........................................................................................................... 299
Figure18.24 Classroom Innovation and trend in expenditure per student ................................................................................................... 300
Figure18.25 Overall education innovation and change in 8th grade maths teacher satisfaction .............................................................. 302
Figure18.26 School innovation and change in 8th grade maths teacher satisfaction ................................................................................... 302
Figure18.27 Classroom innovation and change in 8th grade maths teacher satisfaction ........................................................................... 303
Figure18.28 Overall education innovation and change in 8th grade student satisfaction ........................................................................... 305
Figure18.29 School innovation and change in 8th grade student satisfaction ............................................................................................... 305
Figure18.30 Classroom innovation and change in 8th grade student satisfaction ........................................................................................ 306

TABLE OF CONTENTS
Tables
Table 5.1. Effect sizes for changes in front-of-class practices and use of independent work .................................................................... 118
Table 6.1 Effect sizes for changes in relating lessons to real life, interpreting data and text, and reasoning.............................................133
Table 7.1. Effect sizes for changes in the use of grouping, self-directed work and individualisation in classrooms........................................... 152
Table 8.1. Effect sizes for changes in the use of textbooks ................................................................................................................................. 165
Table 9.1 Effect sizes for changes in assessment methods ................................................................................................................................ 174
Table 10.1 Effect sizes for changes in the availability of computers and the internet in classrooms....................................................... 184
Table 11.1 Effect sizes for changes in the use of computers in classrooms.................................................................................................... 200
Table 12.1 Effect sizes for changes in the provision of special education ....................................................................................................... 214
Table 13.1 Effect sizes for changes in teacher collaboration practices ............................................................................................................ 226
Table 14.1 Effect sizes for changes in benchmarking, monitoring and feedback in school education.................................................... 234
Table 15.1 Effect sizes for changes in school evaluation and hiring practices ............................................................................................... 252
Table 15.1 Effect sizes for changes in school evaluation and hiring practices (continued)......................................................................... 253
Table 16.1 Effect sizes for changes in school external relations practices ...................................................................................................... 268
Table 16.1 Effect sizes for changes in school external relations practices (continued)................................................................................. 269
Table 17.1 Overall composite education innovation index, 2000-2011............................................................................................................. 275
Table 18.1 Correlations between innovation indices in education and educational outcomes................................................................. 307
Table A.1. REFLEX-HEGESCO sample sizes.............................................................................................................................................................. 311
Table A.2. TIMSS sample sizes: Principals............................................................................................................................................................... 312
Table A.3. TIMSS sample sizes: Teachers................................................................................................................................................................. 313
Table A.4. TIMSS sample sizes: Students................................................................................................................................................................. 314
Table A.5. PIRLS sample sizes: Principals, teachers and students..................................................................................................................... 315
Table A.6. PISA sample sizes: Principals................................................................................................................................................................. 316

Executive summary
The ability to measure innovation is essential to an improvement strategy in education. Knowing

whether, and how much, practices are changing within classrooms and educational organisations,
how teachers develop and use their pedagogical resources, and to what extent change can be linked
to improvements would provide a substantial increase in the international education knowledge base.
Measuring Innovation in Education offers new perspectives to address this need for measurement
in educational innovation through a comparison of innovation in education to innovation in other
sectors, identification of specific innovations across educational systems, and construction of
metrics to examine the relationship between educational innovation and changes in educational
outcomes.
Key findings on innovation

in the education sector
Some of the innovation indicators are derived from a survey covering tertiary graduates in 19
European countries, and casting light on several dimensions of innovation in education and other
sectors of the economy (or society). Here are some key findings:
Contrary to common belief, there is a fair level of innovation in the education sector, both
relative to other sectors of society and in absolute terms. 70% of graduates employed in the
education sector consider their establishments as highly innovative, on par with the economy
average (69%).
Within education, innovation intensity is greatest in higher education, with secondary and
primary education approximately equal.
Compared to other sectors, knowledge and method innovation is above average in education,
product and service innovation is below average, and technology innovation is at the average
sectorial level.
Education is at or below the average in terms of the speed of adoption of innovation: 38%
of graduates reported that their educational establishment was mostly at the forefront in
adopting innovations, new knowledge or methods (against 41% on average in the economy).
Higher education stands out in terms of speed of adopting innovation, above the economy
average, and well above the rate in primary and secondary education.
The education sector has significantly higher levels of innovation than the public administration
on all our indicators and is at least as innovative as the health sector on each measure.

EXECUTIVE SUMMARY
Key findings on innovation

in primary and secondary education
Another approach to measuring innovation in education is based on assessing significant

changes in key practices in educational establishments, be they pedagogic or organisational. This
approach was implemented using international surveys such as PISA, TIMSS and PIRLS. Some of the
key findings are as follows:
There have been large increases in innovative pedagogic practices across all countries covered
in areas such as relating lessons to real life, higher order skills, data and text interpretation
and personalisation of teaching.
In their pedagogic practice, teachers have innovated in their use of assessments and in the
accessibility and use of support resources for instruction.
Educational organisations have innovated in the areas of special education, creation of

professional learning communities for teachers, evaluation and analytics and relationship
building with external stakeholders, such as parents.
In
general, countries with greater levels of innovation see increases in certain educational
outcomes, including higher (and improving) 8th grade mathematics performance, more
equitable learning outcomes across ability and more satisfied teachers.
Innovativeeducational systems generally have higher expenditures than non-innovative

systems; however, their students are no more satisfied than those in less innovative systems.
Overall, innovation has been higher with regards to classroom practices than school practices
between 2000 and 2011.
Taking all practices together in an overall composite innovation index, countries in which
there has been the most innovation at the classroom and school levels in primary and
secondary education include Denmark (37 points), Indonesia (36 points), Korea (32 points)
and the Netherlands (30 points). Countries where there has been the least innovation
include the Czech Republic (15 points), Austria (16 points), New Zealand and the United
States (both 17 points). The OECD average is at 22 points. (These points can be read as an
average effect size multiplied by 100.)
Towards surveys on innovation

in education?
While this report uses existing international datasets, improved measures would entail more
specific studies. Our preferred approach to measuring innovation in education would be to develop
a dedicated international survey or at least survey instrument. This survey would ideally:
Adopt and adapt the organisational change approach using matched employer-employee-
user surveys.
Be administered to the central educational administration (ministries, relevant local

authorities) and to educational establishments in primary, secondary and tertiary education
Question three levels of stakeholders (principal/president, teachers/faculty and students)

about the state and changes in their work practices and work environment.
Infer
innovation by comparing whether the investigated practice was used (or used to the
same extent) at the time of the survey and, say, three years before.

EXECUTIVE SUMMARY
Ask respondents their opinion about the impact of these practices (or change in these practices)
on different educational goals (e.g. learning outcomes, equity, access, cost-efficiency).
Capture the sources and objectives of planned innovations, to what extent these planned
improvements are implemented and perceived on the ground, and the extent of unplanned
innovations.
Cover the broad innovation areas: products and services offered by educational organisations
to their users/clients (e.g. textbooks, study programmes); pedagogic practice (e.g. pedagogies,
introduction of new teaching or administrative equipment); organisational practice (e.g.
organisational routines, human resource practices, knowledge management practices;
support for the introduction of new ideas and practices, participation in training and retraining
courses); external relations (e.g. relationships with parents, employers, research organisations,
other academic institutions, advertisement practices).
Collect information about the broader environment in which these practices take place, such
as information about size of establishment and classrooms, number of classes, competition
with other schools in the neighbourhood, regulation and regulatory changes.

Overview
Why and how to measure

innovation in education
This Overview highlights the importance of measuring innovation in

education, presents the methodology, objectives and findings of the book,
and proposes new ways to improve measures of educational innovation in
the future.

OVERVIEW
Objectives of the book

The OECD Innovation Strategy called for new perspectives on the measurement of innovation
(OECD, 2010a, 2010b). In particular, it called for measures of innovation in the public sector,
including in the education sector. Measuring innovation in education responds to this call, offering new
perspectives on measuring innovation in education. It pursues several objectives.
The first objective is informative. The book gives readers new international comparative information
about innovation in education compared to other sectors, and documents change in a variety of
dimensions of school practices between 1999 and 2011. It is a key resource for readers interested in
educational innovation, but given the variety of practices covered, it also provides material of interest
to a wider audience. The main substantive findings of the book are summarised below.
The second objective is methodological. The book illustrates two basic but very different methods
of measuring innovation in education, along with their advantages and disadvantages. It identifies
the breadth of relevant data that can be incorporated within such measures, and makes visible the
type of information that new data collections based on the two broad measurement approaches
would yield. In particular, it shows how the publication of indicators about innovation in education
based on two different definitions would enable innovation to be viewed in more objective, factual
terms that are not conditioned solely by expert opinion.
The third objective is heuristic. Measuring innovation in education is a large scale pilot on the
measurement of innovation. It proves that such measurement is possible within the education
sector, shows what the production of such information could offer educational policy makers and
illustrates how it could place discussion of innovation in education on a firmer footing. What kind of
knowledge would we gain if we had better data on innovation? Which questions could we answer?
The fourth objective is prospective. The increasing wealth of educational data at the international
level has made this pilot possible, but the lessons drawn from this effort should lead to improved
measures and inform the development of targeted data collection on innovation in education. We
propose below new approaches to measuring innovation in education in the future.
This book is thus the beginning of a new journey and calls for innovations in the field of
measurement and not just of education.
Why measure innovation in education: context and rationales

Before presenting how innovation in education is measured in this book, let us start by recalling
why measuring innovation in education matters.
In the last few decades, innovation has been increasingly regarded as a crucial factor in
maintaining competitiveness in a globalised economy. Innovation can breathe new life into slowing
stagnant markets, and act as a mechanism to enhance any organisations ability to adapt to changing
environments (Damanpour and Gopalakrishnan, 1998; Hargadon and Sutton, 2000). Both innovation
policies and theory have mainly focused on the business sector (Lekhi, 2007). Businesses need to
innovate in order to keep up with competition by introducing new products or services, improving
the efficiency of their production processes and organisational arrangements, or enhancing the
marketing of their activities in order to guarantee their survival.
Much more recently, policy interest has extended this innovation imperative from private
organisations to the provision of public services. Although public services, including education,
tend neither to operate within competitive markets nor have the same incentives to innovate as
businesses do (Lekhi, 2007), there are important arguments to push for innovation in education
as a means to maximise the value of public investment. Several recent national innovation

OVERVIEW
strategies include provisions for more innovation in the public sector (e.g. Australia, Finland, the
Netherlands, Norway and the United Kingdom). Demographic pressures, burgeoning demand for
government services, higher public expectations and ever-tighter fiscal constraints mean that
the public sector needs innovative solutions to enhance productivity, contain costs and boost
public satisfaction.
Innovation in the public sector in general, and in education in particular, could be a major driver
for significant welfare gains. Governments provide a large number of services in OECD countries
and the share of these services as a proportion of national income is considerable. Government
expenditure in OECD countries represents above 40% of GDP on average (48% in 2011), and in
some cases corresponds to more than half of the national GDP. Education is a major component of
government services: in 2010, public expenditure on educational institutions accounted for 5.5% of
the national income on average for OECD countries. Innovations improving the effectiveness and
efficiency of such a large area of government spending could yield important benefits.
Why innovation in education matters

How could innovation add value in the case of education?
First of all, educational innovations can improve learning outcomes and the quality of education
provision. For example, changes in the educational system or in pedagogies can help customise the
educational process. New trends in personalised learning rely heavily on new school organisations
and the use of ICT.
Second, education is perceived in most countries as a means to enhance equity and equality.
Innovations could also help enhance equity in the access to and use of education, as well as equality
in learning outcomes.
Third, public organisations are often under as much pressure as businesses to improve efficiency,
minimise costs and maximise the bang for the buck. Mulgan and Albury (2003) argue that there has
been a tendency for costs in all public services to rise faster than those in the rest of the economy,
and education is no exception. While this could be attributed to Baumols cost disease, i.e. to the
nature of the public service provision (which faces ever-rising labour costs and limited scope for
transformative productivity gains), this may also be due to a lack of innovation (e.g. Foray and Raffo,
2012). Innovation, then, is regarded as a stimulus for a more efficient provision of these services.
Finally, education should remain relevant in the face of rapid changes to society and the national
economy (Barrett, 1998:288). The education sector therefore should introduce the necessary changes
that permit it to adapt to societal needs. Education systems, for example, face a need to adopt
teaching, learning or organisational practices that have been identified as beneficial to fostering
skills for innovation (Dumont et al., 2010; Schleicher, 2012; Winner et al., 2013). The results of the
Programme for International Student Assessment (PISA), Trends in International Mathematics and
Science Study (TIMSS), Progress in International Reading Literacy Study (PIRLS) and of the OECD
Survey on adult skills point to the need for innovation to improve results in literacy, numeracy or
scientific literacy in many countries.
The book proposes an exploration of the association between school innovation and different
measures related to these educational objectives.
Why a measurement agenda

Evidence-based policy requires reliable measurement. Policies supporting innovation in the
private and public sectors need relevant and reliable indicators that help monitor the innovation

OVERVIEW
process, and evaluate the success of innovation policies. Moreover, comparable international data
and benchmarking facilitate international policy learning.
A measurement agenda is essential to an innovation and improvement strategy in education.

Knowing whether and to what extent practices are changing within classrooms and educational
organisations, how teachers develop and use their pedagogical resources, and to what extent this
change can be linked to improvements would allow for a substantial increase in the international
knowledge base. We still lack too much information on actual classroom and school practices and
resource use as well as how these factors are changing over time. Developing indicators that allow
us to identify changes within classrooms and schools, and to what extent and for whom they are
an amelioration, is key to the improvement of education. Such indicators are also a key for policy
makers to understand the impacts of policy reforms in the classroom.
At the international level, the OECD PISA and Teaching and Learning International Survey
(TALIS) programmes have started to meet the need for measurement, as have TIMSS and PIRLS
(conducted by the International Association for the Evaluation of Educational Achievement [IEA]).
However, further efforts to collect precise data are still necessary. Key knowledge gaps include: the
state of teaching; the introduction of new or improved educational resources; how policy-driven
or user-driven innovations actually change learning and teaching within classrooms as well as
organisational efficiency within educational establishments. In addition, there is currently no
international data collection comparable to the above-mentioned surveys at the higher education
level or in vocational education and training establishments.
An important first step for a measurement agenda is to agree on operational definitions. This
is necessary before we can better understand the drivers of innovation in education and refine our
understanding of the effects of innovation policy levers in the education sector, an objective that
goes well beyond the scope of this book. Understanding the limitations of existing measures will
help improve the future measurement of the most important dimensions of innovation in education.
How innovation in education is measured in this publication

There has been a long-standing effort to develop innovation indicators for the private sector.
Indicators derived from research and development (R&D) statistics and innovation surveys, for
example, have gained collective support. National innovation surveys, such as the EU Community
Innovation Survey (CIS), have, for example, been harmonised internationally since 1992.
However, the measurement of innovation and its effectiveness in the public sector, and in
education in particular, is in its infancy. Despite the relative wealth of indicators in education, we still
lack data that measure the innovation performance of education systems or that link innovations in
classroom and school practices to actual improvement. This lack of available data could jeopardise our
understanding and monitoring capacity of innovations in education, and thus hinder improvement.
The nature of the indicators to measure educational innovation may be different from those
developed for the business sector. Given some of the specificities of the innovation process in
education, better understanding both the amount and quality of innovation in education, as well as
of the process leading to it, may require different forms of measurement.
Defining innovation in education

Many definitions of innovation are used in different contexts and disciplines, though, for statistical
purposes, the most widely accepted definition of innovation comes from the Oslo Manual (OECD/
Eurostat, 2005). It defines innovation as the implementation of a new or significantly improved
product (good or service) or process, a new marketing method, or a new organisational method in

OVERVIEW
business practices, workplace organisation or external relations. In this definition, implementation

is used to refer to the introduction of a product to the market, whilst it refers to the actual use of
processes, marketing methods and organisational methods. The use of the word new indicates that
innovation contains a degree of novelty at the level of the organisation, the market, or the world.
This definition has been widely applied to the private sector and can also be applied to
education with small modifications. Educational organisations (e.g. schools, universities, training
centres, education publishers) introduce (1) new products and services, e.g. new syllabi, textbooks
or educational resources (2) new processes for delivering their services, e.g. use of ICT in e-learning
services, (3) new ways of organising their activities, e.g. ICT to communicate with students and
parents, and (4) new marketing techniques, e.g. differential pricing of postgraduate courses. These
new practices are intended to improve the provision of education in one way or another, and
therefore, innovations in education should be regarded as improvements.
However, the notion of improvement in many public services, including education, can be
elusive and the use of this definition has been challenged. The perception of improvement depends
on the perspective of the stakeholders, who may wear several hats: consumer, citizen and tax-payer
(Parston, 2007). Assessing the success of companies in the private sector by profit, sales or growth is
widely accepted: whatever their objectives, ultimately they have a single bottom line which prevails
over their other objectives. By contrast, whether public organisations stay in business or close
is usually bound to a political decision (rather than a market sanction). Public organisations are
assessed on a multiplicity of objectives, such as increased quality, equity, coverage and efficiency,
which are less commensurable and can even conflict.
As a result, improvements in education can be perceived differently depending on which

objective is examined or on the point of view of the observer. Moreover, cultural values, social policies
and political goals can lead to differing prioritisation of these different objectives across countries.
Prioritisation can also change over time as the result of shifts in circumstances and citizens
expectations. This has consequences for the validity and limitations of the information gathered.
This implies that, ideally, innovation indicators in the education sector should be linked to
specific social and educational objectives (e.g. learning outcomes, cost efficiency, equity, and public
satisfaction). It should also be measured at different levels and, when they cannot be objective,
measured according to different stakeholders perspectives.
Measuring innovation in education: two broad approaches

Available innovation indicators provide an insight into the occurrence of innovation within some
sectors of the economy but leave important gaps in our knowledge about innovations happening
within the public sector. A number of recent initiatives have aimed at bridging these gaps and
exploring the development of indicators better suited to public services in general. These initiatives
could potentially be applied specifically to the education sector.
The recent initiatives can be grouped into two broad approaches to measuring innovation: (1) the
adaptation of the innovation surveys to the public sector (including education), and (2) the analysis
of organisational changes through employer-employee surveys. The two approaches explored are
presented in the section below.
Measuring innovation in education is a pioneering attempt to apply these two broad approaches
by presenting indicators based on existing international datasets. We thus provide education policy
makers with an estimated order of magnitude of innovation and change in education, but also
show the types of information that targeted data collections based on these approaches could
generate. We also propose new methods to start more systematic collections of reliable measures of
innovation in education.

OVERVIEW
The adaptation of innovation surveys to the education sector

The first approach to measure innovation in the public sector is the adaptation of existing
national innovation surveys, such as the EU Community Innovation Survey. Such surveys offer
well-established means of measuring innovation, performed over several decades for (selected)
parts of the private sector.
Part I takes this approach to measuring innovation in education and presents indicators
based on the analysis of two surveys (REFLEX and HEGESCO) in line with the methodology of the
Community Innovation Survey. The Community Innovation Survey draws on the Oslo Manual
and its definition of innovation.
REFLEX and HEGESCO survey higher education graduates in European countries and Japan
five years after their graduation. Survey questions cover respondents current employment,
including the intensity of innovation in their organisation, the type of innovation (product,
process, organisation and marketing), and the extent to which they are involved in the
innovation process. The data thus provide an assessment of innovation within different sectors,
including education. The data allow for the comparison of innovation by type and level across
different sub-sectors of education (primary, secondary, tertiary) and also with the innovation
levels observed in other sectors of the economy. Detailed information about the surveys can be
found in Appendix I.
There are a few differences between the REFLEX and HEGESCO surveys and national innovation
surveys. First, REFLEX and HEGESCO include a small number of questions about innovation, but
innovation is not their focus. Second, REFLEX and HEGESCO cover all sectors of the economy while
national innovation surveys typically cover a few sectors considered ex ante as more innovative.
Third, the respondents are drawn from different groups: REFLEX and HEGESCO employed tertiary
educated individuals, whilst national innovation surveys are answered by a representative
answering for the responding firm. As most tertiary education graduates are employees rather than
employers, REFLEX-HEGESCO are essentially employee surveys, while national innovation surveys
are employer surveys.
Whilst at first glance the innovation categories reported in REFLEX and HEGESCO seem different
from those of the Community Innovation Survey (and from the Oslo Manual), there is in fact a direct
correspondence between the two. The definition used in the REFLEX and HEGESCO surveys is based
on the second (rather than the third) edition of the Oslo Manual (OECD, 1997). The second version of
the manual distinguished between two types of technological innovation (product or service, and
process), and also considered a third type called non-technological innovation, covering innovation
in organisation and method. This definition can easily be reconciled with the definitions of the 3rd
edition of the Oslo Manual, whose four types of innovation have been presented above. Product (or
service) remains the same; process innovation refers to technology or tools; and organisation and
marketing innovations are grouped as innovations of knowledge or method.
Contrary to innovation surveys, REFLEX and HEGESCO do not include questions about the
drivers of and obstacles to innovation. The innovation indicators reported in Part I therefore focus
exclusively on innovation levels at a point in time.
The adaptation of organisational change surveys to the education sector

The second approach that has been used to improve the measurement of innovation in the
public (and business) sector is based on surveys of organisational change. These surveys have
been developed more recently than innovation surveys and have been implemented in national
or European surveys (e.g. the Organisational Change and Computerisation [COI] survey in France
or the Measuring the Dynamics of Organisation and Work [MEADOW] project in Europe). These

OVERVIEW
surveys capture innovation with a mix of subject-based and object-based approaches. They
typically measure the dissemination of specific innovations in the economy, for example computers
or organisational practices. In terms of method, they implement matched employer-employee
surveys, asking workers and employers about their current working conditions or tools compared
to those in the past. The difference between the current and past situations makes it possible to
determine whether there has been innovation across different dimensions of interest. However, as
of 2014, to our knowledge no international database using this approach covers the education sector.
Part II applies the working definition of innovation as the implementation of a new or significantly
changed process, practice, organisational or marketing method observed at the education system
level through micro-data collected within schools. The emphasis is particularly placed on change in
practices. Given that we cannot directly observe whether these processes, practices and methods are
improved, we have to depart from the Oslo Manual definition and use change as a proxy measure.
It can be assumed that change occurs because of a belief that the new version is an improvement
of some educational goal. The book thus presents a range of indicators based on an approximation
of the traditional innovation definition. It captures innovation as a significant change in some key
practices in educational establishments by drawing on the PISA, TIMSS and PIRLS databases. Further
information about these data sources can be found in Appendix I of the book. Whilst these studies
are designed to measure student outcomes, they also collect information about educational and
teaching practices at a point in time. The repeated cross-sectional nature of the studies makes it
possible to map trends over time. Our indicators are therefore based on the analysis of responses
to questions that have been asked in at least two waves of the study in order to identify changes in
professional practices or in classroom or school resources.
In contrast to innovation surveys, the proposed indicators are object-based; they do not only
give a level of innovation by country, but also identify what these innovations or practices are.
The indicators report information about which practices have changed and which have remained
constant. They also report on the direction of the observed changes: one can innovate by doing
significantly less of something, or significantly more. (This is also true in the business sector: a
device can be significantly improved by reducing its inputs (such as power use) or its outputs (such as
gas emission) whilst maintaining the same functionality.) Moreover, in this section we focus on the
school sector, which does typically not introduce products to a market. Gault (2012) has suggested
that this potential issue in the definition can be overcome by introducing the explanation that a
product is implemented when it is made available to potential users. In the case of education, this
would typically be students.
How much change counts as a significant change? One difficult question with this methodology
is to determine how much a variable needs to change before we consider that we are observing an
innovation, that is, what constitutes a significant (or noteworthy) change. There is no definitive
answer to this question, which requires, in any case, some subjective judgment. For example,
the degree to which the adoption of a teaching practice by 10% more teachers can be considered
innovative depends on the context: it may be considered a more significant change in a country in
which 10% of teachers used the practice than in a country in which 70% of teachers already used it.
In this publication, summary tables providing effect sizes assist the reader in making this judgment.
Effect sizes give a standardised measure of these changes and help interpret the relative magnitude
of the change: the greater the effect size, the higher the magnitude (and likely significance) of
change over time.
The practices presented have been selected and grouped in two categories: classroom and
school changes (or innovations).
Classroom practices (i.e. teaching and learning) are often the most difficult practices of education
to change, and the literature on research and innovation shows that classroom practices are often

OVERVIEW
left unaffected by educational reform. An important reason to measure change in classroom

practices is precisely to see whether the intended effects of reform, professional development
efforts, advocacy or educational research do materialise in changed practices. We look at three
dimensions: instructional practices (Chapters 5, 6, 7, 9), use of educational resources (Chapters 8, 11)
and availability of resources for teaching (Chapter 10). In the frame of the Oslo Manual, noteworthy
changes in any of these dimensions would correspond to a process innovation.
School practices are more easily influenced by education policy and decision makers, and
perhaps also easier to measure. They encompass practices that can affect students directly, for
example through the provision of special programmes (Chapter 12), or indirectly, through new
organisational and human resource management practices (Chapters 13, 14, 15) and new ways of
engaging or relating to parents (Chapter 16). Here again, there is a straightforward correspondence
with the types of innovation of the Oslo Manual as changes to school practices cover innovations in
organisation and marketing (such as external relations with parents), even though these categories
could be broader. In sum, our implementation of the the organisational change approach can easily
be related to the innovation framework of the Oslo Manual.
The variety of indicators that can be included in the organisational change approach gives
us much more information and detail about ongoing innovations than the innovation survey
approach. It also allows us to accommodate an expert view of educational innovation. Because
of the lack of innovation measures, many observers and experts have their own opinion about
education systems innovation intensity and about what is innovative or not. Some people would
consider that an increase in the use of ICT is a sign of innovation. Others would consider some
pedagogical practices as innovative for example active pedagogies such as problem-based
learning or student-centred pedagogies because these pedagogies are less common than others,
an impression that they form from school visits or knowledge of the field. The indicators compiled
in this book also provide information on whether practices subjectively perceived as innovative (in
a different meaning from the Oslo manual) are becoming more common or not and also the extent
to which they are mainstream.
One downside of applying the organisational change method through the use of international
databases not purposely designed to measure innovation is that the data do not give us a synthetic
measure of innovation. While innovation is often made up of many small changes, it is important to
get a synthetic idea of the magnitude of overall innovation and of different dimensions of innovation.
To that purpose, in each chapter of this publication we provide a visual representation of

aggregated change across the different dimensions covered in the chapter. While this aggregate
indicator cannot be compared across chapters (given the diversity of the format of the questions), it
helps the reader to spot where there has been more change in the broad category that we examine
across countries.
We also compute a composite innovation index and several other sub-indices (Chapter 17),
which synthesise the information captured throughout the organisational change part of the
publication. We present aggregate estimates of innovation in schools and classrooms and more
focused detail of its location in terms of grade and subject. The construction of the indices and the
reasons to interpret them with caution are explained in Appendix II.
Better understanding the role of innovation in education

Innovation may or may not achieve its stated goals of improvement. The same is true of policy
reform. One possible reason is that the changes necessary (or expected) to reach the goal do not
occur. For this reason, monitoring innovation and change is important, irrespective of whether or
not we can prove the effect of innovation (or noteworthy changes in practices and resources) on

OVERVIEW
valued outcomes in the education sector. Such monitoring allows education policy makers and other
education stakeholders to know whether their decisions and actions have led to the anticipated
level of innovation. This would help better understand the issue at stake, as captured by the title of a
recent book: So much reform, so little change (Payne, 2010). In turn, regardless of reform, understanding
what is changing in the educational landscape is a key to informed decision-making.
One could also hope that measuring innovation will help to better understand the effects of
specific types of innovation on educational outcomes. Part III of this publication explores how
innovation in primary and secondary schools is associated with a variety of educational outcomes
related to student performance, equality and equity, expenditure per student, and teacher and
student satisfaction. The analysis is descriptive in nature, and cannot explain the direction of any
observed relationship or be used to infer causality, but it kick-starts a discussion on what more is
needed in order to measure the impact of innovation and to cast light on the contexts that make
innovation more likely to succeed.
Summary of the main findings

Different pictures of innovation in education can be drawn from the indicators presented in this
book, depending on readers interests. In this section we propose one possible picture with a high
level summary of some key findings.
Innovation intensity in the education sector

Contrary to what is often believed, there is a fair level of innovation in the education sector, both
relative to other sectors and in absolute terms. Innovation intensity is greater in higher education,
with secondary and primary education approximately equal. Compared to other sectors, knowledge
and method innovation is above average in education; product and service innovation is below
average, while technology innovation is at the average sectorial level.
A first dimension of innovation concerns the prevalence of highly innovative organisations, as

reported by the tertiary education graduates they employ.
70% of graduates employed in the education sector consider their establishments as highly
innovative, on par with the economy average (69%).
Educational establishments are however slightly less likely than average to combine high levels
of innovation across all types of innovation (product, technology, and knowledge/method).
In higher education, 80% of professionals in Europe were employed in highly innovative

organisations in relation to at least one type of innovation. That is similar to the manufacturing
sector (79%) and the highest share of all sectors of the economy.
A second dimension is whether organisations adopt innovations, new knowledge or new

methods quickly. Rapid adoption is a sign of being a lead innovator.
Education is at or below the average in terms of the speed of adoption of innovation: 38%
of graduates reported that their educational establishment was mostly at the forefront in
adopting innovations, new knowledge or methods (against 41% on average in the economy).
Higher education stands out in terms of speed of adopting innovation, above the economy
average, and well above the rate in primary and secondary education. Overall, 46% of higher
education professionals reported that their educational establishment was mostly at the
forefront in adopting innovations and new knowledge, compared to 31% in primary education
and 30% in secondary education.

OVERVIEW
A third dimension of innovation lies in the prevalence of highly innovative jobs in the sector, a
criterion that is more selective than the prevalence of highly innovative organisations. We define
highly innovative jobs as those in highly innovative organisations regarding at least one type of
innovation and where the employee plays a role in introducing those kinds of innovations.
58% of tertiary graduates indicated that they had highly innovative jobs (against 55% on
average in the economy).
Highly innovative jobs in education were more likely to concern knowledge and methods
innovation (47%); this was the most common type of innovation in the education sector in all
countries covered.
Only the manufacturing sector (64%) had significantly more highly innovative jobs than the
education sector across the countries analysed.
68% of graduates working in the higher education sector in Europe had highly innovative
jobs in 2005 or 2008. This figure was 53% for secondary and 55% for primary education, while
the likelihood of graduates having highly innovative jobs was nearly twice as high for those
working in higher education as in secondary or primary education.
There are big variations across countries in the reported levels of innovation. Interestingly, the
education sector has significantly higher levels of innovation than the public administration on all our
indicators and is at least as innovative as the health sector on each measure. On average, education thus
seems to have higher levels of innovation than the other public sectors for which we have information.
Pedagogic and organisational innovation in primary and secondary education

Our analysis of the magnitude of change at the classroom and school levels in primary and
secondary education gives us an idea of what some of the innovations measured above may look
like at these levels.
Many observers of educational innovation in primary and secondary education are interested
in pedagogic innovation because teaching (rather than teachers) is the closest factor to the learner
allowing learning improvement in formal settings. It is often believed that pedagogic practices in
the classroom change little, and that teachers are reluctant to adopt active pedagogies that give
a more central role to students in directing their own learning or that motivate them more for
learning (Dumont et al., 2010). Our indicators confirm this overall impression to some extent but
also highlight important nuances.
Very little change has occurred in terms of prevalence of lecture-style presentations at 8th
grade between 2003 and 2007. While this is a more contentious pedagogic practice, the use of
instructional time for student independent work (without guidance) has also stayed mostly
constant. On average, teaching practices that put teachers at the centre of instruction have
thus not diminished. Interestingly, teachers and students disagree on the prevalence of these
practices (students perceive them as much more prevalent than teachers do).
In contrast, between 2001 and 2011, there has been a large increase in other pedagogic practices
generally considered as innovative, with increasing emphasis on: relating lessons to real life;
higher order skills (reasoning and leaving discretion to students in solving problems, designing
experiments or choosing their reading materials); data and text interpretation; and personalisation
of teaching (responsiveness to individual students needs, individualised instruction). This is true
across grades and subjects. Students group work has also increased on average.
In a couple of cases, average innovation levels hide a mixed picture across countries or
disciplines. Whilst individualised instruction increased on average by 11 percentage points,

OVERVIEW
it has sharply decreased or increased in a few countries. Whilst students group work has
increased in maths and reading, it has slightly decreased in science.
Student assessment has become a central focus of education policy in many countries (OECD,
2013a). It has also been a major area of pedagogic innovation in primary and secondary education,
with considerable change in assessment methods and hopes that more testing will lead to better
learning outcomes. Indeed, there seems to have been much more testing of students in the 2000s:
the use of all types of tests has either increased or remained stable on average. Even though trends
in some countries may give the impression of an increased use of standardised tests by teachers,
this does not seem to be an international trend:
Overall, the use of tests developed by teachers has been the subject of innovation, with an
absolute change of 8% points on average. Across countries, though, changes have occurred in
both directions: tests developed by teachers were more frequently used in 2009 than in 2003
in six countries, while the opposite was true for four countries.
There has been an absolute change of 3% points on average in the frequent use of standardised
tests in schools, with an average change of only 1% point when all directions of change are
taken into account. This practice became less frequent in three countries, and more frequent in
eight countries, including countries where the increase has been very large (the Netherlands,
Indonesia, Poland and the Russian Federation).
Very large changes in the use of student assessments not based on classroom tests can be
observed in some countries between 2003 and 2009, with increases in Poland (by 67% points),
Finland (62% points) and Denmark (27% points), and decrease in the United States (25% points).
Between 2003 and 2009, there was a noteable increase in the use of student portfolios as
assessments in schools in Indonesia (by 40% points), Denmark (36% points) and Greece (29%
points) and a decrease in this practice in Norway (29% points).
Pedagogic innovation is often related to the accessibility and use of support resources for
instruction, such as textbooks and computers. Knowing to what extent teachers use different
pedagogic resources is critical to find the best ways to support their teaching and empower them
(Krkkinen, 2012; Avvisati et al., 2013; OECD, 2012). Among these resources, many observers attach
particular importance and interest in the use of information and communication technology (ICT)
sometimes even merely assimilating innovation to the use of ICT in school.
Teachers have significantly increased their use of textbooks as a basis of instruction for
science and maths classes between 2003 and 2011. Consistent with this, as textbooks became
a primary resource for instruction, there was a reduction in the extent to which textbooks were
used as supplementary resources. Only for reading instruction has the increase in the use of
textbooks been modest: the three countries exhibiting an increase in textbook use equalised
those countries with diminishing use.
Whilst availability of computers has increased on average in 4th grade classrooms between
2003 and 2011, it has slightly decreased in 8th grade classrooms (and computers are not
available for use by all 4th or 8th grade maths and science students). There have been increases
in some countries and decreases in other countries. In New Zealand, for example, computer
availability increased in maths at 4th grade but decreased in maths at 8th grade level between
2003 and 2011. There may be several explanations for the unexpected direction of these
changes, including the fact that computers have become only one ICT device among others.
As mobile learning becomes more prevalent, tablets, e-readers, mobile phones, clickers and
other portable devices may displace computers from classrooms, at least at the upper levels
of schooling in OECD countries.

OVERVIEW
Inevitably, there is overlap between the percentage of classrooms with a computer and those
with access to the Internet. Similarly, there is overlap with classrooms using computers to
practice skills and procedures, search information, analyse data, undertake experimentation,
or for reading or writing. Large changes in both directions were observed, and were typically
more pronounced and positive at 4th grade than at 8th grade. The Russian Federation is the
only country exhibiting consistent increases in computer use across pedagogical purposes,
disciplines and levels between 2003 and 2011.
The analyses also capture another type of innovation in primary and secondary education:
organisational innovation at the school level. Organisational changes typically aim at improving
learning in the classroom by changing the supporting environment offered by the school to students
or staff. These changes can correspond to a new pedagogic offer, to new professional practices of
teachers, such as collaboration or teacher evaluation.
In terms of educational offer proposed by schools, innovation in special education has mostly
resulted in more remedial and enrichment education at 8th grade level between 1999 and
2007. However, there are important exceptions: for example, large decreases in enrichment
education for 8th grade science took place in the Russian Federation (by 50% points), Israel (30%
points) and Hong Kong (25% points); and for mathematics, notable decreases took place in the
Russian Federation (43% points) and Ontario (26% points).
The constitution of professional learning communities entailing more collaboration and

peer learning among teachers is often considered to be a critical driver of innovation and
improvement in teaching practices (OECD, 2013b; Vieluf et al., 2012). Two key related practices are
teacher collaboration and more intensive patterns of class observation, for both of which there
is evidence of innovation. Peer discussions about instruction have increased on average, and
regular classroom observation has increased as well, but to a much lesser extent. Collaboration
in preparing instructional materials has decreased, perhaps as a result of the more intensive
use of textbooks as a basis for instruction. Interestingly, innovation related to these professional
learning communities has not been uniform across countries and has moved in both directions
(more or less of these practices) between 2003 and 2011. There has been more change in this
area at 8th grade than 4th grade across OECD countries in absolute terms.
Thanks to new avenues opened by data analytics, educational data are being used for comparing
and giving feedback to schools or teachers, and for accountability purposes. While student
assessment data have clearly increasingly been used for district or national benchmarking
in secondary education and for monitoring yearly school progress, there has been a strong
reduction in their use for comparing schools. The administrative tracking of achievement
data has remained constant on average, but has sharply increased in some countries (Korea,
Denmark, Norway, Poland, Israel) and decreased in others (the Netherlands, Germany).
Teacher evaluation has also shown signs of innovation, with the use of assessment data,
external evaluation and peer evaluation typically increasing across grades and subjects.
Direction of innovation in recruiting and retaining practices differed across education systems
with regard to incentives use, with little overall change concerning OECD average level.
Finally, organisational innovation can be focused on new relationships with external

stakeholders such as parents, employers, funders or the public at large. This type of innovation
could also be characterised as marketing innovation. The analyses notably capture innovation in
relationships with parents, which are highly important given the importance of family involvement
for student achievement.
Assessment data are extensively used as tools to inform parents, though there has been little
change between 2000 and 2009. Similarly, parental perception regarding school provision of

OVERVIEW
regular information on students performance did not change significantly between 2006 and
2009. The United States (21% points) registered the largest increase in the use of assessment
data to inform parents about their childs performance.
More schools asked for parental involvement in schools between 2003 and 2007. Principal
and school engagement in public relations and fundraising activities slightly decreased across
OECD countries in the timeframe between 2001 and 2009.
Overall, innovation has been higher with regards to classroom practices than school practices
between 2000 and 2011. Taking all practices together in an overall composite innovation index, 13
out of 28 education systems are seen to be above the OECD mean (22 points) in terms of the extent
of change across school and classroom practices. Countries in which there has been the most
innovation at the classroom and school levels in primary and secondary education include Denmark
(37 points), Indonesia (36 points), Korea (32 points) and the Netherlands (30 points). Countries where
there has been the least innovation include the Czech Republic (15 points), Austria (16 points), New
Zealand and the United States (both 17 points). The points of the indices are based on (and can be
interpreted as) effect sizes (multiplied by 100).
While some practices have changed (generally increased) in the same direction across countries
(e.g. relating lessons to real life and reasoning, using text books and involving parents), there are
still many practices that exhibit no clear international direction of change. Innovation concerning
these practices corresponded to large moves in both directions across countries (as with methods
of assessment) or by subject/grade (as with network accessibility in New Zealand, which increased
in 4th grade maths but decreased in 8th grade maths between 2003 and 2011). Some practices have
also changed significantly in a few countries (such as the increased extent to which parents in
the United States have been informed about their childs performance), but not seen widespread
variation. While there may be an international consensus about the adoption of some practices, this
is clearly not the case for all.
Composite innovation indices, and the association between these and outcomes
The overall extent of innovation in primary and secondary education in a country is correlated
(usually positively) with certain educational outcomes. In particular, more innovative countries have
higher and improving maths performance at 8th grade, more equitable learning outcomes across
ability, and more satisfied teachers. However, they have typically increased their expenditure more,
and their students are no more satisfied than those in less innovative countries.
Whilst there is considerable variation, education systems that have experienced more change
in school and classroom practice, as indicated by a higher position on the overall composite
innovation index, typically have higher maths scores than those with a lower position.
Analysis of the association between innovation and improvement in maths performance

between 2003 and 2011 indicates a positive association between overall innovation and trends
in 8th grade mathematics outcomes. There is no association between school-level innovation
and trends in maths scores.
There is a positive association between innovation and equality in learning outcomes:

education systems that have innovated the most are also the most equitable in terms of
students learning outcomes. However, the correlations between innovation and equity of
learning opportunities are weak.
Most of the education systems included in the analysis have increased their level of educational
expenditure per student between 2000 and 2010 by similar amounts, regardless of their
level of overall innovation. Among the education systems registering the largest increases

OVERVIEW
in educational expenditure per student, Korea and the Czech Republic exhibit two opposite
behaviours regarding educational innovation, where the former is above average in terms of
innovation whilst the latter is near the bottom of the innovation index.
Analysis of the relationship between overall innovation and changes in 8th grade maths teacher
satisfaction shows a strong positive correlation, indicating that teachers in more innovative
education systems have become more satisfied over time whilst satisfaction has fallen in less
innovative places. The finding is even stronger when looking at classroom innovation.
In contrast with the previous finding, classroom level innovation appears to have no association
with student satisfaction, although this is in part driven by the slight reduction in satisfaction
amongst students in Indonesia. Israel and Korea stand out as countries that have innovated at
the classroom level and seen an improvement in student satisfaction.
At this stage, no conclusion can be drawn about the direction of any causality underlying these
associations. Further work, both qualitative and quantitative, should investigate the complex
interplay between innovation in general, specific innovations and various educational outcomes.
More systematic and targeted data collections about innovation in education should be a key aspect
of this policy research agenda.
How to improve the measurement of innovation in education

Measuring Innovation in Education allows us to explore the potential of two different methods to
measure innovation in the education sector: an innovation survey approach and an organisational
change approach. Both approaches yield valuable information. Their implementation has enabled
better understanding of the limits of each method. In this final section, we translate what we have
learned during this exercise into a proposal for developing new, improved measures of innovation
in education. We then discuss how it could be complemented by other measurement initiatives.
Designing a new survey on innovation in education

Our preferred approach to measuring innovation in education would be to develop a dedicated
international survey or at least survey instrument. Adopting and adapting the organisational
change approach using matched employer-employee-user surveys appears as the most
fruitful way forward for the education community. Such a survey could be administered to the
central educational administration (ministries, relevant local authorities) and to educational
establishments in primary, secondary and tertiary education, targeting three levels of stakeholders:
principal/president, teachers/faculty and students. The survey would question the three different
stakeholders about the state and changes in their work practices and work environment.
Innovation would be inferred by comparing whether the investigated practice was used (or used
to the same extent) at the time of the survey and, say, three years before. The survey would also
ask respondents their opinion about the impact of these practices (or change in these practices)
on different educational goals (e.g. learning outcomes, equity, access, cost-efficiency). Like in
traditional innovation surveys, questions to employers could allow one to capture the sources and
objectives of planned innovations, while questions to other stakeholders would enable capturing
to what extent these planned improvements are implemented and perceived on the ground, and
the extent of unplanned innovations.
As for practices, the survey would cover the broad innovation areas highlighted by the Oslo
Manual: products and services offered by educational organisations to their users/clients (e.g.
textbooks, study programmes); pedagogic practice (e.g. pedagogies, introduction of new teaching
or administrative equipment); organisational practice (e.g. organisational routines, human resource
practices, knowledge management practices; support for the introduction of new ideas and

OVERVIEW
practices, participation in training and retraining courses); external relations (e.g. relationships with
parents, employers, research organisations, other academic institutions, advertisement practices).
Information about the broader environment in which these practices take place should also be
collected, such as information about size of establishment and classrooms, number of classes,
competition with other schools in the neighbourhood, regulation and regulatory changes.
Advantages of a new survey

A new matched employer-employee organisational change survey has at least three principal
advantages for education stakeholders.
First, it allows identifying the main areas of innovation (and even specific innovations) in the
education sector and not only measuring the intensity of overall innovation. This is one advantage of
object-based surveys, which collect more specific information because they tend to focus on one type
of social activity. For example, the second part of the report is object-based in that it tells us what the
innovations are at the system level: a different way to involve parents, the increased use of a pedagogic
practice. Their disadvantage is that comparison with other sectors becomes difficult, if not impossible.
Surveys covering several sectors generally need to be subject-based and capture different types of
innovation in a generic way so that they can be relevant for different types of activities. Or they need to
focus on a specific type of innovation that cuts across different activities, for example ICT innovation.
Second, this approach avoids the ambiguities of the innovation concept, a term that, even
when defined, can be understood in different ways by respondents across different countries. Here,
innovation is mainly inferred from factual questions about work or learning practices.
Third, asking the views of several stakeholders gives a rich picture about current practices,
about innovations and about their perceived impact. Innovation surveys typically report
innovations from the viewpoint of the employers (firms), which gives a high-level view covering the
entire establishment. This viewpoint is important in order to understand the intended practices.
Surveying employees and users provides additional understanding of whether these innovations
are implemented, or perceived as innovative by employees (regarding organisational innovation
for example) or by end users or consumers (regarding new products, services or processes). There
are also innovations that are not intended by employers or (or not perceived as innovation), and
employees and users are much better placed to report on those. A matched survey would thus
provide us with a multiplicity of perspectives on different types of innovations.
Challenges of a new survey

A matched organisational survey approach to measuring educational innovation presents
(at least) three big intellectual and practical challenges: identifying the major relevant practices
and innovations that matter the most in the education sector, which can prove difficult at the
international level; getting accurate reporting of pedagogical practices and resource use; and linking
the changes to observed and perceived outcomes.
Identifying the major relevant educational practices and innovations internationally can be
challenging. As documented in our report, innovations in pedagogic and school practices are neither
similar nor synchronous in all countries: from one country to the other, innovations may concern
different types of practices over a specific period of time. As a survey can only cover a limited number
of practices, the selection of the most relevant practices can prove difficult internationally. Even
within a single country, innovation can in principle concern any product, process or practice. The
lack of representativeness of the surveyed practices can thus be worrisome. However, this does not
imply that we cannot agree on core practices that deserve to be monitored in all countries and on
some other practices that are important or interesting to cover from an innovation perspective, even

OVERVIEW
though their prevalence differs a lot from one country to the other. The second kind of practices
could be emerging practices or practices considered to have a high transformational potential for
education.
Within the broad areas of innovations and practices mentioned above, we could select major
practices of interest by building on three different types of inputs: teachers/faculty and employers
could be asked in a pre-survey about the practices they see as the most important to cover in such
a survey; educational innovation and improvement experts could be asked the same, using for
example the Delphi method; and a literature review of practices that are known (or believed) to have
an important impact on key educational goals could be a third input. The challenge and objective
of the final questionnaire would be to identify a small but relevant number of core educational
practices, but also emerging or promising practices of interest that would be balanced and
interesting to both stakeholders and countries. Should the survey be repeated, modules focusing
on specific dimensions of interest (for example the use of specific organisational routines or of
tools such as learning management systems) could allow a more in-depth analysis of the uptake of
specific practices.
The second main difficulty lies in the appropriate reporting of pedagogic practices and of their
occurrence given that teaching and learning involves a mix of so many different types of activities. If
we decided that we would like to know whether teachers use more of, say, metacognitive instruction
or problem-based pedagogies, how would we collect accurate information? Ideally, this would be
done through observations rather than self-reported responses by stakeholders, as these responses
are subject to discretionary interpretation. In that respect, large-scale video studies could be seen
as an ideal option: they would allow us to capture teachers teaching practices in an objective
manner. The Measures of Effective Teaching (MET) project, a USD 52 million study sponsored by
the Bill & Melinda Gates Foundation, has partnered and tested different tools with 3 000 US teacher
volunteers, including tools to measure the effectiveness of instruction as observed by students
and by peers (see http://www.metproject.org). These tools or similar ones could possibly be used to
cover some aspects of the classroom environment and instruction. Most of them tend to focus on
an assessment of teaching effectiveness rather than a description of pedagogic practice, and their
model may also be prone to cultural variability. For example, being caring or managing student
behaviour may well be interpreted differently in, say, the United States and in Japan. One could
however develop different coding tools to describe the pedagogic practice as well as to assess the
quality of their implementation.
Even though it is routinely done in many surveys, another underestimated challenge is to collect
accurately the frequency or amount of time teachers or pupils spend on specific practices. One way
to capture different types of pedagogic practices more objectively than a traditional questionnaire
would be to use the experience sampling method developed by Larson and Csikszentmihalyi (1983),
i.e., asking different stakeholders to describe in real time what activity they are doing (e.g. via
mobile phone or other mobile device). Csikszentmihalyi and Schneider (2001) use this method in an
educational setting to assess how much time is devoted to different learning or teaching activities.
Although it is a declarative method, in principle it gives a more accurate picture of time spent on
different activities than questions about retrospective use of time would typically do.
The third challenge in our method is to link change (innovation) and outcomes in a satisfactory
way. It is the impact of innovation (or of the innovation process) that matters. Moreover, innovation
does not necessarily have the positive impact it seeks. Ideally, we would thus like impacts to be
measured objectively at the establishment level. This is difficult in practice, but the rise of longitudinal
data systems that follow students from kindergarten to their transition to the labour market will
make this possible in the near future (OECD, 2010c). Our questionnaires matched organisational
change survey instruments should ideally be used within countries or jurisdictions as part of these
educational longitudinal information systems that track student, teacher and school characteristics

OVERVIEW
and outcomes over time. This link would allow us to identify whether observed changes led to
improvements in specific outcomes, all other (observed) things being equal. At the aggregate level,
establishing such a causal link is very difficult, even though a dedicated survey could easily improve
our understanding of the associations between outcomes and observed changes compared to what
could be done in this report. Asking different surveyed stakeholders about their perception of impact
is for example useful. First, because their perception may be accurate. Second, and perhaps more
importantly, because their subjective answers could also help understand why some changes are
embraced or resisted, regardless of their objective effects.
Measuring innovation in education following the organisational change methodology would

thus give the educational community a picture of current pedagogical and organisational practices,
of how they have changed or are changing, and would lead to identify some important innovations.
The accuracy of these measures would of course increase over time as they are repeated. While
computing a composite innovation index could be done again, a more straightforward way to
capture the overall extent of innovation would be to include a few questions using the subject-
based approach of innovation surveys, which ask directly about innovation according to some
defined categories (product, process, organisation, marketing; new to the world, to the sector, to
the company/organisation). The two approaches could indeed be mixed and, as the present report
demonstrates, provide complementary types of innovations. A few questions could, for example,
help capture the main sources of innovation in education, as well as identify some important
barriers. In our multilevel approach, the employer questionnaire could investigate many of the
dimensions that commonly form part of innovation surveys (for example the intended impact of
innovations).
The development of a specific survey or survey instrument on innovation in education along

the lines proposed above is feasible. It has development and implementation cost, like other surveys
that are routinely undertaken in many different areas. One objective of Measuring Innovation in
Education is to show the value such an innovation survey would have for education decision makers.
It could naturally be less ambitious than what is proposed above.
Taking advantage of new efforts to measure public sector innovation

The proposed approach is not exclusive from other measurement approaches. In fact, measuring
innovation through several approaches would enrich our stock of knowledge, and also speed up the
development of our understanding of innovation in the education sector.
There is rising interest in the measurement of innovation in the public sector, and taking
advantage of ongoing efforts should be another means to improve the knowledge base about
educational innovation. A number of recent initiatives have explored the development of innovation
indicators suited to public services. Innovation in education could be measured as part of these
surveys. These approaches follow either the subject-based innovation survey or the object-based
organisational change approach. Any new effort to develop new measures of innovation in the
education sector should partly build on the lessons and instruments designed by these initiatives.
One big advantage of these initiatives is that they allow one to compare innovation in education and
innovation in other sectors (or public sectors).
The adaptation of innovation surveys to the education sector

As of 2014, the OECD Working Party of National Experts on Science and Technology Indicators
(NESTI) is conducting work to contribute to the development of a measurement framework for public
sector innovation1. The project examines the feasibility of adapting the framework commonly used
for measuring business innovation (the Oslo Manual) to a public sector setting and has started

OVERVIEW
developing a questionnaire inspired by traditional innovation surveys. The envisaged survey

is largely subject-based but covers not only the types of innovation (goods and service, process,
organisational, communication) but also their perceived effects, drivers and strategies, investment
in terms of training and funding. One section of the survey could ask respondents to describe the
innovation they perceive to be the most important in the last two years, thus bringing some additional
object-based information about what some of these innovations look like. One possible option for
the administration of the survey would be to target similar units within specific sub-sectors, such
as health or education (e.g. schools or universities). Should such a survey be implemented, covering
education would be very valuable.
This new effort complements and builds on other attempts to measure innovation in the
public sector.
In Australia, the 2011 Australian State of Service Agency and Employee Surveys tested a
module on innovation covering all agencies (employers) and a sample of employees for all agencies
with more than 100 employees (thus using the employer-employee method). The agency questions
asked about the strategies in place for promoting innovation, the extent of innovation, and their
assessment of innovation capability. Employees were also asked about the extent of innovation,
about the most significant one, about innovation barriers and also whether the work climate was
conducive to innovation (Arundel et al., 2012; Arundel and Huber, 2013). A model questionnaire was
developed and may be fully implemented in 2014.
The project Measuring Public Innovation in the Nordic Countries (MEPIN) also proposed a
methodology and a first pilot for measuring innovation in public services in Denmark, Sweden,
Norway, Finland and Iceland. Bloch (2011) presents the framework, methodology and some results
of the pilot studies. In the five countries, the pilot study covered public sector institutions at both
central and local/regional levels, including ministries and government agencies (central) as well
as municipalities, schools and hospitals (local/regional). At the local level, only Denmark covered
the education sector (upper secondary schools). Some of the innovations mentioned in the report
come from the education sector though, with an example of interdisciplinary cooperation between
administration, nurseries and settling in schools for organisational innovations, and the mention
of the international marketing of education for communication innovation.
The EU Innobarometer 2010 (EC, 2010) provided interesting information on innovation in

public administration, including education, as well as a questionnaire adapting traditional
innovation surveys to the public sector. A 2012 Eurobarometer investigated the perception of public
sector innovation by those working in the private sector, while a 2014 Eurobarometer examined
the supporting role of the public sector in the commercialisation of innovation (EC, 2012, 2014).
Information gathered through the 2010 and 2012 barometers, along with other data sources, feeds
a pilot European public sector innovation scoreboard (EC, 2013), which highlights a number of ways
through which public sector innovation contributes to a countrys overall performance. While
the pilot study does not tell apart the variety of activities of the public sector, further efforts may
differentiate different types of public sector innovation and allow comparing education with other
public sector activities.
The Innovation Index developed by the UK National Endowment on Science Technology and
the Arts (NESTA) is another example of such efforts, covering the health sector. This could inspire
further work in education.
The adaptation of organisational change surveys to the education sector

Given that innovation is defined as positive change (significant improvement or useful novelty),
measuring change at the organisation level captures innovation to the extent that the change can be

OVERVIEW
linked to actual positive outcomes. These positive outcomes may be either perceived by stakeholders
or directly measurable. Linking organisational change with outcomes can also potentially help
identify novelties with no or negative impacts. Organisational innovation and change have thus
gained more attention in the past years as a way to capture innovation and some new measurement
efforts have started (Greenan and Lorenz, 2013).
Funded by the European Commission and coordinated by the Centre dEtude de lEmploi (CEE)
and the University of Nice, the MEADOW project (Measuring the Dynamics of Organisation and Work)
has built on various European efforts to set out guidelines for collecting and interpreting harmonised
data at the European level on organisational changes as well as their social and economic impacts
(www.meadow-project.eu). This project has developed survey instruments to inquire about the state
and change of the organisation and the work experience of both the employer and the employee.
In particular, the survey asks about the drivers, the strategy or policy of the organisation, the use
of management practices and ICT, the structure of the organisation, the employer and employee
outcome, the work organisation and the working conditions (MEADOW Consortium, 2010). As of
2014, pilot surveys have been carried out in Sweden (Statistics Sweden, 2011) as well as in Finland.
As they are meant to cover both the business and the public sectors, the implementation of surveys
inspired by these guidelines in the education sector would provide education policy makers with
very valuable information.
In France, the 2006 survey on Changements Organisationnels et Informatisation (Organisational

Changes and IT diffusion), a linked multi-level employer-employee survey was first designed for
the business sector and then adapted to the health sector. The survey was administered at the
levels of the central administration and of hospitals. It assessed the changes within health central
administration and hospitals as well as their outcomes, focusing on the diffusion of management
techniques and practices, the use of IT and other organisational changes and features. It also
tried to capture drivers for changes such as new demand, availability of new tools, changes in
regulatory control, public service modernisation agenda, etc. The survey identified the current
organisational situation as well as changes through retrospective questions. It used a panel survey
structure. Bigi et al. (2013) give an example of how it can be used to compare changes in public
and private hospitals. Some useful information could also be derived from the coverage of the
education sector.
Using existing national or international school-teacher and other educational surveys

A final avenue would be to rely on existing educational surveys, as we have done in this book.
A wealth of measures of different aspects of the education systems exist. In recent years, statistical
collections and surveys on education have expanded, providing crucial data for understanding
and comparing the education sector between countries. System-level information relevant to
understand drivers of change at the central administration or regulatory level is already collected.
A number of teacher, school or school-teacher surveys also exist at the national level, either run by
statistical agencies or by academic researchers (e.g. the London School of Economics (LSE) Survey of
Headteachers in England).
International surveys such as PISA, PIRLSS, TIMSS or TALIS are repeated and continuously
refine internationally validated questions that allow pedagogic and organisational practices to be
captured. These multi-level surveys sometimes collect similar data from several stakeholders (for
example teachers and students), therefore allowing the answers to be compared across surveys.
One way forward would be to develop core questions for these surveys that would be repeated over
their different cycles, as well as some specific modules about new emerging practices. Like for any
established survey, the difficulty is to make space for new questions.

OVERVIEW
Conclusion
Whether ambitious or modest, using either new or using existing surveys, there are several ways
to move forward the measurement agenda on educational innovation. Now that we know how this
could be done, and what kind of information we would get, and how this could lead to educational
improvement, we need to weigh the costs and benefits of investing in specific efforts to better
identify what is changing significantly and what the main drivers and effects of these innovations
are. A measurement agenda is key to any innovation strategy.

References
Arundel, A. and D. Huber (2013), From Too Little to Too Much Innovation? Issues in Measuring
Innovation in the Public Sector, Structural Change and Economic Dynamics, 27, 146-159.
Arundel, A., K.R. OBrien and N. Torugsa (2012), Measuring Innovation in the Australian Public Sector,
Australian Innovation Research Centre, Hobart, Tasmania
Avvisati, F., S. Hennessy, R. B. Kozma, and S. Vincent-Lancrin (2013), Review of the Italian Strategy for
Digital Schools, OECD Education Working Papers, No. 90, OECD Publishing.
Barret, D. (1998), Paradox Process: Creative Business Solutions Where You Least Expect to Find
Them, AMACOM, New York-USA
Bigi, M., N. Greenan, S. Hamont-Cholet and F. Lanfranchi (2013), Y-a-t-il convergence de lengagement
des salaris du secteur priv et des agents de la fonction publique dEtat face aux changements
organisationnels ?, Revue dconomie politique, 123(3), 377-401.
Bloch, C. (2011), Measuring Innovation in the Nordic Countries (MEPIN), Danish Centre for Studies in
Research and Research Policy (CFA), http://www.nordicinnovation.org/Global/_Publications/
Reports/2011/201102_MEPIN_report_web.pdf.
Csikszentmihalyi, M. and B. Schneider (2001), Becoming Adult: How Teenagers Prepare for the World of
Work, Basic Books.
Damanpour, C. and S. Gopalakrishnan (1998), Theories of Organizational Structure and innovation

Adoption: The role of Environmental change, Journal of Engineering and Technology Management
15.1: 1-24
Dumont, H., D. Istance and F. Benavides (2010), The Nature of Learning. Using Research to Inspire
Practice, OECD Publishing, Paris.
European Commission (2010), Innovation in Public Administration, Flash Eurobarometer 305.
European Commission (2012), Innovation in The Public Sector: Its Perception in and Impact on Business,
Flash Eurobarometer 343.
European Commission (2013), European Public Sector Innovation Scoreboard 2013: A Pilot Exercise, http://
ec.europa.eu/enterprise/policies/innovation/files/epsis-2013_en.pdf.
European Commission (2014), The Role of Public Support in the Commercialisation of Innovations, Flash
Eurobarometer 394.
Foray, D. and J. Raffo (2012), Business-driven innovation: is it making a difference in education? An

analysis of educational patents, OECD Education Working Papers, N 84, OECD Publishing.
Gault, F. (Ed.) (2013), Handbook of Innovation Indicators and Measurement, Edward Elgar.
Gault, F. (2012), User Innovation and the Market, Science and Public Policy, 39: 118-128.

REFERENCES
Greenan, N. and E. Lorenz (2013), Developing harmonized measures of the dynamics of work organizations
and work, in F. Gault Ed., Handbook of Innovation Indicators and Measurement, Edward Elgar.
Hargadon, A. and R. Sutton (2000), Building and innovation factory, Harvard Business Review
Higher Education as a Generator of Strategic Competences (HEGESCO) (database),

http://www.hegesco.org/.
Hughes, A., K. Moore and N. Kataria (2011), Innovation in Public Sector Organisations: A pilot survey for
measuring innovation across the public sector, NESTA, http://www.nesta.org.uk/sites/default/files/
innovation_in_public_sector_orgs.pdf.
Krkkinen, K. (2013), Bringing About Curriculum Innovation. Implicit Approaches in the OECD
Area, OECD Education Working Papers, No. 82, OECD Publishing.
Larson, R. and M. Csikszentmihalyi (1983), The experience sampling method, New Directions for
Methodology of Social and Behavioral Science, 15, 41-56.
Lekhi, R. (2007), Public Service innovation, A research report to the Work Foundations Knowledge
Economy Programme, London UK
MEADOW Consortium (2010), The MEADOW Guidelines, Grigny, France, http://meadow-project.eu/

images/2013/meadowguidelines.pdf.
Mulgan, G. and D. Albury (2003), Innovation in the Public Sector, Strategy Unit, Cabinet Office, London.
OECD (1997), The Oslo Manual. 2nd Edition, OECD Publishing, Paris.
OECD (2010a), Measuring Innovation. A New Perspectives, OECD Publishing, Paris, http://dx.doi.
org/10.1787/9789264059474-en.
OECD (2010b), The OECD Innovation Strategy: Getting a Head Start on Tomorrow, OECD Publishing, Paris,
http://dx.doi.org/10.1787/9789264083479-en.
OECD (2010c), Educational Information Systems for Innovation and Improvement, Workshop
Summary Report, http://www.oecd.org/edu/ceri/46413418.pdf.
OECD (2012), Connected Minds: Technology and Todays Learners, Educational Research and Innovation,
OECD Publishing, http://dx.doi.org/10.1787/9789264111011-en.
OECD (2013a), Synergies for Better Learning: An International Perspective on Evaluation and
Assessment, OECD Reviews of Evaluation and Assessment in Education, OECD Publishing,
http://dx.doi.org/10.1787/9789264190658-en.
OECD (2013b), Leadership for 21st Century Learning, OECD Publishing,

http://dx.doi.org/10.1787/9789264205406-en.
OECD/Eurostat (2005), The Oslo Manual. 3rd Edition, OECD Publishing, Paris,
http://dx.doi.org/10.1787/9789264013100-en.
Payne, (2010), So Much Reform, So Little Change, Harvard Education Press, Cambridge, MA.
Programme for International Student Assessment (PISA) (database), http://www.oecd.org/pisa/.
Progress in International Reading Literacy Study (PIRLS) (database), http://timssandpirls.bc.edu/.
Research into Employment and professional Flexibility (REFLEX) (database).
Schleicher, A.(Ed.) (2012), Preparing Teachers and Developing School Leaders for the 21st Century: Lessons
from around the World, OECD Publishing, Paris.

REFERENCES
Statistics Sweden (2011), Learning Organisations Matter, http://www.scb.se/statistik/_publikationer/

NR9999_2011A01_BR_NRFT1101.pdf.
Trends in International Mathematics and Science Study (TIMSS) (database), http://timssandpirls.bc.edu/.
Vieluf, S., D. Kaplan, E. Klieme and S. Bayer (2012), Teaching Practices and Pedagogical Innovation: Evidence
from TALIS, OECD Publishing.
Winner, E., T.R. Goldstein and S. Vincent-Lancrin (2013), Art for Arts Sake? The Impact of Arts Education,
OECD Publishing, Paris.

Readers guide
How to read the figures and tables

Part I of this book presents the levels of innovation across different sectors of the economy. Bar
charts are used to display the percentage of graduates employed in each sector who reported that,
for example, they played a role in introducing innovation.
The bar charts show how education (represented by the horizontal bars) compares with health,
manufacturing, and business activities (first three bars reading from the left, for each country), as
well as how it compares with the economy as a whole (right hand bar for each country) (See Figure
1). The country mean provides an average across the countries surveyed.
Figure 1 How to read Figures describing innovation in the education sector in Part I
The horizontal bars indicate the values for education
Health Manufacturing Business activities All Economy Education

%
100
80
60
40
20
0
e
nc gar
y al blic ain ny an nd um way onia ania rkey stria land ly and ds nia dom
ug Sp erma y me Pola elgi Ita l lan love
Fra Hun Port Repu No
r
Es
t hu Tu Au itzer Fin her ing
ch
G nt
r B Lit t S
dK
e u Sw Ne t e
Cz Co Un
i
The education systems are arranged according to their

values for education, from lowest to highest

READERS GUIDE
Figures are also used in Part I to depict the odds ratio of a graduate reporting some aspect of
innovation (namely product or service, technology, tools or instrument, or knowledge or methods)
according to the level of education they are employed in (primary, secondary or tertiary). As shown
in Figure 2, secondary education is the reference point for the analysis reported in these Figures. An
odds ratio of less than one indicates that there is a lower likelihood of something occurring, whilst a
ratio of more than one suggests an increased likelihood. Significance tests have been used to check
whether the odds ratio is significantly different from one
Figure 2 How to read Figures describing innovation in the education sector in Part I
Product Technology, tools Knowledge

or service or instruments or methods
odds ratio
3
2.01*** 1.91***
2
1.44***
1.17
1.06
1
0.96
0,5
Primary education Higher education
A value below one indicates a lower likelihood A value of around 2 indicates there are twice
of this situtation in primary education the odds that this situation will occur in higher
than in secondary education, because secondary education as in secondary education.
education is the refrerence category The *** indicate that this is a statistically
significant difference at the 0.01 level

READERS GUIDE
The Figures in Part II provide graphical representation of changes across time. Horizontal bars
indicate the percentage-point change between the first and last measurement for each education
system; the right hand axis indicates their absolute values, whilst the direction of change is shown
by the colour of the bar and the values displayed above the graph (a negative change implies
a decrease in the practice, whilst a positive change implies an increase, see Figure 3). Negative
change is represented by a dark bar, and positive change, by a white bar. Vertical bars represent
the percentage level of the considered practice in each education system for two or more years
measured on the left hand axis.
Figure 3 How to read change Figures in Part II
A negative value shows that there has been a reduction

in this particular practice for this country. This is also The horizontal bars indicate absolute % point change
indicated by the darker shading on the horizontal bars. and are measured on the right hand axis
2003 2007 2011 Negative change Positive change OECD (average absolute change)
% -13 -7 -6 -6 -2 -0 +1 +2 +3 +7 +10 +10 +12 +12 +12 +12 +14 +15 +17 +18 +20 +24 +28 +31
% point
+36
100 40
90
35
80
70 30
60 25
50 20
40
15
30
20 10
10 5
0 0
na tts nd rea tes ota bec hile land ore* ry** ey** n*** ay** na** n*** rio** lia** ca*** n*** ia*** el*** d.*** ly*** ia***
dia se ala Ko Sta nes Que C ng p ga rk ea w hi apa nta tra fri ede en sra Fe Ita nes
In achu Ze d n E inga Hun Tu D m Nor g, C J O Aus th A Sw Slov I an
s w ite Mi S n ss
i do
s e n C o u In
Ma N U OE g K S o R u
n
Ho
Education systems are arranged from those with the
largest decrease to largest increase over time, as
The vertical bars are measured on the left hand axis indicated by the horizontal bars.
Asterisks are used on the horizontal axis to indicate countries where the difference in the
percentages measured on the left hand axis are significantly different from zero. The smaller the
significance level, the more confident the reader can be that there is a genuine difference between
the two time periods.
*** = significant difference in percentages at the 0.01 level;
** = significant difference in percentages at the 0.05 level;
* = significant difference in percentages at 0.1 level.
The example Figure presented above shows that there was a decrease by 24% in the practice
reported in Hungary (horizontal bars are read on the right axis, and values are given on top of the
Figure), and that this decrease was significant at the 0.01 level (i.e., there is 1 chance out of 100 that
the change estimate is due to chance). The vertical bars show that this was caused by a decrease
from just over 70% to less than 50% between 2003 and 2009.

READERS GUIDE
Summary Figures provide combined information about changes, drawing on the data represented
in the module (see Figure 4). They illustrate the typical direction of change for the block of related
practices: bars above the zero horizontal axis (intercept) indicate that average changes were
positive (i.e. an increase) within that system, whilst bars below the bar indicate negative changes
(a reduction). Bars that cross the zero intercept show countries where the direction of change was
not consistent across the various categories represented. If the bar is longer above than below the
horizontal axis (e.g. for Portugal in the graph below), then it shows that systems have increase these
broad categories of practices, even though some related practices were also decreased.
Figure 4 How to read the summary Figures in Part II
The bars are measured on the left hand OECD values are indicated at the far right. The OECD column refers to
axis. Those above the line show an the actual change across the OECD in each of the practices discussed.
increase in a certain practice over time; The second bar shows the sum of the absolute change in each of
those below the line show a decrease. these and so indicates the overall magnitude of the changes occurring
Compared to students in the same grade in other schools Compared to national or regional benchmarks Compared to other students
%
60
40
20
-20
-40
-60
d ey lic ia n ly ia ry ia es d ay in n d ea ce m rk el ds d g l a a n y ile d ia n n )
lan rk ub str ede Itastral nga venStat alanorw Spa JapainlanKor ree lgiu ma Israrlan lanbourtugatoni nad atio manCh elanonesmeameaange
Ire TuRep Au Sw u u l
A H S ite w Z
o d e N F G e
B D e n e Po o r s
E C d Ga e r e r Ic nd CD D ch
h th em P Fe I E EC s.
z ec U n Ne Ne L ux n O O (ab
C si a
u s
R
Education systems are arranged from smallest absolute change across the
practices measured to largest absolute change.
Effect size tables show the effect size of each change and indicate whether this is small (0.2 to
0.5), medium (0.5 to 0.8) or large (over 0.8). The purpose is to present a standardised measure of the
changes over the different indicators presented. It is these effect sizes that have been used in the
creation of the composite innovation indices (see below). The letter (m) indicates missing data.

READERS
1. HIGHLY INNOVATIVE WORKPLACES IN EDUCATION AND OTHER GUIDE
SECTORS
Table
Table 1 How
1 How to read
to read thethe summary
summary data
data tables
tables in in Part
Part II II
Change in remedial education Change in enrichment education
8th grade 4th grade 8th grade 4th grade
Remedial reading
Informal reading
mathematics
mathematics
mathematics
mathematics
Enrichment
Enrichment
Enrichment
Enrichment
initiatives
specialist
Remedial
Remedial
Remedial
Remedial
science
science
science
science
99-07 99-07 03-07 03-07 01-06 99-07 99-07 03-07 03-07 01-06
Australia 0,06 -0,23 0,04 0,07 m -0,16 -0,06 -0,07 -0,11 m
Ontario m m -0,46 -0,27 -0,05 m m -0,51 -0,11 -0,12
Quebec m m 0,37 0,30 -0,33 m m 0,13 0,39 0,03
Czech Republic 0,08 -0,29 m m m -0,45 -0,33 m m m
France m m m m -0,01 m m m m 0,05
Germany m m m m -0,07 m m m m 0,02
Hungary 0,38 0,44 0,09 0,01 0,07 0,26 -0,13 -0,06 -0,05 0,10
Iceland m m m m 0,16 m m m m -0,14
Israel -0,19 -0,36 m m -0,04 0,15 -0,65 m m -0,02
Italy -0,08 -0,46 -0,02 -0,20 -0,15 0,00 0,03 0,13 0,00 0,09
Japan 0,51 0,42 0,12 0,10 m 0,76 0,61 0,53 0,21 m
Korea 1,15 0,34 m m m 0,85 0,34 m m m
Netherlands m m 0,11 -0,04 -0,16 m m 0,01 -0,14 0,37
New Zealand m m -0,10 -0,11 0,13 m m -0,02 -0,32 -0,06
Norway m m m m -0,02 m m m m 0,06
Slovak Republic m m m m 0,45 m m m m 0,29
Slovenia -0,29 -0,06 0,00 0,02 -0,05 -0,37 -0,25 0,29 0,01 0,00
Sweden m m m m -0,32 m m m m 0,15
Turkey 0,56 0,47 m m m -0,42 -0,42 m m m
England 0,86 0,78 0,07 -0,04 0,14 0,65 0,70 0,14 0,38 0,09
Scotland m m 0,05 0,24 0,02 m m 0,11 0,02 0,21
United States 0,32 -0,10 -0,02 -0,09 0,13 0,00 0,09 -0,14 -0,30 0,00
OECD (average) 0,30 0,09 0,02 0,00 -0,01 0,24 0,08 0,04 -0,04 0,07
OECD (average absolute) 0,42 0,36 0,12 0,12 0,14 0,36 0,32 0,18 0,17 0,11
Hong Kong, China 0,43 -0,01 -0,13 -0,09 -0,11 -0,07 -0,53 0,09 0,13 0,36
Indonesia 0,15 -0,02 m m m -0,36 -0,49 m m m
Russian Federation 0,14 0,03 0,53 0,30 0,32 -0,98 -1,15 0,39 0,19 0,26
Singapore -0,30 -0,23 -0,15 0,29 -0,06 0,20 0,39 -0,08 -0,01 -0,12
___ Education
Education systems
systems are arrangedare arranged
by OECD by OECD and Partner countries.
and Partner Theyindicates
Light shading are listed alphabetically.
small effect sizes.
countries. They are listed alphabetically. Medium effect sizes are indicated by moderate
___ A negative value indicates a decrease in the activity over time. The letter (m) indicates that
shading. Dark shading indicates large effect sizes.
data is missing.
A negative value indicates a decrease in the activity over
___ Education systems
time. The are
letter (m) arranged
indicates by OECD
that data and Partner countries. They are listed alphabetically.
is missing.
A negative value indicates a decrease in the activity over time. The letter (m) indicates that data
is missing. Light shading indicates small effect sizes. Medium effect sizes are indicated by moderate
shading. Dark shading indicates large effect sizes.
MEASURING
MEASURING INNOVATION
INNOVATION IN EDUCATION
IN EDUCATION : A NEW
: A NEW PERSPECTIVE
PERSPECTIVE OECD
OECD 20142014 5147
READERS GUIDE
The final chapter reports the potential associations between composite innovation indices and
various education system outcome measures, including performance and equity. Scatter plots are
used to depict the levels of innovation and outcome measures for each country. A line of best fit
is added to indicate whether or not there is a trend and the extent to which the points are close
to the line (indicating that the two measures are correlated). Quadrants are determined by placing
horizontal and vertical lines at the appropriate mean. In the case of innovation measures, this is the
mean of available data from OECD countries, while for the outcome measures the mean is drawn
from the original data.
Figure 5 How to read the summary data tables in Part III
Here the line of best fit shows a positive slope, suggesting that higher values on
TIMSS 8th grade
the x axis are associated with higher values on the y axis. Countries are scattered
to either side, indicating that the two measures are not highly correlated.
maths score - 2011
650
SGP KOR
600 HKG
JPN
US-MA
US-MN RUS
550 CA-QC
US-IN CA-ON
USA DEU ISR
ENG HUN
TIMSS mean
500
AUS ITA NOR SVN
NZL SWE TUR
450
CHL
OECD mean
400
10 15 20 25 30 35
Overall innovation
Vertical line identifies the OECD mean for the The horizontal line is drawn at the mean value for TIMSS
composite innovation index being used. 8th grade science across all participating countries.
The country codes used in the Figures in part III are:
AUS Australia HKG Hong Kong KOR Korea CA-QC Quebec
AUT Austria HUN Hungary US-MA Massachusetts RUS Russian Federation
CHL Chile US-IN Indiana US-MN Minnesota SGP Singapore
CZE Czech Republic IDN Indonesia NLD Netherlands SVN Slovenia
DNK Denmark ISR Israel NZL New Zealand SWE Sweden
ENG England ITA Italy NOR Norway TUR Turkey
DEU Germany JPN Japan CA-ON Ontario USA United States

PART I
Comparing innovation in education

with other sectors

CHAPTER 1
Highly innovative workplaces

in education and other sectors
A proportion of graduate employees have characterised their workplace

as having high or very high levels of innovation. These highly innovative
workplaces may focus on one type of innovation or incorporate several types.
The existence of highly innovative workplaces within the education sector
can be compared with other sectors, such as manufacturing or other public
services.
Exploring differences in the proportion of such highly innovative workplaces
at different levels of education provides insights into how these are spread
across primary, secondary and higher education.

1. HIGHLY INNOVATIVE WORKPLACES IN EDUCATION AND OTHER SECTORS
Innovation occurrence and types within the education sector
General findings
Highly innovative workplaces within the education sector are very common in Europe (Figure1.1).
On average, 69% of graduates employed in the education sector considered that their organisation
was highly innovative regarding at least one type of innovation in 2005. This was the case for at least
half of the graduates in all 19 countries reported and for more than 70% of graduates in 11 countries.
Nevertheless, highly innovative workplaces in education do not typically cover each of the
different forms of innovation (Figure1.2). Of graduates working in education, 19% reported that
their organisation was highly innovative concerning all three types of innovation. In no country did
this figure exceed 35%.
Knowledge or methods innovations are more common than other types of innovation (Figure1.3).
On average 59% of graduates in the education sector worked in organisations that were highly
innovative in terms of knowledge or methods. For technology, tools or instruments innovation this
was the case for only 36% of graduates and regarding product or service innovation, 38%. Knowledge
or method innovation was the most common type of innovation in a large majority of countries,
followed by product or service innovation and technology, tools or instruments innovation. However
product or service innovation was more common than technology, tools or instruments innovation
in five countries, while the opposite was true only in Estonia.
Country specificities
The United Kingdom (followed by Slovenia and Italy) has an above average proportion of highly
innovative workplaces in education and exhibits more types of innovation. In 2005, the United
Kingdom had the largest share of graduates that considered their workplace to be highly innovative
regarding both at least one type of innovation (79%) and all three types of innovation (33%). The
United Kingdom also ranked above average across the countries studied for all three types of
innovation individually. In terms of both innovation occurrence and type, both Slovenia (78% and
23%) and Italy (73% and 23%) outperformed an average European country. Italy also ranked above
average in product or service and technology, tools or instruments innovation, while Slovenia did so
in knowledge or methods innovation.
In contrast, the French and Hungarian education sectors seem to contain few highly innovative
workplaces. The share of graduates employed in highly innovative workplaces regarding both at
least one type of innovation and all three types of innovation was below the European average in
France (51% and 9%) and Hungary (55% and 10%). In addition, France ranked below average for all
three types of innovation individually, while this was the case for Hungary concerning knowledge
or methods and product or service innovation.
The Netherlands and Switzerland show that workplaces in the education sector can be highly
innovative regarding certain types of innovation. In 2005, the Dutch education sector outperformed
an average country in highly innovative workplaces regarding at least one type of innovation (77%)
as well as knowledge or methods and product or service innovation individually. However, the Dutch
education sector ranked below European average both regarding the range of types of innovation
(16%) and technology, tools or instruments innovation. Although the occurrence of innovation (73%)
was high in Switzerland, the country ranked below the average regarding both the range of types of
innovation (13%) and product or service innovation.

Figure1.1 Education professionals in highly innovative workplaces regarding

at least one type of innovation, by country
Percentage of graduates working in the education sector in highly innovative workplaces, 2005 or 2008
%
90
80 77.0 78.4 78.7
71.9 72.4 72.6 73.1 73.2 74.7
69.0 69.4 69.6 70.2 70.3
70 63.6 64.9 65.7
60 59.6
54.7
51.0
50
40
30
20
10
0
** y ** ** ** n y n d y a a y a * *
d*
* ** ** ***
e* r al c * Spai man mea olan lgium rwa toni uani urke ustri land Italy s* a*
ra nc nga rtug ubli e r y P B e N o E s t h T A e r n lan land veni gdom
F u o ep G t r L i it z Fi r o in
H P
hR un th
e Sl
ec Co Sw Ne dK
Cz n ite
U
1 2 http://dx.doi.org/10.1787/888933081853
Notes: ***=difference with the country mean significant at the 0.01 level; **=difference with the country mean significant at the 0.05 level; * =difference
with the country mean significant at 0.1 level. Year 2008 for Hungary, Lithuania, Poland, Slovenia and Turkey, year 2005 for other countries.
Source: Authors calculations based on REFLEX (2005) and HEGESCO (2008)
Figure1.2 Education professionals in highly innovative workplaces

across three types of innovation, by country
%
90
80
70
60
50
40 33.0
32.1
30 22.9
20.8 20.8 21.6 21.9 22.5 22.6 22.8
20 17.0 17.6 18.6 19.1
13.1 15.5
9.3 9.9 10.2 11.3
10
0
**
e * ny *
** *** y *** d *** ** ia ic ia n m d al ia * d * in *** ***
nc ry a ds ston publ ustr mea lgiu nlan rtug uan enia olan Italy Spa key om
ra m
a
n ga orw rlan rlan E R e A r y B e Fi Po i t h l ov P u r g d
F r e e t L T
Ge Hu N itz eth ec
h un
S in
Sw N Cz Co dK
n ite
U
1 2 http://dx.doi.org/10.1787/888933081872
Notes: ***=difference with the country mean significant at the 0.01 level; **=difference with the country mean significant at the 0.05 level; *=difference
Box 1.1 Data source details for Chapter 1

REFLEX (2005) and HEGESCO (2008) surveys asked higher education graduates five years after their
graduation How would you characterize the extent of innovation in your organization or your work?
regarding product or service, technology, tools or instruments and knowledge or methods innovation.
High innovation corresponds to values 4 and 5 in the scale from 1 (very low) to 5 (very high). The education
sector includes primary, secondary and higher education as well as other non-specified education activities.

Figure1.3 Education professionals in highly innovative workplaces, by innovation type and country
% Knowledge or methods
80
71.0 72.3
70 66.3 67.2 67.7
61.8 62.6 63.9 64.6
58.9 59.1 60.1 61.4
60 53.2 55.2
49.8 50.2 51.4
50 43.6
40 37.6
30
20
10
0
*** *** c *** al
*
a*
* * ly n ey any ium way and and ia ** ia ** d *** *** *** ***
n ce ary bli t ug toni Spa
in Ita mea urk m l g o r rl ol a n tr n o m nds n ia
a g u r y T r e z e P s l a d a e
Fr Hun Rep Po Es tr Ge B N it hu Au Fin ng rl v
un Sw Lit Ki ethe Slo
z ec
h Co i t ed N
C Un
% Product or service
80
70
60
50.3 52.3
50 45.9 48.4
42.0 44.2
37.6 39.1 39.3 40.5
40 35.5 36.4 37.3
30.9 32.7 32.9 33.8
29.2
30
21.8 22.3
20
10
0
*** *** *** l c a * *** s *** y ** ***
ce ary and nia ga rway any ubli tonia land ean pain gium stri nia d *
a n g l h ua ortu o r m ep E s Po m S e l Au l ove lan Italy n d r ke dom
Fr Hun tzer t N e y n a u
Li P G hR un
tr B S Fi erl T
Ki
ng
Sw
i ec Co
th
Cz Ne i t ed
Un
% Technology, tools or instruments

80
70
60
51.2 53.0
50 44.2 45.3 45.7
40.4 42.9
40 35.8 36.1 36.4 37.1 37.3
32.7 33.9
28.6 28.8 30.4
30 23.7 25.5
19.1
20
10
0
*** *** *** *** ry nd nd ia nd m an stria blic ugal in ** ia ** ania ly *** y *** ***
a ny nds way nce unga Pola inla oven erla elgiu me u pu ort p a t o n h u I t a r k e o m
a r a F l z y A e S d
rm rl No Fr H S it B tr hR
P Es Lit Tu ng
Ge ethe Sw un ec Ki
N Co Cz i t ed
Un
1 2 http://dx.doi.org/10.1787/888933081891
Source : Authors calculations based on REFLEX (2005) and HEGESCO (2008)

Innovation occurrence within the education sector compared with other sectors
General findings
The education sector stands at the average level in terms of innovative workplaces when
compared to other sectors of the economy in Europe (Figure1.4 and Figure1.5). Similarly to the 69%
average across sectors in 2005, 70% of graduates in the education sector considered that they were
employed in highly innovative workplaces (excluding Portugal). The proportion of highly innovative
workplaces in education was similar to that across sectors in most individual countries, being above
in three countries and below only in Portugal and the Czech Republic. The manufacturing sector
(79%) had more highly innovative workplaces than the education sector in all countries, with the
exception of Slovenia. The education sector ranked below most other sectors of the economy only
in the Czech Republic.
The education sector is below average in terms of the presence of three types of innovation
(Figure1.6 and Figure1.7). In the education sector, 19% of graduates considered that their workplace
was highly innovative concerning all three types of innovation compared with the 23% average for
the economy. In the majority of individual countries the education sector ranked below the economy
as whole. Highly innovative workplaces with regard to the three types of innovation were most
common in the manufacturing sector (34%), which outperformed education in the great majority of
individual countries as did business activities.
However, highly innovative workplaces are more common in the education sector than in other
public services (Figure1.4 to Figure1.7). In 2005, fewer graduates were employed in workplaces that
were highly innovative regarding at least one type of innovation in public administration (53%) and
health (64%) than in education (70%). In this respect, the health sector ranked below the education
sector in five European countries as well as on average the opposite was true only in the Czech
Republic. When looking across types of innovation, the education sector (19%) ranked above the
health (17%) and public administration (12%) sectors. This was the case for the health sector across
three countries, whereas the case was reverse for the Czech Republic and Slovenia.
In terms of innovative workplaces, education compares relatively well to other sectors in
Switzerland. In 2005, innovative workplaces regarding at least one type of innovation were more
common in the Swiss education sector than in the economy as a whole (by 6% points) or the health
sector (by 19% points). The Swiss education sector also outperformed the health sector (by 4% points)
regarding the occurrence of several types of innovation in workplaces.
In contrast, workplaces in the education sector are less innovative than those in other sectors
in the Czech Republic. Most other sectors of the economy including health, manufacturing and
business activities outperformed the Czech education sector in terms of both occurrence and
range of types of innovation.
The case of Slovenia highlights that even less innovative education workplaces can do
comparatively well concerning some types of innovation. Although the Slovenian education sector
ranked below most other sectors regarding the range of types of innovation in workplaces, it
outperformed both the economy as a whole (by 7% points) and the manufacturing sector (by 7%
points) for at least one type of innovation.

Figure1.4 Professionals in highly innovative workplaces regarding at least

one type of innovation, by sector
Percentage of graduates working in highly innovative workplaces, 2005 or 2008
%
90
80 76.1 78.7
73.4 74.4 74.7
67.1 68.8 68.8 69.8
70 63.3 64.4 64.8 66.3
60 53.0
50
40
30
20
10
0
c *** *** s* ** *** *** ***
bli *** ity ts ge ale on d g
Pu tion t i on a lth t ric v ice u ran vera les c ati e an inin tion t ies n ce r ing
a c e c r a A o u r m a i a tu
str u H Ele rs
e
res
t h Ed ltu nic tiv Fin
ini str he
W cu mu ac fac
m C on Ot a nd gri m e ss a nu
a d l A o si n M
te C
Ho Bu
1 2 http://dx.doi.org/10.1787/888933081910
Notes: ***=difference with the education sector significant at the 0.01 level; **=difference with the education sector significant at the 0.05 level;
*=difference with the education sector significant at 0.1 level. Year 2008 for Hungary, Lithuania, Poland, Slovenia and Turkey, year 2005 for other
countries. Portugal is excluded.
Figure1.5 Professionals in highly innovative workplaces regarding at least one type of innovation,
by sector and country

%
100
80
60
40
20
0
e
nc gar
y al blic ain ny an nd um way onia ania rkey stria land ly and ds nia dom
ug Sp erma y me Pola elgi Ita l lan love
Fra Hun Port Repu No
r
Es
t hu Tu Au itzer Fin her ng
h G n t r B Lit t S Ki
e c u Sw Ne t ed
Cz Co Un
i
1 2 http://dx.doi.org/10.1787/888933081929
Notes: Year 2008 for Hungary, Lithuania, Poland, Slovenia and Turkey, year 2005 for other countries.

Figure1.6 Professionals in highly innovative workplaces across three types of innovation, by sector
%
90
80
70
60
50
40 34.3
32.4
30 26.0 26.4 28.6
19.5 21.5 21.7 22.0 22.8
20 15.6 17.1 19.0
12.4
10
0
c
bli *** ants **
ice
s on on ici
ty sa
le
e*
** *** nd *** *** n*
**
g*
**
Pu tion r lth rv ati cti ctr ole rag
ce e a ng iti
es tio rin
t ra s t au H ea
r se d uc s t ru l e h ve i n an l t ur ini t iv i c a t u
s re he E n E W A F u m ac n fac
ini nd Ot Co ric ss mu nu
ad
m la Ag ine om Ma
t e s C
Ho Bu
1 2 http://dx.doi.org/10.1787/888933081948
Figure1.7 Professionals in highly innovative workplaces across three types of innovation,


%
100 100
80 80
60 60
40 40
20 20
0 0
e y ry way and nds nia bli
c
tri
a n d al nia nia and ly in key
nc an ga ea ium an ug Ita a do
m
Fra erm Hun Nor tzer
l
e rla Esto epu Aus y m Belg Finl Port thua love Pol Sp Tu
r
ing
G i t h R t r L i S K
Sw Ne ec
h un ite
d
Cz Co Un
1 2 http://dx.doi.org/10.1787/888933081967

Innovation types within the education sector compared with other sectors
General findings
Education outperforms other sectors of the economy in terms of highly innovative workplaces in
knowledge or methods innovation the most common innovation type among economic sectors across
the countries covered (Figure1.8 and Figure1.9). Against the average of 49%, and more than in almost
any other sector, 60% of graduates in the education sector considered that their workplace was highly
innovative with regard to knowledge or methods innovation in 2005. Education surpassed most other
sectors in knowledge or methods innovation in Switzerland and the Netherlands and the economy
as a whole in nine other countries. As in education, knowledge or methods innovation was the most
common innovation type for highly innovative workplaces on average and across five other sectors.
In contrast, highly innovative workplaces in product or service innovation are less common in
the education sector than in the rest of the economy in Europe (Figure1.8 and Figure1.10). Only
38% of graduates employed in education worked in highly innovative organisations in terms of
product or service innovation in 2005. This was below the average (47%) and outperforming only
public administration (30%). With regard to the product or service innovation, education was among
the sectors with the smallest share of graduates in highly innovative organisations in the vast
majority of countries it only significantly outperformed health in Finland. While product or service
innovation was the second most common innovation type characterising workplaces in education,
it was the most common type for manufacturing, finance, whole sales and hotels and restaurants.
Also with regard to technology, tools or instruments innovation, highly innovative workplaces
are less common in education than in other sectors of the economy (Figure1.8 and Figure1.11).
In education, 36% of graduates worked in workplaces that were highly innovative in terms of
technology, tools or instruments innovation in 2005 against the average of 42% for the whole
economy. Education ranked below seven sectors of the economy, but above public administration
(28%) and health (33%). Manufacturing outperformed education in 13 countries, business activities
in 10 countries and health in five countries. Health ranked lower than education only in Switzerland
and the United Kingdom. Technology, tools or instruments innovation was the least common type
of workplace innovation overall and in five individual sectors.
Switzerland, and to a lesser extent, Finland and the Netherlands, are countries where education
compares relatively well to other sectors in terms of highly innovative workplaces. In 2005, the Swiss
and Dutch education sectors outperformed most other sectors of the economy in terms of knowledge
or methods innovation. Outranking health by 11% points, the education sector in Switzerland
stood at a similar standing to the economy as a whole regarding technology, tools or instruments
innovation. The Finnish education sector ranked above most other sectors in knowledge or methods
innovation and unlike in any other country outperformed health in product or service innovation
(by 8% points).
In contrast, the Czech Republic has few innovative education workplaces. In 2005, the Czech
education sector ranked below most other sectors for product or service and technology, tools
or instruments innovation. As to knowledge or methods innovation, education was at a similar
standing or below the rest of the Czech economy.
Germany provides an example of a country with high occurrences of some, but not all types of
innovation. While the German education sector outperformed most other sectors regarding highly
innovative workplaces in knowledge or method innovation, it ranked below most other sectors for
technology, tools or instruments innovation.

Figure1.8 Professionals in highly innovative workplaces, by sector and innovation type

Product or service Technology, tools or instruments Knowledge or methods

%
70
60
50
40
30
20
10
0
d ts lic on le on s n e ty re g ce h g ies on
an ran Pubtrati sa cti ice tio rag ici ltu nin inan alt rin vit ati
telstau i s h ole t ru s erv n ica Ave
l e ctr ricu mi F He ac
tu
a cti d uc
o s f
H re m
in W
Co
n he
r
mu E Ag and nu es
s E
ad Ot Co
m Ma sin
Bu
1 2 http://dx.doi.org/10.1787/888933081986
Notes: Year 2008 for Hungary, Lithuania, Poland, Slovenia and Turkey, year 2005 for other countries. Portugal is excluded.
Figure1.9 Professionals in highly innovative workplaces regarding knowledge

or methods innovation, by sector and country

%
80
60
40
20
0
e
nc gar
y
bli
c al ia in taly ean key any ium way and and ia ia d m s ia
ug ston Spa I r rl l an ustr nlan gdo rland ven
Fra Hun Repu Port E r ym Tu erm Bel
g r
No itze Po ithu A Fi i n e S lo
h t G L K t h
ec un Sw ite
d Ne
Cz Co Un
1 2 http://dx.doi.org/10.1787/888933082005

Figure1.10 Professionals in highly innovative workplaces regarding product

or service innovation, by sector and country

%
80
60
40
20
0
e
nc gar
y d ia al y y ic ia d n in m ia ia d ly ds ey m
lan an ug orwa man publ ston olan mea Spa lgiu ustr ven nlan Ita rlan urk do
Fra Hun tzer ithu Port N er R e E P ry Be A Slo Fi e T ing
i L G t t h K
Sw ec
h un Ne ite
d
Cz Co Un
1 2 http://dx.doi.org/10.1787/888933082024
Figure1.11 Professionals in highly innovative workplaces regarding technology, tools or

instruments innovation, by sector and country

%
80
60
40
20
0
y s y e c l
an land rwa ranc ary nd nd nia nd um an tria bli ga pain onia ania ly key
Ita do
m
rm ng Pola Finla love zerla elgi y me Aus epu ortu S t hu Tu
r
ing
Ge ethe
r No F Hu S i t B t r R P Es
Lit K
N Sw o un z e ch ite
d
C C Un
1 2 http://dx.doi.org/10.1787/888933082043

Innovation within the education sector by level of education
Innovation occurrence and types

Within the education sector, highly innovative workplaces are more common in higher
education than at either primary or secondary level (Figure1.12 and Figure1.13). Against 69% for the
education sector in general (including Portugal), 80% of graduates working in the higher education
sector in Europe were employed in highly innovative organisations in relation to at least one type
of innovation. This compares with 63% for secondary and 65% for primary education. The likelihood
of higher education workplaces being highly innovative in at least one type of innovation was twice
that of secondary or primary education. Moreover, the higher education sector outperformed most
other sectors of the economy including the manufacturing sector in terms of highly innovative
workplaces. While primary education stands at the average level when compared to other sectors,
secondary education ranked below the average.
Higher education has the largest share of highly innovative workplaces in terms of the three
different types of innovation, whereas secondary education ranks last in this respect (Figure1.12 and
Figure1.13). Against the 20% for the whole education sector, 28% of graduates in higher education
worked in innovative organisations regarding three innovation types. These figures were only 14%
for secondary and 17% for primary education. Graduates employed in higher education were more
than twice as likely to work for an organisation concentrating on all three types of innovation than
those working in secondary education. In comparison with the rest of the economy, higher education
had above average level of highly innovative workplaces in terms of the range of innovation types,
whereas both secondary and primary education ranked below average in this respect.
Innovation types
Highly innovative workplaces across the different levels of education are mostly focused on
knowledge or methods innovation. This was the most common type of innovation across all levels
of education in terms of graduates working in highly innovative organisations in 2005 (Figure1.12).
While product or service innovation was more common than technology, tools or instruments
innovation in primary and secondary education, the opposite was true for higher education.
Higher education also outperforms other levels of education regarding all different innovation
types (Figure1.12 and Figure1.14). Of all levels of education, higher education employed the largest
proportion of graduates to highly innovative workplaces with regard to knowledge or methods (71%),
product or service (44%) and technology, tools or instruments (50%) innovation. Higher education
also outperformed all other sectors of the economy regarding knowledge or methods innovation
and ranked above average for technology, tools or instruments innovation. Compared to secondary
and primary education, the odds of working in an innovative organisation in higher education
were more than twice greater for technology, tools or instruments innovation and twice greater for
knowledge or methods innovation. Relative to secondary education, the odds were one and a half
times greater for product or service innovation.
Secondary and primary levels of education have less highly innovative workplaces with respect to
different innovation types (Figure1.12 and Figure1.14). Of all education levels, innovative workplaces
in secondary education were the least common with regard to knowledge or methods innovation
(54%) as well as to product or service innovation (32%). As to technology, tools or instruments
innovation, primary education (29%) had fewer highly innovative workplaces. Although primary and
secondary education outperformed the economy average in knowledge or methods innovation, both
ranked below the average in product or service and technology, tools or instruments innovation.

Figure1.12 Education professionals in highly innovative workplaces, by innovation type

and education level
At least one type of Three types of Product Technology, tools Knowledge

% innovation innovation % or service or instruments or methods
90 90
80 80
70 70
60 60
50 50
40 40
30 30
20 20
10 10
0 0
Primary Secondary Higher Total Primary Secondary Higher Total
education education education education education education education education

Innovation likelihood compared with secondary education Innovation likelihood compared with secondary education
At least one type Three types Product Technology, tools Knowledge

of innovation of innovation or service or instruments or methods
odds ratio odds ratio
3
2.23*** 2.23*** 2.26*** 2.09***

2 2
1.54***
1.19 1.14
1.01
1 1
0.97 0.92
0,5 0,5
Primary education Higher education Primary education Higher education

1 2 http://dx.doi.org/10.1787/888933082062
Notes: ***=odds ratio significant at the 0.01 level; **=odds ratio significant at the 0.05 level; *=odds ratio significant at 0.1 level. Spain is excluded. Year
2008 for Hungary, Lithuania, Poland, Slovenia and Turkey, year 2005 for other countries.

Figure1.13 Professionals in highly innovative workplaces across three types of innovation,

by sector and education level
At least one type of innovation Three types of innovation

%
90
80
70
60
50
40
30
20
10
0
c n n n h y n s s e le n g n s e g n
bli tio tio tio alt tricit tio ice and ant rag sa tio and nin tio es nc rin tio
Pustra truc u ca He c u ca s erv tel taur Ave ole duca ure mi nica u sin Fina c tu u ca
i s d l e d h B f a d
in
m Con
e E e he
r Ho res W E
cu
lt mu nu er
e
ad ary ary Ot gri m Ma igh
o nd r im A Co H
c P
Se
1 2 http://dx.doi.org/10.1787/888933082081
Notes: Year 2008 for Hungary, Lithuania, Poland, Slovenia and Turkey, year 2005 for other countries. Portugal and Spain are excluded.
Figure1.14 Professionals in highly innovative workplaces, by innovation type,

sector and education level

%
80
70
60
50
40
30
20
10
0
d ts lic on le on s n e ty nd ing nce alth g n on ss on on
an ran Pubtrati lesa cti ice tio rag trici a n ina rin tio ati ine ati ati
telstau i s h o t ru s erv n ica Ave
l e c u re mi F He a ctu duca duc B us d uc d uc
o in W s r u E t f e e E e
H re
Co
n he m ul nu er
ad
m Ot m ric Ma da
ry ary gh
Co Ag on rim Hi
c P
Se
1 2 http://dx.doi.org/10.1787/888933082100

CHAPTER 2
Employee participation
in innovation in education
and other sectors
Employees may or may not have a role to play in introducing innovations

into their workplace. For those who do, such participation may be focused on
a specific type of innovation, or across various types.
The extent to which employees participate in innovation within the
education sector can be compared with other sectors of the economy such as
manufacturing or other public services. Differences across primary, secondary
and higher education can also be explored to understand whether participation
is more common at some levels than others.

2. EMPLOYEE PARTICIPATION IN INNOVATION IN EDUCATION AND OTHER SECTORS
General findings
Employee participation in innovation within the education sector is common in Europe
(Figure2.1). On average, 76% of higher education graduates working in the education sector in 2005
reported that they participated in introducing at least one type of innovation in their organisation.
With the exception of Hungary and Spain, this was the case for at least 70% of graduates.
However, employees in the education sector do not necessarily take part in introducing each
of the different forms of innovation (Figure2.2). Of graduates working in education, on average
23% reported that they participated in introducing three types of innovation in their organisation.
Only in Turkey, Estonia and the Czech Republic did this figure exceed 30% of the higher education
graduates, while it remained at 15% or below in four countries.
Employee participation in innovation is particularly high regarding knowledge or methods

innovation, but lower for product or service innovation and especially technology, tools or
instruments innovation (Figure2.3). In 2005, 70% of higher education graduates in the education
sector played a role in introducing knowledge or methods innovations in their organisations. For
product or service innovation this was the case for 44% of graduates and regarding technology, tools
or instruments, of 36% of graduates. Knowledge or methods innovation was the most common type
of innovation in all countries. Technology, tools or instruments innovation was less common than
product or service innovation in most countries the opposite was true only in Spain.
The Czech Republic, Estonia and Lithuania stand out as countries with above average level range
of employee participation in innovation in the education sector. In 2005, the share of graduates
participating in at least one type of innovation and all three types of innovation was above average in
Czech Republic (87% and 32%), Lithuania (86% and 30% in 2008) and Estonia (85% and 36%). Both the
Czech Republic and Estonia also ranked above the average regarding the three different innovation
types individually, whereas this was the case for Lithuania concerning knowledge or methods and
product or service innovation.
In contrast, employees in Hungary, Poland, Germany, Switzerland and Austria participate less
in innovation in education. Hungary had the smallest share of graduates taking part in any type
of innovation (69%) in their workplace and below average share regarding also all three types of
innovations (18%) as well as individually. Regarding employee participation in a range of types
of innovation in education, Poland (11%), Germany (12%), Switzerland (13%) and Austria (15%)
stood below the European average. More specifically, these four countries had below average level
participation in product or service and technology, tools or instruments innovation.
In Spain, the Netherlands, Italy and Slovenia employee participation is high for some innovation
types, but low for others. In 2005, Spain ranked below the average in the education sector employee
participation in any type of innovation as well as specifically in knowledge or methods and product or
service innovation. Although the Dutch education sector ranked below the average both in terms of
the range and technology, tools or instruments innovation, the case was reverse concerning product
or service innovation. In Italy, education employee participation was high for product or service
and technology, tools or instruments innovation, but low for knowledge or methods innovation, the
reverse of Slovenia.

Figure2.1 Participation of education professionals in the introduction of at least

one type of innovation, by country
Percentage of graduates working in the education sector who play a role in introducing at least one type of innovation, 2005 or 2008
%
100
90 85.7 87.3
77.6 78.0 78.7 85.3
80 72.6 74.9 75.1 75.4 76.1 76.1 76.7 76.9
70 71.7 72.1 72.6
70.6
60 68.9 69.3
50
40
30
20
10
0
*** n ** nc
e al d m m and tria way ean key and ia ly y ds a *** ia ** c*
**
ary Spai ug olan gdo lgiu r
inl Slov
en Ita man rlan bli
g Fra Port P i n B e t z erl Aus Nor ry m Tu F e r e t o ni uan u
un dK i t G th Es Lith ep
H
ite Sw un Ne hR
Un Co ec
Cz
1 2 http://dx.doi.org/10.1787/888933082119
Figure2.2 Participation of education professionals in the introduction

of three types of innovation, by country
Percentage of graduates working in the education sector who play a role in introducing three types of innovation, 2005 or 2008
%
100
90
80
70
60
50
35.7 38.0
40 29.1 29.2 29.8 32.4
30 22.6 23.3 23.3 24.2 25.3 26.3
18.0 18.4 18.7 19.9
20 11.0 12.2 13.2 14.8
10
0
*** *** *** *** s ** *** ce * l *** ** * *** *** **
ly nia um ean ga ay nd ia ***
l a nd any land stria and gary Fran Ita love elgi y m ortu Norw Finla pain gdom uan ublic onia ey *
Po erm tzer u e r l n S B t r P S n t h p s t r k
A u i i e E
G i th H un dK
L
hR dT
u
Sw Ne Co ite ec ite
Un Cz Un
1 2 http://dx.doi.org/10.1787/888933082138

REFLEX (2005) and HEGESCO (2008) surveys asked higher education graduates five years after their
graduation Do you play a role in introducing these innovations in your organisation? regarding product
or service, technology, tools or instruments and knowledge or methods innovation. The answer options
were Yes, No and Not applicable, no innovations. The education sector includes primary, secondary and
higher education as well as other non-specified education activities.

Figure2.3 Participation of education professionals in innovation, by innovation type and country

Percentage of graduates working in the education sector who play a role in introducing different types of innovation, 2005 or 2008
100
90 80.6 80.8 83.1
70.8 70.8 71.9 73.0 74.2 74.3
80 67.0 67.1 67.9 68.0 68.2 69.1 70.3
63.0 66.1
70 59.2 60.3
60
50
40
30
20
10
0
*** *** y *** * al d d y n d ia y s ** y ** ** **
ly nce r ain ium ug olan rlan gdom urke mea nlan ustr orwa land nia man nia * lic * nia *
Ita ga Sp Belg Port P tze Kin T Fi A er love Ger to ub a
Fra Hun i t ry N t h S E s ep h u
Sw ited un Ne hR Lit
Un Co ec
Cz
100
90
80
70 62.3 64.1
60 49.2 49.6 52.0 52.3
46.0 47.1 47.8
50 41.2 42.3 43.5
35.8 35.8 36.7 37.3 38.7
40 34.9
30.2
30 24.3
20
10
0
** ** *** y *** y ** l* ce in ** nia ium ean y * * * * * * *
d* d* ia n ga ran e om wa d * aly * ds * nia lic ** ia ** ey **
r
lan lan ustr nga rma ortu F S pa Slov Belg ry m ingd Nor inlan It rlan sto pub uan Turk
Po tzer A Hu e P t K F e E e h
i G un ed th hR Lit
Sw Co Unit Ne ec
Cz

100
90
80
70
60 48.3 50.3
50 41.5 43.0 45.1
37.3 38.0 39.3 40.2 40.7
31.9 33.2 35.7 35.7
40 30.5 30.5
25.9 26.8
30 18.8 20.6
20
10
0
** ** ** ** ** ** d* ay an ia ce ey l * * * * **
d * ny * ria * ds * ary ia m m ga nd * ly ** in ** ic ** ia
an orw me elgiu huan Fran ngdo Turk tu l
lan a st rlan ung oven zerl N try B t i P or inla Ita Spa pub s ton
Po erm Au e S l i t L i K F e E
G th H
Sw un ite
d
hR
Ne Co Un ec
Cz
1 2 http://dx.doi.org/10.1787/888933082157

General findings
Compared with other sectors of the economy, employee participation in innovation in the
education sector is common (Figure2.4 and Figure2.5). Above the 71% average for the economy in
2005, 76% of graduates working in education reported having played a role in introducing at least one
type of innovation in their organisation. In eight individual countries, participation in innovation
in education was significantly above that of the whole economy. Together with manufacturing
(78%), employee participation in innovation in education was among the highest of all sectors. Such
participation in innovation in education stood at a similar level to that of manufacturing in most
countries, the opposite being the case only for Switzerland and Turkey.
In terms of introducing different innovation types, the education sector stands at the average
level (Figure2.6 and Figure2.7). Across all sectors, 24% of graduates participated in introducing all
three types of innovation, compared with 23% of those working in education. While at the level
of or even above the economy as a whole in most individual countries, the education sector
ranked significantly below the whole economy only in Austria. Employee participation in three
types of innovation was the highest in agriculture and mining (30%), manufacturing (27%) and
business activities (26%), and significantly above education. The education sector ranked lower than
manufacturing in eight countries and lower than business services in nine, and below the European
country mean. Only in three countries did the education sector outperform manufacturing and, in
two others, business services.
Employees take part in introducing innovation more in education than in other public services
(Figure2.4 to Figure2.7). In 2005, fewer higher education graduates participated in introducing at
least one type of innovation in public administration (61%) and health (68%) than in education (76%).
In this respect, the health sector was significantly behind the education sector in nine European
countries, as well as on average. When looking at several types of innovation, education (23%) ranked
above public administration (14%), but stood at similar level to health (22%). Education outperformed
health in three countries, whereas the opposite was true only for Estonia.
The Czech Republic stands out as a country where employee participation in innovation in the
education sector is particularly common when compared to other sectors of the economy. In 2005,
the Czech education sector outperformed the economy as a whole in terms of both the level and
types of employee participation in innovation. The Czech education sector outperformed business
services by 4% points and manufacturing by 8% points in participation in at least one type of
innovation, and outperformed manufacturing by 9% points across the three types of innovation.
In contrast, employee participation in innovation in the Turkish education sector is less common
relative to other sectors and countries. In Turkey in 2008, fewer graduates employed in education played
a role in introducing at least one type of innovation than those employed in manufacturing (16% points
difference) or business activities (14% points difference). Education ranked below business activities by
24% points and manufacturing by 14% points regarding participation in various types of innovation.
The case of the Netherlands illustrates that employee participation in innovation in education
can be relatively high for some types of innovation. The Dutch education sector outperformed the
economy as a whole by 8% points, health by 14% points and business activities by 9% points in terms
of participation in at least one type of innovation. Yet, in terms of participation in various types of
innovation, education ranked below both manufacturing (15% points less) and business activities
(6% points less).

Figure2.4 Professionals participating in innovation, by sector

Percentage of graduates who play a role in introducing at least one type of innovation, 2005 or 2008
%
90
75.4 76.3 77.7
80 71.4 72.6 73.9 73.9 74.1
68.0 68.7 70.1 70.4
70 66.5
60.7
60
50
40
30
20
10
0
c *** *** * **
bli *** *** ts y* rag
e le
tio
n ** ice
s d g on g*
Pu tion ce lth on ran cit sa s* an nin cati
ra n an ea cti a u c tri Ave h ole n ica i tie s erv ure mi d u t u rin
t i H u e u v r t
ini
s F
ns
tr es
t El W m cti he ul E
uf
ac
Co dr m sa Ot ric an
ad
m an Co n es Ag M
te l s i
Ho Bu
1 2 http://dx.doi.org/10.1787/888933082176
Figure2.5 Professionals participating in innovation, by sector and country

Percentage of graduates who play a role in introducing at least one type of innovation, 2005 or 2008

%
100
80
60
40
20
0
e l ds lic
ary pain anc tuga land dom gium land stria rway ean rkey land enia ly y ia ia
Ita man rlan ston uan pub
ng S Fr Por Po ng Bel itzer Tu v
Hu Ki Au No try m Fin Slo r
Ge ethe E h
Lit ch R
e
i t ed Sw o un N e
Un C Cz
1 2 http://dx.doi.org/10.1787/888933082195

Figure2.6 Professionals participating across three types of innovation, by sector

Percentage of graduates who play a role in introducing three types of innovation, 2005 or 2008
%
90
80
70
60
50
40
26.1 26.8 27.1 29.5
30 22.4 23.2 23.5 23.6 24.8 25.3 25.5
22.1
20 14.2 16.3
10
0
c
bli *** e*
** h*
*
ale cit
y
tio
n
rag
e
tio
n
ice
s
tio
n
s*
** g*
* ts nd ***
Pu tion nc e alt o les c tri ca ve i ca e rv uc i tie rin u ran re aning
a a h e u A n s t r v t u t a u i
str Fin H W El Ed mu he
r ns cti ac es ult m
ini m Ot Co sa an
uf dr ric
ad
m Co i n es M l an Ag
s te
Bu Ho
1 2 http://dx.doi.org/10.1787/888933082214
Figure2.7 Professionals participating across three types of innovation, by sector and country
Percentage of graduates who play a role in introducing three types of innovation, 2005 or 2008

%
100 100
80 80
60 60
40 40
20 20
0 0
y ia ds gary nce l c
lan
d
an
d
lan ustr rlan Ita
ly nia y
um ean tuga rwa lan
d ain om nia bli nia rkey
Po erm tzer n Fra ve elgi m or Sp ingd hua epu Esto Tu
G i A h e Hu Slo B t ry P No Fin K L it R
t
Sw N e un t ed ec
h
Co Un
i C z
1 2 http://dx.doi.org/10.1787/888933082233
Source: REFLEX 2005 and HEGESCO 2008

General findings
Education outperforms most other sectors of the economy regarding employee participation
in knowledge or methods innovation in Europe (Figure2.8 and Figure2.9). Against the average of
61% and more than in most other sectors, 71% of graduates working in education played a role
in introducing knowledge or methods innovation in their organisation in 2005. The education
sector outperformed most other sectors of the economy in five countries. Employee participation
in knowledge or methods innovation was more common than participation in other types of
innovation in nearly all sectors of the economy including education.
A smaller proportion of graduates participate in product or service innovation in the education

sector than the rest of the economy on average (Figure2.8 and Figure2.10). In 2005, 44% of
graduates working in education had played a role in introducing product or service innovation,
below the average of sectors (47%) and outperforming only public administration (35%). Education
ranked significantly below most other sectors of the economy in six countries and on average
outperforming manufacturing only in Lithuania. Participation in product or service innovation was
the second most common for education and most other sectors of the economy, while holding
similar standing with knowledge or methods innovation for the hotel and restaurants sector.
Employee participation in technology, tools or instruments innovation in education is at the

average level when compared to other sectors of the economy (Figure2.8 and Figure2.11). Graduates
working in the education sector reported a participation rate of 35% for technology, tools or
instruments innovation in 2005, close to the 36% economy average. Education ranked below the
sectors of agriculture and mining (46%) and manufacturing (43%), but significantly above public
administration (24%) and health (30%). Manufacturing and business activities outperformed
education in eight countries. In contrast, education ranked above manufacturing in Czech Republic
and Estonia and above health in five countries. Amongst the different innovation types, participation
in technology, tools or instruments innovation was the least common for most sectors of the
economy.
With regard to employee participation in innovation, education ranks highly in terms of different
types of innovation in Spain. Spain was among countries where education outperformed most other
sectors of the economy regarding participation in knowledge or methods innovation in 2005. The
Spanish education sector also ranked significantly above the economy as a whole as well as health
(by 11% points) and business activities (by 9% points) in terms of technology, tools or instruments
innovation.
In contrast, in Poland education employee participation in innovation is relatively low. Together

with health, the Polish education sector ranked below most other sectors regarding participation in
product or service and technology, tools or instruments innovation in 2008.
The cases of Germany and Austria illustrate examples of countries where education employees
participate in some, but not all types of innovation. The German and Austrian education sectors
outperformed most other sectors of the economy with regard to participation in knowledge or
methods innovation in 2005. Yet, education ranked below manufacturing and business activities
concerning both product and service innovation and technology, tools or instruments innovation.

Figure2.8 Professionals participating in innovation, by sector and innovation type

Percentage of graduates who play a role in introducing innovation, 2005 or 2008

%
80
70
60
50
40
30
20
10
0
nc
e c n
bli tio nd nt
s n le on e g
alt
h es cit
y
ice
s re g on
a Pustra tel aaura nica
tio sa cti rag rin iti tri ltu nin ati
Fin ole t ru Ave ctu He ctiv c serv ricu mi uc
in i o
H re s t
mu W
h
n s
nu
f a
sa Ele er Ag an d E d
ad
m m Co Ma
es Ot
h
Co sin
Bu
1 2 http://dx.doi.org/10.1787/888933082252
Figure2.9 Professionals participating in knowledge or methods innovation, by sector and country

Percentage of graduates who play a role in introducing knowledge or methods innovation, 2005 or 2008

%
100
80
60
40
20
0
ly nce ry l y ia ay ds enia any c
Ita ga Spa
in ium ga an
d
an
d om rke ea
n d
lan ustr orw rlan nia bli nia
Fra Hun lg rtu Pol zerl ingd Tu ry m Fin ov rm sto epu thua
Be Po i t K t A N t h e S l G e E R L i
Sw ited un Ne ec
h
Un Co Cz
1 2 http://dx.doi.org/10.1787/888933082271

Figure2.10 Professionals participating in product or service innovation, by sector and country

Percentage of graduates who play a role in introducing product or service innovation, 2005 or 2008

%
100
80
60
40
20
0
ia ry y l e nd Italy nds onia lic ania rkey
d d
lan lan ustr nga an tuga ranc ain nia um an om ay
ub
Po tzer erm Por F Sp love elgi y me ingd Norw Finla erl
a
Es
t
ep ithu Tu
i A Hu G S B t r K t h R L
Sw un ed Ne ec
h
Co Unit C z
1 2 http://dx.doi.org/10.1787/888933082290
Figure2.11 Professionals participating in technology, tools or instruments innovation,

Percentage of graduates who play a role in introducing technology, tools or instruments innovation, 2005 or 2008

%
100
80
60
40
20
0
y a s y a a e y l ly c
lan
d
an tri nd ngar veni rlan rwa
d y n
ea gium uani ranc gdom urke tuga lan
d
Ita ain bli tonia
Po erm Aus erla u l o z e o y m e l h F i n T o r F i n Sp epu Es
S t N tr B Li t P
G
Ne
th H
Sw
i
un dK hR
Co ite ec
Un Cz
1 2 http://dx.doi.org/10.1787/888933082309

Innovation occurrence and types

Employee participation in innovation is more frequent in the higher education sector than in
secondary or primary education (Figure2.12 and Figure2.13). Against 76% for the education sector
in general, 81% of graduates working in the higher education sector had played a role in introducing
at least one type of innovation in 2005. This compares with 74% of those in employment within
secondary education and 73% within primary education. Graduates working in higher education
were nearly 50% more likely to participate in at least one type of innovation than those working
in secondary or primary education. Moreover, the higher education sector outperformed all other
sectors of the economy including manufacturing regarding employee participation in at least
one type of innovation. Higher education stands out as the education sector with most employee
participation in various types of innovation (Figure2.12 and Figure2.13). Against 24% for the
whole education sector, 26 % of graduates in higher education played a role in introducing the
three different types of innovations. This figure was only 20% for secondary and 24% for primary
education. Graduates employed in higher education were 30% more likely to take part in different
types of innovation than those working in secondary education. Higher education was also among
the sectors of the economy with above average (24%) levels of employees participating in different
types of innovation, whereas secondary education ranked below the average in this respect.
Innovation types
Knowledge or methods innovation was the most common type of innovation across all levels
of education in terms of graduate participation in innovation in 2005 (Figure2.12). Employee
participation in product or service innovation was more common than in technology, tools and
instruments innovation.
Higher education outperforms other levels of education in terms of the introduction of all three
innovation types (Figure2.12 and Figure2.14). It showed the greatest share of graduates participating
in innovation in terms of knowledge or methods (76%), product or service (46%) and technology,
tools or instruments (40%) innovation. Higher education also outperformed all other sectors of the
economy regarding employee participation in knowledge or methods innovation and was among the
most innovative sectors in technology, tools or instruments innovation. Compared to secondary and
primary education, the likelihood of participating in introducing knowledge or methods innovation
was almost 50% greater for graduates working in higher education and 40% higher with regard to
In secondary education, employee participation in innovation was least common with regard
to technology, tools or instruments innovation (31%), product or service innovation (39%) and
knowledge or methods innovation (68%). Although primary and secondary education outperformed
the economy average in participation in knowledge or methods innovation, both ranked significantly
below the average for product or service and technology, tools or instruments innovation.

Figure2.12 Professionals participating in innovation, by innovation type and education level

Percentage of graduates working in the education sector who play a role in introducing innovation, 2005 or 2008

90 90
80 80
70 70
60 60
50 50
40 40
30 30
20 20
10 10
0 0

3 3
2 2
1.47***
1.28*** 1.39*** 1.47***
1.14 1.20***
1.02 1.04
1 1
0.94 0.99
0,5 0,5

1 2 http://dx.doi.org/10.1787/888933082328
Notes: ***=odds ratio significant at the 0.01 level; **=odds ratio significant at the 0.05 level; *=odds ratio significant at 0.1 level. Year 2008 for Hungary,
Lithuania, Poland, Slovenia and Turkey, year 2005 for other countries. Spain is excluded.

Figure2.13 Professionals participating in innovation across three types of innovation,

by sector and education level and sectors of the economy

%
90
80
70
60
50
40
30
20
10
0
c n
bli tio nc
e
alt
h on d ts ity ge ale on ss es on n nd ing tion rin
g on
Pustra Fina cti an ran tric era oles ati ine vic ati tio ati
i He tru otelstau Elec Av h nic B us r ser uc u ca ure a min duca a ctu d uc
n s u e d d t E f e
i
Co
n H re W m he e ul nu er
ad
m m Ot ary dary gric Ma gh
Co im on A Hi
Pr c
Se
1 2 http://dx.doi.org/10.1787/888933082347
Figure2.14 Professionals participating in innovation, by innovation type, sector and education level

%
80
70
60
50
40
30
20
10
0
e ale ge s s
nc
c n s n
bli tio nd nt tio cti
on ing lth es cit
y
ice nd in
g ion on
ati
on
ati
on
a Pu stra tel aauranica les era ctur ea sin ectri rv re a min ucat ati
Fin h o t r u Av a H u s e uc uc uc
ini Ho rest mu W n s u f B E l
he
r l t u e d e d E d e d
m Co n
Ot gric
u
ary ary er
ad Co
m Ma nd gh
A co Pr
im Hi
S e
1 2 http://dx.doi.org/10.1787/888933082366

CHAPTER 3
Speed of adoption of innovation

in education and other sectors
A workplace may sometimes be at the forefront of adopting innovations or

new knowledge or methods (described as lead innovation adoption), whether
or not it is highly innovative. Analysis of lead innovation adoption can identify
the extent to which such activities are observed in the education sector within
more innovative organisations. The education sector can also be compared
with other sectors of the economy such as manufacturing or other public
services and across primary, secondary and higher education.

3. SPEED OF ADOPTION OF INNOVATION IN EDUCATION AND OTHER SECTORS
Level of Innovation within the education sector
General findings
Lead innovation adoption occurs when organisations are at the forefront in terms of adopting
innovations or new knowledge or methods. Instances of lead innovation adoption in education are
widespread in Europe and Japan (Figure3.1 and Figure3.2). On average, 73% of graduates working
in the education sector in 2005 indicated that their organisation was a lead adopter; higher than
the percentage describing their organisation as highly innovative (described in Chapter 1). This was
the case for more than half of graduates in all 20 countries for which data was available. Combining
lead adoption with reports on highly innovative workplaces, on average across the education sector,
56% of graduates were working in a highly innovative workplace that was at least sometimes at the
forefront of innovation adoption (Figure3.2). This was the case for at least half of the graduates in
most of the 19 European countries with available data.
Graduates within the education sector were less likely to report that their workplace was mostly
(as opposed to sometimes) at the forefront of innovation adoption (Figure3.3 and Figure3.4). In 2005
only 37% of graduates in education sectors in Europe and Japan saw their organisations as being
mostly at the forefront. The percentage of graduates that felt this was the case was below 50% in all
countries except Switzerland and Finland. Figure3.4 shows that around a third of graduates in the
education sector in Europe (32%) considered their organisation to be highly innovative and mostly at
the forefront when adopting innovations. This figure exceeded a half only in Switzerland.
Switzerland, Finland and Austria stand out as countries with above average levels of lead
innovation adoption in education. In 2005, the share of graduates employed in education that
considered their workplace to be at least sometimes at the forefront of innovation adoption was
over 80% in Switzerland (84%) and Austria (82%). In the case of graduates who also reported that
they worked in highly innovative workplaces, the proportion exceeded 65% for both countries.
When looking at lead adoption in all workplaces, at least half of the graduates considered their
organisation to be mostly at the forefront of innovation adoption in Switzerland (57%) and Finland
(51%). For these countries and Austria, these figures exceeded 40% of graduates employed in highly
innovative workplaces in the education sector.
On the contrary, Poland, France, Japan and Hungary are countries with relatively little lead
innovation adoption in education. Below average levels of graduates working in the education
sector considered their organisation to be at least sometimes at the forefront of innovation adoption
in France (60%), Japan (65%), Poland (67%) and Hungary (68%). When looking at highly innovative
workplaces in particular, the figures for the three European countries reached a maximum of 50%.
The percentage of employees considering their workplaces in education to be mostly at the forefront
of adopting innovations, new knowledge or new methods was low in Poland (17%), France (18%),
Portugal (22%), Hungary (27%) and Japan (28%). France (14%), Poland (16%) and Portugal (17%) ranked
also particularly low in terms of highly innovative organisations being mostly at the forefront.

Figure3.1 Education professionals in workplaces at least sometimes quick

to adopt innovation, by country
Percentage of graduates working in the education sector in workplaces at least sometimes
at the forefront of adopting innovation, 2005 or 2008
%
100
82.3 83.7
80 76.1 77.3 77.5 78.6 79.1 79.2
73.0 74.1 74.6 74.8 75.1
68.0 70.9
64.1 64.9 65.2 67.1 67.1
60.0
60
40
20
0
** nia * l * * ** n n y ly s ** ** d ** ** nia ** ** **
e* n* ga y * nd ry Spai mea man gdom Ita land ium ia y* c* a* d*
ra nc thua Japa Portu urke Pola nga y e r i n e r l g v en nlan rwa Esto ubli stri rlan
F Li T Hu tr G dK th Be Slo Fi No ep Au itze
un ite Ne hR
Co Un z ec Sw
C
1 2 http://dx.doi.org/10.1787/888933082385
Figure3.2 Education professionals in highly innovative workplaces at least sometimes quick

Percentage of graduates working in the education sector in highly innovative workplaces at least sometimes at the forefront of adoting
innovation, 2005 or 2008
%
80
70 67.0 69.5
63.7 65.4
59.4 60.3 61.5 62.3
60 56.3 57.5 58.4 59.1
54.0 55.9
47.4 49.6 50.0
50 45.2
41.6 42.4
40
30
20
10
0
** y ** ** * * * * * ** ** ** **
e* r al ey nd nia pain Italy ean any gium tonia rway lic * nds nd * m a* a* d*
ra nc nga rtug Turk Pola thua S y m erm
B el E s N o p ub erla inla g do veni ustri rlan
i r
F Hu Po L
un
t G e
h R Net
h F in Slo A itz
e
Co ec dK Sw
Cz n ite
U
1 2 http://dx.doi.org/10.1787/888933082404

REFLEX (2005) and HEGESCO (2008) surveys asked higher education graduates five years after their graduation
Is your organization normally at the forefront when it comes to adopting innovations, new knowledge or
new methods, or is it more a follower?. In the scale from 1 (mainly at the forefront) to 5 (mainly a follower),
mostly at the forefront corresponds to values 1 and 2 and sometimes at the forefront to values 1, 2 and 3. The
graduates were also asked How would you characterize the extent of innovation in your organization or your
work? regarding three types of innovation separately. High innovative organisations correspond to values 4
and 5 in the scale from 1 (very low) to 5 (very high) regarding at least one type of innovation. The education
sector includes primary, secondary and higher education as well as other non-specified education activities.

Figure3.3 Education professionals in workplaces mostly quick to adopt innovation, by country

Percentage of graduates working in the education sector in workplaces mostly at the forefront of adopting innovation, 2005 or 2008
%
100
80
60 56.5
49.5 50.7
41.9 42.0 42.1 42.2 43.3
40 35.9 36.8 37.1 37.5 37.7 38.6 38.9
27.3 28.1 29.9
22.0
20 17.3 17.6
0
** ** *** y *** n *** nia ain ean nds any taly key way m *** ** *** ** *** d *** d ***
d * ce * al r a Sp y m erla erm I r do blic enia um onia tria
o lan ran rtug nga Ja pa ithu r Tu Nor ing u v l g i s t u s lan rlan
P F Po Hu L t
un Ne
t h G K
ed h R
ep Sl o
Be E A Fin itze
C o i t c Sw
Un Cze
1 2 http://dx.doi.org/10.1787/888933082423
Figure3.4 Education professionals in highly innovative workplaces mostly quick

Percentage of graduates working in the education sector in highly innovative workplaces mostly at the forefront
of adopting innovation, 2005 or 2008
%
80
70
60
52.1
50 44.9
43.6
40 35.5 35.9 36.6 38.2 38.4
32.2 33.0 33.9 34.1 34.2
27.1 27.2 29.8
30
20.8
20 13.8 16.1 16.7
10
0
** ** ** ** ia * ly n y ds ay in * ** * ** * *** d *** d ***
e* d * gal * ary * uan rkey Ita mea man rlan orw Spa gium blic tonia enia dom tria
nc lan n n
Fra Po o rtu ung Lith Tu t ry G er the N B el epu E s l o v i ng Aus Finla zerla
P H un Ne ch
R S dK it
Co e i t e S w
Cz Un
1 2 http://dx.doi.org/10.1787/888933082442

Lead innovation within the education sector compared with other sectors
General findings
The education sector stands at the average regarding the spread of lead innovation adoption
when compared to other sectors of the economy in Europe and Japan (Figure3.5 and Figure3.6).
Similar to the 74% average across economies in 2005, 73% of graduates working in education
reported that their workplace was at least sometimes at the forefront in adopting innovations, new
knowledge or methods. In line with the average of the economy (57%), a little more than half of
graduates worked in organisations that were both highly innovative and exhibited lead innovation
adoption (57%). Education ranked significantly below five other sectors including business activities
(80%) or manufacturing in terms of lead innovation adoption (79%), but above public administration
(58%). Amongst graduates who reported that they were in highly innovative workplaces that were
also at the forefront of adoption, education ranked below five other sectors, but above both public
administration (39%) and health (53%). Both manufacturing and business activities outperformed
education in terms of lead innovation adoption in eight individual countries in general, but
education ranked above health in six countries. In the case of highly innovative workplaces with
lead innovation adoption, manufacturing outperformed education in most countries and business
activities in nine, whereas health ranked significantly below education in five countries. Only in
Slovenia did education rank above manufacturing for all workplaces and those also described
as highly innovative workplaces. Health outperformed education in France and Estonia in all
workplaces, while it outperformed the Czech Republic in highly innovative workplaces.
Conversely, the education sector falls below the average in terms of being mostly at the forefront of
innovation (Figure3.7 and Figure3.8). In 2005, 37% of graduates working in education considered their
organisation as being mostly at the forefront of innovation adoption in Europe and Japan compared
with 41% for the average sector of the economy. This was the case for 33% of graduates in education
who also considered their workplace to be highly innovative, below the 36% economy average in Europe.
Ranking below six other sectors, education outperformed only public administration (27%) in general
and both public administration (21%) and health (30%) regarding highly innovative workplaces at the
forefront of innovation adoption. Both in general and in the case of highly innovative organisations,
manufacturing and business activities outperformed education in most countries. Education
outperformed health in five countries in general, the opposite being true for three countries.
In comparison to other sectors of the economy, education in Slovenia and Switzerland does
relatively well in lead innovation adoption (Figure3.9 to Figure3.12). In 2008 and 2005, the Slovenian
and Swiss education sectors significantly outperformed the health sectors with workplaces
sometimes or mostly at the forefront of innovation adoption in general and highly innovative
workplaces being at the forefront. In both countries, education stood at least at the similar level
with other sectors of the economy in being at least sometimes, but also mostly, in forefront of
adopting innovations, new knowledge or methods.
The Czech Republic and, to a lesser extent, France and Hungary compare less well to other
sectors in terms of lead innovation. In 2005, the Czech education sector ranked below all other
sectors of the economy in the share of graduates being employed by all or only highly innovative
organisations that were mostly at the forefront of innovation. This was also the case for workplaces
that were mostly at the forefront of innovation adoption in the Czech Republic. The economy as a
whole, manufacturing and business activities significantly outperformed education in both France
and Hungary regarding graduates being employed in highly innovative workplaces that were mostly
at the forefront of adopting innovations.

Figure3.5 Professionals in workplaces at least sometimes quick to adopt innovation, by sector

Percentage of graduates in workplaces at least sometimes at the forefront of adopting innovation, 2005 or 2008
%
100
77.2 78.7 79.0 79.1 79.9

80 72.3 72.8 73.3 73.4 74.1 74.7 74.9
70.3
60 58.3
40
20
0
c ** *** ** re *** **
bli *** d nts on ice
s
alt
h
tio
n ge ale ici
ty
ce g* ltu g s*
Pu tion el anura cti rv He ca era les ctr an on rin ricu inin ivitie
ra t sta tru se u Av ho l e i n ati t u
s t o
H re n s
he
r Ed W E F n i c
fa c g
A d m ac
t
ini Co Ot mu nu an ss
m
ad Co
m Ma s ine
Bu
1 2 http://dx.doi.org/10.1787/888933082461
Figure3.6 Professionals in highly innovative workplaces at least sometimes quick

to adopt innovation, by sector
Percentage of graduates in highly innovative workplaces at least sometimes at the forefront of adopting innovation, 2005 or 2008
%
100
80
64.3 64.4 67.7
63.8 63.9
60 55.1 56.8 56.9 57.7
50.9 51.2 52.3 52.9
40 39.2
20
0
c ** re *** ** ** *** **
bli *** on d ts
an an ici
ty ice
s
tio
n
rag
e
sa
le
ltu g ce n* g*
Pu tion ructi el taur ctr ea
lth
erv ca ve ole cu inin an tio iti
es rin
ra s t o t l e H s d u A h r i i n c a iv t u
st n H res E he
r E W Ag d m F ni ac
t ac
ini Co Ot an mu ss uf
m m ne an
ad C o s i M
Bu
1 2 http://dx.doi.org/10.1787/888933082480

Figure3.7 Professionals in workplaces mostly quick to adopt innovation, by sector

Percentage of graduates in workplaces mostly at the forefront of adopting innovation, 2005 or 2008
%
100
80
60
50.8 51.2
46.9 47.0 47.1
41.4 41.6 42.7
40 35.5 36.4 36.5 37.5 37.8
27.5
20
0
c ** *** *** re *** *** ***
bli *** on d ts alt
h on ice
s
e* ici
ty ***
Pu tion ructi an ran ati ale ce es cultuing ing on
ra s t o telstau He
d uc s erv e rag l e ctr l es n an iv iti r i in t ur c ati
s t n H re E r v E o Fi t g m c i
ini Co he A h ac A d fa n
m Ot W ss an nu mu
ad s ine Ma C om
Bu
1 2 http://dx.doi.org/10.1787/888933082499
Figure3.8 Professionals in highly innovative workplaces mostly quick to adopt innovation, by sector
Percentage of graduates in highly innovative workplaces mostly at the forefront of adopting innovation, 2005 or 2008
%
100
80
60
46.8 47.1
41.9 42.5 43.2
40 33.5 35.9 37.0
29.5 29.5 30.0 31.6 32.8
21.4
20
0
c ** re *** ** **
bli *** d nts on h* ice
s on ici
ty e* *** *** *** n* g*
Pu tion el anura cti alt rv ati ale ce es cultuing
ra o t sta s tru He se d uc l e ctr e rag l es n an iv iti r i in c a tio
t u rin
t r v o i t g
A dm i
ini
s H re
Co
n he E E A h F ac n fac
m Ot W ss an mu nu
ad s ine C om Ma
Bu
1 2 http://dx.doi.org/10.1787/888933082518

Figure3.9 Professionals in workplaces at least sometimes quick to adopt innovation,

Percentage of graduates in workplaces at least sometimes at the forefront of adopting innovation, 2005 or 2008

%
100 100
80 80
60 60
40 40
20 20
0 0
e a l y d ary ain an any y s a y a c a
n
nc ani apa tuga rke lan om Ital and gium eni land rwa toni ubli stri lan
d
Fra ithu J or Tu Po Hun
g Sp y me erm ingd erl Bel lov Fin No Es ep Au itzer
L P t r G K t h S R
un ite
d Ne ec
h Sw
Co Un Cz
1 2 http://dx.doi.org/10.1787/888933082537
Figure3.10 Professionals in workplaces mostly quick to adopt innovation, by sector and country
Percentage of graduates in workplaces mostly at the forefront of adopting innovation, 2005 or 2008

%
100 100
80 80
60 60
40 40
20 20
0 0
d ce al ry pan nia ain ean nds any taly key way om bli
c
nia ium nia tri
a d d
lan ran rtug nga Ja hua Sp y m erla erm I ur an
or ingd epu love elg Esto Aus Finl zerl
an
Po F Po u i t r T N S B t
H L t
un Ne
th G dK hR Sw
i
Co ite Czec
Un
1 2 http://dx.doi.org/10.1787/888933082556

Figure3.11 Professionals in highly innovative workplaces at least sometimes quick

to adopt innovation, by sector and country
Percentage of graduates in highly innovative workplaces at least sometimes at the forefront of adopting innovation, 2005 or 2008

%
100 100
80 80
60 60
40 40
20 20
0 0
e
nc gar
y al y d ia in taly ean any ium ia ay lic ds nd m ia ria land
ug urke olan uan Spa I ton orw epub rlan inla ngdo oven ust
Fra Hun Port T P h y m erm elg Es e F l A er
L i t t r G B N R t h K i S itz
o un e ch Ne t ed Sw
C z i
C Un
1 2 http://dx.doi.org/10.1787/888933082575
Figure3.12 Professionals in highly innovative workplaces mostly quick to adopt innovation,

Percentage of graduates in highly innovative workplaces mostly at the forefront of adopting innovation, 2005 or 2008

%
80 80
70 70
60 60
50 50
40 40
30 30
20 20
10 10
0 0
e l s c
nc lan
d ga ary nia key Italy ean any and rway pain gium ubli tonia enia dom stria land land
Fra Po rtu ung thua Tur m erm erl No
S l ep Es v g Au Fin itze
r
P o H Li t ry G t h Be
h R Slo Kin
u n N e e c e d S w
Co Cz it
Un
1 2 http://dx.doi.org/10.1787/888933082594


Lead innovation adoption is more widespread in higher education than in secondary or primary
education (Figure3.13 and Figure3.12). Against 73% for the education sector in general, 80% of
graduates working in the higher education sector were employed in organisations at least sometimes
at the forefront in adopting innovations in 2005. The relevant figure for the secondary education
sector was 67% and 70% for primary education. As to workplaces considered to be highly innovative
and at the forefront of innovation adoption, higher education (69%) outperformed all other sectors of
education and the economy except the manufacturing sector (68%). In contrast, both primary (53%)
and secondary education (49%) ranked slightly below the economy average (57%). The likelihood of
graduates working in an organisation at the forefront of adopting innovation was almost twice as
high for those working in higher education as in secondary or primary education and more than
twice as high in the case of highly innovative organisations.
Higher education stands out with regard to workplaces in the education sector that are mostly
at the forefront of adopting innovation (Figure3.13 and Figure3.14). Above the 36% for the whole
education sector, 46% of graduates working in higher education reported that their workplace was
mostly at the forefront in adopting innovations, new knowledge or methods in 2005. In comparison,
just 30% of those in employment within secondary education and 31% in primary education reported
that their organisations were mostly at the forefront. In comparison with other sectors of the
economy, higher education also outperformed the average (41%) whilst the opposite was true for
both secondary and primary education. The picture was similar when looking at the proportion of
graduates reporting that their workplace was both highly innovative and mostly at the forefront of
adopting innovation. Higher education (43%) significantly outperformed both the education sector
in general (32%) and the economy average (36%); it also compared favourably with the 25% for both
secondary and primary education. Graduates employed in the higher education sector were nearly
twice as likely to work in an organisation that was mostly at the forefront of adopting innovation
compared with those employed in secondary or primary education. In the case of highly innovative
workplaces, these odds were more than double.

Figure3.13 Professionals in workplaces quick to adopt innovation, by sector and education level
Percentage of graduates in workplaces at the forefront of adopting innovation, 2005 or 2008
At least sometimes at the forefront Mostly at the forefront

%
90
80
70
60
50
40
30
20
10
0
c n ion s n n s h on e le ty ce n g re g s on
bli tio t nd nt tio tio rvice alt ati rag sa ici an tio turin ultu inin ines ati
Pustra uca tel aaura uca truc s e He uc Ave ole lectr Fin n ica c r ic d m Bus uc
n i e d o s t d s r d h E u f a g e d
i H re ry e n he E W m nu A an er
m
ad nda
ry a Co Ot m Ma gh
o rim Co Hi
c P
Se
Innovation likelihood compared with secondary education
At least sometimes Mostly at the At least sometimes Mostly at the

% at the forefront forefront at the forefront forefront
100 odds ratio
3
80
60 1.90*** 1.85***
2
40
1.07
20 1
0.98
0
Primary Secondary Higher Total
0,5
education education education education Primary education Higher education
1 2 http://dx.doi.org/10.1787/888933082613

Figure3.14 Professionals in highly innovative workplaces quick to adopt innovation,

Percentage of graduates in highly innovative workplaces at the forefront of adopting innovation, 2005 or 2008
At least sometimes at the forefront Mostly at the forefront

%
80
70
60
50
40
30
20
10
0
c n ion
bli tio on nd ants icity on h s n e le ture ing nce n es
s g n
t cti ati alt ice tio rag sa l n tio rin tio
Pustra uca t
a
ru otel taur lect
r
uc He s erv duca Ave hole ricu mi Fina nica u sin ctu duca
i d s d d B f a
in y e n H res E e he
r E W Ag an mu nu er
e
m
ad nda
r Co ary Ot m Ma gh
o rim Co Hi
c P
Se
Innovation likelihood compared with secondary education
At least sometimes Mostly at the At least sometimes Mostly at the

% at the forefront forefront at the forefront forefront
100 odds ratio
3
80
2.19*** 2.20***
60
2
40
1.10
20 1
0.94
0
Primary Secondary Higher Total
0,5
education education education education Primary education Higher education
1 2 http://dx.doi.org/10.1787/888933082632
:
Notes: ***=odds ratio significant at the 0.01 level; **=odds ratio significant at the 0.05 level; *=odds ratio significant at 0.1 level. Spain is excluded. Year
2008 for Hungary, Lithuania, Poland, Slovenia and Turkey, year 2005 for other countries.

CHAPTER 4
Highly innovative jobs
It is possible to measure the percentage of recent graduates who have

highly innovative jobs - that is jobs in highly innovative workplaces where
they themselves participate in introducing the innovation - by deriving a new
variable from their responses to the two questions reported in Chapters 1 and
2. Such highly innovative jobs may combine various kinds of innovation or
focus on a specific type of innovation.
The education sector can be compared with other sectors of the economy
such as manufacturing or other public services to see which have the largest
proportion of highly innovative jobs. Exploring differences between innovative
jobs across levels of education is also important, highly innovative jobs may
not be equally distributed amongst primary, secondary and higher education.

4. HIGHLY INNOVATIVE JOBS
General findings
Highly innovative jobs are not uncommon in Europe (Figure4.1). On average, 58% of graduates
working in the education sector in 2005 considered that they had a highly innovative job they
reported that they worked in highly innovative organisations regarding at least one type of innovation
and that they themselves also played a role in introducing those kinds of innovations. This was the
case for at least half of the graduates in a large majority of the 19 European countries for which data
was available, with the exception of France, Portugal and Hungary.
However, innovative jobs in education do not necessarily incorporate each of the three forms
of innovation (Figure4.2). Of graduates working in education in an average European country just
9% reported that their organisation was highly innovative concerning three types of innovation and
that they also participated in introducing those innovations. In no country did this figure exceed
20% of graduates.
Highly innovative jobs in education are most likely to cover knowledge or methods innovation,
while other types of innovation are less common (Figure4.3). In 2005, 47% of graduates played a role
in introducing knowledge or methods innovations in their highly innovative organisations across
the countries studied. For technology, tools or instruments innovation this was the case for only 21%
of graduates and regarding product or service innovation, 25% of graduates. Knowledge or method
innovation was the most common type of innovation in all countries. In eight countries, product or
service innovation was more common than technology, tools or instruments innovation.
The United Kingdom and Finland stand out as countries with more highly innovative jobs than
average across different types of innovation in the education sector. In 2005, the United Kingdom had
the largest share of graduates (17%) considering their jobs to be highly innovative in all three types
of innovation. Innovative jobs in education concerning product or service as well as technology,
tools or instruments innovation were significantly more common in the United Kingdom than on
average. Finland was among the countries with above average levels of innovative jobs concerning
at least one type of innovation (64%) as well as knowledge or methods and product or service
innovation specifically.
In contrast, Hungary, France and Poland have few highly innovative jobs in the education sector.
The share of graduates participating in the introduction of three types of innovation in highly
innovative organisations was significantly below the European average in Hungary (3%), France (5%)
and Poland (6%). Regarding at least one type of innovation, France (43%) and Hungary (48%) were
among countries with the lowest share of graduates in innovative jobs. Hungary ranked below the
European average for all three types of innovation, whilst France was below average for knowledge
or methods and product or service innovation and Poland was below average for product or service
and technology, tools or instruments.
The Dutch and Spanish cases illustrate that jobs in the education sector may not be highly
innovative across all types of innovation. In 2005, the Netherlands had the largest share of highly
innovative jobs covering at least one type of innovation (65%) with above average levels for both
knowledge or methods and product or service innovation. Yet the level of highly innovative jobs in
technology, tools or instruments innovation in the Dutch education sector was below the European
average. In Spain, highly innovative jobs in education were below average when considering at least
one type of innovation (52%) and knowledge or methods innovation. However, the Spanish education
sector was among the most innovative when looking at technology, tools or instruments innovation.

Figure4.1 Education professionals in highly innovative jobs regarding at least one type of innovation,
by country
Percentage of graduates working in the education sector in highly innovative workplaces regarding at least one type of innovation
and playing a role in introducing innovation, 2005 or 2008
%
70 65.3
62.3 63.6 64.0 64.0 64.4
58.7 59.3 60.2 60.4 60.5 60.7 61.1
60 58.1 58.2
52.2 54.2
50 46.4 47.8
43.4
40
30
20
10
0
** l *** ** ** d n ia y y ic ly ay m nd a* ** *** * ***
e* ry in olan lgium mea ustr man urke ubl Ita rw gdo erla toni land nia nia nds
a nc tuga nga Spa P e A r T ep No Kin e ua
r r u B t r y G e R i tz Es i n v t h r l a
F Po H un ec
h d Sw
F Slo Li th
e
Co Cz ite Ne
Un
1 2 http://dx.doi.org/10.1787/888933082651
Figure4.2 Education professionals in highly innovative jobs in education

across three types of innovation, by country
Percentage of graduates working in the education sector in highly innovative workplaces across three types of innovation
%
70
60
50
40
30
20 17.1
12.0 13.1 14.4
9.4 10.4 10.7 11.2 11.4 11.4 11.6
10 5.9 6.1 7.5 8.3 8.4
3.0 4.8 5.4 5.4
0
*** d *** y *** e *** y ** nd * way nds tria ean ium c l * ***
ain bli nia nia nd ga nia ey
ry
ga rlan
l c n
Ita ran rma Pol
a or erl
a s
Au try m Bel
g Sp epu love hua Finla ortu Esto urk do
m
n e F N t h R S L i t P T g
Hu itz Ge Ne un ec
h in
Sw Co Cz dK
n ite
U
1 2 http://dx.doi.org/10.1787/888933082670

REFLEX (2005) and HEGESCO (2008) surveys asked higher education graduates five years after their graduation
How would you characterize the extent of innovation in your organization or your work? regarding product
or service, technology, tools or instruments and knowledge or methods innovation. High innovation
intensity corresponds to values 5 and 4 in the scale from 1 (very low) to 5 (very high). The graduates were
also asked Do you play a role in introducing these innovations in your organisation? regarding the three
innovation types. The education sector includes primary, secondary and higher education as well as other
non-specified education activities.

Figure4.3 Education professionals in highly innovative jobs in education,

by innovation type and country
Percentage of graduates working in the education sector in highly innovative workplaces across three types of innovation
70
60 57.2 57.7 58.1
52.4 52.8 52.8 54.8
50.4 51.7
50 47.5 48.4 48.8
44.9 45.3 46.0
41.7
40 35.4 37.5
32.3 33.8
30
20
10
0
** ** ** ** ** y ia lic ean ium and and m ny ria * ay * d *** *** *** ***
e* y * aly * gal ain urke ton ub l l do rma ds enia ania
ra nc gar It ortu S p T E s ep try m Bel
g Po tzer i ng e
t
us Norw nlan rlan v
n R i K G A i u
F Hu P h
ec Cou
n Sw ited F
th
e Slo Lith
Cz Un Ne
70
60
50
40 36.0 37.2
30.2 31.6
30 27.2 27.3 27.8 28.9
24.7 24.7 25.3 25.9 26.1
21.3 23.1
20 19.4
13.4 14.1 14.4 15.6
10
0
*** *** e *** d *** gal any ain an tria way ly ia ia lic ia m * *** *** y*
*
nd gary nc tu Sp y me Aus r Ita ton huan epub oven elgiu gdom land nds ke
l a lan Por Germ No Es l r
Po Hun Fra tzer t r t
Li ch R S B in Fin her
l a T u
i u n e dK t
Sw Co Cz n ite Ne
U

70
60
50
40
33.2
30 26.5 28.9
24.9 26.2
20.5 20.7 22.6 22.7 23.0
18.2 18.2 19.2 20.3 20.5
20 14.5 16.6
12.2 14.2
10.7
10
0
** ** ** ** ay ria ia ce m al nd ean ania ly d ey *** c *** *** ***
d * ny * s * ary rw ust oven Fran elgiu rtug erla Ita lan urk ain nia
lan a nd g No m hu Fin T bli om
Po erm erla Hun
A Sl B Po i tz try Lit Sp epu ingd Esto
G e th Sw oun h R ed K
N C ec t
Cz Uni
1 2 http://dx.doi.org/10.1787/888933082689

General findings
The education sector is amongst the economic sectors where highly innovative jobs are relatively
common (Figure4.4 and Figure4.5). Compared with a 55% average for the whole economy in 2005
(excluding Portugal), 59% of graduates working in the education sector had highly innovative jobs.
In six individual countries, the level of highly innovative jobs in education was significantly above
that of the economy as a whole. Only the manufacturing sector (64%) had significantly more highly
innovative jobs than the education sector across the countries analysed.
The education sector stands at the average level in comparison with other sectors in Europe
for jobs that are highly innovative regarding three types of innovation (Figure4.6 and Figure4.7).
Both across the economy and in the education sector, 9% of graduates had highly innovative jobs
concerning three different types of innovation at the same time. Such highly innovative jobs were
the most common in manufacturing, communication and business activities (13% each). The
education sector ranked significantly lower than manufacturing in eight countries and lower than
business services in nine countries.
Highly innovative jobs are more common in the education sector than in other public services
(Figure4.4 to Figure4.7). In 2005, fewer graduates were employed in highly innovative jobs regarding
at least one type of innovation in public administration (40%) and health (50%) than in education
(59%). In this respect, the health sector was significantly behind the education sector on average and
in seven European countries the opposite was true only in the Czech Republic. When looking at
the occurrence of three types of innovation, the education sector (10%) clearly ranked above public
administration (5%) and health (7%) sectors. This was confirmed for the health sector in Finland and
Belgium, whereas the opposite was the case for the Czech Republic.
Lithuania and, to a lesser extent, the Netherlands and Estonia stand out as countries where
highly innovative jobs in education are common when compared to other sectors of the economy.
The education sector outperformed most sectors of the economy in Lithuania in 2008 and in the
Netherlands in 2005 when considering graduates working in highly innovative jobs regarding at least
one type of innovation. Unlike in most European countries, highly innovative jobs were significantly
more common in education than in manufacturing in Lithuania (by 17% points) and Estonia (by
9% points) however the level of highly innovative jobs in manufacturing was below the European
average in both countries.
On the contrary, relative to other sectors and countries, in the Czech and Italian education
sectors few graduates are in highly innovative jobs. As to employee participation in at least one type
of innovation in highly innovative workplaces, most sectors of the economy outperformed education
in the Czech Republic in 2008. Unlike the other countries analysed, the Czech education sector
ranked below the health sector regarding both occurrence of innovation (by 5% points) and across
three types of innovation (by 3% points). Italy stood out as the only country where the education
sector ranked significantly below the economy as a whole across three types of innovation (3%
points) as well as below most other sectors in 2005.

Figure4.4 Professionals in highly innovative jobs, by sector

Percentage of graduates working in highly innovative workplaces regarding at least one type of innovation
and playing a role in introducing those innovations, 2005 or 2008
%
70
64.4
59.0 59.3 59.5 60.5
60 54.4 54.9 55.4 55.6
52.0 53.3
49.7 50.4
50
40 39.5
30
20
10
0
c ** ** *** ** * on **
bli *** n* h* ici
ty d ts
an an nce e* s* ale ati
re g
ltu inin catio
n
iti
es g*
Pu tion tio alt el taur rag ice les uc tiv rin
ra u c e l e ctr o t s n a ve e rv o E d r icud m n i
a c t u
st H Fi s g
ini ns
tr E H re A
he
r W
h A an mu es
s fac
m Co Ot Co
m
sin nu
ad Bu Ma
1 2 http://dx.doi.org/10.1787/888933082708
Figure4.5 Professionals in highly innovative jobs, by sector and country

Percentage of graduates working in highly innovative workplaces regarding at least one type of innovation

%
80
70
60
50
40
30
20
10
0
e l ry in and ium ean ia ny rkey c ly ay nd onia land enia ania ds
nc uga ga Spa l str rma bli Ita
m
rw gdo erla lan
Fra Port Hun Po Belg ry m Au e Tu Repu No Kin tz Es
t v
Fin Slo Lithu her
t G i t
un ec
h
ite
d Sw Ne
Co Cz Un
1 2 http://dx.doi.org/10.1787/888933082727

Figure4.6 Professionals in highly innovative jobs across three types of innovation,

by sector
Percentage of graduates working in highly innovative workplaces regarding three types of innovation
%
70
60
50
40
30
20
11.5 12.8 13.2 13.4
10 7.1 7.5 7.6 7.9 9.0 9.1 9.5 9.5 9.5
5.1
0
c
bli *** d ts ** s* e on ty e n le re g *** ** **
Pu tion el anuran ealth ice nc cti ici rag tio sa ltu inin es on g*
ra t t a H e rv Fina t ru e ctr Ave u ca
h ole gricud m iv iti c ati t u rin
st o
H res rs s l d A an t i
ini he Co
n E E W ac mu
n fac
m Ot ss nu
ad s ine C om Ma
Bu
1 2 http://dx.doi.org/10.1787/888933082746

Percentage of graduates working in highly innovative workplaces regarding three types of innovation

%
80 80
70 70
60 60
50 50
40 40
30 30
20 20
10 10
0 0
ce s c l
ary nd Ita
ly n
y d y
an olan rwa land stri
a n
ea gium Spai
n
bli nia nia and tuga tonia rkey dom
ng erla Fra rm P o r u m e l pu love thua Finl r Es Tu ng
Hu itz G e N th
e A tr y B
hR
e S Li Po Ki
Sw Ne un ec t ed
Co C z
Un
i
1 2 http://dx.doi.org/10.1787/888933082765

General findings
Education outperforms other sectors of the economy in terms of highly innovative jobs in
knowledge or methods innovation the most common innovation type in most sectors of the
economy in Europe (Figure4.8 and Figure4.9 ). Against the average of 37% and more than in any other
sector, 48% of graduates in the education sector held highly innovative jobs in knowledge or methods
innovation in 2005. The education sector surpassed all other sectors in knowledge or methods
innovation in five countries. In no country had education the smallest share of highly innovative
jobs. As in education, knowledge or methods innovation was the most common innovation type for
highly innovative jobs in on average across the economy and in seven other specific sectors.
As to product or service innovation, the education sector employs a smaller proportion

of graduates in highly innovative jobs than the rest of the economy in Europe (Figure4.8 and
Figure4.10). While 25% of graduates in education worked in highly innovative jobs in terms of product
or service in 2005, this was below the average (29%) and the manufacturing sector (37%). Education
outperformed manufacturing only in Lithuania. Alongside health, education was among the sectors
with the smallest share of graduates in highly innovative jobs in six countries. Whereas product or
service innovation was the second most common innovation type for education, it held a similar
standing with knowledge or methods innovation in manufacturing, communication, wholesale and
hotels and restaurants.
Highly innovative jobs in technology, tools or instruments innovation in education are at the
average level when compared with other sectors of the economy despite the fact that this is the
least common type of innovation for education (Figure4.8 and Figure4.11). Both in the education
sector and on average, 21% of graduates worked in highly innovative jobs regarding technology, tools
or instruments innovation in 2005. Education ranked below the leading sector of manufacturing
(29%), for example, but significantly above both public administration (13%) and health (16%).
Manufacturing outperformed education in eight countries the situation was reversed only in
Estonia. In contrast, the health sector ranked lower than the education sector in five countries. Only
in Slovenia, Poland and the Netherlands was education together with health among the sectors
with the smallest share of highly innovative jobs in technology, tools or instruments innovation.
Similarly to education, technology, tools or instruments innovation was the least common type of
innovation overall and in nine other sectors.
With regard to highly innovative jobs, education ranks comparatively well across three types
of innovation in Lithuania. This country was among those where education outperformed all other
sectors of the economy in 2008 regarding knowledge or methods innovation. It was also the only
country where education outperformed manufacturing in product or service innovation (27%
against 12%).
In contrast, Poland has relatively few highly innovative jobs in education. In 2008, the Polish
education sector ranked poorly against other sectors both regarding product or service and
The cases of Germany, Switzerland and the Netherlands illustrate education sectors with
a focus on certain types of innovation. The German, Swiss and Dutch education sectors had the
largest share of highly innovative jobs in terms of knowledge or methods in 2005. Yet education
performed relatively poorly concerning product or service innovation in Germany and Switzerland
and regarding technology, tools or instruments innovation in the Netherlands.

Figure4.8 Professionals in highly innovative jobs, by sector and innovation type

Percentage of graduates working in highly innovative workplaces and playing a role in introducing those innovations, 2005 or 2008

%
60
50
40
30
20
10
0
n d ts on le ce n s e g h y re g ies on
c
bli tio an an cti sa an tio ice rag rin alt cit ultu inin vit ati
Puistra otel taur ru ole Fin ica erv ve ctu He tri cti uc
s s t h u n r s A f a Ele
c
g ric d m a E d
in H re n W m he nu A an s
m Co Ot es
ad Co
m Ma sin
Bu
1 2 http://dx.doi.org/10.1787/888933082784
Figure4.9 Professionals in highly innovative jobs regarding knowledge or methods innovation,


%
70
60
50
40
30
20
10
0
l
e
nc gar
y
Ita
ly ga pain rkey onia ublic ean ium land land dom any stria rway land ands enia ania
Fra Hun rtu S Tu Es
t
ep try m Bel
g Po tzer ng rm Au No Fin her
l v u
Po Ki Ge Slo Lith
hR n i
Sw ited Ne
t
z ec Cou
C Un
1 2 http://dx.doi.org/10.1787/888933082803
:

Figure4.10 Professionals in highly innovative jobs regarding product or service innovation,


%
70
60
50
40
30
20
10
0
d ry e
nc lan
d al y in n ia y ly ia ia lic ia m m nd ds ey
lan ga ug man Spa mea ustr orwa Ita ton huan pub oven elgiu ngdo inla rlan Turk
Po Hun Fra tzer Port er ry A N Es e Sl i F e
i G t Lit ch R B K t h
Sw un e ite
d Ne
Co Cz Un
1 2 http://dx.doi.org/10.1787/888933082822
Figure4.11 Professionals in highly innovative jobs regarding technology, tools

or instruments innovation, by sector and country

%
70
60
50
40
30
20
10
0
d y s
nd gar
y y ia ia e al and ean nia ly and ey ain bli
c
nia
lan an wa ustr ven ranc lgium rtug rl ua Ita l rk om
Sp epu ingd Esto
Po erm erla Hun N or A S l o F B e Po tze try
m
ith Fin Tu R
G t h i L K
Ne Sw oun ec
h
ted
C Cz Uni
1 2 http://dx.doi.org/10.1787/888933082841

Innovation occurrence and type

A larger proportion of graduates have highly innovative jobs in the higher education sector than
in secondary or primary education (Figure4.12 and Figure4.13). Against 58% for the education sector
in general (including Portugal), 68% of graduates working in the higher education sector in Europe
had highly innovative jobs in 2005. This figure was 53% for secondary and 55% for primary education,
while the likelihood of graduates having highly innovative jobs was nearly twice as high for those
working in higher education as in secondary or primary education. Moreover, the higher education
sector outperformed all other sectors of the economy including the manufacturing sector in the
proportion of highly innovative jobs. Both primary and secondary education stood at the average
level when compared to other sectors of the economy.
Higher education stands out as the education sector with the largest share of highly innovative
jobs with regard to three types of innovation, whereas secondary education is at the end of the rank
in this respect (Figure4.12 and Figure4.13). Against the 10% for the whole education sector, 14% of
graduates in higher education worked in innovative organisations regarding three innovation types
and played a role in introducing those innovations. This figure was only 6% for secondary and 9% for
primary education. Graduates employed in higher education were more than twice as likely to hold
highly innovative jobs regarding three innovation types than those working in secondary education.
Higher education was also among the sectors of the economy that had the greatest share of highly
innovative jobs in terms of innovation across the different types, whereas secondary education
ranked below average in this respect.
Innovation types
As within the education sector as a whole, most highly innovative jobs found at different levels
of the education system concern knowledge or methods innovation (Figure4.12). This was the most
common type of innovation for all levels of education in 2005. While product or service innovation
was more common than technology, tools and instruments innovation in primary and secondary
education, the two types held an equal standing in the higher education sector.
Higher education outperforms other levels of education regarding the three different innovation
types (Figure4.12 and Figure4.14). Across all levels of education, higher education employed the
greatest share of graduates to highly innovative jobs in terms of knowledge or methods (59%),
product or service (29%) and technology, tools or instruments (29%) innovation. Higher education
also outperformed all other sectors of the economy regarding knowledge or methods innovation and
was among the most innovative sectors in technology, tools or instruments innovation. Compared
to secondary and primary education, the odds of having an innovative job in higher education were
twice as high for technology, tools or instruments innovation and nearly twice as high for knowledge
or methods innovation. Relative to secondary education, the odds were almost one and a half times
as high for product or service innovation.
It can be seen that within secondary and primary education the proportion of graduates in
innovative jobs across the various types of innovation falls short of that in higher education (Figure4.12
and Figure4.14). Secondary education had the lowest proportion of graduates in highly innovative
jobs with regard to knowledge or methods innovation (42%) and product or service innovation (20%).
In terms of technology, tools or instruments innovation, both primary and secondary education (16%)
employed a low proportion of graduates in highly innovative jobs. Although primary and secondary
education outperformed the economy average in knowledge or methods innovation, both ranked
below the average in product or service and technology, tools or instruments innovation.

Figure4.12 Education professionals in highly innovative jobs, by innovation type and education level
Percentage of graduates employed in education working in highly innovative organisations
and playing a role in introducing innovations, 2005 or 2008

80 80
70 70
60 60
50 50
40 40
30 30
20 20
10 10
0 0

3 3
2.30***
1.86*** 2.01*** 1.91***
2 2
1.35 1.44***
1.17
1.01 1.06
1 1
0.96
0,5 0,5
1 2 http://dx.doi.org/10.1787/888933082860


Percentage of graduates working in highly innovative organisations and playing a role in introducing innovations, 2005 or 2008

%
80
70
60
50
40
30
20
10
0
c n n h ty nd nts tion e e es le n n ure ing ion es g n
bli tio tio alt ici a a nc rag rvic sa tio tio t t iti rin tio
Pustra truc He ctr tel taur duca Fina Ave s e ole duca u ca icul min nica c tiv ctu u ca
i s l e o r h d r d a f a d
in
m Con
E H res ry e he W e E Ag an mu ess nu er
e
ad da Ot ary m in Ma gh
on rim Co us Hi
c P B
Se
1 2 http://dx.doi.org/10.1787/888933082879
Figure4.14 Professionals in highly innovative jobs, by innovation type, sector and education level
Percentage of graduates working in highly innovative organisations and playing a role in introducing innovations, 2005 or 2008

%
70
60
50
40
30
20
10
0
c n s
bli tio nd nt tio
n ale ance on ice
s ge rin
g lth ity re ng ion ss on on on
P stra tel aaura truc
u les n ati rv era tu ea ctric cultu ini cat ine ati ati ati
h o Fi n i c s e Av c H i m u u s uc uc uc
ini Ho rest ons W u
he
r u f a E l e g r d
A an ry e d B e d E d e d
m m
Ot
n ary er
ad C
Co
m Ma nd
a gh
co Pr
im Hi
S e
1 2 http://dx.doi.org/10.1787/888933082898

PART II
Innovation as change
in classrooms and schools
CHAPTER 5
Innovation in teaching style
Innovation in the classroom in terms of teaching style could incorporate

more or less use of front-of-class teaching such as lecturing in class, reading
aloud or demonstrating science experiments. The aim of innovation with
regard to increasing the use of front-of-class teaching could be, for example, to
ensure that basic principles are explained to the whole class, whilst a reduction
may occur as a result of the introduction of a more individualised approach to
classroom teaching.
Innovation could also take the form of more or less time devoted to
independent work. An increase in independent work may reflect a move
towards engendering greater autonomy whilst a decrease could indicate a
change towards more direct teacher guidance.

5. INNOVATION IN TEACHING STYLE
Use of front-of-class teaching
General findings
Innovation in the classroom in secondary school is not widely observed in terms of a change
in the time spent on lecture-style presentations. The absolute change in the percentage of 8th grade
mathematics class-time spent on this style of teaching across the OECD between 2003 and 2007
was 2% points; in science the equivalent absolute change amounted to 3% points (Figure5.1 and
Figure5.2). According to student reports, the absolute change in the time spent on lecture style
presentations in maths across OECD countries was 5% points compared with 3% points in science
(Figure5.2 and Figure5.4). Secondary school students reporting also indicated an absolute change
in demonstrations of experiments across OECD countries of 5% points (Figure5.5). Student reports
show that the incidence of lecture style presentation in maths fell markedly in Japan (34% points)
and increased in the Russian Federation (26% points). These changes had medium to large effect
sizes. Students also reported a reduction in the extent to which teachers demonstrated experiments
in the Basque country (11% points) and an increase in Korea (15% points). These two education
system changes had small effect sizes. In total, with at least small effect sizes, students reported a
reduction in lecture style presentations in maths lessons in three countries and an increase in one.
Innovation in 4th grade classes has meant increases and decreases in the amount of time spent
on lecture-style presentations and reading to the whole class. The average absolute change across
the OECD area between 2003 and 2007 in lecturing during 4th grade maths was 2% points, while the
equivalent absolute change in reading aloud to 4th grade students between 2001 and 2011 was 4%
points (Figure5.5 and Figure5.6). The OECD absolute change in teachers demonstrating experiments
to the class, according to student reports, was 9% points. Only Quebec (9% points) exhibited an
increase in the use of lecture-style presentations for 4th grade mathematics. Slovenia (10% points)
increased the use of front-of-class science demonstrations between 2003 and 2007 according to
students, whilst several countries, including Japan (19% points), the Netherlands (17% points) and
Australia (14% points) reduced the use of this teaching approach. Teacher reports from Hong Kong
(18% points), Quebec (9% points) and the Netherlands (7% points) indicate the greatest increases
in reading aloud between 2001 and 2011, whilst reading aloud reduced in the Slovak Republic (11%
points), the Russian Federation (6% points) and Slovenia (4% points). These education system level
changes recorded small to medium effect sizes. Overall, and with at least small effect sizes, there
was an increase in 4th grade students reporting that teachers demonstrated experiments in one
country and a decrease in five, compared with an increase in 4th grade teachers reading aloud to
their class in three education systems and a decrease in four others.
The Netherlands is an example of a country that reduced the use of front-of-class science
demonstrations between 2003 and 2007 according to 4th grade students, but increased the extent to
which teachers read aloud to 4th grade students between 2001 and 2011.
In contrast in the Russian Federation, there was a marked increase in science demonstrations
at secondary school level between 2003 and 2007, whilst the incidence of reading aloud in primary
school decreased between 2001 and 2011.

Figure5.1 Time spent on lecture-style presentations in 8th grade maths, according to teachers
Percentage of class time and change over time
2003 2007 Negative change Positive change OECD (average absolute change)
% -5 -2 -2 -2 -1 -1 -0 -0 +0 +0 +1 +1 +1 +2 +2 +3 +3 +3 +3 +4
% point
40 6
35
5
30
4
25
20 3
15
2
10
1
5
0 0
*** e* d.** * nd a ly io ia ec *** rael pan alia ntry *** *** ea ** ***
sia or e ary la hin Ita tar ven ueb ean Is Ja ustr es ay Kor land den
o ne ngap an F u ng Scot g, C On Slo Q m A c ou Stat orw n g w e
In
d Si ussi H
Ko
n CD ue ited N E S
OE sq
R ng Ba Un
Ho
1 2 http://dx.doi.org/10.1787/888933082917
Notes: *** = change significant at the 0.01 level; ** = change significant at the 0.05 level; * = change significant at 0.1 level
Source: Authors calculations based on TIMSS (2003 and 2007)
Figure5.2 Time spent on lecture-style presentations in 8th grade maths, according to students
% -34 -9 -9 -5 -5 -5 -4 -3 -3 -3 -2 -2 -0 +0 +0 +1 +2 +4 +6 +26
% point
100 35
90
30
80
70 25
60 20
50
40 15
30 10
20
10 5
0 0
* *
*** ong ea*** io*** ay**
* * *
e** an** and* gary tes* ntr
* y
be
c
Ita
ly nd ralia rael sia** en** ia*** d.***
an K r
tar orw gap
or e tl ta ou ue gla t Is ne wed oven n Fe
Jap ong Ko n m c o u n
H ted S e c Q En Aus do S ia
H O N
Sin OEC
D S i qu In Sl ss
Un Bas Ru
1 2 http://dx.doi.org/10.1787/888933082936

Figure5.3 Time spent on lecture-style presentations in 8th grade science, according to teachers
% -5 -4 -3 -3 -3 -3 -2 -2 -1 -1 +0 +1 +1 +2 +3 +4 +6 +6 +7 +7
% point
50 10
40 8
30 6
20 4
10 2
0 0
n ** * l io sia* aly* ia re es alia n*** and rea
* * ** y**
* ** d***
sia .*** ry* ebec srae tar It en po tat tr
ay na n*
Ko orw , Chi ede untr apa tlan
n*
s
Ru Fedunga Qu I On don
e
lov inga ed S Aus ea ngl J
S m E N g w o c o
H In S it CD n S c S
Un OE Ko ue
ng sq
Ho Ba
1 2 http://dx.doi.org/10.1787/888933082955
Figure5.4 Time spent on lecture-style presentations in 8th grade science, according to students
% -8 -5 -4 -4 -2 -2 -2 -2 -1 -1 -0 +1 +2 +3 +3
% point
100 10
80 8
60 6
40 4
20 2
0 0
*** *** c**
*
l**
* io d** try or
e ** es an ly lia ** d*
rea ina ar n* tat Jap Ita a ay n
be rae nt lan un ap ea str rw gla
Ko , C h
Q u e I s O c o t
e c o
S i n g m t e dS Au No En
ng S u C D i
Ko sq OE Un
ng Ba
Ho
1 2 http://dx.doi.org/10.1787/888933082974

Figure5.5 Watching the teacher demonstrate an experiment or investigation in 8th grade science,
according to students
Percentage of students watching the teacher demonstrate an experiment or investigation in at least half the lessons
and change over time
% -11 -7 -5 -4 -4 -4 -3 -0 +1 +3 +3 +4 +6 +6 +15
% point
80 16
70 14
60 12
50 10
40 8
30 6
20 4
10 2
0 0
ue ** *** a**
* el* ly*
*
an
* ay ea
n d
ar
io nd
*
es
*
e**
*
a**
* ***
sq ry* ec Isr
a
Ita Jap rw lan nt tla tat or ali rea
Ba unt ueb C hin No D
m
E ng O o S a p t r Ko
co Q g, C Sc ite
d g
Au
s
on OE
Un Sin
n gK
Ho
1 2 http://dx.doi.org/10.1787/888933082993
Box 5.1 Data source details for Chapter 5.1 to 5.5

TIMSS (2003 and 2007) surveys asked 8th grade teachers In a typical week of mathematics/science lessons
for the TIMSS class, what percentage of time do students spend on each of the following activities? [...] b)
Listening to lecture-style presentations. TIMSS (2003 and 2007) asked 8th grade students How often do you
do these things in your mathematics/science lessons? [] k/l) We listen to the teacher give a lecture-style
presentation and b) We watch the teacher demonstrate an experiment or investigation with answer options
Every or almost every lesson; About half the lessons; Some lessons; Never. The same data restrictions as
in International TIMSS and PIRLS Reports by the International Association for the Evaluation of Educational
Achievement (IEA) apply to these data.

Figure5.6 Time spent on lecture-style presentations in 4th grade maths, according to teachers
% -2 -2 -2 -2 -1 -1 -0 -0 +0 +0 +1 +1 +1 +1 +1 +1 +9
% point
40 10
35
8
30
25 6
20
15 4
10
2
5
0 0
** * ** e* d ly d. ds lia nd ea
n an tes io ia a y* ***
nd ary apor lan Ita Fe lan tra tla Jap Sta Ont
ar en hin wa ebec
ala u ng n g E ng s ian ther us S co D
m
d S lov g ,C N or u
Ze H Si s A
OE
C it e
Ko
n Q
w Ru Ne Un
Ne ng
Ho
1 2 http://dx.doi.org/10.1787/888933083012
Figure5.7 Watching the teacher demonstrate an experiment or investigation in 4th grade science, ac-
cording to students
Percentage of students watching the teacher demonstrate an experiment or investigation once or twice a month
or more and change over time
% -19 -17 -14 -12 -10 -9 -8 -8 -7 -7 -4 -3 -0 -0 +0 +10

% point
100 20
90 18
80 16
70 14
60 12
50 10
40 8
30 6
20 4
10 2
0 0
*** s**
* *** ** *** ** d**
*
a**
* ** c** ** d re ly tes ***
an lia ry* ay io* n* be nd an po Ita nia
Jap and tra ga rw tar lan hin ea ue gla c otl ga Sta ve
erl Au
s
H u n
N o O n Z e a
ng
, C
CD
m Q E n S S i n
it e d
S l o
th w Un
Ne Ne Ko OE
o ng
H
1 2 http://dx.doi.org/10.1787/888933083031

Figure5.8 Reading aloud to 4th grade students, according to teachers

Percentage of students whose teacher reads aloud to the whole class weekly or more and change over time
% % point
-11 -6 -6 -4 -4 -4 -3 -3 -3 -1 -1 -1 -0 -0 +0 +1 +1 +1 +2 +2 +3 +3 +3 +3 +5 +6 +7 +9 +18
100 20
90 18
80 16
70 14
60 12
50 10
40 8
30 6
20 4
10 2
0 0
vakic*** d.** ark ia** re** nd* ates ary ain erta blic way and taly ean rica den any ario tria rael nch and ada sia* nce ds** ec** a***
g
Slopubl n Feenmovengapo ngla d St un Sp Alb epu Nor Zeal I m Af e m
D h Sw Ger On A
t us Is re ol an ne Fra an eb hin
F P C do
Reussi
a D Sl in E ite H R EC Sout um In erl Qu g, C
ch ew th
R
S U n
Cz
e N O lgi Ne Ko
n
Be ng
Ho
1 2 http://dx.doi.org/10.1787/888933083050
Notes: *** = change significant at the 0.01 level; ** = change significant at the 0.05 level; * = change significant at 0.1 level; Change between 2006 and 2011
instead of 2001 and 2011 for Austria, Belgium French, Canada (Alberta), Denmark, Indonesia, Poland, South Africa and Spain. OECD average includes
all OECD education systems for which data is available for all years concerned.
Source: Authors calculations based on PIRLS (2001, 2006 and 2011)

TIMSS (2003 and 2007) surveys asked 4th grade teachers In a typical week of mathematics lessons for
the <fourth-grade> students in the TIMSS class, what percentage of time do students spend on each of the
following activities? [...] b) Listening to lecture-style presentations. TIMSS (2003 and 2007) asked 4th grade
students In school, how often do you do these things? [] b) I watch the teacher do a science experiment
with answer options At least once a week, once or twice a month, a few times a year, Never. PIRLS
(2001, 2006 and 2011) surveys asked When you have reading instruction and/or do reading activities with the
students, how often do you do the following? [...] a) Read aloud to the class with answer options Every day
or almost every day; Once or twice a week; Once or twice a month; Never or almost never. The same data
restrictions as in International TIMSS and PIRLS Reports by the International Association for the Evaluation
of Educational Achievement (IEA) apply to these data.

Independent work
General findings
Innovation in terms of independent work has not been widely observed in 8th grade maths or
science classes between 2003 and 2007 (Figure5.9 and Figure5.10). Only in Quebec (8% points) did
innovation result in a reduction in independent work in mathematics. This change was characterised
by a small effect size.
No significant innovation was observed in 4th grade maths between 2003 and 2007, but innovation
in reading took the form of a greater emphasis on reading alone between 2001 and 2011 (Figure5.11
and Figure5.12). The average absolute change in reading alone across the OECD was 4% points.
Norway (12% points) and Hong Kong (20% points) exhibited noticeable increases in this aspect of
classroom practice. These country level changes were characterised by medium effect sizes whilst
the OECD average and average absolute change exhibited small effect sizes. Overall, with at least
small effect sizes, there was an increase in teachers asking students to read alone in 13 countries.

Figure5.9 Time spent on independent work in 8th grade maths, according to teachers
% -8 -4 -3 -3 -2 -2 -2 -1 -1 -0 -0 -0 -0 -0 +1 +1 +1 +2 +3 +3 % point
30 10
9
25 8
20 7
6
15 5
4
10 3
5 2
1
0 0
*** ia*** a*** ry** na** l** e** es an** nia ly io ay en ia an ed.* ntry land land
ec e ga , Chi
ae or Stat ve Ita tar orw wed ones Jap
eb stral Kor Isr gap m
e
Slo On ian
F
co
u ot Eng
u
Q Au u n
H on g i n i t e d
C D N S In
d
ss que Sc
S
K Un OE R u s
ng Ba
Ho
1 2 http://dx.doi.org/10.1787/888933083069
Figure5.10 Time spent on independent work in 8th grade science, according to teachers
% -4 -2 -2 -2 -1 -1 -1 -1 -1 -1 -0 -0 +0 +0 +0 +1 +1 +1 +1 +2
% point
25 5
20 4
15 3
10 2
5 1
0 0
*** lia* a*** n a es ore* ry n ** ia ia ry el d c io d y* **
ay a e de hin tat nt Japa ean nes ven nga Isra tlan uebe ntar glan Ital d.*
o rw ustr Kor Swe g, C ed S ngap cou m ndo S lo H u S c o Q O E n n Fe
A n Si ue CD I ia
N Ko Unit sq OE ss
ng Ba Ru
Ho
1 2 http://dx.doi.org/10.1787/888933083088

TIMMS (2003 and 2007) asked 8th grade teachers In a typical week of mathematics/science lessons for the
TIMSS class, what percentage of time do students spend on each of the following activities? [] d) Working
problems on their own without your guidance.
The same data restrictions as in International TIMSS and PIRLS Reports by the International Association for
the Evaluation of Educational Achievement (IEA) apply to these data.

Figure5.11 Time spent on independent work in 4th grade maths, according to teachers
% -6 -3 -2 -1 -1 -1 -1 -0 +0 +1 +1 +1 +1 +1 +1 +3 +5
% point
50 10
45 9
40 8
35 7
30 6
25 5
20 4
15 3
10 2
5 1
0 0
c**
* ay a
tes
* d ia re n ia io ly ary d. d ds an
*
d**
*
e rw hin lan stral po ea en nt
ar Ita ng Fe lan rlan Jap
eb No ,C Sta Scot u ga D
m lov O Hu ian ea e g lan
Qu n g
ite
d A S i n C S s s w
Z t h E n
Ko OE Ru Ne Ne
ng Un
Ho
1 2 http://dx.doi.org/10.1787/888933083107
Figure5.12 Students reading silently alone in 4th grade, according to teachers

Percentage of teachers asking students to read silently on their own once a week or more and change over time
% % point
-1 +0 +0 +0 +0 +1 +1 +1 +1 +2 +2 +3 +3 +3 +3 +4 +4 +4 +4 +5 +6 +7 +7 +8 +9 +9 +12 +20 +28
100 30
90
80 25
70
20
60
50 15
40
30 10
20
5
10
0 0
w d* ce d h ta lic n ia d. d* io ia el es re y* *** ** *** *** *** ** *** *** *** *** *** *** ***
Ne alan ran glan renc lber pub ede nes n Fe olan ntar ustr Isra Stat apo ngar ada enia ean nds any ublic bec ark ain taly way ina rica
Z e F En F A Re Sw do ia P O A d ng Hu Can Slov m erla erm Rep Que enm Sp I or Ch Af
um h In uss ite Si D N ng, th
lgi z ec R Un O EC Neth G vak D o ou
B e C Slo n gK S
H o
1 2 http://dx.doi.org/10.1787/888933083126

TIMMS (2003 and 2007) asked 4th grade teachers In a typical week of mathematics lessons for the <fourth-
grade> students in the TIMSS class, what percentage of time do students spend on each of the following
activities? [] d) Working problems on their own without your guidance. PIRLS (2001, 2006 and 2011) asked
teachers When you have reading instruction and/or do reading activities with the students, how often do
you do the following? [...] c) Ask students to read silently on their own with answer options Every day or
almost every day; Once or twice a week; Once or twice a month; Never or almost never. The same data

Summary
Innovation in terms of teaching styles has been more notable in terms of changes in the use of
front-of-class teaching than in the extent to which students carry out independent work, although
there have been changes in reading alone in primary schools. Front-of-class teaching at 8th grade
has both increased and decreased over time, but has typically been in the same direction for maths
and science within a country and has been more marked in maths. There has also been consistency
in the direction of change in the extent to which teachers demonstrate science experiments to
primary and secondary students within a country (albeit over different time periods), which in most
cases has meant reducing this practice.
Figure5.13 Change in time spent on lecture-style presentations in 8th grade, according to teachers.
lecturing-science lecturing-maths
%
20
-20
nia ralia Ita
ly
tar
io
po
re tes be
c
an
d ina orea rway gary ation land apan eden ea
n n )
ea ge
ve st On ga Sta Que Engl g, Ch K No un er co
t J m m han
Slo Au i n e d n H e d S Sw CD D . c
S it F OE
C
Un Ko
ian OE (abs
ng ss
Ho Ru
1 2 http://dx.doi.org/10.1787/888933083145
Figure5.14 Change in watching the teacher demonstrate an experiment or investigation

in at least half the lessons, according to students.
Teacher experiment demo- 4th grade Teacher experiment demo- 8th grade
%
20
-20
-40
es ly nd d or
e nd nia tario ary ina rway be
c ds lia an n n )
tat Ita gla an ala ve ng lan tra Jap ea ea ge
S En otl g ap e l o n u , Ch o Q ue r us m m an
ited S c
Sin ew Z S O H n g N t h e A EC
D
C D . ch
Un N Ko Ne O OE (abs
ng
Ho
1 2 http://dx.doi.org/10.1787/888933083164
Note : For details please see Figures5.1, 5.3, 5.5 and 5.7.
Note on Israel
The statistical data for Israel are supplied by and under the responsibility of the relevant Israeli authorities.
The use of such data by the OECD is without prejudice to the status of the Golan Heights, East Jerusalem and
Israeli settlements in the West Bank under the terms of international law.

Table 5.1. Effect sizes for changes in front-of-class practices and use of independent work
Change in front of class practices Change in independent work

8 grade
th
4 grade
th
8th grade 4th grade
Lecture-Style presentation
Lecture-style presentation
in science (student resp.)
in maths (student resp.)
presentation in science
presentation in maths
presentation in maths
Reading on their own

Science experiments
Science experiments
demonstration
demonstration
Reading aloud
Lecture-Style
Lecture-style
Mathematics
Mathematics
lecture-style
to the class
Science
03-07 03-07 03-07 03-07 03-07 03-07 03-07 01-11 03-07 03-07 03-07 01-11
Australia 0.03 0.00 0.00 0.02 0.13 0.00 -0.28 m -0.06 -0.04 -0.01 m
Canada m m m m m m m 0.14 m m m 0.27
Ontario -0.01 -0.07 -0.04 -0.03 0.06 0.01 -0.19 0.15 0.00 0.01 0.01 0.17
Quebec 0.00 -0.03 -0.04 -0.07 -0.15 0.15 -0.13 0.26 -0.16 0.01 -0.12 0.48
Czech Republic m m m m m m m -0.02 m m m 0.03
France m m m m m m m 0.17 m m m 0.01
Germany m m m m m m m 0.03 m m m 0.38
Hungary -0.04 -0.04 -0.06 m m -0.05 -0.31 -0.08 -0.05 0.00 0.02 0.21
Israel 0.02 0.02 -0.04 -0.06 -0.09 m m 0.15 -0.05 0.00 m 0.13
Italy -0.01 -0.01 -0.04 0.01 -0.09 -0.03 0.00 -0.02 -0.01 0.04 0.03 0.29
Japan 0.02 -0.41 0.06 -0.01 -0.08 0.01 -0.47 m 0.02 -0.01 0.04 m
Korea 0.04 -0.17 0.02 -0.14 0.30 m m m -0.08 -0.08 m m
Netherlands m m m m m 0.00 -0.33 0.27 m m 0.02 0.46
New Zealand m m m m m -0.08 -0.16 -0.02 m m 0.02 -0.09
Norway 0.06 -0.07 0.05 0.05 -0.07 0.04 -0.20 -0.03 0.00 -0.06 -0.03 0.57
Slovak Republic m m m m m m m -0.30 m m m 0.21
Slovenia 0.00 0.11 -0.02 m m 0.03 0.26 -0.22 -0.01 0.00 0.00 0.22
Basque country 0.04 -0.03 0.09 -0.03 -0.23 m m m 0.05 -0.02 m m
Sweden 0.09 0.05 0.08 m m m m 0.06 0.00 -0.03 m 0.14
England 0.06 0.00 0.02 0.05 0.01 -0.03 -0.10 -0.22 0.04 0.02 0.08 0.01
Scotland -0.03 -0.06 0.19 -0.05 0.07 0.01 -0.06 m 0.04 0.01 -0.02 m
United States 0.05 -0.04 -0.01 -0.01 0.07 0.01 0.00 -0.14 -0.02 -0.02 -0.02 0.19
OECD (average) 0.02 -0.05 0.02 -0.02 -0.01 0.00 -0.15 0.01 -0.02 -0.01 0.00 0.22
OECD (avevage absolute) 0.03 0.07 0.05 0.04 0.11 0.04 0.14 0.19 0.04 0.02 0.03 0.24
Hong Kong, China -0.01 -0.13 0.04 -0.09 -0.10 0.01 -0.16 0.53 -0.05 -0.03 -0.03 0.55
Indonesia -0.08 0.04 -0.03 m m m m m 0.01 0.00 m m
Russian Federation -0.04 0.31 -0.06 m m -0.01 m -0.24 0.04 0.04 0.02 0.08
Singapore -0.04 -0.08 -0.02 -0.03 0.12 -0.03 -0.01 -0.17 -0.05 -0.03 -0.01 0.15
1 2 http://dx.doi.org/10.1787/888933083183
Notes: OECD average includes all OECD education systems for which data is available for all years concerned
= Effect size (from -0.2 to -0.5 or 0.2 to 0.5)
= Effect size (equal or above -0.8 or equal or below 0.8)
Source: Authors calculations based on TIMSS (2003 and 2007) and PIRLS (2001, 2006 and 2011).

CHAPTER 6
Innovation
in instructional practices
Innovation in instructional practices could incorporate changes in the extent to

which students apply their knowledge and skills to their real lives or to activities
such as interpretation of data or reasoning. The aim of such innovation may be
to encourage engagement and motivation by making lessons more salient or to
encourage students critical thinking skills. A reduction in these practices may
occur if teachers explore innovative alternatives or seek to spend the time on
different activities.

6. INNOVATION IN INSTRUCTIONAL PRACTICES
Relating lessons to real-life
General Findings
Innovation in the classroom encompasses an increase in the practice of asking students to relate
what they learned to their daily lives. Across the OECD area the absolute change in 8th grade teachers
asking students to relate what they learned in maths to real-life between 2003 and 2011 was 13%
points; the equivalent change in secondary school science lessons was 15% points (Figure6.1 and
Figure6.3). The average absolute change in students in OECD countries reporting that they related
what they learned to their daily lives between 2003 and 2007 was 8% points for maths and 2% point
for science (Figure6.2 and Figure6.4). In maths, teachers in Indonesia exhibited the largest increase
(36% points), whilst increases in Italy (31% points) and the Russian Federation (28% points) were also
marked. Indonesia (28% points) also stood out in 8th grade science, as did Ontario (27% points), Korea
(25% points), Italy (22% points), Singapore (22% points) and Israel (20% points). These education
system changes presented large to medium effect sizes. Based on teacher reports, the OECD average
and average absolute change for 8th grade maths and science showed small effect sizes. Overall,
and with at least small effect sizes, there was an increase in teachers asking students to relate
maths lessons to daily life in 14 countries, and a decrease in one. Students own reporting in maths
indicated an increase in five countries with small effect sizes. In 8th grade science, increases were
observed in 20 education systems with at least small effect sizes.
Fourth grade teachers innovated by increasing the extent to which they asked their students to
relate their maths, science and reading lessons to their daily lives (Figure6.5 to Figure6.7). Between
2007 and 2011 the average absolute change across OECD countries in students being asked to relate
what they learned in maths to real-life was 11% points, whilst in science the change between 2003
and 2011 amounted to 18% points. In reading, the average absolute change in students relating
their reading to their own experiences between 2001 and 2011 in OECD countries amounted to 11%
points. Norway (22% points) and Denmark (22% points) exhibited the largest increases in relating
lessons to real-life maths, with small effect sizes. Italy (45% points), Belgium (Flemish; 36% points),
Hong Kong (32% points) and Quebec (32% points) showed large increase in teachers asking students
to relate science lessons to real-life between 2003 and 2011 with a large effect size. Norway (38%
points), Indonesia (25% points), Netherlands (24% points), Hong Kong (24% points) and Ontario (18%
points) were characterised by a large increase with regards to reading. OECD average and average
absolute change effect sizes for 4th grade mathsmscience and reading were small. Overall, with
small effect sizes, there was an increase in the extent to which 4th grade maths students related
what they learned to daily life between 2007 and 2011 in 10 education systems; similarly, and with
at least small effect sizes, an increase can be observed for 4th grade science students in 13 countries.
Significant increases, with at least small effect sizes, occurred in the extent to which reading was
related with real-life experiences for 4th grade students in 13 countries.
Indonesia and Italy stand out as countries where teachers reports indicate that there has been
widespread innovation across subjects and grades with regards to relating lessons to students daily
lives. In Italy, for example, a greater proportion of students were asked to relate their mathematics
and science lessons to their real life experiences in 2007 than 2003 in 8th grade as well as in 4th grade
science with at least medium effect sizes.

Figure6.1 Relating 8th grade maths learning to students daily life, according to teachers
Percentage of students whose teachers ask them to relate what they learn in class to their daily life
in at least half their lessons and change over time
% -13 -7 -6 -6 -2 -0 +1 +2 +3 +7 +10 +10 +12 +12 +12 +12 +14 +15 +17 +18 +20 +24 +28 +31 +36
% point
100 40
90
35
80
70 30
60 25
50 20
40
15
30
20 10
10 5
0 0
na tts nd rea tes ota bec hile land ore* ry** ey** n*** ay** na** n*** rio** lia** ca*** n*** ia*** el*** d.*** ly*** ia***
dia se ala Ko Sta nes Que C ng p ga rk ea w hi apa nta tra fri ede en sra Fe Ita nes
In achu Ze d n E inga Hun Tu D m Nor g, C J O Aus th A Sw Slov I an
ss New n ite Mi S C n si In
do
M a U O E
g Ko
S ou R us
n
Ho
1 2http://dx.doi.org/10.1787/888933083202
Notes: *** = change significant at the 0.01 level; ** = change significant at the 0.05 level; * = change significant at 0.1 level; Change between 2007 and
2011 instead of 2003 and 2011 for Turkey and United States (Massachusetts and Minnesota). OECD average includes all OECD education systems for
which data is available for all years concerned.
Source: Authors calculations based on TIMSS (2003, 2007 and 2011)
Figure6.2 Relating 8th grade maths learning to students daily life, according to students
% +0 +2 +2 +3 +4 +5 +5 +6 +6 +6 +7 +7 +8 +9 +9 +12 +14 +17 +18 +19

% point
70 20
18
60
16
50 14
40 12
10
30 8
20 6
4
10
2
0 0
s * * * * * * * * * * * * * ** ** **
ng tate ebec rael rea** re** lia** rio** aly** try** ay** nd** an** nd** ry** sia** en** ia* d.* n*
g Ko S Q u Is Ko a po stra nta It u n r w g la e t la n ga n e e d v en n Fe Japa
n d co o m o o
Ho Unit
e g
Au O N En CD Sc Hu Ind Sw Slo ssia
Sin ue
sq OE Ru
Ba
1 2 http://dx.doi.org/10.1787/888933083221
***=change significant at the 0.01 level; **=change significant at the 0.05 level; *=change significant at 0.1 level

Figure6.3 Relating 8th grade science learning to students daily life, according to teachers
% -0 +3 +5 +5 +5 +8 +8 +9 +10 +10 +10 +11 +11 +13 +15 +15 +16 +17 +18 +20 +22 +22 +25 +27 +28
% point
100 30
90
80 25
70
20
60
50 15
40
30 10
20
5
10
0 0
nia tt
s ey ica e** s** na n** *** * * * a y** *** .*** *** *** *** l*** *** *** *** *** ***
l y nd ec an ot a a y e a o a
ve use Turk Afr Chi tate ndia ede gar eala ueb Jap nes rwa ean Fed hin trali land srae Ital por ore tari esi
Slo ach u t h d S I w n
S Hu w Z Q i n o
N D m sian g, C Aus Eng I
n ga K On
d on
s s o i t e M C s n S i I n
Ma
S
Un Ne OE Ru g Ko
n
Ho
1 2 http://dx.doi.org/10.1787/888933083240
Figure6.4 Relating 8th grade science learning to students daily life, according to students
% -5 -3 -3 -3 -2 -1 -1 -1 -0 +1 +1 +1 +2 +2 +5
% point
70 6
60 5
50
4
40
3
30
2
20
10 1
0 0
** io* a ly* c ry rea n tes re y nd an a ***
ae
l*
tar ali Ita be t ea po wa tla Jap
in d
Isr str ue co
un Ko m Sta ga or co Ch lan
On Au Q C D e d i n N S g,
En
g
ue OE it S
Ko
n
as
q Un ng
B
Ho
1 2 http://dx.doi.org/10.1787/888933083259
***=change significant at the 0.01 level; **=change significant at the 0.05 level; *=change significant at 0.1 level
Box 6.1 Data source details for Figures 6.1 to 6.4

TIMSS (2003, 2007 and 2011) surveys asked 8th grade teachers In teaching mathematics/science to this
class, how often do you usually ask students to do the following? [...] h/i) Relate what they are learning in
mathematics/science to their daily lives; with answer options Every or almost every lesson; About half
the lessons; Some lessons; Never. TIMSS (2003 and 2007) asked 8th grade students How often do you do
these things in your mathematics/science lessons? [...] h/j) We relate what we are learning in mathematics/
science to our daily lives with answer options Every or almost every lesson; About half the lessons; Some
lessons; Never. The same data restrictions as in International TIMSS and PIRLS Reports by the International
Association for the Evaluation of Educational Achievement (IEA) apply to these data.

Figure6.5 Relating 4th grade maths learning to students daily life, according to teachers
% -4 -3 +1 +2 +4 +6 +6 +7 +8 +8 +8 +8 +9 +10 +10 +11 +12 +15 +17 +18 +18 +22

% point
+22
100 25
90
80 20
70
60 15
50
40 10
30
20 5
10
0 0
nd ina blic en nd an ed. ed. ia* bec re** ria* ry** rio n*** ds* ic** a*** s*** y*** a*** y*** k***
ala Ch epu wed ngla Jap an F an F oven Que apo ust nga Onta ea rlan publ bert tate an trali rwa mar
Ze ng, R S E ss
i
ss
i Sl g A
Hu
m e e Al ed S Germ Aus No Den
w o
Ne ng K lova
k
Ru Ru Sin CD Neth ch R
S OE z e U nit
Ho C
1 2 http://dx.doi.org/10.1787/888933083278
Figure6.6 Relating 4th grade science learning to students daily life, according to students
% -7 -0 +3 +3 +5 +5 +7 +7 +8 +8 +9 +11 +13 +17 +19 +22 +24 +26 +28 +31 +32 +32 +36
% point
+45
100 50
90 45
80 40
70 35
60 30
50 25
40 20
30 15
20 10
10 5
0 0
.
nd ed tria blic ny blic nd rta lia an rio* y*** s*** n*** e*** y*** s*** n*** a*** k*** c*** a*** h*** y***
ala n F Aus epu rma epu ngla Albe ustra Jap nta gar tate ea por rwa land ede veni mar ebe hin mis Ital
Ze ssia Ge h R E O un d S D m nga No er Sw lo en Qu g, C Fle
w Ru kR A H ite EC Si th S
Ne va
Cz
ec Ne
D n
Ko ium
Slo Un O
ng Belg
Ho
1 2 http://dx.doi.org/10.1787/888933083297
2011 instead of 2003 and 2011 for Austria, Canada (Alberta), Czech Republic, Denmark, Germany, Slovak Republic and Sweden. OECD average includes

Figure6.7 Relating 4th grade reading to students own experience, according to teachers
Percentage of students whose teachers ask them to relate what they read with their own experience
% -7 -5 -2 +1 +2 +2 +3 +4 +5 +5 +6 +7 +7 +7 +8 +8 +8 +9 +9 +10 +12 +13 +18 +19 +20 +24 +24 +25 +38
% point
100 40
90 35
80
70 30
60 25
50 20
40 15
30
10
20
10 5
0 0
ec ch rk d ny ta el d. lic lic d ia s** ia ce ** en *** *** *** d* *** *** *** *** *** s** *** ***
eb en ma lan a ber Isra n Fe pub pub alan ven tate ustr ran Italy ed gary ore ean glan ada ario ain rica ina and esia way
Qum FrDen PoGerm Al sia Re Re Ze Sl ed S
o A F Sw un gap m En Can Ont Sp Af , Ch erl on Nor
g iu R us ech vak New i t H Sin ECD h
ut ng eth In
d
Be
l Cz Slo U n O So g Ko N
n
Ho
1 2 http://dx.doi.org/10.1787/888933083316

TIMSS (2003, 2007 and 2011) surveys asked 4th grade teachers In teaching mathematics/science to this
class, how often do you usually ask students to do the following? [...] g/h) Relate what they are learning in
mathematics/science to their daily lives with answer options Every or almost every lesson; About half the
lessons; Some lessons; Never. PIRLS (2001, 2006 and 2011) asked 4th grade teachers: How often do you ask
the students to do the following things to help develop reading comprehension skills or strategies? [...] d)
Compare what they have read with experiences they have had with answer options Every day or almost
every day; Once or twice a week; Once or twice a month; Never or almost never. The same data restrictions as

Data and text interpretation
General Findings
Innovation in classroom practice has included both increases and decreases in the extent to
which students interpret data and text.
Across the OECD area the average absolute change in 8th grade students reporting that they had
interpreted data in maths between 2003 and 2007 was 7% points (Figure6.8). A larger proportion
of students in Japan (18% points), Hong Kong, (14% points), Korea (14% points) and the Russian
Federation (10% points) interpreted data in 2007 than 2003. In contrast, students in Hungary (21%
points) indicated a large reduction in data interpretation over this time period. These education
system changes presented small effect sizes.
The average absolute change between 2001 and 2011 for teachers reporting that 4th grade
students made generalisations and drew inferences when reading came to 16% points (Figure6.9).
Such text interpretation increased notably in Hong Kong (38% points), the Slovak Republic (37%
points) and France (32% points). These education system level increases showed medium to large
effect sizes whilst the OECD average and average absolute change for 4th grade reading showed small
effect sizes. Overall, and with at least medium effect sizes, there was an increase in the extent to
which students were asked to make generalisations and draw inferences in five countries.
Hong Kong stands out as an education system where students were asked to interpret data and
text more across disciplines and levels. In Hong Kong a larger proportion of students were asked to
interpret data in 2007 than 2003 in 8th grade mathematics and to draw inferences from text in 2011
than 2001 in 4th grade reading, with at least small effect sizes.

Figure6.8 Interpreting data in tables, figures or graphs in 8th grade maths, according to students
Percentage of students reporting that they interpret data in tables, figures or graphs
in half of the classes or more and change over time
% -21 -4 -3 -3 -0 +0 +2 +2 +3 +3 +4 +5 +7 +8 +8 +9 +10 +14 +14 +18

% point
80 25
70
20
60
50 15
40
30 10
20
5
10
0 0
* * * ** ** ** ** ** ** ** g, *** **
ry*
* * ly
Ita orw
ay d try en tes an** and tario lia* el* ore* nia* sia* bec* ed.* rea* Kon ina an*
lan oun wed Sta e t l ra ra
ga N g
En ue c S ed m Sco O n s t Is a p v e n e e F o
K on g h
C Jap
n Au ing Slo Indo Qu sian
Hu s q U nit ECD S u s H
Ba O R
1 2 http://dx.doi.org/10.1787/888933083335
Figure6.9 Students making generalisations and drawing inferences from a text in 4th grade reading,
according to teachers
Percentage of students whose teachers ask them to make generalisations and draw inferences
from a text once a week or more and change over time
% -3 +0 +2 +3 +3 +4 +6 +7 +8 +8 +9 +11 +11 +13 +14 +15 +16 +17 +18 +19 +20 +21 +22 +23 +23 +26 +32 +37 +38
% point
100 40
90 35
80
70 30
60 25
50 20
40 15
30
10
20
10 5
0 0
h c y k . a * * a * * * * * * * * * * * * * * * * * * * * *
ec bli an ar ed rt ry* d* tri el* d* s** d** ica n* o** e** n** a** h** c** a** y** s** y** a** n** e** ic** a**
Cz epu erm enm ian F Albe nga olan Aus Isra glan tate alan Afr ede tari apor mea nad renc ebe veni Ital land rwa nesi Spai ranc ubl hin
R G D ss Hu P En ed S Ze uth Sw On ing CD Ca m F Qu Slo er No ndo F Rep g, C
Ru it w So S E th I k n
Un Ne O
e lgi
u Ne
l ova g Ko
B S on
H
1 2 http://dx.doi.org/10.1787/888933083354

TIMSS (2003 and 2007) asked 8th grade students How often do you do these things in your mathematics
lessons? [] d) We interpret data in tables, Figures, or graphs Every or almost every lesson; About half the
lessons; Some lessons; Never. PIRLS (2001, 2006, and 2011) asks 4th grade teachers How often do you ask
the students to do the following things to help develop reading comprehension skills or strategies? [...] g)
Make generalizations and draw inferences based on what they have read with answer options Every day
or almost every day; Once or twice a week; Once or twice a month; Never or almost never. The same data

Reasoning
General findings
Innovation in instructional practices can also be indicated through students explaining and
elaborating their answers more widely during mathematics and science lessons in secondary
education (Figure6.10 to Figure6.13). Between 2007 and 2011 OECD average absolute change for
students being asked to observe and describe natural phenomena was 20% points, while it amounted
to 12% points for students being asked to explain what they are studying during science classes. As
for mathematics classes, the average absolute change within the OECD area between 2003 and 2011
was 10% points for students being asked to explain their answers, similar to the one reported by
students (6% points) between 2003 and 2007. During science classes, a larger proportion of students
in England (34% points), Australia (32% points) and the United States (31% points) in particular have
been asked to describe natural phenomena, whereas in Quebec (31% points) and Hong Kong (24%
points) a larger proportion of students was asked to explain what they were studying. Significantly
more teachers in Indonesia (30% points), Singapore (22% points), Australia (21% points) and Japan
(20% points) asked their students to explain their answer during mathematics lessons, while
more students reported having to explain their answer in Japan (44% points). These changes were
characterised by large and medium effect sizes, while the OECD average and average absolute
changes showed small effect sizes. Overall, with at least small effect size, the extent to which
students described natural phenomena increased in 20 education systems and the extent to which
they explained what they were studying during their science lessons increased in 16. There was
an increase in teachers asking students to explain their answers during mathematics classes in 13
countries and a decrease in one, whereas students reported having explained their answers more in
two countries and less in one, although within a shorter time period.
As to primary education, classroom innovation was also illustrated through an increase

in students being asked to explain their answers and what they were studying (Figure6.14 and
Figure6.15). OECD average absolute change regarding students being asked to explain what they
were studying in science lessons was 13% points between 2007 and 2011, whilst it amounted to
12% points for students being asked to explain their answers in mathematics between 2003 and
2011. The Netherlands (37% points), Sweden (30% points) and the Slovak Republic (21% points) are
countries where there was an increase in students being asked to explain what they were studying
in their science classes. For mathematics, the education systems showcasing the largest increases
in students explaining their answers were Norway (40% points), Hong Kong (28% points), Alberta
(23% points), Australia (23% points) and the United States (17% points) These system level changes
exhibited from medium to large effect size, while the OECD average and average absolute change
effect sizes were small for both mathematics and science. Altogether, with at least small effect sizes,
primary school students were asked more for explanations in 17 education systems during both
mathematics and science classes.
Innovation in the form of an increase in the practice of asking students to elaborate on their
answers and the topics they studied occurred across disciplines and levels in Australia. With at least
small effect sizes, there was an increase in Australian students being asked to explain their answers
and the topics of their studies both in 8th and 4th grade and to describe observed natural phenomena
in 8th grade.

Figure6.10 Students explaining answers during 8th grade maths lessons, according to teachers
Percentage of students whose teachers ask them to explain their answers in at least half their lessons and change over time
% -13 -4 -3 -2 +1 +1 +3 +5 +6 +7 +7 +7 +8 +8 +9 +11 +11 +13 +16 +17 +19 +20 +21 +22 +30
% point
100 35
90
30
80
70 25
60 20
50
40 15
30 10
20
10 5
0 0
*** taly ary Fed. way key bec ina etts n*** ile** nia* el** s*** nd* ana rio** nd** n*** a*** ta** n*** ia*** re*** ia***
rea I ng n r Tur ue Ch us ea Ch ve Isra ate ala di ta a e ic o a l o s
Ko Hu ssia No Q g,
n sach CD m Slo St Ze In On Engl wed Afr nnes Jap stra gap one
u o s e d w S t h i Au in nd
R g K a E n i t Ne o u M S I
n M O U S
Ho
1 2 http://dx.doi.org/10.1787/888933083373
Figure6.11 Students explaining answers during 8th grade maths lessons, according to students
Percentage of students whose teachers ask them to explain their answers in at least half their lessons and change over time
% -11 -5 -3 -2 -2 -2 -2 -0 +0 +0 +1 +1 +2 +2 +2 +3 +6 +9 +13 +44

% point
100 50
90 45
80 40
70 35
60 30
50 25
40 20
30 15
20 10
10 5
0 0
,
** y** ry d ay ec rael ates re d ian ion io ia ng a a* *** *** *** ** **
ry* Ital ount glan orw ueb s po otlan uss erat ntar stral g Ko hinKore ean den esia ia* n*
n g a c n N Q I
d S t
n g a c R e d O u o n C m e n ven Japa
E e i S A w o Sl o
Hu ue it S F H CD S In
d
sq Un OE
Ba
1 2 http://dx.doi.org/10.1787/888933083392

Figure6.12 Students explaining what they are studying during 8th grade science lessons,
Percentage of students whose teachers ask them to explain what they are studying in at least half their lessons and change over time
% -3 +0 +4 +6 +8 +8 +10 +10 +10 +10 +11 +11 +12 +13 +13 +17 +17 +18 +20 +24 +31
% point
100 35
90 30
80
70 25
60 20
50
40 15
30 10
20
5
10
0 0
. * * * * * s ** rio* ** d*** a*** ** a*** e*** ** a*** c***
ary Fed rkey enia rael aly** sia** tes** sota sett ay n* n* i r n*
ng n Tu lov Is It e a n e u rw Onta mea glan Kore Japa tral apo de hin ebe
Hu ssia S do
n S t n
Mi ssa
ch N o
En s g e
Sw ng, C Qu
Ru In nited a EC
D Au Sin o
U M O K
ng
Ho
1 2 http://dx.doi.org/10.1787/888933083411
Figure6.13 Students observing and describing natural phenomena during 8th grade science lessons,
Percentage of students whose teachers ask them to observe and describe natural phenomena
% +11 +12 +13 +13 +14 +14 +15 +17 +19 +19 +20 +21 +24 +26 +26 +26 +28 +31 +31 +32 +34
% point
80 35
70 30
60 25
50
20
40
15
30
20 10
10 5
0 0
** ** s
y**
*
a**
*
n*
*
y**
* ** ** y*** ** ** ** ** ** ** ** ** ** ** **
ec ea ett n* el* n* ed.* ina* ary* sia* rio* ore* tes* ota* lia* nd*
eb Kor hus rwa veni Japa Ital ede Isra urke mea F h g e ta p ta nes ra gla
Qu a c o
Sl o w T a n , C u n o n n g a S s t n
ss N S CD ssi ng H In
d O
Sin nited Min Au E
Ma OE Ru g Ko U
n
Ho
1 2 http://dx.doi.org/10.1787/888933083430

TIMSS and PIRLS (2003, 2007 and 2011) survey asked teachers In teaching mathematics/science to this
class, how often do you usually ask students to do the following? g) Explain their answers a) Observe natural
phenomena and describe what they see h) Give explanations about something they are studying, with
answer options Every or almost every lesson; About half the lessons; Some lessons; Never. TIMSS (2003
and 2007) survey asked students How often do you do these things in your mathematics lessons? [] g) We
explain our answers with answer options, with answer options Every or almost every lesson; About half the
lessons; Some lessons; Never. The same data restrictions as in International TIMSS and PIRLS Reports by the
International Association for the Evaluation of Educational Achievement (IEA) apply to these data.

Figure6.14 Students explaining answers during 4th grade maths lessons, according to teachers
Percentage of students whose teachers ask them to explain answers in at least half their lessons and change over time
% -0 -0 +1 +1 +2 +6 +8 +8 +9 +10 +11 +11 +12 +12 +12 +12 +14 +16 +17 +18 +23 +23 +28
% point
+40
100 45
90 40
80 35
70
30
60
25
50
40 20
30 15
20 10
10 5
0 0
um ish ec Fed. gary Italy nd* den an* lic** ia** d*** lic*** re** y*** io*** n*** rk*** ia*** es*** ds*** ia*** ta*** a*** ay***
l i m eb
g gla Swe Jap pub oven alan pub gapo man ntar mea ma ustr Stat lan tral lber Chin orw
Be Fle Qu sian Hun En
s e Sl Ze Re Sin Ger O CD Den A ed er Aus A g, N
Ru hR th on
z ec N ew vak
o OE U nit Ne gK
C Sl n
Ho
1 2 http://dx.doi.org/10.1787/888933083449
Figure6.15 Students explaining what they are studying during 4th grade science lessons,
Percentage of students whose teachers ask them to explain what they are studying in at least half their lessons and change over time
% +0 +2 +3 +5 +5 +6 +6 +7 +11 +11 +12 +12 +13 +13 +14 +15 +15 +16 +19 +19 +21 +30 +37
% point
100 40
90 35
80
30
70
60 25
50 20
40 15
30
10
20
10 5
0 0
nd blic d.*** stria way tes* nia* ly*** rta** ina** nd** ny*** re*** n*** ry*** lia** io*** n*** ec*** rk*** lic*** en*** ds***
ala u e r a
Au No d St Slov
e Ita be h la a po ea ga tra ar Japa ueb ma ub ed lan
Ze Rep ian F Al g, C Eng erm nga D m Hun Aus Ont Q Den Rep Sw her
w h
Ne Czec Rus
s n ite o n G S i E C k t
U g K O va Ne
Ho
n Slo
1 2 http://dx.doi.org/10.1787/888933083468

TIMSS (2003, 2007 and 2011) surveys asked 4th grade teachers: In teaching mathematics/science to this
class, how often do you usually ask students to do the following? [] g) Explain their answers-[] g) Give
explanations about something they are studying with answer options Every or almost every lesson; About
half the lessons; Some lessons; Never. The same data restrictions as in International TIMSS and PIRLS Reports
by the International Association for the Evaluation of Educational Achievement (IEA) apply to these data.

Summary
Innovation in the extent to which students have been asked to relate what they are learning
to their daily lives has almost always manifested as an increase, and the practice has typically
increased more in maths than science at both 8th grade and 4th grade. Innovation or significant
change is also illustrated through an increase in the extent of student reasoning and self-directed
work across disciplines and educational levels between the time frames analysed.
Figure6.16 Change in students relating what they learn to their daily lives in 8th grade
Relate to real life- science Relate to real life- maths

%
80
60
40
20
-20
ile tes ec nd ry ia na nd an ay en na re ea lia rio on ael Italy esia ean ean ge)
Ch Sta ueb eala unga oven ndia ngla Jap orw wed , Chi gapo Kor stra nta erati Isr n m n
d Q Z H S l I E N S g n u O d do CD CD mcha
ite w on Si A Fe In E OE bs.
Un Ne ng
K
ia n O ( a
Ho ss
Ru
1 2 http://dx.doi.org/10.1787/888933083487
Figure6.17 Change in students relating what they learn to their daily lives in 4th grade
Relate to real life- science (2007-2011) Relate to real life- maths (2003-2011)
%
60
40
20
-20
k ic n n ia d d ic n ta y io ry rk lia re s a s ia h c y ly n n )
va bl ede tio str lan lan ubl Japa lber man ntar nga ma stra apo State Chin land ven mis uebe orwa Ita mea mea nge
SloRepu Sw dera Au Zea Eng Rep A Ger O u e n Au ng d , e r l o l e Q N a
Fe w h H D S g
i ite on eth S
m
F
E C ECDs. ch
D
ian Ne Cz
ec Un ng K N lgi
u O O (ab
ss Ho Be
Ru
1 2 http://dx.doi.org/10.1787/888933083506

Figure6.18 Change in students reasoning in 8th grade
Observe and describe- science Explain subjects- science Explain answers- maths (2003-2011)
%
80
60
40
20
-20
ile nd na ica tts ey on ay aly ia ry ael ea ta ec rio tes an en na nd sia re lia an an e)
Ch eala ndia Afr use Turk erati orw It oven unga Isr Kor neso ueb nta Sta Jap wed , Chi ngla one gapo stra me me ang
Z I h h N l n Q O ed S ng E Ind Sin Au CD CD . ch
ut ac ed S H Mi
Ne
w So ass a nF U nit g Ko OE OE(abs
M s i o n
s
Ru H
1 2 http://dx.doi.org/10.1787/888933083525
Figure6.19 Change in students reasoning in 4th grade
Explain subjects-science Explain answers- maths (2003-2011)

%
60
40
20
-20
c
m sh lic on ria taly and erta any ary nia and bec ark ubli tes pan ore ario den alia ina ay nds ean an ge)
l giu mi ub rati ust I al Alb rm ung love ngl Que nm Rep w
Sta Ja ngap Ont Swe ustr g, Ch Nor erla D m m han
e
Be Fle Rep ede A Ze G e H S E e
D ak t e d S i A on t h EC ECD . c
h F w v Un i e s
ec n
Cz ssia Ne Slo ng
K N O O (ab
Ru Ho
1 2 http://dx.doi.org/10.1787/888933083544
Notes: For details please see Figures 6.1, 6.3, 6.5, 6.6, 6.10, 6.12, 6.14 and 6.15.
Note on Israel

Table 6.1 Effect sizes for changes in relating lessons to real life, interpreting data and text, and reasoning
Data and text

Change in student relating to real life Change in students reasoning
interpretation
8th grade 4th grade 8 th
4 th
8th grade 4th grade
grade grade
Explain answers
Explain answers
Explain answers
Explain subjects
Explain subjects
(student resp.)
(student resp.)
(student resp.)
(student resp.)
Observe and
Mathematics
Mathematics
Mathematics
Mathematics
describe in
in science
in science
in maths
in maths
in maths
Reading
Reading
Science
Science
Science
science
03-11 03-07 03-11 03-07 07-11 03-07 01-11 03-07 01-11 03-11 03-07 07-11 07-11 03-11 03-11
Australia 0.30 0.10 0.37 -0.05 0.38 0.15 m 0.09 m 0.48 0.04 0.41 0.67 0.63 0.29
Austria m m m m 0.17 m m m m m m m m m 0.10
Belgium Flemish m m m m m 0.78 m m m m m m m -0.01 m
Canada m m m m m m 0.30 m 0.43 m m m m m m
Alberta m m m m 0.31 m m m m m m m m m 0.26
Ontario 0.30 0.11 0.60 -0.07 0.20 0.20 0.52 0.07 0.43 0.32 0.04 0.27 0.57 0.36 0.35
Quebec 0.02 0.04 0.22 -0.04 0.17 0.68 -0.14 0.19 0.42 0.08 -0.04 0.74 0.24 0.00 0.40
Chile 0.06 m 0.23 m m m m m m 0.22 m m m m m
Czech Republic m m m m 0.29 m 0.12 m -0.06 m m m m m 0.06
Denmark m m m m 0.45 m m m m m m m m m 0.38
France m m m m m m 0.16 m 0.67 m m m m m m
Germany m m m m 0.35 m 0.04 m 0.01 m m m m m 0.26
Hungary 0.22 0.19 0.34 m 0.22 0.36 0.31 -0.43 0.19 -0.18 -0.23 -0.06 0.59 0.08 0.33
Israel 0.49 0.05 0.51 -0.10 m m 0.11 0.11 0.30 0.23 -0.04 0.22 0.41 m m
Italy 0.63 0.12 0.53 -0.05 0.15 1.02 0.22 -0.08 0.43 -0.11 -0.09 0.35 0.30 0.06 0.32
Japan 0.30 0.41 0.21 0.04 0.12 0.16 m 0.37 m 0.41 0.92 0.35 0.29 0.21 0.36
Korea -0.11 0.11 0.63 -0.02 m m m 0.34 m -0.28 0.04 0.37 0.24 m m
Netherlands m m m m 0.23 0.49 0.52 m 0.47 m m m m 0.45 0.77
New Zealand -0.12 m 0.23 m -0.08 -0.13 0.18 m 0.34 0.22 m m m 0.32 0.00
Norway 0.24 0.14 0.27 0.03 0.44 0.44 0.78 -0.06 0.48 0.02 -0.05 0.23 0.39 0.82 0.11
Slovak Republic m m m m 0.03 m 0.16 m 0.86 m m m m m 0.54
Slovenia 0.43 0.34 -0.01 m 0.19 0.75 0.18 0.15 0.45 0.17 0.28 0.12 0.28 0.28 0.16
Basque country m 0.13 m -0.02 m m m -0.01 m m -0.07 m m m m
Sweden 0.37 0.30 0.21 m 0.04 m 0.16 0.00 0.26 0.34 0.12 0.41 0.41 m 0.60
Turkey m m m m m m m m m m m 0.10 0.40 m m
England 0.06 0.15 0.39 0.11 0.09 0.17 0.26 -0.05 0.27 0.36 -0.05 0.34 0.71 0.22 0.28
Scotland m 0.18 m 0.03 m m m 0.06 m m 0.01 m m m m
United States -0.05 0.04 0.20 -0.01 0.40 0.30 0.26 0.04 0.39 0.22 -0.01 0.27 0.64 0.55 0.14
Indiana -0.26 m 0.22 m m m m m m 0.24 m m m m m
Massachusetts m m m m m m m m m m m 0.26 0.26 m m
Minnesota m m m m m m m m m m m 0.27 0.64 m m
OECD (average) 0.25 0.16 0.35 -0.01 0.22 0.38 0.25 0.05 0.40 0.16 0.06 0.29 0.44 0.33 0.30
OECD (average absolute) 0.27 0.16 0.35 0.05 0.23 0.41 0.27 0.14 0.40 0.25 0.14 0.30 0.44 0.33 0.30
Hong Kong, China 0.26 0.01 0.36 0.04 -0.06 0.69 0.53 0.31 0.82 0.10 0.04 0.58 0.53 0.58 0.25
Indonesia 0.80 0.25 0.74 m m m m 0.15 m 0.78 0.19 0.37 0.54 m m
Russian Federation 0.59 0.36 0.39 m 0.15 0.00 0.13 0.21 0.16 -0.08 0.03 0.01 0.46 0.12 0.21
Singapore 0.15 0.10 0.50 0.02 0.17 0.44 0.26 0.14 0.40 0.46 0.00 0.45 0.61 0.25 0.33
South Africa 0.36 m 0.13 m m m m m m 0.41 m m m m m
1 2 http://dx.doi.org/10.1787/888933083563
Source: Authors calculations based on TIMSS (2003, 2007 and 20011) and PIRLS (2001, 2006 and 2011)

CHAPTER 7
Innovation
in class organisation
Innovation in the classroom can also be seen through different ways of

organising the class for different instructional purposes. Teachers may innovate
by adapting the organisation of the class according to the subject and type of
content they are delivering. Teachers may also give more or less autonomy to
students through self- directed work or provide students with individualised
instruction. The aim of increasing these types of instructional practices could
be, for example, to facilitate collaborative learning between students, or to
address specific educational needs whilst a decrease might reflect a wish to
reduce the extent to which students regroup and move around the classroom
or to increase the time they spend learning directly from the teacher.

7. INNOVATION IN CLASS ORGANISATION
Group work
General findings
Innovation in the classroom also included more or less use of group working within the class in
secondary education (Figure7.1 and Figure7.2). Between 2003 and 2007 the OECD average absolute
change regarding students working in small groups was 13% points for mathematics classes and 4%
points for science classes. Japan (73% points) stands out as the country where the practice of having
students working in small groups during mathematics lessons increased the most according to
student own reporting. Korea (10% points), on the other hand, illustrates the case of a country where
students group work decreased the most during science classes. These country changes exhibited
medium and small effect sizes, while the OECD average exhibited small effect size in the case of
mathematics lessons. Overall, student group work increased with at least small effect sizes in 12
countries during mathematics classes. In the case of science lessons the increases exhibited small
effect sizes in one education system only.
Innovation in primary education classrooms also encompassed more or less use of group
working during reading and science classes (Figure7.3 and Figure7.4 4th ). Between 2001 and 2006,
the OECD average absolute change for students reading aloud in small groups or pairs was 8% points,
while it amounted to 6% points in the case of students reporting that they worked in small groups
during science lessons between 2003 and 2007. Hong Kong (26% points) was the education system
showing the largest increase in group working during reading classes. In science classes, Slovenia
(13% points) showed the largest increase in group working, whereas the Netherlands (22% points)
showed the largest decrease in this respect. These education system changes presented medium
and small effect size. Altogether, with at least small effect sizes, student group work in reading
classes increased in seven education systems and decreased in one.
Hong Kong stood out as the education system where innovation was illustrated through more
group working practices across disciplines and levels. There was an increase in the extent to which
students in Hong Kong were grouped during 8th grade mathematics lessons between 2003 and 2007
with small effect size and during 4th grade reading lessons between 2001 and 2006 with medium
effect size.

Figure7.1 8th grade maths students working in small groups, according to students
Percentage of students working together in small groups in at least half their lessons and change over time
% -3 +1 +3 +3 +5 +8 +8 +8 +8 +9 +11 +13 +13 +13 +13 +16 +17 +18 +23 +73
% point
100 80
90 70
80
60
70
60 50
50 40
40 30
30
20
20
10
10
0 0
ue try ry a** * ** *** *** *** *** * * * * * * * * * * *
sq un ga Kore ly* ay* es rael Fed. den a** lia** nia** nd** an** rio** sia** e** nd** ec** an**
Ita orw Stat hin stra or b
Ba co Hun Is n e C v e
o t l a m
e
n t a n e a p g l a u e Ja p
N d ia Sw ng, Au Slo Sc ECD O do ing En Q
ite ss o In S
Un Ru gK O
n
Ho
1 2 http://dx.doi.org/10.1787/888933083582
***=Change significant at the 0.01 level; **=change significant at the 0.05 level; *=change significant at 0.1 level.
Figure7.2 8th grade science students working in small groups, according to students
% -10 -10 -7 -6 -4 -4 -4 -3 -3 -2 -1 -1 +0 +0 +1
% point
100 12
80 10
8
60
6
40
4
20 2
0 0
*** ** ** * c ** an * e** ly d d a io s
rea
l* ay try be n* ina or Ita n n ali tar te
ae rw un ue ea Jap Ch ap gla tla str Sta
Ko Isr o c o Q m g , g En Sc
o
Au On d
N
ue CD n Sin ite
sq OE Ko Un
Ba o ng
H
1 2 http://dx.doi.org/10.1787/888933083601
Box 7.1 Data source details for Figures 7.1 and 7.2
TIMSS (2003 and 2007) surveys asked 8th grade students How often do you do these things in your
mathematics/ science lessons?[] m) We work together in small groups [] e) We work in small groups on an
experiment or investigation with answer options Every or almost every lesson; About half the lessons; Some
lessons; Never. The same data restrictions as in International TIMSS and PIRLS Reports by the International

Figure7.3 4th grade students reading aloud in small groups or pairs, according to teachers
Percentage of students whose teachers ask them to read aloud in small groups or pairs at least weekly and change over time
% -10 -5 -4 -1 -1 +1 +1 +1 +1 +5 +5 +7 +8 +9 +10 +11 +13 +17 +19 +21 +26

% point
100 30
90
80 25
70 20
60
50 15
40
30 10
20 5
10
0 0
k l
va lic* rae an
d n d s d. c ly ** ** y s* e* ** ** ** ** ** y*** a***
de an nd Fe uebe Ita and* ean* man tate por nce* land enia rio* ary* a
Slopub Is Zeal Swe cotl erla ian Q e l m e r S nga Fra g ov ta n g rw hin
R e
ew S e t h
u s s I c
C D G
i t e d S i E n S l O n
H u N ng, C
o
N N R O E U n Ko
ng
Ho
1 2 http://dx.doi.org/10.1787/888933083620
Source: Authors calculations based on PIRLS (2001 and 2006)
Figure7.4 4th grade science students working in small groups, according to students
% -22 -9 -9 -7 -7 -5 -4 -3 -3 -2 -1 -0 -0 +3 +5 +13
% point
100 25
90
80 20
70
60 15
50
40 10
30
20 5
10
0 0
*** c**
* *** *** ia*
* y** ** or
e nd es an ly ry nd *** ***
s nd rio wa n* tat Ita a a nia
nd be a a ral ea ap gla Jap ng otl
a
hin
erl
a
Qu
e a l n t s t o r m ng En dS Hu Sc C ve
th Z e O A u N
C D S i i t e
n g, Slo
Ne Ne
w OE Un Ko
ng
Ho
1 2 http://dx.doi.org/10.1787/888933083639
PIRLS (2001, and 2006) surveys asked teachers When you have reading instruction and/or do reading
activities with the students, how often do you do the following? [] c) Ask students to read aloud in small
groups or pairs with answer options Every day or almost every day; Once or twice a week; Once or twice
a month; Never or almost never. TIMSS (2003 and 2007) surveys asked 4th grade students: In school, how
often do you do these things? [] e) I work with other students in a small group on a science experiment or
investigation with answer options At least once a week; Once or twice a month; A few times a year; Never.

In-class ability grouping
General findings
Innovation in the classroom has typically resulted in slightly higher use of in-class ability
grouping in primary and secondary education but some large changes are observed in both directions
(Figure7.5 to Figure7.7 4th ). Between 2006 and 2009 the OECD average absolute change regarding
the use of in-class ability grouping amounted to 9% points for secondary education students. In
primary education the average absolute change in the regular use of same ability groups was 5%
points, whereas OECD average absolute change for the regular use of mixed ability groups amounted
to 2% points between 2001 and 2011. Indonesia (40% points), Denmark (36% points) and Greece
(29% points) are the countries that showed the largest increase in use of ability grouping policies,
while Norway (29% points) outranked all other countries for the decrease in this respect. The Czech
Republic (70% points) exhibited the largest reduction in the regular use of mixed ability groups for
4th grade reading instruction. These country changes showed from medium to large effect sizes.
Overall, ability grouping policies increased in seven education systems and decreased in two with
small effect sizes. Similarly, regular use of same ability grouping increased in seven countries and
decreased in one with small effect sizes, while regular use of mixed ability grouping increased and
decreased in two education systems.
Indonesia is a country where innovation in the classroom was exemplified in a greater regular
use of same ability grouping across educational levels. In Indonesia same ability grouping policies
increased significantly between 2006 and 2009 in secondary education as regular use of same ability
grouping increased significantly in primary education between 2001 and 2011, with at least small
effect sizes.

Figure7.5 Within class, same-ability grouping policy in secondary education

Percentage of 15 year-old students grouped by ability within class for at least some subjects and change over time
% -29 -12 -9 -8 -8 -5 -5 -5 -3 -2 -2 -0 +0 +1 +2 +3 +3 +4 +4 +4 +5 +6 +7 +7 +7 +9 +9 +10 +14 +18 +19 +29 +36 +40

% point
100 45
90 40
80 35
70 30
60 25
50
20
40
15
30
20 10
10 5
0 0
*** *** n* es y* d y* *** ey ia m d ia lic a *** d ia ry le m n *** d. el d* s* *** ** *** *** *** *** ***
ay rg pa tat an lan tal ia rk str iu lan ral ub nad an lan on ga hi do ede nd Fe sra an nd co nd in al ce rk sia
o rw bou Ja d S erm Fin Ioven Tu Au BelgZeaAustRep Ca me Po EstHun C ing Sw cela sian Itzerl erla exiIrela Spa rtugGreenmaone
N em ite G Sl w h CD dK I us i th M Po De Ind
Lu
x Un Ne Czec OE ite R Sw Ne
Un
1 2 http://dx.doi.org/10.1787/888933083658
Source: Authors calculations based on PISA (2006 and 2009)
Figure7.6 4th grade students in same-ability groups for reading, according to teachers
Percentage of students whose teachers often or always create same-ability groups when doing reading instruction or reading activities
% -4 -3 -3 -2 -2 -2 -1 -1 -1 -1 +0 +1 +1 +1 +1 +1 +1 +1 +1 +3 +4 +5 +5 +6 +7 +9 +12 +13
% point
80 80
70 70
60 60
50 50
40 40
30 30
20 20
10 10
0 0
* n e ly ic d ia k io a a ia c d. y h ic ia ** ** s ** * ** l* ** ** **
ary de nc Ita ubl olan nes mar tar nad Chin ven uebe n Fe man renc ubl ustr pain ean* tate ore* berta way srae nds land land
u ng Swe Fra ep P do en On Ca g, l o Q i a e r F ep A S m d S ap Al or I rla g ea
H hR In D n S ss G um k R e g N e En Z
ec Ko Ru lgi va CD nit Sin th w
Cz ng Be Slo OE U Ne Ne
H o
1 2 http://dx.doi.org/10.1787/888933083677

Figure7.7 4th grade students in mixed-ability groups for reading, according to teachers
Percentage of students whose teachers often or always create mixed-ability groups when doing reading instruction
or reading activities and change over time
% -70 -8 -3 -3 -2 -2 -2 -2 -1 -1 -1 -1 +0 +0 +1 +1 +1 +1 +1 +2 +2 +3 +3 +3 +4 +5 +5 +12
% point
75 75
73 73
71 71
69 69
20 15
15
10
10
5
5
0 0
h * l * * *** na a**
ec *** .** ds lic ay ce ae aly rta nd en nd ark an nd ch rio da ria ec nia ore y* tes n* i i
C ublic Fed rlanepub orw Fran Isr It Albe Pola wed ngla nm me eala Fren nta ana Aust ueblove gap man Sta Spai gary , Ch nes
z
e N S e D O C Q d n
p a n h
Re ssi Net vak
R E D E C
ew
Z
i u m S i n e
S G nitr e H u o ng Indo
O N lg K
Ru Slo Be
U ng
Ho
1 2 http://dx.doi.org/10.1787/888933083696
PISA (2006 and 2009) surveys asked school principals and heads of schools Some schools organise
instruction differently for students with different abilities. What is your schools policy about this for students
in <national modal grade for 15-year-olds>?[] b) Students are grouped by ability within their classes with
answer options For all subjects; For some subjects; Not for any subject. PIRLS (2001, 2006 and 2011) surveys
asked 4th grade teachers: When you have reading instruction and/or do reading activities, how often do you
organize students in the following ways? [] b) I create same-ability groups [] b) I create mixed-ability
groups with answer options Always or almost always; Often; Sometimes; Never. The same data restrictions
as in International TIMSS and PIRLS Reports by the International Association for the Evaluation of Educational

Self-directed work
General findings
Innovation in the classroom has also resulted in a change in the use of student self-directed work
during lessons in secondary education (Figure7.8 to Figure7.11 ). OECD average absolute change in
students being asked to decide procedures to solve complex mathematics problem was 8% points
between 2003 and 2011, equivalent to students own reporting figure between 2003 and 2007. As
for science classes, the average absolute change among OECD countries regarding students being
asked to design and plan experiments was 8% points between 2003 and 2011, higher than students
reported figure of 5% points in the 2003-2007 time frame. In particular, there was an increase in
students in Ontario (26% points) being asked to decide procedures for solving complex mathematics
problems. In Minnesota (29% points) and England (26% points) teachers increased the extent to
which they asked students to design or plan experiments. These education system changes exhibited
medium effect sizes. Overall, during mathematics lessons, student self-directed work increased in
seven education systems according to teachers reporting; it increased in one and decreased in
four education systems according to students own reporting, with at least small effect sizes. Self-
directed work during science classes increased in 12 countries according to teacher reporting, with
at least small effect sizes. However, within a shorter timeframe, it decreased significantly in one
education system, according to students own reporting.
Innovation in the classroom was also illustrated through increases and decreases in student
self-direct work during lessons in primary education (Figure7.12 to Figure7.15 ). Between 2003
and 2011 the average absolute change at OECD level for students being asked to design and plan
scientific experiments was 10% points, as for student reported data between 2003 and 2007. OECD
average absolute change was 11% points for students reading books of their own choice according to
teacher reporting, whereas it was 6% points according to student reporting between 2001 and 2011.
In Denmark (29% points), Singapore (28% points), Sweden (20% points) and Hong Kong (15% points)
teachers asked students to plan and design experiments far more; on the contrary, the Netherlands
(42% points) stood out as the country registering the largest decrease in this respect according
to students own reporting. Norway (34% points), Hong Kong (25% points) and Italy (24% points)
exhibited the largest increase in students choosing the books they would read during reading classes,
while, according to students own reporting, the Czech Republic (29% points) registered the largest
decrease in this respect. These education system changes presented large and medium effect sizes.
The OECD average absolute change in student reported self-directed work in science showed a small
effect size; whereas OECD average and average absolute changes exhibited small effect size with
regard to teacher reported self-directed work in science and reading and average absolute change in
science classes according to student reporting showed a small effect size. Altogether, with at least
small effect sizes, self-directed work in science increased in 14 countries, while it increased in one
education system and decreased in three according to student own reporting. In reading classes,
self-directed work increased in 16 countries while decreasing in one according to teacher reports;
student reported data registered an increase in one education system and decreases in five, with at
least small effect sizes.
Australia, New Zealand and South Africa are countries where innovation was illustrated in more
self-directed work across disciplines and levels. South Africa, for example, experienced a significant
increase in students being asked to decide procedures for solving complex mathematics in 8th grade
between 2003 and 2011, as well as in students being asked to read books they chose themselves
during 4th grade reading classes in the period from 2001 to 2011.

Figure7.8 8th grade maths students asked to decide on procedures for solving complex problems,
Percentage of students whose teachers ask them to decide on their own procedures for solving complex problems
% -9 -4 -4 -2 -1 -1 +0 +1 +2 +4 +5 +5 +6 +7 +7 +7 +8 +9 +12 +12 +14 +18 +22 +23 +26

% point
100 30
90
80 25
70
20
60
50 15
40
30 10
20
5
10
0 0
l* rea ec en tes nia na ed. aly ina *** tts ary sia re* an ay nd d** e*** a*** y*** a*** a** o***
ae b d a a F It Ch ean use ng ne apo Jap orw gla lan hil ric ke ali sot ari
Isr Ko Que Swe d St love Indi ian g, m ch Hu Indo ing N En Zea C Af Tur str ne nt
e S s
i t
Ru
s on CD ssa S th Au Min O
Un g K OE Ma N ew S ou
o n
H
1 2 http://dx.doi.org/10.1787/888933083715
Figure7.9 8th grade maths students asked to decide on procedures for solving complex problems,
according to students
Percentage of students deciding on their own procedures for solving complex problems in at least half their lessons
% -15 -14 -11 -11 -10 -8 -7 -7 -7 -6 -5 -5 -4 -2 -1 -1 +0 +3 +14

% point
100 16
90 14
80
12
70
60 10
50 8
40 6
30
4
20
10 2
0 0
*** * *
c** try** ael**
* *** *** *** *** *** n*
** d**
*
a**
* *** de
n a re ly d. ***
an be ea gary ates and ario ea tlan trali ay hin apo Ita n Fe enia
Jap Que un Isr K or n S t gl n t m o s o rw S we g, C n g ia v
o u n n i s l o
c H ite
d E O CD Sc Au N
Ko
S
Ru
s S
ue OE
sq Un ng
Ba Ho
1 2 http://dx.doi.org/10.1787/888933083734

Figure7.10 8th grade science students asked to design or plan experiments, according to teachers
Percentage of students whose teachers ask them to design or plan experiments or investigations in at least half their lessons
% -3 -3 -1 -1 -0 +1 +1 +2 +4 +5 +7 +8 +8 +9 +9 +10 +10 +12 +12 +13 +13 +14 +15 +26 +29
% point
50 35
45
40 30
35 25
30
20
25
20 15
15 10
10
5
5
0 0
a l n a c y y e ** y ** a n ** ** ** ** s ** ** a ** ** ** **
ric Israe ede veni ebe rwa urke Chil gary Ital an* nesi Japa rea Fed* ore* ina sett tes* and dian rio* lia* nd* ota*
Af Sw Slo Qu No T n e
m Indo Ko an gap , Ch chu Sta eal In nta stra ngla nes
u th H u i g a Z O u
So CD ss Sin on ass ited ew A E
Mi
n
OE Ru K M
Un N
ng
Ho
1 2 http://dx.doi.org/10.1787/888933083753
Figure7.11 8th grade science students asked to design or plan experiments, according to students
Percentage of students whose teachers ask them to design or plan experiments or investigations in at least half their lessons
% -11 -8 -6 -4 -3 -3 -1 +1 +2 +3 +3 +6 +6 +6 +8
% point
70 12
60 10
50
8
40
6
30
4
20
10 2
0 0
ue ** ** ** ** * y n* io c * a
e**
* *** *** d**
*
sq ry* el* n* ly* nd wa ea ar be es ali ea ina
B unt Isra
a pa Ita la r nt ue tat str or or an
Ja ng No m O Q S u a p K C h t l
co E CD ite
d A g g, Sc
o
OE Un Sin on
n gK
Ho
1 2 http://dx.doi.org/10.1787/888933083772

TIMSS and PIRLS (2003, 2007 and 2011) surveys asked teachers In teaching mathematics/science to this
class, how often do you usually ask students to do the following? [] i) Decide on their own procedures for
solving complex problems []c) Design or plan experiments or investigations with answer options Every
or almost every lesson; About half the lessons; Some lessons; Never.. TIMSS (2003 and 2007) surveys asked
students: How often do you do these things in your mathematics/science lessons? [] i)

Figure7.12 4th grade science students asked to design or plan experiments, according to teachers
Percentage of students whose teachers ask them to design or plan experiments or investigations at least once a month
% -5 -1 -1 +1 +1 +3 +6 +7 +7 +8 +9 +9 +10 +12 +14 +15 +15 +15 +16 +18 +19 +20 +28 +29
% point
80 30
70
25
60
50 20
40 15
30
10
20
5
10
0 0
lic and nds tria ish way ry*** enia pan d.*** ic*** n*** y*** lia** es*** rta** a*** ly*** nd** io*** ec*** n*** re*** rk***
ub l a s m r Ja Fe ubl ea an tra tat lbe hin Ita gla tar eb ede apo ma
ep Zea herl Au Fle No unga Slov
R
k ew e t m H s i an Rep D m Germ Aus ed S A g, C En On Qu Sw ing Den
a u s C n
Slo
v N N
e lgi Ru zech OE U nit g Ko
S
B C n
Ho
1 2 http://dx.doi.org/10.1787/888933083791
Figure7.13 4th grade science students asked to design or plan experiments, according to students
Percentage of students who design or plan experiments or investigations in at least half their lessons and change over time
% -42 -19 -15 -9 -8 -8 -8 -7 -6 -2 -2 -2 -1 +1 +3 +12

% point
100 45
90 40
80 35
70
30
60
25
50
20
40
30 15
20 10
10 5
0 0
*** *** ** *** *** *** ** d**
* ** ly nd re* ina tes c **
ds an io* nd lia ay n* ry* Ita be ia*
an ar tra ea lan ga gla po Ch Sta ue en
erl Jap On
t otl
a
s o rw m a n En ing
a
ng
, d Q v
th Sc Au N CD Ze Hu S o ite Slo
Ne OE Ne
w
n gK Un
Ho
1 2 http://dx.doi.org/10.1787/888933083810

We decide on our own procedures for solving complex problems []c) We design or plan an experiment or
investigation with answer options Every or almost every lesson; About half the lessons; Some lessons; Never.

Figure7.14 4th grade students reading books that they choose themselves, according to teachers
Percentage of students whose teachers give them time to read books of their own choosing at least once a month and change over time
% -13 -12 -12 -9 -1 -1 +1 +1 +2 +2 +3 +4 +5 +6 +6 +7 +7 +8 +9 +11 +12 +14 +16 +17 +19 +22 +24 +25 +34
% point
100 40
90 35
80
30
70
60 25
50 20
40 15
30 10
20
10 5
0 0
** ** *** e* d ta d n en es in re s** ** c* *** lic *** *** *** c** ** *** *** *** *** *** *** ***
nia ch nd nc lan er lan tio d at pa po d rio be da ub an rk el li sia ry ria ny ca ly na ay
l ove Fren ola Fra Zea Alb Eng dera Swe d St S inga erlan nta Que ana Rep me nma Israepub one nga ust rma Afri Ita Chi orw
S m P Fe ite O C ch CD De d u A e th , N
u Ne
w
Un
S th k R In H G ou ng
e lgi s ian Ne C ze OE ova S Ko
B s S l n g
Ru Ho
1 2 http://dx.doi.org/10.1787/888933083829
Figure7.15 4th grade students reading books that they choose themselves, according to students
Percentage of students reading books that they choose themselves at least once a month and change over time
% -29 -16 -13 -12 -12 -10 -6 -4 -4 -2 -2 -1 -1 -1 -0 +0 +0 +1 +1 +1 +2 +2 +3 +4 +6 +8 +9 +14

% point
100 35
90
30
80
70 25
60
20
50
40 15
30 10
20
5
10
0 0
h * *** *** .*** *** *** *** y** el *** e** nd ria es in ry na nd a** *** k* *** *** *** *** *** ***
ec *** c** l a t a a i c r s
C ublicubli any enia Fed land ance ench Ita Isr ean apor eala Aust Sta Sp ung , Ch ngla lbert ebe nma tario eden and esia fricarway
z
p p r m ov n Po Fr Fr m ng Z ed H ng E A Qu De On Sw erl don h A No
Re k Re Ge Sl ssia C D i
S New i t o h In out
Ru um OE Un K
Ne
t
va lgi ng S
Slo Be Ho
1 2 http://dx.doi.org/10.1787/888933083848

TIMSS (2003 and 2007) surveys asked students: In school, how often do you do these things? []c) I design or plan
a science experiment or investigation [] with answer options At least once a week; Once or twice a month; A few
times a year; Never. PIRLS (2001, 2006 and 2011) asked 4th grade teachers: When you have reading instruction and/or
do reading activities with the students, how often do you do the following? [] d) Give students time to read books of
their own choosing with answer options Every day or almost every day; Once or twice a week; Once or twice a month;
Never or almost never. PIRLS (2001, 2006 and 2011) surveys also asked students: In school, how often do these things
happen? [] b) I read books that I choose myself with answer options Every day or almost every day; Once or twice a
week; Once or twice a month; Never or almost never. The same data restrictions as in International TIMSS and PIRLS
Reports by the International Association for the Evaluation of Educational Achievement (IEA) apply to these data.

Individualisation
General findings
Innovation in the classroom has also been indicated through an increase in individualisation
practices in primary and secondary education (Figure7.16 to Figure7.18). Between 2000 and 2009 the
OECD average absolute change in the ability of teachers to meet individual students needs amounted
to 10% points, while it amounted to 5% points with regard to student perception of their teachers
willingness to help students individually in secondary education. In primary education, OECD average
absolute change concerning the regular use of individualised instruction during reading classes was
17% points between 2001 and 2011. Greece (38% points), Indonesia (26% points) and Portugal (25%
points) are the countries where teachers increased their capacity to address students needs the
most. Likewise, Israel (47% points), Hungary (32% points), the Slovak Republic (29% points) and the
United States (27% points) outranked all other countries by using regular individualised instruction
more widely than before. These education system changes showed from medium to large effect
sizes, while OECD average absolute effect size was small concerning student perception of teachers
willingness to help. The OECD average and average absolute change effect sizes in the regular use
of individualised instruction in reading were small. Overall, teachers responsiveness to individual
students needs increased in 11 countries with at least small effect sizes, while student perception
of teachers willingness to help increased in four. Regular use of individualised instruction increased
with at least small effect sizes in 13 education systems and decreased in five.

Figure7.16 Teachers meeting individual students needs in secondary education

Percentage of 15 year-old students in schools where the principal reports that learning is not at all hindered, or hindered very little, by
teachers not meeting individual students needs and change over time
% -8 -8 -4 -4 -4 -3 -2 -1 -1 -1 -1 +1 +2 +3 +4 +4 +6 +6 +7 +7 +12 +13 +14 +14 +15 +17 +18 +20 +25 +26 +38
% point
100 40
90 35
80
30
70
60 25
50 20
40 15
30
10
20
10 5
0 0
* s y a a c * e a * y l * * * * * * * * * * *
an and nd Ital stri arknad ubli nd**Chil exicolandedentrali giumlandlandan** an srae tes* nd** rg** m** ry** in** ed.** ay** gal** sia** ce**
JapZeal erla A u enm Ca ep ela M Fin Sw us Bel tzer Ire me erm I Sta ola bou gdo nga Spa n F orw rtu ne ree
w eth D h R Ic A
Sw
i
CD
G
ite
d P em Kin Hu
ss
ia N Po ndo G
Ne N Cz
ec OE Un
x
Lu ited Ru
I
U n
1 2 http://dx.doi.org/10.1787/888933083867
Notes: *** = change significant at the 0.01 level; ** = change significant at the 0.05 level; * = change significant at 0.1 level; OECD average includes all
OECD education systems for which data is available for all years concerned.
Source: Authors calculations based on PISA (2000, 2003 and 2009)
Figure7.17 Student perception of teacher willingness to help them individually in secondary education
Percentage of 15 year-old students who agree or strongly agree that if they need extra help, they will receive it from their teachers and
change over time
% -3 -2 -1 -1 -1 +0 +1 +1 +1 +1 +1 +2 +2 +2 +3 +3 +3 +3 +4 +4 +4 +4 +4 +5 +5 +5 +7 +7 +8 +8 +11 +12 +16 +16

% point
100 18
90 16
80 14
70 12
60
10
50
8
40
30 6
20 4
10 2
0 0
l * * * * * * * * * * * * * * * * * * * * * * * * *
ae rk co lia an ain om a* ece nd ay en ds e* y** g** d** d** .** y** ** d** a** s** ** c** a** e** d** a** y** l** g** **
Isrnma exi stra Jap Sp ngd nadGre inla orwwedrlan rancItal Kon lan rlan Fed gar eanlannesi tategiumubli ore Chil lan stri antuga our land
M u i a F N S e F e n n m a S l K e u m r b o
De A dK C th ng Ir itzessia Hu CD ZeIndoted Be Rep Ic A Ger Po em P
ite Ne Ho Sw Ru OE New Uni ec
h
Lu
x
Un Cz
1 2 http://dx.doi.org/10.1787/888933083886

Figure7.18 4th grade students receiving individualised instruction in reading, according to teachers
Percentage of students whose teachers always or almost always use individualised instruction for reading and change over time
% -27 -20 -20 -15 -13 -7 -5 -2 -2 -2 +1 +3 +4 +8 +8 +9 +11 +11 +12 +15 +17 +19 +20 +23 +27 +29 +32 +47
% point
80 50
70 45
40
60
35
50 30
40 25
30 20
15
20
10
10 5
0 0
** ** ** ** ** n y y ly ia h e k * c* ** ** ** ** ** ** ** ** ** ** ** ** **
ia* d* ic* d.* na de wa an Ita nes renc ranc mar land ebe stria ore* ean* ain* ada* and* rta* rio* nds* tes* blic* ary*rael*
v en olanpubl n Fe , Chi Swe Nor erm d o F F en ea Qu u ap m Sp an gl lbe nta rla Sta pu ng Is
Sl o P e ia g G In ium D w Z A g D C En A O he ed Re Hu
h R ss on lg Ne Sin EC t t k
ec Ru ng K Be O Ne Uni ova
Cz o Sl
H
1 2 http://dx.doi.org/10.1787/888933083905
2011 instead of 2001 and 2011 for Austria, Belgium French, Canada (Alberta), Denmark, Indonesia, Poland and Spain. OECD average includes all OECD
education systems for which data is available for all years concerned.

PISA (2000, 2003 and 2009) surveys asked secondary principals In your school, to what extent is the learning
of students hindered by the following phenomenon? [] e) Teachers not meeting individual students needs
with answer options Not at all; Very little; To some extent; A lot. They also asked secondary students: How
much do you disagree or agree with each of the following statements about teachers at your school? [] d) If
I need extra help, I will receive it from my teachers with answer options Strongly disagree; Disagree; Agree;
Strongly agree.
PIRLS (2001, 2006 and 2011) surveys asked 4th grade teachers: When you have reading instruction and/or
do reading activities, how often do you organize students in the following ways? [] d) I use individualized
instruction for reading with answer options Always or almost always; Often; Sometimes; Never. The
same data restrictions as in International TIMSS and PIRLS Reports by the International Association for the
Evaluation of Educational Achievement (IEA) apply to these data.

Summary
Innovation or significant change in instructional practices has been illustrated through an
increase in the extent of student self-directed work across disciplines and educational levels
between the analysed time frames. Similarly, innovation in instructional practices in relation to
students working in small groups resulted in an increase across levels in mathematics and reading
and a decrease in science.
Figure7.19 Change in 8th grade students working in small groups, according to students
Group work- science Group work- maths

%
80
60
40
20
-20
es ntry alia way l c
ga
ry ly n
Ita ratio ede Stat
n nia ina orea tario land srae land pore ebe apan ean eannge)
n w o u str Nor ove , Ch K n o t I n g g a u J m m ha
Hu d e S d
ite que
c Au S l g O S c E Sin
Q D
EC ECDbs. c
n Fe n Kon O
i a U a s g O ( a
ss B n
Ru Ho
1 2 http://dx.doi.org/10.1787/888933083924
Figure7.20 Change in 4th grade students working in small groups, according to students
Group work- science (2003-2007) Group work- reading (2001-2006)

%
40
20
-20
-40
y l y c c e s e s
ian on an Ital eden tland land srae an ubli ebe anc tate por land land gary and tario rway enia hina ean ean ge)
ss ati Jap I Fr d S nga Zea Eng Hun erl On No Slov g, C D m D m han
Rueder Sw Sco Ice e rm Rep Qu
G k ite Si ew th n C C .c
F va Un N Ne Ko OE OE(abs
Slo ng
Ho
1 2 http://dx.doi.org/10.1787/888933083943
Notes : For details please see Figure7.1, 7.2, 7.3 and 7.4

Figure7.21 Change in 8th grade students being asked to decide and design procedures
Plan experiment - science Decide procedures- maths

%
60
40
20
-20
nia en bec Italy pan way rael tion gary ates iana orea hina hile esia pore land land ralia ario ean ean ge)
ve wed Que Ja r Is era un St Ind C n a a g t t n
S l o S No d H ited
K
g,
C do ing Ze En Aus On CD mCD mcha
Fe on In S ew E OE bs.
n U n K N O
ia ng (a
ss Ho
Ru
1 2 http://dx.doi.org/10.1787/888933083962
Figure7.22 Change in 4th grade students being asked to decide and design procedures
Plan experiment - science (2003 - 2007) Choose books - reading (2001 - 2011)
%
60
40
20
-20
n d s n c n e l c a s d y a y o c e y y a n
um ish de lan and apa ubli atio anc srae ubli trali tate lan an veni gar tari ebe por rwa Ital hin ea eannge)
lgi m e a l J ep er Fr I ep
us ed S Eng Germ Slo Hun On Qu inga No C m m
Be Fle Sw Ze ther R e d R A t S n g, CD CD . cha
w Ne h k i Ko OE OE abs
Ne ec nF va Un
Cz ssia Slo ng (
Ru Ho
1 2 http://dx.doi.org/10.1787/888933083981
Notes : For details please see Figure 7.8, 7.10, 7.12 and 7.14
Note on Israel

Table 7.1. Effect sizes for changes in the use of grouping, self-directed work
and individualisation in classrooms
Group work In-class ability Change in students self-directed work Individualisation

grouping
8th grade 4th grade 8th 4th grade 8th grade 4th grade 8th grade 4th
grade grade grade
Design and plan experiments

Reading mixed ability groups
Reading same ability groups
Decide procedures in maths
Choose books to read
instruction in reading
Use of individualised
Choose book to read
willingness to help
Teachers meeting
individual needs
(student resp.)
(student resp.)
School policy
Perception of
Mathematics
in science
in science
Reading
Science
Science
03-07 03-07 01-06 03-07 06-09 01-11 01-11 03-11 03-07 03-11 03-07 03-11 03-07 01-11 01-11 00-09 00-09 01-11
Australia 0.26 0.00 m -0.14 0.01 m m 0.47 -0.11 0.33 0.06 0.26 -0.17 m m 0.06 -0.03 m
Austria m m m m -0.05 m m m m m m m m m m -0.09 0.17 m
Belgium m m m m -0.04 m m m m m m m m m m 0.08 0.13 m
Belgium Flemish m m m m m m m m m m m 0.08 m m m m m m
Canada m m m m 0.05 -0.04 0.07 m m m m m m 0.30 m -0.05 0.03 0.35
Ontario 0.29 0.00 0.36 -0.16 m -0.05 0.07 0.54 -0.14 0.34 0.01 0.43 -0.30 0.29 0.14 m m 0.43
Quebec 0.47 -0.08 0.02 -0.19 m 0.05 0.08 -0.07 -0.29 -0.01 0.04 0.43 0.07 0.25 0.08 m m 0.21
Chile m m m m 0.09 m m 0.31 m 0.05 m m m m m -0.02 0.15 m
Czech Republic m m m m 0.01 -0.16 -1.71 m m m m m m 0.15 -0.59 -0.04 0.12 -0.42
Denmark m m m m 0.78 m m m m m m m m m m -0.10 -0.05 m
Estonia m m m m 0.07 m m m m m m m m m m m m m
Finland m m m m -0.11 m m m m m m m m m m 0.02 0.03 m
France m m 0.28 m m -0.15 -0.13 m m m m m m -0.20 -0.14 m 0.05 0.08
Germany m m 0.15 m -0.16 0.10 0.24 m m m m m m 0.38 -0.28 0.15 0.22 -0.05
Greece m m m m 0.69 m m m m m m m m m m 0.78 0.02 m
Hungary 0.03 m 0.41 0.00 0.09 -0.20 0.30 0.14 -0.17 0.14 m 0.30 -0.12 0.35 0.01 0.45 0.09 0.66
Iceland m m 0.12 m 0.15 m m m m m m m m m m -0.02 0.19 m
Ireland m m m m 0.30 m m m m m m m m m m 0.12 0.08 m
Israel 0.18 -0.19 -0.11 m 0.16 0.23 -0.05 -0.20 -0.23 -0.06 -0.16 m m 0.30 -0.04 0.14 -0.06 1.01
Italy 0.11 -0.05 0.02 0.00 -0.11 -0.29 -0.07 0.05 0.00 0.16 -0.13 0.32 -0.05 0.52 -0.08 -0.10 0.07 -0.04
Japan 1.64 -0.08 m -0.03 -0.19 m m 0.16 -0.32 0.17 -0.12 0.14 -0.42 m m -0.19 -0.02 m
Korea 0.07 -0.22 m m m m m -0.09 -0.20 0.22 0.17 m m m m m 0.17 m
Luxembourg m m m m -0.25 m m m m m m m m m m 0.28 0.33 m
Mexico m m m m 0.21 m m m m m m m m m m -0.01 -0.04 m
Netherlands m m 0.01 -0.47 0.19 0.29 -0.12 m m m m -0.06 -0.96 0.26 0.19 -0.09 0.06 0.47
New Zealand m m -0.10 -0.19 0.00 0.27 0.06 0.24 m 0.30 m -0.03 -0.13 -0.08 -0.03 -0.15 0.12 0.17
Norway 0.13 -0.13 0.42 -0.09 -0.58 0.25 -0.13 0.16 -0.07 0.02 -0.06 0.12 -0.16 0.88 0.38 0.40 0.03 -0.10
Poland m m m m 0.07 m m m m m m m m m m 0.34 0.34 m
Portugal m m m m 0.41 m m m m m m m m m m 0.52 0.33 m
Slovak Republic m m -0.21 m m 0.12 -0.13 m m m m m m 0.26 -0.35 m m 0.60
Slovenia 0.36 m 0.26 0.30 -0.11 0.17 0.10 -0.01 0.28 -0.03 m 0.17 0.25 -0.28 -0.32 m m -0.59
Spain m m m m 0.36 m m m m m m m m m m 0.40 0.00 m
Basque country -0.08 -0.13 m m m m m m -0.22 m -0.25 m m m m m m m
Sweden 0.22 m -0.03 m 0.13 -0.17 -0.03 -0.04 -0.03 -0.03 m m m 0.11 0.14 0.04 0.05 -0.17
Switzerland m m m m 0.17 m m m m m m m m m m 0.10 0.09 m
Turkey m m m m -0.05 m m m m m m m m m m m m m
United Kingdom m m m m 0.11 m m m m m m m m m m 0.31 0.02 m
England 0.40 -0.03 0.25 -0.06 m 0.25 -0.03 0.17 -0.14 0.63 -0.07 0.32 -0.05 0.05 0.03 m m 0.37

Table 7.1. Effect sizes for changes in the use of grouping, self-directed work
and individualisation in classrooms (continued)
Group work In-class ability Change in students self-directed work Individualisation

grouping
8th grade 4th grade 8th 4th grade 8th grade 4th grade 8th grade 4th
grade grade grade

Reading mixed ability groups
Reading same ability groups
Choose books to read
Choose book to read
willingness to help
Teachers meeting
individual needs
(student resp.)
(student resp.)
School policy
Perception of
Mathematics
in science
in science
Reading
Science
Science
03-07 03-07 01-06 03-07 06-09 01-11 01-11 03-11 03-07 03-11 03-07 03-11 03-07 01-11 01-11 00-09 00-09 01-11
Scotland 0.36 -0.03 -0.02 0.07 m m m m -0.11 m 0.15 m -0.19 m m m m m
United States 0.16 0.03 0.19 -0.05 -0.16 0.10 0.11 -0.02 -0.14 0.25 0.06 0.31 0.02 0.11 -0.02 0.26 0.13 0.58
Indiana m m m m m m m 0.00 m 0.29 m m m m m m m m
OECD (average) 0.31 -0.08 0.12 -0.08 0.08 0.06 0.02 0.10 -0.13 0.16 -0.02 0.23 -0.17 0.21 -0.02 0.14 0.10 0.24
OECD (av. absolute) 0.32 0.08 0.17 0.13 0.19 0.18 0.11 0.16 0.16 0.18 0.11 0.24 0.22 0.29 0.15 0.11 0.20 0.37
Hong Kong, China 0.27 -0.08 0.52 0.12 m 0.01 0.15 0.07 -0.02 0.26 0.13 0.52 -0.02 0.52 0.02 m 0.09 -0.27
Indonesia 0.34 m m m 0.83 m m 0.15 m 0.18 m m m m m 0.63 0.11 m
Russian Federation 0.23 m 0.02 m 0.15 0.04 -0.29 0.02 0.05 0.25 m 0.25 m 0.02 -0.28 0.37 0.09 -0.33
Singapore 0.40 -0.06 0.18 -0.05 m 0.23 0.08 0.15 -0.01 0.32 0.13 0.68 -0.05 0.13 -0.04 m m 0.31
South Africa m m m m m m m 0.28 m -0.06 m m m m m m m m
1 2 http://dx.doi.org/10.1787/888933084000
Source: Authors calculations based on TIMSS (2003, 2007 and 2011), PIRLS (2001, 2006 and 2011) and PISA (2000, 2006 and 2009).

CHAPTER 8
Innovation
in the use of textbooks in classrooms
Innovation in the classroom can also incorporate different approaches to the

use of textbooks as instructional material. Teachers may choose to make more
or less use of textbooks either as a basis for instruction or as supplementary
tools. The aim of innovation with regard to the use of textbooks could be, for
example, to align classroom curriculum with standards through a stricter
or a more lenient adherence with textbook contents, while reduced use of
textbooks could reflect an intention to introduce alternative sources such as
Open Educational Resources, or to champion more active pedagogies.

8. INNOVATION IN THE USE OF TEXTBOOKS IN CLASSROOMS
Use of textbooks as primary resources
General findings
Innovation in the classroom has been illustrated through an increased use of textbooks as a
basis for instruction during mathematics and science classes in secondary education (Figure8.1 and
Figure8.2). Between 2003 and 2011 the average absolute change within the OECD area regarding the
use of textbooks as a basis for instruction amounted to 15% points in both 8th grade mathematics
and science. Turkey (42% points), Israel (40% points), South Africa (37% points), Slovenia (35%
points), Italy (34% points), Indonesia (21% points), and Chile (19% points) are countries registering
the largest increases in the use of textbooks as primary resources for mathematics instruction.
Likewise, textbooks were employed considerably more extensively as a basis for science instruction
in Indonesia (75% points), Turkey (37% points), Sweden (35% points), South Africa (31% points), Israel
(29% points) and Slovenia (25% points). These changes were characterised by large and medium
effect sizes, while the OECD average and average absolute changes presented small effect sizes in
8th grade mathematics and science . Overall and with at least small effect sizes, textbooks were used
more widely as a basis for mathematics instruction in ten education systems and less widely in four,
while they were used more widely for science instruction in 14 countries and less in three, with at
Classroom innovation regarding the use of textbooks as a basis for instruction in primary
education showed slightly more variation than those characterising secondary education (Figure8.3
to Figure8.5). The OECD average absolute change regarding use of textbooks as primary resources
for instruction was 14% points in 4th grade science and 13% points in 4th grade mathematics, between
2003 and 2011. It amounted to 8% points in the case of 4th grade reading instruction, although
within a shorter time period from 2001 to 2006. Slovenia (63% points), Italy (38% points), Austria
(32% points), Norway (30% points), the Slovak Republic (30% points) and the Czech Republic (25%
points) were the leading education systems in terms of an increase in the use of textbooks as a
basis for science instruction. Similarly, textbooks were used more widely as a basis for mathematics
instruction in Italy (34% points), the Slovak Republic (32% points) and Slovenia (32% points); while
opposite was true in Flanders (46% points). The Netherlands (43%) stood out as the country that used
textbooks more widely as a basis for reading instruction in the 4th grade. These education system
changes presented from medium to large effect sizes; while the OECD average absolute change
effect sizes across all subjects were small and both average and average absolute change effect
sizes were small for science. Altogether, textbooks were used more as primary science instruction
resources in 12 education systems and less in one, with at least small effect sizes, while in the case
of mathematics instruction textbooks were more widely used in eight countries and less in five.
As for reading instruction, the number of countries (three) exhibiting an increase in textbook use
equalised that of those countries diminishing use, with at least small effect sizes.
Innovation in Italy and Slovenia is indicated through a comprehensive increase in the use of
textbooks as a basis for instruction across disciplines and levels. In Slovenia, for example, textbooks
were used more widely as primary resources for mathematics, science and reading instruction both
in 8th and 4th grade, with at least small effect size.
In contrast, innovation in England manifested through a lower use of textbooks as a basis

for instruction across disciplines and levels. English teachers used textbooks less as a basis for
mathematics, science and reading instruction in 2011 than in 2003 in both primary and secondary
education, with small effect sizes.

Figure8.1 Textbooks as a primary resource for 8th grade mathematics instruction

Percentage of students whose teachers use textbooks as a basis for instruction and change over time
% -20 -17 -16 -15 -7 -4 -2 -2 +2 +2 +5 +5 +5 +7 +7 +8 +9 +12 +19 +21 +34 +35 +37 +40 +42
% point
100 45
90 40
80 35
70
30
60
25
50
40 20
30 15
20 10
10 5
0 0
** ** ** ** a d c ts d. y io a ia ** n ** ** ** ** ** ** ** ** ** **
na nd es* re* sot lan ebe set Fe rwa tar hin ral en Japa rea* ean* gary hile* sia* taly* nia* rica* rael* key*
dia ngla Stat gapo inne Zea Qu chu sian No On g, C Aust wed Ko m un C n e I v e Af I s Tur
In E ed in a n S o o
Sl uth
it S M ew ss Rus Ko CD H In
d
Un
N Ma ng OE So
Ho
1 2 http://dx.doi.org/10.1787/888933084019
Figure8.2 Textbooks as a primary resource for 8th grade science instruction

% -21 -12 -8 -4 -4 +2 +5 +5 +7 +9 +10 +10 +12 +13 +14 +15 +16 +21 +23 +25 +29 +31 +35 +37 +75
% point
100 80
90 70
80
70 60
60 50
50 40
40
30
30
20 20
10 10
0 0
** a d a s c y d ta etts ea** an* rio* n*** lia** .*** ly*** y*** le*** ia*** el*** a*** n*** y*** ia***
re* ian lan hin ate ebe wa lan so p d r i c e
a po Ind Eng g, C d St Qu Nor Zea nne chus Kor Ja Onta mea stra n Fe Ita nga Ch ven Isra Afri ede urk nes
i Slo Sw T do
Sing K o n ite
n ew M
a s s a
E C D Au sia
s H u
u th I n
ng U N M O R u S o
Ho
1 2 http://dx.doi.org/10.1787/888933084038

Figure8.3 Textbooks as a primary resource for 4th grade mathematics instruction

% -46 -17 -15 -9 -4 -3 +0 +0 +2 +3 +4 +6 +6 +8 +8 +9 +10 +10 +11 +12 +12 +32 +32 +34
% point
100 50
90 45
80 40
70 35
60 30
50 25
40 20
30 15
20 10
10 5
0 0
* * * * * . * * * * * * * * * * * *
g i umish** nd** tes** nd** nds eden ebec ark erta Fed pore an** pan alia any* ina* tario ay** ary* blic* ria** ia** lic** ly**
e l m la ta
B le ng S l a
a he r l a w Qu
S e n m lb an
A i n g a
m
e Ja ustr rm Ch On orw ng pu ust ven pub
, u Ita
F E d Z e t D s s i
S CD A e
G on g N H hR e A l
S kRo e
ite w Ne Ru OE K ec
Un Ne ng Cz lov
a
H o S
1 2 http://dx.doi.org/10.1787/888933084057
Figure8.4 Textbooks as a primary resource for 4th grade science instruction

% -17 -7 -3 -2 -1 +0 +1 +1 +4 +5 +8 +8 +9 +10 +11 +12 +14 +18 +25 +30 +30 +32 +38 +63
% point
100 70
90 60
80
70 50
60 40
50
40 30
30 20
20
10 10
0 0
*** re rta* and nds ario tes and alia pan ary* ark ish* na** n*** d.*** n*** y*** ic*** ic*** y*** ia*** ly*** ia***
ec po l a l r Ja ng nm lem Chi ea Fe ede man ubl ubl rwa str Ita ven
eb nga Albe Eng erla Ont d St Zea ust
Q u Si th ite ew A Hu De m F ng, m
D ssian Sw Ger Rep Rep No A
u
Slo
Ne U n N g u
i gKo
O E C u ch vak
l R e
Be Hon Cz Slo
1 2 http://dx.doi.org/10.1787/888933084076

Figure8.5 Textbooks as a primary resource for 4th grade reading instruction

% -19 -12 -11 -9 -7 -5 -4 -3 -3 -0 +1 +1 +3 +3 +3 +5 +5 +6 +10 +10 +43

% point
100 50
90 45
80 40
70 35
60 30
50 25
40 20
30 15
20 10
10 5
0 0
* * * *
d** ance land ates ede
n ny* ore ina bec lic and ean ed.* ario rael gary way ly** and ia** s***
an a p h e ub l F t Is un r Ita Icel en nd
tl Fr Eng d St S w erm nga g, C Qu Rep Zea CD m ian On H No lov erla
Sco i t e G S i o n a k ew E s s S h
K O Ru t
Un ng Slo
v N Ne
Ho
1 2 http://dx.doi.org/10.1787/888933084095

TIMSS (2003, 2007 and 2011) surveys asked 4th and 8th grade teachers When you teach mathematics/
science to this class, how do you use the following resources? [] a) textbooks with answer options Basis
for instruction; Supplement; Not used. PIRLS (2001 and 2006) survey asked 4th grade teachers: When you
have reading instruction and/or do reading activities with the students, how often do you use the following
resources? [] a) textbooks with answer options Every day or almost every day; Once or twice a week;
Once or twice a month; Never or almost never. The same data restrictions as in International TIMSS and
PIRLS Reports by the International Association for the Evaluation of Educational Achievement (IEA) apply
to these data.

Use of textbooks as supplementary resources
General findings
Innovation in classrooms was exemplified also by a reduction in the use of textbooks as
supplementary resources for instruction in secondary education (Figure8.6 and Figure8.7). At the
OECD level, average absolute change regarding the use of textbooks as supplementary resources for
mathematics and science instruction amounted to 13% points between 2003 and 2011. Textbook use
as a supplementary resource for mathematics instruction diminished strikingly in Israel (38% points),
Turkey (36% points), Slovenia (35% points), South Africa (33% points), Italy (31% points) and Indonesia
(22% points). As for science instruction, textbooks were used significantly less as supplementary
resources in Turkey (34% points), Sweden (33% points), Israel (28% points), South Africa (28% points)
Slovenia (26% points), Hungary (22% points) and Indonesia (20% points). These country changes
presented large and medium effect sizes; while the OECD average and average absolute change
effect sizes weresmall for textbook use as a supplementary resource for mathematics and science
instruction. Overall, textbooks were used more as supplementary resources for mathematics
instruction in three education systems and less in ten, with at least small effect sizes, while in the
case of science instruction textbooks were more widely used in two countries and less in 11.
In comparison with secondary education, innovation resulted in some reduction in textbook

use as a supplementary resource in primary education instruction (Figure8.8 and Figure8.9). The
average absolute change at OECD level in the use of textbooks as supplements for both mathematics
and science instruction was 11% points in the period from 2003 to 2011. The Slovak Republic (33%
points), Slovenia (31% points), and Italy (25% points) exhibited the largest decrease in the use of
textbooks to supplement mathematics instruction, whereas the opposite was true in the Flanders
(24% points). Textbooks were used significantly less as supplements to science instruction in Slovenia
(46% points), Italy (33% points), the Slovak Republic (30% points), the Czech Republic (27% points)
and Norway (25% points), whereas New Zealand (29% points) and Quebec (23% points) showed an
increase in supplementary use of textbooks in science classes. These education system changes
exhibited large and medium effect sizes, while the OECD average absolute change effect size was
small. Altogether, textbooks were used more widely as a supplement for mathematics instruction
in three education systems and less widely in ten, while their use as a supplement for science
instruction increased in five countries and decreased in 11, with at least small effect sizes.
Innovation regarding supplementary use of textbooks in the United States and Singapore is
indicated through a significant increase across disciplines in primary and secondary education. In
the United States, for example, teachers used textbooks as supplementary instruction tools for both
4th grade mathematics and science classes between 2003 and 2011, with small effect sizes.
Italy and Slovenia, on the contrary stood out as countries where innovation exemplified in a lower
use of textbooks for supplementary instruction across disciplines and levels. Italian teachers used
textbooks significantly less as supplementary resources for mathematics and science instruction in
4th and 8th grade between 2003 and 2011, with at least small effect sizes.
The direction of innovation concerning textbook use was not consistent across disciplines and
levels in England. English teachers used textbooks more widely as supplementary resources for 8th
grade mathematics instruction and less for 4th grade science instruction between 2003 and 2011.

Figure8.6 Textbooks as a supplementary resource for 8th grade mathematics instruction

Percentage of students whose teachers use textbooks as a supplementary resource and change over time
% -38 -36 -35 -33 -31 -22 -12 -9 -7 -7 -6 -6 -5 -5 -3 -3 -2 -2 -2 +1 +7 +9 +12 +13

% point
+18
80 40
70 35
60 30
50 25
40 20
30 15
20 10
10 5
0 0
* * * * * * * * * * . * * * *
l** ey** ia** ca** ly** ia** ary* an** pan en** tario hina hile rea* setts ralia way Fed ebec land sota tes* re** ana nd*
ae n i Ita es Ja ed On , C C Ko u ust Nor ian Qu Zea nne Sta apo ndi gla
Isr Turk love Afr o n ung me w g ch A s i g I n
S uth In
d H CD S
Ko
n
ss
a
Ru
s w M ited Sin E
So OE ng Ma Ne Un
Ho
1 2 http://dx.doi.org/10.1787/888933084114
Figure8.7 Textbooks as a supplementary resource for 8th grade science instruction

% -34 -33 -28 -28 -26 -22 -20 -16 -16 -15 -11 -10 -8 -7 -6 -6 -6 +2 +3 +3 +3 +4 +5 +11 +12
% point
80 40
70 35
60 30
50 25
40 20
30 15
20 10
10 5
0 0
*** *** *** *** *** *** *** *** *** *** *** io n ts ea ta y* ia ec d a d es a ***
ey den rael rica nia ary esia hile taly Fed. ean ntar Japa uset Kor eso rwa stral ueb alan Chin glan Stat dian ore
rk Is Af love ung on I n n No e g, n d n gap
Tu Swe h H Ind
C
ia CD
m O
a ch i n Au Q Z
on
E
ite
I
ut S ss ss M w
Ne ng K Un Sin
So Ru OE Ma
Ho
1 2 http://dx.doi.org/10.1787/888933084133

Figure8.8 Textbooks as a supplementary resource for 4th grade mathematics instruction

% -33 -31 -25 -15 -12 -12 -11 -11 -10 -9 -9 -7 -7 -5 -5 -3 +1 +3 +3 +3 +4 +14 +18 +24
% point
100 40
90 35
80
30
70
60 25
50 20
40 15
30
10
20
10 5
0 0
k .
va c*** a*** y*** c*** y*** a*** re** y*** rio* y*** lia* na* n*** pan ebec Fed ark land den nds* land es*** ta*** sh***
Sloubli veni Ital ubli gar stri apo an nta rwa stra Chi ea Ja Qu an m ea e a g t r i
p p n u g
A Sin Ger m O o u , m i e n
D wZ Sw herl En Sta Albe lem
Re Slo h Re Hu N A ong CD
R us
s
e et t ed F
K E N N i m
ec ng O Un lgi
u
Cz Ho Be
1 2 http://dx.doi.org/10.1787/888933084152
Figure8.9 Textbooks as a supplementary resource for 4th grade science instruction

% -46 -33 -30 -27 -25 -22 -12 -10 -10 -9 -8 -5 -5 -3 +1 +3 +8 +9 +10 +10 +19 +20 +23
% point
+29
70 50
45
60
40
50 35
40 30
25
30 20
20 15
10
10
5
0 0
** ** ** ** ** ** * ** ** y* y* n n ** s e k ** io ** ** ** ** **
n ia* taly* blic* blic* ay* tria* land hina ed.* an gar Japa ede ean land apor mar erta tar tes* ish* alia* bec* and*
ve I pu pu orw us n g C F r m u n S w m e r g e n l b O n ta m t r u e a l
Slo e e N A E g, ian Ge H CD Neth Si
n D A d S Fle Aus Q Ze
kR hR on ss OE ite m w
l ova zec n g K Ru Un elgiu Ne
C
S Ho B
1 2 http://dx.doi.org/10.1787/888933084171

Summary
Innovation or significant change in the use of textbooks during lessons varied across countries
and extent. Change between 2003 and 2011 in the OECD area is illustrated through a wider adoption
of textbooks as a basis for instruction rather than as supplementary resources, both in primary and
secondary education and consistently across disciplines.
Figure8.10 Change in the use of textbooks as a primary resource for 8th grade instruction
Primary resource-science Primary resource-maths

%
100
80
60
40
20
0
-20
-40
-60
ec ay d a io n n ea lia es d a ry re ile den Italy enia rael esia ean ean ge)
eb orw alan Chin ntar Japa ratio Kor stra Stat glan dian nga apo Ch e v Is n m n
Qu N Ze ng, O d e u
A ite d E n I n Hu ing Sw Slo do CD CD mcha
w o Fe S In E OE bs.
Ne ng K ia n U n O (a
Ho ss
Ru
1 2 http://dx.doi.org/10.1787/888933084190
Figure8.11 Change in the use of textbooks as a primary resource for 4h grade instruction
Primary resource-science Primary resource-maths

%
100
80
60
40
20
0
-20
-40
-60
ds io nd pore an lia ion tate
s ec nd ary hina rway mish ly nia ea
n n e)
an tar ala Jap ustra erat eb gla ung Ita ve ea g
mchan
erl On Ze Sing
a
d dS Qu En H ng
,C No Fle Slo D
m
D
eth w A
nF
e
ni t e
Ko ium OE
C C
OE (ab
s .
N Ne i a U n g l g
ss Ho Be
Ru
1 2 http://dx.doi.org/10.1787/888933084209
Notes : For details, please see Figures 8.1, 8.2, 8.3 and 8.4.

Figure8.12 Change in the use of textbooks as a supplementary resource for 8th grade instruction
Supplementary resource-science Supplementary resource-maths

%
40
20
-20
-40
-60
-80
nd lia bec way ina orea ates pan ario tion hile land pore iana gary den esia Italy enia rael ean ean ge)
ala ustra Que r h K t Ja Ont era C ng a Ind n e n v Is m n
Ze No ng, C dS d E Sing Hu Sw Indo Slo D D m ha
w A o ite Fe EC OECbs. c
Ne ng
K U n
ia n O (a
Ho ss
Ru
1 2 http://dx.doi.org/10.1787/888933084228
Figure8.13 Change in the use of textbooks as a supplementary resource for 4th grade instruction
Supplementary resource-science Supplementary resource-maths

%
60
40
20
0
-20
-40
-60
-80
-100
ds an ion por
e nd ina rio ary tate
s ec lia nd ay sh ly nia ea
n n e)
an Jap erat gla , Ch Onta ung eb tra eala rw lemi Ita ve ea g
mchan
erl d i ng
a
En g H dS Qu Aus Z No F Slo D
m
D
eth Fe S
Ko
n i t e w um OE
C C
OE (ab
s .
N ian ng Un Ne lgi
ss Ho Be
Ru
1 2 http://dx.doi.org/10.1787/888933084247
Notes : OECD average includes all OECD education systems. For details, please see Figures 8.5, 8.6, 8.7 and 8.8.
Note on Israel

Table 8.1. Effect sizes for changes in the use of textbooks
As primary resource for instruction As supplementary resource for instruction
Mathematics
Mathematics
Mathematics
Mathematics
Reading
Science
Science
Science
Science
03-11 03-11 03-11 03-11 01-06 03-11 03-11 03-11 03-11
Australia 0.10 0.30 0.20 0.14 m -0.06 0.03 -0.19 0.49
Belgium Flemish m m -1.00 0.20 m m m 0.59 0.43
Ontario 0.09 0.24 0.20 0.01 0.07 -0.12 -0.20 -0.20 0.19
Quebec -0.03 0.04 0.00 -0.37 -0.06 -0.04 0.06 -0.10 0.51
Chile 0.60 0.49 m m m -0.12 -0.34 m m
France m m m m -0.24 m m m m
Germany m m m m -0.17 m m m m
Hungary 0.25 0.52 0.29 0.22 0.15 -0.25 -0.52 -0.32 -0.22
Iceland m m m m 0.19 m m m m
Israel 1.06 0.60 m m 0.08 -1.03 -0.59 m m
Italy 0.70 0.35 0.81 0.79 0.24 -0.64 -0.35 -0.53 -0.67
Japan 0.18 0.21 0.19 0.13 m -0.19 -0.18 -0.17 -0.13
Korea 0.33 0.27 m m m -0.21 -0.16 m m
Netherlands m m -0.21 -0.03 0.96 m m 0.26 0.03
New Zealand -0.08 0.14 -0.29 0.05 0.02 0.01 0.08 0.06 0.66
Norway 0.10 0.16 0.40 0.66 0.15 -0.10 -0.19 -0.37 -0.58
Slovak Republic m m m m 0.00 m m m m
Slovenia 0.84 0.58 0.66 1.39 0.24 -0.84 -0.59 -0.70 -1.04
Sweden 0.28 0.73 m m -0.14 -0.29 -0.69 m m
England -0.36 -0.24 -0.45 -0.09 -0.26 0.36 0.10 0.08 -0.23
Scotland m m m m -0.38 m m m m
United States -0.33 -0.07 -0.30 0.02 -0.18 0.19 0.11 0.29 0.22
Indiana -0.42 -0.24 m m m 0.28 0.23 m m
OECD (average) 0.25 0.28 0.13 0.24 0.04 -0.25 -0.24 -0.16 -0.06
OECD (average absolute) 0.36 0.33 0.33 0.32 0.21 0.33 0.29 0.27 0.42
Hong Kong, China 0.14 -0.14 0.25 0.36 -0.13 -0.16 0.11 -0.21 -0.39
Indonesia 0.50 1.78 m m m -0.52 -0.62 m m
Russian Federation 0.07 0.35 0.12 0.37 0.19 -0.06 -0.35 -0.14 -0.32
Singapore -0.32 -0.43 0.08 -0.16 -0.18 0.26 0.25 -0.24 0.06
South Africa 0.77 0.62 m m m -0.68 -0.57 m m
1 2 http://dx.doi.org/10.1787/888933084266
Source: TIMSS 2003, 2007 and 2011 and PIRLS 2001, 2006 and 2011.

CHAPTER 9
Innovation
in the methods of assessment
used in classrooms
Innovation in classrooms can include a change in the methods teachers use

to assess the students over time. Teachers may innovate by administering
different types of tests, as well as by evaluating students through their daily
activities and outputs. The aim of innovation in this respect could be, for
example, to change the type of assessment in order to better monitor student
performance or to better address students needs and identify potential
solutions for improving their learning outcomes.

9. INNOVATION IN THE METHODS OF ASSESSMENT USED IN CLASSROOMS
Test based assessment
General Findings
Innovation in classrooms has meant more or less use of different types of tests for assessing
students (Figure9.1 and Figure9.2). The use of teacher-developed tests for 15-year-olds across the
OECD has shown an average absolute change of 8% points between 2003 and 2009 of, while the use
of standardised test registered an average absolute change of 3% points among OECD countries. In
Turkey (23% points), teacher-developed tests were used considerably more widely in 2009 than in
2003, while schools in Hungary (24% points) used this method significantly less for assessment in
2009 than 2003. In the Netherlands (21% points) and Indonesia (16% points) schools significantly
increased the use of standardised tests for student assessment. These country changes presented
medium effect sizes. Overall, teacher-developed tests were used more frequently in 2009 than in
2003 in six countries, while the opposite was true for four countries, with at least small effect sizes.
Similarly, use of standardised tests became more frequent in eight countries and less in three, with
at least small effect sizes.
Poland, Denmark and the Russian Federation are countries where innovation is illustrated
through an increase in the use of test based assessments. Danish schools, for example, tested their
students more widely both through standardised and teacher-developed tests in 2009 than 2003
with small effect sizes.
Turkey illustrates the case of a country where the direction of innovation regarding use of
test based assessment varied according to the approach under investigation. Between 2003 and
2009, Turkish schools used more teacher developed tests and less standardised test to assess their
students, with small effect sizes.

Figure9.1 Use of standardised tests as an assessment method in classrooms

Percentage of 15-year old students enrolled in schools where standardised tests are used for assessment at least monthly
% % point
-11 -6 -5 -2 -2 -2 -1 -1 -1 -1 -0 -0 -0 +0 +0 +0 +0 +0 +1 +1 +2 +2 +2 +3 +5 +7 +7 +10 +10 +16 +21
35 25
30
20
25
20 15
15 10
10
5
5
0 0
*** ey* nd y** taly ece ary denxico pan alia *** om tes blic g*** ium ay n** ada tria d* nd nd lic* l*** *** .*** *** *** s***
ain rk ala an I Gre ung Swe Me Ja ustr landingdd Sta epu ourBelg Norwmea Can Aus inlan erlaIrelapub tuga ark Fed land esiaand
Sp Tu Ze erm H A Ice d K ite h R mb F itz e r nm n Po don erl
w G
ite UnCzec uxe CD Sw k R Po De ssia In eth
Ne OE va Ru
Un L Slo N
1 2 http://dx.doi.org/10.1787/888933084285
Figure9.2 Use of teacher-developed tests as an assessment method

Percentage of 15-year old students enrolled in schools where teacher-developed tests are used for assessment at least monthly
% -24 -15 -12 -11 -8 -7 -5 -4 -3 -2 -1 -1 +1 +1 +1 +2 +2 +3 +4 +5 +7 +7 +7 +8 +8 +9 +9 +12 +17 +22 +23

% point
100 25
90
80 20
70
60 15
50
40 10
30
20 5
10
0 0
c s
*** c*** *** *** d** om nd gal bli den ate *** nd ay ean ada ium nds any Italy stria rg*** in**eece esia ed.* an** nd* k*** d*** y***
g aryubli ralia xicoalaningd erlaortu epu Swe d St land inlaNorw m n lg la
D Ca Be ther erm
r
Au bou Spa Gr don an F Jap Irela ma olan urke
t e z R F
n p s
Huh Re Au Mw Zited Swi
e K t P k
a n ite Ice O EC e G e m In ussi D en P T
v U N x
ec Ne Un Slo Lu R
Cz
1 2 http://dx.doi.org/10.1787/888933084304

PISA (2003, 2009) surveys asked school principals Generally, in your school, how often are students in <national
modal grade for 15-year-olds> assessed using the following methods? [] Standardised tests,[] Teacher
developed tests, [] Teachers judgmental ratings, [] Students portfolio, [] Student assignments/
projects/homework , with answer options Never; 1-2 times a year; 3-5 times a year; monthly; more than once
a month. Same data restrictions as in PISA international surveys (2000, 2003, 2006, 2009) apply to these data.

Non-test based assessments
General Findings
Innovation in classroom assessment methods typically occurred as an increase in the use of non-test
based teacher-guided assessments among OECD countries (Figure9.3). Between 2003 and 2009 the OECD
average absolute change regarding the use of teachers judgmental ratings as tools for assessment amounted
to 13% points. Poland (67% points), Finland (62% points) and Denmark (27% points) are countries where
teachers judgmental ratings use increased most, while teachers in the United States (25% points) used
this method of assessment significantly less in 2009 than in 2003. These country changes presented from
medium to large effect sizes, while the OECD average and average absolute effect sizes were small. Overall
use of judgmental ratings as an assessment method increased in 12 education systems and decreased in two.
Innovation in the use of non-test based assessment also took the form of a more widespread
evaluation of student outputs (Figure9.4 and Figure9.5). OECD average absolute change was 9%
points for the use of students portfolios for assessment, while it amounted to 8% points regarding
the of students assignments, projects or homework. Between 2003 and 2009, there was a noteable
increase in the use of student portfolios as assessments in schools in Indonesia (by 40% points),
Denmark (36% points) and Greece (29% points) and a decrease in this practice in Norway (29% points).
With regard to the frequency of use of student assignments, projects or homework for assessment,
Hungary (38% points) stood out as a country showing the largest decrease. These country changes
presented from medium to large effect sizes. Altogether, use of students portfolios increased in
seven countries and decreased in two with at least small effect sizes, while students assignments
were used more widely for assessment in eight countries and less in four.
Country Specificities
Portugal and Greece are countries where innovation was exemplified in additional use of non-test based
methods to assess students. In the period between 2003 and 2009, Portuguese and Greek schools assessed their
students more by evaluating their portfolios and assignments, projects or homework with at least small effect sizes.
The cases of Luxembourg and Norway illustrate that innovation in non-test based assessment methods
could occur as changes in both directions. Between 2003 and 2009, schools in Norway assessed their students
more through teachers judgmental ratings and less through students portfolios, with at least small effect sizes.

Figure9.3 Use of teachers judgmental ratings as an assessment method

Percentage of 15-year olds assessed by teachers judgmental ratings at least monthly and change over time
% -25 -15 -6 -5 -2 -2 -1 +2 +2 +2 +2 +3 +4 +4 +5 +7 +8 +8 +11 +11 +13 +14 +15 +15 +16 +16 +17 +22 +27 +62 +67
% point
100 70
90
60
80
70 50
60 40
50
40 30
30 20
20 10
10
0 0
* * ia ia y y o al ce lic an ny * ria *** *** *** d** *** *** *** *** *** *** *** *** *** ***
i tedes** d.**tral nes rke Ital exic landlandgdomug ee b ap a um st a n n n ry d rg ic n s y k d d
UnStatn FeAus ndo Tu M zer Ire Kin Port GrRepu J erm elgi Au nadmea edeeala nga elan bou ubl Spai land rwa marnlan olan
s ia I w i t d ch G B CaCD Sw Z Hu Ic em Rep er NoDen Fi P
s S it e e w x k th
Ru Un Cz OE Ne Lulova Ne
S
1 2 http://dx.doi.org/10.1787/888933084323
Figure9.4 Use of students portfolio as an assessment method

Percentage of 15-year old students enrolled in schools where students portfolios are used for assessment at least monthly
% -21 -14 -11 -10 -8 -5 -4 -4 -4 -3 -2 -1 -1 -1 -0 +1 +2 +2 +2 +2 +3 +3 +5 +5 +5 +6 +7 +8

% point
+8 +14 +14
80 25
70
20
60
50
15
40
10
30
20
5
10
0 0
h * * * * * ia ly d y co es ia n d d d l a ay e * * k * * * *
z ec lic**ary** lic** ion*key*stral Ita alanman exi Stat landustr mea elan rlan nlanlgiumtugaanadorw reec nds eden rg**mar pain apanom* nd** sia**
C ub ng ub rat ur u Ze Ger Mted Po A CD Ir tze Fi Be Por C N G erla Sw bou en S J gd la ne
p
Re Huk Re Fed
p e T A w i i m D in Ice do
Ne Un OE Sw N eth xe e dK In
a
v sia n L u i t
l o
S us U n
R
1 2 http://dx.doi.org/10.1787/888933084342

Figure9.5 Use of students assignments/projects/homework as an assessment method

Percentage of 15-year old students enrolled in schools where students portfolios are used for assessment at least monthly
% -38 -17 -14 -13 -9 -8 -8 -4 -3 -2 -2 -1 +0 +0 +0 +2 +2 +2 +4 +4 +4 +8 +8 +8 +9 +10 +11 +11 +11 +14 +14
% point
100 40
90 35
80
30
70
60 25
50 20
40 15
30 10
20
10 5
0 0
** ** ** ** ** ** y y in d. ** m an es ia da d k d ic d ** ** e* l* ** ** ** ** ** **
ry* m* ly* n* lic lia wa rke pa Fe rg* iu e tat es a an ar an bl lan ria ico ec ga d* ds nd d* y* an
n ga gdo Ita ede pub stra Nor Tu S sian bou Belg D med S don Can Finl enm IrelRepuZea ust ex Gre ortu elan rlan erla olan man Jap
Hu Kin Swh Re Au s m C t In D A M P Ic he itz P er
Ru uxe OE Uni k w
va Ne t G
ite
d
ze
c L Slo Ne Sw
U n C
1 2 http://dx.doi.org/10.1787/888933084361

Summary
Innovation or significant change regarding assessment methods has meant, with a few
exceptions, a more frequent use of non-test based assessments in classroom between 2003 and
2009. Similarly, but to a lesser extent, use of test based assessment methods increased among OECD
countries. Interestingly, these different types of assessment methods seemed to be regarded as
complementary, with no significant substitution trends observed between test and non-test based
assessments.
Figure9.6 Change in the use of test-based assessment methods in classrooms

15-year old students enrolled in schools where different types of tests are used for assessment
Use of teacher developed tests Use of standardised tests

%
80
60
40
20
0
-20
-40
-60
-80
es nd ay m en nd da ny ly nd rg ria ce an al lia co nd nd in sia rk ds ry ey nd an an e)
S tat cela orw elgiu wed inla ana rma Ita erla bou ust Gree Jap rtug stra exi eala rela Spa one nma rlan nga Turk Pola me me ang
I N B S F C Ge itz em A Po Au M Z I d e e Hu CD CD . ch
ted In D eth
Un
i Sw Lux Ne
w N OE OE(abs
1 2 http://dx.doi.org/10.1787/888933084380
Figure9.7 Change in the use of non-test-based assessment methods in classrooms

15-year old students enrolled in schools where assessment occurs outside of a formal testing practice
Use of students' portfolio Use of judgmental ratings Use of students' assignments

%
80
60
40
20
0
-20
-40
-60
-80
ey ium ada alia and tria and taly any pan urg gal den and ates nds ain ece esia ary way ark and and ean ean ge)
rk r l s l I m Ja bo rtu e el t la Sp Gre on ung or nm inl Pol m m an
Tu Belg Can ust Zea Au Ire r m Po Sw Ic ed S ther F
A w Ge xe it Ne In
d H N De CD CD . ch
Ne Lu U n OE OE(abs
1 2 http://dx.doi.org/10.1787/888933084399
Notes : For details, please see Figures 9.1,9. 2,9.3 and 9.4.

Table 9.1 Effect sizes for changes in assessment methods
Use of test based assessments Use of non-test based assessments

Use of students
Use of standardized Use of teacher- Use of teachers Use of assignements,
tests developed tests judgmental ratings students portfolio projects and
homework
03-09 03-09 03-09 03-09 03-09

Australia -0.03 -0.24 -0.11 0.13 -0,19
Austria 0.12 0.13 0.19 -0.02 0,20
Belgium 0.02 0.06 0.14 0.07 -0,03
Canada 0.09 0.06 0.16 0.06 0,08
Czech Republic 0.02 -0.31 0.09 -0.53 0,04
Denmark 0.34 0.38 0.83 0.12 0,04
Finland 0.29 0.02 1.34 0.12 0,26
Germany -0.31 0.07 0.11 -0.09 -0,87
Greece -0.04 0.15 0.06 0.12 0,25
Hungary -0.16 -0.50 0.33 -0.53 0,18
Iceland -0.12 -0.02 0.34 0.28 -0,30
Ireland 0.15 0.24 0.04 -0.01 0,29
Italy -0.06 0.10 -0.04 -0.08 0,17
Japan -0.04 0.27 0.11 0.18 0,22
Luxembourg 0.02 0.16 0.32 0.25 0,10
Mexico -0.04 -0.22 -0.02 -0.05 -0,15
Netherlands 0.52 0.11 0.39 0.21 0,29
New Zealand -0.12 -0.17 0.25 -0.12 0,21
Norway 0.06 0.03 0.45 0.09 0,07
Poland 0.37 0.46 1.57 -0.03 -0,09
Portugal 0.55 -0.10 0.11 0.07 -0,30
Slovak Republic 0.22 -0.06 0.41 -0.30 0,23
Spain -0.50 0.19 0.33 0.15 -0,38
Sweden -0.07 -0.04 0.29 0.16 0,01
Switzerland 0.13 -0.14 0.04 0.06 0,01
Turkey -0.21 0.52 -0.05 -0.24 0,20
United Kingdom -0.03 -0.14 0.05 0.23 0,01
United States 0.01 -0.06 -0.57 -0.07 -0,04
OECD (average) 0.05 0.04 0.24 -0.02 0.01
OECD (average absolute) 0.17 0.18 0.31 0.16 0.20
Indonesia 0.51 0.17 -0.10 0.32 0.01
Russian Federation 0.31 0.20 -0.31 -0.26 -0.04
1 2 http://dx.doi.org/10.1787/888933084418

CHAPTER 10
Innovation
in the availability of computers
and the internet in the classroom
Innovation in the classroom can take the form of providing students with
access to computers and the internet. Schools may choose to invest in more
computer and network equipment in their classes to be used as a tool for
instruction during lessons, or they may reduce computer use in the classroom,
possibly in favour of using technology in other ways or collecting ICT together
in dedicated technology suites. The aim of innovation with regard to increasing
computer and internet availability could be, for example, to make students
familiar with the use of ICT and to facilitate the pedagogical use of technology
in classrooms.

10. INNOVATION IN THE AVAILABILITY OF COMPUTERS AND THE INTERNET IN THE CLASSROOM
Computer availability
General findings
Innovation in classroom practices translated into a greater availability of computers for
pedagogical purposes in secondary education in some education systems, and less availability in
others (Figure10.1 and Figure10.2). Between 2003 and 2011, the average absolute change within the
OECD area regarding computer availability was 12% points in 8th grade mathematics classes and
9% points in 8th grade science classes. the Russian Federation (33% points) particularly increased
computer availability during 8th grade mathematics classes relative to other countries, while
computer availability decreased significantly in New Zealand (42% points) and Japan (28% points).
The Russian Federation (41% points) also led the increase in computer availability in 8th grade
science classes, whereas Japan (30% points) registered the largest decrease in this respect. These
changes were characterised by large and medium effect sizes, while the OECD average absolute
change in 8th grade maths classes presented a small effect size. Overall, computer availability in 8th
grade mathematics classes increased with at least small effect sizes in six education systems and
decreased in eight. As for 8th grade science classes, computer availability increased in four countries
and decreased in five, presenting at least small effect sizes.
As to primary education, innovation in the form of students having more or fewer computers
available was as pronounced as in secondary education (Figure10.3 and Figure10.4). OECD
countries registered an average absolute change of 12% points in computer availability in 4th grade
mathematics and science classes, indicating the magnitude of innovation between 2003 and 2011.
The Russian Federation (27% points) showed the most prominent increase in computer availability
during classes in 4th grade mathematics; on the contrary, Japan (27% points) and Denmark (24%
points) decreased significantly in this respect. The Russian Federation (30% points) also showcased
the largest increase in computer availability during 4th grade science classes, closely followed by
the Netherlands (26% points). These country changes presented from medium to large effect sizes
whilst OECD average absolute changes presented small effect sizes. Altogether, computer availability
during 4th grade mathematics classes significantly increased in four countries and decreased in eight
with at least small effect sizes. During 4th grade science classes, computer availability increased in
seven countries and decreased in five, with at least small effect sizes.
Computer availability for students during classes increased simultaneously in several levels or
disciplines in the Russian Federation and Norway. With at least medium effect sizes, the Russian
Federation, for example, experienced increases in computer availability during mathematics and
science classes both at 8th and 4th grades between 2003 and 2011.
By contrast, computer availability decreased in a comprehensive manner in Japan. Both

during mathematics and science classes computer availability decreased in Japanese primary and
secondary education with at least small effect sizes between 2003 and 2011.
Direction of innovation regarding computer availability has not been consistent across
education levels and disciplines in New Zealand. As for mathematics classes, computer availability
significantly increased in 4th grade from 2003 to 2011, while decreasing substantially with large effect
size in 8th grade.

Figure10.1 Computer availability in 8th grade maths lessons

Percentage of students with computers available to use and change over time
% -42 -28 -20 -17 -15 -15 -12 -11 -6 -5 -5 -4 -2 -2 -1 +2 +2 +4 +6 +9 +11 +11 +16 +20 +33
% point
100 45
90 40
80 35
70
30
60
25
50
40 20
30 15
20 10
10 5
0 0
w *** *** s* *** ** d* l** *** ia le n a es n ly ta ey ca io a* ** a* *** *** ***
Neand pan sett rea hina glan srae ore ven Chi ede dian Stat mea Ita eso urk Afri ntar nesi ebec trali ary way Fed.
a l Ja hu Ko , C n I a p Slo S w In d D n n T h O o u s n g or
Ze ac g E g e
nit OE
C Mi ut In
d Q Au Hu N sian
ss on Sin So s
M a
n g K U Ru
Ho
1 2 http://dx.doi.org/10.1787/888933084437
Figure10.2 Computer availability in 8th grade science classrooms

% -30 -23 -22 -20 -10 -10 -9 -8 -5 -4 -3 -3 -1 -1 +0 +1 +2 +4 +4 +8 +8 +9 +14 +15

% point
+41
100 45
90 40
80 35
70
30
60
25
50
40 20
30 15
20 10
10 5
0 0
*** *** *** *** na el* d d es lia ** ia en ey tts taly ota rica bec ry* rio ile* a*** y*** .***
an ore ina rea dia Isra alan glan Stat stra ean oven wed urk use I es Af ue nga nta Ch esi rwa Fed
Jap gap , Ch Ko In Ze En ted Au D m Sl S T ch inn uth Q Hu O n
i n g w i C s a M do No ssian
S on Ne Un OE as So In
K M Ru
ng
Ho
1 2 http://dx.doi.org/10.1787/888933084456

Figure10.3 Computer availability in 4th grade maths classrooms

% -27 -24 -16 -16 -14 -13 -13 -13 -12 -9 -9 -8 -7 +1 +1 +2 +4 +6 +7 +7 +17 +18 +20
% point
+27
100 30
90
80 25
70
20
60
50 15
40
30 10
20
5
10
0 0
*** *** *** *** *** a** ** ** a* ic* lic ia* en ean taly tes any alia nds nia d*** y*** y*** .***
an ark ish bec ore in and ario ert ubl ub str ed I Sta m str rla ve an wa ar Fed
Jap nm lem Que gap g, Ch Engl Ont Alb Rep Rep Au Sw CD m r
ed Ge Au ethe Slo Zeal Nor ung ian
e F i n n h E i t H
D m S Ko k
va Czec O Un N w ss
lgi
u
ng Slo Ne Ru
B e H o
1 2 http://dx.doi.org/10.1787/888933084475
Figure10.4 Computer availability in 4th grade science classrooms

% -16 -15 -14 -11 -10 -9 -7 -4 -4 -3 -3 -3 -2 -1 -1 +1 +3 +12 +12 +18 +18 +21 +26
% point
+30
100 35
90
30
80
70 25
60 20
50
40 15
30 10
20
10 5
0 0
* * * * * * c a c s y a a a c d * * * * * * * *
d** re** n** rk** rio den bli rali ebe ate an ert hin stri bli lan ean ary* aly** nia** ay** ish** ds** ed.**
g lan apo Japa ma Onta Swe Repu ust Qu d St erm Alb g, C Au Repu Zea m ung It ve orw em rlan n F
En Sing n A ite G n D Sl o N Fl the ssia
De va
k
Un Ko h w
ec Ne OE
C H
Slo ng Cz um Ne Ru
Ho e lgi
B
1 2 http://dx.doi.org/10.1787/888933084494

TIMSS (2003, 2007 and 2011) surveys asked 8th grade teachers Do the students in this class have computer(s)
available to use during their mathematics/science lessons? with answer options Yes/No. TIMSS-PIRLS
(2001, 2006 and 2011) survey asked teachers Do the <fourth-grade> students in the <PIRLS/TIMSS> class have
computer(s) available to use during their mathematics/science lessons?, with response options Yes/No.

Network accessibility
General findings
Innovation in classrooms is also observed through an increased or reduced availability of
Internet network accessibility during lessons. At OECD level, average absolute change in accessibility
to the Internet during 8th grade mathematics classes was 12% points, representing the magnitude of
change between 2003 and 2011(Figure10.5 and Figure10.6). Likewise, OECD average absolute change
in Internet accessibility during science classes was 10% points. As for mathematics classes, the
country showing the most significant increase was the Russian Federation (33% points), in contrast
with Japan (28% points) outranking all other countries for a decrease in Internet accessibility during
classes. The Russian Federation (36% points) also stood out in terms of an increase in Internet
accessibility during science classes, while decreases were particularly remarkable in Japan (30%
points) and Korea (23% points). These country changes presented large and medium effect sizes.
OECD average absolute changes exhibited small effect sizes for maths and science. Overall, Internet
accessibility during mathematics classes increased substantially in six countries and decreased in
seven, with small effect sizes. As for science classes, the increase in Internet availability had at least
small effect sizes in four countries, while the decrease had small effect sizes in five countries.
In comparison with secondary education, innovation in the form of a change in Internet

accessibility during classes was slightly more pronounced in primary education (Figure10.7 and
Figure10.8). Between 2003 and 2011, the average absolute change for the OECD area regarding
Internet availability during 4th grade classes amounted to 14% points for mathematics and 12%
points science. In 4th grade mathematics classes the most prominent increases were registered in
Norway (33% points), the Russian Federation (23% points), New Zealand (22% points) and in Hungary
(19% points), whereas the decreases were substantial in Japan (27% points) and in Denmark (24%
points). As for 4th grade science classes, the Flanders (35% points) and Netherlands (28% points)
stood out as education systems with the largest increases in Internet accessibility during classes.
These education system changes presented medium effect size, while the OECD average absolute
changes had small effect sizes. Overall, Internet accessibility during 4th grade mathematics classes
increased in seven and decreased in six education systems, with at least small effect size; as for
Internet accessibility in 4th grade mathematics, the increases exhibited at least small effect sizes in
six countries while decreases were significant in five countries.
The Russian Federation, Norway and Hungary stand out as education systems where innovation
took the form of a widespread increase in Internet accessibility simultaneously across disciplines
and educational levels. For example, with at least small effect sizes, Norway saw increases in the
availability of Internet connected computers during mathematics and science classes, both in
primary and secondary education between 2003 and 2011.
On the contrary, Japan, Singapore and Hong Kong showcased that innovation may result
in a simultaneous decrease in Internet accessibility during classes of different disciplines and
educational levels. Especially Japan experienced a consistent decrease in the availability of Internet
connected computers in both mathematics and science classes of 4th and 8th grade with at least
small effect sizes.
New Zealand illustrates the case of a country where innovation concerning Internet accessibility
in classrooms has not been consistent across education levels. New Zealand schools made available
more Internet connected computers for mathematics classes in 4th grade and less in 8th grade in the
period from 2003 to 2011.

Figure10.5 Internet availability in the 8th grade maths classrooms

Percentage of students with Internet access in the classroom and change over time
% -28 -24 -19 -18 -13 -13 -10 -9 -7 -6 -6 -6 -4 -1 -1 +1 +3 +4 +6 +8 +10 +11 +17 +23
% point
+33
100 35
90
30
80
70 25
60 20
50
40 15
30 10
20
10 5
0 0
*** *** *** ts* *** a** d na ia ta el en ile es an ca ly ey io a** lia *** *** *** ***
an rea nd set ore in glan dia ven eso Isra wed Ch Stat me Afri Ita urk ntar esi stra ebec gary way Fed.
Jap Ko eala chu gap g, Ch En In Slo inn S d D h T O o n u r
Qu Hun No sian
Z sa Sin on M ite OEC Sout In
d A
w
Ne Ma
s
g K Un R us
n
Ho
1 2 http://dx.doi.org/10.1787/888933084513
Figure10.6 Internet availability in the 8th grade science classrooms

% -30 -23 -21 -18 -9 -8 -4 -3 -3 -2 -2 -1 -0 +0 +1 +2 +3 +8 +9 +9 +9 +11 +13 +23

% point
+36
100 40
90 35
80
70 30
60 25
50 20
40
15
30
20 10
10 5
0 0
*** *** *** *** na el d es ca d ia an en tts lia ey ly ec io le* ta y** *** *** ***
an rea ore ina dia Isra glan Stat Afri alan oven me wed use stra urk Ita ueb ntar Chi neso gar esia way Fed.
Jap Ko gap , Ch In En ted uth Ze Sl CD S ach Au T Q O i n Hun don Nor ian
Sin ong Un
i So
Ne
w OE ss M In ss
gK Ma Ru
n
Ho
1 2 http://dx.doi.org/10.1787/888933084532

Figure10.7 Internet availability in the 4th grade maths classrooms

% -27 -24 -12 -11 -11 -11 -10 -7 -7 -3 -0 +0 +1 +5 +7 +8 +9 +12 +14 +18 +19 +22 +23 +33
% point
100 35
90
30
80
70 25
60 20
50
40 15
30 10
20
10 5
0 0
*** *** *** a* c* d* ta en io lic taly tes blic n*** ish ny* ria* ia** a*** s*** y*** d*** .*** y***
an ark ore hin ebe lan lber wed ntar pub I Sta pu ea em ma st ral ni nd ar n ed a
Jap nm gap g, C Qu Eng A S O Re d e m Fl Ger Au ust love erla ung eala ian F Norw
De Sin Kon a k n ite ch R CD ium A S t h H Z ss
v U Cze OE elg Ne w u
ng Slo B Ne R
Ho
1 2 http://dx.doi.org/10.1787/888933084551
Figure10.8 Internet availability in the 4th grade science classrooms

% -17 -14 -12 -12 -10 -6 -5 -4 -2 -2 -2 -2 +2 +4 +5 +6 +6 +13 +18 +21 +24 +24 +28 +35
% point
100 40
90 35
80
30
70
60 25
50 20
40 15
30
10
20
10 5
0 0
** ** ** ** * o a s c a a c c d ** ny aly a*** y*** a*** y*** .*** s*** **
re* n* nd rk* den ari ert ate ebe rali hin bli bli lan n* a It tri r i a d h*
apo Japa ngla ma Swe Ont Alb d St Qu ust g, C Repu Repu Zea mea erm u s n ga ven orw n Fe rland emis
n o
ing E De ite A on k h w
va ec Ne OEC
D G A Hu Sl N sia the Fl
S Un K
ng Slo Cz
s
Ru Ne lgium
Ho B e
1 2 http://dx.doi.org/10.1787/888933084570

TIMSS (2003, 2007 and 2011) surveys asked 4th and 8th grade teachers Do the students in this class have
computer(s) available to use during their mathematics/science lessons? with answer options Yes/No.
Conditional on answering Yes to this question teacher are asked Do any of the computer(s) have access to the
Internet? with answer options Yes/No. The same data restrictions as in International TIMSS and PIRLS Reports

Summary
Innovation or significant change in computer and Internet availability during lessons has
resulted in slightly more availability in primary education and slightly less in secondary education.
Change in the OECD area was balanced across disciplines and levels with some countries showing
consistent increases and decreases across disciplines and levels between 2003 and 2011 as in the
case of the Russian Federation and Japan.
Figure10.9 Change in computer availability in 8th grade classrooms
Science computer availability Maths computer availability

%
80
60
40
20
0
-20
-40
-60
-80
ly den tes nia ana io ile lia ec el nd ia ry re ay na ea d an on an n )
Ita e ta ve ndi tar Ch stra ueb Isra gla ones nga apo orw Chi Kor alan Jap rati me meaange
Sw ed S Slo I On u Q E n d u
H Si n g N g , Z e d e D D h
it A In on w Fe OEC OEC bs. c
Un n gK Ne i an (a
s
Ho Ru
s
1 2 http://dx.doi.org/10.1787/888933084589
Figure10.10 Change in computer availability in 4th grade classrooms
Science computer availability Maths computer availability

%
60
40
20
-20
-40
-60
tes ralia Ita
ly ina land ebe
c io
tar oven
ia or
e nd ary ds ay
rw lemi
sh an ion ea
n n e)
ea g
Sta st Ch a Qu On ap gla ung erlan Jap erat m mchan
ed Au n g, Ze S l i ng En H t h No F e d C D D .
nit Ko Ne
w S
Ne um F OE
C
OE (ab
s
U ng lgi ian
Ho Be ss
Ru
1 2 http://dx.doi.org/10.1787/888933084608
Notes: For details, please see Figures 10.1,10. 2, 10.3 and 10.4.

Figure10.11 Change in Internet availability in 8th grade classrooms
Science internet availability Maths internet availability

%
80
60
40
20
0
-20
-40
-60
-80
l
tes Italy eden enia ralia hile land tario srae iana ebec esia land gary hina pore rway orea pan tion ean ean ge)
Sta v
Sw Slo Aus
t C ng
On
I
In
d Qu
do
n ea Hun g, C inga No K Ja era m m an
ed E In ew Z on ed ECD ECDs. ch
Unit N K
S
an
F O O (ab
ng ssi
Ho Ru
1 2 http://dx.doi.org/10.1787/888933084627
Figure10.12 Change in Internet availability in 4th grade classrooms
Science internet availability Maths internet availability

%
60
40
20
-20
-40
-60
tes Ita
ly
tar
io ina be
c lia nd land pore enia gary an ish ds ion rwa
y
ea
n n e)
ea ng
Sta Ch Que tra ngla a a lov Hun Jap lem erlan erat
ed On n g, A us E Ze Sing S F t h e d No CD m D m. cha
nit Ko Ne
w um Ne ian F OE
C
OE (ab
s
U ng lgi
Ho Be ss
Ru
1 2 http://dx.doi.org/10.1787/888933084646
Notes: OECD average includes all OECD education systems. For details, please see Figures 10.5,10.6, 10.7 and 10.8.
Note on Israel

Table 10.1 Effect sizes for changes in the availability of computers and the internet in classrooms
Change in computer availability Change in network availability
Mathematics Science Mathematics Science Mathematics Science Mathematics Science
03-11 03-11 03-11 03-11 03-11 03-11 03-11 03-11

Australia 0.23 -0.08 0.14 -0.09 0.21 0.02 0.27 -0.06
Belgium Flemish m m -0.32 0.48 m m 0.15 0.76
Ontario 0.13 0.15 -0.26 -0.20 0.11 0.17 -0.14 -0.12
Quebec 0.30 0.09 -0.32 -0.07 0.31 0.17 -0.24 -0.05
Chile -0.10 0.20 m m -0.09 0.19 m m
Hungary 0.34 0.15 0.47 0.27 0.39 0.22 0.53 0.45
Israel -0.24 -0.20 m m -0.16 -0.17 m m
Italy -0.02 0.02 0.02 0.28 0.06 0.06 0.00 0.14
Japan -0.64 -0.65 -0.60 -0.38 -0.62 -0.63 -0.59 -0.36
Korea -0.36 -0.48 m m -0.49 -0.53 m m
Netherlands m m 0.17 0.52 m m 0.40 0.56
New Zealand -0.87 -0.18 0.41 0.02 -0.39 -0.05 0.53 0.09
Norway 0.43 0.34 0.38 0.38 0.48 0.48 0.68 0.50
Slovenia -0.14 -0.06 0.16 0.38 -0.15 -0.03 0.34 0.47
Sweden -0.10 -0.03 m m -0.12 -0.01 m m
England -0.30 -0.17 -0.31 -0.41 -0.21 -0.08 -0.25 -0.30
United States -0.04 -0.11 0.04 -0.07 -0.02 -0.07 0.01 -0.08
Indiana -0.08 -0.23 m m -0.18 -0.21 m m
OECD (average) -0.03 -0.08 0.02 0.05 -0.02 -0.03 0.13 0.10
OECD (average absolute) 0.25 0.19 0.27 0.26 0.26 0.20 0.33 0.27
Hong Kong, China -0.33 -0.44 -0.27 -0.05 -0.28 -0.38 -0.23 -0.04
Indonesia 0.24 0.36 m m 0.32 0.53 m m
Russian Federation 0.78 0.94 0.78 0.87 0.85 0.91 0.78 0.82
Singapore -0.22 -0.49 -0.31 -0.33 -0.26 -0.43 -0.26 -0.35
South Africa 0.11 0.10 m m 0.03 -0.11 m m
1 2 http://dx.doi.org/10.1787/888933084665

CHAPTER 11
Innovation
in the use of computers
in the classroom
Innovation in the classroom includes different possibilities for using

computers during classroom instruction across subjects. Teachers may choose
to integrate their instruction with a wider or narrower use of computers to
serve different pedagogical purposes. The aim of innovation with regard
computer use could be, for example, for students to develop an adequate set
of digital competencies in primary school and to make students more aware
of the usefulness of computers for their learning. A reduction in ICT use in
the classroom may result from innovations such as a decision to provide
computers for the home, or a preference for providing hands on experience
through real objects and experiments rather than virtual ones.

11. INNOVATION IN THE USE OF COMPUTERS IN THE CLASSROOM
Use of computers for practising skills and procedures
General findings
Innovation in the classroom has taken the form of greater and lesser use of computers to practice
skills and procedures during mathematics and science lessons in secondary education (Figure11.1
and Figure11.2). Between 2003 and 2011 the average absolute change within the OECD area for
computer use to practice skills and procedures was 10% points in 8th grade science and 9% points
in 8th grade mathematics. The Russian Federation (41% points) showed the largest increase in the
practice of skills and procedures in science classes through computersand also led as the education
system presenting the largest increase in skills and procedures practice through computers in 8th
grade mathematics classes (33% points). On the contrary, in Korea (26% points) and Japan (12% points)
students practiced skills and procedures less frequently via computer in 2011 than in 2003. These
changes were characterised by large and medium effect sizes, while the OECD average absolute
change for 8th grademaths and science presented small effect sizes. Overall and with at least small
effect sizes, computer use to practice skills and procedures during science lessons augmented in
nine education systems and diminished in three, whereas, in the case of mathematics classes, it
increased in five countries and decreased in three.
Primary education reflected secondary education in terms of the change in the use of
computers to practice skills and procedures (Figure11.3 and Figure11.4). OECD average absolute
change regarding the use of computers to practice skills and procedures was 13% points in 4th grade
mathematics and 10% points in 4th grade science, between 2003 and 2011. In New Zealand (33%
points) and the Russian Federation (24% points) practising skills and procedures through computers
during mathematics classes became particularly more common in 2011 than it was in 2003, while
the opposite was true in Japan (23% points) and Denmark (22% points). The Russian Federation (30%
points) and Norway (25% points) showcased the most notable increases in computer use for skills
and procedures practise during science lessons. These country changes presented from medium to
large effect sizes; while the OECD average absolute change effect sizeswere small. Altogether, use
of computers to practise skills and procedures during 4th grade mathematics classes significantly
increased in four countries and decreased in ten with at least small effect sizes. As for 4th grade
science classes, skills and procedures practice through computers increased in seven countries and
decreased in one, with at least small effect sizes.
Use of computers to practice skills and procedures increased across disciplines and levels in
the Russian Federation, Norway and Hungary. With at least small effect sizes, The use of computers
to practice skills and procedures during mathematics and science lessons by Hungarian students
increased between 2003 and 2011, at both 8th and 4th grades .
In contrast, practising skills and procedures through computers decreased significantly across
disciplines in Korea, Japan and Singapore. Korean students used computers less to practice skills
and procedures during 8th grade mathematics and science classes with at least small effect sizes
between 2003 and 2011.

Figure11.1 Computer use to practice skills and procedures in 8th grade maths
Percentage of students using computers to practice skills and procedures at least sometimes and change over time
% -26 -12 -12 -10 -9 -7 -5 -5 -4 -3 -2 -1 -1 -0 +0 +2 +2 +4 +4 +5 +10 +10 +14 +19 +33

% point
60 35
50 30
25
40
20
30
15
20
10
10 5
0 0
*** tts *** nd re** nia rael ina and hile rica ates key ean esia den taly ana bec ario ota ry** ay** ia*** d.***
rea se an gla o ve Is , Ch eal C Af St Tur m on we I di Que Ont nes nga orw tral Fe
Ko achu Jap En ngap Slo h CD Ind S In
s s S i
g
on ew
Z
o ut nited E Mi
n Hu N Aus sian
s
K S U O
Ma o ng N Ru
H
1 2 http://dx.doi.org/10.1787/888933084684
Figure11.2 Computer use to practice skills and procedures in 8th grade science
% -16 -11 -8 -8 -7 -5 -5 -4 -3 -3 -2 +2 +4 +4 +5 +5 +6 +7 +8 +11 +13 +18 +24 +31

% point
+41
70 45
60 40
35
50
30
40 25
30 20
20 15
10
10 5
0 0
na a** ina re** n** nd tts ael nia tes key ica ota Italy den n*** ebec sia* land ile** rio** ia*** ry*** ay*** d.***
dia Kore , Ch apo Japa ngla huse Isr love Sta Tur Afr nes e a e a Ch ta ral ga Fe
In S ted h Sw me Qu don Ze On ust Hun Nor sian
w
g
on Sin
g E ac
s s n i o ut Min C D I n ew A s
g K a U S E N u
n M O R
Ho
1 2 http://dx.doi.org/10.1787/888933084703

Figure11.3 Computer use to practice skills and procedures in 4th grade maths
% -23 -22 -21 -17 -16 -14 -12 -11 -11 -10 -10 -4 +1 +2 +4 +4 +4 +4 +4 +6 +16 +24 +24 +33
% point
100 40
90 35
80
30
70
60 25
50 20
40 15
30
10
20
10 5
0 0
*** *** *** *** ** *** ** ta io* ina ia** blic any ean nia tes taly den alia nds y*** .*** y*** d***
an ark ore bec and blic ish ber tar h tr u m ve ta I e tr la ar ed a n
Jap nm gap Que Engl epu Flem Al On ng, C Aus Rep Ger CD m Slo ed S Sw Aus ther ung an F Norw eala
De Sin R o a k E n i t e H s i Z
ch ium g K ov O U N u s ew
e
Cz Bel
g n Sl R N
Ho
1 2 http://dx.doi.org/10.1787/888933084722
Figure11.4 Computer use to practice skills and procedures in 4h grade science

% -13 -10 -7 -4 -4 -2 +0 +1 +1 +4 +4 +5 +5 +5 +5 +7 +7 +11 +14 +17 +18 +22 +25 +30

% point
70 35
60 30
50 25
40 20
30 15
20 10
10 5
0 0
* k o c a a a y s a c a e c * * * * * * * *
nd ar ari bli tri ali ert an den ate apan hin ebe eni por ubli an** and* nds* aly** ary** ish** ay** ed.**
gla enm Ont epu Aus ustr Alb erm Swe d St J , C Qu Slov inga Rep me eal erla It ng em orw n F
En D R A G it e n g S ak CD w Z u
H mF l N si a
ec
h Ko th
Cz Un ng
v OE Ne Ne Ru
s
Slo lgi
u
Ho Be
1 2 Sttp://dx.doi.org/10.1787/888933084741

TIMSS (2003, 2007 and 2011) surveys asked 4th and 8th grade teachers Do the students in
this class have computer(s) available to use during their mathematics/science lessons? with
answer options Yes/No. Conditional on answering yes to this question teachers are asked
How often do you have the students do the following computer activities during mathematics/
science lessons? a) Practice skills and procedures. The response codes to this question vary
across questionnaires. The results count all responses except those indicating Never. The
same data restrictions as in International TIMSS and PIRLS Reports by the International

Use of computers for information research
General findings
Innovation in classrooms was also exemplified in a larger or smaller student use of computers
to look up information during lessons in secondary education (Figure11.5 and Figure11.6). At
OECD level, average absolute change in students using computers to look up information during
lessons was 10% points both in 8th grade mathematics and science between 2003 and 2011. Use
of computers for information research during mathematics lessons increased particularly in the
Russian Federation (35% points) while substantially decreasing in Korea (35% points). The Russian
Federation (37% points) also outranked all other education systems in relation to the increase in
student use of computers for information research during science lessons; the largest decreases
in such use were found in Singapore (27% points), Japan (26% points) and Hong Kong (24% points).
These education system changes presented large and medium effect sizes; while the OECD
average absolute change effect sizes were small. Overall, students used computers more to look up
information during mathematics and science classes in four education systems and less in seven,
with at least small effect sizes.
In comparison with secondary education, innovation regarding computer use for information
research purposes was slightly more pronounced in primary education (Figure11.7 and Figure11.8).
The average absolute change at the OECD level in the use of computers to look up information in 4th
grade was 13% points during science lessons and 11% during mathematics lessons in the period from
2003 to 2011. Students in science classes used computers more to look up information especially in
the Netherlands (27% points) and the Russian Federation (26% points). During mathematics classes,
use of a computer for information research increased particularly in the Russian Federation (20%
points), while decreasing significantly in Denmark (28% points). These changes presented large and
medium effect sizes while the OECD average absolute change effect sizes were small. Altogether,
use of computers for information research purposes increased in seven education systems and
decreased in seven during 4th grade science classes, whereas it increased in five systems and
decreased in six during mathematics lessons, with at least small effect sizes.
The Russian Federation and Hungary stood out as education systems where innovation in the
form of a broader use of computers to look up information during classes occurred simultaneously
across disciplines and educational levels. For example, with at least small effect sizes, Hungarian
students used computers more often to look up information during mathematics and science
classes and in primary and secondary education between 2003 and 2011.
Singapore, England and Hong Kong are education systems where innovation is illustrated in
reduced use of computers across disciplines and levels to look up information. Fourth-grade and
8th grade English students, for example, used computers significantly less to look up information
during mathematics and science lessons in the period from 2003 to 2011 with small effect sizes.

Figure11.5 Computer use to look up ideas and information in 8th grade maths
Percentage of students using computers to look up ideas and information at least sometimes and change over time
% -35 -20 -16 -15 -13 -11 -8 -7 -7 -5 -5 -4 -3 -2 -2 -1 +1 +2 +2 +4 +6 +6 +10 +16 +35

% point
70 40
60 35
50 30
25
40
20
30
15
20
10
10 5
0 0
*** d** *** a** na** s*** n*** hile way nia and n*** den rael etts rica key ario ota esia alia bec* ly** ry*** d.***
rea an ore ian i te a C or ove eal ea Swe Is hus Af Tur nt nes on str ue Ita nga e
Ko Engl gap Ind g, Ch Sta Jap N Sl Z m c h O in nd Au Q nF
Sin o n ited ew CD
E s sa Sout M I Hu ssia
K N O a Ru
ng Un M
Ho
1 2 http://dx.doi.org/10.1787/888933084760
Figure11.6 Computer use to look up ideas and information in 8th grade science
% -27 -26 -24 -20 -13 -12 -11 -10 -9 -7 -5 -5 -4 -3 -1 +1 +2 +2 +3 +3 +6 +10 +13 +21 +37
% point
80 40
70 35
60 30
50 25
40 20
30 15
20 10
10 5
0 0
e**
* *** *** ***
na
* tts d* s** nd lia en ael nia n** ey ica taly ota hile bec ario ia*** ay** y*** d.***
p or apan hina orea dia use glan tate eala stra wed Isr love mea Turk Afr I es C ue nt es rw gar Fe
ga J C K I n ch E ed S w Z Au
n S S D th nn Q O on No un an
Sin ng
,
ss
a
it Ne EC S ou Mi In
d H ssi
K o M a n O Ru
U
ng
Ho
1 2 http://dx.doi.org/10.1787/888933084779

Figure11.7 Computer use to look up ideas and information in 4th grade maths
% -28 -17 -15 -14 -13 -12 -9 -8 -8 -3 -3 -1 +1 +2 +2 +3 +6 +6 +6 +9 +11 +19 +20 +23
% point
80 30
70
25
60
50 20
40 15
30
10
20
10 5
0 0
* * a** o** d** ta* * * lic en lic * * * .*** ***
k** c** i sh n* tes ria an ds ore ny nia lia y** y** y**
ar ebe Chin tar glan lber emi Japa epub wed epub Sta Aust me rlan gap rma love stra Ital gar rwa Fed land
nm u
Q ng, O n
En A F l
hR
S
k R ite
d C D e
th Si n G e S Au
Hu
n
No sia n
Ze
a
De o um ec va Un OE Ne s w
n gK e lgi Cz Slo Ru Ne
Ho B
1 2 http://dx.doi.org/10.1787/888933084798
Figure11.8 Computer use to look up ideas and information in 4th grade science
% -19 -15 -15 -14 -12 -11 -10 -7 -6 -4 -4 -4 -4 -1 -1 -1 +3 +10 +18 +20 +22 +22 +26 +27
% point
100 30
90
80 25
70
20
60
50 15
40
30 10
20
5
10
0 0
* * * * * * a * a c c c a a y d * * * * * * * *
d** en* rk** re** an* rio hin tes* ert bli ebe bli rali stri an lan an* taly* nia** ish** ary** ay** ed.** ds**
g lan wed ma apo Jap Onta g, C Sta Alb Repu Qu Repu ust Au erm Zea me I ve
e m n g orw n F rlan
En S n
De Sin
g n
Ko nite
d h k A G w CD Slo Fl Hu N sia the
ng Cz
ec va Ne OE m s
Ru Ne
o U Slo e l giu
H B
1 2 http://dx.doi.org/10.1787/888933084817

computer(s) available to use during their mathematics/science lessons? with answer options Yes/No.
Conditional on answering yes to this question teachers are asked How often do you have the students do
the following computer activities during mathematics/science lessons? b) Look up ideas and information
with answer options Every or almost every day; Once or twice a week; Once or twice a month; Never or
almost never. The same data restrictions as in International TIMSS and PIRLS Reports by the International

Use of computers for data analysis and scientific experimentation
General findings
Increased and reduced use of computers to analyse data during lessons in secondary education
has also been observed across education systems (Figure11.9 and Figure11.10). Between 2003 and
2011 the OECD average absolute change in the use of computers to analyse data was 10% points
during mathematics lessons and 7% points during science lessons. Students use of computers
to analyse data during mathematics lessons increased especially in the Russian Federation (22%
points), while decreasing remarkably in England (29% points) and Korea (25% points). Again, the
Russian Federation (27% points) stood out as the country where students used computers more
often for data analysis purposes during science classes in 2011 than 2003. These country changes
exhibited medium effect sizes, while the OECD average absolute presented a small effect size in
computer use during science lessons. Overall, students used computers more often to analyse data
during mathematics lessons in 2011 than in 2003 in four education systems, while the opposite was
true for six, with at least small effect sizes. Use of computers for data analysis during science classes
increased in three education systems and decreased in six, with small effect sizes.
Innovation in classrooms also encompassed a greater or lower use of computers for scientific
experimentations (Figure11.11 and Figure11.12). Between 2003 and 2011, OECD average absolute
change concerning students using computers to conduct scientific experiments during 8th grade
science lessons was 11% points, while in 4th grade classes it amounted to 8% points with regard to
the use of computers for studying natural phenomena through simulations. The Russian Federation
(21% points) outranked all other countries in the increase of students using computers to conduct
experiments during 8th science classes, whereas the opposite trend was particularly pronounced in
Korea (32% points). Similarly, the Russian Federation (18% points) exhibited the largest increase in
the use of computers for studying natural phenomena in 4th grade. These country changes presented
medium effect sizes, whereas OECD average absolute change showed a small effect size as per 4th
grade students using computers for the studying natural phenomena. Altogether, 8th grade students
made larger use of computers to conduct scientific experiments in eight education systems, while
the opposite was true for seven education systems between 2003 and 2011, with at least small effect
sizes. Likewise, with small effect sizes 4th grade students used computers more for studying natural
phenomena in 11 education systems.
The Russian Federation, Hungary and Norway stood out as education systems where innovation
was illustrated in a broader use of computers for data analysis and scientific experimentation
simultaneously across educational levels. For example, with at least small effect sizes, Norwegian
students used computers more to analyse data during mathematics and science classes and to
conduct scientific experiments in primary and secondary education between 2003 and 2011.
In contrast, innovation in England and Korea was characterised by lower use of computers to
process and analyse data and conduct scientific experiments simultaneously across disciplines.
Eighth-grade Korean students, for example, used computers significantly less to analyse data during
mathematics and science lessons and to conduct scientific experiments in 2011 than 2003 with at

Figure11.9 Computer use to analyse data in 8th grade maths

Percentage of students using computers to analyse data at least sometimes and change over time
% -29 -25 -15 -15 -14 -7 -6 -6 -6 -5 -5 -3 -3 -2 -1 -1 -1 -0 +0 +1 +4 +5 +10 +11 +22

% point
70 30
60 25
50
20
40
15
30
10
20
10 5
0 0
* * * * * a * s l * n e o e y y s a a a y * * * *
d** ea** ia** nd* na* hin pan tate srae an** ede por tari Chil Ital rke sett fric nesi esot gar ebec ralia ay* ed.**
g lan Kor ven eala ndia g, C Ja d S I
m
e Sw
n g a On Tu hu
h A do
n n u n u
Q s t orw F
n
En Slo w Z I on ite CD Si ac
ss Sou
t In Mi H Au N sia
Ne ng
K Un OE Ma Ru
s
H o
1 2 http://dx.doi.org/10.1787/888933084836
Figure11.10 Computer use to analyse data in 8th grade science

% -24 -21 -15 -12 -11 -11 -8 -6 -5 -4 -2 -2 -2 -2 -0 +0 +1 +2 +5 +8 +9 +9 +9 +15 +27

% point
70 30
60 25
50
20
40
15
30
10
20
10 5
0 0
*** *** *** tts d* *** s** el ey en an d ay na aly rio ica ec nia lia ia** ile* ota *** .***
na pore orea use glan pan tate Isra urk wed me alan orw Chi It nta Afr ueb ve stra es Ch es ary Fed
dia K ach En Ja d S T S Ze N ng, O th Q Slo Au don nn ng n
In inga s e C D
E ew o u In Mi Hu ssia
S a s i t O N K So
M U n n g Ru
Ho
1 2 http://dx.doi.org/10.1787/888933084855

TIMSS (2003, 2007 and 2011) surveys asked 8th grade teachers Do the students in this class have computer(s)
available to use during their mathematics/science lessons? with answer options Yes/No. Conditional on
answering yes to this question teachers are asked How often do you have the students do the following
computer activities during mathematics/science lessons? c) Process and analyse data with answer options
Every or almost every day; Once or twice a week; Once or twice a month; Never or almost never. The
same data restrictions as in International TIMSS and PIRLS Reports by the International Association for the

Figure11.11 Computer use to conduct scientific experiments in 8th grade science

% -32 -16 -15 -15 -12 -10 -9 -3 -3 -1 -0 +1 +1 +3 +3 +4 +8 +9 +10 +11 +14 +16 +17 +19
% point
+21
90 35
80
30
70
25
60
50 20
40 15
30
10
20
10 5
0 0
*** *** tts *** a** na *** ael tes ota nd ica an en ey aly rio c** *** *** a** *** *** *** .***
rea nd se ore in dia an Isr Sta nes eala Afr me wed Turk It nta uebe esia enia trali rway gary hile Fed
Ko ngla achu gap g, Ch In Jap d n Z h D S O Q n v s n C an
E ss Sin on ite Mi ew Sout OEC do Slo Au No Hu si
Un In us
Ma n g K N R
Ho
1 2 http://dx.doi.org/10.1787/888933084874
Figure11.12 Computer use to study natural phenomena through simulations in 4th grade science
% -9 -7 -4 -3 -3 -3 -1 +0 +1 +5 +6 +7 +7 +8 +8 +8 +9 +10 +10 +10 +11 +11 +14

% point
+18
70 25
60
20
50
40 15
30 10
20
5
10
0 0
an rio nd ny tria en blic rta blic n*** ark s** ry* ia** ds* lia h** c** d** re** y** na* y*** .***
Jap Onta ngla rma Aus wed epu Albe epu ea nm State nga ven rlan ustra mis uebe alan apo orwa Chi Ital Fed
E Ge S
hR kR D
m De d
e Hu Slo the A Fle Q Ze Sing N ng, sia
n
Cz
ec l ova OEC
U nit Ne g i um N ew g Ko R us
S l n
Be Ho
1 2 http://dx.doi.org/10.1787/888933084893

computer(s) available to use during their science lessons? with answer options Yes/No. Conditional on
answering yes to this question teachers are asked How often do you have the students do the following
computer activities during science lessons? c) Do scientific procedures or experiments, d) Study natural
phenomena through simulations with answer options Every or almost every day; Once or twice a week; Once
or twice a month; never or almost never. The same data restrictions as in International TIMSS and PIRLS Reports

Use of computers for reading and writing
General findings
Innovation in the classroom has also resulted in either increased or decreased student use
of computers to read and write text in primary education (Figure11.13 and Figure11.14). Between
2001 and 2006 the OECD average absolute change regarding students using computers to read text
was 7% points, while it amounted to 8% for the use of computers to write text in primary education.
Hong Kong (39% points), the Netherlands (19% points) and the Slovak Republic (12% points) are
the education systems showing the largest increases in the use of computers for reading purposes
with medium to large effect sizes.The OECD exhibited a small average absolute change effect size
on this aspect of computer use at 4th grade. The Slovak Republic (11% points) exhibited the most
substantial increase concerning writing text on computers in primary with a medium effect size.
Overall, reading text on computers increased in nine education systems and decreased in three with
at least small effect sizes, while writing text on computers broadened in six education systems and
diminished in four with at least small effect sizes.
The Slovak Republic, the Netherlands and Norway are countries where innovation was exemplified
in additional use of computers for reading and writing. In these countries, students used computers
more widely to read and write text between 2001 and 2006, with at least small effect sizes.
In contrast, use of computers to read and write text particularly decreased in New Zealand and
Israel. Fourth grade students in these countries made less use of computers to read and write text in
2006 than in 2001, with small effect sizes.

Figure11.13 Computer use to read text in 4th grade

Percentage of students using computers to read stories or other texts at least once or twice a week and change over time
% -15 -10 -7 -3 -3 -0 +0 +1 +2 +3 +4 +4 +4 +5 +6 +7 +7 +12 +15 +19 +39

% point
50 40
35
40
30
30 25
20
20 15
10
10
5
0 0
w *** ** l* es io en nd ce * *** ry* ore any * * ** ** c*** d*** s*** ***
Ne and ebec Israe Stat ntar wed otla Fran n* ia* taly ed.* ay na
l S c ea land nga gap erm ven I F i
rw publ glan land , Chi
a u d O S m e u i n o ia n o
Ze Q ite CD Ic H S G Sl ss N Re En ther ong
Un OE Ru va
k
Ne ng K
l o
S Ho
1 2 http://dx.doi.org/10.1787/888933084912
Figure11.14 Computer use to write text in 4th grade

Percentage of students using computers to read stories or other texts at least once or twice a week and change over time
% -21 -15 -12 -12 -8 -6 -2 -1 -1 -1 -1 +0 +0 +2 +3 +4 +7 +9 +11 +15 +17

% point
70 25
60
20
50
40 15
30 10
20
5
10
0 0
*
l** nd**
* s** d** nd de
n io ean nia d any nce ary be
c
a** por
e ** ** ** ** y***
ar e lan d.* ds blic* aly*
a ate tlan ngla Swe Ont D m Slov ce Germ Fra ung Que Chin nga Fe rlan It orw
a
Isr eala d St o E C I H , i n e p u
Z i t e Sc OE o ng S
s sia eth k Re N
w
Ne Un ng
K Ru N va
Ho Slo
1 2 http://dx.doi.org/10.1787/888933084931

PIRLS (2001 and 2006) surveys asked 4th grade teachers: Are computers available for use by your class? with
answer options Yes/No. Conditional on answering yes to this question teachers are asked How often do
you have students do the following computer activities? c) Read stories or other texts on the computer, e) Use
the computer to write stories or other texts with answer options Every or almost every day; Once or twice
a week; Once or twice a month; Never or almost never. The same data restrictions as in International TIMSS
and PIRLS Reports by the International Association for the Evaluation of Educational Achievement (IEA) apply
to these data.

Summary
Innovation or significant change in the use of computers during lessons varied across countries
and pedagogical purposes. Change in the OECD area was unequal across disciplines, where use of
computers touched primarily mathematics or science depending on the pedagogical purposes, and
levels. Notably the Russian Federation is the only country exhibiting consistent increases across
pedagogical purposes, disciplines and levels between 2003 and 2011.
Figure11.15 Change in the use of computers to practice skills and procedures in 8th grade
Practice of skills and procedures-science Practice of skills and procedures-maths

%
80
60
40
20
0
-20
-40
-60
l
tes Italy eden esia ebec srae enia land hina hile land pore apan tario iana gary ralia orea rway tion ean ean ge)
Sta Sw Indo
n
Qu I v a C C ng a J On Ind Hun Aus
t K No ra D m m han
d Slo w Ze ng, E Sing de D
ite o Fe OEC OEC bs. c
Un Ne ng K a n (a
o s si
H Ru
1 2 http://dx.doi.org/10.1787/888933084950
Figure11.16 Change in the use of computers to practice skills and procedures in 4th grade
Practice of skills and procedures-science Practice of skills and procedures-maths

%
60
40
20
-20
-40
ali
a tes nia ina ar
io nd
s
Ita
ly
eb
ec
po
re an nd ary mish land rway ation ea
n n e)
ea g
str Sta love g, Ch Ont rla Qu ga Jap ngla ung ea m mchan
Au ited S n h e i n E H F le Z No eder C D D
t S um New C s.
Un Ko Ne F OE OE (ab
ng lgi ian
Ho Be u ss
R
1 2 http://dx.doi.org/10.1787/888933084969
Notes: For details, please see Figures11.1, 11.2, 11.3 and 11.4.

Figure11.17 Change in the use of computers to look up ideas and information in 8th grade
Information research-science Information research-maths

%
80
60
40
20
0
-20
-40
-60
l en nia bec hile Italy ralia land esia way ates iana land pan gary hina pore orea tion ean ean ge)
ae rio
Isr nta wed love Que C st ea on Nor d St Ind Eng Ja un ,C ga K era m n
O S S Au w Z Ind H ng Sin d D D m ha
ite Ko Fe OEC OEC bs. c
Ne Un n g i a n ( a
Ho ss
Ru
1 2 http://dx.doi.org/10.1787/888933084988
Figure11.18 Change in the use of computers to look up ideas and information in 4th grade
Information research-science Information research-maths

%
60
40
20
-20
-40
es lia pore ly an ec nd nia rio ina mish and
s nd ary rway ation ea
n n e)
tat tra ga
Ita Jap eb ala love Onta , Ch rl gla ung m ea g
mchan
S
Au
s Qu Ze F l e e n N o e r D
ted Sin w S n g th E H ed C CD s.
Un
i
Ne Ko gium Ne nF OE OE (ab
o ng B el s sia
H Ru
1 2 http://dx.doi.org/10.1787/888933085007
Notes: OECD average includes all OECD education systems. For details, please see Figures 11.5, 11.6, 11.7 and 11.8.
Figure11.19 Change in the use of computers to analyse data in 8th grade
Data analysis-science Data analysis-maths

%
60
40
20
-20
-40
-60
ly io ec den ina esia rael hile way ates and pan alia ary nia ore ana and rea tion ean ean ge)
Ita ntar ueb e h n Is C r t al Ja ustr ung love gap ndi ngl Ko era m n
O Q Sw ng, C Indo No ed S Ze H S in I E d D D m ha
K o n i t e w A S
n Fe OEC OECbs. c
n g U N i a ( a
Ho ss
Ru
1 2 http://dx.doi.org/10.1787/888933085026

Figure11.20 Change in the use of computers to study and conduct scientific experiments
Study of natural phenomena-4th grade Scientific experiments-8th grade

%
60
40
20
-20
-40
en ds ates ish iana and ario taly pan enia hile bec and alia ina gary ore way rea tion ean ean ge)
ed lan t m nd al Ont I Ja lov C e l tr h p r Ko era n
Sw ther ted S Fle I Ze S Qu Eng Aus g, C Hun inga No d
m
D D m ha
e i m ew o n S Fe OEC OECbs. c
N Un elgiu N n g K
i a n ( a
B Ho ss
Ru
1 2 http://dx.doi.org/10.1787/888933085045
Notes: OECD average includes all OECD education systems. For details, please see Figures, 11.9, 11.10 ,11.11 and 11.12.
Figure11.21 Change in the use of computers to read or write text in 4th grade
Writing text Reading text

%
60
40
20
-20
-40
e d y y io den nia ore bec and ion tes taly and blic y s l d a n an e)
nc lan gar an ar wa nd Israe alan hin ea g
e p e l
Fra Ice Hun erm Ont Swe Slov nga Qu cot dera d St
t a I
n gl epu
N or erla e , C D m mehan
E Z g D
G Si S Fe
n ite a kR eth ew Kon OEC OECbs. c
ia n U v N N (a
ss Slo ng
Ru Ho
1 2 http://dx.doi.org/10.1787/888933085064
Notes : For details, please see Figures 11.13 and 11.14.
Note on Israel

Table 11.1 Effect sizes for changes in the use of computers in classrooms
To practice skills and procedures To lookup information To analyse data To read/

To conduct write text
scientific
experiments
8th grade 4th grade 8th grade 4th grade 8th grade 4th grade
Mathematics
Mathematics
Mathematics
Mathematics
Mathematics
Write text
Read text
8th grade
4th grade
Science
Science
Science
Science
Science
03-11 03-11 03-11 03-11 03-11 03-11 03-11 03-11 03-11 03-11 03-11 03-11 01-06 01-06
Australia 0.38 0.37 0.09 -0.03 0.13 -0.14 0.13 -0.08 0.21 0.16 0.29 0.16 m m
Belgium Flemish m m -0.25 0.45 m m -0.20 0.43 m m m 0.23 m m
Ontario 0.12 0.30 -0.21 -0.14 0.04 0.11 -0.29 -0.22 -0.06 0.01 0.18 -0.16 -0.09 -0.04
Quebec 0.12 0.14 -0.38 0.11 0.23 0.07 -0.43 -0.08 0.23 0.04 0.27 0.26 -0.28 0.07
Chile -0.07 0.22 m m -0.15 0.07 m m -0.05 0.18 0.40 m m m
France m m m m m m m m m m m m 0.03 0.00
Germany m m m m m m m m m m m m 0.12 -0.02
Hungary 0.25 0.59 0.43 0.47 0.40 0.45 0.36 0.54 0.10 0.36 0.48 0.21 0.23 0.03
Iceland m m m m m m m m m m m m 0.08 -0.02
Israel -0.12 -0.09 m m -0.05 -0.09 m m -0.13 -0.12 -0.08 m -0.21 m
Italy 0.05 0.11 0.09 0.48 0.26 0.05 0.27 0.23 -0.03 -0.01 0.12 0.47 0.20 0.39
Japan -0.53 -0.27 -0.59 0.11 -0.30 -0.60 -0.25 -0.24 -0.20 -0.33 -0.39 -0.18 m m
Korea -0.54 -0.22 m m -0.72 -0.40 m m -0.53 -0.30 -0.71 m m m
Netherlands m m 0.14 0.35 m m 0.05 0.56 m m m 0.25 0.56 0.22
New Zealand -0.09 0.21 0.73 0.23 -0.12 -0.18 0.46 -0.01 -0.35 -0.04 -0.01 0.20 -0.34 -0.29
Norway 0.28 0.66 0.49 0.60 -0.15 0.28 0.44 0.44 0.23 -0.03 0.37 0.31 0.25 0.43
Slovak Republic m m m m m m m m m m m m 0.50 0.51
Slovenia -0.16 -0.07 0.08 0.13 -0.13 -0.08 0.15 0.40 -0.38 0.11 0.30 0.21 0.23 -0.05
Sweden 0.04 0.13 m m -0.08 -0.10 m m -0.13 -0.08 0.08 m 0.00 -0.12
England -0.20 -0.12 -0.34 -0.25 -0.41 -0.22 -0.27 -0.43 -0.59 -0.22 -0.34 -0.07 0.33 -0.17
Scotland m m m m m m m m m m m m 0.01 -0.27
United States -0.03 -0.05 0.07 0.08 -0.26 -0.20 -0.02 -0.15 -0.13 -0.16 -0.05 0.14 -0.08 -0.27
Indiana 0.08 -0.31 m m -0.37 -0.28 m m -0.34 -0.49 -0.20 m m m
OECD (average) -0.03 0.11 0.05 0.18 -0.08 -0.07 0.05 0.08 -0.11 -0.04 0.04 0.15 0.09 -0.01
OECD (average absolute) 0.22 0.24 0.30 0.25 0.24 0.22 0.26 0.28 0.23 0.15 0.28 0.22 0.21 0.20
Hong Kong, China -0.15 -0.19 -0.22 0.10 -0.32 -0.51 -0.34 -0.20 -0.19 -0.04 -0.26 0.24 1.06 0.23
Indonesia 0.00 0.22 m m 0.18 0.30 m m 0.01 0.27 0.33 m m m
Russian Federation 0.83 1.01 0.73 0.96 0.92 0.88 0.67 0.85 0.62 0.73 0.65 0.72 0.31 0.41
Singapore -0.18 -0.16 -0.43 0.11 -0.34 -0.56 0.07 -0.29 -0.07 -0.45 -0.31 0.22 0.09 0.14
South Africa -0.06 0.09 m m -0.03 0.03 m m -0.01 0.04 0.06 m m m
1 2 http://dx.doi.org/10.1787/888933085083
Source: Authors calculations based on TIMSS (2003, 2007 and 2011) and PIRLS (2001 and 2006)

Box11.6. Conditional probabilities approach (1/2)

So far in this section, innovation in computer use has been identified by a change in computer use per se, to give
a clear idea about the combination of changes in classroom practices and computer availability between 2003 and
2011. However, TIMSS and PIRLS answers on computer use can also be analysed using the conditional probability
approach where the percentages are calculated over the subset of those answering yes to the question: Do the
students in this class have computer(s) available to use during their mathematics/science lessons?. The Figures
presented below employ this approach; they describe the effective change in use among those classrooms that had
a computer available during lessons. This method controls for the change in availability of computers and therefore
provides a clearer picture of the extent of any change in use. By comparing the results of these analyses with those
above, it is possible to see whether changes relate primarily to a change in the percentage of classrooms with access
to a computer, or changes in the proportion of classrooms making specific use of the computer.
Computer use to look up ideas and information in 8th grade mathematics in classrooms with computers.
% -35 -29 -27 -23 -19 -17 -14 -11 -9 -7 -5 -4 -4 -3 -1 +0 +5 +8 +9 +13 +15 +21 +21 +33
% point
+44
100 50
90 45
80 40
70 35
60 30
50 25
40 20
30 15
20 10
10 5
0 0
*** *** a** s*** d** *** ica ina hile pan an* nia ario den alia key ota bec rael etts ary esia nd** ly*** d.***
rea ay ian te an ore fr h C Ja me love Ont we ustr Tur nes Que Is hus ung on ala Ita Fe
Ko Norw Ind Sta Engl gap th A ng, C S A n H Ind Ze
d i n ou Ko CD S
Mi ss
ac ian
i t e S S O E w ss
Un o ng Ma Ne Ru
H
1 2 http://dx.doi.org/10.1787/888933085102
2011 instead of 2003 and 2011 for Turkey and United States (Massachusetts and Minnesota). OECD average includes all OECD education systems
for which data is available for all years concerned.
Computer use to look up ideas and information in 4th grade science in classrooms with computers
% -13 -11 -7 -7 -6 -5 -4 -4 -4 -2 -2 -1 -1 -0 +0 +0 +2 +4 +4 +5 +10 +11 +25

% point
+30
100 30
90
80 25
70
20
60
50 15
40
30 10
20
5
10
0 0
g, * * c * c c s * .
on na* en bli es* ark pan Italy land berta ebe land tario ralia pore stria mish ean any ubli enia and rway ary* Fed
n g Chi wed epu Stat enm Ja
K
E n g Al Q u
Z e a On ust ga Au le
F D
m erm ep lov erl No n g ian
Ho S h R ed D A Si n C G kR S th Hu uss
w um OE Ne
ec nit Ne lgi va R
Cz U Be Slo
1 2 http://dx.doi.org/10.1787/888933085121
Notes: *** = change significant at the 0.01 level; ** = change significant at the 0.05 level; * = change significant at 0.1 level; Change between 2007
and 2011 instead of 2003 and 2011 for Austria, Canada (Alberta), Czech Republic, Denmark, Germany, Slovak Republic and Sweden. OECD average
includes all OECD education systems for which data is available for all years concerned.

Box11.6. Conditional probabilities approach (2/2)

Computer use to analyse data in 8th grade mathematics in classrooms with computers
% -33 -32 -24 -24 -20 -15 -11 -10 -9 -8 -8 -7 -3 -3 -1 -1 +0 +1 +2 +3 +3 +5 +8 +11

% point
+18
100 35
90
30
80
70 25
60 20
50
40 15
30 10
20
10 5
0 0
* * * * * * * s a y y n n y a e l a e a d a . c s
d** ia** na* ea** ry* rio* n** ate fric rke wa de apa Ital nesi Chil srae esot por hin lan trali Fed ebe sett
g lan ven ndia Kor unga nta mea d St h A Tu Nor Swe J
d o I
n n nga g, C Zea us ian Qu chu
En Sl o I O e t In i i n A s s a
H CD Unit Sou M S
Ko New Ru ss
OE ng Ma
Ho
1 2 http://dx.doi.org/10.1787/888933085140
2011 instead of 2003 and 2011 for Turkey and United States (Massachusetts and Minnesota). OECD average includes all OECD education systems
for which data is available for all years concerned.
Computer use to write text in 4th grade reading in classrooms with computers
% -15 -14 -12 -12 -8 -7 -5 -4 -3 -2 -2 -1 -1 +3 +3 +4 +7 +9 +15 +18 +20

% point
80 20
70
60 15
50
40 10
30
20 5
10
0 0
ae
l y n
d** tes** nd** land man ede veni ranc
a e n ario nd** ary be
c *
na po
re lic s** *** *** .***
ea ub and Italy way Fed
Isr alan
S t a t la n g e r S w Slo F m Ont cela ung Que , Chi nga ep r l r
o E D I H i
Ze ted Sc G C ng S k R he No sian
w i OE Ko va Net s
Ne Un ng Slo Ru
H o
1 2 http://dx.doi.org/10.1787/888933085159

CHAPTER 12
Innovation
in the provision of special education
in schools
Innovation in schools can take the form of providing special education

opportunities for students. Schools may choose to offer remedial education to
students who need additional support to catch up or keep up with the required
skill level of their grade. Schools may also innovate by offering enrichment
education for students who have specific interest in a certain discipline and
would flourish with extra challenges. The aim of innovation with regard to
increasing the use of special education could be, for example, to reduce the
inequality in terms of student outcomes and avoid grade repetition, while
giving talented students the opportunity to reach their full learning potential.
A reduction in the use of remedial education may reflect an innovation such
as a move to whole class activities.

12. INNOVATION IN THE PROVISION OF SPECIAL EDUCATION IN SCHOOLS
Remedial education
General findings
Much innovation has taken place in the form of providing secondary education mathematics and
science students with more or less remedial education (Figure12.1 and Figure12.2). Between 1999
and 2007, the average absolute change within the OECD area regarding remedial education provision
was 17% points for 8th grade mathematics and 16% points for 8th grade science. In mathematics, the
increases were the most prominent in Korea (54% points), England (33% points), Turkey (27% points),
Quebec (24% points) and Japan (21% points). The decrease was notable in Ontario (19% points).
Regarding science, the largest increases took place in Quebec (47% points) and England (37% points).
All these changes in individual education systems corresponded to large or medium effect sizes. In
contrast,the OECD average and average absolute changes in maths, and average absolute changes in
science, presented small effect sizes. Overall, the provision of remedial mathematics increased with
at least small effect sizes in eight education systems and decreased in three others. The percentage
of 8th grade students attending schools that offered remedial science decreased in seven education
systems, while it increased in six others with at least small effect sizes.
Compared to secondary schools, innovation concerning remedial education provision has been
less apparent in primary school mathematics, science and reading although within a shorter
period of time (Figure12.3 to Figure12.5). For both 4th grade science and mathematics remedial
education, the average absolute change for the OECD area was only 5% each between 2003 and
2007. As to having a remedial reading specialist always available in 4th grade classrooms, the average
absolute change was 4% points between 2001 and 2006. In mathematics, the increase was prominent
in the Russian Federation (26% points) with a medium effect size. In general, the percentage of 4th
grade students attending schools that provided remedial mathematics increased with at least small
effect sizes in two education systems and decreased in one system. For science, four education
systems experienced increases with small effects, while two others saw decreases. With small effect
sizes, two education systems saw decreases and two others increases in the percentage of students
whose teachers reported that a reading specialist is always available in their classroom to work with
students with reading difficulties.
The provision of remedial education for students increased simultaneously in several levels or
disciplines in Quebec, England, Israel and the Russian Federation. With at least small effect sizes,
Quebec, for example, experienced increases in the percentage of students attending schools that
provided remedial mathematics and science both at 8th and 4th grades within the period 2003 to 2007.
By contrast, remedial education provision decreased in a comprehensive manner in Ontario.

The provision of both remedial mathematics and science education decreased both at 4th and 8th
grades in the Ontario education system from 2003 to 2007 with at least small effects.
Direction of innovation concerning remedial education has not been consistent across education
levels and disciplines, as illustrated by the cases of Hong Kong and Singapore. For example in
Singapore, remedial mathematics and science in the 8th grade decreased from 1999 to 2007, but
increased in science 4th grade within the same period.

Figure12.1 Remedial mathematics provision in 8th grade

Percentage of students in schools that offer remedial mathematics and change over time
% -19 -9 -8 -6 -5 -3 +0 +2 +3 +4 +4 +7 +14 +14 +15 +20 +21 +24 +27 +33 +54
% point
100 60
90
80 50
70
40
60
50 30
40
30 20
20 10
10
0 0
** el ry ** ** ly d lia sia en lic Fed n*** s*** y*** a*** n*** c*** *** d*** a***
io* Isra ount enia pore Ita tlan stra one wed pub a te r n a e ey n e
n tar e c l ov g a S co Au Ind S R e s ian me Sta unga , Chi Jap ueb Turk ngla Kor
s Q
O
sq
u S Sin z ec
h Ru ECD ited H ong E
B a C O Un K
ng
Ho
1 2 http://dx.doi.org/10.1787/888933085178
instead of 1999 and 2007 for Canada (Ontario and Quebec), Scotland,Spain (Basque country) and Sweden. OECD average includes all OECD education
systems for which data is available for all years concerned.
Figure12.2 Remedial science provision in 8th grade

Percentage of students in schools that offer remedial science and change over time
% -22 -17 -15 -13 -12 -11 -5 -4 -3 -2 -0 -0 +1 +4 +10 +14 +19 +22 +23 +37 +47
% point
100 50
90 45
80 40
70 35
60 30
50 25
40 20
30 15
20 10
10 5
0 0
** ** *** ** rio* lia** re** ates enia d esia . * * * * * *
ina Fed ean* eden ea** an** ry** ey** nd** ec**
*
ly* ry* rael lic t an
Ita unt I s p ub Onta stra gapo ed S Slov cotl don g, Ch ian m S w Ko r
Ja p nga urk gla ueb
co Re Au Sin nit S In n ss D Hu T En Q
ue ch U Ko Ru OEC
a sq C ze o ng
B H
1 2 http://dx.doi.org/10.1787/888933085197

Figure12.3 Remedial mathematics provision in 4th grade

Percentage of students in schools that offer remedial mathematics and change over time
% -20 -5 -4 -1 -1 -1 +0 +0 +1 +2 +2 +2 +3 +4 +16 +26

% point
100 30
90
80 25
70
60 20
50
15
40
30 10
20
10 5
0 0
io*
** nd a tes ly or
e nia ea
n
nd
s d
ali
a d ary an *** d**
*
la hin Sta Ita p ve rla otl
an
str lan ng Jap ec Fe
tar Z ea
g ,C d ga Slo D
m e c u E ng u u eb n
On w n ite Sin C t h S A H Q ss
i a
Ne Ko Un OE Ne Ru
ng
Ho
1 2 http://dx.doi.org/10.1787/888933085216
Figure12.4 Remedial science provision in 4th grade

Percentage of students in schools that offer remedial science and change over time
% -12 -9 -3 -3 -2 -2 -1 -0 +0 +1 +1 +5 +7 +10 +12 +13

% point
100 16
90 14
80
12
70
60 10
50 8
40 6
30
4
20
10 2
0 0
** ly* es a d d ds ea
n ry ia lia an d* e** c** ***
tar
io Ita Sta
t hin lan lan an ga en tra Jap tla
n or be ed
On d g ,C Z ea E ng erl D
m un lov us o g ap ue nF
it e n w t h C H S A Sc Si n Q
ss
i a
Un Ko Ne Ne OE Ru
ng
Ho
1 2 http://dx.doi.org/10.1787/888933085235

Figure12.5 Availability of specialists to work with 4th grade students with reading difficulties
Percentage of students in schools with reading specialists always available in classrooms to work with students with reading difficul-
ties and change over time
% -10 -8 -8 -2 -2 -2 -2 -2 -1 -1 -1 -0 -0 +1 +2 +5 +5 +5 +6 +7 +9
% point
40 12
35 10
30
8
25
20 6
15
4
10
2
5
0 0
** ** s l e ** e y ** s ** **
n* ec nd rio nia ina rae y
an por
y n d d d
wa taly* mea ranc tlan ngar blic* glan alan tate and* ed.*
e de ueb erla Onta love , Ch Is
e rm nga Nor I D F c o u u n e d S e l F
Sw Q eth S g Si S H p E Z Ic sia n
Ko
n G
OE
C e w nite
N kR Ne s
ng va U Ru
Ho Slo
1 2 http://dx.doi.org/10.1787/888933085254

TIMSS (1999, 2003 and 2007) surveys asked school principals and department heads Does your school do
any of the following for students in the <eighth-grade> / <fourth-grade>? [] Offer remedial mathematics
[] science with answer options Yes/No. PIRLS (2001 and 2006) survey asked teachers Are the following
resources available to you to deal with students who have difficulty with reading? [] A <reading specialist>
is available to work in my classroom with those students, with response options Always; Sometimes; Never.

Enrichment education
General findings
Innovation is apparent through several decreases and some increases in the provision of
enrichment education in secondary education mathematics and science (Figure12.6 and Figure12.7).
Within the period 1999 to 2007 the average absolute change in the OECD area was 16% points for
8th grade mathematics and 15% points for 8th grade science enrichment. The largest decreases in
enrichment education for 8th grade science took place in the Russian Federation (50% points), Israel
(30% points) and Hong Kong (25% points), while England (34% points) and Japan (30% points) saw the
greatest increases. As for mathematics, the decreases were notable in the Russian Federation (43%
points) and Ontario (26% points) and the increases in Korea (41% points), Japan (37% points) and
England (31% points). Changes in individual education systems presented large to medium effect
sizes, whereas, in the case of the OECD average and average absolute, effect sizes were small for 8th
grade mathematics and average absolute effect sizes were small for 8th grade science. The percentage
of students attending schools that offer enrichment education in mathematics decreased in seven
education systems and increased in six others, with at least small effect sizes. The provision of
enrichment education in science decreased in eight education systems, while increasing in four
others with at least small effect sizes.
Within a shorter time period, innovation with regard to enrichment education in primary school
mathematics, science and reading has meant increased or decreased provision in some systems
(Figure12.8 to Figure12.10). Between 2003 and 2007, the average absolute change for the OECD
area regarding 4th grade enrichment education was 8% points for mathematics and 7% points for
science. As to the provision of informal reading initiatives in the 4th grade, the average absolute
change for the OECD area was 4% points between 2001 and 2006. The increase in mathematics was
prominent in Japan (26% points) and the decrease notable in Ontario (25% points) as illustrated
by medium effect sizes. Overall, the percentage of 4th grade students attending schools that provide
mathematics enrichment classes increased with medium to small effect sizes in three education
systems and decreased in one system. For science, three education systems experienced increases
with small effect sizes, while two others saw decreases. In addition, five education systems saw
increases in the percentage of 4th grade students whose schools have informal initiatives such as
book clubs to encourage students to read,with small effect sizes.
Japan and England stand out as education systems where innovation took the form of increased
enrichment education provision in several education levels or disciplines simultaneously. For
example, with at least small effect sizes, England saw increases in the percentage of students
attending schools that provided enrichment classes in 8th grade mathematics and science as well as
in 4th grade science during the period 1999 to 2007.
However, enrichment education provision decreased across several disciplines or education

levels in Turkey, Ontario and Indonesia. For example, enrichment education in both mathematics
and science in the 8th grade decreased in Indonesia between 1999 and 2007 with small effects.
The cases of Slovenia, the Russian Federation and Hong Kong illustrate that innovation is
not always consistent across disciplines and levels of education. For example, although 8th grade
mathematics and science enrichment education provision decreased in the Russian Federation
from 1999 to 2007 with large effect sizes, it increased in 4th grade enrichment mathematics with
small effect sizes. The provision of informal reading initiatives also increased in Russian primary
schools between 2001 and 2006.

Figure12.6 Enrichment mathematics provision in 8th grade

Percentage of students in schools that offer enrichment mathematics and change over time
% -43 -26 -20 -18 -15 -9 -8 -7 -6 -4 -4 -2 -0 -0 +7 +7 +12 +13 +31 +37 +41
% point
100 45
90 40
80 35
70
30
60
25
50
20
40
30 15
20 10
10 5
0 0
** ** *** ** ** *** ry lia ia*** c a d s y l * ** ** d*** ** a***
d.* io* en blic* key* esia nt tra n be hin tlan tate Ital Israe pore ary an* n*
Fe ntar wed u Tur on c ou Aus love Q ue g, C c o d S g a n g
m
e g l an Japa Kore
n p S Hu CD
ss
ia O S
hR
e In
d ue S Ko
n ite Sin En
Ru ec sq ng Un OE
Cz Ba Ho
1 2 http://dx.doi.org/10.1787/888933085273
Figure12.7 Enrichment science provision in 8th grade

Percentage of students in schools that offer enrichment science and change over time
% -50 -30 -25 -14 -14 -14 -9 -7 -6 -5 -5 -5 -3 +2 +4 +4 +7 +13 +15 +30 +34
% point
100 55
90 50
80 45
70 40
60 35
50 30
25
40 20
30 15
20 10
10 5
0 0
* * *
d**
*
l**
* a** ey*** lic*** sia** io nia* gary nd ntr
y
de
n a y ** s c ** **
ali Ital ean tate ebe ore* rea* an* nd*
** **
Fe Israe Chin ar
n u r k u b ne O nt love Hun cotla cou Swe ustr m d S Q u ap K o Ja p g l a
ia g, T Rep ndo S S ue A CD nite g En
Ru
ss on h I sq OE U Sin
n gK z ec Ba
C
Ho
1 2 http://dx.doi.org/10.1787/888933085292

Figure12.8 Enrichment mathematics provision in 4th grade

Percentage of students in schools that offer enrichment mathematics and change over time
% -25 -7 -3 -3 -2 -1 +0 +2 +5 +5 +6 +6 +7 +8 +15 +26

% point
100 30
90
80 25
70 20
60
50 15
40
30 10
20
5
10
0 0
** s a ary re d ds n a d c ly d ** d**
* ***
rio
* te ali po lan an ea hin otlan be Ita gla
n
nia
*
an
ta Sta str ng ga ea erl m ,C ue
En ve Fe ap
On ite
d Au Hu S i n Z t h C D n g S c Q
Slo i a n J
Un Ne
w Ne OE Ko ss
ng Ru
Ho
1 2 http://dx.doi.org/10.1787/888933085311
Figure12.9 Enrichment science provision in 4th grade

Percentage of students in schools that offer enrichment science and change over time
% -15 -12 -6 -5 -5 -3 -0 -0 -0 +0 +1 +5 +6 +10 +13 +15

% point
100 16
90 14
80
12
70
60 10
50 8
40 6
30
4
20
10 2
0 0
*** *** ds lia io ry ea
n ly re an
d nia ina d* an
* *** ***
lan
d es an tra tar ga Ita ap
o
otl ve Ch Fe
Jap ec nd
a tat herl us On un D
m g Sc Slo g, ian u eb gla
Ze d S t A H C Sin n s s Q E n
w ite Ne OE Ko Ru
Ne Un ng
Ho
1 2 http://dx.doi.org/10.1787/888933085330

Figure12.10 Provision of informal initiatives to encourage 4th grade students to read

Percentage of students in schools that offer informal initiatives to encourage students to read and change over time
% -3 -2 -2 -2 -1 -0 +0 +1 +1 +2 +2 +2 +3 +3 +4 +4 +5 +5 +11 +12 +18

% point
100 20
90 18
80 16
70 14
60 12
50 10
40 8
30 6
20 4
10 2
0 0
d io d re el es ia y ec d ce ay *** ** ly ry den and lic** d.*** s***
lan ntar alan apo Isra Stat ven man ueb glan ran orw ean hina Ita nga e tl b e nd
Ice O Ze Sing d l o e r Q E n F N m C u Sw Sco epu an F erla
ew n ite
S G
C D ong, H
k R ssi t h
N U O E K va Ru N e
ng Slo
Ho
1 2 http://dx.doi.org/10.1787/888933085349

TIMSS (1999, 2003 and 2007) surveys asked school principals and department heads Does your school do any
of the following for students in the <eighth-grade> / <fourth-grade>? [] Offer enrichment mathematics []
science with answer options Yes/No. PIRLS (2001 and 2006) survey asked principals Does your school have
the following? [] Informal initiatives to encourage students to read (for example, book clubs, independent
reading contests, school-wide recreational reading periods) with answer options Yes/No The same data

Summary
Innovation or significant change regarding special education provision has meant both more
and less provision in secondary and primary education. Change in the OECD area was balanced
across disciplines and unbalanced across levels. Its direction can only be determined in the case
of secondary education, where it translated into a significant increase in remedial and enrichment
education offerings between 1999 and 2007.
Figure12.11 Change in the provision of remedial education in 8th grade
Science remediation Maths remediation

%
80
60
40
20
-20
-40
esi
a n ia ore alia bli
c tes a el ly ary an ey rea nd ean eannge)
tio ven ap str pu Sta hin Isra Ita ng Jap rk Ko gla
on era Slo ing Au Re g, C Hu Tu En D m ECD m ha
Ind F ed S h i ted o n
OE
C
O (abs. c
an ec Un K
ssi Cz ng
Ru Ho
1 2 http://dx.doi.org/10.1787/888933085368
Figure12.12 Change in the provision of remedial education in 4th grade

%
60
40
20
-20
-40
ia ds alia ngary gland tes nd ina an
d an Ita
ly ore ec tar
io nv ean eannge)
ven lan str Sta ala , Ch Scotl Jap ap eb tio
Slo ether Au Hu En d Ze ng g Qu On era D m ECD m ch
a
N n i te
e w K o Sin F ed O EC O b s.
U N ng i a n ( a
Ho R uss
1 2 http://dx.doi.org/10.1787/888933085387
Notes: For details, please see Figures 12.1,12. 2,12.3 and 12.4.
Note on Israel

Figure12.13 Change in the provision of enrichment education in 8th grade

%
80
60
40
20
0
-20
-40
-60
-80
-100
-120
ly tes alia ia ary ore esi
a a y
bli
c el rea nd an n an an e)
Ita Sta str ven ng ap hin rke pu Isra Ko gla Jap tio me D mehang
ed Au Slo Hu ing nd
on g, C Tu Re En era CD
i t S I o n h Fed E E C s. c
Un ng
K ec ian O O (ab
Cz uss
Ho R
1 2 http://dx.doi.org/10.1787/888933085406
Figure12.14 Change in the provision of enrichment education in 4th grade

%
60
40
20
-20
-40
ore ary d ly s ia ia a nd ec tes n nd io an an an e)
ap ng tlan Ita rla
nd stral ven hin ala eb Sta tio gla tar Jap me D mehang
g Hu Sco e Au Slo g, C w Ze Qu d era En On D
Sin N eth K o n e n ite Fed OEC C
OE(abs.
c
ng N U an
Ho ssi
Ru
1 2 http://dx.doi.org/10.1787/888933085425
Notes: OECD average includes all OECD education systems. For details, please see Figures 12.5, 12.6, 12.7 and 12.8.
Figure12.15 Change in the provision of special education in 4th grade reading
Reading specialist [teacher reported] Informal reading initatives [principal reported]

%
60
40
20
-20
nia ce ael ny ay ore taly ario ina and ary ates and and bec and den blic tion nds ean ean ge)
ve Fran Isr rma orw gap I t h tl g t l l e l e u a a
Ice Sw Rep der herl CD m D mchan
Slo Ge N On ng, C Sco Hun ed S Eng Zea Qu
Sn
i o n it e w a k F e e t E OECbs.
K U N v n N O (a
o ng Slo ssia
H Ru
1 2 http://dx.doi.org/10.1787/888933085444
Notes: OECD average includes all OECD education systems. For details, please see Figures12.5 and 12.10.

Table 12.1 Effect sizes for changes in the provision of special education
Change in remedial education Change in enrichment education
Remedial reading
Informal reading
mathematics
mathematics
mathematics
mathematics
Enrichment
Enrichment
Enrichment
Enrichment
initiatives
specialist
Remedial
Remedial
Remedial
Remedial
science
science
science
science
99-07 99-07 03-07 03-07 01-06 99-07 99-07 03-07 03-07 01-06
Australia 0,06 -0,23 0,04 0,07 m -0,16 -0,06 -0,07 -0,11 m
Ontario m m -0,46 -0,27 -0,05 m m -0,51 -0,11 -0,12
Quebec m m 0,37 0,30 -0,33 m m 0,13 0,39 0,03
Czech Republic 0,08 -0,29 m m m -0,45 -0,33 m m m
France m m m m -0,01 m m m m 0,05
Germany m m m m -0,07 m m m m 0,02
Hungary 0,38 0,44 0,09 0,01 0,07 0,26 -0,13 -0,06 -0,05 0,10
Iceland m m m m 0,16 m m m m -0,14
Israel -0,19 -0,36 m m -0,04 0,15 -0,65 m m -0,02
Italy -0,08 -0,46 -0,02 -0,20 -0,15 0,00 0,03 0,13 0,00 0,09
Japan 0,51 0,42 0,12 0,10 m 0,76 0,61 0,53 0,21 m
Korea 1,15 0,34 m m m 0,85 0,34 m m m
Netherlands m m 0,11 -0,04 -0,16 m m 0,01 -0,14 0,37
New Zealand m m -0,10 -0,11 0,13 m m -0,02 -0,32 -0,06
Norway m m m m -0,02 m m m m 0,06
Slovak Republic m m m m 0,45 m m m m 0,29
Slovenia -0,29 -0,06 0,00 0,02 -0,05 -0,37 -0,25 0,29 0,01 0,00
Sweden m m m m -0,32 m m m m 0,15
Turkey 0,56 0,47 m m m -0,42 -0,42 m m m
England 0,86 0,78 0,07 -0,04 0,14 0,65 0,70 0,14 0,38 0,09
Scotland m m 0,05 0,24 0,02 m m 0,11 0,02 0,21
United States 0,32 -0,10 -0,02 -0,09 0,13 0,00 0,09 -0,14 -0,30 0,00
OECD (average) 0,30 0,09 0,02 0,00 -0,01 0,24 0,08 0,04 -0,04 0,07
OECD (average absolute) 0,42 0,36 0,12 0,12 0,14 0,36 0,32 0,18 0,17 0,11
Hong Kong, China 0,43 -0,01 -0,13 -0,09 -0,11 -0,07 -0,53 0,09 0,13 0,36
Indonesia 0,15 -0,02 m m m -0,36 -0,49 m m m
Russian Federation 0,14 0,03 0,53 0,30 0,32 -0,98 -1,15 0,39 0,19 0,26
Singapore -0,30 -0,23 -0,15 0,29 -0,06 0,20 0,39 -0,08 -0,01 -0,12
1 2 http://dx.doi.org/10.1787/888933085463
Source: Authors calculations based on TIMSS (1999, 2003 and 2007) and PIRLS (2001 and 2006)

CHAPTER 13
Innovation
in the extent of teacher collaboration
in schools
Innovation in schools can take the form of increased or reduced collaboration

among teachers in different ways.Teachers may collaborate with their colleagues
by sharing knowledge or by preparing instructional materials together, or they
could work independently to develop materials that are highly tailored to their
specific class. The aim of innovation with regard to teacher collaboration could
be, for example, to foster the diffusion of particularly effective practices and to
favour collaborative learning environments for teachers. Additionally, teachers
could learn about new practices by observing what happens in their colleagues
classrooms. Alternatively, change may occur to reduce the potential stress of
being observed, or the time commitment required to observe colleagues.

13. INNOVATION IN THE EXTENT OF TEACHER COLLABORATION IN SCHOOLS
Instructional collaboration
General findings
Innovation in primary and secondary schools regarding instructional collaboration has meant
increased and decreased discussion among teachers on different ways of teaching a particular
concept (Figure13.1 to Figure13.3). Between 2003 and 2011 the OECD average absolute change
regarding frequency of peer discussions of this type was 12% points for 8th grade mathematics, 10%
points for 8th grade science and 5% points for 4th grade. Teachers in 8th grade mathematics discussed
more frequently especially in Israel (24% points), Slovenia (21% points), England (17% points), United
States (14% points) and Chile (13% points). On the contrary, discussion became less frequent in
the Russian Federation (21% points), Sweden (19% points), Hong Kong (14% points), Norway (12%
points) and South Africa (10% points). In 8th grade science Israel (34% points) presented the largest
increase in the frequency of discussions, whereas the decrease was noteworthy especially in the
Russian Federation (27% points). Primary education teachers discussed significantly more often
in Singapore (14% points), while the opposite was true in the Netherlands (15% points) and the
Russian Federation (14% points). These changes had from medium to small effect sizes, whereas
the OECD average absolute change in teacher peer discussions presented a small effect size in 8th
grade mathematics. Overall, teacher discussions in mathematics increased in six education systems
and decreased in six with at least a small effect size, whereas collaboration among science teachers
increased in two and decreased in five education systems. In primary education, teachers increased
their collaboration in one education system and decreased in three, with at least small effect size.
Innovation in instructional collaboration manifested also through more or less cooperation in

planning and preparing instructional materials (Figure13.4 to Figure13.6). Between 2003 and 2011,
OECD average absolute change amounted to 12% points in science and 10% points in mathematics 8th
grade, whereas it was 8% points for teachers in 4th grade. Eighth grade science teachers increased their
cooperation especially in Israel (24% points), whereas noteworthy decreases were observed in the
Russian Federation (35% points) and Hungary (34% points). Cooperation increased among 8th grade
mathematics teachers most notably in Israel (24% points), whereas decreases were remarkable in
Indonesia (33% points) and the Russian Federation (29% points). With regard to 4th grade, collaboration
in preparing instructional materials especially decreased in the Slovak Republic (30% points). These
changes were characterised by medium effect sizes. OECD average absolute effect sizes were small
for joint preparation of materials at 8th grade in both maths and science. Overall, teacher collaboration
in preparing instructional materials increased in three and decreased in nine education systems
for 8th grade science. Similarly, it increased in two education systems and decreased in eight in 8th
grade mathematics it with at least small effect size. In primary education, collaboration in preparing
instructional materials increased in two education systems and decreased in six.
Teacher collaboration practices increased across disciplines in Israel and United States. With
at least small effect sizes, teachers in these countries increased both peer discussions and joint
preparation of materials in 8th grade mathematics and science, between 2003 and 2011.
In contrast, teacher collaboration decreased substantially across disciplines in the Russian

Federation and Sweden. In these countries teachers in 8th grade mathematics and science decreased
the levels of peer discussions and joint preparation, with at least small effect sizes.
Direction of innovation regarding teacher collaboration has not been consistent among practices
in Norway. Norwegian teachers increased joint preparation with small effect size in 8th grade
mathematics and decreased peer discussions in 8th grade science between 2003 and 2011.

Figure13.1 Peer discussions amongst 8th grade mathematics teachers

Percentage of students who have a teacher who discusses with other teachers how to teach a particular topic once a week
or more and change over time
% -21 -19 -14 -12 -12 -10 -10 -8 -3 -2 -2 +1 +1 +4 +5 +5 +5 +6 +7 +11 +13 +14 +17 +21
% point
+24
80 25
70
20
60
50 15
40
30 10
20
5
10
0 0
*** *** ** ** c* ** d ry ey re ia lia n ** ea ts ly a ta o* ** *** ** *** ***
ed en ina way ebe rica lan ga rk po nes tra Japa ean Kor uset Ita dian eso tari hile ates land enia rael
i a n F wed g, Ch Nor Qu h Af Zea Hun Tu inga ndo Aus D
m ch In inn On C St ng
E l ov Is
ss S n t w S I a M d S
o ou Ne EC ss ite
Ru K S O M a n
ng U
Ho
1 2 http://dx.doi.org/10.1787/888933085482
Figure13.2 Peer discussions amongst 8th grade science teachers

Percentage of students who have a teacher who discusses with other teachers how to teach a particular topic once a week or more
% -27 -11 -10 -10 -10 -8 -6 -5 -5 -3 -1 +2 +3 +4 +4 +4 +4 +6 +7 +7 +8 +9 +9 +12

% point
+34
80 35
70 30
60
25
50
20
40
15
30
20 10
10 5
0 0
*** ** d* ** *** ay d ia ca a ey n re ta ia ec n ly a ts ile a* s** o* ***
ed den lan orea gary orw glan ven Afri Chin urk mea apo eso ones ueb Japa Ita dian uset Ch trali tate tari rael
i a n F Swe Zea K un N En Slo uth ng, T D
C i n g inn d
n Q In ach u s
d S O n Is
ss S I A te
Ru Ne
w H So g Ko OE M ss i
n Ma Un
Ho
1 2 http://dx.doi.org/10.1787/888933085501

Figure13.3 Peer discussions amongst 4th grade teachers

Percentage of students who have a teacher who discusses with other teachers how to teach a particular topic once a week or more
% -15 -14 -10 -9 -8 -7 -6 -6 -6 -4 -3 +0 +1 +1 +2 +2 +3 +3 +3 +4 +5 +6 +8 +14

% point
80 16
70 14
60 12
50 10
40 8
30 6
20 4
10 2
0 0
** d*** sh* rta ark bec en ria ina ay nia an lic nd lic aly nd tes ary ny rio an lia e***
ds i e d t w e b a b It ala Sta ng ma nta Jap stra or
l an n Fe lem Alb enm Que Swe Aus g, Ch Nor love D m epu Engl epu Ze d r p
e r a F D S
EC ak R hR w nite Hu Ge O Au inga
e th ussi um Kon O v e c e S
N R lgi n g l o C z N U
Be S
Ho
1 2 http://dx.doi.org/10.1787/888933085520
instead of 2003 and 2011 for Austria, Canada (Alberta), Czech Republic, Denmark, Germany, Slovak Republic and Sweden . OECD average includes all

TIMSS (2003, 2007 and 2011) surveys asked teachers How often do you have the following types of interactions with
other teachers? [] Discuss how to teach a particular topic [] Collaborate in planning and preparing instructional
materials [] Visit another classroom to learn more about teaching., with response options Never or almost never; 2
or 3 times per month; 13 times per week; Daily or almost daily. Same data restrictions as in International TIMSS and
PIRLS Reports by the International Association for the Evaluation of Educational Achievement (IEA) apply to these data.

Figure13.4 Teacher collaboration in preparing instructional materials in 8th grade mathematics

Percentage of students who have a teacher who collaborates in planning and preparing instructional materials with other teachers
once a week or more and change over time
% -33 -29 -18 -18 -18 -15 -10 -9 -9 -8 -7 -5 -5 -1 -1 -0 +0 +3 +4 +5 +5 +6 +10 +14 +31
% point
90 35
80
30
70
60 25
50 20
40 15
30
10
20
10 5
0 0
** ** ** a*** y*** e*** nd nd ly** ea* bec ina ile lia ota an rio key na ay tts an s** a*** l***
ia* d.* n* a a ta r Ch stra nes me nta Tur ndia orw use Jap tate eni srae
n es n Fe ede fric ngar apor Zeal Engl I Ko Que g, Ch N ach S
d o ia
s S A
w th Hu ng w
i o n Au Min ECD O I
s ed Slo
v I
In us u S Ne gK O as nit
R So n M U
Ho
1 2 http://dx.doi.org/10.1787/888933085539
Figure13.5 Teacher collaboration in preparing instructional materials in 8th grade science

% -35 -34 -22 -21 -18 -18 -13 -13 -8 -8 -8 -3 -2 -1 +2 +2 +2 +5 +5 +5 +6 +7 +11 +11 +24
% point
80 40
70 35
60 30
50 25
40 20
30 15
20 10
10 5
0 0
.*** *** *** *** *** *** d** a** y** e** ile ey an na ta na lia ec ia* rio tes an ay* tts l***
ed ary rica sia den and lan Kore Ital apor Ch Turk me ndia neso , Chi stra ueb ven nta Sta Jap orw use srae
a n F ung h Af done Swe eal Eng n g C D I i n ng Au Q Slo O ed N ach I
ss
i H ut In w
Z Si OE M Ko nit as
s
Ru So N e n g U M
Ho
1 2 http://dx.doi.org/10.1787/888933085558

Figure13.6 Teacher collaboration in preparing instructional materials in 4th grade

% -30 -21 -16 -13 -13 -13 -9 -9 -8 -4 -4 -3 -1 +1 +1 +1 +3 +5 +6 +6 +7 +9 +14 +16

% point
80 35
70 30
60
25
50
20
40
15
30
20 10
10 5
0 0
*** ry*** ta*** k*** ed** ly*** blic ario nd* bec tria den ean alia and ore ina pan ates way nds ish* y*** ia***
lic r F Ita pu Ont eala Que Aus Swe D m ustr Engl ngap g, Ch Ja d St Nor erla lem man ven
p ub nga lber nma ian Re Z F er Slo
Re Hu A De ss ch
C A Si on ite th
k R u e Ne
w OE K Un Ne gium G
va Cz ng l
Slo Ho Be
1 2 http://dx.doi.org/10.1787/888933085577

Classroom observations
General findings
Innovation in teacher collaboration practices included some change to the proportion of teachers
making regular visits to other classrooms to learn teaching practices in secondary education
(Figure13.7 and Figure13.8). Between 2003 and 2011, the OECD average absolute change was 4%
points concerning teachers visits to other classrooms in 8th grade science and 3% points in 8th grade
mathematics. Regarding science, increases in classroom visits occurred in Indonesia (10% points),
Ontario (10% points), Hong Kong (8% points), Slovenia (6% points) and Israel (5% points). On the other
hand, a decrease was remarkable only in Norway (11% points). In mathematics the increases were
noteworthy mostly in Slovenia (11% points) and Ontario (9% points), decreases instead were more
pronounced in countries such as Indonesia (13% points) and Norway (8% points). The effect sizes of
all these education system changes were small. Overall, teachers visits to other science classrooms
increased in six education systems with small effect sizes and decreased in one with medium
effect size. Teachers visits in mathematics classrooms increased in three education systems and
decreased in another three with at least small effect size.
In comparison with secondary education, primary education teachers increased regular visits
to their colleagues to learn more about teaching practices (Figure13.9). The OECD average absolute
change in the period between 2003 and 2011 was 8% points concerning classroom observations
of other teachers work in 4th grade, although it should be noted that OECD average change and
absolute change are heavily driven by changes in Slovenia. The largest increases were observed in
Slovenia (59% points), Denmark (11% points), England (8% points) and Singapore (5% points); on the
contrary Norway (11% points) showed the largest decrease in teachers visits to other colleagues
in 4th grade. With the exception of Slovenia showing a large effect size, all changes within these
education systems and the OECD average absolute presented small effect sizes. Altogether, with a
small effect size, increases in classroom observation practices were significant in four education
systems, whereas decreases were significant in one.
Classroom observation practices in the form of teachers visits to other colleagues classrooms
increased simultaneously across disciplines and levels in Slovenia and across disciplines in Ontario.
Between 2003 and 2011 teachers visits to other classrooms increased in 8th grade science and
mathematics in Slovenia and Ontario with at least small effect size. In Slovenia there was also a
significant increase at 4th grade, with a large effect size.
On the contrary, Norway was the only country which experienced reductions across disciplines
and levels with regard to classroom observation practices. Norwegian teachers reduced the
frequency of their visits to other colleagues both in secondary science and mathematics and in
primary education.
The direction of change concerning classroom observation practices has not been consistent in
Indonesia across disciplines in secondary education. Indonesian teachers increased the frequency
of their visits to other colleagues in 8th grade science between 2003 and 2011 and decreased in 8th
grade mathematics.

Figure13.7 Classroom observations in 8th grade mathematics

Percentage of students who have a teacher who visits another classroom to learn more about teaching once a week or more and
change over time
% -13 -8 -3 -3 -1 -1 -1 -0 -0 +0 +0 +1 +1 +2 +2 +2 +2 +2 +3 +4 +4 +4 +4 +9 +11
% point
40 14
12
30
10
20 8
15 6
10 4
5 2
0 0
** *** tts ed. en ota key ary ore aly rea bec na *** an lia nd tes el** ina d** ica ile io** a***
sia ay e F d s r g p It Ko ue dia ean Jap stra gla Sta sra Ch lan Afr Ch tar eni
ne orw chus sian Swe inne Tu Hun nga Q In m En ted I g, Zea uth
d o N ssa us Si Au on On Slov
In M CD i o
Ma
R OE Un n g K New S
Ho
1 2 http://dx.doi.org/10.1787/888933085596
Figure13.8 Classroom observations in 8th grade science

Percentage of students who have a teacher who visits another classroom to learn more about teaching once a week or more and
change over time
% -11 -2 -2 -1 -1 -1 -0 -0 -0 +0 +0 +0 +1 +1 +1 +1 +2 +3 +4 +5 +5 +6 +8 +10 +10

% point
24 12
20 10
16 8
12 6
8 4
4 2
0 0
*** .* es ey a le n d ec ta ry ly ca ea ts re *** d ia l** n *** *** *** **
ay Fed tat urk dian Chi ede alan ueb eso nga Ita Afri Kor uset apo ean glan stral srae Japa nia ina ario esia
o rw ian d S T In S w Ze Q inn Hu t h ch ing m En Au I ove Ch nt o n
N uss nit e w M u a S CD Sl ng , O Ind
Ne So ss
R U Ma OE Ko
o ng
H
1 2 http://dx.doi.org/10.1787/888933085615

Figure13.9 Classroom observations in 4th grade

Percentage of students who have a teacher who visits another classroom to learn more about teaching once a week or more
% -11 -5 -2 -2 -1 -1 -0 -0 +1 +1 +1 +1 +2 +3 +4 +4 +4 +4 +5 +6 +6 +8 +11 +59

% point
60 60
55 55
50 50
45
20 45
20
15 15
10 10
5 5
0 0
* * a . y y c s ish lic nd en an * a * o ** ** ** a ** ** **
y** ly* tri Fed ar an bli nd b a d Jap hina trali ebec tari ates ore* an* bert nd* ark* nia*
r wa Ita Aus ian ung erm epu erla Flem epu Zeal Swe , C Aus Qu On d St gap me Al ngla nm love
N o s s H G R t h R g E De
Ru va
k
Ne lgium zech New Ko
n ite Sin ECD S
Slo Be C ng Un O
Ho
1 2 http://dx.doi.org/10.1787/888933085634

TIMSS (2003, 2007 and 2011) surveys asked teachers How often do you have the following types of
interactions with other teachers? [] Visit another classroom to learn more about teaching., with response
options Never or almost never, 2 or 3 times per month, 13 times per week and Daily or almost daily.
Same data restrictions as in International TIMSS and PIRLS Reports by the International Association for the

Summary
Innovation or significant change regarding teacher collaboration has meant more and
fewer peer discussions, joint material preparation and classroom observations among teachers in
secondary and primary education. While there has been clear change in the OECD area, direction of
innovation has not been consistent across education systems and therefore little can be said on the
overall trajectory of change between 2003 and 2011 for OECD countries.
Figure13.10 Change in the frequency of teachers peer discussions
4th grade 8th grade science 8th grade maths

%
80
60
40
20
0
-20
-40
-60
-80
ey esia sota pan etts ana taly rea rica alia ore ary hile ina and bec way and ates ario nia den rael tion ean ean ge)
rk Ja hus ndi I Ko Af str ap ng C , Ch eal Que or ngl St Ont love we Is era n
Tu don inne I h Au ing Hu g Z N E ed S d
m
D D m ha
In M ac ut S on ew it
S
Fe OEC OECbs. c
a ss So K N U n n
M ng ia (a
Ho ss
Ru
1 2 http://dx.doi.org/10.1787/888933085653
Figure13.11 Change in the frequency of teacher collaboration in preparing instructional materials

%
80
60
40
20
0
-20
-40
-60
-80
c e rio ec tts an rea tes ay nd re lic aly nia nd en sia ael ry on an an e)
ota lia na ey li ina il o a i
n es stra ndia Turk epub , Ch Ch nta ueb use Jap Ko Sta orw ngla gap epub It love eala wed one Isr ung erat me me ang
i n A u I R g O Q ch e d N E in R S Z S Ind H d C D CD ch
n a t S k ew e .
n F OE OE(abs
M h s ni
z ec g Ko as U l ova N i a
C on M S s s
H Ru
1 2 http://dx.doi.org/10.1787/888933085672
Notes: For details please see Figure13.1, 13.2, 13.3, 13.4, 13.5 and 13.6

Figure13.12 Change in the frequency of classroom observations

%
80
60
40
20
-20
-40
s a c c s e e s
nd sot bli ary rea key ana den be ett and Italy Chil ation por tate apan ralia land hina tario rway enia ean ean ge)
e rla nne epu ung Ko Tur Indi Swe Que hus Zeal e r
n g a
d S J ust ng , C On
E ng
v
No Slo CD m m n
D ha
th Mi ch R H ac d Si
Ne ss New n Fe n ite A
Ko O E OECbs. c
e a U (a
Cz M ss
ia ng
Ru Ho
1 2 http://dx.doi.org/10.1787/888933085691
Notes: For details please see Figure13.7, 13.8 and 13.9
Note on Israel

Table 13.1 Effect sizes for changes in teacher collaboration practices
Change in instructional collaboration Change in classroom observations
Teachers peer
Visits of other
discussions
classrooms
Joint preparation Joint prepa- Visits of other teachers
teachers
Teachers peer discussions of materials ration of classrooms
materials
Maths Science Maths Science All All Maths Science All
03-11 03-11 03-11 03-11 03-11 03-11 03-11 03-11 03-11
Australia 0,03 0,18 -0,02 0,05 0,17 0,01 0,13 0,14 0,11
Belgium Flemish m m m m -0,21 0,18 m m 0,06
Ontario 0,22 0,24 0,00 0,11 0,10 -0,18 0,33 0,34 0,14
Quebec -0,24 0,09 -0,15 0,10 -0,13 -0,08 0,04 0,00 0,18
Chile 0,27 0,17 -0,10 -0,16 m m 0,16 -0,05 m
Hungary -0,16 -0,21 -0,39 -0,76 0,07 -0,42 -0,01 0,01 -0,05
Israel 0,51 0,69 0,64 0,48 m m 0,20 0,24 m
Italy 0,11 0,12 -0,22 -0,22 0,04 -0,25 0,02 0,02 -0,19
Japan 0,03 0,09 0,14 0,17 0,12 0,11 0,06 0,21 0,07
Korea 0,11 -0,22 -0,17 -0,26 m m 0,03 0,08 m
Netherlands m m m m -0,31 0,17 m m -0,01
New Zealand -0,19 -0,21 -0,20 -0,36 0,06 -0,16 0,19 0,00 0,05
Norway -0,25 -0,15 0,10 0,24 -0,09 0,13 -0,37 -0,68 -0,47
Slovenia 0,42 -0,10 0,29 0,12 -0,06 0,32 0,66 0,28 1,65
Sweden -0,37 -0,22 -0,37 -0,37 m m -0,07 -0,02 m
England 0,35 -0,11 -0,19 -0,28 0,02 0,03 0,10 0,14 0,35
United States 0,28 0,18 0,19 0,12 0,06 0,12 0,12 -0,07 0,17
Indiana 0,13 0,14 0,08 -0,03 m m 0,08 -0,08 m
OECD (average) 0,08 0,04 -0,01 -0,04 0,00 -0,02 0,09 0,05 0,17
OECD (average absolute) 0,24 0,20 0,22 0,25 0,10 0,16 0,16 0,17 0,29
Hong Kong, China -0,30 -0,06 -0,14 0,05 -0,12 0,07 0,22 0,43 0,19
Indonesia -0,05 0,08 -0,69 -0,43 m m -0,28 0,27 m
Russian Federation -0,43 -0,57 -0,65 -0,80 -0,29 -0,26 -0,09 -0,08 -0,05
Singapore -0,05 0,06 -0,32 -0,17 0,28 0,03 0,00 0,08 0,37
South Africa -0,21 -0,09 -0,36 -0,45 m m 0,14 0,03 m
1 2 http://dx.doi.org/10.1787/888933085710
Source: Authors calculations based on TIMSS (2003, 2007 and 2011

CHAPTER 14
Innovation
in feedback mechanisms in schools
Innovation in schools can take the form of a change in the use of

benchmarking, monitoring and feedback activities. Student assessment and
achievement data may be used for comparing a schools performance against
national benchmarks, against other schools or for monitoring its progress over
time to better understand its strengths and weaknesses. Feedback received
from assessments can be used for further instructional and curriculum
improvement. The aim of innovation with regard to increasing the use of
benchmarking, monitoring and feedback activities could be, for example, to
improve teaching quality through increased feedback. A decrease may result
from the desire to decrease between-school competition or limit the burden of
data collection by reducing these activities.

14. INNOVATION IN FEEDBACK MECHANISMS IN SCHOOLS
Progress benchmarking and monitoring
General Findings
Much of the innovation in progress monitoring has resulted in less use of student assessments for
between-school comparisons and more for comparison with district or national benchmarks across
schools teaching 15-year-olds (Figure14.1 and Figure14.2). The average absolute change within the
OECD area concerning use of assessments for school comparisons was 23% points between 2003 and
2009 whereas it amounted to 15% points for comparison with district or national performance from
2000 to 2009. Hong Kong (68% points), Spain (45% points), Italy (45% points), Iceland (42% points), Poland
(40% points), Portugal (39% points) and Hungary (34% points) showed prominent reductions in the use
of student assessments for comparison between schools. Increase was noteworthy only in Denmark
(19% points). As for comparison with district or national benchmarks, Luxembourg (54% points), Canada
(29% points), Portugal (28% points), Denmark (27% points), Switzerland (24% points), United States
(23% points) and Germany (21% points) experienced the largest increases. Illustrating the magnitude
of change, all increases in these individual education systems were characterised by at least medium
effect sizes, whereas the decreases had large effect sizes. OECD average and average absolute effect
sizes were small to medium for between-school comparisons and the average absolute effect size was
small for comparison with district or national benchmarks. Overall, with at least small effect sizes,
assessment use for between-school comparisons decreased in 24 education systems and increased in
one. Comparisons with district and national benchmarks increased in 16 systems and decreased in two.
Innovation has also occurred in the form of increased monitoring over time and tracking of the
achievement of 15-year-old students by schools and administrations (Figure14.3 and Figure14.4).
On average absolute level, change between 2000 and 2009 within the OECD area was 12% points
regarding the use of student assessments for monitoring school progress over time. It amounted to
7% points between 2006 and 2009 for tracking of achievement data by an administrative authority
and to 3% points for parental perception on schools monitoring student progress. In particular,
Luxembourg (40% points), Belgium (34% points), Greece (34% points), the United States (24%
points) and the Russian Federation (14% points) increased assessment use for monitoring school
progress. Decreases in tracking achievement data were prominent in the Netherlands (25% points)
and Germany (25% points). Changes in individual systems corresponded to medium to large effect
sizes, while the OECD average effect size was small for monitoring school progress as well as for
parental perception. There was no notable change for administrative tracking of achievement data
on average for the OECD. Overall, with at least small effect sizes, 14 education systems increased the
use of student assessment to monitor schools progress overtime, whereas the opposite was true for
three systems. Administrative achievement data tracking increased in eight education systems and
decreased in three others with at least small effect sizes.
Denmark stands out as the only country where innovation took the form of increased use of
assessment data. With at least small effects, assessment use for comparisons between schools and with
national benchmarks as well as for monitoring and tracking increased from 2000, 2003 or 2006 to 2009.
In the Netherlands, innovation took the form of decreased monitoring. With at least small effect
sizes, assessment use for between-school comparisons and/or district or national performance
decreased from 2000 or 2003 to 2009. Also administrative tracking of data decreased from 2006 to 2009.
The cases of Germany and Mexico illustrate that innovation can mean increases in one type
of monitoring and decreases in other. In these two countries assessment use for comparison with
district or national performance and progress monitoring increased with at least small effects
between 2000 and 2009, while use for between-school comparisons decreased from 2003 to 2009.

Figure14.1 Use of student assessments for district or national benchmarking

Percentage of 15-year old students in schools where assessments are used for comparing school
to district or national performance and change over time
% -24 -23 -8 -2 +1 +2 +2 +3 +4 +5 +5 +8 +8 +9 +10 +11 +11 +12 +12 +13 +14 +17 +18 +21 +21 +23 +24 +27 +28 +29 +54
% point
100 55
90 50
80 45
70 40
60 35
50 30
40 25
20
30
15
20 10
10 5
0 0
* * * * * l * * * * * * * * * * * * * * *
s** d** nd nd om tria en sia ain m ion ary nd lia ce* ae n** y* le* y** n** o** d** y** c** s** d** k** l** a** g**
l a nd elan Finla ealaingd Aus weddone Spelgiu erat ung Irela stra ree Isr ea orwa Chi ItalJapa exicolan man ubli tate rlan mar tuga nad our
er Ic Z S B Fed H Au G m p S
M P Ger Re ed itze en Po Ca em r b
th w dK In CD N
D
Ne Nenite n OE h it
ec n Sw Lu
x
U s sia Cz U
Ru
1 2 http://dx.doi.org/10.1787/888933085729
Figure14.2 Use of student assessments for between-school comparisons

Percentage of 15-year old students in schools where assessments are used to compare the school
with other schools and change over time
% -68 -45 -45 -42 -40 -39 -38 -37 -34 -33 -31 -28 -27 -26 -23 -22 -22 -21 -21 -21 -16 -14 -13 -11 -9 -3 -2 -2 -2 +4 +4 +9 +19
% point
100 70
90 60
80
70 50
60 40
50
40 30
30 20
20
10
10
0 0
g, * * * * * * * * * * * * * * * * * * * * * * * * * y s d a a * *
K onina*a* in**aly** nd**nd**gal** nd** ria**ary** lia** m** an** lic** nd** lic** ny** ds**ico** an** da** ay** nd**om**eeceden* rke taterlan ationnesi ore rg**ark**
g Ch Sp It ela ola rtu nla ust ng stra lgiu Jap pub ela pub ma lan ex me ana orw ala gd Gr we Tu d S tze der do K ou m
n i
Ic P Po F A Hu Au Be r r
e I Re Ger he M D C N Ze Kin b
Ho S ite wi Fe In m en
kR h t
OE
C w d Un Ssian xe D
va ec Ne Nenite Lu
Slo Cz U R u s
1 2 http://dx.doi.org/10.1787/888933085748

Figure14.3 Use of student assessments for monitoring progress over time

Percentage of 15-year old students in schools where assessments are used for monitoring progress from year to year
% -17 -9 -8 -6 -2 -2 -1 +0 +0 +2 +3 +4 +4 +6 +7 +7 +8 +9 +9 +9 +10 +13 +14 +16 +19 +19 +22 +24 +34 +34 +40
% point
100 45
90 40
80
35
70
30
60
50 25
40 20
30 15
20 10
10 5
0 0
* s * * y d * a l a d d y d * * * * * * * * * * * * * * * *
d** nd ile* ary Ital lan domand* stri sraenesi lan lan an rlan raliaden* ark* ico** an** pangal**ion** lic** ay** da** ain**tes** ce** m**urg**
lan erla Ch ung a g l
Ze Kin Ice A
u I do Ire Po erm ze st e m ex
m
e Ja tu at ub rw na Sp ta ree giu bo
Fin eth H w d I n G i t Au S w e n M
CD
o r e
P ed R
r ep N o C a d S G e l
B xem
N Nenite Sw D ite
OE n F ec
h
Un Lu
U
s sia Cz
Ru
1 2 http://dx.doi.org/10.1787/888933085767
Figure14.4 Administrative tracking of school achievement data

Percentage of 15-year old students in schools where achievement data is tracked over time by an administrative authority
% -25 -25 -11 -10 -10 -7 -6 -5 -5 -4 -3 -3 -1 -1 -1 -1 -1 -0 -0 -0 +0 +0 +0 +1 +1 +2 +5 +5 +6 +9 +10 +11 +12 +13 +20 +22 +24
% point
100 30
90
80 25
70
20
60
50 15
40
30 10
20
5
10
0 0
Netherlands***
Germany***
Belgium**
Finland*
Austria*
Luxembourg***
Iceland***
Australia
Japan
Mexico**
Switzerland
Turkey
Estonia
Hong Kong, China
Russian Federation
Slovenia
Canada
United States
Ireland
Spain
Chile
OECD mean
United Kingdom
Portugal
Sweden
Czech Republic
Greece
Italy*
New Zealand**
Slovak Republic**
Hungary*
Indonesia**
Israel**
Poland***
Norway***
Denmark***
Korea***
1 2 http://dx.doi.org/10.1787/888933085786

PISA (2000, 2003 and/or 2009) surveys asked principals In your school, are assessments of students in
<national modal grade for 15-year-olds> used for the following purposes []To compare the school to <district
or national> performance [] To compare the school with other schools [] To monitor the schools progress
from year to year with answer options Yes/No. PISA (2006 and 2009) asked In your school, are achievement
data used in any of the following <accountability procedures> [] tracked over time by an administrative
authority? with answer options Yes/No.

Curriculum and instruction development
General findings
Innovation in the form of increased use of assessment data for instructional and curriculum
improvement has taken place in many education systems in schools with 15-year-old students
(Figure14.5). Between 2003 and 2009, the average absolute change within the OECD area for using
student assessments to identify improvement aspects for instruction and curriculum amounted to
8% points. The most noteworthy increases took place in Denmark (38% points) and Indonesia (22%
points). Hungary (27%) experienced the largest decrease in using assessments to identify points for
instructional and curriculum improvement. Change in these individual countries was associated
with large to medium effect sizes. In general, the percentage of 15-year-olds in schools where
assessments were used to identify points for instructional and curriculum improvement increased
with at least small effect sizes in six education systems, while decreasing in only one.
Overall, little innovation has taken place regarding time investment for pedagogical and
curriculum development at the primary school level (Figure14.6). The average absolute change
within OECD area regarding the time principals in the 4th grade schools devoted to curriculum and
pedagogical development was 3% points between 2001 and 2006. Hungary (10% points) stands out as
the only country where principals notably reduced the time devoted to curriculum and instruction
development and it is the only country where the change presents a small effect size.
Innovation in Denmark has taken the form of an increase in the use of student assessments for
instructional and curriculum improvement. Within the period 2003 to 2009, the share of 15-year-olds
attending Danish schools where student assessments were used to identify points for instructional
and curriculum improvement increased with a large effect size.
In contrast, innovation took the form of a clear decrease in (data based) curriculum and
instruction development in Hungary. In Hungary, the use of student assessments for instructional
and curriculum improvement in the schools of 15-year-olds decreased with medium effect size from
2003 to 2009. In addition, the time principals dedicated to curriculum and pedagogy development
fell with small effect size between 2001 and 2006.

Figure14.5 Use of student assessments for instructional or curriculum improvement

Percentage of 15-year old students in schools where assessments are used to identify instructional or curriculum improvement
% -27 -9 -6 -5 -5 -5 -4 -3 -3 -2 +0 +1 +2 +3 +3 +3 +3 +4 +4 +4 +5 +5 +6 +6 +7 +11 +22 +22 +22 +38

% point
100 40
90 35
80
70 30
60 25
50 20
40 15
30
10
20
10 5
0 0
*** d* ds gal m lic *** *** ria nd ay ion en nd ain *** ico om da* nd an aly lia* ce s*** y** *** *** *** ***
g aryinlanrlanortu elgiuepub land ourgAust erla orwerat wed eala Sp eanMex ngd ana Pola Jap It stra Greetate man rkey esia land ark
n F the P B R Ice mb itz N ed S Z m Ki C r Tu don Ire enm
Hu h ew D Au d S Ge
Ne z ec u xe Sw
i a nF N O EC i t ed n ite In D
C L ss Un U
Ru
1 2 http://dx.doi.org/10.1787/888933085805
Sources: Authors calculations based on PISA (2003 and 2009)
Figure14.6 Principals time devoted to curriculum and pedagogy development at the 4th grade
Percentage of principals time devoted to developing curriculum and pedagogy in their school and change over time
% -10 -4 -4 -3 -2 -2 -2 -2 -2 -1 -1 -1 -1 -0 -0 -0 +0 +0 +1 +4 +5
% point
30 12
25 10
20 8
15 6
10 4
5 2
0 0
** c*** y*** e*** d el d* d d. io *** es ce ds d a ec ay ny** ia*** n***
ry* i l r lan Isra alan otlan n Fe ntar ean Stat ran rlan elan Chin ueb orw
n ga publ Ita apo Eng e c i a O m d F e I c g , Q N
a n
rm love Swe
de
Hu k Re g Z S s e et h on e
Sin ew R us E CD Unit N K G S
v a N O n g
Slo Ho
1 2 http://dx.doi.org/10.1787/888933085824
Sources: Authors calculations based on PIRLS (2001 and 2006)
PISA (2003 and 2009) surveys asked school principals In your school, are assessments of students in
<national modal grade for 15-year-olds> used for the following purposes: [] To Identify aspects of instruction
or the curriculum that could be improved with answer options Yes/No. PIRLS (2001 and 2006) surveys asked
principals and department heads As principal of this school, approximately what percentage of your time is
devoted to the following activities? [] Developing curriculum and pedagogy for your school, with answers
expressed in percentages. The same data restrictions as in International PIRLS Reports by the International

Summary
Innovation regarding benchmarking, monitoring and feedback practices has meant more or less
use of student assessment in secondary schools for external comparison and internal evaluation
purposes. Innovation took the form of an increase in performance monitoring for comparison with
national benchmarks between 2000 and 2009 and a decrease for comparison among schools among
OECD countries between 2003 and 2009. On the contrary, in general, the use of assessment for
curriculum and instruction development has exhibited little change between 2003 and 2009.
Figure14.7 Change in the use of student assessments for external comparison
Compare with other schools (2009-2003) Compare with district or national performance (2009-2000)
%
80
60
40
20
0
-20
-40
-60
-80
Indonesia
Russian
Federation
Israel
Sweden
Chile
United
Kingdom
New Zealand
Greece
United States
Switzerland
Norway
Ireland
Belgium
Mexico
Austria
Hungary
Japan
Australia
Germany
Czech
Republic
Finland
Denmark
Netherlands
Spain
Canada
Poland
Italy
Iceland
Portugal
OECD mean
OECD mean
(abs. change)
Luxembourg
1 2 http://dx.doi.org/10.1787/888933085843
Notes : For details please see Figures14.1 and 14.2
Figure14.8 Change in the use of student assessments to monitor progress and to improve
curriculum in secondary education
Compare with other schools (2009-2003) Compare with district or national performance (2009-2000)
%
50
40
30
20
10
0
-10
-20
-30
-40
-50
Israel
Austria
New Zealand
United Kingdom
Iceland
Italy
Poland
Switzerland
Sweden
Mexico
Australia
Japan
Russian Federation
Netherlands
Germany
Portugal
Norway
Czech Republic
Turkey
Canada
Spain
Indonesia
Finland
Ireland
United States
Hungary
Belgium
Luxembourg
Denmark
OECD mean
OECD mean
Greece
(abs. change)
1 2 http://dx.doi.org/10.1787/888933085862
Notes : For details please see Figures14.3 and 14.6
Note on Israel

Table 14.1 Effect sizes for changes in benchmarking, monitoring and feedback in school education
Change in progress monitoring Change in curriculum and

instruction development
Use of assessment data Use of achieve- Use of assessment data

ment data
Comparison with Comparison with Monitoring school Tracked over time Instructional Principal time
district/national other schools progress by admin authority or curriculum devoted to curric-
performance improvement ulum development
00-09 03-09 00-09 06-09 03-09 01-06
Australia 0,18 -0,69 0,18 -0,14 0,15 m

Austria 0,06 -0,78 0,01 -0,19 -0,05 m
Belgium 0,17 -0,80 0,75 -0,21 -0,09 m
Canada 0,59 -0,45 0,46 -0,02 0,11 m
Chile 0,24 m -0,30 0,00 m m
Czech Republic 0,43 -0,54 0,41 0,04 -0,13 m
Denmark 0,74 0,52 0,20 0,46 0,83 m
Estonia m m m -0,03 m m
Finland -0,15 -0,78 -0,34 -0,19 -0,19 m
France m m m m m -0,12
Germany 0,51 -0,48 0,12 -0,51 0,23 0,18
Greece 0,29 -0,24 0,70 0,09 0,13 m
Hungary 0,16 -0,87 -0,19 0,20 -0,67 -0,95
Iceland -0,48 -0,94 0,01 -0,13 -0,18 -0,03
Ireland 0,16 -0,54 0,07 -0,01 0,45 m
Israel 0,21 m 0,04 0,26 m -0,20
Italy 0,29 -0,94 -0,04 0,12 0,14 -0,40
Japan 0,38 -0,60 0,20 -0,14 0,13 m
Korea m 0,08 m 0,50 m m
Luxembourg 1,64 0,19 1,38 -0,16 -0,06 m
Mexico 0,36 -0,55 0,25 -0,13 0,10 m
Netherlands -0,49 -0,44 -0,22 -0,58 -0,13 -0,03
New Zealand -0,07 -0,44 -0,11 0,20 0,15 -0,21
Norway 0,24 -0,33 0,43 0,41 0,00 0,02
Poland 0,36 -1,06 0,17 0,36 0,15 m
Portugal 0,61 -0,82 0,35 0,01 -0,12 m
Slovak Republic m -0,75 m 0,22 m -0,62
Slovenia m m m -0,01 m 0,42
Spain 0,11 -0,94 0,49 0,00 0,09 m
Sweden 0,05 -0,23 0,24 0,04 0,04 0,62
Switzerland 0,55 -0,05 0,14 -0,07 -0,05 m
Turkey m -0,08 m -0,07 0,44 m
United Kingdom 0,04 -0,43 -0,03 0,02 0,14 m
England m m m m m -0,20
Scotland m m m m m -0,17
United States 0,68 -0,08 0,75 -0,02 0,33 -0,09

Table 14.1 Effect sizes for changes in benchmarking, monitoring and feedback in school education
(continued)
Change in progress monitoring Change in curriculum and

instruction development
Use of assessment data Use of achieve- Use of assessment data

ment data
Comparison with Comparison with Monitoring school Tracked over time Instructional Principal time
district/national other schools progress by admin authority or curriculum devoted to curric-
performance improvement ulum development
00-09 03-09 00-09 06-09 03-09 01-06
OECD (average) 0,28 -0,48 0,22 0,01 0,07 -0,11

OECD (average absolute) 0,37 0,54 0,31 0,17 0,20 0,26
Hong Kong, China m -1,51 m -0,02 m 0,00
Indonesia 0,07 0,10 0,15 0,23 0,75 m
Russian Federation 0,13 -0,09 0,56 -0,17 0,22 -0,15
Singapore m m m m m -0,28
1 2 http://dx.doi.org/10.1787/888933085881
Source: Authors calculations based on PIRLS (2001 and 2006) and PISA (2000, 2003, 2006 and 2009)

CHAPTER 15
Innovation
in evaluation and hiring in schools
Innovation in schools can also include changes in practices such as

evaluation, hiring and retention. Teachers may be evaluated externally by
inspectors or through internal review by their peers. Schools may also innovate
by changing the extent to which they use achievement and assessment data
to evaluate teacher and principal performances. Innovation in schools could
also concern the use of incentives for recruitment and retention purposes. The
aim of innovation with regard to teacher evaluation could be, for example, to
improve the quality of teaching and teachers effectiveness, while an increased
use of incentives could be a response to a greater degree of competition among
schools, to ensure the presence of talented teachers on the staff.

15. INNOVATION IN EVALUATION AND HIRING IN SCHOOLS
Use of data for teacher and principal evaluation
General findings
Innovation in evaluation practices also included either increased or decreased use of assessment
and achievement data in secondary education (Figure15.1 and Figure15.2). Between 2003 and 2009
the OECD average absolute change in the use of assessment data to make judgements about teachers
effectiveness was 7% points; the average absolute change in the use of achievement data between
2006 and 2009 amounted to 6% points. Use of assessment data to judge teachers effectiveness
increased especially in Turkey (38% points). Achievement data were used more widely in countries
such as Greece (17% points), Denmark (16% points), Norway and Slovenia (11% points), whereas
their use declined in Portugal (21% points), Estonia (13% points), Hungary (9% points) and Czech
Republic (9% points). The reported changes in the use of assessment data presented medium effect
sizes, while effect size was small for changes in achievement data use (reflecting the shorter time
period over which changes were measured). Overall, the use of assessment data to judge teachers
effectiveness increased in eight education systems and decreased in two with at least a small effect
size, while achievement data were used more widely to evaluate teachers performance in five
education systems and less in four, with small effect sizes.
Innovation also entailed wider use of achievement data to evaluate principals performance
in the majority of education systems (Figure15.3). Between 2006 and 2009, OECD average absolute
change amounted to 7% regarding this particular practice. The most prominent increase was
found in Slovenia (49% points), while other countries such as Norway (17% points), Denmark (15%
points), Greece (12% points), New Zealand (11% points) and Iceland (8% points) also showed notable
increases. Decreases were noteworthy in the case of Estonia (13% points), Hungary (13% points) and
Hong Kong (11% points). These changes presented small effect sizes, with the exception of Slovenia,
where the effect size was large. Altogether, principals performance evaluatios were more likely to
incorporate achievement data in 2009 than in 2006 in six education systems and less in three, with
at least small effect size.
Slovenia, Norway, Denmark and Greece are countries where innovation was exemplified in
additional use of achievement data for evaluation purposes. In these countries, achievement data
were used more widely to evaluate both teachers and principals performance between 2006 and
2009, with at least small effect sizes.
In contrast, use of achievement and assessment data for evaluation purposes particularly
decreased in Hungary. Hungarian schools made less use of assessment and achievement data to
evaluate the performance and effectiveness of teachers and principals in the time span between
2003 and 2009, with small effect sizes.
The direction of innovation regarding use of data for evaluation has not been consistent
among instruments and targets in Hong Kong. Schools in Hong Kong increased the use of student
assessment data to judge teachers effectiveness between 2003 and 2009 and reduced the use of
achievement data to evaluate principals performance, with small effect sizes.

Figure15.1 Use of assessment data to make judgments about teachers effectiveness

Percentage of 15-year old students in schools where assessment data are used to make judgments
about teachers effectiveness and change over time
% -16 -10 -8 -3 -3 -1 -1 -1 -0 +0 +1 +1 +3 +4 +4 +4 +4 +4 +5 +5 +6 +7 +7 +8 +9 +10 +10 +11 +12 +13 +13 +20 +38
% point
100 40
90 35
80
30
70
60 25
50 20
40 15
30
10
20
10 5
0 0
** a* d ly n ic ** n al n ** o s a d k* ** y d ic e d * s ** ** ** ** ** ** ** **
ry* tri an Ita Japa ubland*ratiortug ede urg* gdomexic tate nad rlanmar ean* rwaolan ubl reecalan painlandany*ralia um*orea ong sia*and*key*
n ga Aus Finl e p el e o Sw bo in M d S Ca ze n m No P ep G Ze S e r t gi K g K one rel ur
Hu h R Ic Fe
d P m K e it e kR th ermAus Bel n d I T
ec a n u xe ited U nit Sw D ECD va Ne
w Ne G Ho In
Cz s s i L U n O S l o
Ru
1 2 http://dx.doi.org/10.1787/888933085900
Figure15.2 Use of achievement data for evaluating teachers performance

Percentage of 15-year old students in schools where achievement data are used for evaluating teachers performance
% -21 -13 -9 -9 -8 -8 -7 -6 -5 -4 -4 -4 -3 -3 -3 -2 -2 -2 -2 -1 -1 -1 -1 -1 -0 -0 +0 +0 +3 +3 +3 +5 +10 +11 +11 +16 +17

% point
100 25
90
80 20
70
60 15
50
40 10
30
20 5
10
0 0
Portugal***
Estonia***
Hungary**
Czech Republic**
Hong Kong
Ireland
Spain*
Israel*
Sweden
Mexico
Canada*
Germany
Finland
Turkey
Iceland***
Japan
Poland
Netherlands
Slovak Republic
Russian Federation
Austria
OECD mean
United Kingdom
Chile
Australia
United States
Indonesia
Italy
New Zealand
Luxembourg***
Switzerland
Belgium
Korea*
Slovenia***
Norway**
Denmark***
Greece***
1 2 http://dx.doi.org/10.1787/888933085919

Figure15.3 Use of achievement data for the evaluation of principals performance

Percentage of 15-year old students in schools where achievement data are used for evaluating the principals performance
% -13 -13 -11 -6 -5 -5 -4 -4 -3 -3 -2 -2 -1 -1 -1 -0 +1 +2 +2 +2 +3 +3 +3 +4 +4 +4 +5 +6 +6 +7 +8 +11 +12 +15 +17 +49

% point
100 50
90
80 40
70
60 30
50
40 20
30
20 10
10
0 0
Estonia***
Hungary**
Hong Kong
Czech Republic
Canada*
Ireland**
Turkey
Australia
Italy
Russian Federation
Indonesia
Portugal
Japan
Chile
Germany
Austria
Slovak Republic
Switzerland
Finland
United Kingdom
Spain
OECD mean***
Netherlands
Korea
Poland
Belgium*
Sweden
United States
Mexico*
Israel
Iceland***
New Zealand**
Greece***
Denmark***
Norway***
Slovenia***
1 2 http://dx.doi.org/10.1787/888933085938

PISA (2003 and 2009) surveys asked school principals In your school, are assessments of students in
<national modal grade for 15-year-olds> used for the following purposes [] To make judgments about
teachers effectiveness with answer options Yes/No. PISA (2006 and 2009) asked In your school, are
achievement data used in any of the following <accountability procedures>? [] Achievement data are used in
evaluation of teachers performance [] Achievement data are used in evaluation of principals performance
with answer options Yes/No.

External evaluation
General findings
In secondary education, innovation in the evaluation of teachers translated into more observations
of teachers practices by inspectors or other persons external to the school, across the vast majority
of OECD countries (Figure15.4 and Figure15.5). Between 2003 and 2011 the OECD average absolute
change regarding performance evaluation through inspectors external observations was 11% points
for 8th grade mathematics and 10% points for 8th grade science. In 8th grade mathematics the increase
was most prominent in Israel (33% points), South Africa (31% points), Chile (23% points) and Ontario
(15% points). Similarly in 8th grade science, education systems with the largest increases were South
Africa (34% points), Israel (26% points) and Ontario (12% points). These changes exhibit medium
effect sizes, whereas the OECD average and average absolute changes showed small effect sizes
both for 8th grade mathematics and science. Altogether, teacher observation by inspectors or other
persons external to the school increased in 13 education systems and decreased in one, with at least
a small effect size in 8th grade mathematics classes. In the case of 8th grade science classes, external
observation increased in 14 education systems and decreased in one, with at least small effect sizes.
Similarly, innovation in external evaluation in primary education is illustrated through more

frequent observations of teachers practices by inspectors or other persons external to the school
(Figure15.6). The OECD average absolute change in teacher evaluation via external observation
between 2003 and 2011 amounted to 13% points in 4th grade classes. Education systems where
external observation of teachers practices increased the most were New Zealand (26% points), Hong
Kong (26% points), Norway (26% points), Singapore (24% points) and Ontario (16% points). These
changes were characterised by medium effect sizes, while the OECD average and average absolute
presented small effect sizes. Overall, 4th grade teacher evaluation through external observations by
inspectors or other persons external to the school increased in 12 education systems with at least a
small effect size, whereas the opposite was true only in the case of one education system.
Teachers evaluation via inspectors or external observation increased simultaneously across
disciplines and levels in Hong Kong and Ontario. Increases in external evaluation were substantial
in Ontario, showing medium effect size across levels, whereas in the case of Hong Kong, this was
true only for teachers in 4th grade in the period from 2003 to 2011.
By contrast, Slovenia was the only country showing a simultaneous decrease across disciplines
and levels with regard to evaluation through inspectors or external observations, with small effect
sizes between 2003 and 2011, both for 8th grade and 4th grade teachers.

Figure15.4 Evaluation of teachers practices by inspector observation in 8th grade mathematics

Percentage of students in schools in which observations by inspectors or other persons external to the school are used
to evaluate the practice of teachers and change over time
% -11 -7 -4 -1 +1 +1 +1 +5 +6 +6 +6 +7 +7 +9 +12 +12 +15 +17 +18 +18 +20 +20 +23 +31 +33
% point
100 35
90
30
80
70 25
60 20
50
40 15
30 10
20
10 5
0 0
** y a ts n ly ia ia y* n ** d. y ** ** * ** a ** ** ** ** ** ** **
ia* rke ian et Japa Ita tral nes rwa ede bec* n Fe ngar ean* ore* land rio* esot rea* ina* tes* land hile* rica* rael*
ven Tu Ind chus u s do
N o S w ue ia Hu m a p ea nta nn Ko Ch Sta ng C A f I s
Slo a A In Q uss g Z O M i g, E h
ss R CD Sin ew on ite
d ut
Ma OE N
g K Un So
n
Ho
1 2 http://dx.doi.org/10.1787/888933085957
Figure15.5 Evaluation of teachers practices by inspectors observation in 8th grade science

% -13 -7 -3 -1 -0 -0 +1 +3 +5 +5 +6 +8 +8 +11 +12 +12 +12 +14 +14 +18 +21 +21 +21 +26 +34
% point
100 35
90
30
80
70 25
60 20
50
40 15
30 10
20
10 5
0 0
** a y ia ly ts n ia y n ** y ** a ** * ** ** ** ** ** ** ** ** **
ia* ian rke al Ita set Japa nes rwa ede bec* ngar ean* esot Fed. land rio* orea ore* tes* ina* and* hile* rael* rica*
ven Ind Tu ustr ch u d o No Sw ue u m n n n e a nta K ap Sta Ch gl C I s A f
Slo A a In Q H i i a Z O g , En h
ss CD M uss ew Sin nited ong ut
Ma OE R N U ng K So
Ho
1 2 http://dx.doi.org/10.1787/888933085976

Figure15.6 Evaluation of teachers practices by inspectors observation in 4th grade

% -16 -3 +1 +3 +4 +4 +5 +5 +6 +6 +7 +8 +9 +9 +10 +12 +13 +15 +16 +19 +24 +26 +26 +26
% point
100 30
90
80 25
70
20
60
50 15
40
30 10
20
5
10
0 0
** n ly ic d a h ** * y* d y ic ** ** ** ** ** ** ** ** ** ** **
ia* de Ita ubl n Fe lbert mis bec* ark an glan ngar ubl ates ean* pan alia* nds rio* tria* ore* ay* ina* and*
ven Swe ep ia
s A F l e ue n m rm n
e E u ep St m Ja str r l a nta us a p orw Ch al
R e H R
Slo h Rus um
Q D G k ed D Au eth
e O A ing N ng, Ze
ec lgi va nit OEC S o ew
Cz Be Slo U N
gK N
n
Ho
1 2 http://dx.doi.org/10.1787/888933085995

TIMSS and PIRLS (2003, 2007 and 2011) surveys asked principals and heads of schools: In your school,
are any of the following used to evaluate the practice of <eighth-grade> mathematics/science teachers? []
Observations by inspectors or other persons external to the school, with response options Yes/No. The
same question was asked to 4th grade school principals and heads of schools. The same data restrictions as

Peer evaluation
General findings
Innovation in peer evaluation practices typically took the form of more use of teacher peer
review in secondary education (Figure15.7 and Figure15.8). Between 2003 and 2011, OECD average
absolute change regarding peer review activities was 12% points for 8th grade science teachers and
11% points for 8th grade mathematics teachers. South Africa (34% points), Hong Kong (27% points),
Korea (22% points) and New Zealand (21%) were the education systems showing the largest increases
in the use of peer review among 8th grade science teachers, while Norway (17% points) had the most
prominent decease in this respect. Similarly, the use of peer reviews for evaluating mathematics
teachers also increased especially in South Africa (36% points), Hong Kong (26% points) and Korea
(22% points), whereas the decrease was particularly marked in the case of Indonesia (13% points).
These education system changes showed medium effect sizes except in the case of Korea where the
effect size was large. The OECD average absolute change in teacher peer review use presented small
effect sizes both for 8th grade mathematics and science teachers. Altogether, use of teacher peer
review as a tool for evaluation increased in 11 education systems and decreased in five in both 8th
grade science and mathematics, with at least small effect sizes.
At the primary school level, innovation in the form of greater adoption of teacher peer review
for evaluation was also widespread (Figure15.9). The OECD average absolute change in the period
between 2003 and 2011 amounted to 12% points for teachers in 4th grade. The largest increase was
observed in Austria (34% points), while in the case of Flemish teachers (24% points), use of peer
reviews significantly diminished in comparison with other countries. These changes had medium
effect sizes, whereas the OECD average absolute effect size was small. Overall, with at least small
effect sizes, use of teacher peer reviews as an evaluation tool increased in 11 education systems and
decreased in four.
Teacher evaluations through peer review increased simultaneously across disciplines and levels
in Hong Kong and New Zealand. Between 2003 and 2011 teacher peer reviews aimed at evaluating
other colleagues practices increased in these education systems in 8th grade mathematics and
science and in 4th grade, with at least small effect size. Similarly, more 8th grade Korean schools
conducted evaluations through peer reviews across disciplines between 2003 and 2011, with large
effect size.
The cases of Norway and Italy, on the contrary, illustrate that innovation can also take the
form of a reduction across disciplines and levels in the adoption of teacher peer reviews as a tool
for evaluation. In these countries, use of peer reviews decreased consistently in 8th grade science,
mathematics and 4th grade between 2003 and 2011, with at least small effect sizes.

Figure15.7 Peer review evaluation of teachers practices in 8th grade mathematics

Percentage of students in schools in which peer review is used to evaluate the practice of teachers and change over time
% -16 -15 -13 -11 -10 -4 +2 +4 +4 +5 +6 +7 +8 +10 +11 +13 +14 +16 +20 +21 +21 +22 +23 +26 +36
% point
100 40
90 35
80
30
70
60 25
50 20
40 15
30
20 10
10 5
0 0
*** *** *** y* ** ec ed ry ia d io *** n ta a *** s* ** *** *** *** *** *** *** ***
ile ey sia wa Italy ueb an F nga ven glan ntar ean Japa eso dian ates sett eden alia and ore rael orea ina rica
Ch Turk one Nor Q ssi H u Slo En O m i n n In St h u w str al ap Is K , Ch Af
In
d Ru CD M t ed ssa
c S Au Ze Sing ng uth
E i a ew Ko So
O Un M N n g
Ho
1 2 http://dx.doi.org/10.1787/888933086014
Figure15.8 Peer review evaluation of teachers practices in 8th grade science

% -17 -17 -16 -14 -13 -6 +1 +2 +3 +4 +5 +6 +7 +8 +11 +13 +18 +19 +19 +19 +19 +21 +22 +27 +34
% point
100 35
90
30
80
70 25
60
20
50
40 15
30 10
20
10 5
0 0
*** *** ia*** ile** ly*** bec Fed. and ario ota ary n*** nia pan ana s*** n*** el*** tts** ia*** re*** d*** a*** a*** a***
ey ay s g e Ja di te ic
u rk orw nes Ch Ita Que ian Engl Ont nne un mea Slov e a e l o n
In Sta wed Isr hus stra gap eala Ko , Ch Afr
re in
T N do ss Mi H D
In Ru E C i t ed S
s ac Au Sin w Z o n g uth
s o
O Un Ma Ne ng
K S
Ho
1 2 http://dx.doi.org/10.1787/888933086033

Figure15.9 Peer review evaluation of teachers practices in 4th grade

% -24 -14 -12 -9 -6 -4 +1 +3 +4 +6 +6 +7 +8 +8 +8 +9 +9 +10 +12 +12 +20 +20 +23 +34
% point
100 35
90
30
80
70 25
60
20
50
40 15
30 10
20
10 5
0 0
* * * * k c c a * y * * * s * * * * * * * *
l g iumish** ay* ec** taly* mar ubli ubli bert ed.* an an** gary rio* and land eden tes** pore nia* pan* na** lia** nd** ria**
Be lem Norw ueb I en ep ep Al n F
e r m e un nta erl n g w t a g a v e Ja h i
t r a ala s t
m E S o , C us Ze u
F Q D kR hR ss
ia G D H O eth d S Sin Sl ng A w A
va ec Ru OE
C N ite Ko e
Slo Cz Un n g N
Ho
1 2 http://dx.doi.org/10.1787/888933086052

TIMSS and PIRLS (2003, 2007 and 2011) surveys asked principals and heads of schools: In your school,
are any of the following used to evaluate the practice of <eighth-grade> mathematics/science teachers?
[]Teacher peer review, with response options Yes/No. The same question was asked to 4th grade school
principals and heads of schools. The same data restrictions as in International TIMSS and PIRLS Reports by
the International Association for the Evaluation of Educational Achievement (IEA) apply to these data.

Incentive use for recruitment and retention of teachers
General findings
Innovation in teacher recruitment and retention in secondary education took the form of
increases and decreases in the use of different types of incentives (Figure15.10 and Figure15.11).
The OECD average absolute change regarding incentives use between 2003 and 2011 amounted
to 6% points and 5% points for teachers in 8th grade mathematics and science respectively. Use
of pay, housing or signing bonus incentives particularly increased in Singapore (40% points), the
Russian Federation (28% points), Israel (18% points) and Japan (13% points) for teachers in 8th grade
mathematics. Similarly, Singapore (40% points), the Russian Federation (30% points) and Israel
(16% points) were the countries that increased the use of incentives to recruit and retain 8th grade
science teachers. These changes registered from large to medium effect sizes, while the OECD
average absolute changes had small effect sizes. Overall, the use of incentives for recruiting and
retaining teachers increased in 11 education systems and decreased in four in the case of 8th grade
mathematics teachers, with at least small effect size . Incentive use for recruiting and retention of
8th grade science teachers increased in ten education systems and decreased in five, with at least
small effect sizes.
In comparison with secondary education, change in the use of incentives for recruiting and
retaining teachers in primary education was less pronounced (Figure15.12). In this respect, the
OECD average absolute change in the period between 2003 and 2007 amounted to 3% points for 4th
grade teachers. The largest increases in the use of incentives to recruit and retain teachers in 4th
grade were observed in the Russian Federation (11% points), Hungary (9% points), and Singapore (7%
points). In addition to statistical significance these changes presented small effect sizes. Altogether,
with a small effect size, increases in the use of incentives were significant in eight education systems.
Incentive use for teacher recruitment and retention increased simultaneously across disciplines
and levels in the Russian Federation, Singapore and Hungary. Between 2003 and 2011 incentives to
recruit and retain teachers in 8th grade and 4th grade were used significantly more in these countries,
with at least small effect size.
On the contrary, use of incentives to recruit and retain teachers significantly decreased across
disciplines in Indonesia, Norway and Turkey. Fewer schools in these countries used incentives for
8th grade mathematics and science teacher recruitment and retention between 2003 and 2011, with
small effect size.

Figure15.10 Use of incentives for teacher recruitment and retention in 8th grade mathematics
Percentage of students in schools which currently use any incentives to recruit or retain teachers and change over time
% -10 -10 -8 -7 -5 -4 -4 -2 -0 -0 +0 +1 +2 +2 +3 +4 +5 +6 +6 +11 +13 +18 +28 +40

% point
70 45
60 40
35
50
30
40 25
30 20
20 15
10
10
5
0 0
* ** y** nd ota en nd nia tes lia na ica rio an* ec* tts a** a** ry* e*** ** ** d*** e***
sia y* e a d a a a r a e e n a l n* el*
o ne rwa urk ngl nnes Swe eal love d Sta ustr Indi h Af Ont me ueb hus Kor Chi ung Chi Japa Isra n Fe apor
d N o T E Z S te ut CD Q ac g,
In Mi w i A
So ss on
H ia
ss Sin
g
Ne Un OE Ma Ru
n gK
Ho
1 2 http://dx.doi.org/10.1787/888933086071
Figure15.11 Use of incentives for teacher recruitment and retention in 8th grade science
Percentage of students in schools which currently use any incentives to recruit or retain teachers and change over time
% -18 -15 -10 -8 -8 -4 -4 -1 -1 +0 +1 +1 +1 +1 +2 +4 +4 +5 +6 +7 +9 +16 +30 +40

% point
70 45
60 40
35
50
30
40 25
30 20
20 15
10
10
5
0 0
*** d** ey** ota ay** and den alia nia ana ates bec ean rica ario ina* etts ea** an** ry** ile** el*** d.*** re***
sia an k es l e tr e i t e f t h s r p a Ch a e o
o ne ngl Tur inn Norw Zea Sw Aus Slov Ind ed S Qu CD m th A On g, C chu Ko Ja ung Isr an F gap
d E it E Sou n ssa H i
In M
N ew
U n O Ko a u ss Sin
ng M R
Ho
1 2 http://dx.doi.org/10.1787/888933086090

Figure15.12 Use of incentives for teacher recruitment and retention in 4th grade
Percentage of students in schools which currently use any incentives to recruit or retain teachers
% -5 -2 -1 -0 +0 +0 +1 +1 +2 +2 +2 +2 +4 +7 +9 +11
% point
40 12
35 10
30
8
25
20 6
15 4
10
2
5
0 0
nd lia nd ina es ay n* an nd
s nd io be
c ia* re*
*
ry*
*
d.*
*
la str
a
otl
a
Ch tat rw ea Jap rla ala tar ue en po ga Fe
ng c , S o m e e n Q ov a n n
E Au S ng ite
d N
CD th Z O Sl g Hu ia
Ko Un OE Ne Ne
w Sin ss
ng Ru
H o
1 2 http://dx.doi.org/10.1787/888933086109

TIMSS and PIRLS (2003, 2007 and 2011) surveys asked principals and heads of schools: Does your school
currently use any incentives (e.g. pay, housing, signing bonus, smaller classes) to recruit or retain <eighth-
grade> teachers in the following fields? [] Mathematics [] Science [] Others, with response options Yes/
No. The same data restrictions as in International TIMSS and PIRLS Reports by the International Association
for the Evaluation of Educational Achievement (IEA) apply to these data.

Summary
Innovation or significant change regarding teacher evaluation has meant more and less use of
achievement and assessment data, a greater degree of external evaluation by inspectors in primary and
secondary education, as well as a greater use of teacher peer reviews for these purposes between 2003
and 2011. The direction of innovation in recruiting and retaining practices differed across education
systems with regard to incentives use, with little overall change concerning OECD average level.
Figure15.13 Change in the use of achievement and assessment data for school evaluation
Achievement to evaluate principals Achievement to evaluate teachers Assessment to judge effectiveness

%
80
60
40
20
0
-20
-40
-60
ile rg an on aly tes en ria nd nd da nd ael lia ds ny sia lic in m nd al ia a d y rk ce ry ey nia an an e)
Ch bou Japerati It Sta wed ust Pola cela ana inla Isr stra rlanrma one pub Spaelgiueala rtug ston Koreelanorwama GreeungaTurklove me me ang
xe
m ed ed S A I C F Auethe Ge Ind h Re B Z Po E Ir N en H S CD CD . ch
Lu a nF U nit N z ec Ne
w D OE OE(abs
i C
ss
Ru
1 2 http://dx.doi.org/10.1787/888933086128
Figure15.14 Change in the evaluation of teachers practices by inspectors observation

%
80
60
40
20
0
-20
-40
-60
-80
ly sh ia n a n ia ds ec n ry ea ay ia io le es d d re el a n n )
Ita emi nes ede dian Japa stral rlan ueb ratio nga Kor orw ven ntar Chi Stat glan alan apo Isra Chin mea mea nge
Fl Indo Sw In u he Q de Hu N Slo O d E n Ze ing g , D D h a
A t e it e n C C c
Ko OE OE abs.
um F w S
lgi Ne ian Un Ne
Be ss ng (
Ru Ho
1 2 http://dx.doi.org/10.1787/888933086147
Notes: For details, please see Figures15.1, 15.2, 15.3, 15.4, 15.5 and 15.6.

Figure15.15 Change in peer review evaluation of teachers practices

%
100
80
60
40
20
0
-20
-40
-60
-80
s y
ion nd and ar ario an ni
a a
be
c ish an sia ile aly en es el ay ea re lia nd ca na an an e)
rat erla ngl ung Ont Indi love Que Flem Ja don
p e Ch It
w ed Stat Isra orw Kor gapo stra eala Afri Chi me me ang
e ,
ed th E H S In S e d N
Si n Au w Z uth ong ECDECDs. ch
a n F Ne g ium U nit Ne So g K O O (ab
i l n
ss Be Ho
Ru
1 2 http://dx.doi.org/10.1787/888933086166
Figure15.16 Change in the use of incentives for teachers recruitment and retention

%
100
80
60
40
20
0
-20
-40
-60
-80
es ds rica alia ario bec enia den and ina esia rea way pan hile gary and rael tion ore ean ean ge)
tat lan f tr t e v e al h n Ko Nor Ja C un gl Is era gap m m an
e d S ther uth A Aus On Qu Slo Sw Ze ng, C Indo H En ed Sin ECD ECDs. ch
i t w
U n N e S o e
N ng Ko
i a nF O O (ab
Ho ss
Ru
1 2 http://dx.doi.org/10.1787/888933086185
Notes: For details, please see Figures15.7, 15.8, 15.9, 15.10, 15.11 and 15.12
Note on Israel

Table 15.1 Effect sizes for changes in school evaluation and hiring practices
Change in data use for Change in external evaluation Change in peer evaluation Change in use of incentives
evaluation
Achievement data to
to evaluate teachers
evaluate principals
to judge teachers
Achievement data
Assessment data
Change in Change in peer review Change in use of incentives
effectiveness
performance
performance
inspectors observations evaluation of teachers for teachers recruitment
of teacher practices practices and retention
8th grade 8th grade 4th 8th 4th grade 8th 4th grade
grade grade grade
maths science all maths science all maths science all
03-09 06-09 06-09 03-11 03-11 03-11 03-11 03-11 03-11 03-11 03-11 03-07
Australia 0,21 -0,01 -0,07 0,04 -0,02 0,40 0,40 0,38 0,41 -0,01 -0,05 -0,10
Austria -0,21 -0,03 0,00 m m m m m m m m m
Belgium 0,26 0,12 0,15 m m m m m m m m m
Belgium Flemish m m m m m 0,10 m m -0,65 m m m
Canada 0,08 -0,11 -0,12 m m m m m m m m m
Ontario m m m 0,53 0,51 0,50 0,19 0,10 0,24 0,24 0,24 0,25
Quebec m m m 0,41 0,43 0,47 -0,13 -0,19 -0,43 0,20 0,06 0,31
Chile m -0,01 -0,01 0,54 0,50 m -0,34 -0,30 m 0,33 0,27 m
Czech Republic -0,03 -0,24 -0,12 m m m m m m m m m
Denmark 0,19 0,35 0,36 m m m m m m m m m
Estonia m -0,32 -0,27 m m m m m m m m m
Finland -0,19 -0,08 0,12 m m m m m m m m m
Germany 0,27 -0,09 -0,02 m m m m m m m m m
Greece 0,17 0,47 0,39 m m m m m m m m m
Hungary -0,33 -0,28 -0,27 0,16 0,17 0,17 0,10 0,13 0,21 0,21 0,25 0,28
Iceland -0,02 -0,06 0,28 m m m m m m m m m
Ireland 0,45 -0,19 -0,17 m m m m m m m m m
Israel m -0,18 0,16 0,68 0,52 m 0,44 0,38 m 0,55 0,51 m
Italy -0,08 0,01 -0,09 0,05 -0,03 0,05 -0,23 -0,30 -0,21 m m m
Japan -0,07 -0,05 -0,03 0,01 0,03 0,25 0,16 0,16 0,25 0,59 0,35 0,24
Korea 0,25 0,21 0,09 0,37 0,29 m 0,84 0,82 m 0,26 0,26 m
Luxembourg 0,02 0,13 m m m m m m m m m m
Mexico 0,06 -0,11 0,13 m m m m m m m m m
Netherlands 0,18 -0,04 0,07 m m 0,30 m m 0,20 m m 0,28
New Zealand 0,15 0,05 0,23 0,25 0,24 0,53 0,50 0,53 0,49 -0,12 -0,13 0,08
Norway 0,12 0,23 0,35 0,22 0,21 0,74 -0,24 -0,36 -0,29 -0,33 -0,31 0,02
Poland 0,13 -0,05 0,09 m m m m m m m m m
Portugal 0,00 -0,49 -0,05 m m m m m m m m m
Slovak Republic 0,13 -0,03 0,01 m m m m m m m m m
Slovenia m 0,23 1,02 -0,32 -0,38 -0,40 0,10 0,15 0,28 -0,13 -0,05 0,24
Spain 0,16 -0,15 0,08 m m m m m m m m m
Sweden 0,01 -0,10 0,09 0,18 0,16 m 0,34 0,36 m -0,18 -0,14 m
Switzerland 0,08 0,11 0,08 m m m m m m m m m
Turkey 0,79 -0,06 -0,09 m m m m m m m m m

Table 15.1 Effect sizes for changes in school evaluation and hiring practices (continued)
Change in data use for Change in external evaluation Change in peer evaluation Change in use of incentives
evaluation
Achievement data to
to evaluate teachers
evaluate principals
to judge teachers
Achievement data
Assessment data
Change in Change in peer review Change in use of incentives
effectiveness
performance
performance
inspectors observations evaluation of teachers for teachers recruitment
of teacher practices practices and retention
8th grade 8th grade 4th 8th 4th grade 8th 4th grade
grade grade grade
maths science all maths science all maths science all
03-09 06-09 06-09 03-11 03-11 03-11 03-11 03-11 03-11 03-11 03-11 03-07
United Kingdom 0,02 -0,03 0,08 m m m m m m m m m
England m m m 0,45 0,49 0,22 0,12 0,06 0,18 -0,17 -0,37 -0,18
Scotland m m m m m m m m m m m -0,10
United States 0,07 0,00 0,12 0,41 0,39 0,20 0,29 0,28 0,23 -0,01 0,02 0,02
Indiana m m m -0,11 -0,18 m 0,27 0,29 m 0,00 0,00 m
OECD (average) 0,10 -0,03 0,07 0,25 0,21 0,29 0,18 0,15 0,13 0,10 0,06 0,11
OECD (average absolute) 0,16 0,14 0,16 0,30 0,28 0,35 0,28 0,28 0,28 0,24 0,22 0,17
Hong Kong, China 0,27 -0,17 -0,27 0,41 0,47 0,56 0,54 0,56 0,45 0,32 0,25 -0,01
Indonesia 0,48 0,02 -0,06 0,14 0,08 m -0,52 -0,68 m -0,24 m m
Russian Federation -0,08 -0,08 -0,09 0,16 0,26 0,10 0,09 0,07 0,21 0,57 0,60 0,24
Singapore m m m 0,41 0,45 0,70 0,43 0,39 0,20 0,98 0,98 0,26
South Africa m m m 0,63 0,70 m 0,79 0,74 m 0,03 0,05 m
1 2 http://dx.doi.org/10.1787/888933086204
Source: Authors calculations based on TIMSS (2003, 2007 and 2011) andPISA (2003, 2006 and 2009)

CHAPTER 16
Innovation in schools external relations
Innovation in schools can incorporate changes to external relation practices

whether aimed at informing parents on their childs performance or to involve
them in certain activities as well as for self-promotion purposes. The aim of
innovation with regard to external relations could be, for example, to create
a stronger and supportive sense of community between schools, parents
and students as well as to promote the schools and extend their outreach to
previously under served students.

16. INNOVATION IN SCHOOLS EXTERNAL RELATIONS
Informing parents
General findings
In general, innovation in school external relations has not resulted in changes in the use of
different methods to inform parents about their childs progress (Figure16.1 and Figure16.2).
Assessment data were extensively used as tools to inform parents in 2000 and there has been little
change between then and 2009. Similarly, parental perception regarding school provision of regular
information on students performance did not change significantly between 2006 and 2009. OECD
average absolute change regarding the use of student assessment data to inform parents about their
childs progress amounted to 2% points in the period between 2000 and 2009. Likewise, parental
perception of regular performance information provision registered the same OECD average absolute
change between 2006 and 2009. The United States (21% points) registered the largest increase in
the use of assessment data to inform parents about their childs performance. This change was
characterised by a medium effect size. Altogether, use of assessment data to inform parents
increased in two education systems and decreased in three, with at least a small effect size.
As to the modalities through which parents are informed about their childs performance,
innovation is indicated through both increased and decreased provision of information on student
comparative performance against different points of reference (Figure16.3 to Figure16.5). Between
2006 and 2009, OECD average absolute change regarding provision of information on student
performance relative to other students in the school was 10% points, while it amounted to 7%
points when information on student performance was presented in comparison with regional or
national benchmarks. OECD average absolute change regarding performance information provision
comparing group of students to same grade students from another school was 7% points. Decreases
were particularly pronounced in terms of information on student performance being presented
in comparison with those of others in the same class in the Slovak Republic (33% points) and the
Russian Federation (21% points). Indonesia (25% points) exhibited the largest decrease in the use of
national or regional benchmarks as a point of reference in student performance comparisons. The
Slovak Republic (29% points) showed the largest decrease in performance information provision
presented in comparison to same grade but different school students. These changes had from
large to medium effect size, while the OECD average absolute change presented small effect sizes
when performance information was presented in comparison with those of others in the same
class. Overall, other students were more frequently used as a reference for performance comparison
in two education systems and less in 18, while regional or national benchmarks were used more
frequently in five education systems and less in seven, with at least small effect sizes. Information
on performance compared with same grade but different school students was used more frequently
in four education systems and less in eight, with at least small effect sizes.
Innovation in providing more information on student performance was particularly pronounced
in Portugal. Portuguese schools provided parents with more information about their childs
performance relative to students of other schools and in comparison with national or regional
benchmarks between 2006 and 2009, with small effect sizes.
In contrast, innovation in the Slovak Republic was characterised by a lower degree of comparative
performance information provision to parents. Slovak schools provided less comparative information
about student performance in comparison with other classmates or students from other schools
between 2006 and 2009, with small effect sizes.

Figure16.1 Parental perceptions of receiving regular information about their childs progress
Percentage of 15-year old students whose parents agree or strongly agree that they receive regular information
about their childs progress and change over time
% +1 +1 +2 +2 +2 +4 +5
% point
100 6
90
80 5
70
4
60
50 3
40
30 2
20 1
10
0 0
ly d y a* * ** l* ** ***
Ita ala
n an hin an ga ar
k
Ze rm C e tu
Ge g, m
Po
r nm
Ne
w on C D De
gK OE
Hon
1 2 http://dx.doi.org/10.1787/888933086223
Figure16.2 Use of assessment data to inform parents about their childs progress
Percentage of 15-year old students in schools where assessment data are used to inform parents about their childs progress
% -3 -3 -3 -2 -1 -1 -1 -1 -1 -1 -0 -0 -0 -0 -0 +0 +0 +0 +0 +0 +0 +0 +1 +1 +1 +1 +2 +2 +2 +7 +21
% point
100 25
90
80 20
70
60 15
50
40 10
30
20 5
10
0 0
c d y d ** a y e l g k n n
nd ile ny* ary bli an tal an lia esi ium da wa ec ga nd ur ar nd de tio nd n** om an ico ds ael in* ia* s***
e rla Chrma ung epu Finl I Zeal stra don Belg anaNor Gre ortu Icelamboenm IrelaSwe era Polamea ngd JapMex rlan Isr Spa ustrtate
it z R u n C P e d D Ki e A S
Ge H ch w A I
Lu
x D Fe th d
Sw e Ne ian EC ited Ne ite
Cz s s O
U n U n
R u
1 2 http://dx.doi.org/10.1787/888933086242
Notes: *** = change significant at the 0.01 level; ** = change significant at the 0.05 level; * = change significant at 0.1 level.

Figure16.3 Information provided to parents about their childs performance relative

to other students in the same school
Percentage of 15-year old students in schools that provide information to parents on their childs academic performance relative
to other students in the school and change over time
% -33 -21 -21 -19 -19 -19 -19 -17 -15 -14 -14 -13 -12 -12 -11 -10 -9 -9 -7 -7 -5 -4 -4 -3 -3 -2 -1 -1 -0 +1 +1 +2 +3 +5 +10 +20
% point
100 35
90
30
80
70 25
60 20
50
40 15
30 10
20
10 5
0 0
Slovak Republic***
Germany***
United Kingdom***
Poland***
Iceland***
Israel***
Netherlands***
Korea***
Estonia***
Chile**
Canada***
United States**
Spain***
Slovenia***
Portugal*
Hungary*
OECD mean***
Indonesia***
Italy*
Greece
Switzerland
New Zealand
Finland
Czech Republic
Sweden
Mexico
Norway
Ireland
Belgium
Austria
Turkey
Luxembourg***
Australia***
Japan***
Russian Federation***
Hong Kong, China**
1 2 http://dx.doi.org/10.1787/888933086261
Figure16.4 Information provided to parents about their childs performance relative to national or
regional benchmarks
Percentage of 15-year old students in schools that provide information to parents on their childs academic performance relative to
national or regional benchmarks and change over time
% -25 -18 -14 -12 -11 -11 -9 -7 -7 -6 -6 -5 -5 -4 -4 -4 -3 -3 -3 -2 -0 -0 +0 +1 +2 +2 +3 +4 +4 +6 +7 +8 +8 +10 +20 +21 +22
% point
100 30
90
80 25
70
20
60
50 15
40
30 10
20
5
10
0 0
Indonesia***
Finland***
United Kingdom***
Denmark**
Estonia**
Iceland***
Canada***
Chile*
Slovak Republic
Sweden**
Germany
Hong Kong
Italy*
Austria*
Greece
Poland
Australia
Netherlands
Ireland
Japan
OECD mean
United States
Turkey
New Zealand
Israel
Switzerland
Russian Federation
Korea
Slovenia***
Czech Republic
Hungary
Norway
Spain**
Portugal*
Luxembourg***
Belgium***
Mexico***
1 2 http://dx.doi.org/10.1787/888933086280
Source: Authors calculations based on PISA (2006 and 2009.

Figure16.5 Information provided to parents about student group performance relative to students in
the same grade in other schools
Percentage of 15-year old students in schools that provide information to parents on the academic performance of students relative to
students in the same grade in other schools and change over time
% -29 -17 -12 -12 -12 -11 -10 -10 -8 -7 -5 -5 -4 -4 -4 -3 -3 -3 -3 -3 -2 -1 +0 +0 +1 +1 +1 +2 +2 +2 +4 +4 +8 +13 +14 +15
% point
100 30
90
80 25
70
20
60
50 15
40
30 10
20
5
10
0 0
United Kingdom*
Slovak Republic***
Chile***
Denmark**
Indonesia**
Germany***
Iceland***
Russian Federation**
Canada***
Netherlands**
United States
Switzerland*
Korea
Israel
Italy**
Turkey
Luxembourg***
Sweden
Austria
Ireland
OECD mean***
Czech Republic
Hungary
Belgium
Spain
Hong Kong
Mexico
Finland
Slovenia***
Australia
Estonia
Poland
Portugal***
Norway***
New Zealand***
Greece***
1 2 http://dx.doi.org/10.1787/888933086299

PISA (2000, 2003 and/or 2009) surveys asked school principals: In your school, are
assessments of students in <national modal grade for 15-year-olds> used for any of the
following purposes? []To inform parents about their childs progress with answer options
Yes/No. PISA (2006 and 2009) surveys asked parents: How much do you agree or disagree with
the following statements? [] My childs school provides regular and useful information on my
childs progress with answer options Strongly Agree; Agree; Disagree; Strongly disagree.
PISA (2006 and 2009) surveys asked school principals: Does your school provide
information to parents of students in <national modal grade for 15-year-olds> on their childs
academic performance relative to other students in <national modal grade for 15-year-olds>
in your school?, Does your school provide information to parents of students in <national
modal grade for 15-year-olds> on their childs academic performance relative to national
or regional <benchmarks>?. Does your school provide information to parents on the
academic performance of students in <national modal grade for 15-year-olds> as a group
relative to students in the same grade in other schools?. Answer options were Yes/No.

Parental involvement
General findings
In secondary education, innovation in terms of schools openness to parental involvement with
the school was indicated through both increased and reduced invitations to parents to join school
committees and in volunteering in projects, programmes or trips in their childs school (Figure16.6
and Figure16.7). Between 2003 and 2007 the OECD average absolute change in the practice of asking
parents to serve on school committees was 10% points, while the change in terms of asking parents
to volunteer in projects, programmes and trips amounted to 6% points. Invitations for parental
participation in 8th grade school committees increased mostly in countries such as the United
States (15% points), Hungary (15% points), Indonesia (14% points), Hong Kong (13% points), Korea
(10% points), the Russian Federation (9% points) and Australia (7% points). It decreased in Slovenia
(22% points), Italy (16% points) and Israel (13% points). Parent volunteering in school projects,
programmes or trips increased especially in Singapore (15% points), Australia (13% points), Norway
(13% points), United States (7% points) and the Russian Federation (6% points). Decreases were
particularly remarkable in the case of Indonesia (18% points) and Italy (11% points). These changes
presented small effect size as did the OECD average absolute change in schools asking paretns
of 8th grade studetns to serve on committees. Overall, parental involvement in school committees
increased in eight education systems and decreased in four, while parent volunteering increased in
five education systems and decreased in two, with small effect sizes.
Innovation in parental involvement at the primary school level was also illustrated through
increased and decreased invitations to participate in school committees or volunteering in projects,
programmes and trips across OECD countries (Figure16.8 and Figure16.9). OECD average absolute
change regarding parents being asked to serve on 4th grade school committees was 8% points,
while it amounted to 4% for parents being asked to volunteer in school projects, programmes and
trips between 2003 and 2007. Requests for parental participation in school committees increased
particularly in Hungary (14% points), Scotland (10% points), the Russian Federation (8% points)
and New Zealand (6% points), while decreases were substantial in Slovenia (19% points) and Italy
(12% points). Parents were asked more frequently to volunteer for projects, programmes and trips
especially in Norway (7% points), Australia (7% points), and Singapore (3% points), while it was the
opposite in Italy (13% points). These changes were characterised by medium effect sizes. Altogether,
school requests for parental involvement in committees increased in six education systems and
decreased in two with at least small effect sizes, while demand for parental volunteering in projects
programmes and trips augmented in four education systems and reduced in one.
Singapore illustrates the case of an education system where requests for parental involvement
increased across levels and practices. Between 2003 and 2007, schools in Singapore asked more
parents to serve on school committees both in 8th and 4th grade and to volunteer in projects,
programmes and trips in 4th grade, with small effect sizes.
On the contrary, Italy showed simultaneous decreases across practices and levels with regard to
requests for parental involvement. Italian schools decreased their invitations for parents to serve
on committees as well as to volunteer for projects, programmes and trips in both primary and
secondary education between 2003 and 2007.
Indonesia is the only example of a country where secondary schools asked for more parental
participation in committees and less volunteering in projects, programmes and trips between 2003
and 2007; both changes were characterised by small effect sizes.

Figure16.6 Schools asking parents of 8th grade students to serve on committees

Percentage of students in schools that ask parents to serve on school committees and change over time
% -22 -16 -13 -12 -1 -1 -1 +0 +5 +5 +6 +7 +7 +9 +10 +13 +14 +15 +15

% point
100 25
90
80 20
70
60 15
50
40 10
30
20 5
10
0 0
*** *** l** io* * * *** a** ia*** ary** es***
ae
e
or apan rway ea
n en try land lia* ebec on**
nia Ita
ly
Isr tar gap J m ed oun t ra u i rea hin s
ng tat
ve n N o
CD S w c c o s t Q a t K o C ne
Slo
O Sin ue
S Au de
r g, do Hu ted S
OE s q F e K o n I n i
Ba ian ng Un
ss Ho
Ru
1 2 http://dx.doi.org/10.1787/888933086318
Figure16.7 Schools asking parents of 4th grade students to serve on committees

Percentage of students in schools that ask parents to serve on school committees and change over time
% -19 -12 -6 -2 +1 +3 +3 +5 +6 +7 +8 +8 +10 +10 +11 +14

% point
100 20
90 18
80 16
70 14
60 12
50 10
40 8
30 6
20 4
10 2
0 0
** ly*
* io ay n an lia ds ** * ec ** ** e* * **
ia* tar rw ea Jap tra an nd tes eb ion tland or ina ary
ve
n Ita On No m s erl ala Sta Qu rat g ap Ch ng
Slo C D Au t h Z e d e c o
Si n g , u
OE Ne w ite ed S
Ko
n H
Ne Un nF ng
s sia H o
Ru
1 2 http://dx.doi.org/10.1787/888933086337

Figure16.8 Schools asking parents of 8th grade students to volunteer for projects
Percentage of students in schools that ask parents to volunteer for projects, programmes and trips and change over time
% -18 -11 -8 -6 -5 -3 -3 +1 +2 +2 +2 +4 +4 +6 +7 +9 +13 +13 +15

% point
100 20
90 18
80 16
70 14
60 12
50 10
40 8
30 6
20 4
10 2
0 0
** * a
ly* apan hin tland ebe
c io ia a n el ry ry on** *** n* *** lia** re***
ia* Ita ar en Kore mea Isra nga nt i es de way
n es J , C Sco Qu O nt lov D u c ou erat tat Swe r str
a
ap
o
do g S H S o
In Ko
n
OE
C e d
qu n Fe nited
N Au Sing
g s
n Ba ia U
Ho ss
Ru
1 2 http://dx.doi.org/10.1787/888933086356
Figure16.9 Schools asking parents of 4th grade students to volunteer for projects
Percentage of students in schools that ask parents to volunteer for projects, programmes and trips and change over time
% -13 -3 -2 -1 -1 +1 +1 +1 +1 +1 +3 +3 +4 +5 +7 +7
% point
100 16
90 14
80
12
70
60 10
50 8
40 6
30
4
20
10 2
0 0
* s * * ** **
ly* nd ina an ar
io ea
n
eb
ec nd tes ga
ry an
d
re* ion en
ia
lia ay
Ita rla Ch Jap On
t m Qu ala Sta un otl po rat ov tra rw
th
e g ,
CD Ze d H S c g a e S l s o
Ne Ko
n
OE w ite Sin ed Au N
ng Ne Un nF
Ho s sia
Ru
1 2 http://dx.doi.org/10.1787/888933086375

TIMSS (2003 and 2007) asked principal and head of schools in 8th and 4th grade Does your
school ask parents to do the following? [] Volunteer for school projects, programs, and trips
[] Serve on school committees (e.g., select school personnel, review school finances) with
response options Yes/No. The same data restrictions as in International TIMSS and PIRLS
Reports by the International Association for the Evaluation of Educational Achievement (IEA)
apply to these data.

Public relations
General findings
Innovation in schools public relations can be illustrated by more or less self-promotion through
achievement data and more or less engagement of the principal in public relations and fundraising
among OECD countries (Figure16.10 and Figure16.11). The OECD average absolute change between
2006 and 2009 concerning public posting of achievement data amounted to 9% points in secondary
education, whilst OECD average absolute change regarding principal engagement in public relation
and fundraising was 10% points between 2003 and 2007. Slovak Republic (35% points) and New
Zealand (30% points) stood out as countries where the practice of posting achievement data publicly
was used more frequently in 2009 than in 2006. In Korea (19% points) and Canada (15% points)
principals devoted substantially more time into public relations and fundraising activities, while
they spent less time to these ends in Slovenia (19% points), Norway (19% points) and United States
(17% points). Changes in the posting of achievement data at country level showed medium effect
sizes, while changes related to principals engagement in public relations and fundraising showed
small effect sizes. OECD average absolute change in principals public relations and fundraising
showed a small effect size. Overall, with at least small effect sizes, public posting of achievement
data increased in six education systems and decreased in eight. The time spent by principals in
fundraising and public relations increased in two education systems, whereas decreases were
observed in four, with small effect sizes.
Innovation in primary education school public relation activities has taken the form of modest
changes with regard to principals engagement in community and parent relations (Figure16.12).
The OECD average absolute change between 2001 and 2006 in percentage of 4th grade principals
time devoted to this end was 2% points.
Innovation in Korea translated into schools engaging more in public relation activities in
secondary education. More Korean schools posted achievement data publicly in 2009 than 2006,
while principals spent more time in fundraising and public relations in 2007 than in 2003. These
changes were characterised by small effect sizes.

Figure16.10 Publicly posting school achievement data

Percentage of 15-year old students in schools where achievement data are publicly posted and change over time
% -19 -18 -15 -15 -14 -13 -10 -8 -8 -7 -7 -6 -5 -4 -4 -4 -3 -3 -2 -2 -2 -1 -1 -1 -1 -1 +0 +2 +4 +4 +5 +13 +14 +16 +17 +30 +35
% point
100 40
90 35
80
30
70
60 25
50 20
40 15
30
10
20
10 5
0 0
Russian Fed.
Netherlands***
Estonia***
Czech Republic***
Luxembourg***
United Kingdom***
Australia***
Israel**
Hong Kong
Canada**
Ireland**
Japan**
Norway
Sweden
Mexico
Spain
Switzerland*
Germany
Belgium*
Finland
Portugal
Italy
Austria
OECD mean*
United States
Chile
Greece
Slovenia
Denmark
Iceland
Hungary
Poland**
Turkey**
Korea***
Indonesia***
New Zealand***
Slovak Republic***
1 2 http://dx.doi.org/10.1787/888933086394
Sources: Authors calculations based on PISA (2006 and 2009)
Figure16.11 Public relations and fundraising by school principals

Percentage of 15-year olds in schools in which the principal spends 10% or more of his/her time on public relations and fundraising
% -19 -19 -17 -8 -8 -7 -6 -6 -5 -4 -4 -2 -0 +1 +1 +4 +5 +6 +15 +19

% point
100 20
90 18
80 16
70 14
60 12
50 10
40 8
30 6
20 4
10 2
0 0
** y*** s*** ry d rio pan taly ry en* n*
* ina or
e
ed ali
a
es
ia nd rael ec*** ea***
ia* e nt an ta Ja I ga d ea , Ch gap an F ustr don gla
ven orwa Stat c ou c otl On un Swe m g i n i n Is eb Kor
Slo N ed qu
e S H
CD Ko
n S us
s A I n E Qu
i t s E
O on g R
Un Ba
H
1 2 http://dx.doi.org/10.1787/888933086413
Sources: Authors calculations based on TIMSS (2003 and 2007)

Figure16.12 Time devoted to community/parent relations by 4th grade principals
% -3 -3 -3 -3 -3 -3 -2 -2 -2 -2 -2 -1 -1 -1 -1 -1 +0 +1 +2
% point
25 5
20 4
15 3
10 2
5 1
0 0
*
e** tes**
* *** el*** ia*** *** *** *** .*** ** d ** d ly y** ry ds and
*
n*
*
ay ina pore land Fed lic tlan n* lan Ita an unga rlan de
nc a rw a n
Isr love , Ch ub o ea ea l
ng Swe
Fra d St N o g a I c e
i a n e p S c m Z e rm H h e E
ite
S ng Sin ss kR CD New G
Ne
t
Un Ko Ru lova OE
o ng S
H
1 2 http://dx.doi.org/10.1787/888933086432

PISA (2006 and 2009) surveys asked principals and heads of schools in 8th grade: In your
school, are achievement data used in any of the following <accountability procedures>? []
Achievement data are posted publicly (e.g. in the media), with response options Yes/No.
TIMSS (2003 and 2007) survey asked principals and head of schools: By the end of this school
year, approximately what percentage of time in your role as principal [] Public relations
and fundraising, with answer option expressed in percentages. PIRLS (2001 and 2006) asked
principals of 4th grade classes As principal of this school, approximately what percentage of
your time is devoted to the following activities? []Parent and community relations with
answers expressed in percentages. The same data restrictions as in International TIMSS and
PIRLS Reports by the International Association for the Evaluation of Educational Achievement
(IEA) apply to these data.

Summary
Innovation or significant change regarding external relations practices has meant less
use of achievement and assessment data for the provision of comparative information on the
students performance between 2006 and 2009. On the other hand, more schools asked for parental
involvement in schools between 2003 and 2007. Principal and school engagement in public relations
and fundraising activities slightly decreased across OECD countries in the timeframe between 2001
and 2009.
Figure16.13 Change in provision of comparative information on students performance
Compared to students in the same grade in other schools Compared to national or regional benchmarks Compared to other students
%
60
40
20
-20
-40
-60
d ey lic ia n ly ia ry ia es d ay in n d ea ce m rk el ds d g l a a n y ile d ia n n )
lan rk ub str ede Itastral nga venStat alanorw Spa JapainlanKor ree lgiu ma Israrlan lanbourtugatoni nad atio manCh elanonesmeamea nge
Ire TuRep Au Sw u u
A H S ite wl o d Z e N F G B e e n e Po m o r E s a e
C ed Gr e r Ic nd CD D cha
h D th xe P I E EC s.
ec Un Ne Ne Lu nF O O (ab
Cz s i a
s
Ru
1 2 http://dx.doi.org/10.1787/888933086451
Figure16.14 Change in school involvement by parents of 8th grade students
Parents to serve committees Parents to volunteer in projects

%
60
40
20
-20
-40
an try bec nd orea way den tion rael ario pore gary hina ralia ates enia ly ia
Ita nes mea meaange
n n )
Jap oun Que cotla K Nor e a Is t t t
On inga Hun ng, C Aus ed S Slo
v
e c S Sw eder In
d o
C ECD s. ch
D
qu F S o n it O E
s ia n K U O (ab
Ba ss ng
Ru Ho
1 2 http://dx.doi.org/10.1787/888933086470
Notes : For details, please see Figures 16.1, 16.2, 16.3, 16.4 and 16.5.

Figure16.15 Change in school involvement by parents of 4th grade students
Parents to serve committees Parents to volunteer in projects

%
60
40
20
-20
-40
an land io ds es ec ay lia on nd ina re ry ia ly n n )
Jap tar lan tat eb rw tra erati tla Ch po ga en Ita ea ea ge
Ze
a On ther dS Qu No us co g, n ga un lov D
m m han
w t e A e d S n S i H S C C D . c
Ne Ne Un
i F Ko OE OE (abs
ian ng
ss Ho
Ru
1 2 http://dx.doi.org/10.1787/888933086489
Figure16.16 Change in public relation practices of schools
Principal community and parent relations Principal public relations Achievement data posted
4th grade 01-06 8th grade 03-07 publicly 06-09
%
60
40
20
-20
-40
ile ce ria rk gal nd re m on ce in nd ny nd da aly en ry nd an ey lia rg ia ael ds sia ia es ay d ea an n e)
ChGree ust nmaortu inlagapoelgiueratiFran Spacela rmaIrela ana It wedungaPola Jap Turkstra bouston Isr rlanoneovenStat orw alanKor me meaang
A e P F in B ed I Ge C Au xem E e e
D S
S H th Ind Sl ited Nw Z CD CD . ch
a nF Lu Ne n N e OE OE (abs
i U
ss
Ru
1 2 http://dx.doi.org/10.1787/888933086508
Notes : For details, please see Figures 16.6, 16.7, 16.8, 16.9 and 16.10.
Note on Israel

Table 16.1 Effect sizes for changes in school external relations practices
Change in information provision to parents Change in parents involvement Change in public relations
Principals time to communi-

Parental perceptions on pro-
parent on child performance

vision of regular information
serve on school committees
School asking parents to be

Achievement data to inform
students in the same grade

relative to national/regional
Schools asking parents to
Public relations and fund-

volunteer for school proj-
relative to other students
ects, programs and trips
Achievement data to be
Students performance
As a group relative to
raising by principals
ty/parent relations
in other schools
posted publicly
benchmarks
8th grade 8th grade 4th grade 8th grade 4th grade 8th grade 4th grade
06-09 06-09 06-09 06-09 06-09 03-07 03-07 03-07 03-07 06-09 03-07 01-06
Australia m -0.20 0.22 -0.06 0.05 0.30 0.13 0.28 0.35 -0.26 0.03 m
Austria m 0.21 0.05 -0.18 -0.10 m m m m -0.06 m m
Belgium m -0.06 0.03 0.50 0.05 m m m m -0.17 m m
Canada m -0.04 -0.30 -0.19 -0.22 m m m m -0.16 m m
Ontario m m m m m -0.26 -0.14 -0.12 -0.03 m -0.14 m
Quebec m m m m m 0.16 0.17 -0.07 0.04 m 0.32 m
Chile m -0.21 -0.29 -0.19 -0.36 m m m m -0.02 m m
Czech Republic m -0.09 -0.04 0.12 -0.03 m m m m -0.30 m m
Denmark 0.11 0.00 m -0.24 -0.25 m m m m 0.03 m m
Estonia m m -0.31 -0.22 0.09 m m m m -0.38 m m
Finland m -0.10 -0.08 -0.36 0.04 m m m m -0.11 m m
France m m m m m m m m m m m -0.10
Germany 0.04 -0.16 -0.42 -0.12 -0.29 m m m m -0.10 m -0.04
Greece m -0.10 -0.08 -0.12 0.45 m m m m -0.02 m m
Hungary m -0.12 -0.20 0.14 0.01 0.29 0.28 0.12 0.05 0.12 -0.13 -0.03
Iceland m 0.00 -0.42 -0.22 -0.25 m m m m 0.11 m -0.05
Ireland m 0.02 0.01 -0.06 -0.12 m m m m -0.17 m m
Israel m 0.09 -0.38 0.04 -0.10 -0.26 m 0.06 m -0.22 0.12 -0.07
Italy 0.02 -0.08 -0.14 -0.13 -0.16 -0.32 -0.24 -0.23 -0.25 -0.03 -0.16 -0.03
Japan m 0.11 0.41 -0.04 m -0.02 0.06 -0.19 -0.05 -0.28 -0.16 m
Korea m m -0.36 0.09 -0.08 0.31 m 0.03 m 0.38 0.40 m
Luxembourg m 0.00 0.14 0.48 -0.09 m m m m -0.30 m m
Mexico m 0.08 -0.02 0.44 0.03 m m m m -0.08 m m
Netherlands m 0.15 -0.38 -0.08 -0.30 m 0.16 m -0.12 -0.42 m m
New Zealand 0.02 -0.21 -0.07 0.03 0.28 m 0.20 m 0.20 0.64 m -0.04
Norway m -0.02 0.00 0.16 0.26 -0.02 -0.07 0.35 0.29 -0.12 -0.44 -0.10
Poland m 0.05 -0.42 -0.09 0.09 m m m m 0.26 m m
Portugal 0.11 0.00 -0.21 0.22 0.32 m m m m -0.04 m m
Slovak Republic m m -0.84 -0.14 -0.59 m m m m 0.72 m -0.05
Slovenia m m -0.28 0.09 0.10 -0.45 -0.39 0.03 0.10 0.01 -0.38 -0.08
Spain m 0.16 -0.24 0.22 0.04 m m m m -0.12 m m
Basque country m m m m m 0.20 m 0.10 m m -0.18 m
Sweden m 0.02 -0.03 -0.22 -0.07 0.11 m 0.20 m -0.10 -0.21 0.06
Switzerland m -0.11 -0.07 0.05 -0.15 m m m m -0.17 m m

Table 16.1 Effect sizes for changes in school external relations practices (continued)
Change in information provision to parents Change in parents involvement Change in public relations
Principals time to communi-

Parental perceptions on pro-

vision of regular information
serve on school committees
School asking parents to be

students in the same grade

relative to national/regional
Schools asking parents to
Public relations and fund-

volunteer for school proj-
relative to other students
ects, programs and trips
Achievement data to be
As a group relative to
raising by principals
ty/parent relations
in other schools
posted publicly
benchmarks
8th grade 8th grade 4th grade 8th grade 4th grade 8th grade 4th grade
06-09 06-09 06-09 06-09 06-09 03-07 03-07 03-07 03-07 06-09 03-07 01-06
Turkey m m 0.09 0.01 -0.07 m m m m 0.29 m m

United Kingdom m 0.07 -0.39 -0.32 -0.15 m m m m -0.41 m m
England m m m m m m m m m m 0.10 0.04
Scotland m m m m m 0.17 0.34 -0.10 0.16 m -0.18 -0.05
United States m 0.66 -0.24 0.00 -0.11 0.38 0.19 0.31 0.08 -0.04 -0.34 -0.09
OECD (average) 0.04 0.01 -0.16 -0.01 -0.05 0.04 0.06 0.05 0.07 -0.05 -0.09 -0.04
OECD (average absolute) 0.11 0.06 0.22 0.17 0.16 0.23 0.20 0.16 0.14 0.20 0.22 0.06
Hong Kong, China 0.05 m -0.23 -0.15 0.05 0.27 0.22 -0.17 -0.16 -0.17 -0.04 -0.08
Indonesia m -0.05 -0.31 -0.61 -0.25 0.31 m -0.38 m 0.42 0.08 m
Russian Federation m 0.11 -0.55 0.07 -0.20 0.27 0.23 0.22 0.18 0.09 0.03 -0.05
Singapore m m m m m -0.02 0.21 0.49 0.24 m -0.01 -0.07
1 2 http://dx.doi.org/10.1787/888933086527
Source: Authors calculations based on TIMSS (2003 and 2007), PIRLS (2001 and 2006) and PISA (2002, 2006 and 2009)

CHAPTER 17
Composite indices of innovation

in classrooms and schools
Combining information about the extent to which school and classroom

practices have changed provides important insights into the extent and focus
of innovation within education in different education systems.
An education system may be widely innovative, changing many practices at
different levels and across subjects, or it may focus on certain aspects more
than others. A focus on school change rather than classroom change may
indicate innovations designed to improve whole school results, whilst those
education systems with more innovation at 8th grade than 4th grade may be
seeking innovations that improve higher education options and labour market
opportunities for students. Innovation activities that focus on one subject
over another may be designed to address identified weaknesses or to build on
perceived strengths within the wider economy, for example.

17. COMPOSITE INDICES OF INNOVATION IN CLASSROOMS AND SCHOOLS
Composite innovation indices

The wide range of changes identified within an education system creates the opportunity to
generate composite indices of innovation for 28 of the education systems discussed. These indices
can be used to investigate which education systems are more or less innovative in terms of school
or classroom practices, and can also show whether innovation more typically occurs in a particular
grade or within a certain subject.
Aggregating information at both school and classroom level to provide an overall index identifies
13 education systems that are above the OECD mean (22 points) in terms of the extent of change
across school and classroom practices between 2000 and 2011 (Figure17.1). These include Denmark
(37 points), Indonesia (36 points), Korea (32 points) and the Netherlands (30 points). At the other
end of the range are New Zealand (17 points), Austria (16 points), the Czech Republic (15 points) and
Massachusetts (14 points).
There is some variation in the extent to which the change occurs at classroom or school level
(Figure17.2). Denmark, Hungary, Israel, Korea, the Russian Federation and Slovenia stand out with
scores above the relevant OECD mean at both school (23 points) and classroom level (22 points),
suggesting that their education systems are characterised by innovation at both levels. In contrast,
Australia, Austria, the Czech Republic, Germany, Italy, Japan, Minnesota, Massachusetts, Sweden and
the United States, and exhibit scores below the OECD average across the two indicators.
In the case of England (33 points), Hong Kong (32 points) and Quebec (26 points) change occurs
primarily in school practices. It is also noteworthy that the four education systems at the lower end
of the range in terms of overall innovation have more innovation at the school level than within the
classroom. In contrast, in most of the top ranking education systems in terms of overall innovation,
Denmark (40 points), Indonesia (42 points), Korea (36 points), the Russian Federation (31 points) and
Hungary (28 points), innovation is most apparent with regard to classroom practices.
Subject level composite innovation indices highlight the high level of innovation in the Russian
Federation, in relation to both science (49 points) and maths (41 points) (Figure17.3). In comparison,
the OECD means for both maths and science are 24 points. Japan also stands out as having relatively
high level of innovation in maths (28 points) and science (29 points), despite below average overall
levels of innovation. In addition, innovation is above the means for both maths and science in
Indonesia, Korea, the Netherlands, Israel, England, and Norway. Slovenia (29 points) and New
Zealand (26 points) stand out as education systems with above average innovation in maths, but
below in science (Slovenia 22 points and New Zealand 16 points). Conversely, Singapore (32 points),
Hungary (31 points) and Hong Kong (27 points) have above average levels of innovation in science
but below in maths.
The composite indices show small differences in levels of innovation at 4th grade and 8th grade
in many education systems (Figure17.4), and the majority of systems are either above or below the
relevant OECD mean on both indices. Denmark and Indonesia rank the highest at both grade levels,
just as they rank highest overall. Only England stands above the OECD mean for innovation in the
8th grade (24 points compared with an OECD mean of 22 points) but below for innovation at 4th grade
(19 points compared with an OECD mean 23 points). Similarly Quebec is the only education system
to have a score above the OECD mean at 4th grade (24 points) but below at 8th grade (16 points).

Figure17.1 Overall composite education innovation index, 2000-2011
Overall Innovation
%
40
35
30
25
20
15
10
5
0
a a s y a l a e y y a e y y c a s a a c s
ar esi ore nd tion gar hin srae eni por landrwa tario ean rke apan ian Chil eden Ital an ebe trali tate sot land stri ubli sett
k
e nm don K erladera Hun g, C I lov ga ng o
S n E N O n D m Tu J Ind S w e rm Qu us d S nne Zea Au Rep chu
D In th e on Si C G A ite Mi w h a
Neian F gK OE Un Ne ec ass
ss on Cz M
R u H
1 2 http://dx.doi.org/10.1787/888933086546
Figure17.2 School and classroom level composite innovation indices, 2000-2011
Classroom change School change

%
45
40
35
30
25
20
15
10
5
0
k
nm on K erladera Hun g, C I lov ga ng o
N O n D m Tu J Ind w e rm Qu us d S nne Zea Au Rep chu
De Ind t h e on
S S i n E C S G A e i
it M ew h a
Neian F gK OE Un N ec ass
ss on Cz M
R u H
1 2 http://dx.doi.org/10.1787/888933086565
Figure17.3 Subject-level composite innovation indices for maths and science education, 2000-2011
Mathematics Science
%
60
50
40
30
20
10
0
k
e nm don K erladera Hun g, C I lov ga ng o
S Sin E N On CD m T
u J Ind
Sw
e a
erm Qu Aus ed S inn Ze Au Repachu
D In t h e n G t M ew h
Neian F Ko OE Un
i
N ec ass
ss ng Cz M
Ru Ho
1 2 http://dx.doi.org/10.1787/888933086584

Figure17.4 Composite innovation indices for 8th grade and 4th grade, 2000-2011
8th grade 4th grade

%
40
35
30
25
20
15
10
5
0
k
nm on K erladera Hun g, C I lov ga ng o
N On CD m T
u J Ind
Sw
e a
erm Qu Aus ed S inn Ze Au Repachu
De Ind t h e n S Sin E G t M ew h
Neian F Ko OE Un
i
N ec ass
ss ng Cz M
Ru Ho
1 2 http://dx.doi.org/10.1787/888933086603

The composite innovation indices are based on average absolute effect sizes of changes reported elsewhere
in this chapter. A large value on the index shows that changes have occurred across different aspects of that
education system, whether reductions or increases in a particular practice. Each set of indices, including the
overall indices is calculated separately; it is not possible to sum two or more sets of indices to replicate this
overall measure.
Note on Israel

Table 17.1 Overall composite education innovation index, 2000-2011
Overall Classroom School

Countries Mathematics Science 8th grade 4th grade
Innovation change change
Denmark 36.93 40.09 32.20 38.43 38.90 37.30 35.33

Indonesia 36.43 41.94 28.16 33.60 39.96 36.02
Korea 31.57 33.58 28.55 35.31 35.91 31.45
Netherlands 30.00 30.28 29.59 31.09 36.43 30.00 30.62
Russian Federation 29.05 30.62 26.69 41.42 48.76 37.21 32.31
Hungary 27.40 28.01 26.48 22.74 31.36 27.74 24.82
Hong Kong, China 26.70 22.20 31.81 22.11 26.79 28.26 26.27
Israel 25.97 26.68 24.90 35.85 32.67 28.34 28.96
Slovenia 25.44 23.34 28.58 29.09 22.14 25.05 33.07
Singapore 24.55 26.69 18.62 22.28 31.86 26.89 27.43
England 24.14 21.70 33.07 26.69 26.83 23.74 19.63
Norway 23.56 24.65 21.91 28.25 28.49 24.65 32.81
Ontario 22.99 21.97 25.82 20.95 22.77 21.74 21.11
OECD mean 22.31 22.10 23.12 23.92 24.26 22.23 23.04
Turkey 21.41 22.66 19.53 20.40 20.85 19.82
Japan 20.53 21.83 18.58 28.46 29.62 24.30 24.91
Indiana 20.33 23.04 12.82 19.32 22.10 20.53
Chile 19.98 18.34 22.45 20.63 22.80 19.68
Sweden 19.65 21.44 16.97 19.12 23.98 17.15 20.88
Italy 18.97 21.63 14.98 15.62 20.68 15.99 18.50
Germany 18.52 17.82 19.57 16.28 16.26 18.21 14.81
Quebec 18.31 15.45 26.26 15.59 20.35 16.10 24.35
Australia 18.13 19.46 16.15 21.86 17.73 17.06 21.39
United States 17.24 16.30 18.65 16.64 14.08 16.24 15.80
Minnesota 17.18 18.16 14.43 17.24 15.90 16.18
New Zealand 17.09 12.97 23.27 25.57 15.68 19.85 18.61
Austria 15.58 12.61 20.05 17.91 16.61 16.74 17.21
Czech Republic 15.13 13.22 17.98 15.78 15.27 15.76 11.86
Massachusetts 13.89 12.35 18.15 16.07 16.86 16.19
1 2 http://dx.doi.org/10.1787/888933086622

PART III
Change in educational outcomes

CHAPTER 18
Innovation and educational outcomes
Measures of innovation can be used to better understand how innovation

relates to educational outcomes. The extent of innovation may be associated
with various types of educational outcome, including learning outcomes,
equality and equity. It may also be related to expenditure trends.
A positive association between innovation and specific outcomes might occur
if the innovation led to some improvements at the classroom or school level.
This could also be implied through a positive association with innovation and
changes in outcomes across time. Innovation may also be positively associated
with equity, particularly if innovation occurs to tackle previous inequalities or
to drive improvements across the whole school. Conversely, certain outcomes
my lead to more innovation at classroom or school level, either because they
increase the freedom or the pressure to innovate.
If no association between innovation and outcomes is observed, it may be that
innovations are expected to have an impact over the longer term or that other
confounding factors have prevented an improvement in outcomes. Indeed, it is
possible that innovation was necessary in order to arrest a potential fall in outcomes
caused by issues such as budget cuts or staff shortages. Alternatively, it is possible
that these innovations were not intended to target the outcomes analysed.
This chapter explores some of the complex relationships between innovation
and outcomes in education and considers plausible explanations for the patterns
found. More regular assessment would help to build on this knowledge to better
understand the driving factors behind relationships observed.

18. INNOVATION AND EDUCATIONAL OUTCOMES
Composite innovation indices and learning outcomes
Innovation and test scores

Are education system level changes in classroom and school practices associated with outcome
measure taken from TIMSS, PIRLS and PISA studies, using the most recent outcomes available,
or with change in outcomes across time? In other words, is there a link between innovation and
learning outcomes?
The relationship between innovation and learning outcomes across education systems might
be driven by two opposing factors. On the one hand, systems with low levels of performance may
innovate in order to improve this important outcome. On the other hand, education systems with
high levels of performance may exhibit continuous innovation as a means of keeping performance
levels high. In other words, high levels of innovation can be associated both with high and low levels
of performance. The relationship may not be causal, and at this stage in our knowledge, it should
not be concluded that innovation results in a particular outcome.
Innovation and TIMSS 8th grade mathematics score

Higher levels of innovation are associated with higher maths scores (Figure18.1 to Figure18.3).
Whilst there is considerable variation, education systems that have experienced more change in
school and classroom practice, as indicated by a higher position on the overall innovation index,
typically have higher maths scores than those with a lower position.
Education systems in the top right quadrant of Figure18.1 (including Korea, the Russian
Federation, Hungary and Hong Kong) have innovation scores that are above the OECD mean, and
above average maths scores on TIMSS. These systems have innovated to keep their scores high,
or may have achieved high scores as a result of the innovations that have occurred. Conversely,
education systems in the bottom left quadrant (such as Sweden, Chile and Turkey) have lower than
average maths scores and low levels of innovation.
However, there are also a number of education systems with high outcomes and relatively low
levels of innovation (top left quadrant); these systems may have practices in place that they do not
intend to change while performance is high.
Figure18.2 and Figure18.3 provide additional detail about how the outcomes are associated with
school and classroom practices. The association between maths scores and classroom innovation
is higher than that of school innovation, and whilst the data does not show causation, this pattern
is consistent with the argument that changes in classroom practices may be more likely to lead to
better performance than school level changes.
The top two quadrants of Figure18.2 and Figure18.3 show that there are many education
systems with above average maths scores across all levels of school innovation, whilst the highest
scoring systems have above average innovation at the classroom level.

Figure18.1 Overall education innovation and 8th grade mathematics outcomes
TIMSS 8th grade

maths score - 2011
650
SGP KOR
600 HKG
JPN
US-MA
US-MN RUS
550 CA-QC
US-IN CA-ON
USA DEU ISR
ENG HUN
TIMSS mean
500
AUS ITA NOR SVN
NZL SWE TUR
450
CHL
OECD mean
400
10 15 20 25 30 35
Overall innovation
1 2 http://dx.doi.org/10.1787/888933086641
Figure18.2 School innovation and 8th grade mathematics outcomes

TIMSS 8th grade
maths score - 2011
650
SGP KOR
600 HKG
JPN
US-MA
US-MN
550 RUS
CA-QC
US-IN ISR
USA DEU ENG
HUN SVN
500 AUS SWE CA-ON
ITA NOR TIMSS mean
NZL
TUR
450
CHL
OECD mean
400
10 15 20 25 30 35
Innovation in schools
1 2 http://dx.doi.org/10.1787/888933086660

Figure18.3 Classroom innovation and 8th grade mathematics outcomes
TIMSS 8th grade

maths score - 2011
650
SGP KOR
600
HKG
JPN
US-MA
US-MN RUS
550 CA-QC
US-IN
DEU CA-ON ISR
USA AUS SVN HUN
ENG
500 ITA
NOR TIMSS mean
NZL SWE
TUR
450
CHL
OECD mean
400
10 15 20 25 30 35
Innovation in classrooms
1 2 http://dx.doi.org/10.1787/888933086679
The vertical lines on Figures in this section indicate the OECD mean for innovation, whilst the horizontal line indicates the mean value for the
outcome, as reported in the original data
Box 18.1. How to read the graphs

The graphs used in this section show the association between the selected innovation and
outcome variables. The innovation score is usually represented on the x axis, while the outcome
variable is represented on the y axis. The vertical line dividing the quadrants represents the
innovation score OECD mean, whereas the horizontal line represents the average of the selected
measure taken from the original source. The trend line represents the line of best fit between the
two variables..
Box 18.2. Data source details for Figures 18.1 to 18.6

In this chapter, Innovation is measured using various composite innovation indices as presented
in Chapter 18. An overall indicator gathers information about educational change across grades
and subjects at school and classroom level, whilst two separate indices look at innovation in school
and classrooms separately. Innovation at 8th grade combines innovation in classrooms or schools
with 8th grade students (using data from TIMSS), and schools with 15-year-olds students (using
data from PISA). A 4th grade innovation index combines information from classrooms or schools
that have 4th grade students, while the mathematics innovation index combines innovation in 8th
or 4th grade in classrooms or schools that relates to mathematics practices.
The outcome measure used in the above figures is 2011 8th grade mathematics results from
TIMSS. 8th grade mathematics results are available for 23 of the 28 education systems for
which innovation scores have been calculated, excluding Austria, the Czech Republic, Denmark,
Indonesia and the Netherlands.

Innovation in mathematics education and mathematics outcomes

It is plausible that maths score outcomes would be most directly associated with innovation in
mathematics. This is explored in Figure18.4, which shows that there is some association between
innovation in maths (across 8th and 4th grades) and maths performance, but that it is not the strongest
of the associations explored. A possible explanation could be that some organisational, non-subject-
specific factors might also have an indirect impact on performance. In particular, there are many
education systems with above average maths scores that have not been particularly innovative in
maths, including high scoring education systems such as Singapore and Hong Kong.
Figure18.5 investigates the association between 8th grade maths performance and innovation
occurring across 8th grade school and classroom practices. The association is similar to that between
maths scores and the overall indicator of innovation. The education systems in the top right
quadrant are characterised by high levels of innovation at 8th grade and correspondingly high maths
scores. This may reflect the extent to which secondary schools and classroom teachers are working
at maintaining or improving outcomes for their students in this core subject. There are several
education systems with above average maths scores that are not particularly innovative in the 8th
grade, but those with the highest performance (Korea, Singapore, Hong Kong and Japan) are all
above the OECD mean in terms of innovation at 8th grade.
There is also a positive association between 4th grade maths performance and innovation at
school and classroom level in 4th grade. The highest scoring education systems at 4th grade maths
(Singapore, Korea, Hong Kong and Japan) were all above average in terms of innovation at 4th grade.

Figure18.4 Innovation in mathematics and 8th grade mathematics outcomes

TIMSS 8th grade
maths score - 2011
650
SGP KOR
600 HKG
JPN
US-MA
US-MN RUS
550 CA-QC
US-IN ISR
DEU CA-ON ENG SVN
USA
500 SWE AUS HUN
ITA NOR TIMSS mean
NZL
TUR
450
CHL
OECD mean
400
10 15 20 25 30 35 40 45
Innovation in maths
1 2 http://dx.doi.org/10.1787/888933086698
Figure18.5 Innovation at 8th grade and 8th grade mathematics outcomes

TIMSS 8th grade
maths score - 2011
650
SGP KOR
600 HKG
JPN
US-MA
550 US-MIN RUS
CA-QC US-IN
ISR
USA AUS DEU ENG SVN HUN
CA-ON
500 SWE
ITA NZL NOR TIMSS mean
TUR
450
CHL
OECD mean
400
10 15 20 25 30 35 40
Innovation in 8th grade
1 2 http://dx.doi.org/10.1787/888933086717
Figure18.6 Innovation at 4th grade and 4th grade mathematics outcomes

TIMSS 8th grade
maths score - 2011
650
625
HKG SGP
600
JPN
575
550 USA ENG NLD RUS DNK

CA-QC
DEU
525 CZE CA-ON AUS HUN SVN
AUT ITA
SWE NOR
500
NZL TIMSS mean
OECD mean
475
10 15 20 25 30 35 40
1 2 http://dx.doi.org/10.1787/888933086736

Innovation and TIMSS 8th grade mathematics trends

Trend data enable analysis of the association between innovation and improvement in
performance over time. The following Figures indicate that there is a positive association between
overall innovation and trends in 8th grade mathematics outcomes between 2003 and 2011 (Figure18.7).
Some education systems, and particularly the Russian Federation and Korea, have exhibited
large improvements in maths grades and correspondingly large amounts of change in school
and classroom practices. However, positive change has also occurred without such high levels
of innovation: Chile is an example of a country that has improved performance whilst levels of
innovation remain below the OECD mean.
Two high performing education systems in terms of maths scores, Hungary and Hong Kong
provide examples of education systems that have not experienced a positive trend in maths at 8th
grade but have introduced many changes. This may indicate that the marginal return to innovation
has been diminishing. Ontario is also characterised by above average levels of innovation and
negative trends in maths performance.
There is no association between school level innovation and maths trends, with education
systems scattered across the four quadrants (Figure18.8). Of particular note is the right hand side of
Figure18.8, which shows that whilst several innovative systems (at the school level) have achieved
positive change in their maths performance, slightly more have experienced negative trends. A
more detailed analysis of these patterns would be necessary to comprehend the reasons but it may
be that some school level changes are focused on improving test scores whilst others have other
purposes.
The analysis does suggest a positive association between changes in classroom practice and
maths trends (Figure18.9). However, this result appears to be driven by the significant improvements
in the Russian Federation and Korea and their innovative attitude in classroom practices, while of
the education systems analysed in this section most are at or below the OECD mean, making it
difficult to draw.
Figure18.7 Overall education innovation and 8th grade mathematics outcome trends
Trend 2003-2011 TIMSS 8th grade
maths score
40
RUS
CHL
30
KOR
20 ITA
US-MA US-IN NOR
US-MIN SVN
ENG
10 USA
JPN SGP HKG
0 NZL AUS TIMSS mean
CA-ON
CA-QC
-10 SWE
-20 HUN
OECD mean
-30
10 15 20 25 30 35
Overall innovation
1 2 http://dx.doi.org/10.1787/888933086755

Figure18.8 School innovation and 8th grade mathematics outcome trends

maths score
40
RUS
CHL
30 KOR
20
US-IN US-MIN US-MA NOR
ITA SVN
ENG
10 SGP
USA HKG
AUS JPN
0
NZL TIMSS mean
CA-ON
CA-QC
-10 SWE
-20 HUN
OECD mean
-30
10 15 20 25 30 35
1 2 http://dx.doi.org/10.1787/888933086774
Figure18.9 Classroom innovation and 8th grade mathematics outcome trends

maths change in scores
40
RUS
CHL
30
KOR
20 US-MA ITA US-IN

US-MIN NOR
10 ENG SVN
USA
SGP
AUS JPN HKG
0
NZL TIMSS mean
CA-QC CA-ON
-10 SWE
-20 HUN
OECD mean
-30
1 2 http://dx.doi.org/10.1787/888933086793

Innovation and TIMSS 8th grade science trends

In contrast with the associations found between innovation and maths trends, a weak negative
correlation is apparent between innovation and science trends (Figure18.10).
Whilst there is no strong pattern, the association is driven in particular by Indonesia, Hong
Kong and Hungary, as they have exhibited large decreases in science performance despite a large
amount of change in school and classroom practices. This may indicate that the innovations were
not relevant or appropriate for science performance. However, negative trends in science have also
occurred without such high levels of innovation: Sweden and New Zealand are examples of countries
where science performance decreased whilst levels of innovation remain below the OECD mean.
The Russian Federation and Slovenia provide examples of counties that have introduced many
changes and experienced a positive trend in science at 8th grade. Singapore is also characterised by
above average levels of innovation and positive trends in science performance.
Figure18.11 shows a weak negative association between school level innovation and science
trends. England and Hong Kong stands out as education systems with particularly high levels of
change in school practices and substantial decrease in science performance, in contrast with the
Russian Federation and Slovenia where science performance increased alongside innovation.
The analysis also finds a weak negative correlation between changes in classroom practice and
science trends showing that more innovation at the school level is associated with a reduction in
science performance over time (Figure18.12). However, again this result appears to be influenced by
the significant decrease in science performance in Indonesia combined with its particular innovative
strength in classroom practices that alone drives the direction of the correlation. The other education
systems analysed seems to scatter around the quadrants with many of them lying very close to the OECD
average, making it difficult to establish a clear direction of the correlation between the two variables.
The difference in the associations between innovation and changes in maths and science scores
could come from the fact that more innovation is required to improve some subjects than others, or
that innovation is triggered by falling trends in some subjects. It could also be related to the extent
to which change might be expected to improve performance; high performing education systems
are unlikely to see very positive trends in grades over time as they are starting from near the top.
Innovation may also be focused on aspects of education beyond subject level performance, such as
equity or cost efficiency.

Figure18.10 Overall education innovation and 8th grade science outcome trends

science change in scores
50 CHL
40
RUS
30 SVN
20 US-MIN
US-MA ITA SGP
10 JPN
NOR KOR TIMSS mean
USA
0 US-IN
AUS
ENG
-10 NZL SWE
CA-QC
CA-ON HKG HUN IDN
-20
OECD mean
-30
10 15 20 25 30 35 40
Overall innovation
1 2 http://dx.doi.org/10.1787/888933086812
Figure18.11 School innovation and 8th grade science outcome trends

50
CHL
40
RUS
30
SVN
20
US-MIN
ITA US-MA SGP
10 JPN
US-IN NOR KOR
USA
TIMSS mean
0
AUS NZL
CA-ON CA-QC
-10 SWE
ENG
HUN IDN HKG
-20
OECD mean
-30
10 15 20 25 30 35
1 2 http://dx.doi.org/10.1787/888933086831

Figure18.12 Classroom innovation and 8th grade science outcome trends

50
CHL
40
RUS
30 SVN
20 US-MIN
US-MA ITA SGP
10
JPN US-IN KOR
USA NOR TIMSS mean
0
NZL AUS
CA-QC ENG
-10
CA-ON
SWE HUN IDN
HKG
-20
OECD mean
-30
10 15 20 25 30 35 40 45
1 2 http://dx.doi.org/10.1787/888933086850

TIMSS trend data for 8th grade maths is available for 20 of the 28 education systems for which
we have innovation scores, excluding Austria, the Czech Republic, Denmark, Germany, Israel,
Indonesia, the Netherlands, and Turkey. Trends are based on 8 education systems with a negative
value indicating a reduction and 12 with a positive value indicating an increase in outcomes.
TIMSS trend data for 8th grade science is available for 21 of the 28 education systems for which
we have innovation scores, excluding Austria, the Czech Republic, Denmark, Germany, Israel,
the Netherlands, and Turkey. Trends are based on 11 education systems with a negative value
indicating a reduction and 10 with a positive value indicating an increase in outcomes.

Composite innovation indices, equality and equity

One of the desirable outcomes of systemic change or innovation in education is an increase in
equality. Educational equality can include equality in learning outcomes and learning opportunities.
The analysis in this section is based on PISA equality measures. Equality in learning outcomes
has been measured by analysing the performance dispersion of students in reading, while the
equity of learning opportunities can be measured by the percentage of students who come from a
disadvantaged socio-economic background and perform much higher than would be predicted by
their background, as discussed in OECD PISA 2009 Results: Overcoming Social Background- VolumeII.
Innovation and equality in learning outcomes

The association between innovation and the dispersion in learning outcomes is analysed using
the reading scores from PISA 2009 by measuring the distance between 10th and 90th percentile of the
distribution of learning outcomes.
There is a positive association between innovation and equality in learning outcomes;

education systems that have innovated the most are also the most equitable in terms of students
learning outcomes (Figure18.13 to Figure18.15). This is especially the case for Indonesia and Korea.
Furthermore, very few education systems fall into the bottom right quadrant in other words there
are few systems that are innovative but not equal. However, it is important to note that equality in
Chile and Turkey is slightly above countries such as Denmark, which have high innovation scores,
and Israel has considerably lower reading equality than Hong Kong despite being well above the
OECD mean in terms of overall innovation, indicating that there is some variability.
The general, positive pattern holds when looking at school or classroom level innovation separately.
There are however some exceptions, with Turkey and Norway standing out as countries with high
equality and low levels of innovation at the school level, and Chile standing out for its high levels of
equality and low levels of innovation at classroom level. On the other hand, Israel and New Zealand
have innovated at the school level and display low levels of equality, whilst Israel and Singapore exhibit
low levels of equity alongside above average levels of innovation at the classroom level.
When the analysis is restricted to 8th grade innovation the positive association with equality still
holds, although there is less correlation. Of particular note, many education systems are clustered
in the bottom left hand quadrant of the Figure; these exhibit very similar levels of equality and
innovation, with both being below average.
A possible explanation for the lower correlation between innovation at 8th grade rather than
overall innovation centres on the fact that the equality measure focuses specifically on reading. It
might be that reading performance is particularly influenced by changes in primary education than
secondary, in which case innovation at the 8th grade is not fully capturing the potential relationship.
The general positive trend between innovation and equality of learning outcomes could result
from a situation where education systems with higher level of equality in learning outcomes feel
more secure and empowered to change their practices. In turn, it is possible that those with low
equality prioritise some other educational objectives rather than innovating in order to promote
equality. The assumption that more innovation could lead to higher levels of equality is plausible,
though not proven: a high level of equality in learning outcomes may require responsiveness to
students needs, and therefore higher levels of change.

Figure18.13 Overall education innovation and students performance variation in reading
2009 PISA reading performance

equity
140
IDN
120
KOR
100
CHL TUR HKG DNK
80
NOR HUN NLD
CZE ITA SVN RUS OECD mean
60
USA DEU JPN ENG
SWE SGP
40 NZL AUS
AUT
20 ISR
OECD mean
0
10 15 20 25 30 35 40
Overall innovation
1 2 http://dx.doi.org/10.1787/888933086869
Figure18.14 Schools innovation and students performance variation in reading

equity
140 IDN
120
KOR
100
TUR CHL HKG
DNK
80 RUS
NOR NLD
SVN
CZE HUN OECD mean
60 ITA AUSSWE ENG
JPNDEU
AUS
SGP SWE AUT
40 NZL
20 ISR
OECD mean
0
10 15 20 25 30 35
Innovation in school
1 2 http://dx.doi.org/10.1787/888933086888

Figure18.15 Classroom innovation and students performance variation in reading

equity
140 IDN
120
KOR
100
CHL TUR
HKG DNK
80 RUS
CZE SVN NOR NLD

ITA HUN
60
USA AUS SWE ENG SGP OECD mean
AUT DEU JPN
40 NZL
20 ISR
OECD mean
0
10 15 20 25 30 35 40 45
1 2 http://dx.doi.org/10.1787/888933086907
Figure18.16 Innovation at 8th grade and students performance variation in reading

equity
140
IDN
120
KOR
100
CHL TUR HKG DNK
80 NLD RUS
HUN
NOR SVN
CZE
60 ITA SWE DEU ENG JPN SGP OECD mean
USA
AUT AUS NZL
40
20 ISR
OECD mean
0
10 15 20 25 30 35 40
1 2 http://dx.doi.org/10.1787/888933086926
* PISA reading performance variation is calculated across the whole United Kingdom.

Innovation and equity of learning opportunities

Equity of learning opportunities can be assessed through the proportion of disadvantaged
students that show high levels of performance. The correlations between innovation and equity
of learning opportunities are weak (Figure18.17). In particular, across the whole distribution
of innovation scores there are many education systems with poor equity in terms of learning
opportunities. This is in noticeable contrast to the reading performance equality measure described
above. However, the three most equitable education systems in this regard, Hong Kong, Korea and
Singapore, are also above average in terms of overall innovation.
It may be that the societal factors that lead to underperformance amongst disadvantaged
students in some education systems have a much greater impact than the changes observed in
schools and classrooms.
The plots shown in Figure18.18 indicate that the two most equitable education systems, Hong
Kong and Korea are particularly innovative at the school level whilst the third, Singapore, is below
the OECD mean in terms of school level innovation. At the classroom level, Korea and Singapore
both have high levels of innovation and equity, whilst the most equitable education system, Hong
Kong, sits on the OECD mean in terms of classroom innovation (Figure18.19). This indicates that
amongst these education systems changes at either the school or classroom level are associated
with equity in opportunities.
Singapore, Japan and Turkey also stand out as being relatively equitable, with lower than average
levels of innovation. Meanwhile, England and Denmark are noticeable in that they have below
average levels of equity alongside high levels of innovation at the school level.
Denmark and Indonesia stand out for having below average levels of equity in learning
opportunities and high levels of innovation at the classroom level. Given that Denmark is also
innovative at the school level, it may be a country that is making efforts to improve equity by
changing practices across the board.

PISA data on equality in reading performance and on equity of learning opportunities are
available for 23 of the 28 education systems for which we have innovation scores, excluding
Indiana, Massachusetts, Minnesota, Ontario and Quebec.
The reported measure for equality in reading performance is based on the distance between the
10th and the 90th percentile for the PISA reading assessment in 2009 for each education system.
Each equity measure is obtained by subtracting each inter-percentile distance from the maximum
distance found in the sample. The reported measure for equity of learning opportunity corresponds
to the percentage of students described as resilient among disadvantaged students in PISA 2009
Results: Overcoming Social Background, equity in learning opportunities and outcomes, volume II.
The reported analyses focus on the potential correlation between equality in reading performance
and equity of learning opportunities across three innovation scores (overall, school and classroom).

Figure18.17 Overall education innovation and equity of learning opportunities
2009 PISA equity of learning

opportunities
80
HKG
70
60 KOR
SGP
50
JPN TUR
40 NZL
NLD
AUS ITA TIMSS mean
30 SWE NOR SVN HUN
USA DEU CHL IDN DNK
CZE
AUT ENG ISR
20 RUS
OECD mean
10
10 15 20 25 30 35 40
Overall innovation
1 2 http://dx.doi.org/10.1787/888933086945
Figure18.18 School innovation and equity of learning opportunities

opportunities
80
HKG
70
60
KOR
50 SGP
JPN TUR
40 NZL
ITA NLD
AUS USA TIMSS mean
30 SWE
DEU NOR ISR HUN IDN SVN DNK
CZE ENG
AUT CHL RUS
20
OECD mean
10
10 15 20 25 30 35
1 2 http://dx.doi.org/10.1787/888933086964

Figure18.19 Classroom innovation and equity of learning opportunities

opportunities
80
HKG
70
60 KOR
50 SGP
JPN TUR
40 NZL
ITA NLD
AUS
USA OECD mean
30 SWE SVN HUN
CZE DEU CHL ISR DNK IDN
AUT ENG NOR RUS
20
OECD mean
10
10 15 20 25 30 35 40 45
1 2 http://dx.doi.org/10.1787/888933086983

Innovation and variation in outcomes within and across schools

An education system may have more or less variation in outcomes within schools, depending
on whether the school and education system works to improve the weakest students at the school
or system level, and whether they segregate or stream students according to predicted ability. A low
measure of variation across schools may indicate a more equitable system, whilst a low measure
within schools may be more or less equitable depending on the approach used. There is some
correlation between variation within and across schools and innovation, with more innovative
education systems showing less variation (Figure18.19 and Figure18.20).
The three education systems with the lowest performance variation across schools, Norway,
Denmark and Indonesia, also have above average levels of overall innovation. At the other end of the
distribution, two countries with most variation across schools are not particularly innovative (Italy
and Turkey). In contrast, Israel also has a large amount of variation in outcomes across schools, and
is above the OECD average for overall innovation.
Education systems in the top right quadrant of Figure18.20 are characterised by similar
performance levels within schools and high levels of innovation. In particular, Indonesia, the
Netherlands, Hungary and Sweden stand out for having relatively little performance variation at
the school level, and overall innovation above the OECD mean. At the other extreme, Sweden, New
Zealand, Austria and the US have rather high levels of within school variation, and lower levels of
innovation.
Denmark and Indonesia are noteworthy because they are both innovative but whilst Denmark
has a small difference in outcomes across schools, it has a larger than average difference within
schools. In contrast, Indonesia has very little variation either in or between schools, possibly
reflecting lower outcomes overall.

Figure18.20 Overall education innovation and variation in equity between schools
2009 PISA equity

between schools
NOR
90 DNK
80 IDN
SWE RUS
70 NZL AUS KOR

ENG
SGP
60 USA
CZE TIMSS mean

50
SVN
CHL JPN NLD
40
AUT
DEU HUN
30 ISR
TUR
ITA
20
OECD mean
10
10 15 20 25 30 35 40
Overall Innovation
1 2 http://dx.doi.org/10.1787/888933087002
Figure18.21 Overall education innovation and variation in equity within schools
2009 PISA equity

within school
80
IDN
70 NLD
HUN
TUR SVN
60
DEU
ITA
50 CZE AUT CHL
40 JPN KOR
OECD mean
RUS
30 SGP
DNK
USA ENG ISR
20
AUS NOR
10
NZL
OECD mean
SWE
0
10 15 20 25 30 35 40
Overall Innovation
1 2 http://dx.doi.org/10.1787/888933087021

Composite innovation indices and changes in education expenditure

It might be possible to observe an association between overall innovation and change in
expenditure per student as a result of a number of factors, some of them opposing. If innovation
occurs in order to reduce expenditure or to overcome budget cuts one might expect to see a negative
correlation between innovation and change in expenditure. Equally, if education systems with very
low and falling expenditure per student feel obliged to take the chance on high risk innovations, there
might be a negative correlation. Conversely, a positive correlation between change in expenditure
and innovation might be interpreted as an investment in innovation or an indicator that schools
have the freedom of a more generous budget to spend on innovation.
Most of the education systems included in the analysis of innovation and expenditure have
increased their level of educational expenditure per student between 2000 and 2010 by similar
amounts, regardless of their level of overall innovation (Figure18.22). A positive but weak correlation
between innovation and change in expenditure is found. This may suggest that on average for
the education systems covered, innovation requires resources for its implementation. However,
the hypothesis that innovation actually leads to a decrease in expenditure and therefore to more
efficiency in the system cannot be supported as no country, except for Italy, has seen its educational
expenditure per student decrease in the analysed period despite the global financial crisis.
Among the education systems registering the largest increases in educational expenditure
per student, Korea and the Czech Republic exhibit two opposite behaviours regarding educational
innovation, where Korea is above average in terms of innovation the Czech Republic is near the
bottom of the innovation index. Koreas data may suggest that the educational innovation is
costly, although this hypothesis should be verified against the actual percentages of educational
expenditures dedicated to innovation. However, there is also evidence that some forms of innovation
can cost little or nothing: Denmark, for example has increased expenditure by less than average
whilst still exhibiting above average levels of change across the education system, suggesting that
those innovations may not have been as resource consuming.
The general positive pattern seems to hold when looking at school innovation (Figure18.23),
while the relationship between expenditure trends and classroom innovation shows no association
(Figure18.24). These patterns are consistent with the potential scale of the different changes, with
school change likely requiring more resources than classroom level changes.

Figure18.22 Overall education innovation and trend in expenditure per student
2011 TIMSS 8th grade

maths scores
120
KOR
100
CZE
80
60 AUS HUN
40 NLD OECD mean

JPN NOR ISR
AUT SWE
20 USA DNK
ITA
0
OECD mean
-20
10 15 20 25 30 35 40
Overall Innovation
1 2 http://dx.doi.org/10.1787/888933087040
Figure18.23 School innovation and trend in expenditure per student

maths scores
120
KOR
100
CZE
80
60
AUS HUN
40 OECD mean
USA NOR ISR NLD
20 SWE AUT DNK
JPN
ITA
0
OECD mean
-20
10 15 20 25 30 35
1 2 http://dx.doi.org/10.1787/888933087059

Figure18.24 Classroom Innovation and trend in expenditure per student

maths scores
120
KOR
100
CZE
80
60 AUS HUN
40
USA SWE JPN NOR ISR NLD OECD mean
AUT
20 DNK
ITA
0
OECD mean
-20
10 15 20 25 30 35 40 45
1 2 http://dx.doi.org/10.1787/888933087078

The reported analyses focus on the potential correlation between the change in expenditure
per student, and various composite innovation indices. The OECD data on expenditure trends is
an index of change in expenditure per student for all services at the primary, secondary and post-
secondary non-tertiary levels of education between 2000 and 2010 using data from Education at a
Glance 2013. This index takes 2005 as the baseline, and the change in expenditure is calculated as
a percentage of this value.

Composite innovation indices and satisfaction

Innovation in schools and classroom may lead to greater satisfaction for teachers by increasing
autonomy or the possibility to be adaptive or creative within the school setting. Students may also
become more satisfied if changes are designed to better meet their needs or to improve the dynamic
nature of their education. Alternatively change may be unsettling, leading to greater dissatisfaction
amongst teachers and students, particularly if they feel that the innovations are not in their control
or are detrimental to their work or studies.
Innovation and teacher satisfaction

Analysis of the relationship between overall innovation and changes in 8th grade maths teacher
satisfaction shows a strong positive correlation, indicating that teachers in more innovative
education systems have become more satisfied over time whilst satisfaction has fallen in less
innovative places (Figure18.25). Indonesia, Korea and the Russian Federation in particular have high
levels of change in education practices alongside high levels of improvement in teach satisfaction
over time.
Separate analyses of the correlation between teacher satisfaction and innovation at school
and classroom level indicate a particularly strong association between the improvement in maths
teacher satisfaction over time and the amount of innovation in the classroom (Figure18.26 and
Figure18.27). This is consistent with the argument that the teachers have a sense of autonomy
in the classroom and have made changes according to their needs and preferences. However, in
education systems such as Hong Kong and England where innovation is relatively high at the school
level but not at the classroom level, teacher satisfaction has also improved. This may indicate that
certain school level changes have a positive impact on the teachers in these systems.

Figure18.25 Overall education innovation and change in 8th grade maths teacher satisfaction
Trend 2003-2011 in 8th grade

mathematics teacher satistaction
40
KOR IDN
35
30
RUS
HKG
25 JPN SVN
ENG
20 US-MA NZL ITA US-IN SGP
NOR ISR
15
USA CA-QC OECD mean
10 CA-ON
US-MN
AUS TUR HUN
5
CHL
0
-5 SWE
-10 OECD mean
-15
10 15 20 25 30 35 40
Overall Innovation
1 2 http://dx.doi.org/10.1787/888933087097
Figure18.26 School innovation and change in 8th grade maths teacher satisfaction

mathmatics teacher satsfaction
40
IDN
KOR
35
30 RUS
HKG
25 JPN SVN
ENG
20 US-IN ITA
SGP NOR NZL ISR
US-MA
15 USA OECD mean
US-MN CA-QC
10 AUS
TUR CA-ON HUN
5
CHL
0
-5 SWE
-10 OECD mean

-15
10 15 20 25 30 35
1 2 http://dx.doi.org/10.1787/888933087116

Figure18.27 Classroom innovation and change in 8th grade maths teacher satisfaction

mathematics teacher satsfaction
40
IDN
KOR
35
30 RUS
HKG
25 JPN SVN
US-MA ENG
20 NZL ITA US-IN SGP
15 USA NOR ISR OECD mean
10 CA-QC
AUS
US-MN TUR HUN
5 CA-ON
CHL
0
-5 SWE
-10 OECD mean
-15
10 15 20 25 30 35 40 45
1 2 http://dx.doi.org/10.1787/888933087135

Innovation and student satisfaction

There is no correlation between overall innovation and the change in 8th grade student
satisfaction. Education systems are spread across high and low levels of innovation, and both
increases and decreases in student satisfaction.
There is, however, a positive correlation with student satisfaction when focusing specifically
on school level innovation. This may indicate that student satisfaction increases when schools
innovate, or possibly that schools have been innovating in order to raise student satisfaction. Indeed
those systems with the highest values on the school level innovation index also exhibit the largest
improvement in student satisfaction. Nevertheless, the relationship is not particularly strong, and
there are education systems in all four quadrants, indicating that some education systems are
innovating at the school level but satisfaction is falling (such as Hungary and Slovenia), and others
such as Japan have increased satisfaction without innovating at the school level.
In contrast with the findings for school level innovation, classroom level innovation appears to
have no association with student satisfaction, although this is in part driven by the slight reduction
in satisfaction amongst students in Indonesia. Israel and Korea stand out as countries that have
innovated at the classroom level and seen an improvement in student satisfaction, but other
education systems that have innovated in the classroom, such as Hungary and Turkey have not
been so successful at improving satisfaction amongst their students. As with other such patterns
observed, it may be that these systems are not seeking to improve this measure, either because it is
already sufficiently high, or because other goals take priority.

Figure18.28 Overall education innovation and change in 8th grade student satisfaction

students satsfaction
20
CA-QC ISR
15 HKG
10 JPN ENG
SWE
ITA KOR
AUS CA-ON OECD mean
5 NOR RUS
US-MA USA US-IN SGP IDN
NZL
SVN
0 US-MN
TUR
CHL
HUN
-5
OECD mean
-10
10 15 20 25 30 35 40
Overall Innovation
1 2 http://dx.doi.org/10.1787/888933087154
Figure18.29 School innovation and change in 8th grade student satisfaction

students satsfaction
20
ISR CA-QC
15 HKG
10 JPN ENG
SWE
ITA KOR
AUS CA-ON
NOR
5
USA NZL RUS IDN OECD mean
US-MN US-MA SGP
0 US-IN SVN
TUR
CHL
HUN
-5
OECD mean
-10
10 15 20 25 30 35
1 2 http://dx.doi.org/10.1787/888933087173

Figure18.30 Classroom innovation and change in 8th grade student satisfaction

students satisfaction
20
CA-QC ISR
15 HKG
10 ENG JPN
SWE KOR
AUS ITA OECD mean
5
USA CA-ON
US-MA US-IN SGP IDN
NZL US-MN NOR RUS
SVN
0
CHL TUR
HUN
-5
OECD mean
-10
10 15 20 25 30 35 40 45
1 2 http://dx.doi.org/10.1787/888933087192
Box 18.6.Data source details for Figures 18.25 to 18.30

TIMSS (2003, 2007 and 2011) surveys asked 8th grade teachers How much do you agree with
the following statements[] b) I am satisfied with being a teacher at this school with answer
options Agree a lot; Agree a little; Disagree a little; Disagree a lot. Students were asked What do
you think about your school? Tell how much you agree with these statements[] a) I like being in
school with answer options Agree a lot; Agree a little; Disagree a little; Disagree a lot. The same
data restrictions as in International TIMSS and PIRLS Reports by the International Association for
the Evaluation of Educational Achievement (IEA) apply to these data. Change in satisfaction is the
percentage point difference between 2003 and 2011.
Note on Israel
The statistical data for Israel are supplied by and under the responsibility of the relevant Israeli
authorities. The use of such data by the OECD is without prejudice to the status of the Golan Heights,
East Jerusalem and Israeli settlements in the West Bank under the terms of international law.

Table 18.1 Correlations between innovation indices in education and educational outcomes
Classroom Mathematics
Correlation coefficients Overall index School index 8th grade index 4th grade index
index index
TIMSS 8th grade maths score 0,3128 0,1309 0,3056 0,3903 0,2048
TIMSS 4th grade maths score 0,2430
TIMSS 8th grade maths changes in score 0,1646 0,0255 0,2305
TIMSS 8th grade science changes in score -0,1938 -0,2454 -0,1494
PISA reading performance equality 0,6143 0,4276 0,6078 0,4985
PISA equity of learning opportunities 0,1284 0,1384 0,0594
PISA performance variation between schools 0,3791
PISA performance variation within school 0,2699
Change in expenditure per student 00-10 0,1011 0,1821 0,0574
Trends in teacher satisfaction 8th grade maths 0,5914 0,4380 0,5306
Trends in student satisfaction 8th grade 0,0191 0,2311 -0,0559
1 2 http://dx.doi.org/10.1787/888933087211

Annex A
Data sources and methods
Annex A presents the different data sources, the coverage of the statistics
(target, countries, years), as well as sample sizes for the different data.

ANNEX A DATA SOURCES AND METHODS
Data sources: overview

This publication reports the results of secondary analysis of data from several sources.
The first chapter uses REFLEX (Research into Employment and professional Flexibility) and HEGESCO
(Higher Education as a Generator of Strategic Competences) data. REFLEX surveys higher education
graduates five years after their graduation and asks a range of questions about their workplace.
HEGESCO uses the same method and questionnaire, but applied to other European countries.
The remaining chapters report the analysis of data collected in surveys of students, teachers and
principals collected to inform assessments of student outcomes. These data are drawn from PISA
(Programme on International Student Assessment), TIMSS (Trends in International Mathematics
and Science Study) and PIRLS (Progress in International Reading Literacy Study).
PISA, TIMSS and PIRLS have been created to look at student achievements in maths and science
(PISA and TIMSS) and text understanding (PISA and PIRLS). Background questionnaires provide
relevant information about classroom or school practices which has been used to identify the extent
to which they have changed over time. All these surveys are cross-sectional.
Coverage of the statistics

REFLEX data cover recent higher education graduates working as highly skilled professionals.
The target population is graduates from ISCED 5A (2004 definition) who got their degree in the
academic year 1999/2000 (including foreign students, distance learners and those who subsequently
moved abroad, but excluding graduates from ISCED 6) and were surveyed in 2005. HEGESCO carried
out the survey in 2008. Percentages should be read as the percentage of recent higher education
graduates who report..
PISA is designed to assess learning outcomes of 15-year-old students and make comparisons
over time. The PISA assessment focuses on the extent to which students can apply the knowledge
and skills they have learned and practised at school when confronted with situations and challenges
for which that knowledge may be relevant.
PISA uses questionnaires to collect background information from students and data on various
aspects of organisation and educational provision in schools from school principals.
The target population of PISA is 15-year-old students in grade 7 or higher who attend education
institutions, including those enrolled part-time and those in vocational training programmes. It is
important to note that the sample is not designed to be representative of schools or classrooms and has
not been reweighted. Results should be read as the percentage of 15-year-old students who report...
TIMSS and PIRLS are designed to measure student achievement around the world and make comparisons
over time. TIMSS has two target populationsall students enrolled at the 4th grade and all students enrolled
at the 8th grade, although countries may choose to assess either or both student populations. Fourth and
eighth grade represent four and eight years of schooling respsectively, counting from the first year of ISCED
Level 1, providing the mean age at the time of testing is at least 9.5 years/13.5 years.
The target population for PIRLS is all students enrolled at the 4th grade. All schools of all
educational sub-systems that have students learning full-time in the target grade are part of the
international target population, including schools that are not under the authority of the national
Ministry of Education.
TIMSS and PIRLS are designed to pay particular attention to students curricular and instructional
experiences and therefore sample intact classes of students. However, as with PISA, TIMSS and PIRLS
are not designed to be representative of schools or classrooms and data have not been reweighted.
Results should be read as the percentage of 4th /8th grade students who report...

Country coverage
This publication incorporates information from 38 education systems or countries within the
OECD, and 5 partner countries.
19 education systems participated in REFLEX-HEGESCO in 2005 or 2008 [note Portugal is not

included in graphs by sector and Spain is not included in graphs by education level];
34 education systems within the OECD participated in PISA in 2009, 32 participated in 2006, and
30 education systems participated in 2000 and 2003.
38 education systems within the OECD participated in TIMSS in 2011, 27 education systems in
2007 and 23 in 2003.
29 education systems within the OECD participated in PIRLS in 2011, 27 in 2006 and 23 in 2001
23 education systems are included in the composite innovation indices.
Sample sizes
Table A.1. REFLEX-HEGESCO sample sizes
Smallest sample sizes for REFLEX-HEGESCO 2005 or 2008 (based on analysis in Chapter 4)
All sectors Education sector All sectors Education sector
Italy 2,595 264 2,335 225
Spain 3,169 576 2,871 497
France 1,312 275 1,260 256
Austria 1,437 233 1,368 215
Germany 1,451 290 1,373 260
Netherlands 2,954 418 2,884 404
United kingdom 1,275 223 1,224 215
Finland 2,165 435 2,108 415
Norway 1,873 308 1,807 291
Czech republic 5,878 878 5,679 814
Switzerland 4,262 187 4,117 179
Portugal 577 122 557 117
Belgium 1,213 187 1,182 182
Estonia 840 147 783 132
Slovenia 2,578 409 2,461 373
Turkey 1,995 248 1,879 235
Lithuania 783 103 747 96
Poland 1,112 222 1,100 219
Hungary 1,058 139 1,020 138
Total 38,527 5,664 36,755 5,263
Note that education sector sample size includes graduates working in primary, secondary and higher education, with the exception of Spain, where
all education professionals are counted, including those in other kinds of education.

Table A.2. TIMSS sample sizes: Principals
4th grade 8th grade
OECD countries 2003 2007 2011 1999 2003 2007 2011

Australia 204 229 280 201 207 228 277
Austria 196 158
Belgium Flemish 149 142 165 144
Canada 408
Ontario 189 188 146 186 176 143
Quebec 193 186 190 175 170 189
Alberta 146 143 145
British Columbia 150 150
Chile 200 189 195 193
Czech Republic 144 177 162 147
Denmark 137 216
Estonia 151
Finland 145 175 145
Germany 246 197
Hungary 157 144 149 147 155 144 146
Ireland 150
Israel 139 146 146 151
Italy 171 170 202 191 171 170 197
Japan 150 148 149 140 146 146 138
Korea 150 150 149 150 150
Netherlands 130 141 128 182 130
New Zealand 220 220 180 160 169 158
Norway 139 145 119 138 139 134
Poland 150
Portugal 147
Slovak Republic 184 197 145 179
Slovenia 174 148 195 153 174 148 186
Spain 151
Basque country 115 112
Sweden 155 152 159 159 153
Turkey 257 206 146 239
England 123 143 125 206 87 137 118
Scotland 125 139 128 129
United States 248 257 369 240 232 239 501
Alabama 55
California 82
Colorado 53
Connecticut 62
Florida 77 60
Indiana 56 54 56
Massachusetts 47 48 56
Minnesota 50 49 55
North Carolina 46 59
Non OECD countries

Hong Kong 132 126 136 183 125 120 117
Indonesia 168 150 149 153
Russian Federation 205 206 202 194 214 210 210
Singapore 182 177 176 145 164 164 165
South Africa 233 255 285

Table A.3. TIMSS sample sizes: Teachers
4th grade 8th grade maths 8th grade science
OECD countries 2003 2007 2011 2003 2007 2011 2003 2007 2011
Australia 300 360 594 246 251 802 520 496 1049
Austria 356 296
Belgium Flemish 547 268 272 641
Canada
Ontario 255 279 362 203 214 244 206 219 245
Quebec 269 308 300 181 226 265 442 192 323
Alberta 252 235 222 234
British Columbia 313 187 464
Chile 200 215 194 214 194
Denmark 246 341
Estonia 168 521
Finland 310 264 827
Germany 373 312
Hungary 158 255 324 198 289 280 627 987 1005
Ireland 220
Israel 353 394 514 308 270 282
Italy 243 323 314 217 287 205 217 287 205
Japan 178 250 265 146 216 181 146 178 151
Korea 168 365 243 376 357 181 202
Netherlands 141 218 210 130 377
New Zealand 722 609 494 261 354 176 265
Norway 228 310 280 179 270 175 179 264 171
Poland 257
Portugal 240
Slovak Republic 343 422 179 599
Slovenia 180 340 245 238 503 523 528 779 901
Spain 200
Basque country 123 164 124 155
Sweden 396 369 300 491 405 647 680 540
Turkey 263 146 240 146 240
England 165 250 261 139 235 212 503 615 751
Scotland 199 291 172 330 677 859
United States 921 904 767 456 532 559 1090 687 931
Alabama 107 130
California 111 210
Colorado 105 151
Connecticut 120 148
Florida 191 113 212
Indiana 138 128 113 285 146
Massachusetts 156 103 105 114 107
Minnesota 168 104 110 116 147
North Carolina 120 105 114
Non OECD countries

Hong Kong 297 282 267 147 145 148 131 123 124
Indonesia 155 149 170 278 276 259
Russian Federation 206 268 218 215 273 239 855 1083 916
Singapore 246 508 515 332 357 330 336 429 330
South Africa 256 327 255 316

Table A.4. TIMSS sample sizes: Students
4th grade 8th grade
OECD countries 2003 2007 2011 2003 2007 2011

Australia 4321 4108 6146 4791 4069 7556
Austria 4859 4668
Belgium Flemish 4712 4849 4970
Canada
Ontario 4362 3496 4570 4217 3448 4756
Quebec 4350 3885 4235 4411 3956 6149
Alberta 4037 3645 4799
British Columbia 4153 4256
Chile 5585 6377 5835
Czech Republic 4235 4578 4845
Denmark 3519 3987
Estonia 4040
Finland 4638 4266
Germany 5200 3995
Hungary 3319 4048 5204 3302 4111 5178
Ireland 4560
Israel 4318 3294 4699
Italy 4282 4470 4200 4278 4408 3979
Japan 4535 4487 4411 4856 4312 4414
Korea 4334 5309 4240 5166
Netherlands 2937 3349 3229 3065
New Zealand 4308 4940 5572 3801 5336
Norway 4342 4108 3121 4133 4627 3862
Poland 5027
Portugal 4042
Slovak Republic 4963 5616 4215
Slovenia 3126 4351 4492 3578 4043 4415
Spain 4183
Basque country 2514 2296
Sweden 4676 4663 4256 5215 5573
Turkey 7479 4498 6928
England 3585 4316 3397 2830 4025 3842
Scotland 3936 3929 3516 4070
United States 9829 7896 12569 8912 7377 10477
Alabama 2113
California 2614
Colorado 2167
Connecticut 2099
Florida 2661 1712
Indiana 2233 2188 2260
Massachusetts 1747 1897 2075
Minnesota 1846 1777 2500
North Carolina 1792 2103
Non OECD countries

Hong Kong 4608 3791 3957 4972 3470 4015
Indonesia 5762 4203 5795
Russian Federation 3963 4464 4467 4667 4472 4893
Singapore 6668 5041 6368 6018 4599 5927
South Africa 8952 11969

Table A.5. PIRLS sample sizes: Principals, teachers and students
Principals Teachers Students
2001 2006 2001 2006 2011 2001 2006 2011

OECD countries
Australia 519 6126
Austria 158 263 284 5067 4670
Belgium French 150 278 218 4552 3727
Canada 409 1405 23206
Ontario 190 180 222 200 282 4295 3988 4561
Quebec 182 185 187 210 220 3958 3748 4244
Alberta 150 233 219 4243 3789
British Columbia 148 273 4150
Nova Scotia 201 258 4436
Czech Republic 141 141 235 3022 4556
Denmark 145 216 237 4001 4594
Finland 291 4640
France 145 169 228 261 276 3538 4404 4438
Germany 211 405 393 418 222 7633 7899 4000
Greece 145 145 2494
Hungary 216 149 220 196 252 4666 4068 5204
Iceland 133 128 242 239 3676 3673
Ireland 221 4524
Israel 147 149 147 149 166 3973 3908 4186
Italy 184 150 184 198 239 3502 3581 4189
Luxembourg 178 363 5101
Netherlands 135 139 195 207 207 4112 4156 3995
New Zealand 156 243 175 514 434 2488 6256 5644
Norway 136 135 199 228 200 3459 3837 3190
Poland 148 250 257 4854 5005
Portugal 242 4085
Slovak Republic 150 167 176 263 316 3807 5380 5630
Slovenia 148 145 155 315 243 2952 5337 4512
Spain 152 193 403 4094 8580
Andalusia 198 4333
Sweden 146 147 398 258 271 6044 4394 4622
Turkey 154 154 5125
United Kingdom 268
England 131 148 132 186 182 3156 4036 3927
Scotland 118 130 136 202 2717 3775
United States 174 183 176 253 618 3763 5190 12726
Florida 150 2598
Non OECD countries

Hong Kong 147 144 147 144 138 5050 4712 3875
Indonesia 168 168 163 4774 4791
Russian Federation 206 232 206 232 209 4093 4720 4461
Singapore 196 178 196 357 356 7002 6390 6367
South Africa 397 403 111 14657 3515

Table A.6. PISA sample sizes: Principals
OECD countries 2000 2003 2006 2009
Australia 228 314 350 345

Austria 213 192 197 280
Belgium 214 282 269 275
Canada 1098 1066 861 908
Chile 173 199
Denmark 223 205 209 285
Estonia 169 175
Finland 155 197 155 203
France 174 163 179 166
Germany 213 213 225 226
Greece 139 171 189 183
Hungary 193 252 189 187
Iceland 130 129 135 129
Ireland 135 143 164 141
Isral 149 176
Italy 170 406 796 1095
Japan 135 144 181 185
Korea 146 149 154 157
Luxembourg 23 29 31 39
Mexico 182 1102 1128 1531
Netherlands 100 153 183 185
New Zealand 152 171 170 161
Norway 176 180 203 197
Poland 126 163 221 179
Portugal 145 152 172 212
Slovak Republic 281 188 189
Slovenia 356 337
Spain 185 383 686 888
Sweden 159 185 197 189
Switzerland 282 444 509 425
Turkey 159 160 170
United Kingdom 349 361 494 481
Scotland 88 96 94 97
United States 145 262 166 160
Non OECD countries
Hong Kong 145 146 151

Indonesia 344 352 183
Russian Federation 238 211 209 213
Singapore 171

Year coverage
This publication focuses on change across time and therefore requires data from the same
questions asked in different years. There are many such questions in the datasets employed, but it
should be noted that the years in which they were answered varies.
Where possible, analysis focuses on change between 2003 and 2011, although data from PISA for
2000, 2003 and 2009 are also used, and from PIRLS 2001 and 2006. The years included in the analyses
are indicated in the chapters.
In some cases, data are also available for an additional year between the two end points. In this
case, the data from all three data collection exercises are represented in figures but only the end
points are discussed in the text.
Calculation of cross-country means and totals

Given the range of education systems covered in each chapter, cross-country means may not
always incorporate the same countries or the same number of education systems. Where practical,
the average cross-country statistics have been calculated using data from European Union countries
(as in the chapter based on REFLEXHEGESCO data) or from OECD countries (as in PISA, TIMSS, PIRLS).
For each indicator in TIMSS, PIRLS and PISA, the OECD average is computed taking into account the
subset of education systems for which data is available for all years concerned.
The cross-country average is calculated as the unweighted mean of the data values of all relevant
countries for which data are available or can be estimated. It therefore refers to an average of data
values at the level of the national (or regional) systems and can be used to answer the question
of how an indicator value for a given country compares with the value for a typical or average
country. It does not take into account the absolute size of the education system in each country. This
approach is taken for the purpose of comparing, for example, changes within individual countries
with those across the entire cross-country area for which valid data are available, with this area
considered as a single entity.
Calculation of effect sizes

Effect sizes are presented for all analyses in addition to tests of statistical significance.
Tests of significance allow the reader to determine whether the difference between the two
percentages reported could have happened by chance if the actual difference is zero and thus
consider the quality of the instrument used for measurement. However, statistical significance is
dependent on the sample size (the larger the sample and the more confident the reader can be
that even small differences wouldnt have happened by chance) and can, in principle, be improved
simply by increasing the number of observations. Yet this does not tell the reader anything about
how meaningful the observed effects are in real-world terms. For example, a change in classroom
practice could be statistically significant but only amount to a few percentage points of relative
change with no practical meaning.
The effect size provides important information about the size of the relationship between two
statistics. The main difference between effect size and significance is that change is normalized
by the standard deviation as opposed to standard error, which means that the result no longer
depends on sample size. The precise form of calculation depends on the type of question asked, but
is typically calculated as:
X2 - X1
E = _________
21

i.e. as the change between a treatment and control group (or any two subgroups of a sample; or
as in our case - two different years), divided by a pooled standard deviation:
v 21 + v 22
v 21 = 2
Sometimes, the control group standard deviation or more complicated forms of pooled standard
deviations are used instead of the one displayed.
This book looks at effect sizes in two ways. One approach is to calculate country level effect
sizes. Here, means and standard deviations refer to the individual country samples. The effect size
calculation provides information about how much, in terms of their own standard deviation, a
^ ^
country has moved up (or down) over time. For country level effect sizes, 1 and , 2 are estimated
via =SE*n (with n being the sample sizes), which provides a conservative (lower) estimate for the
effect size (as n could potentially be overestimated by including invalid observations).
A second way of looking at effect size is required for questions that evaluate proportions, i.e.
those that deal with categorical variables and ask, for example, How often do you do this activity in
class? Daily? At least weekly? At least monthly? Rarely or never?. In this case, Cohen h is applied
to carry out an arcsin-transformation, whereby h=2(arcsin P1-arcsin P2).
In accordance with common practices, effect sizes are assessed at three different levels. Effect
sizes of less than 0.2 are considered negligible or very small, between 0.2 and 0.5 are considered to
be small, effect sizes between 0.5 and 0.8, medium, and effect sizes above 0.8 are considered large.
While the usefulness of such cut-offs is debatable, this convention is followed by adding a colour
coding in three different shades of blue when displaying effect sizes. The reader should interpret the
colour coding with care as there is little practical difference between an effect size of 0.18 and 0.22,
even if the colour coding is different.
Further resources
The publication uses the OECD StatLinks service. Below each table and Figure is a URL that
leads to a corresponding Excel workbook containing the underlying data for that indicator. These
URLs are stable and will remain unchanged over time. In addition, readers of the electronic version
of this publication (the e-book) will be able to click directly on the links and the relevant workbook
will open in a separate window. The tables in the Excel files contain additional information and
computations that could not be presented in the paper version.
Note on Israel

Annex B
Composite innovation indices
This Annex explains how the composite innovation indices have been
computed.

ANNEX B COMPOSITE INNOVATION INDICES
The analyses reported throughout this book have shown considerable variation in the amount
of change in educational practices and thus the potential extent of innovation. In order to provide
an overview of change across school and classroom practices and to draw some conclusions about
the level of innovation in each country, it may be considered helpful to combine some of this
information and look at the extent and focus of innovation within education in different countries.
There may be important differences between school and classroom level practices, or practices
at different grade levels. For this reason, several indices have been created (specifically, indices at
school, classroom, 8th grade and 4thgrade levels, and by maths and science), as well as an overall
score. This allows readers and policy makers to identify which aspects of countries education
system(s) appear to be relatively innovative, and identifies countries that are innovating throughout
the education system.
Creating the indices

The indices of change draw on the analysis reported in this book. The approach used is broadly
based on the guidance provided in the 2008 OECD handbook on constructing composite indicators.
In particular, the indicators are derived (as far as possible) on the definition of innovation discussed
in the introduction and the process of creating them takes into account the need for appropriate
data and imputing missing values. The data does not need to be weighted or normalised, since we
are using the same measure across all our indicators, and they each add such a small amount to the
overall indicator.
Data sources for each indicator vary only slightly, as shown in the following table.
Table B.1 Data Sources for Indices

Study name Questionnaires used Grades/age covered Incorporated in indices
TIMSS Principals 4 th
Classroom, School, 4th grade
8 th
Teachers 4 th
8 th
PIRLS Principals 4th Classroom, School, 4th grade
Teachers 4th Classroom, School, 4th grade
PISA Principals 15-year-olds Classroom, School, 8th grade
Notes: All questions used in the indices have been asked in more than one wave of data collection, allowing analysis of change over time. Information
from student level questionnaires has not been used due to the lower level of country level coverage and corresponding high level of missing data.
The indices are based on the effect sizes of changes in responses to specific questions. Effect
sizes reflect the size and direction of changes seen across two points in time, with a large positive
effect size indicating a large increase over time and a large negative effect size indicating a large
decrease. Effect sizes give a standardised measure of the change and can thus be easily added
together. Given that both increases and decreases indicate change and can be considered to be
innovative, the absolute value of the effect size has been used to create the indicator. An index that
kept the sign of the effect size would make countries that have large changes in both directions
appear to have no change at all.

Missing values
Variation in the coverage of PISA and TIMSS/PIRLS means that data are not available for all
education systems across both databases. School and classroom change effect sizes are therefore
not available for all education systems across all of the questions asked. Furthermore, data are
missing when certain questions (or questionnaires) were omitted at the national level at certain
points in time. This is not an issue when reporting responses to a single question, but it does pose
a potential problem when seeking to combine information across questions. In order to analyse as
many countries as possible whilst keeping a wide range of questions in the analysis, it has been
necessary to manage the missing data through a combination of deletion and imputation:
An iterative process has been used to manage observations (education systems) and variables
(questions) with missing data, and some systems/countries and questions have had to be omitted
in the construction of the index:
1. Education systems that had effect size data for fewer than 10% of the potential question set
(that is all questions reported in the preceding chapters) were excluded. This left us with 43
education systems.
2. Following this, questions with high proportions of missing data were dropped. Specifically, those
questions with effect size missing for more than 50% of the remaining database were excluded.
3. Education systems with less than 40% valid data on the remaining questions were then
excluded from the analysis. A total of 28 education systems are covered in the final indices.
4. At this point, those questions that still had missing change effect sizes for more than 50%
of the education systems covered have been dropped. A total of 75 questions remained, to
be used to impute missing values (School: 13 PISA questions, 20 TIMSS and PIRLS questions;
Classroom: 8 PISA questions, 34 TIMSS and PIRLS questions). Some of these questions were
not used in the final indices, as discussed below.
The remaining missing values have been replaced using imputed values. The imputation process
uses several stages and takes into account information across all available questions and countries.
The method provides replicable values with the same data (in other words, another analyst would get
the same results by applying this method to the same data), and is equally applicable to different data.
Imputation was undertaken on 4 separate groups of data, incorporating information from both
4th and 8th grade where relevant:
Missing PISA data was imputed using existing PISA data separately at classroom and school
levels.
Missing TIMSS/PIRLS data was imputed using existing TIMSS/PIRLS data separately at classroom
and school levels.
The imputation process uses relative information about how question responses vary across
countries and information about the extent of change within an education system, as follows:
1. For each available question, the mean absolute effect size was calculated across countries (i.e. the
sum of absolute effect sizes on a particular question divided by the number of countries) (A))
2. For each country, the ratio of absolute effect sizes (AES) to this cross country mean was then
calculated for every variable by country [RatioQ =AESQC/AQ]
3. Then, for each country, the mean ratio of absolute effect size to cross country mean was
calculated (i.e. the sum of the ratio on each question, divided by the number of questions with
valid data (MeanRatio))
4. Finally, missing values were replaced by the product of the mean ratio and the mean absolute
effect size: [Imputed value=MeanRatio*A ]

Criteria for including questions in the indices

Valid data from all relevant questions were used in the missing data imputation. However,
highly correlated questions may unduly influence an index that seeks to explore the extent to which
change occurs over different aspects of education, particularly given the existence of missing data.
For this reason, where question effect sizes are highly correlated [0.6 or more using Persons r] and
the wording of the questions is the same across different grades or subjects, only the question
with the highest absolute effect size at the OECD level has been included in the classroom, school
and overall indices. Where the effect sizes of different questions within a module are correlated,
but the wording differs, both questions have been included as separate items within the indicator.
Questions have also been retained for indices at subject and grade level where the possibility of
correlation is not a problem.
Table 0.2 below shows the final number of questions included in the classroom and schools
indices, and the number of imputed values per country. The overall index combines information
from both classroom and school; it is calculated as the sum of the two indices, divided by the total
number of available questions.

Table B.2 Count of imputed values
Overall Classroom School Mathematics Science 8th grade 4th grade

index index index index index index index
% of imputed values
% of imputed values
% of imputed values
% of imputed values
% of imputed values
% of imputed values
% of imputed values
available questions
available questions
available questions
available questions
available questions
available questions
available questions
n. of potentially
n. of potentially
n. of potentially
n. of potentially
n. of potentially
n. of potentially
n. of potentially
Australia 55 4% 33 0% 22 9% 22 0% 25 0% 57 0% 18 11%
Austria 55 51% 33 58% 22 41% 22 82% 25 80% 57 63% 18 22%
Chile 55 35% 33 45% 22 36% 22 32% 25 44% 57 26%
Czech Republic 55 49% 33 67% 22 36% 22 77% 25 72% 57 65% 18 22%
Denmark 55 53% 33 58% 22 45% 22 82% 25 80% 57 65% 18 22%
England 42 0% 33 12% 9 0% 22 0% 25 0% 57 30% 18 0%
Germany 55 47% 33 58% 22 32% 22 82% 25 80% 57 63% 18 11%
Hong Kong 47 23% 25 0% 22 50% 22 0% 25 0% 57 33% 18 0%
Hungary 55 2% 33 3% 22 0% 22 0% 25 4% 57 2% 18 0%
Indiana 34 44% 25 44% 9 44% 22 32% 25 40% 36 75%
Indonesia 55 20% 33 27% 22 9% 22 23% 25 28% 57 2%
Israel 55 22% 33 36% 22 18% 22 23% 25 24% 57 14% 18 89%
Italy 55 0% 33 0% 22 0% 22 0% 25 0% 57 0% 18 0%
Japan 55 5% 33 0% 22 14% 22 0% 25 0% 57 2% 18 11%
Korea 55 25% 33 45% 22 27% 22 23% 25 24% 57 19%
Massachusetts 34 44% 25 44% 9 44% 22 32% 25 40% 36 75%
Minnesota 34 44% 25 44% 9 44% 22 32% 25 40% 36 75%
Netherlands 55 44% 33 52% 22 32% 22 77% 25 76% 57 63% 18 0%
New Zealand 55 9% 33 9% 22 9% 22 9% 25 16% 57 11% 18 0%
Norway 55 4% 33 0% 22 9% 22 5% 25 8% 57 5% 18 0%
Ontario 34 0% 25 0% 9 0% 22 0% 25 0% 36 58% 18 0%
Quebec 34 0% 25 0% 9 0% 22 0% 25 0% 36 58% 18 0%
Russian Federation 55 5% 33 15% 22 5% 22 0% 25 8% 57 9% 18 6%
Singapore 34 0% 25 0% 9 0% 22 0% 25 0% 36 58% 18 0%
Slovenia 55 13% 33 21% 22 27% 22 0% 25 4% 57 23% 18 0%
Sweden 55 5% 33 9% 22 0% 22 5% 25 8% 57 2% 18 11%
Turkey 55 31% 33 42% 22 23% 22 27% 25 36% 57 16%
United States 55 0% 33 0% 22 0% 22 0% 25 0% 57 0% 18 0%
= between 40% and 50% of values used in the composite index were imputed
= more than 50% of values used in the composite index were imputed.
Note that all imputations have been undertaken using the maximum number of valid observations (allowing for a better estimate of the true value),
before creating the composite indices using different combinations of variables.

Developing and reporting the indices

The indices developed are intended to show the extent of change or innovation in one country
when compared with other countries. They can be used to rank countries according to their relative
levels of innovation in terms of school and classroom practices.
The final indices for each country =100 x (sum of absolute effect sizes/number of questions included).
The number of questions included depends on whether data exists in PISA and/or TIMSS/PIRLS
and therefore differs across education systems. It also clearly depends on the indicator itself: up to
22 questions are used at school level (depending on the availability of PISA data), 33 at classroom
level, 45 at 8th grade (including PISA data) and 10 at 4th grade; with up to 55 questions combined into
an overall indicator (combining school and classroom level).
It is possible for the absolute effect size to take a value that is greater than one; however in
practice it typically ranges from zero to one. In theory, it is therefore possible for the indicator to take
a minimum value of zero and a maximum value of over 100.
The use of Principal Components Analysis (PCA) was explored as an alternative approach to
creating indices. Criteria for selecting factors were eigenvalues >=1. Combined factor scores (adding
the individual scores from each factor identified) were found to be highly correlated with the final
approach taken.
Cautions
Question inclusion
The indices combine information from a large and diverse pool of questions asked on different
surveys. On the assumption that each question can provide additional information about the extent
of change and innovation in an education system, the process employed to develop the indices
has drawn on as many of the questions as possible and their inclusion has been determined by
the availability of valid data. However, a more theoretical approach focusing on the most relevant
questions, or a statistical approach to data reduction may provide different results.
Education system coverage

The indices provide some information about a subset of the education systems discussed in
the previous chapters. This subset has been determined by the availability of data. It may be that
other systems sit at the extremes of the ranking. It should be noted that the inclusion or removal
of education systems would also impact on the imputation of missing values. Although it gives a
robust synthesis of change covered by our change indicators, the country ranking should not be
over-interpreted.
OECD average
The OECD average is computed for all the education systems for which data are available for
all years concerned. In calculating the weights of regions that do not correspond to an entire OECD
member the following procedure has been followed. Education systems that are part of a country
for which the overall data is available are not considered this being the case of the different States
in the United States. Conversely, education systems that do not have a figure for the whole country
they belong to have been given weight equal to 1- this being the case, for example of Ontario and
Belgium Flanders among others.

Time periods
The effect size of the change in responses to a particular question is calculated across the same
two points of time for each country but the two points in time may differ by question. The indices
therefore show a tendency to change or innovate across different time periods, rather than the
extent of change over a specific time period.
Interpreting the findings

The indices reported help the reader to consider the benefits of such a composite innovation
indicator based on change measures, but may not provide a fully accurate representation of the
level of change and innovation within a country. Whilst the indicator is based on many questions
and observations, the relatively large proportion of missing data imputations that was needed to
construct the innovation indices invites the reader to be cautious. The innovation indices are mere
indicators of innovation, and small differences in levels are almost certainly not meaningful.
A higher score on the indicator suggests that an education system is characterised by more
change than other systems. However, there is currently no theory that could be applied to describe
the different levels in terms of adequacy of innovation. Similarly, the scale does not provide
information about what is necessary to move from one point to another. Additional work could be
undertaken to develop qualitative descriptions of different points on the scale, but this should be
preceded by improved data collection.
Note on Israel

Countries top 5 areas of classroom-level innovations
Hong Kong, China

Canada (Ontario)
Canada (Qubec)
United Kingdom
United States
New Zealand
Netherlands
Federation
Singapore
Indonesia
(England)
Australia
Hungary
Slovenia
Sweden
Russian
Norway
Subject
Grade
Korea
Japan
Israel
Italy
Science experiments
8th 5
demonstration
Reading on their own 4th 4 5
4th 4 3
ma
8th 3 1 3 2 4 3
Student relating to real life 4th 2 1 2 4 3 3
sci
8th 1 3 1 1 4 3 3
rea 4th 1 3
Data interpretation in
8 th
4 3
maths
Text interpretation 4th 5 5 4 5 2
4th 2 2 3 5
Explain answers in maths
8th 2 2 5 4 4
4th 5 2 2
Explain subjects in science
8th 5 1 5 2 2 4 5
Observe and describe in
8 th
1 2 2 4 3 1 1 2
science
Group work in maths 8th 5 1 3
Class ability grouping school
8th 2
policy
Decide procedures in maths 8 th
4 3
Design and plan 4th 4 1
experiments in science 8th 2 1
Choose book to read 4th 3 1
4 th
1 2 5 2
4th 2 1
ma
8th 1 2 4 1 1
Textbook as primary
4th 2 5 1
resource of instruction sci
8th 1 2 4 1 1 1
rea 4th 1
4th
ma
Textbook as supplementary 8th
resource 4th 3 3 2
sci
8th
4th 5
ma
8th
Computer availability
4th 2
sci
8th 2
4th 5 3 4 4
ma
8th 4 4
Network availability
4th 5 3 4 3 4
sci
8th 4 4
4th 1
ma
Use of computer to practice 8th
skills and procedures 4th 3 4 1
sci
8th 3 1
4th 4 3
ma
Use of computer to look up 8th 3
information 4th 4 4 5 3
sci
8th 4 3
Use of computer to conduct
4 th
5
scientific experiments
Use of computer to read text 4th 5 1
Note: Israel and Indonesia: 8 grade and 4 grade reading; Korea: 8 grade only; The Netherlands and Sweden: no TIMSS for 8 grade; ma: mathematics;
th th th th
sci: science; rea: reading.

The statistical data for Israel are supplied by and under the responsibility of the relevant Israeli authorities. The use of such data by the OECD is
without prejudice to the status of the Golan Heights, East Jerusalem and Israeli settlements in the West Bank under the terms of international law.

Countries top 5 areas of school-level innovations
Hong Kong, China

Canada (Ontario)
Canada (Qubec)
United Kingdom
United States
New Zealand
Netherlands
Federation
Singapore
Indonesia
(England)
Australia
Hungary
Slovenia
Sweden
Russian
Norway
Subject
Grade
Korea
Japan
Israel
Italy
4th 3 3
ma
8th 1 2 2 1 3
Remedial education
4th 3 3
sci
8th 1 2 2 1
4th 3 1 2 4
ma
8th 4 1 3 2
Enrichment education 4th 2 3 1 2 4
sci
8th 1 3 2 5
rea 4th 1 5
4th
Teachers peer discussions ma 1 5 4 4
8th
sci 4 1 5 4
4th 5
Teachers joined for
ma 2 5
preparation of materials 8th
sci 2 5
4th 1 5
Visits of others teachers
ma 2 1
classrooms 8th
sci 2 1 4 5
Use of assessment data for
comparison with district/ 8th 1 5 2
national performance
Use of assessment data
for comparison with other 8th
schools
8th 2 3 1 2
monitoring school progress
tracking over time by admin
8th 4 4 3
authority
instructional or curriculum 8th 2 1
improvement
Principal time devoted to
4th 3 1
curriculum development
Assessment data to judge
8th 5 2
teachers' effectiveness
Achievement data
to evaluate teachers 8th
performance
Achievement data to
evaluate principal's 8th 5 4 2
performance
4 th
2 1 1 2 3 1 3 2 2
Change in inspector's
ma 1 1 3 3 5 3 4 2 2
practices 8 th
sci 1 1 3 3 5 3 4 2 2
4th 1 5 5 4 2 1
Change in peer review
evaluation of teachers' ma 1 5 1 2 2 1 4
practices 8th
sci 1 5 1 2 2 1 4
4th 3 5 3 3 1
Change in use of incentives
for teachers' recruitment ma 3 5 3 4 3 1 1
and retention 8th
sci 3 5 3 4 3 1 1
8 th
3
Information provision
to parents : student
8th 5 4
performance relative to
other students

COUNTRIES TOP 5 AREAS OF SCHOOL-LEVEL INNOVATIONS
Hong Kong, China

Canada (Ontario)
Canada (Qubec)
United Kingdom
United States
New Zealand
Netherlands
Federation
Singapore
Indonesia
(England)
Australia
Hungary
Slovenia
Sweden
Russian
Norway
Subject
Grade
Korea
Japan
Israel
Italy
Information provision to
parents : relative to students
8th 4
in the same grade in other
schools
Parental involvement : to be 4th 4 2 5
serve on school committees 8th 4 2 5 4 5
Parental involvement : to 4th 3 5
be volunteer for school
projects, programs and trips 8th 3 5 4 3
Public relations :
achievement data publicly 8th 1 3
posted
Public relations and
8th 4 5
fundraising by the principal
Note: Israel and Indonesia: 8th grade and 4th grade reading; Korea: 8th grade only; The Netherlands and Sweden: no TIMSS for 8th grade; ma: mathematics;
sci: science; rea: reading.
The statistical data for Israel are supplied by and under the responsibility of the relevant Israeli authorities. The use of such data by the OECD is
without prejudice to the status of the Golan Heights, East Jerusalem and Israeli settlements in the West Bank under the terms of international law.

Country index
Countries and education systems Chapter

OECD countries
Australia 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18
Austria 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18
Belgium 1, 2, 3, 4, 7, 9, 14, 15, 16
Belgium (Fl.) 6, 8, 10, 11, 13, 15
Belgium (Fr.) 5, 6, 7
Canada 5, 6, 7, 9, 14, 15, 16
Alberta 5, 6, 7, 8, 10, 11, 13, 15
Ontario 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 17, 18
Quebec 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 17, 18
Chile 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18
Czech Republic 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18
Denmark 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18
Estonia 1, 2, 3, 4, 7, 9, 14, 15, 16
Finland 1, 2, 3, 4, 7, 9, 14, 15, 16
France 1, 2, 3, 4, 5, 6, 7, 8, 11, 12, 14, 16
Germany 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18
Greece 7, 9, 14, 15, 16
Hungary 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18
Iceland 7, 8, 9, 11, 12, 14, 15, 16
Ireland 7, 9, 14, 15, 16
Israel 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 17, 18
Italy 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18
Japan 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18
Korea 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 17, 18
Luxembourg 7, 9, 14, 15, 16
Mexico 7, 9, 14, 15, 16
Netherlands 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18
New Zealand 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18
Norway 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18
Poland 1, 2, 3, 4, 5, 7, 9, 14, 15, 16
Portugal 1, 2, 3, 4, 7, 9, 14, 15, 16
Slovak Republic 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16
Slovenia 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18
Spain 1, 2, 3, 4, 5, 6, 7, 9, 14, 15, 16
Basque country 5, 6, 7, 12, 16
Sweden 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18
Switzerland 1, 2, 3, 4, 7, 9, 14, 15, 16
Turkey 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18
United Kingdom 1, 2, 3, 4, 7, 9, 14, 15, 16
England 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 17, 18
Scotland 5, 6, 7, 8, 11, 12, 14, 15, 16
United States 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18
Indiana 6, 7, 8, 10, 11, 13, 15, 17, 18
Massachusetts 6, 7, 8, 10, 11, 13, 15, 17, 18
Minnesota 6, 7, 8, 10, 11, 13, 15, 17, 18
Non-OECD countries
Hong Kong 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 17, 18
Indonesia 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18
Lithuania 1, 2, 3, 4
Russian Federation 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18
Singapore 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 17, 18
South Africa 5, 6, 7, 8, 10, 11, 13, 15

ORGANISATION FOR ECONOMIC CO-OPERATION
AND DEVELOPMENT
The OECD is a unique forum where governments work together to address the economic, social and
environmental challenges of globalisation. The OECD is also at the forefront of efforts to understand and
to help governments respond to new developments and concerns, such as corporate governance, the
information economy and the challenges of an ageing population. The Organisation provides a setting
where governments can compare policy experiences, seek answers to common problems, identify good
practice and work to co-ordinate domestic and international policies.
The OECD member countries are: Australia, Austria, Belgium, Canada, Chile, the Czech Republic,
Denmark, Estonia, Finland, France, Germany, Greece, Hungary, Iceland, Ireland, Israel, Italy, Japan, Korea,
Luxembourg, Mexico, the Netherlands, New Zealand, Norway, Poland, Portugal, the Slovak Republic, Slovenia,
Spain, Sweden, Switzerland, Turkey, the United Kingdom and the United States. The European Union takes
part in the work of the OECD.
OECD Publishing disseminates widely the results of the Organisations statistics gathering and
research on economic, social and environmental issues, as well as the conventions, guidelines and
standards agreed by its members.
OECD PUBLISHING, 2, rue Andr-Pascal, 75775 PARIS CEDEX 16

(96 2014 05 1 P) ISBN 978-92-64-21568-9 2014
Measuring Innovation in Education
A NEW PERSPECTIVE
Do teachers innovate? Do they try different pedagogical approaches? Are practices within classrooms
and educational organisations changing? And to what extent can change be linked to improvements?
A measurement agenda is essential to an innovation and improvement strategy in education. Measuring
Innovation in Education offers new perspectives on addressing the need for such measurement.
This books first objective is informative: it gives readers new international comparative information about
innovation in education compared to other sectors. And it documents change in a variety of dimensions of
school practices between 1999 and 2011. Its second objective is methodological: it assesses two approaches
to capturing the extent and type of innovation occurring within and across education systems. The third
objective is exploratory: this book showcases a large-scale pilot that presents over 200 measures of innovation
in education using existing international data. Last but not least, the fourth objective is prospective: this report
proposes new approaches to measuring innovation in education in the future.
This book is the beginning of a new journey: it calls for innovations in the field of measurement and not just
of education.
Contents
Overview: why and how to measure innovation in education
Part I. Comparing innovation in education with other sectors
Part II. Innovation as change in classrooms and schools
Part III. Change in educational outcomes
Consult this publication on line at http://dx.doi.org/10.1787/9789264215696-en.

This work is published on the OECD iLibrary, which gathers all OECD books, periodicals and statistical databases.
Visit www.oecd-ilibrary.org for more information.
ISBN 978-92-64-21568-9
96 2014 05 1 P
9HSTCQE*cbfgij+

E

Încărcat de

Informații document

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

E

Încărcat de

Drepturi de autor:

Formate disponibile

Educational Research and Innovation

Centre for Educational Research and Innovation

Stphan Vincent-Lancrin, Kiira Krkkinen,

Please cite this publication as:

ISBN 978-92-64-21568-9 (print)

Series: Educational Research and Innovation

Corrigenda to OECD publications may be found on line at: www.oecd.org/about/publishing/corrigenda.htm.

Innovation drives improvement, either incrementally by advancing existing processes or more

MEASURING INNOVATION IN EDUCATION : A NEW PERSPECTIVE OECD 2014 3

The book is dedicated to the memory of Claude Sauvageot.

4 MEASURING INNOVATION IN EDUCATION : A NEW PERSPECTIVE OECD 2014

Chapter 1 Highly innovative workplaces in education and other sectors...................................51

Chapter 2 Employee participation in innovation in education and other sectors.......................65

Chapter 3 Speed of adoption of innovation in education and other sectors................................79

Chapter 4 Highly innovative jobs........................................................................................................91

Chapter 5 Innovation in teaching style............................................................................................107

MEASURING INNOVATION IN EDUCATION : A NEW PERSPECTIVE OECD 2014 5

Chapter 6 Innovation in instructional practices ............................................................................119

Chapter 7 Innovation in class organisation.....................................................................................135

Chapter 8 Innovation in the use of textbooks in classrooms........................................................155

Chapter 9 Innovation in the methods of assessment used in classrooms.................................167

Chapter 11 Innovation in the use of computers in the classroom...............................................185

Chapter 12 Innovation in the provision of special education in schools....................................203

Chapter 13 Innovation in the extent of teacher collaboration in schools...................................215

Chapter 14 Innovation in feedback mechanisms in schools.........................................................227

Chapter 15 Innovation in evaluation and hiring in schools..........................................................237

6 MEASURING INNOVATION IN EDUCATION : A NEW PERSPECTIVE OECD 2014

Chapter 16 Innovation in schools external relations....................................................................255

Chapter 17 Composite indices of innovation in classrooms and schools...................................271

Chapter 18 Innovation and educational outcomes........................................................................279

Annex A Data sources and methods................................................................................................309

Annex B Composite innovation indices...........................................................................................319

Countries top 5 areas of classroom-level innovations..................................................................327

MEASURING INNOVATION IN EDUCATION : A NEW PERSPECTIVE OECD 2014 7

8 MEASURING INNOVATION IN EDUCATION : A NEW PERSPECTIVE OECD 2014

Figure4.4 Professionals in highly innovative jobs, by sector................................................................................................................................ 96

MEASURING INNOVATION IN EDUCATION : A NEW PERSPECTIVE OECD 2014 9

10 MEASURING INNOVATION IN EDUCATION : A NEW PERSPECTIVE OECD 2014

Figure10.11 Change in Internet availability in 8th grade classrooms................................................................................................................ 183

MEASURING INNOVATION IN EDUCATION : A NEW PERSPECTIVE OECD 2014 11

12 MEASURING INNOVATION IN EDUCATION : A NEW PERSPECTIVE OECD 2014

Figure18.3 Classroom innovation and 8th grade mathematics outcomes....................................................................................................... 282

MEASURING INNOVATION IN EDUCATION : A NEW PERSPECTIVE OECD 2014 13

14 MEASURING INNOVATION IN EDUCATION : A NEW PERSPECTIVE OECD 2014

The ability to measure innovation is essential to an improvement strategy in education. Knowing

Key findings on innovation

MEASURING INNOVATION IN EDUCATION : A NEW PERSPECTIVE OECD 2014 15

Key findings on innovation

Another approach to measuring innovation in education is based on assessing significant

Educational organisations have innovated in the areas of special education, creation of

Innovativeeducational systems generally have higher expenditures than non-innovative

Towards surveys on innovation

Be administered to the central educational administration (ministries, relevant local

Question three levels of stakeholders (principal/president, teachers/faculty and students)

16 MEASURING INNOVATION IN EDUCATION : A NEW PERSPECTIVE OECD 2014

MEASURING INNOVATION IN EDUCATION : A NEW PERSPECTIVE OECD 2014 17

Why and how to measure

This Overview highlights the importance of measuring innovation in

MEASURING INNOVATION IN EDUCATION : A NEW PERSPECTIVE OECD 2014 19

Objectives of the book

Why measure innovation in education: context and rationales

20 MEASURING INNOVATION IN EDUCATION : A NEW PERSPECTIVE OECD 2014

Why innovation in education matters