Observation – what can(’t) we learn

Daniel Muijs
Head of Research
Ofsted
@ProfDanielMuijs

Observation – what can(’t) we learn Slide 1

Why evidence-informed teaching matters

 Teaching has more impact on pupil outcomes than any other

aspect of schooling
 Explains up to 75% of classroom level variance in pupil
performance
 Collective teacher effectiveness (all the teachers in a school)
can have lasting effects on pupils
 Teaching quality is something we can affect as schools and
educators

Observation – what can(’t) we learn Slide 2

A social justice issue

 Most important for:

 pupils from low socioeconomic status (SES) and disadvantaged
backgrounds
 lower achieving pupils
 pupils at risk
 Less important for more advantaged pupils

Observation – what can(’t) we learn Slide 3

Do we have evidence on what works?

 We do!
 A lot of evidence on effective teaching
 Robust findings in a number of key areas
 Significant evidence on effective schools
 But:
 We don’t know everything – evidence base differs among areas
 Evidence needs to be translated into school and classroom contexts
 Evidence may be contextual: ‘what works where, under what
circumstances’ (Dylan William)
 But again, some things work under most circumstances, some under
few

Observation – what can(’t) we learn Slide 4

How can one judge teaching quality?

 Pupil attainment
 Value added
 Teacher self-report
 Pupil report
 Observation

Observation – what can(’t) we learn Slide 5

Value of observation
 Significant predictor of attainment, but measures distinct
aspects (correlations are moderate)
 Allows more detailed evidence on what actually happens in
classrooms
 Provides one indicator of teaching quality across the schools
 Provides one indicator of (in)consistency of quality of teaching
and approaches to teaching across the school

Observation – what can(’t) we learn Slide 6

Problems with observation

 Creation of standard ‘Ofsted lessons’

 Issues of reliability:
 will different observers come to the same judgement?
 how many observations required?
 Validity:
 what are effective practices? How do we know?
 What is observable, and what isn’t.

Observation – what can(’t) we learn Slide 7

Observation

 International seminar held looking at six international models

 What can Ofsted learn from international best practice in the use of
lesson observation for the evaluation and improvement of teaching
quality?
 What changes should Ofsted consider in revising its inspection
framework?

Observation – what can(’t) we learn Slide 8

Lesson observation

 The Classroom Assessment Scoring System (CLASS)

 Framework for Teaching (FfT)
 The International Comparative Analysis of Learning and
Teaching (ICALT)
 The International System for Teacher Observation and
Feedback (ISTOF)
 The Mathematical Quality of Instruction (MQI)
 Generic Dimensions of Teaching Quality

Observation – what can(’t) we learn Slide 9

Key discussions from the seminar

 Four key issues:

 Level of Inference
 What can(‘t) we observe?
 School and classroom level
 Generic and subject-specific aspects of teaching

Observation – what can(’t) we learn Slide 10

Inference in observation
 An important concept from the seminar was about the level of
inference
 A high inference model requires the observer to make use of
their own judgement to make an evaluation:
Teacher seizes on a teachable moment to enhance a lesson
 A low inference model has indicators which are factual and
directly observable:
Pupils put their hands up to answer questions

Observation – what can(’t) we learn Slide 11

Inference in observation
 Low inference models are more reliable, but don’t necessarily capture what
matters most in effective teaching
 The six models all used scaled indicators but, the experts were clear that
these were still high inference models
 Primacy was still given to expert judgement
 In many cases, qualitative data was still also collected to support the ratings
given in the indicators.

Observation – what can(’t) we learn Slide 12

Teaching vs learning

 The challenge put to us at the seminar was that observation should

only be used for assessing teaching (the focus of all six models).
 The argument is that learning is invisible: you can’t see it happening
through observation. Attempts to measure learning through
observation do not meet typical reliability standards.
 Research indicates, however, that teacher quality is the most
important school factor associated with pupil attainment.
 A focus on teachers and teaching, therefore, ensures validity in an
observation model.
 Learning can still be picked up in other ways to inform judgements,
for instance, through school assessment practices and book scrutiny.

Observation – what can(’t) we learn Slide 13

Teacher level vs school level

 Most of the models focused on individual teacher observation.

 US models often used for individual accountability(!)
 There was agreement that one-off observations of a single
teacher is unreliable for evaluating that teacher.
 The experts agreed that in our context there was no argument
to focus on teacher or lesson level observation.
 Aggregation of observations at the school level was considered
much more likely to lead to valid and reliable judgements.
 There was general agreement that if school level observation is
the preference.

Observation – what can(’t) we learn Slide 14

Generic focus vs subject focus

 Teaching is subject-specific, but there are generic teaching

qualities, and generic measures of teaching are significantly
related to value added (e.g. MET studies)
 Existing instruments are primarily generic
 At the seminar, the experts agreed that subject aspects are
desirable – if difficult
 Subject-expertise would also need to be considered within the
inspection workforce for reliability purposes.

Observation – what can(’t) we learn Slide 15

Two examples: ICALT
 High inference observational tool that also includes low inference indicators
 Developmentally focused - differential 'item difficulty' indicates development needs
 Developed with the Dutch inspectorate
 Meant to facilitate international comparison
 Observation instrument has seven aspects:
− safe and stimulating learning climate
− clear and activating instruction
− efficient classroom management,
− teaching learning strategies,
− adaptive teaching,
− involvement of students,
− reflexivity and discursiveness

Observation – what can(’t) we learn Slide 16

Two examples: ISTOF
 High inference generic observation framework
 Developed as an instrument that would work across borders in international school effectiveness studies
 Also enables the provision of formative feedback on teaching
 Contains items that draw on a variety of teaching styles
 Seven factors included in observation schedule:
− assessment
− clarity of instruction
− classroom climate
− classroom management
− Differentiation
− instructional skills
− promoting active learning
 High-inference nature means observers must be appropriately trained before using the instrument
 Observations carried out either in person or through video.

Observation – what can(’t) we learn Slide 17

Activity

 Please have a look at the ICALT model and the ISTOF

instrument and discuss the following questions:
 How do these approaches compare to what current observation
practice in your school is?
 What do you think the strengths and weakness of the ICALT and ISTOF
approaches are?
 What do you think the strengths and weakness of Ofsted’s current
approach are?

Observation – what can(’t) we learn Slide 18

Implications for Ofsted

 We will be looking at our observation practices, with models to

be tested in Autumn
 We will NOT be grading individual lessons – just as we do not
do now
 We will NOT be using simplistic low inference tick-box
approaches
 We will NOT be using any of the international models we looked
at

Observation – what can(’t) we learn Slide 19

Implications for schools

 Be clear on what observation is for:

 accountability
 performance management
 mutual learning
 Different approaches may be required…

Observation – what can(’t) we learn Slide 20

Ofsted on the web and on social media

www.gov.uk/ofsted
https://reports.ofsted.gov.uk
www.linkedin.com/company/ofsted
www.youtube.com/ofstednews
www.slideshare.net/ofstednews
www.twitter.com/ofstednews

Observation – what can(’t) we learn Slide 21