This article was downloaded by: [University of North Texas]

On: 29 November 2014, At: 02:13

Publisher: Routledge
Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer
House, 37-41 Mortimer Street, London W1T 3JH, UK

Educational Action Research

Publication details, including instructions for authors and subscription information:
http://www.tandfonline.com/loi/reac20

The impact of intensive value for money

performance auditing in educational systems
a
John Elliott
a
University of East Anglia , United Kingdom
Published online: 24 Feb 2007.

To cite this article: John Elliott (2002) The impact of intensive value for money performance auditing in educational
systems, Educational Action Research, 10:3, 499-506

To link to this article: http://dx.doi.org/10.1080/09650790200200197

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the Content) contained
in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no
representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of
the Content. Any opinions and views expressed in this publication are the opinions and views of the authors,
and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied
upon and should be independently verified with primary sources of information. Taylor and Francis shall
not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other
liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or
arising out of the use of the Content.

This article may be used for research, teaching, and private study purposes. Any substantial or systematic
reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any
form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://
www.tandfonline.com/page/terms-and-conditions
 Educational Action Research, Volume 10, Number 3, 2002

THEORETICAL RESOURCES

The Impact of Intensive

Value for Money Performance
Auditing in Educational Systems

JOHN ELLIOTT
University of East Anglia, United Kingdom
Downloaded by [University of North Texas] at 02:13 29 November 2014

ABSTRACT This article examines some of the consequences of value for money
audit systems on professional practices in general and those of teachers in
particular. The article argues that value for money audit actually prevent
professionals from evaluating their practices. This is because it confuses
outputs with outcomes and fails to empirically validate the former as indicators
of effectiveness in particular contexts of action. The article also argues that the
shift from external to internal audit and evaluation processes should not be
misunderstood as a move towards greater professional autonomy and a higher
trust form of accountability. It is argued that internal audit is a more exacting
expression of power over the activities of professionals than external audit.

What is Value for Money Performance Audit?

According to Power (1997) Value for Money (VfM) audit claims to evaluate
performance according to three sorts of criteria that he calls The Three Es;
namely:

Economy the acquisition of resources on the best possible terms.

Efficiency the use of resources to achieve a given level of output.
Effectiveness the match between intentions and outcomes.

He claims that the efficiency and effectiveness components are at tension in

VfM Audit because they stem from different logics of Evaluation. Efficiency
stems from the disciplines of Accountancy and Business Studies. Within
this logic outcomes get defined as intended outputs or measures of the
measurable. Effectiveness on the other hand stems from the Social
Sciences, and refers to both the intended and unintended effects of activities

499
 John Elliott

conducted in complex multi-dimensional contexts, many of which are not

susceptible to precise quantification. Power argues that while VfM Audit
claims to judge effectiveness, it tends to emphasise economy/efficiency.
As a mechanism of accountability for public service professionals,
Power believes that VfM should be viewed in a policy context that aspires to
replace inefficient bureaucratic management of the public sector with ideas
borrowed from the private sector under the label of the New Public
Management (NPM). The key ideas of the NPM are decentralisation and
organisational autonomy and its latent function is to facilitate indirect,
rather than direct regulatory control over performance. In addition, I would
contend that in this context power is divorced from responsibility and
accountability gets interpreted as a mechanism for securing regulatory
compliance.
Power also points out that VfM can shape up as a form of self-audit, as
well as external audit. This has implications for what is currently happening
in the United Kingdom educational system. We are witnessing a transition
from heavy touch external inspection systems to light touch inspections of
internal audit procedures. Within the new policy context the ideal
organisation manages its own compliance through self-audit.
Downloaded by [University of North Texas] at 02:13 29 November 2014

From the policy-makers perspective, VfM Audit is itself a cost-effective

means of driving up standards in our public services. Yet I would argue that
we now have increasing evidence of its counter-productive side effects. VfM
may shape performance in ways that raise standards, as these are defined
by specifications of measurable output, but in doing so it brings into effect
outcomes that threaten to undermine the actual quality of public service
delivery. If this is so, then it bears out Powers thesis that VfM confuses
effectiveness with efficiency.

The Impact of Value for Money Audit

I would suggest that too much stress on VfM shaped performance audit is
counterproductive in the following respects.

VfM audit drains too much trust out of the system where it counts, at the level
of the work place

If accountability for the quality of public services is too weak, there is an

increased danger of service providers neglecting the legitimate interests of
central government, communities of stakeholders and individual consumers;
of internal systems being more liable to corruption and inefficiency and of
the provision failing to fit the needs of a wider policy context. If, on the other
hand, accountability is too strong in its drive to render performance
increasingly transparent to the public gaze, trust in practitioners is drained
away and they, in turn, come to mistrust the integrity of the auditors.
Sustained mistrust turns into a pathological fear of audit and the resulting
despair produces a counterproductive situation in which practitioners, such

500
 VALUE FOR MONEY AUDITS

as teachers, lose motivation and interest in their work, go through the

motions of compliance, and avoid taking reasonable and necessary risks for
fear of being wrong (playing it safe). The outcome for educational
institutions, like schools and higher education institutions, is that they fail
to innovate in order to respond to new challenges and needs that emerge
from a dynamic, unstable and continuously changing social environment.

By confusing outputs with outcomes VfM audit neglects the difficult task of
understanding the complex connections between service activities and their
consequences, which is central to evaluations of the effectiveness of activities

It is one thing to assess performance against certain standardised outputs to

which it should conform (e.g. exam and test results, average waiting times
for patients, police response times to emergency calls, meals on wheels per
thousand of the elderly population) and quite another thing to assess the
impact or outcomes (unintended as well as intended) of performances,
including those which comply to the given standards.
For example, some teachers may be successful in getting large
numbers of students to pass a test or exam at an acceptable level of
attainment, but the methods used may have diminished rather than
Downloaded by [University of North Texas] at 02:13 29 November 2014

enhanced students motivation to learn and the depth of their engagement

with the subject matter (see Galton et al, 1999). Such methods may also
make teachers less motivated to teach, since their focus on producing the
measurable outputs specified by the tests gives them less discretion over the
selection of interesting subject matter for their students to learn. Given
these effects on learners and their teachers, can we say that the test or exam
is an indicator of effective teaching rather than simply its efficiency. I think
not. The negative outcomes nullify the use of the output measure the test
or exam as an indicator of effectiveness. As Power has pointed out output
specifications tend to lack empirical verification as indicators, since they are
defined independently of any understanding of the contextual complexities
that shape the relationship between activities and their actual outcomes.
It may be a fact that teachers and educational institutions are
improving their performance as measured against such indicators as exam
results and attendance rates, but we have very little understanding of the
chain of effects such improvements set off within the educational system
and beyond. For example, is decreasing attention to the 30% that are
unlikely to attain AC at GCSE in United Kingdom schools a direct
consequence of measuring performance against this indicator, or is the
problem of teacher supply in under-performing inner-city schools in any way
connected to the difficulties in such schools of generating performances that
are up to standard? What in these circumstances is effective teaching?
We need to give space in the accountability system for developing a
form of performance evaluation that is more sensitive to the complexities of
connecting processes causally to outcomes (see Power, 1997). In the case
of teachers these would include self, peer and students/parents

501
 John Elliott

evaluation components, in addition to an external evaluation by an

independent agent. Within this form of evaluation, indicators of effectiveness
will be regarded as things to be empirically verified, refined and discovered
in particular contexts of practice. They will also include qualitative as well as
quantitative indicators. Cost-effectiveness considerations would not be ruled
out, but instead of simply measuring the costs of achieving a given level of
output they would involve a more sophisticated weighing up of costs against
evidence of impact.

VfM audit is intolerant of time and insensitive to context and thereby renders
much evidence about the real impact of practices invisible

The prevailing technologies of audit rest upon the assumption of fixed and
immutable performance indicators that do not vary across time and context.
For example, they leave little room for a view of teaching effectiveness as a
time-dependent and context-bound phenomenon. The impact of teaching on
students, the chain of effects it creates in particular contexts, takes time to
unfold, and our understanding of the connections between processes and
outcomes therefore changes over time and is never perfect. Teaching, argues
Downloaded by [University of North Texas] at 02:13 29 November 2014

Strathern (2000), is audited as if immediate assimilability is its goal. Yet we

know that learning takes place over time and may manifest itself weeks,
years, generations, after teaching, and may manifest itself in forms that do
not look like the original at all for the students experiences will introduce
his or her own indirection.
Whereas performance audits aim to make activities within the
organisation perfectly transparent, through the use of timeless indicators,
such an aspiration is largely illusory as a comprehensive basis for
professional accountability. It is simply not the case that quality does not
exist if it cannot be measured against fixed and timeless targets. Indeed, the
reverse is often the case: namely, that the gaze of audit, if too unremitting in
its quest for transparency, makes evidence about the real impact of teaching
on learners invisible and in doing so masks quality.
Clarity of expression is often listed as an indicator of effective
teaching and can be judged within a very short time-scale. I well remember
doing a comparative evaluation of two seminars for serving teachers on a
course I was running; one given by a head teacher and the other by a
professional researcher. Immediately following the seminars, the course
members unanimously praised the head teacher and condemned the
researcher as a teacher on the grounds that the former was crystal clear
while the latter was very difficult to understand. Months later they were still
grappling with the ideas of the latter, but had entirely forgotten what the
former had said. Judgments about the quality of the teaching of one
compared to that of the other changed over time. If the researchers teaching
had been subjected to a formal performance audit at the time the demands
of immediate transparency would have rendered evidence about its true
quality invisible.

502
 VALUE FOR MONEY AUDITS

In yielding only imperfect information at any point in time evaluations

of a schools or teachers effectiveness have a transient use (see Strathern,
2000). They provide feedback that can be used by practitioners to further
develop their practice. Accountability for the effectiveness of teaching
involves a willingness on the part of teachers to publicly demonstrate, when
called upon to do so, how evidence about the impact of their teaching on
learners has been used to further develop practice. This type of
accountability presupposes a growth of evidence-based practice, where
teachers engage both with and in research into the quality of their teaching.
It lies at the heart of quality development in educational institutions and
should not be confused with audit-based accountability. The idea of
evidence-based teaching as a process of rectifying deficiencies, identified
by measuring performance against a normative template of fixed indicators,
rests on a distorted conception of what constitutes relevant evidence about
the relationship between teaching and its outcomes. Evidence of measurable
improvement, defined as bringing performance up to a fixed standard, is not
the same as evidence of development, defined as an open-ended and on-
going process over time.
Accountability for the development of true quality in teaching is
Downloaded by [University of North Texas] at 02:13 29 November 2014

conditional upon teachers being trusted to engage in the process free from
the unremitting gaze of audit. This is because it is a process, which is
essentially invisible to audit. The more the technology of audit the creation
of information systems and audit trails for the purposes of inspection
encroach on and shape teaching and learning, the less space teachers have
for quality development and becoming effective teachers.
It is precisely because quality development in teaching requires an
investment of trust in teachers that society in the form of its
representational bodies has the right at particular points in time to call
them to account for their practice in terms of its outcomes for students and
others. Teachers can exercise accountability by demonstrating at these
points their ability to make transient use of evidence about the impact of
their teaching in ways that improve their practical judgments and decisions.
Such demonstrations would consist of public accounts of reflective practice.
This kind of accountability implies trust, but not unconditional trust.
There comes a point at which the strengthening of performance audit
at the level of classrooms becomes counter-productive. At this point it
disrupts rather than enhances the development of teaching quality, drains
away the necessary amount of trust such development depends on, and
renders teachers largely unaccountable for the outcomes as opposed to the
outputs of their teaching. The strengthening of accountability through
audit may bring more comfort to policy-makers, and perhaps the
consumers of education, as a guarantor of minimal standards, but it is
likely to depress quality development. It is not that accountability within
educational systems is not in need of further strengthening, but this needs
to be based on a different logic of evaluation to the one that shapes

503
 John Elliott

performance auditing. A better balance between evaluation for performance

audit and evaluation for quality development needs to be achieved.

The unremitting gaze of VfM audit is driven by a strong sense of impending

crisis and places teachers in a constant state of activation from which they
find it difficult to find release

Given its intolerance of time for development and its aspiration to render
performance totally transparent to its gaze, performance auditing places
practitioners in the role of the ever-active performer. The audit culture is
intolerant of time because it is shaped by a sense of urgency. From its
perspective, the system is in a state of crisis and disaster is always
imminent. Things have to be done now to ward it off. People within the
organisation therefore have to be kept in a continuous state of activation.
This implies changes to the way in which professional selves are
conceptualised and developed. In the case of teachers the aim of professional
development would be to promote the auditable, competitive and ever-active
performer in the place of the inspiring teacher (see Strathern, 2000).
Downloaded by [University of North Texas] at 02:13 29 November 2014

Concluding Remarks
Before the rise of performance auditing, the accountability procedures for
rofessionals was very weak and it can be justifiably argued that too much
trust was invested in them unconditionally. However, the solution to the
problem is not to keep on extending a compliance model of accountability
derived from the private sector. When it is extended to the point where most
of the trust invested in practitioners has finally drained away and teachers
are kept so busy improving their fitness for audit that they have no time left
over for reflectively developing the quality of their practice, such an
accountability system becomes dysfunctional. The solution is to develop a
new model of professional accountability that invests conditional trust in
practitioners and is underpinned by a different logic of evaluatory practice.

Correspondence
Profesor John Elliott, CARE School of Education and Professional
Development, University of East Anglia, Norwich NR4 7TJ, United Kingdom
(john.elliott@uea.ac.uk).

References
Galton, M., Gray, J. & Rudduck, J. (1999) The Impact of School Transitions and
Transfers on Pupils: pupil progress and attainment, Research Report RR131.
Nottingham: DFEE Publications.
Power, M. (1997) The Audit Society. Oxford: University Press.

504
 VALUE FOR MONEY AUDITS

Strathern, M. (2000) The Tyranny of Transparency, British Educational Research

Journal, 26, pp. 309-321.
Downloaded by [University of North Texas] at 02:13 29 November 2014

505
 John Elliott

REINTERPRETING EVIDENCE-
BASED PRACTICE:
A NARRATIVE APPROACH

A Collaborative Action Research Network Conference

Sponsored by the South Thames Medical and Dental
Education Centre

Thursday October 5th, 2000 at

Senate House, University of London
CONFERENCE NARRATIVES
Downloaded by [University of North Texas] at 02:13 29 November 2014

Edited by Members of the

Collaborative Action Research Network

This report of the conference (approx. 90 pages) contains the

full text of the keynote lectures by Helen Simons and Trisha
Greenhalgh on the role of narrative in professional practice,
including specific studies of education and medicine, and
detailed summaries of all the workshop discussion groups.

Cost 8.30 (7.30 to CARN members)

Obtainable from
Lucila Recart, CARN Secretary,
CARE, University of East Anglia,
Norwich NR4 7TJ, United Kingdom.
Please make cheques payable to CARN

506