POLICY EVALUATION CRITERIA

INTRODUCTION
Policymaking and hence policy evaluation are undertaken in increasingly complex and uncertain
realities. A shift from one-way governing systems towards two-way governance systems
between those governing and those governed has been purported. The often-conflicting interests
of a range of public and private policy actors must therefore be taken into consideration when
prescribing and evaluating policy instruments. Moreover, increasing uncertainties regarding, for
example, the effects of global climate change, the volatility of financial markets and geo-political
instability pose further challenges to public policymaking. In light of these challenges, policy
evaluation has become a growth industry, undertaken by academics, governmental and nongovernmental organisations, private consultancy firms and lobby groups. As such, policy
evaluation is an inherently socio-political practice shaped by the interests, values and beliefs of
policy evaluators and participants in the policy process. The challenge for policy evaluators is
therefore to strive for objectivity in policy evaluation, while acknowledging the inherent
complexities and value-laden nature of this practice.

This section attempts to answer the following questions: what is policy evaluation; when is
policy evaluated; who evaluates policies and why; how are policies evaluated; and what is
evaluated. The section concludes that policy evaluation is an inherently political practice, which
requires the application of a mix of qualitative and quantitative research methods.

WHAT IS A POLICY EVALUATION?

Definitions of policy evaluation abound and vary. Broadly, policy evaluation can be understood
as learning about the consequences of public policy. Policy evaluation can be conceptualised as
either an ongoing activity throughout the policymaking process, or as a separate stage of the
policy cycle. The purpose of policy evaluation is to assess whether the policy objectives pursued
were (or will be) met, and the means employed to achieve these objectives.

TYPES OF POLICY EVALUATION

Three different types of policy evaluation can be discerned. Understanding policy evaluation as a
discrete stage of the policy cycle, results in the conceptualisation of the different types of policy
evaluation.

1. EX-ANTE POLICY EVALUATION

Ex-ante policy evaluation implicates the assessment of policy instruments before they are
chosen, designed and implemented. This practice aims to inform policymakers of the possible
consequences of various policy instruments before these are adopted. Ex-ante evaluation
measures include environmental impact assessment, strategic environmental assessment and
regulatory impact assessment.

2. EX-POST POLICY EVALUATION

Ex-post policy evaluation occurs following the implementation of the policy instrument, and
often (but not exclusively) ahead of its reformulation or termination. Ex-post evaluation
investigates 'how a public policy has actually fared in action.

3. CONTINUOUS POLICY EVALUATION

Continuous evaluation occurs at all stages of the policy process. Continuous evaluation involves
investigating a policy program to obtain all information pertinent to the assessment of its
performance, both process and result. Also termed formative evaluation, this type of
evaluation is undertaken intermittently at all stages of the policy cycle, and particularly during
policy implementation. Formative evaluation requires analysis of the extent to which a program
is being implemented and the conditions that promote successful implementation. This
description corresponds with the two evaluation dimensions presented in the introduction to this
topic. These are the considerations of social, economic, environmental and decoupling impacts
alongside considerations of the implementability of policy instruments. Continuous evaluation
challenges the earlier perceptions of evaluation as a distinctive stage of the policy cycle.

There are clear benefits to this approach, such as ensuring the assessment of public policies not
only before or after implementation but on a regular basis, maintaining dynamic policy

implementation, and dealing with uncertainties and unexpected consequences. However, in

practice, continuous evaluation remains the least implemented type of policy evaluation.
These notions of policy evaluation and particularly the conception of ex-ante and ex-post
evaluation, assume that policymaking is a linear process defined by the discrete stages of the
policy cycle. However, this is rarely the case, as policymaking is never a rational, linear process.
The idea of dividing policy-making in such a way greatly overstates the rational nature of policymaking and gives a false picture of a process which is not a conveyor belt in which agendasetting takes place at one end of the line and implementation and evaluation occurs at the other.
In practice, policymaking in general, and policy evaluation specifically, seldom occur in a linear
fashion. Still, the stagist model of policymaking provides a useful heuristic device 'with which to
orientate and situate a particular academic contribution' . We therefore maintain the distinction
between ex-ante and ex-post evaluation, although in practice policy evaluation can be understood
as a continuous process.

WHO EVALUATES, WHY, AND HOW?

Policy evaluation is conducted by governments and other interested policy actors, who wish to
influence the policymaking process. Policy evaluators include academic scholars, independent
research institutes, consultancy firms, international and national governmental bodies, nongovernmental organisations and business groups. Each of these actors has their own interests,
values and beliefs that may shape the outcomes of the evaluation process.

Two opposing views on the execution of policy evaluation exist. The rational model supports
evidence-based policymaking. In this view, policy evaluation should objectively assess the
achievements of a given policy instrument (or policy packages), creating a natural feedback
cycle to subsequent policy improvement. However, this ideal seldom occurs in practice, and
there is rarely a systematic connection between evaluation and policy improvement.
Interpretivist views of policymaking therefore conceive policy evaluation as a political tool
which shapes all (or some) stages of the policymaking process. These two conflicting views
subsequently

lead

to

diverging

methodological

perspectives

on

policy

evaluation.

WHAT IS EVALUATED?
As mentioned above, policy evaluation is an inherently political practice, influenced by the
interests and values of the evaluator. The selection of evaluation criteria and the perspectives of
different policy actors can therefore lead to different conclusions regarding policy outcomes.
Public policy scholars created numerous categorisations of evaluation criteria. These studies
were often concerned with the process, content, or outcomes of policies and policy instruments,
either separately or in combination. For example, administrative evaluation addresses these three
aspects, as summarised in Table 1. Criteria for policy evaluation, based on the examination of
process, content and outcome. These criteria are constructed in order to overcome the barrier
between positivists and constructivists evaluation approaches.

There are four outlined considerations that affect evaluation efforts. First, positivist verification
examines whether the policy's stated objectives were met. Second, validation of the policy in
terms of its relevance to the problems it aims to address will affect the evaluation outcomes.
Third, vindication examines whether the policy contributes to society as a whole, while social
choice examines the ideological concerns about the policys objectives. These considerations
influence the choice of divergent evaluation criteria, and therefore may lead to different results
of the policy evaluation process. Not only the choice of indicators to assess a specific impact, or
criterion will affect the evaluation outcome, but of course, and first of all the choice of what is
evaluated (before deciding on how it will be evaluated). Thus, evaluation can consider economic
impacts and appear successful only since environmental impacts are not considered.

Table 1:

IMPACT EVALUATION OF POLICY INSTRUMENTS

As mentioned above, the first part of this evaluation methodology examines the economic,
environmental, social, and decoupling impacts of policy instruments that promote servicizing.
These impacts are outlined in more detail below.

1. Economic impacts
Much work has been done on the economic impacts of environmental legislation and
technological innovation. Indeed, environmental economists have over the past decades created
detailed methodologies for economic evaluation, based largely on cost-benefit analyses. In the
SPREE project, our ambitions are more modest. We relate the concept of economic impacts to
the economic effects of policy instruments on governments, service providers, consumers and
other affected parties, or in other words, the value chain of the examined system. We suggest a
series of guiding questions, which will inform an economic analysis. Economic impact can be
considered to include the impact on economic growth, employment and changes in the costs and
benefits to affected parties.
In the context of the SPREE project, and especially the quantitative analysis, the focus is on the
service providers and produces, consumers, and on the chain/sector level. To assess the
economic impacts on these actors the following questions can be used to guide the evaluation.

For service providers and producers:

1. Does the policy instrument creates additional profits, and reduces risks for the service
provider?
2. Is innovation stimulated (dynamic efficiency)?
3. What are the costs in terms of compliance, investment, and marketing?
4. Are intellectual property rights secured?
5. Are environmental externalities internalised?

For consumers:
1. What are the costs for consumers associated with the policy instrument?
2. Are there any utility gains in terms of reduced costs, time savings and/or improved quality of
life (see also social impacts)?
3. What are the likely short and long-term impacts on consumer demand and consumption
behaviour?
For the chain/sector level:
1. What are the overall profits and added value in the system (e.g. knowledge gained, incomes
secured, expertise developed, networks constructed etc.)?
2. What are the labour employment consequences of the change in the system?

3.1.2 Environmental impacts

Servicizing systems and the policy instruments that support their uptake are often intended to
reduce the negative environmental impacts of economic development. One of the main
arguments in favour of the creation of servicizing systems is that gains can be made in resourceuse efficiency. Material products are treated as capital assets rather than consumables.
Servicizing may thus create incentives to maximize resource-use efficiency and product
durability, whilst minimizing costs to the producer, consumer and the environment.
Environmental assessment of servicizing solutions demonstrates that, on average, single-digitfactor improvement was achieved in various case studies, including: washing centres (Weaver,
Jansen et al. 2000; Hirschl, Konrad et al. 2003), car sharing (Sperling, Shaheen et al. 2000), ski
rental services (Hirschl, Konrad et al. 2001), and integrated pest management (Schmidt-Bleek
and Lehner 1998). The relatively modest improvements stem from a lack of focus on the

systemic changes that need to be made in order to deliver services in a more sustainable way. At
the same time, quantitative methods for evaluating environmental impacts of servicizing
solutions are still in short supply. Studies employing life-cycle analysis (LCA) and similar
methods for evaluating servicizing solutions still yield diverging results, owing to differences in
the evaluation systems initial conditions, boundaries and functional units chosen. The evaluation
of environmental impacts therefore constitutes an important challenge to be addressed
throughout the SPREE project.
In the SPREE evaluation of environmental impacts associated with the implementation of policy
instruments to promote servicizing, we distinguish between two evaluation criteria. Maximising
resource-use productivity and minimising environmental impacts. Resource-use oriented
indicators represent the amounts of resources used, while environmental-impact based indicators
relate to the environmental impacts of servicizing systems, including the production and
consumption of goods and services. Some examples of these indicators are outlined in Table 4,
with regards to materials, energy and water resources. Some of these indicators can be evaluated
through

LCA.

However,

we

acknowledge that it may not be possible to include all the suggested indicators in case-specific
evaluations. For example, it may be difficult to assess the impacts of a given policy on air
pollution. Likewise, current LCA techniques have not accounted for some resource-efficiency
indicators, such as metal depletion. Therefore, the factors suggested in Table 4 are more for
illustration while specific indicators for this category will be selected later in the project.

Due to some trade-offs between different environmental impacts, it is crucial to evaluate multiple
environmental impacts of servicizing systems based on an LCA methodology (as explained in
detail in Task 3.1). In the case of the SPREE project, indicators are identified for each of the
case-study sectors. Examples of these indicators include inventory data, such as greenhouse gas
emissions,

energy

use,

water use, and waste generated, as well as impact indicators such as Global Warming Potential
(GWP), acidification, etc.
3.1.3 Social impacts

Within the SPREE project, we examine the effect of policy instruments on the social impacts of
servicizing systems and we examine the effect of policy instruments that promote servicizing on
the behaviour of consumers and businesses. Different types of social impacts can be
distinguished among them are quality of life, the function of well-being (Diener, 2005), access to
resources, job opportunities and equity issues.
The concept of quality of life refers to the extent to which a persons life is desirable, often in
relation to some external factors such as income or environment (Diener, 2005). However, the
measurement of quality of life is not always based solely on objective measurements such as
GDP, life expectancy and education. For example, Pukeliene and Starkauskiene (2011)
differentiate between individual quality of life and societal quality of life. Individual quality of
life is explicit and considers how well individuals live, societal quality of life is implicit and
relates to the stability of society. For the purpose of this task, and in order to simplify the
evaluation procedure, we aim to evaluate individual quality of life, based on the measurements of
objective indicators.
The perception of wellbeing is more subjective. Wellbeing can be understood as a function of
how people feel and operate on a personal and societal level, together with an overall selfevaluation of a persons life (Diener, 2005; Michaelson et al., 2012). Community wellbeing has
been defined by Forjaz et al. (2011: 734) as satisfaction with the local place of residence taking
into account the attachment to it, the social and physical environment, and the services and
facilities. More broadly, Hall et al.(2010) define societal well-being as the sum of human wellbeing together with the condition of the ecosystem, which provides the necessary resources and
ecosystem services to maintain well-being. Thus, there is clearly an overlap between the notions
of quality of life, and well-being. However, for methodological purposes we distinguish between
these concepts.
In evaluating the social impacts of policy instruments to promote servicizing, equity must also be
considered. While policy instruments that promote servicizing may improve the quality of life
and the well-being of society as a whole, they may have negative impact upon certain
individuals,

or

they

might

not improve quality of life and wellbeing equally across society. Hall and his colleagues (2010)
argued that when considering the sum of human well-being, it is important to consider intergenerational and intra-generation equity. Although average material well-being can rise in a

society, if the distribution of well-being is skewed, progress may not be sustainable or equitable
(Hall et al., 2010). For example, some women may feel less comfortable travelling on public
transport late at night than travelling in their own car, and thus could be negatively affected by
measures to improve public transport a servicizing system. Disabled users, who previously
owned specially-adapted equipment, may struggle to benefit from services geared towards the
wider population and which do not take into consideration their particular needs. Therefore,
considerations of equity and fairness must be taken into account when measuring the social
impacts of policy instruments to promote servicizing (these aspects have been studied within task
3.2).
Some guiding questions for evaluating the social impacts of servicizing and policy instruments
that

promote

servicizing

include:

Evaluating Implementation
The previous section examined criteria for evaluating the change in the impacts of servicizing
systems as a result of policy instruments to promote servicizing systems. In this section, we turn
to examine factors that may affect the implementation of these instruments. It is common
knowledge that the impact of policy instruments depends on their implementation in practice
(Parsons, 1995: 547). Or, as Rist (1998: 158-159) observes:
Successful policy implementation involves the translation of policy intent into policy action.
While the failure to craft a thoughtful policy involving the choice of appropriate instruments can
reasonably predict that the policy will not have the intended effects, so also can the failure to
appropriately implement the policy ensure that the intended effects do not occur. Intention
without

execution

leads

nowhere.

Thus, changes in the economic, environmental and social impacts of policy instruments depend
on the implementation of these instruments. Still, most studies on policy instruments have not
adequately addressed their implementation in practice (Jordan et al. 2011: 546). In the remainder
of this section, we introduce some of the factors that may influence the implementabiltiy (that is
the likelihood of successful implementation) of policy instruments. These factors include
feasibility, institutional settings, political legitimacy, social acceptability and flexibility in

dealing with uncertainty. These considerations and their evaluation are discussed in some detail
below. The suggested criteria must not be read as a comprehensive review of the vast literature
on policy implementation, but should be seen as providing general guidelines for the empirical
work which will be undertaken later in this project.

3.2.1 Implementation feasibility: financial, technical, technological and pre-condition

considerations
In order to ensure the successful implementation of policy instruments, they must be feasible
financially, technically and technologically. As put by McDonnell and Elmore (1987: 146),
The selection of policy instrument depends on the constraints a policymaker faces and the
resources available either to diminish the force of those constraints or to enhance the
effectiveness of a given instrument. In simplest terms, identifying resources and constraints is
how policymakers assess what is feasible, given how they define a policy problem.
We therefore need to understand the material constraints under which policy instruments are
chosen and designed. These include financial limitations on both public bodies and the private
actors, who are responsible for the implementation in practice of policy instruments. A policy
instrument cannot be implemented if there are no funds to implement it, if it is not financially
viable, or it could only be partially implemented (e.g. unfinished project). Equally, policy
instruments need to be technically feasible, so that their implementation is not only financially
viable, but also relatively straight-forward to undertake in terms of technical feasibility. For
example, financial resources might exist to build a new high-speed train line, but the expertise to
build the required infrastructure, or the boring machines for the required tunnels are not
available.
Another consideration that must be taken into account when evaluating the implementabiltiy of
policy instruments to promote servicizing is whether the instrument is technologically feasible.
Some policy instruments are adopted in order to promote technological innovation (technologyforcing instruments). For example, the European Union car emissions standards have over
several decades promoted the uptake of new technologies, such as catalytic converters in the
1980s and 1990s (Arp, 2000; Wurzel, 2002), and more recently the uptake of hybrid cars and
other low-carbon car innovations, following EU legislation on car CO2 emissions. However,
when assessing technological feasibility it does not suffice to examine solely the available

technology. Rather, as suggested by Unruh (2000), we must also understand societal and
institutional factors, as well as interdependencies with other technologies and infrastructure in
order to adequately address this criterion. The matter of technological feasibility will need to be
carefully monitored during the project and the implementation of this methodological
framework, as some of these matters are in the heart of the SPREE project and the promotion of
a resource-efficient servicizing economy.
One of the main elements in formulating policy packages is accounting for the relations between
individual policy instruments. An important relation is precondition relation, relating to a
situation whereby the successful implementation of one policy instrument entirely depends upon
the prior successful implementation of another. In other words, a policy instrument will only
work

if

its

preconditions have been implemented (Givoni et al., 2013; Taeihagh et al., forthcoming). In this
respect it is important to identify, when characterising and evaluating a policy instrument
whether it has any preconditions.