Revised Draft 20/11/07

Territorial Cohesion – analysis of environmental aspects of EU

regional policy

Task 1: Evaluate ex-post the effectiveness, efficiency and effects of

implementing the Structural and Cohesion Funds in environment

Storyline for Task 1

Background

The EU regional policy seeks to strengthen the economic, social and territorial ‘cohesion’ of
the Union. Meeting this objective presents significant challenges and opportunities for
environmental protection and sustainable development in Europe.

Evaluation of the effectiveness, efficiency and effects of cohesion measures can help raise
important questions about Regional policy funds and the environment: Are they working?
Are they effective? How can they be improved? However, experience suggests that
more systematic and rigorous evaluation (ex-ante, mid-term and ex-post, and on-going)
focusing on the environment is required to adequately answer these questions.

In 2006, the ENEA (European Network of Environmental Authorities) working group ISFE
(Impact of Structural and Cohesion Funds on Environment) prepared an analytical
description of the contribution of Structural and Cohesion Funds to a better environment.
This provides an overview of how the direct environmental investments have been allocated
to environment sectors and the status of integration of environment into the sectors.

Aim and objectives

The overall aim of this project is to build on the ENEA work and to evaluate ex-post the
effectiveness, efficiency and effects1 of implementing the Structural and Cohesion Funds in
environment within three pilot countries (Austria, Italy and Spain) for the 2000-2006 cycle.
The project will focus in particular on environmental interventions and on certain types of
environmental interventions, including those relating to the waste water treatment sector,
biodiversity (Natura2000 sites) and energy efficiency / renewable energy, and will only
consider two of the Structural Fund themes (the European Regional Development Fund
(ERDF) and the European Social Fund (ESF)), as well as Cohesion Funds.

To achieve this overall aim, the project will seek to address the following specific objectives:

1. Evaluate ex-post the Structural and Cohesion Funds implementation of environmental

policies (overall policies, directives and strategies) and the integration of environment
into non-environmental policies / objectives.

2. Assess and discuss the potential imbalances of the Structural and Cohesions Funds
allocations / priorities to the various economic, social and environmental dimensions of
Regional development taking into account the experiences of previous programming
cycles.

1
These are defined in a 2001 EEA report, Reporting on environmental measures: Are we being effective?, as follows:
o the effects of an environmental measure: the outputs of a measure that can be directly attributed to its
implementation.
o the effectiveness of an environmental measure: a judgement about the outcome: whether or not they have resulted in
the objectives and targets of the policy measure being achieved.
o the cost-effectiveness of an environmental measure: a comparison of the effects of a measure with the costs of
implementing it.
Terrestrial cohesion: 1 Milieu Consortium
Storyline for Task 1
 Revised Draft 20/11/07

The starting point will be to investigate how EC environmental (and sustainable

development) objectives and legislation are incorporated into Structural and Cohesion Funds
spending and how the pilot countries interpret these and integrate them within Structural and
Cohesion Funds operational programmes, axes / priorities, measures and projects. This will
need to include an investigation of the evaluation requirements for the funds and current
practice (ex-ante, mid-term and ex-post).

It will also be important to reflect on the balance of funding between environment, economic
and social sectors, in terms of relative importance of environment expenditures according to
needs and the amount of funds allocated. In addition, whether and how the funding supports
the environmental policy objectives and positively effects / impacts2 the environment. The
absorption capacity of regions to undertake the environmental interventions programmed will
also be considered. Absorption capacity will be principally considered in the financial sense,
but also in terms of administrative capacity.

Overall, the main focus for the project will be on developing an analytical framework /
methodology, or potentially frameworks / methodologies, for undertaking ex-post
effectiveness evaluation (focusing on environmental implications, rather than a full ex-post
evaluation). This framework will be informed by EEA experience of ex-post policy
effectiveness analysis, other available literature and existing practice, including a review of
the methodologies found in the case study countries. It will then be tested in the pilot
countries (and specific case studies within them) and revised through an iterative process. A
key aspect to be considered as part of developing this framework / methodology will be data
and information needs and availability.

Focus sectors and regions

As noted above, the analysis of Cohesion and Structural fund spending will focus on three
sectors: waste water treatment, biodiversity and renewable energy. Due to the different
situations in the three pilot countries, each country will focus on at least two sectors.

• Austria: biodiversity and renewable energy

• Italy: waste water treatment, biodiversity and renewable energy
• Spain: waste water treatment and biodiversity / renewable energy (respective focus of
biodiversity and renewable energy still to be determined for Spain)

For waste water treatment and renewable energy we expect to use existing cross-country
indicators (where possible, these will be linked with indicators in the EEA core set). For
biodiversity, initial discussions have not identified existing indicators that would identify the
impacts and effects of Cohesion and Structural Fund spending. Here the work will be
exploratory: in both Austria and Italy we will test indicators constructed on data available in
the countries.

Where possible, individual case studies will go beyond this “triangle” of core sectors to
capture an idea of others.

The project will focus on the following regions:

• Austria: all nine objective 2 regions (there was only one objective 1 region 2000-2006 -
Burgenland)
• Italy: all six objective 1 regions in 2000-2006 (Campania, Apulia, Basilicata, Calabria,
Sicily and Sardinia)
• Spain: two objective 1 regions in 2000-2006 (Andalusia and Galicia)
2
The terms “effect” and “impact” are often used interchangeably within environmental assessment to mean a change in the
existing environment caused directly or indirectly by the activity or measure being implemented. However, where a distinction is
drawn, an impact is a physical or measurable change to the environment attributable to the activity or measure being
implemented whereas an effect is the result of an impact on a particular resource or receptor. The distinction between effect (or
impact) and effectiveness has already been discussed above.
Terrestrial cohesion: 2 Milieu Consortium
Storyline for Task 1
 Revised Draft 20/11/07

Progress of work

This is the second draft of the project Storyline and reflects comments received following
consultation with the EEA and ENEA working group on an earlier version dated 11/10/07.

At the ENEA working group on 27 November 2007 we will present an interim report with our
initial work, focusing on the review of national evaluations and an analysis of the pilot country
context.

In the next step of work we will focus on the overview of Cohesion and Structural fund
spending and interventions. We see these separate activities as interacting. Moreover,
while evaluation methodology development represents our end-point, this work will interact
with the other activities.

Overview of the main project activities

The process of evaluating the effectiveness, efficiency and effects of cohesion policy is
illustrated in Appendix 1, which highlights the key project activities linked to the two
objectives above (i.e. ex-post evaluation and potential imbalances / absorption capacity).
The specific tasks from the original EEA Technical Annex to be undertaken are listed in
Appendix 2.

The main project activities are detailed below. These will also be reflected in the final report
from the project and include the following topics:

A. Review of existing evaluations, regulations and DG Regio guidance on carrying out

evaluations;
B. Overview of Cohesion and Structural Fund spending and interventions, including
potential imbalances and absorption capacity;
C. Analysis of policy and pilot country context;
D. Evaluation methodology development; and
E. Evaluation within the pilot countries.

Where input and assistance from the ETC-LUCI and ENEA working group is likely to be
needed, this has also been indicated.

A. Review of existing evaluations, regulations and DG Regio guidance on carrying

out evaluations:
1. A brief literature review on the evaluation of Cohesion and Structural Funds and their
environmental implications, highlighting any improvements needed identified as
identified in the literature;
2. A review of the requirements in EU Regulations and guidance (DG Regio) relating to
the evaluation (ex-ante, mid-term and ex-post) of Cohesion and Structural Funds for
the 2000-2006 cycle. Highlighting the evaluation requirements relating to the
environmental in particular, the stages and scales at which the evaluations are required
(including what is / could be required at plan, programme, axis, measure, project levels
etc), the indicators that have to be monitored etc;
3. A comparison of the evaluation requirements (in Regulations, guidance etc) for the
2000-2006 cycle with those for 2007-2013 (and the 1994-1999 cycle where
appropriate) – are there additional requirements in the current cycle, do these address
any of the shortcoming in previous cycle etc?
4. Examples of practice drawn from evaluation reports prepared within the pilot countries /
regions to illustrate specific points and issues arising out of the tasks above and to
illustrate environmental evaluation practice that may have wider application and for
comparison of approaches;

Terrestrial cohesion: 3 Milieu Consortium

Storyline for Task 1
 Revised Draft 20/11/07

5. Findings of the review highlighting potential improvements to the evaluations (how and
when evaluation is undertaken, methodologies used, data and indicators used etc).

B. Overview of Cohesion and Structural Fund spending and interventions:

Overview of spending and environmental interventions:
6. An overview of Cohesion and Structural Fund allocations / commitments / actual spend
for 2000-2006 cycle within the three pilot countries (ideally at NUTS3) broken down by
broad category, with further details on environmental interventions (including, where
available, contextual data on overall national spending on the environment; [ETC-LUCI
and ENEA working group pilot country representatives to assist with data acquisition
and analysis]
7. Examples / illustrations of how Cohesion and Structural Funds axis / measures /
projects have been used to implement EU objectives / specific Directives (e.g. Urban
Waste Water Treatment, Natura2000, Water Framework etc) within the pilot countries /
regions;
8. General conclusions from the review of Cohesion and Structural Funds allocations on
whether the funds have been directed to environmental problems (consider planned
expenditure and environmental situation when programming).

Potential imbalances and absorption capacity

9. An overview of trends and key messages on imbalances and absorption from an
analysis of financial data for the 2000-2006 cycle (and 1994-1999 / 2007-2013 data if
comparable / available), combined into broader categories to reflect economic, social
and environmental dimensions and within the environmental intervention categories (for
example, are the issues for UWWT - building infrastructure is expensive but an
established practice – different to biodiversity - where new types of projects may need
to be developed); [ETC-LUCI and ENEA working group pilot country representatives to
assist with data acquisition and analysis]
10. In pilot countries / regions, comparing initial programming documents with final
programming documents and data of actual expenditure (last data available) and
number of environmental projects for 2000-2006 cycle – are any trends / conclusions
apparent re absorption capacity? Is there a move of money allocated for the
environment to other interventions or vice versa? [ETC-LUCI and ENEA working group
pilot country representatives to assist with data acquisition and analysis]
11. Findings from a review of literature and interviews with the ENEA working group
members in the pilot countries and other representatives interviewed in the countries,
including relevant desk officers in the Commission (DG Regio and DG Environment),
for explanations on potential imbalances and integration of environment into social and
economic interventions; [ENEA working group pilot country representatives to assist
through interviews etc]
12. General conclusions / illustrative examples on factors behind potential imbalances and
the integration of environmental objectives in operation programmes and potential
improvements, including:
• discussion on whether there is anything inherent in, for example,
environmental interventions generally / certain types of environmental
interventions / the organisations involved in delivering them that creates barriers
compared with other (e.g. socio-economic) interventions. Is there any evidence
that environmental interventions are moving from sectoral work to more integrated
work? (the Austrian case study may provide some potential insights here);
• categories/typology of potential barriers to absorption – e.g. transposition of
Directives / national policy implementation, financial, knowledge – and identifying
examples across the different categories from the pilot countries;
• potential examples or elements of good practice approaches for measuring
regional absorption capacity within the pilot countries and their transferability;

Terrestrial cohesion: 4 Milieu Consortium

Storyline for Task 1
 Revised Draft 20/11/07

• ideas on how to operationalise assessments (qualitatively / quantitatively) of

absorption capacity;

C. Analysis of policy and pilot country context:

13. A brief summary of EU environmental objectives, including existing / forthcoming EU
Directives, strategies etc, focusing on the environmental interventions being
considered, i.e. waste water, biodiversity (Natura2000 sites) and energy efficiency /
renewable energy;
14. A brief summaries of each pilot country (and for the case study regions within them) on
how these EU environmental objectives are addressed by Cohesion and Structural
Fund spending at the national / regional level, the key priorities (relating to state,
pressures etc) and how Cohesion and Structural Funds are being used to address
them;
15. Land use / environmental profiles (or characterisations) for the pilot countries / case
study regions drawn from existing sources to illustrate the key priorities, with maps of
key datasets; [ETC-LUCI and ENEA working group pilot country representatives to
assist with data acquisition and analysis]
16. Discussion on the issues and lessons drawn from other groups that have considered
Cohesion and Structural Funds and the environment (e.g. WWF, ESPON, REC, DG
Regio, other ENEA working groups, European Court of Auditors, Economic and Social
Committee and Committee of the Regions).

D. Evaluation methodology development:

17. An overall analytical framework / methodology for undertaking ex-post effectiveness
evaluation of cohesion policies, including identifying potential data availability / needs to
support the framework / methodology; discussing ideas and approaches for evaluation
with the ENEA country representatives, as well as ETC-LUCI experts; and testing and
revising the framework / methodology within pilot countries / regions and comment on
utility. In terms of what this framework / methodology might look like, it could potentially
include a variety of elements with the overall aim of improving the robustness of the
evaluation of effectiveness, efficiency and effects, for example:
• principles for the more effective and practical use of existing forms of
evaluation and assessment, including the information their outputs contain;
• illustrations of how existing data and information could be processed and
analysed to provide further utility on evaluating effectiveness, efficiency and effects;
• the use of evaluations at different scales (e.g. plan, programme, axis,
measure, project levels) and how they interrelate / inform one another, how results /
analysis is aggregated / disaggregated, etc. Also the spatial scale at which data is
collected / analysed;
• examples of existing good practice and recommendations on how these could
be operationalised and transferred more widely;
• examples of additional data / indicators that could be collected and
illustrations of how this could provide further utility; and
• additional tools and/or methods that could be used to support further analysis
or understand different aspects, both as part of existing evaluations and
assessments and ex-post to provide evidence as part of an external auditing or
review process.

E. Evaluation within the pilot countries:

18. A qualitative assessment of what has been achieved (at the pilot country / region level)
in terms of the key environmental indicators and the Cohesion and Structural Fund
contribution, including examples / comments from key local contacts; [ETC-LUCI and
ENEA working group pilot country representatives to assist with data acquisition and
analysis]

Terrestrial cohesion: 5 Milieu Consortium

Storyline for Task 1
 Revised Draft 20/11/07

19. An analysis of the effectiveness of the environmental interventions within Cohesion and
Structural Funds in the pilot country / regions i.e. have the interventions delivered the
outcome intended in terms of their intended objectives and targets; [ETC-LUCI and
ENEA working group pilot country representatives to assist with data acquisition and
analysis]
20. Illustrations, using a few key environmental / spatial datasets for the beginning and end
of the cycle period, of how the environment has changed over the period – discussion
on whether any conclusions can be drawn on the role of Cohesion and Structural Fund
interventions in these changes or at least whether funds have been targeted to
problems / priorities (using a variety of means – data, literature / reports, interviews with
managing authorities etc); [ETC-LUCI and ENEA working group pilot country
representatives to assist with data acquisition and analysis]
21. Identifying potential case studies for the 2009/2010 mid-term review from the pilot
countries / regions. [ENEA working group pilot country representatives to assist
through discussions]

Terrestrial cohesion: 6 Milieu Consortium

Storyline for Task 1
 Revised Draft 20/11/07

Appendix 1: Project Storyline

Draft “sto
Development of an ov
methodology for unde
effectiveness evaluat

Draft “sto policies [1

EU policy
Terrestrial cohesion:
Storyline for Task 1 objectives,
7 Milieu Consortium
 Revised Draft 20/11/07

Appendix 2: Specific project tasks

Task 1A: Ex-post evaluation
General
i. Development of an overall analytical framework / methodology for undertaking ex-post
effectiveness evaluation of cohesion policies.
ii. Description of different national / regional approaches or strategies using Structural and
Cohesion Funds to comply with general EU environmental objectives; if possible, illustration of
using Structural and Cohesion Funds to implement EU Directives related to the environment (e.g.
Nitrates, Natura2000, WFD).
iii. Description of different kinds of “interventions” aimed at environmental objectives: specific
axes inside operational programmes; specific measures; initiatives (particularly, INTERREG);
major projects (if any).
iv. Review of current (existing in EU regulation) tools to assess effectiveness, efficiency and
effects / impacts of Structural and Cohesion Funds: evaluations (according to chapter III of R (EC)
1260/99); environmental impact assessment; strategic impact assessment; Identification of gaps
and scope for improvement, specifically related to environment.
v. General conclusion (qualitative assessment) on contribution of Structural and Cohesion Funds
to the achievement of overall and specific EU environmental objectives (directives and strategies).
If possible, discussion on [expected] combined effects and synergies between EU and national
policies.
vi. Make available data from “InfoView” in a “standardised” and aggregated way with a view to
update the data of the ISFE prepared in 2006 by DG Environment.
Pilot countries
vii. Effectiveness, efficiency and effect / impact analysis of Structural and Cohesion Funds with
regard to environmental objectives. In other words, have the environmental objectives been
achieved and / or changed during the policy cycle, have the funds been applied efficiently, and,
lastly, have the funds created positive environmental impacts? When possible, illustrative
examples of “good practice”.
viii. Identify potential case studies (e.g. pilot countries, operational programmes, axes, measures,
projects, etc.) for 2008 that can feed into the Mid-term review of Regional policy in 2009 / 2010 in
consultation with the ENEA working group, Commission services and EEA / ETC-LUSI.

Task 1B: Potential imbalances/absorption capacity

General:
i. Discussion on likely explanatory / influencing factors behind potential imbalances among
investments devoted to the economic, social and environmental dimensions, in programmes
financed by EU Structural and Cohesion Funds. Suggestions for improvement of better and more
balanced integration of environmental objectives in operation programmes.
ii. Suggest a methodology for measuring and assessing regional absorption capacity by the
number (and if possible funds allocated) of environmental projects identified in the regions
(NUTS3), comparing initial programming documents with final programming documents and data
of actual expenditure (last data available); ideally, for the two programming periods 1994-1999 and
2000-2006. Include a review and revision of existing indicators for monitoring and evaluation and
eventually, the proposal of new indicators.
iii. Discussion on likely explanatory / influencing factors behind the different levels of absorption
capacity, including the proposal of possible recommendations for improvement.
Pilot countries:
iv. Undertake an ex-ante / ex-post comparison, if possible, assessing whether the environmental
interventions were completed compared to the initial strategy in the pilot countries, including
assessing whether these environmental interventions achieved their originally stated objectives.
The analysis should further provide examples of best practice and assess how absorption capacity
can influence the effectiveness, efficiency and effects / impacts of Structural and Cohesion
interventions with regard to environmental objectives (qualitative / quantitative assessment), e.g.
cost-benefit analysis related to action (resources used on environment) or inaction (resources not
used on environment but transferred to other programmes, axes or measures with potential
negative impacts on the environment).

Terrestrial cohesion: 8 Milieu Consortium

Storyline for Task 1