Sunteți pe pagina 1din 217

An Evaluation of Pharefinanced Energy and Environment Programmes Final Report September 1999

This report was produced by "Environment Resources Management" (ERM) for the Evaluation Unit of the Joint External Relation Service (SCR) of the European Commission. The views and opinions expressed herein do not necessarily reflect those of the European Commission. The European Commission does not guarantee the accuracy of the data included in this report, nor does it accept responsibility for any consequence of their use.

CONTENT
1 EXECUTIVE SUMMARY 6

1.1 INTRODUCTION 1.2 METHODOLOGY AND APPROACH 1.3 PROGRAMME DESCRIPTION 1.4 RELEVANCE OF PHARE E&E PROGRAMMES 1.5 FINDINGS AT PROJECT LEVEL 1.5.1 INSTITUTIONAL PROJECTS 1.5.2 INVESTMENT PROJECTS 1.6 RECOMMENDATIONS
WHAT IS PHARE? LIST OF ABBREVIATIONS 1 INTRODUCTION

6 6 7 7 8 9 9 10
13 14 15

1.1 BACKGROUND 1.2 CONTEXT OF PHARE E&E PROGRAMMES 1.3 CHANGES IN PHARE STRATEGIES 1.4 EVALUATION OBJECTIVES 1.5 REPORT STRUCTURE
2 METHODOLOGY AND APPROACHES

15 15 16 17 17
18

2.1 THE INVENTORY OF PHARE E&E PROGRAMMES AND PROJECTS 2.1.1 COLLECTING INFORMATION 2.1.2 CLASSIFICATION 2.2 SAMPLING 2.2.1 SAMPLE SELECTION PROCESS 2.2.2 SAMPLE REPRESENTATIVENESS 2.3 FIELD EVALUATIONS 2.4 LOGICAL FRAMEWORKS 2.5 EVALUATION CRITERIA 2.6 GENERIC LOGICAL FRAMEWORKS FOR PROJECT EVALUATION
3 PHARES E&E PROGRAMME

19 19 19 21 21 23 26 27 27 28
30

3.1 PHARE E&E PROGRAMME DESCRIPTION 3.1.1 COUNTRY AND SECTOR ALLOCATIONS 3.1.2 EXPENDITURE BY SECTOR OBJECTIVE 3.1.3 EXPENDITURE BY PROJECT OUTPUT 3.2 RELEVANCE OF PHARE E&E PROGRAMMES 3.2.1 RELEVANCE TO COUNTRY NEEDS, POLICIES AND STRATEGIES 3.2.2 APPROPRIATENESS OF DESIGN OF PHARE E&E PROGRAMMES 3.3.3 COMPARATIVE ADVANTAGE OF PHARE INVOLVEMENT IN E&E PROGRAMMES
4 PROJECT LEVEL

30 30 31 33 35 36 39 45
50

4.1 INTRODUCTION 4.2 INSTITUTIONAL PROJECTS 4.2.1 PERFORMANCE SCORES 4.2.2 STRATEGY PROJECTS 4.2.3 ORGANISATIONAL REFORM PROJECTS 4.2.4 REGULATION DEVELOPMENT

50 50 50 51 53 55 3

4.2.5 RAISING PUBLIC AWARENESS 4.3 INVESTMENT PROJECTS 4.3.1 PHYSICAL ASSETS 4.3.2 DEMONSTRATIONS 4.3.3 INVESTMENT PREPARATION 4.3.4 E&E INVESTMENT FUNDS
5.1 RECOMMENDATIONS

55 56 57 60 62 63
67

5.2 PROJECT LEVEL RECOMMENDATIONS 5.2.1 INSTITUTIONAL STRENGTHENING PROJECTS 5.2.2 INVESTMENT PROJECTS 5.3 PROGRAMME LEVEL RECOMMENDATIONS 5.3.1 IMPROVING PHARE SECTOR ASSISTANCE 5.3.2 PROGRAMME STRATEGY RECOMMENDATIONS
APPENDIX A: PROJECT EVALUATION SUMMARIES BULGARIAN REGIONAL LAB EQUIPMENT BULGARIAN PORT DREDGING BULGARIAN BAY CLEAN-UP BULGARIAN DISTRICT HEATING BULGARIA DEMAND REDUCTION BULGARIAN TANNERY EFFLUENT CZECH MUNICIPAL LOANS CZECH MIT SUPPORT CZECH NGO SUPPORT ESTONIA WATER MANAGEMENT ESTONIA NETWORK REHABILITATION ESTONIAN POLLUTION MASTERPLAN ESTONIAN REGIONAL LAB EQUIPMENT HUNGARIAN HAZARDOUS WASTE HUNGARIAN SEWER SYSTEM INSTALLATION HUNGARIAN NATIONAL PARK

68 69 72 76 76 78
82 83 87 90 94 98 102 105 108 112 116 121 126 130 134 138 142

HUNGARY BIO-DIVERSITY AWARENESS LITHUANIA ENVIRONMENT INVESTMENT FUND LITHUANIAN ENERGY EFFICIENCY FUND LITHUANIA DH DECENTRALISATION LITHUANIA LSPS RESTRUCTURING POLAND INSTRUMENT COSTING ROMANIAN HOSPITAL ENERGY EFFICIENCY ROMANIAN ENERGY SERVICES ROMANIA ARCE SUPPORT ROMANIAN POLLUTION MONITORING SLOVAK ENERGY AUDIT SLOVAKIA ENERGY DATABASE SLOVAKIA POLLUTION MONITORING SLOVENIA ENERGY EFFICIENCY STRATEGY APPENDIX B: HISTORY OF PHARE E&E PROGRAMMES APPENDIX C: TERMS OF REFERENCE

146 150 153 157 161 168 173 177 180 184 188 191 194 198 201 206

EXECUTIVE SUMMARY

1.1

INTRODUCTION

This evaluation report covers all Energy and Environment (E&E) programmes financed by Phare over the period 1989-1998 in the ten candidate member states to the EU: Estonia, Latvia, Lithuania, Poland, the Czech and Slovak Republics, Hungary, Slovenia, Romania and Bulgaria. These countries have a combined population of about 105 million. These Phare E&E programmes represent a total financial commitment of some Euro 570 million, or more than 7 per cent of the Phare programme, of which Euro 350 million (61 per cent) in Environment and Euro 220 million (39 per cent) in Energy. This evaluation is part of the 1998/1999 work plan of sector evaluations under the Phare Evaluations Programme, conducted by the Evaluation Unit of SCR. The immediate objective is to improve the design and preparation of future E&E programmes, given the current reorientation of the Phare programme proposed by the Commission. The wider objective is to contribute to improving the performance of Phare-financed E&E programmes. This report builds upon findings from the Inventory Phase under this evaluation, presented in an Inventory Report. This is available upon request. A sample of 31 projects has been selected from this inventory for in-depth evaluation. The 31 Project Evaluation Summaries are presented in Appendix A to this Report. 1.2 METHODOLOGY AND APPROACH

An overview of all Phare-financed E&E programmes and projects was not readily available and had to be compiled, absorbing about 40 per cent of the resources for this evaluation (see Inventory Report). Individual projects within programmes were classified by both output and sector objectives. By output, a distinction was made between institutional strengthening and investment projects. Institutional Strengthening projects produce proposals for institutional reforms as outputs. Their objective is to actually achieve the reforms and improve the performance of the institutions concerned. Reforms are further classified by four areas: organisational reform; pricing reform; development of regulations; awareness building. Investment projects produce more tangible physical and financial outputs, in the following areas: demonstration projects; financing of physical assets; investment funds; and investment preparation assistance. Their objective is to achieve some direct economic benefits, or demonstration of these benefits. For the classification of projects by sector objectives, the following six categories have been identified: energy savings; improved energy supply; reduced waste and water use; improved waste services and water supply; pollution reduction; and bio-diversity conservation. From the 792 projects in the inventory, a sample of 31 projects was then selected for detailed evaluation. Selection criteria were designed to achieve maximum coverage of variability of types of project financed by Phare, whilst selecting individual projects on a random basis. The sampled projects were evaluated in accordance with the evaluation criteria in the logical framework approach: efficiency, effectiveness, impact and sustainability. The 6

evaluation criteria of relevance verified whether individual project design was coherent with the overall programme and policy level objectives as well as with the comparative advantages of the donor organisation. Scoring of evaluation criteria was done on a scale from 1 to 5: 5 = excellent, 4 = good, 3 = satisfactory, 2 = moderate, 1 = poor. 1.3 PROGRAMME DESCRIPTION

The inventory shows considerable variation in total Phare E&E allocations per capita, as well as in the percentage of Phare national programmes devoted to E&E. Lithuania, Estonia, Slovakia and Bulgaria have the highest Phare E&E allocations. In Lithuania and the Slovak Republic, this is accounted for by high allocations to both energy and environment. In Bulgaria, energy dominates, whilst Estonia has a relatively high environment allocation. Amongst the countries with relatively low overall E&E allocations, this arises from the absence of significant energy programmes in Poland and the Czech Republic and the absence of large environment programmes in Slovenia and Romania. There is considerable variation across countries in sub-sector budget allocations too. In general, pollution reduction was the main priority, with the notable exception of Slovenia. Other priorities have been water services, energy supply and energy saving with about 20 per cent of the Phare E&E budget each. Bio-diversity has received only a minimal share of the E&E budget. From the point of view of types of outputs, Phare supported 341 institutional strengthening projects, almost half (171) of which were primarily concerned with reforming or strengthening public organisations. Nearly 100 of the institutional projects were concerned with strategy formation, covering a wide range of issues, often including economic and regulatory policy, as well as investment prioritisation. 28 projects were concerned primarily with regulations and 39 with public awareness. Very few projects were primarily concerned with pricing, although pricing issues featured in many strategy projects. Phare also supported 310 investment-related projects, of which more than one-third was primarily concerned with the provision of physical assets and another third with the provision of demonstration projects. The remainder focused on investment funds and investment preparation. 1.4 Table 1 RELEVANCE OF PHARE E&E PROGRAMMES Project Scores
Average Efficiency Effectiveness Relevance 4.0 2.8 2.9 4.2 3.3 3.0 4.0 3.1 2.6 4.2 3.1 3.2 4.1 3.1 2.9 Impact 2.6 2.9 2.3 3.0 2.7 Sustainability Average 2.9 3.0 3.0 2.9 3.0 3.0 3.2 3.0 3.3 3.2

Average Investment Projects Average Institutional Projects Average Energy Projects Average Environment Projects Average scores

Relevance to Country Needs In order to assess relevance Phare E&E programme allocations have been compared to E&E priorities in the countries concerned. Overall, relatively strong positive correlation between Phare allocations and the identified national priorities indicates that the programmes 7

matched the needs, in spite of the fact that there was little formal strategic prioritisation for most of the period concerned. The high needs relevance ratings given confirms this (see Table 1). However, Phare-financed National Environmental Action Plans and National Environmental Strategies took many years to complete and were often not in place until at least 1996. Programme Relevance: Appropriateness and Comparative Advantage Energy Efficiency. Energy efficiency projects showed considerable variation in their appropriateness but were generally good. The institutional strengthening projects and the energy efficiency fund project were found to be appropriate, but the two investment preparation projects and the demonstration project were rated as only moderately appropriate. Phare's comparative advantage in energy efficiency projects was rated as very high in most of the institutional strengthening projects and as good in the demonstration projects. The average comparative advantage score for the projects sampled in this sector is 4.0. However, the fund and investment preparation projects were assessed as having only satisfactory comparative advantage. Energy Supply. The appropriateness of the energy supply projects evaluated was generally good, with their comparative advantage being slightly lower. This is because Phare does not appear to have sufficient leverage in the energy supply sector to have a significant impact on policies affecting organisational structures and pricing, Water and Waste Water. The projects generally provided assistance in a highly appropriate manner to the water sector and in particular to water companies. The assistance Phare provided was both financial support for infrastructure investment and technical assistance to improve the management of the services. The appropriateness was rated as excellent in all the water-related projects except for the Bulgaria tannery effluent project, which was only satisfactory. The projects analysed demonstrated a high comparative advantage for Phare. Air Pollution. The air pollution projects were found to be very appropriate to the context. However, it is difficult to exactly determine the appropriateness of each project addressing pollution reduction, as there are many influencing factors. To promote environmental investment for air pollution reduction by industry needs environmental awareness building, air quality legislation, regulations and enforcement combined with sources of finance. Phares comparative advantage for the air pollution projects was also satisfactory to good. Bio-diversity. Only two projects were evaluated within this sector. Whilst the projects appropriateness on average was satisfactory Phare had a large comparative advantage over other donors in bio-diversity. This is due to the less commercialised aspect of this sector so that Phares issuance of grants is particularly beneficial. This benefit is heightened because of the low level of government budget allocated for bio-diversity. 1.5 FINDINGS AT PROJECT LEVEL

The project performance scores are presented in Table 1.

1.5.1

Institutional Projects

The evaluation shows that there is more variation within the different types of institutional project that between types. This suggests that there are a wide variety of circumstances within each project type and that it is difficult to draw general conclusions which, apply to all projects of a particular type. However, the overall trend is that the scores tend to go steadily lower for efficiency, effectiveness and impact (just below satisfactory). Sustainability is often rated higher than impact because the indirect impact of skill transfer increased the probability of the continuation of activities after project completion. Strategy Projects. The performance of strategy projects has been mixed. Outputs were generally of high quality, but the impact of projects on policy has been disappointing, even when projects addressed issues, which were being actively, discussed and resolved. Some indirect impact on national capacity and policy debate was achieved in limited circles, but this seems to have little immediate effect on policy. The lesson from the strategy projects seems to be that it is difficult for Phare to contribute to policy formation, though more success was achieved in assisting with the implementation of policy. Organisation Projects. Organisation projects were particularly problematic, mainly because they often aimed to influence reforms, which were politically sensitive, before clear policies had been established. As a result, the projects lacked political support, and sometimes lacked the participation of the beneficiary institutions. The projects, which concentrated on strengthening particular technical activities, were more successful than those that addressed larger issues about the role and financial status of organisations. Regulation Projects. The one regulation project, which we evaluated in detail, suggests that regulation projects are similar to other institutional projects, in that it is rare for the recommendations of Phare projects to be adopted directly by governments. However, more limited activities which are related to the implementation of regulations can be effective. Public Awareness Projects. Two of the three public awareness projects evaluated in detail were amongst the most successful Phare projects. The sample projects suggest that the main risk associated with public awareness projects is the possibility that institutional responsibilities are not clearly defined. The projects also suggest that support should be within the context of a continuing publicity programme, often involving NGOs. 1.5.2 Investment Projects

The evaluation shows an overall trend for investment related projects to be of moderate to good efficiency with effectiveness being slightly higher - Investment Preparation is the only output with lower effectiveness and impact slightly lower. Sustainability is on average satisfactory. Physical Assets. Physical assets projects were amongst the most problematic projects. Most of the problems could have been solved by better preparation, to ensure that designs were technically and financially sound, with good participation from beneficiary organisations. Problems were sometimes caused by a lack of complementarity activities, to ensure that incentives exist to make use of the assets. Demonstrations. The majority of demonstrations were based on proven technology already operating in EU member states. All the pilot demonstrations successfully worked in country. Approximately one third of the projects have proven replication, with funding 9

from National funds, municipalities, Phare, industry or homeowners. The demonstration projects were very useful in raising public and investor awareness and in providing valuable physical evidence to polluters on practical means of complying with national legislation and to energy users on lowering costs of energy use. A few projects had no replication potential, as there were no opportunities for using the technologies apart from where demonstrated. It was also found that inadequate attention was paid to dissemination actions in some projects. Investment Preparation. Phare projects to support investment preparation had problems understanding the requirements of the financial sector. This partly reflected the fact that the financial sector itself has few precedents for financing environment and energy savings investments. However, Phare also provided insufficient support to help the financial sector to develop this experience. Funds. Phare has provided extensive support for funds to provide financing for environment and energy savings. There appears to be strong demand for these funds, though this is constrained in some countries by the slow development of legislation to alter incentives. The main problem was in low efficiency in starting Fund operations often resulting from poor integration of Funds and the finance sector. 1.6 RECOMMENDATIONS

Maintain overall approach. At the overall E&E programme level we recommend a continuation of the existing Phare E&E programming policy to be government-led regarding project identification, albeit with some adjustment for each sub-sector. Phare should build on its past strategy of promoting government ownership of Phare country programmes and continue to develop government capacity, particularly for programming and project preparation. In the long-term this will allow Phare support to be transformed into a mechanism which is closer to that used for the EU Structural Fund. Improve risk management. Phare should provide more careful risk management for its E&E projects. The risks identified are often impossible to avoid and Phare projects should develop techniques for minimising the impact of this risk. Aim for programme balance. The objective of allocating 70% of Phare resources to investment should be maintained. However, some resources should still be devoted to complementary technical assistance. This should be achieved by Phare supporting a range of complementary activities through an integrated and balanced programme (including legislation, policy development, enforcement, financing and public awareness). This should be undertaken in coordination with the ECs other financing mechanisms for Candidate countries such as ISPA. Focus on local policy implementation instead of policy making. Phare programming should shift away from national policy development and focus more support for the technical aspects of policy implementation once political commitment at the central level is clear. In view of the success of supporting investment and reform at local levels, Phare should specifically focus more technical support at local decentralised levels (i.e. to local utilities and municipalities, particularly for water and district heating), leaving accession to dominate national policy development. However, effort can and should be made at a project level to increase the influence on political will (see Section 5.2).

10

Increase Phares role as an investment catalyst. Whilst past efforts by Phare to promote investment by national industry and financial sectors have proven difficult for long-term reduction of pollution, further efforts focusing on stimulating investment by the private sector are essential. This includes building working partnerships between government, industry and banks (through forums etc), implementing financial instruments and incentives (e.g. impose significant taxes and pollution charges as current public awareness and enforcement measures are insufficient), building capacity in private banks (through training courses on investing in E&E sectors), supported by project preparation training to industry and utilities and subsidised lending for environmental and energy projects. More detailed recommendations will require a further review of Phares (and other donors) investment preparation projects. Creating an enabling environment. At a country level Phare should strengthen the regulatory environment so that industry has increased financial incentive to invest eg through higher (and fair) pollution charges and associated enforcement. Lack of enforcement cannot be easily rectified. Phare has already supported monitoring projects and now in many countries should shift emphasis towards increasing public awareness. Phare should thus provide TA for implementing PPP at a policy level as well as investment at a local level (for pollution reduction etc). Increase Phares role as a distributor of information and skills. Phare should promote dissemination of its project results and lessons learned between the Phare countries. Phare could facilitate this through a number of ways such as holding regional conferences on technical, policy and managerial case studies and lessons learned from implementing Phare projects, establishing voluntary informal lines of communication between national environment and energy Ministries (eg through forming a website) or by working with the Commissions twinning programme. In-country experience indicates that twinning would increase in value if it could also be between the Phare countries. Whilst experts from Member States will be able to show what needs to be in place, experts from Phare countries will be able to show how the changes can be made. This will allow more advanced Phare country technical policy makers to show the intermediary steps of how to achieve progress, not just where to go. There are also key recommendations made at the sub-sector and project level, which are presented in the main report. The key recommendations for the various types of projects are as follows. Phare should continue to provide physical assets where there are significant externalities (environmental and economic) to be addressed and financing through commercial means would not occur. However, the range of potential assets to be provided will be reduced as ISPA will finance water, wastewater and solid waste projects. Phare will be able to finance pollution abatement and nature protection equipment and facilities. The level of resources allocated to the selection, preparation and appraisal of investment projects has not been sufficient and should be increased. The preparation of investments should include an analysis of complementary activities (including policy reforms, technical assistance and training), and the risks associated with these. Phare should continue to support demonstrations of new technology, equipment and procedures to complement policy reform. Future demonstration projects should focus more on aspects of financial viability and replicability. Phare should increase emphasis on the importance of proactive dissemination of demonstration results, both within the 11

project design and with the responsible organisation after project completion. If Phare is to act as a catalyst for raising financing, then the following types of projects will be increasingly important: energy and environment audits, feasibility studies, and assistance to identify sources of financing and preparing loan applications. Difficulties in the past, particularly with securing financing, need to be overcome. Project designers need to improve their understanding of financial incentives and of the financial sector in order to increase the impact and sustainability by ensuring proper financial support to the subsequent investments. If Phare continues to support - environment and energy Funds it must find more efficient ways of commencing their operations. Designated credit lines to existing National investment funds may be an alternative to setting up new and independent funds. Phare should continue to support public awareness projects. However, awareness raising, without financial incentives, is usually insufficient to change public behaviour. Therefore, future awareness campaigns need to be set in the context of a long-term policy of reform. Phare should change the focus of its strategy projects from broad sector reform and priorities (e.g. energy supply, pricing, NEAPS) to strategies for implementing accession. There should also be more follow-up to recommendations put forward by regulatory and strategy studies, for instance by allocating responsibility to individuals within the beneficiary organisation for pushing forward the recommendations into political debate and implementation.

12

WHAT IS PHARE?
Phare originally stood for Poland Hungary Aid for the Reconstruction of the Economy. It is the European Unions (EU) financial instrument designed to assist the Central and Eastern European countries (CEECs) in their transition from an economically and politically centralised system to a decentralised market economy and democratic society based on individual rights, and to support the reintegration of their economies and societies with the rest of the world and especially with the European Union. It was established in 1989, covering at that time only Poland and Hungary. Within two years the programme was extended, in terms of both countries and budget. By 1997, 13 CEECs have become eligible for Phare support. At the end of 1998, cumulative commitments amounted to EURO 8,918 billion, all in the form of non-reimbursable grants, financed from the EU budget. Being both demand-driven and policy-oriented, Phare has reflected the rate, direction and emphasis of reform programmes in individual countries. In the early stages of the reform process, Phare emphasised critical aid. However, it rapidly moved towards institutional reform activities in support of decentralisation of central government, the transfer of productive assets to the private sector, and related legal and regulatory reforms and institutions. Phare focused on know-how transfers, through technical assistance, for privatisation, restructuring of state-owned enterprises, the modernisation of the banking and financial system, the development of the markets, particularly the labour market and the social safety net, and finally the encouragement of foreign investment. As the reforms began taking hold in a number of countries and the first phase of the legal and regulatory framework came into effect, the requirements began to reflect a mixture of specific transition-related reform needs and pure economic development and investment perspectives. Turning around economies and the restoration of medium-term growth became major objectives. This necessitated more support for the restructuring of the economy with greater emphasis on physical and human capital investments and the changing role and impact of government through public administration reform. Phare took a new turn after the Copenhagen Summit (June 1993) confirmed the prospect of membership for the CEECs and the Essen Summit (December 1994) designated Phare as the main financial instrument to support pre-accession strategies. From 1995 onwards, Phare country programmes added new types of assistance, including technical assistance for the approximation of laws and conformity of standards in the CEECs with those of the EU and more direct financial assistance for investments in transport links, energy interconnections, regional development and cross-border co-operation. Following the publication of the European Commissions Opinions (July 1997) on accession of the candidate member states, the Phare Programme has become accession-driven. From 1998 onwards, Phare programmes are based on Accession Partnerships, which indicate the areas of the Acquis Communautaire (the set of EU legislation and regulations) where candidate member states need to make further progress in order to pave the way for full membership. Phare will provide both technical assistance and investment support to help candidate member states implement their own National Programme for the Adoption of the Acquis.

13

LIST OF ABBREVIATIONS

BG CEEC CR DG1A DH E&E EC ES EU GOS HU IFI LI MoU NEAPs NEK NESs NGO PL PMUs R&D RO SCR SL SMEs SR TA TOR UN

Bulgaria Central and Eastern European Countries Czech Republic Directorate General 1A - External relations with the Europe and the New Independent States, Common Foreign and Security Policy, External Service District Heating Energy and Environment European Commission Estonia European Union Government of Slovenia Hungary International Finance Institutions Lithuania Memorandum of Understanding National Environmental Action Plans National Electricity Corporation National Environmental Strategies Non Government Organisations Poland Programme Management Units Research and Development Romania Common Service for External Relations Slovenia Small and Medium Enterprises Slovak Republic Technical Assistance Terms of Reference United Nations

14

INTRODUCTION

1.1

BACKGROUND

This evaluation report covers all Energy and Environment programmes financed by Phare over the period 1989-1998 in the ten candidate member states to the EU: Estonia, Latvia, Lithuania, Poland, the Czech and Slovak Republics, Hungary, Slovenia, Romania and Bulgaria. Phare Energy and Environment (E&E) programmes represent a total financial commitment of some 573 million, or more than 6 per cent of the Phare programme. This report has been prepared by "Environment Resources Management" (ERM) for the Two other documents are available as a companion to the present report. A comprehensive inventory of all Phare-financed Energy and Environment programmes, compiled in the Inventory Report, which is Project Evaluation Summaries Appendix to this Report. 1.2
ONTEXT OF P

E&E PROGRAMMES

indigenous resources, below world market prices and inadequate revenues. As a consequence there is a widespread and severe inefficiency in the use of energy throughout their eco-systems eroded. Phares Progress and Strategy Paper Environment to the year 2000 states that an estimated quality of the environment in CEECs. The EUs environmental policy, derived from the Treaty, aims towards sustainability based on the integration of environmental protection environmental damage at the source, and shared responsibility. During the early years of Phare, shortly after the fall of the Communist regimes, reform in years of adjustment to market economies. Furthermore, the environment was seen as a relatively new field with rapidly evolving experiences and standards. policy for the EU, were clearly important in shaping decisions in Phare countries. However, the prospect of EU membership was decided at the Copenhagen Council meeting politics of Phare countries. Accession requires countries to comply with basic legislation governing economic and civil market, specific attention is given to energy integration and the need to promote equal access to efficient energy supplies. Environmental concerns also form an important part of

15

candidate countries, and because firms across the EU must be exposed to the same environmental standards, in order to ensure fair competition. The major EU directives for the environment set standards for air and water quality and bio-diversity. Key elements of the energy Acquis comprise Treaty provisions and secondary legislation particularly concerning competition and State aids, internal energy market (including directives on electricity, price transparency etc.), nuclear energy, as well as energy efficiency and environmental rules. Development of trans-European energy networks and support for energy R&D are other important elements of energy policy. In July 1997, the EC published its opinion on each countrys progress with respect to requirements for membership, both in terms of economic criteria and their respective capacity to assume the Acquis. These include obligations for energy and environment. The Opinions describe and assess, amongst others, the current and prospective nature of the energy sector and environmental situation in each country. They make conclusions for each countrys prospects and time-frame for approximation of the energy and environment legislation to the EC Acquis. The conclusions include key sub-sectors (e.g. the water sector) in which countries will find most difficulty achieving compliance with EC Acquis. 1.3 CHANGES IN PHARE STRATEGIES

Since 1989, the Phare programme has provided assistance to overcome the problems, first of transition and later on, accession to the EU. In the early 1990s, Phares approach was driven by the problems of economic, institutional and political transition. Phare essentially responded to government policies and proposals: it was demand-driven. Each country takes the responsibility for running its own programmes. The result of this strategy is that each country has had a separate history of Phare funded projects tailored to the national context yet within a strategic framework. When the objective of accession to the EU was endorsed at the Copenhagen Summit in 1993, Phare gradually evolved towards as accession-driven programme. Financing focused on facilitating the achievement of the criteria for accession, and in particular the adoption of the Acquis Communautaire by the candidate member states. As a result, the overall Phare objectives were now fixed, rather than left to individual country policies and priorities. This increasing focus on Accession finally led to the ECs adoption of a pre-accession strategy, cristalized in the Phare Programme Guidelines adopted in March 1997. These guidelines confirm Phares overall mission to facilitate accession and re-focus programme components on specific accession issues. The Guidelines also propose a split-up in the budget between investment support (70 per cent) and institution building (30 per cent), to be applied from 1998 onwards. While the institution building component will focus primarily on adoption and implementation of specific components of the Acquis, the investment component will provide finance for investments in transport and environment. The latter will of course facilitate implementation of environmental regulations contained in the Acquis.

16

The present evaluation report basically covers Phare E&E programmes designed and been carried out with a foreward-looking view, to produce recommendations that could usefully be applied within this changed policy environment.

EVALUATION

BJECTIVES

This evaluation is part of a series of sector evaluations being undertaken in the framework evaluation reports are public documents, disseminated on a wide scale and available on the Internet1. future E&E programmes. The proposed by the Commission. The is to improve the design and preparation of are to contribute to improving the

and accountability regarding funds allocated to the Phare programme.

S ) applied as

The report starts with a description of the evaluation methodology ( Phare E&E programmes (

). Thereafter, the key findings and conclusions of the ), then at a project level ( ). Section 6

actions to the various sub-sectors ( project designers.

Summary country programme information in Appendix C

Appendix A . The Terms of Reference are

17

METHODOLOGY AND APPROACHES

This chapter explains the way in which the evaluation was set up, in three parts: How the inventory of Phare E&E programmes and projects was done How a sample of projects for evaluation was drawn from that inventory Which methods and criteria have been applied to evaluate programmes and projects Timewise, the evaluation was split up in two phases, the first dealing with the inventory and the sampling of projects, and the second with an in-depth evaluation and report writing. The main steps in the evaluation are shown in Figure 2. Figure 2.1 Evaluation Approach

Collection of Data

Detailed Inventory

PHASE I

Classification of Projects

Database

Analysis of Expenditure Patterns

Selection of Sample Projects for Evaluation

Detailed Project Evaluation

PHASE II Analysis of findings - by project type - sector

Recommendations

18

2.1

THE INVENTORY OF PHARE E&E PROGRAMMES AND PROJECTS

2.1.1

Collecting Information

Before an evaluation can be carried out, an overview of all programmes to be evaluated is required. Such an overview was not readily available and had to be compiled. About 40 per cent of the resources for this evaluation went into this compilation of an inventory. There is no central record of Phare programmes and projects at the EC in Brussels. Under Phares Decentralised Implementation System, programmes are largely managed within the beneficiary countries by Delegations and Programme Management Units (PMUs). Records are also mainly kept there. Only general programme documents are available in Brussels, though for older programmes they may have been archived in the ECs Zaventem archives. A high turnover of task managers also implies loss of institutional memory and often documents. There exists a central database in Brussels, called Desire. This is mainly an accounting tool. It contains financial information on budgets, programmes and contracts, commitments and payments. It contains hardly any meaningful information on the substance of these administrative entities. However, it is a good starting point to get an overview of approved programmes, projects and contracts. Financial records in PMUs and delegations exist, but these are often difficult to obtain for earlier projects. Project reports seem to have been collected largely at the initiative of individual officials in PMUs, delegations and implementing organisations, and in Brussels. As few of these individuals are still in place, it if often difficult to obtain even the most basic project reports. Most of the information was collected by the consultants during field visits to each of the ten countries. The inventory was collected using a standard format, which required information on the coding and title of projects; the main activities; dates of starting and completion; expenditure commitments and disbursements; and outputs. The inventory is based on records of all funds committed to Phare national and multicountry energy and environment programmes between 1990 and 1998, in the ten candidate member states covered in this evaluation. Following classification, recorded projects were coded and entered into a database (operated via Microsoft Access). The database proved to be an effective way of collating the inventory and presenting the results. In total there are 795 projects identified in the database. The inventory is more fully described in the Inventory Report. The full database listing is provided in Annex B.

2.1.2

Classification

Starting from the Desire database implies starting from administrative and financial items such as programme budgets and contracts. These do not necessarily correspond to logically 19

coherent projects, which are the basis for the evaluation. Projects were defined as separately managed actions with distinct outputs. In most cases this coincided with contracts, which also coincided with the various financial records and reports. However, there were some cases where judgement was required on how to interpret this definition. Since the evaluation methodology (see further) starts from the concept of outputs, Phare E&E programmes had to be disaggregated into projects, in order to define meaningful outputs. The remainder of this chapter therefore deals with projects, rather than programmes, unless explicitly mentioned otherwise. In order to develop a framework for comparing similar projects we classified each project by output and sector objectives. The classification was based on the following definitions. Outputs refer to those results of the projects, which are within the control of project management, the point at which the responsibility of project management ends. Sector objectives relate to the ultimate objectives of each project. For the classification of projects by outputs, two key types of outputs are distinguished: Institutional Strengthening and Investment projects: Institutional Strengthening projects produce proposals for institutional reforms as outputs. These proposals should be effectively adopted and implemented in order to achieve the objective of institutional reform. Reforms are further classified in four areas: organisational reform; pricing reform; development of regulations; awareness building. Investment projects produce more tangible physical and financial outputs, in the following areas: demonstration projects; financing of physical assets; investment funds; and investment preparation assistance. Their objective is to achieve some direct economic benefits, or demonstration of these benefits. The reason for making this distinction between both types of project outputs is that they require quite different evaluation approaches and methods (see below). There are a number of projects, which do not fall easily within any of these two categories. These include strategic planning projects, critical aid projects and support for PMUs. Critical aid projects address specific issues at a given time (notably emergency clean-up and import projects). There are also a number of projects with unknown objectives for which insufficient information is available and projects with multiple objectives, which do not fall into any one category. They have been completely separated from the categories of institutional reform or investment projects. In the classification of projects by sector objectives, the following six categories have been identified: energy savings; improved energy supply; reduced waste and water use; improved waste services and water supply; pollution reduction; and bio-diversity conservation.

20

The sector objectives are intended to reflect the main areas where progress is required to associated with accession and represent the main gaps where compliance is required. Table 1.1 presents a matrix that classifies all E&E projects in the inventory by type of output projects with organisation strengthening or reform outputs in the energy savings subsector. 668 out of the 792 projects fit into one of 66 project types. The remaining 127 projects unknown. Table 2.1 Numbers of Projects by output and sector objectives
SECTOR Environment Improved Reduced Improved Reduced Bio- diversity Unknown Total Pollution Conservation /Multiple Energy Supply Waste/ Waste Waste/ Use Water Supply

Output

Energy Energy Saving

Institutional. Strengthening Organisations Pricing Regulations Awareness Sub-total Investments Physical Assets Demonstrations Investment Funds Investment Preparation Subtotal Other Strategy Critical Aid Project Management Unknown/Multiple Subtotal Total

24 0 2 16 42

74 3 16 0 93

2 0 1 0 3

10 0 1 3 14

48 2 4 10 64

2 0 1 2 5

11 0 3 8 22

171 5 28 39 243

4 11 20 31 66

11 11 2 43 67

0 1 0 1 2

26 1 1 12 40

53 11 9 28 101

18 0 0 5 23

3 0 3 5 11

115 35 35 125 310

10 0 1 1 12 120

30 5 4 2 41 201

0 0 0 0 0 5

11 0 0 0 11 65

22 13 14 5 54 219

5 0 0 1 6 34

20 0 82 13 115 148

98 18 101 22 239 792

2.2 2.2.1

SAMPLING Sample selection process

The evaluation could not possibly evaluate in detail all 792 projects in ten countries. A representative sample had to be drawn from the population of projects. Our primary objective in selecting the project sample from the population of 795 projects, was to achieve maximum coverage of variability of types of project financed by Phare, 21

whilst selecting individual projects on a random basis. To achieve this we adopted a highly stratified random sampling approach. Sampling was based on the classification matrix in table 2.1. above. The rows and columns unknown, and other were excluded from the selection, except for strategy projects. We selected one project from each of the remaining 24 relevant project types (cells in table 2.1. with more than 5 projects). In addition, a second project was selected from the three project types with more than 45 projects, since these large categories do contain considerable variability. In each of these categories, projects were selected at random, using randomly generated numbers, to avoid bias. The selected projects were then screened to reject any projects for which implementation is not sufficiently advanced to generate outputs. We also rejected any projects with expenditure of less than 50,000 Euro, as these were deemed too small. Finally, we excluded multi-country projects, since it was felt that evaluation of these would inevitably focus on the value of the multi-country approach. This is not a major interest of the evaluation, particularly as the EC has already decided to reduce the use of this approach: individual countries will become Member of the EU, not multi-country entities. The screening led to the rejection of 4 projects. These were replaced by a further random selection. Additionally some modifications were made during field visits where further analysis showed that projects were different from their inventory classifications or data was unavailable. This led to 4 projects being reclassified or replaced. Finally, three additional projects were added to the sample, when the work programme in countries unexpectedly created the opportunity to study an additional project. Altogether, the sample of projects evaluated in detail is composed of 31 projects. This was accepted as a feasible number, taking into account the resources available for this evaluation and the minimum time required to undertake a meaningful evaluation of each project. Additional data has been collected on Fund and demonstration projects.

22

Table 2.2 presents the final sample of projects. 2.2.2 Sample representativeness

The sample of 31 projects represents about 4 per cent of the total number of projects and Phare financial commitments in E&E. However, from a statistical point of view, sample representativeness is not determined by the percentage coverage of the population of projects but rather by the extent to which the sample covers variety in the population. The sample selection method has ensured maximum coverage of that variety. The representation of financial commitments in the project sample could have been larger. However in order not to bias against smaller projects such as institutional strengthening and in an attempt to increase the variety of projects covered we followed the selection method detailed above. This led to a relatively low level of representation of overall expenditure. Table 2.3 shows the sample representation by output and demonstrates that there is no significant bias in the representation of specific objectives. The main discrepancy is that the sample only covers 3 per cent of expenditure on physical assets, despite containing 6 per cent of the projects. This has occurred because the random process happened to select 7 relatively small physical assets projects.

23

Table 2.2
Sector

Project Sample by Type


Country Prog. Title Expenditure (M ) 2.54 1.20 0.20 0.68 0.10 0.19 0.33

1 2 3 4 5 6 7

Physical Assets ENE BG ENE HU ENV BG ENV ES ENV BG ENV HU ENV HU Demonstrations ENE RO ENE RO ENE BG Investment Funds ENE LI ENV CR ENV LI

9601 9002 9408 9408 9408 9002 9002

Port dredging to improve energy supply Sewer system installation Equipment for regional labs Lab equipment Tannery effluent technology (1) Equipment/facilities for national parks Feasibility study for hazardous waste

8 9 10

9504 9504 9307

Energy Services (2) Hospital Energy Efficiency (2) Improved district heating systems

0.6 0.25 3.01

11 12 13

9403 9002 9503

Seed capital for energy efficiency fund Loans for investments in municipalities Environmental Investment Fund (3)

3.00 0.40 2.3

14 15

Investment Preparation ENE SR 9202 ENE PL 9213 Cleanup Project ENV BG Strategy Projects ENE SL ENE PL ENV RO ENE SR ENV ES Organisations ENE RO ENE LI ENE CR ENE ES ENE ES ENV SR Regulations ENE LI

Energy audits Financial survey of energy companies

0.35 0.16

16

9310

Bay of Vromos Clean-up (3)

0.76

17 18 19 20 21

9302 9102 9105 9406 9408

Energy Efficiency Strategy Permit Costing Plan for National Monitoring System Energy production database Master plan for pollution monitoring

0.57 0.20 1.19 0.05 0.44

22 23 24 25 26 27

9105 9302 9303 9623 9504 9503

Restructuring ARCE LSPS restructuring study Support to MIT in energy policy Water and waste service (County Water) (4) Water Network TA National pollution monitoring

0.44 0.30 0.08 0.90 0.30 0.29

28

9302

Regulations for decentralised district

0.17

24

Public Awareness ENE BG 9207 Demand reduction campaign 0.25 ENV HU 9102 Bio-diversity Awareness (5) 0.18 ENV CR 9102 NGO Awareness (3) 0.45 (1) This was originally classified as a demonstration project (2) Both projects were originally under one title as RO9504 Demo projects on energy efficiency. (3) These were additional projects (4) This project was originally classified as an Investment project (5) The project originally selected (HU 9102.403) was wrongly categorised as a Public Awareness project. This alternative was therefore chosen. 29 30 31

Table 2.3

Representativeness of the Sample


Total M Sample M 2.31 0.0 0.2 0.9 3.4 5.24 3.9 5.7 0.5 15.3 0.76 0.0 0.0 2.45 2.45 21.15 Sample % of Total No. M 3% 0% 1% 5% 3% 3% 20% 17% 1% 5% 0% 0% 0% 4% 2% 4% 4% 0% 4% 8% 4% 6% 8% 9% 2% 5% 0% 0% 0% 5% 2% 4%

No. 171 5 28 39 243 115 37 35 125 312 18 101 23 98 240 795

No. 6 0 1 3 10 7 3 3 2 15 1 0 0 5 5 30

Institutional Strengthening Organisations Pricing Regulations Awareness Sub-total Investment Physical assets Demonstrations Financing Investment preparation Sub-total Other Critical aid Project management Multiple/unknown Strategies Sub-total Total

81.0 1.9 11.1 16.4 110.3 198.4 19.3 34.3 49.2 301.2 34.4 39.5 17.5 70.4 161.8 573.3

Geographic representation was not part of the selection criteria as the main interest of the study is to evaluate the Phare programme as a whole. However, a sample of 31 is likely to provide a reasonable geographic spread, which enables some analysis of the importance of the national context. Table 2.4 summarises the sample by specific objective and region. The regions are divided into Baltics and Poland, the Central Regions (Hungary, Czech Republic, Slovakia and Slovenia) and the Black Sea countries (Romania and Bulgaria). The overall distribution of the project sample is fairly equal with slightly fewer projects located in the Black Sea countries.

25

Table 2.4

Summary of sample projects by output and geographical region


Baltic/Poland 3 0 1 0 4 1 0 2 1 4 0 2 10 Central Region 2 0 0 2 4 3 0 1 1 5 0 2 11 Black Sea 1 0 0 1 2 3 3 0 0 6 1 1 10 Total 6 0 1 3 10 7 3 3 2 15 1 5 31

Project specific objectives Institutional Strengthening Organisations Pricing Regulations Awareness Sub-total Investment Physical assets Demonstrations Investment Funds Investment preparation Sub-total Other Clean up Strategies Total

2.3

FIELD EVALUATIONS

Each of the sample projects was evaluated in more detail during a period of about 1 week in the country concerned. These evaluations were done by a team of 5 evaluators, using the standard report structure presented in annex 2. As part of the standard procedures, evaluators were required to provide scores to summarise their opinions of project performance against each of the evaluation criteria (see logical frameworks below). Consistency between evaluators was maintained by detailed preparatory discussions and by providing standard guidelines for evaluations. This led to a relatively clear comparability of issues across projects. However, it was more difficult to ensure consistency in the allocation of scores. Most of the sample project evaluations had difficulty in obtaining objective information about projects. This meant that evaluations often had to rely on interpreting the opinions of key informants. It was often difficult to identify informants who were familiar with projects but did not have some involvement; which would encourage a particular perspective. The main sources of information for drawing conclusions and recommendations are the inventory and sample evaluations. The inventory provides evidence of the main trends in the Phare E&E programme. It contains information on the intended outputs of each project, which are used for classification. It cannot inform about project relevance and performance. However, any indications in the inventory of the actual achievement of outputs are inevitably based on very rapid assessment and the experience of the sample evaluations suggests that such assessments are not reliable in judging the performance of a project. Thus, the main sources of information on the performance of projects are the sample project evaluations. Conclusions were drawn primarily by comparing the experience of as many projects as possible. This process has been largely inductive, as we have attempted to avoid bringing to the evaluation any preconceptions about the programme.

26

In addition to the inventory and sample evaluations, we collected background information about problems, policies and activities in each country, both during the inventory and the evaluations, which has provided important information on which to assess relevance. 2.4 LOGICAL FRAMEWORKS

The evaluation follows the logical framework approach, as set out in the ToR for this evaluation. The logical framework has become the de facto EC standard for project design and evaluation in many areas of EC activities, including external aid programmes such as Phare. Logical frameworks are simply a system of, mostly chronological, classification of events in a project (see Table 2.4). They identify inputs, outputs, immediate or specific objectives and overall objectives. These have been defined for investment and institutional reform projects. Each of these is associated with verifiable indicators, along with the source from which the indicators can be obtained. Judgement was used in defining the boundaries between outputs, specific objectives and overall objectives. We followed the principle that outputs are under the control of project management. Specific objectives relate to the implementation of the outputs by authorities or events outside the control of the project management. Overall objectives relate to the impact of the project once its outputs are effectively used. Table 2.4 Logical framework and evaluation criteria
Evaluation criteria: Indicators: Long-term viability factors of infrastructure and reformed institutions Investment: economic benefits Institutional reform: changes in the behaviour of target organisations Impact Investment: operational infrastructure Institutional reform: proposals adopted, human resources employed Effectiveness Investment: infrastructure, equipment Institutional reform: proposals for reforms, draft laws, trained human resources Efficiency Investment: finance, equipment, Institutional reform: technical assistance Relevance Policies and needs, comparative advantage appropriate design, complementarity,

Project Logic: Sustainable over time

Wider or objectives overall

Sustainability

Immediate or specific objectives

Outputs

Inputs

Design

2.5

EVALUATION CRITERIA

The criteria used for this project level evaluation can be readily derived from the logical framework. Each criterion defines the performance or achievements in terms of transition from one phase in the project to the next: 27

Efficiency measures the level of inputs required to achieve outputs. Efficiency can sometimes be measured by cost benefit analysis, through estimation of IRRs, cost effectiveness or cost/benefit ratios. However, it is often difficult to quantify benefits, as required in this type of analysis, and efficiency may require a more subjective form of comparative analysis about the quality of the outputs. Effectiveness measures the achievement of specific objectives such as the effective utilisation of infrastructure investments, and the adoption and implementation of proposals for legal and organisational reform, or trained human resources. Impact measures the achievement of overall objectives in terms of the economic benefits derived from investment projects and changes in institutional performance as a result of institutional reform outputs. In theory, specific and overall objectives should be defined at levels where it is possible to isolate the effects of the project from other events. In practice, it may be difficult to observe the effects and it can be necessary to rely on inference from observed outputs. Sustainability is estimated by assessing whether the impact of projects can be expected to continue into the future.

The evaluation criterion of relevance ensures a sort of bridging function between project and programme level evaluation. It verifies whether individual project design or set-up was coherent with overall programme and policy level objectives as well as with the relative advantages of the donor organisation. Relevance therefore groups a number of related elements: The extent to which programmes and their constituting projects were relevant with respect to the policies and needs of the country, region or sector targeted and the extent to which priority needs have been met; The extent to which the design of the programme and its various projects was appropriate to achieve the objectives; The extent to which the donor, in this case Phare, had a comparative advantage in funding the programme and its project, both in terms of its own financial instruments (in the case of Phare, grant subsidies) and the skills and knowledge that the agency can offer. All programmes and projects evaluated are score on a scale from 1 to 5: 5 = excellent, 4 = rather good, 3 = satisfactory, 2 = rather poor, 1 = poor. 2.6 GENERIC LOGICAL FRAMEWORKS FOR PROJECT EVALUATION

Though the logical framework is a standard method for design of programmes within the EC, and within the Phare programme, logframes are often not available, especially not for the earlier programmes. Even when they are available, they are often of questionable quality and hard to use for evaluation purposes: outputs and objectives are defined in a very general or ambitious way so that they become unverifiable. Consequently, evaluators often needed to (re)construct their own logical frameworks, a posteriori to the project. Furthermore, in a wide-ranging evaluation exercise such as the present one, and in order to ensure comparability between different project evaluations, it is important to define evaluation criteria similarly across all projects. For these reasons, we developed generic logical frameworks for each type of project, institutional strengthening and investment. The evaluation thus took place against generically defined outputs and objectives. These may 28

not necessarily correspond to the outputs and objectives as defined in the work programme documents. Figure 2.1 Logical frameworks for Institutional Strengthening Projects
Indicators Organisational output, profit Fulfilment of the organisations objectives Decisions by the authorities or Boards Personnel records Sources Policy statements Further legislation Budgets Legislation Reports from government, agencies and others Trainee tracer studies Rapid beneficiary assessment Reports

Project logic Overall objectives Improved performance of the target organisation Specific objectives Adoption and implementation of proposals Application of training in job Outputs Proposals for organisational change, legal drafts Trained human resources Input TA , equipment Some supporting finance

Quality and scope of proposals Quality and quantity of training Expenditure

Project records

Figure 2.2

Logical frameworks for Investment Projects


Indicators Output, profit Emissions Sources Recipient organisations: accounts, reports, interviews

Project logic Overall objectives Improved output, profits Reduced pollution Loan repayments Specific objectives Use of new techniques Approval of loans/grants Replication of demonstration Output New techniques Financial systems/funds Loan/grant applications Studies Input TA, plant, equipment

Capacity utilisation Number of loans/grants approved New technology sold Number of replications Installation of plant & equipment Applications for funds

Recipient organisations Financing organisations Suppliers

Project reports

Project records

29

PHARES E&E PROGRAMME

This section presents the findings and conclusions of the evaluation at a programme level. First, it describes the programme and considers issues related to Phare programming. The section then considers the relevance of the E&E programme. Because of the varied nature of the energy and environment sectors, they are further divided into sub-sectors. 3.1 3.1.1 PHARE E&E PROGRAMME DESCRIPTION Country and Sector Allocations

Tables 3.1 presents basic programme information about the Phare E&E programmes. Table 3.1
Country Energy Bulgaria Czech Rep. Estonia Hungary Latvia Lithuania Poland Romania Slovakia Slovenia Multi-country Total Environment Bulgaria Czech Rep. Estonia Hungary Latvia Lithuania Poland Romania Slovakia Slovenia Multi-country Total Grand total

Phare E&E Programmes (1990-1997), ranked by relative importance


Number of programmes 8 4 4 5 4 8 4 4 5 3 5 55 4 4 5 7 6 9 5 3 5 0 5 55 110 Year of first 1991 1991 1992 1991 1993 1991 1991 1991 1991 1993 1992 Year of last Number of projects 59 55 18 26 22 48 21 50 50 18 33 400 32 39 20 88 25 19 71 10 35 Committed (M ) 70.8 12.3 5.2 16.5 2.8 21.0 7.5 29.5 11.0 7.2 42.6 226.4 28.2 20.7 10.8 69.7 12.8 19.7 88.8 15.5 16.2 0.0 64.6 346.9 569.38 % of Phare country programme 12.0 1.5 4.0 2.0 2.0 9.0 0.5 3.5 6.0 6.0 5.5 2.8 4.5 6.0 10.0 9.0 7.5 8.5 6.0 2.0 9.0 0.0 8.0 4.5 7.3

1996 1997 1996 1996 1997 1997 1994 1995 1995 1995 1996

1990 1990 1992 1990 1992 1992 1990 1991 1990 1997 1991

1994 1997 1997 1995 1997 1997 1995 1996 1995 1997 1997

1
55 395 826

There have been various exercises to calculate the allocations to the Czech and Slovak Republics before independence. The figures quoted here reflect PMU calculations. Note: differences between levels of allocated and committed expenditure tend to be minor, allowing comparison between them. Source: ERM Inventory Report, 1998.

Overall financial commitments under Phare E&E programmes over the period 1990-98 amount to nearly 570 million, of which 350 million (61 per cent) are in Environment

30

and 220 million (39 per cent) in Energy. This represents 7.3 per cent of the total Phare programme over that period. In Figure 3.1, the distribution of E&E allocations across countries from Table 1 has been put in a graphic format. It presents both per capita (right-hand axis) and percentage allocations (left-hand axis) by country. It demonstrates the considerable variation in total Phare allocations per capita, as well as in the percentage of total Phare spending devoted to E&E (the correlation coefficients between per capita and percentage spending are 0.23 for environment and 0.37 for energy). Lithuania, Estonia, Slovakia and Bulgaria have a high overall E&E allocation. In Lithuania and the Slovak Republic, this is accounted for by high allocations to both energy and environment. In Bulgaria, energy dominates, whilst Estonia has a relatively high environment allocation. Amongst the countries with relatively low overall E&E allocations, this arises from the absence of significant energy programmes in Poland and the Czech Republic and the absence of large environment programmes in Slovenia and Romania. Figure 3.1 Country distribution of E&E allocations (1990-1997)
20% 18% 16% % of Total Phare 14% 12% 10% 8% 6% 4% 2% 0% Slovak Rep Slovenia Romania Latvia Lithuania Hungary Bulgaria Czech Rep Estonia Poland 100 80 70 60 50 40 30 20 10 0 Total Phare (EURO/capita) 90

Energy %

Environment %

Total /capita

3.1.2

Expenditure by Sector Objective

In this section we move down from the overall country allocation level to the allocation of E&E expenditure in each country by sub-sector. The objectives in each sub-sector are the following: Energy Saving. The objective of public intervention for energy efficiency is straightforward and involves reducing energy costs, natural resource use (mainly oil and gas) and associated pollution. Intervention normally takes the form of a range of economic reforms and promotional activities, with important overlap with environmental activities. Energy Supply. Support has been to restructure institutions and reform the regulatory framework in order to establish competitive markets and increase security and diversity of supply. Issues of increased efficiency, lower costs and reduced pollution are also important. 31

Water Services. The water sector faces many environmental problems. The main ones, which face Phare countries seeking integration into the EU are related to achieving EU water quality standards. Improvement of prevalent poor quality groundwater and river and coastal waters is a main priority. Effort is mainly being targeted toward improving the overall functioning of the water utilities. Additionally in areas of high industrial pollution assistance is directly being channelled into regional water resource planning and wastewater reduction. Pollution Reduction. Air quality improvement is a priority in many countries, particularly in traditionally industrial areas. Often industry lacks the incentive to reduce pollution because enforcement of existing regulatory systems is weak and fines, if imposed, are much less costly than investing in pollution abatement measures. Therefore one objective of Phare has been to develop national regulatory systems to promote pollution reduction amongst industry. The systems are trying to adopt tools such as standards, permits, monitoring programmes and fines. The other goal is to provide information about the technological options available for industry to adopt to reduce their pollution. Once the regulatory system is effectively working and industry is aware of potential pollution abatement investments it can make a key goal is to provide available financing for the necessary investments. Bio-diversity. Phare's bio-diversity objectives are covered by the two main EC directives aimed at creating protected areas, the Birds and the Habitats directives. The key objectives are to establish sites for protection and ensure they are managed and protected to certain standards. Table 3.2 presents the allocation of Phare E&E funds across these sub-sector objectives: Table 3.2
Lithuania Latvia Estonia Poland Czech Rep Slovakia Hungary Slovenia Romania Bulgaria

Phare E&E allocations across sub-sectors, by country


Energy Energy Water Pollution Bio-diversity Other2 Saving Supply Services1 Reduction 13% 34% 33% 11% 2% 7% 9% 38% 26% 0% 12% 11% 34% 33% 0% 2% 1% 23% 32% 23% 24% 13% 3% 47% 0% 24% 12% 9% 32% 2% 9% 7% 27% 50% 3% 63% 19% 0% 0% 0% 16% 46% 17% 18% 0% 5% 55% 0% 28% 0% 3% 7% 20% 10% 19% 13% 21% 4% 18% 3% 12% 10%

Overall average 20% 21% 18% 28% 1 Includes waste water (industrial waste is included in pollution reduction) 2 Includes projects classified with multiple or unknown objectives Shaded cells indicate most important allocations for each country Source: ERM Inventory Database

Table 3.2 reveals that pollution reduction was, for most countries, the main priority, with the notable exception of Slovenia. Next come water services, energy supply and energy saving with about 20 per cent of the Phare E&E budget each. Bio-diversity has received only a

32

minimal share of the E&E budget. diversity2.

Only Poland allocated a substantial share to bio-

There is considerable variation across countries in sector budget allocations. The allocation of budget for water services and wastewater treatment varied greatly between countries. Notably, the Czech Republic, Slovakia, Slovenia and Bulgaria have allocated small amounts to this sector. Energy supply has received particularly high budget allocation in Romania, Bulgaria and Lithuania. Energy saving has been a high priority in Slovenia, and somewhat less in Slovakia and the Czech republic. 3.1.3 Expenditure by Project Output

Table 3.3 shows the allocation of the total Phare E&E budget by type of project output (as defined in Section 2). Table 3.3 Commitment of Programme Budget (M ) by type of project output
Energy Physical assets Organisations Strategy Investment preparation Project management Financing Critical aid Demonstrations Awareness Regulations Pricing Multiple Unknown Total 41.9 38.4 24.3 26.4 17.6 25.0 22.4 12.3 6.2 8.4 1.5 1.8 0.2 226.4 Environment 156.5 42.6 46.1 22.7 21.9 9.4 12.0 7.0 10.1 2.7 0.4 6.4 9.1 346.9 Total 198.4 81.0 70.4 49.2 39.5 34.3 34.4 19.3 16.4 11.1 1.9 8.3 9.3 573.3 % of Total 35 14 12 9 7 6 6 3 3 2 <1 1 2 100

Table 3.3 shows that between 1989 and 1998 35 per cent of the Phare E&E programme budget has been committed to financing the provision of physical assets. Organisational reform and strategy outputs had the next highest shares of total budget with 14 per cent and 12 per cent respectively. All other project outputs each received under 10 per cent of total programme budget. It also shows that 7 per cent of the budget went to management of Phare projects (supporting PMUs, etc.). Figure 3.2 presents allocations by project output but puts them in an evolutionary perspective, covering the period from 1990 to 1997. The graph shows that commitments on strategic studies peaked during the early years and dropped off thereafter. Institutional projects have retained a more stable share of between 10 and 20 per cent of the total annual allocations, particularly after 1992/93. We have included critical aid as a separate line to show a peak in the early years and to distinguish this from investments, which are surprisingly high during the early years. The figure also shows important critical aid expenditure in 1994 and 1996 which is accounted for largely by several large uranium mining clean-up projects, which required substantial preparation.

Note the overall averages are averages of each countrys share by sector of their Phare E&E budget. They do not reflect the total Phare E&E programme budget allocation by sector as each country has different absolute amounts for their E&E programmes.
2

33

This appears to be in line with the evolution in the strategies and priorities of the Phare programme itself. The peak in critical aid in the early 1990s indicates the need to resolve urgent problems related to transition and the fall of Communist regimes. Strategy studies also peaked in the early 1990s, when the level of consensus about priorities and how to tackle these was generally vague. However, countries did address this and allocations for strategy studies have been decreasing steadily since 1993. Institutional reform programmes show a mixed trend, with a peak in 1991 and another in 1995. The first peak is more related to the need to set up basic E&E institutions in the wake of transition and to deal with the most urgent issues. The second peak is probably more related to the fact that Accession became an objective of the Phare programme from 1994 onwards, and specific institutions and reforms related to the Acquis Communautaire had to be put in place. Finally, investments have been high throughout the programme, with the exception of 1993, which contained only 4 projects. There is a general pattern for the early physical asset projects to involve provision of monitoring equipment and for the later projects to involve assets for local services, notably for water supply. However, there are several important examples, which are contrary to this general pattern. Table 3.3 Evolution of components of E&E Programmes 1990-97 (as % of total)
1990 6.5 5.0 8.3 1.1 34.8 13.4 3.4 1991 11.8 33.4 27.1 6.5 26.5 9.3 8.9 1992 0.0 7.8 6.6 2.9 36.4 10.1 13.9 1993 1.4 6.8 2.8 3.1 3.8 16.8 6.4 1994 7.7 6.5 10.4 4.2 27.8 20.9 7.8 1995 1.0 4.5 15.5 6.5 35.4 20.1 6.6 1996 6.0 3.6 7.2 3.6 25.7 8.8 5.3 1997 0.0 2.6 3.0 1.4 8.0 3.4 4.8

Critical aid Strategies Organisation Other Institutional Physical Assets Other Investment Other

Table 1.3 shows the percentage of budget committed in each country for each project output. The figure shows that the allocation of Phare E&E budget by project output has been a relatively constant across the different countries. The main exception to this pattern is the high variability in resources devoted to strategies, with nearly 30 per cent in Romania and barely 10 per cent in Bulgaria and Poland. The graph also shows the high concentration of critical aid in Bulgaria and Poland. The generally stable ratio of investment to institutional strengthening expenditure across countries suggests that, at an aggregate level, all countries require a similar balance of project types, despite the different problems faced by the countries and the different sectors in which resources were concentrated.

34

Table 3.4

Budget Commitment by Project Output in each Country (in per cent)


BG CR 0.0 2.3 4.5 24.7 1.5 100.0 ES 0.0 2.8 3.3 7.7 2.2 100.0 HU 0.0 4.6 10.9 65.0 5.8 100.0 LE 0.0 0.1 2.2 10.2 3.1 100.0 LI 0.1 2.8 5.4 28.8 3.7 100.0 PL 8.4 18.7 11.9 45.3 12.0 100.0 RO 0.0 1.4 13.2 28.8 1.5 100.0 SL 0.0 1.4 1.8 3.6 0.4 100.0 SR 0.0 3.7 7.6 13.0 2.8 100.0 ZZ 2.0 26.4 38.8 20.2 19.8 100.0

Critical Aid Strategies Institutional Strengthening Investment PMU/Multiple/Unknown Total

24.0 6.3 10.6 53.8 4.3 100.0

3.2

RELEVANCE OF PHARE E&E PROGRAMMES

The remainder of this chapter evaluates Phare E&E programmes according to the Relevance criterion, defined in Chapter 2. Relevance covers three aspects: the relevance of a programme with respect to the countrys needs and policies; the appropriateness of the approach, and the comparative advantage of Phare involvement in these programmes and projects. We discuss each of these three aspects, first at programme level (this section) and then at the level of each of the sampled projects in this evaluation (next section). Table 3.5 Relevance Scores for Sample Projects
Needs Energy Efficiency Romania energy services Romania hospital EE Slovakia energy audits Poland energy finance Romania ARCE support Slovenia EE strategy Bulgaria demand reduction Lithuania EE Fund Average score Energy Supply Bulgaria port dredging Slovakia energy database Lithuania DH decentralisation Lithuania LSPS restructuring Czech MIT support Bulgaria DH Average score Water & Waste Water Hungary sewer system Estonia county water Estonia water network Average score Pollution Bulgaria regional labs Estonia lab equipment Poland instrument costing Romania pollution monitoring 5 5 5 5 5 5 5 5 5 5 5 4 3 2 5 4 5 4 5 4.7 3 5 5 5 Appropriateness Comparative advantage 4 2 2 2 5 5 5 5 3.8 na 5 3 2 5 4 3.8 5 5 5 5 3 4 3 5 4 4 3 3 5 5 5 3 4 2 5 5 2 3 4 3.5 5 4 3 4 4 4 4 4 Average 4.3 3.7 3.3 3.3 5 5 5 4.3 4.3 3.5 5 4 2.3 3.3 4.3 3.8 5 4.3 4.3 4.5 3.3 4.3 4 4.7

35

Bulgaria tannery Czech municipal loans Czech NGO awareness Estonia pollution masterplan Slovakia pollution monitoring Lithuania Envt Fund Hungary hazardous waste Average score Bio-diversity Hungary biodiversity awareness Hungary national park Average score Cleanup Projects Bulgaria bay cleanup Average Investment Projects Average Institutional Projects Average scores

5 5 5 5 5 5 5 4.8 4 5 4.5 3 4.7 4.5 4.6

3 5 5 4 5 5 2 4.2 2 4 3.0 4 3.6 4.3 3.9

4 3 3 4 3 3 4 3.6 4 5 4.5 4 3.7 3.9 3.8

4 4.3 4.3 4.3 4.3 4.3 3.7 4.2 3.3 4.7 4.0 3.7 4.0 4.2 4.1

3.2.1

Relevance to country needs, policies and strategies

In the early 1990s, the Phare countries started the process of economic transition from a centrally planned to a decentralised market economy. This entails a wide range of institutional reforms (legal, organisational) in the E&E domain. In energy and water utilities, this often entailed a decentralisation of ownership (privatisation) of assets, together with the adoption of regulatory frameworks and setting up controlling authorities to supervise the public service aspects of these utilities. All the countries require a comprehensive revision of legislation and a major strengthening of monitoring and enforcement agencies. However, because of the wide economic and financial implications of privatisation and regulation of utilities, this process did not go smoothly. It is still far from completed in many countries. On the other hand, significant investments were required to improve the physical facilities, ensure a continued supply of energy and water, and improve the capacity to manage these services in a more responsive fashion. The often unfavourable financial situation of utilities, at the start of the transition process and as a result of dramatic changes in price structures, required further government financial involvement, at least for a while. Furthermore, all countries required major shifts in public attitude towards the use of energy, water and waste services. Environmental problems were severe in many cases, which could only be improved through investment in clean-up and better environmental regulation. When Phare shifted towards facilitation of Accession and adoption of the Acquis, from the mid-1990s onwards, the focus also shifted to adoption and implementation of more stringent EU standards in the E&E domains. This led Phare to increase the ceiling on investments in infrastructure, including E&E, to 70 per cent of its total budget, from programme year 1998 onwards. However, this is not reflected in this evaluation, since it covers the period to 1997 only. Since then, Phare Opinions and Partnership Agreements, and the National Programmes for the Adoption of the Acquis that followed, have begun to prioritise, in a limited manner, the differences in the level of environment- and energyrelated problems faced by the different countries and have described progress made to date.

36

a)

Comparison of Phare Commitment with Country Priorities

In order to assess relevance in a more quantitative way, Phare E&E programme allocations have been compared to E&E priorities in the countries concerned. Based on these official documents, in-country interviews with key policy makers and reviews of national policy documents, we have compiled for the purpose of this evaluation a diagrammatic summary of the priority problems by sector in each country (table 3.6)(explain in more detail how you did this). Although this was compiled in 1998 only, it is quite obvious that the priorities of problems have hardly changed in the course of the 1990s. Table 3.6 Energy and Environment Needs in Phare Candidate Countries

Energy Energy Water Industrial Pollution BioSaving Supply Services1 Waste Reduction diversity Lithuania Latvia Estonia Poland Czech Rep Slovakia Hungary Slovenia Romania Bulgaria 1 Includes waste water Key: XXXX = critical; XXX = very important; XX = important; X = not a priority Sources: Agenda 2000 for each country, other official documents and evaluation findings

The table allows a comparison of the relative importance of problems in the different subsectors in each country, and of which countries have the most serious problems in each subsector. The main patterns are: in the Baltic countries, water problems are dominant; air pollution is the main problem for the central area of Poland and the Czech and Slovak Republics, largely because of the reliance on high sulphur coal; Hungary and Slovenia have less serious problems, though water is their main problem; and Romania and Bulgaria are concerned both with energy supply and industrial pollution. Table 3.7 examines whether funding allocations matched priorities by comparing level of priority with percentage allocation of total E&E budget. The correlation coefficients between the figures in the table show that 49 per cent of the variation in Phare national resource allocation to different sectors is explained by the level of priority for the sub-sector3. This suggests that sub-sector priorities in each country influenced Phare E&E programme spending. However, there are other important factors determining resource allocation for example, Phare might have provided a relatively modest share of funds to a high priority sector if government or other donors were particularly active in this sector.

The coefficient excludes the exceptional situation in Hungary

37

On average, where sub-sectors had critical priority; they received 35 per cent of national E&E funding allocations, compared with an average of 16 per cent for very important subsectors and 10 per cent for important sub-sectors. The main exception to this rule is the low budget share committed to bio-diversity compared to it being identified as an important or very important sector in all countries. Additional exceptions include the high funding for pollution in Hungary, which was not identified as a high priority. There are also a few examples where a high level of funding occurred in a sector not identified as critical notably regarding energy saving in the Czech and Slovak Republics and in Slovenia. However, in general, the spending on energy efficiency was concluded to be strongly needed because there are often problems in mobilising political commitment for it, both from central government and energy supply agencies. Tension between interests of those who work in energy supply and those promoting energy saving is high where the system was centrally managed and particularly in countries where energy supply is problematic. Table 3.7 Energy and Environment Needs in Phare Candidate Countries

Energy Energy Water Pollution BioSaving Supply Services1 Reduction diversity Lithuania (13%) (34%) (33%) (11%) (2%) Latvia ( (7%) (9%) (26%) (0%) 38%) Estonia (12%) (11%) (34%) (33%) (0%) Poland (2%) (1%) (23%) (32%) (23%) Czech Rep (24%) (13%) (3%) (47%) (0%) Slovakia (24%) (12%) (9%) (32%) (2%) Hungary (9%) (7%) (27%) (50%) (3%) Slovenia (63%) (19%) (0%) (0%) (0%) Romania ( (46%) (17%) (18%) (0%) 16%) Bulgaria (5%) (55%) (0%) (28%) (0%) 1 Includes waste water Key: XXXX = critical; XXX = very important; XX = important; X = not a priority Sources: Agenda 2000 for each country, other official documents and evaluation findings

b)

Comparison of Phare Commitment with Compliance Cost Estimates

The level of priority needs set out above reflects the severity and urgency of current problems. However, this is not necessary correlated with the expenditure required to meet compliance requirements. Compliance costs are the estimated expenditures required to comply with EU regulations in the matter concerned, and thus to satisfy the conditions for Accession, that is adoption and implementation of the environmental Acquis. Compliance cost estimates have been produced by Disae (a Phare Environmental Approximation Facility) and are reproduced in Table 3.2. Table 3.7 compares per capita compliance costs for water & wastewater and for air pollution (the only sectors with available data) with Phare commitments per capita. 38

Table 3.8

Summary of Compliance Cost Estimates (in


Water and Waste Water Phare Compliance Allocation Costs 3.69 144 1.56 758 3.68 577 0.57 265 0.09 689 0.48 660 2.34 1,089 0.00 932 0.33 274 0.01 181

per capita)

Lithuania Latvia Estonia Poland1 Czech Rep Slovakia Hungary Slovenia Romania Bulgaria
1

Air Pollution Phare Compliance Allocation costs 1.20 0 1.05 299 3.48 615 0.80 394 1.50 621 1.62 358 4.31 267 0.00 278 0.35 18 3.26 71

For Poland, air pollution problems include expenditure on vehicle standards Source: Disae (Phare Environmental Approximation Facility)

The correlation coefficients for Phare spending and compliance costs for both water and air sectors are 0.02 and 0.18 respectively showing hat there is no evident relationship between Phare spending and compliance costs. c) Conclusions on Programme Relevance with Respect to Needs and Policies

Overall, the correlation between Phare spending and the identified national priorities indicates that needs were matched by expenditure in spite of the fact that there was little formal strategic prioritisation for most of the period concerned4. The high needs relevance ratings given to most sampled projects, averaging 4.5, in this evaluation confirms this. The low correlation between Phare spending and compliance cost estimates indicates Phare was less influenced by the financial degree of spending needed for each sector in the long-term and concentrated more on nationally identified sector priorities. 3.2.2 Appropriateness of Design of Phare E&E programmes

This section presents the evaluation findings on the appropriateness of the different project designs and outputs, in a sector context. a) Energy Efficiency

There was considerable variation in the appropriateness of the type of project implemented for energy efficiency. The average appropriateness score for the projects sampled in this sector is 3.8. The appropriateness of the institutional strengthening projects and the energy efficiency fund project were found to be good to excellent, but the two investment preparation projects and the demonstration project were rated as only moderately appropriate. A number of key issues explain the variation in appropriateness of energy efficiency projects. The Balance of Support for Energy Efficiency. Progress in promoting energy efficiency requires appropriate developments on a number of fronts, including: effective institutions for promoting energy efficiency;
Phare attempted to address the lack of national priority identification through funding National Environmental Action Plans (NEAPs) and National Environmental Strategies (NESs). However NEAPs generally took many years to complete and were often not in place until at least 1996, leaving only a year or two for follow-on projects to be selected in annual programmes.
4

39

price increases to create financial incentives; regulations governing energy use and emissions reduction; campaigns to raise public awareness; demonstrations to raise awareness in industry; and available funding sources, either from banks and dedicated funds.
The Slovenia EE Strategy project provided support for institutions and regulations, but failed to support changes in prices and did not resolve financing issues. The Romania ARCE Support project provided support for institutions, but real progress in promoting energy efficiency had to wait until funding mechanisms were in place. The success of the Bulgaria Demand Reduction project in promoting public awareness and changing behaviour was dependent on energy price increases, which occurred shortly before the project.

Several of the energy efficiency projects demonstrated the need for progress to be made on all these components before improvements in energy efficiency can be brought about. Where project design did not take into account necessary complementary reforms or activities, the projects were rated as less appropriate which in turn was reflected in lower impact scores. In particular, the projects show the importance of price reforms taking place to reinforce institutional and public awareness activities.

Financing Energy Efficiency. Energy saving studies in numerous Phare countries revealed considerable and commercially viable energy saving investment potential. However this was not being realised because of lack of available financing and lack of information and resources within SMEs. Because investments in energy efficiency do not generally have a high priority in the private sector, it was highly appropriate for Phare to design energy saving funds, which would offer preferential terms and conditions as shown in Table 3.8, to attract demand, mainly by SMEs. Phare has supported a total of eight energy efficiency Funds. Due to the in-country inexperience in designing such funds Phare also supported feasibility studies, as shown in Table 3.8. These set out the institutional and financial arrangements for the Funds, recommending Review Boards, terms and conditions for loans, funding criteria and target organisations. The Funds, as shown in Table 3.8 have consequently been well targeted to channel funds to achieve energy savings and a return on capital. The Funds varied in their rate of loan subsidies. The Czech Municipal Loans Scheme and Slovak Energy Fund had the greatest level of subsidies with variable rates set at 25% and 35% of market rates respectively. Most of the other Funds provide loans at around 60% of market rates. They also provide grace periods as part of the pay back conditions. These subsidies have been sufficient to attract numerous applications, leading to the high effectiveness of the Funds (see Chapter 4). All the Funds have stringent and transparent screening criteria and loan application evaluation procedures, as described in Table 3.8. These ensure that even if the subsidised lending rates are too low and attract applications for investments which should not be financed, the Fund review Boards will only approve loans for commercially viable and energy saving investments. In addition to providing subsidised capital, in many Phare countries it is appropriate for energy efficiency funds to provide additional technical support to potential lenders to build up their capacity to raise financing for energy saving investments. The Energy Efficiency Fund in Lithuania whilst having worked with companies during the feasibility stage to identify potential investments did not have any on-going provision for an outreach programme to inform companies of the source of financing, how to apply for loans or how 40

to audit their enterprises to identify potential energy efficiency investments. The review of the other funds did not ascertain whether they had such provisions. The Czech Energy Saving Fund however did include a promotional campaign for the Fund. The institutional arrangements developed with the banking sector tended to be designed sub-optimally. In most Phare countries, the banking sector has little experience in lending for energy efficiency. Even in countries where the banking sector has grown substantially over the past 10 years, this growth has tended to be in more conventional activities. There has been increasing experience in governments and in the EU Commission in dealing with the financial sector, but this tends to be limited to financial sector work, and is not easily accessible to those responsible for managing Phare support for energy and environment5. The E&E programmes have not benefited from this experience and remain with a limited understanding of the requirements and practices of the financial sector. This past lack of due consideration in designing appropriate institutional arrangements has delayed many Funds from starting operations. This appears one of the main problems for Phare in producing effective financing mechanisms. The Lithuanian Energy Efficiency Fund has taken over two years to start operating as Phare has not been able to reconcile the requirements of the National Bank of Lithuania who will be custodians of the Fund. Other Funds have had a difficult time signing local banks to act as local partners and disbursement mechanisms because banks were not convinced of the financial benefits and levels of risk based on the proposed arrangements set out in the feasibility studies. Support for District Heating. Within energy efficiency activities District Heating (DH) projects were particularly relevant as DH schemes supply energy for up to 20per cent of houses and public institutions in most Phare countries. Most DH plants rely on old plant and local fuels, which generate serious air pollution in areas of concentrated population. Support for DH is thus highly relevant. However, there is still some debate in Phare countries about the extent to which DH rehabilitation should be dependent on the decentralisation of management and ownership. The projects provide a range of evidence on this debate. They suggest that pilot activities can take place with
The Lithuania DH Decentralisation project was relatively successful in supporting a pilot rehabilitation activity (though this has not been used as a model for replication), but was not successful in influencing the policy debate through the strategy study work. The Bulgaria District Heating was designed more as a technical demonstration project, than an investment. However, it did demonstrate that investments can be made in advance of reforms on decentralisation. Several of the gas conversion investments funded under the Czech Municipal Loans project encountered problems arising from the lack of clear regulations over the ownership of the investment. In one of the loans, the problem was the subject of a legal case. Whilst this was a burden on the municipality and gas company concerned, such legal work is probably part of the natural evolution of regulations.

municipalities and local DH organisations, independently of the status of policy debate on decentralisation. Furthermore, they suggest that it is difficult for Phare to contribute to policy reform through technical studies. Campaigns for Energy Efficiency. The high rating of the Bulgarian Demand Reduction project reflects the appropriateness of public awareness projects complementing energy efficiency policies, especially when these include price rises. It also reveals that follow-on public awareness programmes are important, to sustain public awareness and introduce new

In the Poland Energy Finance project, preparation of the Phare project seems to have been undertaken with very little private sector appreciation and very little research about existing activities. As a result, project outputs were of little use to the private sector.
5

41

techniques. However, these routine public awareness activities are likely to be relatively modest and focused. b) Energy Supply

The appropriateness of the energy supply projects evaluated was generally good. The average appropriateness score for the projects sampled in this sector is 3.8. Restructuring and Decentralisation. Improving efficiency of energy supply through reforming and decentralising energy supply systems was highly relevant. However there were problems in finding appropriate ways of achieving a direct impact on energy supply restructuring. There were also problems in finding appropriate ways of supporting decentralisation policy. The Lithuania LSPS restructuring project was poorly designed in that it did not take into sufficient consideration the resistance within the organisation towards its own reform. Whilst the project output recommendations concerning corporate organisation may have been technically correct, but were either not adopted or adopted in a highly modified form as the commitment of the beneficiary was not yet in place. However, projects designed to work directly with local level organisations (e.g. power companies) were found to support the process of decentralisation. This is evidenced in the Bulgaria where the project focused on diversifying and increasing energy supply to one power plant, thus increasing regional sources of power production. Indirect Impact. The appropriateness of Phare support for energy supply is partly determined by the way in which Phare activities influence reforms. Although project recommendations for energy supply reforms do not seem to have been adopted directly, one would expect project reports to be consulted as background material in preparation for negotiations and policy debates. The sample projects do not provide conclusive evidence on this for energy supply projects. The experience of the evaluation in general suggests that Phare can expect to have such indirect influence, but that this needs to be recognised much more explicitly in project objectives, if it is to be given sufficient attention. Private Sector Initiatives. Phare did not attempt to support private sector initiatives in energy supply during the early years of operations. However, this would have seemed an appropriate type of projects to implement. Phare Energy programmes, which have begun to relate to private sector initiatives have yet to yield clear lessons on what types of support are most appropriate for achieving these objectives. Diversity of Supply. The Bulgaria Port Dredging project was designed to increase the capacity of a regional power plant (Varna) to import coal from sources around the world in order to reduce unit costs. The project concept in theory was highly appropriate. However, the impact of the project indicates that the design may not have been appropriate and a more detailed economic and technical analysis informing the design of the project may have been warranted to optimal economic impacts (see Chapter 4). c) Water and Waste Water

The projects generally provided assistance in a highly appropriate manner to the water sector and in particular to water companies. The average appropriateness score for the projects sampled in this sector is 4.5. The assistance Phare provided was both financial support for infrastructure investment and technical assistance to improve the management

42

of the services. The appropriateness was rated as excellent in all the water-related projects except for the Bulgaria tannery effluent project which was only satisfactory. Infrastructure Investment. Many municipalities lack wastewater treatment facilities and have old water supply treatment plants and water supply and wastewater collection networks. The two Estonian water projects addressed the important issues of network rehabilitation to reduce network leakage, which puts a strain on the system, increase the cost to drinking water and pumping, exploit more water resources and create a health risk from leaking sewage. The projects also provided skills transfer so that the water companies could continue to monitor network problems and integrate investment solutions into their planning procedures. Management Assistance. The ability of water companies to address their need for investment in new infrastructure is dependent on their ability to internally generate capital and raise external financing through business planning, increasing revenue and lowering operating costs which are part of institutional strengthening efforts. Therefore it was appropriate for projects to have a comprehensive approach to improving water companies services and to aim at restructuring management and processes so that they would be able to become financially self-sufficient and cover the required investments. The technical assistance for the water companies, which proved most appropriate was in the following areas: identification of operating costs; calculating cost-recovering tariffs; business plan development; organisational restructuring; and billing and collection. These appeared more important for the development of water companies than engineering services because they already had technically proficient engineers but lacked experience in financial and business planning. There was a general lack of business culture and resistance towards market planning. The efforts made by the projects to focus on the upper management in water companies proved appropriate to change the attitude of the management to become more favourable environmentally effective water service. Public Awareness. Whilst it is appropriate to develop the culture of the water companys management there are other parties influencing the future direction and operation of the water company. This is particularly the case when tariff levels need to be increased. Existing attitudes tend to regard water as a common good and therefore there is often strong resistance to raising water prices. Therefore assistance is also needed to educate
The targeting of management as well as infrastructure investment in the Estonian water management project proved successful. The management of the water company learned how to rehabilitate its network but also how to identify the operating costs of the system. From this exercise they realised the potential cost savings from network rehabilitation. This gave them added incentive to focus investment on network rehabilitation and away from the more traditional supply focus.

to running an self-sufficient, efficient and

In the case of the Estonian water management project, the Municipal council owns the targeted water company and makes the final decisions over tariff increases. So even if the project was effective in increasing the water companys capabilities in tariff setting the Municipal council had final approval over tariffs. The Estonian water management project also highlighted the influence of the public in accepting tariff rises which influencing the improvement of municipal water services.

43

and change the attitude of the local authorities regarding appropriate tariff levels and of consumers regarding the nature of water. Whilst Phare did not always address these issues of public perception in the projects analysed, it must be recognised that within individual projects there is a limit to what Phare can address. Industrial Water Pollution. Phare has also funded the provision of wastewater treatment equipment for highly polluting factories. Projects for this type were most appropriate when they acted as demonstration projects to replicate for other similar industries. However, the appropriateness of the Bulgaria Tannery Effluent project was rated as only satisfactory because the project was implemented in a context of loose regulatory enforcement so that other industry had little incentive to invest in similar effluent treatment equipment. Replication. Due to the number of potential beneficiaries in this sector, replication is key in increasing Phares contribution to utility improvement. An important aspect of promoting replication is to select beneficiaries who are sufficiently typical to act as models for other beneficiaries (whilst also being sufficiently advanced to achieve results). It is also appropriate for projects to include a replication component in the project design. The evaluation found that one successful model is to build capacity in consulting which can then be sold to other municipal water companies. The Estonian Network Rehabilitation project was well designed as it supported the Estonian national water company which now provides project related services to water companies throughout the country. This model has been successful mainly because the beneficiary had a financial incentive to replicate the service. d) Air Pollution

The air pollution projects were found to be very appropriate to the context. The average appropriateness score for the projects sampled in this sector is 4.3. However, it is difficult to exactly determine the appropriateness of each project addressing pollution reduction as there are many influencing factors. To promote environmental investment for air pollution reduction by industry needs environmental awareness building, air quality legislation, regulations and enforcement combined with sources of finance. Monitoring and Regulation. It was judged as appropriate to provide monitoring equipment for laboratories in Bulgaria and Estonia. However, it was felt, particularly in Bulgaria that too many resources were being concentrated into monitoring. A typical national view is that reliable monitoring data is a precondition for pollution reduction. However, Phare support needed more balance between financing investment in monitoring equipment and support for regulation, enforcement and directly for pollution reduction. We found little evidence of any source of investment for direct pollution abatement in the countries receiving lab equipment6. Often areas of high pollution are already known and immediate investment would be more appropriate than further data collection. Where the provision of lab monitoring equipment was part of an integrated strategy and a series of recommendations Phare funding was more effective in contributing to mechanisms for addressing the national pollution problems7.

The fact that the Bulgarian lab equipment project does not appear strategically optimal for pollution reduction in Bulgaria indicates that Phare had not sufficiently researched the fundamental problems facing the sector or considered where best it should intervene (as left it to the host government). 7 The Estonia lab equipment project was different as it was part of an integrated strategy. It was one of a set of initiatives recommended by the Phare funded Pollution Masterplan. The other planned initiatives deal with capacity building and promotion of information sharing. Necessary legal
6

44

Financial and Economic Instruments. The environmental funds were determined to be highly appropriate. Many industries are restricted in their capabilities to invest in pollution reduction equipment because of lack of funds. One of Phare's goals is to promote the adoption of instruments, which strengthen the principle of the polluter pays. Many of the strategies and regulatory studies have promoted various economic instruments for pollution reduction (see Section 4 on Funds). These interventions are a highly appropriate element of the range of support required for the sub-sector. e) Bio-diversity

The average appropriateness score for the projects sampled in this sector is 3.0. The Hungarian National Park projects appropriateness was rated as good as investments in national park facilities were appropriate for achieving the sector objectives. The appropriateness of the range of support provided by the project could have been improved by including more income-generating schemes to increase the parks financial independence. The Hungarian Awareness Raising project was only moderately appropriate. The design was poor as the awareness raising was not sufficiently targeted at the park management to stimulate their interest in the issues. 3.3.3 a) Comparative advantage of Phare involvement in E&E programmes Energy Efficiency

Phare's comparative advantage in energy efficiency projects was rated as very high in most of the institutional strengthening projects and as good in the demonstration projects. The average comparative advantage score for the sampled projects in this sector is 4.0. However, in the fund and investment preparation projects, Phare was assessed as having only satisfactory comparative advantage. In view of the combined economic and environmental benefits to be obtained from increased energy efficiency, and the important international implications of these, it is natural that Phare should seek to support energy efficiency. The ECs success in dealing with municipalities and local governments and municipalities suggest that District Heating projects are likely to be of particular attraction to Phare. Phares capacity to provide grants for public awareness campaigns suggests this is one type of project of importance to energy efficiency in which Phare has a particular comparative advantage. Conversely, providing grants for promoting investments, particularly small ones which will not attract IFI interest has proven a comparative advantage for Phare. However, the difficulties encountered in Phare demonstration and investment preparation show Phares technical limitations in effectively promoting investment for energy efficiency. b) Energy Supply

The average comparative advantage score for the projects sampled in this sector is 3.5. The evidence suggests that the Phare programme does not have sufficient leverage to have a significant impact on policies affecting organisational structures and pricing, at least in the energy supply sector. These changes require larger political development, in which IFIs and
development took place in parallel to the project. However, fines are still low and until they rise industry in Estonia will not have a financial incentive to comply with pollution standards.

45

accession negotiations may play an important part. Consequently, few energy supply projects appears to have tried to impact on deregulation and competition.
The Czech Municipal Loans project financed The reform process is often highly investments in gas conversion, despite a lack of complex, involving changing regulations clarity over the ownership of the assets. In several and negotiations over responsibilities for of the municipalities, legal cases had been brought assets and debts. The definition of to try to clarify ownership and responsibility policies is essential in laying the issues. Although these cases are serious problems groundwork for decentralisation. But for the municipalities involve, they should help to there is then a further stage of developing develop the legal precedents which will establish experience and legal precedents in the the practical working relationships between interpretation of policy and legislation municipalities and power supply companies and and in building capacity at local levels. other utilities. When Phare supports local investments at a time of consolidation of decentralisation reforms, this can help to encourage the development of a practical interpretation of legislation.

There are project evidences that local In addition to the advice on regulatory reform, the investment in energy supply which Phare Lithuania DH project also contained a pilot finances can be successful and therefore investment in a local district heating plant. This appropriate, even without regulatory reform. was successfully implemented as a discrete However, the evidence also suggests that investment, despite a lack of regulatory reform, demonstration effects are less easy to achieve although it had little demonstration effect, partly because of the lack of financial and regulatory until reforms are in place. Phare's incentives arising because of delays in reforms. comparative advantage in the energy supply The Bulgaria District Heating project also contained sector is in providing technical studies related local investments, which were primarily designed to the reform debate and in providing support as demonstrations. Whilst the demonstration for local level investments, as part of a impact was limited, for similar reasons to those in broader decentralisation process. This Lithuania, the investments themselves were comparative advantage is related to the role, considered successful. which the EU Commission is likely to play after accession. The reform process also requires substantial capacity building in local management, for which Phare may have a comparative advantage. c) Water and Waste Water

The projects analysed demonstrated a high comparative advantage for Phare. The average comparative advantage score for the projects sampled in this sector is 4.0. Phares comparative advantage in water and waste water is not restricted to assisting large utilities which need major infrastructure investments. Phare can concentrate on smaller utilities, which need management assistance. The projects analysed mostly targeted Phare assistance to these types of utilities. Many utilities cannot immediately charge full rates for new services such as wastewater treatment and thus benefit greatly from a grant element in financing.

46

d)

Air Pollution

Phares comparative advantage for the air pollution projects was also satisfactory to good. The average comparative advantage score for the projects sampled in this sector is 3.6. Funding masterplans and emission inventories needs grants, especially as most Phare countries are keen to limit the costs of air pollution control, which are met by industry. Monitoring networks do not work at this stage based on revenues. However, grants to this field may discourage the development of enforcement of fines, which would finance the labs and motivate industry to invest in pollution reduction. e) Bio-diversity

Phare had a large comparative advantage over other donors in bio-diversity. The average comparative advantage score for the projects sampled in this sector is 4.5. Due to the less commercialised aspect of this sector Phares issuance of grants is particularly beneficial. This benefit is heightened because of the low level of government budget allocated for bio-diversity. In Hungary Phare was the only donor investing in physical assets for bio-diversity.

47

Table 3.9

Energy Savings Funds supported by Phare


CR 9303 B13 Czech Energy Saving Fund M 4.7 directly into fund plus additional support for technical appraisal of projects (M 0.2) and a promotional campaign for fund LE 9704.01.01 Energy Efficiency Fund Provision of Funds (M 2.6 planned), Feasibility Study (M 0.12) and TA for fund management (M 0.1). The fund is currently under negotiation. There will be an initial deposit made to the Ministry of economics, with co-financing from Banks. 1997 SR 9303.02, 9406.02 Energy Saving Scheme Fund M 3.8 in 3 tranches (93 and 94 programmes) for fund initiation and operation SL 9503.01.01 Energy Saving Fund 0.13 M for feasibility study for fund establishment (1994) and 2M for Fund

Phares contribution (financial and technical)8

Date of establishment Funds received (M )

March 1997

June 1997 11.4

Jan. 1998 13.69

4.7 (From Phare) - plus ESOB (Czech bank) 2.6 (planned from Phare) Additional amounts under negotiation as of matches Phare disbursement for every 30/10/98 loan Variable interest rate but average rate applied during fund operation period = 8.5%, average pay back period 6-7yrs 13-15% Viable project showing significant energy savings (40% of all cost savings achieved by project implementation must be from energy savings) Max. Project value M 1.6. Buildings (flats, schools, hospital, swimming pools), industries, heating plants Loans for 10 year period- long term credits. Interest rates will very and are currently under negotiation 10-20% Phare is supporting a study to develop criteria and procedures for applications (TA to Fund - M 0.1). A handbook is currently being produced.

Terms of loans10

6-7% over 10 years (2 years grace)

Comparable market interest rates Criteria/conditions for loan

18-22% 40% energy savings

Preferential loans only maintained if energy saving goals are achieved c. 6-7% Fund has 60% of market interest rate c. 11 - 12% Energy savings have to provide at least half the return on the investment

Main use of Loans/Grants

Type of organisations applying Primary motivation of organisations applying

Municipal authorities, housing cooperatives, Industrial companies, Heating Plants To reduce energy/production costs, to meet pollution limits

Energy Efficiency projects in any sector - not Energy savings projects in (primarily) necessarily just for buildings. Projects can the industrial sector include: distribution systems energy audits to reduce production costs development of energy efficiency measures Private and public organisations Industrial Companies

80% for energy saving equipment

Companies which are not majority owned by local or national government Cost reductions

Reduction of energy use and costs

8 9

Capital is provided in Euros to the authorised custodian bank (often the National Bank) in-country for further disbursement.
M 2 from Phare, M 1.6 from Government and M 10 from Commercial Bank.

The terms for the rates varies between Funds. Generally, rates are determined within a range depending on risk, current market interest rates and other factors. Once committed the rates are usually but not necessarily fixed for the period of the loan. Interest rates are quoted as of November 1998.
10

48

Phares contribution (financial and technical)12 Date of establishment

SL 9503.03.01 Renewable Energy Grant Support Scheme11 M 0.15 from Phare with cofinancing from GOS 1997

BG 9411.01.02 National Energy Efficiency Fund

HU 9609.02 Energy Saving Revolving Fund

LI 9403.01.01 Energy Efficiency Fund

Provision of TA (M 0.3) for establishment of fund Feasibility Study (M including 0.07 M for loans/grants given by fund Funds (M 7.5) Originally Oct 1992 but stopped. Restored 1997 1997

0.11), and provision of Feasibility study (0.3 M ) and provision of seed capital (2.95 M planned)

Funds received (M

0.35

M 0.3 by Phare and 44 MLeva from the state for grants/loans 3 years for pay back with no gratis period 50% of the basic interest rate of National Bank of Bulgaria Variable but between 5-6% Hospitals and Schools may apply for 50% grant and 50% low interest credit for Energy Efficiency projects. Industries may apply only for low interest credit for Renewable energy projects as well. Energy efficiency and renewable energy projects in industry, households, transportation, energy sector (miming, electricity, heat supply etc.) Hospitals, schools and industrial units Decrease energy use and costs (for 3 reasons: planned increase in energy supply costs in 2001; improving the attractiveness of enterprises for sale due to privatisation policies and a reduction in the import of fuels - Bulgaria currently imports 75% of its fuels)

Terms of loans13

N/A - grants only

Comparable market interest rates Criteria/conditions for loan

N/A Recommendation from advisor, receipt of purchase, minimum collection area of 4 m Grant support to reimburse 500 private households who built solar collectors on their houses Individual residential building owners Hot water supply during summer season

Not yet established (institutional and legal frameworks are established but Fund is awaiting receipt of seed capital from Phare) M 7.5 (from Phare) - this forms 25% of the 2.95 (awaiting disbursement from Phare) funds resources. Remaining 75% will be provided by banks. 25% of loan is provided by Phare - interest 11% for long-term investments free, 75% by Banks Interest rate is the official interest rate of Hungarian National Bank plus 2% if loan repaid in 5 years, 3% if loan longer Applications would not be able to receive 16-18% loans without this Fund Reduction of energy use by at least 50% maximum size loan is M 0.5

Main use of Loans/Grants Type of organisations applying Primary motivation of organisations applying

Implementation of energy efficiency measures energy efficiency measures targeted at small enterprises Municipal government, waste water and heat supply companies Primarily cost reductions, Dictates of public policy for municipal authorities; and Commercial banks do not provide loans for these projects none as yet - planned to be small Objective is to reduce energy intensity rates EU levels and thereby to reduce imports and expenditure on energy.

11

Slovenia also has an Environment Fund - Phare has approved M 4.0 for the Environmental Credit Scheme to be disbursed as loans to industrial enterprises. The first tranche of M? 2 has been received. 12 Capital is provided in Euros to the authorised custodian bank (often the National Bank) in-country for further disbursement. 13 The terms for the rates varies between Funds. Generally, rates are determined within a range depending on risk, current market interest rates and other factors. Once committed the rates are usually but not necessarily fixed for the period of the loan. Interest rates are quoted as of November 1998.

49

PROJECT LEVEL

4.1

INTRODUCTION

This section examines the performance of the nine different project types. The types are defined by their outputs and associated specific objectives, as described in Section 2. The section also describes the main factors, which determined the performance of each type of projects, and this analysis is used to reach the recommendations for improving performance at project level. The section is based on the results of the detailed evaluation of sample 31 projects, and supplementary surveys for selected project types. As the sample includes relatively few projects of each type, the results are not statistically representative and should be treated as illustrative rather than demonstrative. 4.2 INSTITUTIONAL PROJECTS

Phare supported 341 institutional strengthening projects, almost half (171) of which were primarily concerned with reforming or strengthening public organisations. Nearly 100 of the institutional projects were concerned with strategy formation, covering a wide range of issues, often including economic and regulatory policy, as well as investment prioritisation. 28 projects were concerned primarily with regulations and 39 with public awareness. Very few projects were primarily concerned with pricing, although pricing issues featured in many strategy projects. The sample of institutional projects selected for more detailed evaluation included: 5 strategy projects, 6 organisation projects; 1 regulation project; and 3 public awareness projects. Whilst this classification reflects the primary focus of the projects, many of the projects contained features of other types of project, thus widening the base from which conclusions could be drawn on each project type. 4.2.1 Performance Scores

This section presents scores for the efficiency, effectiveness, impact and sustainability of the 15 sample institution projects. The scores concerning the relevance of these projects are considered in Chapter 3. Table 4.1 shows that there is more variation within the different types of institutional project that between types. This suggests that there are a wide variety of circumstances within each project type and that it is difficult to draw general conclusions which, apply to all projects of a particular type. However, the overall trend is that the score, which is high for relevance (between good and excellent), tends to go steadily lower for efficiency, effectiveness and impact (just below satisfactory). Sustainability is often rated higher than impact. This is because the indirect impact of skill transfer increased the probability of the continuance of activities after project completion.

50

Table 4.1

Institutional projects evaluation scores (scale 1-5)


Value Size ( M) Relevance* Efficiency Effectiveness Impact Sustainability

Strategy Projects Slovenia EE strategy Poland instrument costing Romania pollution monitoring Slovakia energy database Estonia pollution masterplan Average score Organisations Romania ARCE support Lithuania LSPS restructuring Czech MIT support Estonia county water Estonia water network Slovakia pollution monitoring Average score Regulations Lithuania DH decentralisation Average score Public Awareness Bulgaria demand reduction Hungary bio-diversity awareness Czech NGO awareness Average score Average Institutional Projects

0.57 0.20 1.19 0.05 0.44

5.0 4.0 4.3 5.0 4.3 4.5 4.3 2.3 3.3 4.3 4.3 4.3 3.8

5 1 3 3 3 3.0 5 1 3 4 3 5 3.5 3 3.0 5 1 5 3.7 3.3

5 3 2 2 3 3.0 1 2 4 3 5 4 3.2 1 1.0 5 1 4 3.3 3.0

3 2 1 3 4 2.6 2 2 2 na 5 4 3.0 2 2.0 5 1 4 3.3 2.9

5 3 1 na 4 3.3 1 3 2 na 4 3 2.6 4 4.0 3 1 5 3.0 3.0

0.44 0.30 0.08 0.90 0.30 0.29

0.17

4.0 4.0

0.25 0.18 0.45

5.0 3.3 4.3 4.2 4.2

0.39

* - average relevance score of needs, appropriateness and comparative advantage

4.2.2

Strategy Projects

Performance. Strategy formulation was a high priority during the early stages of Phare operations, when energy and environmental management required major shifts in policy. This is reflected in the high relevance of all the strategy projects, and the fact that many of the projects took place in the earlier years of Phare activities. However, the performance of the sample strategy projects was variable. In general, the outputs produced by the strategy projects were of high quality, lending credence to the widespread belief that Phare is good at supporting technical assistance, except in the case of the Poland Instrument Costing project. However, there was much more variable success in the direct adoption of the recommendations as official government policy and, hence, little direct success in achieving a wider impact on the sector. In general, effectiveness was satisfactory with the exception of the Slovenia Energy Efficiency Strategy project, which was rated excellent. Impact tended to be lower than effectiveness and sustainability the latter being mainly due to skill transfer during the process of strategy development.

51

Technical Supervision. There is evidence that Phare is not well suited to providing supervision of strategy projects, which involve technical issues. Whilst the technical skills may exist in the Commission and in some individuals in Phare line ministries, an established mechanism for mobilising this experience in an authoritative way has not yet been developed. Thus, the success of technical supervision depends on whether informed individuals have

In the Poland Instrument Costing project, the Commission seems to have provided little contribution to supervising the substance of the work at any stage of the project and failed to provide advice to ensure that the work did not become too academic. In the Slovenia Energy Efficiency Strategy project, government (supported by an energy advisor provided by the SYNERY programme) provided excellent supervision to the extent that they co-ordinated appropriate revisions in the TOR. As a result, expenditure on pricing studies was reduced. Another project, which would have benefited from more active technical supervision by government was the Slovakia Energy Database project, where a stronger insistence on TOR would have been useful.

the opportunity or the authority to make decisions on the project. The problems with technical supervision were particularly evident when draft final reports were considered by Steering Committees. There were examples of Steering Committees which provided substantial technical comments on draft reports, and which held up acceptance of the report whilst these were being dealt with. However, there are very few examples of when this has lead to a rejection of a final report. Political and Beneficiary Commitment. Obtaining political commitment for strategy projects requires complex judgement, as strategy projects have often sought to influence policy at exactly the time when political change has been great and unpredictable. The difficulties of ensuring political commitment have been reduced in some projects by flexible management.
One of the main reasons for the effectiveness of the Slovenia Energy Efficiency project was the commitment and long-term involvement of one key Slovenian official, who was able to ensure that recommendations were acceptable and accepted. In contrast, the Poland Instrument Costing project demonstrates the way in which limited involvement of key institutions results in poor impact. In this case, two of the key potential beneficiary institutions were not at all involved in the preparation and management of the project.

This enabled the project management to reduce or cut activities, which were unlikely to obtain political commitment. The evaluation also suggests that the performance of strategy projects was strongly affected by the participation of all related institutions14. Project success was greatly enhanced if clear and stable institutional responsibilities are defined. Inter-departmental Communication. In common with government administrations across the world, there is poor communication between the different departments of government in many Phare countries. This is understandable where there are potential debates over responsibilities, especially in times of rapid change. However, Phare's support for energy and environment projects has frequently been undertaken with very little consultation with related governments departments. In the Poland Instrument Costing project, the Ministry of Finance

The Estonia Pollution Masterplan project demonstrates the success achieved by getting consensus over priorities amongst the various institutions concerned. The Slovakia Energy Database project illustrates the importance of wide institutional participation, with institutional rivalry frustrating the continuity of activities.
14

52

was generally not informed about the results, despite its importance in ensuring financial sustainability and in negotiating accession, and this has led to a lower impact. Indirect Impact. Although the recommendations of strategy projects may not have been adopted directly, many of the sample projects have been followed by reforms in the directions indicated by the projects. Even if the direct influence of the projects may have been small, the projects do seem to have had an indirect influence on the views of key individuals. This indirect impact of projects can take place even when individuals move positions. In our sample projects, staff turnover within the project lifetime was not high, but there were occasions when individuals moved position after the end of the project, sometimes to equally influential posts15. Local Consultants have proved to be reliable and have contributed to relatively strong performance. In the Poland Instrument Costing project, a large part of the work was undertaken by local consultants. Whilst these consultants required access to international expertise, they could have managed the project well. In the Czech MIT Support project, the use of local consultants helped to ensure that activities were locally acceptable, both within government and in industry. In the Slovakia Energy Database project, it seems clear that costs could have been saved by expanding the responsibilities of local experts, without harming the quality of project outputs. 4.2.3 Organisational Reform Projects

Performance. Projects to strengthen or reform organisations comprised by far the largest category of institutional projects, accounting for half of all institutional projects. In general, the project efficiency was rated satisfactory to good. This was mainly due to the good quality of local and international consultants and to the fact that projects were considered cost-effective. Effectiveness was on average between satisfactory and good. However, there has been limited impact from the organisation projects, with some exceptions (notably the Estonia Water Network project). Political Commitment. Ensuring that project recommendations have political support is particularly important for organisational projects. However, where projects can have an impact on strongly debated reforms, political commitment is often unpredictable. As a result, project activities may be entirely blocked or ignored. This is particularly evident in projects where there are wellestablished political positions, notably in the energy sector. The unpredictability of political commitment is particularly evident where there are strongly competing positions. Politicians, government administration, interested utilities or other firms usually hold different viewpoints. There are no easy solutions to this issue and each situation must be analysed on its own merit. However, in such situations, the
The Lithuania LSPS Restructuring project provides the clearest example of the importance of political commitment in contested reforms. Some parts of government did want to proceed with decentralisation, but the energy supply establishment resisted this and had the support in other parts of government. The strength of political commitment was insufficient, until an impending change of government facilitated stronger political commitment. As a result, study activities were blocked and project recommendations were largely ignored. Similar lessons were experienced by the Romania ARCE Support project, where lack of clear political support for reforms was associated with existing energy supply interests, and project results were largely ignored. The Romania Pollution Monitoring project did not involve the large commercial interests associated with energy supply, but there was little impact because government was insufficiently committed to the institutional reforms required in order to to adopt 15 The two key ministry individuals in the Poland Instrument Costing project subsequently moved the the project recommendations. power sector and have been active in promoting permit costing approaches in the power sector. 53

existence of an independent technical study can help to mould the opinions of key individuals, or to reinforce views, which are already held. Beneficiary Commitment. Problems in obtaining institutional commitment were important for organisation projects. This was often associated with rivalry between institutions. The problems are often most severe when reforms involve major changes in employment. Institutional participation is often particularly important at the early stages of the preparation of TOR. The sample projects included some in which insufficient attention was paid to obtaining full commitment to the TOR from all related institutions. There were several projects in which institutional responsibilities were clearly defined and the project could focus on the one or two key institutions involved. However, it was far more common for projects to have been operating in an environment where institutional responsibilities were evolving. It was therefore difficult to predict who was responsible for the different activities16. Management Training and Study Tours. The experience of the sample organisation projects suggests that technical training in the operation of new equipment was relatively successful. However, it is far more difficult to train managers to adopt new strategic planning activities to best utilise the new equipment. Study tours have been very effective in conveying new ideas providing that those selected remain in a position to influence decision-making. The impact of study tours is not necessarily to be found in the direct replication of methods, but in general exposure to new approaches and methods.
The approach of the consultants in the Estonia County Water demonstrates the value of the on-the-job training for skills transfer. The consultants assisted in providing the technical analysis to define the strategic options for management restructuring, but required management staff to decide on which options to adopt, thus forcing them to become fully engaged in the analysis and impact. In the Hungary Bio-diversity Awareness project, the study tour provided a wider understanding and context within which appropriate country-specific models were discussed. The study tour in the Estonia Municipal Water project showed managers the different options for new water systems. In the Bulgaria Demand Reduction project a study tour was organised to provide the opportunity to visit utilities in an EU member state. However, several of the participants were relatively senior and may retire before fully utilising the experience gained from the study tour.

The timing of study tours is critical. In most cases, it is best for study tours to take place relatively early in a project, so that the participants can incorporate lessons into their activities.17 It is also important that the trainees selected are those that are likely to be involved in the management of the project and in follow-on activities, and that employment turnover is not excessive in the institutions from which participants are drawn. In many cases, this means that the ideal candidates for trainees are amongst younger middle-

16 The Estonia Water Network project provides a good example of where a project had a clearly defined partner organisation, the Estonian Water Company, which was strongly committed to project activities. In contrast, in the Romania Pollution Monitoring project, there was a lack of clear definition of institutional responsibilities which made it difficult to establish a clear partner organisation with whom to collaborate with in implementing the project. 17 If the study tour in the Bulgaria Demand Reduction project had taken place at the beginning of the project, the participants on the study tour would have realised the importance of involving Bulgarian utilities in the awareness campaign, which turned out to be a major constraint.

54

management tiers. It may also mean that candidates from private organisations are good candidates. Indirect impact. Although few of the recommendations for reform proposed by the sample organisation projects were directly adopted and the reports have not been widely referred to, there is evidence that projects have had an indirect effect on the course of subsequent reforms18. The sample project evidence suggests that expecting Phare projects to have a dominant impact on strongly debated political reforms was and still is unrealistic. Even in areas where a project may provide important technical information, which appears to be essential for effective policy analysis, it is still unlikely that a technical Phare study will be the main cause of reforms. 4.2.4 Regulation Development

Performance. The Lithuania District Heating Decentralisation project was the only regulation project. It was relatively efficient but did not achieve its primary objective of leading to the adoption of recommendations. However, the project did have some influence in the longer-term evolution of debate, and was therefore judged to have some sustainability. Technical Supervision. In the Lithuania District Heating Decentralisation project, the Commission services were not active in providing technical supervision despite the fact that they subsequently succeeded in directly negotiating the arrangements for accession. Lack of mechanisms for providing technical supervision was evident. This was particularly problematic as this type of project need to deliver very politically sensitive and practical recommendations. The experience of the Commission should have helped government in insisting on more specific recommendations from the study. Political and Beneficiary Commitment. The main reason for low effectiveness was the lack of political interest to implement district heating decentralisation at that time. This indicates, despite being able to provide useful technical assistance, Phare outputs have limited political influence and project of this nature are highly dependent on the political climate of the beneficiary institution. 4.2.5 Raising Public Awareness

Performance. On the whole, public awareness projects were successful, achieving a wide exposure to messages, with some evidence of substantial changes in public opinion. The Bulgaria Demand Reduction project reached such a high proportion of the population that most of the easily identifiable energy saving techniques were presented. Likewise, the Czech NGO Support project demonstrated the importance of improving public awareness. Only the Hungarian Bio-diversity Awareness project suffered from the low interest of the beneficiary organisation, which was not ready to fully accept the new approaches promoting public participation in nature conservation, and a bad management.

The sustainability of the Estonia County Water project was dependent on the continuity of key staff, who became committed to the project and motivated to ensure continuing activity. Despite not achieving its specific objectives, the Romania ARCE Support project did have an impact on reforms. Some of the individuals involved in the project have since been in influential positions and have promoted energy efficiency policies.
18

55

Project Preparation and Management. Efficiency was quite high in two of the projects. Overall, budgets seemed to have been well judged and appropriate. The Czech NGO Support project design was also well adapted to the situation. It showed how support could be provided effectively through local retailing mechanisms without undermining local initiatives, as proposed by the Phare Partnership Programme.

In the Bulgaria Demand Reduction project, there were consultants on the shortlist who declined to bid, because of the budget, but who had extensive experience in energy reduction campaigns. These consultants may have sought to involve the utilities, and hence solve the one significant problem with the project. In the Hungary Bio-diversity Awareness project, the main limitations were caused by lack of political commitment. However, the low budget also contributed to the limitations of some of the project outputs, as well as the inability to promote a more sustained stimulation of political and public debate.

Finally, reaching the right groups with appropriate materials and mode of communication was key to the success of project impact in the cases of the Czech NGO Support and the Bulgaria Demand Reduction projects. For the Hungarian Bio-diversity Awareness project, the report produced fell short of a operational strategy and could only be used by senior staff. Policy Context. The impact of public awareness campaigns are positively influenced by the policy development and structural reforms being implemented in-country at the same time. In the case of energy conservation, financial incentives were found to be instrumental in achieving change in behaviour. The Bulgaria Demand Reduction project acted as a stimulus to household to respond to concurrent energy price increases. Beneficiary Commitment. In common with other institutional projects, public awareness projects demonstrated the importance of involving all institutions at an early stage in the project. This helps to ensure that institutions are interested and committed to the activities of the project, and that commitments is obtained from the full range of institutions which are likely to be required for effective follow-up. In the Bulgaria Demand Reduction project, the effectiveness of the project was undermined by the fact that the national electricity corporation (NEK) was preparing its own parallel project at the same time as the Phare project was being prepared. The Hungary Bio-diversity Awareness project also demonstrated the importance of involving the national institutions at an early stage as the impact of the project was constrained by the lack of commitment in the administrations. 4.3 INVESTMENT PROJECTS

Phare supported 310 investment-related projects, of which more than one-third were primarily concerned with the provision of physical assets and another third with the provision of demonstration projects. The reminder focused on investment funds and investment preparation.

56

Table 4.2

Investment projects evaluation scores


Value Size (M ) Relevance Efficiency Effectiveness Impact Sustainability

Physical Assets Bulgaria port dredging Hungary sewer system Bulgaria regional labs Estonia lab equipment Bulgaria tannery Hungary national park Hungary hazardous waste Average score Demonstrations Romania energy services Romania hospital EE Bulgaria DH Average score Investment Funds Lithuania EE Fund Czech municipal loans Lithuania Env Fund Average score Investment Preparation Slovakia energy audits Poland energy finance Average score Cleanup Projects Bulgaria bay cleanup Average Projects Investment

2.54 1.20 0.20 0.68 0.10 0.19 0.33

3.5 5.0 3.3 4.3 4.0 4.7 3.7 4.0 4.3 3.7 4.3 4.1 4.3 4.3 4.3 4.3 3.3 3.3 3.3 3.0 4.0

1 5 3 3 3 1 2 2.6 3 2 4 3.0 1 4 1 2.0 4 3 3.5 5 2.8

2 3 2 3 4 5 2 3.0 3 2 4 3.0 1 4 1 2.0 3 2 2.5 5 2.9

2 4 2 3 2 2 1 2.3 Na 1 Na 1.0 4 4 4 4.0 1 1 1.0 5 2.6

4 4 2 4 2 1 na 2.8 na na na 0.0 na 2 na 2.0 4 1 2.5 5 2.9

0.60 0.25 3.01

3.00 0.40 2.30

0.35 0.16

0.76 1.0

The sample of investment-related projects selected for more detailed evaluation included: 7 physical assets projects, 3 demonstration projects, 3 investment funds, 2 investment preparation projects and 1 clean-up project. Performance Scores This section presents scores for the efficiency, effectiveness, impact and sustainability of the 16 sample investment projects. The scores concerning the relevance of these projects are considered in Chapter 3. Table 4.2 shows an overall trend of moderate to good efficiency with effectiveness being slightly higher - Investment Preparation is only output with lower effectiveness and impact slightly lower. Sustainability is on average satisfactory.

4.3.1

Physical Assets

Performance. Nearly all the projects delivered the required assets in a cost-effective manner, though there were examples of over-budgeting in the case of the lab equipment in Bulgaria and the information centre in the Hungarian National Park. The quality of the contractors was generally high. Efficiency in the Bulgaria Port Dredging project was specially poor with long delays. Effectiveness was mixed averaging a satisfactory rating. However, impact generally tended to be moderate to satisfactory and sustainability was just below satisfactory.
57

Preparatory Studies. Several investment projects showed insufficient economic and technical analysis and lacked social and environmental assessments - normally carried out during the pre-feasibility and feasibility stages and prior to project approval, resulting in a poor impact. Economic studies are needed to rationally allocate grants and to ensure that the proposed assets are costeffective. Otherwise, the risk is that project proponents do not rigorously analyse options knowing that repayment of the grant is not required. Technical appraisal is also needed to make sure the asset is a priority for the existing situation and its impact will not be dependent on future complementary investments. This is a key issue as there is a high opportunity cost for Phare grants for physical assets. However, the projects revealed that Phare only allocates a small budget, if any, for consultants to carry out prefeasibility studies. The time Phare allocates to project preparation is low compared to other donors operating in the developing world, including IFIs, UN agencies and bilateral ones.
The feasibility and EIA studies funded by Phare for the Hungarian hazardous waste project failed to identify the high level of local public resistance which eventually halted the entire investment. This shows the consequence of insufficiently assessing social impacts, at least for hazardous waste facilities. The pre-feasibility study for the Bulgaria port dredging project was not available for review in Bulgaria (and there is some doubt about exactly what was done). However, evidence found during the project shows that increases in vessel tonnage were possible without dredging. This indicates that the technical aspects influencing cost saving were not investigated sufficiently. There was no real needs assessment carried out for the Bulgarian lab project. As a result, excessive equipment of a higher quality than required was provided. In the case there were pre-feasibility studies, the budget allocated was insufficient: less than a week for the Estonia network project and the Bulgarian port dredging project. Furthermore, even the larger Phare projects have little resources for preparation, and many of the projects funded by the UN and bilateral agencies are of a similar size to those of Phare, but receive more resources for preparation.

Whilst there is often a difference in size of investments between Phare and other donors, the level of required preparation is not necessarily proportional to the size of the investment. The problem of inadequate studies is compounded because environment and energy departments in Phare countries lack in-house capability to carry out their own feasibility studies and are reliant on consultants. Technical Supervision. The efficiency of implementing investments appears to benefit significantly from technical experts supervising the investments19. The consultants act as a link between the PMU and the contractors supplying and/or installing equipment. This has proven useful and cost-effective for projects supplying equipment with detailed technical specifications and training requirements and those installing complicated infrastructure
The design study commissioned by the Municipal waterworks of Budapest for the sewer system project proved to have inadequate design details. This led to additional work and costs during the project. However, the municipality hired an engineering firm to supervise the sewer works. Their technical ability proved useful in limiting additional costs during the project. In the Estonia County Water project, the PMU benefited from outsourcing consultants to not only prepare the TORs for equipment specification but to assist in the tender evaluation, manage the contractors and be responsible for the provision of high quality and appropriate equipment and materials. The PMU judged that the additional cost was a valuable investment for the outcome of the project.
19

58

systems. However, if projects are straightforward, supervision, additional to the PMU, may only add another layer of management. Suitability of the Financing System. Because grants for physical assets do not need a financial return, they do not create a strong incentive for the project beneficiary to plan carefully the investments or decide whether the expenditure is cost-effective. The lack of financial incentives may have influenced the selection of equipment in the Bulgarian Laboratory Equipment project. The lack of financial incentives may have influenced the selection of equipment in the Bulgarian laboratory equipment project and constrained the sustainability of the Bulgarian tannery effluent project. The allocation of Phare resources through funds and other financing channels rather than direct subsidies - is a response to this. On the longer term, loans encourage utilities and companies to develop financial planning and discipline. This is important for achieving programme goals of capacity building which are important for sustained investment (see Section 7). This is already the case when cost sharing is required. The extensive participation of staff from the beneficiary organisations in the Estonia Water Network project demonstrated the benefits of having good beneficiary cost sharing. Public Involvement. Consulting local communities proved an important component for the development of large infrastructure projects, particularly those with potential environmental or social impacts. Projects evaluated have not devoted sufficient resources to this process. Lack of public consultation was key in halting the Hungarian hazardous waste project. Complementary investment and activities, including training, have influenced the impact of investment projects. Training was necessary when staff at the beneficiary institution did not know how to effectively use the Phare-funded investments. Where training was provided alongside the physical asset the effectiveness and impact of the asset increased. In the case these activities were not undertaken, the impact was seriously reduced.
With: The Estonia water network TA project carried out pilot studies as a form of on-the-job training to use the newly supplied equipment (for leak detection and sewer cleaning). The engineers from the municipalities worked alongside the contractors during the pilot activities. The three targeted municipal water companies all reported that this was the best method for them to become confident in using the equipment. They all demonstrated the ability to carry out the activities themselves. Without: The sewer system installed in Hungary took time to operate because local household did not invest in sewer connections. Conditionality for the local authorities in Hungary to build housing connections to the sewer or upgrading of the treatment plant would have increased the use of the installed system. Phares investment in the Bulgarian port dredging would have had more impact if the power plant had been required to simultaneously invest in associated unloading facilities within a certain time period (e.g. 12 months). The plant is now planning such investments but the combined benefits will accrue only five years after the completion of the Phare investment.

Timing of Phare Payments. In many of the projects evaluated, Phares procedures caused significant delays to the provision of a physical asset. This put several projects evaluated at
59

risk. The Hungarian hazardous waste project was supposed to supply equipment for a hazardous waste facility. The budget was eventually reallocated to another project as delays in the project resulted in the disbursement period being exceeded. The Lithuanian energy efficiency fund nearly lost its M 2 seed capital as delays in its establishment resulted in initial disbursement period has being exceeded. Delays of payments can also cause internal difficulties for a project. A projects efficiency can be adversely affected when Phare is providing a grant to finance local contractors to carry out an investment project which includes large capital expenditure for equipment procurement. Local contractors often have limited internal funds and highly constrained cash flow. Therefore they are dependent on Phare payments to pay for equipment and subcontractors involved in investment projects. In the Bulgaria port dredging project, the local dredging sub-contractor threatened to stop dredging as the contractor could not make payments as they were dependent on late payments from Phare. The project was nearly halted, the activities were slowed and quality dropped, adding costs.

4.3.2

Demonstrations

Performance. The majority of demonstrations were based on proven technology already operating in EU member states. All the pilot demonstrations successfully worked incountry. Approximately one third of the projects have proven replication, with funding from National funds, municipalities, Phare, industry or home owners. The demonstration projects were very useful in raising public and investor awareness and in providing valuable physical evidence to polluters on practical means of complying with national legislation and to energy users on lowering costs of energy use. This is particularly true because the evaluation found that the demonstrated equipment was generally good quality. However, in some cases, it was difficult to judge the potential replication as some of the projects are not yet complete. A few projects had no replication potential as there were no opportunities for using the technologies apart from where demonstrated. Table 4.3 Performance of Demonstration Projects20
Technology Demonstrated Thermal insulation in buildings Very small-scale ball water turbine Coal boiler for district heating Follow-up of 9101.42.1/B1 project Energy saving package for hospitals Remediate of pesticide bunkers Boilers in health resorts Heating technology Hydro plant for small dam Geothermal unit Straw fired boiler for municipal heating Water and wastewater systems Water supply protection 7 renewable energy technologies District heating boilers and meters Coal boiler emission reduction Number and scale of Replications Indirect impact on 10,000 flats 100-200 applications received No information As SR 9101.42.1/B1 No Replication No replication known Info spread by no replication known No Replication Project unfinished Project unfinished 5 known replications 800+ replications in 41 regions Plans in 3 municipalities & some utilities Not known Plans in at least one town All other boilers in utility

Project Code SR 9101.42.1/B1 SR 9502.04.01 SR 9202.42.01 SR 9202.42.08 RO 9305.05.02-L002 PL 9507.04.02 PL 9507.01.02 PL 9102.03.04 LI 9403.04.03 LI 9601.06.03 LI 9403.02.03 LE 9407.02 HU 9203.01.08 HU 9103.07 HU 9103.05.03 HU 9002.202.1

The Evaluation Team classified 30 projects as primarily generating demonstration outputs. To supplement the 3 sample evaluations, we undertook a rapid questionnaire survey of all the demonstration projects, in an attempt to get some indication of the type of technologies involved and the range of experience with replication.
20

60

Project Code HU 9002.108# ES 9302.01 ES 9404.05.01 BG 9408.04.04 BG 9508.02.03 BG 9408.04.01 BG 9310.04.04 BG 9310.03.01 BG 9307.01 SL 9404.01.02 SL 9503.03.02-4 CR 9101.B1/91 CR 9202.B3 CR 9303.D7

Technology Demonstrated 150 bus engines renewed Oil shale reclamation Oil shale power station investigation Remediation of land near uranium mines Modern mining equipment Bus conversion to gas EIA procedures Municipal solid waste landfill. Energy efficiency in district heating Energy efficiency practices Biomass-fired district heating boilers Energy savings investments in flats Energy audits in schools and hospitals Energy savings in hospitals

Number and scale of Replications 800 engines in 23 towns None yet, but some potential At least one 100MW boiler being built Interest in several sites Not known Several major applications No replication 10 municipalities applied Not known Increased interest Applications being prepared/considered No replication Follow-up project Not known

Financial and Legal Incentives. Incentives are highly variable between countries, depending on prices and the legislative and regulatory framework. Industry generally lacked the incentive to adopt pollution control equipment because national regulatory systems were not sufficiently developed or enforced. Fines needed to be higher to give industry the financial incentives to maintain demonstrated equipment and invest in replicating demonstrated measures (particularly pollution control). Fines and charges also needed to be strictly enforced. Industry and district heating companies were influenced by the potential savings from reduced energy use. Energy efficiency measures make more economic sense as energy prices increase. Traditionally energy prices have been low in CEECs, and whilst rising, are still not high enough in some countries to make energy efficiency profitable. Ability of Beneficiaries to Finance Investments. Replication was dependent on the beneficiaries ability to raise finances for the investment, either internally or externally. Demonstration projects were found occurring where sector beneficiaries did not have the budget to spend on demonstrated measures or the ability to raise external financing. This meant that no replication could occur. Considering the limited budgets of beneficiaries and many governments, it is a risk for a project to carry out a demonstration if available funding for beneficiaries to adopt the technology has not been planned.
The energy audits for hospitals in Romania is a good example of where the ability of the beneficiaries (hospitals) to purchase demonstrated energy efficiency measures was not apparently taken into consideration. The hospitals became interested in the energy efficiency investment proposals but were not in control of their budgets (imposed from central government) and thus could not afford to replicate the demonstrated measures. The project could not achieve its objectives unless the budgetary rules for hospitals were changed and they could allocate part of the budget to energy efficiency. The benefits of the monitoring equipment provided for laboratories would be secured if laboratories had greater capacity to generate their own income for equipment maintenance (e.g. through levying of pollution charges) instead of relying on central government budgets. The same is true of the bio-diversity asset project in Hungary.

61

Dissemination. Effectiveness in the sample projects was generally constrained because of the limited efforts to disseminate project results to potential beneficiaries. The main benefit of demonstration projects according to many beneficiaries interviewed during the evaluation is that they provide visible evidence of technological options and ways to improve existing practices.

The TOR for the Romanian hospital demonstration project did not require the consultants to disseminate the results of the project but only to print 250 leaflets. This was a risk for the project as its success depended on a national institution distributing the leaflets. In the end the consultants went beyond their terms of reference and distributed the leaflets themselves. The Romanian industry demonstration project appears to have a better chance of investment by other industry than the Romanian hospital project because part of the project was to train local consultants to assist Romanian industry in implementing energy efficiency issues.

The Bulgarian district heating project benefited from local building residents being able to see the installed equipment working well. Learning by doing remains an important way to convey a message. Information dissemination and awareness building are often the most complex parts of a demonstration project as beneficiaries are diffuse and the results are less tangible and take longer to achieve. One of the main weaknesses found with demonstration projects was when consultants were not given sufficient responsibility, if any, to disseminate the results, or to promote replication either in-country or in other countries in similar situations. There is project evidence that the allocation of responsibility for replicating project activities would increase the chance of projects achieving their overall objectives.

4.3.3

Investment Preparation

Performance. The two projects analysed in detail during the evaluation had moderate to good efficiency but indicated a lack of effectiveness and impact. The timing and focus of the projects were questionable and led to very low impact of the projects. Neither project analysed managed to secure financing for investment proposals. In addition to timing and focus a key weakness in Phares approach is how it deals with the financial sector in Phare countries. Beneficiary Selection. The projects, which identified investments in individual companies, did so quite successfully. Carrying out energy audits for individual companies proved a useful way of identifying energy efficiency investments. Investment assessments were less useful when done at a sector level, which did not provide enough detail. Ability of Beneficiaries to Finance Investments. In order for an institution to carry out an environmental or energy related investment, it needs the capability to raise capital (usually either from government budget, other donor agencies or commercial sources). The projects were found to consistently not accomplish their objectives of promoting environmental investments because of a lack of assessment of the financial resources of the targeted sector; and insufficient assistance for improving the in-house capability of potential beneficiaries to apply for loans Financial Resources. The energy audits for hospitals in Romania is a good example of where the ability of the beneficiary the health sector - to purchase demonstrated energy efficiency equipment was apparently not taken into consideration. The hospitals became interested in
62

the energy efficiency investment proposals but were not in control of their budgets (imposed from central government) and thus could not afford to replicate the demonstrated measures. The project could not achieve its objectives unless the budgetary rules for hospitals were changes and their could allocate parts of the budget to energy efficiency. Loan Application Skills. One of the most important components, evidenced during the evaluation, is the need to assist investors to improve their inhouse capability to apply for loans. Projects sometimes overlooked this component and expected beneficiaries to be able to do this themselves. However, beneficiaries (district heating and water companies as well as polluting industries) in each country vary enormously in size and in-house resources.
The Polish energy investment preparation project failed to have any real impact as it did not specifically target the small or medium sized energy companies, which needed assistance in promoting themselves to banks. As part of the Energy Efficiency Fund project a needs assessment was undertaken. This highlighted the high number of companies, which had potentially suitable investments but did not have the in-house capabilities to identify the investments themselves. The Lithuanian Environmental Investment Fund has already hired an expert, funded by a bilateral donor, to visit industries. The aim is to assist the companies in making loan applications and at the same time to increase the market for Fund lending.

There was an evidenced need to assist the small and medium sized industries and municipalities (responsible for district heating, water, waste) appraise environmental investment, their cost and financial return to the company and put forward an acceptable loan application. This is particularly the case for municipalities and utilities, recently empowered through decentralisation to be responsible for financial management of their operations. One project has already started trying to address this issue through external assistance. Banks' Investment Expertise. There was a general lack of effectiveness of projects targeting banks as potential sources of finance for environmental investments (score: 2). Lack of experience dealing with bank requirements (risk and return) has proved to be an obstacle (4.5.6.b). Additionally, banks were found to lack in-house expertise in assessing energy efficiency and pollution abatement investments. Consequently banks have been reluctant to build new markets in the targeted sectors. 4.3.4 E&E Investment Funds

Performance. Phare has contributed to 15 Funds in various capacities from feasibility studies to co-financing or sole financing. Out of these, The Romanian Energy Efficiency Fund was cancelled before establishment, four have not yet been established (Slovakian Environmental Revolving Fund, Czech Republic Business Environmental Fund, Latvian Energy Efficiency Fund and the Lithuanian Energy Efficiency Fund) and two have commenced but have not disbursed any funds (Bulgarian National Energy Efficiency Fund and Hungarian Energy Saving Revolving Fund). This indicates low efficiency in the establishment and operationalisation of Phare-involved Investment Funds. Seven Investment Funds are operating as shown in Table 4.4. The effectiveness of the Funds, measured by the percentage and value of disbursement, varies between the Funds but has been generally good. The Czech Energy Saving Fund and the Slovenia Renewable Energy Grant Support Scheme have been very effective, disbursing 70 and 77% of loans respectively. The Hungarian Central Environmental Protection Fund, to which Phare contributed M 14, has been highly effective, approving over 1000 loans with a value of
63

M 72. The Czech Municipal Loans Scheme for gas conversions also effectively disbursed all funds (M 1.15) via 4 loans. The Slovenian Energy Saving Fund only commenced operation as of January 1998 but has already approved 7 loans with a value over M 2. The Slovak Energy Saving Fund has also managed to disburse over M 2, although this is only 20% of its total funds. Only the Hungary Energy Saving Revolving Fund has been operational and has not yet approved any loans. The desk review did not evaluate the impact of these loans. Two of the three Funds evaluated in detail had no disbursement so were rated as having poor impact. The Czech Municipal Loans Scheme was rated as having good impact because approximately 10 MW of heating capacity was replaced from coal to gas with the associated savings in coal emissions. However there were some doubts over the impact (see below). Table 4.4 presents summary information on the seven Funds in operation and the two from Lithuania which were evaluated in detail. More general information on the funds regarding terms and conditions are provided in Table 3.9 Table 4.4
Date of establishment Capitalisation of Funds (M ) Phares contribution (in M ) Number of loan applications Number of loans/grants approved Value of loans/grants approved (M ) % disbursement
1

Effectiveness of Funds
SR/Ene 6/97 11.4 CR/Ene 3/97 4.7 CR/Env SL/Ene 8/94 97 1.15 0.35 SL/Ene 1/98 13.6 Le/Env HU/Env 97 92 9.3 LI/Env not yet operational na still awaiting M 2 Phare seed capital na 100% 40 8 1.3 (no disbursal yet) 0 LI/Ene not yet operational still awaiting M 2.95 Phare seed capital 100% 0 0 0

33% 12 4 2.3 20%

100%1 60 17 3.25 70%

100% na 4 1.15 100%

40% 549 505 0.27 77%

15% 15 7 2.1 15%

22% 30 10 na na

1976 1029 72.3 na

Czech bank matches Phare disbursement for every loan Phare contributes only 25% of a loan with the rest provided by commercial banks Phare also provided technical assistance to the Polish Environmental Protection Fund. As this was already in operation its effectiveness not evaluated. na: not available
2

Technical Assistance. Phare funded feasibility studies for nearly every Investment Fund to which it provided capital. The feasibility studies were essential for designing the institutional arrangements and review and disbursement procedures for the Funds. The efficiency of setting up and initiating operations of Funds depends, in part, on the appropriateness of recommendations in the studies. The low efficiency suggests the studies or their implementation did not adequately address the institutional difficulties in setting up a Fund (particularly arrangements with banks described below). The effectiveness of the Funds depends, in part, on technical assistance they can provide, once in operation, to their potential customers. Many of the enterprises the Funds try to target do not yet have in-house capability to identify potential investments, find out about the respective Funds and prepare loan applications. Many Funds do not have a provision

64

for this type of technical assistance. The Lithuanian Environmental Investment Fund, is an example of a Fund which engaged a pollution control investment expert to travel around the country (funded by a bilateral donor) to visit and assist industrial enterprises. At the same time this has increased its market for lending. Difficulties in Establishing Funds. The main efficiency problem concerned the frequent delays in establishing Funds. Eight of 16 Funds supported by Phare encountered delays in starting. Two of the funds analyzed in detail showed bureaucratic difficulties in establishing funds, associated with the difficulties in reconciling the requirements and procedures of Phare, government and central and commercial banks. Each Fund has spent over 2 years attempting to become operational and are still awaiting for Phare and the Lithuanian government to reconcile their requirements to allow the transfer of Phare seed capital into the Lithuanian banks. Level of Integration with the Financial Sector. Many of the projects relied on developing close working relationships with domestic banks. The three Funds that were evaluated all relied on domestic banks as mechanisms for loan disbursement. The two Lithuania Funds highlight the distinct characteristics (procedures, skills, rules and financial accountability) of the domestic and national banks and show the difficulties encountered when these characteristics are not taken into consideration.
The Lithuanian Energy Efficiency Fund has been delayed, in part, because Phare did not take into consideration the legal status of the National Bank of Lithuania, which was planned to be the holder of the seed capital. Therefore, it has taken over two years for the signing of the Financial Agreement for the establishment of the Fund. The Lithuanian Environmental Investment Fund also had difficulties securing partnerships with local banks whom did not feel confident with the financial and risk management arrangements on offer. The arrangements were based on the feasibility study, which did not adequately take into account banks requirements.

Banks were also found to lack in-house expertise in assessing energy efficiency and pollution abatement investments. Consequently banks have been reluctant to build new markets in the targeted sectors21. Neither of the Lithuanian Fund projects had provisions to build capacity within the banks were kept distinct from the technical review board. The banks were expected to learn from demonstration only. These difficulties, in part, reflect Phares and the Governments lack of experience in private financing for investment related activities within the environment and energy sectors and in dealing with the financial sector. Impact of Investments. The survey on which the Table is based did not include an analysis of the quality of applications. However, the detailed sample evaluations revealed that Funds were applying sufficient review and screening combined with explicit loan conditions to ensure projects will have impact. For example several of the Energy Saving Funds will only approve loans for projects which will generate energy cost savings equivalent to around 50% of total cost saving. Complementing National Activity. Funds must be careful to not provide funding for projects which would have been financed anyway by either commercial banks or existing National Funds. The Czech Municipal Loans Scheme generated doubts over the impact of its loans as the beneficiary municipalities would probably have completed the same gas conversion activities without Phare funds, although a year or two later. Thus the main impact of the

This was experienced in the Polish energy investment preparation project, Lithuania Energy Efficiency Fund, Romania industry services project and the Slovakia energy audit project .
21

65

loans were to subsidise the borrowing activities of the municipalities and possibly bring forward a small part of their gas conversion programme.

66

5.

RECOMMENDATIONS

This chapter draws together recommendations arising from the findings and conclusions of the report. We focus the recommendations at the project, sub-sector and programme level. At the project level the key recommendations have some features in common with other recent Phare and TACIS sector evaluations. The main recommendations useful to most types of Phare Energy and Environment projects are the following: Phare should provide more careful risk management for its projects in order to address the problem of constrained project impact resulting from lack of political commitment or lack of complementary activities. Phare should devote more resources to project design activities. This may include carrying out feasibility studies and in the long-term providing training to those in government identifying and designing projects. Phare should use more local consultants for carrying out projects. Whilst international consultants are sometimes valuable in bringing respected and independent external advice to politicised situations, local consultants are increasing in their technical capabilities and bring to projects good local knowledge and a sensitivity to local politics. Phare should provide technical assistance with the provision of physical assets, financed by the Commission, wherever possible so that staff in the beneficiary organisation maximise impact of the investment. This should include providing TA to complement investments funded by ISPA. Phare should increase the involvement of stakeholders and the general public in projects where possible. Consultation is important both to improve the design of projects but also to strengthen acceptability, a sense of ownership and sustainability of projects and policies. The increase in transparency will also increase the accountability of the project to beneficiaries and thus help to ensure impact. Phare should make all project goals explicit, particularly regarding capacity building and skills transfer. These should be put into the project logical frameworks so that the outputs can be properly evaluated. This will also make more transparent any trade-offs between objectives associated with capacity-building and other explicit project objectives. Phare should promote more dissemination and replication of projects. Most problems addressed in a project are numerous throughout a country or even the region. Therefore results need to be disseminated and mechanisms developed so beneficiaries can pass on information and skills to the other potential beneficiaries. Most of the other sector evaluations being undertaken by Phare have reached strong sectorspecific policy recommendations. Our overall programme recommendations are limited as our sample of projects represented such a wide variety of sub-sectors and project types generating diverse findings. It has therefore been difficult to observe strong patterns or to collect comprehensive information on the policy background for each sub-sector in each country. The E&E Programme is composed of five sub-sectors, each with distinct objectives. Based
67

on the evaluation of projects, the key strategic recommendations for future Phare involvement in each sub-sector can be summarised as follows: Energy efficiency should continue to receive support, working with independent agencies for the medium term. Support needs to focus on mobilising financing capacity in future. Energy supply is politically sensitive and Phare should concentrate on limited technical studies. Phare should expand support for local water utilities, through supporting investments financed by ISPA by funding management restructuring and public awareness projects., Attention should be paid to mechanisms which will promote replicability of activities by other utilities. Phare support for the abatement of air pollution needs to turn more to enforcement, having invested largely in monitoring in the past. This requires a range of complementary projects covering enforcement capacity, incentives and financing capabilities. Bio-diversity should receive more Phare support due to the relatively limited levels of financing from national and international sources. Investments should still be rigorous in,obtaining agreements on institutional responsibilities and generating financial returns. At the overall programme level, we recommend a continuation of the existing Phare E&E policy, albeit with some adjustment for each sub-sector. We make this recommendation because our evaluation of programme relevance found that Phares policy taking approach has led to sub-sector expenditure allocation matching country priorities. The project evaluations also showed that projects were generally relevant, appropriate and efficiently implemented. However, effectiveness was only satisfactory and impact just below satisfactory. Additionally, the recent policy changes associated with the Acquis and Phares DIS have already addressed many of the issues raised at the programme level. As a result, our recommendations focus on the project level. This is the level at which greatest impact can be made on the performance of the Phare programme, and which should therefore receive the greatest attention both from Phare governments and from the EC. 5.1 PROJECT LEVEL RECOMMENDATIONS

We provide recommendations here to be incorporated into project selection and design. They should assist officials in Phare governments and in the EC to improve project performance and in particular increase project impact. The recommendations are based on the factors which the evaluations revealed to be the most common constraints on project impact. These relate to addressing risks both external and internal to projects. Managing the risk from external factors. The most common risks to all projects were external to the projects. Many institutional and investment projects suffered from the risks associated with providing support in a rapidly evolving policy and investment environment. The most common and significant risks were: delayed complementary actions, notably on changes in pricing regulations, institutional
68

reforms or pollution reduction enforcement mechanisms; lack of beneficiary interest or resources leading to limited or no implementation of project outputs; lack of complementary equipment and infrastructure to maximise an investment These risks are often impossible to avoid and Phare projects should develop techniques for minimising the impact of this risk. Several risk management techniques should be pursued: assess and address risks to a project as part of project design or even during project selection; avoid projects which are critically dependent on complementary activities which have a high risk of not occurring; where risk is high, propose technical studies, where some impact can be achieved regardless of the external factors; manage projects in a flexible way, to allow alterations to reflect evolving political and economic situations which include flexible TOR and the phasing of projects; maximise the involvement of key organisations and individuals through stakeholder and identification and participation. This will increase stakeholders senses of responsibility to also contribute to managing external risks; allocate responsibility to individuals to implement project outputs; and increase accountability of individuals managing in-country Phare programmes so that projects are more integrated and complementary activities occur where possible e.g. drafting of programme plans, detailing how proposed projects fit into national environmental or energy strategies.

5.1.1

Institutional Strengthening Projects

The one recommendation common to all institutional projects is that Phare should ensure that all stakeholders - executing bodies, direct beneficiaries and partner organisations are clearly identified and involved, particularly in the early stages of project design. This participation will increase political will towards acceptance and implementation of project outputs. The level of such participation required in TORs should be proportional to the level of political resistance to the project. The other recommendations are specific to different types of institutional strengthening project (strategy, organisational reform, regulations and public awareness) and are provided below. Strategy Projects Future Phare Support. Most Phare countries are now undergoing reform and have National Environment Action Plans (NEAPs). Therefore Phare should change the focus of its strategy projects from broad sector reform and priorities (e.g. energy supply, pricing, NEAPS) to strategies for implementing accession. The growing consensus over the priorities dictated by accession means that future strategy work should take place in a more predictable policy evolution. Accession will require constant revision to strategies and action plans, but this is more likely to focus on particular project types (notably concerned with establishing and enforcing standards). Designing strategy projects. Phare should recognise that it is unlikely to dominate policy formation as a result of developing a strategy document, and that the most important benefits of such documents may be in addressing broader Phare objectives, such as building capacity and promoting policy debate. These broader impacts should be included in the
69

logframes for each project, along with related resource allocation and indicators. During strategy projects Phare should promote inter-departmental collaboration at technical levels, as personal collaboration at technical levels often helps stimulate co-ordination at policy levels. Local consultants should be widely used in strategy studies as they are increasingly skilled in providing technical advice, and have growing experience in ensuring that study outputs are acceptable to government. Experience suggests that this should assist in promoting ownership as local consultants may be more sensitive to political forces. It should also help in reducing the cost, although it should be expected that the cost of local consultants increases, as they become more capable and experienced. However, international expertise can still be valuable in adding weight and objectivity to strategies affecting potentially politicised situations. Organisation Reform Projects Further Phare Support. Despite the past difficulties with organisation projects, Phare should continue to provide assistance to help resolve controversial organisation reforms, in order to support accession conditions. Indeed, the obligations of accession mean that the EU is now the leading player in imposing conditionality on Phare countries and Phare projects should be able to build on this conditionality to achieve greater influence. However, the nature and management of the conditionality requires experienced and diplomatic judgement, that is related to the particular circumstances of the country concerned, and needs careful risk management using the techniques listed above. Level of Beneficiary Commitment. Wherever possible, Phare involvement should clearly identify the beneficiary institution and ensure its strong commitment. However, Phare should not necessarily avoid operating in areas of unclear institutional reform, since these are often the areas where most assistance is required. Where it is not possible to identify clear and stable institutional responsibilities, this should be recognised explicitly and Phare projects should adopt strategies to deal with this. This may require setting up a more flexible management system to enable activities to be adjusted to reflect changes in institutional responsibilities, or to enable a change in the beneficiary institution. It may also involve providing discrete technical collaboration, which will be beneficial regardless of the results of institutional reform. Institutionalising skills. The relatively high turnover of staff in public institutions in Phare countries poses a threat to the sustainability of training activities. One of the strongest methods of minimising this threat is to try to ensure that skills are fixed within institutions, rather than in individuals. This can sometimes be achieved by introducing formal models or databases, which are the responsibility of certain departments within an organisation and which are therefore understood by several people. However, any such models should be simple, practical and accessible, and should, preferably, be developed and managed by a team of local staff. Cost Sharing. International experience suggests that donor supported programmes often benefit from a degree of cost sharing with beneficiary organisations, as this improves participation and commitment. The limited evidence from the sample projects suggests that support was effective when it was integrated with beneficiary-funded activities. The degree of cost sharing can be varied to reflect the importance of participation. Cost sharing reduces the sense of aid dependency, which is a serious problem for the Phare programme and was particularly evident in the environment sector. A degree of cost sharing is typical in most
70

EU regional development programmes amongst member states, even when these concern training and technical support. Management Training: Management training is a particular skill, especially when it involves complete changes in attitude, which are associated with a switch from central planning to consumer sovereignty. Managers and technical experts may not be well qualified to perform this training, unless they are specifically trained to do so. One important feature of this training is to include specific assistance to senior management in defining and prioritising between the management options opened up by new technologies and strategies. This should be done in a participatory way, requiring management to reach their own decisions. Supporting TA with Equipment. There is some evidence that TA for organisational strengthening can achieve greater ownership amongst the organisations involved if it is combined with the provision of equipment which enables changes in work practices to be implemented. Whilst it is important that major investments only take place for clear purposes, once all institutions and objectives are in place, it may be advisable to supplement some institutional strengthening projects with modest levels of equipment provision in order to improve skills consolidation and motivation. This applies particularly to the provision of small numbers of computers22. Twinning. Phare should provide more support for twinning arrangements. Expert inputs should be provided to less advanced utilities as well as Ministries in Phare countries. Twinning should also be between the Candidate countries. Whilst experts from Member States will be able to show what needs to be in place, experts from Candidate countries will be able to show how the changes can be made. This will allow to show the intermediary steps of how to achieve progress, not just where to go. The leap for the administration in a country like Bulgaria or Romania to comply with the will be a lot more (and potentially impractical) than from countries such as Poland, Slovenia, Estonia and Czech Republic. It may also be useful to twin with public departments from areas in EU member states which benefit from Structural and Cohesion Funds, to share experience in the management of different EU programmes. Regulation Projects Further Phare Involvement. Phare can provide useful technical advice on the design of new regulations and legislation, based on the varied experience in the EU. Indeed, EU member states are in a constant process of improving adherence to EU standards, and are at varying stages of compliance.. This advice is most useful for those Phare governments which are already fully committed to accession, and are mainly interested in the detailed implementation of accession requirements. Political Commitment. The design and the context of regulation projects are crucial. For countries which are already committed to changing regulations, it is important that the TOR of Phare projects require very practical and specific recommendations, designed to feed directly into the drafting of new regulations and legislation. Furthermore, Steering Committees should insist that these recommendations are provided, before reports are accepted. There should then be follow-up to the recommendations by mechanisms such as allocating responsibility to individuals within the beneficiary organisation for pushing
22 This occurred, in the Estonia Municipal Water project, where provision of equipment was critical in generating commitment and, hence, in promoting the effectiveness of the TA. In contrast, in the Slovakia Energy Database project, the lack of parallel equipment undermined government interest in the study.

71

forward the recommendations from the reports into political debate and, where possible, implementation. Where it is thought that governments are unlikely to be ready to adopt specific recommendations, it is still possible that support can be provided for studies describing the costs and benefits of possible options.. Public Awareness Further Phare Involvement. Phare should continue to support public awareness projects, as it has a good comparative advantage, for several reasons: the EC has experience of promoting public awareness in EU member states; the projects are not generally linked to critical deadlines; the projects are a manageable scale for Phare; and the financial returns are often indirect or long term. Phare support in some countries (such as Estonia, Poland, Hungary and the Czech Republic) will be less critical as changes in public opinion are further advanced and governments and utilities are becoming more sophisticated in dealing with public awareness. Complementary Activity. Awareness raising, without financial incentives, is usually insufficient to change public behaviour. Therefore, future awareness campaigns need to be set in the context of a long-term policy of reform. Also, in the more advanced Phare countries, it is important that future Phare support will be more usefully provided if government strategy evolves to accommodate increased public pressures. Working with NGOs. Phare should also continue its NGO activities. The sample projects evaluated suggest that NGOs are important players in environment and energy efficiency fields. This will often be done most effectively by retailing support through independent local consultants. Follow-up activities. For public awareness projects, there is a clear need for follow-on projects. It should be possible to estimate the optimum level of funding, based on the parameters revealed in initial campaigns. The sample projects evaluated suggest that follow-up activities would require only modest resources to maintain awareness and to introduce any new techniques, which may become available. The possible benefits from requiring an element of cost sharing apply for public awareness projects in a similar fashion to organisation projects.

5.1.2

Investment Projects

Recommendations are provided for all investment related projects: direct investments, demonstrations, investment preparation and investment funds. In all investment projects, Phare should recognise the distinction between projects which aim to stimulate additional investment and those which aim to provide facilities on a grant basis. Where the objective is to stimulate additional investment, Phare should target beneficiaries where support will just tip the balance in encouraging investments. This will generally mean excluding the most wealthy authorities, utilities and companies, who can raise their own finance. It will also mean excluding the most under-resourced, who are unable to repay loans or capitalise on the investment. Where Phare is providing grants for investment, it will generally be most relevant to work in the poorest areas which are able to manage the investment efficiently.

72

Direct Investments Further Phare involvement. Where beneficiaries are richer and capable of becoming financially independent it may be better to provide loans via government, rather than grants23. Even where beneficiaries are not able to fully repay a loan for an investment, it may still be desirable to require some cost sharing to try to ensure ownership and commitment. Investment Preparation. The level of resources allocated to the selection, preparation and appraisal of Phare investment projects has not been sufficient and should be increased. The preparation of investments should include an analysis of complementary activities (including policy reforms, technical assistance and training), and the risks associated with these. The improvement of the preparatory studies will allow Phare to increase the strategic role of the potential investment into a beneficiarys overall activities. Two aspects should be highly prioritised in this area: Phare should build national capacity by providing training to the Ministries of Environment and Energy on the role and scope of pre-appraisal and feasibility studies and the management of consultants in these tasks. Phare should utilise its comparative advantage in providing technical assistance to ensure that collaboration with IFIs is well co-ordinated, which is an essential part of the new orientations for Phare.

Implementation. Particular attention should be paid to two implementation issues when Phare is involved in investment financing: Phare should not finance projects with tight deadlines for completion, particularly where other investments are dependent on the completion of these projects. Phare should develop mechanisms for improving disbursement for local contractors, who are particularly vulnerable to delays in payment.

Demonstrations Further Phare support. Phare should continue to support demonstrations of new technology, equipment and procedures to complement policy reform. Future demonstration projects should focus more on aspects of financial viability and replicability. Financial and Legal Incentives. Phare should improve the identification of financial and legal incentives during project preparation and consider how effective a project may be if it is dependent on the development of prices or regulations. Where the project is important but financial incentives are not yet in place, there is a role for a grant investment, which is less dependent on external incentives being in place. Proactive Dissemination. Phare should increase emphasis on the importance of proactive dissemination of demonstration results, both within the project design and with the
23 Where municipalities have strong finances, as in the case of the Czech Republic, it is possible to provide loans, which may be repaid from general revenue. However, in the majority of municipalities in CEE, municipal finances are weak and repayment may be dependent on careful planning. In many cases, it is likely that a degree of management and financial independence for heating and water services will be required to generate sufficient revenue to repay the loans. In many cases, revenue generation will take time to increase (based on charge increases) which will enable loans to be repaid.

73

responsible organisation after project completion.. The responsibilities within demonstration projects for dissemination of results should be clearly defined. It should be recognised that the skills for dissemination are different from the engineering skills needed to install and operate the equipment and therefore projects should be resourced appropriately. Finally, Phare should not limit the objectives of investment replication to the national level, but should also include inter-regional dissemination in other Phare countries. Natural dissemination. Dissemination by word of mouth between industries or individuals has proven an important mechanism for the adoption of demonstrated technologies/processes (e.g. district heating energy efficiency schemes by community blocks). Therefore it is important to ensure that installed equipment continues to function after the project is complete. This may require training local engineers in equipment maintenance and should be an important component of a demonstration project24. Financing Replication. The design of demonstration projects should include a plan for how replication of the demonstration is going to be financed. For example, demonstration projects should be followed up, as much as possible, with investment preparation projects to help beneficiaries with identifying sources of finance and securing loans for replication., . Physical assets. Where possible, physical assets projects should continue to build in demonstration components. However, the fact that investment projects have a demonstration objective should not be used as a reason to neglect the financial viability of the investment. Investment Preparation Further Phare support. If Phare is to act as a catalyst for raising financing, then this type of project will be increasingly important. Difficulties in the past need to be overcome to achieve investments. However, even the process of bringing together industry and the financial sector is itself important in the long-term. Financing energy and environmental investments. Project designers need to improve their understanding of funding requirements of the financial sector in order to increase the impact by ensuring proposals lead to loans . Phare support should also include a focus towards banks to improve their understanding of energy and environment investments (energy efficiency and pollution abatement) and banks should be more greatly involved in the projects particularly in meeting loan applicants. Project identification. Investment preparation projects need to improve their assessment of the financial capabilities, ie to afford to take out loans, of the potential investors in the target sectors (e.g. district heating)before agreeing to assist them in finding suitable sources of financing. Investment planning capacity. Phare should assist in the building of investment planning capacity, particularly in small companies. This could be done through technical and financial outreach programmes for potential borrowers. Similarly, investment preparation projects should assess the financial sources of potential investors in the sectors (e.g. district heating, polluting industry) before agreeing to assist target companies/institutions.

24 The Bulgarian district heating demonstration projects continued impact is in doubt as engineers have not been trained to maintain the demonstrated equipment.

74

Synergies. Phare projects should try to link, as much as possible, investment preparation with demonstration projects and investment Funds. The right sequencing and coordination of projects should allow greater impact and sustainability ie first investment can be demonstration, then once proven as technically and financially viable Phare should bring together industry and financial sectors and Funds. However, this should be done in a way, which avoids an attitude of aid dependency throughout the investment cycle, by making sure that decisions at each stage are taken independently. Investment Funds Our assessment of Fund operations was limited because two out of the three Funds evaluated were not yet in operation. A more detailed comparative assessment should therefore be undertaken into administrative and economic aspects of Funds. However, from the evidence acquired during this evaluation, we have the following recommendations: Further Phare Funding. Phare should increase their involvement if it can find more efficient administrative ways of commencing Fund operations. Designated credit lines to existing National investment funds may be an alternative to setting up new and independent funds. Preparation and Implementation. One of the key conditions to ensure effectiveness and impact is that Phare improves the efficiency with which it establishes Investment Funds. This implies a better use of the European Commission services experience accumulated so far in the domain and a better allocation of resources during fund set-up. In particular, resources need to be allocated so that the recommendations from the feasibility study are implemented and Phare obtains rapid agreement on the financial and legal arrangements of the Funds with its partners (the responsible Ministries, national and commercial banks). Working with the Financial Sector. The Commission needs to improve its understanding of the financial sector when planning operational partnerships and expecting local commercial banks to play an important role in projects (either promoting, managing or providing investment). Closer working co-operation between Phare-financed Funds and banks - and also the National Funds, where they exist - will increase the efficiency of the technical and financial appraisal procedures and will increase banks understanding of technical aspects of environmental or energy related investments. Phare should also increase its support to its ministerial counterparts and utilities to develop better models for co-operation with the financial sector. This requires long-term/on-going assistance in the form of specialist advisors and manuals in addition to increased discussions during project identification, selection and design in order to focus project objectives and ensure complementary factors are in place. Where Funds are run outside banks they should consider using dedicated technical expertise to assist in making decisions on loan applications. The expert(s) should have practical banking experience, and should be local, if possible. Where the Fund is administered directly by a bank, the banks E&E technical in-house capability should be strengthened through the provision of information (eg delivery of training courses on investing in E&E sectors). Terms and Conditions. Terms and conditions should be maintained reasonably close to commercial norms. An element of subsidy (on average currently one-third of market price)

75

is still generally valuable for environment and energy saving projects to ease the compliance cost burden. In the long-term, programmes should be designed to include a gradual phasing out of subsidies. Outreach Programmes. At the same time, the Funds should develop outreach programmes using financial and environmental consultants to build up capacity of potential borrowers (particularly SMEs) to identify investments, apply for loans and maximise the benefits from the investments. 5.2 5.2.1 PROGRAMME LEVEL RECOMMENDATIONS Improving Phare Sector Assistance

We provide recommendations for programming in each of the six sub-sectors. Energy Efficiency Phare should continue to give high priority to funding energy efficiency projects, because of the relevance of these projects to Phare countries and the limited public expenditure in the area, particularly compared to expenditure for improving energy supply. Support for energy efficiency needs to be balanced across a range of institutional and investment projects. However, projects to support energy savings institutions may need to be considered as temporary, as there are few models of permanent national energy saving institutions in EU member states. It is clear that Phare alone cannot support all the actions required and that Phare support should be co-ordinated with other changes. Where there is substantial doubt about the prospects for complementary changes taking place (particularly energy price increases) Phare projects should incorporate methods of risk management. If it is too difficult to predict progress on essential complementary activities, it may be best for Phare to support related technical studies, rather than preparation for investment and financing. Investments in district heating are particularly appropriate for Phare, as they involve working with local authorities and are not dependent on national reforms. In contrast, funding for projects to support private investment in energy saving should be carefully prepared with a full understanding of private incentives and financial sector practices. Phare support for energy saving campaigns is particularly useful, though this support should be rapidly declining after the initial campaign, and when a national system has been established. In general, Phare has followed the model of the independent energy efficiency agency, attempting to support establishment costs and strategies for continued financing25. In view of the dominant political power of energy supply companies in the energy sector in most Phare countries, this approach is necessary and should continue. However, in some EU countries, energy supply companies have become sufficiently committed to energy saving principles to collaborate actively with energy efficiency agencies and to run their own energy saving campaigns. There is evidence that this could begin to be considered in some Phare countries26.
The Romania ARCE Support project was based on the model for creating a functionally and financially independent energy efficiency agency. 26 The prospect of involving energy supply organisations in energy saving activities was raised in the Bulgaria Demand Reduction project. Although the project did not involve supply companies, the state energy company
25

76

Energy Supply Phare should design support for energy supply reforms in anticipation of a long-term relationship between the EU Commission and Phare countries. However, the EC should expect to have a strong direct influence on reforms through accession conditionality and not through the support of the Phare programme. In this context, the role of the Phare programme should be to complement accession conditionality with associated technical studies. These technical studies can cover various activities, including: the introduction of new technology; financial analysis; improved management in energy companies; or ways to manage different policies. Phare support for direct investment in energy supply can also be justified, especially where this is related to improving supply security. The level and design of this support is likely to be strongly influenced by accession negotiations. However, given the large investments which are likely to be required to meet energy supply security requirements, it is likely that the Phare programme will only be able to provide supporting activities, such as feasibility studies or management systems, and that the main source of resources will be from industry, and/or from IFIs. Some support for investment to promote supply efficiency may be justified. However, the total requirements are large and are best dealt with through IFIs, which can provide much larger resources and are better able to design the financial arrangements in a way which is consistent with the commercial development of energy supply. Phare's support for energy supply efficiency should be focused on areas where there are overlaps with environmental concerns, and where commercial incentives have not yet been developed to stimulate private sector investment. Such support requires much more resources at the appraisal stage Water Supply and Wastewater Services Due to Phares success in this important sector Phare should continue to strongly support water services. Phare should support restructuring management to complement ISPA financed infrastructure investments needed by municipal water companies. Projects should focus on developing management in water companies to become financially accountable and independent. This means raising their understanding of how to raise revenues (tariff levels and structures and improving metering, billing and collection) and reducing costs (network rehabilitation, staff). Due to the large number of similar potential beneficiaries in this sector, Phare should pay particular attention to disseminating and replicating project results, both by selecting, together with ISPA, utilities that can act as models for replication and by stimulating utilities and other organisations in providing consulting to promote replication. This replication should ultimately become self-financing, and Phare should support complementary development of regulations, which provide financial incentives to replicate investments in water improvement. Air Pollution

was preparing its own campaign and government officials became aware of this possibility as a result of the project.

77

In order to achieve air pollution reduction Phare should consider project selection in a more strategic manner instead of limiting assistance to purchasing monitoring equipment. In particular, there are several countries (such as Estonia and Bulgaria) in which the balance of financial support should change to focus more on enforcement than on increasing monitoring. This requires a balanced programme of projects supporting the reform and application of pollution charges and other regulations as well as complementary programmes to promote public awareness and political commitment. Once the full range of conditions are in place operational support for enforcement organisations will be more effective and thus increase in importance. Once legislation is in place and enforcement gains momentum, Phare should also increase its provision of subsidised financing to ease the cost burden to industry of adopting pollution reducing equipment. This subsidised financing should be planned, so that the level of subsidy is gradually reduced, so that there is an increased tendency for industry to incorporate environmental concerns into its own investment decisions. Bio-diversity Bio-diversity was the only sector where the proportion of Phare budget allocation was minimal compared to the degree of assessed national priority. Phare should use its strong comparative advantage in this sub-sector and increase its allocation of programme expenditure for assistance in developing national parks and raising public awareness. Projects should include more income-generating schemes to increase the financial independence of national parks. Sector assistance should be aimed at both strengthening protected areas and increasing public interest in bio-diversity. Specifically, Phare should increase support to NGOs, which are active in promoting public interest in bio-diversity.

5.2.2

Programme Strategy Recommendations

At the level of the whole Phare E&E programme we have a set of strategic recommendations regarding the following: Phares policy towards Phare country governments; Phares role in policy making versus policy implementation; the balance of TA to investment Phares Overall Policy of Support for National E&E Programmes 1. Government Ownership and Direction. The Phare strategy of maximising the responsibility given to government in selecting and designing Phare supported projects to fit within their own national programmes has been correctly aligned and should therefore be maintained. This strategy has encouraged ownership and commitment, which are essential prerequisites for programme success. However, there have been occasions where government capabilities are limited, and these should be addressed by technical support. Phare should build on its strategy by continuing to develop government capacity, particularly for programming and project preparation. In the long-term this will allow Phare support to be transformed into a mechanism which is closer to that used for the Structural Fund. In developing capacity Phare should focus more on training already employed government staff than on expanding government human resources as this has proved difficult in the past (international advisors are being
78

phased out and where PMU staff were external to government they infrequently moved into it). Inter-regional exchange of information and secondments between Phare supported governments will be a valuable component of this capacity building exercise. This should be formalised and could be incorporated into the twinning process. 2. Balanced Programmes. Phares initial strategy has been to phase the types of projects funded. However, there were many occasions where sequencing did not occur as expected and complementarity of different policies, legislation and financing proved successful for example it was found that lending mechanisms (ie Funds) for pollution abatement measures will only be effective if legislation and enforcement measures are in place to stimulate industry to reduce pollution and vice versa industry cannot presently comply with pollution standards unless financial support is available. Phare should therefore begin to focus less on trying to support a certain sequence of types of projects (e.g. legislation development before provision of financing) and focus on supporting a range of complementary activities through an integrated and balanced programme (including legislation, policy development, enforcement, financing and public awareness). This should be undertaken in coordination with the ECs other financing mechanisms for Candidate countries such as ISPA. Phares Role in Policy Making versus Policy Implementation 3. Implementation TA. The evaluation showed Phare was better at supporting technical development rather than influencing political will. Therefore programming should shift away from national policy development and focus more support for the technical aspects of policy implementation once political commitment at the central level is clear. The dominance of accession means that Phare support to policy can concentrate on this implementation aspect of reforms, which already have political commitment (e.g. energy security, Environmental Impact Assessment and Polluter Pays Principle). However, effort can be made at a project level to increase the influence on political will (see Section 5.2). 4. Decentralised Support In view of the success of supporting investment and reform at local levels, Phare should specifically focus more technical support at local decentralised levels (ie to local utilities and municipalities, particular for water and district heating), leaving accession to dominate national policy development. The approach should focus on promoting replication of project benefits to all utilities for example through assisting regionally strong utilities, which can promote replication of Phare TA by providing similar services to smaller local municipalities and utilities. Inter-regional information exchange between utilities will also be essential at this level. 5. Increase Phares Role as a Distributor of Information and Skills. Phare should promote dissemination of its project results and lessons learned between the Phare countries. Phare could facilitate this through a number of ways such as holding regional conferences on technical, policy and managerial case studies and lessons learned from implementing Phare projects (starting with a discussion on the results of the evaluation), establishing voluntary informal lines of communication between national environment and energy Ministries (eg through forming a website) or by working with the Commissions twinning programme. In-country experience indicates that twinning would increase in value if it could also be between the Phare countries. Whilst experts from Member States will be able to show what needs to be in place, experts from Phare countries will be able to show how the changes can be made. This will allow more advanced Phare country technical policy makers to show the intermediary steps of how

79

to achieve progress, not just where to go.

80

The Balance of TA to Investment 6. The objective of allocating 70% of Phare resources to investment should be maintained. It will be extremely important to still devote 30% of budget to complementary technical assistance.. Phare should continue to recognise capacity building as part of its programme strategy. This will be critical in influencing both the impact of future Phare investments (as discussed at project level, Chapter 4) and increasing investments by other sources of financing. 7. Catalysing Investment. One of Phares objectives has been to act as a catalyst for investments. The evaluation found that this has proven difficult but an essential goal and therefore Phare should increase TA to stimulate additional sources of financing. In financing energy and environment projects IFIs have tend to have their own agenda and we therefore do not foresee Phare acting to directly catalyse IFI investment. Therefore, whilst past efforts by Phare to promote investment by national industry and financial sectors have proven difficult for long-term reduction of pollution further efforts focused on stimulating investment by the private sector is essential. This includes building working partnerships between government, industry and banks (through forums etc), financial instruments and incentives (e.g. impose significant taxes and pollution charges as current public awareness and enforcement measures are insufficient), and build capacity in private banks, supported by subsidised lending for environmental and energy projects. At a sector level Phare should provide project preparation training 8. Creating an Enabling Environment. At a country level Phare should strengthen the regulatory environment so that industry has increased financial incentive to invest eg through higher (and fair) pollution charges and associated enforcement. Lack of enforcement cannot be easily rectified (see Section 5.3.1). Phare has already supported monitoring projects and now in many countries should shift emphasis towards increasing public awareness. Phare should thus provide TA for implementing PPP at a policy level as well as investment at a local level (for pollution reduction etc).

81

APPENDIX A: PROJECT EVALUATION SUMMARIES

This Annex to the Final Report presents a summary evaluation for each of the 31 projects evaluated in detail during this assignment. The summaries briefly describe the project and its context; state its overall and specific objectives; assess performance against the five main evaluation criteria and provide lessons learned which were inspired during the project but which do not necessarily pertain just to the project. The evaluation criteria are the following: Relevance (with its three components: relevance to the needs, appropriateness of the design and comparative advantage), Efficiency, Effectiveness, Impact and Sustainability.

82

BULGARIAN REGIONAL LAB EQUIPMENT (3/96-12/97)

0.20 MEURO

INTRODUCTION The project has been identified under the 1994 Phare Environment Programme for Bulgaria. It follows on a series of other projects supporting Regional Laboratories through the provision of monitoring and analytical equipment, lab furniture, ventilation systems and other items. Although the project is registered as a separate project under the 1994 programme, it has in practice been partly merged with a project identified under the 1993 programme with similar contents, namely project BG 9310.02.01 Laboratory upgrading. Project design, support to tendering and implementation support to both projects were provided from funds allocated under the 1993 project. The two projects are worth 1.42 MEURO and include the following activities: project design ; support to tendering; provision of lab furniture, ventilation systems, air condition systems etc. for 7 Regional Labs and for the National Centre for Environment and Sustainable Development (NCESD). The output was the supply of the above lab equipment with the specific objective of improving the working conditions for laboratory staff and conditions of monitoring equipment. The overall objective was to increase the monitoring performance of the regional laboratories. The key services of the lab include: provision of ambient quality monitoring data to local authorities and to the national monitoring network (managed by the NCESD); analysing monitoring samples taken by the Regional Environmental Inspectorates. PERFORMANCE Relevance to Needs (Score 3) The project, together with a larger series of supply projects, was perceived as meeting priority national needs. There was a general perception at the time of project identification that monitoring was most needed to get a better understanding of the state of the environment and the origin of the pollution. It is debatable whether Phare should have focused so strongly on monitoring needs compared to other needs for environmental pollution control. The project was also perceived to meet most pressing sectoral needs, but as with national needs it is debatable whether the wishes put forward by regional labs and the NCESD reflect priority needs only. At the time of project identification there was no comprehensive assessment of needs relating to the development of the monitoring system. Mainly material needs were put forward, not complemented with non-material needs such as those relating to training, communication/information processing, (financial) management etc.

83

Appropriateness of Project Type (Score 3) Given the lack of complementary measures for pollution control e.g. strengthening enforcement, it is debatable whether Phare should have focused so strongly on monitoring needs. Comparative Advantage (Score 4) Phare has a comparative advantage in supplying monitoring equipment in that there is no direct return on capital and thus the laboratories are in no capacity to repay a loan. They thus need provision of a grant where government budget falls short. Efficiency (Score: 3) Except for some minor delays the project was implemented according to schedule. There have been no problems with the delivered equipment and furniture. Cost-effectiveness is one of the major weaknesses of the project. It would probably have been acceptable to supply the regional labs with more modest furniture. This applies particularly to office furniture. It is not clear how much could have been saved through this. It would probably have been acceptable to provide less laboratory furniture and equipment as part of the furnished labs are not (yet) in use and laboratory cupboards and shelves were still partly empty. Funds could either have been used to purchase some necessary monitoring equipment (radiation, BOD) or to furnish an 8th regional lab. Effectiveness (Score: 3) Improvement of working conditions and to improvement of operations of the beneficiary labs was perceived as high by the beneficiary labs and others involved. Indeed the contribution of new furniture, ventilation systems etc. to better working conditions and operation is evident, but to what extent the project under evaluation contributed to these improvements is hard to determine. However the level of equipment utilisation is in general sub-optimal. Additional actions will be necessary (additional staff, basic supplies, monitoring equipment) to guarantee strong effectiveness of the project. Impact (Score: 2) It is difficult to isolate the impact of the project from other developments. In general, the investments in the monitoring systems have mainly resulted in more information about the state of the environment. However, the services of the laboratories are still not easily accessible for all users, both in terms of possibilities for interested parties to obtain environmental data and of the way environmental data are presented. Additional complementary actions are needed to enable the monitoring services to be used for pollution control measures. This requires a set of pollution control instruments to be developed and adopted, investments to be done in pollution control and strengthening of the effectiveness of enforcement efforts. Sustainability (Score: 3) Future operations of the labs are not supported by a sound financial basis at present. Although the future existence and operation of regional labs is not in any doubt, there are no firm guarantees that regional labs will have sufficient staff, budgets for supplies of basic needs and sufficient monitoring equipment.

84

Score
Efficiency Effectiveness Impact Sustainability Relevance to needs 3 3 2 3 3 3 4

Needs Appropriateness Comparative Advantage

1=poor; 3=adequate; 5=good KEY LESSONS LEARNED The timing of providing material support to regional labs may not have been well balanced27. Possibly, a more thorough needs assessment could have contributed to more balanced expenditure on the monitoring system. Such a needs assessment could then also have included the needs of environment inspectors working at the REIs, training needs etc. In retrospect it is understandable that Phare funding has been used to purchase more laboratory furniture than strictly necessary, because at the time of applying for funding there were no real funding alternatives and there was uncertainty about possibilities for future funding (Phare or other). The impression is that as much as possible funding has been applied for as a result of this situation. This has probably not resulted in the most cost-efficient use of funds. The use of Phare funds for equipment is a very efficient use of funds as they are exempted from VAT and other taxes. Purchases done with NEPF funds would be liable to taxes. In a wider sense, it could be argued that some of the funds which have now been spent on material supplies, should have been spent on other activities supporting strengthening of the monitoring system. The focus has very much been on material supplies while more attention could have been paid to non-material needs. These needs have not been identified and addressed in a structured way to date. There seems to be a general trend in the environment sector in Bulgaria to focus on material needs in the identification and design of projects supporting strengthening of organisations, rather than looking at a mix of material and non-material needs. Although the need for physical assets has been obvious in the evaluated project, the project has hardly been complemented with investments in non-material needs. Regarding the project under evaluation, other investments in physical assets (new buildings) by domestic funds justify the use of funds for this purpose at the time. However, the effectiveness and impact of projects investing in physical assets in general can be increased by developing more balanced programmes including not mainly material supplies, but also addressing needs of organisations such as the NCESD and the REIs (both the Control Departments and the Laboratories) in a non-material sense. The importance of guaranteed funds for future use and operation of the provided assets
It would for instance only be logical to purchase equipment and furniture for a lab, which has twice the size of the existing one if there were concrete plans for expansion of the lab in terms of staff finances and technical possibilities. The Regional Laboratory in Velike Tarnovo has been supplied with laboratory furniture in expectation that more staff and monitoring can be purchased. However, the plans for such an expansion are not at all concrete and applications for additional staff and equipment have not been approved yet.
27

85

seems to be not fully considered during project/programme design. There is a real risk that Phare funds are not used efficiently and effectively relating to the lack of trained staff to use and operate the assets and the lack of sound financial arrangements to maintain, supply and replace provided equipment. It is of crucial importance that the financial and staff situation of institutions which receive substantial Phare support is thoroughly assessed before major investments are being made in physical assets. The need for investment in physical assists is often evident, but investments in assets are only justified when they are part of a larger package of investments and actions. There seems to be a general perception that environmental monitoring is the most key requirement to control environmental pollution, but this is in fact not the case. Monitoring is needed to provide information on the state of the environment and on sources of pollution, but only if subsequent policy and legal measures are taken and if real actions are taken to implement these policies and legislation, pollution is being controlled. Provision of monitoring assets alone will not directly result in meeting sectoral objectives.

86

BULGARIAN PORT DREDGING

(4/96-10/97 )

2.54 MEURO

INTRODUCTION In 1996 part of the Kozloduy nuclear power plant was being decommissioned and energy production from elsewhere in the country was required. Varna thermal power plant was target to increase its production. Consequently, concept of the project to dredge port for the Varna thermal power plant (TPP) was developed. The overall objective of the project is to improve coal supply for electricity production through lower cost of coal and improved security and regularity of supply and increased coal reserves. It is unclear if the financial benefit was to be cheaper FOB of coal or cheaper shipping costs for the coal, or merely a more convenient and quicker method for receiving coal deliveries. The specific objective of the project was to allow larger vessels to deliver larger shipments of coal to Varna TPP and to save money on the cost of Phare funded coal purchase. (Phare wanted the project to be completed within 6 months (by November 1996) so that the coal they were to purchase on behalf of NEK could be purchased from Vietnam at a lower price). The dredging of the channel was also to improve the mode of unloading of larger ships. Coal was to be unloaded at Varna TPPs docking facilities as opposed to partial unloading offshore. PERFORMANCE Relevance to Needs (score: 5) Energy production was to be increased at Varna power plant and there was an evident need to diversify energy suppliers in order to reduce the cost of coal. Until 1996 the power plant had imported all its coal from the Ukraine. However prices from suppliers around the world were significantly cheaper ($30/t FOB compared to $35/t FOB) in 1996. Alternatives such as gas or national coal were not feasible. Therefore the objective of the project was highly relevant to national energy supply strategy. Appropriateness of Project Type (Score: na) The purpose of dredging the port was supposed to improve the importing of coal from long distance supply sources by allowing larger vessels to discharge the coal at the power plant. However the fact that there has been no significant increase in tonnage of shipments, depth and type of vessels before and after the dredging raises doubt as to the specific need for dredging to the specified depths (see Effectiveness). Comparative Advantage (Score: 2) Phare lacked the resources to assist designing the optimal investment. Furthermore, tts slow contracting procedures also proved a disadvantage as the inability to complete the project within six months meant that Phare did not realise the intended cost saving on upcoming coal shipments. Efficiency (score: 1) The project experienced many delays in implementation. The Commission delayed signing the contract, paying the advance payment, which prevented the contractors to commence
87

work (due to cash flow difficulties) as well as paying the fifth payment, nearly leading to halt the project (as the contractor did not have sufficient funds to pay for the dredging services without Phare financial support) and to a significant decrease in work rate and quality. The contractors were not able to deal with the technical difficulties encountered during dredging and it took them 7 months instead of the contracted 3 months to complete most of the work. The total completion of the job took an additional 6 months (one year in total). Effectiveness (score: 2) There has been no significant change in the size and type of long-distance vessel delivering coal to the power plant since the legal increase in channel depth. The improvement has been in changing unloading from partial offshore unloading to allowing vessels to enter TPP port and therefore reduce costs and speed up unloading time, allowing more deliveries. Effectiveness has been limited because authorities have not legally recognised full increase in depth of channel (only 37ft instead of 40ft). Therefore the large cost saving vessels (50,000t + Panamex ships) cannot use the port. Also no cost saving was made from Phare funded coal purchases. Impact (score: 2) Total cost savings have been limited because larger ships still cannot use port (above). Therefore no significant reduction in shipment costs to-date. Unloading (discharge) rates at the port are too slow and therefore full cost savings of using larger boats cannot be realised. The main price reduction has been due to increased competition and access to overseas markets. This in conjunction with a drop in world prices has lead to a reduction in coal costs. However as the effectiveness section demonstrates the increase in access to markets has not been due to the dredging. Dredging though has increased the quantity of coal being delivered from long-distance as it has eased the unloading for the incoming ships. Subsequently Varna TPP has managed to increase its supply of coal and increase its energy production. Energy production levels have increased from 1.65 billion kWh to 2 billion in 1996 to 2.3 billion in 1997 and 1998. Coal reserves in the plans storage facility were increased from zero in January 1996 to 200,000 in 1998 which allows the plant security of supply and flexibility with negotiating coal delivery contracts. Sustainability (Score: 4) The improvement of the port has paved the way for the subsequent investment projects the plant is preparing regarding improving power generation technology and port facilities. Fifty foreign companies have so far expressed interest in a joint venture with Varna TPP which will help finance these projects. However the sustainability of the direct benefits from the dredging will only sustainable if the National Energy Company will pay for channel maintenance costs and its commitment to this will only be evident when maintenance is needed in the future.

88

Score
Efficiency Effectiveness Impact Sustainability Relevance to needs 1 2 2 4 5 na 2

Needs Appropriateness Comparative Advantage

1=poor; 3=satisfactory; 5=good KEY LESSONS LEARNED Phare has a procedural disadvantage when providing a grant to finance local contractors to carry out an investment project. Generally, local contractors have limited internal funds and highly constrained cash flow. Therefore they are dependent on Phare payments to pay for equipment and sub-contractors involved in investment projects. This project exemplifies how the common tendency of Phare to be late with their payments exposes a project to risk and delay and thus additional costs. Phare should consider imposing conditionalities with investments when providing grants for physical assets or commitment of co-financing. For example, Phare should have only provided a grant for dredging if the plant invested in associated unloading facilities within a certain time period (e.g. 12 months). As such the plant is now planning such investments but the combined benefits will accrue only 5 years after the completion of the Phare investment. The lack of evidence and opinion surrounding the objectives of the project, the financial justification for the project and the physical impact of the project clearly demonstrates the need for someone to be delegated the responsibility for technical justification of the project and monitoring the impacts (not necessarily the same person). Providing a grant based on limited financial planning with no need for a financial return does not create an incentive for project sponsors to carefully plan their investment or decide whether the money is being spent on the best project (issue of opportunity cost). Financing was a particularly important issue in the mid-90s when national companies were in poorer financial positions and national financial sectors were generally underdeveloped and could not provide loans. Now companies should be more financially viable and if not providing them with a grant may not be helping them in their long term restructuring goals. Phare would benefit from having a set of selection criteria for deciding when it should provide a grant for a physical asset. Assist technically justified projects. Maximise financial benefit to the recipient. The benefit of a grant will be maximised if the project is not affordable through the market. The port project with its supposedly short financial payback should have been promoted for a commercial loan. Avoid undermining the commercial financial sector. Finance projects which have no tight deadlines for completion (i.e. other investments are not dependent on it). The Commission would benefit from having selection criteria to inform about the following three goals: what project types to avoid funding; when (timing) to do a type of project - how to determine if the right climate, macro conditions are in place for a successful project improving project design at a general level for specific project types
89

BULGARIAN BAY CLEAN-UP

0.76 MEURO

INTRODUCTION The project was included in the Phare 1993 programme as an example of how public resources should be used to clean up industrial waste. The project involved a substantial clean-up operation on a beach that had been contaminated with industrial waste from a copper mining and milling company (BMM). The project started with a study to evaluate the BMM proposals for clean up. This study confirmed BMM's assessment that radioactive and heavy metal pollution existed, but that it was largely limited to the beach. The study examined the costs and benefits of a variety of options and supported BMM's proposals to alleviate the worst problems by removing contaminated sand. The preparatory study took place mainly during 1995 and the main engineering work took place from September 1997 for about one year. The specific objectives of the project were to reduce exposure of humans and wildlife to radioactivity and to reduce the risks of contamination from heavy metals. Secondary objectives included promoting public awareness and confidence in the clean up of hazardous waste. The overall objective was to improve bio-diversity and human health, particularly to tourists in the Vromos Bay. The project also aimed to ensure that further copper mining and processing would be done on a sustainable and environmentally acceptable basis. The project also had additional objectives of restoring public confidence in official announcements about environmental safety. PERFORMANCE Relevance to Needs (score: 3) The clean up of industrial waste is a major environmental priority for Bulgaria and there are many important sites of hazardous waste, including some with exposure to radioactivity. Although the problem is known to exist, there is little experience in the financing and management of clean-up operations. The project therefore met a clear national need, which goes beyond emergency clean-up priorities of the first years of democracy, and is likely to continue for several decades. The main question about the relevance of the project centres on whether the project was the one that would achieve the greatest impact on reduced environmental hazards. The main concern arises from the fact that the existing research suggested that harmful doses of radiation were limited to the beach. The project reports do not state how many people live in the 'small settlement of summer cottages' on the beach. There is no written evidence of the reasons why the government selected the project. However, discussions with the Phare PMU suggest that there were a combination of reasons. The waste problems at Vromos Bay were well known, having been studied throughout the 1970s and 80s. Some of the contamination was radioactive, and the area had considerable tourist potential. BMM was keen to collaborate in the clean-up, and had
90

already produced proposals of how to deal with the problem. There was some urgency to deal with the problem, as the company was facing closure and some of the technical options for clean-up involved reprocessing and the use of waste storage sites, which could otherwise be closed. Finally, the scale of the problem was of an order, which could be tackled by a single Phare project. In view of the fact that the selection of the project had to take place without the benefit of a national plan for waste reduction, the selection seems to have been reasonable, though it is not clear that it was optimal. Appropriateness of Project Type (score: 4) The combination of a preparatory study, following rapidly by engineering work using local contractors was appropriate for the project. Comparative Advantage (score: 4) There is also a question as to whether Phare grant financing of the full cost of discrete environmental investments is the best use of resources. Such projects tend to produce clear results without the risks associated with trying to co-ordinate with government policy and donor activity. Some wider benefits can be achieved, through demonstration effects in industry and by promoting public awareness and pressure. However, there is little sustainable institutional framework for direct continuation of the work. In the case of the Vromos Bay project, the approach was justifiable, because the project was identified and initiated in the early years of the democratic period, before it was clear how the institutional responsibilities for clean-up activities were clear. However, in the future, Phare support for such activities should probably be through the existing funds involved in environmental investment, to ensure the widest possible replication. Efficiency (score: 5) The preparatory analysis was well-done and ensured that the most cost-effective solution was found, involving the removal of 200,000 m3 of contaminated sand into an existing tailing pond, and the sealing of this pond, with some modification of surrounding drainage. The initial cost estimates of 1.7 MEURO were reduced to 0.76 MECU, both because the amount of sand to be removed was reduced and because the work was undertaken by Bulgarian contractors. Some optional reprocessing activities were discussed, but the environmental costs of these (amounting to about 0.8MECU) were to be financing from the profits from reprocessing. Thus, the study was efficiently conducted and ensured that the main work was cost-effective. The study also proposed a parallel public information programme, to reduce misconceptions and the risks of visiting the bay. This was effectively conducted, both by local participation and by national broadcasting with leading politicians. The German consulting firm, Uranerzbergbau, co-operated efficiently with BMM and the local contractor and with Sozopol Municipality. Effectiveness (score: 5) The following table presents the international standards for permitted exposure to radiation, along with the levels of exposure before and after the project, for those people taking summer residence at the beach. The table demonstrates the success of the project in reducing exposure to acceptable levels. The testing also showed that there was no risk to

91

the few hundred people living in two nearby villages, as these received normal doses of less than 0.2 mSv/a. Dose Levels Before and After Clean-up (in mSv/a)
Adults Children International Standards 5.0 1.0 Before Clean-up 3.1 6.1 After Clean-up 0.5 1.2

There was a major change in the appearance of the beach, since the most highly contaminated sand was also the blackest. This should have a major impact on the tourist potential of the beach. There is some evidence that bio-diversity in the Vromos Bay has been adversely affected and a smaller number of pollution-tolerant species have become more dominant. However, the preparatory report suggests this is more because of the large quantities of waste deposited on the beach, than because of radioactive contamination. The publicity activities of the project were successful, with recent prime-time TV exposure, on the formal opening of the beach, and substantial discussions amongst local people and within the mining industry. Impact (score: 5) The radiation exposure at the beach was successfully reduced to well below national permitted levels, rendering the beach safe for tourists. It is clear the this will generate important health benefits for those who stay in the summer cottages. It is too early to assess the extent to which this will lead to the expansion of tourism on the beach. Neither the preparatory study nor the final report contained any attempt to describe or estimate the potential economic benefits from expanding tourism. The project has been an important public relations success for the government, generating confidence in their commitment to identifying environmental problems and assisting in the resolution of these problems. Sustainability (score: 5) The preparatory study recommended long-term monitoring of the clean-up. Continued monitoring will depend on the continued funding and staffing of the inspectorate. However, the studies undertaken produce clear and consistent evidence of the stability of the remaining contamination, so that there is little prospect of future increases in exposure. The clean-up work was undertaken on the assumption that it would be part of a decommissioning exercise for the copper processing activities. This is currently being undertaken by the environmental inspectorate.

92

Needs 3 Appropriateness 4 Comparative Advantage 4 1=poor; 3=satisfactory; 5=excellent

Efficiency Effectiveness Impact Sustainability Relevance to needs

Score 5 5 5 5

KEY LESSONS LEARNED The funding of discrete clean-up operations can produce clear results. In the absence of a method of valuing the improvements in human health and bio-diversity, it is difficult to assess the cost:benefit ratio of particular projects. In this context, it is important to have a range of different projects prepared, along with information on the severity of contamination, the numbers of people and importance of the ecosystems involved, the costs and the economic impact of clean-up. Only with this information available can an informed prioritisation of grants for clean up be undertaken. The collaboration of industry managers is essential for effective clean-up operations. Whilst discrete grants for clean-up operations may have been useful in the early years of Phare, future support for environmental clean-up investment should be more integrated with evolving national systems of legislation, incentives and funding, to ensure that the maximum continuity is achieved. Political support for projects can provide important additional benefits from effective publicity.

93

BULGARIAN DISTRICT HEATING (8/95-12/98)

3.01 MEURO

INTRODUCTION The project was identified during the 1991 Phare energy programme and demonstrates new technologies and management practises for energy efficiency in multi-dwelling units supplied by district heating. It is one of five related energy efficiency demonstration projects, covering different types of buildings. One of the other projects was assigned the additional task of drawing together and disseminating the results, conclusions and recommendations of all five. The principal project activities have been to install the equipment and analyse the results to produce conclusions on strategies for optimal district heating. The objective of the project was to raise awareness and interest in energy efficiency measures for district heated multi-dwelling homes in order for residents to adopt such measures in multi-dwelling houses supplied by district heating. The project started in August 1995 and was due for completion by the end of 1998. The project was executed by a consortium of consulting firms including HASKONING (lead consultants), BCEOM and Energoproekt Ltd (local counterparts). The demonstration was organised in one privately owned building in Plovdiv and involved a range of different measures. The final report of the project was delayed by approximately one year. Some onthe-job training was done but this seems to have been limited. The main dissemination activities were done through another project in the same Phare programme. These have included a three-day workshop and there are plans for a further campaign focusing on the target groups and using a range of techniques including mass media, leaflets and targeted information sessions. However, little has yet been achieved and there are concerns that this part of the project is not being given enough emphasis. PERFORMANCE Relevance to Needs (Score: 5) The Bulgarian energy sector is considered to be 2-4 times less efficient than the EU member state average. District heating is in crisis in most places. Energy prices have been increasing steadily and households are finding it difficult to pay their bills. However, the new government has announced a commitment to strengthening the legal framework for the energy sector, for improving energy efficiency and for restructuring and gradual privatisation of the power utilities. Furthermore, in applying for membership of the European Union and in subsequent elaboration of a Partnership Agreement, the Government of Bulgaria is committed to implementing the acquis which contains provisions and secondary legislation concerning pricing, licensing and competition, as well as environmental performance and the achievement of certain energy efficiency standards. The project was therefore highly relevant to a priority concern in Bulgaria. The decision to implement a demonstration project was based on past experience in western Europe showing the effectiveness of such projects in raising awareness on energy conservation measures amongst the residents who determine future activities both through direct pressure and political representation.
94

Appropriateness of Project Type (Score: 4) The project type was appropriate. It was important to raise public awareness about the options available to home dwellers to adopt energy saving measures. The most appropriate way to convince them of the effectiveness of such measures was to install equipment in a building and allow residents to witness the energy savings. This was done and proved effective. Comparative Advantage (Score: 4) Phares comparative advantage is its ability to fund small scale projects such as this which are very important in raising public awareness. Efficiency (Score: 4) The value of the contract was well judged. The balance of international and local consultants was efficient and cost-effective, and the procurement of equipment was efficient. Management was efficient, except that there were delays in signing the consultant's contract and long delays over the selection of the demonstration site. These delays meant that one winter season was lost and the project was delayed by a year. There were also some delays arising from disagreements about payment procedures for sub-contractors. These delays do not appear to have had any damaging effect on efficiency. The Steering Committee laudably included representatives of a range of relevant ministries. However, the participation of ministries other than the Committee of Energy has not been good. The project was highly dependent on the willingness of local residents to take part in the demonstration, and the high level of interest and support for the project provided by Plovdiv District Heating Company (North). The response from local communities was as would be expected; mixed at first then enthusiastic as people began to understand the objectives of the project and how they were expected to participate.

Effectiveness (Score: 4) The specific objective of the project is to increase awareness and interest in energy efficiency measures for district heated multi-dwelling homes. This objective has only been partially achieved, in the sense that those individuals and groups that have been directly benefiting from the project are clearly interested while a wider effect is not yet evident. Interest is particularly strong in the Plovdiv municipal institutions. However, the information dissemination component of the project was only just beginning during ERMs visit. Importantly, the results of the project have shown that while building insulation has a role to play, it is not the most cost effective means of achieving energy savings. Up until present, the received wisdom in Bulgaria was that insulation was the best approach to take in reducing bills. In fact, based on the results of the project, far greater percentage savings are to be achieved via the installation of thermostatic valves and heat allocaters. For the specific objectives of the project to be secured there is a clear need for further

95

discussion and review of the project outputs, focusing not only on financial and technical aspects, but also on wider legal and institutional aspects. This is the responsibility of the Committee on Energy and the Consultants through the mechanism of the Steering Committee. Without this further informed debate in is doubtful whether the necessary commitment from the key players can be achieved. Impact (Score: na) The overall objective of the project is to encourage the adoption of energy efficiency measures in multi-dwelling houses supplied by district heating; and to influence policymakers, planners and designers in the energy sector. The installation of equipment lead to energy savings of up to 49%, according to the Draft Final report of the project. Discussion with the residents affected by the project suggested a reduction in heating bills of 20% to 40%. . However, as the project is coming to a close at the end of this year; there is no detectable replication impact as yet. Providing that the Committee for Energy takes full ownership for the project, it may take several years and a continuing increase in heating bills for any impact to be detectable. The level of impact will depend on continued reform to energy pricing and greater political commitment to energy saving, both at national and municipal levels. This should include greater support to the institutions concerned with energy savings, both in funding and supporting legislation and changes in regulations. Sustainability (Score: na) At present, the long-term sustainability of the project is in question. The CoE is the owner of the project once the consultant input is complete but there appears to be insufficient involvement from other key institutions including the NEEA, the Ministry of Construction and Regional Development, the Ministry of Health, Ministry of Education and the Ministry of Finance. However, continuing increases in energy prices will create strong popular demand for assistance in energy savings, and this should ensure that the impact of the project is sustained.
Efficiency Effectiveness Impact Sustainability Relevance to needs Score 4 4 na na

Needs 5 Appropriateness 4 Comparative Advantage 4 1=poor; 3=satisfactory; 5=excellent

KEY LESSONS LEARNED The installation component of demonstration projects is usually relatively straight forward. By contrast, wider information dissemination and awareness is usually more problematic. The results of raising awareness are always less tangible and take longer to achieve. Often, there is a need to engage other ministries or agencies which need a lot of convincing at first. Also, effective information dissemination normally requires a lot of creative input; skills, which are habitually weakly represented on consultant teams and within the host institution.
96

Because of the difficulties of engaging wider support from demonstration projects, there is a tendency for consultants and other key players to focus too much on the technical/tangible aspects. Unless the focus can be changed and a determined and sustained effort is made to generate wider interest, the objectives of demonstration projects are seldom reached. Difficulties arise when the lead ministry is trying to generate inter-ministerial interest and awareness. Unless these other ministries are clearly involved and interested from the outset there is often insufficient sense of ownership thereafter for the project to create a significant impact. Timing is critical. Ideally, demonstration projects should be timed to coincide with the necessary economic, legal, institutional, and organisational changes that will allow replication. If the project comes too early, then the awareness raising component can be held back for a while, though not for too long since old knowledge is always less convincing. Demonstration projects do generally result in tangible outputs, when so much development assistance is perceived to be paper only. Learning by doing remains an important way of getting the message across. It is for this reason that demonstration projects must be careful to ensure that equipment installed continues to function after the consultants and contractors have left. In so doing, much of the necessary information dissemination will occur naturally. The evidence of this project is that demonstration projects can be a very effective means of convincing the public of the benefits of a new approach particularly when the subject addresses the day to day living conditions of the vast majority of the population. At the same time, they are relatively quick to implement and do not require large-scale investment. Therefore, they are an good mode of intervention for grant funding in general and Phare funding in particular. Phare has considerable experience with demonstration projects, and they are suited to the typical scale and timing of a Phare project. Demonstration projects should be used strategically as a means of promoting more fundamental change. They provide a bit of visible proof to set against the less tangible and often bland (yet essential) offerings of institutional, legal and financial reform. Thus, a key criterion for Phare in identifying the need for a demonstration project in the future relates to where it fits into an overall strategy for intervention in a given sector.

97

BULGARIA DEMAND REDUCTION (12/93 - 7/94)

(0.25 MEURO)

INTRODUCTION The project was identified in late 1992, as an addition to a government request for emergency assistance in averting a power crisis in the winter of 1992/93. The project took place during the 1993/94 winter, after efficient tendering. The project involved a range of publicity campaigns and promotional activities about energy saving, with the following outputs: 3 TV slots; 2 radio slots; bus and tram posters; 700,000 leaflets; 3 press conferences and 2 periods of activity by street promotional teams giving away calendars, badges, hats and leaflets. Some associated market research and evaluation activities were undertaken and a small study tour was also included. The main specific objective was the receipt of information on energy saving by individuals and organisations. Secondary specific objectives included recommendations for future public awareness activity. An implicit specific objective, common to all Phare programmes, was the support for building local capacity in similar projects. The overall objectives were a change in public attitudes to energy savings and, consequently, an increase in energy saving activity. The project was supervised by a Steering Committee, comprising officials from the Committee of Energy (CoE). The lead contractor was a German firm, specialising in energy planning, with local partners covering energy planning (Energoproekt) and marketing (Kres). The market research was subcontracted to a specialist firm (SOVA 5). The national electricity company (NEK) and their subsidiaries were not involved in the project.

PERFORMANCE Relevance to Needs (Score: 5) Bulgaria has the highest level of energy consumption per capita in Europe. This is partly explained by the economic structure of the country, but domestic energy consumption per capita is also very high. Thus, a programme to reduce energy demand was very relevant to Bulgaria needs. Despite this, the government's main concern in the early 1990s was to maintain energy supplies at a time when there were frequent breaks in supply and strong financial constraints. Thus, it was at the Commission's suggestion that the demand reduction project was included in the Phare programme. The National Electricity Company (NEK) was not involved in the project and, as a result, it was not realised that they were also preparing an energy reduction campaign. There is some evidence that these plans were shelved when it became clear that Phare would support a campaign. If the project did undermine a NEK initiative, then this would seriously undermine the need relevance of the project. However, it is not clear that an effective NEK campaign would have taken place. The detailed content of the messages contained in the publicity campaigns deliberately focused on energy savings possibilities, which did not require investment. The final poll confirmed that this approach was correct, since the population were more interested in
98

simple practices which do not require investment. Appropriateness (Score: 5) The provision of technical assistance to support a public awareness campaign was an appropriate type of project at the period when the project was initiated. The whole 1992 energy programme was conditional on an increase in energy prices, which was implemented in May 1993. Thus, the public awareness campaign was consistent with complementary policy reform. Experience in publicity work was rapidly growing, but was mainly involved with private marketing and it was useful to have some foreign technical assistance to facilitate exchange of experience with EU organisations. The only constraint to the relevance of the project was the lack of a partner organisation to continue similar work in future. Comparative Advantage (Score: 5) Phare seems to have a comparative advantage in supporting public awareness projects, because of the wide EU experience, which they can draw on, which is of direct relevance to neighbouring Phare countries, particularly in the context of accession. Furthermore, the fact that public awareness projects are relatively unaffected by deadlines means that Phare's relatively lengthy and inflexible procedures do not constitute a major constraint. Efficiency (Score: 5) The project outputs were generally of a good quality. The TOR required the project to produce conclusions on whether the methods of organising and managing the campaign would be a good model for future campaigns. The Final Report included some conclusions, but did not produce clear recommendations for future work. The project budget appears to have been well judged. Throughout the project, the consultants took the attitude that they should maximise expenditure on publicity. This approach might seem sensible, and perhaps reflected the skills available to the consultants. However, it may also have contributed to the lack of attention to future activities, which has limited the impact of the project. The phasing of the programme seems to have been well designed. In particular, the programme included short market research studies before the programme and at midpoint, to help in setting and revising the targeting of publicity programmes. The main exception to this, is that the training would have been more useful, if it had taken place earlier in the programme. There were shortcomings in the energy savings skills of the consultant consortium, and more experienced consultants would probably have avoided the one shortcoming of the project by involving utilities, and ensuring greater continuity. However, the local marketing skills were good and did not require international support. The campaign described a range of energy saving techniques, but the final opinion poll suggested that people would have liked more of these 'technical tips'. It is not clear whether there are more simple technical tips which could have been included, to improve the quality of the campaign.

99

Effectiveness (Score: 5) Both the project's own ex-post survey and a rapid beneficiary survey conducted for this evaluation suggests that the campaign reached a large proportion of the population and was therefore highly effective in achieving it primary objective. The project was also effective in building local capacity. Impact (Score: 3) The surveys undertaken by the project, and the rapid beneficiary survey for this evaluation, suggest that public interest in energy savings was already high before the project started. However, the surveys also suggested that the campaign did change the attitudes and behaviour of a significant proportion of the population. Indeed, the final project poll suggested that about 2 million people (ie 57% of the 72% interested in the campaign) had followed at least one piece of advice from the campaign. The final project poll found that there was greatest interest in advice which did not involve costs, validating the approach of the campaign to concentrate on low or zero cost energy saving techniques. Public interest was underlined by the fact that the poll revealed strong interest in further technical advice on saving energy. Although the TOR required an analysis of consumption patterns before and after the campaign, it is impossible to isolate the effect of the project from that of climate and economy. Sustainability (Score : 3) Political commitment to the project was ambiguous. The CoE was prepared to support the project, but NEK was only marginally involved. The absence of a clear institutional base for energy saving activity frustrated any concerns for continuity. There has been some subsequent institutional development, resulting in the creation of the National Energy Efficiency Agency (NEEA), but this still has a very limited budget and is not widely effective. In general, government still views energy efficiency as a donor activity. The rapid beneficiary survey suggests that the public interest identified in the project polls still continues to exist, but that the specific changes in attitudes and behaviour have become diluted over the past four years. The project seems to have acted as a stimulus to households to respond to price changes. The strengthening of the price signals since the project has probably reinforced the changes in practices. The project was successful in supporting the technical capacity within Bulgaria to mount an energy demand reduction campaign. This capacity still exists, and international support would not be required to organise another campaign. However, some exchange of international experience on the optimal frequency of repeating campaign could be useful, possibly in the form of a workshop.

100

Score Efficiency Effectiveness Impact Sustainability Relevance to needs 5 5 3 3 5 5 5

Needs Appropriateness Comparative Advantage

1=poor; 3=satisfactory; 5=excellent KEY LESSONS LEARNED The project suggests that occasional national campaigns of energy savings awareness can make an important contribution to reducing energy demand. The primary function of the campaigns is to reinforce energy savings attitudes, and the campaigns should also have a direct effect on demand. For a public awareness campaign to be successful, it needs to be set in the context of a long-term policy, with clear expectations for strategy, funding and institutional commitment. For example, it should take place at a time when other changes in the sector are reinforcing the message. In particular, prices should be changing in a way which strengthens the message and structural changes and economic trends should also reinforce the message that changes are required to adjust to a new set of circumstances. All these conditions were fulfilled in the Bulgarian Demand Reduction Campaign. Despite strong requirements in the TOR for evaluation activities and follow-up recommendations, these seem to have been ignored and no follow-up has been undertaken. An internal budget has not been created for further public awareness campaigns and government has not requested further assistance from donors. Consequently, some of the potential impact of this success story has been wasted. Where projects include related training activities, it is useful if these can be organised early in the project, so that trainees can feed the lessons learned into the management of the project. Also, it is important that the trainees selected are likely to be involved in the management of the project and of follow-on activities. In many cases, this means that the ideal candidates for trainees are amongst younger middle-management tiers. The funding of public awareness projects seems to suit Phare's comparative advantage, in that they are not generally linked to very specific time schedules and there is no clear source of private funds. However, the annual nature of Phare programming makes it more difficult to support a continuing activity, and Phare should be careful to try to ensure that plans exist for follow-on work. In the medium term, it is the energy utility companies which should manage energy efficiency campaigns, and Phare must support them in this, rather than displacing them by direct activity. Most CEECs are now likely to have local marketing companies who are capable of managing high quality publicity campaigns.

101

BULGARIAN TANNERY EFFLUENT (10/97-12/98) INTRODUCTION

0.10 EURO

Sevko applied for Phare Funding for mechanical treatment technology for an on-site wastewater treatment plant. The project became part of the 1994 Phare Work Programme. The project consisted of purchasing and supplying the equipment to Sevko factory. The specific objective was for the efficient operation by the factory of the new treatment equipment. The overall objective was to significantly reduce wastewater pollution by the factory into the river Rositza. PERFORMANCE Relevance to Needs (Score: 5) The present WWTP - constructed in 1978 - is out-dated, overloaded and not functioning properly. The tannery is discharging its effluent into the river Rositza, which feeds Al. Stamboliysky dam, which is situated downstream of the town and flows from there into the Yantra River, which de-waters into the Danube. The water in the reservoir before the dam is being used for irrigation and plans exist to use it for drinking water as well. The water in the reservoir is reported to suffer from eutrophication. The high nitrogen load from Sevko, is expected to contribute substantially to the eutrophication problems. In the Danube basin nitrogen and phosphorous have been identified as priority pollutants. From the available data appears that Sevko is a major polluter of surface water and reduction of the emission of pollutants is urgently required. Appropriateness of Project Type (Score: 3) There were two concerns over the appropriateness of the project, both relating to the lack of demonstration effect. Firstly there was no component in the project for disseminating the results of the investment to other tanneries. Secondly choice of technology is questionable as other tanneries need biological, not mechanical treatment, to comply with national legislation. Comparative Advantage (Score 4) Phares comparative advantage was contributing a grant to the National Environment Fund thus making the subsequent loan to the industrial beneficiary more affordable. This is important when trying to promote pollution abatement investments in weak regulatory environments. Efficiency (Score: 3) The project was managed efficiently. Phare co-ordinated payment and delivery without the occurrence of any significant problem. It further arranged for a visit to the plant by the Investment Control Team, to check on quality of the supplies delivered by Ecolex. However, the process leading up to the signing of the contract was slow. This led to, perhaps unnecessarily large, problems concerning the actual value of funds allocated by the Fund (due to inflation and exchange rate fluctuation) - Phare money itself was allocated in ECU. It further led to a problem when disbursement for 93 and 94 projects was to be closed
102

by 30 June 1998, and not all payments had been made yet. Effectiveness & Impact (Scores: 4 & 2) With the operation and official opening of the mechanical treatment technology still a few weeks away, no firm conclusion can be drawn with regard to the level of effectiveness or impact of the project. However, the technology has been installed. Emission levels will be measurable shortly after the plant starts operating. It was stated that the reduction of contamination by the plant will be decreased by only around 30% with the installation of the mechanical treatment plant. Full treatment including biological treatment will be needed to reduce emission levels to comply with Bulgarian legislation. Sustainability (Score: 2) The direct sustainability of the project will depend on whether funds and staff will be available and capable to maintain the WWTP at Sevko. The indirect sustainability of the project (what is that ) is undermined by its limited scope. It appears that the value of the project beyond pollution abatement for the Sevko tannery was insufficiently considered during the project identification process. Current regulatory and economic climate also are not conducive for securing sustainable results. Score
Efficiency Effectiveness Impact Sustainability Relevance to needs 3 4 2 2 5 3 4

Needs Appropriateness Comparative Advantage

1=poor; 3=satisfactory; 5=excellent KEY LESSONS LEARNED Phare must ensure it has the technical capacity to identify sound the most cost-effective investment / supply projects, based on world-wide best available techniques (BAT) and best environmental practices (BEP). Possibly such capacity can be found in-house, but is likely that specialists must be contracted. Furthermore, more careful assessment of investment need and capacity in this early stage is essential. Phare should consider the advantages of adding an explicit demo component to its procurement projects. The sector targeted by a project should be stimulated to replicate technology, and Phare should consider playing a more pro-active role in this. It appears in accordance with sound economic principles to provide credits to industry rather than grants. Nonetheless, it need to be considered if the economic and regulatory climate is sufficiently conducive, so that industry can afford to invest. If not, than a grant may be more appropriate. Investment projects should be combined with explicit plans for improving the investment climate in the country itself. Providing loans can aid the transition from central to market economy. The switch is well-timed where a country is in the middle of privatisation. Phare needs to carefully
103

assess if economic development has been sufficient for industry to be able to afford loans (and not go bankrupt). Phare needs to strengthen its methodology to weigh macroeconomic issues before projects are designed. When it is decided that a loan is most appropriate, it is doubtful that Phare has a comparative advantage in this area. Providing loans, especially in economically unstable countries, requires strong financial expertise. It is likely that international financial institutions have the comparative advantage.

104

CZECH MUNICIPAL LOANS (8/94-ONGOING)

0.40 MEURO

INTRODUCTION The project provided 0.40 MEURO for loans to four municipalities to assist in conversion to gas, either for the replacement of coal boilers with gas boilers (both in central heating facilities and municipal buildings) or for the distribution of gas to individual houses. There was no technical assistance, and government selected the first loans to be made, without assistance from the project. The funds are administered by a commercial bank according to a simple contract with each municipality. Repayments have been made into a Revolving Fund and the future of this fund is currently being discussed. The overall objectives of this project were to increase investment in conversion to gas, and hence to reduce consumption of coal and associated air pollution. The specific objectives of the project are the use of equipment by municipality to provide, in a more efficient manner, heating services and the more efficient management of investments for heating by municipalities. The specific loans are as follows: a gas distribution network (400,000 ECU); conversion of one central heating scheme; one gas network; boilers for nine public buildings, including 5 schools (250,000 ECU); conversion to gas of two central heating schemes (400,000 ECU), and; boilers for 5 schools (100,000 ECU). PERFORMANCE Relevance to Needs (Score: 5) In many municipalities, the main source of localised air pollution has been the use of lignite for domestic heating. In recognition of this gas conversion was a national priority. Indeed, the government had already decided to support a major national programme of gas conversion, though the State Environment Fund with a budget of Kc 6 bn (170 MECU). Appropriateness of Project Type (Score: 5) There is concern over whether the loans substituted financing to be raised by the municipalities. The municipalities selected already had well-developed plans for conversion to gas, and had organised their own lending programme. The main incremental effect of the Phare was to reduce the cost of borrowing, and thus hasten and expand investment plans. As Phare lending amounted to less than 10% of total lending, it is very unlikely that the Phare programme had much impact on the decision to proceed with conversion to gas. Against this background, it could be argued that the Phare project would have had a greater impact if it had provided more widespread training and support to middle-level municipalities to help them pursue investments which might otherwise not have taken place. Comparative Advantage (Score: 3) Phares comparative advantage is that its grant contribution to the fund makes the loans more affordable. This is particularly important when promoting municipalities to carry out
105

environmental investments. Efficiency (Score: 4) The outputs of the project are the loans to the municipalities to enable them to invest in conversion of their heating systems from coal to gas boilers. Overall there have been only minor implementation problems in the project. The loan scheme was efficiently managed by the bank and the municipalities. The costs of the Phare financed work were the same as those of the other gas conversion investments. The loans appear to have been used to purchase high quality equipment and was contract work undertaken quickly. Effectiveness (Score: 4) The four loans financed the provision of equipment for central heating systems and the heating of public buildings. All equipment has been installed within the last year and is now operational. The 2 loans for new gas networks have also been effective. There has been a good use of the new gas networks with 90% of houses having connections in areas where gas supply has been available for several months. Impact (Score: 4) The overall impacts of the project are reduced emissions by municipal heating companies and the replication of lending by attraction of other funds. It is difficult to measure the reduction in emissions resulting from the project. We estimate that the total heating capacity provided on the first round of loans was about 10MW, indicating the savings in coal emissions generated by the project. However, the beneficiary municipalities would probably have completed the same gas conversion activities without Phare funds, although a year or two later. Thus the main impact of the project was to subsidise the borrowing activities of the municipalities and possibly to bring forward a small part of their gas conversion programme. Sustainability (Score: 2) The Revolving Fund will be managed by the Ministry of Environment, though the EU Delegation will retain some influence over the fund, both through the MoU and through its ability to share experience with the Ministry of Environment and encourage them to adapt activities to reflect EU experience. To-date the Fund has had a good experience with loan repayments. The first repayment of the loans was made in November 1997 and the repayments will be completed in two further tranches in November 1998 and 1999. The repayments will then be used for subsequent loans with part of the interest repayment to cover the cost of operating the Fund.
Efficiency Effectiveness Impact Sustainability Relevance to needs Score 4 4 4 2 5 5 3

Needs Appropriateness Comparative Advantage 1=poor; 3=satisfactory; 5=excellent

106

KEY LESSONS LEARNED Municipalities are important managers of environmental investment, particularly for heating and water treatment. Where municipalities have strong finances, as in the case of the Czech project, it is possible to provide loans which may be repaid from general revenue. However, in the vast majority of municipalities in Central and Eastern Europe (CEE), municipal finances are weak and repayment may be dependent on careful planning. In many cases, it is likely that a degree of management and financial independence for heating and water facilities will be required to mobilise revenue from the services for repayment of loans. In many cases, revenue generation will not be sufficient to repay loans, and an element of grant will be required. In several CEE countries, there are now major national environmental funds, which rely mainly on national sources of income and devote a substantial part of their resources to loans and grants for municipalities. Although it may be possible to run a parallel loan scheme, supported by Phare funds, there is a risk that this will undermine national funds. Such a scheme also makes important demands on scarce human resources. Therefore, Phare support should be integrated with the management of national funds, unless there are serious questions for doubting the efficiency of the funds, or if there are particular gaps in the coverage of the funds. Municipalities are major investors in the reduction of pollution and national policies on municipal finance need to ensure that municipalities have access to resources, both income and capital. Municipal financing of environmental investment also requires supporting regulatory and economic incentives, including environmental penalties and/or charges. It also needs to take place at the same time as developments in the management of environmental facilities. Ideally, newly independent management should be involved in the design of new facilities, with some responsibility for repayment plans. Support needs to be provided to CEEC governments to help them understand how Structural Funds are managed, so that they can align their mechanisms for municipal and regional investment to be compatible with Structural Funds. Although the Czech Ministry of Finance has information on the financial situation of all municipalities, this is not widely analysed, particularly from the perspective of environmental investment capacity. As future EU support for environmental investment is likely to involve local government, if the models currently applied to EU Structural Funds are applied, it would be useful to provide some assistance to the Ministry of Finance to help in analysing the available data and promoting debate about the regional targeting of support for environmental investment. This data is currently treated as confidential in the Czech Republic, and it may be necessary to require a more open approach to regional planning, for such assistance to be effective. Some parts of the Ministry of Finance have already requested this change.

107

CZECH MIT SUPPORT

(4/97 - 6/97)

0.08 MEURO

INTRODUCTION The project was put together in an opportunistic fashion at the end of the programme, to help the Ministry of Industry and Trade (MIT) finalise key activities, utilising interest on the programme account. Three activities were covered: support for energy policy implementation; evaluation of the Energy Saving Fund (ESF); and energy efficiency in the chemical industry. Much of the support consisted of ad hoc advice to officials and general support to the PMU, such as preparation of TOR for future projects, response to OMAS requests; implementation of the ESF and representational trips for the Ministry. In addition to the activities included in the TOR, the consultants spent much time on other activities, including: ad hoc advice to officials; preparation of TOR for future projects; general support to the PMU; response to OMAS requests and missions; implementation of the ESF; and representational trips for MIT. The specific objectives were to: finalise policy, strategic, legislative and regulatory framework; prepare new Phare programmes; improve the working of the Energy Saving Fund; and implement energy strategy for chemical industry. The overall objective of the project was to contribute to making the energy sector more efficient through reform and investment. The PMU prepared the project. The consultants selected were senior PMU officials whose contracts were with MIT were coming to an end. The contract was handled through the framework contract, though the framework consultants provided no substantial input. The work began in September 1997 and ran until May 1998, when the Final Report was submitted. The Final Report has a large Annex in the Czech language, although according to the proposal all reports are to be submitted in both languages. PERFORMANCE Relevance to Needs Score: ( 2 5 3)

The last official version of the Czech Energy Policy was approved in 1992. By the time of the project, there had been many significant changes in the sector and the need to revise the policy was accepted on all sides. Key changes included the division of Czechoslovakia, various privatisations and linking the Czech electrical system to UCPTE. Completion of the nuclear power plant at Temelin remained a contentious issue. In addition the policy needed to reflect aspects of the EU acquis. Various draft Energy Policies were produced in 1996 and 1997, but several issues remained contentious, including : pricing; brown coal extraction; the completion of Temelin; reconciliation with energy savings and the new Environmental Act; and the regulatory activities of the new Energy Regulatory Agency. During the same period there was also an increasing recognition of the lack of progress towards legislative alignment. Thus, there had been substantial movement within government on energy policy reform and there was only moderate need for further coordination. The Ministry was subject to criticism over the delays in implementing the Energy Efficiency
108

Fund, and it was highly relevant to allocate short-term management resources to this. Finally, the Chemical Industry faced particularly large problems resulting from highly polluting fuels and poor efficiency, and it was therefore useful to provide specific support for this. Appropriateness of Project Type (Score: 5) There was a strong need within MIT for support of the kind that was provided by the project. Indeed, the TOR reflected only poorly the many requirements. The provision of long-term support accessible to MIT staff on a daily basis, complemented by short-term specialists was generally welcomed. Comparative advantage (Score: 3) Phare has little comparative advantage in supporting ad hoc local consulting. There are risks associated with conflicts of personal interest and Phare depends on relatively cumbersome procedures to maintain independence from such pressures. The best source of finance for this sort of advice is probably the government itself, which would then have the greatest incentives for making the best use of ad hoc local consultants. Efficiency (Score: 1 3 5)

The Final Report is poorly presented and not well focused on the TOR or on revised objectives. In particular, the Final Report provides limited information on any advice provided on updating energy policy and on draft regulatory principles and norms. There is also little documentary evidence of support to Phare programming. However, there is some documentary evidence that the general support provided to the PMU was appreciated by MIT. Support for the ESF formed a major part of the TOR. There is little documentary evidence of this support and much of the report is taken up with general description of decisionmaking and subsidy for energy efficiency. However, it is widely recognised that the project played an important role in the initial stages of implementation of the ESF, through participation of the consultants in discussions and meetings. The support provided to the chemical industry was documented in some detail in the Czech sections of the Final Report. The industry itself seems to have found the support useful. Cost-effectiveness. The volume and quality of advice given seems to have been costeffective, though savings could have been made by excluding the project from the Phare framework agreement. Management. Management of the project was light. There seems to have been no Steering Committee or regular meetings to examine progress, which is not surprising because of the ad hoc nature of much of the work. Effectiveness (Score: 1 3 5)

The project seems to have made little demonstrable contribution to the development of the Energy Policy, although the Ministry maintains that useful advice was provided on a day to day basis. There was some effective contribution to the development of the regulatory framework and the establishment of the Regulatory Agency.

109

The project made an effective contribution to developing the next stages of the Phare programme. There was a major turnaround in the success of the ESF during the project, but it is difficult to attribute any of this to the project. The study on energy efficiency in the chemical industry is the main reference document for the industry and has been influential in promoting energy efficiency awareness and planning in the industry. Impact Score (Score 1 3 3)

The project probably made a small impact on the policies and plans of the new government to promote energy efficiency. The small impact of the project on the establishment of the ESF will have had some effect on energy savings, but the activities of ESF are, as yet, very small. It is too early to identify an impact of the project on energy saving in the chemical industry. Sustainability (Score: 1 3 3)

In those areas were the project was successful the work was sustained by subsequent government and Phare work. The integration of the project within government structures ensured full government ownership and good institutional transfer of experience. Although the future of ESF is not yet clear, the general concept of lending to promote energy efficiency should be sustained. The indications are that the project study has successfully initiated a sustained programme of work within the chemical industry.

Score
A Needs 2 Appropriateness 5 Comparative Advantage 3 Efficiency 1 Effectiveness 1 Impact 1 Sustainability 1 1=poor; 3=satisfactory; 5=excellent Relevance to needs B 5 C 3

3 3 3 3

5 5 3 3

KEY LESSONS LEARNED The execution of the project was rather unsatisfactory. Due regard was not paid to observing the ToR; the purpose of the Inception Report was missed; the Final report does not do justice to the work done; many of the key outputs are only in Czech and not even a summary is provided in English. The discipline of agreeing TOR and reporting is important to ensure efficiency, commitment and dissemination of project results. Relatively small sums of money, made available for local specialists can be effective in overcoming specific difficulties, and may be useful in providing follow-up from studies to implementation. There now exists in CEECs a growing pool of local consultants who are well-informed about EU practices and requirements, as well as being intimately familiar with national conditions. These consultants are a very valuable resource for CEEC governments. Because they are also highly valued by the private sector, they can be expensive. Phare
110

should continue funding such local consultation for government, at least during a further transition period, whilst well-qualified advice is in short supply. It may be desirable to supplement large studies and technical assistance projects with budgets held jointly by the delegation and the PMU for local consultants to carry out clearly specified follow-up tasks, following clear procedures and, possibly, fixed fee rates. This arrangement could replace many of the functions of the framework contracts. The value of such arrangements is likely to be further enhanced if the lower limit for projects rises to 2 MECU. Phare would be the most appropriate donor to fund such activity.

111

CZECH NGO SUPPORT

(1996)

0.45 MEURO

INTRODUCTION The project took place largely during 1996, as part of three projects supported by Phare designed to support education for environmentally sustainable development. The other two projects dealt with formal education, particularly at higher levels, whilst the NGO project support a wide range of NGOs. The project was identified in the 1991 programme, which was delayed and revised to reflect the division of Czechoslovakia. Table 1 Summary of Evaluation Survey
How important was the project1 Were Effect on Effect follow-on member on reveffects -ship enue important Effect Comments Timing on on manage- of work benefic ment done2 -iaries Would you have done work without Phare?

Videos Rezekvitek A-VIDEO Skyfilm Spolecnost CSOP Kosenke Publications UNESCO MAB Knezice/Chalou pky PRODOS TASK klub Special Activities Slunakov Nadace Agentura Koniklec Hnut Duha

c c b b c c b b c

yes yes yes yes yes yes yes minor some

minor no no some no yes no

minor 1000s TV no wide no wide

none delays good inflexible inflexible

c b c c c c c a c

yes yes difficult difficult difficult failed try to yes difficult

no yes no yes

minor 15,000 delays + Yes complex 4,200+ delays

b b c

yes yes high

yes some yes

yes some yes

60,000 good + 100s none good

a c c c c b b c c a c c

try to yes try to yes yes, but yes, but yes yes try to yes yes difficult

SEVER b yes complex ROSA b yes complex Nadace A b yes good Nadace B b high none CSOP b high simple VERONICA b yes simple Zeleny kruh c no ZOCSOP Botic b yes good ZOCSOP c yes inflexible Veronica 1 a = the most, b = one of the most, c = important, d = not important 2 main work actually before: a) discussions, b) contract, c) payment

30,000 + yes 100s minor minor 1000s no no 4000+ no no 1000s indirect no Yes yes yes 6000+ no no 30++ yes no 100+ some

The project supported the production of 6 videos and 9 publications, along with 8 separate public awareness activities, such as presentations at exhibitions, and 8 NGO strengthening and skill transfer projects. The specific objective of the project was to support the activities of the NGOs. The overall objective of the project was to increase public awareness of environmental issues.

112

PERFORMANCE Relevance to Needs (Score: 5) The importance of the role of NGOs in the promotion of a sustainable economic society was widely recognised in the Czech Republic and within the Phare programme. Most Phare support for NGOs was provided through a specific programme of support to governance, with activities across many sectors, as well as through the support to REC. However, the Ministry of Environment chose to focus additional attention on the activities of environmental NGOs, to try to raise public support for environmental management. This decision was enlightened, since the basis of the growth in political support and commercial pressure in western countries has been the strong growth in public interest in environmental issues. Appropriateness of Project Type (Score: 5) Providing relatively large scale financial support to NGOs is a complex and difficult process, as it is easy to destroy the focussed commitment of key members of NGOs by generating internal divisions over how to utilise external finance. The way in which the programme was managed minimised this risk by providing small grants to a wide range of NGOs and allowing the NGOs to identify and propose the activities which they wished to have financed. Comparative Advantage (Score: 3) Support to NGOs is best provided from institutions which are as far removed from government as possible, in order to minimise the risks of a loss of independence. The Phare national programmes are normally under relatively close government management, which means that support for NGOs would have to be organised through a channel which retains independence from government. Efficiency (Score: 5) The quality of the activities supported by the project was generally high. Twenty six of the NGOs fully completed their activities, three partially completed their activities and one did not complete. In view of the relatively short period of operation of many of the NGOs this record is remarkable, and shows that the NGO selection process was well managed. The project was efficiently tendered and managed by a local consulting office (DHV), who drew up sub-contracts with the NGOs selected. Relations between DHV and the many participating NGOs were generally good. The evaluation survey of NGOs shows that there were relatively few problems with management. Those NGOs who had experience of dealing with international donors, appreciated the simplicity of the procedures used by DHV, whilst those who had little experience found the procedures complex. This suggests that DHV were successful in simplifying the procedures which they accepted from Phare. There were some problems for the NGOs in absorbing the cash flow requirements of the project, but these were resolved. At the end of the project, the remaining savings in the budget were allocated to provide further support to those activities, which were deemed to have been most successful, as measured by the public response to products. This was a good way of maximising the
113

efficiency of the resources. Effectiveness (Score: 4) The evaluation survey suggests that all participating NGOs were very positive about the project. Most of the NGOs reported that the project activity was one of the most important activities they had been able to do. About half the NGOs reported that the project had some effect on membership and revenue. The one area of concern about the project is that several of the NGOs reported that they were already committed to the activities before Phare support was provided. Indeed, most NGOs reported that they would have tried to do the activities even without Phare support. However, many of them suggested that they would not have been able to do the activities as well, or on as large a scale. Impact (Score: 4) The Evaluation survey suggests that over 100,000 people were affected by the project's activities. Those projects which reached smaller numbers of people were usually involved in more focussed activities, often targeted at schoolchildren. Because very few of the activities had a national publicity campaign, it is not possible to undertake a general public opinion poll, though this could have produced results in several of the focus locations. However, demand for all the products seems to have been high and NGOs did not report any difficulties in distributing the products which the project supported. There is some concern that the project financed mainly the production of materials, and did not provide support for dissemination, relying instead on the activities of the NGOs. Where the project did attempt to use funds to support the distribution of books, this was rejected by the EU delegation, on the suprising grounds that Phare could not provide support which might benefit the private bookshops involved in distribution. In practice, the NGOs were well organised to manage distribution and leaving responsibility for dissemination with NGOs probably assisted in ensuring full ownership and maximum sustainability of activities. Sustainability (Score: 4) The full ownership of the project by the NGOs should ensure a strong sustainability of activities. Almost all NGOs reported important follow-on effects. Score
Relevance to needs Needs Appropriateness Comparative Advantage 5 5 3 5 4 4 4

Efficiency Effectiveness Impact Sustainability

1=poor; 3=satisfactory; 5=excellent

114

KEY LESSONS LEARNED There is a huge potential demand for NGO public awareness products and activities. Supporting NGOs is an important complement to other forms of support in the environment sector, as political commitment and commercial pressure are strongly affected by public opinion. Phare can provide effective support for NGOs. The model of using a local consultant to distribute small amounts to many NGOs seems to suit the demands and absorptive capacity of NGOs. Operating through a local consultant also has the advantage of enabling them to simplify the procedures, which NGOs have to follow. Some degree of independent monitoring of NGO support projects is useful, to try to ensure that resources are focussed on the most successful. These conclusions support the conclusions and recommendations of the Phare Partnership Programme.

115

ESTONIA WATER MANAGEMENT

(3/98-ONGOING)

0.90 MEURO

INTRODUCTION The project comprises two types of institutional strengthening: firstly, integrated water resource management for Ida-Viru County (component I); and, secondly, and business planning and rehabilitation for the Jarve Vesi Water Company (component II). The activities in both these components are based around on-the-job participation in pratical work, to support formal training. There are a total of 25 consultants (15 international and 10 local) and 7 staff from Eesti Veervack, the national water company, working on this project. Training. A training needs assessment was completed by June 1998. This led to the design of a training programme for the Ministry of Environment, County Environment Department and Kohtle-Jarve water company. The training included: databases; GIS; IWRMP; water supply management; water tariffs and business planning; leakage control management; and cost-effective distribution management. The training courses were supplemented by a study tour to Denmark for Component I and one to Finland for Component II. There will be 2 seminars on integrated water resource planning attended by all counties. The Eesti Veevark training manager worked closely with the international and local consultants in setting up and carrying out the training courses. To-date approximately 75% of the intended training courses have been delivered with the remainder to be provided in 1999. Component I. The consultants have been working with the County Environment Department in collecting the data required in the planning stage, setting up the databases, and developing the integrated water resource plan. One of the key components is for the consultants to show the Department how to organise and use all the previously collected data for water management (including updating of REDOS groundwater database). The project is to provide this assistance in a form which helps the Department in the preparation of a development plan. The approach has been to ask the staff to complete certain tasks which feed into the overall processes. However it appears that the consultants are preparing large sections of the report and not leaving them to the County recipient specialists. Eesti Veevark has one water resource specialist who works with the Environment Department and consultants to understand what they are doing and to assist in their tasks. However he does not appear to work very closely with his counterparts. Component II a. The project will assist local officials in preparing a report on improvement of the public water supplies in the project area with a planning horizon of 15 years including: assessments of the urban water supply situation and the institutional and financial situation for water supply; and prioritised recommendations and cost estimates for improvements. The approach taken by the consultants has been to teach Jarve Vesi staff (primarily the Director) the key issues and rules of the game e.g. on what grounds are they trying to raise tariffs - inflation, operation cost recovery, or operation and investment cost recovery. Then to provide them with options but let them come to their own decisions as to which is the most appropriate option. This approach has been taken on the development of the business plan in particular. The main output to date has been a working paper financial and
116

administrative review of the system to act as a platform for discussion in the development of a five year business plan. Component II b. The project will involve an analysis of the Johvi-Ahtme water distribution network, and the provision of some equipment for the survey needed to calibrate the system. It will also include a detailed rehabilitation and renewal plan for the Johve-Ahtme distribution system, along with the identification and procurement of the equipment, materials and works needed to carry out the priority part of the rehabilitation plan. The consultants with engineers from Jarve Vesi have reviewed the network and developed the network modelling skills, along with a pilot plan for rehabilitation. Due to the technical problems (identified in the technical review) and the complex ownership structure of the network and supply system it has been agreed for the pilot project to focus on rehabilitating and improving common assets. This includes the water intake facility, pumping station and main water supply pipe. The project will also include some pipe network rehabilitation. Jarve Vesi, with assistance from Eesti Veevark are now in the process of preparing tender document. Relevance to Needs (Score: 4) The project was developed from the recommendations for action in the previous REDOS I Project. There is a question as to whether the project was the most strategic for the water sector and whether assistance to another water company may have been more useful for replicating efforts elsewhere. However the integral involvement of Eesti Veervack means that the experiences gained from these projects have the potential to be transferred to other interested institutions. Appropriateness of Project Type (Score: 5) The combination of different training activities, including training of the national institution to provide further training, was ideal for achieving the objectives of the project. Comparative Advantage (Score: 4) Phare should have a comparative advantage in supporting such training projects, because of the range of EU expertise on which it can draw.

Efficiency (Score: 4) There have been no significant delays and the quality of reporting to date from the consultants is high. The consultants appear to be working well with beneficiary personnel in most cases, though there are some instances where there has been insufficient beneficiary participation. Effectiveness (Score: 3) Effectiveness can only be estimated as many tasks in the project have not been completed. Effectiveness is scored for each of the 3 beneficiaries. The scores are based on interviews and not report outputs as no outputs were complete. Ida-Virumaa County Environment Department. The project seems to be succeeding in

117

capacity building regarding monitoring and data organisation. It has so far been less effective on building planning and decision making capabilities, but this generally takes longer than purely technical capacity-building, and greater success should be achieved later in the project. The overall effectiveness is constrained by limited human resources in the beneficiary institution. Jarve Vesi. The objective in Jarve Vesi is to increase the capacity of Jarve Vesi staff in: water supply management systems; financial control and accounting; and engineering (including leak detection, network modelling and operation). There is also an objective of increasing the capacity of Eesti Veevark to carry out future training in the management of water supply systems and leakage control. The institutional strengthening of this water company is particularly difficult as it has a complicated system and attitudes inherited from the Soviet system. The project appears effective in developing the attitudes of the senior management and raising their awareness of key financial and techncial issues to improve the running of their water company and network. Furthermore, they appear ready to adopt the business plan under development. It is difficult to gauge at this stage if the assistance will be sufficient to make significant long term changes. Eesti Veevark believes it benefited greatly from working with the international consultants in preparing the training courses which it can now develop for other water companies and interested institutions. However it believes that it has not gained any major new technical skill or service as it did not gain that much benefit from working with either of the other beneficiaries. The project will end up strengthening EVs existing services offered to other water companies as opposed to providing it with a range of new services. However, Eesti Veevark officials believe that there will be little interest from most counties in integrated water resources management, which will constrain their ability to have a major sustained impact. Impact (Score: na ) The overall objectives are: the implementation of the IWRM in Ida-Viru; the development of IWRM plans in other counties; implementation of the rehabilitation plan by Jarve Vesi; and dissemination and training on IWRM to other water companies by Eesti Veevark. The project is still at too early a stage to estimate impact. However, the indications are that there will be substantial impact for Ida-Viru and Jarve Vesi. The wider dissemination may be more difficult, as other counties may be less receptive to new management techniques. Sustainability (Score: na) Sustainability will depend greatly on whether those few personnel who have benefitted from assistance and collaboration with consultants continue in their jobs and be provided with assistants. However the emphasis on leaving tools such as the database also means that, if personnel leave, new staff can takeover and use the planning tools relatively easily.

118

Score
Relevance Needs Appropriateness Comparative Advantage 4 5 4 4 3 na na

Efficiency Effectiveness Impact Sustainability 1=poor; 3=satisfactory; 5=excellent

KEY LESSONS LEARNED This project provides a good example of integrated and comprehensive planning trying to include all components which will increase the probability that the project will be successful ie have the desired impact. This is in contrast to many projects which are more discrete and rely on too many external factors to be in place and therefore often have their impact constrained. Phare projects should not merely channel assistance into institutions with limited human resources because it is better than nothing. During project design and justification there is a need to consider how large the project should be, compared with absorptive capacity, and whether the beneficiary institution has the incentive and capacity to increase its personnel targeted for capacity building during the project. Depending on the dynamics of the potential beneficiary institution, possibilities should be reviewed to replace directors with young dynamic modern thinking managers (e.g. in Tartu) instead of spending years and millions of EURO to educate existing directors (who may retire in five years anyway). Implementation of this recommendation would be difficult and would probably depend on mutual consent from all parties. As Phare has limited financial and human resources to implement projects it must prioritise replication of projects. It is not sufficient to finance demonstration projects. Phare should finance projects which themselves develop the replication abilities incountry. Potential examples are given below. If projects aim to be replicated across the country there needs to be a qualified organisation in place with appropriate incentives to carry out or ensure the replication of the service/action. Therefore part of the project must be building the capabilities of the assigned replicating agency. In this project Eesti Veevark was a beneficiary expected to replicate IWRMP training across the country. Also a good approach adopted was permitting other institutions to participate as observers in the Advisory Committee Meetings so that they could understand how to manage such a project and what type of outcomes (and quality of outputs) they should expect. There does not need to be a central coordinating agency, like Eesti Veervack. If a utility is strengthened (like Jarve Vesi) it could then share its information and new skills with the other utilities. This could be charged as a consulting service, which also further benefits the utility being targeted for strengthening. This way national expertise and experience can be maximised in further sectoral development. Is it better to focus effort on one company and push through change on all aspects from the top down or try to improve one aspect across all water companies? The difficulties arising in this project demonstrate the need for broad projects, which address all the critical barriers to change. Improving financial planning capabilities can only occur through a change in attitude, usually at Director level. However, changing attitude can only occur through institutional reviews, identifying problems and suggesting solutions,
119

which can then be discussed between management and consultants. Therefore just one component (training or institutional reviews) will probably not work in isolation.

120

ESTONIA NETWORK REHABILITATION (5/96-12/98)

0.30 MEURO

THE MAIN ACTIVITIES The main activities were: supply of equipment for leak detection, sewer cleaning and CCTV inspections to Eesti Veervack (the Estonian water company which provides consulting services to the municipal water companies), together with execution of pilot surveys in Narva, Tartu and Rapla municipal water companies. This was the 2nd contract and was carried out by a Finnish consortium led by the contractor Painehuutelu; supply of pipes and fittings for the pilot water main network rehabilitation component. This was the third contract and was carried out by CNH contractors. In addition, there was a preparatory first consulting contract to assist Eesti Veervack, in preparing tender documents and tender evaluations for the other Phare funded components of the project, and assist in supervision of the pilot network rehabilitation, thereby providing on-the-job training. The outputs are project management reports, contractual documents, training courses, operations and maintenance manuals, cleaning and leak detection surveys, maps showing the location and construction of the renovated and cleaned sections of the piping network in each pilot municipality. The specific objectives are: ownership by Eesti Veervack of leak detection, cleaning and CCTV inspection equipment; Eesti Veervack staff trained in use of equipment; pilot cleaning service in the three municipalities (Narva, Tartu, Rapla); supply of materials (pipes etc) for the three pilot projects; and civil works for rehabilitation. The overall objectives of the project are as follows: Eesti Veervack to carry out cleaning and leakage detection services for water companies (other than pilot) Pilot municipalities to install supplied materials Improved understanding by municipalities of the need for network cleaning and leakage detection and reduction and plans for future system cleaning and repair. PERFORMANCE Relevance to Needs (Score: 4) Leakage in many of the water company networks is high (40%). The EBRD loan was to cover investment in water supply and wastewater treatment. Donor money was needed for pipe rehabilitation. The only question is whether Tartu and Narva were the best selections (as opposed to other towns) as now appears not to have critically needed the support.
121

However when the pilot water companies were selected in 1996 the level of development may have been significantly lower (as rapid change over last 2 years due to progressive management). Appropriateness of Project Type (Score: 5) Because of the clear requirements in the water sector, strengthening the key institutions to deal with the investment required was an appropriate type of support for Phare to provide. The combination of institutional strengthening and some equipment was particularly appropriate. Comparative Advantage (Score: 4) Whilst the combination of institutional strengthening and related investment was an effective combination, this type of integrated and phased approach is difficult for Phare to manage, in the light of the rigidities and delays in Phare procedures. Efficiency (Score: 3) The overall project will finish by the intended deadline. However both the 2nd and third contracts were delayed. The 3rd contract was delayed due to Phares hesitance to accept a guarantee from an insurance company instead of a bank. The quality of equipment and materials and training supplied was high indicating good TOR and supervision and selection of contractors. However the delay in starting the 2nd contract meant that 3rd contract had to be issued (with technical specifications) before the survey work was carried out. This meant that the materials were not necessarily those identified as needed from the pilot surveys. However, because much of the equipment was standard, the extent of sub-optimal specification was limited. Effectiveness (Score: 5) The project successfully organised and supervised the supply of equipment to Eesti Veervack and the municipalities and all the training courses. The material is now being supplied later than expected but this is not causing any difficulties to the beneficiaries. The main benefit has been from the practical working with new equipment supplied and seeing its efficiency and effectiveness. This made the workers enthusiastic and confident to use it in the future. Impact (Score: 5) The main benefit was intended for Eesti Veervack. This main objective has been successfully achieved. Eesti Veevarck is now very active carrying out sewer cleaning and to a lesser extent leak detection services to water companies around Estonia. There is therefore strong evidence of the transfer of technical and consulting skills to Eesti Veevarck and national demand for the equipment involved and associated services. The actual benefit to the municipalities was limited because of their respective existing situations. Narva water company were very enthusiastic about their exposure to the new equipment and benefited from seeing work carried out in their system. They purchased their own leak detection equipment and repaired all leakage identified during the

122

contractors survey. They also plan to hire Eesti Veervack for further sewer cleaning. However they lacked understanding of the role of the activities in their overall planning for network rehabilitation. Tartu was less impacted by the project. It has a progressive management and already had plans in place to purchase sewer cleaning equipment and to systematically rehabilitate their entire network. In their case the surveys, practical work and supply of materials were helpful but not critical in increasing their awareness of network rehabilitation. Rapla water company was also not significantly impacted by the project. Whilst benefiting from being exposed to the new equipment and services being a small company with no budget they believe they cannot afford to hire Eesti Veevark for further leak detection or regular cleaning services. Sustainability (Score: 4) The level of impact to Eesti Veevarck provides strong evidence that the project is highly sustainable. This is because the correct incentives are in place for both Eesti Veervack and the pilot water companies and because the training was sufficient for skill transfer so that the Estonians can now use competently the equipment supplied. Score
Relevance to needs Needs Appropriateness Comparative Advantage Efficiency Effectiveness Impact Sustainability 4 5 4 3 5 5 4

1=poor; 3=satisfactory; 5=excellent KEY LESSONS LEARNED Phare should continue to consider investment packages where supply of physical assets are supplemented by training in both techniques to utilise the physical assets (in this case use of equipment and interpretation of results) and also in planning (e.g. the role of the equipment, when to use it, when it is more cost-effective to use other technical options). From this project it is evident that in one Municipal water company whilst the training was valuable in educating staff on how to use equipment (leak survey and sewer cleaning) and indeed stimulated their enthusiasm for using the equipment they did not have the capacity to think strategically about the role of the equipment - how often to monitor vis--vis cost. One reason for the projects success was because it was part of a long/large well planned investment strategy for the whole sector (supported by EBRD) with a high level of interest from the benefactors. The project was a success because Eesti Veervack was allocated responsibility for implementation of project outputs. Importantly the implementing agency had experience in the water sector and a financial incentive to succeed (although a question as to how efficient and motivated Eesti Veervack actually has been was made by Tartu water company). Project selection in a country programme to achieve EU accession goals can either
123

concentrate resources on specific utilities (e.g. municipal water companies) or spread resources across a sector (e.g. water). The goal for the latter approach is for the whole sector potentially to move forward together (e.g. the investment preparation project reviewed in Poland - showing diffuse and low impact results). The former approach could target one or two companies/utilities (committed, forward thinking etc). These companies/utilities (e.g. Tartu water company) could then act as models for progress and pull the other similar companies up to their level of management and investment planning. The success of this approach would depend in part on the co-operation between actors in the sector (in competition or not) and how easily/difficult company specific investment could be replicated in other companies (e.g. if the management gulf is too large). One option to accelerate meeting sectoral objectives is to replicate in other countries the model in the Estonian water sector of having a central co-ordinating unit which transfers skills through provision of consulting services, to all the decentralised utilities in the sector. This way, skills can be imparted (by Phare consultants) to the central coordinating unit that then passes them on to the other actors in the sector. This will only work if there is commitment from the central coordinating unit to pass on such skills. The commitment can come if there is a financial incentive such as in Estonia where Eesti Veervack charges for its services (e.g. sewer cleaning, leak detection, investment planning, training). If such a unit charges for its services then there needs to be a demand (affordability, awareness issue) from the utilities/companies in the sector to purchase the service. This could be applicable for district heating companies and power generation companies as well as water companies. Needs assessment for physical asset supply and associated training is an important issue. The question of whether it should take place before a project as part of the project appraisal stage or as part of the project itself. This will depend on the type of project. In general investment needs should be determined before a project (as lump sum). Training associated with investments can be determined at the beginning of a project. This will be more efficient if carried out by the same consultants who will carry out the training. The absorptive capacity of recipients of technical assistance can be determined by their levels of education, awareness and commitment. If commitment is present then education and awareness building can be part of the project. Phare should consider which of these factors characterise the implementers and beneficiaries of each project to be funded. There appears no resolution to the experience that a non-committed beneficiary will lead to severely limited project impacts. Capacity building (training etc) should be channelled more into the organisations with lower staff turnover so the impacts can be sustained and maximised. Therefore during project design it should be taken into consideration that turnover in private or municipal companies tends to be lower than in central government departments where much is dependent on elections etc. One scheme Phare should consider is to increase information sharing between PMUs. There is a lot of very good work being carried out by many PMUs and the consultants working on the Phare projects. At the same time each country is aiming for the same objective - accession, and is carrying out (at different times) similar types of projects (e.g. Funds, sewer rehabilitation, water supply investment etc). The information from specific types of projects should be shared; replicated; and built upon between countries. This could be achieved through study tours by PMUs to different countries, particularly those closer to accession (e.g. Estonia). This way the PMUs can see possibilities, successes and quality (technical rigour) they should expect from their consultants. PMUs need to spend more time on identifying not just suitable but optimal projects.
124

Projects are often funded as there are no alternatives and otherwise the available budget would not be spent. This is because in general PMUs are not technically proficient and rely on the government to request projects. Whilst this is the plan of Phare and cannot be altered PMUs could at least hire consultants to work with the government to carry out strategic needs assessment and thus assist them in selecting sector strategic projects.

125

ESTONIAN POLLUTION MASTERPLAN (2/96-6/97)

0.44 MEURO

INTRODUCTION The project has been identified under the 1994 Environmental Programme which was dedicated entirely to pollution monitoring and enforcement, including both the masterplan and four supporting projects. Implementation of the masterplan project started in February 1996, and the final report was produced in June 1997. A consensus on priority projects had already been reached earlier in 1997 and the four Phare projects started towards the end of 1997, after a short delay. The project main activities include the development of a Masterplan for Pollution Monitoring and Enforcement (Phase 1) and the design and supervision of the four priority projects (Phase 2). The main beneficiaries were the Environment Inspectorate and the Environment Information Centre. The contractor was VKI Water Quality Institute. The preparation of the masterplan involved: an analysis of national policy; a review of existing activities and capacity; and a review of priorities for projects. The masterplan was to include both the priority projects and recommendations for legislation and institutional strengthening for enforcement. The implementation of the masterplan involved assistance in tendering, contracting and supervision of the selected projects. The specific objectives of the project were formal adoption of the Masterplan by relevant institutions, including adoption of recommendations for changes in legislation, enforcement, the EMP, information systems, laboratory capability and sector monitoring. The overall objectives of the project include the application of Masterplan in funding followon projects and further legislation decisions as well as improvement of the services of the beneficiary Institutions. PERFORMANCE Relevance to Needs (Score: 5) The first half of the 1990s was characterised by a focus on clean-up of hot spots identified by HELCOM (the environment body for the Baltic Sea region) and a focus on developing environmental legislation and institutions. In 1995 the Government started developing the National Environment Strategy, with Phare assistance. The strategy has 3 priorities: greater public awareness, clean technology and reduced emissions from the energy sector. The strategy also sets out the importance of EU approximation. Pollution control in Estonia is based on a system of environmental permits supported by fees and fines. However, the enforcement capacity is limited by the fact that permits and fines are often based on old unattainable levels of ambient pollution and data does not exist to monitor this effectively. New legislation is now being prepared to try to make enforcement more effective. In view of the above policies and priorities, the project seems highly relevant to all beneficiary institutions and follows on recommendations for action made in the context of
126

previous projects. It is also highly relevant in the context of EU approximation. The TOR for the Masterplan project state that at least five issues must be tackled to ensure effective implementation of the, then, forthcoming National Environmental Action Plan: Government policy must link enforcement and monitoring more tightly; institutional capacity building is required for both monitoring and enforcement agencies; the enforcement agency must steer what monitoring information is produced and how it is presented, as they are the primary users of such information; an information management system must be developed which more accurately delivers required information in a user-friendly manner; a rehabilitation/procurement programme must be undertaken to strengthen the infrastructure of both monitoring and enforcement systems.

The Phare project has been designed to make a start in addressing these five issues by improving understanding of the current situation and the priority actions required to address the key issues, and is thus relevant to national needs. Appropriateness of Project Type (Score: 5) The project type seems appropriate for the needs at the time of starting the project, with well designed technical assistance. However, the term Masterplan is somewhat misleading as it suggests the development of a strategy document, whereas a sector survey combined with a project preparation facility would have been a more appropriate label. In fact, the 4 follow-up projects to be implemented as Phase 2 of the Masterplan project have been, much more than the Masterplan project, designed as projects provoking organisational change and building capacity. Comparative Advantage (Score: 4) The Ministry of Environment made a careful decision to apply for funding to Phare after comparing different multilateral and bilateral donors. Phare funding was chosen because Phare allows for the highest input of local consultants and pays lower fees for international consultants, thus providing better value for money. Whilst the involvement of local consultants was probably an advantage for the project, it is not clear that obtaining the cheapest international consultants is a good policy. Had the motivation for using Phare money been more associated with obtaining consultants who are experienced in accession related pollution monitoring, this would have been better. Efficiency (Score: 3) The consultant focused strongly on the development of consensus over the priorities of different projects, and this was the most well-developed part of the masterplan. The quality of preparation of priority projects was acceptable. The consultants deliberately prepared relatively general TOR, to try to ensure flexibility in dealing with the evolving national situation. In the event, the follow-up projects were started relatively quickly and the TOR could have been more specific. However, this is not a criticism, as it is generally wise to build flexibility into management, given the length of time which it normally takes to lead to implementation. The quality of the policy analysis and recommendations contained in the masterplan was high, and they were used in managing the follow-up projects.

127

The two phase structure of the project should have been efficient and led to advantages in the tendering of the follow-up projects. However, the role of the lead consultants in supervising follow-up projects was not effective, as it complicated the supervisory responsibilities of the PMU, which was probably capable of supervising the follow-up projects directly. The budget for the project was reasonable, especially in view of the level of detail required in the masterplan. At the time of identifying the project, the Ministry of Environment's decision to apply for funding to Phare, was based partly on the fact that Phare allows for the highest input of local consultants and pays lower fees for international consultants than some bilateral donors, providing therefore more value for money. Co-ordination with other donor support Estonia is a relatively small country, which claims to have the highest amount of foreign donor assistance per head of the population of all Phare countries. Projects funded by Phare are particularly sensitive to this problem as some other donor agencies pay beneficiaries to participate, whilst Phare does not. This problem had only a minor influence on the project, suggesting that Phare's practices may be rather efficient. Effectiveness (Score: 3) The Masterplan has been accepted by the key institutions involved as a useful working document, but has not been formally adopted. by any of the beneficiary institutions and certainly not by any official statement. Some of the recommendations in the masterplan have been included, in a more general way, in the National Environmental Strategy and the National Environmental Action Plan, and these documents are seen to have superseded the Masterplan. Regional Environmental Departments were hardly aware of the masterplan, although there is a strong awareness of the existence and scope of the priority projects. The masterplan has been influential in guiding subsequent project expenditure, notably by Phare. The Masterplan project has resulted in greater awareness of the need for changes in the beneficiary institutions, though no structural management changes have been planned as yet. Key people in the central institutions seem to have been appreciative of the project and committed to implementing recommendations, but commitment in regional institutions was lower. One of the major constraints to project effectiveness has been the shortage of skilled staff in the key institutions. The Masterplan has mainly contributed to streamlining developments in relation to environmental monitoring and enforcement. Most of the issues addressed would, according to key individuals, have been addressed by Estonian environmental institutions in any way, but the Masterplan has enhanced understanding of the implications of the issues and has increased the pace of actions taking place. Impact (Score: 3) To date, the four Phare follow-up projects have been implemented, and some of these will lead to improved beneficiary services. Some of the recommended actions have also been taken up by other external donors. Although it is not clear how much influence the project had on these decisions, the strong emphasis on consensus building for priority projects should has probably ensured an impact. There is political commitment to continue with

128

priority actions and to proceed with reform of legislation and policy. However, commitment to national funding is limited, and is threatened by a declining political priority for environment. Sustainability (Score: 3) The sustainability of the project impact is not yet clear, but the rapid and strong commitment to reform in government and in most beneficiary institutions suggests that sustainability should be high, and the financial situation of the beneficiary institutions is expected to be stable. Score
Relevance to needs Needs Appropriateness Comparative Advantage 5 5 4 3 3 4 4

Efficiency Effectiveness Impact Sustainability

1=poor; 3=satisfactory; 5=excellent KEY LESSONS LEARNED The project is perceived to contribute substantially to meeting sector and accession objectives. The project came for all beneficiaries at the right time and has helped them to enhance understanding of key issues and seems to have resulted in concrete plans for change, although none of the beneficiary organisations have yet committed themselves officially to adopting new ideas. Timely interventions involving the beneficiaries have higher effectiveness and impact. The project provides an excellent model for preparing an investment project thoroughly and implementing the project within the same contract, in an efficient manner. Complementary actions to enhance the impact of the project are required, in the field of legislation, institutional change and others. Particularly in the legal area, substantial efforts have already been made to develop new legislation in the relevant areas. These legal developments have taken place parallel to the project and have in several cases taken MPIP recommendations and results into account.

129

ESTONIAN REGIONAL LAB EQUIPMENT

(11/97-6/98)

0.68 MEURO

INTRODUCTION The project, identified under the Phare COP 1994, was focused on environmental monitoring and enforcement. The project follows on the 1994 project Masterplan for Environmental Monitoring and Enforcement and forms one of the four Masterplan Implementation projects which were funded under the 1994 Phare COP. The Terms or Reference were designed by the contractor who was also responsible for management of the Masterplan. This project purchased basic and advanced field equipment for environmental laboratories and inspectorates and supply of hydrometric monitoring stations to the Institute of Hydrology worth almost 700,000 ECU. The Contractor was required to deliver and install the equipment where relevant and to provide training for the use of some of the equipment. Spare parts, accessories and consumables, maintenance and after sales services were to be provided for a period of three years as well as a 12 months warranty. The overall objective was to improve the monitoring performance of the beneficiary laboratories through use of modern equipment. PERFORMANCE Relevance to Needs (Score: 3) The project has been designed based on a thorough assessment of the needs to improve environmental monitoring, particularly in its function to support enforcement, and based on discussions with Heads of Labs and other institutions. The supplies are therefore not the response to a shopping list provided by the beneficiaries, but are there result of a analysis of gaps in basic and specific equipment and materials. In wider context, the need to strengthen enforcement in order to improve general pollution control in Estonia, had been identified by the Ministry of Environment. HELCOM supported the urgent need for action in this area. The SALES project, which is in fact a survey of the existing monitoring and enforcement system. Appropriateness of Project Type (Score: 3) The project type is appropriate in relation to the needs. The supplies delivered are complemented by the other related projects. Comparative Advantage of Phare (Score: 4) Phares comparative advantage is that there were no other donors interested in financing the project. Efficiency (Score: 1) As no financial data could be provided by the PMU or the Contractor, no evaluation on the cost of the supplied equipment could be made. Most supplies were needed, including the supplies of glassware and chemicals, although it can be argued that the beneficiaries should be able to provide these types of supplies themselves. Water purification equipment ,
130

balances and sampling equipment were particularly appreciated by the beneficiaries. Serious doubts relate to the provision of more sophisticated equipment, particularly gas chromatographs. None of the labs seemed to have the gas chromatograph in full use, relating to insufficient training on the use of it, insufficient staff to operate the equipment and a lack of orders for gaschromatograph analyses. Most labs intended to use the gas chromatograph for analysis of oil products and would not have the means in the short term to use it for additional analyses, such as pesticides and heavy metals. It can be doubted whether these gas chromatographs should have been supplies, or whether they should have been supplied to all labs. Project implementation has only been partly successful, although all supplies have been delivered and installed. Problems occurred after delivery and before final acceptance been carried out. Recipients apparently did not receive clear instructions concerning opening of the delivered packages and more than necessary time was spent to check whether all parts and pieces of equipment were present. In a few cases spare parts and chemicals still need to be delivered. The problems appear to relate mainly to inefficient management by the Contractor of the SALES project. However, the company contracted for supervision (VKI) might also bear some responsibility. In the end, the problems relating to delivery did not result into any pieces of equipment being incomplete. They have resulted in some delays in taking equipment in operation and probably caused some confusion amongst recipients. All beneficiaries were content with the quality of the provided equipment and materials. All analysing equipment meets national and international standards and their supply has brought labs closer to meeting certification requirements. The recipients were less content with the provided training on the use of major pieces of equipment. Training for the use of the gaschromatograph and spectrophotometer was found too short or insufficient. In most labs staff is still not able to use the gaschromatograph and they will still have to make a detailed study of provided manuals. The same counts for a spectrophotometer provided. Effectiveness (Score: 2) Although the effectiveness of the project can not be easily isolated from other projects supporting the same beneficiary institutions, some conclusions can be drawn on this issue. The project has contributed substantially to improvement of efficiency of operations of the beneficiaries. Not all equipment is in use due to lack of trained staff, lack of market for the use of the equipment. Also the more expenses pieces of equipment provided are not expected to be used for their full range of possibilities. Most of the materials and equipment provided in the framework of the SALES project were in full use at the time of the evaluation. However, problems to use delivered gaschromatographs occurred in a number of environmental laboratories, as staff was not yet fully prepared to operate the equipment or was not available or as there was no market for gaschromatograph analyses. Most equipment appeared to be in full and mostly in constant use. Most pieces of equipment and material had already resulted in reduction in time for the production of data

131

and therefore in a larger quantity of data produced per day. Also, new pieces of equipment such as the gaschromatograph and the spectrophotometer enable labs to analyse more substances. All recipients claim that the quality of the data they produce has significantly improved. Although none of the beneficiaries considered reliability of services as a major issue, the SALES project will clearly have an impact on reliability. As many operations have been automised, analytical services and the time they cost to be provided, can be more accurately estimated. However, complementary actions are needed to increase the overall reliability of services. Most beneficiaries expected that operational costs would reduce as a result of more time efficient equipment, reducing staff costs and/or increasing revenues. No calculations had been made of concrete cost savings, however. Complementary actions are being taken by the beneficiaries, although further actions are necessary. There is a strong commitment amongst environmental laboratories to apply for Estonian and possibly international accreditation in the next years. Impact (Score: 2) The general evaluation is that the SALES project has contributed to the improvement of services of the beneficiary institution. Most regional labs believe that without the SALES project they could not have achieved the level of quality they have now and that they could not have considered to apply for accreditation. One of the beneficiary labs has already been accredited according to Estonian standards and claims that this would not have been possible without the supplied equipment. Complementary actions are needed to strengthen enforcement, to improve the effectiveness of economic instruments and to review the permitting system. Sustainability (Score: 3) The sustainability of the project depends largely on the financial viability of the beneficiary institutions. The regional environmental laboratories have recently undergone major changes, now operating as businesses. Labs are responsible for generating their own revenues and covering all their costs. This process has, for instance, been one of the driving forces for accreditation of the labs, so that they can compete with private labs and/or labs from private companies. The Ministry of Environment and the CEDs are for most labs the main clients, although some also provide services for industrial enterprises. Although the situation is probably the best possible, the labs are still highly dependent on orders from government institutions. Therefore, it would still be necessary to establish more long term working relations with clients and to develop business and/or operational plans. Score
Needs 3 Appropriateness 3 Comparative Advantage 4 Efficiency 1 Effectiveness 2 Impact 2 Sustainability 3 1=poor; 3=satisfactory; 5= excellent Relevance to needs

132

KEY LESSONS LEARNED Supply of equipment and materials can be very useful to strengthen organisations and systems. However supply of equipment should only be considered in the context of other complementary actions taking place. The success of the present project relates to the fact that it does not respond to a shopping list provided by the beneficiary institutions alone, but that it responds to needs identified in the context of strengthening enforcement and environmental monitoring, although the need for some of the equipment provided is still questionable. The success of the project also relates to the fact that are projects were taking place in parallel which , to some extent, stimulated the beneficiaries to review and upgrade their services provided. A key lesson regarding this type of projects should also be that the supply of equipment should be sufficiently accompanies by training on the use of it and should take into account the capability of the beneficiary institution to operate it. The need for investment in physical assists is often evident, but investments in assets are only justified when they are part of a larger package of investments and actions. In Estonia, the SALES project was clearly part of a larger package of actions and investments, not only provided by Phare, but also, although to a lower extent, by other donors and some domestic contributions. The link between upgrading the environmental monitoring system and strengthening enforcement has in general been made in the context of Phare assistance, although the Environment Inspectorate generally benefited less from the overall package of 1994 Phare assistance than the monitoring institutions. Nevertheless, a strong feature of the project is that it was designed to support enforcement and pollution control and not just the monitoring system, although complementary actions will still need to be taken to enhance the impact of the project. EU accession is expected to be a major driving force for further complementary actions in the field of enforcement and pollution control.

133

HUNGARIAN HAZARDOUS WASTE

(6/92-12/93)

0.33 MECU

INTRODUCTION The project consisted of preparatory studies and of the procurement of equipment for a hazardous waste pre-treatment facility and a landfill. Stationary pre-treatment equipment, two mobile pre-treatment dewatering units and a laboratory for the landfill were to be purchased with the Phare funds. The overall objective is the construction of the facilities for the safe treatment of the hazardous industrial waste in the Transdanubia Region. The specific objective of the project was a sound investment proposal for the construction of a treatment facility, based upon the recommendations made in the studies and installed equipment was the facility would have been constructed. A feasibility study, an EIA and a technical design study were carried out. These studies gathered the necessary data on the amounts and types of waste generated and produced recommendations for the establishment of a hazardous waste pre-treatment and disposal facility. However, in the first half of 1993, the initial Phare budget allocation of 2.02 MEURO was reduced to 550.743 ECU, the funds for procurement of equipment being reallocated to other projects. This is because it was decided in the course of the technical design study and the EIA that the treatment facility would never have been constructed by end of 1993, the moment at which the 1990 Phare-programme budget had to be committed. The purchase of equipment before the actual construction of the facility would not have been appropriate. Therefore the equipment was never procured and the remaining part of the budget (i.e. the part that not spent on studies) was re-allocated to another project. In the end the public rejected the facility which was thus never constructed. The technical study was stopped after public rejection. PERFORMANCE Relevance to Needs (Score: 5) Large quantities of hazardous industrial waste are generated in Hungary, which is mostly stored or treated on site, mainly because of the lack of treatment facilities. The number and the condition of the existing treatment facilities are insufficient. There was thus a clear need to investment in facilities for the safe and environmentally sound treatment and disposal of waste, and of hazardous industrial waste in particular. The north west region of Hungary, due to its industrial composition and the types and quantity of waste it generates, is a good location for the establishment of such a facility. Hungary had acquired experience with engineering and constructing during the construction of a state-of-the-art incinerator and landfill,. There was less experience with pre-treatment. This justifies the use of Phare funds to procure the equipment, which was not available in Hungary. Appropriateness of Project Type (Score: 2) The fact that the overall project was eventually halted because of public opposition
134

identified during a public consultation exercise carried out after the completion of the Phare funded preparatory studies shows that the type of studies funded were not appropriate. For such a publicly sensitive development a comprehensive public consultation exercise should have been funded before commencing a detailed feasibility study. Comparative Advantage of Phare (Score: 4) The main comparative advantage is that Phare is willing to provide relatively small financing i.e. below $5 million. This allows the government to access funds for facilities too small to be of interest to the World Bank and EBRD. The grant makes the investment more affordable to the government. This is particularly important for such a facility which is introducing a new service (hazardous waste pre-treatment) which would be difficult to levy charges, at least initially, thus reducing the investors ability to repay a loan. Efficiency (Score: 2) The expected outputs of the project were installed equipment and preparatory studies. The pre-feasibility study and the EIA were conducted, while the technical design study was halted and the equipment not purchased. No project reports were made available during the evaluation. Cost-effectiveness may have been improved if the international consultants had taken into account the earlier studies conducted on behalf on the Hungarian government regarding the hazardous waste facility. There was there a good relationship between all parties involved in the management of the project, but the responsibilities of each party were not clearly delineated , which had an adverse effect on the progress of the project. For example, officially, the project was managed by a consultant on behalf of the Ministry of Environment, while in practice a local consulting firm, Transdanubia Waste Ltd., was managing. This might be due to the fact that the Phare project was not clearly distinguished from the management of the overall project, namely the construction of the facilities. The implementation of the project, as it was originally conceived, was seriously delayed by Phares administrative procedures, the TORs for the preparatory studies were not drafted on schedule and evident lack of interest from the national political authorities. The delays had an adverse effect on the project and eventually the remaining funds as disbursement deadlines were exceeded. Effectiveness (Score: 2) The project did not achieve its specific objectives. The Phare project was stopped and the funds allocated to the purchase of equipment were re-allocated to another project. The main reason for this was the fact that the construction of the facilities would never have started before the deadline for commitment of the Phare funds and procurement of equipment before actual construction is not appropriate. Therefore project was cancelled because of exceeding disbursement deadline. The delay was in part due to loss of political commitment to the entire project and in part due to Phare procedures. However, the overall investment did not proceed in the end. Firstly the government decided that industry should finance the investment and not the public sector. Then Waste Management International ,a potential private investor, decided to pursue the investment independently. However it too had to abandon his investment plans after public rejection of the construction of the facilities. This public rejection was not triggered by the Phare

135

funded studies (not identified during the EIA). It was identified during the public consultation organised independently by WMI after a request by the local public. Impact (Score: 1) The remaining funds for this project were re-allocated and the investment plans were cancelled. The project did not have any impact. At is best, it can be argued that the project resulted in a better understanding of hazardous waste generation in the north west of Hungary and of the management of waste. While this might be true for the local industries that were surveyed, we did not find any reference to the studies in relevant publications from the authorities. The mayor of Vrtesacsa was hardly aware of the fact that part of the project was funded by Phare and officials of the Ministry for Industry and the Ministry of Environment can hardly recall Phare involvement and were not aware of any studies conducted. Sustainability (Score : na) The project did not have any results, that could be sustainable. Score
Efficiency Effectiveness Impact Sustainability Relevance to needs 2 2 1 na 5 2 4

Needs Appropriateness Comparative Advantage

1=poor; 3=satisfactory; 5=excellent KEY LESSONS LEARNED The design of highly sensitive projects should consider public consultation and participation. Local opposition (the NIMBY-Not in my backyard syndrome) constitutes one of the biggest single obstacles to establishment of a hazardous waste facility. A public awareness study should have been one of the first components of this project. Any hazardous waste-related project, should start with informing the public about the real issues and the actual risks for health and environment. The public in Hungary is so sensitive towards waste treatment facilities that in the next five years it will be difficult to have a facility accepted. If Phare would grant funds for waste related projects in the next 5 years, it should be for municipal waste collection and disposal. Before funds are allocated to highly sensitive projects, such as the establishment of a hazardous waste treatment facility, it should be assessed whether there is political commitment, especially if an important part of the investment funds are to be provided by the government. It is clear the political climate in which the project was identified (before 1990) was different than after 1990, when the project had to be implemented and the facility had to be constructed. After 1990, the role of the Government changed and the responsibility for the construction of hazardous waste management facilities was laid upon industry. The change of policy of the government after 1990, is one of the main reasons that the project had no chance to succeed. This change of policy should have been taken account of before inclusion of the project in the list of Phare funded projects. The issue of the required investment is also important. The Phare fund was relatively
136

minor compared to the total cost of the project and to the cost that the Hungarians would bear. The proposed method for financing was as follows: 30 % of the investment cost from the Central Environmental Fund, 10 % from Phare and the remaining 60 % from both Hungarian and foreign private investors. The Phare funds were meant for the procurement of equipment. Evidently, the construction of the facility is a prerequisite and requires important investment. If Phare funds are for components of a larger investment project Phare should obtain guarantees that the overall project will be completed. The project was both an investment preparation and physical assets project. Phare funds should not be allocated to the procurement of equipment for facilities for which the preparatory studies still have to be conducted.

137

HUNGARIAN SEWER SYSTEM INSTALLATION (10/92-4/94)

1.2 MEURO

INTRODUCTION The project consisted of the construction of a sewer system in Tahitotfalu town on Szentendre Island. The objective of installing the sewer system was to reduce contamination to groundwater which is extracted to supply water to Budapest. The specific objective of the project is the use of the sewer system. The municipality of Tahitotfalu had hired upon its own expenses an engineer for the daily management of the project and a consulting firm for the overall technical supervision. Preparatory activities, such as the drafting of a detailed design study, were conducted by the Municipal Waterworks of Budapest. The Phare component was financing of sewer equipment. PERFORMANCE Relevance to Needs (Score: 5) The support for a sewer system was in line with national environmental priorities and the level of resources devoted to this element of the environment programme was appropriate. More than 90 % of Hungarys drinking water is coming from subsurface sources and groundwater contamination is thus a critical issue. Szentendre Island has been classified by Decree of the National Water Authority as a Class 2/2 (important and sensitive) subsurface water resource. Szentendre Island is clearly an important drinking water zone, providing 60% of Budapests drinking water. The quality of the groundwater is threatened with contamination from septic tanks discharges. The treatment of the household waste water, rather than seeping through the highly permeable soil to the drinking water aquifers, helps preserving the drinking water resources from the city of Budapest. Appropriateness of Project Type (Score: 5) The Phare contribution to the sewer project was appropriate. The grant made the investment in much needed infrastructure more affordable. It may have also been appropriate to ensure plans were in place for complementary investment financed by other sources. Comparative Advantage of Phare (Score: 5) The grant made the investment in much needed infrastructure more affordable. This was important at this stage when the municipality was introducing the wastewater collection service and could not set high tariffs immediately to cover costs. Efficiency (Score: 5) The specifications of the new system were set at a proper level, consistent with the local needs. In the opinion of the various parties involved, including the Danube Regional Waterworks, the operator of the system, similar outputs could not have been achieved with lower costs. The proposal from the selected contractor was technically sound and fitted in the budget that was made available by Phare. The contractor also proved to be flexible and technically capable to deal with unexpected technical work variations, entailing from
138

inadequate design details in the preparatory study. The municipality had hired a local engineer on a contractual basis for the daily management of the project. The municipality hired upon its own expenses an engineering firm to supervise the works. This proved to be a very cost-effective measure because the firms technical ability helped to limit the additional costs resulting from the inadequate design details in the preparatory study. The project manager and the supervising firm, were critical for the progress of the project. The sewer system was constructed as required in the TOR and met the expectations of all parties involved, including the municipality and the operator of the system. The Danube Regional Waterworks accepted to operate the system on a contractual basis, which is proof of their trust in the quality of the system. Since installation it has not experienced any operating problem indicating that the system was properly constructed. Effectiveness (Score: 3) The specific objective of the project is the use of the sewer system. If measured by the number of households that are connected to the system, the effectiveness was initially very low. The connection rate though has raised gradually from 10 % to 57 %. The connected households represent about 80 % of the waste water. People that are still not connected is because they cannot afford it, such as pensioners which is a social constraint. The connection rate was never given serious consideration at the time when the project was identified and selected for funding. The cost of getting connected to the system is relatively high for a household. There is a strong commitment to the project from the local municipal government, which is promoting connection amongst the households. Indeed, the Hungarian government is considering the introduction of a tax on the use of natural environmental resources. Such a tax would be higher for the households that are not connected to a sewer system. Such a legislative action would create the right environment to maximise the effectiveness of the investment. Impact (Score: 4) The overall objective is the reduction of the risk of groundwater by septic task discharges to conserve the quality of drinking water resources in the project area. The Municipal Waterworks of Budapest provides Budapest and surrounding villages on the northern side with drinking water extracted on the island. Monitoring results from the Waterworks show that the quality of the groundwater on the island where the municipality is located, has improved since the construction of the sewer system acceptable to EU standards. The risk of contamination of the drinking water aquifers has been reduced, but to substantially reduce the risk, a sewer system should also be constructed in the three other villages of the island. The risk of contamination by other sources of pollution, such as agriculture on the Island, has never been seriously considered. The water treatment facility in Szentendre, treating the waste water from Tahitotfalu, has been upgraded, in terms of both capacity and technical treatment, resulting in a better quality of the effluent. The costs for this upgrade were partially covered by a grant that four villages, amongst which Tahitotfalu, had applied for with the Hungarian government. The upgrade is an indirect consequence from the Phare funded project and was a necessary complementary action.

139

The Phare grant for the sewer equipment has triggered a grant application by these two villages with the Hungarian government for the construction of a sewer system in those villages and therefore the project has had a replication impact. Sustainability (Score: 4) The sustainability of the sewer system is closely related to the ability to finance the operation and maintenance. The sewer system is owned by the municipality of Tahitotfalu, but operated by the Danube Regional Water Works, with whom the municipality has concluded a contract, both to operate the system and to treat the household waste water. In the Technical Management Agreement concluded between the Phare PMU and the municipality of Tahitotfalu, the latter had to ensure that a local contribution would be made available to ensure the proper management of the sewage system and that the waste water would be accepted for treatment by the Szentendre plant of the Regional Waterworks. The households that are connected to the sewage system, pay for the service directly to the Danube Regional Waterworks, which issues one bill, covering the operational costs of the system, the treatment of the waste water and the supply of the drinking water.

Score
Efficiency Effectiveness Impact Sustainability Relevance to needs 5 3 4 4 5 5 5

Needs Appropriateness Comparative Advantage

1=poor; 3=satisfactory; 5=excellent KEY LESSONS LEARNED Investing in physical assets, is appropriate for achieving the objective of an operational sewer system. It would have been valuable to also finance the household sewage connections given that this was a critical factor in initially limiting effectiveness of the project and given that the total cost of connecting households is minor compared to the construction cost. When Phare provides a grant for an investment project, the beneficiary should clearly identify the follow-up and complementary actions, such as the upgrade of the treatment facility in this project, necessary to make the investment effective. Phare could assess whether the project would highly benefit from inclusion of the follow-up actions and complementary in the project activities to be funded. The efficiency with which the project was conducted, can be partially accredited to the involvement of a technically skilled Phare PMU officer. Phare would benefit from a follow-up of the projects by technically skilled personnel. To maximise the impact of a project that aims to reduce a risk, Phare should assess before funding, which other risks exist. In this project, while it was clear that septic tank discharges constituted a risk for the groundwater, it has not been considered whether there were other risk factors, such as agriculture. Phare would benefit from having a set of selection criteria for deciding on the projects it provides a grant for: Fund win-win investments. The project was beneficial for the environment and
140

helped preserving an economical good at the same time; Assist technically justified projects, where the beneficiary gives proof of having the required technical skills or can demonstrate that he will call upon the appropriate expertise. The efficiency of this project was maximised through the management and the supervision of respectively a local engineer and a consulting firm. Fund projects where the applicant is also the direct beneficiary of the output of the project. In this project, the direct beneficiary of the sewer system was not the municipality, the beneficiary of the grant, but the water company providing Budapest, with drinking water. This explains why the connection of the households to the system progressed so slowly. If the direct beneficiary is not the applicant, the direct beneficiary should be fully and formally involved in the project. In this project the direct beneficiary was involved through the financing of the preparatory work, but its involvement stopped when the works started.

141

HUNGARIAN NATIONAL PARK

(1993-94)

0.19 MEURO

INTRODUCTION The Parks primary objective and purpose is nature conservation. Secondary objectives include research, education and tourism. The project provided grants for the construction of new buildings (visitor centre, research buildings) for the Park and the procurement of installation of furnishings and equipment. Government contributed with provision of 1.1 MEURO of land. The overall objective was to improve conservation within the park. The specific objective was the operational use of the park administration buildings, research, education and tourism facilities. PERFORMANCE Relevance to Needs (Score: 5) The project was identified by the National Authority for Nature Conservation to support the newly created Fert-Hansg National Park (1991), an internationally important wetland and lake providing habitat for a range of migratory birds. Therefore a Phare project was highly useful. Appropriateness of Project Type (Score: 4) The concern is over the design and cost of the administrative building and visitor centre which are not appropriate given the primary objective of the park is nature conservation. It could be argued that a less costly structure would have served just as well for the purposes of nature conservation, and that a high-cost structure is only justified if used for income generation from the public. However, a less impressive building might not achieve the same level of public acclaim and interest, including frequent visits from senior politicians. Furthermore, due to its unique qualities greater potential exists for generating revenue in the future. At the same time, the high cost of the administrative/visitor centre meant that resources had to be drawn away from other components of the project which has reduced the combined operational effectiveness of the measures originally proposed by the project. Comparative Advantage of Phare (Score:5) Such a project promoting public awareness for conservation should not be expected to raise sufficient revenues to repay a loan as any user charges would have to start low to attract visitors to the park. Therefore Phare has a major comparative advantage funding such conservation related projects as it is grant based Efficiency (Score: 1) Implementation problems arose because of misunderstandings over financing and procurement procedures, particularly in relation to VAT payments on equipment. While the Financing Agreement for HU9002 stated that no stamp duty should be paid on equipment purchased, this is mandatory under Hungarian law. As a result duty had to be paid initially on all equipment purchased and then reclaimed - a process which took several months each time resulting in delays.
142

Further delays were caused because of a lack of familiarity with EU procedures (in particular tendering procedures), and because - in the interests of political expediency - the project preparation and design phase was extremely short, leading to the need for later revisions and alternations both to project scope and budget. Furthermore, delays were also caused because the PMU was seriously understaffed at the time and because of organisational change within the Ministry of Environment. For these various reasons, project completion was delayed by two years. Effectiveness (Score: 5) The specific objective of the project is the operational use of administrative and associated buildings for park administration, research, education and tourism. This has been achieved. The Centre now receives 10,000 visitors per year and the total number of visitors to the park has increased 10 fold since 1994. The Park currently works with 3 secondary schools and 15 primary schools in surrounding villages and in the principal local town of Sopron. During 1995 and 1996 summer seasons more than 8,000 pupils visited the Park of which more than half visited the Centre at Sarrd. During 1995 and 1996 the Park hosted 2 international symposiums and 6 national conferences and workshops. Over a similar period 10 Masters theses were completed and one PhD. The Park is formally collaborating with 5 Colleges and 3 Universities in Hungary. These figures are not altogether conclusive in the sense that they do not necessarily suggest that the facilities are being use to their maximum. However, they do convincingly demonstrate that the Park is known and is active in providing a service to a wide range of stakeholders, a situation that could not have arisen without investment from Phare. Impact (Score: 2) The overall objective of the project is to improve and conserve wildlife habitat within the Fert-Hansg National Park and to provide a focus for secondary activities associated with research, education and tourism. The project has gone some way towards reaching its overall objective though this is always difficult to measure accurately in terms of habitat created/protected or species numbers increased. The Park stated that illegal activity has been declining in the past five years which they attribute to the Parks activities in relation to tourism, education and use of the local media. The Park is confident that the habitat in the core zone has been conserved and improved since establishment of the Park. The most convincing indicator is that there has been no noticeable decline in numbers of wildfowl - therefore populations are currently stable. Additionally a conservation management plan has been produced and the park is relatively active in working with the community on forest activities. Sustainability (Score: 1) The long-term sustainability of the project and of the Park is all about funding and motivating public interest. There is no question that with sufficient funding the Park can continue to operate effectively and the use of the buildings will be sustained or increased. This is because the project is relevant to the needs of the Park and staff are committed to ensuring that the objectives of the Park are met.

143

However the financial sustainability of the Park is in question. At present the Park expects to continue to be funded primarily from central funds. However, the level of funding is limited and insufficient to permit the Park to broaden the scope of its activities. Over dependency on central funding increases the Parks vulnerability to future policy change. In order to reduce this vulnerability, the Park will need to refocus its priorities more towards the development of income generating activities. The most obvious way of increasing revenue would be to levy a small fee for all visitors. With an estimated volume of 200,000 visitors per year a small charge of the order of 200-300 HUF/capita would make a lot of difference. Charges for accommodation at the Centre are also very low and could be raised. Score
Efficiency Effectiveness Impact Sustainability Relevance to needs 1 5 2 1 5 4 5

1=poor; 3=satisfactory; 5=excellent KEY LESSONS LEARNED The project demonstrates that investments in physical assets for bio-diversity conservation can have a positive environmental pay-off, but that attention should be paid to the cost efficiency and sustainability of such investments. This is critical to ensuring long-term use for intended purpose. Although investments in bio-diversity are never likely to be fully self-sustaining, measures for matching the level of investment to potential for income generation should be made more explicit in project design. If investments are to be made in high-cost facilities, project design should incorporate more details on how these facilities will be used to best effect. During our visit to the visitor centre we noted that the educational and exhibition materials on view were limited. We believe that a more proactive and imaginative approach to information displays is justified for a building of this value and will increase impact. The project appears to be at odds with government policy on the role of national parks in nature conservation. From the design and cost of the structures now in place, one concludes that their primary function is to draw more visitors into the Park. In fact the Parks primary objective is nature conservation, while education and recreation are only secondary objectives. In the future, the match between purpose of investment and existing policies needs to be more clearly stated. Investments in physical assets for nature conservation can play a key role in biodiversity conservation and in helping countries to meet accession objectives in this area. The requirements of accession include harmonisation of legislation and institutional reform, as well as physical evidence of policy implementation. Clearly this cannot be achieved without investment in national park infrastructure. Based on the evidence of this project, such investments can lead to visible environmental impacts within a relatively short space of time, as well as increasing the participation and interest of the public. Nature conservation is rarely an appropriate mode of intervention for the development banks with the exception perhaps of ecotourism. Sources of financing for nature conservation can come from government, from the public, from the private sector and
144

from grants such as those provided by the EU and the bilateral donors. At present government aid to nature conservation in Hungary is limited and the level of public awareness is not sufficiently high to be able to generate significant revenue. Therefore their is a potentially an important role for Phare in this sector. Until such time as core funding can be increased (either from central government or DGXI) and/or supplemented via funds raised from the public and private sector, Phare should continue to provide an appropriate means of funding physical investments in conservation under certain conditions. They should be as strategic as possible: - perhaps focusing on: the most important conservation sites; those sites currently under threat; sites most likely to encourage public interest; or sites where there is a significant existing level of public interest. It is significant that the GoH has not been able to attract further funding to invest in physical assets for nature conservation from bilateral donors currently active in the nature conservation sector. These other donors have tended to fund institutional and policy reform, i.e. putting in place the necessary conditions to encourage investment. By contrast, the Phare programme has provided assistance both in institutional reform and in investments. Thereby, it has played a key role in the nature conservation sector in Hungary and can continue to do so in the future. One reservation is that the National Authority for Nature Conservation (NANC) currently views Phare procedures to be cumbersome, opaque and a significant disincentive to applying for EU funds. While the NANC appreciates the positive role that Phare has played, it now considers Phare as a lender of last resort. Investments in physical assets can lead to clearly visible outputs motivating project staff and other project stakeholders. In Hungary there is a common perception that most donor assistance is paper only. Successful investments in physical assets can help to redress this imbalance. The critical factors influencing the successful implementation of the project were local ownership of project design, commitment of project staff, and the fact that it built on an already established initiative rather than attempting to start something new.

145

HUNGARY BIO-DIVERSITY AWARENESS

(12/95-ONGOING)

0.18 MEURO

INTRODUCTION The project introduces western European principles of public awareness into Hungary through the National Authority for Nature Conservation (NANC) and its nine nature conservation Directorates. It was implemented with the assistance of the European Centre for Nature Conservation (ECNC). The project had a long history, paralleling the institutional development of nature conservation in Hungary. Project preparation was initially delayed, through lack of agreement over activities and was then rushed, taking place over the winter of 1994/95. The TOR were scant and there was no tendering. This enabled formal disbursement to be completed by the deadline of end 1995. However, project activities are still incomplete, three years later. The project started with a review of existing attitudes to nature conservation in Hungary, which provided useful background information. This was supported by a study tour of the UK, which helped raise awareness amongst national park officials, and was thought to be very useful by those who participated. The main activity was the development of a strategy and action plan for public participation in nature conservation. The specific objective of the project was the acceptance and implementation of the strategy for raising public awareness of nature conservation. The overall objective of the project was to increase public awareness of nature conservation and the role of protected areas in Hungary.
PERFORMANCE

Relevance to Needs (Score: 5) There are two relatively distinct models of nature conservation and protected area management in Europe: the 'conservator' model, followed by France and Germany, in which the primary responsibility of the state is to conserve wildlife; and the 'participatory' model, followed by the UK, in which conservation is dependent on public awareness and support, including integration with the local economy. As the Hungarian approach is already close to the conservator model, the consultants felt that there was most to be learned from introducing more participatory principles to nature conservation. For this reason, the project was implemented with the assistance of the UKs conservation agency, English Nature. The project started with a review of existing levels of public awareness in Hungary, which concluded that nature conservation was generally not considered to be important. Conservation was often viewed as a barrier to economic development. Those groups who were more positive generally disagreed with NANC approaches. Furthermore the interests of the public are of secondary importance to NANC, compared with its function as a conservator. However there was some variation in this attitude expressed by the different Directorates which we visited. Whichever model of conservation is to dominate the Hungarian approach, there was clearly an important need to accommodate more awareness raising activities, in order to educate and promote public support.

146

Appropriateness of Project Type (Score: 2) The provision of technical assistance for a major study was premature, in view of the fact that major shifts in management thinking were required. Some preliminary sensitisation would have helped prepare for the project. In addition, the formal training activities were not sufficiently participatory and did not succeed in involving key management officials. Comparative Advantage (Score: 4) In theory, Phare should have a high comparative advantage in providing assistance in defining the approach to public participation, by stimulating exchange of experience across various EU countries. Efficiency (Score: 1) The consultants scope of work was ill defined, with poor reporting and monitoring. The launch workshop was of limited use, partly because the use of the English language excluded some key participants. The report produced from this activity was well presented and provides an interesting and useful reference guide on public awareness. However, it falls short of an operational strategy. As a set of guidelines, the project documents may yet be of use to senior management staff within the directorates. Printed brochures have been produced by the project, which were a significant improvement on those being used by the Directorates at that time. These new brochures have also been used as a model for subsequent materials produced by a number of Directorates. The project should have undertaken a pilot public awareness project as a means of demonstrating aspects of the proposed strategy. However, due to disagreements between NANC, Phare PMU and the consultants over what the objectives of the pilot should be, none has been carried out to date. This situation has arisen largely because the terms of reference are vague about the objectives of the project and the consultants responsibilities. The NANC now plans to use the remaining project funds for the construction of a geological park at NANC HQ in Budapest. Although this is not in line with the objectives of the project, the EU Delegation and Phare PMU appear to have limited interest in, or leverage over, the use of the remaining funds. The budget and project scope was not sufficient to achieve the project objectives. Serious problems with implementation arose largely because the project had no clear political priority within government, and the project preparation was rushed without full government participation. The lack of commitment was further enforced by the perception in the nature conservation sector in Hungary that Phare is the most difficult development partner to work with, due to its complex and lengthy procedures, the lack of clear direction from Phare and the frequent staff changes in Brussels. Effectiveness (Score: 1) A draft strategy has stimulated some debate within the NANC and Directorates. However, it is far from being implemented, largely because there is a lack of clear political direction in this area. All Directorates which we contacted (six out of a total of nine) had received the Strategy, but had been influenced by its recommendations. In the meantime, at least three Directorates have taken the initiative to develop their own public awareness and tourism

147

development plans. A secondary specific objective was achieved, involving development and distributing of national park brochures. Approximately 100,000 were produced and distributed, leading to reprints of 50,000 and a request for further reprints by the end of the year. Impact (Score: 1) Predictably, the impact of the project on this objective has been marginal, as there has been no acceptance or implementation of the proposed strategy. However, the published material has had some influence, largely limited to reinforcing the interest of those already aware of nature conservation. Despite this lack of impact, there is now a trend of increasing numbers of visitors to national parks. Furthermore, some parks are increasingly devoting their attention to attracting and managing visitors. This would indicate that, despite the political and institutional problems faced by the project, the timing was right. Indeed, it is still possible that the project documents could be useful in future policy formulation and strategy development. Sustainability (Score: 1) Although the objectives of the project have not been attained, it is possible that the project could have a longer term influence through the introduction of new ideas about public awareness to individuals who are in positions of authority, or who may latter be appointed to positions of authority. If this does result in greater public awareness, this should contribute to bio-diversity conservation by reducing conflicts with protected areas and by changing attitudes not only towards protected areas but also to the wider management of the environment outside protected areas. Score
Relevance to needs Needs Appropriateness Comparative Advantage Efficiency Effectiveness Impact Sustainability 5 2 4 1 1 1 1

1=poor; 3=satisfactory; 5=excellent KEY LESSONS LEARNED The effectiveness and impact of introducing new methods or ideas is dependent (at least in the short to medium term) on the degree to which senior management staff can be encouraged to be involved. Project design should allow for sufficient resources to be allocated to defining the meaning and role of public awareness in nature conservation within a society and the implications of this for institutional and organisational reform. This is particularly important when seeking to encourage major changes in policy, such as that involved in shifting from a conservator to a participatory approach to bio-diversity conservation. During the course of this project, a consensus was assumed when this was not the case. The purpose of the project was therefore perceived differently by the different parties
148

involved, as were the principal outputs. In view of the increasing level of public interest in Hungarys national parks, it appears that the timing of this project was right. Therefore project relevance was high. The limited effectiveness and impact of the project can be attributed mainly to poor design and poor management during the early stages. As a result the most of the key players lost interest in the outputs. Study tours can be very effective in conveying new ideas providing that those selected remain in a position to influence decision-making. The impact of study tours is not necessarily to be found in the replication of methods learned, but in the provision of wider understanding and context within which appropriate country-specific models can be designed. External consultants have a key role to play in bringing in new knowledge and in providing impartial and independent advice. However, the usefulness of such advice is dependent upon the degree to which it complements local knowledge and experience. The challenges faced by different national parks in interacting with the public vary considerably. Strategic approaches should tailor plans more closely to the individual needs of different national parks. This is important not only to produce more effective plans, but also to ensure sufficient local ownership of the plan. Investments in public awareness for nature conservation can help to justify and strengthen the acceptability and role of protected areas in the community, leading to longer-term pay-offs in relation to bio-diversity conservation. This public support is essential in meeting the increased pressures on bio-diversity conservation. In some CEECs (such as Estonia, Poland, Hungary and the Czech Republic), governments are already becoming more sophisticated in dealing with public awareness in nature conservation. In these countries, it is likely that future Phare support will be more usefully provided to NGOs, allowing government strategy to evolve to accommodate increased public awareness. Any role that Phare (or its successor) might play in this sector in the future is likely to be limited by the strongly held perception that grants from the EU are hard to obtain, expensive in terms of time spent preparing project proposals, and complex to manage owing to existing procedures. As a result, there is a tendency to apply for EU funds only after other potential sources have been considered.

149

LITHUANIA ENVIRONMENT INVESTMENT FUND (96-ONGOING)

2.3 MEURO

INTRODUCTION The main activity of the project was to establish the Environmental Investment Fund. This included a feasibility study, the development of an Operating Manual and the transfer of 2 MEURO seed capital. The Fund was established in November 1996. It is allowed to lend up to $375,000 per loan and up to 70% of the total external financing. The overall objective of the Fund was to promote investments by industry for pollution control and energy efficiency. The specific objective of the project is to disburse loans. PERFORMANCE Relevance to Needs (Score: 5) In Lithuania the major part of environmental investments have been directed for the construction of wastewater treatment plants. The Ministry of Environmental Protection saw the need for an environmental investment fund to support the realisation of Lithuanias other environmental protection priorities (waste and pollution minimisation), to increase the volume of environmental investments and strengthen private sources of environmental financing. The establishment of the Phare funded Energy Efficiency Fund may reduce the need for an environmental fund focusing on energy efficiency and not just pollution reduction. Appropriateness of Project Type (Score: 5) A Fund providing subsidised loans was an appropriate mechanism to promote environmental investments. Comparative Advantage of Phare (Score: 5) Phare has shown weaknesses in establishing Funds. Its inability to reconcile Phare and national requirements led to long delays in the establishment of the Fund. Thus Phares rigid procedures combined with lack of research into national requirements are a disadvantage. Efficiency (Score: 1) The feasibility study and Operations manual were produced as expected. However, to-date the 2 MEURO seed capital has not been transferred because of major procedural hurdles to be overcome between Phare, the Ministry of Environment and the Environment Fund. Phare took over six months to disapprove of the proposed Financial Agreement between the Ministry of Environment and the Fund. Phare requested that the Agreement replicate the terms and conditions for the Slovenian Energy Efficiency Fund. Whilst this was approved by all parties it did not take into account of national complexities with setting up a bank account. The Environment Fund in order to communicate with Phare must go through the Ministry of Environment which itself communicates via the EU Delegation. This complex communication structure has not helped facilitating an efficient resolution to
150

the disagreements. Additionally, only one commercial bank has already signed an operating agreement and one refused to sign. This is because the Operating Manual has not taken into account obligations and concerns of the commercial banks. The Fund is therefore having to learn to negotiate with the local banks. Effectiveness (Score: 1) The Fund has received 40 applications for loans worth $6 million. The Fund, with assistance from the consultants prepared in more detail 12 of the investment proposals, valued at $2.5 million. The Fund presented the project to the supervisory board which approved eight of them. These total $1.2 million. These loans are now being passed to the commercial banks for approval (regarding financial risk). However until Phare transfers the capital to the Fund no loans will be able to be disbursed. Meanwhile the Ministry of Finance has not committed any national budget to the Fund limiting effectiveness to the Phare seed capital when it arrives. Impact (Score: 4) The overall objective of the project is to reduce pollution and energy use as a result of the loans funding projects. There has been no impact to-date because no loans have yet been disbursed. Sustainability (Score: na) The fund will charge for the loans so it can be revolving. No disbursement so dont know yet about repayments. Experience from an already operating Fund in Lithuania shows good signs of repayment. The level of applicants shows that companies are prepared to take out loans at the specified interest rates. USAID has provided a consultant to visit municipalities, Associations and industries around Lithuania identifying suitable projects for the Fund. This will help bring in quality and numerous loans to keep the Fund active.

Score
Efficiency Effectiveness Impact Sustainability Relevance to needs 1 1 4 na 5 5 3

Needs Appropriateness Comparative Advantage

1=poor; 3=satisfactory; 5=excellent

151

KEY LESSONS LEARNED Tailoring to the National Situation If Phare is going to get involved in projects which require legislative development then Phare should consider the cost and benefit of offering a package of assistance including assistance in developing the related legislation and institutional arrangements. The Commission must tailor contracts to be in accordance with national legislation. This will hopefully now already be happening due to the decentralised system of having incountry CFCUs and task managers in each country Delegation. Implementation In general, where project outputs require an additional stage of actions for the implementation of recommendations from a study (in this case the stage between the feasibility study and transfer of funds to the Fund) an action plan should be developed by the PMU with a series of activities laid out which if completed and monitored will lead to the successful and efficient transfer of funds. This planning will allow the Commission to co-ordinate its activities and requirements with the respective national institutions (Ministries and banks). Determining Goals This project highlights a trade-off which is occurring for Phare in achieving its objectives. There are both the projects overall objectives which if met will directly benefit the sector and then there are capacity building objectives which will increase the sustainability of the given process. In this case, creating a Fund under ownership of the government, financially serviced by the National Bank and funds distributed locally by commercial banks has led to difficulties in making contracts agreeable to every institution. A Trust Fund managed by a Phare consultant providing grants or loans directly to enterprises would have been more efficient to establish. However then the opportunity to build of capacity within the government, Bank of Lithuania and the commercial banks would have been lost. Sometimes it may be more effective for a project to separate out its goals. Learning by doing may be the best. However if the project is not working well, such as the EEF then it may not be a good model to follow. Learning by seeing an efficiently set up and operating model (e.g. a Trust Fund) may have a more positive impact which commercial banks would wish to replicate.

152

LITHUANIAN ENERGY EFFICIENCY FUND (11/96-ONGOING) 3 MEURO

INTRODUCTION Phare financed a feasibility study for the establishment of an Energy Efficiency Fund from budget in its 1994 National Energy Programme. This included a Review of Energy Projects to identify potential Fund users and viable investments in Lithuania. The final component was to be the transfer of 2.95 MEURO to the Fund, at the end of 1996, to act as seed capital. The overall objectives are for the loans to finance enterprise investments, which lead to both energy and financial savings in addition to repayment of the loan. The positive results to be achieved by the investments are supposed have a demonstration effect with the objective of attracting private capital for similar types of projects in the future. The specific objective of the feasibility study was to make recommendations on the establishment of the Fund and prepare the various documents, which would enable the Fund to operate soon after the completion of the Study. The specific objective of the seed capital is for the Fund to provide soft loans to small and medium scale projects, which have a reasonable payback period but are unable to be financed solely through commercial financing. However, since the completion of the feasibility study at the end of 1996 there have been several complications delaying the signing of the Financial Agreement between Phare and the Government of Lithuania. As of 10 October 1998 the 2.95 MEURO has still not been transferred to Lithuania and the Fund remains inactive. However the institutional and legal framework has been set up by the Government and is now in place and the Fund is ready to function on receipt of the seed capital. Loans will be for a maximum of 500,000 EURO to target small scale investments. This will satisfy the projects objective of increasing small enterprises access to cheaper capital. PERFORMANCE Relevance to Needs (Score: 5) The restructuring and commercialisation of the energy sector has been a priority of the GOL throughout the 1990s. Within the energy sector the National Energy Strategy (1994) identified increasing energy efficiency as a key goal. This is because enterprises have inherited from central planning, extremely high energy utilisation performances. This has significantly increases their cost of operations and national expenditure on imported fuel. Lithuanias toe (tons of oil equivalent) per $1000 GDP in 1994 was 1.6 compared to 0.4 for the EU. As a result of this very low energy intensity rate national oil and gas imports in 1994 exceeded $1 billion. If Lithuania could reduce its energy intensity rate to EU rates it could potentially save $0.75 billion on import expenditure annually. At an enterprise level there has been a demonstrated demand for energy efficiency investment. Many system components are old and need replacement. The average age of boilers is over 20 years old. Also many boilers could be converted from oil to wood chip. There had been a series of USAID, DG XVII and Danish government funded studies all investigating and showing demand for energy efficiency investments with short term paybacks.

153

Appropriateness of Project Type (Score: 5) It is still very difficult to raise external financing due to the limited availability and high cost of financing from commercial banks due to the underdevelopment of the financial sector. Therefore a Fund is an important mechanism for rapidly distributing capital to satisfy the demonstrated need for investment Comparative Advantage of Phare (Score: 3) Providing a grant to allow soft loans is essential to gain industries interest in investing in energy efficiency measures. Efficiency (Score: 1) There have been significant delays in the process by both the Commission and the GOL (Ministry of Finances), totalling two years. The delay in gaining final approval for the Financial Agreement of the Fund has resulted in the non-transfer of the 3 MEURO and the non-commencement of the Funds operations. The most notable is a gap of 5 months taken by the GOL (Ministry of Finances and Ministry of Economy) between approving Fund regulations in January 1997 and requesting by Ministry of Finance to transfer of funds in May 1997. Then it took Brussels 6 months to respond to the request for the transfer of funds. The main stumbling block has been the inability to reconcile difficulties and legal requirements between Phare and the Ministries, particularly regarding the status and role of the National Bank of Lithuania and the expiry dates of the Financial Memorandum. Effectiveness (Score: 1) The feasibility studys recommendations formed the basis for the development by the GOL of the national regulations for the management and operations of the EEF. The institutional and legal framework is now set up and the relevant parties are ready to start operating the Fund. The project approval criteria, appraisal guidelines and application formats are all developed. One local commercial bank which will financially appraise projects and disburse the loans has been selected. However formal agreements with the bank will only be prepared once the Fund capital is present. The Fund has not started practical operation and no loan applications have been requested as the seed capital has still not been transferred from Phare to the Fund. Therefore the projects specific objectives have not been achieved. However, the review of energy efficiency projects combined with existing energy efficiency investing being undertaken by bilateral donor agencies demonstrates there will be a market for the Fund and that once in operation disbursement should occur within the expected time frame (50% in 6 months and the remainder within 12 months). Once in operations the Funds effectiveness will be increased because the capital in the Fund is to be enlarged. The PMU structured two Phare-funded investment projects to provide grants (worth 2.1 MEURO and 2.5 MECU) to the GOL which are then disbursed as loans to the Lithuanian Gas company which must repay them into the Fund. Consequently, each year the Funds supply of capital for investments will increase by 600,000 ECU.

154

Impact (Score: 4) There has been no impact from the Fund because the fund has not yet become operational and no loans have been made. No other sources of financing have provided capital to the Fund. The feasibility study estimates an energy saving of 150,000 toe at the end of the ten year life of the Fund. This compares to a total estimated technically feasible energy saving in Lithuania of 486,00028 thus contributing to 30% of possible energy efficiency investments. An additional benefit will be the further development of the capacity and interest of local commercial banks to make loans for energy efficiency investments. Sustainability (Score: na) In theory the Fund should be sustainable as the interest repayments are to be reinvested in future energy saving projects. This though cannot be determined until the Fund is operational and loan payback starts. Score
Efficiency Effectiveness Impact Sustainability Relevance to needs 1 1 4 na 5 5 3

Needs Appropriateness Comparative Advantage

1=poor; 3=satisfactory; 5=excellent KEY LESSONS LEARNED Banks have their own procedures, rules and financial accountability which must be taken into account during the feasibility study and particularly when developing agreements with them. Efficiency of establishing and operating a Fund may be increased if there is a dedicated technical expert working within the financial institution(s) making decisions on loan applications. This increases the synergy between the two appraisal procedures and will increase the banks understanding of the technical dimension. The dedicated technical expert could then also carry out the technical assistance outreach programme (see below). Cost-effectiveness should not be the most important criteria in developing fund management. There are two options where increasing expenditure may improve the effectiveness of the Fund: Using staff from other institutions to advise and contribute to a Funds operations with no remuneration may not achieve the best results as there is no (financial) motivation. For Fund effectiveness it may be better to have a committed team or individual administering the Fund. It may be also be worth the cost to engage on a full time basis a financial and technical expert to implement a technical assistance outreach programme and visit small companies (potential borrowers) and assist them in identifying investment projects and understanding the financial analysis needed to ascertain their viability. This will
28

Feasibility study. 155

increase the capacity of small companies to improve energy efficiency (or other environmental goals of the Fund) and at the same time increase the market for Fund lending. As the financial sector develops Funds may not be needed and Phare, like the World Bank, could have a Credit Line, which is disbursed directly through local commercial banks. Phare appears to experience serious difficulties with large complicated projects which take a long time to implement and have to pass through national bureaucratic hurdles. On the one hand the EC cannot pressurise a government into signing a Financial Agreement. On the other hand if the budget is not committed before the expiry of the Financial Memorandum (FM) and then disbursed within one year of the completion of the FM the budget should be withdrawn (according to Phare procedures). This may mean that in the future Phare should avoid agreeing to fund large and complicated projects requiring significant government involvement and alterations to national legislation e.g. establishing Investment Funds (unless the necessary resources can be input to complete the entire process). It appears critical for establishing Funds that responsibility is allocated to a party to work with the relevant institutions to see the process through and ensure the Financial Agreement is approved in an efficient manner. As the consultants responsibilities ended with the preparation of the feasibility study there is justification for the PMU in this (and other similar circumstances) to either be given more authority than to just act as a channel of communication between the Commission and the GOL or to hire a consultant to see the process through.

156

LITHUANIA DH DECENTRALISATION

(4/94-9/95)

0.17 MEURO

INTRODUCTION The project was identified by the Ministry of Energy in 1992 at a time of increasing interest in the decentralisation of district heating. The TOR were prepared in 1994, in response to official statements in support of decentralisation, defining a project to provide assistance to a government commission on decentralisation in preparing the rules, regulations and legislation to direct the process (Task A) and in producing recommendations for how to conduct a pilot action in one municipality (Task B). Task A includes recommendations on national requirements for the decentralisation process and an Action Plan, including recommendations on pricing and asset valuation. The study also examined means of mitigating the social consequences of decentralisation. Task A was completed in April 1996 and task be in July 1996. Project activities started in September 1995. The project was managed jointly by the PMU, the Energy Agency and the Municipality of Birzai. The contractor was Mannheimer Versorgungs- und Verkehrsgesellschaft (MVV), with the Lithuania Energy Institute (ETNA) and UAB Terma. The Final Report was presented to the Commission in mid 1996 and the relevant legislation was adopted in April 1997. The main specific objective of the project was the adoption of recommendations on rules and regulations for decentralised energy distribution and the establishment of the institutions necessary to implement these rules and regulations. In addition, the pilot project aimed to demonstrate the desirability and feasibility of decentralisation. It is also inferred that stimulation of discussion was an indirect objective of the project. The projects overall objective is successful decentralisation of district heating sector in general and in Birzai in particular. This implies financially healthy utilities and an energy efficient district heating network with minimal system losses and lower production costs. PERFORMANCE Relevance to Needs (Score: 4) The financial management of heat supply in Lithuania in the early 1990s was a complex mesh of relationships, including subsidies, cross-subsidies, delayed payments, charge collection problems and unexplained losses. As a result, it was impossible to determine the commercial viability of district heating. However, it was clear that heating supplies were extremely inefficient and that substantial investments would be required to improve efficiency. At a time of widespread privatisation, it was recognised that decentralisation was required in heating services, in order to mobilise resources and manage the new equipment and services. This was also supported strongly by the World Bank and the IMF. However, there were complex problems in legislating for the transfer of responsibilities and assets which would be involved in decentralisation, and then in building the capacity to manage a new approach to service delivery. The project addressed these concerns and was thus highly relevant to national needs.

157

Appropriateness of Project Type (Score: 3) Phare opted to undertake a regulatory study. This could have been of great value to government. However, the execution of the study was more in the nature of a general strategy study, as it did not provide practical and focussed recommendations to steer the drafting of the law. Thus, the original design of the project was appropriate, but the way it was actually implemented was less appropriate. In addition, a regulatory project of such scale (impacting on national, regional and municipal levels) required an additional training component, to guide the institutions involved prepare for the transition. The combination of a regulatory study and a pilot activity in one municipality was relevant and should have been effective, if the regulatory study had been more focused. This moderate appropriateness of the project in practice, was probably dictated by the relatively small allocation of resources to the national study. Comparative Advantage (Score: 3) Phare was well suited to funding regulatory studies, with supporting training. Efficiency (Score: 3) The reports cover the range of subjects identified in the TOR, but are rather general and poorly structured and do not make full use of the local information produced in the reports of the local consultants. Furthermore, the recommendations and Action Plan are not practical enough and cannot readily be implemented by government. Much of the responsibility for the poor project outputs lies with the TOR, which were generally vague, without a clear statements of expected outputs and with confusing references to other energy sub-sectors. The quality of outputs relating to Task B is more satisfactory. Although there are still some criticisms that the report on the pilot actions were too general, the conclusions appear accurate and a list of recommendations was prepared, concerning transfer of assets, financing losses/debts, development of training programmes, and search for more costeffective strategy, including cheaper fuel. Management was generally efficient and co-operative, with the exception of the collaboration between ETNA and the Lithuania State Power System (LSPS), which was problematic and was never solved. The resistance of LSPS to decentralisation in general and to particular elements of the proposed decentralisation was a major constraint to the project. The project budget was modest, for the TOR. Only 25% of resources were devoted to the first task and 75% to the pilot project, which probably explains the quality of the respective outputs. A higher allocation to the national exercise would have been better. This meant that there were constraints both in the cost effectiveness of each component and the balance of appropriateness of the project as a whole. Effectiveness (Score: 1) During the 1996/97 winter, a change in government, combined with strong international pressure, led to the adoption of a law governing energy decentralisation. However, its provisions were generally more varied and flexible than recommended by the project. Also,

158

the new law does not address the pricing issues raised in the project. The implicit objective of improved discussion was achieved. The pilot project was implemented and district heating in Birzai was decentralised, with the assets being transferred to the municipality even before the new Law was passed in April 1997. However, the pilot study has not yet been used as a demonstration model. Furthermore, the training objectives of the pilot project were not undertaken. Political opposition and indifference appeared an important obstruction to effectiveness regarding the output of Task A. It is possible that no matter how good the output of the project, it would have been largely disregarded in the process of drafting the law. Decentralisation was unpopular with many parties. LSPS were opposed in general and the municipalities were concerned about taking on long-standing debt. Politicians were also concerned about the impact on prices and public opinion. Impact (Score: 2) The overall objective of decentralising DH were not achieved because of the low effectiveness of the project. However, the project seems to have had an indirect impact, through the promotion of discussion, which has influenced the attitudes of key individuals to changes which have taken place, even though these changes have not been directly following project recommendations. At present, difficulties remain. Many municipalities are facing financial problems, due to increasing debt, arrears of payment, inadequate tariffs, and operational problems, such as system losses that are increasing due to neglected maintenance, etc. Some municipalities/utilities appear to be on the verge of bankruptcy. Pricing commission established too late. In interim period, tariffs could not cover costs and utilities continued to make losses. It has been argued that the 1997 law, merely transferring assets, should have contained (or be rapidly followed by) provisions to guide the process - as was intended by the project. Sustainability (Score: 1) It is too early to judge the sustainability of the limited indirect impact of the project. Although political commitment to decentralisation now seems irreversible, there are major financial problems in many municipalities which will create problems with the sustainability of decentralisation.

Score
Relevance to needs Needs Appropriateness Comparative Advantage Efficiency Effectiveness Impact Sustainability 4 3 3 3 1 2 1

1=poor; 3=satisfactory; 5=excellent

159

KEY LESSONS LEARNED It is difficult for regulatory preparation projects to achieve their specific objectives of realising subsequent legal arrangements. Regulatory preparation projects can only act as a catalyst and there is a need for many complementary factors to be in place to ensure that a law, taking into account the recommendations, is actually adopted. New regulation are bound to affect parties with different interests and these need assistance in adjusting to the changes. The economic climate should alleviate the inevitable financial problems of adjustment ad some temporary compensatory measures may be necessary. The drafting of TOR needs to create a sense of ownership. Careful study of political will, and realistic expectations of project needs, is necessary before TOR are designed. TOR need government approval, but there appears some room for creativity and for the EU Commission to influence the output required in TOR. Absence of clear specification of activities and objectives in the TOR can lead to disagreement between contractor and beneficiary. TOR should avoid such general phrases as assistance to governmental commission, but require clear output from consultant, to make sure that a certain level and quality of output is guaranteed. Regulatory projects can be successful, but design must be done carefully, geared toward special circumstances. A general output provides for useful information but lacks guidance for action. It may be best to involve more input from local consultants in the drafting of legislation, to ensure that it is closely adapted to local circumstances, though many governments will still wish to see some international input. Whilst, there are good arguments for decentralising project management to government officials and agencies, the EU Commission should retain an involvement in project management to ensure transparency and to provide checks and balances. The Task Managers and PIU should pay more attention to forging initial agreement among key actors when specifying project objectives and outputs. Phare should consider creating a system for systematic local follow-up to completed Phare projects. The main reform steps for restructuring and privatising the energy sector still lay ahead. Private sector initiatives, like foreign direct investments, are lacking in the sector and were not sufficiently encouraged neither by government or by Phare programmes. Future inventions will need to focus on further encouraging private sector involvement, demonopolisation and reform of pricing and tariff policy as important preconditions for increasing the efficient use of energy. These may be followed by a demand side reduction programme. There is a need for focus on electricity and gas sectors - policy cannot successfully focus on one sector only, since this would create imbalances. The decentralisation of gas and electricity distribution is a politically sensitive issue that still needs to be addressed in context of accession. Project impacts can be diluted if goals are not explicit and there is no clear measurement of their attainment Phare needs to make a strategic decision on what goals are most important and ensure they are met. In particular, it needs to be clear whether implicit objectives of promoting debate and capacity building are sufficient to justify a project. The risks that beneficiaries do not adopt or maintain strategies, methodologies or tools developed for them during the course of the project (internal achievement risk) needs to be more thoroughly considered, and this needs to be part of a more general risk assessment in future programme design.

160

LITHUANIA LSPS RESTRUCTURING

(3/95-9/95)

0.30 MEURO

INTRODUCTION This project arose out of a requirement imposed by the IBRD on the grant of a loan to the Lithuanian State Power System (LSPS) for rehabilitation of the power sector in Lithuania. The TOR originated largely from the IBRD and progress with LSPS reforms was a condition of the IBRD loan. Tendering took place in late 1994 and was won by Price Waterhouse with ESB International, the consulting practice of the Irish Electricity Board. The project started in May 1995, and the Inception Report immediately raised questions about the scope of the work, which were never really resolved. The TOR limited the project to a diagnosis of the problems, a schedule of implementation of the existing action plan for reform and assistance with internal restructuring. The consultants proposed devoting some attention to the restructuring itself, since this was not as advanced as the TOR had assumed it would be. The LSPS sought to limit the extent of the reforms to be covered. The project activities included discussions, workshops, support for task forces and the design and promotion of reform proposals. The project also provided lessons from international experience. The expected outputs of the project concentrated on reviewing the current situation and a series of detailed activities to assist in the improvement of internal management at LSPS, concentrating on more transparent financial procedures. The specific objectives of the project were to achieve agreement on a restructuring schedule and to improve internal management. The overall objective of the project was to improve financial performance and efficiency of the heat and power sector. A draft Interim Report was produced, describing current arrangements and options for the future. This report was not well received by LSPS, which felt that it extended beyond the TOR and was too critical of the economic situation of LSPS and the management competence. Further discussions were not constructive and little progress was made. The Final Report was issued in September 1996, one year late, after a draft which had received similar reactions to the Interim Report. PERFORMANCE Relevance of Needs (Score: 3) Prior to independence Lithuania occupied an important position within the Soviet Unions West Electricity region, largely because of the nuclear plant at Ignalina. However, the power sector in Lithuania suffered from most of the problems typical in former centrally planned economies, with dependence on ageing nuclear capacity, low charges, an inadequate legal framework, centralised management, and a lack of financial transparency. The IBRD programme attempted to address these problems by combining resources for investment with economic and institutional restructuring. This programme was supported by a GoL Statement on energy strategy, but this was general in nature and the Phare project was intended to elaborate on this strategy. Thus the project met a serious need for assistance in energy restructuring.
161

Appropriateness of Project Type (Score: 2) It is arguable that the project was premature in that neither the commitment of the Ministry to restructure nor the legal framework were in place to support the restructuring. It would have been more appropriate to focus the project directly on a broader discussion of restructuring option. However, this would not have met with the time-scale required for the IBRD lending programme. Comparative Advantage (Score: 2) The decentralised approach to decision-making which is taken by Phare means that it is not well suited to participate in projects which require firm conditionality. In theory, it should be well-suited to provide technical studies to influence contentious policy debates, but, in these cases, it should first be clear that such technical studies will be useful regardless of the outcome of hotly debated reforms. Efficiency (Score: 1) The review of the current situation was limited. The legal basis was covered only superficially and there was no reference to the action plan which had been developed by government. Work on improving internal LSPS management was patchy. Three was a review of the management and management information systems. Reviews of financial and proposals for improvements were included, but generally lacked detail. Some actions were omitted entirely, including a review of auditing requirement and the affordability of current tariffs and an assessment of work required for a revaluation of LSPS assets. There is some suggestion that these were omitted because they had already been covered by different studies, but the documentation on this is not clear. In contrast, the final report includes substantial work on proposals for restructuring LSPS, even though this was not included in the TOR. Thus, the efficiency of the project was poor, with little achievement of expected outputs. Management. The main reason for poor outputs and efficiency was a serious breakdown of relations between the consultant and the beneficiary. This breakdown in co-operation was perhaps inevitable. The TOR required the consultant to elaborate the implementation of reforms which had not been agreed. It is often tempting to promote a difficult study by limiting the formal TOR to relatively technical implementation issues. This can enable technical arguments to push forward the process of reform. However, it cannot be guaranteed to take help resolve reforms and may lead to expensive delays, which further entrench positions. In the event, the consultants were unable to perform their duties without agreement on reforms and the LSPS were not prepared to participate in defining reforms with external consultants, especially when this threatened their continued existence. Cost Effectiveness. The outputs that were originally expected from the project could not have been produced at lower cost. However, the outputs that were achieved could have been developed more cheaply.

162

Effectiveness (Score: 2) Since the election of the new government in December 1996, there have been major reforms in the energy sector. The government approved the action plan proposed in the Final Report and a significant part of the actions have actually been implemented. The government intervened directly to rationalise the financial situation in LSPS, and the Energy Pricing Commission was strengthened to enable price rises to take place. The government has also announced its intention to implement substantial restructuring in the energy sector. The reforms do not refer directly to the project and many of the government actions do not follow the project proposals. The Final Report is not well know amongst senior government officials, with the exception of the Ministry representative on the board. It was also unknown to EBRD, which is a major investor in the energy sector, and was even unknown to staff at the EU Delegation. Furthermore, there was no copy of the report in LSPS for some time. However, the report is regarded as important with the State Pricing Commission and the Chairman of the Working Group set up to recommend on the break-up of the distribution assets claimed it was important in his thinking, although there is very little detail in the report pertinent to that exercise. There are also suggestions that, although senior official are not familiar with the detail of the report, the project does provide some independent analytical justification for proceeding with reform. According to LSPS, none of the recommendations concerning corporate organisation, financial practices and corporate planning have been adopted. Progress has been made in the improvement of tariffs for electricity and heat, but it is not apparent that the project had any perceptible influence on this improvement. In retrospect, it would have been more effective to support a small restructuring project within the Ministry looking at options and experience. Some activities to support LSPS management could take place independent of structural reforms, but many would be dependent on reforms being agreed. Phare is not well suited to provide support in these circumstances, as the pace of political reform is unpredictable, and Phare procedures are lengthy and not responsive to political change. Impact (Score: 2) Whilst there have been improvements in the performance of the sector, it is difficult to identify any direct impact of the project. This is largely because the political commitment to reform took place after the project activities. Sustainability (Score: 3) There were few direct project results to sustain. The main sustained effects of the project will be to have reinforced the appreciation of key officials of the importance which the EU attaches to energy sector reforms. Thus, despite the poor impact of the project, as measured by achieved of explicit objectives, there may be some sustained indirect results. The poor relations between LSPS and the project may have contributed to the recent collapse of a subsequent Phare project to support performance auditing in LSPS, suggesting the wider impact of the project has been negative in the recent past.

163

Score
Relevance to needs Needs Appropriateness Comparative Advantage 3 2 2 1 2 2 3

Efficiency Effectiveness Impact Sustainability

1=poor; 3=satisfactory; 5=excellent KEY LESSONS LEARNED It is hard, if not impossible, for projects of institutional strengthening to succeed if the beneficiary finds the character of the project or the approach of the consultant to be threatening and if they then withdraw their co-operation. It should be a pre-condition of funding that the beneficiary fully supports explicit TOR and has a genuine interest in the work and agrees to put up resources of its own, albeit in kind. For projects which explore structural change, it should be ascertained that there is an institution which can implement changes. This institution should be a partner in the project, not just an interested bystander. In most cases this institution is a Ministry, supported by cabinet and Parliament. Organisations restructuring projects have an important political dimension. The experience of international development banks suggest that such projects are most effective as a part of a sustained political pressure on government. Traditionally, the Commission has not provided this pressure, though it may have linked its support to the activities of IFIs. However, the Commission is asserting increasingly strong political pressure through the conditions associated with accession. Accession provides a clear agenda and a link to political processes, but the detailed mechanisms by which Phare activities are linked to this agenda require substantial preparation. In particular, the lengthy nature of Phare procedures create difficulties for Phare in supporting this type of project, compared with other donors. The principle of subsidiarity, and the variety of conditions in EU member states, mean that CEECs will retain considerable flexibility in interpreting accession conditions. The role of Phare in influencing CEEC developments will be an important complement to political negotiation, in the same way the Commission analysis influences political developments within member states. The grant element of Phare was fundamental in getting the project accepted by LSPS. However, there is also a danger in the use of grant funds, since LSPS were less concerned to approve the TOR of the project, and to be committed to the project, than if they had been taking a loan. The delays associated with Phare procedures were evident in this project. However, in this project they did not affect the effectiveness of the project, because political changes were also delayed. Collaboration with IBRD was productive, as the Bank's objectives were broadly congruent with the accession requirement of a competitive economy and well managed power sector. However, there remains a difference in approach, with the Bank intervening more actively in the definition of solutions. The project activities followed the Bank's interventionist approach in preference to that of Phare. Both approaches can play an important role in fostering reform, but it is important that the Phare approach is not forgotten, particularly in attempting to generate support for reforms within the institutions affected.
164

POLAND ENERGY FINANCE

(12/95-12/96) 0.16 MEURO

INTRODUCTION The project was to act as an investment catalyst for electricity distribution utilities, district heating companies and co-generation plants. The project was carried out by a local consulting firm, Narodowa Agencia Poszanowania Energii (NAPE). The main activity was a questionnaire survey to companies in the sub-sectors and analysis. The findings along with other data contributed to the preparation of the following outputs: a sub-sector market survey published as a booklet; energy company database put on disc; and a seminar on investment opportunities in the sub-sectors. There were several specific objectives of this project as follows: increase investor knowledge and interest in the energy sub-sectors; increase communication between energy companies and financial institutions; and increase awareness in energy companies of the financial data needs of financial institutions for loan approval. The projects overall objective is to increase long term external financing for investments in district heating and electricity distribution companies. PERFORMANCE Relevance to Needs (Score: 5) Poland has been progressing toward market liberalisation and attempting to involve the financial sector in contributing towards the economys investment needs. The need for private capital in the energy sector was determined to be particularly strong in 1995 because it was one of the most important sectors in Poland requiring major restructuring and investment for rehabilitation and modernisation and the State would not be able to meet all investment demand. The report subsequently estimated investment needs until the year 2,000 are greater than 5 BECU29. The large level of estimated investment need in the subsectors demonstrates the need for a project promoting investment. The DoE believed in 1995 that investors were aware of investment opportunities with the major commercial players in the energy sector - power generation companies, but had limited experience with Electricity Distribution Utilities, District Heating Companies and Co-Generation Plants. There was therefore a need for an investment preparation project for these sectors. Appropriateness of Project Type (Score: 2) The project was not very appropriate as the exercise to gather and publicise commercial information about the energy companies proved commercially sensitive and failed to deliver the expected outputs. The broad range of targeted beneficiaries also led to the type
29

Baed on 4.1 PLN = 1 EURO (1/99) 165

of assistance being inappropriate to many of the energy companies, which were either too large or too small to benefit. Comparative Advantage of Phare (Score: 3) The project provides information on a sector and not just for one company thus making a grant more appropriate. This gave Phare a clear advantage for funding the project. Additionally energy consultants were needed to provide the analysis on the sector. Phare also has a comparative advantage in funding relatively small budget research projects. Efficiency (Score: 2) There were several complications during the implementation of the project. This appears mainly due to poor project design and as a result with the District Heating Association. The energy companies, despite being the projects intended beneficiaries, were reluctant to provide information requested in the questionnaire indicating that the purpose and design of the detailed questionnaire was inappropriate. The project did not take into consideration the varied needs of the energy companies within these sectors or the dynamics of the financial sector. As a result beneficiary companies did not respond well to the study. Considering the difficulties encountered and the scope of the project the outputs were good quality. Effectiveness (Score: 2) The project only contributed to increased investor knowledge and interest in the energy sub-sectors in a limited manner. The Department of Energy did not proactively distribute the report or database catalogue. It only gave out copies to visiting investors expressing interest in the energy sector. As a result of the outputs being disseminated in a low profile ad-hoc basis the level of utilisation of project outputs was low. However, 300 booklets on the market survey were distributed at following seminars and contributed to the debate in the field if not directly to any investment action. There was also a final seminar which provided opportunity for bringing together key players (commercial banks and energy companies). This was well attended and successful at least in the short term (see Impact). Impact (Score: 1) Out of the ten energy companies surveyed during the evaluation none remembered the study or seminar. One main problem is the quick turnover of staff meant that the effectiveness of the seminar was short lived as those attending did not remain in positions responsible for investments, which also partly explains the lack of institutional memory regarding project outputs. Furthermore, companies now still seek low interest credit from National Environment Fund instead of commercial banks. Banks interviewed were not aware of company database. They tend to use their own contacts into the sub-sector. There was a view that the Financial Survey would be useful in helping banks define their sector strategies. However banks are not yet preparing such sector strategies. They are still mostly providing short term credit (working capital) to the larger district heating companies and have not started making medium to long term investments within the sub-sectors.

166

Sustainability (Score: 1) The database, the vehicle for providing on-going financial information to investors on the potential investments in the targeted sub-sectors, has not been updated (presents historical information) and therefore is not useful. The outputs were not sufficient to stimulate on-going effort by sub-sector companies in identifying and proposing investments. There was no follow-up projects to further promote investments in the sub-sectors. The overall objective (of increasing investment) seems to have been left by the Department of Energy. Score
Relevance to needs Needs Appropriateness Comparative Advantage 5 2 3 2 2 1 1

Efficiency Effectiveness Impact Sustainability

1=poor; 3=satisfactory; 5=excellent KEY LESSONS LEARNED The project was not comprehensive or take into account other factors within the targeted sectors which would constrain its impact. For a project to succeed in actually generating investments it must be more comprehensive. Two main external constraints for investment preparation projects are the following: macroeconomic conditions providing legal and financial incentives for investment for both the investee and the financial institutions. internal institutional reform. The organisations looking for investment need direct assistance in strengthening their capabilities in financial information collection, analysis and presentation. There were too many beneficiaries (over 500 energy companies) all with different inhouse capabilities for investment planning. The study did not focus on any particular type and therefore ended up not being beneficial to any. Sectors should be broken down and the project tailored exactly to the type of institution to be assisted. Lack of focus will only lead to diffuse impacts such as in this project. Project impacts can be diluted if goals are not explicit and there is no measurement of their attainment. During the evaluation many comments inferred that the project was a success because whilst the explicit project goals were not fully met secondary benefits and Phare objectives of capacity building, skills transfer and demonstration value were achieved. Phare needs to make a strategic decision on what goals are most important and ensure they are met. The projects looked at in Poland suggest that up until now Phare has been satisfied with a project not achieving its explicit objectives e.g. promoting investment, because it helped further the debate and acted as a demonstration of where the field could be headed. If this is not sufficient then Phare will need a system in place which checks to make sure all the macroeconomic conditions are already in operation (and not upcoming) which will allow the project to have a significant impact on its beneficiaries.

167

POLAND INSTRUMENT COSTING

(91-12/94)

0.20 MEURO

INTRODUCTION The project was identified during the first year of Phare's operations in Poland. Most of the work on the study was done during the last half of 1994. Following an extended period of comments and redrafting, the final report was accepted in early 1996, after several extensions. The activities were based on economic modelling and analysis of the implementability of different instruments. They also included preparation of checklists of the issues for subsequent analysis of alternative instruments. The main expected output was a report describing: alternative economic instruments; a methodology for calculating the costs of these instruments; and policy recommendations. The project also held workshops to promote public debate. These outputs were produced. However, the debate on the report focused on the technical description and costing of pollution abatement technology, rather than the more strategic implications. Attendance at the workshops was varied and widespread, including senior civil servants and academics, industry representatives and several politicians. The project was supervised by a Steering Committee, chaired by a Professor and including all Department directors in the Ministry of Environment, plus the Phare PAO. The project was initiated and strongly supported by several people on the Steering Committee, along with an international adviser and the academic community. Implementation was done by consultants from the Cracow Academy of Science in Poland and Grontmij Consulting Engineers with TME from the Netherlands. The project ran concurrently with another related Phare project to fund a detailed study of the possible implementation of tradable permits. The project aimed to promote awareness and understanding of different instruments and assist government in reaching conclusions on these instruments. The overall objectives were that the analysis and recommendations in the report and seminars should be used in subsequent policy formation. PERFORMANCE Relevance of Needs (Score: 5) The project was prepared soon after the change to democratic government in Poland, when it was already clear that Poland would be seeking membership of the EU. With some of the largest environmental problems in Eastern Europe, it was also clear that compliance with EU environmental standards would be a major issue in preparing for and negotiating accession. It was thus critical that the government should consider the most cost-effective strategy for compliance, to minimise requirements for investment and operation costs related to environmental protection. Poland has a long tradition of environmental regulation, based on a combination of permits and charges. In 1990, Poland was energetically embracing market reforms and it was thus natural that the country should be exploring options for environmental management, which were viewed as consistent with these market reforms. A theoretical awareness of the
168

various policy options for managing pollution abatement was fairly widespread in academic, government and industrial circles. However, there was a need for applied analysis of the theoretical arguments. Thus, the project was designed to be relevant to an important national need. In the event, the focus of tradable permits meant that the project was less relevant, as this was not the policy option pursued by the EU. Relevance of Project Type (Score: 3) Although the provision of technical assistance to help with applied policy analysis should have been an appropriate type of support, a number of problems undermined the appropriateness of the project. Firstly, several other similar initiatives were taken, which proved to be more effective. Secondly, there was inadequate attention to skill transfer, capacity and dissemination. Comparative Advantage (Score: 4) Phare has a comparative advantage in providing technical studies, which aim to support and inform policy debate. However, the project demonstrates that these studies should be applied, rather than academic, as Phare is not able to provide highly technical supervision. Efficiency (Score: 1) Cost-effectiveness. Although the project was amongst the smallest 30% of strategy projects funded by Phare, it would still have been possible to achieve the analytical objectives of the project with a much smaller budget. But the project can be considered cost-effective because it also contained elements of expenditure, which supported the broader Phare objectives of capacity building through collaboration between Polish and international consultants, and some personal training. Management. The relationship between Polish and Dutch consultants was only partly successful. At a technical level there was some good collaboration amongst selected individuals over the use of MOSES. However, beyond this input it would probably have been more efficient for international expertise to have been provided through participation in a panel of experts. The Steering Committee appears to have worked efficiently, and provided substantial input, particularly in the form of comments on the Draft Report. Quality. Government officials are generally happy with the report and it is frequently referred to as a base source of information, normally in the context of the arguments in favour of economic instruments. Nevertheless, the core of the report does not provide clearly argued analysis and most of the observations have to be taken on trust. Furthermore, there are no explicit recommendations. Also, it would have been useful to address the issue of corruption in the analysis of implementability. Finally, there may be problems with the consistency between the figures in the Executive Summary and the main report (though it is only possible to infer this form indirect quotation of the Executive Summary, as an copy is no longer available. The main responsibility for efficiency problems must lie with those who drew up the Terms of Reference. Either the TOR should have had less ambitious modelling requirements, or the objectives should have been explicitly expanded to include preparing the Ministry for the management of environmental compliance. Some responsibility also rests with the consultants, who should have adapted their analysis to reflect this basic problem, instead of remaining locked into the use of restrictive models.

169

Effectiveness (Score: 3) The project did play an important role in promoting debate, which has continued to spread widely amongst government and the academic community. However, the analysis done in the report is not widely understood, and the analytical methods have not been adopted in the Ministry of Environment. The project has not had a direct effect on wider public debate, but may have had an indirect effect through the work of key individuals involved in the project. The report does not include explicit recommendations for the adoption particular policies. However, there are clear implicit conclusions about the advantages of tradable permits for environmental management in Poland, and the implicit recommendation is to adopt this policy. Implicit objectives included a general interest in capacity building in policy analysis, and this has been achieved and is continuing to spread through academic training. A 'hidden' objective, which has only become apparent since the project was undertaken, was the stimulus to data collection which will be essential for effective management of compliance in the future. There has been good commitment from the Ministry of Environment. Unfortunately, there appears to have been no involvement from the Ministry of Finance, despite the importance of their role in negotiating accession. This was a major limitation to the project.

Impact (Score: 2) The implicit recommendations for adoption of tradable permits have not yet been implemented by the government and there is no sign that plans are being made to introduce tradable permits. However, this should be seen in the light of developments across Europe, where there is an ongoing debate on the potential value of tradable permits. If this is taken forward, Poland will be well-placed to participate. The study has contributed to several subsequent pieces of analysis applied across different CEECs. Significantly, this analysis has been focussed on costing accession, rather than on evaluation of policy alternatives, and it is possible that the main longer term impact of the project will be in providing supporting analysis for accession negotiations. The report has been widely circulated within the Ministry of Environment, but is not well known outside the Ministry. However, The study has made a contribution to the analytical skills in Polish academia, although the restrictive licensing associated with formal modelling has limited this. Finally, the project has drawn attention to the strong need for further data-gathering activity. Sustainability (Score: 3) The direct sustainability of the methods of analysing costs was undermined by the use of models which can only be operated by a limited number of people. However, the status associated with such modelling may have helped to promote debate over alternative strategies, and this would help to promote public acceptance of environmental issues in general, as well as to help refine strategies.

170

Score
Relevance to needs Needs Appropriateness Comparative Advantage 5 3 4 1 3 2 3

Efficiency Effectiveness Impact Sustainability

1=poor; 3=satisfactory; 5=excellent KEY LESSONS LEARNED Phare should ensure that it has the technical capacity to manage strategic studies, either from in-house staff or contracted specialists. This support should be available from the beginning of the project, and approval of the TOR should be conditional on prior 'peer review' of the proposals. Phare should avoid funding excessively academic studies, as they do not have the experience to guarantee relevance and quality, or to ensure that results are sustained. Strategic studies should be combined with explicit plans for further follow-up, even though it will not be clear what the detailed contents of this follow-up might be, until the results of the study begin to become available. Studies must include specific recommendations, along with associated justifications. Phare may have an important responsibility to try to encourage inter-Ministerial collaboration, notably between technical ministries and the Ministry of Finance. Poland has some relatively advanced debate on environmental instruments, as well as some strong analytical capacity, especially in academia. The Phare principle of supporting local initiative should have been extended to allow a home-grown analytical method to be used to analyse the environmental costs of accession. There is a potential conflict of interests between Phare's role in supporting local capacity in candidate countries, and the need for the Commission to manage accession negotiations. This may become increasingly obvious, with Phare providing country support and other parts of the Commission taking particular positions in negotiations.

171

STRATEGIC STUDIES AND AWARENESS-BUILDING ACTIVITIES SHOULD BE FUNDED BY GRANTS, RATHER THAN LOANS. HOWEVER, IT IS ALSO IMPORTANT THAT THEY ARE FUNDED BY ORGANISATIONS WHICH CAN PARTICIPATE IN THE TECHNICAL SPECIFICATION AND MANAGEMENT OF THE WORK. PHARE APPEARS TO HAVE LITTLE ABILITY TO PROVIDE THIS SORT OF TECHNICAL ENGAGEMENT, AND THERE IS THEREFORE A RISK THAT PHARE-FUNDED STUDIES CAN LOSE FOCUS, THROUGH THE INFLUENCE OF PARTICULAR INTEREST GROUPS.

172

ROMANIAN HOSPITAL ENERGY EFFICIENCY (12/95-9/97)

0.25 MEURO

INTRODUCTION The project was identified under the Phare 1992 Programme and aimed to assist four hospitals in adopting energy saving measures which could then be demonstrated to other hospitals in Romania. Consultants were also required to assist each hospital in drafting investment plans. The investment plans would help the hospitals secure financing for identified energy efficiency investments. Lastly, they were to draft an energy conservation plan for the health sector. During the project, energy surveys were undertaken, equipment was supplied and energy efficiency measures were implemented, training was provided to staff, and leaflets were prepared and a workshop organised to disseminate information on the energy saving measures to relevant parties. The overall objective of the project is the replication of the new energy efficiency measures (techniques and equipment). The project had several specific objectives as follows: implementation of a range of no cost and low cost measures dissemination of the results of the project adoption of the energy conservation plan for the MoH acceptance of financial proposals by banks PERFORMANCE Relevance to Needs (Score: 5) In the 1997 Partnership Agreement, energy policy appears on the list of medium-term objectives. Energy efficiency is mentioned as one of several efforts that need to be stepped up considerably. The Acquis also requires improvement of energy efficiency. In the early 1990s when the project was initially identifies, there was a clear need for implementation of energy efficiency measures. Energy intensity was very high, compared to European standards. At the time that the projects were identified, there was no clear energy strategy for Romania. However, with government and IFIs (including Phare) focusing mostly on energy supply, it appears that implementing a project addressing energy demand would be useful. Within the health sector the hospitals are not energy efficient, and energy efficiency has not been a priority of the MoH. For most or all hospitals, there is a need for assistance in energy auditing, implementation of energy conservation measures, metering and monitoring, project management, and financing. Appropriateness of Project Type (Score: 2) Audits were a necessary first step to demonstrate to hospitals where energy savings could occur. There is a question as to the timing of the project because hospitals still do not have control over their budget so whilst convinced by the energy saving potential of investments they do not have control of their budgets to invest in them. Therefore the projects impacts were always going to be constrained.
173

Comparative Advantage of Phare (Score: 4) Phares comparative advantage is that this type of assistance needed a relatively small value grant to several beneficiaries with no large potential investment to follow. This means that IFIs would not be interested. Phare is well suited to provide technical assistance for these types of project with multiple beneficiaries and limited lending needs being generated. Efficiency (Score: 2) The requirements of the TOR were met and the consultants managed to identify and implement beneficial measures and calculate associated energy-savings. However, the quality of the deliverables was not fully satisfactory. The energy surveys were generally too superficial. The final report was poorly written and was insufficient particularly regarding lessons learned and recommendations. It did not even discuss the major problems found with investment for hospitals in final report. Bankable projects, as requested in the TOR, were not completed. The PMU and ARCE were also not satisfied with the Energy Conservation Plan, which according to them should have been a strategy. Generally, it does contain some useful ideas, but is rather superficial, does not set targets and does not consider local circumstances. Because ARCE (the beneficiary) was not satisfied with the deliverables of the consultant the completion of the project was delayed30. Effectiveness (Score: 2) Energy efficiency measures were assessed, and implemented in all four hospitals. All equipment was installed. According to the consultants, 2000 leaflets were disbursed through the Association of Hospitals. There were very few representatives from other hospitals limiting the dissemination occurring at the final workshop. The National Conservation Plan was prepared, but was not implemented. Impact (Score: 1) The project has not had an impact so far. There has been no evidence of replication of the demonstration projects and hospitals are still not equipped to undertake energy efficiency measures. Whilst the project has increased the interest in hospitals in energy savings (as the demonstrations have proven that a considerable saving can indeed be achieved) the hospitals still do not have staff to develop an energy policy, nor do they have the budgetary power to implement any. Sustainability (Score: na ) The targeted hospitals will continue to finalise their bankable projects and submit investment proposals to banks but there are doubts about the sustainability of the process due to the limited budgets available to the hospitals.

Problems on the side of the local supplier caused a further delay in the procurement of the required equipment, and subsequent monitoring of energy use. According to the Consultants, this was due to complicated Phare procedures.
30

174

Score
Efficiency Effectiveness Impact Sustainability Relevance to needs 2 2 1 na 5 2 4

Needs Appropriateness Comparative Advantage

1=poor; 3=satisfactory; 5=excellent KEY LESSONS LEARNED The hospital project had a design weakness in that it did not impose a responsibility on the consultant to disseminate information (it only required the printing of leaflets), and incorrectly approached the financial issue by aiming at designing bankable proposals where the primary problem was with the structure of the health sector. In this project an initial step was required: restructuring of the health sector and in particular the budgetary rules, so that hospitals would have more liberty to allocate parts of their budget to energy efficiency. Again, it shows the risk to projects of uncontrollable external factors. The question then becomes how far can Phare become involved in either resolving the more macro conditions or at least conditionally linking the projects to their resolution. A demonstration project can increase its impact if the results are widely disseminated. Dissemination should be the responsibility of the consultant and required in the TOR. The project suggests it is critical to consider whether the appropriate external conditions are in place, and if not, whether complementary components could help to overcome barriers. If a project is designed to cover aspects that can make or break its success, there is a better chance that effectiveness and impact are achieved. Such integrative approach can lead to better results than a single approach, where success is still determined by other factors, but where the project itself does not try to influence those. Project identification and design are also critical. The hospital project may have been more successful if the TOR had been stronger. It is recommended that Phare procedures allow for more attention to project design and identification. The PMUs themselves cannot deliver sufficient expertise on each topic that is or needs be addressed - such would require a rather large team of experts. Instead, the following is advised involve other government and non-government institutions in passing and judging ideas during project identification and design. It should be noted that the role of the external adviser to the PMU has been a useful one. Having state-owned companies as beneficiaries of projects may be risky as they tend to be less motivated. It is proven again that beneficiaries need to be further involved if anything is to be achieved. Involvement will be a proof of commitment during the project and may sustain a sense of ownership afterwards, which would increase chances of political change. In the case of the hospital project, the MoH was on the Steering Committee, but
175

does not appear from files or interviews that it had any sense of ownership of the reform project. There needs to be involvement and sense of ownership at sufficiently high level, both in government (MoH) and the institutions were measures are implemented. In case of hospitals, were management decisions are taken by doctors, it is not sufficient to work with and train technical staff that has no impact on management decisions. Investigate whether it is better to have one big demonstration project instead of a number of smaller ones. In any case, every demonstration should be significant enough to have clear demonstration value i.e. dont expect a cumulative effect from a number of small demonstrations. Beneficiaries could be required to provide part of the funding (as is case in GEF project mentioned above). Such will allow for bigger project. And will more likely guarantee commitment. It can be argued that Phare should not in each case wait until a regulatory framework is in place. Nonetheless, it might be better to focus first on sectors where replication can be achieved (because there are no institutional barriers), such as industry. Demonstration projects can provide piece of the puzzle, a bit of visible proof. But it is key that Phare identifies where and when it fits into an overall strategy for intervention in a given sector. Requires co-ordinated approach to other on-going activities in the sector.

176

ROMANIAN ENERGY SERVICES

(9/96-9/98)

0.6 MEURO

INTRODUCTION The project was identified in 1992 by the Romanian Agency for Energy Conservation (ARCE). The TOR was not prepared until May 1995 and the project did not start until September 1996. The overall objective was for the replication of the demonstrated energy efficiency measures in other companies. This project had the following specific objectives: provision of assistance to five enterprises to implement effective energy management; creation of a long term link between Romanian consultants and enterprises for provision of technical assistance; this is described in the TOR as integrated energy services; dissemination of results of demonstrations; and acceptance of financial proposals by relevant institutions. International consultants, NIFES, undertook the energy audits in each company, to identify and quantify energy efficiency measures. Seven energy efficiency projects were selected as demos, on the basis of their replicability and a quick return on investment. Four investment projects (in three enterprises) were identified and elaborated to a bankable status. This comprised a technical feasibility study and an analysis of the financial performance of the sponsors in a form suitable for presenting to lending institutions. In two of the participating enterprises it was not possible to agree an investment project. PERFORMANCE Relevance to Needs (Score: 5) In the 1997 Partnership Agreement, energy policy appears on the list of medium-term objectives. Energy efficiency is mentioned as one of several efforts that need to be stepped up considerably. The Acquis also requires improvement of energy efficiency. In the early 1990s when the project was initially identifies, there was a clear need for implementation of energy efficiency measures. Energy intensity was very high, compared to European standards. At the time that the projects were identified, there was no clear energy strategy for Romania. However, with government and IFIs (including Phare) focusing mostly on energy supply, it appears that implementing a project addressing energy demand would be both novel and useful. Romania has experienced a slow and complicated Romanias economic recovery. Romania ended 1997 with an inflation rate of 151.4 %. Privatisation has not taken off as planned, and the delays have contributed to a worsening of economic and financial conditions. Most of industry is in poor condition, and the financial sector is reluctant to lend money. Therefore energy efficiency has not been targeted and is an area where assistance is valuable. Appropriateness of Project Type (Score: 4) The main component of the project was highly appropriate. With low energy intensity providing audits to industry was the first necessary step to identify where energy efficiency
177

investments could occur. As many of the major energy saving measures are costly and industry would need loans it was important that the project assisted in securing financing for the investments, which it attempted. Comparative Advantage of Phare (Score: 4) Phares comparative advantage is that this type of assistance needed a relatively small value grant to several beneficiaries with no large potential investment to follow. This means that IFIs would not be interested. Phare is well suited to provide technical assistance for these types of project with multiple beneficiaries and limited lending needs being generated. Efficiency (Score: 3) The outputs delivered are good quality. These included: completion of audits; preparation of energy management programmes (for participating enterprises); identification of seven demonstration projects; completion of four investment projects and identification of funding for two sponsors; ordering of demonstration equipment (for low cost measures); and preparation of five case studies (leaflets) and 200 copies of each. However, the project was not completed within the official contract period of two years. There were delays in the approval of participating enterprises and approval of the selected demonstration and investment projects and delays in the implementation of the EBRD energy saving scheme (to be the source of financing for the proposed investments). Overall, the 0.6 MEURO appears a reasonable value for the project considering the number of inputs expected. Effectiveness (Score: 3) The seven identified energy saving demonstration project had not yet taken place. Consultant has prepared 200 leaflets on each case, however, and agreed with the Romanian Agency for Energy Conservation to distribute these upon completion of the project. Financing sources from commercial banks have been found for two investment projects. The three local consultants as well as the industries participating have been highly involved in the project and it is probable that a transfer of know-how (on energy efficiency auditing and project preparation) occurred. Impact (Score: na) The overall objective of the projects is for the replication of energy efficiency measures in other companies and continued co-operation between consultants and companies in Romania on energy efficiency issues. Since the project is still not complete the overall objectives have not yet been achieved. This project, however evidence during the project suggests that is has good potential to achieve its overall objectives31. Sustainability (Score: na) There was no element of sustainability to detect as the project was not yet complete.
industries have a sufficient degree of flexibility in their financial planning to allocate budget to energy efficiency measures;the Consultants have provided good case studies and, unlike the consultants in the hospital project, are responsible for disseminating the information under the project; and local consultants were trained with the objective to assist Romanian industry in implementing energy efficiency measures.
31

178

Score
Efficiency Effectiveness Impact Sustainability Relevance to needs 3 3 na na 5 4 4

Needs Appropriateness Comparative Advantage

1=poor; 3=satisfactory; 5=excellent KEY LESSONS LEARNED Once convinced of the benefits of the technology, companies/individuals can become powerful channel for dissemination. Financial and political constraints make it less likely that a demonstration will be successfully replicated. However, the demonstrations, especially if results are widely disseminated, may contribute to making industry and government aware of the need to lift these constraints. In addition, the projects under evaluation were made up of additional components that suggested a comprehensive approach, aiming to address also these financial and political constraints. The project increased its chance of success because it recognised that demonstration alone is not enough when it is clear that the beneficiaries also need help in raising funds for medium- and high cost measures and therefore consisted of (1) demos; (2) dissemination; (3) financial component; (4) policy component, for hospitals. Helping beneficiaries to design financial plans is a useful component of the industry project.

179

ROMANIA ARCE SUPPORT (91-9/94)

0.44 MEURO

INTRODUCTION The Romanian Energy Conservation Agency (ARCE) was set up in 1991 and was the first energy conservation agency to be set up in CEECs. It was formed from an existing state body which relied on inspection and regulation, whilst ARCE was to rely on technology transfer. The request to Phare originated in ARCE, which was looking to learn from international experience. The French Environment and Energy Management Agency (ADEME) was selected in Brussels by Phare to carry out the work, and Phare managed the project. It is not clear if the project was tendered and the TOR are no longer available, either in Brussels, the PMU or ARCE. The project had two components: provision of a long-term adviser and a study, which was conducted from the end of 1992 to mid 1993. Work was monitored by a Steering Committee comprising representatives from the Romanian Government, the European Commission, ARCE and ADEME. The main outputs of the project were recommendations for the structure and funding of ARCE and for the legislative structure for energy efficiency. The specific objectives were the adoption of the various proposals and use of the skills transferred. The overall objectives were to create an effective agency and design and implement useful programmes, thus leading to increased energy efficiency in Romania. PERFORMANCE Relevance To Needs (Score: 3) At the time Romania emerged from the revolution of 1989, the political context was complex, with reformist and conservative tendencies in conflict. The energy sector was highly political, but there was a widespread perception of wasteful energy use and there was some receptiveness to the need to reform institutions and adopt different tools of economic management. It was also clear that improved energy use would be an essential for greater participation in Europe, both for Romanian industry to compete in Europe and for environmental reasons. It was perceived by reformist elements within the Ministry that management of energy use in the industrial sector should proceed through the encouragement of proper methods and the best technology rather than by regulation and penalties. It was also realised that this would require a major process of adjustment in management skills and practices. The staff of ARCE had a background based on regulation and penalties and it was clear that they would need reorientation, both to manage the promotion of new technology and the development and adoption new regulations and legislation. The new Agency had an ad hoc legal and institutional form and no strategy or business plan. The project was therefore highly relevant to national needs.

180

Appropriateness of Project Type (Score: 5) It was relevant to provide institutional support to ARCE, before embarking on other programmes of training or demonstrations, and the combination of long and short term assistance was well-designed. Comparative Advantage (Score:5 ) Phare should have a good comparative advantage in providing institutional strengthening for energy efficiency agencies. The longer term nature of this work means that it is not greatly harmed by Phare's lengthy procedures. Furthermore, EU countries have strong and varied experience which Phare resources can be used to mobilise. This experience may be of particular relevance when planning for energy efficiency in the context of accession. Efficiency (Score: 5) The are no records of planned or actual use of resources for different activities. None of the persons interviewed suggested that the activities had in any way been fewer or less comprehensive than expected. The Final Report is in five volumes. They are fairly short and lacking in detail, but are generally of high quality and focus on the important issues. Furthermore, they should be judged against the enormous challenges that would have been faced in the country at that time. The principal failing of the reports is that they do not review many options but specify a single recommended line of development. The recommendations provided by the project were of good quality and followed international practice. Proposals were made by the consultants for a programme of dissemination of material to energy users in all energy using sectors. Detailed proposals were also made for training in energy efficiency technology and management. The key elements of a strategy for renewable energy were also defined. The project was required to disseminate ideas regarding energy efficiency. The main vehicle for this seems to have been the Steering Committee, which was also concerned with project management. It is difficult to judge the degree of success in disseminating information, and it is possible that too little emphasis was put on this aspect. Good data on energy use in industry was available and a data-base was established for this data. The MEDEE demand model was modified to cope with the special requirements of Romania and on-the-job training was provided. It is not obvious that such modelling activity should be a core-function of ARCE and its relevance could be questioned. Finally, the project proposed a programme for industry including ten specific actions. The analysis was thoroughly performed and was used as a basis for subsequent actions, including those supported by Phare, and provided useful data about convergence costs. Cost Effectiveness. In the absence of TOR and other project documentation it is hard to be certain about the efficiency of the project. However, the nature and quality of the work in the difficult context of the time suggest a cost-effective implementation.

181

Management. There is no documentary record of the management. No managerial problems were identified during interviews. The beneficiaries generally expressed considerable satisfaction with the project and all the interview evidence suggests that cooperation from the beneficiaries was willing and enthusiastic. The quality of the consultants seems to have been universally applauded. Several interviewees indeed expressed a view that the early Phare projects were generally superior to later ones in that the beneficiaries were keen to acquire the new knowledge and approaches that consultants brought. As this relatively easy phase of knowledge transfer was overcome and the harder phase of implementation was entered so the limitations and constraints of technical co-operation became visible and a certain disillusion set in. Such work could only have been supported by grant. No other donor seems to have been approached. At the time Phare procedures were less onerous and indeed the time from the creation of ARCE to the implementation of the project was remarkably short. Effectiveness (Score: 1) Most importantly, the project's proposals for the institutional and financial independence of ARCE were not adopted. The energy sector is still dominated by powerful energy supply interests and this has severely hampered ARCE's effectiveness. Proposals for legal development have also not been adopted, though there is now some possibility that a draft currently before Parliament will be passed. The project proposals failed mainly because the political support for change was not sustained, as a result of the dominance of short-term concerns to ensure energy supplies and a consolidation of conservative forces. There is still very limited commitment to ARCE. There seems to have been very little use of transferred skills, and the training impact of the project has been very low, partly because of constraints on ARCE's activities and partly because of high staff turnover at ARCE. The main effect of the project was that many of the proposals for future work were subsequently funded by Phare. Indeed almost all subsequent Phare projects can be traced back to the Strategy proposals. However, this was a relatively minor part of the project. Impact (Score: 2) The impact on ARCE has been small, because government has not followed the projects' proposals. The EC does not seem to have promoted the project proposals. However, there has been some delayed impact on the preparation of legislation as some of the individuals outside ARCE were involved in the project. A wide range of follow-up projects has been performed with ARCE as beneficiary. Most of these projects follow closely the recommendations of the Strategy project. Whilst the projects seem well conceived, many seem not to have been effective, either because of poor design or because of a lack of complementary legislative, strategic, institutional and financial actions.

182

Sustainability (Score: 1) The project was not sustainable largely because the complementary actions did not materialise within the Romanian government. The problems with the subsequent project to support an energy efficiency fund further undermined the sustainability of ARCE. It is unfortunate that the EU has not stressed the issue in the Accession Partnership or its ongoing political dialogue with government, despite the importance of energy efficiency, both in the Acquis and in fostering economic efficiency. Score
Relevance to needs Needs Appropriateness Comparative Advantage 3 5 5 5 1 2 1

Efficiency Effectiveness Impact Sustainability

1=poor; 3=satisfactory; 5=excellent KEY LESSONS LEARNED The main reason for ineffectiveness was an inability to gather political support for the changes, exacerbated by a shift in the political conjunction. In all probability this could not have been foreseen and the project was a risk well worth taking. However, it is more doubtful that subsequent support should have been provided to ARCE, when it was evident that there was no serious intention within government to adopt the project's key proposals. A more rapid identification of such problems should be possible if the Delegation were to relate the outputs of institutional strengthening projects more closely to their political dialogue with government. This is becoming even more necessary as the emphasis of Phare shifts to accession. This will involve the design and management of more conditionality in EU projects. The lack of an effective programme in energy efficiency is a serious impediment to Romanias intentions to accede to the EU. The very poor level of efficiency in energy use will lower the competitiveness of its industry and will lead to direct conflicts with the requirements of the Acquis.

183

ROMANIAN POLLUTION MONITORING

(91-11/95)

1.19 MEURO

INTRODUCTION The project provided technical assistance for the preparation of a masterplan for pollution monitoring and for training in the operation of two mobile labs. The project was coordinated with that of the other main project in the 1991 programme, to support the Ministry of Water, Forests and Environmental Protection (MWFEP), which took place largely during 1992. The masterplan project was undertaken by SGS-Ecocare from Belgium. Work on the masterplan took place during late 1992 and early 1993, with the final report being produced in mid 1993. The technical support for the operation of the mobile labs was intended to provide immediate piloting of the masterplan, but was seriously delayed because the labs did not arrive until 1995. The Master Plan provides a framework in which already active structures can be progressively developed into an integrated environmental monitoring system, combining all environmental sectors. The Plan identified 39 Master Plan Projects (MPPs), required to implement the Plan, which were prepared in collaboration with the Romanian Agency for Development (RAD). The project involved three main stages: a review of the existing system; an assessment of the future information needs; and a strategy for implementing a new system. The specific objectives of the project were the adoption of proposals for organisational change and the implementation of the projects identified as necessary in the Master Plan for its practical implementation, including the procurement of new equipment. The overall objective was an effective, integrated monitoring system, resulting in an improvement of the accuracy of environmental data, within the context of the sector objectives of reducing pollution levels. PERFORMANCE Relevance to Needs (Score: 5) The most serious pollution in Romania is mainly found around the metallurgical, chemical and machine-building industries, which are energy intensive and generate serious air and water pollution, along with a range of hazardous wastes. Before 1989, little attention was paid to environmental control and the resources devoted to environmental investment are still low, even by the standards of Central and Eastern Europe. Some first steps towards more comprehensive environmental management were made in the early 1990s, including changes to the MWFEP and an environmental law which recognised the polluter pays principle. A policy framework was firstly established in the 1992 Environment Strategy Paper (ESP), prepared with the assistance of a range of major donors. The ESP recommended a '10 Point Plan of Action'. Debate on integrated environmental management continued for several years, during which the project took place, and the Government finally approved an Environmental Protection Strategy in December 1995.

184

Romania submitted its application for membership of the European Union in June 1995, along with a national pre-accession strategy. This led to the formal Accession Partnership in December 1997, which serves as a basis for the pre-accession strategies for all candidate countries, and aims to help in mobilising resources for convergence. Institutional strengthening and environmental protection are key priorities in these documents. The regular gathering of information on pollution is a pre-requisite for environmental protection and building this capacity required starting with a national plan. Thus, the project was directly relevant to national priorities. Relevance of Project Type (Score: 5) The provision of technical assistance to help the definition of future activities, combined with some training to assist in the implementation of a priority activity appears to have been a relevant way of reaching the objectives. The concentration on practical action plans was probably correct, with some related input into policy-level strategy. Comparative Advantage (Score: 4) The use of Phare resources for supporting masterplans seems to have been relevant. The only disadvantage of this approach is that it can lead to a sense of donor dependence, in which future actions are assumed to be the responsibility of international partners. This dependence could extend to dependence on EU instruments such as Cohesion and Structural Funds, after accession. The fact that the only projects in the action plan which yet have committed financing are both funded by Phare illustrates the dangers involved. Efficiency (Score: 3) The review of the existing system was comprehensive and efficient and involved wellstructured use of questionnaires and workshops. The analysis of the main problems in the system and needs for improved information was also detailed and comprehensive. The strategy for the development of the integrated monitoring network is the core of the report and is based on 39 separate projects. In general, the funds were spent cost-effectively and the project was managed efficiently. In particular, the timing and design of the project meant that co-ordination with the Phare institutional project and with other assistance from IBRD and bilateral donors was good. However, the project was relatively expensive, and, in theory, it should have been possible to make savings by making better use of government expertise and local consultants, particularly for the preparatory data gathering activities. In practice, there was very little experience with managing this type of project in Romania, and a strong involvement from external consultants, with management from Brussels, was probably essential. Indeed, communication was severely hampered in the early years by problems with language and telecommunication facilities. Effectiveness (Score: 2) The level of effectiveness was relatively low. Most importantly, there has been little progress on the organisational changes which are an essential pre-requisite for effective investment in the monitoring system. Two of the 39 identified projects in the masterplan have been implemented, both funded
185

under the 1992 Phare Programme. The first provided 45% of the basic requirements for monitoring equipment and the second supported industry self-monitoring in one pollution hot-spot. Proposals for subsequent Phare environment programmes in 1995 and 1997 have focused on further improving the quality of data collection and communication. However the EC has rejected these proposals because institutional responsibilities are still unclear. The lack of effectiveness has resulted largely because the frequent changes in government have made it difficult to sustain any substantial institutional reform. The large number of existing institutions involved in pollution monitoring, increase the vested interests and the level of commitment which will be required to make reforms. This will require clear statements of the authority of the MoE in supervising pollution monitoring, which will require changes to the legislation. Impact (Score: 2) The level of impact was very low, because of the low effectiveness described above. However, some impact was felt through the provision of the basic equipment included in the Phare 1992 programme. This equipment is used efficiently to produce useful information. Sustainability (Score: 1) The main impact of the project was that it led to the provision of basic monitoring equipment. This organisations using the equipment have the technical capacity to manage the equipment and have sufficient funds to finance the operating costs of the equipment. This limited impact is therefore sustainable.

Score
Relevance to needs Needs Appropriateness Comparative Advantage Efficiency Effectiveness Impact Sustainability 5 5 4 3 2 2 1

1=poor; 3=satisfactory; 5=excellent KEY LESSONS LEARNED The implementation of a masterplan requires a strong organisation, with clear responsibilities, accepted by all other organisations involved in monitoring. The responsibilities should be clearly determined before progressing to investments and pilot projects. The project has been conducted in a constantly changing political environment. Stronger political commitment, guaranteed for a longer period, is required, in a form, which can be sustained through changes in government. However, at the time the project was contracted, there appeared to be good political commitment to reforming environmental management. It was reasonable to hope for some continuity in this commitment. In theory, it would have been possible to design the
186

project so that the second part (preparation of investments) was dependent on the implementation of institutional changes. However, in practice, it was probably useful and necessary to illustrate the strategy with the investment requirements, in order to promote political commitment. The project partner was the MoE. The ownership of a project should have been spread to all Ministries involved and to all public and commercial stakeholders in the sector. Part of the project consisted of the gathering of information by the consultant. The MoE should have been able to provide this information, though this may have required some institutional strengthening. There are many complementary activities, which need to take place in Romania to meet accession requirements. Much new legislation is required, particularly in solid and hazardous waste management. Public awareness is low and progress is required on financing strategies. Some immediate progress can be made on these activities, but substantial progress will be dependent on implementing the reforms necessary for successful monitoring. In supporting this strategy project, Phare had a comparative advantage because most bilateral donors intervene in specific cities or provide training. Also the project required a level of grant funding which would have been difficult to obtain from other donors.

187

SLOVAK ENERGY AUDIT

( 1993-96)

0.35 MEURO

INTRODUCTION This is a Phare 1992 project which required the preparation of energy audits and preinvestment studies for six companies, one university and one medical hospital in the Slovak Republic. For each audit, local and foreign consultants together undertook an audit at the premises of the beneficiary. The consultants estimated energy savings and together with the beneficiary selected energy savings measures varying from low to high cost. For the medium to high cost measures the consultants assisted the beneficiary in developing investment proposals in order to raise financing to implement the measures. The investment proposals were to be submitted an Energy Savings Scheme (ESS) to be established as part of the 1993 Phare Programme. The overall objective of the project was to reduce energy use by high energy users through demonstration audits and promoting investment in energy efficiency measures. The specific objectives of the project were the following: implementation of low cost energy saving measures; investment proposals to be considered by an Energy Saving Scheme (to be established in the following years programme). There was also an implicit objective to promote transfer of skills from foreign to local consultants and beneficiaries on audit techniques and building awareness of potential energy saving measures. PERFORMANCE Relevance to Needs (Score: 5) There was clear need for improving energy saving in Slovakia. There was also a particular need to improve auditing techniques because whilst audits were carried out they were of poor quality and did not meet international standards. New techniques in auditing were needed to be demonstrated. The preparation of investment studies from the audits was also necessary. Appropriateness of Project Type (Score: 2) The concept to provide audits to identify areas for energy saving followed by assistance in securing financing for energy saving investments is a valuable combination for the beneficiaries. However, the timing of trying to promote investment is questionable as financing is still expensive and to-date there has been limited success in securing financing for the identified energy saving measures. Comparative Advantage (Score: 3) Phares comparative advantage is that it can contract relatively small value projects. This has allowed a host of small enterprises, too small to be of interest to other IFIs, to benefit from international assistance.

188

Efficiency (Score: 4) The project outputs were the audits and investment proposals. There were no major delays in the completion of all eight audits and investment proposal reports. Generally, the quality of the audits was regarded highly and co-operation between the beneficiaries and the consultants was good. However Phare PMUs management was limited leading to a lack of information being provided to consultants on the establishment of the Energy Saving Scheme and its requirements for financial proposals. As such the financial proposals were not satisfactory as they did not cover the proposal requirements set by the ESS. Effectiveness (Score: 3) Companies, where possible, implemented low cost measures. However, the medium to high cost measures which need investment and the consultants helped write financing proposals to obtain financing from the ESS have not been financed as planned. Initially the ESS was not in operation when the project was completed. Furthermore, when the ESS was finally established, it was found that most of the investment proposals did not meet ESS financing requirements. Where financing criteria were met the interest rate charged by the ESS (7-8 %) was considered too high by the companies. Although some companies are now preparing to submit revised proposals to the ESS. Overall, the effectiveness of the project was too dependent on the establishment of the ESS and its approval of investment proposals, and was limited when this process failed. In all but one audit there was skills transfer from the foreign to local consultants. In one audit the contribution by foreign consultants was limited. Slovak companies are now capable of providing good energy audits. It appears that the Phare audit-projects with their aims of transferring skills have contributed to this. They are considered to have complemented the training programmes. Impact (Score: 1) Until today, most of the companies and institutions audited have been unable to implement medium and high cost energy saving measures. They have not received funding from the ESS or alternative financing sources to facilitate implementation. All the beneficiary companies still have the audit recommendations and intend to implement them. It is likely, therefore, that when (or if) funding becomes available, the projects will achieve their specific and overall objectives. To this end, some companies are looking for foreign joint ventures which will allow them to finance such investments. Sustainability (Score: na) The sustainability of the measures cannot be determined as they have not yet been funded. The sustainability of auditing services in Slovakia now appears high. Auditing will now be required under the new energy law, and Slovak companies are better/well equipped to undertake the audits. In fact, the Energy Agency is providing training in auditing to local

189

companies, geared to increasing energy efficiency32. Score


Efficiency Effectiveness Impact Sustainability Relevance to needs 4 3 1 na 5 2 3

Needs Appropriateness Comparative Advantage

1=poor; 3=satisfactory; 5=excellent KEY LESSONS LEARNED The success (effectiveness) of one Phare project should not be made fully dependent on the success of another future Phare - or international donor - project. Success of a project is very much dependent on external factors (that Phare cannot control). Hence, each project should be able to stand on its own. Energy saving audit projects should be accompanied by investment proposals that are ready to be submitted to targeted financial institutions. Where audits are supposed to result in investment Phare should consider risk and cost issues of the investments and whether investments are possible in the sector (for company and lender) before initiating and investment preparation project. In addition, if risks have been assessed and it is decided to go ahead with an investment preparation project, Phare should ensure that individuals at sufficiently high (management) level are involved from the beginning. This will ensure not only that a sense of commitment and ownership will be established, but also that it will be at sufficient level. If management does not want to co-operate, than Phare should consider implementing the project for another company where management is more committed. The combination of practical audits and a theoretical training seems to be effective. Results of practical audits should be shared with other companies. They do institute a form of demonstration (even though there are no physical investments), and information on measures can be spread through means of leaflets and flyers.

32

It has already trained about 180 auditors. 190

SLOVAKIA ENERGY DATABASE

(92-94) 0.05 MEURO

INTRODUCTION The project was identified in 1993 as a follow-up to a project in the 1991 Phare Programme. The 1991 project reported in January 1995, and led to the establishment of an energy database at the Power Research Institution (EGU), which was then part of the Ministry of Economy (MoE) and was privatised in 1997. The 1994 project was intended to elaborate the database and make proposals for a more complete information system. Project tendering took place during the second half of 1996. The French company Enerdata, which had completed the 1991 project together with EGU, was selected to implement the project. The Final Report was produced in July 1997. The report assesses the current data availability, and makes a comparison with the reporting requirements of the IEA and OECD to identify the additional needs of a data base. It identifies three key gaps (final energy consumption by branch, household heat consumption and non-energy use of oil products) and provides recommendations on the improvement of annual and monthly surveys and on publications. The report outlines various options for managing energy statistics. The report makes observations about the methods normally adopted in the EU, but does not provide firm recommendations. Finally, the report provides advice on software, hardware, and related costs. No concrete measures have yet been taken to set up an information system but there are currently active discussions on institutional arrangement for this. The specific objective of the project was for the government to take actions to improve the energy data base and information system. The overall objective of the project is that the implementation by government of changes stimulated by the project, leading to improved energy statistics. PERFORMANCE Relevance to Needs (Score: ) Since the formation of the country in late 1992, Slovakia has faced the difficult task of building a market economy and, at the same time, establishing the institutions of an independent state. Political instability and lack of a broad social consensus on the path of economic reform have presented significant obstacles to the transformation process. After the division of Czechoslovakia in 1992, the Czech Republic inherited the federal statistical office in Prague. Slovakia did have a Statistical Office (SO), but data gathering and processing capacity needed strengthening. EGU has been running the energy database set up under the 1991 project. Although the design of the database was compatible with international standards, the data collection done by the SO was still taking place under old systems, leading to some important weaknesses. Some institutional rationalisation was also needed, to maximise efficiency, and there was a need to include energy efficiency in the database. The need for reform of energy data management is stressed by the 1993 and 1997 Energy Strategy. Thus, there is little doubt about the relevance of the project objectives to
191

national needs. Appropriateness of Project Type (Score: ) The provision of technical assistance was appropriate for the objectives of the project. Comparative Advantage (Score: ) The provision of technical assistance for activities which have little commercial return is an area of comparative advantage for Phare. Furthermore, the fact that there were no critical deadlines to meet also supported the view that Phare had a comparative advantage in supporting this project. Efficiency (Score: 2) The budget for this project was 49,500 ECU. Foreign participation was necessary, to bring experience on arrangements in other countries, but some work could have been done by local consultants, which would have been more cost effective. In particular, a local partner could have done much of the liasing with local institutions, both to resolve the minor logistical problems which occurred, but also, more importantly, to try to promote more involvement and ownership from all the organisations involved, some of whom (notably the SO) did not feel involved. With a relatively small additional budget, more training and some hardware could have been provided, which would have helped generate interest and ownership in the project. However, without agreed institutional responsibilities, there would have been a strong risk that this would be wasted by being allocated to the wrong institution. Effectiveness (Score: 2) There have, as yet, been no moves to adopt any of the options discussed in the report, though there is some evidence that government is continuing with discussions. Co-operation and consultation between the stakeholders (MoE, EA, SO, and EGU) was lacking. The study would have provided an ideal opportunity to encourage collaboration through working jointly on a project important to several interest groups. The Commission should have had a greater role in encouraging this collaboration. The MoE did not treat the project as not a priority, especially after the one official interested in the project left the Ministry. However, the Energy Authority (EA) has been interested in the project and has used it to support it's plans to take over responsibility for data gathering and analysis. The SO and EGU appear not to have been involved and it is not clear if they will resist losing activities. No plans have yet been made to fund the additional activities which would be required. Impact (Score: 3) No impact has yet been established, though the study has helped bring forward discussions on reorganisation. One of the causes of delayed impact is that the TOR and the study failed to involve all stakeholders. The study should have attempted to foster a forum within which

192

implementation could be managed. Sustainability (Score: na) With no impact yet achieved, it is unclear whether there will be any sustainability. Some training will be required to ensure a critical mass of individuals are able to manage the database and information system, once the institutional arrangements are agreed.

Score
Relevance to needs Needs Appropriateness Comparative Advantage 5 5 3 2 2 3 na

Efficiency Effectiveness Impact Sustainability

1=poor; 3=satisfactory; 5=excellent KEY LESSONS LEARNED An energy database is fundamental, and its set-up and implementation can be a good project for Phare to finance since it is well-defined. It is unclear whether the additional provision of hardware would have helped the project, but it is likely that the possibility of future provision of hardware could act as an incentive to resolve institutional reforms. Phare annual programming and subsidiarity makes conditional phasing of projects difficult. There is an implicit conditionality in annual programming, arising from the fact that governments are aware that follow-up projects will be dependent on their response. It should be possible for the EC delegation to exploit this implicit conditionality where necessary, without prejudicing principles of subsidiarity. Local institutions should normally be given a significant role in most studies, including the design of TOR and implementation of the project. All institutions affected must be involved, to try to maximise ownership. Phare and the EC delegation can play an important role in fostering collaboration between government bodies. It is important that studies are funded by organisations, which can participate in the technical specification and management of the work. In the above case, Phare showed little ability to provide this sort of technical engagement and this contributed to the rather weak response from government.

193

SLOVAKIA POLLUTION MONITORING (95-10/98)

0.29 MEURO

INTRODUCTION The project represents approximately 35% of the budget of the 1995 environment sector programme in Slovakia, and was based on a first air pollution project implemented under the 1993 Phare programmes. The project was selected from two options for supporting air pollution because its scope and objectives were clearly defined and it was felt that the Ministry of Environment was more likely to fund the other project. The project consists of two components: firstly, TA for the development of a National Emission Inventory System (NEIS), integrating two existing inventory systems, each operated for different purposes by the Slovak Hydrometeorological Institute (SHMI); and, secondly, some limited supplies of computer hardware to operate the system. The project produced a functioning new central database and related software, along with new working practices and manual to assist in the collection, analysis and presentation of information. The project was implemented from July 1997 to October 1998, mainly by a Slovak project team in co-operation with international consultants. The project was delayed by a total of about 3 months, for several reasons; firstly, there were some problems optimising the consultants team; secondly, there was a need to adjust to evolving national legislation; and, thirdly, the beneficiary was more intensively involved than had been foreseen. The specific objectives of the project were the implementation and use of the new systems. The overall objectives of the projects are increased efficiency and reliability of data collection. PERFORMANCE Relevance to Needs (Score: 5) Air Pollution is one of the priority environmental concerns in the Slovak Republic. The 1995 National Environmental Policy and the National Environmental Action Plan both list air protection among the main priorities in Slovakia. During the past three years, there has been substantial progress in adopting new legislation to implement pollution control, and Slovakia is now one of the more advanced Phare countries in meeting the pollution legislation requirements. The Ministry of Environment identified the development of the national air quality and emission inventory systems as crucial to support the substantial efforts being undertaken to develop the legal and policy instruments for air pollution control and to increase the reliability and efficiency of air pollution control measures. The project was well-suited to meet the needs, and complemented other investments to support the implementation of pollution monitoring. Appropriateness of Project Type (Score: 5) The provision of a combination of local and external technical assistance to help with the creation of a database was relevant to the needs of the sector and complemented the other
194

activities in the sector well. Comparative Advantage (Score 5) There seem to have been no substantial problems with the choice of Phare as the source of funds for the project. The ability to use Phare funds for the relatively large share local of local consultants was important for the success of the project. Efficiency (Score: 5) The outputs were of a high quality and comprehensive. The TOR were well designed, but, in view of the technical nature of the work, some refinement was required during inception, which involved some minor delays. The software was based on standard MS Office applications and is therefore highly accessible to beneficiary staff, and can be adapted very easily. This has led to a strong sense of ownership by the beneficiary. Cost effectiveness was high as about 90% of the inputs were provided by local consultants and the beneficiary also devoted substantial staff resources at no cost to the project. However, there is a belief in Slovakia that international inputs, which used 30% of the budget, could have been reduced still further, thus improving cost effectiveness. It is difficult to judge whether this is indeed the case. The cost effectiveness of the hardware procurement is demonstrated by the fact that 6 more computers were supplied than were originally envisaged in the budget. Project management has been rather efficient. Some minor inefficiencies occurred as a result of communication problems, particularly between the Slovak sub-contractor and the international consulting firm. The changes in legislation halfway through the project should have been foreseen by the Ministry and the delays at this stage could then have been avoided. The Framework Contract in force for Slovakia does not seem to have worked efficiently for the project, as the framework roster of experts did not include the specialists required for the project, and the framework thus led to delays and costs with little advantage to the project. Effectiveness (Score: 3) Conclusions concerning effectiveness of the project can only be preliminary as the project has just been finalised. Those locations where the system has been tested seem enthusiastic about using the system. However, it is still too early for the new system to be formally adopted, and full implementation in all regions requires further investment in hardware and a major training effort. The government has not yet produce a financial plan for the implementation of the new system for emission inventories and fee calculations. However, during project implementation, maximum efforts have been made to involve the Ministry of Interior and Regional Environment Offices in the development of the new system, which guarantees a maximum level of understanding and appreciation of the new system. In addition to the SHMI, both the Ministry of Interior and the Regional Environment Offices seem to be convinced of the need for the new system and they expect to put substantial efforts in convincing the government to fund implementation. Unfortunately, these efforts take place at a time of strong pressure on government finances, which will make securing additional resources difficult.

195

Impact (Score: 3) Overall Objective As the project was finalised in October 1998, this evaluation can only provide some predictions for the impact of the project. If the system is implemented, cost efficiency should be assured through the rationalisation of the two old data collection systems, as well as through the automation of tasks and a reduction in bureaucracy. The reliability of the air emission inventory on national level is expected to increase, particularly as more point sources can be covered with the new system, and the system should ensure more efficient and transparent calculation of fees. Finally, the project should improve public access to data and should ensure compatibility with international standards, including those in CORINAIR. Sustainability (Score: 3) The sustainability of the project seems to depend on two main factors: the commitment of the main beneficiaries to implement the project and the financial viability of the beneficiary institutions. The commitment of all direct beneficiaries, but particularly the SHMI, is valued as very high and this provides the guarantee that key persons within the beneficiary institutions will put the project forward. The financial sustainability of the beneficiaries is less secured, though. The Environment Offices of the State Administration are also vulnerable as they depend for their funding on the State Administration. Although there is no reason to believe that the Environment Offices will not continue to exist in their present form, their financial situation is relatively insecure. Score
Relevance to needs Needs Appropriateness Comparative Advantage Efficiency Effectiveness Impact Sustainability 5 5 5 5 3 3 3

1=poor; 3=satisfactory; 5=excellent KEY LESSONS LEARNED The project seems to have been developed in a time when the sector was in movement and when developments in legislation and policy were taking place. The present project supports further developments in the sector and is, if further implemented, expected to improve the efficiency and effectiveness of the system of monitoring air emissions on national level and of the system which implements the economic instrument of air pollution fees. Although the sector of air pollution seems relatively well developed compared to some other sectors, substantial efforts are still necessary to improve air pollution control. The system of calculating air pollution fees may improve through the Phare project, but this will only have an impact on the sector if pollution fees form a real incentive for air pollution control The same is valid for air fines which are charged when air emission limits are exceeded. The effectiveness of enforcement of air pollution legislation needs to be assessed and, where necessary, improved.
196

It can be expected that the process of EU approximation will stimulate the effectiveness further improvements in air pollution control. The project under evaluation will play a role in assessing this effectiveness. Future Phare projects in this sector could play a important role in increasing awareness particularly in relation to implementation and enforcement of legislation. This will enable the relevant parties to assess the effectiveness of current legislation and policy Selected Non-Attainment Areas is currently being prepared under the Phare 1998 Framework Directive and the EC Large Combustion Plant Directive. This project could project under evaluation and could contribute substantially to meeting sector needs.

197

SLOVENIA ENERGY EFFICIENCY STRATEGY

(93-6/96)

0.57 MEURO

INTRODUCTION The project was designed to provide the foundation for a comprehensive national energy conservation programme in Slovenia. The project was undertaken by ETSU, starting in April 1994 and finishing by July 1996, with the production of the final report. The government was fully engaged in the project and provided guidance and assistance throughout the project and detailed comments on reports. This participation was based around a core group of individuals. There were substantial delays, arising initially from data gathering problems and later from delays in providing comments and in ETSU's response to the detailed comments. Task 1 was to develop a National Energy Survey, based on secondary sources, to be used by other tasks. Task 2, to identify constraints and solutions to energy efficiency, could have been better focused on practical issues, such as identifying the most effective target groups for public awareness campaigns. Task 3 produced useful recommendations for the AAEU and its relationship with the Ministry and the private sector. Task 4 produced a useful review of the current laws and regulations in Slovenia and compared them to the options used elsewhere in EU countries, which were used in the subsequent preparation of legislation. Resources for task 5, on pricing and taxation policy, were reduced during implementation, to reflect lack of clear policy support. Task 6 provided a review of information and awareness activities and of financial instruments, with recommendations for Government sponsored activities. Task 7 produced a database of energy conservation information, but was not institutionalised. Task 8 was effectively the concluding activity of the report, summarising the findings of the other tasks and the recommended energy conservation programmes for implementation by the Agency. This has formed the basis of Government activities in energy conservation to be implemented by the Agency. The outputs additionally included: on-the-job training; two workshops; a Study Tour. There were four specific objectives for the project: guiding the development of AAEU; assisting AAEU in implementing the energy efficiency strategy; changing regulations covering energy efficiency; and changing energy pricing and taxation. The overall objective was the acceptance by consumers of programmes promoted by the Agency. PERFORMANCE Relevance of Needs (Score: 5) In common with many CEEC countries, Slovenia has a high energy intensity and is committed to reducing this, both for accession to the EU and through participation in international agreements. The project TOR were drawn up during 1993, prior to the preparation of a Strategy for Efficient Energy Use and Supply in Slovenia, and a draft Energy
198

was modified to providing assistance in the elaboration of the new strategy for implementation by the newly formed Agency. Use (AEEU) and provided useful assistance to AEEU. There were, however, some problems. The support for pricing and taxation policy was generally relevant but the directly adopted. Also, with hind-sight, the support for an energy conservation database was less relevant than other activities because the Government adopted common IT database developed as part of the project. Appropriateness of Project Type (Score: 5) of energy efficiency strategy was relevant to the requirements at the time. The study

Comparative Advantage (Score: 3) of democratic government when the broad experience of EU countries was of particular

Efficiency (Score: 5) The total size of the project was about right for the importance of the requirements, though the pricing and taxation task could have been further cut, or cancelled experts on data gathering would have been useful, though not at the expense of the EU Management. which the Slovenians placed on the outputs of the project. Project management structures revision in TOR to reflect evolving conditions in Slovenia. This was influenced by the The only management problem appears to have been some disagreement between Quality. consistent with the Terms of Reference, with the exception of the task 5, mentioned above.

Effectiveness (Score: 5) development of AAEU; this was largely achieved, though some recommendations are still assist AAEU in implementing the energy efficiency strategy; this objective was largely

199

achieved and AAEU has followed closely the project recommendations. Thirdly, the project aimed to change to regulations covering energy efficiency, and has succeeded in that most recommendations are now included in the proposed Energy Law. Finally, the project objectives of changing to energy pricing and taxation was not effective. The project covered a broad range of activities, intended to provide complementary advances. The main constraint to effectiveness was the lack of progress on pricing and taxation policy, and on metering systems for household energy. Impact (Score: 3) The substantial operations of AAEU suggest that some success has been achieved in the overall objective of generating acceptance of AAEU activities. The ultimate objective of the project is to reduce energy intensity and, in so doing, preserve energy resources and reduce environmental pollution. It is impossible to measure project impact on this objective, but economic statistics suggest that energy intensity is now declining. Sustainability (Score: 3) The project made a significant contribution to capacity building in Slovenia, both through the involvement of experts and government officials and through the institutional capacity of AAEU. Score
Relevance to needs Needs Appropriateness Comparative Advantage Efficiency Effectiveness Impact Sustainability 5 5 3 5 5 3 5

1=poor; 3=satisfactory; 5=excellent KEY LESSONS LEARNED The problems relating to pricing policy suggest that the project design should take account of the powers and responsibilities of the institutions, which will implement the recommendations. Projects should be designed either for one institution or with the active involvement of all those institutions involved. It can often be easier for EU consultants funded by Phare to provide action-oriented recommendations, rather than high level policy advice, since they are unlikely to be sensitive to the national political and institutional sensitivities. The project provides a good example of how flexible management is required, especially for Phare projects, where implementation often takes place several years after the original TOR are prepared. One of the reasons for the effectiveness of the project was the commitment and involvement of the Slovenian officials to the project. This was partly because the project recommendations were being produced at exactly the right stage in the establishment of the AAEU.

200

Czech Republic There have been four Phare Energy Programmes in 1991, 1992, 1993 and 1997, with a total development and energy conservation, including support for the establishment of an Energy Saving Fund. There were also minor studies in clean coal utilisation, regional these programmes, partly arising from the division of Czechoslovakia. The 1997 programme contained one small project to support the establishment of a regulatory body. Estonia The support for conventional energy in Estonia consisted of five programmes, with modest support the establishment of the PIU, and to fund a national energy database. The 1994 efficiency work, and the 1996 programme consolidated this.

Phare support for Energy in Hungary consisted of four programmes between 1991 and awareness and management support. The next programme in 1994 concentrated on public focused on support for investment preparation in firms and municipalities, and in programme was originally designed to support gas storage and international electricity amongst donors, and the funds were switched to boost the resources of the Energy Saving Latvia The first project in 1993 funded the establishment of a PIU including a long-term adviser. of an Energy Fund, harmonisation of legislation; and a commercialisation study. The 1996 with various projects to consolidate existing activities.

Phare Support for conventional energy in Lithuania consisted of eight programmes between capita. The 1991 and 1992 programmes focused on national energy strategies, particularly restructuring and market reforms, whilst the 1994 programme introduced a focus on energy
201

efficiency. The 1995 and 1996 programmes contained a wide range of projects to consolidate existing activities. In addition, there were special projects in 1995 and 1997 to fund part of a gas pipeline and gas storage facility. Poland Support for energy in Poland consisted of four programmes between 1991 and 1994. With a total budget of 8 MECU, it was the smallest programme per capita. All the programmes provided support for strategic energy planning, initially to central government and subsequently to relevant agencies. Energy sector restructuring was introduced in 1992 and was the focus of the 1993 and 1994 programmes. Slovakia The Slovakian energy programmes took place between 1991 and 1994 and had a total budget of 11 MECU. The 1991 and 1992 programmes were initiated in Czechoslovakia and then split, whilst the programmes in 1993 and 1994 were part of Infrastructure Programmes. About 80% of funds have been devoted to energy efficiency, including an Energy Saving Scheme funded in the later programmes. Almost all the programmes included support for studies of energy development in different regions. In addition, programmes have provided ad-hoc assistance in institutional strengthening and training. Slovenia There have been four energy programmes since 1993, with a budget of 7.3 MECU. It was decided not to have a programme in 1996, as a result of the slow progress with the 1995 programme. The 1993 programme focused on support for energy efficiency institutions and strategies, and for laws governing energy supplies. In 1994, support continued to focus on energy efficiency, including investment preparation in industry. The 1994 programme also funded a study for an Energy Saving Fund, which then received nearly half the funding under the 1995 programme. The remainder of the 1995 programme was divided between renewable energy and legal approximation, and this support was continued in the 1997 programme. Romania Romanian conventional energy support consisted of 4 programmes in 1991, 1992 and 1993 and 1995, totalling 41 MECU. Unlike other countries, the 1991 programme started with support for the restructuring of energy institutions, without preparatory support for strategy development. This was supplemented in 1992 by modest support for strengthening energy institutions. However, the main focus of the 1992 programme was on studies and demonstration projects for energy efficiency. The 1993 programme continued the pattern of supporting institutional strengthening and energy efficiency. A coal restructuring project was included in 1993, and a follow-up project funded in the 1995 programme. Half the 1995 programme was devoted to investment to rehabilitate energy supply facilities. The 1995 programme also provided further institutional strengthening in energy efficiency, although an attempt to establish an energy efficiency fund was abandoned. Bulgaria Phare support for conventional energy in Bulgaria consisted of eight programmes between 1991 to 1996. A total of 71.8 MEURO was budgeted, making it the largest Phare energy

202

electricity or coal to maintain energy supplies. Of the other four, the first in 1991 contained a wide range of studies and the next in 1993 was mainly concerned with demonstrations for energy supply, and this was followed in 1995 by support for harmonisation of energy policy. Lack of progress with energy sector reforms has led to delays in approving further

History of Phare National Environment Programmes Bulgaria effective enforcement system to complement the fines and other incentives, which have existed since the 1980s. The first programme purchased equipment for air monitoring and existing legal system for pollution control but lacked the capacity to monitor pollution monitoring activities and also had a few projects relating to ecotourism and public industrial waste. The 1994 programme funded several demonstration projects. There have Czech Republic demand driven and was mainly used for providing monitoring environmental equipment. delay until 1996 until the next Programme (due to disbursement problems with the 1990/91 and cleaner technology for industry, although these have not yet been committed. Projects process of environmental approximation.

The first initiative was to provide a long term environmental advisor. The second step was environmental policy document and set out a framework of priorities. Attention was then Monitoring Masterplan and subsequent funding of lab equipment. There was then an water sector. These started with a regional water management strategy which was followed improving network rehabilitation (building skills and providing equipment).

The first set of projects in the early 1990s covered a wide range of activities (training, conservation). Throughout the mid and late 1990s the majority of support was for

203

environmental investment, primarily for municipal water supply and wastewater and partly for district heating. The remainder provided technical assistance for the implementation of accession-based environmental policies. Part of the 95 programme channelled investments for SMEs. Latvia The first initiative was to provide a long term environmental advisor. There were also two preparation projects for municipal water supply and wastewater investments. The 93 and 94 programmes targeted assistance in wastewater treatment investments, both for municipalities and small towns. This turned into a national programme to improve small town water services. Subsequent projects focused on institutional building - management for the on-going water investments and the establishment of an environmental investment fund. Projects in the 97 programme are about to commence and assist on-going projects (fund and water programme). Lithuania The first initiative was to provide a long term environmental advisor. In the mid-1990s funding was for general assistance to the Ministry of Environment, water and waste utilities as well as some clean-up projects. Later projects (95 -97) were aimed at institution building e.g. (establishing an environmental investment fund, a monitoring database and regulations for protected areas. In 1997, there was a project to develop legislation to further the accession process. The remaining budgets (95-onwards) have been focused on investing in a few large wastewater treatment facilities. Poland The first programme focused on clean up in all the priority sectors: air pollution, water protection, waste treatment and nature conservation. The next phase of projects was broad ranging, focusing on institutional strengthening (of PMU, Ministry of Environment and state monitoring system). The Programmes between 1992 and 1994 focused on forestry projects (which included a responsive forest pest campaign). The final programme in 1995 invested in environmental infrastructure (equipment for air emission abatement, wastewater treatment and waste treatment). Romania There were only three environment programmes 1991, 1993 and 1996. The first project were to identify priority solutions and measures for specific problems. Some TA and monitoring equipment were also provided. The 1992 Programme had three themes: improving the state monitoring and enforcement system through the provision of equipment and a long term expert (to assist with procurement); institutional strengthening of the Ministry; and monitoring environmental pollution in a poor quality region. The 1996 programme was cofinanced with EBRD to invest in the upgrading of wastewater and water supply infrastructure in a heavy mining region. Slovakia Programming commenced in 1990 in the Czech and Slovak Federal Republic. It was demand driven and provided assistance for environmental hot spots. In 1992, the programmes were split between Czech and Slovakia in 1992. The subsequent projects in the

204

1992 programme focused on critical issues (water and health) but made a shift to more longterm developments via assistance in education and awareness raising activities. The 1993 Programme focused on the state of air and water quality with a large component on monitoring equipment. Subsequent projects under the 94 and 95 Programmes funded institutional strengthening projects. This continued addressing air quality monitoring and started assistance in the waste sector. There have been no more programmes since 1995 although a new allocation is expected under the 1998 Programme. Slovenia The first programme was not until 1997. This is because Slovenia had a limited national Phare contribution and allocated the funds to other sectors. It addressed some environmental issues through cross-border programmes. The 97 programme was all put into an existing environmental fund (both TA and capital).

205

APPENDIX C: TERMS OF REFERENCE

EVALUATION OF PHARE-FINANCED ENERGY AND ENVIRONMENT PROGRAMMES


1. Coverage of the evaluation

This study evaluates all Phare-financed programmes in the domains of energy (energy production and distribution, rational energy policies) and environment (soil, water and air quality, solid, water and hazardous waste, degradation of the ecosystems) in the candidate member states (Estonia, Latvia, Lithuania, Poland, Hungary, Czech Republic, Slovakia, Slovenia, Romania and Bulgaria)33, both under national and in multi-country programmes. Altogether they account for approximately 750 MEURO (see annex A). The evaluation does not cover the nuclear energy sector and energy projects financed under the Cross-Border Co-operation (CBC) programmes. 2. Background

In the domain of energy supply, and despite several years of transition, the CEECs are still facing a number of common difficulties in their energy sectors, including high energy subsidies, high global energy intensity and weak environmental performance, low productivity an unreliable supplies with heavy dependency on a single energy supplier and bottlenecks in trans-border energy networks. From the start, the Phare programme has provided assistance to overcome these problems, both through national and multi-country programmes. Whereas Phare initially focused exclusively on technical assistance, it moved progressively towards direct as well as co-financed investments. These have largely focused on extending credit lines to banks in order to provide energy efficiency loans to SMEs. Furthermore, Phare has moved away from an initial ad-hoc approach on a country-by-country basis towards a more coherent strategy in line with the requirements of accession and the Acquis Communautaire. The CEECs environmental problems are among the worst in Europe. Environmental concerns include soil, air and water quality (especially waste water and water supply); solid and hazardous waste and the degradation of ecosystems. These are compounded by inadequate policies and institutions, inadequate public awareness about environmental issues and scarce (private and public) capital for environmental investments. Basic environmental policy reforms were already taking place by 1991. In 1992, this was complemented by the development of national environment funds in Poland and Hungary. This fund initiative was then replicated in the Czech Republic and then other CEEC countries. Phares environment strategies is now also evolving towards a pre-accession phase, whereby CEECs are to begin a process of review and alignment of their legislation
For Bulgaria and Slovakia, the evaluation will take into account the results of on-going country-level evaluations
33

206

with the EU Acquis. Furthermore, the Phare re-orientations provide for a very substantial shift towards investments to resolve problems in meeting these environmental standards. The move towards more Phare spending on investment is now enshrined in the New Orientations for the Phare programme, with 70% of Phares budgetary envelope being allocated to investments in various sectors related to the Acquis, including in E&E. For this reason, this evaluation is timely. The energy and environment sectors are put together in a single evaluation because of the strong linkages and overlaps between both in the Phare programme. Energy programmes have strong environmental linkages and the reforms of energy utilities also envisage improved environmental performance. Likewise, improved environmental regulations cannot be effective without reforms in the energy sector. Water supply issues are included because of their strong connection to environmental issues. 3. Objectives of the evaluation

The wider objective of this evaluation is to contribute to an improvement in the performance of Phare-financed energy and environmental (E&E) programmes. The immediate objective is to provide inputs into the preparation of future E&E programmes, in the context of the re-orientation of the Phare programme proposed by the Commission; The output of this evaluation will be a report covering the following issues: (a) an overview of the state of E&E policy, the performance of major utilities and the remaining obstacles to convergence with EU energy and environment Acquis; (b) an inventory of outputs of Phare E&E programmes ; (c) an in-depth analysis of the performance of Phare programmes in these sectors ; (d) recommendations on Phares future institution building and investment activities in E&E in the context of the New Phare Guidelines. The objectives are expected to be achieved through feeding the recommendations back into the Phare programming cycle. For this purpose, special attention will be paid to the dissemination of the evaluation report, both at headquarters level and in the field, both among programme managers and a wider public of decision makers and interested parties. 4. Evaluation criteria

The evaluation will be based on the Logical Framework methodology. It will establish empirically verifiable indicators for each of the criteria discussed below, discuss programme performance as measured by these indicators and elaborate on the underlying assumptions. Annex B can be used as a basis from which to draft a logical framework, by updating the objectives, and drafting a more comprehensive list of activities and indicators. The evaluation team should establish objectively verifiable indicators for each of these criteria and for all programmes and projects to be evaluated. A final list of indicators will be agreed upon between the team and the Evaluation Unit after the first field visits, after a clear picture on types of outputs and availability of information has been assembled. The report should also cover both the programme or policy level as well as the concrete

207

projects carried out under each programme. The evaluation should make a distinction between institutional reform and investment activities in the Phare programmes. This classification is based on an output criterion. Institutional reform activities primarily aim at changing the behaviour of an institution (a company, a government department or other authority). Institutional reform projects mostly use technical assistance inputs only (some equipment may be involved). The outputs of institutional reform activities are trained staff or proposals for policy, regulatory, legal and organisational changes. Investment projects aim to increase/change the stock of physical investments and thus the technical behaviour of a company or organisation. The outputs of investment activities are financial or physical investments. The latter concern mainly Phare activities in energy efficiency and environmental funds34 and in pilot and large-scale investment projects. Since Phare is, so far, predominantly a technical assistance programme, investment activities are rather accessory to technical assistance and usually envisage a demonstration impact. Relevance and design The relevance and design assessment is carried out from three points of view: (a) the specific and wider programme objectives as compared to sector needs and country policies (is this a useful thing to do?), (b) the appropriateness of the intervention strategies (is this the right way to do it?) and (c) the comparative advantage of Phare in achieving the objectives (why should Phare do this?). This assessment is carried out both at programme (overall policies) and at project (concrete actions) level. Relevance also refers to other donors programmes. Efficiency The efficiency criterion assesses the extent to which inputs and activities have effectively produced the expected outputs. Institutional reform programmes yield two types of outputs: training of human resources and proposals for legislative, policy, regulatory and organisational reforms. Measures of cost-efficiency may be applicable to a limited extent only. Emphasis will be put on management efficiency and the quality - rather than quantity and cost - of outputs. Concerning the output of proposals for legal, regulatory and organisational changes, the evaluators will measure to what extent the expected outputs have been delivered by the Phare programmes, the quality of these outputs and their cost. For outputs of trained staff, the evaluators will look into selection criteria, curriculum quality and cost-effectiveness of the training projects. The outputs of environmental and energy awareness activities consist of information for the wider public. Here, selection and targeting, content of the campaigns and managerial and financial indicators may be appropriate. Investment projects yield two types of outputs too: either financial investments (loans and equity financing schemes) or physical investments (often pilot projects, infrastructure and equipment). Here, financial indicators will be more appropriate to measure efficiency: the
The Funds provide credit lines to financial institutions and support the potential beneficiaries in preparing business plans, energy efficiency and environment feasibility studies, selection procedures.
34

208

internal rate of return of the projects, and compared to market rates, where available, in order to estimate the implicit or explicit subsidy element in the financing operation. This will also constitute an important input into the assessment of financial sustainability of the operations. In this respect, the evaluation will examine in particular the operations of the Energy Saving Funds and the Environmental Protection Funds under Phare E&E programmes. The evaluation will include a detailed inventory of all outputs produced by these Funds (credit decisions). This includes an analysis of types of beneficiaries and allocations of financing. Energy saving and environment indicators will be compared for a sample of loans with the initial targets and estimations. Indicators of efficiency will be set up to compare outputs with costs of producing these outputs, including management, technical assistance and other costs born by the donor or by government subsidies. The evaluation will also investigate how the programme management and the institutional set-up affects investment performance and how alternative institutional arrangements might improve this performance. Allocation procedures for scarce credit funds at nonmarket-clearing interest rates will be thoroughly investigated. Effectiveness Effectiveness refers to the extent to which the proposed reforms are effectively being applied and the trained human resources effectively used in practice (i.e. the extent to which the programme outputs have helped to reach the programme specific objectives). The effectiveness of institutional reforms will be analysed at three levels : Sector policy level: The evaluators will provide an assessment of the degree to which policy proposals, legal and institutional changes in E&E, proposed by Phare technical assistants, have effectively been adopted by the competent authorities. Operators: At the level of energy utilities operators and environmental agencies, the evaluation will assess the extent to which institutional and organisational reform proposals have been effectively implemented, and to what extent their trained staff is effectively in a position to utilise the acquired skills and knowledge. For this latter purpose, tracer studies will be undertaken for a sample of trainees. Consumers: Where feasible, the evaluation will try to asses the extent to which E&E awareness and information campaigns have effectively reached the wider public and changed behaviour (impact) in the relevant domains. The effectiveness of investment activities covers both the E&E funds and other direct investments. E&E Funds: The evaluation will assess to what extent financial support from E&E funds has effectively improved environmental standards and/or enhanced energy efficiency in the targeted companies and institutions. Other investments: The effective impact of physical investments in particular utilities or institutions on their economic and technical performance (with respect to international standards or E&E norms) will be examined.

209

Impact The impact criterion measures the extent to which the direct programme objectives have contributed to enhance the overall performance of the institution(s) for which they were intended or the achievement of global E&E standards and policy targets. The institutional impact analysis will be based on internationally agreed indicators as well as indicators defined by the evaluators for the specific purpose of the evaluation. Sector policy level: The evaluation will assess the extent to which effectively adopted policy changes and legal and regulatory outputs have changed performance in the E&E sectors. In the energy sector, impact indicators will concentrate on reliability and diversity of supply, energy efficiency and economic pricing. In the environmental sector, international environmental standards and indicators will be used. Care should be taken to distinguish between the impact of specific Phare programmes and projects on these indicators, and the impact of factors not related to Phare. Operators: The impact of Phare organisational changes and trained human resources on the behaviour of energy utilities and environmental institutions will also be assessed. The impact assessment will be undertaken, if possible, against the initial situation i.e. before the Phare Programme started. The impact assessment of investment activities will involve: For projects that were set up mainly for demonstration purposes, the extent of spill-over to other operators and institutions will be examined. For investment projects without demonstration effects, their impact on the achievement of global policy objectives will be investigated. As far as demonstration effects of E&E Funds are concerned, the evaluators will examine: the extent to which learning effects in financial institutions have lowered transaction costs and lowered institutional obstacles for potential beneficiaries financing of energyefficient or environment-friendly investments and accessing financial markets. the extent to which Phare direct financial support has increased energy and environment investments of SMEs, non profit-oriented organisations and local authorities and improved immediate and long-term profitability of the projects. the demand effect on the suppliers of energy-efficient and environment-friendly equipment and services. Sustainability This criterion assesses the long-term viability of the induced institutional reforms following the withdrawal of the technical or financial assistance programme. Sustainability will be assessed both at sector and at operator/institution level. While most technical assistance activities provide a one-off input, some of these may have included medium-term support operations, the long-term sustainability of which needs to be assessed. For operators that benefited from financial support, the long-run financial and technical viability of the operation will be assessed. As far as Funds are concerned, the evaluators will assess:
210

the sustainability of such a strategy (loan schemes) in the framework of the energy efficiency and environment policies and the financial sustainability of the direct financing set-up, which could be measured, amongst others, by a positive internal rate of return, net of donor or government subsidies ; the sustainability of the demonstration effect on the SME sector and on the banks ; legal and organisational sustainability of institutional structures for direct financing, created or supported by Phare, including arrangements for exit strategies. The issue of ultimate ownership of the revolving Funds and responsibilities of owners will also be addressed. 5. Organisation of the evaluation

The evaluation is expected to start in April 1998. A draft report should be available in October 1998. At the start of the evaluation, a briefing session will be organised for the consultants in Brussels. The consultants can meet with the task managers, obtain some relevant documents and information (most documents will have to be obtained locally) and finalise the work programme and timing. It will also permit to have a first discussion on evaluation methodology with the Evaluation Unit and to define in details the scope of the evaluation in order to avoid potential overlapping with other on-going or foreseen evaluations. Thereafter, a first field visit will take place, in order for the consultants to familiarise with the each of the country programmes as well as with the multi-country programmes. This will permit to make an inventory of projects under the various programmes and assess data availability. Shortly after this field visit, a two-day workshop will be organised for the entire evaluation team (including local consultants), together with the Evaluation Unit, to present the findings of the first field visit, propose and discuss a selection of projects under each programme for further analysis and discuss the details of the evaluation methodology to be applied (including logical framework and data collection methods). This will lead to an inception report for this evaluation, outlining in detail the evaluation methodology, planning and questionnaires and survey techniques to be used in the course of the evaluation, to be approved by the Commission services concerned, before implementation. Thereafter, the second round of field visits can start, with detailed evaluation studies. Following the field work, a draft evaluation report shall be submitted to the Evaluation Unit (DG 1A/F/5) in 50 copies (and on diskette). This report will not exceed 60 pages, excluding annexes. It will include an abstract (1 page), an executive summary ( 5 pages), lessons learned from the evaluation and conclusions and recommendations. The annexes will include, amongst others, the terms of reference, and a summary presentation of all data collected. The Evaluation Unit will ensure distribution of the draft report to the Commission services concerned, SME programme managers and the beneficiary countries authorities, for comments. It will be discussed with the Commission services concerned at a meeting in Brussels, before finalisation. Within one month after receiving the comments, the consultants will prepare a final report, in 2000 copies (plus diskette), in accordance with Phare Information Unit lay-out standards. It will contain an appropriate disclaimer, to be decided upon and provided by the Evaluation Unit.

211

6.

Composition of the team

The composition of the team is largely left to the contractor, but some important elements The team should comprise at least of: 1. one expert in energy policies and regulations, who is familiar with EU policies in this domain and has extensive experience in the Phare countries. Ideally, s/he should also 2. domain and extensive experience in the Phare countries. 3. one expert/economist with very extensive theoretical and practical experience in evaluation methodology, possibly in the role of team leader. the use of local consultants in the beneficiary countries in indispensable for a better to relevant data and information. 7. Budget

For this evaluation, a maximum budget of EURO 500.000 has been allocated from the Phare

212

Annex A : Phare Energy and Environment Multi-Country and National Programmes (*)
BG Energy Environment Energy and Environment Total 91,500,000 44,000,000 5,000,000 8,500,000 61,800,000 29,700,000 CS 9,000,000 35,000,000 5,000,000 CZ ES 5,000,000 3,500,000 HU 17,000,000 59,000,000 15,500,000 91,500,000 4,500,000 16,650,000 114,300,000 34,400,000 6,400,000 4,000,000 134,000,000 LE 900,000 3,600,000 LI 13,150,000 3,500,000 PL 4,300,000 110,000,000 RO 21,000,000 13,400,000 SL 6,400,000 SR 3,000,000 1,000,000 MCP 43,000,000 91,000,000 Total 184,550,000 359,500,000 15,500,000 559,550,000

(*) This table does not include General Technical Assistance Facility programmes, nor the water component of the Infrastructure programmes.

213

ANNEX B: Phare Energy and Environment Programmes 1990-1997 In both cases, the support was in the form of technical assistance and in the form of equipment supply or investments / co-financing. Phare Energy Programme Sector Energy Long-term objectives (1) Long-term objectives (2) : Convergence towards EU Rational behaviour (operator Convergence of CEEC with EU energy and consumers) sector

Problems to be solved Inefficient allocation of scarce resources (too much emphasis on the supply side)

Specific objectives Up-dated energy policy, in accordance with EC guidelines

Activities Set up of demand-side databases Least-cost development studies Development of energy policies, based on realistic demand-side analysis Sector restructuring policy Audit of existing energypricing systems Strategy on energy prices and taxation Development of accountancy and billing systems Definition of a comprehensive energysaving strategy

Aggregated Indicators

Inadequate energy tariffs (enterprises have in fact paying for the households)

Price transparency and regulatory framework

Energy prices real costs

214

High energy intensity

Specific energy efficiency strategy

Weak environmental and safety programmes

Integrated environment and energy

Low productivity and low quality of service by energy companies (monopolies)

Market orientation

Heavy dependency on one energy supplier or one energy resource

Security of supply

Audits related to energysaving potential in different sectors Assistance to the creation of energy agencies or to existing ones Training for energy managers and decision makers Development of financial incentives and tools (energysaving funds complementing commercial schemes) Development of energyefficiency standards Pilot projects Development of consumer awareness and information Development of Energy Saving Funds : soft loans for SMEs Environment impact assessment studies Audits of existing environment regulations and recommendations for future adaptation Environment up-grading strategy for energy companies Co-financing of investments for improvement Development of clean-coal technologies Development of secondary legislation (license scheme) Screening of existing regulation Commercialisation, restructuring and privatisation of energy companies Screening of existing regulation Strategic study on diversification of supplies Strategic study on use of local and renewable energy resources Feasibility studies related to storage or harbour facilities

Energy consumption unit of GDP

per

CO2, SO2, NOx per capita

Import/consumpti on ( )

Interconnection capacity

215

Bottlenecks transborder networks

in

Interconnection European networks

of energy

Interconnection feasibility studies Technical assistance to operators to help them fulfil conditionalities of interconnection Twining programmes Co-financing of equipment or of interconnection infrastructures

TENs

216

Phare Environment Programme Sector Environment Long-term objectives (1) Quality Long-term objectives (2) : Convergence towards EU EU environment legislation and standards

Problems to be solved Human resource and institutional weaknesses

Specific objectives Strengthening / building up of human resources and adequate institutions

Activities Establishment of new institutions Building capacities in existing institutions Decentralisation of institutional capabilities Improving public investment programming Improving project preparation and management skills Information systems Public register and accountability Education programmes Public participation programmes Support to NGOs National environment strategies National environment action plans Legislation for specific areas Environmental standards Enforcement legislation Harmonisation with EU and international standards Developing environmental financing strategies Master plan development Preparing projects and feasibility studies Rehabilitation and maintenance programmes Pilot projects Priority new capital projects Set up of an environmental monitoring system Enforcement of standards and law and fining/penalties system

Aggregated Indicators

Insufficient awareness of environmental issues

Greater awareness and participation

Inadequate policy and regulatory frameworks

Effective policy

environmental

Scarce resources

financial

Priority capital investments and technical assistance

Poor system

monitoring

Effective monitoring and law enforcement

217

S-ar putea să vă placă și