PLANNING AND ORGANISING USEFUL EVALUATIONS

UNHCR INSPECTION AND EVALUATION SERVICE

JANUARY 1998

TABLE OF CONTENTS
DEFINING THE FUNCTION
UNDERSTANDING THE PURPOSE
IDENTIFYING THE EVALUATION NEEDED
PLANNING THE EVALUATION
PREPARING TERMS OF REFERENCE
SELECTING THE EVALUATORS
IDENTIFYING THE METHODOLOGY
SUPERVISING THE EVALUATION
PREPARING A FINAL REPORT
FOLLOWING UP ON EVALUATION FINDINGS
ANNEX : Using evaluative frameworks and questions to
analyze activities.

Introduction
These guidelines are intended to assist UNHCR field and headquarters staff in
evaluating and analysing the broad range of operational activities undertaken by
UNHCR. The guidelines attempt to provide an elementary understanding of the
function, a description of the different approaches that can be considered, as well
as suggestions as to how evaluations should be planned, implemented and
followed up. The methods suggested are based on commonly applied
professional approaches and techniques as well as the experience gained in
supervising and carrying out evaluations in a wide variety of operational
environments.
These guidelines were prepared by Lowell Martin, UNHCR, Evaluation
Co-ordinator.
DEFINING THE FUNCTION
1. Evaluation is often defined and conceptualised in such a broad fashion that the
function takes many forms and is carried out in a variety of ways. Consequently,
evaluations can range from relatively thorough project monitoring that questions
assumptions and assesses whether project objectives have been achieved, to
more rigorous in-depth evaluations designed to comprehensively analyse all
aspects of an activity or operation.
2. In UNHCR, evaluations are often thought of as an examination of the
appropriateness of operational aims and the extent to which they are being
achieved. In most instances, evaluations also examine whether activities are
having the desired impact and are being carried out in an efficient and
cost-effective manner.
3. A commonly employed and somewhat broader definition of evaluation used by
many UN agencies describes the function as a "process which attempts to
determine as systematically and objectively as possible the relevance,
effectiveness and impact of activities in the light of their objectives". Definitions of
evaluation tend to vary somewhat depending on how organisations use the
function. Consequently, it is often equally important to understand what the
function is intended to do.
UNDERSTANDING THE PURPOSE
4. UNHCR evaluations can serve many purposes. The principal aim is to provide
analyses and proposals regarding issues and concerns the organisation is
attempting to address. In most cases, evaluations are intended to serve as a
catalyst for change by furthering understanding and assisting decision-making.
Hence, evaluations often take on both a learning and trouble-shooting character.
Evaluations can, for example, permit programme managers to answer questions
as to whether projects, programmes and activities are achieving their objectives
as well as whether there are better ways of attaining goals.
5. Evaluations are commonly thought of as an action-oriented management tool
and organisational process for improving activities still in progress as well as
future planning. They may also serve many useful functions such as providing an
occasion to bring operational staff or programme partners together to review
activities for which they are jointly responsible. Although evaluations are
sometimes intended as a means of providing analytical information on results that
can be used for control or accountability, they are generally much less successful
in this role.
IDENTIFYING THE EVALUATION NEEDED
6. The type of evaluation required will depend on the purpose and scope of the
evaluation, as well as the time and expertise of those carrying out the work. In
most cases, a relatively simple common sense approach applied objectively and
systematically will meet field managers' needs. Such an approach will usually
provide an approximation of results and causes that is sufficient for a general
understanding of what has transpired and generally adequate for taking decisions.
7. The most common UNHCR evaluation called for is the ad hoc evaluation
carried out during implementation, often before a decision is made to enter a new
phase. Such evaluations are generally requested when there are misgivings
about implementation or the direction the activity or operation is taking.
8. Lessons learned evaluations intended to improve future operations are most
commonly carried out after or during the late stages of an emergency or
repatriation. Lessons learned evaluations may also be organised to analyse new
or innovative approaches. In most instances, they are carried out in response to a
Headquarters' request.
9. A quick and simple self-evaluation should generally be given first consideration
when the need for decision-making information arises. Although more elaborate
evaluations can be considered when circumstances dictate, complex approaches
can quickly become costly and burdensome.
10. Staff who have a general understanding of the activity to be evaluated, as well
as good analytical and listening skills, should be able to carry out a quick and
elementary self-evaluation of most field activities or operations. Such an
evaluation can rely on programme documents and data, as well as the
perspective of those associated with an operation who are in a position to judge
its results. An evaluation of this nature would, for example, seek the views of
those who support or are responsible for activities, including government
personnel, NGO and UNHCR staff, as well as donor officials and project
beneficiaries.
11. Self evaluations have a number of advantages which include :
providing immediate and direct feed-back; G
allowing staff participation in the process which increases their commitment
and sense of responsibility for the evaluation;
G
providing a quick and low cost means of organising an evaluation; and, G
facilitating broad evaluation coverage. G
allowing staff participation in the process which increases their commitment
and sense of responsibility for the evaluation;
G
providing a quick and low cost means of organising an evaluation; and, G
facilitating broad evaluation coverage. G
12. Where questions of objectivity, independence or the need for a fresh
perspective arise, an outside evaluation using a consultant or staff from another
operation is a commonly used approach. Furthermore, if external staff or
consultants are experienced in undertaking evaluations and can be devoted
exclusively to the exercise, it is evident that a more rigorous exercise can be
carried out. In attempting to undertake a large multi-faceted evaluation it is almost
imperative that staff are solely dedicated to the exercise.
13. Using internal staff to the extent they are available not only allows staff an
opportunity to improve their analytical and problem-solving skills, it also permits
them to become familiar with the activities and problems of other parts of the
organisation as well as increases the potential for "cross fertilisation" of
successful ideas, techniques and strategies within UNHCR. Thus, staff
participation in evaluations can provide an important means of personal career
and organisational development.
14. A joint evaluation with another partner, the government or a donor may also
be considered. Joint evaluations can prove useful in developing recommendations
that are acceptable to more than one party or agency. They can also offer the
advantage of providing a broader perspective, more expertise, promoting a better
dialogue and facilitating future collaboration. On the other hand, they can be
cumbersome to organise and schedule and often lead to delays and more costly
evaluations. In addition, the absence of standardised evaluation approaches and
procedures can also lead to both friction and results which are a diluted
compromise that fails to satisfy any of the evaluation partners.
PLANNING THE EVALUATION
15. To be successful, evaluations require thorough planning. Among the tasks
involved are deciding on the goals of the evaluation as well as focusing the
exercise on the issues and activities to be covered. In addition to clarifying the
information needs of decision-makers, a number of fundamental questions should
also be asked at the outset, such as:
Why is the evaluation being undertaken? G
What is it expected to achieve? G
How will the results be used and by whom? G
16. It is crucial that users' needs are identified and effectively integrated into the
planning process. An initial step in planning an evaluation involves identifying the
appropriate decision-makers and determining their information needs. A series of
discussions and consultations that involve decision-makers as well the staff who
implement activities on a day-to-day basis and any other interested parties should
enable those planning the evaluation to gain a sufficient sense of the information
and analysis required as well as finalise the goals of the evaluation.
17. Before embarking on a full-scale evaluation, mull over alternative ways in
which the objectives can be achieved. There may be a simpler approach that can
be used, such as a mission, a meeting, or a small study. The problems, their
causes, and the various solutions may already be clear. Before commencing,
some thought should also be given to the various interests involved in the issue or
activity to be evaluated, the potential cost of the exercise, and the availability of
qualified staff to carry out the evaluation.
Clarifying objectives
18. At a very early stage, planners should also begin discussing the aims of
particular programmes, projects or activities. In many instances there is a lack of
clarity about the precise goals and as a consequence, planners will often have to
collaborate with policy makers and operational staff to better define objectives,
taking into account the original intent of the programme or activity as well as new
thinking. In addition to goals that are ambiguous, unrealistic, or need prioritising,
there may also be competing or conflicting aims that need to be considered along
with the unintended effects of the programme or activity.
Identifying vested interests
19. The various interests in the programme or activity to be evaluated should be
given some consideration. Planners will often find that many parties will be
involved in the use of evaluation results. Their potential reaction to the findings
should be borne in mind and their co-operation sought.
20. In addition to the decision-makers most directly involved, there is often a long
list of persons with an interest in the evaluation such as policy makers, donors,
operational partners, organisations with competing interests, beneficiaries and
various parts of the host government. The extent to which these groups may seek
involvement in the evaluation will not always be clear. Nor will it be evident how
the interest of each are engaged or how they will respond to a particular outcome.
Such aspects should, however, be given some consideration in consultations and
planning.
21. Planners should be sensitive to the interests that are likely to lead to
evaluation results being intentionally misused. Misuse can take many forms, but
the most common types tend to be :
using an evaluation to justify decisions already taken; G
undertaking a study in order to delay action; and, G
organising a partisan evaluation with the aim of advocating or defending
particular positions.
G
22. There are situations in which either evaluations should not be attempted or
special approaches should be considered. For example, this may be the case
when resistance is so great that a valid and reliable study cannot be made.
Finally, if an evaluation is only being undertaken because of bureaucratic or
formalistic reasons, it is likely that the results will be ignored. In such cases, the
evaluation is likely to be a poor use of resources.
23. In conceptualising the assignment, some thought should be given to the
availability of persons to carry out the evaluation. Planners should, however,
avoid the all too frequent tendency to quickly decide on who should carry out the
assignment and the locations to be visited before the work is even
conceptualised.
Considering the feasibility and timing
24. Consideration must be given to the appropriateness and feasibility of an
evaluation. After some reflection planners may decide that for several reasons it is
not the appropriate time to undertake a particular evaluation. The reasons most
frequently cited are that activities are undergoing significant change, or are so
new that an evaluation would be premature.
25. Situational reasons may also be invoked when a postponement is sought. An
emergency or conflict in the area that would lead to staff security risks often can
lead to delays. In such cases, it may be decided that the evaluation should be
rescheduled or that the issues or decisions can be addressed in another manner.
26. It is sometimes suggested that an evaluation should not be undertaken
because data on results is not yet available or because the activities are
undergoing change. These reasons, however, should not necessarily preclude an
evaluation. In general, any project, programme, or activity can be sufficiently
evaluated at some level to adequately serve the purposes of decision-makers.
27. The timing of an evaluation is extremely important. Consequently, the annual
programming cycle is often a crucial consideration in planning. It is essential that
analysis is provided at a time when it can influence decision-making.
28. While it is useful to anticipate and budget for evaluations, in most instances a
degree of flexibility in scheduling is desirable. Problems or changes in the activity
to be evaluated often make precise, pre-established plans and schedules
somewhat unrealistic.
Gaining co-operation
29. Regardless of the focus of the evaluation, it is critical that all concerned
understand and accept from the outset the principal audience and purpose of the
evaluation. To those responsible for the operation being reviewed, an evaluation
can be a sensitive matter, particularly when the evaluation seeks to document and
analyse action taken at all levels with a view to identifying the effectiveness and
impact of operational activities.
30. Staff generally feel somewhat apprehensive in view of their commitment to the
activities they manage and their fear that a superficial or unobjective evolution
could produce unwarranted judgements. The extent to which controversy is
anticipated can be a good indicator of the need to make every effort to involve
those concerned in the evaluation's planning.
31. Successfully carrying out an evaluation is normally dependent upon obtaining
the trust and co-operation of those likely to be affected by it. This includes those
with overall responsibility as well as staff responsible for implementation in the
field. In addition, there are likely to be many other partners who have the capacity
to enhance or frustrate the evaluation. Apprehension can often be reduced and
acceptance facilitated when evaluation and operational personnel sit down
together before the evaluation to discuss goals, build a consensus, and plan the
evaluation approach that will be taken.
Foreseeing follow-up
32. The importance of planning how the evaluation will be followed up can not be
over emphasised. In general, planning is even more important in producing
change than strict follow up procedures which can easily become a mere
formality. In most cases, the key to successful follow-up is simply foreseeing that
the evaluation addressees the immediate problems and issues confronting
decision-makers and then ensuring that the recommendations are provided in a
timely fashion so they can be incorporated into a decision or solution.
33. Finally, it is always wise to try to keep the exercise simple. An in-depth
evaluation may not be required. In general, there is a tendency to take decisions
without the necessary information and analysis when very little additional
knowledge is required. On the other hand, once a decision is made to evaluate,
the exercise frequently becomes overly complex and comprehensive relative to
the information actually required to take the necessary decision. As a
consequence, it is often helpful to continually reassess the need for an evaluation
as the exercise is being conceptualised and planned, as well as useful to look for
alternative approaches that are simpler and more timely.
PREPARING TERMS OF REFERENCE
34. Precise terms of reference are essential for each evaluation in order to reduce
the likelihood of misunderstanding during the study. In most cases, they constitute
the planner's most important tool for linking the evaluation's design with the
decisions to be taken. The terms of reference should provide a formal record of
agreement as to what will be done and should outline the obligations of all
participants in the evaluation. Submitting the terms of reference to management
for their approval may also help in gaining commitment to the evaluation.
35. Detailed terms of reference for a programme evaluation study should
generally include :
the reasons for the evaluation and its objectives; G
a statement of the scope and specific issues to be addressed in the study; G
a detailed workplan of the evaluation indicating what questions are to be
answered and tasks are to be carried out including, if possible, what
information is to be collected and how;
G
the locations to be visited; G
which participants are responsible for various tasks; G
a statement of the expected output; G
a timetable indicating when various tasks will be completed as well as the
due dates and recipients of any periodic reports or outlines; and,
G
a budget indicating the costs associated with the evaluation. G
36. Terms of reference, or an accompanying agreement prepared for an external
consultant who will be undertaking the evaluation, should specify that an initial
report will be submitted in draft and provide for time to make corrections or
changes to the draft once it has been reviewed. It is also useful to stipulate that:
the report will be written in accordance with an accepted outline; any substantive
changes are to be agreed upon; and that the final report and all information
gathered during the evaluation are the property of UNHCR and will not be used
for other purposes. In addition, UNHCR should retain the right to make editorial
revisions. Finally, the agreement should specify that final payment for the
evaluation should be based on submission of an acceptable final report rather
than the end of the time period foreseen to carry out the assignment.
37. Staff organising the evaluation should try to avoid the temptation to limit terms
of reference to a few sketchy generalities such as to examine "the issues" and
"make recommendations for improvement". Poorly planned evaluations often then
delegate the more detailed terms of reference to the consultant.
Consultants are seldom in a position to interpret the aims and issues to be
addressed by the evaluation and the work can too easily follow the interests of the
consultant rather than the needs of the decision-makers.
38. Allowing an external consultant employed to undertake an evaluation to draft
the terms of reference can also indicate a neglect of three important maxims that
should always be followed in the use of consultants. First, never let the consultant
take control. Second, to get something out of a consultancy the staff managing
the assignment must have a clear sense of what they expect from it. Third, the
benefit from the consultancy, is proportionate to the time invested in it by those
responsible for the work.
SELECTING THE EVALUATORS
39. There are advantages to using persons external to the operation to undertake
an evaluation as well as benefits in having those who have been responsible for
implementation carry out an assessment of the operation. In many situations,
however, an outside evaluation is the most practical approach in view of the
pressure operational staff are under to perform their regular duties. If internal staff
are used, it is particularly important to work out a role that ensures a degree of
autonomy from programme management.
40. An outsider with no vested interest in the operation can generally be more
objective than staff directly involved in implementation, but a great deal of time will
also be required in familiarising an outsider with the activities under review. Using
staff skilled in evaluation work also has advantages, but there are approaches
that can be easily understood and carried out by persons with modest expertise, if
they are familiar with UNHCR's general policies and practices and the means the
organisation follows in achieving its aims.
41. In general, personal qualities and substantive experience are more important
factors in the selection of evaluation staff than specific technical or methodological
knowledge. Ideally, personnel engaged for an evaluation should be inquiring,
analytical, thorough, detached, as well as have the ability to think systematically
and rigorously. An evaluator should also have an openness to new ideas and a
sensitivity to the complexity and constraints associated with UNHCR's activities. It
is also important, but sometimes overlooked, that the evaluator has credibility with
the managers who will implement the evaluation recommendations.
42. The need to use outside consultants in order to ensure independence and
objectivity is frequently debated. Generally, a consultant or evaluator from outside
UNHCR is more credible if the intended users of the evaluation are also external,
such as the host government or a donor to the operation. On the other
hand, if the evaluation recommendations and lessons are to be used by UNHCR
managers, they will generally find the conclusions and recommendations more
relevant if they are an internal product. This does not, however, preclude the use
of a consultant who is aware of the organisation's basic purposes and style of
working, particularly one who has been able to establish a relationship of trust and
confidence at some previous time.
43. Regardless of a consultant's evaluation skills, if an individual or team does not
have some experience with refugee operations and a familiarity with the political
contexts common to refugee situations, they will be greatly hampered in carrying
out a comprehensive evaluation. A sectoral consultant can function more
effectively without such experience, but even then there is an increased risk that
recommendations will be made in isolation from the operation's overall aims and
without sufficient regard to the context. Consultants with only limited experience of
refugee operations will also require a time-consuming briefing that can be
burdensome for those called upon to supply the briefing.
44. UNHCR personnel who have not been directly associated with the operation
can often bring the same objectivity as an outsider. Seconding staff from other
parts of the organisation can frequently provide a combination of both experience
and objectivity, particularly if seconded staff are combined with a consultant
perceived as completely independent.
45. Self-evaluation is another approach that can be employed. Self-evaluations
are sometimes viewed as an efficient and inexpensive way of carrying out
evaluations and documenting lessons learned since they enable staff who have
been directly involved with the programme to contribute what they know without
having to filter their observations through an outsider. In most cases,
self-evaluations and lessons learned or problem-solving workshops are
approaches that should be given first consideration.
46. On the other hand, self evaluations often suffer from the tendency of staff
responsible for implementation to be overly critical of those they were dependent
upon for support or implementation, while at the same time placing their own
efforts in the best possible light. Furthermore, those preparing a self-evaluation
may not be sufficiently knowledgeable about all aspects of the operation, to make
an accurate assessment. In the worst of cases, a staff member performing a
self-evaluation may have difficulties with superiors or colleagues if the
self-evaluation is perceived as being critical and thus may be forced to downplay
or suppress any negative results.
IDENTIFYING THE METHODOLOGY
47. Carrying out evaluation studies does not involve routine and repetitive tasks.
The issues addressed vary greatly from evaluation to evaluation, as do the range
of analyses that can be employed in carrying out the actual study. As a
consequence, there can be no detailed step-by-step procedures that are
applicable in evaluating all operations and activities. General principles guide the
work, but an evaluation must be carefully designed for each situation.
48. When selecting evaluation methodology it is generally better to avoid
elaborate and complex designs. Most methodologies nevertheless involve a
variety of methods and techniques in collecting and analysing data that, for
example, include :
a review of plans and policy guidance; G
interviews, group discussions and possibly questionnaires; G
analysis of reports and data; G
on site observations and possibly statistical sampling and inferences; and, G
comparisons and analysis of standards, costs and performance. G
49. Of paramount importance is the need to work systematically and objectively
through an analysis of issues and activities. In most situations, a variety of
common sense approaches normally employed in informal research will produce
acceptable results. More rigorous techniques can be employed when the situation
and the skills of the evaluator permit.
50. Rigorous and relatively complex evaluations require considerable resources
and time and are best carried out by persons with some background in evaluation
work or at least a degree of expertise. Such persons have the advantage of
understanding how to apply the variety of analytical approaches and evaluation
techniques that can potentially be used.
51. There are many analyses and techniques which are based on certain
assumptions, conditions and principles which need to be understood by the
evaluator in order to avoid misusing the analysis and drawing erroneous
conclusions. Although often attempted, sampling, for example, is susceptible to
misuse without some knowledge of sampling theory. Similarly, preparing and
interpreting questionnaires generally requires a degree of expertise for the results
to have meaningful validity.
52. For most UNHCR evaluations, a general exposure to a variety of analytical
approaches is sufficient. Staff familiar with the planning and monitoring of refugee
operations, but without any particular expertise in evaluation, can use an
approach primarily based on interviewing. Such an approach provides an
approximation of results and causes that is sufficient for a general understanding
of what has transpired and for providing advice regarding decision-making.
Despite limitations to the approach, it is perhaps the most commonly employed. It
is also one of the easiest to carry out and should not be difficult for anyone with a
general understanding of the activities to be evaluated.
53. In evaluating complex programmes or operations, complicated research
methodologies are generally inappropriate, although more sophisticated designs
can be considered in evaluating various sectoral activities. Furthermore, the lack
of hard data in most situations often means that evaluations of field activities must
rely heavily on more subjective information obtained through interviews and
on-the-spot assessments.
54. Most evaluations of refugee activities are not viewed as a scientific research
activity but rather as an aid to decision-making and management. UNHCR
evaluations, therefore, generally aim at producing objective but not necessarily
conclusive results. Consequently, evaluators must often aggregate inferences
obtained in a variety of ways rather than seeking a definitive assessment through
a more complex and rigorous method.
55. Once staff planning the evaluation have held thorough discussions concerning
the operation, its activities, issues and any problems being encountered, many
steps involved in the study should be reasonably clear. The evaluator must
develop a conceptual understanding of the operation as a process for achieving
certain ends. At the same time, the evaluator should essentially disaggregate the
operation, viewing each element of management, (planning, staffing, etc) and
each activity and sector, from a number of different perspectives. (See Annex for
examples of several frameworks and questions that can be considered.)
56. The overall operation and its parts should be analysed and performance
assessed, using whatever measures are available, including implicit or stated
objectives, criteria and standards. Much of the evaluation work may be devoted to
gaining an understanding of what can reasonably be expected in a given context.
Organisational, national or international standards are often useful measures, but
in most instances can only be tentatively applied.
57. The most essential characteristic of evaluation studies is their attempt to
measure results against objectives. Unfortunately it is not always easy to achieve
clarity and precision in objectives. Objectives are often vague, general and difficult
to measure. In many instances, there are several objectives with different
priorities and complex inter relationships which may, in fact, not even be
complementary.
58. Multiple and ambiguous objectives need not be a major obstacle in carrying
out useful evaluations. In most cases a wide range of evaluative perspectives can
be applied rather than simply assessing the extent to which the operation and its
activities have achieved their stated objectives. For example, the evaluation could
also analyse the impact of programme or project activities have had on the
refugee target group or in other words, how the operation or activities have
contributed to the refugees' well-being or significantly changed their condition. In
addition, the evaluation could attempt to analyse the cost effectiveness of
activities and how efficiently resources have been used. Finally, the evaluation
could consider the economic, cultural and institutional impact of programme
activities, as well as any unanticipated effects.
59. Evaluations should not only examine the results, or likely results, but also try
to understand how various interventions actually produce the final outcome. An
analysis of the relationships which can be seen to exist between activities is often
more illuminating than a simple analysis of accomplishments. Casual
relationships are particularly important since the evaluation's conclusions and
recommendations regarding a situation will normally emerge from the causes.
60. In trying to document lessons, evaluators may want to ask such questions as:
if the operation were being mounted again what approaches, systems or
procedures could be changed to provide a better framework within which to
operate, or conversely, what policies were pursued in the operation that were
restrictive or inappropriate to the situation.
SUPERVISING THE EVALUATION
61. It is essential that staff responsible for planning and organising the evaluation
thoroughly guide and supervise those who carry out the exercise particularly
during the initial and final phases of the work. Of particular importance is a
thorough discussion of the assignment's objectives, assumptions, potential
outcomes and the interests of various parties involved during the early stages of
the evaluation.
62. It is often useful to accompany an outside evaluator to early meetings. At a
later stage the topics covered during the evaluation can be discussed with a
sample of those being interviewed to ensure the relevant questions are being
asked. Discard the notion that external consultants should be as independent and
self-sufficient as possible and instead endeavour to support the evaluators as
much as possible. If the capacity is available, do basic research and provide
documents. This will save time and help to ensure the work is properly done.
Finally, try to make office space available which can ensure regular consultations
and permit private meetings.
63. Experience has shown that one of the most crucial points of supervisory
involvement occurs when conclusions and recommendations are formulated and
when the report's structure is developed. It is at this stage that external staff often
need assistance in shaping evaluation findings and formulating recommendations
that are feasible and realistic. Guidance in developing the report outline can also
help ensure that the most important messages are brought to the fore.
PREPARING A FINAL REPORT
64. A report presenting the evaluation conclusions and recommendations is an
important management tool and an essential element of any evaluation. The
report should serve as a focal point for debate and discussion, facilitate
decision-making, and serve as an important record for the future. A written
document also provides those who did not have an opportunity to contribute to the
evaluation to add their comments later. For those who may not have been directly
involved in the operation, such as staff responsible for training or similar
operations elsewhere, the report may provide a wealth of ideas and information
that can help guide other operations or serve as a case study for planning and
training.
65. Reports should be tailored to the audience and the material being presented.
For most readers, however, the more concise, analytical and balanced the report,
the more useful it will be. Unnecessarily lengthy reports are a common error that
should be avoided. This includes the temptation to present superfluous
information in the hope it will be of "use to someone", demonstrate how much
work was done, or show the thoroughness of the analysis.
66. Reports should normally include a summary or an overview that is thoughtfully
prepared in anticipation that most of the reports recipients will not have time to
thoroughly read the whole report. Some background should be included at the
outset that explains the evaluation approach used, what was done, and why.
Essential background on the operations or activity should also be presented
briefly explaining its origin and purpose, organisation and resources involved. In
most cases, a historical description should, however, be kept to a minimum.
67. In general, evaluation reports should communicate information in a way that
can be easily understood and inspire action. Conclusions should be logical,
justified and follow from the evidence presented. Recommendations should also
be well supported by the body of the report which should include sufficient data,
examples, argumentation, or expert opinion to sustain the findings.
68. Recommendations should be feasible and clearly identify who is expected to
take action. Issues or questions which need further study or consideration should
be identified and discussed. Evaluators should, however, avoid or very judiciously
make commonplace recommendations for more study, more resources and better
co-ordination which have become evaluation clichs. The same general advice
would apply to simplistics exhortations for improved performance.
FOLLOWING UP ON EVALUATION FINDINGS
69. There is always significant potential for an evaluation report to become an end
in itself. It is therefore imperative that plans are developed that will ensure
appropriate follow-up action on report recommendations and any decisions taken.
Decisions should be incorporated into an action plan for follow-up indicating how
changes will be made, by whom and when.
70. Responsibility for follow-up should normally be assigned to staff with at least
some direct responsibility for the functional areas in which changes are to be
made. There are, however, many different ways to approach follow-up. A few of
the most common possibilities include :
convening a meeting to review the evaluation's conclusions and
recommendations and to direct implementation of the follow-up decisions;
G
using the evaluation as input to a working group; G
using the evaluation to formulate the agenda of a policy setting or strategic
planning meeting;
G
incorporating evaluation findings into policy papers, guidelines or training
material;
G
using the evaluation report to support UNHCR's negotiating position in
discussions with government, donor or other agencies;
G
using the report as a basis for a meeting with other agencies or NGOs;
and,
G
widely disseminating the evaluation report to all countries where similar
activities or operations have taken place.
G
71. An independent organisational component is often assigned responsibility for
monitoring follow-up. Under such an arrangement, those responsible for
monitoring follow-up are kept informed of any action taken or changes produced
as a consequence of the evaluation. Ideally, the entire process should culminate
with an implementation report describing any decisions taken on the basis of the
evaluation and what steps have been taken as a result.
Annex
Using evaluative frameworks and questions to analyse activities
1. In simplest terms, carrying out an evaluation often means making overall
judgements about an activity as well as identifying problems and their causes so
that they can be corrected and avoided in the future. To judge an activity, the
evaluator should analyse it as a whole from a number of perspectives using a
variety of measures and criteria. In most cases, an analysis is facilitated by
separating the activity into logical parts in order to examine and assess each of
the parts that make up the activity.
2. In breaking an activity or operation into logical parts, it has always been
common in UNHCR to separate and analyse activities by each programme sector,
such as water, food, health etc., or by functional components such as protection,
assistance, external relations, etc. The phases of an activity can also provide a
useful way of understanding and analysing it. A useful but less commonly
employed framework in UNHCR is the "logical framework" model that breaks an
activity into inputs, activities, outputs and objectives.
3. Once the most identifiable or essential elements of an activity have been
identified, each element can then be analysed using basic evaluative concepts
such as:
Effectiveness which considers the extent to which an activity has
achieved or is likely to achieve its objectives;
Efficiency which considers how well resources are used to
undertake activities and achieve objectives;
Economy (or cost-effectiveness) which considers whether
objectives could be accomplished at a lower cost, examines waste,
and assesses whether results or benefits justify the cost; and
Impact which normally considers an activity's contribution to the
well-being of the beneficiaries, and assesses change brought about.
Other aspects often considered include :
Relevance which considers whether an activity or project is well suited or
appropriate to the needs or situation.
Unanticipated consequences which considers any significant unforeseen
effect of an activity, either beneficial or detrimental.
4. A framework based on management functions can also be usefully employed to
disaggregate or analyse. Using such a framework, an operation, activity, or
sector, can be further reviewed in terms of its :
planning; G
organisation; G
staffing; G
directing (or implementation); and G
controling (including monitoring). G
5. Various frameworks can provide useful starting points for reflection and
analysis as well as provide a way of ensuring aspects have not been overlooked.
Many staff find question based frameworks the easiest to understand and apply.
6. The simplest, and perhaps most frequently applied framework in monitoring
and evaluation is a form of :
what is right? G
what is wrong? G
why? and, G
what needs to be done? G
7. As the assessment becomes more thorough, other elements are normally
added to the framework, such as a statement of objectives and assessment of
their achievement. Inevitably, any discussion of objectives leads to further
questions regarding the assumptions that the objectives are based upon, and the
continuing validity of the original objectives.
8. A simple but slightly more complex framework that is often employed as a basis
for self-evaluation includes questions such as :
what was the objective? G
was the objective met? If not why not? G
Were the plans and resources adequate? G
What changes and alternatives should be considered? G
what lessons were learned? G
9. Contributions of the various conceptual framework and questions described
above provide a good starting point, but it will quickly become apparent to most
staff that the number of possible questions is almost unlimited. Consequently, an
important aspect of planning and carrying-out an evaluation is determining which
questions and answers are the most crucial in understanding and assessing an
activity. In undertaking most evaluations and in organising a final report, it is
generally useful to keep in mind that the following elements will have to be fully
developed and analysed for each finding or section of the report:
What is the condition, situation or problem that requires correction? G
What effect or significance does the problem have? G
What is the cause or reason for the condition? G
What conclusions can be drawn and recommendations made to address
the causes and correct the problem?
G
10. Such frameworks serve as tools to be used and adopted as needed. They
should, however, not be applied so rigidly that they become cumbersome. Rather,
they should be modified and adapted to suit the information and analytical
requirements of a particular situation.
11. An example of the kinds of questions that would be reflected upon and asked
in reviewing a UNHCR country operation are presented below in order to serve as
an illustration of the many aspects that should be considered as well as provide a
series of questions that may serve as a starting point in carrying out such a
review.
12. Many of these commonly asked questions can assist staff in preparing terms
of reference as well as aid in carrying out the actual evaluation. They can also
stimulate thinking and help in generating many other essential questions.
I. UNDERSTANDING THE SITUATION
What access do we have to refugees G
Are they of clear concern to UNHCR G
What would happen if refugees were not granted asylum, assisted G
Are lives in danger G
Are any refoulements, or deportations of asylum seekers G
Are communities located and organised in a way that provides security G
Are new movements imminent or has the situation stabilized G
How rapidly are circumstances changing G
Are refugee needs being seen in isolation from other affected persons G
What are the principal concerns of the host government G
Have refugees had a destabilizing effect or other impact on the area G
What is known about the refugees G
What types of contacts are maintained with refugees G
How are they organized G
What is the number and make-up of the group in terms of origins,
vulnerability, age, gender, skills and social-economic background
G
Are there significant gaps in statistics and knowledge G
How accurate are populations figures G
Are events in the country of origin understood G
What is known about refugees' intentions and desires G
What options are available to the refugees G
What efforts have been made to identify and seek solutions G
What is known about the safety and feasibility of repatriation G
What is the likelihood of local settlement G
How self reliant are the refugees G
Will refugees settle spontaneously G
What are the prospects for integration G
To what extent will refugees have to find their own way G
How will the experience change the refugees and effect the solution G
What is the likelihood of refugees returning to their traditional way of life G
Are there aspects of the situation that attract refugees G
What are their needs, and how are they evolving G
What information is available regarding the beneficiaries or situation,
including studies, reports, data, etc.
G
What will happen if refugees are not assisted G
Would assistance attract opportunity seekers G
Are refugee privileged in any way compared to nationals G
How does the refugee presence affect nationals or the host country G
How long have refugees been receiving assistance G
What are their attitudes toward assistance G
How do standards compare to national and country of origin standards G
What are the limitations to assisting or protecting refugees G
What are the prospects for self reliance, in what ways G
How can more self-reliance be achieved, what are the barriers G
II. REVIEWING THE PLAN AND APPROACH
Has there been an adequate diagnosis of the situation and analysis of the
potential beneficiaries
G
Has data been systematically collected and analysed G
What data, assessments and research are required G
Has feedback and experience been taken into account G
What is UNHCR's potential role G
Are there any precedents for UNHCR actions G
What are the reasons for involvement and criteria for phase out G
What are UNHCR's comparative strengths in the situation G
What should be the nature, extent and duration of UNHCR involvement G
What are our principal problems and constraints G
Have all critical factors been considered including the political, economic
and cultural factors, as well as the weather and logistical considerations
G
Are there lessons from other situations that can be drawn upon G
What aspects of the situation are likely to change G
Have various scenarios been taken into account G
What are the prospects for the future G
How can refugees be assisted toward solutions G
Have needs been prioritized G
What major assumptions were made in designing the activity G
Have the assumptions proved accurate G
Were there major considerations or problems that should have been
foreseen but were not
G
Are assumptions and expectations realistic G
Have the objectives and strategies been sufficiently clarified G
Were alternative approaches to achieving the objectives considered G
Are objective precise or vague statements of intent G
Are the objectives sufficiently explicit so that their achievement be
determined
G
Are there specific indicators against which the attainment of objective can
be assessed and impact determined
G
What are the principal strategies G
Were alternate approaches and strategies considered G
Are activities within UNHCR's mandate and appropriate G
Which needs are a priority G
What activities are the most critical G
To what extent do activities correspond to actual needs G
What were the main inputs, activities and outputs G
What is the status of activities G
What major changes, external, institutional, political, or economic have
occured, since the activity was planned and will have a substantial impact
on results
G
What changes are being sought or planned G
Do activities serve a sufficient number of potential beneficiaries G
Can activities be reduced without negatively affecting refugees G
What are the potential difficulties associated with activities G
What are the principal constraints G
Are activities consistent with solutions G
Are activities and services linked to longer term integration or repatriation G
Will activities contribute to rehabilitation and reconstruction G
Do activities discourage initiative and lead to dependence G
Does the activity logically fit and meaningfully relate to an overall
framework
G
Does the activity relate to similar parallel, prior, or subsequent activities G
What administrative and material support is required G
What are the links between protection and assistance activities G
Is the plan comprehensive and coherent or a justification for piece-meal
inputs
G
Is there potential to rationalise and harmonize activities G
Do activities have a planned completion, can a date be set G
Do activities make sufficient provision for phasing out G
How have activities begun and developed G
Are activities developing a life of their own separate from needs G
Do activities imply permanence G
Once begun are activities likely to expand G
How was the programme to be implemented e,g, how was it organised G
Who are the principal beneficiaries and how were they to benefit G
What special needs attention is being given to women and children, other
groups
G
Are environmental concerns being considered G
What are the implications of new activities and approaches G
What are the operational requirements G
What skills or knowledge is required to formulate needed projects G
Do staff have the experience and expertise to carry out the activities
envisaged
G
Does the general planning framework permit more detailed technical
planning
G
Are plans coherent G
Did planning anticipate and make provisions for over-coming obstacles G
How extensive and comprehensive is the planning G
Are headquarters and the field in agreement regarding the approach and
strategy
G
What are the key policy dilemmas G
Has headquarters provided solid backing and support G
What political or moral support can UNHCR provide to partners G
Can reductions be negotiated, what are the barriers G
What can be outsourced G
What are the immediate, medium and long term requirements according to
levels of assistance foreseen, numbers of beneficiaries and priority of
needs,
G
Is there sufficient capacity to perform the required activities G
Will the activity be able to secure resources G
Can funds be raised to support the activities G
What role can be played without resources G
Which activities need outside support in order to be sustained G
Have all the likely problems been anticipated G
What is the outlook for success G
III. ASSESSING THE CONTEXT AND LINKAGES
What are national attitudes and interests regarding refugees, assistance G
What are the government's objectives, how are they different G
Are refugees considered in national plans G
Can national resources be mobilised G
What are the critical or potential constraints imposed by the host
government
G
How vulnerable is UNHCR to various pressures, G
What are the prospects for a change in government policies G
Can assistance be used to shape and liberalise policies G
What are the principal areas of conflict with local interests, government,
and partners
G
What potential interest do donors have, how can interest be developed G
Are there related country or regional initiatives G
Are related initiatives successful, can they be drawn upon G
Is there external expertise which can be utilised G
In which areas do organizations have a presence G
What do others have to offer G
Do roles conflict or result in duplication G
What activities can or should be carried out by others G
Are there particular areas and sectors in which partners are interested or
uninterested
G
Do partners have the mandate, expertise, material resources including staff
and money to assume activities
G
What commitments has UNHCR made G
Is UNHCR being drawn into development activities G
Is there a mismatch of mandates, objectives or approaches G
How is UNHCR perceived G
Have partners capabilities been identified G
Do partners have the capacity and skills required, how can we support
them
G
Do partners share common goals G
Was there adequate consultation and participation with those related to or
affected by the plans
G
Is there agreement among partners on UNHCR's plan and strategy G
What are the links and arrangement with others G
Are activities well coordinated G
Are sectoral activities or responsibilities successfully divided among
partners
G
To what extent is there competition among agencies G
How can UNHCR collaborate more effectively with other organisations G
What type of support and assistance is available from donors, partners,
other agencies and host government
G
Is the support provided by partners adequate, good G
Can interest and support be increased G
Have public information activities been adequately covered G
What are the principal external constraints G
IV. ANALYZING IMPLEMENTATION
Have assumptions proved accurate G
Have plans been followed G
Have the objectives changed during implementation G
What major changes have occured since activities were planned G
Were there problems or considerations which should have been foreseen G
Is preparedness adequate G
Are activities fragmented and disconnected G
Are assistance activities contributing to protection G
Have roles and responsibilities been logically delineated and
communicated
G
Are lines of authority and reporting clear G
Do all levels have the decision-making authority they need G
Are the level and forms of activities consistent among locations G
Are there staff or skill shortages G
Have staff received sufficient training and orientation G
Does the absence of refugee participation and training weaken activities or
hinder hand-over
G
Were national resources, local expertise and materials used to the extent
possible
G
Do material, staff and financial resources match the needs, G
Were resources mobilised as required G
What problems were created by the unavailability of resources G
Would additional or fewer resources make a significant difference G
Have there been delays in the preparation, receipt and signing of
instructions and agreement
G
Have schedules been adhered to G
Which problems have lead to delays and what were the consequences G
Has information been communicated in a timely manner G
Is information shared among partners and staff G
Has adequate guidance been provided G
Was decision-making delegated to the appropriate level G
Are decisions made in a timely way and at the right level G
What slows or hampers decision-making G
Have financial and administrative procedures been followed G
Are activities in compliance with UNHCR policies and standards G
Have procedures been developed and formalised as required G
Are activities consistent with policies, practices and guidelines G
What guidelines, policies or decisions are needed G
Are co-ordinating arrangements adequate in preventing duplication and
ensuring all sectors, areas and groups are adequately covered
G
Are contractual arrangements adequate G
How well were activities actually coordinated G
Has UNHCR been able to influence and guide agency programmes G
Are the actions of all partners directed toward solutions G
Have partners performed adequately G
What are the principal source of conflicts G
Can UNHCR be less operational; what can others do G
What sectors or activities is UNHCR most involved in, least involved in G
What stages have activities passed through G
Have activities begun well or poorly G
What were the strengths and weaknesses of implementation G
Have structures, systems and procedures facilitated or obstructed activities G
Have approaches been cost effective G
What measures could potentially reduce costs G
Have skills been developed and operational capacities created G
How well are refugees or returnees being prepared G
Are activities routinely monitored and corrective action taken as required G
What are the principal constraints or impediments to implementation G
What changes would improve implementation or services G
V. EVALUATING RESULTS
Is the plan logical and coherent G
Were measures to provide security and protection adequate G
Are the most needy beneficiaries targeted G
Have activities contributed to solutions G
Have activities helped to prevent further population displacements G
Was social conflict prevented or limited G
Are measures adequate to ensure activities will be sustained after UNHCR
phases out
G
Were needs overlooked or neglected G
Are the trends satisfactory or worrisome G
What initiatives offer potential G
Which activities or sectors have not achieved the desired standard G
Was self reliance encouraged and achieved G
Have we helped to draw attention and mobilize support G
Has the operation had a substantial impact on government policies or
public opinion
G
Have we honoured commitments G
What long term effects is assistance having G
What will happen after our withdrawal G
Is it likely that the activity will have the expected impact G
Have activities made a significant difference G
Has the operation made a substantial contribution to the refugees' well
being
G
Have activities significantly changed or improved conditions of the
beneficiaries
G
What important successes or failures occured G
Which sectors and activities have been the most successful, the least
successful
G
What are the most successful and unsuccessful aspects of activities,
reasons why, and their potential
G
What approaches or strategies were the most successful or problematic G
What are the principal problems G
What weaknesses or strengths have surfaced G
What has been accomplished to date G
How do achievements compare to plans and objectives G
Did the assistance and services cover needs G
To what extent is the activity likely to achieve its objectives, if not, why G
Were the resources effectively used G
Were activities efficiently carried out G
Could the objectives have been accomplished at a lower cost G
What areas or activities offer potential cost savings G
Do results or benefits justify the costs G
Were activities excessively wasteful G
Which activities were most costly relative to their priority G
Does the activity complement, duplicate, overlap, or work at
cross-purposes with other programmes
G
Was assistance significant or marginal value to beneficiaries G
Were objectives relevant and worth pursuing G
What are the most useful activities and sectors G
Is the activity having any significant unexpected effects, either beneficial or
detrimental
G
Why did the operation succeed or fail G
What factors affected project performance, how G
Were there better ways to address the needs G
What should we have done differently G
What should we now be doing more or less of G
To what extent can activities be reoriented G
What would happen if parts of the operation were eliminated G
What alternatives to the current plan merit consideration G
Where is improvement needed G
What new programmes, policies, activities, systems or guidelines are
required
G
What lessons can be learned G