FORUM 2009
as the USAID-supported Demographic
How Do We Measure
What Really Counts?
Session discusses assessing the impact of complex
development interventions and applying results.
BY HILARY NALVEN, SENIOR PROGRAM ASSOCIATE FOR STRATEGIC IMPACT, INTERACTION
I
NTERNATIONAL DEVELOPMENT
actors confront two challenging
demands: the need to respond to
complex development problems
with multi-faceted interventions and,
at the same time, to demonstrate
impact. In response, InterAction’s
Evaluation and Program Effectiveness
Working Group hosted the Forum ses-
sion entitled How do we measure what
really counts? Assessing the impact of
complex development interventions and
applying results. The workshop was
moderated by Carlisle Levine, Senior
Technical Advisor for Advocacy Evi-
dence Systems and Impact Assess-
ment at CARE USA. Panelists shared
their experiences successfully measur-
ing the impact of empowerment pro-
grams and using evidence to influence
decision-makers.
Ms. Levine noted that, while there is
38 MONDAY DEVELOPMENTS September 2009
not a single definition for impact evalu-
ation, this session would use the one
that centers on assigning attribution
by establishing a credible counterfac-
tual, i.e. evidence that without a spe-
cific action some change would not
have occurred.
Howard White, Executive Director of
the International Initiative for Impact
Evaluation (3ie), put forth an objec-
tive, “to counter the view that [orga-
nizations should] only measure what
is easy to measure and not what mat-
ters.” To successfully do this, one must
first clearly define the concept that is
to be measured. While at the World
Bank, Mr. White measured women’s
empowerment using indicators at the
national, intermediary and local levels
and allowing for them to remain cul-
turally specific.
There are a number of surveys such
and Health Survey that collect various
measures of women’s agency, the ques-
tions varying by country. A promising
new approach is that of “vignettes”,
which provide culturally specific refer-
ence points through short descriptions
on three or four people in a familiar
local setting. Respondents are asked to
rate their own circumstances relative to
these vignettes. This ranking, although
difficult and costly, allowed the World
Bank to accurately quantify how
empowerment had changed over time.
Mr. White emphasized that impact
evaluations require costly data collec-
tion and are not always appropriate.
Therefore, it is important to identify
when to complete impact evaluations.
Namely, decisions should be techni-
cally, practically, and politically fea-
sible, and: “be issues driven, not meth-
ods driven, pilot projects, or important
projects [those that are large-scale or
expected to have substantial impact];
test ‘conventional wisdom’; and rep-
resentative of an agency’s activities.”
When impact evaluations are not
advisable, consistent performance
monitoring can be carried out.
Dr. Anju Malhotra, Vice President for
Social and Economic Development at
the International Center for Research
on Women (ICRW) concentrated on
four points. First, it is important to try
to measure complex concepts such as
justice, equity and poverty alleviation.
Second, classifying issues as soft or
hard is a false dichotomy; most have
both hard (tangible) and soft (intan-
gible) measures. Third, we should
measure the “what” and the “how” of
development interventions. Finally,
evaluators and implementers must be
closely connected.
Dr. Malhotra believes women’s
empowerment is one of those complex
concepts that can and should be mea-
sured. She defined women’s empower-
ment as “the ability to make strategic
life choices,” and clarified this further
stating, “having a cow does not mean
empowerment, rather it is the ability
to use the cow to sell milk and make
money.” She compared measuring
women’s empowerment to the complex
issue of poverty—something people
have been trying to measure for years.
For example, like poverty, women’s

empowerment looks different in vari-
ous contexts and may take decades to
see progress.
Furthermore, the overall picture and
the sub-parts of complex interven-
tions must be considered. Specifically,
since women’s empowerment consists
of large and small changes over the
long-term, it should be broken up into
“slices.” A key question is, “What slice
do we take on?” and, based on this,
develop appropriate indicators and
evaluation design. ICRW concentrates
on the “empowerment slice”, using as
measures the actors involved, the con-
text of the project, and the levels at
which ICRW works.
Both panelists addressed how to use
evaluation findings to influence deci-
sion-makers.
Dr. Malhotra hypothesized that eval-
uating mid-level issues might be more
conducive to influencing change, as
they identify more actionable, feasible
solutions (e.g. while women’s empow-
erment is broad and complex, child
marriage is a mid-level issue). They
advised that evaluations must be well
designed and technically rigorous. It
must address relevant questions for
both program participants and policy-
makers, and include all actors from the
beginning to ensure necessary buy-in.
Notably, findings can be context spe-
cific; to generalize results we need a
wide-ranging body of evidence. Finally,
those in power must be educated about
the issues.
After hearing the presentations per-
haps a more accurate title for this ses-
sion would be, “you can measure what
really counts,” with an addendum,
“but it isn’t easy.” MD
Further reading
Measuring Empowerment in Practice
Structuring Analysis and Framing
Indicators by Ruth Alsop, Nina
Heinsohn
Comparing Incomparable Survey
Responses: New Tools for Anchoring
Vignettes by Gary King and Jonathan
Wand
Measuring Empowerment: Cross-
Disciplinary Perspectives by Deepa
Narayan
Managing Risk and
Litigation Exposure
Panel offers ways to improve security and limit liability.
BY SHANNON DAVIS, SECURITY ASSOCIATION MANAGER, INTERACTION
D
O NGOS NEED TO BE CON-
cerned with litigation? Ray
Biagini, a partner at McKenna,
Long, and Aldridge, says yes.
Yan Bui, Account Executive at Clem-
ents International, and Victor Ferreira,
Senior Consultant at Control Risks,
discussed ways in which an NGO is
at risk and the measures it can take
to protect itself from litigation at the
InterAction workshop Increasing Risk
and Litigation Exposure.
There is a rising trend of NGOs fac-
ing litigation due to the increased risk
NGO staff encounter in hostile environ-
ments. Although there are not many
reported cases, the presenters believed
there are an increasing number of NGO
legal liability lawsuits, typically taking
the form of a negligence (tort) claim.
John Schafer, Senior Security
Coordinator at InterAction, offered a
hypothetical scenario: two NGO staff
members, member Alpha and member
Bravo, leave for a trip to Haiti. Alpha
received security training, the coun-
FORUM 2009
try risk assessment, and information
about the organization’s policies and
procedures for staff travelling overseas.
Bravo received very little information
and no training. Upon arriving at the
airport, Alpha was picked up by pre-
arranged driver. No pre-arrangements
had been made for Bravo so he used a
local driver and was attacked en route
to his destination and taken captive.
After 37 days, Bravo was released from
captivity, suffering from hepatitis C,
body injury, malnutrition, and psycho-
logical ailments. Is the NGO at fault?
The answer is yes. Using this example,
the presenters discussed the steps
NGOs can take to protect their staff.
Victor Ferreira noted that the orga-
nization has a responsibility to pro-
vide structures and tools to ensure the
safety and security of its employees,
otherwise referred to as the organiza-
tion’s duty of care. InterAction member
organizations are required to comply
with the Minimum Operating Security
Standards, or MOSS, which can limit
exposure to negligence claims. NGOs
should provide risk assessments to
staff working in or travelling to specific
regions or countries that may be dan-
gerous. Staff members should also be
asked to review and possibly sign some
form of written documentation that
acknowledges their understanding of
the risks they may encounter.
NGOs should consider having some
form of insurance to help mitigate
risk. However, even with insurance
coverage, NGOs should be proactive
in ensuring that all staff receive secu-
rity training. If the NGO has a security
infrastructure, this helps to reduce
the premium rates as well as liability.
Security standards should be continu-
ously monitored and communicated
according to the potential risks that
staff may encounter overseas. MD
MONDAY DEVELOPMENTS September 2009 39