Bhasin K. Are We on the Right Track?

Report of a Workshop on Participatory

Evaluation // Knowledge shared. Participatory Evaluation in Development Cooperation /
Ed. by Edward T. Jackson and Yusuf Kassam IDRC/Kuarian Press! "##$%
Are We on the Right Track? Report of a Workshop on
Participatory Evaluation
Kala &hasin
In January 1!" seven o# us who had $een involved with rural development #or several years
spent three days in %ecundera$ad& India& discussing how to evaluate the process o# participatory
development. 'ur meetings were held in the o##ice o# the (ural Development )dvisory %ervice&
and #ive o# us also stayed there. *etting to +now one another during and outside the meeting was
as rewarding an e,perience as the discussions themselves. )t the end o# the three days& all o# us
#elt satis#ied with the consensus and mutual understanding we had managed to arrive at and the
#riendships we had $uilt. -his chapter is an attempt to share our discussions with those who were
not with us $ut who are as interested in these issues as we were and continue to $e.
.hat #ollows is a more or less ver$atim reproduction o# our discussions& al$eit arranged a little
more systematically. .e are sharing our discussions with you in the hope that this document will
lead to #urther discussion and greater clarity a$out the evaluation o# development activities.
&ackgroun'
Di##erent development approaches and strategies re/uire di##erent +inds o# evaluation methods
and techni/ues. I# development pro0ects are top1down& started $y people #rom outside the
community 2governmental or nongovernmental organi3ations or agencies4 to provide services
such as health and education and to $ring a$out certain changes in production methods and
techni/ues& then the local people are merely recipients& targets& or o$0ects o# development.
People #or whom development is supposedly intended have little or no say in the content
-he wor+shop participants were )$ha Bhaiya& )runa (oy& Datta %avale& Kamla Bhasin& 5. Kurian&
5. 6. %hastri& and 6i+as Bhai.
)n earlier version o# this chapter was #irst pu$lished $y 7reedom #rom 8unger Campaign/)ction #or
Development Program o# the 7ood and )griculture '##ice in 9ew Delhi& India.
and direction o# such e##orts. .ith hindsight it can $e said that top1down& centrali3ed
development pro0ects seldom help the really poor and needy $ecause the real causes o# poverty
are le#t un/uestioned and unchallenged. -he evaluation o# such pro0ects is also& /uite logically&
top1down& geared and done $y the decision ma+ers without any participation o# the local people.
-hose #rom whom in#ormation and opinions are gathered are not even in#ormed a$out the
evaluation outcome. In #act& o#ten even the pro0ect holders have no say. 7or them& more o#ten
than not& evaluation is li+e an inspection $eing carried out $y outsiders at the insistence o#
#unding agencies& and they #eel threatened $y it. -he main purpose o# such an evaluation is
clearly one o# #inancial accounta$ility& and emphasis is on physical targets. Because this model
o# development does not insist on starting a process o# consciousness1raising& increasing
awareness& and mo$ili3ation& little attention is paid to the assessment o# intangi$les such as
people:s participation& the decision1ma+ing process& level o# awareness& and practice o#
democracy.
Development& however& can also $e understood as a means o# helping the poor to collectively
analy3e the socioeconomic& political& and cultural structures that +eep them poor and get
organi3ed to challenge these structures. In such a development model& the oppressed people are
seen as su$0ects& not merely o$0ects o# their own development. -he program is a partnership
$etween the local masses and outsiders. Its strength is concomitant with that o# the people:s
organi3ations 2P's4 that emerge& their democratic #unctioning& and the actions they ta+e to tilt
the $alance o# power and resources in #avor o# the e,ploited masses. -he evaluation o# such
e##orts #or development and organi3ation has a di##erent purpose and demands other methods&
techni/ues& and indicators.
.e who met in %ecundera$ad were interested only in the evaluation o# the second type o#
development e##orts. )ll o# us #elt that although a large num$er o# action groups 2)*s4 are now
concentrating on the mo$ili3ation and organi3ation o# the poor& there is little clarity on how these
e##orts should $e assessed. -he purpose o# our tal+s was to achieve some common understanding
on the $asis o# the e,perience and ideas we all had on evaluation.
(oe Iportant Consi'erations
.e agreed that the evaluation o# people1centered and people1oriented e##orts at consciousness1
raising& mo$ili3ation& organi3ation& and action should consider several +ey points.
Evaluation Is Reflection on Action
Evaluation& as we see it& is collective re#lection on the actions ta+en $y individuals within a
group and $y the group itsel#& and on the methods o# #unctioning o# a group. Its purpose is
improvement $oth in the understanding and analysis o# reality and issues and in #uture action.
-hus seen& it is an important method o# group education and learning.
Built-in and n!oin! Evaluation
7or a group interested in improving not only the socioeconomic position o# the poor $ut also the
methods o# #unctioning and the understanding o# everyone involved in the wor+& evaluation has
to $e $uilt1in and ongoing. (e#lection $ased on concrete in#ormation has to $e closely lin+ed to
action. In addition to ongoing evaluation& at the end o# one or two years& there can $e an overall&
time1$ound evaluation that is a cumulative assessment o# what has ta+en place over a decade.
-he e,perience o# the Comprehensive (ural 'perations %ervice %ociety 2C('%%4 shared $y 5.
Kurian illustrated very well the method and importance o# ongoing evaluation and its
culmination into an annual e,ercise. -he village sanghams 2small& #ace1to1#ace groups o# rural
poor4 initiated $y C('%% assess their activities and the per#ormance o# the #unctionaries every
month. In addition& they assess every ma0or action underta+en $y them. Evaluation sessions are
also organi3ed every three months at the cluster and area level. )part #rom these evaluations $y
the local people& C('%% sta## meets once a month to ta+e stoc+ o# its activities and methods o#
#unctioning. )nnual sel#1evaluation is done in January o# every year.
Em"#asis on $elf-Evaluation
-he emphasis o# a people1centered and people1oriented program or organi3ation has to $e on
sel#1evaluation in which the people and the organi3ers not only participate $ut also decide a$out
its parameters& #orm& and methods. -he #inal 0udges o# a program:s e##ectiveness must $e the
people themselves.
Evaluation of Tan!ibles% Intan!ibles% and &rocesses
I# the o$0ectives o# development are $oth tangi$les 2such as improved economic status& improved
health4 and intangi$les 2such as increased awareness& people:s participation& and democratic
decision ma+ing4& then o$viously evaluation must also #ocus on $oth these aspects. -here are
techni/ues availa$le #or assessing tangi$les& $ut we need to develop methods and indicators as
#ar as intangi$les and processes are concerned.
-he process a group goes through to reach decisions and act is as important as the outcome o#
the action. .e have to understand how people move toward the achievement o# their o$0ectives.
It is necessary to understand how the processes within P's and )*s are related to general
processes in society and how they a##ect each other. -heir conte,t has to $e understood.
Just as there is a close relationship $etween action and re#lection& theory and practice& there is
also one $etween tangi$le o$0ectives& such as increased access to land or higher wages& and
intangi$le ones& such as improved level o# awareness and strength o# P's. Ideally& the
achievement o# one should lead to improvement o# the other.
P's might $e #ighting #or economic $ene#its& $ut unli+e the usual development pro0ects& P's
emphasi3e the processes and use each struggle to educate and strengthen themselves. )#ter
achieving some small victories& P's cannot sit /uietly and smugly& $ut need to constantly as+
how much space has $een created $y a campaign and how that space should $e used #or #uture
action. 7or them& the process o# structural change should $e an ongoing one that does not stop at
any particular point. -his is di##erent #rom target1$ound pro0ects& considered terminated on
completion o# a certain num$er o# wells& the installation o# pump sets& the production o# $iogas&
and so #orth.
'alse (ic#otomy between )onsciousness-Raisin! and Economic
(evelo"ment
.hen the entire emphasis o# development programs is on material development& /uantitative
analysis is primary. But when the emphasis o# development e##orts is on the growth o# people
and their organi3ation& /ualitative analysis assumes more importance. Because material
development and the development o# people:s consciousness and their organi3ation do 2and
must4 go together& /uantitative and /ualitative analysis cannot $e e,clusive o# each other. %ome
groups ta+e an e,treme position and re0ect all /uantitative data and measurement o# material
development. -hey tal+ only o# intangi$les li+e consciousness1raising or increasing the level o#
awareness. .e #elt a need to have a good synthesis o# evaluating tangi$les and intangi$les&
/uantitative and /ualitative results. I# one is wor+ing with the really poor& their material
conditions have to $e improved #ast 2mainly& o# course& through their own e##orts4. -he poor are
not going to $e interested in consciousness1raising #or its own sa+e. )ll consciousness1raising
must lead to an improvement in their material conditions& and vice versa. In #act& this dichotomy
$etween organi3ational wor+ and programs #or economic development is #alse and misleading.
*roups primarily doing organi3ational wor+ also improve the economic status o# the poor at
least as much& i# not more& as the so1called pro0ects #or income generation do. 'rgani3ations such
as Bhoomi %ena& %hrami+ %angathana& and C('%% have achieved tremendous economic
$ene#its #or the poor through their struggles to recover alienated lands& provide higher wages and
employment opportunities& lower interest rates& #ight corruption& reduce the power o# middlemen&
and so #orth. -he economic position o# the poor can $e improved $y removing insecurity and
e,ploitation& and i# these two tas+s go on simultaneously& it is ideal.
-he attempts to organi3e the poor also improve their receiving mechanism and $argaining power
and there$y ena$le them to ma+e increased use o# government schemes& $an+ loans& and the li+e.
-o recapitulate& economic development and people:s organi3ations and action are;and should
$e;dialectically related. Every struggle $y the oppressed should create more space #or their
economic development& and their improved economic status should in turn strengthen their
organi3ation.
*eed to +ook at T#ree Kinds of &rocesses
.e need to evaluate processes in three areas or realities& and also to loo+ at the interplay among
these three<
1. -he )*:s reality and the processes within it.
=. -he community within which the )* is wor+ing and the processes within the community.
". -he larger socioeconomic and political reality in which $oth )*s and oppressed
communities are situated.
It is important to analy3e and understand why some people #orm an )*& why they want to relate
to a certain oppressed community& what their perception o# the larger reality and structures is&
what conception o# change they have& and what their goals and aspirations are. Is there any
homogeneity $etween the aspirations and understanding o# the )*s and those o# the community
within which they wor+> 8ow realistic are the o$0ectives set $y them in the conte,t o# opposition
#orces>
Inter"lay of As"irations and Reality
It is also important to loo+ at the o$0ectives and aspirations o# )*s and P's in the conte,t o# the
#orces o# reality. .e have to see the dynamics $etween $oth. -he reality e,ists and operates
independently o# aspirations o# )*s and P's that intervene to change it according to their own
understanding. %o we must understand the totality o# the #orces o# society and see what the
intervention has succeeded in achieving.
9ot only is there need to assess the e,tent to which the o$0ectives and aspirations have $een
achieved& $ut they& themselves& have to $e constantly reviewed and read0usted according to
changing reality and changes in )*s: and P's: understanding. .e need methods and tools to
assess the )*s: and P's: goals in the conte,t o# their aspirations and hypothesis& and o# the
larger reality.
It is only when action is ta+en a#ter a systematic analysis o# the overall situation and reality that
it $ecomes meaning#ul and e##ective. 7or e,ample& i# one does community theater without
understanding the conte,t& and i# it is not related to any action& it provides& at $est& some
entertainment. People:s theater can inspire and lead to action only i# it is done with a perception
o# reality& and o# the needs and aspirations o# the masses. .hen divorced #rom P's and #rom
action& theater& non#ormal education& or consciousness1raising e##orts are uninspiring and
uninnovative and lead to no change in the oppressive situation and structures.
The Role of )utsi'ers in (elf*Evaluation
Emphasis on sel#1evaluation does not mean that we too+ the e,treme position that local people
and )*s can assess their wor+ themselves. .e recogni3ed that every perception has its
limitations. Just as outsiders: perception might $e limited $ecause o# their lac+ o# +nowledge and
ac/uaintance with local realities& local people:s perception might $e limited $ecause o# their
particularity. -he interaction o# perceptions and views 2$oth o# insiders and o# outsiders4 can
there#ore $e very $ene#icial.
-he presence o# an e,perienced and sensitive outsider can encourage the group to #ormulate and
articulate its thoughts more systematically and o$0ectively. ) sensitive outsider can enrich the
discussions $y $ringing in other e,periences& perceptions& perspectives& and dimensions. -here
can $e areas that local people either #orget to loo+ at or do not want to loo+ at. It is the outsiders:
role to $ring these #orgotten elements or reality into discussion& however unpleasant this might
$e. ?ocal people and )*s have to $e helped to reali3e that unpleasant #acts cannot $e wished
away. )n outsider plays an important role $y as+ing the right +ind o# /uestions and providing
use#ul insights #or dealing with dilemmas and uncertainties.
'utsiders can play this role e##ectively only i# they are actually insiders in more than one way.
-hey have to $e +nown and accepta$le to the people who are assessing themselves& should
identi#y with the group:s o$0ectives& and should $e involved in the same +inds o# struggles and
processes& although in another area or at a di##erent level. Inso#ar as they are involved and have a
commitment to the same goals& they are not @o$0ective@ evaluators. 8as not the myth o#
evaluation $eing o$0ective $een e,ploded>
7or helping in assessing various aspects o# wor+& we might need di##erent +inds o# outsiders& #or
e,ample& someone ac/uainted with health issues when it is a$out a community health program.
It must $e remem$ered that an insensitive outsider can ruin all e##orts at a genuine sel#1
evaluationA instead o# leading to a common understanding she or he can #urther divide the people
and generally harm the organi3ation and action.
In order to $e e##ective& an outsider has to $e thoroughly prepared $y gathering whatever
in#ormation is availa$le a$out the organi3ations and the local and natural realities within which
they are operating and that they want to change through their interaction.
It was pointed out that )*s can also help each other in their sel#1evaluation. -he same is
possi$le $etween communities and groups. E,perienced mem$ers o# one group can help others
in their sel#1evaluation. %uch interaction strengthens the lin+s $etween di##erent groups and thus
increases their 0oint strength.
(elf*Evaluation Is Possi+le )nly If the A, Is Rea'y for It
It was stated that all )*s do not recogni3e the need #or an honest sel#1evaluation. %ome o# them
consider it a waste o# time. -hey want to get along with action and see re#lection as separate
#rom it. 7or them& re#lection is unnecessary theori3ing that delays action. '# course& when ta+en
to an e,treme +ind o# @hair1splitting&@ re#lection can indeed delay actionA in #act& at times& it
$ecomes its su$stitute. But re#lection is a$solutely necessary 2in right measures4& especially to
avoid the other e,tremes o# activism.
%ome )*s might recogni3e the need #or sel#1evaluation $ut might not $e ready #or it& $ecause it
analy3es all aspects o# wor+ and relationships& and this can $e a very pain#ul process& especially
in the $eginning. It re/uires a certain sel#1con#idence& the a$ility to loo+ at onesel# critically and
to listen to criticism without getting de#ensive or aggressive.
It is only when at least some mem$ers o# the )* recogni3e the need #or a sel#1evaluation that its
process can $e started. )s the latter goes on& other mem$ers might also recogni3e its use#ulness
and importance and 0oin it.
(elf*Evaluation- An Illustration
)runa (oy shared with us the e,perience o# her group& the %ocial .or+ and (esource Centre
2%.(C4& with a sel#1evaluation process. 8er case study shows how& through it& changes too+
place in their understanding and analysis o# the reality around them& and their own role vis1B1vis
this reality.
%.(C started wor+ in 1C=. Initially it was primarily a group o# pro#essionals trying to provide
technical and managerial solutions to the pro$lem o# poverty and in0ustice. )t that time %.(C
did not wor+ e,clusively with the poor& nor did it have their organi3ation as its o$0ective. In the
course o# the #irst three to #our years& some /uestions cropped up in some o# the wor+ers: minds
a$out the larger reality& the community within which the )* should wor+& the ade/uacy o#
technical solutions& the role o# pro#essionals& and so #orth. -his /uestioning $y individuals within
the )* led to some creative tension and changes in the wor+& $ut #or another two years there was
neither a collective /uestioning nor a clearly e,pressed need #or evaluation. In 1C!& eight to ten
mem$ers started to concreti3e the issues& and a de$ate $egan within the group on the need #or
sel#1/uestioning. -his small group started meeting in#ormally to #ormulate the /uestions that
were in their minds. -hey re#lected on all issues $othering them and on the relationship o# this
/uestioning to their understanding& their wor+& and local reality. -hey also identi#ied pro$lems in
the #ollowing areas o# their wor+ and group #unctioning<
D Communication within the )* itsel# and $etween its mem$ers and local peopleA
D Di##erent +inds o# ine/ualities and di##erences in status within the )*A
D Concentration o# decision ma+ing in a #ew hands& and the need to create structures that
would ensure $roader participation and reduce the e,ercise o# in#ormal powerA and
D Place o# economic development and its relationship with politics& social change& and so
#orth.
-he group& small at the $eginning& gradually e,panded to reach eighteen to twenty mem$ers.
-hey once sat almost every day #or a$out si, wee+s during which their own wor+ was more or
less suspended. -his activity was not seen very #avora$ly $y some other )* mem$ers& $ut they
did not o$0ect to it. .atching cautiously& they even 0oined some o# the sessions& $ut distrustingly.
-he $all that had $een set rolling moved on. ?ater a group o# #orty had two #our1day sessions
with eight outsiders well +nown to them and who& it was #elt& would $e a$le to help them deal
with certain dilemmas #aced and /uestions they had regarding the nature and direction o# their
wor+& the role o# an institution li+e theirs& development programs versus organi3ation& and the
li+e.
-here was a tremendous heterogeneity among the mem$ers in terms o# their social and
educational $ac+grounds& understanding and articulation o# issues& and commitment to change.
-he pace o# discussions was there#ore slow& and everyone did not participate e/ually.
-he +inds o# /uestions raised and answers attempted are given here in )runa:s own words< @.e
demanded openness and a$ility to discuss even personal commitments and aspirations. .e $ro+e
the $arriers $etween our pro#essional and personal lives. .e reali3ed that our o$0ectives had
$een too general. .e narrowed them down. .e decided we should wor+ mainly with the poor.
.e #ormulated a decision1ma+ing process which was participatory. .e wanted a #orum in which
every wor+er could e##ectively ta+e part. .e decided we should evaluate ourselves 2our
attitudes& $ehavior& understanding4 once a year;how honest are we& how democratic& how open&
how caste1minded> .hat is our understanding o# issues> .e discussed /uestions li+e what is
more important #or a wor+er;a Ph.D. or a capacity to communicate with people and elicit
people:s participation> But this process o# personal evaluation when related to salary structures
was not very success#ul. %u$0ective #actors played too important a role and did not allow #or the
personal evaluation to $ecome operational in relation to 0udgments $y peers on one another:s
salaries. -his power was vested $y the group in its director& accepting its own #ailure.
@)lso at the village level we had tal+s with people who had participated in our programs. .e
met them at one o# the #ive #ield centers once a month on the new moon day and reviewed the
various programs. 5eetings were sometimes held with special1interest or program groups li+e
cra#ts group& health group& and so on.
@.e concluded that there was a role #or an institution li+e ours. .e discussed its role in
development& :agitation&: in trying to $ring a$out structural changes. .e also discussed whether it
was possi$le #or a development group li+e this to shi#t gear and go into organi3ational activity.
%ome #elt it could $e done& others that it could not& and should not.@
-he long tal+s o$viously led to several changes in their wor+& in the decision1ma+ing process&
and in interpersonal relationships. -hese changes led to the need #or more discussion and clarity.
In the end& a dialectical relationship seems to have $een esta$lished $etween action and
re#lection& theory and practice.
Perceive' A'vantages of (elf*Evaluation
)ccording to )runa& these sel#1evaluation sessions were e,tremely use#ul. )t the end o# it all&
most participants reali3ed that this +ind o# communication and openness is necessary #or
improving a group:s e##ectiveness and impact.
%el#1evaluation can help everyone to thin+ and learn collectively& to articulate $etter. I# carried
on sensitively& it can ma+e every participant more honest& sensitive& analytical& and open to
change. It changes every$ody:s awareness and consciousness& as well as people:s attitudes& and
helps them to cope $etter with con#licts.
%el#1evaluation can improve a group:s inner #unctioning $y creating $etter relationships $etween
the di##erent )* mem$ers. 'pen discussion on certain issues removes unnecessary
misunderstandings. By tal+ing #ran+ly& even a$out sensitive issues& people $egin to see and
appreciate others: viewpoints.
%el#1evaluation helps in evolving a common perspective& a shared commitment to action& and
thus trans#orms a loose group o# individuals into a cohesive and e##ective )*. )s the analysis o#
the group improves& it understands $etter the larger realities and the interaction o# its wor+ with
them. By ma+ing mem$ers critically conscious o# their actions& it improves $oth a group:s inner
#unctioning and the wor+ it does with people. )ccording to )runa& @an attempt to resolve our
own dilemmas and con#licts led to greater clarity.@
%uch a process alters the relationships within the group and the relationship o# the )* with the
people. Because the )* $ecomes a cohesive group and develops a certain #ocus& )* mem$ers
do not say di##erent things a$out their wor+& and this improves the )*:s image vis1B1vis the
people. -he misunderstanding or con#usion that people might have a$out the )*:s role& real
motivation& and so on is reduced when it develops an open dialogue with local people and also
involves them in the assessment o# the wor+ initiated.
%ystematic sel#1evaluation re/uires that )*s develop methods o# gathering and documenting
in#ormation and o# conducting #ree interaction and discussions and +eeping records o# these.
)*s also have to loo+ #or indicators o# consciousness and articulation. Because o# all these
conscious e##orts at evaluation& the )*s: wor+ improves.
In addition to improvement in the a$ove1mentioned areas& which are mainly intangi$le&
e,perience shows that sel#1evaluation improves the achievement o# tangi$le results. -his
happens $ecause action $ecomes much more relevant& conscious& and #ocused. )s part o# their
sel#1evaluation& %.(C also did /ualitative analysis and #ound that the #ormer had led to $etter
tangi$le results. 2-his was also Kurian:s e,perience in C('%%.4
(oe E.aples of &a' E.ternal Evaluations
.e also heard e,amples o# some $ad e,ternal evaluations conducted $y social scientists and
rural development and management e,perts& using the latest cost1$ene#it and social cost1$ene#it
analysis. In order to get a good analysis o# their wor+& C('%% got an evaluation done $y a well1
+nown organi3ation. )t the end o# the ela$orate /uestioning& data collecting& and processing&
what C('%% got was merely a description o# its wor+ without any analysis. -he evaluation
#ailed to provide any guidelines #or #uture action& which was the main purpose o# having it done&
and ended up giving C('%% a very good chit and a su$stantial $ill. %imilarly& some management
people had gone to %.(C to conduct social cost1$ene#it analysis& and its outcome was not
help#ul either& at least not to the )* and local people.
-he sharing o# these e,periences made us reali3e that there are no ready1made @scienti#ic@ tools
availa$le #or the evaluation o# e##orts to raise people:s consciousness and mo$ili3e them.
Esta$lished academic institutions cannot& #or o$vious reasons& $e e,pected to provide the
necessary help in this matter. )*s and mass organi3ations& together with some sensitive
academics& will have to evolve methods and tools #or assessing their wor+.