MEASURING THE IMPACT OF

JUDICIAL TRAINING
INPROL Consolidated Response (07-005)

With contributions from With contributions from Greg Gisvold, Mira Gur-Arie, Renee Dopplick,
Meghan Stewart, Linda Bishai, Sermid Al-Sarraf, Andrea De Maio, Colette Rausch, Ab Currie,
Patrick Murphy, William Brunson, Irene-Maria Eich, Carl Baar, Livingston Armytage, Richard
messick, Karen Widess

Prepared by Leigh Toomey

 MEASURING THE IMPACT OF JUDICIAL TRAINING
INPROL Consolidated Response (07-005)
July 14, 2007

Submitted by: Libor Chlad, EUJUST LEX Program, Council of the European Union

Drafted by: Leigh Toomey, INPROL Rule of Law Facilitator

With contributions from:

1. Greg Gisvold, Senior Democracy and Governance Specialist, Chemonics International

2. Mira Gur-Arie, Director, International Judicial Relations, United States Federal Judicial
Center
3. Renee Dopplick, Research Assistant, International Network to Promote the Rule of Law
4. Meghan Stewart, Senior Peace Fellow, Public International Law & Policy Group
5. Linda Bishai, Senior Program Officer, Education Program, United States Institute of
Peace
6. Sermid Al-Sarraf, Executive Director, International Institute for the Rule of Law, Iraq
7. Andrea De Maio, Researcher for Peace Building Unit, International Development Law
Organization
8. Colette Rausch, Deputy Director, Rule of Law Program, United States Institute of Peace
9. Ab Currie, Principal Researcher, Access to Justice and Legal Aid, Research and
Statistics Division, Department of Justice, Canada
10. Patrick Murphy, Senior Rule of Law Advisor, United States Agency for International
Development, Russia
11. William Brunson, Director of Special Projects, National Judicial College, United States
12. Irene-Maria Eich, Course Coordinator and Director, Training Unit, Zentrum für
Internationale Friedenseinsätze (ZIF)
13. Carl Baar, Professor Emeritus, Department of Political Science, Brock University
14. Livingston Armytage, Director, Centre for Judicial Studies, Australia
15. Richard Messick, Senior Public Sector Specialist and Co-Director, Law and Justice
Thematic Group, World Bank
16. Karen Widess, Head of Program Development, International Development Law
Organization

The full text of the responses provided by these INPROL members can be found at
http://www.inprol.org/node/1991. INPROL invites further comment by members.

Note: All opinions stated in this consolidated response have been made in a personal capacity
and do not necessarily reflect the views of particular organizations. INPROL does not explicitly
advocate policies.

INPROL is a project of the United States Institute of Peace with facilitation support from the Center of Excellence for Stability
Police Units, the Pearson Peacekeeping Centre, and the Public International Law & Policy Group.
 MEASURING THE IMPACT OF JUDICIAL TRAINING

Query:

What factors must be taken into consideration when evaluating a judicial training course,
particularly in a country emerging from conflict? Are there any performance indicators
that can assist in measuring the impact of judicial training? What are the unique
challenges involved in evaluating judicial training?

Response Summary:

Measuring the impact of a judicial training program involves making an assessment of

how the training improved the participants’ knowledge, skills, attitudes and overall
professional competence to administer justice. Given the complex and inherently
subjective nature of judicial work, no single performance indicator can accurately
measure these changes. Instead, evaluators must carefully select a range of indicators
to measure both the outputs and outcomes of the training, relying on both objective and
subjective indicators. A thorough assessment of judicial training will include:

 Baseline Assessment: Appropriate baseline indicators of judicial performance must

be established before the training. In countries emerging from conflict, this baseline
will often be quite low and difficult to discern because decisions reached by individual
judges or courts may not have been published and language barriers can make it
difficult for outside evaluators to gather relevant data. Nonetheless, an initial
assessment should identify the weaknesses of the judges to be trained both to
design the program and to perform an evaluation at the end of the process.

 In-Training Evaluations: Once baselines are established, it is important to gather

data on the training as it is taking place. This could include evaluation forms given to
participants after units are completed, tests to measure knowledge and retention of
the material (although this may be a sensitive subject when dealing with
professionals who consider themselves to be at the top of their field), and interviews
with key experts who may sit in on or actually conduct the trainings.

 Post-Training Analysis: After training is completed – or simultaneously if the

trainings are conducted occasionally over a long period of time – external factors
such as case processing time or comparative evaluations of written judgments can
be the most effective means of assessing the performance of judges.

 Outputs and Impacts: It is also important to keep in mind at each stage of the
evaluation process the difference between measuring outputs (e.g. numbers of
participants, days of sessions, etc) and impacts (such as whether judges have
actually become more knowledgeable, improved their rulings, or are acting with
greater integrity) which reflects the ultimate goals of the training. The former are
easier to measure and speak more to the effectiveness of a project’s administration
(and is thus also important to track fiscal responsibility in the provider). The latter are
far more subjective and must be considered in the context of the myriad factors

July 2007 INPROL Consolidated Response (07-005) Page 1 of 9

 beyond individual competence (corruption, lack of resources, lack of political will) that
can prevent even the best judges from having an impact on improving the justice
system.

 Best Practices and International Standards: There are many resources available
on design and evaluation of rule of law training generally, some of which are listed
below. Moreover, evaluators of judicial training must be familiar with the relevant
international standards that impact the work of the judiciary. These standards
provide important guidance for assessing judicial performance. Among the most
important of these international standards are the Basic Principles on the
Independence of the Judiciary, The Bangalore Principles of Judicial Conduct, the UN
Convention against Transnational Organized Crime, and the UN Convention Against
Corruption. Other relevant standards are found below in the compilation of
resources.

Following is an outline of the key components of a judicial training evaluation program.

Establishing Performance Baselines:

During periods of conflict, the judicial sector often suffers more than other forms of
government administration. Judges are dispersed; those that remain receive little
training from the beginning of the conflict; and few gain knowledge of current
international standards. In addition, legal (and primary) education is often weak or non-
existent; therefore recruiting new judges that are competent to carry out the complex
task of issuing consistent, reasoned judgments is a challenge.

A recent evaluation by the International Development Law Organization (IDLO) of its

judicial training programs in several developing countries, including the Democratic
Republic of the Congo (DRC) (see the compilation of resources section below) bears
this out. In the DRC, the evaluators noted that the effects of war and the resulting
inadequacy of court infrastructure and use of obsolete equipment; deficiencies in
existing texts and laws; little investment in the justice sector (including payment of a
living wage to judges), and a general lack of confidence in the administration of justice
prohibited the transfer of learning following extensive training by IDLO. A pre-training
assessment should take into account these unique factors and the structure of the
justice system itself to ensure that training is effective and can be accurately measured.
Key considerations include:

 Legal System: Any pre-training assessment must first consider the different
attributes of the legal system in question. For example, a judge working in a country,
which applies the civil law, requires training in inquisitorial, rather than adversarial,
trial procedure. In societies which maintain both a formal justice system and an
informal means of resolving disputes, judicial training might need to cover customary
systems, at least in a rudimentary way, in order to give judges an understanding of
how the systems interact. The evaluation of the training should therefore account for
how well the training accounted for these needs.

 Broad Scope of Judicial Work: In order to perform effectively, judges need to

develop and maintain knowledge and skills in a variety of areas, including legal
research, analysis and writing; civil and criminal trial procedure; the rules of
evidence, and judicial ethics. In a country emerging from conflict, this may also

July 2007 INPROL Consolidated Response (07-005) Page 2 of 9

 require training on subjects such as international human rights, with a focus on due
process and fair trial standards; international humanitarian law, and anti-corruption
measures.

 Flexibility in the Face of Reform: As the legal system develops, so too will the
range and complexity of legal matters which judges will encounter in their work.
Unlike other areas of rule of law training, such as corrections or police training whose
subject matter is relatively static and discrete, a judicial training curriculum must be
regularly updated to reflect legal developments at the local, state or provincial,
national, regional and international levels in a wide range of legal subjects.

 Trainers: A training design must address the supply side as well by selecting
trainers that are mutually acceptable to both donors/programmers and the judges to
be trained. Failure to do so can influence the effectiveness of the training. Several
INPROL members who responded to this query pointed out that judges are,
understandably, often unwilling to participate in training unless the trainers have a
high level of expertise and can command respect in judicial circles. This may
preclude administrative experts who have valuable knowledge about court
operations but are not seen by participants as appropriate mentors. The trainer’s
background and understanding of the cultural, historical or religious context in which
training takes place is critical. For example, as one practitioner noted, a trainer with
expertise in Sharia Law is likely to be more effective in transferring skills to judges in
an Islamic country.

Measuring the Training Program Output:

To evaluate the extent which a judicial training program effectively meets the needs of
the participants and the objectives of the trainers, two main factors come into play:
evaluation of the process and evaluation of the impact or result. Each may in turn be
measured by both objective and subjective criteria. Given the difficulty of measuring
many of the components of judicial training, a mix of methods and criteria can help to
enhance the reliability of evaluation criteria.

Process Indicators: To measure the effectiveness of the training in serving the

interests of the participants one can use “Process Indicators” to measure elements
internal to the project and evaluate whether the training is fulfilling its terms of reference.

 Quantitative Process Indicators – First, one can measure the quantitative features
of the training – such as the number of judges trained, the number and length of
training sessions conducted, the number of materials distributed, the schedule and
length of training, etc. Another example would be an indicator measuring judges’
participation in training, in terms of whether a specific minimum threshold for
attendance was met. These indicators are objective and easily quantified, and
usually allow the evaluation question to be answered with a “yes” or “no” response.
An evaluator will also want to know whether the training was conducted on schedule
and within budget. In that case, the indicators are the date by which the training was
to have been completed and the actual cost of the training. An example of a training
evaluation checklist is found in the compilation of resources section below.

 Qualitative Process Indicators – A qualitative approach to process indicators

would be to measure the appropriateness and depth of course content and materials

July 2007 INPROL Consolidated Response (07-005) Page 3 of 9

 and appropriateness of and receptivity of course participants to the trainers. For
example, evaluators of judicial training commonly ask judges in a survey completed
after the training whether they were satisfied with the training and whether they felt
that the training made a difference to their knowledge, skills and attitudes (the
indicator being the number of affirmative responses). This is sometimes extended to
questions on whether the judges intend to make improvements in judicial service
delivery as a result of the training. Sample evaluations of judicial training can be
found below in the compilation of resources section. These qualitative process
indicators allow evaluators to measure perceptions of the training value and, if
necessary, to make appropriate changes in future training sessions.

Course materials should also be reviewed for scope and compatibility with other
necessary subjects or priority areas of reform. Questions to ask are:

 Were the materials broad enough to cover the full scope of the judicial enterprise?
 If the materials are broad in scope, was enough time spent to absorb each?
 What other trainings have been given or are planned, and do they dovetail with the
particular training being evaluated?
 How does the training correlate with other rule of law initiatives being undertaken in
the country or region in question?

Impact Indicators: The evaluation of judicial training is not complete until both the
outputs and the impacts of the training have been assessed. Evaluation of the
outcomes or impact of judicial training involves consideration of its longer-term effects,
particularly the improvement in the way that judges perform their work as a result of the
training and how that change contributed to judicial reform. These indicators are known
as “Impact Indicators” because they measure elements external to the project and how
those elements contribute to enhancing the quality of justice.

For example, an evaluator will consider whether there has been a change in judicial
performance by looking at factors such as:

 whether there is a higher quality of judgments;

 the number of identified behavioral changes attributed by participants to the training;
 the results of surveys or interviews completed by those who regularly interact with
judges and are in a position to offer an impartial review of the change in the judges’
behavior;
 whether there has been a change in judicial performance in crucial areas such as the
reduction of impunity and injustice, the administration of justice, judicial
independence and accountability, respect for human rights, and equality before the
law. Specific metrics for each of these are provided in the “Metrics Framework for
Assessing Conflict Transformation and Stabilization” developed by the United States
Institute of Peace.
 whether the change is consistent with international best practices and benchmarks.
For example, by comparing behavior with measures used by Transparency
International in its Corruption Perceptions Index (whether there have been any
reported incidents of judicial corruption, public perceptions of judicial integrity etc).

That said, it is extremely difficult to determine exactly what constitutes improved

performance, given the inherent subjectivity of assessing the quality of judgments
rendered by judges. It is also difficult to isolate the impact of training on judicial reform

July 2007 INPROL Consolidated Response (07-005) Page 4 of 9

and the rule of law, particularly if it involves just a single training course. It is therefore
considered best practice to use a combination of indicators to obtain the most accurate
results possible.

Quantitative Impact Indicators -- The data used to measure the impact of judicial
training can be obtained in a number of ways, including:

 Comparison of the results of the pre-training needs assessment and any

examinations conducted post-training (particularly in legal systems where judicial
promotion depends on taking an examination, which presumably ensures that the
training participants attempt to provide the best answers possible);

 Answers given in questionnaires completed by the judges after the training; and
 Judicial management data such as court statistics.

Depending on the specific topic of the training delivered, evaluators can also examine
case statistics to see if there has been any discernible change between the pre and
post-training periods. This would include statistics on conviction rates; the number of
new cases each year and the number of case disposals; case processing time; the
number of appeals and the percentage of successful appeals, as well as the number and
nature of complaints against the judiciary and their outcomes. This type of
measurement can be used to complement (and perhaps contradict) the subjective
reporting intrinsic in surveys or questionnaires completed by trainees themselves.
However, the accuracy of this method depends upon reliable data being regularly
compiled and available from court administrators. This may rarely be the case in a
country emerging from conflict, but will be vital to develop if international efforts are to be
sustainable.

Qualitative Impact Indicators -- Experience in evaluating judicial training has yielded a

number of possible subjective methods, which can be used by evaluators to measure
the impact of the training. These include:

 Surveys (self, peer and external).

 Interviews with training participants. Depending on the number of judges in the

training, evaluators can undertake a sampling set of interviews of participants, asking
them whether and how they are using the training. Examples of questionnaires used
in this type of survey can be found below in the compilation of resources section of
this consolidated response.

 Key informant interviews. Evaluators can also conduct what is known as “key
informant interviews” of court users which, depending on the type of legal system,
might include members of the public, court employees and clerks, notaries, lawyers
and other judges who did not participate in the training. These individuals often have
the unique opportunity to observe and offer valuable insights regarding actual
changes in judges’ ability, decorum, behavior and skills. A combination of asking the
participants themselves and a control group in a position to interact with judges can
provide useful data.

 Appellate review. If the jurisdiction has a reasonably sophisticated appellate

structure, a survey and/or interviews with higher court judges or a review of higher

July 2007 INPROL Consolidated Response (07-005) Page 5 of 9

 court statistics can produce data about the improvement of lower court decisions and
whether there has been a reduction in the number of decisions overturned. If the
judges who participated in the training are members of the highest national court, or
the appellate courts do not exist or produce irregular data, this will not be a suitable
method of measuring the impact of judicial training.

 Observation and appraisal of judicial performance by experts. The experts (either

lawyers or judges with no connection to those trained) consider matters such as the
awareness and enforcement by trained judges of time limits for accused persons to
be brought before a court; whether the trained judges properly apply rules and
standards for assessing whether evidence has been illegally obtained, and whether
the trained judges ensure that the human rights of those charged with a crime are
fully respected (the right to counsel etc).

 Focus groups. Another method of measuring impact, albeit a less reliable one, is to
ask select members of civil society, community representatives and public interest
groups about their impressions of the performance of the judiciary after the training
and their confidence in the integrity of the judicial system. The criteria used to
measure satisfaction with judicial services would include the degree to which the
judge in question protects human rights, the judge’s accessibility, openness,
efficiency, transparency and conduct. It may not, however, be easy to select
appropriate representatives from these groups, nor to satisfy statistical validity in how
that selection was made. This method is also highly subjective in that the responses
of those surveyed may reflect their own biases.

Other Challenges of Evaluating Judicial Training:

Evaluators of judicial training will also face a number of methodological and other
challenges.

 Subjective Nature of Judicial Practice: Many aspects of judicial practice are

subject to multiple interpretations, such as whether a judge has correctly interpreted
a code or case law or assessed the credibility of a witness during a trial. Subjective
interpretations of performance should be tempered with more objective statistical
criteria to gain a more accurate picture of performance.

 Judicial Independence: There are concerns that efforts to assess the impact of
judicial training may undermine or threaten judicial independence. For example, a
judge who issued an unpopular ruling may face allegations of incompetence, as
evidenced by a negative assessment of his or her judicial performance. To some
extent, this is unavoidable as judges require training, and evaluation is a critical part
of that process. Where possible, results of an evaluation should only be made
available to participants and their supervisors, rather than to the public or other
branches of government. Some countries have established judicial training programs
under the auspices of an independent national judicial training center to ensure
critical evaluations while maintaining judicial independence.

 External Factors: External factors such as ongoing conflict, widespread corruption

or years of neglect in reforming the legal system often minimize the impact of an
otherwise successful training program, particularly in a country striving to emerge
from conflict. It is usually impossible to isolate the impact of training and to show a

July 2007 INPROL Consolidated Response (07-005) Page 6 of 9

 direct cause-and-effect relationship between the training and the change in behavior
of participants. If changes do occur in the performance of judges, they may be the
result of a cumulative effect of training, discussions with colleagues, exposure to new
ideas in the media and ongoing evolution of personal views and positions.
Nevertheless, an evaluation which indicates that a judicial training course contributed
toward positive changes is still a worthwhile outcome for any program.

 Long-term Nature of Judicial Reform: The behavioral change that is the basis for
sustainable judicial reform takes time to register an impact. While mileposts along
the way are necessary, the impact of a judicial reform project may not be measured
in time frames of a year or two. Rule of law professionals should not be tempted to
concentrate solely on quantifiable outputs (such as the provision of legal textbooks
and materials) at the expense of qualitative measures taken to improve judicial
performance over the long term.

As this Consolidated Response indicates, judicial training is one of the most difficult
forms of training to evaluate. INPROL would welcome further comment by members on
their experience in designing an evaluation process and selecting performance
indicators to assess judicial training, particularly in countries transitioning from war to
peace.

________________

Compilation of Resources:

This Consolidated Response draws from many of the following resources, which are
useful reference tools for policing practitioners. All listed documents with a hyperlink are
uploaded to the INPROL Digital Library.

GENERAL LITERATURE ON EVALUATION

 “USAID Tips: Performance Monitoring and Evaluation: Conducting a Participatory

Evaluation”, United States Agency for International Development, 1996.
 “USAID Tips: Performance Monitoring and Evaluation: Conducting Key Informant
Interviews”, United States Agency for International Development, 1996.
 “Metrics Framework for Assessing Conflict Transformation and Stabilization”, July
2007, United States Institute of Peace.

RESOURCES ON JUDICIAL TRAINING

Selected International Standards

 Convention for the Suppression of the Traffic in Persons and of the Exploitation of
the Prostitution of Others (1949)
 The Geneva Conventions (1949)
 Convention relating to the Status of Refugees (1951)
 United Nations Basic Principles on the Independence of the Judiciary (1985)
 United Nations Convention against Transnational Organized Crime (2000)

July 2007 INPROL Consolidated Response (07-005) Page 7 of 9

 The Bangalore Principles of Judicial Conduct (2002)
 United Nations Convention Against Corruption (2003)

Best Practices

 “Training: Judicial Education/Adult Education Project (JEAEP)”, Judicial Education

Reference, Information and Technical Transfer (JERITT) Project, 1992 (This
resource contains examples of participant questionnaires and training evaluation
checklists).
 “Devising Measures to Evaluate Judicial Reform”, Livingston Armytage, May 1998
(available at http://www.educatingjudges.com) (This Consolidated Response relies
extensively on this source of information and INPROL extends its thanks and
acknowledgment to the author).
 “USAID Latin America: Judicial Training and Justice Reform”, United States Agency
for International Development, 1998.
 “Diagnosing Judicial Performance: Toward a Tool to Help Guide Judicial Reform
Programs”, Linn Hammergren, World Bank (draft paper), 2002.
 “Human Rights in the Administration of Justice: A Manual on Human Rights for
Judges, Prosecutors and Lawyers”, International Bar Association and Office of the
United Nations High Commissioner for Human Rights, 2003.
 “Evaluating Judicial Education Organizations: What Can and Should be Measured?”,
Commonwealth Judicial Education Institute, 2004.
 “The Quality of the Training of Magistrates and Common European Standards for
Judicial Training”, Seventh Plenary Meeting of the Lisbon Network, 2005.
 “United Nations Primer for Justice Components in Multidimensional Peace
Operations: Strengthening the Rule of Law”, United Nations Department of
Peacekeeping Operations, 2006.

Related INPROL Materials

 INPROL Consolidated Response on Judicial Independence and Financial

Disclosure, May 2007.
 INPROL Discussion Forum on National Legal Training Centers, June 2007.
 INPROL Consolidated Response on Planning and Evaluating Corrections Training,
September 2007.

EXAMPLES OF EVALUATION TOOLS

 Judicial Training Needs Assessment

 Sample Judicial Training Course (in Spanish)
 Potential Topics of Judicial Training
 Evaluation of a Judicial Training Course
 Evaluation of Judicial Training Courses Developed by the IDLO (This resource
includes examples of evaluation questionnaires. The evaluation covers 6 country
programs (e.g. Democratic Republic of the Congo, from p.26 of Volume II).

USEFUL INTERNET RESOURCES

 American Bar Association (CEELI Judicial Reform Index)

(http://www.abanet.org/rol/publications/judicial_reform_index.shtml)
 Canadian Judicial Council (http://www.cjc-ccm.gc.ca)

July 2007 INPROL Consolidated Response (07-005) Page 8 of 9

 Centre for Judicial Studies (Australia) (http://www.educatingjudges.com)
 Commonwealth Judicial Education Institute (http://cjei.org)
 Federal Judicial Center (United States) (http://www.fjc.gov)
 École Nationale de la Magistrature (France) (http://www.enm.justice.fr)
 European Judicial Training Network (http://www.ejtn.net)
 International Development Law Organization (http://www.idlo.int)
 Judicial Education Reference, Information and Technical Transfer (JERITT) Project
(http://jeritt.msu.edu)
 Justice Studies Center of the Americas (http://www.cejamericas.org)
 Lisbon Network (part of the legal cooperation program of the Council of Europe)
(http://www.coe.int/t/dg1/legalcooperation/judicialprofessions/lisbon/default_en.asp)
 National Association of State Judicial Educators (United States) (http://nasje.org)
 National Judicial College (United States) (http://www.judges.org)
 National Judicial Institute (Canada) (http://www.nji.ca/nji/index.cfm)
 Transparency International (http://www.transparency.org)
 United Nations Asia and Far East Institute for the Prevention of Crime and the
Treatment of Offenders (http://www.unafei.or.jp/english/index.htm)
 United Nations Interregional Crime and Justice Research Institute
(http://www.unicri.it)
 United Nations Office on Drugs and Crime (http://www.unodc.org)
 World Bank (http://www.worldbank.org)
See also:
http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTLAWJUSTINST/0,,contentM
DK:20756999~menuPK:1990189~pagePK:210058~piPK:210062~theSitePK:1974062,0
0.html

Note: All opinions stated in this consolidated reply have been made in a personal
capacity and do not necessarily reflect the views of particular organizations.
INPROL does not explicitly advocate policies.

Information:
New Queries: To send a new query, please send an email to inprol@inprol.org.
Documents: To submit a document to INPROL, please login to INPROL and visit
http://www.inprol.org/uploadcontent or send an email (with the document attached) to
inprol@inprol.org.

July 2007 INPROL Consolidated Response (07-005) Page 9 of 9