Evaluating

Learning and Development

Activities
Formal Report
(approx. word count 3.000 words)

Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May
21, 2016 1

Contents
Table of figures...................................................................................................... 3
Introduction........................................................................................................... 4
Methodology.......................................................................................................... 5
Nature of Evaluation.............................................................................................. 6
Purposes of Evaluation.......................................................................................... 7
Related Terms and Processes................................................................................. 8
Purposes of evaluation for 3 different stakeholders.............................................10
Return on investment and return on expectation and its measurement..........12
Evaluation Methods............................................................................................. 14
Evaluation Tools................................................................................................... 17
Conclusions.......................................................................................................... 21
References........................................................................................................... 22

Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May
21, 2016 2

Table of figures
Figure 1 Ripple Model of Evaluation
Figure 2 Phillips ROI model
Figure 3: The Logic of Training (Robert O. Brinkerhoff)

Introduction
Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May
21, 2016 3

Firstly, I will discuss nature and some of the purposes of conducting an

evaluation and how the term evaluation differs from terms such as validation,
assessment, testing, monitoring and review. I will then further elaborate on
some of the purposes and/or benefits of evaluation for different stakeholders.
Subsequently, I will try to explain the concepts of return on investment and
return on expectation and possible ways of measuring them. Afterwards, I will
present 3 different evaluation methods and I will wrap up this report by providing
various evaluation tools and their advantages and drawbacks.

Methodology
I drew on the following primary research:
Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May
21, 2016 4

Learning and Development Practice 2nd edition

ASTD Handbook
Telling Aint Training
CIPD material
How to Measure Training Results
Designing and Delivering Training
Current evaluation practices in our organisation
Discussion with colleagues from L&D Graeme Turnbull and Chris Evans
Discussion with Operations Trainer Ian Sommerville
I also drew on the following secondary research:

Online material on CIPD website

Nature of Evaluation
There are many definitions of the term evaluation. Lets look at some of
the definitions found in dictionaries of English language.
A judgment about how good, useful, or successful something is
Longman Dictionary of Contemporary English (5th Edition)
Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May
21, 2016 5

The process of studying all the available information about sb/sth and forming
an opinion about them/it
Oxford Business English

Beevers and Rea (2010) define the term evaluation as:

Measuring and analysing various aspects of L&D provision, with a view to
determining the effectiveness and value of the provision and informing decisions
about how it can be improved

Based on a fact sheet Evaluating Learning and Development from CIPD website
the term evaluation is defined in detail as follows:
The evaluation of learning and development (L&D) is the formal or informal
assessment of the quality and effectiveness of an employers learning and
development provision, usually by some measure of the merit of the provision
itself (the input, for example the quality of course content and presentation)
and/or by monitoring its impact (the outcomes, for example improved
skills/qualifications or enhanced productivity/profitability).
In general, learning is the desired goal of any training programme. However, if
learning does not create the desired change in behaviour in the workplace, we
have not successfully accomplished the overall goal of applying learning to the
workplace. This is sometimes quite difficult to measure in the first place. Lets
have a look at purposes of evaluation.

Purposes of Evaluation
There are many reasons for evaluating learning programmes. In Chapter IV
Measuring and Evaluating Impact of ASTD Handbook by Biech (2014), Donald L.
Kirkpatrick summarised some of them in the below list:
Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May
21, 2016 6

 to determine whether the particular programme should continue or be

dropped
to learn how a learning programme can be improved
to justify the training budget
to ensure learning compliance
to maximise the value of training
to align training with strategy

Beevers and Rea (2010) look at evaluation in terms of different stakeholders. In

any evaluation, there will be a learner, trainer and organisational interests
considered. They also list other internal and external partners, some of them are:

L&D and HR Team

Finance function
Management
Awarding, accrediting and regulatory bodies

Related Terms and Processes

I have tried to draw on some of the definitions of the term evaluation.
Lets discuss some other terms which are associated with evaluation but which
have slightly different meanings. Those terms are:
Validation
Assessment

Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May
21, 2016 7

 Testing
Monitoring and review
Return on investment (ROI)

Beevers and Rea (2010) define the above terms in a more detailed way.
Validation
Validation is usually included within evaluation process. Revisiting learning
outcomes at the end of a learning session or programme and making sure they
are closely tied in with organisational goals and especially achieved is just one
example of validation. The evaluation of a learning intervention must prove its
relevance, alignment and must be measurable (RAM approach). In order to
achieve that, all four of the Kirkpatricks evaluation levels should be established
as each bit of evidence can help to build the overall case of the programmes
effectiveness.
Assessment
Assessment tracks individual progress and can be measured by a range of
methods. It can be undertaken prior to, during or after a learning activity.
Assessment can be formal or informal depending on the learning outcomes.

Testing
Testing is similar to assessment. It can be described as an assessment activity
and its format (oral or written) will vary depending on what is tested. Stolovitch
and Keeps (2002) state the following key points about testing:
Testing is a natural part of learning. It helps both learners and trainers confirm
performance objective attainment or identify where something is missing and
requires corrective feedback.
Testing doesnt need to be threatening but sometimes it is.

Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May
21, 2016 8

 Testing is an excellent way to teach. It lets the learner try out her or his
learning with a bit of a challenge.
Testing requires that feedback is given following the test. It either confirms
objective attainment or offers corrective information specific to how the
learner performed on the test.
Because testing requires active learner engagement, it should be used
frequently. Meaningful engagement enhances comprehension and retention.
However, we need to be careful as testing frightens adult learners, particularly
those who may not have had great success with it at school.
Monitoring and review
Monitoring and review is a collection of activities to gauge learner progress
towards goals. Information gained from monitoring and review of learning
activities is often useful to the evaluation process. It identifies development
areas and whether any further learning intervention is needed.

Purposes of evaluation for 3 different stakeholders

The main stakeholders in the evaluation process are:
1. Learner
2. Trainer
3. Organisation
Lets have a look at some of the purposes and benefits of evaluation for the
above stakeholders.
Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May
21, 2016 9

Potential purposes (and benefits) of evaluation:

Learner

Enable learner to express their opinion and influence future development of

learning programme
Demonstrate a recognition of learners inputs
Helps to identify any further training needs and/or identify future talents
Enable learner to identify his/her preferred learning styles

Trainer

Feedback on trainers performance

Identification of areas of development
Challenges the trainer to think out of the box and maybe makes us rethink

our ingrained approach

Achievement of trainers own objectives
Establish whether learner enjoyed the learning and whether selected learning

methods were effective

Helps trainer to test out new techniques and methods
Helps to improve the overall learning experience
Trainer can justify the validity of the learning to management
Gives evidence towards trainers credibility

Organisation

Align with organisational goals

Financial expectations - ROI
Audit reasons
Increased productivity, decreased attrition, KPIs
Increases competitiveness
Enables the organisation to look for exceptional employees
Enables to maintain and gain professional qualification and accreditation
Establish costs of L&D
Supports the role of L&D function within an organisation

Those were just some of the purposes of evaluation. I will now discuss the terms
ROI and ROE.

Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May
21, 2016 10

Return on investment and return on expectation and

its measurement
Beevers and Rea (2010) define the term return on investment (ROI) as a
measure of the financial impact of training/learning activities on an organisation.
ROI is usually considered to be the highest level of evaluation. In order to
calculate the ROI, I need to be able to assess costs of training provision and its
financial benefits.
Measuring ROI should be a requirement in most organisations. ROI was
popularised by Jack Phillips. There are different formulas which can be used to
measure ROI. In simple terms, the ROI formula is:
(Return Investment)
Investment

Its typically expressed as a percentage or as a cost-benefit ratio. ROI can be

seen as an extension of Kirkpatricks evaluation model. Thus, ROI can be seen as
Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May
21, 2016 11

level 5 of Kirkpatricks model. Collecting level 4 data helps to convert them to

monetary values.

Return on Expectation
Beevers and Rea (2010) state that return on expectation (ROE) is more
concerned with stakeholder satisfaction and whether learning expectations of
key stakeholders have been met rather than monetary values.
According to a CIPD factsheet The value of learning: A new model of value and
evaluation, we should be asking the following key questions which underpin a
return on expectation approach:

What were the original expectations of organisation stakeholders for the

learning or training? Have those expectations since changed?

What changes have occurred as a result of the learning processes?
To what extent have stakeholder expectations been met?

Measures of ROE can be achieved through focused dialogue, real understanding

and mutual trust between key stakeholders senior management and L&D
function. Measuring ROE requires softer and qualitative data compared to hard
numerical data. CIPD has been championing this approach for some time now.
ROE is described in detail in one of CIPD publications The Value of Learning:
From return on investment to return on expectation by Valerie Anderso.

Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May
21, 2016 12

Evaluation Methods
There are various approaches (frameworks) to evaluation. I have listed
some of them below:

Ciro Model
ROI
The Ripple Model of Evaluation
Triangulation in evaluation
Kirkpatricks Model

Let us examine some of them in more detail. Simmonds (2002) explains in his
book Designing and Delivering Training so-called Ripple model of evaluation.
Figure 1 depicts visually the Ripple Model of evaluation.

Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May
21, 2016 13

Figure 1 Ripple Model of Evaluation

Simmonds (2012) illustrates learning evaluation as constantly moving, dynamic

and organic and not as something static or linear. The satisfaction of learners
needs as a requirement is in the centre. This is translated into learning which is
transferred to the workplace and changed job performance. Finally, its value is
determined to the wider organisation. So-called ripples also flow back towards
the centre. The learning will be constrained by organisational culture, financial
constraints and stakeholder expectations. The organisation will affect the scope
of change and learning in the department as well as on individual level.

I have briefly explained the concept of ROI by Phillips in the previous section.
Phillips explained his ROI model in his book How to Measure Training Results
(2002). Collecting level-five data (hard and soft) and converting them to
monetary value is quite demanding and complex. Phillips model is illustrated in
Figure 2 below.
Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May
21, 2016 14

Figure 2 Phillips ROI model

The most widely used evaluation model is Kirkpatricks four level of evaluation. It
consists of:

Level
Level
Level
Level

1
2
3
4

Evaluation:
Evaluation:
Evaluation:
Evaluation:

Reaction
Learning
Behaviour
Results

All four levels are described in Evaluating Training Programs: The four levels
(Kirkpatrick and Kirkpatrick, 2006). Most companies conduct the evaluation at
least on level 1 and level 2. They are generally less costly and easier to measure
(happy sheets, formal and/or informal assessment tests, quizzes, etc.) whereas
evaluation on level 3 and 4 requires more planning and is more difficult to
measure. It also requires some time in order to be measurable. Nonetheless, its
vital to have a system of evaluation in place which aims for all four levels.

Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May
21, 2016 15

Evaluation Tools
We have different ways to evaluate the learning session and determine if
the transfer of knowledge, skills and change of behaviour in the workplace has
been successful. Even though the change of behaviour as a result of learning
intervention is much more difficult to measure, we can assess the learners
progress (knowledge and skills) throughout the training as well as at the very
end. Assessment can be classified as part of evaluation (Kirkpatricks level 2). We
generally distinguish between formative and summative assessment. Under
formative assessment, we understand continuous assessment throughout the
learning session or programme. It is used to map the learners progress. The goal
of summative assessment is to evaluate student learning at the end of the
learning session against some standard or benchmark. In the more lecturerbased environment (such as school/uni environment), we can come up with the
following overview of different assessment tools.
Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May
21, 2016 16

Formative and Summative Assessment tools (source: Cambridge Community)

Lets again further examine testing. Stolovitch and Keeps (2002) state that tests
must be linked to overall objectives of the learning session and must be created
with caution as failing the exam may be associated by the learner with I am a
failure. However, tests are generally a good assessment/evaluation tool. Tests
throughout the training are used in my workplace and both declarative and
procedural knowledge is tested. In order to put the learners at ease, I do a funny
learning activity such bubble busters, who wants to be a millionaire, etc. before
every test.
Stolovitch and Keeps (2002) present a variety of ways how a test can look like.
Here are just some of them:

Binary test
Matching test
Multiple-choice test
Completion test
Short-answer closed question test
Open-ended essay test

Each of them has its advantages and drawbacks. It is good practice to mix them
up depending whether we want to test procedural or declarative knowledge.
After each learning programme, which takes 4 weeks, my learners fill in
immediate evaluation forms (Kirkpatricks level 1), so called happy sheets. This
questionnaire is designed to collect both quantitative and qualitative data. It
contains open, closed questions and different scales of 1 to 10. Nancy
Kristiansen (2014) in Chapter 28 of ASTD Handbook lists some of the benefits
and advantages of using a well-designed level 1 evaluation strategy:
Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May
21, 2016 17

Improve decision-making
Prepare for levels 2, 3, and 4 cross-functional strategies
Meet regulatory compliance requirements
Reinforce a culture of learning
Involve participants as contributors to the betterment of the training process
Inquire directly whether or not what was learned met participants
expectations
and objectives

Lets have a look at Kirkpatricks level 3 evaluation in greater detail. Level 3

gathers the information on how (and how well) the skills, knowledge and
attitudes, that were acquired in a learning programme, are being used in
workplace performance (Brinkerhoff and Mooney, 2014, p. 524). In one word,
level 3 is about the application. The below figure depicts the value to the
organisation through improved job performance as a result of a learning
intervention.

Figure 3: The Logic of Training (Robert O. Brinkerhoff)

Evaluation tools on level 3 can range from interviews, direct observation, indirect
observation (e.g. call monitoring), analysis of KPIs (analyses effects of behaviour
rather than the behaviour itself), surveys, the success case evaluation method
(Brinkerhoff, 2003, 2006). Currently, I utilise direct and indirect observation and
surveys (3 months evaluation form) as available evaluation tools on level 3.
Using surveys leave a lot to be desired, though. People may feel surveyed out
and resistant to taking the time to complete them. They are also subject to the
bias of people telling you what they think you want to hear. Typical answers are:
Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May
21, 2016 18

Of course, Im using my training, Im supposed to after all. Sitting with the

employee and watching his job performance directly provides me or team
manager

with

much

more

accurate

data

and

is

less

susceptible

to

misinterpretation bias. However, its impractical and time-consuming and the

person being observed might feel he or she is being watched. This way, its
closer to level 2 evaluation rather than a level 3. Choosing and scoring random
calls and giving immediate feedback is more appropriate for level 3. This is also
part of weekly PEP sessions conducted in my workplace. Lets look at the most
common question regarding the timing of level 3 evaluation.
How soon after training should we follow up with trainees?
Brinkerhoff and Mooney (2014, p.536) answer the question as follows:
It is a mistake to use a standard timeframe, such as after three months, six
months, nine months, and so forth. The right way to determine the correct lag
time prior to level 3 inquiry is to ask: How soon after the training is it reasonable
to expect that people should have been able to apply their learning? In some
cases, this may be a matter of days, such as with call-centre employees learning
a technique for routeing calls more effectively. In other cases, such as with
purchasing executives , it may be necessary to wait a full quarter. But always it
is wise to remember this guideline: most training, if its not applied in the first
two to three weeks after training, will probably never be used.

Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May
21, 2016 19

Conclusions
I tried to illustrate the importance and the benefits of having some kind of
framework for evaluation in place. Failing to so will have drastic consequences
on the overall business unit/organisations performance in todays highly
competitive environment. I hope this report will provide some valuable insight for
other L&D practitioners in my organisation as well and further improve and
extend the evaluation system we have in place.

Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May
21, 2016 20

References
Books
Biech, E., ed. (2008). ASTD Handbook for Workplace Learning Professionals.
Alexandria, VA: ASTD Press.
Beevers, K. and Rea, A. (2010). Learning and development practice. London:
Chartered Institute of Personnel and Development.
Kirkpatrick, Donald L., and James D. Kirkpatrick. 2006. Evaluating training
programs: The four levels. 3d ed. San Francisco: Berrett-Koehler
Stolovitch, H. and Keeps, E. (2002). Telling ain't training. Alexandria, VA: ASTD.
Simmonds, D. (2003). Designing and delivering training. London: Chartered
Institute of Personnel and Development.
Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May
21, 2016 21

Phillips, J. and Stone, R. (2002). How to measure training results. New York:
McGraw-Hill.
Online
CHARTERED INSTITUTE OF PERSONNEL AND DEVELOPMENT. (2015) Evaluating
learning and development [online]. Factsheet. London: CIPD.
Available at http://www.cipd.co.uk/hr-resources/factsheets/evaluating-learningtalent-development.aspx (Accessed 20 May 2016)
http://www.cipd.co.uk/toolclicks/learning/training-tools/evaluatingtraining/default.aspx
CHARTERED INSTITUTE OF PERSONNEL AND DEVELOPMENT. (2015) Evaluating
learning and development [online]. Factsheet. London: CIPD.
Available at http://www.cipd.co.uk/hr-resources/factsheets/costing-benchmarkinglearning-development.aspx (Accessed 20 May 2016)
http://www.trainingzone.co.uk/deliver/coaching/cipd-tells-trainers-to-move-awayfrom-traditional-evaluation (Accessed 20 May 2016)
CHARTERED INSTITUTE OF PERSONNEL AND DEVELOPMENT. (2007) The Value of
Learning: From return on investment to return on expectation. Factsheet. London:
CIPD
Available at Available at http://www.cipd.co.uk/hrresources/factsheets/evaluating-learning-talent-development.aspx (Accessed 20
May 2016)
http://cambridge-community.org.uk/professional

development/gswafl/index.html

(Accessed 20 May 2016)

Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May
21, 2016 22