TRAINING EVALUATION & METRICS

Evaluation is the systematic process of determining the worth, value or meaning of any process or activity.

PURPOSE
FEEDBACK

CONTROL
PURPOSE OF TRAINING EVALUATION

RESEARCH

POWER

INTERVENTION

REASONS
ALIGN WITH BUSINESS NEEDS JUSTIFY THE INVESTMENT RIGHT DIRECTION IDENTIFICATION OF GAPSIMPROVING FUTURE TRAINING EFFORTS BUILD CREDIBILITY

BARRIERS
LACK OF TOP MANAGEMENT SUPPORT LACK OF SKILLS TO EVALUATE KNOWING WHAT CRITERIONS TO EVALUATE ON RISKY AND EXPENSIVE ENTERPRISE

TYPES OF EVALUATION OBJECTIVES

Formative
Formative evaluation provides ongoing feedback to the curriculum designers and developers to ensure that what is being created really meets the needs of the intended audience.

Summative
Outcome evaluation determines whether or not the desired results (e.g., what participants are doing) of applying new skills were achieved in the short-term

Process evaluation provides information about what occurs during training. This includes giving and receiving verbal feedback.

Impact determines how the results of the training affect the strategic goal

DIFFERENT APPROACHES
MODE / FRAMEWORK KIRKPATRICK CIPP(GALVIN) CIRO BIRKERHOFF Levels of evaluation Four levels: Reaction, learning, behavior, results Four levels: context, input, process and product Context, input reaction and outcome Six stages: goal setting, program design, program implementation, Immediate outcomes, intermediate or usage outcomes and impacts and worth Four sets of activities: inputs, process, output and outcome

SYSTEMS APPROACH(IPO)

KRAIGER, FORD AND SALAS A classification scheme that specifies three categories of learning outcomes(cognitive ,skill- based, affective) and proposes evaluation measures appropriate for each category of outcomes KAUFMAN ANS KELLER Five levels: enabling and reaction, acquisition, application, organizational outputs, and societal outcomes Identifies five categories of variable and relationships among them: secondary influences, motivation elements, environmental elements, outcomes , ability/enabling elements

HOLTON

KIRKPATRICK/ PHILLIPS MODEL

KIRKPATRICK-1959 LEVEL 1-4 PHILLIPS -1970 LEVEL 5 ROI

LEVEL I
REACTION

LEVEL2
LEARNING

LEVEL 3
BEHAVIOUR

LEVEL 4
RESULTS

LEVEL5
ROI

PROCESS
ROI RESULTS
LEVEL 5 : 5%-10%

LEVEL 4; 10%-20%

BEHAVIOUR

LEVEL 3:30%-40%

LEARNING REACTION & PLANNED ACTION

LEVEL 2: 40%-50% LEVEL 1: 90%100%

PROCESS

BEFORE

DURING
AFTER

Evaluation
Plan the Evaluation during Design of the Training Program Develop Data collection tools to be used in each stage

Reaction Level Evaluation Level I

Reaction - What Is It?

How favorably participants react to the training (Customer satisfaction)
Collects reactions to instructor, course, and learning environment Communicates to trainees that their feedback is valued Can provide quantitative information

Reaction - What It Looks Like

Feedback Questionnaire
Program Objectives Content Methods of delivery Course materials Instruction tools Assignments Media Facilitator Facilities Trainee Motivation to Learn

Reaction - How to Perform

Determine what you want to find out Design a form to collect/quantify reactions Do it Immediately- Give it tomorrow Develop acceptable scoring standards Follow-up as appropriate- Count Feedback to Feedback Utilise and incorporate

Questionnaire
Open Ended Questionnaire Check list Two way questions Multiple Choice Questions Ranking

Uses of Reaction Data

Monitoring customer satisfaction Identifying strengths and weaknesses Evaluating facilitators Evaluating facilities Evaluating planned improvements Linking with follow up data Marketing programs

Limitations
Subjective, based on the feelings at the time of testing Participants may be too polite or too rude It is also subject to misuse A good rating is no assurance that participants will practice what has been taught For those who believe, no proof is required For those who do not, no proof is sufficient

Factors associated with Rating

The time of the day that session is offered, participants desire to attend the training, the gender, physical attractiveness of trainer (Baldwin, 2004;Oliver & Sautter, 2005) Feedback system is inappropriate when they are used to evaluate learning in an environment where participants are expected to share their experiences and learn (Smith, 2004)

Learning Level Evaluation Level II

Objectives of L II Evaluation
Providing individual feedback Improving training program
Objectives, Content & Delivery

Evaluating instructors

What is learnt

Knowledge Skills Self-Concepts Traits

Motives

Measuring Learning with Tests

Based on medium
Paper pencil test, simulations , actual piece of work and computer based test

Based on test design

Oral exams, essay test, objective tests, normreferenced test, criterion-referenced test and performance testing

Tests to Measure Learning

Norm referenced tests
Compare trainees with each other rather than to specific instructional objectives

Criterion referenced test

An objective test with a predetermined cut-off score for specific instructional objectives

Performance testing
Allows trainees to exhibit a skill that was learned in the program

Measure Learning with Simulation

This involves construction and application of a procedure or task that models activity for the program being conducted They can provide accurate evaluation if the performance in simulation is objective & can be clearly measured Role Plays Business Games

Learning - How to Perform

Use a control group, if feasible Evaluate knowledge, skills, and/or abilities before and after Get 100% participation or use statistical sample Follow-up as appropriate

Exercise
1.Analysis of Feedback Forms 2.Analysis of Compiled Ratings

Behaviour Level Evaluation Level III

Behavior
Transfer of knowledge, skills, and /or abilities to the real world
Measure achievement of performance objectives

Involve the immediate supervisor/s Observe performer, first-hand Survey key people who observe performer Use checklists, questionnaires, interviews, or combinations

Organisation support and Transfer Climate

Organizational policies and practices Extent of encouragement to apply new learnings Consequences when applying Reinforcement and Coaching

Trainee Characteristics
Ability to learn
Aptitude & specific intelligence

Motivation to learn
WIIFM, belief in training, perceived need for KSA improvement, perceived back to job situation

Attitude
Job satisfaction, low organizational commitment, intention to leave

Personality (The BIG 5 Factors)

Initiative, Openness to experience, extraversion , agreeableness, conscientiousness

Demonstrated Capability - Trainee

Ability to transfer (self efficacy) Intention to transfer (WIIFM & perceived support) Initiation Partial transfer Conscious maintenance Unconscious maintenance

Behavior - How to Perform

Evaluate before and after training Allow ample time before observing Survey key people Consider cost vs. benefits
100% participation or a sampling Repeated evaluations at appropriate intervals Use of a control group

Data Collection Questionnaire

Action plan implementation Use of program materials KSA application Frequency of application Measurable improvements Improvements linked to program Monetary impact Barriers Enablers

Level IV Results

Results - What Is It?

Assesses bottom line, final results Definition of results dependent upon the objectives of the training program

Results - What It Looks Like

Depends upon objectives of training program
Quantify

Proof vs. Evidence

Proof is concrete Evidence is soft

Results - How to Perform

Use a control group Allow time for results to be realized Measure before and after the program Consider cost versus benefits Be satisfied with evidence when proof is not possible

L III & L IV Data Collection

Method L III L IV

Follow-up Questionnaires
Observation Interviews with Participants Action Planning Performance Contracting

Yes
Yes Yes Yes Yes

Yes

Yes Yes

Performance Monitoring

Yes

Tools for Evaluation

Tools/Learning Participants/Questionnaires or reports Manager questionaires or reports Reactions 4 4 Learnings 4 4 Behavious 4 4 Results 4 4

Written test or examination

Practical test or demonstration Customer survey Employee survey Interviews Observation on the job 4

4
4 4 4 4 4 4 4 4 4 4 4

Metrics

Metrics for measuring T & D efficiency

Areas Training Cost Training Hours Training courses Training satisfaction Training budget Training results Key Performance Indicators Company training expenditure (% of salaries & wages) Avg. no. of training hours per employee No. of courses offered No. of courses implemented Employee Satisfaction with training % of HR budget spent on training Avg. training costs per employee % of employees gone through training Avg. time to competence % employees reach competence after training % of employees completing the course compared to total no. of employees E-learning courses utilized % of e-learning pass rate

Training penetration rate E-learning training

I am Provided Effective Coaching By My Supervisor To Enhance My In-Clinic Performance

4 3.5 3 2.5 2 1.5 1
Total Responses Maxima Critica Intima

3.2

3.1

3.2

3.1

I get adequate support in handling customer queries

4 3.5 3 2.5 2 1.5 1
Total Responses Maxima Critica Intima

3.2

3.2

3.3

3.2

Post training data across batches Leaders Academy

Step 1: Be Supportive Step 2: Define the Topic and Need 7.00 6.00 5.00 4.00 3.00 2.00 1.00 0.00 Batch 8 Batch 1 Batch 2 Self Manager Direct Reports 7.00 6.00 5.00 4.00 3.00 2.00 1.00 0.00 Batch 8 Batch 1 Batch 2 Self Manager Direct Reports

Step 3: Establish the Im pact

Step 4: Initiate a Plan

6.00 5.00 4.00 3.00 2.00 1.00 0.00 Batch 8 Batch 1 Batch 2 Self Manager Direct Reports

7.00 6.00 5.00 4.00 3.00 2.00 1.00 0.00 Batch 8 Batch 1 Batch 2 Self Manager Direct Reports

Post training data across batches

Step 5: Get a Com m itm ent Step 6: Confront Excuses & Resistance 6 5 4 3 2 1 0 Batch 8 Batch 1 Batch 2 Self Manager Direct Reports 6.00 5.00 4.00 3.00 2.00 1.00 0.00 Batch 8 Batch 1 Batch 2 Self Manager Direct Reports

Step 7: Clarify Consequences

Step 8: Don't Give Up

7.00 6.00 5.00 4.00 3.00 2.00 1.00 0.00 Batch 8 Batch 1 Batch 2 Self Manager Direct Reports

7.00 6.00 5.00 4.00 3.00 2.00 1.00 0.00 Batch 8 Batch 1 Batch 2 Self Manager Direct Reports

Impact : Certification Metrics

Criteria for certification : Time Status
Oct 06 Nov 06 1. Develop and present district business plan to RBMs 2. Territory wise analysis as per case study model

3. Demonstrate business turnaround in two territories

Present L4 metrics

6 months

DM / HQ : Nagpur 2 Project PSOs chosen for project Before Enrolment (YTD Sept 06) Achievement Growth

RBM : After enrolment (YTD Mar 07) Achievement Growth

HQ - Bilaspur
HQ - Raipur

79%
98%

2% 6%

110% 106%

18% 10%

Impact : Advanced Retailing Skills (for Consumer Health Care)

Feedback From a PSO - L3 & L4 metrics
FEB 06
(Pre Trainng) Retail Calls Outlet Conversion Personal Order Booking Rs. Average booking per Outlet Rs.

APR 06
(Post Trainng)

IMPACT METRICS

242
8%

253
16%

More No. of retailers covered Improved Outlet Conversion

65,000 Rs.288

84,000 Rs.373

29.2% Increase in POB value

Enhanced

Order booking as %age of TGT

11.7%

15.3%

Growth

30%

Productive Calls : Increased from 3 to 5 amongst 10 Retailers covered

RETURN ON INVESTMENT
PARTICIPANTS INPUT IN LEADERSHIP PROGRAM OF MANAGER
PARTICIPANT ANNUAL IMPROVEMENT VALUE Reasons CONFIDENCE ISOLATION FACTOR ADJUSTED VALUE

11

$ 36000

Improvement in efficiency of group

85%

50%

$ 15300

42

90000

Turnover reduction

90%

40%

$ 32400

74

24000

Improvement in customer response time

60%

55%

$ 7920

55

2000

5% improvement in my effectiveness

75%

50%

$ 750

96

10000

Absentiesm reduction

85%

75%

$ 6375

Training Failure Analysis

Create Focus/ Build Intentionality
Trainees

Provide Quality Learning Interventions

Trainees

Support Performance Improvement

No

Reason

did not really need the training Training was focused on the wrong people Trainees were not prepared/motivated to learn Training was not aligned w/ performance needs Trainees were not prepared to apply learning on the job

could not learn material; instruction was not good enough Training design was flawed Exercises/simulatio ns were irrelevant Facilitator did a bad job

opportunity to use the training Trainees did not get support from manager when trying to use training Trainees gave up on new skills too quickly when running into obstacles Lack of peer support No incentive to use it Lack of feedback/ coaching when trying to use it

Thank You

Accelerate

Your partners in building Excellence

Remember learning to ride a bike?

Leveraging Great Learning

Learning x Follow-through = Results

The Literature on Follow-Through

Improvement correlates to follow-up
Goldsmith

Months of practice
Goleman

Involvement of manager
Brinkerhoff & Montesino

Why Different Results?

Background:

8,000 Fortune 100 managers All received 360 feedback and leadership training Leadership effectiveness evaluated18 months later
Little or no improvement Moderate improvement Significant sustained improvement

Three distinct groups:

Same course, same company, different results. Why?

The Difference is Follow-Through

No Follow Through
35 30

Consistent Follow-Through
60 50
percent

percent

25 20 15 10 5 0 -3 -2 -1 0 1 2 3

40 30 20 10 0

Less effective

No Change

More Effective

Some Follow-through
45 40 35 30 25 20 15 10 5 0

-3

-2

-1

0
No Change

percent

Less effective

More Effective

Consistent or periodic follow-up had a dramatic, positive impact.

Less effective
-3 -2 -1

No Change

More Effective

Goldsmith, M: Ask, Learn, Follow-up, and Grow, in Hesselbein et al: Leaders of the Future, 1996

Conclusions - 8 company study

Goldsmith and Morgan, Best Practices in Organizational Development, in press, 2003

Real leadership development is a process. Almost any follow-up is better than none. One of the greatest weaknesses in most training and development is the insufficient attention paid to follow-up. The biggest challenge for most leaders is not understanding the practice of leadership; it is practicing their understanding of leadership.

Months of Practice are Required

Why does emotional intelligence competence take months rather than days? Because the emotional centers of the brain, not just the neocortex, are involved. To master a new behavior, the emotional centers need repetition and practice.

The more often a behavioral sequence is repeated, the stronger the underlying brain circuits become.
At some point, the new neural pathways become the brains default option.
Goleman, Leadership that Gets Results Harvard Business Review March 2000

Manager Involvement
Learners who had pre/post course discussions with their managers (on new skills, applications, etc.) reported significantly higher skill levels and success.
Brinkerhoff & Montesino, Partnerships for Training Transfer, HRD Quarterly Fall 1995

What Hinders Follow-Through

Barriers to Transfer High vs. Low Performance Factors The Knowing/Doing Gap

Barriers to Transfer
Lack of reinforcement on the job Non-supportive organizational climate Learners: new skills are impractical, irrelevant Separation from instructional source Negative peer pressure

Broad & Newstrom, Transfer of Training, 1992

High vs. Low Performance Factors

High performance learners
Explored content before training Pre/post discussions with managers Clear idea on how to apply new skills Frequent practice after training

Low performance learners and their managers

Had none of the above factors supporting use of new skills
Feldstein & Boothman, In Action: Transferring Learning to the Workplace, 1997 ASTD

Final thoughts
Training should move from a service provider to an internal consultant- Passion Develop rapport proactively with internal KOLsspeak the customers language Move out of the comfort zoneInnovate and benchmark with other industries Adopt new technology @ speed of thought--ensure stakeholder buy-in Relentless focus on top line and bottom line Take pride in your roleyou are a Life Changer !...

HOW EFFECTIVE IS YOUR TRAINING PROGRAM?

Broad and Newstrom (1992) report studies have shown less than 30% of what is actually taught transfers to the job in a way that enhances performance.
Source : Broad, M., & Newstrom, J. W. (1992). Transfer of training: Action packed strategies to ensure high payoff from training investments. Reading, MA: Addison-Wesley.

THANK YOU

Research
If we knew what we were doing, it wouldnt be research. - Albert Einstein

The Indian Scenario a research

The sample size consisted of the following seven companies: Godfrey Phillips India Ltd. Mahindra & Mahindra Limited Indian Oil Corporation. Tata Power Ltd. Larsen & Toubro Ltd. Tata Motors Ltd. Johnson & Johnson Ltd. The Indian Hotels Company Ltd.

Training Evaluation - Kirkpatrick

Kirkpatrick's four-level model, each successive evaluation level is built on information provided by the lower level.

The Indian Scenario a research

Organisation: ITES Need: Team Leader Effectiveness Specific area evaluated: Team Huddles Process: Pre/Post Training scores- 8 E Outcomes: Improved scores Drop in absenteeism Reduced attrition

Training Evaluation Recommendations

Build capacity by doing research & benchmarking Train the Team to understand basics of TE Quantify information before intervention get stakeholder involvement Set specific targets for evaluation at all levels Decide on specific measurements Allocate resources

Training Evaluation Recommendations

Start small-- one course as a pilot Focus on a small sample size simplify Share results with trainers and the entire organization Celebrate success stories with stakeholders Design improvement plans

Parting Words
If you always do what you always did, youll always get what you always got.
- Anon

The Last Word

In the last analysis Management is practice Its essence is not knowing but doing Its test is not logic but results Its only authority is Performance Peter Drucker

Thank You