Level 2 Rubric: Learning Evaluation

Excellent

Moderate

Poor

Five Steps for Learning Evaluation (Coscarelli & Shrock, 2008, p. 4)

Analysis What is being tested? The effectiveness of the training
program at Starbucks six to nine years ago. Two choices of analysis:
1.
Performance
Tests
Observable
performance
on a rating
scale, and
these tests
were used at
Starbucks.

The instructor sets

aside a slow time at
work to test the
learner. There are
two to three people
observing the
learner's behavior,
who are experienced
instructors, and are
honest with no bias.
The instructors
watch the learner's
performance during
the test and rate
their performance
fairly. The observers
are all professional
and nice, so the test
is not as stressful.
The learner shows
them what they
learned in training
on how to make the
coffee drinks and
other tasks.

2. Top of the
Skills
Hierarchy
tests highlevel skills on
the hierarchy

The instructor tests

the learners on
higher skills from
the training they
went through to
make coffee drinks.
The test is not about
memorizing, so it

The instructor set

aside a time without
paying attention to
busy times at the
store. The instructor
only has one other
experienced
instructor with them
sometimes. Honesty
and bias can go
either way with the
instructors
analyzing the
learner's
performance. They
are professional but
show no emotions,
and it makes the
test stressful. The
learner tests their
performance for
them, and the score
can be fair or unfair
scoring. When there
are fewer
instructors during
the testing, then
there can be a gap
between the results
scoring the learner.
The instructor tests
the learner on
higher skills from
the training the
trainers went
through to make
coffee. The
instructor tests on

The instructor pulls

the learner aside
during a rush to do
their performance
test. The instructor
is the only one
analyzing the
learners
performance, so
honesty and bias are
big issues because it
becomes more of a
test of if the
instructor likes the
learner or not. The
instructor is not
professional because
no one is watching
their performance,
so the entire test is
stressful and
uncomfortable, and
it makes it highly
likely that the
learner will mess up
from their nerves.

The instructor does

not test the learner
on any higher skills
they learned from
the training. The
instructor goes
straight to
memorization

3. Validity
Does the test
measure
what it
intends to
measure?

focuses on how the

learner can apply
their knowledge to
the job, and other
skills higher on the
hierarchy. The test
shows the instructor
that the learner
understands more
than just how to
make the drinks.
The test is helpful
when the learner
wants to show their
competence in the
training and
knowledge they
attained. It shows
the instructor that
the learner does not
just know how to
perform, but they
are knowledgeable
and able to use it in
many areas. A
coffee tasting and
pairing will show the
instructor that the
learner can apply all
of the material
learned during the
test. The scoring is
accurate because
the instructor is
testing for the right
material.
The instructor tests
the learner on the
training; to see if
they learned the
information, and to
make sure the
training is effective
and valid for the job.
Experts look over
the test first to

memorization and a
couple of higher
skills. The test has
nothing to do with
memorization, but
the instructor is not
handling the testing
correctly. The
learner's
competence can be
shown with both
ways of testing, but
that is not the focus
of this type of test,
so the results and
scores are not
accurate for the
test.

testing, which is a
completely different
type of test. The test
and scores do not
apply because the
test was given in the
wrong manner.

The instructor tests

the learner on parts
of the training, but
missing some
important parts.
The instructor is
trying to see if they
learned the
information, and to
make sure the

The instructor tested

the learner on
random details or
facts about what to
do when they were
at work. The
instructor did not try
to find out how
much had been
learned, so they

legally claim that

the test is valid. The
test is for how much
was learned,
understood, and had
knowledge of the
material for making
drinks and more.

training the
instructor uses is
effective and valid.
Experts are
supposed to look
over the test first to
make it legally
valid, but this test
had not been made
valid yet. The test
did not do a great
job showing how
much was learned,
understood, and
had knowledge of
the material. It was
not a completely
valid test, so it
should not have
been used,
technically.

could not tell if it

was effective or
valid. Experts are
supposed to look
over the test first to
make it legally valid,
but this test was
never looked at as a
test, let alone a valid
test. The test did not
show how much was
learned, understood,
and had knowledge
of the material that
the learner attained
during training. The
test was not valid,
and it should not
have been used, or
even considered any
type of testing. It
was more of a
conversation with
random topics,
which did not relate
to the leaners
training.

4. Construction How to write the test elements and details?

Checklists
breaks down
the process
or product
into a series
of binary
choices, for
example,
present or
absent,
(Coscarelli &
Shrock, 2008,
p. 6). They
are not
scored like
rubrics,
which use
words and

The training has a

checklist of points to
cover in
performance tests.
The instructor
follows the checklist
to make sure
everything is
covered, and no
information is
missed on the
checklist. The uses
of binary choices are
available instead of
a number scale for
the performance
tests.

The training has a

checklist of a few
points to use in the
performance test,
but the checklist
does not cover very
much information.
The instructor may
end up using a
numerical scale to
rate evaluations
instead of the
binary rating for
performance tests.
The change will
make the
performance test
scores inconsistent,

The training has a

checklist with no
information or a
couple of points to
cover for a
performance test. It
is not enough
information for a
performance test.
The instructor does
not use the binary
rating for the
performance test.
The checklist is not
even usable in this
situation.

number
scales.
Multiple
choice items
above the
memory
level
(Coscarelli &
Shrock, 2008,
p. 6)

The test is a
multiple choice
which the instructor
will make higher
cognitive level test.
The test is more in
depth than a
memorization
multiple choice test.
The test has
questions about the
material that was
presented, but the
question will be
asked differently.
The learner has to
use their higher
cognitive thinking
skills to connect the
question with an
example they have
not experienced or
covered in the
training. The test
does not have
questions that are
memory based
because the
questions are
supposed to make
the learners think in
depth about each
topic.

and not reliable for

proof of
performance.
The test is a
multiple choice that
is supposed to tap
into the learners
higher cognitive
level. The test has
mistakes where
some of the
questions are
memory based
questions. The
questions do not
use the right
examples to
connect to the
information from
the training. The
learner does not
have to think as in
depth as they were
lead to believe.

The test is a multiple

choice that does not
go over the memory
level of thinking. So,
this makes the test
easy for the
learners, and they
do not have to use
cognitive skills to
figure out examples
for themselves. The
test does not help
the learners at work
because if a problem
occurs that they
have not
experienced, then
they are not
prepared to think
cognitively at a
higher level. The
test is dumb downed
because of the
individuals lack of
knowledge and
understanding of the
training.

5. Standard Setting Establish a legally defensible cut-off or

mastery score. Three approaches to a score that is standard:
Informed
Judgment

The instructor did

their research to
make an informed
decision on what the
standard score
should be for
mastery after
training. The

The instructor did

some research, but
is still unsure of
what to make the
standard score for
mastery after
training. They end
up using their

The instructor uses

themselves or
nothing as an
informed judgment
on what they
personally think the
standard score
should be for

Conjecture

Contrasting
Groups

judgment can be
backed up with
references and
individuals who
helped the
instructor make the
informed judgment
on the standard
score. The methods
they use give
learners a more
complete and
accurate standard
score.
At least two SME's
look over the
training and
calculate the
probability of an
individual with basic
competence can
pass each part of
the training for
mastery level. By
using a numeric
scale, the SME's can
judge what the
average mastery
score will be by
debate and
agreement. How in
depth the mastery
score level is
attained makes it
effective and legally
valid.

research and
personal opinion for
their decision. Their
judgment can only
be backed up with
references they
found, that created
the standard score.
The method they
used does not give
a completely
accurate standard
score for learners.

mastery after
training. The
judgment cannot be
backed up on paper
in any way. The
method of choosing
the score
themselves gives
the learners an
inaccurate standard
score for learners.

They had one to

two SMEs looking
over the training to
calculate the
probability of an
individual with basic
competence can
pass each part of
the training for
mastery level. The
bar is set lower than
it should because
there are not
enough opinions.
The numerical scale
is used, but they
are not as
experience, so the
master score is just
a decision or
agreement. The
mastery score level
can still make it
effective but not as
legally valid.

The instructor and

SME's gather
information together
about similar and
different tests and

The instructor
gathers a few
examples of
contrasting group
standard scores, but

There was only one

SME in training
looking over the
training to calculate
the probability of an
individual with basic
competence passing
each part of the
training for mastery
level. There is set
way too low for
competence
because there are
only one persons
opinions. They do
not pay attention to
the numeric grading
because of their lack
of experience, so the
mastery level score
is just a decision by
the one SME. The
mastery level
scoring is not
effective, and it is
definitely not legally
valid.
The instructor does
not gather
contrasting group
standard scores
because they think

test scores. They

contrast each of the
groups to find the
standard median
score. The different
examples and
standard scores give
all of them a better
idea on how to
change the mastery
score for the
training. The
contrasting groups
approach helps
learners because it
is fairer because it
shows where their
actual level should
be after training.

does not do a lot of

research for
multiple groups to
contrast. They need
to find the standard
median score, but
with little
information to
contrast group
scores, it is harder
to come up with a
more accurate
conclusion. The few
examples they
attained will at least
help to give them a
different
perspective and
idea about changing
the mastery score
for training. The
contrasting groups
approach helps
instructors and
learners, but there
is not enough
information to make
a concrete standard
score after training.

that they can choose

what they want.
There is no way to
find the median
score because of the
lack of contrasting
groups as examples
and comparisons.
The absence of
examples makes it
impossible to find
the standard score
of mastery level
after training. The
missing group
examples to contrast
makes the quoted
standard score by
the instructor is
inapplicable.