Summative Assessment:

Rubrics and Tests

Effective Teaching and Learning
Baker College

Outcomes

Apply a systematic process for creating a

test blueprint
Identify attributes of effective test questions
Explain the advantages and disadvantages
of different types of test questions
Assess the quality of tests and test items
Create samples of effective questions

What type of assessment?

Procedural knowledge
Declarative knowledge

Test Writing Process

Planning the test

Writing test items
Selecting test items
Formatting the test
Assessing the test
Revising the test
Using the test
After the test
4

Planning the Test

Content Blueprint
Learning outcomes
Weight

Length
Types of items
Number of items

Test Blueprint
Learning outcomes
Apply a systematic process for creating a test
blueprint
Identify attributes of effective test questions

Weight

Explain the advantages and disadvantages of

different types of test questions
Assess the quality of tests and test items
Create samples of effective questions
6

Types of Items

Recognition

True-false
Multiple-choice
Multiple-answer
Matching
Ordering

Recall
Short Answer
Completion
Essay

Average Response Time

Item Type
True-false

30 seconds

Multiple-choice and Multipleanswer

60 90 seconds

Matching and Ordering

30 seconds per response

Short Answer

120 seconds

Completion

60 seconds

Essay

10 30 minutes
8

Writing Test Items

Simple and direct wording

Avoid jargon
Avoid trivia items
Match items to learning outcomes
Each item has an agreed upon correct
answer
Write more questions than you will
need
9

Multiple-choice Items

Stem
Direct question
Incomplete statement

Responses
One correct answer
Multiple distracters

10

Stem

Clearly worded
One idea
Avoid the use of negatives
Enough information to answer the question
Direct questions preferred
Blanks at the end of the stem
Include words repeated in all responses
11

Responses

3-5 per item

Avoid all of the above and none of the
above
Grammatically correct with stem
Similar length and structure
Avoid absolute words
Listed in a logical order
Mutually exclusive and not overlapping
12

Distracters

Plausible
Common misconceptions
Logical misinterpretations
Clichs
Partial answers
Technical terms or jargon
13

Example
What is the minimum number of
responses for a multiple-choice item?
A) 2
B) 3
C) 4
D) 5

14

Application Example
What problem exists in the following multiplechoice stem:
________ is the most common type of test item.
A) Absolute words should be avoided in the stem.
B) The stem contains more than one idea or
concept.
C) Not enough information is presented to answer
the question.
D) The fill-in the blank should come at the end.
15

Analysis and Evaluation

Example Stem
An instructor was asked the following question: "Briefly list and explain how
you develop a test. As an answer, this instructor wrote the following:
I begin by going through the chapter and writing questions based on the
material in the text and my lectures. Then I decide how many questions I want
and select the best questions from the list that I have developed. I format the
test and add instructions. After a few days, I review the questions and make
any revisions that need to be done and remove any jargon or wording lifted
directly from the text. Then I use the test in class. Based on how the class
does, I may make changes for the next time I teach the class.

16

Analysis and Evaluation

Example Responses
Based on the process described in Effective
Classroom Tests, how would you judge this answer?
A) EXCELLENT (all steps in the right order with correct,
clear, and complete descriptions)
B) GOOD (all stages correct in the right order, but the
descriptions are not as complete as they should be).
C) MEDIOCRE (one or two stages are missing, OR the
stages are in the wrong order, OR the explanations are
not complete, OR the explanations are irrelevant)
D) UNACCEPTABLE (one or more stages are missing AND
the explanations are not complete AND/OR they are
irrelevant)
17

Poor Question 1
Good multiple choice items:
A) are easy to write
B) can only test memorized content
C) are better than essay items
D) there is no such thing
E) can test a wide range of content

18

Poor Question 2
Which of the following characteristics is
not true of completion test items but is
an important distinguishing attribute of
matching tests, multiple-choice
questions, and true-false items?
A) They are objective test items.
B) They require knowledge recognition but
not production.
C) Much more difficult to construct.
19

Poor Question 3
Which of the following statements is FALSE?
A) Misfeasance is the improperly doing of an illegal
act.
B) Nonfeasance is improperly doing a legal act.
C) Nonfeasance is the failure to do an act that one
must do legally.
D) Misfeasance is the failure to PROPERLY do an
act that one has a duty to perform.
E) None of the above.
20

Poor Question 4
__________ is/are the best method to
determine if students have learned
something.
A) Comprehensive Exam
B) Homework Assignments
C) Pop Quizzes
D) Research Paper
21

Selecting Test Items

Outcome Weight x # Questions by Type =
#Questions of Each Type for Outcome

22

Formatting the Test

Group items by type

Sort items by increasing difficulty
Add instructions
Review layout and pagination
Write answer key

23

Assessing the Test

Self

2-3 days after writing the test

Clarity
Clues in items to other items

Non-expert

Clarity
Contextual clues

Peer

Content
Weighting to outcomes
Answer key

Students

Clarity
Content
24

Test Taking Procedures

Use of notes or other materials

Time limits

25

After the Test

Item Analysis
Areas for review
Test revisions

26

Activity

Write two test questions on any topic

One question should be an example of
a good test item
One question should be an example of
a poorly written test item

27

Share

Share your two questions with a

partner
Can they determine which is good and
which is bad?
Can they explain what makes one
poorly written?
As a team, how can you fix the poorly
written questions?
28

Intermission

29

Outcomes

Determine what characteristics are

important in evaluating student work
Evaluate rubrics, analytic scales, and
other evaluation methods
Describe the contents of a good rubric
Identify rubrics already in use at Baker
College
Begin work on a rubric for a class
30

What is a Rubric?
A rubric is a scoring tool or guide that lists
the specific criteria and the ranges for
multiple levels of achievement for a piece
of work or performance. A rubric consists
of a set of well-defined factors and criteria
describing the dimensions of an
assignment to be assessed or evaluated.
31

Parts of a Rubric

Scale (columns)
Dimensions (rows)
Criteria descriptions (cells)

32

Benefits of Rubrics

Communicates the instructors

expectations
Streamlines the process for feedback
to the student
Facilitates equitable grading
Standardizes assessment across
different instructors
33

Uses for Rubrics

Papers
Presentations
Projects
Essays
Homework

Case Studies
Participation/Class
Discussion
Portfolios

34

Types of Rubrics

Analytic
Page 11

Holistic
Page 12 and 13

Check List
Page 14

Scoring Guide
Page 15

35

Creating a Rubric

Identify components/outcomes of the

assignment
Determine a scale
Add criteria
Assign points
Set component weights (optional)
Assess the rubric
Test and revise
36

Activity

Split into groups of 3-4

Determine team roles
Select an assignment that needs a
rubric
Can be a specific assignment, such as a
research paper for ENG 102
Can be of a more general nature such as
a class presentation
37

Step 1: Identify
Components

List 5 major objectives/outcomes of

the assignment
Write these items as the row headers
of the sheet provided

38

Step 2: Determine a Scale

Aim for 3-5 levels

Can use an odd or even number of
items
Use the headings on the next slide for
ideas
Write these as column headings on the
sheet provided
39

Potential Column
Headings

Outstanding | Accomplished | Proficient | Developing | Beginning

Accomplished | Average | Developing | Beginning

Excellent | Good | Needs Improvement | Unsatisfactory
Exceptional | Acceptable | Marginal | Unacceptable
Expert | Practitioner | Apprentice | Novice
Professional | Adequate | Needs Work | Youre Fired
Exceeds Expectation | On Target | Beginning
Exemplary | Competent | Developing
High | Medium | Low
Outstanding | Proficient | Shows Potential

40

Step 3: Add Criteria

Create descriptions for each level of

performance for each criteria in the
cells of the rubric
Bullet points
Paragraphs

Write these criteria in the cells of the

sheet provided
41

Step 4: Assign Points

Assign points for each level of

performance
Can use either of the following:
Discrete values (5, 4, 3, 2, 1)
Ranges (10-9) for each level

Indicate the point value on the sheet

provided
Normally placed with the scale
42

Step 5: Set Component

Weights

Allows for different levels of importance

Spelling/grammar more or less important than
content?

Determine if weights are necessary for your

rubric
Assign weights accordingly
See example on page 17-18 of the handout

43

Step 6: Assess the Rubric

Assess your rubric using a metarubric

See examples on page 11-15 of your handout

Conduct a peer review

Ask one or two other instructors to review your
rubric

Provide time for student review

Allow students to ask questions and make
comments

44

Group Project

Trade rubrics with another group

Assess the rubric using a metarubric
from page 11-15

45

Discussion

What metarubric(s) did you use? Why?

What did you see on the other teams
rubric that you liked?
Could you understand the assignment
easily by reviewing the rubric?

46

Step 7: Implement and

refine

Refine your rubric based on feedback

from other instructors and students
Make notes each time you use the
rubric for continuous improvement
purposes
Share with others

47

Rubric Reliability & Validity

Reliability
the likelihood that a given measurement
procedure will yield the same description of a
given phenomena if the measurement is
repeated.

Validity
the extent to which a specific measurement
provides data that relate to commonly accepted
meanings of a particular concept.

Babbie, 1986
48

Reliability Requires

Instructor should reach same

conclusion each time
Different instructors should reach
similar conclusion (interrater reliability)

49

Interrater Reliability

Independently score a set of student

samples
Review responses for consistent and
inconsistent responses
Discuss and reconcile inconsistencies
Repeat with second group of samples
Maki, 2004, p. 127
50

Validity Requires

Reliability
Comprehensiveness
Cover all outcomes

Economy
Space is usually limited, so be selective
about what goes into the rubric

Balanced scoring and weighting

51

End of lecture
Any question

52