Documente Academic
Documente Profesional
Documente Cultură
Item Writing
Rubric Design
1|Page
2|Page
Multiple-Choice Items
The multiple-choice item is composed of a stem and alternatives, which include foils or distractors, and a
keyed response.
Which one of the following makes its own food?
A.
B.
C.
D.
Black Cat
Fish
Grasshopper
Green Plant
STEM
FOILS OR DISTRACTORS
KEYED RESPONSE
The stem serves two primary functions: presenting the problem and setting an appropriate frame-of-reference.
Presenting the central problem in the stem as a question usually results in a better item. Stems asking for the
only alternative that does not correctly answer the central question are more difficult to answer, are more
ambiguous, and discriminate less well between good and poor students. Unless needed to efficiently assess a
desired behavior, it should not be used.
Alternatives serve to refine and delimit the frame-of-reference established by the stem so that one and only one
alternative will appeal to the well-informed individual. The alternatives provide closure to the item. Each
alternative must be grammatically consistent with and appropriate to the stem. Linguistically, the alternative
should be equally attractive. Keep the alternatives short. Try to incorporate in the stem those words that are
repeated in each alternative. Alternatives should be of about the same length and complexity. Avoid overlapping
alternatives. Arrange the alternatives for clarity and ease of reading. If there is a numerical or logical order among
the alternatives, they should be so ordered.
The keyed response must be a better answer to the question than any other alternative. There is a tendency for
the correct response to be longer than the foils. If the keyed response is frequently located in a particular choice
position, this can serve as a clue to the correct answer. After selecting the items for a test, count the number of
times each response position is used.
The primary purpose of foils, or distractors, is to offer the student who lacks requisite knowledge
plausible choices that may be chosen instead of the keyed response. The foils should not appeal to those
who actually know the answer to the question. When developing foils, keep in mind that the distractors
must be grammatically consistent with the stem. A foil should not trick the student; it should provide a
plausible choice if the answer is not known. The greater the similarity among all alternatives, the more
difficult the item, all other things being equal.
There are two types of multiple-choice items: closed-stem and open-stem. Closed stem items pose a
question and end with a question mark. The item should clearly spell out what is being asked. Items that
use figures, diagrams, tables, etc., must have a closed stem. Open stem items have an incomplete
sentence that needs to be completed by one of the options. The incomplete sentence must be direct and
focused with all options written to be grammatically compatible with the stem.
3|Page
Donts
Language should be simple, direct, and free from Give grammatical clues tense conflicts, misuse of
ambiguity; the item should not unnecessarily test articles, and singular-plural conflicts.
reading ability or vocabulary.
Establish a clear frame-of-reference in the stem.
Structure the item around one central idea that is Have correct answer stems with best answer
clearly presented in the stem.
alternatives.
Use a logical ordering of alternatives (alphabetical, Give away the correct answer or the answer to
numerical)
another question in the same test.
Use items stated in the positive. (For example, Avoid items stated in the negative. (For example,
Which of the following is)
Which of the following is NOT) Or use sparingly
The item must have a single best answer among Never use negatives in both the stem and the
the options!
answer choices.
Use common misconceptions and errors as Use items that ask students for their opinions.
possible answer choices.
Use answer choices that are parallel in length, Use answer choices that are a mix of long and short
grammar, semantics, and syntax.
responses.
Answer choices are presented in a logical order Provide superfluous information in the item.
(e.g., alphabetical, numerical, sensible).
Use plausible, but incorrect, answer choices.
Use questions that assess the specific standards to Include trivial or artificially tricky items.
be measured.
The stem should include words or phrases that
otherwise would have to be repeated in each
answer,
Use an incomplete statement only if you capture
the essence of the problem in the stem.
4|Page
Matching Items
Matching items occur in clusters. A cluster of matching items must contain an introductory statement, a set
of related premises, and a list of alternatives to be shared by all the premises in the cluster
For each civil liberties issue in Column I, select the letter from Column INTRODUCTORY STATEMENT
II that helped clarify it. Print the letter for your choice on the line
preceding the question number. A letter will not be used more than
once.
Column I
__1. Exclusionary rule
__2. Flag-burning as symbolic speech
__3. Freedom from self-incrimination
__4. Right to an attorney
__5. Right to privacy
Column II
A. Gideon v. Wainwright
B. Griswold v. Connecticut
C. Hazelwood V. Kuhlmeier
D. Mapp v. Ohio
E. Marbury v. Madison
F. Miranda v. Arizona
G. Texas v. Johnson
H. Tinker v. Des Moines
PREMISES
ALTERNATIVES OR
RESPONSES
The introductory statement must set a general frame-of-reference for responding to the items of the
cluster and clearly indicate to the student how to proceed in selecting responses.
The premise is similar to the stem of the multiple-choice item. The number of premises in a cluster should
rarely exceed six or seven. It is difficult to find even this many homogeneous ideas that are worthy of
measuring, and even more difficult to avoid sampling bias. At the lower end, four premises are minimal.
Even with as many as three premises, the standard multiple-choice format is likely to be more efficient.
The list of alternatives, or responses, to the problems presented in the premises are placed in Column II
or the right-hand column. It is important to communicate to students whether or not an alternative can be
the correct answer for two or more premises. If an alternative cannot be a correct response for two or more
items, then the number of alternatives must be greater than the number of premises. If the directions
indicate that an alternative may be correctly used more than one time, then alternatives must occasionally
be used two or more times to make this direction plausible.
The reason that we do not use the same number of premises and alternatives, with each alternative serving
as a correct answer, is to make it more difficult for a student to use a process of elimination to arrive at a
correct answer. The number of alternatives should be somewhat limited since students must read the list of
alternatives each time they answer a premise.
5|Page
Donts
Provide clear directions. Inform the students if an Use unnecessary, technical terminology.
item may be used more than once.
Use a brief (maximum length is 10) list of items to Have more than 10 possible matches.
be matched; shorter is better.
Offer more response options than items needing Have clues that link premises with the responses.
to be matched.
Use correct grammar.
Use for who, what, when, or where knowledge Expect to measure higher-order mental processes.
items.
6|Page
7|Page
Donts
Write items that are entirely true or false as Use items that are partially true and partially false.
presented.
Write concise, unambiguous, and grammatically Use unnecessary words and phrases.
correct statements.
Use one main idea or concept in the item.
Use items that are germane to the area being Have irrelevant clues.
tested.
Have about the same number of true and false Have a pattern in the order of the responses.
statements.
Include enough items to adequately sample the Use qualifying terms: all, none, some, few, or
standard being assessed.
many.
Avoid vague words and absolutes.
8|Page
Donts
Dont let line length give away answer.
Consider using a list of words from which students Dont use passages or statements right from the
can choose. Best to have more word choices than textbook.
items.
Come to the point; ask one question, get one Dont place blank at the beginning of the
answer.
statement.
Provide adequate space for the answer.
Use simple, but precise, language.
Be sure there is only one correct answer for each
blank.
Be specific about expected level of detail:
Give two (not some) examples
Calculate (not What is) the pressure
Describe three (not What are several)
9|Page
Donts
10 | P a g e
Dos
Donts
12 | P a g e
13 | P a g e
Look at the following assessment items. Some of the items follow quality item writing guidelines, and some
do not. Identify whether or not each item follows quality item writing guidelines.
Mark the Yes box if the item follows all writing guidelines. Mark the No box if the item breaks one or
more of the guidelines. For each no item, identify the guideline(s) broken and fix the item so it follows the
guideline(s).
YES
NO
1.
2.
3.
4.
5.
___1. Texas
A. $7,200,000
___2. Hawaii
B. Chicago
C. Mardi Gras
___4. Illinois
D. Austin
___5. Alaska
E. 50th state
14 | P a g e
YES
NO
6.
7.
1.
2.
3.
4.
5.
6.
7.
Constellation
Prominence
Revolution
Moon
Earths revolutions in one year
Sunspot
Rotation
True or False:
From the Continental Divide, located in the Appalachian Mountains, water flows into either the
Pacific Ocean or the Mississippi River.
8.
True or False:
The animal cell has a definite shape; the plant cell does not.
9.
Instruments located in the percussion section of the orchestra are ____________,
_____________, ______________, and _____________.
10.
get?
B.
2
C.
3
D.
4
15 | P a g e
Activity Directions: Review the assessment and rubric for scoring the Advise FDR
performance task, then review the rubric for the 7th grade Book Report. Discuss:
How are the guidelines for performance assessment evident in the sample? What
guidelines were not followed? What would you change to make the task stronger?
How are the guidelines for rubrics evident in the two samples? What guidelines were
not followed? What would you change to make the rubrics stronger?
Organizational Preparation:
1. At least a week before giving the assessment, check with Special Education Department to decide
adaptations of the assessment for IEP students.
2. Make sure that you have all the materials for the test (including directions, etc.)
3. Teach all required background information before the test.
4. Be sure to practice graphic organizers with students using different topics.
5. Teach scoring rubric to students.
Assessment Administration:
1. Check for student I.D. numbers, names and appropriate male, female label circled on papers.
2. Suggested time limit: one class period.
3. No papers leave the room.
4. Read through and explain task with students without defining terms, answering questions or
allowing discussion on possible answers. Answer clarifying questions not content
questions.
16 | P a g e
Title:
Advise FDR
Background: Its 1939, England is the only major European power fighting Nazi Germany. You have
studied about World War II and you know that many difficult decisions have to be made
during wartime.
Task:
You are an advisor to F.D.R. and the President has asked you for a
written
summary of his options concerning the situations in
Europe. You must help him
decide what his policy will be in Europe. You have to analyze and weigh the positive and
negative consequences (effects) of your options and show that you have considered many
different consequences.
Audience:
The Oval Office at the White House, in a meeting with the President, Secretary of Defense,
Secretary of State, and you.
Procedures:
1.
Think of the options you want to give the President.
2.
Now write each option one per box, in the boxes labeled options.
3.
Under each option, write the positive and negative consequences for each option, in the spaces
labeled Positive and Negative options.
4.
In paragraph form, recommend the option you think is best. Fully explain and support the option
you choose.
17 | P a g e
Student I.D.
Circle One:
Male
Female
OPTION
OPTION
OPTION
OPTION
POSITIVE
CONSEQUENCES
NEGATIVE
CONSEQUENCES
18 | P a g e
Recommend to FDR the one option you think is best. State and fully explain the one option you choose.
Make sure you support your choice using the positive and negative consequences you considered. Focus
on only one option!
19 | P a g e
ACCEPTABLE OPTIONS:
1. Give war supplies, military advisers and money OR Start lend-lease act
Positive:
If give war supplies, etc. is option than start lend-lease act is an acceptable positive
keeps USA out of fighting (no casualties)
saves American lives
helps England (another democracy)
shows our support for our allies
satisfies isolationists in USA
USA gets money and military bases
good for USA businesses and public relations
buys time for England - lets them survive longer
Negative:
England not helped adequately
angers Germany
scares some people in USA who dont want to be in the war
first step toward full scale war
USA could be attacked/damaged
2. Join the war with England (USA commits our forces to fight)
Positive:
liberate captured European territory
England more likely to win - necessary for democracy to continue
Hitler stopped (more likely) from taking over Europe
other countries may follow suit
help stop persecution of certain groups in Europe (Jews)
we would help France and other allies who need help
could help U.S. economy
gain prestige
Negative:
Americans will die
will cost USA money
pulls USA into 2nd European conflict in 20 years
many Americans will become angry at FDR
chance we will lose
Germany may increase their warfare (American shipping more at risk)
U.S. could be attacked/damaged
20 | P a g e
3. Join the war with Germany (USA commits our forces to fight)
Positive:
have a good chance of winning
could take over/gain territory
Negative:
goes against American tradition (Germany not a democracy)
abandons long time ally - England
much of USA population would be furious
allows continued persecution of Jews
will cost USA money
USA could lose
pulls USA into 2nd European conflict in 20 years
Americans will die
U.S. could be attacked/damaged
4. Stay out of war (remain neutral)
Positive:
Americans not at risk (USA lives saved)
USA saves money
keeps isolationists happy
allows USA to focus on its own affairs (depression)
Negative:
Germany continues to expand
England (democracy) is not helped
all of Europe is at risk (potentially the rest of the world)
continued persecution of European people
shows weakness
5. Make an attempt to settle the war (mediate between England and Germany)
Positive:
saves lives (no fighting - just talking)
saves money
makes USA look good
saves Europe
takes place of ineffective League of Nations
Negative:
may not solve problem
parties may not be trustworthy
time consuming
shows weakness
UNACCEPTABLE OPTIONS:
1. use/invent atomic bomb
2. kamikaze attacks
21 | P a g e
22 | P a g e
Rubrics
Rubrics are scoring guides with features identified that will be used to evaluate student work. Rubrics
include descriptions of different levels of performance for each of the defined features. Rubrics provide
information on student progress toward achieving standards.
Rubrics are powerful communication tools when shared with students and parents. A well-written rubric
clarifies the vision of excellence and conveys that vision to students and parents.
A quality rubric should:
Focus instruction intentionally
Guide feedback descriptively
Characterize desired results objectively
Operationalize performance standards purposefully
Develop self-assessment competence constantly
Empower students to evaluate their own work - critically
Involve students thoughtfully
Yield consistent results even when administered by different scorers reliably
Critical Components of a Rubric:
Features the major attributes or categories of learning which will be evaluated. The features must
be based on the standards, skills, and concepts for mastery.
Scale the possible points to be assigned.
Descriptors statements that describe the various levels of performance.
Mid-point the middle point on the rubric which separates student work that is still developing
from work that is considered developed.
Rubric Visual
2
Features
Descriptors
Mid-point
Scale
23 | P a g e
Definition
Advantages
Disadvantages
Each feature is
evaluated
separately.
Analytic
All features
are evaluated
simultaneously
Holistic
24 | P a g e
Definition
Description of work
gives characteristics
that apply to a whole
family of tasks (e.g.,
writing, problem
solving)
General
TaskSpecific
Advantages
Disadvantages
Lower reliability at first than
with task-specific rubrics
Requires practice to apply
well
25 | P a g e
Donts
Select the most important attributes of the task as Use when assessing acquisition of simple
the features for the rubric
information
Write the rubric so it clearly indicates what quality Write a rubric to count the number of times a skill
work looks like and how that should be evidenced in was performed (i.e., finds 1 resource, finds 2
the work
resources, etc.)
Write the descriptors for meeting, or proficient, first Automatically write the lowest level on a rubric as a
(this description should be what is expected by the null category. Lowest-level indicators are not
standard, target, or objective)
always a total absence of knowledge or skill
Write descriptions different enough from level to Write the rubric to guarantee students will score at
level that work can be categorized (it should be every point on the scale
possible to match examples of work to performance
descriptors at each level)
Write the highest scale point to represent truly Use extraneous criteria as features (i.e., not
exemplary performance; it may be that no student aligned to unit standards)
will attain the highest scale point
Address all relevant content and performance Use a scale larger than you can clearly define
targets using different features
Define standards and help students achieve them
by providing criteria to evaluate their own work
Write the rubric in student friendly language
Use an even number scale whenever possible (i.e.,
4, 6)
Determine whether scores will be added and
contribute equally to the total (if different length
scales are used it will impact the contribution of
different features to the total score)
26 | P a g e
Activity Directions: Look at each of the following scoring systems. First determine if the
system is a rubric or not. If it is a rubric, determine whether it is an analytic or holistic rubric
and whether it is general or task-specific.
Explanation
Knowledge
Unable to provide
positive or negative
consequences
OR
Consequences
provided do not fit
with the identified
decision.
Appropriate
positive OR
negative
consequences
provided for the
identified decision.
Some appropriate
positive AND
negative
consequences
provided for the
identified decision.
No evidence or
completely
erroneous.
Minimal
explanation with
little or no support.
Partially explained
with limited amount
of general support.
Adequate
explanation with
moderate amounts
of specific support.
STILL DEVELOPING
Fully explained
with generous
amount of specific
support.
DEVELOPED
28 | P a g e
(10)
You have included your title, author, copyright date, publisher, and number of
pages
(5)
(30)
TOTAL
(100)
29 | P a g e