Developing Selected-Response Items

Two General Types of Paper and Pencil Tests

Selected Response Items:
Binary response items (e. g., true-false). Multiple-choice items. Matching items.

Constructed Response Items:

Fill-in-the-blank items. Completion and short answer items. Essay items.

General Item Writing Rules

(These rules are available on the website for this course)

Provide clear and understandable directions to students about how to respond. Be sure the items themselves are clear (unambiguous) to students. Do not provide unintentional cues regarding the correct response. Use grammar and vocabulary consistent with the source of instructio.

General Item Writing Rules

Keep reading level below students ability. Format the item for efficient scoring. Be sure content experts would agree on the correct answer. WRITE THE ITEM SO THAT IT MEASURES THE SPECIFIED LEARNING TARGET.

Additoinal rules for Binary-Choice Items Binary-choice (or alternate-choice) items present a proposition for which one of two opposing options represents the correct answer. Several variants exist:
True-false. Fact-opinion. Right-wrong. Yes-no.

Binary-Choice Items
(Variations: Embedded true-false items) Indicate whether each underlined word is used as a verb (V) or as something other (O) than a verb.
Sailing has many advantages as a recreational sport. You can sail by yourself or with others. While basic techniques can be learned quickly, you can spend a life-time developing your sailing skills. Answers V O has V O with V O spend V O as V O While V O sailing V O techniques V O can V O sail V O learned

Binary-Choice Items
(Variations: Multiple true-false items)

Read each option and indicate which are correct.

In comparison with multiple-choice items, an advantage of true-false items is
1. more items can be administered within a given time. 2. higher reliability is obtained from a given number of items. 3. each test item can be developed in less time. 4. students will select the correct answer only when they have achieved the skill being assessed.

Binary-Choice Items
Advantages and Limitations

Limitations: Advantages: Highly susceptible to Allows adequate guessing. sampling of (usually, knowledge- Can be used only when dichotomous answers level) content. represent sufficient Relatively easy to response options. construct. Usually, only indirectly Objectively and assess intellectual skills. efficiently scored.

Binary-Choice Items
Qualities of Good Binary-choice Items

Good binary-choice items should.

Measure the specified skill (learning target).
This requires some serious thinking

Require appropriate level of reading skill. Emphasize adjectives or adverbs when they alter or reverse the meaning of the item. Have one (of two) response options that is
unequivocally correct. Continued on next slide

Binary-Choice Items
Qualities of Good Binary-choice Items Exclude adjectives and adverbs that imply an indefinite degree. Avoid adjectives and adverbs that imply absolute meaning Be stated as simply as possible (e.g., should exclude window dressing). Should be written so that the incorrect response is plausible. Should present a single proposition (not a doublebarreled proposition)

Binary-Choice Items
Tip for improving the quality: Use contrasts

Without contrast: The reliability of shortanswer tests is unaffected by guessing. With contrast: The reliability of shortanswer tests is less affected by guessing than is the reliability of multiple-choice tests.

Binary-Choice Items
Examples of double-barreled propositions

Although essay tests require less time to construct than do multiple-choice tests, they require more time to score. Classroom tests should be reliable and yield consistent scores across time.

Binary-Choice Items
Evaluation

Learning target: Information. Identify

qualities desired in multiple-choice items. Poor Item TRUE or FALSE: It is not important for a multiple-choice item to contain five options. Improved Item TRUE or FALSE: A multiple-choice item can contain as few as two options.

Binary-Choice Items
Evaluation

Learning target: Information. Identify qualities desired in multiple-choice items. Poor Item T or F: Sometimes multiple choice items are superior to true-false items. Improved Item T or F: A 10-item multiple-choice test typically will be more reliable than a 10-item true-false test.

Binary-Choice Items
Evaluation

qualities desired in multiple-choice items. Poor Item T F Good multiple-choice items measure important skills. Improved Item T F If plausible distracters are easy to develop, a table of specifications is of little value when constructing multiple-choice items.

Learning target: Information. Identify

Multiple-Choice Items

Anatomy of Multiple-Choice Items

MC items consist of .
A stem,
Either a direct question, or An incomplete statement to be completed.

A correct answer, and Two or more distracters or foils.

Multiple-Choice Items:
Advantages and Limitations Advantages Provide for a wide sampling of content. Effectively structure the problem to be addressed. Can be quickly and objectively scored. Limitations Somewhat susceptible to guessing. Indirectly measure targeted behaviors. Time-consuming to construct.

Multiple-Choice Items
Example: Due to lack of parallel content this item may have more than one correct answer: Which of the following represents the warmest temperature? A. 100 degrees Celsius B. 200 degrees Fahrenheit C. 300 degrees Kelvin D. an oven set a medium

Multiple-Choice Items Qualities Continued

Options avoid repetitive words.
Example: Criterion-referenced A. refers to how a test is constructed. B. refers to how a test is interpreted. C. refers to how a test is scored. D. refers to how a passing score is established.

Multiple-Choice Items Qualities Continued

Extraneous content (window dressing) is excluded (example on next slide). Adjectives or adverbs are highlighted when they reverse or alter the meaning of a stem.
Words like not and except should be emphasized. These can be used, but only when it is important to do so.

Multiple-Choice Items Which item stem contains window dressing? A. What is the highest numerical value of a reliability coefficient? B. Although usually not obtainable, the maximum value of a reliability coefficient E is 1.0.

Multiple-Choice Items Examples of uses of not and except

1. Which of the following qualities least affects the reliability of a test? 2. All of the following represents types of validity EXCEPT 3. The quality that is not an advantage of multiple-choice items is 4. ALL BUT WHICH ONE of the following is...

Multiple-Choice Items
(Continued)
Sample item with equally plausible distracters:

Which item format requires students to spend the greatest portion examination time actually solving problems presented by the items: A. Essay B. Short-answer C. True-false D. Multiple-Choice

Multiple-Choice Items
(Continued)

Qualities desired in M-C items, continued

Options contain grammar consistent with the item stem. The use of all of the above or none of the above used only when necessary. Options are arranged in natural or logical order.

Evaluating M-C Items

Poor item:
Internal consistency is high A. when students who scored high on the first half of the test score high on the second half of the test. B. when students who scored high on the first half of the test score low on the second half of the test. C. when students who scored high on the first half of the test score in an unpredictable manner on the second half of the test. D. when all of the above are true.

Evaluating M-C Items

Improved item: If the internal consistency of a test is good, how will a group of students score on the second half of the test if they got the highest scores on the first half of the test? A. Highest scores. B. Lowest scores. C. Unpredictable scores.

Evaluating M-C Items

Learning Target: Identify characteristics of formal and informal assessments. Which of the following is an example of informal assessment? A. Allowing students a choice of which questions they will answer. B. Not allowing students a choice of which questions they will answer. C. Observing which students are paying attention.

Evaluating M-C Items

Poor item:
Various item formats have specific advantages and limitations. An advantage the essay format has over the multiple-choice format is: A. the essay item can assess more skills in a given amount of time. B. the essay item can assess students ability to evaluate ideas. C. the essay item can be reliably scored. D. the essay item requires students to communicate ideas in writing.

Evaluating M-C Items

Improved item: Which is an advantage of essay over multiple choice items? A. Assess more skills in a given amount of time. B. Assess students ability to evaluate ideas. C. Evaluate students ability to communicate ideas. D. Facilitate reliable scoring of answers.

Multiple-Choice Items: Item-writing Guidelines

1. Does the stem present a clearly stated problem or question? 2. Is extraneous content (window dressing) excluded from the stem? 3. Are adjectives or adverbs emphasized when they reverse or significantly alter the meaning of a stem or option? 4. Are negatives avoided wherever possible or highlighted where necessary? 5. Are the correct answers equally distributed across all choice categories?

Multiple-Choice Items: Item-writing Guidelines

6. 7. 8. 9. Are options parallel in form and content? Do the options avoid repetitive words? Is each distracter plausible? Is the grammar in each option consistent with the stem? 10. Does the item exclude options equivalent to all of the above and none of the above? 11. Unless another order is more logical, are options arranged alphabetically?

Matching Items
Anatomy of a matching item:
Consist of Premises (or stimuli) and Responses. Advantages Provides for wide sampling of knowledge targets. Relatively easy to construct. Can be scored objectively and efficiently.

1. 2. 3. 4. 5.

Include homogeneous premises and responses. Use more responses than premises. Make sure directions are clear to students. Keep responses short and logically ordered. Use four to ten premises (and restrict to one page). 6. Avoid grammatical clues to correct answers.

Specific Item-Writing Guidelines for Matching Items

Developing Constructed-Response Items

Paper & Pencil Constructed-response items, that is.

Developing Constructed-Response Items

Major advantage of constructedresponse items:
They elicit responses that more closely resemble real-life behavior.

In general, however, if a selectedresponse item can provide the same evaluative information as a constructed-response item, use the selected-response item.

Short-Answer Items:
Advantages and disadvantages Advantages: 1. Easy to construct. 2. Require the student to supply and answer. 3. Many such items can be included in a test. Disadvantages: 1. Generally limited to knowledge-level skills. 2. More likely scored erroneously than are selectedresponse items.

Short-Answer Items: Item-writing rules

1. Use direct questions rather than incomplete statements. 2. Write items so that the correct response is concise (a few words or a short phrase). 3. Write items so that they can be scored efficiently. 4. Be sure there is a highly limited set of correct responses. 5. Think of the correct response, then write the item.

Completion Items:
Item-writing rules
Same advantages/disadvantages of short-answer items. Same rules applicable to completion items, plus these additional four:
1. 2. 3. 4. Be sure the blank represents a key word or phrase. Position blank at or near the end of the item. Keep blanks the same length. Use no more than two or three blanks.

Essay Items: Advantages

Unique advantage: Can assess ability to communicate in writing (synthesize, evaluate, compose). Other advantages:
1. Provide more direct measures of behaviors specified in performance objectives. 2. Require the student to produce a response.

Essay Items: Limitations

Scoring is less reliable (more subjective).
1. Inconsistent within teachers across multiple scorings of the same responses. 2. Inconsistent within teachers across students. 3. Inconsistent among teachers on the same responses.

Provides less adequate sampling of content domain. More time-consuming to score.

Essay Item-Writing Rules

1. Convey a clear idea of how extensive a response is expected:
Ten minutes or less (typical for a restricted-response essays). Specify a range for the number of words or the amount of time to be spent on the response. Make the distribution of points obvious. Would different readers assign the same score? Describe what constitutes a correct and complete response. The rubric should be obvious to knowledgeable students. You do not have an essay item unless you have a rubric!

2. Develop a suitable scoring plan (rubric):

Essay Item-Writing Rules (Continued)

3. 4. 5. 6. Do not allow a choice of which items to answer. Evaluate all responses one item at a time. Vary the student order when reading responses. Decide on the weight grammar and vocabulary will carry beforehand. 7. Conceal the identity of students, if possible. 8. Use multiple scorers, when possible.

Multiple-Choice Item Flaws

Examples

M-C Item Flaws Which best describes what happens when work is done?
A. B. C. D. A force operates through a distance. A force is exerted. Energy is destroyed. Potential energy is changed to kinetic energy.

[A]Flaw: Using stereotyped phrases. Item can be answered correctly based on recall of verbal information as well as through understanding of the principal involved.

M-C Item Flaws Which of the following has helped most to increase the length of human life?
A. B. C. D. Fast driving. Avoidance of overeating. Wider use of vitamins. Wider use of inoculation.

[D]Flaw: highly implausible distracter. Choice A is unreasonable, reducing the item to a three-choice item.

M-C Item Flaws Horace Greeley is known for his

A. B. C. D. advice to young men not to go west. discovery of anesthetics. editorship of the New York Times. humorous anecdotes.

[C]Flaw: Verbal trick in distracter: choice A inserts the word not into a phrase otherwise attributable to Horace Greeley.

M-C Item Flaws Slavery was first started

A. B. C. D. at Jamestown settlement. at Plymouth settlement. at a settlement in Massachusetts. a decade before the Civil War.

[A]Flaw: Non-parallel distracters. Choices A and B give specific places, C designates a more general area, D specifies a time. This ambiguity makes more than one choice correct.

M-C Item Flaws

In purifying water for a city water supply, one process is to have the impure water seep through layers of sand and fine and course gravel. Here many impurities are left behind. Below are four terms, one of which will describe this process better than the others. Select the correct one. A. Sedimentation C. Chlorination B. Filtration D. Aeration

[B]Flaw: Stem includes an instructional aside.

M-C Item Flaws While ironing her formal, Jane burned her hand accidentally on the hot iron. This was due to a transfer of heat by
A. conduction. B. radiation. C. conversion. D. absorption.

[A]Flaw: Stem includes window dressing. The introduction implies a practical problem when the item only involves knowledge of technical terms.

M-C Item Flaws In the definition of a mineral, which of the following is incorrect?
A. B. C. D. It was produced by geologic processes. It has distinctive physical properties. It contains one or more elements. It has a variable chemical composition.

[D]Flaw: Uses a negative in the stem; tends to be confusing. These types of items are rarely found outside the classroom.

M-C Item Flaws Which event is more important in American history?

A. B. C. D. Braddocks defeat. Burrs conspiracy. Hayes-Tilden contest. Webster-Hayne debate.

Flaw: No best answer. Whos to say which is more important. Even experts would not agree.

M-C Item Flaws The population of Denmark is about A. 2 million. B. 15 million. C. 4 million. D. 7 million. Flaw: Unnatural sequence of responses. It would be better to order from 2 million to 15 million.

M-C Item Flaws The balance sheet report for the Ajax Canning Company would reveal (A) the companys profit for the previous fiscal year, (B) the amount of money owed to its creditors, (C) the amount of income tax paid, or (D) the amount of sales for the previous fiscal period. [A]Flaw: Placing distracters in tandem with the item stem.

M-C Item Flaws Which is the best definition of a vein?

A. B. C. D A blood vessel carrying heart. A blood vessel carrying A blood vessel carrying A blood vessel carrying heart. blood going to the blue blood. impure blood. blood away from the

[A]Flaw: Needless repetition in the distracters.

End