Creating Tests That Assess Higher Order Thinking Skills

Innovative Educators Dr. Jennifer Hurd October 13, 2010

Objectives
When you complete this workshop, you should be able to:
Write different types of questions that will assess more than knowledge and comprehension on the Blooms Taxonomy Scale. Analyze test items to determine if these test items assess higher order thinking skills. Avoid signaling the correct answer by knowing common errors in teacher-developed exams.

Blooms Taxonomy

Developing Objectives Using Blooms Taxonomy

Objectives must be measurable Objectives must be attainable Objectives must have a limiting factor

Objectives must:
Contain the same key words that we will discuss for developing questions. Map to the mission statement or goals for the entire course or department Demonstrate the skills have been attained for each level of Blooms Taxonomy

Example of an Objective
According to the course material, 85% of the students will be able to design a business plan for a new business of their choice.

Example of an Application Objective

According to the course content, 90% of the students in this course will be able to complete a mock interview with three of the instructors

Learning Outcomes
Are the primary building blocks of good curriculum design.
Support the goal or learning outcome in that each is a small step to arriving at what the learner is supposed to know or be able to do. Are carefully worded with qualifiers to restrict the conditions and terms under which the learning outcomes are met.

Are measurable!

Learning outcomes must contain verbs that describe observable, measurable actions.

Learning Outcomes should:

Provide the basis to select or design instructional materials, content or assessments
Provide a framework within which learners can organize their efforts to complete the learning task

Planning a test
Determine the method of assessment Demonstrating knowledge and understanding Assessing critical thinking skills Assessing problem solving skills Assessing performance of procedures Assessing application skills Assessing collaborative learning skills Assessing research skills

Planning: Specifying What is to be Assessed

Clarify instructional learning outcomes Define the purpose of the test
Pre-instructional (pre-test) Tests and quizzes Final exams

Specify what will be tested and at what level Develop a test blueprint

Blueprint for a Test

Content/ Skill Area Knowledge/ Understanding Critical Thinking Skills Skill Comprehension Application Critical Thinking Problem Solving Performance of Procedures/ Demonstrati ng Techniques Collaborative Learning Skills Integrate Learning Into Professional Practice Assessing Research Skills

Basic Tenants of Test Item Writing

Reading Level Write items as simply as possible Test for important details Avoid direct quotes or verbatim statements Identify any opinions or authority that are used in test items

Anatomy of an Exam Question

Should have three parts:
A key word that tells the students what they are expected to do An object that tells them what general information to use to perform their task A limiting factor or factors that tells what specific information to include in the answer

Blooms Taxonomy Lets look at each level of Blooms Taxonomy to see how we can develop questions for each level.

Knowledge Questions
remembering; memorizing; recognizing; recalling identification and recall of information Who, what, when, where, how ...? Describe

An example of a knowledge question

According to the instructor, define the key words for knowledge questions.

An example of a knowledge question

According to the instructor, list the key words for knowledge questions.
Can you change this question into a M/C question?

Writing a question

According to the instructor, which of the following are the key words for knowledge questions?
A. B. C. D. E. list, synthesize, evaluate, define, write List, define, dramatize, solve List, define, duplicate, memorize None of the above A and B only

Application Question Key Words

Imagine Pretend Make Believe Visualize Assume Put yourself in the situation of Use your knowledge to Suppose

In groups of three or four write a three-part application question on any topic or subject area using one of the key words listed above. Be prepared to share your question.

Imagination Activity
Imagine that you are creating your own company You can hire 5 key people to help you run your company These 5 people can be anyone from the past or present Who would you hire and why? You have 3 minutes to decide, then you will discuss your response with your table partners. One person from each table will share with the group.

Evaluation Question Key Words

Rank Order Judge Grade Measure Determine Convince persuade Decide Support Explain Critique Criticize Recommend Suggest

Sample Three-part Evaluation Question

According to the number of safety-related features each contains, determine which compact car is safest to drive.
Key Word: Determine Object: to find out which compact car is safest to drive Limiting Factor: cars must be compact Only safety-related features to be examined

A/B Visit Argue Workshop

This workshop allows students to persuade, convince, determine, decide, recommend, suggest, support, and explain parts of course content. Applying course content to other parts of the course or to their own life makes it even more meaningful and worthwhile.

A/B Visit Argue Workshop

Set up the room for A/B visits, or have students get into pairs. Your topic is to argue whether or not there should be a student success course for college credit on your campus. The people on the outside (As) will take the pro stance. The people on the inside (Bs) will take the con stance.

Synthesis Question Key Words

Combine Integrate Modify Revise Improve Rearrange Reconstruct Hypothesize Substitute Create Generate Devise Design Compose Plan Predict

Synthesis Sample Question

Rearrange the amendments to the U.S. Constitution in order from least to most important in everyday life. As well, devise any new amendments that you think would improve everyday life. Justify your amendments according to your text or your notes.

Rearrange the amendments to the U.S. Constitution in order from least to most important in everyday life. As well, devise any new ammendments that you think would improve everyday life.
Key Words: rearrange and devise Object: To discover which constitutional amendments you consider most important and which you would like to add to improve everyday life Limiting Factor: deal only with amendments to the U.S. Constitution.

Other Synthesis Assessments

Ask students to create mnemonic devices to remember course content. Students can create crossword puzzles that center on a particular part of course content Create a visual product to show course content. Example: Create a company logo for XXX company.

Analyzing Test Items to Determine if they Assess Higher Order Thinking Skills
In the next three questions, use the chat feature to tell me whether the question assesses higher order thinking skills. hy/why not.

According to the text, good money management skills include A. Planning, budgeting, and living within your means B. Living within your means and using credit cards to help. C. Setting goals, budgeting, living within your means, and using credit cards wisely D. Living on a budget and using credit cards when necessary E. Living on a budget without credit cards.

Use your knowledge from the text and the class discussions of Hitlers strategies to suggest how he lost World War II.

According to the text and class discussions, which of the following would be a reaction of an insecure/avoidant baby to the strange situation research technique? A. The baby would show great upset when the parent departed and would cling to the parent upon his/her return. B. The baby may or may not show great upset when the parent departed, but would cling to the parent upon his/her return. C. The baby would show great upset when the parent departed, but would ignore the parent upon his/her return. D. The baby may or may not show upset when the parent departed, but would ignore the parent upon his/her return.

Common Errors Made by Teachers in Constructing Test Items

Dont give the answer away! Avoid these common errors.

Test Construction Errors to Avoid

Dont use All of the above or None of the above only when they are the correct answer. Avoid distractors that are a joke or an insult.

Test Construction Errors to Avoid

Make sure the stem of the question and the distractors/answer are grammatically correct Avoid having one choice that is extremely longer than the other choices.

Test Construction Errors to Avoid

If all the distractors are numbers, 90% of the time the correct answer is not the highest number or the lowest number. Avoid using absolute determiners in the answer choices.

Test Construction Errors to Avoid

Avoid using more inclusive options. Avoid answer options that state a reason.

Learning how to play the piano while singing occurs in the

A. Affective domain B. Motor domain C. Cognitive domain D. Motor and cognitive domains E. None of the above

According to the text, those who believe in the work ethic believe that

A. All people will do good work. B. Hard work is always rewarded. C. Work is good and important. D. Only good workers should receive pay.

The normal adult temperature of 98.6 degrees F is equivalent to

A. 18 degrees C B. 27 degrees C C. 37 degrees C D. 42 degrees C

Summary
Use three-part questions to make sure you are asking questions that cover all of Blooms Taxonomy Write objectives to measure all aspects of Blooms Taxonomy Have students create questions and grade questions Use other methods instead of just paper/pencil tests to assess students in higher order thinking skills