Vo l um e 2

2.06
Issue 06 JAN 2014

Electronic Newsletter on

faculty & resident instructional development

|Med/ /Ed eNews

Editor Karen Spear Ellinwood, PhD, JD, EdS

this issue

Cover: Writing Effective Exam Items

coop

Residents as Educators (RAE) Programnew activity report form online!

Read more

A
T

The Scoop: RAE Program

ssessment: Writing Effective Exam Items

Resident Development with Dr. Pritchard FEATURE (continued) Save the date

C
Doceri

Karen Spear Ellinwood, PhD, JD, EdS he latest AMES/OMSE FID Series seminar addressed assessment from the perspective of the exam item author. The presenter was Chris Cunniff, MD, a Professor of Pediatrics, also with appointments in the Departments of Medicine and Obstetrics and Gynecology. Dr. Cunniff has written exam items for the American Board of Genetics and contributes to exam items as a lecturer in blocks for the undergraduate medical education program at the University of Arizona College of Medicine. The seminar focused on one type of exam item: the single best answer, multiple choice question. After describing the item anatomy, Dr. Cunniff made several key points about writing effective items of this type:

heck out these

online tools for enhancing learner

engagement: Poll Everywhere

Socrative Instructional technology

EBM Search Tools

AMES/OMSE FID Series 2013-14 Videos online!

Determining reliability Measuring validity Describing the NBME approach to authoring items, and Offering several sample questions for critique and discussion.

This article presents some key points and guidance for writing single best answer items based upon Dr. Cunniffs presentation on February 3, 2014. For those who want to view the entire session, OMSE FID has posted the video online. The link is is in the sidebar.

Item Anatomy
Every multiple choice item has a stem plus options. The stem is vignette containing fact scenario upon which the stduent will base their response. The options are the choices. Incorrect options are referred to as distractors, although how we write distractors is as important as how we prenset cor-

Participants debated concepts and analyses of sample items, including a few contributed by our colleague from the UA College of Nursing, Sue Habkirk.

Continued on page 4
M e d E d / e N e w s [1]

Scoop | Residents as Educators Program

The UA COM instructional development policy requires all residents to participate in the Residents as Educators [RAE] orientation to teaching and complete about 2 hours of instructional development activities each year for the duration of their residency. LCME requires UA COM to maintain central records documenting attendance and completion of this requirement by all residents who teach medical students. OMSE documents such attendance for RAE events it sponsors. The RAE Program provides support to residents and residency program directors in ensuring resident compliance with the FID policy in two ways. First, the RAE Program can develop and conduct instructional development activities that satisfying the policy requirement for residents to participate in ongoing educator development throughout their residencies. Second, the RAE Program can assist residency program directors or departments in creating their own programming to satisfy the requirement. RAE Program workshops address a wide range of topics covering four ACGME competencies: Practice-based learning and Improvement; Interpersonal and Communication Skills; Professionalism; and Systems-based Practice. The RAE Program does not assume which skills residents ought to learn or improve. Instead, we conduct a faculty instructional development (FID) needs assessment to identify teaching strengths as well as practices in need of improvement, in close collaboration with departments, residency program and clerkship directors. The RAE needs assessment involves up to four sources of data: (1) Self-assessment survey by residents; (2) Careful review of de-identified student feedback surveys or clerkship evaluations; (3) Direct observations of resident teaching; and (4) Input from residents and clerkship program co-directors. We provide a report of the needs assessment to the Residency Program Director, who then decides whether to request that we develop a workshop based on the assessment or for assistance in developing program-sponsored RAE activities. LCME Standard ED-24 also requires the College of medicine to centrally monitor whether residents are complying with the policy. To ensure compliance, we are asking Residency Program Directors to:

SUBMISSION GUIDELINES
Med Ed eNews has 2,576 reads through Scribd.com, our publishing host!, in addition to the UA COM faculty readers. If you would like to submit a piece, please follow these guidelines: 1) 2) 1000 words maximum; Address issues relevant to teaching, assessment or technology in classroom or clinical settings; Include references to scholarly works on (medical).

3)

SE! ct OM Conta up in& gro e n r o e-on ent fo For on al developm l a u ion individ struct t s or n ptors e e c m e t r depar ing p d lu c y, in facult ! idents s re! & re ut mo d Fin o

thas go gram o r P AE t The R t star a grea o t f f evelten o ents d m t r a 5 dep ors with educat s a s t n reside oping . mming progra re >>> ut mo Find o

Maintain attendance records and document the number of hours and residents who participate in department or program-sponsored instructional development activities; and Report this information to OMSE by the end of May each year.

Call for submissions Would you like to share your ideas or experiences about developing or enhancing clinical or classroom teaching and assessment practices?
Please submit your manuscript to Karen Spear Ellinwood.

[NOTE: Download information card with active links and QR codes to take you to the appropriate forms]

Support
Depending upon your residents needs or requests, instructional development activities may be conducted one-on-one, in small groups or larger groups, using interactive methods of teaching and learning. [Continued on p. 3] M e d E d / e N e w s [2]

Resident Development

Scoop

[continued from p. 2

RAE Program Feedback

Residents and program directors will be asked to evaluate the RAE activities and suggest improvements or request additional support.

upport for Residents

Gail Pritchard, PhD

The Office of Resident Developments mission is to provide educational support in a structured environment for helping Residents and Fellows achieve their educational goals. The ORD provides individual support for developing: A curriculum vitae and cover letter
Senior Interim Learning Specialist Offices of Medical Student Education and Graduate Medical Education

[520.626.2390]

An appropriate study schedule for STEP 3, in-Service exams, and specialty boards Communication skillsi.e. Giving/ receiving feedback, developing presentation skills

Contact
T. Gail Pritchard, PhD 520.626.2390 tpritcha@medadmin.arizona.edu Karen Spear Ellinwood, PhD, JD Ph. 520.626.1743 kse@medadmin.arizona.edu

Projectsi.e. Efficiently reading and reviewing research articles; synthesizing literature into comprehensive reviews; effective preparation for group presentations

dents and fellows that focus on developing learning strategies, test-taking strategies, and organization and time management strategies, and enhancing presentation skills, such as in learning strategies fro improving how to present Grand Rounds. Visit the Resident Development site

Strategies for learning content material related to exams Strategies for test-taking The ORD also provides department level support by conducting workshops for resi-

FID Series #8
Bedside Teaching Date: 3 March, 2014
The goal of the presentation is to offer two learnercentered models of bedside teaching. One is based on the UA COM Societies pro-gram mentoring model and the other is Patient-centered rounding for learner centered bedside teaching. The presenters prefer an interactive seminar experience, and encourage questions and discussion.

Time: 10:00 11:30 am Location: COM-3230 RSVP: Click on RSVP stamp Refreshments will be served.

M e d E d / e N e w s [3]

[Continued from page 1|COVER] rect answers.

Item Selection
Items should adhere to the following four principles of item selection. An item should: 1. 2. 3. 4. Address one or more learning objectives; Give priority to testing foundational basic science knowledge; Address common or catastrophic clinical issues; and Apply accepted criteria of item construction applicable to the context of the examination.

The commonly accepted criteria for item construction in medical education comes in part from NBME approach to item writing based upon the work of Case and Swanson (2003), published by the National Board of Medical Examiners (NBME). Case and Swanson is available online and provides a detailed explanation of the concepts summarized here. Dr. Cunniff addressed these key criteria: validity, reliability, item difficulty and item discrimination.

CBI R

Facilitator esources
There are a variety of resources online for

facilitators of case-based instruction (CBI), including links to short videos demonstrating ThinkShare (formerly ThinkSpace), a facilitator guide to the developmental curriculum for CBI in the first two years of the undergraduate curriculum, and examples of student work using the eTools ThinkShare (Formerly GroupShare) and CBI ThinkShare. Learn more:

Validity & Reliability

In assessing exam items faculty should consider validity and reliability. Validity indicates that the item accurately measures what it purports to measure (Cunniff) and reliability indicates the item measures knowledge in a reproducible manner, from student to student or one student group to the next. No item can be valid without being reliable. But an item can be reliable without being valid. The coincidence of reliability and validity is a good measure of whether an item remains on an exam.

Item Difficulty & Item Discrimination

In addition to exam item validity and reliability, item writers should consider whether the item presents an appropriate challenge. Always consider the purpose of the question when constructing the stem and options. Do you want the student to recall information? Are you asking the student to discriminate from among finer grained choices? Recall items are important in determining whether students have internalized foundational knowledge. Questions that call for differentiation identify students ability to apply concepts to clinical situations. The former tests memory and the latter requires deeper thinking and reasoning. Blooms taxonomy is a useful guide to determine item difficulty and ensure that the phrasing of the question matches the intention behind what you are asking the student to do. There are many resources online which serve as a guide to using Blooms Taxonomy (example: Iowa State Universitys interactive guide; OMSEs guide).

A structured approach to medical problem-solving

Reflective Teaching/Learning CBI eTools

Items also ought to enable educators to discriminate fairly between the performance of students with knowledge and those without. Calculating the point biserial of an item is one way of measuring its ability to discriminate. The point biserial of item performance indicates that a correct response on that item is indicative of positive performance on the exam overall (Jones, 2011). UA COM block directors work with key faculty in their respective blocks to review exam items using the point biserial as one measure in deciding whether to retain, revise or reject items.

Writing Stems and Options

[Box 1] Dr. Cunniff offered general guidelines for writing stems and options when conA 5 year old boy has difficulty arising from the floor. He structing single best answer multiple choice questions to improve ability to discrimhas decreased motor strength and enlarged calf muscles. inate and create items with appropriate difficulty. CPK level is 20,000 U/L. What is the most likely diagnosis? Writing an Effective Stem The stem ought to be longer than the options. In other words, students should be doing the work of comprehending what the issue is and identifying possible answers as they read the fact scenario. There should be no extraneous information, unless the seemingly irrelevant information is intended to promote differentiation among the options. An example of long-stem, short option item appears in Box 1, A. Becker muscular dystrophy B. Duchenne muscular dystrophy C. Emery Dreifuss muscular dystrophy D. Limb girdle muscular dystrophy E. Spinal muscular atrophy

M e d E d / e N e w s [4]

[Continued from page 4|COVER] [Box 2] A 4 year old boy is diagnosed with an autism spectrum disorder. Relative to the general population, his family is most likely to exhibit which trait?

above. Writing Effective Options The option is not the place for long, detailed explanations. Options should be short and each should focus on related concepts that require the student to differentiate the best answer among them, rather than to ask students to respond as if each were a true/false statement or enable students to disregard obviously irrelevant choices (see Box 1).

A. Rural area residency B. SES higher than the general population C. SES lower than the general population D. SES similar to the general population What should you avoid? Dr. Cunniff offered guidance in what not to do when constructE. Younger parental age ing options for exam items: Options should:

Not be worded in parallel (see Box 2, right). If there are 2 or 3 options with parallel phrasing, students will ignore all other options and,
therefore, will not be encouraged to consider all options and differentiate among them.

Not use always and never. Such options are disregarded as patently false. Avoid short stems with wordy options. These become a series of true/false questions. Not include facts that trigger Pavlovian responses. In other words, do not include fact clinchers that make the answer obviously right or
wrong. This renders the question too easy.

Avoid red herrings or tricky optionsunless you intend to measure whether students can distinguish between relevant and irrelevant
information. Even then, your red herrings should be plausible (see below).

Avoid EXCEPT items. Avoid complex calculations, unless that is the point of the question. When knowing the formula is important, options should consist of various ways to depict the formula, rather than the final answer in applying the formula. The student, then, has to select the right path to the correct answer and is not penalized for committing a simple mathematical error even if they used the right formula. What should you be sure to do? There are also a few tips for what you ought to do when presenting options:

All options should relatively homogenous so they require students to differentiate rather than guess. For example, does the item call for
recall or application of criteria related to assessment or treatment? Pharmacology? Calculations or formulas? Ideally options should be on a continuum of least to most correct. This requires differentiation rather than or in addition to recall. Alphabetize options so students do not attempt to outsmart the test by guessing which option is correct based on the order or position of choices.

All options should be grammatically consistent. All options must be realisticplausible choices. Plausibility is an important feature. Options that are not plausible are immediately dismissed, ensuring lower item difficulty. If you are interested in contributing to the review of exam items and teach in preclinical years, contact the directors of the block you teach. If you would like feedback on exam items you have written or are writing, please contact Dr. Chris Cunniff or Karen Spear Ellinwood. Below are additional resources you might find helpful.

Related instructional resources

Caballero J, WR Wolowich, S Benavides & J Marino. Difficulty and discrimination indices of multiple -choice examination items in a college of pharmacy therapeutics and pathophysiology course sequence. International Journal of Pharmacy Practice. International Journal of Pharmacy Practice 2014, 22, pp. 7683; 2013. Case SM & DB Swanson . Constructing Written Test Questions For the Basic and Clinical Sciences, 3rd Edition (revised), NBME:Philadelphia, PA; 2003. Haladyna TM, SM Downing & MC Rodriguez. A Review of Multiple-Choice Item-Writing for Classroom Assessment. Applied Measurement in Education 15(3); 2002 Jones AT. Comparing Methods for Item Analysis: The Impact of Different Item-Selection Statistics on Test Difficulty Applied Psychological

Measurement October 2011 35: 566-571.

OMSEs resource on Blooms Taxonomy and multiple choice items Interactive guide to Blooms taxonomy, Iowa State University.

SOS Workshops

Something NEW!

SOS
education program.

Workshop Series
Just as OMSE provides workshops for faculty

The SOS Program will begin February 27, 2014.

The primary facilitator for these workshops will be Karen Spear Ellinwood, PhD, JD, OMSE Faculty instructional development. AHSC BioCommunications staff may provide additional assistance or source materials. Please click on the icons below to sign up or to find out more about the program schedule, workshop topics to be offered, .

to learn how to use technology for teaching, OMSE will now offer a series of workshops for support staff so they can enhance the effectiveness and efficiency in support of faculty, students and the medical

The Support for Office Staff (SOS) workshop series is not a certified skills program, such as is offered by The University of Arizona through its professional development courses in information technology and the IT Academy for Microsoft Products as part of its software licensing benefits (see Related Resources below). The SOS workshop series is aimed at assisting OMSE staff, including clerkship and elective/selective coordinators, in further developing skills in using office applications, such as Word, Excel, Project, Visio, and PowerPoint and Adobe Pro, in ways targeted to support the medical education program. Some workshops may offer opportunities to learn new software, while others will focus on learning to use more effectively the software we use every day.

SOS-RSVP form. If you are UA COM faculty and would like your support staff to participate in these workshops, please forward them the link to sign-up form online.

Program Description. Click on the download icon to find out more about the program.

FID Series #11

Strategies for Inquiry-based Learning
The goal of the presentation is to present strategies for engaging students in interactive, inquiry-based learning in classroom and clinical settings, and to provide an opportunity to practice the craft of framing questions for a variety of uses, such as formative assessment, feedback conversations, interactive lecture. Please send questions in advance or bring them to the seminar for discussion.

Date: 22 April, 2014 Time: 12:00 1:30 pm Location: COM-3230 RSVP: Click on RSVP stamp Refreshments will be served.

[Teaching with Technology|TWT]

T
effectively.

echnology Tips
OMSE provides a variety of opportunities for faculty to enhance teaching and assessment practices, including how to incorporate

Web-based tools for teaching & learning

technology into planning, teaching and formative assessment. Our new FID website contains many materials, strategies and ideas for podcasting, inquiry-based learning strategies. The TWT Workshop Series is a program for advancing faculty knowledge and skills in how to integrate technology into everyday teaching practice. The SOS Workshop Series offers indirect faculty support by providing hands-on guidance to support staffblock and clerkship coordinators, for example, in learning how to use office software more efficiently and

Sound Builder ED Radiology 2.0

The Teaching with Technology (TWT) Series offers six 2-hour workshops throughout the year and covers a variety of apps for clinical and classroom teaching. Below is the Series line-up: 1) 2) Poll everywhere, audience response software as polling technology [9 October 2013] Doceri as remote presentation app [9 December 2013] er detail (see Vol.2, No. 5). Mike Griffith, MS, Associate Director of AHSC BioCommunications, is the primary facilitator for all TWT Workshops. Mr. Griffith has a masters degree in education technology and many years experience working with a wide variety of devices and applications in the medical education context. Karen Spear Ellinwood, PhD, JD, OMSE Faculty instructional development, assists Mr. Griffith during workshops and provides follow-up tech support for participants learning or practicing with new technologies.

3D Brain

Notability

AnkiMobile Flashcards iAnnotate

3) Doceri as created material!

4) 5) 6) Flipping: Using Panopto to distribute your didactic material ahead of class Creating self-assessments using Apples iBook! Building online learning: using the AHSL studio.

Would you like to get more techno?

If you would like to learn how to use a specific technology or how to incorporate it into educational strategies for clinical or classroom teaching, please join us for the TWT Series.

Some of these applications can be accessed by using the icons above (upper right). The feature article of the December issue of Med/Ed eNews summarized the purpose and functionality of these apps in great-

NEXT TWT Workshop! RSVP

Date: 20 February 2014Time: 9-11 amRoom:

Please check out our website for faculty instructional development! More resources; new look & feel! Click the image below

M e d E d / e N e w s [7]

Save the date!

Date/Time Presentation Title Presenters

P rofessional D evelopment
The AMES\OMSE FID Series presents a topic relevant to teaching, assessment and/or medical education research from August through May each year. Please subscribe to our newsletter for current information on topics, presenters, and

3 March 2014 10:00 11:30 am [COM-3230]

FID Series Bedside Teach- Amy Waer, MD, and Paul ing Gordon, MD

Meet the FID team!

Director, Faculty Instructional Development Chris Cunniff, MD Office of Medical Student Education (Comstock House) Em. ccunniff@peds.arizona.edu Ph. 520.626.5173

P R
C
error. Click&Go!

special events.

OM accreditation selfstudy summary report available receptor

Instructional Development for all teaching faculty

esources

Karen Spear-Ellinwood, PhD, JD, EdS Associate Specialist for Faculty Development Office of Medical Student Education (COM-3215) Em. kse@medadmin.arizona.edu Ph. 520.626.1743

Accreditation visit dates: There are a variety of resources online for cliniJanuary 12-15, 2014 cal and classroom educators, including links to
learning modules, a guide for engaging students in inquiry learning, Microskills for teaching, and encouraging students to examine for cognitive

Educational Support for Residents and Fellows The AMES\OMSE Faculty Instructional Development (FID) Series is video recorded, so you can access recordings of seminars onlinesimply T. Gail Pritchard, Ph.D. Interim Senior Learning Specialist Office of Medical Student Education (COM-3210) Em. tpritcha@medadmin.arizona.edu Find out more about team learning , CBI and other instructional methods and the recently developed etools, ThinkShare (formerly GroupShare) and CBI ThinkShare (formerly, ThinkSpace). Check out our Faculty Instructional Development Calendar online! Assessment of Student Performance Susan Ellis, EdS, MA Program Manager for Assessment of Student Performance Office of Medical Student Education (COM-3215) Em. sellis@medadmin.arizona.edu Ph. 520.626-3654 Ph. 520-626-2390

MedEd eNews Volume 02 Issue 06 January 2014

Office of Medical Student Education ~ 1501 N. Campbell Avenue ~ Tucson, AZ 85724 ~ 520.626.1743 ~ fid.medicine.arizona.edu