Critical Thinking in Graduate Medical Education: A Role for Concept Mapping Assessment?

Daniel C. West; J. Richard Pomeroy; Jeanny K. Park; et al.

Online article and related content current as of November 10, 2008. JAMA. 2000;284(9):1105-1110 (doi:10.1001/jama.284.9.1105) http://jama.ama-assn.org/cgi/content/full/284/9/1105

Correction Citations Topic collections Related Articles published in the same issue

Contact me if this article is corrected. This article has been cited 20 times. Contact me when this article is cited.

Medical Practice; Medical Education

Contact me when new articles are published in these topic areas.

September 6, 2000
JAMA. 2000;284(9):1175.

Subscribe
http://jama.com/subscribe

Email Alerts
http://jamaarchives.com/alerts

Permissions
permissions@ama-assn.org http://pubs.ama-assn.org/misc/permissions.dtl

Reprints/E-prints
reprints@ama-assn.org

Downloaded from www.jama.com at Penn State Milton S Hershey Med Ctr on November 10, 2008

ORIGINAL CONTRIBUTION

Critical Thinking in Graduate Medical Education

A Role for Concept Mapping Assessment?
Daniel C. West, MD J. Richard Pomeroy, PhD Jeanny K. Park, MD Elise A. Gerstenberger, MD Jonathan Sandoval, PhD
URING THE PAST FEW DE cades medical curricula have increasingly reflected the belief that students should understand concepts, as opposed to rote memorization of facts or procedural algorithms. Many medical schools have adopted alternative teaching and learning strategies that challenge students to become self-directed learners who continuously expand the limits of their knowledge and seek evidence for clinical decision making.1 The root of these ideas lies in the constructivist theory of learning, which states that meaningful learning occurs when students assimilate new knowledge into a preexisting conceptual framework.2-9 This evolving conceptual framework supports expertise in critical thinking and clinical problem solving.10-12 Despite its critical importance in the education of future physicians, tools to evaluate the quality of students conceptual frameworks are limited because of the difficulty in designing and administering objective measures. In this article, we describe an application of concept mapping assessment (CMA) that provides a window into learners organization of knowledge at various points in training. Concept mapping is a technique developed by Jo-

Context Tools to assess the evolving conceptual framework of physicians-intraining are limited, despite their critical importance to physicians evolving clinical expertise. Concept mapping assessment (CMA) enables teachers to view students organization of their knowledge at various points in training. Objective To assess whether CMA reflects expected differences and changes in the conceptual framework of resident physicians, whether concept maps can be scored reliably, and how well CMA scores relate to the results of standard in-training examination. Design, Setting, and Participants A group of 21 resident physicians (9 first-year and 12 second- and third-year residents) from a university-based pediatric training program underwent concept map training, drew a preinstruction concept map about seizures, completed an education course on seizures, and then drew a postinstruction map. Maps were scored independently by 3 raters using a standardized method. The study was conducted in May and June 1999. Main Outcome Measures Preinstruction map total scores and subscores in 4 categories compared with postinstruction map scores; map scores of second- and thirdyear residents compared with first-year residents; and interrater correlation of map scores. Results Total CMA scores increased after instruction from a mean (SD) preinstruction map score of 429 (119) to a mean postinstruction map score of 516 (196) (P=.03). Second- and third-year residents scored significantly higher than first-year residents before instruction (mean [SD] score of 472 [116] vs 371 [102], respectively; P= .04), but not after instruction (mean [SD] scores, 561 [203] vs 456 [179], respectively; P=.16). Second- and third-year residents had greater preinstruction map complexity as measured by cross-link score (P=.01) than first-year residents. The CMA score had a weak to no correlation with the American Board of Pediatrics In-training Examination score (r=0.10-0.54). Interrater correlation of map scoring ranged from weak to moderate for the preinstruction map (r= 0.51-0.69) and moderate to strong for the postinstruction map (r= 0.74-0.88). Conclusions Our data provide preliminary evidence that concept mapping assessment reflects expected differences and change in the conceptual framework of resident physicians. Concept mapping assessment and standardized testing may measure different cognitive domains.
JAMA. 2000;284:1105-1110 www.jama.com

seph Novak in the 1970s in which students create a 2-dimensional diagram outlining their understanding of the relationships between and among important concepts within a given subject area.13-15 Unlike most conventional ex-

Author Affiliations: Department of Pediatrics, School of Medicine (Drs West, Park, and Gerstenberger) and Division of Education (Drs Pomeroy and Sandoval), University of California, Davis. Corresponding Author and Reprints: Daniel C. West, MD, Department of Pediatrics, University of California, Davis, 2516 Stockton Blvd, Sacramento, CA 95817 (e-mail: dcwest@ucdavis.edu). 1105

2000 American Medical Association. All rights reserved.

(Reprinted) JAMA, September 6, 2000Vol 284, No. 9

Downloaded from www.jama.com at Penn State Milton S Hershey Med Ctr on November 10, 2008

CONCEPT MAPPING ASSESSMENT

aminations, CMA may provide unique insights into how an individual organizes his or her knowledge or comes to think about a problem. While concept mapping has been used frequently as a learning tool, especially in high school and college science education, it has only rarely been used as an assessment method, primarily because there is limited information regarding its validity and reliability.12,16-18 Therefore, we undertook this preliminary study to assess whether CMA can measure predicted differences and expected changes in the conceptual framework of resident physicians. In addition, we sought to determine whether concept maps could be scored in a reliable way. METHODS
Study Subjects and Design

All pediatric residents (33 total) at our institution were eligible to participate in this study, which was approved by our institutional human subjects committee. Written informed consent was obtained. A total of 21 residents enrolled: 9 first-year residents, 8 secondFigure 1. Example Concept Map
Water
changes needed by made of

year residents, and 4 third-year residents. Each participant underwent concept map training followed by drawing a preinstruction concept map about the topic seizures. They then participated in a 3-session seizure education course followed by drawing a postinstruction map about seizures. All study subjects participated in a standardized concept map training exercise based on previously described methods.14 Within 2 to 3 weeks of completing map training, participants were given 30 minutes to independently, and without study aids, draw their concept of seizures on a blank sheet of paper (preinstruction map). Residents then participated in a 3-session (1 hour per session) seizure education course consisting of case-based instruction. One week after completing the course, residents drew a postinstruction concept map about seizures under the same conditions as before.
Concept Maps

We used a modification of the hierarchical technique of concept mapping in which a concept map consists of 5

HIERARCHY

States

LEVEL 1

Living Things

Molecules
determines (cross-link) are in

can be

can be

can be

Solid

Liquid

Gas

LEVEL 2

is a type of is a type of

Motion
as in as in

as in

as in

Plants
example

Lake
example

Steam

LEVEL 3

oak tree

Animals
example

Snow

Ice Lake Tahoe

components: concepts, concept links, hierarchies, cross-links, and examples (see FIGURE 1 for an example concept map of water).14,19 A concept is defined as a perceived regularity in events or objects designated by a label and, when depicted on a map, enclosed with a circle. Creating a concept map involves connecting related concepts using arrow lines with a statement or proposition written above the line describing how the concepts are related. A concept linked to another concept by an arrow line with a propositional statement, called a concept link, represents the foundational unit of a concept map.3,14,20,21 In Figure 1, the concept water connected to living things by the proposition needed by represents a concept link. The expression of the hierarchy between concepts is indicated by the direction of the arrow in the concept link with the most general concepts arranged at the top and more specific concepts arranged hierarchically below. Three levels of hierarchy are represented in Figure 1. The linking of a series of related concept links in a hierarchical fashion represents a domain of knowledge. Cross-links are the method by which the map author indicates how different domains of knowledge are related to each other and are a critical measure of map complexity. In a manner similar to propositions, crosslinks are labeled with a phrase that describes the relationship of the domains. In the example map, motion connected to states by the proposition determines defines a cross-link. Examples are labeled and connected to the related concept in the most subordinate position on the map. In Figure 1, Lake Tahoe is a specific example of a lake. Theoretically, as a learners conceptual framework changes, he or she links new propositions sequentially (concept links), general to specific (hierarchy), and across domains (cross-links), thus demonstrating integration of knowledge.22
Scoring Method

my dog

Example map of water demonstrating the basic components of hierarchical concept maps. Note the cross-link (bold arrow) between the concepts motion and states. Reproduced with permission from Cambridge University Press.14 1106 JAMA, September 6, 2000Vol 284, No. 9 (Reprinted)

Maps were scored independently by 3 different raters, blinded to the identity

2000 American Medical Association. All rights reserved.

Downloaded from www.jama.com at Penn State Milton S Hershey Med Ctr on November 10, 2008

CONCEPT MAPPING ASSESSMENT

of the map author, and using a modification of the structural scoring method previously described.7,14 Each rater was a board-certified pediatrician who underwent a 30-minute concept map scoring training exercise immediately prior to grading the maps. Maps were scored in 4 categories: concept links, hierarchy, cross-links, and examples. The value of each scoring category represents the depth of thinking required to form the category.7,14 Each valid concept link was given 2 points, and each level of hierarchy was given 5 points. A cross-link was given 10 points, while an example counted for 1 point. Invalid links or concepts were given 0 points. A total score and subscore for each category (concept links, crosslinks, hierarchy, and examples) were generated for each map. Standard total scores and standard subscores consisting of the sum of individual rater total scores and subscores, respectively, were assigned to each map.
Data Analysis

study, while third-year residents took the examination 10 months prior to completing the study. We then tested the relationship of concept map scores to ITE scores using Spearman rank correlation. RESULTS All 21 participants completed all phases of the study including concept map training, the seizure education course, and both preinstruction and postinstruction concept maps.
Concept Map Scores Based on Level of Training

Statistical analysis was done using STATA, version 6.0 (College Station, Tex). The correlation of each raters total scores and subscores for each map was tested using Spearman rank correlation. All other data analyses were done using standard total scores and standard subscores. We compared preinstruction map to postinstruction map scores using the Wilcoxon signed rank test. To assess whether there was a difference in map scores based on level of training or experience, we combined the more experienced second- and thirdyear residents (n=12) into 1 group and compared their scores with those of the first-year (n = 9) residents using the 2-sample Wilcoxon rank sum test. To assess the relationship of concept map scores to standard written examinations, we compared the most recent American Board of Pediatrics Intraining Examination (ITE) scores of second- and third-year residents to firstyear residents using the 2-sample Wilcoxon rank sum test. First- and secondyear residents took the examination within 1 month of completing this

F IGURE 2 shows a complete lowscoring concept map from a first-year resident, while FIGURE 3 shows a highscoring map from a resident in the second- and third-year group. Note the complexity, with numerous crosslinks in the map from the second- and third-year group compared with the linear simplicity of the first-year residents map. TABLE 1 summarizes the mean standard total and mean standard subscores of study participants by level of training. The second- and thirdFigure 2. Low-Scoring Concept Map
Seizures
includes many is treated with

year residents had significantly higher total (P =.04) and cross-link scores on the preinstruction map (P=.01) compared with first-year residents. There was no significant difference between concept link, hierarchy, and example scores based on level of training. The postinstruction maps demonstrated that while the total and subscores for second- and third-year residents remained higher, these differences did not reach significance. Of particular note is that the significant difference in total score and cross-link score seen on the preinstruction maps was no longer apparent after instruction (P =.16 and .15, respectively).
Concept Map Scores After Educational Intervention

Total scores increased significantly after the educational intervention (P=.03). In addition, cross-link (P=.02) and concept link (P=.01) subscores also increased significantly. Subscores for hierarchy and examples both decreased slightly, but this change did not reach significance. Preinstruction and postinstruction maps from 1 subject are

has numerous

Types Medications
includes includes including

Etiologies
includes includes

Secondary
includes

Primary

Phenytoin
including including

Vascular
includes

Trauma

including

includes

includes

Phenobarbital Absence
includes includes includes

Carbamazepine Stroke

Infections
includes

Myoclonic Jerks

includes

Valproic Acid

Electrolyte Disturbances
example example includes

Meningitis

Tonic-Clonic

Complicated Febrile

low sodium low glucose

Encephalitis

Common Febrile

Reproduction of a hand-written concept map of seizures by a first-year resident. Note the absence of crosslinks and the consistent use of 2 to 3 levels of hierarchy, resulting in a low score. (Reprinted) JAMA, September 6, 2000Vol 284, No. 9 1107

2000 American Medical Association. All rights reserved.

Downloaded from www.jama.com at Penn State Milton S Hershey Med Ctr on November 10, 2008

CONCEPT MAPPING ASSESSMENT

Figure 3. High-Scoring Concept Map

Seizures
can be caused by caused by caused by can be usually treated with

Medications
can be can be

Epilepsy
requires caused by

Fever
due to

Generalized sometimes
treated with type of

Diet

Lorazepam Phenobarbital
can cause can cause

example

Education
as in prevention of as in as in

Infection Abnormal Glucose

can be as in can be

Respiratory
leads to

ketogenic diet

Toxicity

Medications

Trauma Low Glucose

due to

Increased Intracranial Pressure

can cause

type of Neurologic type of

Hypoxia
type of

Respiratory Depression

911 Call

Nonaccidental
type of as in

Brain Abscess
caused by caused by caused by

Encephalitis
type of caused by

can result in due to

Shaken Baby
caused by

Fall

Accidental

Ingestion

High Glucose
due to

Meningitis
caused by caused by caused by Virus spread by prevented by Insects spread by spread by example Humans prevented by

can result in

Neuronal Shear causes

leads to requires causes as in as in requires

as in often implicated

Poor Intake

Fungus

Cerebral Edema
results in

Diabetes
can be

Tumor
can be

Bacteria

Acute Hypertension

treated with

Intracranial Bleed
detected by

Motor Vehicle Accident Prevention

required by

Alcohol Bike Accident

Primary
as in

Metastatic
as in

mosquito

Osmotic Diuretic
example

Astrocytoma

Breast Cancer

Immunizations

NeuroImaging

mannitol as in

Law
as in

Helmets

Seat Belts

Reproduction of a hand-written concept map of seizures by a resident in the second- and third-year group. Note the complexity of the map as manifested by frequent cross-links and the consistent use of 5 to 7 levels of hierarchy, resulting in a higher map score.

Table 1. Comparison of Total Concept Map and Subscores in Second- and Third-Year vs First-Year Residents*
Secondand Third-Year Residents

Component Total Cross-link Concept link Hierarchy Examples Total Cross-link Concept link Hierarchy Examples

First-Year Residents

P Value .04 .01 .86 .74 .80 .16 .15 .24 .59 .75

Preinstruction Map 472 (116) 371 (102) 161 (87) 71 (46) 182 (83) 173 (58) 68 (25) 59 (11) 64 (19) 67 (27) Postinstruction Map 561 (203) 456 (179) 215 (136) 131 (111) 233 (78) 211 (78) 62 (20) 61 (25) 59 (19) 54 (27)

sure. The vague concept cerebral irritation included as a focal point in the preinstruction map was omitted in the postinstruction map. TABLE 2 summarizes the mean standard score and subscores of all subjects before (preinstruction map) and after (postinstruction map) the seizure education course. Total scores increased significantly after the educational intervention (P =.03). In addition, cross-link (P=.02) and concept link (P = .01) subscores also increased significantly. Subscores for hierarchy and examples both decreased slightly, but this change did not reach significance.
Concept Map Scores and ITE

There was weak to no correlation between CMA scores (total and subscores) and ITE scores (r=0.10-0.54). The strongest correlation was between ITE score and the example score on the preinstruction (r =0.54; P=.01) and postinstruction map total score (r = 0.53; P = .01). The correlation between ITE scores and postinstruction map total scores, preinstruction map cross-link scores, and postinstruction map cross-link scores was r=0.29, 0.34, and 0.20. respectively.
Interrater Correlation of Concept Map Scores

*Data are presented as mean (SD) standard scores. See

Scoring Method subsection for explanation of scoring method. For comparison by 2-sample Wilcoxon rank sum test.

shown in FIGURE 4. The resident added a new conceptual domain for prevention and education as well as new crosslinks to increased intracranial pres-

The mean (SD) ITE score of secondand third-year residents was 358 (135) and of first-year residents, 243 (94). The ITE scores of second- and third-year residents were significantly higher than those of first-year residents (P = .05).

The interrater correlation was weak to moderate (r=0.51-0.69) for the preinstruction map and moderate to strong for the postinstruction map (r =0.74-0.88). The interrater correlation for subscores demonstrated an almost identical range to that seen in the analysis of total scores.

1108 JAMA, September 6, 2000Vol 284, No. 9 (Reprinted)

2000 American Medical Association. All rights reserved.

Downloaded from www.jama.com at Penn State Milton S Hershey Med Ctr on November 10, 2008

CONCEPT MAPPING ASSESSMENT

COMMENT We found that concept map scores of residents increased significantly following completion of an educational program after which conceptual framework would be expected to change. This change is perhaps best illustrated by qualitative comparison of the preinstruction and postinstruction maps (eg, Figure 4), which demonstrates the incorporation of additional concepts into the postintervention map, further differentiation of existing concepts, and a change in cross-linking between concepts. Quantitatively, these changes are reflected in a statistically significant increase in total map score and concept link and cross-link subscores. Despite the small sample size, we also found that CMA scores of residents with more training and expertise were significantly higher than those with less training. This difference was the result of greater map complexity, as reflected in the significantly higher cross-link scores of second- and third-year residents. Thus, more experienced residents appeared to organize their knowledge differently, expressing links between concepts that were not observed by less experienced residents. It is important to note that this difference narrowed, as one might anticipate, after all residents participated in the same educational intervention. While not conclusive, these findings provide important preliminary evidence supporting our hypothesis that CMA is a valid measure of conceptual framework and the expected change in that framework with new learning. We also found that while ITE scores of second- and third-year residents were significantly higher than those of firstyear residents, these scores did not correlate well with CMA scores. Thus, residents who scored highly on CMA were not the same residents who scored highly on the ITE examination. This finding is consistent with other reports, which note that CMA scores of college students do not correlate with conventional measures of learning, such as final course grades or scores on standardized tests.16 While there may be other explanations, the absence of a positive correla-

Figure 4. Partial Reproduction of Preinstruction and Postinstruction Concept Maps From the Same Subject
Preinstruction Map
Seizures
treated by may be may be

Focal
caused by caused by

Antiepileptic drugs
example

Generalized
caused by

Dilantin
example

example

Phenobarbital Tegretol Cancer

can be

Cerebral Space Irritation Occupying Masses caused by Meningitis

caused by leading to

Infection
leading to

can be

Toxins
like

Encephalitis
caused by caused by caused by

Infectious

Lead

Bacteria

Viral

Postinstruction Map
Seizures
can be caused by can be caused by can be caused by as prevention of as in

Trauma
as in

Fever

can be caused by

Education

can be caused by

like

Nonaccidental

Toxin Accidental
as in as in

Abnormal Glucose
which leads to

Increased Intracranial Pressure

can be

Infection
type of

as in as in

Respiratory

Shaken Baby
causing

causing causing

Neurologic
leads to

Blood
as in

Bike Accident
can be as in as in due to

Neuronal Shear
leading to

Intracranial Bleed
requires

Fall

High Low
due to

as in as in

Encephalitis

which leads to requires

Cerebral Edema

Motor Vehicle Accident

often implicated in

Ingestion
as in

Brain Abscess

Meningitis

Diabetes

Prevention

Alcohol

Reproduction of approximately 50% of both a preinstuction and postinstruction concept map from the same subject.

tion suggests that CMA measures a different knowledge characteristic than do multiple choice examinations. The issue of scoring reliability is critically important to any future use of CMA. We found that interrater reliability was moderate to strong and, notably, was greater on postinstruction maps. This level of scoring reliability is similar to that reported by others and suggests that gains in scoring experience lead to increased consistency across raters.23 Therefore, rater training would be important to any future applications. The vast majority of previous research regarding concept mapping has involved high school and college science education. In reviewing this litera-

Table 2. Comparison of Total Concept Map and Subscores Before and After Educational Intervention*
PrePostinstruction instruction P Component Map Map Value Total 429 (119) 516 (196) .03 Cross-link 123 (85) 179 (130) .02 Concept link 178 (72) 223 (83) .01 Hierarchy 65 (20) 61 (22) .44 Examples 65 (32) 57 (23) .06
*Data are presented as mean (SD) standard scores. See
Scoring Method subsection for explanation of scoring method. For comparison by Wilcoxon signed rank test.

ture, there is clear evidence that concept mapping can be an effective learning tool in a variety of classroom settings including the education of health care
1109

2000 American Medical Association. All rights reserved.

(Reprinted) JAMA, September 6, 2000Vol 284, No. 9

Downloaded from www.jama.com at Penn State Milton S Hershey Med Ctr on November 10, 2008

CONCEPT MAPPING ASSESSMENT

professionals.16-18,24-30 However, evidence supporting the validity of concept mapping as an assessment tool is much more limited, probably because there is no criterion standard measure of conceptual framework to which CMA can be compared. Traditional measures of competence, such as board scores or even newer competency-based assessments (eg, Objective Structured Clinical Examinations), may not represent appropriate comparisons since they shed little light on how a learner thinks about a problem.12 Thus, other investigators have used the same strategy to build a case for the validity of CMA. For example, in a variety of secondary school and undergraduate settings, CMA has consistently been found to measure conceptual change in situations in which such change would be expected and to detect differences between students with more training in a particular field compared with those with less.14,16-18,22,31,32 In medical education, we identified only a
REFERENCES 1. Tosteson DC. New pathways in general medical education. N Engl J Med. 1990;322:234-238. 2. Piaget J. The Language and Thought of the Child. 3rd ed. London, England: Routledge & Kegan Paul; 1959. 3. Ausubel DP. The Psychology of Meaningful Verbal Learning. New York, NY: Grune & Stratton; 1963. 4. Ausubel DP, Novak JD, Hanesian H. Educational Psychology: A Cognitive View. 2nd ed. New York, NY: Holt Rinehart & Winston; 1978. 5. Markow PG, Lonning RA. Usefulness of concept maps in college chemistry laboratories: students perceptions and effects on achievement. J Res Sci Teach. 1998;35:1015-1029. 6. Bodner GM. Constructivism: a theory of knowledge. J Chem Educ. 1986;63:873-878. 7. Mintzes JJ, Wandersee JH, Novak JD. Teaching Science for Understanding: A Constructivist View. San Diego, Calif: Academic Press; 1998. 8. Posner GJ, Strike KA, Hewson PW, Gertzog WA. Accommodation of a scientific conception: toward a theory of conceptual change. Sci Educ. 1982;66:211-227. 9. von Glaserfeld E. The Practice of Constructivism in Science and Mathematics Education. Washington, DC: American Association for the Advancement of Science Press; 1992. 10. Glaser R, Bassok M. Learning theory and the study of instruction. Annu Rev Psychol. 1989;40:631-666. 11. Royer J, Cisero CA, Carlo MS. Techniques and procedures for assessing cognitive skills. Rev Educ Res. 1993;63:201-224. 12. Ruiz-Primo MA, Shavelson RJ. Problems and issues in the use of concept maps in science assessment. J Res Sci Teach. 1996;33:569-600. 13. Novak JD. Audio-tutorial techiques for individualized science instruction in the elementary school. In:

single related study in which Pathfinder networks, a substantially different form of concept mapping, was used to assess student understanding of pulmonary physiology.33 In that study, the investigators found that, after instruction, student Pathfinder networks became more similar to the networks generated by faculty experts. These findings appear consistent with our study, and while not conclusive, provide additional evidence suggesting the predictive validity of CMA. We have provided encouraging preliminary evidence supporting the validity of CMA as a way to measure conceptual change and differences among resident physicians. Our findings also indicate that rater training is critical to the use of CMA. Future research must focus on further validating this method and addressing issues of reliability before CMA can be applied on a larger scale. Nevertheless, CMA has the potential to evaluate how students or resiTriezenberg H, ed. Individualized Science: Like It Is. Washington, DC: National Science Teachers Association; 1972:14-30. 14. Novak JD, Gowin DB. Learning How to Learn. Cambridge, England: Cambridge University Press; 1984:15-54. 15. Novak JD, Musonda D. A twelve year study of science concept learning. Am Educ Res J. 1991;28: 117-153. 16. Novak JD, Gowin DB, Johansen G. The use of concept mapping and knowledge vee mapping with junior high school science students. Sci Educ. 1983;67: 625-645. 17. Markham KM, Mintzes JJ, Jones MG. The concept map as a research and evaluation tool: further evidence of validity. J Res Sci Teach. 1994;31:91-101. 18. Beyerbach BA. Concept mapping as an approach to assessment of students representation of structural knowledge. Dissertation Abstracts Int. 1986; 46:2622A. 19. Novak JD. Concept mapping: a strategy for organizing knowledge. In: Glynn SM, Duit R, eds. Learning Science in the Schools: Research Reforming Practice. Mahwah, NJ: Lawrence Erlbaum Associates; 1995: 229-245. 20. Novak JD. Applying learning psychology and philosophy of science to biology teaching. Am Biol Teach. 1981;43:13-20. 21. Novak JD. Concept mapping: a useful tool for science education. J Res Sci Teach. 1990;27:937-949. 22. Arnaudin MW, Mintzes JJ, Dunn CS, Shafer TH. Concept mapping in college science teaching. J College Sci Teach. 1984;14:117-121. 23. McClure JR, Sonak B, Suen HK. Concept map assessment of classroom learning: reliability, validity, and logistical practicality. J Res Sci Teach. 1999;36:475-492.

dents organize and use knowledge in a way that traditional objective tests cannot. It also could provide a tool for educators to identify unique distortions in students understanding of content and to identify errors of omission. Finally, CMA might provide insight into why some residents score well on objective written examinations but have difficulty applying this knowledge to clinical situations.
Funding/Support: This work was supported in part by grants from the Childrens Miracle Network Research Fund and the Hibbard E. Williams Research Fund at the University of California, Davis. Previous Presentation: Presented in part at the Pediatric Academic Societies Joint Meeting, Boston, Mass, May 16, 2000. Acknowledgment: We thank Luann Cruys, Debbie Barnes, and Sally Colvin for invaluable administrative support without which this study could not have been completed and Anthony F. Philipps, MD, Department of Pediatrics, for critical review of the manuscript. We also are indebted to Constance Bowe, MD, Department of Neurology, University of California, Davis, for her dedication to medical education and developing and teaching the seizure education course. Finally, we thank the pediatric residents at UC Davis for participating in this study and their willingness to explore a new way of thinking about assessment.

24. Roletto E, Regis A, Albertazzi PG. Concept maps in chemistry education. J Chem Ed. 1996;73:10841088. 25. Lahtinen V, Lonka K, Lindbloom-Ylaenne S. Spontaneous study strategies and the quality of knowledge construction. Br J Educ Psychol. 1997;67: 13-24. 26. Coleman EB. Using explanatory knowledge during collaborative problem solving in science. J Learn Sci. 1998;7:387-427. 27. Heinze-Fry JA, Novak JD. Concept mapping brings long-term movement toward meaningful learning. Sci Educ. 1990;74:461-472. 28. Edmondson KM, Smith DF. Concept mapping to facilitate veterinary students understanding of fluid and electrolyte disorders. Teach Learn Med. 1998;10: 21-33. 29. Chastonay P, Papart JP, Laporte JD, et al. Use of concept mapping to define learning objectives in a master of public health program. Teach Learn Med. 1999; 11:21-25. 30. Edmondson KM. Concept maps and the development of cases for problem-based learning. Acad Med. 1994;69:108-110. 31. Bar-Lavie BZ. Enhancing meaningful learning in an environmental education program: a case study of class empowered through the use of Novaks and Gowins principles of learning how to learn, concept mapping, interviewing, and educating. Dissertation Abstracts Int. 1988;48:2590A. 32. Stuart HA. Should concept maps be scored numerically. Eur J Sci Educ. 1985;7:73-81. 33. McGaghie WC, Boerger RL, McCrimmon DR, Ravitch MM. Learning pulmonary physiology: comparison of student and faculty knowledge structures. Acad Med. 1996;71(suppl):S13-S15.

1110 JAMA, September 6, 2000Vol 284, No. 9 (Reprinted)

2000 American Medical Association. All rights reserved.

Downloaded from www.jama.com at Penn State Milton S Hershey Med Ctr on November 10, 2008