Documente Academic
Documente Profesional
Documente Cultură
By Jane Metzger, Emily Welebob, David W. Bates, Stuart Lipsitz, and David C. Classen
doi: 10.1377/hlthaff.2010.0160
David C. Classen is an
associate professor of
medicine at the University of
M
any people have suggested In this application of clinical decision support, Utah in Salt Lake City, and is
that electronic health rec- physicians are made aware of potential safety also with CSC Healthcare.
ords represent essential infra- issues that can result—for example, when ampi-
structure for the provision of cillin is given to a patient with a known allergy to
safe health care in the United penicillin, or the dose being ordered for a pedi-
States. For several years, the Institute of Medi- atric patient is much higher than the therapeutic
cine, the Leapfrog Group, the National Quality range for a child of this age and weight. Prescrib-
Forum, and other national groups concerned ing errors such as these can lead to anaphylaxis
about patient safety have recommended, in par- or seizures, which are known as adverse drug
ticular, widespread adoption of electronic health events, if the medications are actually adminis-
records with computerized physician order tered. The goal of medication safety decision
entry.1–4 support in computerized physician order entry
is to prevent these types of serious errors as the
orders are being written.
Background On Decision-Support A study demonstrated that one in ten patients
Tools hospitalized in Massachusetts suffered an ad-
With computerized physician order entry, phy- verse drug event that could be prevented by
sicians and other licensed clinicians write their decision-support tools in computerized physi-
orders for hospitalized patients electronically. cian order entry.8 The study spurred the passage
This recommendation is based in large part on of legislation requiring Massachusetts hospitals
demonstrations by pioneering organizations. to implement computerized physician order
The organizations have shown that important entry by 2012 as a condition of licensure.9
improvements in safety can be achieved when More recently, “meaningful use” of computer-
rules-based decision support aids in averting ized physician order entry and clinical decision
medication errors and adverse events by provid- support has been singled out as a requirement
ing advice and warnings as physicians write or- for hospitals to qualify for new financial incen-
ders using a specially programmed computer.5–7 tives. These incentives will be offered under the
health information technology (IT) stimulus scribing errors in “live” hospital settings.
provisions of the American Recovery and Rein- The tool was intentionally designed to give
vestment Act (ARRA) of 2009. individual hospitals detailed and specific feed-
Computerized physician order entry interacts back on their performance and to give purchas-
with other applications in the suite of digital ers, through Leapfrog, an overall score for the
tools that constitute the inpatient electronic hospital that can be used for benchmarking
health record (for example, to obtain infor- purposes.17,18
mation on allergies and patients’ weight) and This assessment complements efforts by the
is typically one of the later modules to be imple- Certification Commission for Health Informa-
mented. Hospitals’ adoption of the compu- tion Technology (CCHIT) to evaluate the capa-
terized physician order entry module is in- bilities available in vendors’ electronic medical
creasing, but slowly.10,11 Several reports have record products “on the shelf.” It evaluates how
suggested that the successful application of products were implemented and are actually
decision support achieved among pioneering being used in hospitals.
organizations is not being replicated and that The development of the tool was initially
implementation can create new problems.12–15 funded by the Robert Wood Johnson Foun-
This report summarizes results for sixty-two hos- dation, the California HealthCare Founda-
pitals across the United States that used a new tion, and the Agency for Healthcare Research
simulation tool to assess their use of medication and Quality, and was completed in 2006. In
safety decision support in electronic health April 2008 the assessment was incorporated into
records with computerized physician order the Leapfrog Annual Safe Practices Survey for the
entry. first time, and hospitals completing the assess-
ment received a feedback report. Beginning in
2009, assessment results were also factored into
Study Data And Methods determining the extent to which the computer-
History Of The Assessment Tool The impetus ized physician order entry implementation met
for developing the assessment tool was initially the Leapfrog standard.
the standard developed by the Leapfrog Group. Design Of The Assessment Tool The assess-
This is an employer group that seeks to accom- ment methodology is modeled after tools that are
plish breakthroughs, or “big leaps,” in hospital commonly used in other industries. It mimics
patient safety through a combination of public what happens when a physician writes an order
awareness and rewards to higher-quality provid- for an actual patient in the implemented elec-
ers. The group selected computerized physician tronic health record with computerized physi-
order entry as one of the first three leaps in 2001. cian order entry. But it uses test patients—in
There was accumulating evidence concerning effect, fictitious patients created for purposes
the frequency, tragic consequences, and finan- of the assessment—and test orders.
cial costs of adverse drug events in hospitalized A group of experts on adverse drug events, as
patients—and computerized physician order en- well as the use of decision support in computer-
try and decision support had demonstrated the ized physician order entry to decrease adverse
ability to help avert many of them.16 drug events, developed test orders that are
The Leapfrog standard includes two elements judged likely to cause serious harm (rather than
of meaningful use to ensure that computerized those with low potential for harm). The test
physician order entry has been implemented in orders belong to the categories of adverse
such as way as to improve medication safety. drug events (such as drug-to-allergy or drug-
According to the standards, physicians and other to-diagnosis contraindication) that prior re-
licensed providers must enter at least 75 percent search shows cause the most harm to patients.
of medication orders using computerized entry. In most cases, they are actual orders that have
Clinical decision support must also be able to caused adverse drug events, taken from primary
avert at least 50 percent of “common, serious adverse drug event data collection studies. The
prescribing errors.”16 assessment offers a one-time, cross-sectional
Clinical decision support in this setting is the look at whether decision support provides advice
logic built into the computerized physician order to a physician writing such an order.18,19
entry system that, for example, checks to see if Decision support for this purpose is a set of
ampicillin has been ordered for a patient who is tools or logic that can be integrated into the
known to be allergic to penicillin. This tool was computerized physician order entry system to
developed to specifically measure the ability of suggest appropriate orders (such as a dose cal-
implemented electronic health record systems culator or a reminder to consider renal function)
with computerized physician order entry to de- or to critique them once they have been entered,
tect and avert these common yet serious pre- as through a message or an alert. The assessment
such as test order detected (yes, no), percentages that can be addressed by basic decision support
were calculated. However, to account for the (61 percent) than for those requiring more ad-
hierarchical nature of the order data, or the test vanced decision support (25 percent).
orders nested within hospitals, the 95 percent Aggregate Results: Adverse Drug Events
binomial confidence intervals for percentages When results for all hospitals were pooled, the
were adjusted using the approach described by adverse drug event category detected most reli-
David Williams.23 ably was drug-to-allergy contraindication. Much
All analyses were conducted using the statis- higher scores were obtained for each of the cat-
tical software package SAS 9.2. All tests were two- egories addressed by basic clinical decision sup-
tailed, and a p value less than 0.05 (therefore port than for those requiring advanced tools
not likely to be due to chance) was considered (Exhibit 4).
statistically significant. Drug-to-diagnosis contraindication includes
pregnancy, which was also analyzed separately.
These potential adverse drug events were only
Study Results detected 15 percent of the time.
The types of hospitals included in the sample The set of test orders for each hospital includes
were broadly representative of larger U.S. hospi- four that are judged to result in patient fatality.
tals (Exhibit 1). Nearly two-thirds were teaching When results for this subset were analyzed, we
hospitals. The higher representation of teaching found that 47 percent (95 percent confidence
hospitals and lower representation of smaller interval: 36.9–57.6) were not detected by the
hospitals is consistent with the current pattern decision support in use in these hospitals.
of adoption of computerized physician order en- Although hospitals do have pharmacy and nurs-
try in hospitals.10,11 Among the sixty-two hospi- ing review processes in place that sometimes
tals, all but one reported using electronic health catch orders like these before the medication
record applications including computerized reaches the patient, these medication orders
physician order entry from one of seven commer- are far outside safe limits and would never be
cial vendors. appropriate physician orders.
Individual Hospitals Scores for individual Contributing Factors The information
hospitals ranged from 10 percent to 82 percent available for exploring contributing factors
of test orders detected. The scores for the top was limited to the vendor software solution in
10 percent, or six hospitals, ranged from 71 per- use, teaching status, hospital size by number of
cent to 82 percent. Scores for the six hospitals beds, and whether or not the hospital was part of
with the lowest scores ranged from 10 percent to a health system. We assessed the relationship
18 percent (Exhibit 2). between performance on the assessment and
Mean hospital scores (Exhibit 3) were higher these factors.
for orders that would lead to adverse drug events High-low scores for hospitals using the same
EXHIBIT 1
Characteristics Of Hospitals Participating In The Study Of Computerized Physician Order Entry (CPOE), 2008
Participating hospitals
Characteristic Number Percent All U.S. hospitals (%) Hospitals with CPOE (%)
HOSPITAL SIZE (BEDS)
<50 0 0 32.9 8.9
50–99 3 4.8 15.9 11.8
100–199 14 22.6 22.4 12.8
200–299 8 13 12.7 15.3
300–399 9 15 7.7 30.1
400–799 23 37 7.3 33.9
800+ 5 8 0.9 41.2
TEACHING STATUS
Teaching 39 63 21.7 –
Nonteaching 23 37 78.3 –
SYSTEM STATUS
Independent 15 24 45 –
Multihospital health system 47 76 55 –
SOURCES: Authors’ analysis; and Notes 10 (for hospital size data) and 11 (for teaching and system status) in text.
658 HE A LT H A FFA IR S A P R I L 2 0 10 2 9 :4
EXHIBIT 2
Hospital Scores For Detection Of Test Orders That Would Cause An Adverse Drug Event In An Adult Patient According To
The Software Product (Vendor) Implemented
Hospital score (percent)
computerized physician order entry software In a multiple regression model, vendor choice
product ranged by as much as 40–65 percent was significantly correlated with performance
(Exhibit 2). Some hospitals using each product (p ¼ 0:009, or not likely to be due to chance).
detected at least 50 percent of the potential ad- This means that there is good statistical evidence
verse drug events. The six top-performing hos- to suggest that choice of vendors does have some
pitals used six different software products: one positive effect on performance. However, vendor
homegrown solution and five vendor products. choice accounted for only 27 percent of the total
EXHIBIT 3
Hospital Mean Scores For Detecting Test Orders For Adult Patients Corresponding To Adverse Drug Event Categories
Addressed By Basic And Advanced Decision Support
Percent of test orders detected
Adverse drug event categories grouped according to
level of clinical decision support Mean (SE) Median Interquartile range
Basic—relatively easy to implementa 61.4 (2.4) 61.1b 53.9–76.2
Advanced—requires more configuration or customizationc 24.8 (2.6) 18.8b 5.9–38.9
Overall score 44.3 (2.3) 41.7 31.6–57.1
SOURCE Authors’ analysis. NOTES N ¼ 62 hospitals. SE is standard error. Interquartile range is the range in which the middle 50 percent
of the observations are seen—25 percent below the median, 25 percent above. ap < 0:0001 using a paired t-test for Basic
Score = Advanced Score. bDrug-to-drug or drug-to-allergy contraindication, inappropriate single dose, therapeutic duplication, and
inappropriate route. cInappropriate cumulative (daily) dose, inappropriate dose (patient weight), age or diagnosis contraindication,
contraindication based on renal function or other condition indicated by laboratory tests, lack of monitoring.
AP R I L 2 0 1 0 2 9 :4 HE A LT H A FFA IR S 659
Focus On Quality
EXHIBIT 4
Pooled Hospital Scores For Detecting Test Orders For Adults In Various Adverse Drug Event Categories
Lower 95 percent Upper 95 percent
Adverse drug event category Percent detected confidence interval confidence interval
a
ADDRESSED BY BASIC CLINICAL DECISION SUPPORT
SOURCE Authors’ analysis. NOTE N ¼ 62 hospitals. aImplementation of applicable decision support is relatively straightforward.
b
Implementation of applicable decision support requires more effort to configure or customize.
AP R I L 2 0 1 0 2 9 :4 HE A LT H A FFA IR S 6 61
Focus On Quality
for comparison of these results. However, during and provide advice or an alert concerning a medi-
development of the assessment tool, the reliabil- cation order that would result in serious harm to
ity and validity were extensively evaluated and an adult patient. Some hospitals performed very
found to be very high. well, while others performed very poorly. In ad-
Finally, because results are self-reported, hos- dition, the studied hospitals as a group were
pitals may have reported better performance using basic decision support far more than the
than was actually assessed. However, it should more advanced tools needed to detect types of
be noted that public reporting of overall scores, orders that are major contributors to adverse
which could influence hospitals’ behavior, was drug events in chart-review studies.
not implemented until 2009, and hospitals were These findings point to the importance of eval-
asked to participate in part to aid them with uations of the use of clinical decision support by
improving their decision support. To address hospitals to help guide their continuing efforts
the potential for gaming, several standard safe- to improve medication safety. In addition, incen-
guards were built into the test to detect patterns tives to hospitals relating to computerized physi-
of use that suggest that gaming may be going on. cian order entry should include some type of
Any assessment that included these character- demonstration that clinical decision support is
istics was excluded from analysis. actually being employed, instead of being based
solely on whether computerized physician order
entry is in use.
Conclusions The broader use of this type of assessment
In sixty-two hospitals using an inpatient elec- of meaningful electronic health record use
tronic health record with computerized physi- should be explored for other software applica-
cian order entry, we found significant vari- tions used in direct clinical care. ▪
ability in the use of decision support to detect
The authors thank Peter Kilbridge, Fran development of the assessment tool of the manuscript. Four of the authors
Turisco, and the project expert advisers, used in this study was supported by (Jane Metzger, Emily Welebob, David W.
as well as individuals in many hospitals, grants from the California HealthCare Bates, and David C. Classen) were
health systems, and software vendor Foundation, the Robert Wood Johnson involved in the development of the
companies, for their involvement and Foundation, and the Agency for assessment tool. The content is solely
assistance in some phase of the Healthcare Research and Quality the responsibility of the authors and
development and testing of the (Contract no. 290-04-0016, Subcontract does not necessarily represent the
assessment tool used in the study. They no. 6275-FCG-01). These organizations official views of the funding agency.
also thank the Leapfrog Group for did not sponsor and were not involved
access to the data for analysis. The in the reported analysis or preparation
NOTES
1 Institute of Medicine. Preventing Clemmer TP, Weaver LK, Orme JF 10 Pedersen CA, Gumpper KF. ASHP
medication errors: quality chasm Jr., et al. A computer-assisted man- survey on informatics: assessment of
series. Washington (DC): National agement program for antibiotics and the adoption and use of pharmacy
Academies Press; 2007. other anti-infective agents. N Engl J informatics in U.S. hospitals—2007.
2 Institute of Medicine. Crossing the Med. 1998;338(4):232–8. Am J Health Syst Pharm. 2008;65:
quality chasm: a new health system 7 Teich JM, Merchia PR, Schmiz JL, 2244–64.
for the 21st century. Washington Kuperman GJ, Spurr CD, Bates DW. 11 American Hospital Association.
(DC): National Academies Press; Effects of computerized physician Continued progress: hospital use of
2001. order entry on prescribing practices. information technology [Internet].
3 Aspden P, Corrigan JM, Wolcott J, Arch Intern Med. 2000;160(18): Chicago (IL): AHA; 2007 [cited 2010
Erickson SM. Patient safety: achiev- 2741–7. Feb 21]. Available from: http://
ing a new standard of care. Wash- 8 Adams M, Bates DW, Coffman G, www.aha.org/aha/content/2007/
ington (DC): National Academies Everett W. Saving lives, saving pdf/070227-continuedprogress.pdf
Press; 2003. money: the imperative for comput- 12 Nebeker JR, Hoffman JM, Weir CR,
4 National Quality Forum. Safe prac- erized physician order entry in Bennett CL, Hurdle JF. High rates of
tices for better healthcare: a con- Massachusetts hospitals [Internet]. adverse drug events in a highly
sensus report [Internet]. Washing- Boston (MA): Massachusetts Tech- computerized hospital. Arch Intern
ton (DC): National Quality Forum; nology Collaborative and New Med. 2005;165(10):1111–6.
2003 [cited 2010 Feb 21]. Available England Healthcare Institute; 2008 13 Walsh KE, Landrigan CP, Adams
from: http://www.ahrq.gov/qual/ [cited 2010 Feb 21]. Available from: WG, Vinci RJ, Chessare JB, Cooper
nqfpract.pdf http://web3.streamhoster.com/ MR, et al. Effect of computer order
5 Bates DW, Leape LL, Cullen DJ, Laird mtc/cpoe20808.pdf entry on prevention of serious
N, Petersen LA, Teich JM, et al. Ef- 9 Getz L. On the right track in Mas- medication errors in hospitalized
fect of computerized physician order sachusetts. For the Record [serial on children. Pediatrics. 2008;121(3):
entry and a team intervention on the Internet]. 2008 Nov 11; 20(23) e421–7.
prevention of serious medication [cited 2010 Feb 21]. Available from: 14 Koppel R, Metlay JP, Cohen A,
errors. JAMA. 1998;280(15):1311–6. http://www.fortherecordmag.com/ Abaluck B, Localio AR, Kimmel SE,
6 Evans RS, Pestotnik SL, Classen DC, archives/ftr_111008p16.shtml et al. Role of computerized physician
AP R I L 2 0 1 0 2 9 :4 HE A LT H A FFA IR S 6 63