Documente Academic
Documente Profesional
Documente Cultură
ratios that met the "Panel-worthy" criteria, with articles published in recent years more
likely to meet all of the criteria. Conclusions. The comprehensive league table (avail-
able on the Web) provides a useful reference, but ratios may not be comparable be-
cause of methodologic variations. The standardized table focuses on studies meeting
basic methodologic criteria, potentially allowing for better comparison with future Ref-
erence Case analyses. Future studies should investigate the quality of analyses’ un-
derlying assumptions in addition to whether certain key procedural protocols were met.
Key words: cost-utility analysis; league tables; economic evaluation. (Med Decis
Making 2000;20:451-467)
Ranked listings of the cost-effectiveness ratios of not know whether a program is a good value until
various health and medical interventions, often it is compared with the benefits derived from re-
called &dquo;league tables,&dquo; have been used to facilitate sources expended in other programs. These league
comparisons across cost-effectiveness analyses. tables (so called after the tables used for British soc-
Comparisons of ratios are essential because we do cer league standings) have often been criticized,
however.1-4 The major criticism is that listing anal-
Received October 22, 1999, from the Program on the Eco- yses in one table &dquo;implies a certain degree of ho-
nomic Evaluation of Medical Technology, Harvard Center for mogeneity in study methodology,&dquo; a homogeneity
Risk Analysis, Harvard School of Public Health, Boston, Massa- that is usually missing.2 Drummond and colleagues
chusetts (RHC, PWS, EAS, CB, PJN), the University of Rochester list methodologic features that are particularly im-
School of Nursing, Community and Preventive Medicine, Roch-
ester, New York (PWS); the Institute of Medical Science, Univer-
portant to consider when interpreting rankings
within league tables, including: the discount rate,
sity of Toronto, Toronto, Ontario, Canada (CB), and the Depart-
ment of Health Policy and Management, Harvard School of utility valuation methods, the included range of
Public Health (PJN) Revision accepted for publication March 22, costs and effects, and the choice of comparison pro-
2000 Presented in part at the annual meeting of the Society for
gram. Mason proposed a similar &dquo;minimum data-
Medical Decision Making, October 4, 1999, in Reno, Nevada. Sup-
set&dquo; that he recommended should be included with
ported entirely by a grant from the National Science Foundation each entry in a cost-effectiveness league table.3 To
and Merck & Co., Inc. under the joint NSF/Private Sector Re-
search Opportunity Initiative (SBR-9730448). Mr Chapman was reduce the effect of variability in source-study meth-
supported by a training grant from the Agency for Health Care ods, these authors have recommended that league
Policy and Research, and Dr. Bell held a fellowship from the tables include only studies that are standardized on
Medical Research Council of Canada, at the time of this study
Address correspondence and reprint requests to Dr. Neu-
key analytic practices.
mann. Harvard Center for Risk Analysis, 718 Huntington Avenue,
While complete standardization of league-table
Boston, MA 02115, telephone (617) 432-1312; fax. (617) 432-0190; entries is the ideal, most published analyses do not
e-mail. ≺pneumann@hsph.harvard edu). allow this. A recent systematic audit of 228 cost-
451
452
utility analyses (CUAs) published from 1976 to 1997 terventions and their cost/QALY ratios, and 2) a stan-
revealed wide variations in methods for estimating dardized sub-table that includes only ratios from
and reporting costs, effectiveness, and preference analyses that adhere to key methodologic recom-
weights, and little evidence of improvement over mendations of the U.S. Public Health Service
time.5,6 This confirmed earlier findings of variations (USPHS) Panel on Cost-Effectiveness in Health and
in methods for estimating QALYS in CUAs.7Others Medicine. 16,17 The Panel has set out guidelines for
have reported that many published cost-effective- the &dquo;Reference Case&dquo; analysis-an analysis using
ness analyses do not adhere to commonly accepted standard methodologic practices, intended to in-
standards.1,8-15 Furthermore, inadequacies of re- form resource-allocation decisions across studies.
porting and the space constraints of journal publi-
cation often mean that the methodologic details of
CUAs are not disclosed.55
Methods
The purpose of this study was to use data from a
comprehensive review of published CUAs to create Details of the study design, data collection, and
two league tables: 1) a comprehensive listing of in- analytic plan are presented below.
453
SOURCE STUDIES AND DATA COLLECTION sive league table, indicating the two cases where this
occurs.
We conducted a computerized literature search For each cost-per-QALY ratio, we recorded a brief
through 1997 of the following databases: Medline, description of the evaluated health-care interven-
HealthSTAR, CancerLit, all editions of Current Con- tion, a description of the baseline comparator, and
tents, and EconLit. We searched for the Medical the target population for which the intervention is
Subject Headings and/or text key words: &dquo;quality- intended (following a practice begun by Graham and
adjusted,&dquo; &dquo;QALY,&dquo; and &dquo;cost-utility,&dquo; and identified Vaupel19). We also classified each intervention into
approximately 1,500 candidates for the database. We the appropriate disease category (based on the ICD-
then screened the titles and abstracts to remove ob- 9 classification system), intervention type, and level
viously non-eligible articles, such as methodologic, of prevention (primary, secondary, or treatment).
review, and non-CUA studies. A final list of 228 orig-
inal CUAs published in English from 1976 through
1997 was compiled. A data-collection (or audit) form ADJUSTMENTS TO PUBLISHED
Ratios were reported as costs per QALY gained, cost- A comprehensive league table (published on the
saving (if the intervention was less costly but pro- Web at (www.hcra.harvard.edu/medical.html)) pre-
duced more QALYs than its comparator), or domi- sents the cost-per-QALY ratios as reported in the lit-
nated (if more costly and less effective). It is also erature, with each study’s costs inflated to a base
possible for the evaluated intervention to be both year. We then present several descriptive summary
less costly and less effective than the comparator tables based on the information gathered in the da-
program, yielding a CU ratio with negative numbers tabase. These tables summarize the studies in the
in both the numerator and the denominator. This database, and stratify the information by the type of
was true for two of the evaluated interventions in intervention, prevention stage, and disease category.
our database (neither of which is in the &dquo;Panel- The methods used in each analysis were also
worthy&dquo; subset of articles). Although a single (posi- compared with key recommendations in the recent
tive) ratio can be reported in these cases, the inter- report of the USPHS Panel on Cost-Effectiveness in
pretation of these ratios is not the same as when the Health and Medicine} 1617 to determine the extent to
intervention is more costly but more effective. Re- which the studies adhered to standard practices and
placing the comparator (status quo) program with whether there is any evidence that methods have
this type of intervention would yield a certain been improving over time.
amount of savings per QALY foregone, so a higher A second league table (presented in the appendix
ratio is actually considered more cost-effective than and also posted on the Web at (www.hcra.harvard.
a lower ratio.18 Because of this different interpreta- edu/medical.html)) includes only ratios from those
tion, we have added a footnote to the comprehen- studies that conformed to selected Panel recom-
454
mendations for estimating cost-effectiveness ratios. is, those of the general population) are best for Ref-
This table is designed to address criticisms that erence Cases, because they are more likely to rep-
cost-effectiveness ratios from different studies are resent societal preferences for resource allocation.
not comparable because methods may differ widely. However, the Panel also noted that patient prefer-
ences for health states would be acceptable in a Ref-
erence Case analysis, if community preferences are
SELECTION CRITERIA FOR STANDARDIZED not available and are prohibitively difficult to mea-
LEAGUE TABLE sure. We therefore include studies that used either
the general population or patients as the source of
We used the following Reference Case recommen- health-state preferences (thus excluding studies that
dations to identify the subset of data points to in- relied on clinicians’ or authors’ judgments of pref-
clude in the standardized league table: adoption of erence weights).
a societal perspective, community or patient pref- Use of net costs. Health
care interventions not only
erence weights, use of net costs, appropriate incre- incur short-term costs or benefits, but also may off-
mental comparisons, and discounting of costs and set downstream costs by preventing long-term mor-
QALYs at the same rate. These recommendations bidity. Reference Case analyses should therefore cal-
are summarized below. culate costs net of any such downstream savings.
Societal perspective. Reference Case analyses are The Panel also recommended the inclusion of direct
intended to aid societal resource allocation deci- health care costs as well as direct non-health care
sions. The Panel noted that to allocate societal re- and time costs. While almost all of the studies in this
sources efficiently, all relevant costs and benefits database (99%) included some direct health care
should be included, and therefore Reference Case costs, fewer (17%) included any non-health care or
analyses should adopt a societal perspective. Be- time costs.6 We included only analyses that calcu-
cause only about half of the studies in the database lated costs net of any downstream savings, as rec-
explicitly stated the adopted perspective, we used ommended by the Panel, but did not screen studies
our own judgment of each study’s perspective. Also, for the types of costs included.
because relatively few studies (53, or 23%) used a Appropriate incremental comparisons. Some stud-
true societal perspective (e.g., including non-health ies reported an average cost-effectiveness ratio for
costs such as time or travel costs), we relaxed this a program, rather than incremental ratios between
criterion to include data points from any study that programs. An average cost-effectiveness ratio is one
we judged took a societal or broad health-care sec- calculated in comparison with a hypothetical no-
tor perspective (i.e., inclusion of all health-care costs cost, no-life-expectancy baseline rather than in com-
285)
regardless of who incurred them). parison with the next-most-effective program.16(P
Preference weights from community or patients. For studies that reported sufficient data, we con-
The Panel argued that community preferences (that verted average CU ratios to incremental ratios
Prevention level
Results FIGURE 2 Cost-utility ratios by intervention’s level of prevention
(outliers excluded) Thick lines indicate medians, and boxes the
The 228 CUAs included data for 647 intervention-
interquartile ranges.
comparator pairs, which are collected in a compre-
hensive league table, along with associated cost-
effectiveness ratios in 1998 dollars per QALY. The
comprehensive league table is not shown here, but tions). However, the median CU ratios were under
is available on our Web site, (www.hsph.harvard. $70,000/QALY for all intervention types except those
edu/organizations/hcra/hcra.html). The interven- classifiedas other medical procedures. The other-
tions are grouped by disease category and by type medical-procedures category includes some inter-
of intervention within disease categories. Sixty-nine ventions that have the highest estimated CU ratios
(10.6%) of the interventions were reported to be cost- in our database, such as preoperative autologous
saving and 64 (9.9%), dominated. The distribution of blood donations. The numbers of ratios recorded in
estimated CU ratios ranged from cost-saving to the database also varied greatly by intervention type
$52,000,000/QALY, with an overall median (excluding (table 2), with the largest categories being pharma-
dominated interventions) of $12,000/QALY (figure 1). ceutical and surgical interventions (with 233 and 119
The distribution is positively skewed, with 69% of the ratios, respectively).
647 interventions estimated to have CU ratios of The interventions were also classified according
$50,000 or less, and 78%, $100,000 or less. to level of prevention: primary, secondary, or terti-
When categorized into different types of interven- ary (that is, treatment). Primary prevention included
tions, the median CU ratios varied considerably (ta- measures intended to prevent the onset of targeted
ble 2), ranging from a median of $2,000/QALY for conditions; secondary prevention was defined as
immunizations to $140,000/QALY for &dquo;other medical measures aimed at identifying and treating asymp-
procedures&dquo; (again excluding dominated interven- tomatic persons who have already developed risk
456
*Includes intenrentions thatwere not aimed at specific diseases, or included more than one disease category.
fThe congenital anomalies category included one non-dominated intervention, estimated to be cost-saving; the pregnancy, childbirth, and puerpenum
category included one intervention, with an estimated ratio of $1,500/QALY.
factors or pre-clinical disease.24 Tertiary prevention erperium, hadten or fewer interventions for which
refers to treatment for clinical diseases or condi- CUAs had been conducted. The median CU ratios
tions. As shown in figure 2, the median CU ratios ranged from cost-saving for interventions targeted
are comparable for health-care interventions aimed at diseases of the blood and blood-forming organs
at primary secondary, and tertiary prevention (at to $40,500/QALY for interventions in respiratory sys-
$14,000, $14,000, and $11,000 per QALY, respec- tem diseases (table 3).
tively). However, the ratios for primary prevention Applying our selected Reference Case criteria, we
interventions were much more varied than those for found that CUAs for 95% of the interventions were
secondary or tertiary prevention (with an interquar- judged to take a societal or broad health-care sector
tile range of $407,000 for primary vs $34,000 for sec- perspective (table 4). Analysts used the general com-
ondary and $39,000 for tertiary prevention). munity or patients as the source of preference
The numbers of analyzed interventions varied weights for 39% of the intervention-comparator
considerably by disease category (table 3), with some pairs. Net costs (defined as costs net of any down-
relativelywell studied, such as circulatory system, stream savings) were calculated for 83% of the ratios.
infectious and parasitic, and neoplasms. Other dis- Appropriate incremental comparisons had been
ease categories, such as congenital anomalies, injury performed for 49% of the ratios. Analysts appropri-
and poisoning, and pregnancy, childbirth and pu- ately discounted both costs and QALYs for 80% of
the CU ratios. Overall, 112 of the 647 interventions
(or 37 of the 228 articles) used all of these recom-
Table 4 * Numbers of interventions and Articles Meeting mended basic analytic methods, representing 17% of
Selected Reference Case Criteria the ratios in the database (table 4; appendix). The
number of interventions meeting the selected Ref-
erence Case criteria over time is shown in figure 3.
Discussion
The comprehensive league table includes 647
cost-per-QALY ratios that vary across a range from
457
Even with these limitations, listing the results of Economic analysis in randomized control trials Med Care.
1992;30:231-43.
analyses together in a league-table format can reveal 9. Udvarhelyi IS, Colditz GA, Rai A, Epstein AM. Cost-effective-
useful information. The analyses presented here, ness and cost-benefit analyses in the medical literature. are
along with publication of the comprehensive league the methods being used correctly? Ann Intern Med. 1992;
table on the Web, provide a snapshot of the state of 116:238-44.
the field through 1997, and reveal which diseases or 10 Nord E. Toward quality assurance in QALY calculations. Int
J Technol Assess Health Care. 1993;9:37-45.
intervention types have been relatively well studied
and which neglected. By standardizing on basic
11. Briggs A, Sculpher M. Sensitivity analysis in economic eval-
uation: a review of published studies. Health Econ. 1995;4:
methodologic criteria, the &dquo;Panel-worthy&dquo; league ta- 335-71.
ble in the appendix allows some assessment of CU 12. Blackmore CC, Magid DJ. Methodologic evaluation of the ra-
ratios across diverse health care interventions. It diology cost-effectiveness literature. Radiology. 1997;203:87-
91.
also provides a benchmark with which to compare
13. Graham JD, Corso PS, Morris JM, Segui-Gomez M, Weinstein
future reference case analyses. MC. Evaluating the cost-effectiveness of clinical and public
We believe that posting these league tables on the health measures Annu Rev Public Health. 1998;19:125-52.
Web will increase their usefulness to policymakers 14. Jefferson T, Smith R, Yee Y, Drummond M, Pratt M, Gale R.
and health-care decision makers. We plan to update Evaluating the BMJ guidelines for economic submissions:
these tables and make them increasingly interactive prospective audit of economic submissions to BMJ and The
Lancet. JAMA. 1998;280:275-7.
in future versions, allowing users to examine the 15. Gerard K, Smoker I, Seymour J. Raising the quality of cost-
cost-effectiveness of interventions in various ways. utility analyses: lessons learnt and still to learn. Health Policy.
For example, a user could prepare disease-specific 1999;46:217-38.
16. Gold MR, Siegel JE, Russell LB, Weinstein MC (eds.). Cost-
league tables to address the question of what is the Effectiveness in Health and Medicine. Oxford, U.K.: Oxford
most efficient way to reduce mortality and morbidity
University Press, 1996.
from cancer, heart disease, and other conditions, or 17. Weinstein MC, Gold MR, Kamlet MS, Russell LB,
Siegel JE,
search for interventions targeting the same popula- for the Panel on Cost-Effectiveness in Health and Medicine
tion to compare the cost-effectiveness of pharma- Recommendations of the Panel on Cost-Effectiveness in
Health and Medicine. JAMA. 1996;276:1253-8.
ceutical and surgical interventions for a specific
18. Laupacis A, Feeny D, Detsky AS, Tugwell PX. How attractive
condition. Finally, we believe that one of the most does a new technology have to be to warrant adoption and
valuable aspects of this review is to examine the ev- utilization? Tentative guidelines for using clinical and eco-
olution of the field over time. nomic evaluations. Can Med Assoc J 1992;146:473-81.
19. Graham JD, Vaupel JW. Value of a life: what difference does
it make? Risk Analysis. 1981;1:89-95.
The authors are grateful to John D. Graham and two anonymous 20. Federal Reserve Bank of St. Louis. Monthly exchange rates
reviewers for helpful comments on drafts of the manuscript. series; extracted March 17, 1999. Available from: ≺http://
www.stls.frb.org/fred/data/exchange.html⟩.
21. Bureau of Labor Statistics. Consumer Price Index—All Ur-
ban Consumers. Series ID: CUUROOOOSAO; extracted March
References 17, 1999. Available from: ≺http://146.142.4.24/cgi-bin/survey
most?bls).
1. Gerard K. Cost-utility practice: a policy maker’s guide
in to 22. Aaron HJ. Health care spending: an analytical forum: I.
the state of the art. Health Policy. 1992;21:249-79. medical costs. Health Affairs. Winter
Thinking straight about
2. Drummond M, Torrance G, Mason J. Cost-effectiveness 1994:8-13.
league tables: more harm than good? Soc Sci Med. 1993;37: 23. Huskamp HA, Newhouse JP. Health care spending: an ana-
33-40.
lytical forum: III. Is health spending slowing down?. Health
3 Mason JM. Cost-per-QALY league tables: their role in phar- Affairs. Winter 1994:8-13.
macoeconomic analysis. PharmacoEconomics. 1994;5:472- 24. United States Preventive Services Task Force. Guide to Clin-
81. ical Preventive Services. 2nd ed. Baltimore, MD: Williams and
4. De Graeve D, Nonneman W. Pharmacoeconomic studies: pit- Wilkins; 1996.
falls and problems. Int J Technol Assess Health Care. 1996; 25. Elixhauser A, Halpern M, Schmier J, Luce BR. Health care
12:22-30. CBA and CEA from 1991 to 1996: an updated bibliography.
5. Neumann PJ, Stone P, Chapman RH, Sandberg E, Bell C. The Med Care. 1998;36:MS1-MS9, MS18-MS147.
quality of reporting in published cost-utility analyses, 1976- 26. Elixhauser A, Luce BR, WR, Reblando J. Health care
Taylor
1997. Ann Intern Med.
2000;132:964-72. CBA/CEA: an update on the and composition of the
growth
6. Stone PW, Chapman RH, Sandberg EA, Liljas B, Neumann literature. Med Care. 1993;31:JS1-JS138.
PJ. Measuring costs in cost-utility analyses: variations in the 27. Tengs TO, Adams ME, Pliskin JS, et al. Five-hundred life-
literature. Int J Technol Assess Health Care. 2000;16:111-24. saving interventions and their cost-effectiveness. Risk Anal-
7. Neumann PJ, Zinner DE, Wright JC. Are methods for esti- ysis. 1995;15:369-90.
mating QALYs in cost-effectiveness analyses improving? Med 28 Brown ML, Fintor L. Cost-effectiveness of breast cancer
Decis Making. 1997;17:402-8. results of a systematic review of the
screening: preliminary
8. Adams ME, McCall NT, Gray DT, Orza MJ, Chalmers TC. literature. Breast Cancer Res Treat. 1993;25:113-8.