Socio-Economic Planning Sciences 37 (2003) 141163

Advances in physician proling: the use of DEA

Janet M. Wagner*, Daniel G. Shimshak, Michael A. Novak
Department of Management Science and Information Systems, University of Massachusetts Boston, 100 Morrissey Blvd., Boston, MA 02125, USA

Abstract Insurers, health plans, and individual physicians in the United States are facing increasing pressures to reduce costs while maintaining quality. In this study, motivated by our work with a large managed care organization, we use readily available data from its claims database with data envelopment analysis (DEA) to examine physician practices within this organization. Currently the organization evaluates primary care physicians using a prole of 16 disparate ratios involving cost, utilization, and quality. We employed these same factors along with indicators of severity to develop a single, comprehensive measure of physician efciency through DEA. DEA enabled us to identify a reference set of best practice physicians tailored to each inefcient physician. This paper presents a discussion of the selection of model inputs and outputs, the development of the DEA model using a stepwise approach, and a sensitivity analysis using superefciency scores. The stepwise and superefciency analyses required little extra computation and yielded useful insights into the reasons as to why certain physicians were found to be efcient. This paper demonstrates that DEA has advantages for physician proling and usefully augments the current ratiobased reports. r 2003 Elsevier Science Ltd. All rights reserved.
Keywords: DEA; Physician proling; Health care efciency

1. Introduction One important way for health plans and insurers to cope with the pressure to increase productivity while maintaining quality is analyze the practices of their physicians. This paper describes the use of data envelopment analysis (DEA) to improve methods of measuring physician efciency in a study undertaken for a large New England managed care organization. Since physicians consume sizable portions of health care spending and control access to numerous other resources, better management by and of physicians can have a signicant impact on controlling the spiraling costs of health care.
*Corresponding author. Tel.: +1-617-287-7890; fax: +1-617-287-7725. E-mail address: janet.wagner@umb.edu (J.M. Wagner). 0038-0121/03/$ - see front matter r 2003 Elsevier Science Ltd. All rights reserved. doi:10.1016/S0038-0121(02)00038-1

142

J.M. Wagner et al. / Socio-Economic Planning Sciences 37 (2003) 141163

Analysts to date have had difculty in establishing a useful and equitable notion of physician efciency and pinpointing specic improvement strategies [1]. In manufacturing and nancial businesses, for example, it is often possible to develop a single summary measure of performance, such as prot or market share. In health care settings, however, there must be multiple measures of performance because outcomes such as patient health and satisfaction are considerations as vital as revenues and costs. In addition, in health care, it can be difcult to determine precisely the amounts of resources or input levels required to efciently yield the desired results or output levels. Currently the organization we worked with prepares reports or proles on a regular basis for each primary care physician (PCP). The proles include detailed information regarding a PCPs costs, utilization of resources, and outcome quality (e.g., cost per admission, hospital admissions per thousand patients, and readmission rates). These indicators, many of which are in the form of ratios, are compared against performance averages for other physicians. Proling involving multiple ratios can more effectively identify over- and underutilization of services, uncover problems with the efciency and quality of care, and assess physician performance [2]. Proles are designed to generate a specic action if the performance indicators for a particular physician differ from the average by a certain amount. This attempt to identify and correct outliers is sometimes referred to as the search for bad apples [2,3]. The ratio-based indicators used in physician proling attempt to highlight physician performances that are exceptionally high or low. For example, a large number of hospital admissions per number of members served in the physicians array of patients or panel would stand out as an indication of inefciency. However, because indicators are limited to one measure of input and/or one measure of output, they cannot easily accommodate situations where multiple outputs are produced using multiple inputs, as is true for physicians. To compensate for the one-dimensional nature of the indicators, a large set of ratios and normative values needs to be calculated in the prole reports. Unfortunately, with multiple indicators, there is no objective way of identifying inefcient physicians. For example, a physician whose admissions per thousand patients are greater than the average value might be considered potentially inefcient. However, it is not possible to determine how much larger than the average a physician must be to be considered inefcient or even if the average itself is efcient. Additionally, with multiple indicators, a physician may appear efcient for one group of measures but inefcient for another group. Without an objective means of prioritizing these indicators, identication of a truly efcient physician becomes difcult. Also, existing methods based on multiple ratios provide very little guidance on how physicians can change their practices to improve their overall performance. In order to overcome the limitations associated with multiple indicators, we will explore physician proling using a tool known as the DEA. Based on linear programming, DEA converts multiple input and output measures into a single, comprehensive measure of efciency without requiring that the relative weights of the measures be known a priori [4]. DEA measures efciency by constructing an empirically based best-practice or efcient frontier and by identifying peer groups. Each physician, referred to by the generic name decision-making unit (DMU), is compared to a composite unit that is constructed as a weighted combination of other units in its peer group [5]. Each input and output variable can be measured independently in any useful unit, without being transformed into a single metric, provided the same variables are

J.M. Wagner et al. / Socio-Economic Planning Sciences 37 (2003) 141163

143

utilized for every DMU [6]. In this way, non-comparable measures can be incorporated into the analysis. A DMU (i.e., a physician) is inefcient when another physician, or a combination of two or more of them, exists which can produce the same outputs with fewer inputs or can produce more outputs with the same inputs. When the above is not the case, it is possible to conclude that the physician being evaluated is efcient. DEA also develops, for each inefcient physician, a tailored comparison group of efcient physicians to whom the inefcient physician should look for improvements. DEA focuses on the best practice or efcient physicians for the purpose of improving overall performance. Unlike traditional physician proling, DEA searches for good apples, that is, the efcient physicians who will serve as role models. DEA can then be used to identify physicians, who are relatively inefcient, measure the magnitude of the inefciency, and evaluate alternative paths to reduce inefciencies. The goal is to nd a redistribution of resources to make an inefcient physician as efcient as the best-practice physician. Our paper makes several contributions to the existing work on physician proling and DEA. The data used for this study are real data taken from the health care organizations claims database. This database contains data on cost, utilization, and severity that have been commonly used in DEA. Our analysis also includes measures of quality that have been tracked by this organization, such as avoidance of readmission and complications (for inpatients) and survey ratings (for outpatients). To our knowledge, no prior studies have employed these particular elements of the process. In addition, many of the existing papers on DEA treat the input and output values used in their studies as simply given and treat the DEA model as a black box. In contrast, we focus on the reasons for choosing the specic DEA model to employ and use an approach of adding variables to the model in a stepwise manner so as to better understand the DEA results (see Norman and Stoker [7] for a presentation of one version of stepwise DEA). As far as we know, this stepwise method has not previously been used in the study of health care problems. Lastly, this paper includes an additional set of analyses using an extension to DEA called superefciency [810] to provide a type of sensitivity analysis that can provide information about physicians with a high inuence on the efcient frontier (similar to the identication of points with very large residuals in multiple regression). This paper should be of interest to health care managers and health care researchers alike. The paper is organized as follows. Section 2 of this paper includes a literature review of the application of DEA in health care. Section 3 presents the data used in this study, including a discussion of the available measures of outcome quality. Section 4 discusses the DEA modeling process including the stepwise technique of adding variables to build a DEA model and sensitivity analysis is performed using the superefciency model. Finally, Section 5 presents the overall conclusions of this study and discusses the ongoing work with this data set.

2. Literature review Substantial research has been done on DEA applications to the health care sector. Most early studies concerned themselves with the efciency of hospital services, although more recently attention has shifted to physician services.

144

J.M. Wagner et al. / Socio-Economic Planning Sciences 37 (2003) 141163

In an early application of DEA to hospitals, Sherman [5] examined the efciency of seven teaching hospitals in Massachusetts. The study used inputs of full-time equivalents (FTEs), cost of supplies and purchased services, and number of beds and outputs of patient days, nurses trained, and internsXresidents trained. This study found that two of the seven hospitals were inefcient and suggested specic input reductions for the inefcient hospitals. Grosskopf and Valdmanis [11] conducted a similar analysis of 82 California hospitals measuring acute care, intensive care, surgeries, and ambulatory and emergency care. Using a sample of 3000 urban hospitals, Ozcan and Luke [12] looked at the relationships between four hospital characteristics (size, membership in a multihospital system, ownership, and payer mix) and hospital efciencies. In an interesting study of hospital efciency, Ozcan [13] analyzed the sensitivity of various input and output variables in DEA models. Two studies focused on nursing service efciency within hospitals. Nunamaker [14] used inpatient costs as inputs and patient days as outputs to study routine nursing service efciency at 17 Wisconsin hospitals. He also estimated the potential cost savings for each inefcient hospital if it were to become efcient. Dittman et al. [15] examined the efciency of medicalsurgical nursing units in 105 hospitals by using three DEA models that considered different combinations of inputs and outputs. The application of DEA has been extended to other types of health care institutions, including Veterans Administration medical centers [16], rural health programs [6], public health centers in Spain [17], regional agencies with programs on area aging [18], organ procurement organizations [19], and health maintenance organizations [20]. In one of the rst studies to explicitly include quality considerations, Salinas-Jimenez and Smith [21] attempted to measure the quality of services provided by the medical practitioners in family health service authorities in England. They used medical expenditures as inputs and seven quality indicators as outputs, including, for example, the percentage of practices employing a nurse practitioner, and the percentage of female patients who have had a cervical smear in the previous ve and a half years. More recently, DEA research in health care has been devoted to studying the efciency of physician practices. Most of the research in physician proling has dealt with the collection and analysis of physician data, including the selection of indicators [22], sources of data [2,22,23], and statistical methods used for analysis [3,24,25]. Chilingerian and Sherman [26] evaluated 15 hospital-based cardiologists. Inputs consisted of length of stay and ancillary costs, while outputs were the successful treatment of low-severity and high-severity heart failure and shock patients. Of the 15 physicians, three were found to be efcient. Further, the study looked at the physicians effectiveness as measured by the proportion of patients treated by each physician that resulted in morbidity or mortality. Other DEA studies by Chilingerian [1,27] considered 24 internists and 12 surgeons in a major teaching hospital. In one study, examination of physician practice characteristics (including age, medical background, HMO afliation, practice specialty, and number and mix of cases) revealed why some physician practices, identied through DEA, appeared to be less than efcient [27]. In a later study, Tobit analysis identied the characteristics of the efcient physicians [1]. Chilingerian and Sherman [28] used DEA to study 326 physicians, including those in family and general practice, internists, and subspecialists, who were members of a large Independent Practice Association (IPA), in order to explore the utilization of medical resources between generalist and specialist physicians. Inputs consisted of the utilization of visits, tests, procedures, and hospital days. The panel size, i.e., the quantity of patients, for seven

J.M. Wagner et al. / Socio-Economic Planning Sciences 37 (2003) 141163

145

different agesex categories (in order to account for case mix) served as output measures. Although the study found one-half of the physicians to be inefcient, best practice physicians included both specialists and generalists. Some recent DEA studies have attempted to benchmark physician performance through the use of claims data. Ozcan [29] analyzed the behavior of 160 physician practices in the treatment of otitis media. He selected as inputs the number of hospitalizations, physician visits, specialist visits, lab tests, and prescriptions, with their related costs and included as outputs the number of treated patient episodes categorized into three severity levels. The study found only 28.8% of the physicians to be efcient and revealed a clear pattern that attributed excessive resource consumption and higher costs to inefcient physicians. The study also found that physician efciency varied by geographic region. Studying 176 physicians who treated sinusitis, Ozcan et al. [30] discerned no clear differences in efciency between generalists and specialists but found that specialists used more resources and incurred higher service costs than the generalists. In an earlier paper dealing with physician evaluation, Wagner and Shimshak [31] performed a case study and demonstrated the messy details of using data available from an actual claims database to do a DEA analysis. This paper continues that work with a greater focus on and more development of the methodology. The innovations in this work include our use of output quality measures (including patient survey results), a stepwise approach to give a greater understanding of the model results, and superefciency scores.

3. Data This study uses data from the computerized claims database of a large managed care organization in order to evaluate the performance of its member physicians. The claims database contains data for a rolling 3-year period; the data for this study were collected for calendar year 1995. We started by using the indicators in the existing, ratio-based physician proles. We added two severity measures: average inpatient case weight and panel health status, which were available from the claims database. We then incorporated a member-generated quality measure obtained from separate data source of responses from patient surveys. 3.1. Sample The literature review cites several studies that compared the types of physicians (both generalists and specialists) and practices across geographic regions. However, we sought to examine the differences in the physicians practice, as opposed to the physicians environment; so we chose the study group to be as homogeneous as possible. Thus, we selected a group of PCPs who were all internists and who were practicing in a small geographic region within the city where this managed care organization was headquartered. These internists all referred patients to the same set of metropolitan hospitals and all had a panel size of about 100 or more members from this managed care plan. A group of 81 PCPs met these criteria. However, the health care organization had practical and condentiality concerns about releasing the data for every member in this practice group. In the end, the organization allowed the use of only every third physician for inclusion in a pilot study, resulting in an initial sample of

146

J.M. Wagner et al. / Socio-Economic Planning Sciences 37 (2003) 141163

27 physicians. Without doubt, to truly investigate the efciency of this particular practice group, all 81 physicians would need to be included. While we are not intending this analysis to be generalizable across all provider groups, we believe this sample is large enough to illustrate the point of this papernamely that DEA is a useful technique for physician proling, and one that has advantages over the physician proling reports now used by this organization. Once we obtained the data, we performed various validity checks to ensure that the data were reasonable. One striking feature of the data was that a small group of these physicians had very few outpatient encounters with their panel members. For some physicians in this small group, there were simply no encounters, while for others, their panel members had the majority of their encounters with physicians other than their own PCP. Assuming that these low encounter rates indicated some anomaly, or a practice type very different from the usual internist, we dropped PCPs from our sample if they had less than 0.5 outpatient encounters per panel member per year. (For reference, the lowest value for this encounter ratio for the remaining PCPs was 0.70, with an average of 1.35 outpatient encounters for all remaining physicians.) As a result of dropping some physicians, the nal study group size was 21 PCPs. (In this paper, the 21 physicians were identied only as physician A, physician B, etc.) 3.2. Measures Starting with the measures used in the current ratio-based physician proles, we divided the set of possible measures into inputs and outputs (see Table 1). The desired results of the physician practice represented the outputs, namely increased output measures indicated increased productivity. Here the goal was that physicians see an increased panel size and/or show increased levels of quality. Additionally, we developed measures of severity for use in the DEA models. These were also treated as outputs. Again, the desired result, from an efciency point of view, was that physicians be able to handle a more severely ill population. The remaining measures of utilization (admissions, hospital days, and physician encounters) and costs (hospital and encounter) were classied as input measures; using fewer of these resources would realize productivity gains. Although other studies discuss the importance of using quality variables in DEA models of physician efciency, few actually have quality data available. In this study, we used three measures of quality as outputs. To indicate a gain in quality, we had to transform the two quality measures currently used by the health care organization by subtracting them from 1. Thus, for inpatient variables of quality, we used the percentage of inpatients not readmitted to the hospital within 15 days and the percentage of inpatients who did not develop complications as a result of treatment during the course of hospitalization. While it would have been possible to use the absolute numbers of inpatients without readmission or complications, we left these quality measures as percentages for a number of reasons. First, the absolute numbers of patients who were readmitted or who had complications were small, which we feared could make distinguishing among them difcult. Second, ratios are how the health care organization currently measures quality and are thus familiar to the users in that form. Third, prior studies combining cost and quality have also used ratio quality measures [21,26]. For the outpatient quality measure, we used a composite score resulting from a member survey regarding access to their PCP, satisfaction with their PCP, and reenrollment with their

J.M. Wagner et al. / Socio-Economic Planning Sciences 37 (2003) 141163 Table 1 Input and output variables used for DEA analysis INPUTS ADMITS HOSPDAYS $PHYSICIAN $ROOM&BOARD $ANCILLARY ENCWPCP ENCNOTPCP $OPWPCP $OPNOTPCP $INPATIENT $OUTPATIENT $TOTAL

147

Sum of number of members inpatient admissions Sum of length of stay in days for inpatient admissions Sum of gross payments for inpatient admissions physician costs Sum of gross payments for inpatient admissions room and board costs Sum of gross payments for inpatient admissions ancillary costs Sum of the number of encounters by members with their PCP where an encounter is dened as all the services received by a member with a PCP in a single day Sum of the number of encounters by members with health care providers other than their PCP Sum of gross payments for encounters by members with their PCP Sum of gross payments for encounters by members with health care providers other than their PCP Sum of total gross payments for inpatient admissions ($PHYSICIAN+$ROOM&BOARD+$ANCILLARY) Sum of total gross payments for all encounters by members ($OPWPCP+$OPNOTPCP) Sum of total gross inpatient and outpatient payments ($INPATIENT+$OUTPATIENT)

OUTPUTS PANEL AVGCASEWT

%NOREADMISSION %NOCOMPLICATIONS HEALTHSTAT

QSCORE

Number of members who have chosen this provider as a PCP to be responsible for the coordination of the members health care Average relative weights for inpatient admissions based on Diagnosis-Related Groups (DRGs) and used to represent the resource intensity of the admissions (where larger values imply greater resource intensity of admissions) % of inpatient admissions not readmitted to the hospital within 15 days % of inpatient admissions with no complications as a result of treatment during course of hospitalization Indicator of health status of panel using the diagnosis case mix to determine the level of health care resource utilization anticipated to be incurred by members of the panel in a particular year from an analysis of prior years health care utilization (based on an average of 1, where values > 1 imply that members are sicker than average) Quality measure of PCP based on members survey regarding access to PCP, satisfaction with PCP, and reenrollment with PCP (where larger values imply greater quality)

PCP. The inclusion of these quality variables allowed for greater discrimination among physicians.

4. DEA models and analysis In this section, we discuss the development of the DEA models and present our analysis of the data.

148

J.M. Wagner et al. / Socio-Economic Planning Sciences 37 (2003) 141163

4.1. Development of DEA model In designing this study, we faced a number of decisions about the specic DEA model to employ. The rst of these involved the rationale behind physician proling and led to the choice of an input orientation rather than an output orientation model. We assumed that physicians resources or inputs could be changed (controllable variables), while the majority of physicians outputs, specically panel size, average case weight, and health status, were basically givens (uncontrollable variables). Therefore, the efciencies calculated in this paper represented the proportion by which it was expected that a physician could reduce his or her input use and still attain the same outputs. Additionally, the model used for this preliminary study did not examine the issue of non-zero slacks; technical or weak efciency was considered efcient [32]. Second, a decision was needed about whether to use constant returns to scale (CRS) or variable returns to scale (VRS). For the types of resources or inputs in this study, which relied heavily on the utilization of hospital or physician resources, we did not expect scale effects, i.e., more patients would be expected to need proportionally more resources. Thus a CRS frontier seemed appropriate. CRS models have been used in a number of health care studies; for example, Ozcan et al. [33], and Chilingerian [27] who argued that since there is no reason to believe that the act of increasing caseloads has a scale effect on the productivity of inputs, variable returns to scale did not seem justied. Additional support for the choice of a CRS model can be found in Fig. 1, which presents a scatterplot of a DEA model with one output (PANEL) and one input ($TOTAL), with the CRS and VRS frontiers indicated. In this gure, the VRS frontier was often heavily inuenced by only a few physicians with large practices. We felt that this VRS frontier in the vicinity of these large practices indicated only that the physicians had few neighbors rather than identifying that these large-practice PCPs were particularly efcient. For these reasons, the CRS frontier was used for all analyses in this study.

2500000

2000000
Total Payments ($TOTAL)

1500000 PCPs CRS Frontier VHS Frontier A R 500000 G U 0 0 100 200 300 400 500 600 700 800 Panel Size (PANEL)

1000000

Fig. 1. Comparison of efcient frontiers.

J.M. Wagner et al. / Socio-Economic Planning Sciences 37 (2003) 141163

149

One of the goals of this paper was not only to present the nal set of DEA model variables but also to demonstrate the process used in selecting these particular variables from the large set of potential variables. In forthcoming sections, we will show how a DEA model can be built in what we call a stepwise approach, i.e., starting with one input variable and one output variable and then adding additional variables in steps. As a rule of thumb, the total number of input and output variables should be less than one-third the number of DMUs [32, p. 252]. This rule will inuence the maximum number of input and output variables allowed with this sample. We should note that adding input or output variables to a DEA model is done by adding constraints to the underlying linear program. Thus adding additional variables will either leave the efciency scores unchanged or will increase them [34]. Therefore, as additional variables are added to the analysis, the set of physicians rated as fully efcient usually grows. One result of using this stepwise approach was that the specic factors leading a particular PCP to be identied as efcient were more readily apparent. In explaining this stepwise technique to non-technical DEA users, we call it the yes, but approach. For example, looking at total payments vs. panel size using the CRS frontier in Fig. 1, only one PCP is efcient. Yes, but if you add in the health status indicators, more PCPs may become efcient. And yes, but what is the impact of quality measures? In contrast to the usual papers that present a list of variables fed into the black box of a DEA model, we found the stepwise approach produced a greater understanding of the data and useful insights into the managerial question of how efciency can be improved. We started our analysis by combining both inpatient and outpatient measures. For this combined model, we examined the effect of adding variables in a stepwise manner. We then analyzed inpatient and outpatient performance separately. Splitting the analysis in this fashion identied some interesting results that were not obvious from a single DEA analysis. As used in this paper, the stepwise approach is suggested as an ad hoc procedure, guided by the users understanding of the production system. 4.2. Search for a parsimonious model In our use of DEA, one aim was to nd a parsimonious model, using as many input and output variables as needed but as few as possible. A parsimonious model typically shows generally low correlations among the input and output variables, respectively [1,35,21]. In fact, high correlations among input or output variables could cause difculties in DEA analysis. Since adding more variables will likely increase the efciency scores [34], adding highly correlated input or output variables would indicate that many PCPs were efcient, when they were actually differentiated solely by small and essentially random uctuations. Possible input measures for this model are listed in Table 1, including measures of resource utilization (ADMITS, HOSPDAYS, ENCWPCP, and ENCNOTPCP), individual measures of cost ($PHYSICIAN, $ROOM&BOARD, $ANCILLARY, $OPWPCP, and $OPNOTPCP), and several aggregated measures of costs ($INPATIENT, $OUTPATIENT, and $TOTAL). Possible outputs included each physicians panel size (PANEL), measures of severity (AVGCASEWT) and HEALTHSTAT, and the three quality measures, %NOREADMISSION, %NOCOMPLICATIONS, and QSCORE.

150

J.M. Wagner et al. / Socio-Economic Planning Sciences 37 (2003) 141163

Table 2 Input variable correlations

ADMITS HOSP- ENCW- ENCNOT- $PHYSICIAN $ROOM& $ANCILLARY $OPW- $OPNOT- $IN$OUTDAYS PCP PCP BOARD PCP PCP PATIENT PATIENT ADMITS HOSPDAYS ENCWPCP ENCNOTPCP $PHYSICIAN $ROOM&BOARD $ANCILLARY $OPWPCP $OPNOTPCP $INPATIENT $OUTPATIENT $TOTAL 1.000 0.948 0.625 0.937 0.918 0.949 0.978 0.587 0.949 0.972 0.944 0.976

1.000 0.585 0.861 0.956 0.993 0.953 0.543 0.879 0.986 0.873 0.938

1.000 0.820 0.563 0.561 0.641 0.943 0.767 0.607 0.786 0.732

1.000 0.839 0.847 0.928 0.794 0.991 0.896 0.994 0.977

1.000 0.950 0.908 0.550 0.852 0.966 0.848 0.914

1.000 0.958 0.512 0.868 0.989 0.861 0.932

1.000 0.591 0.947 0.982 0.941 0.978

1.000 0.726 0.566 0.750 0.693

1.000 0.914 0.999 0.988

1.000 0.909 0.966

1.000 0.986

Examining the correlation matrix in Table 2 for the set of possible input variables led to several insights. First, the total inpatient payments was extremely highly correlated (0.966 or above) with all inpatient inputs measuring utilization and cost. For this data set the correlation analysis shows that measures of admissions, total hospital days, and the assorted payments (whether physician, room and board, or ancillary) varied in unison. While any one of these variables could be used to represent inputs, we chose total inpatient payments. Not surprisingly, the correlations between the number of encounters and the payments for encounters (for both PCP and non-PCP encounters) were also extremely high. Thus again, only the outpatient cost variables and not the outpatient utilization variables were selected as input measures in the DEA analysis. The correlation matrix also indicated that total inpatient and total outpatient payments were fairly highly correlated (0.909). While we might have thought that a push to reduce inpatient costs might lead to increases in outpatient costs, the high positive correlation value showed that this was not the case. Moreover, total payments (combined inpatient and outpatient) were highly correlated with some cost variables but not others. After further analysis of the correlation matrix found in Table 2, two choices of input variables seemed supportable: (1) the completely aggregated measure of $TOTAL and (2) the less aggregated set of $INPATIENT, $OPWPCP, and $OPNOTPCP. Correlations among outputs, as shown in Table 3, were generally not high. We debated the need for two severity variables; however, the two measures of severity, AVGCASEWT and HEALTHSTAT, showed a correlation of only 0.368. These variables appeared to measure different things (perhaps because even in an overall healthy panel, it was possible for a few members to have a number of acute episodes due to accidents or other unforeseen events). Similarly, the quality measures were all correlated at very low levels with one another, implying that the measures were not contributing the same information to the analysis. A review of the correlation matrix in Table 3 justied the inclusion of all output variables in the DEA analysis. 4.3. Combined inpatient and outpatient analysis In our rst analysis, we combined inpatient and outpatient measures. The inputs for this model were total inpatient payments, payments for PCP encounters and payments for non-PCP

J.M. Wagner et al. / Socio-Economic Planning Sciences 37 (2003) 141163 Table 3 Output variable correlations PANEL PANEL AVGCASEWT HEALTHSTAT %NOREADMISSION %NOCOMPLICATION QSCORE 1.000 0.329 0.134 0.018 0.212 0.508 AVGCASEWT HEALTHSTAT %NOREADMISSION

151

%NOCOMPLICATIONS

1.000 0.368 0.430 0.122 0.361

1.000 0.024 0.290 0.151

1.000 0.082 0.074

1.000 0.141

encounters. For outputs we selected panel size, average case weight, panel health status, and the three quality measures, percentage of inpatient admissions not readmitted within 15 days, percentage of inpatients who did not develop complications, and the quality score from the patient survey. This analysis, referred to as Set 1, follows: Set 1 Inputs: $INPATIENT, $OPWPCP, $OPNOTPCP Outputs: PANEL, %NOREADMISSION, %NOCOMPLICATIONS, QSCORE AVGCASEWT, HEALTHSTAT Table 4 presents the efciency scores and reference set for each physician, as calculated by this model. This analysis identied 11 out of the 21 physicians as efcient. The scores of the inefcient physicians ranged from 0.972 for physician M to 0.645 for physician J. Efcient physician R appeared in most of the reference sets (6); efcient physicians H, L, and Q appeared in no reference sets. This analysis also identied the set of efcient physicians and a reference set for each inefcient physician of best practice physicians with similar practice characteristics. Because it provides an objective way of identifying the best practice physicians, DEA represents an improvement over the current ratio-based physician proling methods. However, this method alone does little to explain why certain physicians are efcient and others are not. In order to uncover the causes of physician efciencies, we studied the DEA model using the stepwise approach. Thus, we considered the following three additional sets of input and output measures for the combined inpatient and outpatient analysis: Set 2 Inputs: Outputs: Set 3 Inputs: Outputs: $TOTAL PANEL $TOTAL PANEL, %NOREADMISSION, %NOCOMPLICATIONS, QSCORE Set 4 Inputs: $TOTAL Outputs: PANEL, %NOREADMISSION, %NOCOMPLICATIONS, QSCORE, AVGCASEWT, HEALTHSTAT

152

J.M. Wagner et al. / Socio-Economic Planning Sciences 37 (2003) 141163

Table 4 Results for Set 1 (combined inpatient and outpatient model) DMU A B C D E F G H I J K L M N O P Q R S T U Scores (all PCPs) 1.000 0.651 0.968 0.789 0.740 1.000 1.000 1.000 1.000 0.645 0.849 1.000 0.972 0.732 1.000 1.000 1.000 1.000 0.827 0.791 1.000 Reference Set O G, P, U O F, R, U

G, R F, R F, I A, O, R

A, O, R, U A, O, R, U

Table 5 presents the efciency scores for Sets 14. As previously discussed, adding variables in this manner will likely cause the set of efcient physicians to grow. This effect was seen in the analysis. Using a CRS frontier (see Fig. 1), Set 2 identied only physician R as efcient. This physician treated the most patients for the least dollars. Efciency scores for the inefcient physicians ranged from 0.975 for physician A to 0.454 for physician B. Set 3 shows what happens when considerations of quality are added. In this analysis, physicians G and U also became efcient. Physician U stood out as a quality leader since he or she had perfect scores on %NOREADMISSION and %NOCOMPLICATIONS and among the highest QSCORE values. The reasons for physician Gs shift to efciency were not so apparent. A few other physicians also showed a marked change from Set 2. For example, adding quality measures changed the efciency score for physician H from 0.556 to 0.744. However, the addition of quality changed the efciency scores of the inefcient physicians only slightly, if at all. Set 4 shows the effect of adding measures of severity. In this analysis, physicians F, H, I, and O also became efcient. Physician A still had the highest efciency score (0.975) and physician B the lowest score (0.460) among the inefcient physicians. A comparison of Set 4 containing one input ($TOTAL) with Set 1 containing three inputs ($INPATIENT, $OPWPCP, and $OPNOTPCP) shows the effect of disaggregating the cost variables. Disaggregating the cost variables does change the results of the analysis. All of the physicians identied as efcient when using only one total cost input (Set 4) were still identied as

J.M. Wagner et al. / Socio-Economic Planning Sciences 37 (2003) 141163 Table 5 Efciency scores for combined inpatient and outpatient models Set 1 Inputs: $INPATIENT $OPWPCP $OPNOTPCP PANEL %NOREADMISSION %NOCOMPLICATIONS QSCORE AVGCASEWT HEALTHSTAT 1.000 0.651 0.968 0.789 0.740 1.000 1.000 1.000 1.000 0.645 0.849 1.000 0.972 0.732 1.000 1.000 1.000 1.000 0.827 0.791 1.000 Set 2 $TOTAL Set 3 $TOTAL Set 4 $TOTAL

153

Outputs:

PANEL

PANEL %NOREADMISSION %NOCOMPLICATIONS QSCORE

PANEL %NOREADMISSION %NOCOMPLICATIONS QSCORE AVGCASEWT HEALTHSTAT 0.975 0.460 0.918 0.571 0.733 1.000 1.000 1.000 1.000 0.519 0.832 0.785 0.921 0.696 1.000 0.885 0.862 1.000 0.804 0.780 1.000

A B C D E F G H I J K L M N O P Q R S T U

0.975 0.454 0.820 0.571 0.692 0.904 0.798 0.556 0.711 0.517 0.756 0.485 0.808 0.694 0.923 0.670 0.523 1.000 0.804 0.771 0.923

0.975 0.457 0.879 0.571 0.716 0.972 1.000 0.744 0.742 0.518 0.784 0.524 0.837 0.696 0.957 0.723 0.560 1.000 0.804 0.777 1.000

efcient when using disaggregated costs (Set 1). Physicians A, L, P, and O were also found to be efcient by the disaggregated model. In addition, physician B still had a low efciency score, but physician J had the lowest score (0.645) using the disaggregated model. These additional analyses made it possible to identify the causes of physician efciency. Physician R became efcient due to low resource utilization, physicians G and U due to high quality, and physicians F, H, I, and O due to high severity. Comparing the aggregated and disaggregated cost models also revealed a substantial change in physician efciency scores. Although we were aware of cross-effects between inpatient and outpatient variables, we believed that separate analysis of inpatient and outpatient practices might produce additional insights. These analyses follow.

154

J.M. Wagner et al. / Socio-Economic Planning Sciences 37 (2003) 141163

4.4. Inpatient analysis The input measure used for inpatients was the total inpatient payments. Possible outputs included panel size, average inpatient case weight, panel health status, and two quality measures, percentage of inpatients admissions not readmitted within 15 days and percentage of inpatient admissions who did not develop complications. Following a stepwise procedure, we looked at four sets of input and output variables: Set 5 Inputs: Outputs: Set 6 Inputs: Outputs: Set 7 Inputs: Outputs: Set 8 Inputs: Outputs: $INPATIENT PANEL $INPATIENT PANEL, %NOREADMISSION, %NOCOMPLICATIONS $INPATIENT PANEL, %NOREADMISSION, %NOCOMPLICATIONS, AVGCASEWT $INPATIENT PANEL, %NOREADMISSION, %NOCOMPLICATIONS, AVGCASEWT, HEALTHSTAT

Fig. 2 shows the scatterplot of PANEL vs. $INPATIENT (Set 5), with the CRS frontier indicated. Table 6 presents the efciency scores for Sets 58. We were somewhat surprised that in the analysis of all the four sets, only physician U was found to be efcient. From the combined analysis in Section 4.3 we found that physician U had perfect inpatient quality measures, as well as the lowest total inpatient payments. A closer look at these results showed the effect of introducing additional variables. For example, physician H had an efciency score of 0.233 when we considered only panel size as the output measure in Set 5. His or her efciency score rose to 0.288 when quality measures were included in Set 6. However, in
800000
Total Inpatient Payments( $INPATIENT)

700000 600000 500000 400000 N 300000 200000 100000 0 0 L Q I K H F M E G PC O U 100 200 300 400 500 600 700 T R S A J B

PCPs CRS

800

Panel Size (PANEL)

Fig. 2. Inpatient (Set 5) CRS frontier.

J.M. Wagner et al. / Socio-Economic Planning Sciences 37 (2003) 141163 Table 6 Efciency scores for inpatient models Set 5 Set 6 Set 7 Set 8 Set 8 (without physician U)

155

Inputs: Outputs:

$INPATIENT PANEL

$INPATIENT PANEL %NOREADMISSION %NOCOMPLICATIONS

$INPATIENT PANEL %NOREADMISSION %NOCOMPLICATIONS AVGCASEWT

$INPATIENT PANEL %NOREADMISSION %NOCOMPLICATIONS AVGCASEWT HEALTHSTAT 0.658 0.177 0.811 0.277 0.469 0.623 0.683 0.608 0.736 0.214 0.499 0.370 0.564 0.347 0.654 0.935 0.464 0.404 0.480 0.409 1.000

$INPATIENT PANEL %NOREADMISSION %NOCOMPLICATIONS AVGCASEWT HEALTHSTAT 1.000 0.275 1.000 0.421 0.751 0.779 0.885 0.685 0.787 0.328 0.644 0.396 0.670 0.541 0.915 1.000 0.497 0.633 0.746 0.641

A B C D E F G H I J K L M N O P Q R S T U

0.658 0.177 0.618 0.277 0.469 0.428 0.339 0.233 0.311 0.214 0.357 0.157 0.362 0.347 0.519 0.522 0.176 0.404 0.480 0.409 1.000

0.658 0.177 0.618 0.277 0.469 0.428 0.423 0.288 0.311 0.214 0.357 0.157 0.362 0.347 0.519 0.522 0.176 0.404 0.480 0.409 1.000

0.658 0.177 0.618 0.277 0.469 0.623 0.423 0.607 0.736 0.214 0.499 0.370 0.564 0.347 0.654 0.935 0.464 0.404 0.480 0.409 1.000

Set 7 when we gured in average case weight, physician H scored a much improved 0.607. This score changed to only 0.608 with the addition of health status in Set 8. Interestingly, adding in quality did not change many scores, and for those that did the change was very slight. Considerations of case weight had a greater effect. For example, physician Ps efciency score increased from 0.522 to 0.935. Adding in panel health status also affected relatively few scores. Physicians C and G did show some improvement in efciencies when health status was included. 4.5. Outpatient analysis Possible input measures for outpatients included payments for PCP encounters, payments for non-PCP encounters, and total outpatient payments. Possible outputs included panel size, panel

156

J.M. Wagner et al. / Socio-Economic Planning Sciences 37 (2003) 141163

health status, and quality score from the patient survey. Similar to the analysis performed on the inpatient data, we looked at four sets of input and output measures: Set 9 Inputs: Outputs: Set 10 Inputs: Outputs: Set 11 Inputs: Outputs: Set 12 Inputs: Outputs: $OUTPATIENT PANEL $OPWPCP, $OPNOTPCP PANEL $OPWPCP, $OPNOTPCP PANEL, QSCORE $OPWPCP, $OPNOTPCP PANEL, QSCORE, HEALTHSTAT

Fig. 3 shows the scatterplot of PANEL vs. $OUTPATIENT (Set 9), with the CRS frontier indicated. Table 7 presents the efciency scores for Sets 912. For the outpatient data, adding more variables increased the set of efcient physicians. In Set 9, with one input ($OUTPATIENT) and one output (PANEL), only physician R was efcient. When we separated PCP and non-PCP payments in Set 10, physicians G and O also came out efcient. Figuring in the quality indicator in Set 11 showed physician F to be efcient. Finally, the inclusion of the severity measure in Set 12 resulted in no additional efcient physicians. For the PCPs who were not in the set rated efcient, the addition of new variables improved those physicians scores dramatically. For example, physician H had one of the lowest efciency scores (0.542) for the rst outpatient model described in Set 9. Converting total outpatient payments into two variables in Set 10 yielded a small improvement in the efciency score (0.590). However, when the quality score was introduced in Set 11, the efciency score of physician H increased considerably (0.812). The efciency score increased further (0.935) with the addition of health

1400000
Total Outpatient Payments ($OUTPATIENT)

1200000 1000000 800000 B 600000 T 400000 E 200000 0 0 100 200 300 400 500 600 700 K O PC I HU Q M F G R N J S A

PCPs CRS

800

Panel Size (PANEL)

Fig. 3. Outpatient (Set 9) CRS frontier.

J.M. Wagner et al. / Socio-Economic Planning Sciences 37 (2003) 141163 Table 7 Efciency scores for outpatient model Set 9 Inputs: $OUTPATIENT Set 10 $OPWPCP $OPNOTPCP PANEL Set 11 $OPWPCP $OPNOTPCP PANEL QSCORE Set 12

157

$OPWPCP $OPNOTPCP PANEL QSCORE HEALTHSTAT 0.877 0.651 0.764 0.789 0.580 1.000 1.000 0.971 0.768 0.645 0.774 0.872 0.848 0.729 1.000 0.651 0.771 1.000 0.746 0.720 0.813

Outputs:

PANEL

A B C D E F G H I J K L M N O P Q R S T U

0.744 0.469 0.603 0.505 0.527 0.812 0.768 0.542 0.671 0.508 0.681 0.601 0.751 0.604 0.758 0.488 0.619 1.000 0.643 0.650 0.621

0.877 0.651 0.635 0.789 0.557 0.923 1.000 0.590 0.758 0.645 0.721 0.715 0.794 0.729 1.000 0.504 0.771 1.000 0.746 0.720 0.676

0.877 0.651 0.762 0.789 0.571 1.000 1.000 0.812 0.764 0.645 0.749 0.759 0.832 0.729 1.000 0.640 0.771 1.000 0.746 0.720 0.813

status in Set 12. We noted with interest that physician U, the only efcient PCP in all of the inpatient analyses, scored only in the mid range of efciency in the outpatient analyses. 4.6. Superefciency model One common issue with DEA analyses is that the results can be very sensitive to outliers, i.e., DMUs that are markedly dissimilar from the rest. Sometimes these outliers deserve special attention because they are breakthrough DMUs. Alternatively, they can represent an error in the model data. Either way, these outliers have a strong inuence on the efciency results of the DEA analysis. Insights into the analysis can be gained by studying these DMUs and examining the degree to which they inuence the nal efciency solutions. One method that examines outlier DMUs is called superefciency. This method involves rerunning the DEA model, removing, in turn, each efcient DMU, and calculating a measure of the resulting change. Andersen and Petersen [8] rst proposed the superefciency approach and determined how to calculate the resulting scores. Thrall [10] made the connection that the

158

J.M. Wagner et al. / Socio-Economic Planning Sciences 37 (2003) 141163

resulting scores were measures of extremeness. ONeill [36] applied superefciency to the health care sector by calculating superefciency scores for a DEA model using data from 27 large, urban hospitals. A history of superefciency methods can be found in Dula and Hickman [9]. The superefciency model is an extension of the traditional DEA model. In the traditional model, a DMU can always be in its own reference set. Therefore, the traditional DEA model always has a feasible solution, and the efciency scores will never be higher than 1. In the superefciency model, a DMU is not allowed to be in its own reference set. For an inefcient DMU, the efciency score will not change. However, since efcient DMUs must now have a reference set of other DMUs (a superefciency reference set), the efciency score may now be equal to or greater than 1. DMUs with an efciency score that is greater than 1 are described as superefcient. Fig. 4 shows an example to illustrate the concept of superefciency. The example involves three DMUs that have two outputs (as graphed), and all have the same input. All three DMUs are efcient. The DEA frontier is shown as a solid line. If DMU 2 is removed from its own reference set, the efcient frontier seen by DMU 2 moves toward the origin, as represented by the dotted line. The distance from the origin to the point where DMU 2 is projected onto the shifted efcient frontier is dened as 1. The original position of DMU 2 was 33 percent farther from the origin than the projected point, and thus its superefciency score is calculated as 1.33. The superefciency score for a particular DMU then is a measure of how much the efcient frontier is shifted toward the origin by the removal of that DMU. Thus, superefciency can be interpreted as a measure of a DMUs inuence.

6
Original Frontier

DMU 1

Superefficient Frontier for DMU 2

4
Output 2

Efficiency= 1.33

DMU 2

Efficiency=1

DMU 3

0 0 1 2 3 Output 1 4 5 6

Fig. 4. Superefciency example.

J.M. Wagner et al. / Socio-Economic Planning Sciences 37 (2003) 141163 Table 8 Superefciency scores and related results for combined inpatient and outpatient model (Set 1) DMU Original efciency scores 1.000 0.651 0.968 0.789 0.740 1.000 1.000 1.000 1.000 0.645 0.849 1.000 0.972 0.732 1.000 1.000 1.000 1.000 0.827 0.791 1.000 Original reference set O G, P, U O F, R, U 1.148 2.582 1.193 1.203 G, R F, R 1.039 F, I A, O, R 1.383 1.134 1.372 1.294 A, O, R, U A, O, R, U 2.097 C, P F, G, U H, I, U L, O F G, H, Q G, I, M, U O, Q G, I, L H, O, Q Superefciency scores 1.050 Reference set for superefcient PCPs O, R, U Scores without physician G 1.000 0.651 1.000 0.789 0.740 1.000 1.000 1.000 0.648 0.849 1.000 0.972 0.732 1.000 1.000 1.000 1.000 0.827 0.791 1.000

159

Scores without physician U 1.000 0.651 1.000 0.789 0.789 1.000 1.000 1.000 1.000 0.645 0.849 1.000 0.972 0.732 1.000 1.000 1.000 1.000 0.829 0.804

A B C D E F G H I J K L M N O P Q R S T U

We calculated superefciency scores for all of the models examined in this study. These scores provide some interesting insights into the issue of physician efciency. In this study, all of the linear programs for the superefcient DEA analyses were feasible (infeasibility is technically possible). Table 8 includes superefciency values for the combined inpatient and outpatient model (Set 1). Two PCPs had superefciency scores over 2, namely physician G (2.582) and physician U (2.097). Their scores may indicate that these physicians were of high inuence. However, removing these physicians only affected the efciency scores of inefcient physicians with physicians G or U in their reference set. To check the actual inuence, two more models were run: one with physician G removed and one with physician U removed (also in Table 8). Even with these physicians removed, none of the efciency scores changed very much. When either physician G or physician U was removed, physician Cs efciency increased from 0.968 to 1. Otherwise the greatest change was in the efciency score of physician E. This score increased from 0.740 to 0.789 when physician U was removed. Therefore, for this model, physicians with superefciency scores of 2 do not appear to have a high inuence. Table 9 presents the superefciency values and reference sets for the other models (Sets 212). For all of these analyses, the superefciency scores were relatively low, i.e., below 2.1. We found the highest superefciency values were for physician U in the inpatient models (Sets 58). This result was not unexpected since physician U was the only efcient PCP in all of these models. We can observe the high inuence of physician U by considering, for example, Set 8. With

160

J.M. Wagner et al. / Socio-Economic Planning Sciences 37 (2003) 141163

Table 9 Superefciency scores for sets 212 DEA set 2 3 Efcient PCP R G R U F G H I O R U U U U U R G O R F G O R F G O R Superefciency scores 1.519 1.078 1.030 1.030 1.110 1.212 1.143 1.175 1.002 1.040 1.103 1.519 2.091 2.097 2.097 1.232 1.006 1.033 1.294 1.027 1.926 1.033 1.294 1.027 1.999 1.033 1.294 Superefcient reference set A U A,U F,G,R G,H,I,U H,U G,I F,H F,R A,O F,G,R A C C,P C,P F O,R G F G,R O,Q G F G,R L,O G F

5 6 7 8 9 10

11

12

physician U removed, physicians A, C, and P, previously with efciency scores of 0.658, 0.811, and 0.935, respectively, became efcient. Scores for the other inefcient physicians also increased, some, such as physicians E and O, by almost 0.3. Certainly physician U would be of interest in this analysis. Is this physician highly skilled at diagnosis and able to quickly determine the resources needed to treat his or her panel, thus having both high quality and low resource usage? Or have we made some misidentication, so that physician U is actually a statistical outlier? While superefciency does not have the answer, it does have the capability to highlight these issues. In summary, superefciency is a post-modeling sensitivity analysis technique. For the combined models and for the outpatient models, removing the PCPs of high inuence had only small effects on the efciency scores. For the inpatient models, however, removing the PCP with the high superefciency score did seem to have a signicant impact on the efciency scores of the remaining physicians. Our study of the superefciency models together with the traditional DEA models provided further insight into the efciency of the physicians.

J.M. Wagner et al. / Socio-Economic Planning Sciences 37 (2003) 141163

161

5. Conclusions This study shows the feasibility and advantages of using DEA for physician proling. In this paper, we discuss the development of DEA models for physician proling using resource utilization, severity, and quality measures, taken from existing information in a claims database of a managed health care organization. With DEA we combined multiple inputs and multiple outputs to provide a single comprehensive efciency measure incorporating cost, utilization, severity, and quality, thus quantifying with a single value the degree to which each physician is inefcient. Additionally DEA also identied a reference set of efcient PCPs to which each inefcient PCP can be compared. Essentially, DEA ascertained those PCPs who can serve as role models to the other physicians. The identication of the reference groups represents perhaps the greatest benet of DEA. This health care organization might want to study the efcient physicians to see what best practices could be determined and then generalized to other PCPs. We would also recommend to the organization managers that instead of (or in addition to) distributing quarterly proles, they set up meetings of their physician groups in order to provide an opportunity for the efcient physicians to interact with the other PCPs. This paper has also demonstrated how building the model in a stepwise manner, by adding the input and output variables into the DEA models, one by one, led to greater insight into the reasons why certain physicians were identied as efcient. The traditional method of including all the variables in one analysis led to the conclusion that 11 out of the 21 physicians were efcient (see Section 4.3: Combined Inpatient and Outpatient Analysis) but gave little information about what was allowing some PCPs to be ranked efcient and others not. Building the model in a stepwise manner and dividing the model into separate inpatient and outpatient models revealed reasons for physician efciency. We saw, in this case, that inpatient and outpatient efciencies often differed for the same PCP. We also explored the use of superefciency scores to identify physicians with high inuence on the efcient frontier. The superefciency analysis was easy to implement and posed no technical problems. For inpatients, at least, superefciency did identify a physician of high inuence who would be of particular interest to this health care organization. We believe this study has shown that DEA can be used as a powerful tool to give managers a multitude of options to improve efciency. By determining efciency with a single measure and by identifying a small set of desirable physician practice patterns, the HMO may realize new approaches for managing and controlling costs and for rewarding physicians. Better methods of evaluating physician performance will not only help individual PCPs to improve their practices but also will allow the managed health care organization to better oversee its operations and improve planning and policy-making ability. This study involved only a small group of physicians, as a pilot study. An analysis involving a larger group of physicians could further rene these models. For example, instead of using only one output measure, such as PANEL, it would be possible and desirable to break the panel into subgroups by age and gender or even by health status. This breakdown has commonly been done in other studies. Future work will likely undertake such renements. The next major step for this health care organization is to implement DEA models in routine evaluation of physicians, as either an addition to or even a replacement for the more

162

J.M. Wagner et al. / Socio-Economic Planning Sciences 37 (2003) 141163

traditional ratio-based measures currently used. Such work will include developing the management information systems needed to reliably report DEA results in a production environment and present the results so that they will be understood and used by the physicians involved.

References
[1] Chilingerian JA. Evaluating physician efciency in hospitals: a multivariate analysis of best practices. European Journal of Operations Research 1995;80:54874. [2] Lasker RD, Shapiro DW, Tucker AM. Realizing the potential of practice pattern proling. Inquiry 1992;29: 28797. [3] Salem-Schatz S, Moore G, Rucker M, Pearson SD. The case for case-mix adjustment in practice proling: when good apples look bad. JAMA 1994;272:8714. [4] Charnes A, Cooper WW, Rhodes E. Measuring the efciency of decision making units. European Journal of Operations Research 1978;2:42944. [5] Sherman HD. Hospital efciency measurement and evaluation: empirical test of a new technique. Medicine Care 1984;22:92238. [6] Huang YL, McLaughlin CP. Relative efciency in rural primary health care: an application of data envelopment analysis. Health Services Research 1989;24:14358. [7] Norman M, Stoker B. Data envelopment analysis: the assessment of performance. Chichester, England: Wiley, 1991. [8] Andersen P, Petersen NC. A procedure for ranking efcient units in data envelopment analysis. Management Science 1993;39:12614. [9] Dula JH, Hickman BL. Effects of excluding the column being scored from the dea envelopment lp technology matrix. Journal of the Operational Research Society 1997;48:100112. [10] Thrall RM. Duality, classication and slacks in dea. Annals of Operations Research 1996;66:10938. [11] Grosskopf S, Valdmanis V. Measuring hospital performance: a non-parametric approach. Journal of Health Economics 1987;6:89107. [12] Ozcan YA, Luke RD. A national study of the efciency of hospitals in urban markets. Health Services Research 1993;27:71939. [13] Ozcan YA. Sensitivity analysis of hospital efciency under alternative outputXinput and peer groups: a review. Knowledge & Policy 19921993;5:129. [14] Nunamaker TR. Measuring routine nursing service efciency: a comparison of cost per patient day and data envelopment analysis models. Health Services Research 1983;18:183205. [15] Dittman DA, Capettini R, Morey RC. Measuring efciency in acute care hospitals: an application of data envelopment analysis. Journal of Health & Human Research Administration 1991;14:89108. [16] Sexton TR, Leiken AM, Nolan AH, Liss S, Hogan A, Silkman RH. Evaluating managerial efciency of veterans administration medical centers using data envelopment analysis. Medical Care 1989;27:117588. [17] Pina V, Torres L. Evaluating the efciency of nonprot organizations: an application of data envelopment analysis to the public health service. Financial Accounting and Management 1992;8:21324. [18] Ozcan YA, Cotter JJ. An assessment of area agencies on aging in Virginia through data envelopment analysis. The Geront 1994;34:36370. [19] Ozcan YA, Begun JW, McKinney MM. Benchmarking organ procurement organizations: a national study. Health Services Research 1999;34:85574. [20] Draper DA, Solti I, Ozcan YA. Characteristics of health maintenance organizations and their inuence on efciency. Health Services Management Research 2000;13:4056. [21] Salinas-Jimenez J, Smith P. Data envelopment analysis applied to quality in primary health care. Annals of Operations Research 1996;67:14161. [22] McNeil BJ, Pedersen SH, Gatsonis C. Current issues in proling quality of care. Inquiry 1992;29:298307.

J.M. Wagner et al. / Socio-Economic Planning Sciences 37 (2003) 141163

163

[23] Chaiken BP. Impact on provider proling. Virginia Medical Quarterly 1966;123:23840. [24] Christiansen CL, Morris CN. Improving the statistical approach to health care provider proling. Annals of International Medicine 1997;127:7648. [25] DeLong ER, Peterson ED, DeLong DM, Muhlbaier LH, Hackett S, Mark DB. Comparing risk-adjustment methods for provider proling. Statistics in Medicine 1997;16:264564. [26] Chilingerian JA, Sherman HD. Managing physician efciency and effectiveness in providing hospital services. Health Services Management Research 1990;3:315. [27] Chilingerian JA. Exploring why some physicians hospital practices are more efcient: taking dea inside the hospital Data Envelopment Analysis. In: Charnes A, Cooper WW, Lewin AY, Seiford LM, editors. Theory, methodology and applications. Boston: Kluwer Academic Publishers, 1994. p. 16793. [28] Chilingerian JA, Sherman HD. Benchmarking physician practice patterns with dea: a multi-stage approach for cost containment. Annals of Operations Research 1996;67:83116. [29] Ozcan YA. Physician benchmarking: measuring variation in practice behavior in treatment of otitis media. Health Care Management Science 1998;1:517. [30] Ozcan YA, Jiang HJ, Pai CW. Do primary care physicians or specialists provide more efcient care? Health Services Management Research 2000;13:906. [31] Wagner JM, Shimshak DG. Physician proling using data envelopment analysis: a case study. International Journal of Health Technology and Management 2000;2:35874. [32] Cooper WW, Seiford LM, Tone K. Data envelopment analysis: a comprehensive text with models, applications, references and dea-solver software. Boston: Kluwer Academic Publishers, 2000. [33] Ozcan YA, Watts J, Harris JM II, Wogen SE. Provider experience and technical efciency in the treatment of stroke patients: dea approach. Journal of the Operational Research Society 1998;49:57382. [34] Thrall RM. Classication transitions under expansion of inputs and outputs in data envelopment analysis. Managerial and Decision Economics 1989;10:15962. [35] Golany B, Roll Y. An application procedure for dea. OMEGA 1989;17:23750. [36] ONeill L. Multifactor efciency in data envelopment analysis with an application to urban hospitals. Health Care Management Science 1998;1:1927.