Article

pubs.acs.org/est

Structured Expert Judgment to Characterize Uncertainty between

PM2.5 Exposure and Mortality in Chile
Pamela C. Cisternas,*,, Nicolas C. Bronfman,, Raquel B. Jimenez, Luis A. Cifuentes,,
and Cristobal De La Maza

Engineering Sciences Department, Universidad Andres Bello, Santiago 8370146, Chile

National Research Center for Integrated Natural Disaster Management CONICYT/FONDAP/15110017, Santiago, Chile

Industrial and Systems Engineering Department, Ponticia Universidad Catolica de Chile, Santiago, Chile

Chile Foundation, Santiago, Chile

S Supporting Information
*

ABSTRACT: To further the understanding and implementation of expert elicitation methods in the evaluation of public
policies related to air pollution, the present studys main goal
was to explore the potential strengths and weaknesses of
structured expert judgment (SEJ) methodology as a way to
derive a C-R function for chronic PM2.5 exposure and premature
mortality in Chile. Local experts were classied in two groups
according to background and experience: physicians (Group 1)
and engineers (Group 2). Experts were required to provide an
estimate of the true percent change in nonaccidental mortality
resulting from a permanent 1 g/m3 reduction in PM2.5 annual
average ambient concentration across the entire Chilean
territory. Cookes Classical Model was used to combine the
individual experts assessments. Experts mortality estimations varied markedly across groups: while experts in Group 1 delivered
higher estimations than those reported in major international cohort studies, estimations from Group 2 were, to varying degrees,
anchored to previous studies. Accordingly, combined distributions for each group and all experts were signicantly dierent, due to
the high sensitivity of the weighted distribution to experts performance in calibration variables. Results of this study suggest that,
while the use of SEJ has great potential for estimating C-R functions for chronic exposure to PM2.5 and premature mortality and its
major sources of uncertainty in countries where no studies are available, its successful implementation is conditioned by a number of
factors, which are analyzed and discussed.

INTRODUCTION
Air pollution is a major risk factor for public health, since public
exposure to air pollution has been linked to numerous negative
health impacts. Over the last two decades, time series and
cohort studies have found evidence of a relationship between
short- and long-term exposure to ne particulate matter
(PM2.5) and increased incidence of adverse health impacts
such as cancer, cardiovascular and respiratory diseases, and
premature mortality.110 In these studies, the relationship
between ambient concentration of air pollutants and the
incidence of a specic health eect is described using
concentrationresponse (C-R) functions. More specically,
results from cohort studies suggest signicant relationships for
long-term PM2.5 exposure and increased mortality from general
and cardiovascular diseases. The most cited studies were
developed in the 90s.11,12 More recent studies have been
performed reanalyzing data from previous work2,1318 and
using new data.1923 While, in general, these studies have
reached some degree of consensus regarding the existence of
this relationship, signicant variability is observed in CR
2014 American Chemical Society

functions across studies (see, for example, results from Pope

et al. (2002)2 (P50 = 0.6, CI = 0.161.1), versus results from
Laden et al. (2006)16 (P50 = 1.6, CI = 0.72.6)).
Additionally, there remains signicant uncertainty about the
true nature of the relationship between PM2.5 ambient
concentration and increased incidence of health end points,
provoking debate among experts in the eld. The major sources
of uncertainty are related to (i) the existence of a causal
relationship between health impacts and PM2.5 exposure, (ii) if
there is a relationship, what is the shape of the CR function
and does it present a threshold for eects, (iii) the time window
between exposure and the rst manifestation of eects, and (iv)
the use of PM2.5 mass as a measure of its potential toxicity.24
Epidemiological evidence from long- and short-term studies
provide the informational basis for assessing air pollution
Received:
Revised:
Accepted:
Published:
9717

January 28, 2014

June 27, 2014
July 7, 2014
July 7, 2014
dx.doi.org/10.1021/es500037k | Environ. Sci. Technol. 2014, 48, 97179727

Environmental Science & Technology

Article

calibration variables are dened in section 2.4), eight related to

premature mortality from short- and long-term PM2.5 exposure,
two to the toxicity of PM2.5 components, and two related to
health eects from exposure to the Kuwait oil res in 1991.
Additionally, 12 calibration questions were included and were
combined with expert estimates using the Classical Model (see
the Classic Model section for a description).
Both studies conrm that implementing expert judgment to
quantify uncertainty in the PM2.5-mortality CR function
enables the combination of empirical results with expert
judgments to achieve a comprehensive approach to the
uncertainty inherent in assessing adverse health eects from
PM2.5 exposure.
Studies in Chile. Despite the high levels of air pollution
observed in medium-size and large cities across Chile, only a
few studies addressing public health impacts from exposure to
atmospheric pollutants have been developed locally. Cifuentes
et al.46 estimated a 4.2% increase in nonaccidental daily
mortality attributable to PM2.5 ambient concentrations of
64 g/m3in Santiago. Sanhueza et al.47 found strong relationships
between PM10 exposure and daily cardiovascular and respiratory
mortality in Temuco (southern Chile). Other studies have focused
on increased rates of hospital4851 and emergency admissions.5254
So far, no study has been developed in Chile to estimate the
increment in premature mortality attributable to long-term
PM2.5 exposure. As a result, regulations and standards to protect
public health from adverse eects of air pollution have been
developed using results directly transferred from international
studies. For example, in determining the primary environmental
quality standard for PM2.5 in Chile, nonaccidental mortality
attributed to chronic PM2.5 exposure was quantied using results
from Pope et al.2
It is worth noting that the direct transference of results
assumes that the mortality risk from PM2.5 is independent of
the area of analysis, that is, independent of demographic
characteristics, health status of the population, pollutant
mixture and climatological features. As the understanding of
how these factors modify the eects of PM2.5 exposure is rather
limited, is reasonable to assume that the direct transference of
results from international studies adds uncertainty to local
estimations of health benets from air quality improvements.
Within this context, the development of local studies provides
relevant information to better understand how PM2.5 exposure
impacts local population. Including this information in the
assessment of public health impacts from pollutant exposure
will provide results that better reect the local context, as it
complements the international literature with more specic,
local information for a more complete characterization of
impacts and uncertainties in this process.
Overview of the Present Study. The economic
evaluation of public health risks and benets from reductions
in air pollution ambient concentrations using CR functions is
fundamental to evaluating public policies and normative
instruments aimed at improving air quality. In Chile, air quality
standards for PM2.5 have been evaluated using CR functions
directly transferred from international studies, as no study has
been conducted in this country to estimate the relationship
between premature mortality and long-term PM2.5 exposure.
While directly transferring results adds uncertainty to this
process, developing cohort studies involves high costs and long
periods of time to get results, both fundamental limitations to
the widespread implementation of this type of study. For these
reasons, in countries where epidemiological cohort studies of

management strategies and normative instruments. For

example, CR functions are key inputs in assessing public
health benets from reductions in adverse health eects
expected as a result of air quality improvements. Nevertheless,
there are a limited number of cohort studies addressing longterm exposure to PM2.5, mainly because of the high economic
costs involved in their development and the amount of time it
takes to get conclusive results. This has led to the use of
alternative methodologies for estimating CR functions in
countries where no cohort studies are available. These
alternatives include directly transferring results from studies
developed elsewhere and using expert judgment to adjust such
results (both central estimates and uncertainty intervals) to
better represent local impacts from exposure to this pollutant.
In the last few decades, the structured expert judgment
(SEJ)a tool used to assess uncertainty in complex problems
where parameters are unknown or data is scarcehas been used
to quantitatively characterize the state of knowledge and
uncertainties about health eects from exposure to ambient
pollutants. The main dierence between expert judgment and
SEJ is that the latter uses two measures of performance
(calibration and information detailed in section 3 of the
Supporting Information (SI)), by which a weight is assigned to
each expert based on their performance on both measures. This
allows to transmit the virtues of the good experts (those who
have good performance in both measures) to the combined
distribution (for more detail refer to refs 25 and 26).
SEJ was originally developed to address the high uncertainty
embedded in nuclear plant accident risk and consequence
assessments in the U.S.2735 In 2000, Cooke and Goossens36
developed a methodological guide for implementing SEJ for
uncertainty assessments in quantitative models, thus opening
the door to this tools application outside of the nuclear power
community. Since then, expert judgment has been used in risk
assessments for dierent areas, such as pollution in the food
production chain,37,38 volcanic eruptions,39 inhalation of toxic
chemicals40 and exposure to air pollution.4143 In this study, we
have implemented a SEJ in order to explore the potential
strengths and weaknesses when used in the estimation and
uncertainty assessment of local CR functions for regulatory
and policy assessment in Chile.
Structured Expert Judgment and Its Application in
PM2.5 Health Eects. In 2002, the U.S. National Research
Council (NRC) suggested the U.S. Environmental Protection
Agency (U.S. EPA) the use of expert judgment to determine
the probability distributions of key sources of uncertainty in the
information sources, process and key assumptions used to
estimate health benets from air pollution reductions.44 In
2006, after the successful implementation of a pilot study in
2004,43 the U.S. EPA developed a formal study incorporating
expert judgment elicitation, in which a 12 expert panel (eight
epidemiologists, three toxicologists and one physician) was
implemented in order to address the major uncertainties
surrounding the relationship between PM2.5 exposure and
mortality. Additionally, experts made quantitative estimates of
the impact of a 1 g/m3 reduction in PM2.5 annual average on
nonaccidental mortality in U.S. adults. However, it did not
include a combination of expert estimates in a single probability
distribution that reected the experts degree of uncertainty.42,43
More recently, Tuomisto et al.45 implemented a SEJ, in
which six European experts quantied uncertainty in the
relationship between PM2.5 exposure and premature mortality.
The protocol included 12 query variables (query variables and
9718

dx.doi.org/10.1021/es500037k | Environ. Sci. Technol. 2014, 48, 97179727

Environmental Science & Technology

Article

relation to a background distribution (uniform or log-uniform),

selected by the researchers overseeing the study.
Such weights are assigned using Equal Weight criteria, where
all experts have the same weight, or performance-based weight
criteria, in which experts estimates are weighted according to
their performance over the whole set of seed variables (Global
Weight) or in each seed variable (Item Weight).
Participants. Two Chilean experts in the eld were
contacted to identify potential participants by listing renowned
local experts. In this way, a list of 16 potential participants was
created based on the experts nominations, ISI journal
publications, and other criteria, all of which have directed or
participated in epidemiological studies related to air pollution
exposure and adverse health eects in Chile. Seven of them
agreed to participate in the study. Participation in the study was
voluntary, and no economic or material compensation was
oered. To maintain condentiality, experts are designated
throughout this document as Expert A through Expert G.
In order to assess potential dierences between experts from
dierent backgrounds, the expert panel was divided into two
groups according to eld of expertise. Experts C and D
comprised Group 1 based on their training as physicians and
experience in the elds of environmental epidemiology and
environmental and public health. Group 2 comprised experts A,
B, E, F, and G, with experience in a variety of elds, including
air pollution dispersion and exposure modeling, environmental
epidemiology and cost-benet analysis for evaluating environmental regulatory instruments. Also, experts in Group 2 have
been actively involved in air pollution policies and regulatory
decision-making processes.
Seed Variables. Seven seed variables (S1S7) were dened
based on Tuomisto et al.45 Three variables were related to the
number of days where daily average PM10 ambient concentration exceed 150 g/m3 in at least one monitoring station in
the Metropolitan Region (MR) in the years 2000 (S1), 2005
(S2), and 2010 (S3). Two variables were related to the eect of
PM10 on all-cause mortality, estimated as the number of
nonaccidental deaths in the week with the highest average PM10
concentration to the number of nonaccidental deaths in a
typical week in years 2000 (S4) and 2005 (S6). The last two
variables were related to the impact of PM10 on cardiovascular
mortality, estimated as the ratio of the number of cardiovascular
deaths in the week with the highest average PM10 concentration
to the number of cardiovascular deaths in a typical week in
years 2000 (S5) and 2005 (S7). A complete description of seed
variables is provided in the SI.
It is worth noting that in Chile, as well as in several other
countries, high pollution concentration episodes are observed
more frequently in the winter, inuenced by atmospheric and
meteorological conditions, such as lower temperatures, high
atmospheric stability, and low ventilation. Such factors, in turn,
strongly inuence higher cardiovascular and respiratory
morbidity and mortality rates observed in winter, compared
to annual tendencies. Therefore, it is expected that experts
estimates for variables S4, S5, S6, and S7 are greater than one.
Experts delivered their valuations for each seed variable using
5th, 25th, 50th, 75th, and 95th percentiles.
The information used to estimate the real values of seed
variables was obtained from death certicates and historical
records of PM10 ambient concentration from seven of the 11
air-quality monitoring sites in the MR. The real values are
shown in Figure 1.

PM-mortality are not available, the use of SEJ has great

potential to provide probabilistic characterizations of the eects
of chronic exposure to ne particulate matter.
To further the understanding and implementation of this
instrument, the present studys main goal is to explore the
potential strengths and weaknesses of SEJ as a way to derive
CR functions for regulatory policy assessment and to
characterize the uncertainty in these estimates.

MATERIALS AND METHODS

Structured Expert Judgment: The Protocol. As dened
by Cooke and Goossens,36 the SEJ protocol has three main
stages: (i) preparation for elicitation, (ii) elicitation, and (iii)
postelicitation.
Preparation for Elicitation. First, the problem to be
addressed, that is, the case study is dened. The main goal
and uncertain parameters that experts must estimate are
established. Once the case study is dened, query variables are
dened to assess and quantify uncertain parameters directly
related to the studys goal. Then, calibration or seed variables
are dened. These variables are used to quantify experts
performance as subjective probability assessors and to enable
performance-optimized combinations of expert distributions.
Seed variables also provide valuable feedback to experts,
helping them to gauge their subjective sense of uncertainty
against quantitative measures of performance.34 The experts
estimates for query and seed variables must be delivered in
percentiles (i.e., P5, P25, P50, P75, and P95).
Next, potential participants are identied, usually using peer
nomination techniques. Then, a dry-run session is performed to
verify that the elicitation documents contents, structure and
format are suitable. Prior to the elicitation, a training session
must be carried out to ensure that the experts are familiar with
the procedure.
Elicitation. In the elicitation stage, experts must make
estimates of seed and query variables using percentiles and
clearly stating all assumptions and fundaments that support
their probabilistic estimation.
Post-Elicitation. For each query variable, experts estimates
are combined into a single probability distribution, the decision
maker distribution (DM), which quanties the uncertainty
inherent to the query variable. The DM is obtained using
Cookes Classic Model, which is explained in the following
section. Robustness analyses are performed to evaluate the
signicance of information loss in the combined distribution
when experts or seed variables are omitted from the DM
estimate. Individual estimates are sent back to each expert, with
their respective scores for calibration, information and weight in
the combined distribution. With this information, experts can
modify their estimates. Finally, all relevant information and data
are consolidated in a formal report to be presented to the
projects Decision Managers and the experts.36
Classic Model. The Classic Model25 allows researchers to
estimate a combined distribution for each query variable by
making a linear combination of experts estimates assigning
weights according to two measures of performance in seed
variables: calibration and information. A good expert is one
with high calibration and information scores.31 The calibration
score is the p-value of falsely rejecting the hypothesis that an
experts probability statements are statistically accurate. The
maximal value is 1, and the minimal is 0. Information refers to
the degree to which the experts distribution is concentrated in
9719

dx.doi.org/10.1021/es500037k | Environ. Sci. Technol. 2014, 48, 97179727

Environmental Science & Technology

Article

Figure 1. Expert panel estimates for seed variables S1S7. Horizontal lines indicate the true value of each variable, unknown by experts. A complete
description of seed variables is provided in SI.

Query Variables. The elicitation protocol included seven

query variables Q1Q7, one of which was the quantitative
question (Q7). A complete description of query variables is
provided in Table 1. Variables Q1 and Q2causality
questionswere developed based on U.S. EPA 41 and

addressed uncertainty in the causality of the relationship

between long- and short-term PM2.5 exposure and premature
mortality in Chile. Variable Q3the question on transference
of resultsaddressed the experts position regarding the
validity of using results from epidemiologic studies developed
9720

dx.doi.org/10.1021/es500037k | Environ. Sci. Technol. 2014, 48, 97179727

9721

Based on your interpretation of the evidence, do you want to incorporate a threshold in your characterization of the CR relationship?

What is your estimate of the true percentage change in annual mortality from nonaccidental causes in the adult population of Chile (over 18 years) due to a permanent reduction of 1 g/m3 in annual average
PM2.5 concentrations (over the entire concentration range of 1035 g/m3)? In formulating its response, you must consider eects of reductions in PM2.5 concentrations on both the short and long-term
mortality. To characterize the uncertainty in the CR function, you must provide your estimate as percentiles (fth, 25th, 50th, 75th, and 95th).

Q6

Q7

Note: Question Q4 refers to the shape of the CR function. Questions Q5 and Q6 are related to threshold. Question Q7 is the quantitative question on mortality estimation in Chile.

Do you want to specify if the CR function for PM2.5 and mortality diers across the specied concentration range? If your answer is yes, indicate the form of the functions, the approach to incorporate causality,
and if it considers a concentration threshold.

Do you believe that a concentration threshold for annual nonaccidental mortality associated with population exposure to PM2.5 is detectable in any currently available study

Q5

Query Variables Related to Mortality Estimates for PM2.5 Exposure in Chile.

Q4
Determine the shape of the CR function for long-term exposure (you can make a sketch of the general shape of the function).

Query Variables Related to Transference of Results.

Q3
Despite the growing number of studies about health eects of exposure to air pollution, the extrapolation of results obtained in one location to estimate impacts in another is very frequent. The assumption of the
validity of transference of results states that the risk of mortality from exposure to PM2.5 is independent of the area of analysis. It is assumed that the demographic characteristics, pollutant mixture, the health
status of the population and climatic features are transferable between study areas because the current understanding of how these factors modify the eects of exposure to PM2.5 is rather limited. Do you
consider this as a reasonable assumption? Please specify your arguments.

What evidence do you considered to support or reject the possibility of a causal relationship between reductions in annual average exposure to PM2.5 (including long and short-term exposure reductions) and
changes in mortality in Chile (at annual average 1035 g PM2.5/m3). Do you wish to make a distinction between short-term and long-term relationship? Briey specify your arguments to make such distinction.

Q2

query variables related to causality between PM2.5 exposure and mortality in Chile.

In your opinion, is there a causal relationship between short/long-term exposure to PM2.5 and mortality? The answer must be delivered according to the U.S. EPA standard on weight of evidence for the
determination of causality.24

Q1

Table 1. Query Variablesa

Environmental Science & Technology

Article

dx.doi.org/10.1021/es500037k | Environ. Sci. Technol. 2014, 48, 97179727

Environmental Science & Technology

Article

short-term exposure and likely to be causal for long-term, and

experts A and E declared that there is a probable causal
relationship in both short- and long-term exposure.
The experts based their causality judgments for short-term
exposure on studies conducted in Chile,46,50,56 and studies
conducted in other countries,57.58 The experts cited international
cohort studies,11,12,2,5960 and the WHO guidelines61 when making
their causality judgments for long-term exposure and mortality.
Transference of Results. Experts C, D, and F stated that they
did not agree with the direct transference of results and the
underlying assumption that mortality risk from PM2.5 exposure is
independent of the demographic characteristics and, health status
of the population, pollutant mixture and climatological features
(Q3 in Table 1). In general, these experts argued that the
assumption was not valid since it would violate the existence of four
determinants: biological, socio-cultural, environmental variability
and variability in healthcare systems. On the other hand, experts A,
B, E, and G thought that direct transference was sensible.
Mortality Estimates. Marked dierences in estimations for
the mortality questions, especially in the quantitative question,
were observed between the two groups. Regarding the shape of
the CR function (Q4), experts in Group 2 determined that it
was loglinear, whereas experts in Group 1 said the shape was
loglinear by segments. However, none of the latter groups
members gave cuto values for the loglinear function
segments, due to the lack of evidence. While experts B, C, D,
E, and F believed there was no threshold in the CR function
(Q5), experts A and G said there was a threshold. Even so, no
expert included the existence of a threshold in their mortality
estimates (Q6), within the range of ambient PM2.5 concentration (1035 g/m3) dened in this study.
Figure 2 summarizes the expert panels estimates in the
quantitative question (Q7), as well as the results of the four
major cohort studies and the methodological recommendations
from the WHO.61 Experts in Group 1 delivered the highest
estimates in all percentiles, and showed the widest uncertainty
intervals (fth to 95th percentile). This is consistent with their
experts responses to Q3, where they indicated that the direct
transference of results was not valid.
Estimates provided by experts from Group 2, who in general
considered the direct transference assumption to be valid (all
except expert F), were very similar to results from previous
cohort studies. Experts A and B were concentrated in a region
of values between those reported by Pope et al.,2,12 and lower
than those of Dockery et al.11 and Laden et al.,16 while expert F
provided similar values to those reported by Dockery et al.11
Experts E and G provided a distribution similar to the value
recommended by the WHO.61 The main sources of uncertainty
considered by each expert when making their estimates for Q7
are detailed in Table S6 in the SI.
Combining Experts Estimates. Figure 2 shows distributions combined under equal, global and item schemes for the
entire panel, and for each group, considering calibration factors
estimated based on the experts performance in seed variables
(see Table 2). Results reveal signicant dierences across the
DM for the entire panel and groups 1 and 2not only in their
mean values, but also in experts calibration scores. These results
illustrate how experts estimates are combined using performancebased weights: the main virtue of the SEJ methodology.
While calibration scores for Group 1 in the DM obtained
using the equal weight (EW-1), global weight (GW-1) and item
weight (IW-1) schemes are signicantly dierent (0.3925,
0.0358, and 0.2627, respectively), the DM distributions are

in one country to estimate health impacts from PM2.5 exposure in

another country, where no local studies have been developed.
Variables Q4Q7, mortality questions, related to the CR
function for mortality from long-term exposure to PM2.5 in
Chile, were developed based on U.S. EPA.41 Experts were
required to determine the shape of the CR function (Q4) the
potential existence of a threshold (Q5), as well as to decide
whether to incorporate a threshold into the estimated CR
function (Q6). Finally, the quantitative question (Q7) required
experts to estimate the percent change in annual nonaccidental
mortality in adults caused by a permanent 1 g/m3 reduction in
PM2.5 ambient concentration throughout the entire Chilean
territory. Experts estimates for the quantitative question were
made considering annual average of ambient PM2.5 concentration in the range of 1035 g/m3, and were delivered as
probability distributions, including maximum and minimum
values, and fth, 25th, 50th, 75th, and 95th percentiles.
Complementary Material: Information and Resources.
To avoid potential information gaps, the study team developed
an elicitation document, a webpage with relevant information
and a graphical tool to aid experts with visualizing their estimated
functions.
Dry-Run Session. A pilot session was implemented in order
to test and validate the elicitation process and the instruments
developed for the study. This session comprised three work
meetings with two professionals with extensive experience in
the eld of interest and who were not part of the expert panel.
Procedure. Elicitation Session. Three individual 1 h
meetings with each expert were held. In the rst meeting, the
problem, goal, methodology, and seed and query variables were
presented. After 2 weeks, a second meeting was held to clarify
any questions and elicit seed, causality and transference
variables. Finally, in the third meeting, the experts estimates
for the mortality variables were elicited.
Post-Elicitation. Experts estimates were combined following
the Classical Model to estimate a probability distribution for the
quantitative question (Q7) using the software EXCALIBUR.55
Combined distributions for Group 1, Group 2, and the entire
expert panel were estimated.
A summary of results for the estimates, weights, and
performance was sent to each expert for them to approve or,
if necessary, modify their answers. Three dierent combination
schemes were used: equal weight, global weight, and item weight.

RESULTS
Seed Variables. Only Expert B modied his original
estimates after seeing his individual calibration and information
scores. Experts estimates for seed variables are shown in Figure 1.
In general, experts performance in S1, S2, and S3 was good,
although only experts C and D included the real value of these
three seed variables in their estimations. Note that these two
experts provided the largest uncertainty intervals for S1, S2, and S3.
As expected, the entire panel estimated ratios greater than
one for every percentile of seed variables S4, S5, S6, and S7;
nonetheless, overall experts performance in these variables was
low. In general, all experts with the exception of experts C and
E, provided narrow condence intervals for these variables,
missing the real value by large margins.
Query Variables. Causality. All participants agreed that
there is some degree of causality between exposure to PM2.5
and premature mortality: experts B, D, and F considered that
the relationship is causal both in the short and long-term,
experts C and G considered that the relationship is causal for
9722

dx.doi.org/10.1021/es500037k | Environ. Sci. Technol. 2014, 48, 97179727

Environmental Science & Technology

Article

Figure 2. Uncertainty distributions for the relationship between PM2.5 and nonaccidental mortality in Chile. Box plots represent experts
distributions, the three dierent combination schemes: equal (EW), global (GW) and item weight (IW) for each group (1 and 2) and for all the
experts (A), and the results from four major international cohort studies and the WHO guidelines.

assessments and to characterize the uncertainty in these

estimates. It is worth noting that the SEJ procedure is
characterized by its intrinsically high level of methodological
complexity, and its use in the specic context of environmental
epidemiology and air pollution is very limited. While results of
this study suggest that SEJ can provide relevant information to
better understand how PM 2.5 exposure impacts local
populations in countries where no cohort studies have been
developed, its successful implementation is conditioned by a
number of factors, which are discussed below.
Seed Variables. The seven seed variables included in this
study were intended to quantify the experts performance and
their ability to express self-knowledge regarding two aspects of
the evolution of air pollution and exposure-related mortality
in the study area: changes in daily levels of PM10 in the MR
(S1, S2, and S3) and changes in mortality in periods of high
PM10 ambient concentration. Questions S4 and S6 address the
eect of PM10 on all nonaccidental mortality. Questions S5 and
S7 are similar but address the eect of PM10 on cardiovascular
mortality.
Regarding questions related to the second aspect, experts, in
general, delivered estimates considerably dierent from the true
values for these variables. While all experts provided ratios
greater than onethus recognizing that nonaccidental and
cardiovascular mortality rates should be higher in periods of
higher ambient pollutant concentration in Chiletheir
estimates diered signicantly from the actual values.
We think these results may arise from the experts possibly
having developed these distributions considering only the
isolated eect of air pollution on mortality (i.e., relative risk),
rather than the complete set of factors that aect mortality in
winter. Thus, this group of experts would underestimate the
ratio of nonaccidental and cardiovascular mortality during
periods of high pollution and average mortality trends.
Also, we believe that the design of seed variables S4S7 had
a strong inuence on experts performance in these variables.

markedly similar, all revealing the same value for the 50th
percentile. The three DM for Group 1 are approximately two
times higher than estimates from Dockery et al.,11 and Laden et
al.,16 while presenting greater uncertainty intervals. The clear
similarity is due to the high weighting of experts C and D in the
combined distribution using the global and item schemes,
estimated based on their performance in the seed variables.
The DM for Group 2, where all experts are equally weighted
(EW-2), received a calibration score of 0.0146. Figure 2 shows
that the combined distributions are closer to results reported by
Pope et al.2,12 On the other hand, the calibration score obtained in
the global weight DM of Group 2 (GW-2) was signicantly lower
than the obtained in the item weight DM of Group 2 (IW-2).
DM distributions combined under the equal, global and item
schemes for the complete panel (EW-A, GW-A, and IW-A,
respectively) present important dierences in their central
estimates (EW-A P50 = 0.9294, GW-A P50 = 2.7830, and IW-A
P50 = 2.4480) as well as in uncertainty intervals. While EW-A
incorporate the results from four major international cohort
studies and the WHO guidelines, the GW-A and IW-A are
similar to the DMs of the Group 1, as experts C and D
obtained the highest calibration scores of the panel.
Robustness Analysis. The robustness analysis for variables
revealed that the calibration score for Group 1 is particularly
sensitive to seed variables S1, S3, and S6, while Group 2, is
more sensitive to the seed variables S4, S5, and S6. For all the
experts, the rst three seed variables (S1, S2, and S3) retained
the greatest inuence on the model.
In relation to the models sensitivity toward experts, the
analysis shows that the calibration score of Group 1 is not
sensitive to the experts and the calibration score of Group 2
showed greater sensibility to Experts B and E.

DISCUSSION
In this study we explore the potential strengths and weaknesses
of SEJ as a way to derive CR functions for regulatory policy
9723

dx.doi.org/10.1021/es500037k | Environ. Sci. Technol. 2014, 48, 97179727

0.001 05
0.000 10
0.000 04
0.000 12
0.000 12
0.01464
0.02825
0.16680

0.011 02
0.004 46
0.39250
0.03583
0.26270
1.340 00
6.610 00
0.783 40
1.994 00
1.882 00
0.582 50
0.833 30
1.105 00

0.583 40
1.671 00
0.325 20
0.523 30
0.553 00

all
variables

1.488 00
7.474 00
0.792 00
2.262 00
2.049 00
0.654 10
0.932 50
1.251 00

0.647 20
1.885 00
0.363 50
0.589 70
0.623 40

seed
variables

relative information

0.001 57
0.000 73
0.000 03
0.000 27
0.000 25
0.009 58

0.007 13
0.008 40
0.142 70

unnormalized
weightc

0.2
0.2
0.2
0.2
0.2

0.5
0.5

0.126 10
0.059 02
0.002 30
0.021 80
0.019 75
0.77110

0.045 07
0.053 08
0.90180

with DM

normalized weight
without
DM

0.026 34

0.001 57
0.000 73
0.000 03
0.000 27
0.000 25

0.021 13

0.007 13
0.008 40

unnormalized
weightc

0.550 70
0.257 80
0.010 05
0.095 21
0.086 26

0.459 20
0.540 80

without
DM

0.90260

0.053 65
0.025 12
0.000 98
0.009 28
0.008 40

0.57630

0.194 50
0.229 10

with DM

normalized weight

global weight

0.208 70

0.001 57
0
0
0.000 27
0.000 25

0.16380

0.007 13
0.008 40

unnormalized
weightc

without
DM

0.99010

0.007 43
0
0
0.001 29
0.001 16

0.91340

0.039 77
0.046 84

with DM

normalized weight

item weight

Three combination schemes are compared: equal, global and item weight. bThe calibration score is the p-value of falsely rejecting the hypothesis that an experts probability statements are statistically
accurate. The maximal value is 1, the minimal value is 0. cRepresents the weight assigned to each expert, calculated as the product of the calibration score (second column) and the information score
relative to seed variables (fourth column). dDM_EW: Equal weight combination scheme. eDM_GW: Global weight combination scheme. fDM_IW: Item weight combination scheme.

Expert A
Expert B
Expert E
Expert F
Expert G
DM_EW2d
DM_GW2e
DM_IW2f

Group 2

Expert C
Expert D
DM_EW1d
DM_GW1e
DM_IW1f

Group 1

expert

calibration
Scoreb

equal weight

Table 2. Results from Using Seed Variables to Obtain the Combined Distribution of the Expert PanelS Estimates for the Mortality Questiona

Environmental Science & Technology

Article

9724

dx.doi.org/10.1021/es500037k | Environ. Sci. Technol. 2014, 48, 97179727

Environmental Science & Technology

Article

linked to the experts backgrounds and professional experience.

While almost the entire panel had previous experience in
epidemiological studies, the experts came from two dierent
elds. It is possible that the high estimates from Group 1 were
inuenced by the medical backgrounds of experts C and D,
since they have had more direct contact with patients suering
from respiratory and cardiovascular illnesses worsened by air
pollution exposure and are more knowledgeable about the
toxicological mechanisms through which pollutants aect the
respiratory and cardiovascular systems. Therefore, their
estimates were more independent from previous international
cohort studies.
On the other hand, mortality estimates from the experts in
Group 2, who have backgrounds in engineering and
professional experience in exposure assessments and costbenet analyses, are notably similar to estimates from
international studies. Basically, these experts are consumers of
mortality estimates reported in cohort studies, as they rely on
CR functions reported in the literature to estimate health
impacts from air pollution exposure and to perform regulatory
cost-benet analyses. Consequently, it is not surprising that
their estimates show an anchoring bias toward familiar CR
functions.
The marked dierences between combined distributions
considering all experts (EW-A, GW-A, and IW-A) evidence the
strong sensitivity of combined distributions with expert
calibration to experts performance in seed variables: for
example, the GW-A and IW-A are clearly dominated by
estimates of experts C and D. These experts were calibrated
with weights signicantly higher than the rest of the panel, even
though overall performance of these experts in seed variables
was not good, mainly because poor performance of the rest of
the experts in the seed variables. These results illustrate how
experts estimates are combined using performance-based
weights: the main virtue of the SEJ methodology.
Expert Panel. The expert panel members agreed to
participate in this study anonymously and voluntarily, without
any kind of incentive or remuneration for their collaboration.
The estimates they made required several hours of work
focused on the examination, evaluation and discussion of
background information. Anonymous participation without
direct compensation (beyond personal satisfaction derived from
collaborating) might have aected the expert panel members
disposition to committing the time and eort necessary for this
type of study.
The research team suggests that future studies consider
abandoning the anonymous participation method, as well as
providing the appropriate incentives to motivate experts to
dedicate the necessary time needed to make estimates that
reect their actual level of knowledge and expertise.
Beyond the experts apprehensions and uncertainties
regarding directly transferring results from cohort studies
developed in other countries for use in impact and benet
assessments of changes in air pollution ambient concentrations
due to activities, public policies and normative instruments, the
resources and time necessary to develop cohort studies make it
virtually impossible to conduct local studies in every country. In
this context, the SEJ is a potential alternative method for
estimating the increase in death risk and other health end
points attributable to long-term PM2.5 exposure in countries
where this information is not available, such as Chile.
The results from this study provide valuable insight and
experience into implementing a larger study to assess the true

Making estimates using such an unusual metric (the quotient

of a specic weeks mortality rate over the annual average
weekly mortality rate for dierent causes of deaths and for two
specic years) demanded additional eorts and dedication from
the experts. On the other hand, estimates for these variables
required the experts to have specic knowledge of annual
variations in weekly mortality rates for dierent causes of death.
While the necessary information to make these estimates is
publicly available, large amounts of time and data are required
in order to do so. Moreover, there is no reason to assume that
experts are familiar with how nonaccidental and cardiovascular
mortality rates vary weekly over a specic year in the MR, since
annual average mortality rates are usually used in epidemiology
studies and health impact assessments for air pollution
exposure, as these measures are less sensitive to particular
events that aect daily or weekly mortality rates.
Additionally, for seed variables S4S7, a group of experts
provided highly concentrated probability distributions. Similar
behavior is observed for the same seed variables in Tuomisto
et al.,45 who suggest that these dierences could possibly owe
to overcondence in a group of experts (revealed in the small
condence intervals compared to the rest of the panel).
Seed variables are helpful at determining whether experts can
express when they are knowledgeable and when they are just
guessing in the estimation of seed variables. It could be
expected that experts (especially in Group 2) would have
recognized the diculty embedded in this set of seed variables,
by delivering broader uncertainty intervals in their estimations.
In light of our results, we believe that experts comprehensive
understanding of seed variables (what are they used for, and
how to deliver estimations) is fundamental for experts to make
and communicate their estimations. This highlights the
importance of conducting a workshop with the entire expert
panel at the beginning of the study to ensure experts proper
understanding of calibration variables.
Finally, it is worth noting that the high complexity of the
seed variables, as well as the time, research and dedication
required to make good, accurate estimates, could undermine
experts enthusiasm and commitment to the studys main goal,
aecting the quality of their estimates for query variables.
Whatever the cause, low performance in estimates of these seed
variables explains not only low individual calibration scores, but
also the low calibration scores of combined distributions.
These results evidence the relevance of dening a proper set
of calibration variables in order to obtain a robust method for
calibrating experts, in which a larger number of seed variables is
considered and where such variables are clearly dened.
Mortality Estimates for Chile. Mortality estimates
delivered by experts in Group 1 were markedly similar, had
the largest uncertainty intervals, were signicantly dierent
from the rest of the panels estimates, and, in turn, diered from
previous studies. This outcome is not surprising, as experts C
and D declared that directly transferring results from other
countries was not valid. Therefore, it was expected that they
would venture to make their own estimates, instead of relying
heavily on results reported in previous cohort studies.
Conversely, experts from Group 2, who in general considered
the transference assumption to be valid, provided estimates that
were very similar to results from previous studies. This outcome
reveals that their estimates are, to varying degrees, anchored to
values reported by major international cohort studies.
A plausible explanation for the signicant dierences
observed between mortality estimates by groups 1 and 2 is
9725

dx.doi.org/10.1021/es500037k | Environ. Sci. Technol. 2014, 48, 97179727

Environmental Science & Technology

Article

pollution and mortality in six US cities. N. Engl. J. Med. 1993, 329

(24), 17531759.
(12) Pope, C. A., III; Thun, M. J.; Namboodiri, M. M.; Dockery, D.
W.; Evans, J. S.; Speizer, F. E.; Heath, C. W., Jr Particulate air pollution
as a predictor of mortality in a prospective study of US adults. Am. J.
Respir. Crit. Care Med. 1995, 151 (3), 669674.
(13) HEI. Reanalysis of the Harvard Six Cities Study and the American
Cancer Society Study of Particulate Air Pollution and Mortality; Health
Eects Institute, 2000.
(14) Krewski, D.; Burnett, R.; Goldberg, M.; Hoover, B. K.;
Siemiatycki, J.; Jerrett, M.; Abrahamowicz, M.; White, W. Overview of
the reanalysis of the Harvard six cities study and American Cancer
Society study of particulate air pollution and mortality. J. Toxicol.
Environ. Health, Part A 2003, 66 (1619), 15071552.
(15) Pope, C. A., III; Burnett, R. T.; Thurston, G. D.; Thun, M. J.;
Calle, E. E.; Krewski, D.; Godleski, J. J. Cardiovascular mortality and
long-term exposure to particulate air pollution epidemiological
evidence of general pathophysiological pathways of disease. Circulation
2004, 109 (1), 7177.
(16) Laden, F.; Schwartz, J.; Speizer, F. E.; Dockery, D. W. Reduction
in fine particulate air pollution and mortality. Am. J. Respir. Crit. Care
Med. 2006, 173 (6), 667672.
(17) Jerrett, M.; et al. Spatial analysis of air pollution and mortality in
Los Angeles. Epidemiology 2005, 16 (6), 727.
(18) Krewski, D.; Burnett, R.; Jerrett, M.; Pope, C. A.; Rainham, D.;
Calle, E.; Thurston, G.; Thun, M. Mortality and long-term exposure to
ambient air pollution: Ongoing analyses based on the American
Cancer Society cohort. J. Toxicol. Environ. Health, Part A 2005, 68
(1314), 10931109.
(19) Beelen, R.; et al. Long-term effects of traffic-related air pollution
on mortality in a Dutch cohort (NLCS-AIR study). Environ. Health
Perspect. 2008, 116 (2), 196202.
(20) Enstrom, J. E. Fine particulate air pollution and total mortality
among elderly Californians, 19732002. Inhalation Toxicol. 2005, 17
(14), 803816.
(21) Crouse, D. L.; et al. Risk of nonaccidental and cardiovascular
mortality in relation to long-term exposure to low concentrations of
fine particulate matter: A Canadian national-level cohort study.
Environ. Health Perspect. 2012, 120 (5), 708714.
(22) Puett, R. C., Yanosky, J.D.; Hart, J.E.; Paciorek, C.J.; Schwartz,
J.D.; H. H. Suh MacIntosh, Speizer, F.E.; , Laden, F.; , Chronic ne
and coarse particulate exposure, mortality, and coronary heart disease
in the Nurses Health Study. Environ. Health Perspect., 2009.
(23) Puett, R. C.; et al. Chronic particulate exposure, mortality, and
coronary heart disease in the nurses health study. Am. J. Epidemiol.
2008, 168 (10), 11611168.
(24) EPA. Integrated Science Assessment for Particulate Matter; U.S.
Environmental Protection Agency: Washington, DC, 2009.
(25) Cooke, R. Experts in Uncertainty: Opinion and Subjective
Probability in Science; Oxford University Press, 1991.
(26) Morgan, M. G. and Small, M. Uncertainty: A Guide to Dealing
with Uncertainty in Quantitative Risk and Policy Analysis; Cambridge
University Press, 1992.
(27) NUREG-1150 Severe Accident Risks: An Assessment for Five U.S.
Nuclear Power Plants; Nuclear Regulatory Commission, 1990.
(28) .Public attitudes toward nuclear power. In Nuclear Power in an
Age of Uncertainty, Chapter 8; Oce of Technology Assessment, 1984.
p 218219.
(29) Harrison, J.; Khursheed, A.; Phipps, A.; Goossens, L.; Kraan, B.;
Harper, F. Uncertainties in biokinetic parameters and dose coefficients
determined by expert judgement. Radiat. Prot. Dosim. 1998, 79 (14),
355358.
(30) Goossens, L.; Harper, F.; Kraan, B.; Metivier, H. Expert
judgement for a probabilistic accident consequence uncertainty
analysis. Radiat. Prot. Dosim. 2000, 90 (3), 295301.
(31) Cooke, R.; Goossens, L. Procedures guide for structured expert
judgement in accident consequence modelling. Radiat. Prot. Dosim.
2000, 90 (3), 303309.

change in health end points from exposure to PM2.5 in Chile. It

is worth noting the enormous potential of this methodological
tool for aiding regulatory and normative analyses in Chile,
extending beyond the eld of air pollution.

ASSOCIATED CONTENT

S Supporting Information
*

History of expert judgment, SEJ methodology, details of the

Classical Model, description of the seed variables and other
main results are presented in Supporting Information. This
material is available free of charge via the Internet at http://
pubs.acs.org.

AUTHOR INFORMATION

Corresponding Author

*Phone: +56 (2) 26618644; fax: 00-562-6618623; e-mail:

pamecisternaso@gmail.com.
Notes

The authors declare no competing nancial interest.

ACKNOWLEDGMENTS
This research was partially funded by Chiles National Science
and Technology Commission (Conicyt) through the National
Fund for Scientic and Technological Research (Fondecyt,
grant 1130864), and by the National Research Center for
Integrated Natural Disaster Management CONICYT/FONDAP/
15110017.

REFERENCES

(1) Ostro, B.; Sanchez, J. M.; Aranda, C.; Eskeland, G. S. Air

pollution and mortality: Results from a study of Santiago, Chile. J.
Exposure Anal. Environ. Epidemiol. 1996, 6 (1), 97.
(2) Pope, C. A., III; Burnett, R. T.; Thun, M. J.; Calle, E. E.; Krewski,
D.; Ito, K.; Thurston, G. D. Lung cancer, cardiopulmonary mortality,
and long-term exposure to fine particulate air pollution. J. Am. Med.
Assoc. 2002, 287 (9), 11321141.
(3) Pope, C. A., III; Dockery, D. Health effects of fine particulate air
pollution: Lines that connect. J. Air Waste Manage. Assoc. 2006, 56 (6),
709742.
(4) Slaughter, J. C.; Kim, E.; Sheppard, L.; Sullivan, J. H.; Larson, T.
V.; Claiborn, C. Association between particulate matter and emergency
room visits, hospital admissions and mortality in Spokane,
Washington. J. Exposure Anal. Environ. Epidemiol. 2004, 15 (2),
153159.
(5) Dominici, F.; Peng, R. D.; Bell, M. L.; Pham, L.; McDermott, A.;
Zeger, S. L.; Samet, J. M. Fine particulate air pollution and hospital
admission for cardiovascular and respiratory diseases. J. Am. Med.
Assoc. 2006, 295 (10), 11271134.
(6) Zanobetti, A.; Schwartz, J. Air pollution and emergency
admissions in Boston, MA. J. Epidemiol. Community. Health 2006, 60
(10), 890895.
(7) Laden, F.; Neas, L. M.; Dockery, D. W.; Schwartz, J. Association
of fine particulate matter from different sources with daily mortality in
six US cities. Environ. Health Perspect. 2000, 108 (10), 941.
(8) Tainio, M.; Tuomisto, J. T.; Hanninen, O.; Aarnio, P.; Koistinen,
K. J.; Jantunen, M. J.; Pekkanen, J. Health effects caused by primary
fine particulate matter (PM2.5) emitted from buses in the Helsinki
metropolitan area, Finland. Risk Anal. 2005, 25 (1), 151160.
(9) Ostro, B.; Broadwin, R.; Green, S.; Feng, W. Y.; Lipsett, M. Fine
particulate air pollution and mortality in nine California counties:
Results from CALFINE. Environ. Health Perspect. 2006, 114 (1), 29.
(10) Dockery, D. W.; Schwartz, J.; Spengler, J. D. Air pollution and
daily mortality: Associations with particulates and acid aerosols.
Environ. Res. 1992, 59 (2), 362373.
(11) Dockery, D. W.; Pope, C. A.; Xu, X.; Spengler, J. D.; Ware, J. H.;
Fay, M. E.; Ferris, B. G., Jr; Speizer, F. E. An association between air
9726

dx.doi.org/10.1021/es500037k | Environ. Sci. Technol. 2014, 48, 97179727

Environmental Science & Technology

Article

(52) Ilabaca, M.; Olaeta, I.; Campos, E.; Villaire, J.; Tellez-Rojo, M.
M.; Romieu, I. Association between levels of fine particulate and
emergency visits for pneumonia and other respiratory illnesses among
children in Santiago, Chile. J. Air Waste Manage. Assoc. 1999, 49 (9),
154163.
(53) Ostro, B. D.; Eskeland, G. S.; Sanchez, J. M.; Feyzioglu, T. Air
pollution and health effects: A study of medical visits among children
in Santiago, Chile. Environ. Health Perspect. 1999, 107 (1), 69.
(54) Roman, A. O.; Prieto, M. J.; Mancilla F, P.; Astudillo O, P.;
Acuna S, C.; Delgado B, I. Aumento del riesgo de consultas

cardiovasculares por contaminacion atmosferica por particulas:

Estudio
en la ciudad de Santiago. Rev. Chilena Cardio. 2009, 28 (2), 159164.
(55) Cooke, R. and Solomatine, D. EXCALIBR Integrated System for
Processing Expert Judgements version 3.0; Users Manual, Prepared under
Contract for Directorate-General XII, Delft University of Technology:
Delft, 1992.
(56) Cakmak, S.; Dales, R. E.; Vidal, C. B. Air pollution and mortality
in Chile: Susceptibility among the elderly. Environ. Health Perspect.
2007, 115 (4), 524.
(57) Clancy, L.; Goodman, P.; Sinclair, H.; Dockery, D. W. Effect of
air-pollution control on death rates in Dublin, Ireland: An intervention
study. Lancet 2002, 360 (9341), 121014.
(58) Katsouyanni, K.; et al. Short term effects of air pollution on
health: A European approach using epidemiologic time series data:
The APHEA protocol. J. Epidemiol. Commun. H. 1996, 50 (Suppl 1),
S12S18.
(59) Pope, C. A., III; Ezzati, M.; Dockery, D. W. Fine-particulate air
pollution and life expectancy in the United States. N. Engl. J. Med.
2009, 360 (4), 376386.
(60) Son, J.-Y.; Bell, M. L.; Lee, J.-T. Survival analysis of long-term
exposure to different sizes of airborne particulate matter and risk of
infant mortality using a birth cohort in Seoul, Korea. Environ. Health
Perspect. 2011, 119 (5), 725.
(61) WHO. Air Quality Guidelines: Global Update 2005: Particulate
Matter, Ozone, Nitrogen Dioxide and Sulfur Dioxide; World Health
Organization, 2005.

(32) Kraan, B.; Cooke, R. Processing expert judgements in accident

consequence modelling. Radiat. Prot. Dosim. 2000, 90 (3), 311315.
(33) McKay, M.; Meyer, M. Critique of and limitations on the use of
expert judgements in accident consequence uncertainty analysis.
Radiat. Prot. Dosim. 2000, 90 (3), 325330.
(34) Goossens, L.; Wakeford, R.; Little, M.; Muirhead, C.;
Hasemann, I.; Jones, J. Probabilistic accident consequence uncertainty
analysis of the late health effects module in the COSYMA package.
Radiat. Prot. Dosim. 2000, 90 (3), 359364.
(35) Erhardt, J.; Jones, J.; Goossens, L. Probabilistic accident
consequence uncertainty analysis of the whole program package
COSYMA. Radiat. Prot. Dosim. 2000, 90 (3), 365371.
(36) Cooke, R.; Goossens, L. Procedures Guide for Structured Expert
Judgment; European Commission, 2000.
(37) Van der Fels-Klerx, H.; Cooke, R. M.; Nauta, M. N.; Goossens,
L. H.; Havelaar, A. H. A structured expert judgment study for a model
of campylobacter transmission during broiler-chicken processing. Risk
Anal. 2005, 25 (1), 109124.
(38) Boone, I.; Van der Stede, Y.; Bollaerts, K.; Messens, W.; Vose,
D.; Daube, G.; Aerts, M.; Mintiens, K. Expert judgement in a risk
assessment model for Salmonella spp. in pork: The performance of
different weighting schemes. d. 2009, 92 (3), 224234.
(39) Aspinall, W. Structured elicitation of expert judgment for
probabilistic hazard and risk assessment in volcanic eruptions. J. Geol.
Soc. London 2006, 1530.
(40) Goossens, L.; Cooke, R.; Woudenberg, F.; Van der Torn, P.
Expert judgement and lethal toxicity of inhaled chemicals. J. Risk Res.
1998, 1 (2), 117133.
(41) Industrial Economics, I. Expanded Expert Judgment Assessment of
the Concentration-Response Relationship Between PM2.5 Exposure and
Mortality; U.S. Environmental Protection Agency, 2006.
(42) Roman, H. A.; Walker, K. D.; Walsh, T. L.; Conner, L.;
Richmond, H. M.; Hubbell, B. J.; Kinney, P. L. Expert judgment
assessment of the mortality impact of changes in ambient fine
particulate matter in the US. Environ. Sci. Technol. 2008, 42 (7), 2268
2274.
(43) Industrial Economics, I. An Expert Judgment Assessment of the
Concentration-Response Relationship Between PM2.5 Exposure and
Mortality; U.S. Environmental Protection Agency, 2004.
(44) National Research Council. Estimating the Public Health Benets
of Proposed Air Pollution Regulations; The National Academies Press:
Washington, DC, 2002.
(45) Tuomisto, J. T.; Wilson, A.; Evans, J. S.; Tainio, M. Uncertainty
in mortality response to airborne fine particulate matter: Combining
European air pollution experts. Reliab. Eng. Syst. Safe 2008, 93 (5),
732744.
(46) Cifuentes, L. A.; Vega, J.; Kopfer, K.; Lave, L. B. Effect of the
fine fraction of particulate matter versus the coarse mass and other
pollutants on daily mortality in Santiago, Chile. J. Air Waste Manage.
Assoc. 2000, 50 (8), 12871298.
(47) Sanhueza, P.; Vargas, C.; Mellado, P. Impact of air pollution by
fine particulate matter (PM10) on daily mortality in Temuco, Chile.
Rev. Med. Chile 2006, 134 (6), 754.
(48) Dales, R. E.; Cakmak, S.; Vidal, C. B. Air pollution and
hospitalization for headache in Chile. Am. J. Epidemiol. 2009, 170 (8),
10571066.
(49) Dales, R.; Cakmak, S.; Vidal, C. Air pollution and hospitalization
for venous thromboembolic disease in Chile. J. Thromb. Haemostasis
2010, 8 (4), 669674.
(50) Sanhueza, P. A.; Torreblanca, M. A.; Diaz-Robles, L. A.;
Schiappacasse, L. N.; Silva, M. P.; Astete, T. D. Particulate air pollution
and health effects for cardiovascular and respiratory causes in Temuco,
Chile: A wood-smoke-polluted urban area. J. Air Waste Manage. Assoc.
2009, 59 (12), 14811488.
(51) Castro, P.; Vera, J.; Cifuentes, L.; Wellenius, G.; Verdejo, H.;
Sepulveda, L.; Vukasovic, J. L.; Llevaneras, S. Polucion por material
particulado fino (PM 2.5) incrementa las hospitalizaciones por
insuficiencia cardiaca. Rev. Chilena Cardio. 2010, 29 (3), 306314.
9727

dx.doi.org/10.1021/es500037k | Environ. Sci. Technol. 2014, 48, 97179727