Documente Academic
Documente Profesional
Documente Cultură
Systematic Errors
arXiv:1502.03249v1 [astro-ph.IM] 11 Feb 2015
Gerrit Spengler1
Oscar Klein Centre for Cosmoparticle Physics, Physics Department, Stockholm University,
Albanova University Centre, SE-10691 Stockholm
Abstract
1. Introduction
1 gerrit.spengler@fysik.su.se
NON + NOFF
NOFF = NOFF () = . (2)
1+
2
and [4]) and extensive air shower arrays (see e.g. [5]).
The High Energy Stereoscopic System (H.E.S.S.) is an array of IACTs that op-
erates in the Namibian Khomas Highland since 2003. Observations of the Crab
nebula with H.E.S.S. result in the detection of approximately 100 background
events that pass standard Hillas -ray event selection criteria per 30 min obser-
vation time for a normalization factor of = 0.2 (see [6] for details). Similar
background event rates hold for analyses of data from observations of point like
-ray sources with other current generation IACT arrays such as MAGIC [7]
and VERITAS [8]. According to Eq. 3, the relative error on the normalization
factor must be much smaller than / 25% when SLiMa is used to calculate
the significance of a -ray signal with NOFF 100 and = 0.2. More precisely,
it will be shown later that in this case / must be known to about 3% to
justify the application of SLiMa .
For the planned Cherenkov Telescope Array (CTA, see [9]), the increase in the
number of telescopes will (compared to current generation IACTs) lead to a
factor of 10 larger effective area. An increase of the background event rate
for CTA of a similar factor of 10 (again compared to current generation IACT
arrays) is expected from the enlarged effective area (see [10]). Consequently,
approximately 103 background events per 30 min observation time are expected
in a typical point source analysis of CTA data with = 0.2. In this case, it is
estimated with Eq. 3 that the relative error on the normalization factor must be
much smaller than 8%. Otherwise, an error on the normalization factor must be
considered. Again more precisely, it will be shown later that the relative error
on the normalization factor must be known to about one order of magnitude
better than 8% for the application of SLiMa in this situation.
In analyses of extended -ray sources (e.g. Supernova Remnants, [11]) or galac-
tic dark matter searches (e.g. [12]), the increased size of the signal region leads
easily to background event rates which are an order of magnitude larger than for
point source analyses. It is obvious that in those cases, the normalization factor
must be known with an even better precision than in point source analyses.
It is arguable whether the normalization factor is in general known with the
3
precision that is required for the application of SLiMa . This holds in particular
for analysis of data from the planned CTA experiment.
This paper extends the method for the calculation of the statistical significance
of a -ray signal first proposed in [2] to include an error on the normalization
factor. In addition to other authors, who discussed the same problem (e.g. [13],
[14]), a simple expression for the calculation of the significance of a measured -
ray signal is derived. Moreover, the criterion given by Eq. 3 to decide whether a
given error on the normalization factor must be considered in the calculation of
the statistical significance of a -ray signal is tested in Monte Carlo simulations.
The structure of the paper is as follows: The effect of the neglection of a sys-
tematic error on the normalization factor on SLiMa is quantitatively discussed
in section 2. A modified expression for the significance calculation, which con-
siders an error on the normalization factor, is derived in section 3.
Section 4 discusses more specifically the influence of systematic errors on the
calculation of statistical significances in VHE -ray astronomy with IACTs like
H.E.S.S. or CTA.
4
400 No error on
350
8% error on
300
15% error on
Entries
250
200
150
100
50
0
-5 -4 -3 -2 -1 0 1 2 3 4 5
Significance
Figure 1: Distribution of SLiMa (Eq. 1) when NON = Pois(b) and NOFF = Pois(b) are
independently Poisson distributed with b = 500 events. The normalization factor, , is
distributed like a Gaussian with mean 0.2 and relative width according to the legend. The
width of the Gaussian fit to the significance distribution is compatible with being one if the
error on the normalization factor is vanishing. However, the width of the Gaussian fit to the
significance distribution increases with the relative error on the normalization factor.
5
Li & Ma
Width of Significance Distribution
Modified
10
10-1
1 10
Figure 2: Fitted Gaussian width of the distributions of the significance as calculated with
SLiMa (Eq. 1, red points) and SModified (Eq. 12, black points) as a function of (see Eq. 4).
The parameters NOFF , and / are randomly distributed within NOFF [100, 10000],
[0.1, 2] and / [0, 15%].
here and in the following restricted to small relative errors on the normalization
factor ( / . 15%). For large relative errors on the normalization factor, the
distribution cannot in general be assumed to be Gaussian, e.g. because the
normalization factor must be larger than zero for physical reasons. However,
the assumption of small relative errors on the normalization factor is reasonable
because large relative errors can easily be identified and corrected for in analy-
ses.
The mean of all fitted significance distributions that are shown in Fig. 1 is
compatible with zero. However, the width of the Gaussian fit to the significance
distribution is only compatible with being one when the error on the normal-
ization factor vanishes. For non-zero relative errors on the normalization factor,
the width of the Gaussian fit to the significance distribution increases with the
relative error on the normalization factor. In other words, the absolute value of
the significance is overestimated by SLiMa when the relative error on the nor-
6
malization factor is not vanishing.
The examples in Fig. 1 hold only for special values for the mean number of
background events and the normalization factor. In general, the criterion for
the applicability of SLiMa given in Eq. 3 translates into the condition
s
NOFF
= 1. (4)
( + 1)
Figure 2 shows the width of the Gaussian fit to the distribution of the signif-
icance as a function of . Here, the parameter is calculated for a random
selection of , and NOFF . The simulated number of background events,
NOFF , is Poisson distributed with uniformly distributed means in between 100
and 10000. The normalization factor, , is Gaussian distributed with uniformly
distributed means in between 0.1 and 2 and with uniformly distributed relative
widths in between / = 0 and / = 15%. Figure 2 shows that the width
of the Gaussian fit to the distribution of SLiMa is indeed larger than one if Eq.
4 does not hold. More specifically, Fig. 2 shows that the width of the Gaussian
fit to the distribution of SLiMa becomes larger than one if & 0.1.
SLiMa is, under the assumption of a vanishing error on the normalization factor,
constructed in [2] to be standard normal distributed when the null hypothesis
is true, i.e. when no signal events are present. In turn, SLiMa is not applica-
ble when the systematic error on the normalization factor causes a distribution
of SLiMa which is not compatible with being a standard normal distribution
although the null hypothesis is true. This means that SLiMa is applicable if
. 0.1 (5)
is fulfilled.
p
Equation 5 translates into the condition / . 0.1 ( + 1)/( NOFF ) for the
relative error on the normalization. This condition is more precise than Eq.
3. Also, this condition was already used in the introduction when the maximal
relative error on the normalization factor that allows the application of SLiMa in
typical point source analyses with current IACTs and the planned CTA detector
was estimated.
7
For cases where a systematic error on the normalization factor cannot be ne-
glected, a modified expression for the calculation of the statistical significance
of a -ray signal is derived in the next section.
Again, this likelihood function is in this work assumed to describe the case of
small relative errors on the normalization factor, i.e. / . 15%, to avoid e.g.
negative . The general likelihood function given by Eq. 6 has already been
investigated in the literature (see [13], [15], [14]). However, no simple result for
the statistical significance of a measured signal under the assumption of Eq. 6
was derived before. Also, no simple criterion to decide whether Eq. 6 should be
applied was stated before. This is done in the following.
The likelihood-ratio test statistic is used with the likelihood function given by
Eq. 6 to compare the null hypothesis, i.e. that no signal is measured (s = 0),
with the alternative hypothesis (s 6= 0). The likelihood for the null hypothesis
is
L0 = Pois(NON , b) Pois(NOFF , b) Gaus( , , ) (7)
and the two nuisance parameters b (i.e. the mean number of background events)
and (i.e. the Gaussian distributed normalization factor with mean ) are
eliminated with the profile likelihood method (see e.g. [16]). The profile likeli-
hood condition L0 /b = 0 leads to b = NOFF ( ) where NOFF ( ) is defined
in Eq. 2. The second profile likelihood condition, L0 / = 0, leads to the
cubic expression
8
Equation 8 can have up to three real solutions. Since, for physical reasons,
2 NON 0, it follows that (0) 0. Together with lim ( ) = , it
follows that there is always at least one real and positive solution to Eq. 8. For
example, if = 0, Eq. 8 has three solutions at = 1, = 0 and = .
Since L0 is not defined for = 1 and L0 = 0 for = 0, the solution =
maximizes the likelihood for the null hypothesis.
When ( + 1)2 > 32 NOFF , the cubic equation ( ) = 0 has in general a local
maximum at and a local minimum at + where
1 p
= 1 ( + 1)2 32 NOFF . (9)
3
If Eq. 8 does not have local extrema or sign((+ )( )) = +1, there is only
one real solution to Eq. 8 which can be found numerically in the interval (0, ).
The parameters s = NON b, b = NOFF and = maximize the likelihood
function (Eq. 6) and lead to the likelihood
1
L1 = Pois(NON , NON ) Pois(NOFF , NOFF ) (11)
2
for the alternative hypothesis. The alternative hypothesis has one more param-
eter (s) than the null hypothesis. Following Wilks theorem [17], it is assumed
that the likelihood ratio test statistic TS = 2 ln(L0 /L1 ) is approximately dis-
tributed like a 2 -distribution with one degree of freedom. The validity of this
assumption is justified with Monte Carlo simulations. The evaluation of the
9
likelihood ratio test statistic TS = 2 ln(L0 /L1 ) gives the modified significance
SModified(NON , NOFF , , ) = sign(NON NOFF ) TS
s
2
= sign(NON NOFF ) 2
SLiMa (NON , NOFF , ) + .
(12)
Note that SModified SLiMa when , i.e. when 0. This means that
SModified reduces to SLiMa when systematic errors are negligible.
Equation 12 is tested in a parameter range (NOFF [100, 10000], [0.1, 2]
and / [0, 15%]) which is in practice relevant for analyses of data from
IACTs. The result is shown in Fig. 2. The distribution of SModified is compat-
ible with being standard normal if NON = Pois(b) and NOFF = Pois(b) are
independently Poisson distributed with fixed b for all tested combinations of ,
NOFF and . In particular, the width of the Gaussian fit to the distribution
of SModified shown in Fig. 2 as a function of is compatible with being one.
It is concluded that SModified is standard normal distributed when the null hy-
pothesis is true and systematic errors are too large for the application SLiMa ,
i.e. if & 0.1. For negligible systematic errors, SModified reduces to SLiMa . The
range of possible application for SModified is thus larger than for SLiMa .
10
10
6
Li & Ma
5
0
102 103 104
Background Events per Run
can e.g. be atmospheric differences, bright stars, varying night sky brightness
or electronic problems. An order of magnitude for the deviation of the event
acceptance from rotational symmetry around the observation position in an ob-
servation with H.E.S.S. is estimated in [1] to be 3%. Figure 3 shows SModified for
an error of 3% on the normalization factor as a function of the number of back-
ground events per observation run when the number of signal events is chosen
such that SLiMa = 5. It is evident that in a typical H.E.S.S. point source analy-
sis with approximately 100 background events per observation run (see [6]), the
influence of expected deviations from rotational symmetry on the calculation of
the signal significance is almost negligible. However, it is also obvious that the
influence of systematic errors becomes larger when the number of background
events per observation run increases. This situation can easily occur in analyses
of data from extended signal regions where a spatially larger background region
is used than in point source analyses. As discussed in the introduction, approx-
imately 103 background events are expected in a typical point source analysis
11
of data acquired with the planned CTA experiment ([9], [10]). Figure 3 shows
clearly that this increase of the expected number of background events per ob-
servation run can decrease the source detection potential of CTA when analyses
are performed under conditions that are otherwise similar to e.g. H.E.S.S. data
analyses.
Typically, multiple (i = 1 . . . K) observations of a signal region are combined in
an IACT analysis such that the exposure ratio i is the same for all observations,
i.e. i = . However, deviations between the exposure ratio and the normal-
ization factor from observation run to observation run lead to a distribution of
normalization factors with mean . Again, the distribution of normalization
factors is assumed to be Gaussian with width . For the combined dataset,
the resulting error on the mean normalization factor is / K. The total num-
P
ber of signal (NON = i NON,i ) and the total number of background events
P
(NOFF = i NOFF,i ) are used together with the exposure ratio as normaliza-
tion factor to calculate the significance of the total -ray event signal. Consider
P
the mean number of background events, hNOFF i = 1/K i NOFF,i . The crite-
rion given by Eq. 5 for the application of SLiMa to data from one observation
becomes s s
NOFF hNOFF i
Combined = = . 0.1 (13)
K ( + 1) ( + 1)
when data from multiple observations are combined. Equation 13 means that
given a normalization factor and an error on the normalization factor, the mean
number of background events per observation is relevant to decide whether
SLiMa is applicable or not when multiple observations are combined. If Eq. 13
is not fulfilled, Eq. 12 can be used to calculate the statistical significance of the
combined signal as SModified(NON , NOFF , , / K).
The argumentation given above doesnt cover an important exception that con-
cerns the case where an analysis is set up such that the exposure ratios are not
equal for data from differing observations. This situation can in general only be
treated with a likelihood function that is binned in the exposure ratios (see [13])
or even binned in more parameters that characterize the observation conditions
12
(see [14]).
The considerations in this section show that it is, in particular for the planned
CTA experiment, important to estimate the typical systematic error on the nor-
malization factor for an observation run in an IACT data analysis. Depending
on the mean number of background events per observation run, the exposure
ratio and the estimated error on the exposure ratio, the usage of SModified is
preferred to the usage of SLiMa for the calculation of the statistical significance
of a -ray signal.
5. Conclusion
6. Acknowledgements
The author acknowledges valuable discussions with Jan Conrad. The re-
search of GS is supported by a grant of the Swedish Research Council (PI: Jan
Conrad).
13
References
References
[2] T.-P. Li and Y.-Q. Ma, The Astrophysical Journal 272 (1983) 317324.
[8] V.A. Acciari et al., The Astrophysical Journal Letters 693 (2009) L104.
14