GPS workshop, Montpellier 11-13 Oct 2010 1 / 19

GPS validation: statistical analysis

according to ISO 5725-2

Hervé KERDILES, Krasimira GANISHEVA

GeoCAP, JRC Ispra
GPS workshop, Montpellier 11-13 Oct 2010 2 / 19

Outline
A. Detection of outliers
1.Visual examination
2.Cochran’s test
3.Grubbs’ test on single observations
4.Grubbs’ test on 2 outlying observations
5.Order of application of these tests
B. Checking the bias
C. Checking effect of operator, direction, parcel (ANOVA)
D. Calculating the R-limit
GPS workshop, Montpellier 11-13 Oct 2010 3 / 19

Visual examination by expert

• Cf ISO 5725-2:1994, par 7.2.5: responsibility of statistical expert
to discard data – need to report on discarded data

6
5
GPS workshop, Montpellier 11-13 Oct 2010 4 / 19

Cochran’s test
• Context: validation test with p sets (with n repetitions per set)
• Objective: detect set(s) with a too high (within-set) variance

• Let S1, S2, … Sp be the variances of the p sets

2

C= S max
C
• Compute C as p Outlier
∑S
2
i CV 1%
i =1

• Ex for p=9, n=4: Straggler

ƒ Critical value (CV) 1% = 0.481 CV 5%
ƒ Critical value (CV) 5% = 0.403 OK
0
GPS workshop, Montpellier 11-13 Oct 2010 5 / 19

Example of outliers detected by Cochran

Set of measurement (e.g. 4 repetitions, same GPS constellation, same operator)

outlier
outlier
GPS workshop, Montpellier 11-13 Oct 2010 6 / 19

Grubbs’ test on single observations

• Objective: detect value (min or max) too far from the mean
• Data: p set means (or n repetitions of a set)
• Order the p values: X1, X2, … Xp

• LetX and S be the mean and std deviation of the n values

X − X1
• Compute G1 & Gp as G1 = G
S Outlier
Xp −X CV 1%
Gp =
S Straggler
• Ex for p=4 repetitions:
CV 5%
ƒ Critical value (CV) 1% = 1.496
ƒ Critical value (CV) 5% = 1.481 OK
0
GPS workshop, Montpellier 11-13 Oct 2010 7 / 19

Example outlier detected by Grubbs

10750

10500
Outlier
10250
area (m2)

10000

9750
Ref area
9500

9250

9000
0 1 2 3 4 5 6 7 8 9 10

GeoXT, SI, p7 Set

GPS workshop, Montpellier 11-13 Oct 2010 8 / 19

Grubbs’ test for two outlying observations

• Objective: detect pairs (min or max) too far from the mean
• Data: p set means p−2

∑ i p −1, p
( X − X ) 2

• Compute G (with 2 highest values) as G= i =1

p
where Xp-1,p = mean without the 2 highest values ∑ i
( X − X
i =1
) 2

• idem with 2 lowest values

G
OK
CV 5%
• Ex for p=9 means: Straggler
ƒ Critical value (CV) 1% = 0.085
CV 1%
ƒ Critical value (CV) 5% = 0.149
Outlier
0
GPS workshop, Montpellier 11-13 Oct 2010 9 / 19

Application order of tests (1/2)

Cochran: find set with too high var

1. Set X is outlier 2. Set X is straggler

3. No outlier, no straggler

Grubbs on single Grubbs on single

observations: observations:
max(G1,Gp) max(G1,Gp)

1 outlier 0 outlier 1 outlier 0 outlier

Discard obs, test Discard obs, test

other extreme, other extreme
discard if outlier (discard if outlier)
Grubbs on set means
Discard set
GPS workshop, Montpellier 11-13 Oct 2010 10 / 19

Application order of tests (2/2)

Grubbs on set means
2. No set is outlier
1. One set is outlier
(highest or lowest mean)
Grubbs for 2 outlying means
Discard set
1. One pair of 2. No pair of
outlying means outlying means
Test other
extreme mean,
discard if outlier Discard 2 sets & test
other extreme pair of
means (discard if outlier)

End search for outliers

Do not discard more than 2/9 of data
GPS workshop, Montpellier 11-13 Oct 2010 11 / 19

Checking the bias (1/2)

• Reprod Std deviation calculated with mean, not ref area (RMSE)
• Assume mean = ref area
• If not the case, there is a bias -> need to be corrected (e.g
recalibrate, check for possible SW bug) or modeled (/)

• Let d = Mean of set means – reference area

• There is a bias if d is significantly ≠ 0

• Test 1 from ISO 5275 (cf. wikiCAP “testing the bias” page)
ƒ 95% confidence interval of d = [d – X, d + X]
ƒ If 0 inside [d – X, d + X], no bias, else bias at 95%
ƒ For X, see wikiCAP page or ISO 5275-4:1994 parag 4.7.2
GPS workshop, Montpellier 11-13 Oct 2010 12 / 19

Checking the bias (2/2)

• Let d = Mean of set means – reference area

• Test 2: Student test

ƒ Let M1, M2, … Mn be the means of set 1, 2,.. N
ƒ Let µ be the mean of M1, M2,…Mn
ƒ Let σ be the std deviation of M1, M2,… Mn

µ − Aref
ƒ If no bias (i.e. µ = Aref) then T= ≈ t( n−1)
σ
n
ƒ Look for critical value at 5%, 1% of t(n-1), 2 tails
GPS workshop, Montpellier 11-13 Oct 2010 13 / 19

Example of bias (Trimble ProXRS, field D, CY)

Receiver Trimble GPS Pathfinder Pro XRS+TDS RECON handheld_standalone, Field D : level
7400

Reference area
accepted data
outlier data
overall mean
True_value

7350
Measurement mean and range, sqm

Set 1 Set 9
Set 2
Set 7
7300

Set 6
Set 3

Set 8

Set 5
Set 4

7250

7200
Date of measurement
GPS workshop, Montpellier 11-13 Oct 2010 14 / 19

ANOVA
• Purpose: for each parcel check a possible effect of
ƒ operator: does A measure more (or less) than B?
-> look at mean measured area per operator
ƒ direction of measurement (CW/ACW): do we obtain larger
areas when measuring in clockwise (or anti CW) direction?
-> look at (signed) error on perimeter per direction
ƒ parcel: is error on perimeter larger for certain parcels?
-> look at abs value of |error on perimeter| per parcel

• Hope no effect or negligible effect, otherwise need to review

model of error (cannot aggregate results of all operators, or
cannot average results of all parcels…)
GPS workshop, Montpellier 11-13 Oct 2010 15 / 19

Repeatability & reproducibility std deviations

• Repeatability conditions: conditions where independent test results
are obtained with the same method on identical test items (parcels) in
the same lab by the same operator using the same equipment within
short intervals of time -> repetitions by same operator with same
GPS constellation for a given parcel = set
• Reproducibility conditions: conditions where test results are
obtained with the same method on identical test items in different labs
with different operators using different equipment -> all sets together
• Reprod Variance = Repeat Variance + between sets Variance
• See formulae on the poster made by A. SIMA; WikiCAP Link:
http://marswiki.jrc.ec.europa.eu/wikicap/index.php/GPS_equipment_-
_validation
GPS workshop, Montpellier 11-13 Oct 2010 16 / 19

The Reproducibility limit

• Definition: Maximum difference at 95% between 2 operators
A & B using the same tool / method

• If measurements from A, XA ~ N(µ,σ)

• If measurements from B, XB ~ N(µ,σ)
• and XA and XB are independent
• Then difference X A − X B ≈ N (0, 2σ )

•So in 95% of the cases, X A − X B ≤ 1.96 2σ = 2.8σ

=> Reproducibility limit = 2.8 Reproducibility Std deviation

GPS workshop, Montpellier 11-13 Oct 2010 17 / 19

Conclusion
• General concept of R-limit from chemistry ISO norm 5725:1994
adapted to GPS measurement of areas using the “buffer x
perimeter” model for estimating the error on area
• Statistical analysis of data not complex, but not that simple
either
• Many mistakes made by MS up to now
• Please consult JRC in case of doubt
• We would like to publish your results (once checked) as they
may help other MS – Please send us your protocol, results &
data (cf WikiCAP page on validated GPS)

Overview of GeoXT validation results in 5 MS ->

GPS workshop, Montpellier 11-13 Oct 2010 18 / 19

Results from tests with GeoXT (std alone or EGNOS)

1,20 JRC_PL std alone
SI Egnos
1,00 LV std alone
0,80 FR std alone
R-limit (m)

FR egnos
0,60 Trimble UK egnos

0,40

0,20

0,00
0 10000 20000 30000 40000
Parcel area (m2)

37 Parcels in 5 MS (PL, SI, LV, FR, UK): 0.06 ha to 3.9 ha

 GPS workshop, Montpellier 11-13 Oct 2010 19 / 19

Results from GeoXT tests: average R-limits

1,20
JRC_PL std alone
No mask mask No distinction
SI Egnos
1,00 between
LV std alone
FR std alone
- 2005 and 2008
0,80
R-limit (m)

FR egnos
GeoXT series,
0,60 Trimble UK egnos - continuous or
vertex method
0,40

0,20

0,00
0 5000 10000 15000 20000
Parcel area (m2)

Regional differential correction

JRC_PL std alone (2005):0.60m SI Egnos (2005): 0,54m
LV Latpos (2005): 0.39m
LV std alone (2005): 0.51m FR Egnos (2008):0.35m
FI VRS (2008): 0.39m
FR std alone (2008): 0.39m UK Egnos (2008):0.44m
Navcert VRS (2008): 0.17m
 Annexes: test parcels
GPS workshop, Montpellier 11-13 Oct 2010 20 / 19

JRC-PL test parcels

Parcel A buffer = 1.09 m

Average buffer of the 5
other test parcels = 0.50 m
GPS workshop, Montpellier 11-13 Oct 2010 21 / 19

LV test parcels
Buffer = 0.72m

Buffer = 1.03m

Average buffer of the 4 other test parcels = 0.33m

 GPS workshop, Montpellier 11-13 Oct 2010 22 / 19

Trimble UK-En test parcels

Average buffer of the 4 other test parcels = 0.33m

Buffer parcel F = 0.76m

Buffer parcel E = 0.57m

GPS workshop, Montpellier 11-13 Oct 2010 23 / 19

SI test parcels
Parcel 2
Buffer = 0.84m

Parcel 7
Buffer = 0.83m

Average buffer of other 5 parcels: 0.42m