DRAFT

(please send comments to hdj@dfm.dtu.dk)

Practical guide for the estimation of uncertainty in testing

Hans Dalsgaard Jensen Danish Fundamental Metrology B 307, Matematiktorvet, DK-2800 Kgs. Lyngby and George Teunisse Nederlands Meetinstitut Schoemakerstraat 97 NL-2628 VK Delft

Abstract: Measurement uncertainty plays a fundamental role in the documentation of traceability and the reliability of measurement results. Physical testing often relies on quantitative test results, and such are to be regarded as any other quantitative measurement, and thus requires a reliable uncertainty statement for conclusions to be drawn. However, a tradition and routine in calculating measurement uncertainties for test results is only in the beginning of its development. This guide seeks to develop and present some tools for the physical testing community, showing how to approach the problem of assigning uncertainty using the philosophy and methods of the ISO Guide to the Expression of Uncertainty in Measurement.

Danish Fundamental Metrology B 307 Matematiktorvet DK-2800 Kgs. Lyngby, DENMARK

DFM- 2004-R78 6406 HDJ 2004-11-30

Phone: +45 4593 1144 Fax: +45 4593 1137 Web: www.dfm.dtu.dk

1 Table of contents 1 2 3 4 5 Table of contents ................................................................................................................ 2 Introduction ........................................................................................................................ 2 Connection with other Guides............................................................................................ 2 Considerations for evaluation of uncertainty in testing ..................................................... 3 4.1 Why uncertainty in the first place? ............................................................................ 4 Methods for uncertainty estimation.................................................................................... 4 5.1 General - GUM........................................................................................................... 5 5.1.1 Hints on establishing model functions ............................................................... 8 5.2 Generic models........................................................................................................... 9 5.3 Repeatability - reproducibility ................................................................................. 14 5.3.1 Basic assumptions and use of nested designs................................................... 15 5.3.2 Intercomparison data ........................................................................................ 16 5.4 Other basic tools....................................................................................................... 17 5.4.1 Translation of instrument specifications....................................................... 17 5.4.2 Calculation, use and uncertainty of calibration curves .................................... 18 5.4.3 Drift .................................................................................................................. 21 5.4.4 Handling ill-defined measurement conditions ................................................. 21 Examples from electrical testing ...................................................................................... 22 6.1 Measurement of temperature using thermocouples ................................................. 22 6.2 Heating in Black Test Corner................................................................................... 24 6.3 Glow wire test .......................................................................................................... 25 6.4 EMC - Radiated Immunity Test (IEC 61000-4-3) ................................................... 26 Appendix .......................................................................................................................... 28 7.1 Table of distribution functions often used for type-B evaluation: ........................... 32 References ........................................................................................................................ 33

7 8

2 Introduction The widespread use of accreditation in testing as a means to ensure the proper level of process and quality control of test results has lead to an increased focus on the estimation of uncertainty of quantitative test results. The introduction of ISO 17025 as the laboratory standard has further increased the focus on estimation of uncertainty in quantitative testing. A main problem is the distinction between measurand and characteristic, where the latter is a property determined by a test. However, for quantitative testing, where a quantitative characteristic is determined, there is no practical distinction to that of a measurement quantity. Another problem is the fact that due to ill defined test set up or test environment in quite a number of standards, the result of a test can be strongly influenced by these conditions. The evaluation of all the influences on the measurement quantity could be a very complex and time-consuming operation. This guide aims to supply solutions to some of these problems. 3 Connection with other Guides There exist several other guides for the evaluation of uncertainty in testing. Accreditation bodies (e.g. A2LA, UKAS), regional cooperation organisations (e.g. EA) and laboratory associations (Eurolab) have all published documents concerning this issue, see section 8 References. Some specific information on classical concepts in testing, such as repeatability
6406 HDJ 2004 11 30

3 and reproducibility, are described in detail in ISO 5725 and brought into the GUM framework in ISO 21748. Some of the documents fall in the category of a more or less policy documents, and do not provide specific guidance for the combination and calculation of uncertainty. Other documents provide examples of calculation of uncertainty, but with very specific examples without overall guidance on how to handle specific types of uncertainty contributions. In the present document we will try to bridge this gap and focus on the method of assigning uncertainty, generalize some of the specific uncertainty contributions, open a toolbox of generic measurement methods and discuss how to assign uncertainties for the results from use of these methods. 4 Considerations for evaluation of uncertainty in testing Paper standards (norms) from the beginning of their existence have served as a means to be able to produce or reproduce a property by the extensive description of methods for manufacturing or testing of an object. This method of describing in itself causes the first sources of uncertainty. Differences in interpretation could already lead to different outcome. Therefore it is necessary that there always is some platform where people can exchange their information in order to diminish the possible influence of this source. It even appears (which is often the case in e.g. EMC test methods) that the testing conditions are ill defined. The increasing accuracy and measurement techniques have lead to a growing awareness of the imperfections in many of the described methods in standards, especially when the measurand is presented as an absolute measurement value. However due to the fact that most of the standards (norms) for testing often have been applied for many years it will lead to a heavy oppositions to modify the described methods and especially in the case that these standards do not contain any margins to the measurand in terms of uncertainty. Would an inventory been made on the uncertainty contributions and their values in these standards then there could be made a comparison between such a method and any new method to be introduced. Like mentioned this is not often the case so there will be no strong arguments to change a method to a more sophisticated method even when this appears to be a far more reproducible method. The only elegant way to alter this defensive attitude to a more constructive attitude is to clearly show the benefits of the knowledge of the uncertainties in the measurements by introducing in the standards a dependency of the values of the testing limits from the magnitude of these uncertainties. Of course when starting such an approach it will need an inventory on the measurement uncertainties in all existing (applicable) standards. Though this inventory could be assumed a quite time consuming and extensive job it does not at the same time mean that it is neither impossible nor unprofitable. The following chapters will show that when applying a systematic method in which a distinction is made between the contributions coming from the instrumentation on the one hand and from the test set up on the other hand.
6406 HDJ 2004 11 30

4 It will also be shown that in a number of cases it is not always necessary that the testing lab produce the values of all the contributors to the uncertainty as long as the normalization committees use the same methods of evaluation when integrating test methods in the standards they produce. In the next chapter first of all the use of a uniform method for the estimation of uncertainty will be described, which is already common practice for accredited calibration laboratories. Extending this method for testing set ups is possible when introducing some additional tools. 4.1 Why uncertainty in the first place? The argument has been heard, that uncertainty in testing is not worth the effort, because the test is performed within the limits of the influence factors as specified in the standard, and the uncertainty of the measurement results themselves are small in comparison with the overall test result, hence there is no reason to evaluate or report measurement uncertainty. This of course does not make sense; compliance with as well the conditions of the test as the result of the test itself, does require evaluation of uncertainty. But there are more important arguments for performing a thorough investigation of the uncertainty of measurements, although the task can be time-consuming. These arguments are also stressed by the accreditation organisations: Reporting a qualified measurement uncertainty statement assists in issues such as risk control and the credibility of test results, and can represent a direct competitive advantage by adding value and meaning to the result. The knowledge of quantitative effects of single quantities on the test result improves the reliability of the test procedure. Corrective measures may be implemented more efficiently and hence become more cost-effective. The evaluation of measurement uncertainty enables one to optimise the test procedures and calibration costs may be reduced if it is found that particular influence quantities do not contribute substantially to the uncertainty. When evaluating the result of a test and stating compliance with a specification, it is necessary with information on the uncertainty associated with results. 5 Methods for uncertainty estimation Quantitative measurement results without a stated uncertainty are basically meaningless. However, we must realize that in many, maybe even most, cases in our daily lives this is how results are reported. Often there is an implicit assumption on the accuracy of the number given: When taking of the age of a person, we implicitly assume an uncertainty of less than one year, when reading a reference value in a data book we assume the accuracy is of the order of the last significant digit. When we buy a measurement instrument, we assume that most of the digits shown have meaning; otherwise the manufacturer would probably have chosen a cheaper display. However, for scientifically based measurements or for measurements in which we wish to put just the slightest amount of trust, these implicit assumptions simply are not enough. We need some quantitative indication of the quality of the result; otherwise we have no means of comparing results among themselves, with other measurement results, with reference values or limits. And most importantly, we must be very explicit on the assumptions that are made in the process, because assumptions will still be necessary.

6406 HDJ

2004 11 30

5 According to the newest version of the Vocabulary of Basic and General Terms in Metrology (VIM, 3rd ed), measurement uncertainty is defined as a parameter that characterizes the dispersion of the quantity values that are being attributed to a measurand, based on the information used. Hence, the purpose of estimation of uncertainty is to quantify this dispersion, and the remainder of this paper is devoted to the discussion of the methods used for this. It is the expressed policy of the accreditation bodies as well as of the national metrology institutes, that measurement uncertainty shall be estimated as a realistic, scientifically based quantity, and the method by which to estimate uncertainty should be universal (applicable to all kinds of measurements), consistent (independent of the grouping and decomposition into subcomponents) and transferable (the results of one uncertainty calculation can be directly applied in a second calculation). The method universally agreed upon is described in The ISO Guide on the Expression of Uncertainty in Measurement, or short, GUM. 5.1 General - GUM The GUM, is the Master document used on the calculation and reporting of uncertainty for accredited laboratories. The GUM employs the BIPM Method developed by BIPM in the late 1980s, which takes the basic starting point of handling all measurement quantities, including their factors of influence, as stochastic variables. A stochastic variable describes a quantity as having many possible different outcomes, with a given probability distribution.
Value Fig. 1 A stochastic variable as a quantity with a probability distribution Probability

The value of an input quantity is the estimate of the expectation value of the distribution function, and the associated standard uncertainty of the value of the quantity is the square-root of the estimate of the variance of the distribution function. The methods to ascribe the distribution function of the input quantities may be performed either through statistical means (called type-A evaluation) or by other means (called type-B evaluation), e.g. by assumption of a probability distribution. For a primer on how to evaluate and assign probability distribution functions, see section 7. The measured quantity or the output quantity is described in terms of a model function in terms of the input quantities. Finally, the distribution function of the output quantity/-ies is found using the calculation rules for the propagation of variances. The GUM method is described in various details with examples in many other documents, so a thorough guide will not be given here. For completion, the method may be summarised to comprise of the following steps: 1. Specification of the measurand. Consider the nature of the quantity intended as the result of the measurement. Is it a direct value, is it a deviation from some reference, e.g. the nominal value, is it a dispersion from an average (or is it the average itself) or is it some ensemble property, e.g. the average temperature of a selection of measurement points. The specification of the measurand is naturally critical in any kind of assignment of uncertainty. Often the focus is only on uncertainty components, and are collected without regard to their comparability and how the value of the component combines with the result, thus, the measurand becomes implicitly defined; one of the most dangerous starting points for uncertainty evaluation.
2004 11 30

6406 HDJ

6 2. Input quantities. Identify all sources that have a direct relation to the end result, e.g. instrument readings, applied correction values, reference data, and any source that may affect the result of the measurement indirectly, i.e. influence quantities such as environmental conditions, operator variability, Model function. Set up a model function, which relates the final result (the output quantity), the value of the measurand, to the sources in (2) (the input quantities). The model function may reflect the physical measurement process or may be a purely empirical relation. The starting point for any model function should naturally reflect the way in which the measurement result is derived, even if it consists of a direct reading of an instrument. The model function must then be expanded to take into account all the input quantities identified in (2). For tips on taking the indirect quantities into account, see below. Uncertainty distributions. For each of the input quantities, an expectation value and the variance must be retrieved, either from a statistical treatment or from an assumed distribution function of the input quantity. This part of the process sometimes require just as much thought as setting up the model function itself. Again the documents mentioned above and listed in the appendix provides many examples of derivations for different types of influence quantities. This subject is treated more extensively below. Consider possible correlations between the input quantities, e.g. when using differences between readings. Consider whether two or more of the input quantities are related to each other, even via a third parameter, so that a change in one would be observed at the same time as a change in the other. For example, two quantities may depend on temperature. If this underlying temperature fluctuates, the values of both parameters fluctuate, in the same direction or in the opposite; there is a relation (a correlation) between them. When this interdependence is found or expected to be significant, the degree of correlation must be taken into account when calculating measurement uncertainty. To the extent possible and practical, such a third parameter should be put into the model function as an independent input quantity, and the influence it has on the two parameters in question be described by the model function. The important aspect is the conversion of any uncertainty statement that may be the initial point of discussion, to that of a standard uncertainty in terms of the square root of the estimate of the variance of the distribution function assigned to the quantity. For different types of uncertainty statements, e.g. the very common rectangular distribution a, calculate the standard uncertainties by applying the factor to the parameter appropriate for the distribution function assigned, e.g. for the rectangular distribution a 1 3 , and in the case of statistical treatment based on averages, be sure to consider whether the quantity refers to one observation or an average value of several observations. See section 7.1 for a list of distribution functions, their parameters and conversion to standard uncertainties.

3.

4.

5.

Propagation of uncertainty. Apply the rule of propagation of uncertainty, i.e. calculate the sensitivity factors for the model function for each input quantity and multiply by the standard uncertainty of that quantity, and include the terms due to correlation. It is recommended to use some software calculation tool, to avoid the tedious, error prone calculations when the model function is more than a simple sum/difference or product/quotient.

6406 HDJ

2004 11 30

7 6.
Coverage probability. Due to the nature of the calculation of uncertainty based on standard uncertainties, the interval covered by the result plus and minus the standard uncertainty, will only cover 68% of the expected outcomes, if all the assumptions hold. It is therefore common practice to quote an expanded uncertainty, U, in that the standard uncertainty is multiplied by a coverage factor to achieve either 95 % or 99 % coverage probability. For an output quantity that follows a Normal distribution, a coverage factor of k = 2 is used to achieve approximately 95% coverage probability. If the output quantity is dominated by, say a rectangular distributed input quantity, the appropriate coverage factor is k = 1,65.

In cases where the confidence in the evaluation of the individual uncertainties is small, it is necessary to take into account an appropriate number of degrees of freedom. When the combined uncertainty has a small number of degrees of freedom (less than 10), the factor k = 2 above is not sufficient to achieve a 95 % coverage probability. The relevant coverage factor may be found from a table of the two-tailed Students t-distribution with the given number of degrees of freedom. 7.
Reporting the result. A measurement result should be quoted: x U and provide information on the coverage probability and the coverage factor used.

Two simple cases are mentioned here, which may be applied in maybe 90 % of the cases relevant for the intended audience: A sum/difference of contributions and a product/quotient of contributions.
Sum and difference

Given a measurement of a quantity y found as a sum or differences of other quantities, say a direct reading of a value plus a series of corrections, all stated in the same unit as the quantity being measured (an important point: you cannot add apples and pears!) The model function then becomes a simple sum: y = x1 + x2 + x3 + ... = xi
i

The corrections applied to the direct reading could be a known deviation, an estimated drift or the dispersion determined by experience of the type of instrument used. Again, it is important to stress, that both a value and an uncertainty must be assigned to each of the input quantities xi (even if the appropriate value is 0, thus indicating that no numerical correction is performed). If the input quantities are uncorrelated, the combined standard uncertainty for the output quantity becomes
u ( y ) = u ( x1 ) 2 + u ( x2 ) 2 + u ( x3 ) 2 + ... =
Product and quotient

u( x )
i i

A model function in the form of products and quotients may be handled in a similar way. If the measurand may be modelled in the form y = x1 x2 x3 + ... = xi
i

where the factors are e.g. corrections factors to the main input quantity, say x1. Such correction factors will then typically have a value of 1 (hence not changing the numerical value of the result) and their uncertainty could be in units of % or or similar. For such a
6406 HDJ 2004 11 30

8 product (or quotient) model, it is convenient to work in relative uncertainties w(x) = u(x) / x. If the input quantities are uncorrelated, the combined standard uncertainty for the output quantity becomes w( y ) = w( x1 ) 2 + w( x2 ) 2 + w( x3 ) 2 + ... = and u(y) = w(y) y. In an very large number of cases, and especially in the cases where more empirically based model functions are used, the sum/difference or the product/quotient, or a combination of the two, is all that is needed. This is due to the consistency of the method; it does not depend on any specific way of grouping the uncertainties, and may be evaluated in an arbitrary number of steps. For example, given the expression
y=

w( x )
i i

(x1 + x2 x3 ) + x4
x5 + x 6

This may be reduced in the following way:

x 7 = x 2 x3 x8 = x5 + x6 x9 = x1 + x7 + x4 y = x9 x8

with the calculation of uncertainty done at each step as well, i.e. for the auxiliary quantities x7, x8, and x9. If the quantities are correlated or the same quantity appears in several places in the expression, this approach cannot be used without taking the correlations into account in the steps.
Software

It is strongly recommended to use a software package for the tedious calculation of uncertainty to avoid the necessity for reductions as shown above. Choose a program that allows you to input quantities, distribution functions, correlations and most importantly general model functions, and that calculates the sensitivity coefficients, uncertainty contributions and handles limited degrees of freedom.
5.1.1 Hints on establishing model functions

The establishment of a model function is most often an iterative process. Always start out by the relation that is used to calculate the measurement result, even if it as simple as y = x, e.g. the reading of an instrument. In that case always consider a reading as simply a number thrown at you with no direct meaning; not until a calibration (establishing the relation between the indication of an instrument and the physical quantity) is performed. Even a fixed standard with a stamped on nominal value, say a 100 resistor, should not be considered as a resistance of 100 without taking into account that there is a difference between the physical quantity presented and the collection of symbols 1-0-0- stamped on the box. There will always be a term describing the difference between the nominal value and the true physical value; in simple cases it may have the value zero, but its standard uncertainty will enter the equation for the standard uncertainty of the quantity being measured. Given a quantity that is known to affect the total uncertainty of a measurement process, but for which it may be difficult to establish the functional relationship with the output
6406 HDJ 2004 11 30

9 quantity, one hint is to exaggerate its influence, assume perfect knowledge of its value and determine how a correction would be applied. First give the quantity a name and a symbol so that it may be handled algebraically. E.g. in a test, the influence of the experience of the operator is known to affect, maybe not the result, but at least the expected variation in results. Using the hint, an assumption to make in the process of setting up the model function could be: I known that the operator always measures a value for quantity A which is 25% lower than the true value. If that was so, one would correct quantity A and instead use A/(100 % 25 %) = A/0,75. Hence, the quantity I look for f is the influence quantity, the relative magnitude by which the operator measures too low, and it enters as A/(1 f ). Now, given an observation that for a specific test, operator variability is found as a standard deviation in the results equal to 2%, and the fact that no specific bias (too high values or too low) may be assigned, and hence no correction can be applied, the influence quantity f may now be specified as f = 0, u(f ) = 0,02. It is important to list all possible sources of influence and quantify them in a form that may be expressed in a model function. Whether that needs to be in absolute or relative quantities, with one or another unit (remember also scaling factors in the model function; e.g. not mixing mV and V and similar!) depends of course of the situation, but a decision on some specific quantity is necessary. The method above may help in that respect, when formulating a specific exaggeration, and that will then aid in the proper inclusion into the model function. Second, it is not necessary to start by formulating the model function directly in terms of the quantity to be determined = an expression with all the input quantities. Write up the physical relations, add the appropriate influence components and then perform the algebra to achieve the direct functional form. E.g. say we wish to determine the mass of water in a bottle and we need to take into account the thermal expansion of water and bottle given that the temperature is different from a reference temperature. The physical volume in the flask equals the physical volume of the water, hence we have the physical relations, and the result of the derivation:
V flask ,no min al (1 + glass T ) = Vwater ,no min al (1 + water T ), mwater = waterVwater mwater = waterV flask ,no min al (1 + glass T )/ (1 + water T ) waterV flask ,no min al (1 + ( glass water )T )

5.2 Generic models

Many measurement processes are covered by three generic measurement methods: 1) direct measurement, 2) substitution, and 3) balance or difference measurement. For each of these three types, it is possible to set up a generic model function
Direct measurement

Direct measurements come in two flavours: generation of a known signal onto a measurement instrument (DUT = device under test), or determination of an unknown signal on a calibrated indicator. Calibration of either device is typically performed using the complementary method.

6406 HDJ

2004 11 30

10 When calibrating a measurement instrument, the measurand is the error of indication on the measurement instrument, x_err. The direct input quantities may be specified as:

x_indicate x_res x_stab

DUT
x_err

x_source

The setting x_source

of

calibrator/source,

The indicated value on the DUT, x_indicate.

x_transfer

The first approximation of the model function can then be formulated as, the error of indication equals the indication minus the setting of the source,
x_err = x_indicate x_source.

Now we may build up other influence factors to the model. If the indication (reading of the meter) is limited by instrument resolution or the need to interpolate an observation on a analog scale, it is appropriate to add a quantity for resolution loss, x_res, containing the part of the reading not accessible, e.g. the digits truncated off the display, or the inability to determine the subdivision of a scale. The value of the physical quantity generated by the source is not equal to the setting (dial setting or an entered value), but there is some deviation, x_transfer, which has been determined in a prior calibration. Hence we must add this correction (or subtract a deviation, depending on the how this information is obtained) to the value of the setting of the source. The manufacturer of the source may furthermore have specified the performance of the instrument under different conditions, i.e. the change in the value of the generated physical quantity with respect to the setting, x_stab, e.g. drift over time, influence of environmental conditions and other influence factors. Combining these we get the model function
x_err = (x_indicate + x_res) (x_source + x_transfer + x_stab)

One can then collect information on each of these input quantities:

x_indicate: From repeated observations we may find an average value and a standard deviation, which may then serve as our estimate for the mean and the standard uncertainty, respectively. If the indication is stable, i.e. a constant value, we may take the uncertainty of indication as zero (the resolution is taken care of by the x_res). x_res: In the first case an appropriate assumption is that digit is lost due to rounding, i.e. digit < x_res < + digit. The value of x_res is given equal probability in this interval, hence a rectangular distribution is appropriate, and finally we may state x_res = 0, u(x_res) = 1digit / 2 3. For readings on an analog scale it is common to use not less than one fifth of the scale marking as a limit to the mental subdivision that a skilled operator may perform. x_source: Depending on the type of instrument, we may treat this differently. For an instrument where we type in a value we may safely set the uncertainty to zero. For a knob setting it is appropriate to take into account the variability in the setting, including hysteresis, e.g. as an equal probability distribution of some appropriate interval, similar to that of x_res.
6406 HDJ 2004 11 30

11
x_transfer: This quantity is the difference between the value of actual physical quantity transferred to the DUT from the setting or nominal value of the source. This quantity would be determined by a calibration of the source, and the information needed should be available from a calibration certificate. To find the standard uncertainty we must examine the calibration certificate to find the confidence interval or the coverage probability or the coverage factor. For most accredited calibration certificates a coverage factor of k = 2 is used, hence we find the standard uncertainty as the quoted uncertainty divided by 2: u = U / k. x_stab: The influence of the environment or usage conditions to the output of a source is often described by the specifications of the manufacturer. For a more detailed discussion on the use of manufacturers specifications, see section 5.4.1. The most used strategy, given no other information, is to assign a rectangular distribution to the manufacturers specification, i.e. we obtain x_stab = 0, and u(x_stab) = specification / 3.

If determining the indication requires skill and hence depend on the operator, it may be appropriate to include such a component, i.e. add x_operator to x_indicate, with a value of zero and standard uncertainty equal the experimental standard deviation of readings by different operators under repeatability conditions. The model function is that of a simple sum/difference, hence we obtain the resulting standard uncertainty as the root-sum-square of the standard uncertainties of the components. The use of a calibrated instrument for direct measurement follows closely the discussion above, with a slight rearrangement of the components. The basic relation for a direct measurement, is to consider the indication on the instrument equal the value of the physical quantity presented by the object being measured (DUT). The basic model is thus:
x_dut = x_indicate

x_indicate

INSTR
x_err
x dut

x_res

x_stab

But again we have the influence quantities as described above: The limited resolution of the indication, , the instrument error, as determined above, , and the influence of environment, usage and time on the instrument, . All of these quantities would simply add to the . However, not that the definition of error is error = indication true value, hence the error determined above would need to be subtracted the indication. We arrive at the model function
x_dut = x_indicate + x_res x_err + x_stab

The source and handling of the components is equivalent to that discussed above.
Substitution and scaling

Substitution and scaling: A transfer instrument is used to generate an indication equal to (or with a small difference from) the measurement object and a known standard using an assumption of a stability and linearity of the indicating instrument. The quantity to determine is the value of the DUT, x_dut. The basic quantities that determine the value of x_dut are: the value of the reference, x_ref, and the indications on the transfer instrument, x_i, ref and x_i, dut.

6406 HDJ

2004 11 30

12 The basic assumption used in the substitution method is, that the relationship between the indication on the transfer instrument, when measuring the reference and the DUT is equal to the relationship between the value of the reference and the DUT. The basic model function, i.e. the way we calculate the value is derived from the relation
x _ dut x _ i, dut = x _ ref x _ i, ref x _ dut = x _ ref x _ i, dut x _ i, ref

x_stab

INSTR
x_dut

x_i, ref x_i, dut

x_res x_transfer x_res

x_ref

Again, we may now expand the basic model function to take into account relevant influence factors. The value of the reference, which we assume to know from its calibration, may still be subject to a change over time or operating conditions. This influence we assign as an additive value x_stab. When reading out the indication we may again be limited by the resolution of the instrument or the readability of the scale. We assign the quantity x_res for the missing part of the reading due to rounding. In this type of measurement setup we have two contributors to the loss in resolution, one from the indication of the reference and one from the DUT. Note that we may assume that the two quantities are independent; there is no immediate connection between the rounding in the two cases. The final contribution we consider is the capability of the transfer instrument to perform the transfer, x_transfer. One essential part of the quantity is the linearity of the scale, however more important in the present case is the variation of the deviation from linearity. To illustrate this point, let us assume the relation between the reading on the instrument and the physical quantity is x_true = x_indicate (1 + lin_err), where lin_err is the deviation from the scale factor 1. For such an instrument we find that the ratio of two readings is unaffected by the linearity error. Hence, we must quantify the deviations from linearity and the stability over the measurement time. Thus, we assign x_transfer as the change in linearity over the part of the scale used, and will include it in our model function as a factor on the ratio of the indications. Adding the components to the basic model function we obtain:
x _ dut = ( x _ ref + x _ stab ) x _ i, dut + x _ res1 (1 + x _ transfer ) x _ i, ref + x _ res2

We collect the information for each of the input quantities: x_i,ref and x_i,dut: From repeated observations we use the average and the standard deviation. For a constant reading, u(x_i,..) = 0 and the uncertainty is handled by x_res. x_res1,2: Again the appropriate contribution is x_res = 0 and u(x_res) = 1 digit / 2 3. x_ref: The value and the uncertainty should be obtained from a calibration certificate. x_stab: The expected change in the reference value over time or over usage conditions may be found either from own analysis based on the results of previous calibrations, experience or from specifications from the manufacturer of the standard. Depending on the source of the information different methods of arriving at the value (which would be 0 if
6406 HDJ 2004 11 30

13 no correction is applied) and the standard uncertainty may be employed. E.g. a manufacturers specification may be assigned a rectangular distribution and treated accordingly, or if a historic drift curve is used, the calculated standard uncertainty from the linear regression analysis is appropriate. x_transfer: The transfer component will at best be characterised by the short term reproducibility of the indicating instrument. Thus it may be characterised by performing repeated measurements. If more information is available, such as a specific relation for the short term stability of the indicating instrument, this should be taken into account. In case even slightly different values are read, the non-linearity of the scale should be considered, either from a calibration or from information such as the manufacturers specification. The model function is unfortunately not a simple sum/difference form hence other calculation tools must be taken into use, or calculated in steps by evaluating the various sums separately and combine the factors using relative uncertainties.
Balance or difference method

The unknown quantity is balanced against a known reference, with a small difference nulled out or with an absolute measurement of a small difference.

x_stab

INSTR
x_dut

x_null
x_ref

An unknown quantity is balanced against x_res a known reference and a small difference x_adjust is measured or nulled out using an adjustment. The measurement principle is similar to that of substitution, except the DUT and reference will typically be physically connected, and the indicating instrument will be used on its lowest range. The quantity to determine is the value of the DUT, x_dut. The basic quantities that determine the value of x_dut are: the value of the reference, x_ref, the indication on the null instrument, x_null and the value of the adjustment x_adjust. The basic model function when balanced, i.e. the way we calculate the value, is found as

x _ dut = x _ ref + x _ adjust Again, we may now expand the basic model function to take into account relevant influence factors. The value of the reference, which we assume we know from its calibration, may still be subject to change over time or operating conditions. This influence we assign as an additive value x_stab. When the null indication is achieved, we may again be limited by the resolution of the instrument or the readability of the scale. We assign the quantity x_res for missing part of the reading due to rounding. The measurement of the null is also subject to uncertainty. If the instrument is used as a true null detector by applying an adjustment signal x_adjust to achieve a null reading, our x_null should reflect the ability to achieve a true null, e.g. if the null instrument is a voltmeter, a comparison with a short circuit is appropriate. If polarity reversal is possible, the null value may be derived from the forward and reverse polarity readings. Finally, the x_adjust may be either the direct difference reading on the null instrument if no adjustment is performed, or the value of the additional signal we apply to achieve a null.
6406 HDJ 2004 11 30

14 The demands on the relative uncertainty of x_adjust are much less stringent than that of x_ref; if, say, x_adjust is 1 % of x_ref and its relative uncertainty is 1 %, the overall relative uncertainty contribution becomes 10-4. Adding the components to the basic model function we obtain:
x _ dut = ( x _ ref + x _ stab ) + x _ adjust + ( x _ null + x _ res)

We collect the information for each of the input quantities: x_ref: The value and the uncertainty should be obtained from a calibration certificate. x_stab: The expected change in the reference value over time or over usage conditions may be found either from ones own analysis based on the results of previous calibrations, experience or from specifications from the manufacturer of the standard. Depending on the source of the information different methods of arriving at the value (which would be 0 if no correction is applied) and the standard uncertainty may be employed. E.g. a manufacturers specification may be assigned a rectangular distribution and treated accordingly, or if a historic drift curve is used, the calculated standard uncertainty from the linear regression analysis is appropriate. x_adjust: The additional signal applied may come from a source, and hence handled similar to x_ref. If x_adjust is a direct measurement of a small difference, we may thus need the manufacturers specification for the range used and the value read. x_null: From repeated observations we use the average and the standard deviation. For a constant reading, u(x_i,..) = 0 and the uncertainty is handled by x_res. x_res: Again the appropriate contribution is x_res = 0 and u(x_res) = 1 digit / 2 3. The model function is again a simple sum/difference and the root-sum-square of the uncertainty contributions gives the standard uncertainty of the result.
5.3 Repeatability - reproducibility

Often in testing, series of trails are used to validate the measurement set-up, the method employed as well as in the measurement process itself. Employing statistical methods on the final test results is a powerful method of obtaining an overall estimate of the total uncertainty. However, the requirement for this to be true is that the influence quantities are varied suitably over their full range during such a series of trails. The ISO 5725 series of standards cover much of the detail on the design of experiments and the analysis. The basic idea is to quantify from each level of repetition, the variance of one or more influence quantities in terms of a standard deviation. From the results of two or more levels, it is then possible to quantify the consistency of the assumption on the number of independent influence quantities. This process of determining repeatability and reproducibility may be summarized with the term precision. Secondly when it becomes important to consider comparability outside the laboratory, the concept of trueness, or the determination of bias, must be taken into account. At this point it is important to stress: Precision, i.e. repeatability and reproducibility, is a measure for the ability of the lab to generate consistent results, and can be estimated from internal sources.

6406 HDJ

2004 11 30

15 Trueness or bias, is a measure for the ability of the lab to generate correct results or comparable results, and therefore, must be estimated from external sources: calibration, reference materials, intercomparisons, etc.
5.3.1 Basic assumptions and use of nested designs

It is important to stress, that the methods described in ISO 5725 and ISO 21748 are based on a pragmatic or empirical model function and not model functions based on the physics of the measurement procedure. There is no contradiction with the philosophy of the GUM and the methods are fully applicable for this type of treatment. The empirical model can be stated as (ISO 21748)
y = m + + B + ( ci xi ) + e

where y is the observed value, m is the ideal or true (unknown) value, is the intrinsic method bias, B is the laboratory bias and e is an error term. The sum of ci xi represents other effects than those giving rise to and B. Hence, m in this treatment could be the reading of the (complex) measurement equipment in the form of a black box, which we attribute to the quantity we wish to assign a value to. The parameter is the bias or error the particular method of measurement has been assigned or that one must assume exists when comparing results based on (very) different measurement principles or similar. The parameter B is also a bias or error, but related to the laboratory implementation of the method, i.e. if several laboratories implement the same measurement using the same method, the parameter B would signify the dispersion expected; i.e. the method reproducibility as implemented in the laboratory. In some cases it may not be possible or relevant to distinguish between and B. The terms ci xi are physically modelled influence factors. The formulation used allows one to, e.g. take environmental influences into account, say it is assumed the temperature in the laboratory affects the result with 1 %/C (i.e. the sensitivity factor), one would then be able to take into account the uncertainty in temperature combined with sensitivity factor. Finally, e signifies the pure random errors, or the repeatability term (an average value or other statistically derived value would be part of m). The assumptions in the treatment are, that the expectation of B and e is zero (hence no correction is applied for the laboratory bias!), and have variances L and r respectively. The quantity e basically describes the repeatability of the measurement, while B signifies the reproducibility under ideal conditions.
2 The reproducibility observed would be the combination R = 2 L + r .

The whole idea of the exercise is to find from the observed reproducibility and the observed repeatability to find whether the ideal reproducibility is significant or not, and hence if there are contributors to the total uncertainty that must be described separately. Given two sources of variability, the purpose of the nested design experiments and ANOVA is to determine the significance of one over the other. Let x_A be a component that signifies the within-day variability of some measurement results, and let x_B be a component that signifies the between-day variability. How, and to what extent can we determine the significance of x_B and does it make sense to try and split the observed dispersion into these two components?

6406 HDJ

2004 11 30

16 Lets assume a balanced nested design. Each day we perform N measurements. The ex2 of one of the within-day results would be an estimate of the perimental variance s A variance of the individual measurements. The average of the within-day results would 2 N , where N is the number of measurement then have an estimated variance of s A within a day. Hence, when we calculate the observed variance of the between-day re2 , each result being the average of the results of that day, we should expect a sults, s a
2 2 value compatible with s A N . If the observed s a is significantly larger than this, we 2 must assume there is a contributor x_B with variance s B to the variability and which we 2 2 2 have not taken into account, so that s a = s A N + s B .

To test for significance, an f-test may be performed, by comparing the critical f value 2 2 . One we obtain by pooling the s A values, which with the ratio of two estimates of s A 2 2 for the balanced design is the average of these, sb = s A M , where M is the number of days we conduct our experiment; this estimate has (N 1) M degrees of freedom the 2 2 2 , which has M 1 degrees of freedom. We calculate the ratio s a sb and other as N s a 1 compare to the critical value F0,95(M1, (N1)M) . E.g. if we do 5 measurements each day for 10 days, the critical value is F0,95(9, 40) = 2,12.
2 If the ratio is smaller than the critical value, we cannot conclude that s a is significantly 2 larger than s A N , hence there is no x_B component. 2 If the ratio is larger than the critical value, we can conclude that s a is significantly lar2 ger than s A N , hence there is a x_B component. We may even estimate the magnitude 2 2 2 2 of s B as s B = s a sA N (but be careful if these numbers are close; remember the uncertainty of the variance estimates!)

Thus, appropriate nested designs allow us to extract information of separate uncertainty components based on an empirical model and statistical treatment of the measurement data. However, the method has its limitations. A simple systematic effect such as drift in the measurement value, e.g. caused by a change in a reference standard or the measurement instrument, cannot be distinguished from purely random sources. Hence, it is important, before relying on empirical models, to examine the measurement problem from a physical-technical standpoint, and incorporate the influences identified into the model function. The general expression above allows partly taking such physically modelled effects into account through the terms ci xi.
5.3.2 Intercomparison data

From the discussion above, it is found that the characterisation of repeatability and inhouse reproducibility, together with the technical-physical characterisation of the measurement problem may give an overall estimate of the total uncertainty. However, it is not until a comparison is made with an outside source that a systematic laboratory bias may be uncovered. In order for the laboratory bias, i.e. the systematic difference between the laboratory in question and the outside world, may be stated to be in control, the laboratory must
In Excel, the F-test critical value can be generated as FINV(1-p, 1, 2), where p is the confidence level (e.g. 95%), and 1 and 2 are the degrees of freedom of the variance estimates being compared.
6406 HDJ 2004 11 30
1

17 establish that its reproducibility with variation of several of the influence factors covered by the statistical treatment is within the standard deviation of the betweenlaboratory results of a comparison. Unfortunately, intercomparisons in the field of LVD testing are so infrequent and with a small number of participants, that a good statistical basis for this evaluation is very difficult to obtain. Another method of establishing the magnitude of any bias, is the use of certified reference materials. Again, in the field of LVD testing, such items or materials are usually not available.
5.4 Other basic tools

To generate a realistic uncertainty budget many sources must be taken into consideration. However, many of these a common to measurement situations across the fields of metrology. Below are given some examples of such effects that may be treated in the same way independent of the field of measurement.
5.4.1 Translation of instrument specifications

Measurement instruments are typically accompanied by a set of specifications, indicating the performance to be expected by the instrument under specific conditions of use, as determined by the manufacturer. These are often given as direct or relative limits of the error of indication or generation of the measurement quantity in question. However, there are almost as many ways of indicating the performance of instruments, as there are manufacturers. First, it is necessary to determine to which level of confidence the specifications are given. In few cases statements such as 99% confidence or 95% confidence are given. If it assumed, that the specification may be interpreted as a quantity with a Normal distribution, a 99 % confidence interval corresponds to a coverage factor of k = 2,58 and a 95 % interval corresponds to k = 2,58. To use the specification one would divide the specification limit with the appropriate k-factor to get the standard uncertainty. However, often no such indication is given and it is most appropriate to assume a rectangular distribution for the limit, and hence divide by a factor 3. Second, it is necessary to determine which limitations apply to the specifications given. Do they cover all secondary settings (e.g. integration time, filter settings), or are there additional terms which must be added, e.g. for measurement of resistance, specifications are often quoted for four-wire measurements, and an additional term is quoted for twowire measurement. Third, in many cases, specifications are often quoted in not-so-rigorous language, and caution must be applied in translating these into more strict mathematical form, including separating absolute value terms and relative value terms. In some cases, absolute value terms are even given as a relative term of a range or maximum value, which may not be the possible maximum reading for a range, and even visa-versa. E.g. for some instruments a range may be called 1, but has a maximum reading of 2 and the constant contribution for a measurement on that range is to be calculated from 2 times the range uncertainty. An example is a common statement such as a table heading (% reading + % range) and table entries for specific measurement ranges, e.g. 100 mV 0,005+0,004. This shorthand would then be interpreted for a reading of V, as the interval (510-5 V +
6406 HDJ 2004 11 30

18 4 V). With the assumption of a rectangular distribution, we achieve the standard uncertainty u(V) = 2,910-5 V + 2,3 V.
5.4.2 Calculation, use and uncertainty of calibration curves

Under many measurement conditions, some transducer is used to convert from one physico-chemical property into a property, say voltage, resistance or similar, that is more easily handled and measured. To characterise the transducer and allow the transfer back and forth the two properties, a calibration curve or transfer function is needed. The calibration curve or transfer function is in some cases given by a parameterised, but standardised expression, e.g. the R(T) for resistance thermometers and thermistors, or the V(T) for thermocouples. A calibration curve will typically be mapped by performing a calibration: applying a series of known signals and recording the responses. From the values and standard uncertainties of the inputs and outputs, and appropriate parameterised calibration curve may be determined, by determining the value of the parameters and their covariance (because the parameters are all determined from the same set of inputs, they will be correlated, thus we must derive and use the full covariance matrix of the parameters). With the parameters and covariances in hand, we may for a given observed output derive the appropriate input and its standard uncertainty.

x
Fig. 1 Calibration curve

The red points signify the calibration points, known x with observed y, giving the green calibration curve Given an observation (blue) we derive the stimuli (orange).

Hence the starting point is a set of applied stimuli {xi , u(xi)}, the measured responses {yi , u(yi)} and an empiric or physical bond f(x, y) = 0. The simplest transfer function is a linear relation, f(x,y;a,b) = a+ bx y, with parameters a and b.
Linear regression, y = a + b x

If the uncertainties in x are so small that we may ignore their contribution, we obtain the standard formulas (following Ref. 7):
a= b= S xx S y S x S xy SS xy S x S y

2 = SS xx (S x )
1 2 i =1 u ( yi ) N x2 S xx = i 2 i =1 u ( y i ) S =
N

where

xi 2 i =1 u ( y i ) N xy S xy = i i 2 i =1 u ( y i ) Sx =

Sy =

yi 2 i =1 u ( y i )

6406 HDJ

2004 11 30

19
Uncertainties

The standard uncertainties calculated from the propagation of the uncertainties in y are
u (a) = S xx , u (b) = S , u (a, b) = S x

To test whether the regression line is an appropriate model (i.e. consistent with the uncertainties in y), it would be necessary to compare the Chi-squared value
a + bxi yi ( a, b) = u( y ) i =1 i
2 N

with the degrees of freedom N 2.

If the uncertainties in y are not available, and a good fit is assumed (check a graphical representation!!) the calculation above can be performed using u(yi) = 1, and the uncertainties for the parameters are found by multiplying the calculated u(a) and u(b) with the factor

2 /( N 2) . The covariance is multiplied by the factor 2 /( N 2) .

Using the fitted line

With the parameters a and b, and their uncertainties, in hand we can calculate a predicted output value from an input, e.g. an inter- or extrapolation,
2 ycalc = a + bxmeas , u ( ycalc ) 2 = u (a ) 2 + xmeas u (b) 2 + 2 xmeas u (a, b) + b 2 u ( xmeas ) 2

From a measured output we may derive the value of the input. xcalc = ( y meas a) / b This acts as a model function and we find the standard uncertainty of xcalc as u ( xcalc ) 2 = 2( y meas a) 1 u ( y meas ) 2 + u (a) 2 + ( y meas a) 2 u (b) 2 u ( a, b) 2 b b

with the assumption that the ymeas is independent of the y-values used for the determination of the linear regression parameters. An uncertainty in the values of x used to determine the calibration curve, will also affect the uncertainty of the parameters a and b. An approximation that may be used to take this into account is to scale the uncertainty of xi onto the y-scale using the b parameter calculated without taking u(xi) into account. Then one replaces u(yi)2 by u(yi)2 + b2 u(xi)2 in the formulas above, calculates new values for a and b. A couple of iterations should give a consistent set of values.
General linear least squares

The linear regression above as well as general linear least squares estimation may conveniently be put in a matrix formulation, again with an assumption of a negligible uncertainty for the x-values: Given are the linear (in the parameters) combination y = a j x j = a x , where a = (a1,
j =1 M

a2, , aM) are the parameters to determine and x = (x1, , xM) are generalised inputs (each xi could be a function of a single input parameter, e.g. t, and could include constants, i.e. xi = 1). We then use the set of calibration points {xi, yi} and the covariance matrix ij = u(yi, yj). We can put the x-values in a matrix X with elements Xij = (xj) i and

6406 HDJ

2004 11 30

20 to the least square estimation (the estithe y-values in the vector Y, then the solution a mate of a) becomes = C X T 1 Y, where C = ( X T 1 X) 1 . u (a, a T ) = C a
= Xa with and a predicted value for an input becomes y = x a The fitted values are Y u ( y ) 2 = xCxT

Linear regression in matrix notation

How does this look in an implementation? For the linear regression above we have a = (a, b) and x = (1, x). With a data set of {xi, yi, u(yi)} without correlations between the yvalues we have
u ( y1 ) 2 0 0 y1 1 x1 X = 1 M , Y = M , = 0 O 0 0 y 1 x 0 u( y N ) 2 N N We then get (using the notation above)
S XT 1 X = S x Sx C = XT 1 X S xx

1 S xx Sx

Sx , S

(X

Sy 1 Y = S xy

and finally
= C X T 1 Y = a

1 S xx Sx

S x S y 1 S xx S y S x S xy 1 S xx = , u (a ,a T ) = S S xy SS xy S x S y Sx

Sx S

which is the same result as quoted above. If the y-values are correlated, the matrix above is replaced by the general covariance matrix for y. The inclusion of uncertainties in the x-values can be included by replacing the covariance matrix by u ( x1 ) 2 u ( x1 , x2 ) L 2 O M = y + b x , x = u ( x1 , x2 ) M L u( xN ) 2 For a polynomial expression we would have something like x = (1, x, x2, x3, ) and the procedure to derive the formulas and expressions similar to that shown above. Using the general linear least squares We may obtain the combined uncertainty when using the calibration curve to look up the value from a measured output y including its uncertainty u(y). One requirement is that the y is independent from the y-values used to determine the calibration curve and hence not correlated with the calibration curve parameters. Let us for simplicity assume that x = x(t), e.g. x(t) = (1, t, t2); e.g. if a = (R0, A, B) this could be the calibration curve for a resistance thermometer, with t as temperature and y as resistance: y = a x = R0 + A t + B t2. The general formula for the least squares solution above gives

6406 HDJ

2004 11 30

21 x(t )Cx(t ) T + u ( y ) 2

u (t ) 2 =

(a (x t ) )

T 2

y = a x(t ) the denominator term becomes

For the temperature curve above, T a (x t ) = ( R0 , A, B) (0, 1, 2t ) T = A + 2 B t

The calculation of the covariance matrix C is in this case a bit more complicated, but easy to handle numerically. Such least squares algorithms can easily be implemented in a spreadsheet program such as Microsoft Excel, which supports matrix algebra. The inclusion of uncertainties in x may be included in the determination of the model parameters, as was shown for the linear regression above. It requires the scaling of u(xi) onto y using the first approximation of a. In the case where x = x(t) and we assume no covariances, one would replace u(yi)2 by u ( yi ) 2 + [a (x(t i ) t )]u (t i ) 2 .

Non-linear general least squares

If it is necessary to be able to take both uncertainties in x and y into account, e.g. if the uncertainties in the independent variable are significant, numerical methods must be used. An algorithm for such full calculation is given in Ref. 8. The method described there allows also for implicitly given model functions, i.e. general f(x, y) = 0, and describes also how to test for the consistency of the fit obtained.

5.4.3 Drift
Most material standards are affected by internal and external sources, which changes their characteristics over time. Hence the value represented by the standard changes over time. In many cases the change may be sufficiently small and the change is insignificant. Regular calibration may be used to quantify this change and one may be able to predicted the drift between calibrations. Thus, the change may be compensated and/or the knowledge of the time-dependence can be used to fix calibration intervals; if the standard proves stable enough, longer intervals are possible and visa-versa. The most common method is prediction from linear regression. Hence the methods and formula described in the section on calibration curves may be applied to drift as well. A drift component would then be handled as an additive contribution to the value of the standard, with a standard uncertainty as calculated from the linear regression.

5.4.4 Handling ill-defined measurement conditions

If the variability of the test results is caused by a variation in the measurement conditions, this variability should be quantified by including the influence quantities into the modelling of the measurement situation. This may be done by direct physical modelling or by including a empirical model quantity of the form ci xi. However, it could be argued, that the variation due to ill-defined measurement conditions is not the responsibility of the testing laboratory, but should be part of the method validation studies. That is, such studies should be performed in the process of developing the test method, and should be information provided in the standard. It is the responsibility of the laboratory to evaluate measurement uncertainties of the measurements performed. It would then be appropriate to compare the measurement uncertainty against such method uncertainty in the validation of a specific test setup, and in the process of intercomparisons.
6406 HDJ 2004 11 30

22 Alternatively, in order to facilitate comparison of test results, the actual measurement conditions should be specified when reporting the test result. If differences are found, e.g. in a comparison, and it is observed that measurement conditions vary, it would be appropriate for an investigation as to the magnitude of the influence factor. If, however, the dispersion is much smaller than the test limits, this will only be an academic exercise and not influence the conclusions or decisions drawn from test results.

6 Examples from electrical testing 6.1 Measurement of temperature using thermocouples

In the next two examples of basic LVD testing, the measurement of temperature using thermocouples is the basic physical measurement used. We therefore present here an example of an uncertainty budget for such basic measurement.

Note, however: This example is constructed to present some of the considerations, interpretations and arguments in setting up the model function and assigning uncertainties, and should not be taken as a direct example on what uncertainty to assign a thermocouple temperature measurement!
Thermocouples are made from two wires of dissimilar metals brought together in a joint. Due to the thermoelectric effect and different coefficient for different metals, different voltage drops will occur along the two wires from the common joint, and a voltage difference can be measured at the open end of the thermocouple wires. Given a thermocouple of a specific type, there are standard expressions for the voltage-temperature relationship, e.g. for type K thermocouples a 9th degree polynomial expression plus an exponential term is the reference expression. An inverse expression, giving the temperature as a function of the voltage difference, is also available stated in the same form as a 9th degree polynomial. The conversion error is typically 0,05 C. To avoid the additional thermal voltages when connecting the thermocouple to the voltmeter, an additional thermocouple is inserted held at a standard temperature, 0 C, allowing for the same junction type at the voltmeter. Alternatively, a correction based on the temperature of the junctions at the voltmeter is applied. For the purpose of the measurements for heating testing, we do not seek the best uncertainties, and will assume an internal reference junction compensation is used. The basic for the measured temperature thus becomes T = t(Vt) with Vt = Vm V0, where t(Vt) is the reference conversion curve, Vm is the thermocouple voltage difference and V0 is the compensation voltage. Assume we wish to measure 950 C using a type K thermocouple. The reference voltage is Vref = 39,3139 mV and the sensitivity c950 = dT/dV = 25,3077 C/mV. We perform a first order expansion of the reference curve and obtain T = 950 C + c950(Vm V0 Vref). We may then begin to expand the model function based on other contributions. First, the deviation of the thermocouple in question to that of an ideal thermocouple. We will state that as the additional voltage generated by the thermocouple at the temperature T, Vtc. Second, for the voltmeter we have its measurement error Vm from calibration, the resolution VI, the drift and operating conditions from the specification VD, and the error in the
6406 HDJ 2004 11 30

23 compensation voltage (internally based on the reference curve and the measured temperature on the connection block) V0. Finally, we have the conversion error itself Tc. To expand the model function we may argue: If the meter reading is perfect Vm and the temperature is T, the thermocouple generates too high a voltage Vtc. Hence we must subtract this from the reading. Also the reading presented is corrected for the cold junction voltage, so V0 is in reality included in Vm. Next, the reading is not perfect, but is in error by Vm as determined by calibration, hence we must also subtract this from the reading. Also the drift and operating conditions give too high a reading, so we must subtract VD. The meter compensates the cold junction by calculating the cold junction voltage and subtracting it to the measured voltage, however, this voltage is in error by V0, hence we must subtract that from V0 i.e. add to Vm. Finally, the imperfectness of the temperature-voltage conversion gives an error Tc in temperature, which we must subtract the reference temperatur. With these arguments, we arrive at the expanded model function T = 950 C + Tc + c950(Vm + V0 Vtc Vm VD Vref). Now we examine each of the components and derive the value to assign and the standard uncertainty. Tc: The reference curve specification gives the conversion error as 0,05 C. We will assume there is equal probability in this range, and assign a rectangular distribution function. Hence we obtain the value Tc = 0 C, with standard uncertainty u(Tc) = 0,05 C/3 = 0,029 C. Vm: During measurement we acquire 10 readings. We calculate an average 39,151 mV and a standard deviation 0,011 mV. Hence, Vm = 39,151 mV, u(Vm) = 0,011 mV/10 = 0,0034 mV. V0: The instrument specifies that the cold junction compensation is done with an accuracy of 0,5 C. Converted to voltage, this corresponds to 20 V. We choose to understand this, as the compensation voltage is accurate in the interval 20 V to +20 V. Hence, we assign a symmetric rectangular distribution and get V0 = 0 V and u(V0:) = 20 V / 3 = 11,5 V. Vtc: The thermocouples are made in the laboratory using good quality thermocouple wire. A batch of thermocouples are made and calibrated in an oven and measure their voltage output in a specialised setup. It is found at 950 C that the average voltage generated is 39,250 mV with a standard deviation of 21 V. The uncertainty of the generated temperature is estimated as 0,2 C measured by a standard thermometer, and the uncertainty of the voltage measurement is 5 V. The average voltage error is (39,250 mV 39,314 mV) = 0,064 mV. The 0,2 C temperature uncertainty corresponds to 8 V uncertainty in the generated voltage, hence the combined uncertainty of the voltage error is (212 +52 + 82) V = 23 V. Hence, for a thermocouple from this batch we use Vtc = 0,064 mV, u(Vtc) = 0,023 mV. Vm: The meter has been calibrated at an accredited laboratory. At 40 mV on the 100 mV range, the error was found to be 6 V with an uncertainty of 5 V. The uncertainty is stated as a 95% confidence value, i.e. overage factor k = 2. Hence we use Vm = 6 V, u(Vm) = 5 V / 2 = 2,5 V. VD: The specification of the meter is stated in the form 0,02 % reading + 0,03 % range. The meter is used on the 100 mV range, thus 0,03 % range is 30 V. The reading is 40 mV and 0,02 % reading is 8 V. Added we get 38 V. This is the manufacturers estimate of
6406 HDJ 2004 11 30

24 the absolute limits of variation of the error for 1 year and under the prescribed measurement conditions. Hence, we interpret this as a symmetric rectangular distribution, (0 38) V, i.e. VD = 0 V, u(VD) = 38 V / 3 = 22 V. The resolution is much lower than this term, hence we regard this as included in the specification. We will evaluate the uncertainty in steps: The voltage terms form a sum and difference expression; hence we may combine the uncertainties directly by root-sum-square. We thus obtain V = 0,105 mV, and u(V) = 34 V, dominated by the meter specification and the thermocouple error. We may note, that the uncertainty found for the reading is a factor of 10 lower, hence does not contribute significantly. We may therefore without loss of accuracy reduce the number of readings to, say, four; we have an estimate of the standard deviation, 11 V, and the average of four readings would give a standard uncertainty of 5,5 V. With this value the combined voltage uncertainty remains 34 V within the rounding. The combination of the voltage and the scale factor is a product, but we have not assigned an uncertainty to the scale factor because the conversion uncertainty is covered by Tc. Hence the uncertainty in voltage scales directly to an uncertainty in temperature, and we obtain Tc = c950V = 25,3077 C/mV (0,105) mV = 2,65 C and u(Tc) = c950u(V) = 25,3077 C/mV 0,034 mV = 0,86 C. The final result is thus T = 950 C + 0 C 2,65 C = 947,35 C. The standard uncertainty is the combination of the converted voltage uncertainty and the reference curve uncertainty. The latter is more than a factor of 10 lower, thus we may safely ignore it and obtain u(T) = 0,86 C.

6.2 Heating in Black Test Corner

The tests specified in EN 60335-1 relates to heating of household appliances. The test requires the measurement of temperature on and close to the surface of the appliance under normal operating conditions. A black test corner is implemented as an array of thermocouples connected to Cu disc mounted on wood plates. By using a number of plates, a corner with dimensions of the order 1 m x 1 m x 1 m can be constructed. This arrangement is typically used to measure side and bottom temperatures of large appliances. If we look at the measurement situation, we again must measure the voltage of the thermocouples. The voltage of the thermocouples is related to their temperature, but we are interested in the temperature at the surface of the appliance. Thus we must take into account the difference in temperature from the thermocouple and the position of the surface. Alternatively, we may apply a temperature to the thermocouple array and thus indirectly take into account the transfer, however, we will then be subjecting the thermocouple wire to a different temperature distribution under calibration than under use. Hence we must carefully specify our measurand: Do we report the temperature of the thermocouples in the arrangement specified by the standard, or do we try to evaluate the temperature of the surface of the appliance? In the first case, our measurement model is clear: We have the voltage measurement, translated into temperature using the standard transfer curve, combined into the variable TV, the calibration measurement of the thermocouple voltages when subjected to temperatures generated in an airbath, combined into the variable Tc, and, if we treat all thermocouple
6406 HDJ 2004 11 30

25 alike, and do not perform individual corrections, we have the deviation of the thermocouple voltage from the reference voltage, again scaled to temperature, Tc. Our model function: T = TV + Tc + Tt , with standard uncertainty u (T ) = u (TV ) 2 + u (Tc ) 2 + s (Tt ) 2 Given, for example, the voltage measurement uncertainty and the reference curve conversion has specification 0,4 C, we use u(TV) = 0,4 C / 3 = 0,23 C, the total calibration uncertainty of the temperature bath u(Tc) = 0,04 C, and the observed standard deviation of the individual thermocouple temperatures, u(Tt) = s(Tt) = 0,69 C. Combined we achieve u(T) = 0,73 C, and we see the dominating contribution comes from the spread of the thermocouples. Because this component is near a Normal distribution, we may use coverage factor k 0 2 and achieve U(T) = 1,5 C. If we were to specify our measurand as the surface temperature, we would need to evaluate the possible temperature difference between our thermocouple and the surface, across the airgap present. One way of evaluating this would be to subject the thermocouple plate to a calibration plate. The difference may then be measured and an appropriate component added, with the uncertainty evaluated from, e.g. the maximum deviation combined with the temperature measurement uncertainty.

6.3 Glow wire test

The objective of a glow wire test (EN 60695-2), is to investigate the non-flammability of insulation materials. A piece of material, typically plastic, is subjected to a hot wire, typically NiCr, heated by an electric current. It is observed that the material does not ignite after contact with the wire for a specific time at the specified temperature. The temperature is measured by a thermocouple in the wire near the point of contact. There are some physical dimensions (diameter of wire and penetration) and mechanical specifications (pressure applied) that must kept for the test procedure to be fulfilled. Also the construction of the thermocouple and its integration into the glow wire is specified in the standard. The measurement itself is specified as the temperature of the glow wire. The specification for the thermocouple to use is 0,4 % T (in C), and a verification measurement specified is the generation of the melting of silver with a tolerance of 15 C; a somewhat relaxed requirement in comparison. Let the thermocouple voltage be read by a voltmeter with an internal calculation of a type K thermocouple temperature. The 1-year specification of the instrument is given in the form:

The specification for a Type K thermocouple is thus 1,0 C. We interpret this as a rectangular distribution of the instrument error of (0,0 1,0) C, hence the instrument error is TV = 0 C, u(TV) = 1 C / 3 = 0,58 C.

6406 HDJ

2004 11 30

26 A certified class 1 type K thermocouple, the tolerance (above 375 C) is given as 0,4 % of the temperature. Hence, when the voltage generated equals the reference value at temperature T, the true temperature is in the interval T (1 0,4 %). The temperature indicated on the display of the meter TI will be that of the reference equation, the error due to the probe id thus Tc = 0 C, with u(Tc) = 0,4 % / 3 TI . The value of TI will be given by the reading, and let us assume a constant reading is obtained, i.e. s(TI) = 0. There will then be a contribution from the resolution of the display, a rounding of 0 0,05 C; hence TI = 0, u(TI) = 0,05 C / 3 = 0,029 C. Our basic model function becomes: T = TI + TI + Tc + TV The uncertainty is derived from the simple sum/difference formula: u(T)2 = (0 C)2 + (0,029 C)2 + (0,23 % TI)2 + (0,58 C)2, varying from 2,1 C at 900 C to 2,4 C at 1000 C. Because our dominating term is the tolerance of the thermocouple, which we assigned a rectangular distribution, the correct coverage factor is in fact k = 1,65. The expanded uncertainty at 950 C thus becomes U(950 C) = 3,7 C. Is it necessary to evaluate uncertainty contributions from the measurement conditions to the result? No, it should not! The laboratory must document that the conditions specified (mechanical properties and dimensions) are fulfilled, but it would be the task of the standardization committee to translate the measurement conditions into an overall method uncertainty. This method uncertainty would then be the appropriate uncertainty to compare against in interlaboratory comparisons. The uncertainty reported by a laboratory for a quantitative test result should be the measurement uncertainty obtained for the measurement.

6.4 EMC - Radiated Immunity Test (IEC 61000-4-3)

For EMC compliance an instrument must be immune to an applied electromagnetic field. A test is performed where an instrument is placed in a electric field of known field strength and the ability of the instrument to function is monitored. To fulfil the test conditions, the field distribution must be uniform over the test volume within 6 dB, and the field strength be known. Thus, the test conditions rely on a calibrated field probe and measurement equipment, and compliance with the test specification of a uniform field. The measurement uncertainty budget below is based on the assumption that it has been demonstrated during calibration that the 6 dB field uniformity has been achieved. Once the expanded uncertainty has been calculated then the value is related to the standard severity level (e.g. 3 V/m) to provide an increased test level, which accounts for the uncertainty in the test configuration. The mathematical model for the measurement process is assumed to be: F = FSM + FSAW + PD + PAH + FD + RS + REUT It should be noted that this model does not account for the unknown influence of the unspecified semi-anechoic chamber and the significant variations caused by reflections from
6406 HDJ 2004 11 30

27 the EUT. The contribution form the test set up will need to be evaluated and given a value. In the mean time the 6 dB field uniformity restriction is a means to at least have some control over this contribution for an empty chamber. Thus, to include the lack of information/characterisation of the measurement setup, the 6 dB requirement should be added to the uncertainty for FD.
Symbol FSM FSAW PD PAH FD

Quantity Field Strength monitor Field Strength acceptability window Forward power Measurement Drift Power Amplifier Harmonics Effect of field disturbance Mismatch Measurement System Repeatability

Units dBV dB dB dB dB

Comment As reported on the Field Probe calibration certificate Usually a set parameter to allow the software to accept a value for field strength that is within an acceptable window of the calibration level. A function of the power meter or other device connected to monitor forward power. Contribution due to inclusion of amplifier distortion in power meter reading. Possible effects due to table and/ supporting structures. Not applicable. The repeatability of the measurement system is determined by obtaining the standard deviation of a series of repeated readings on a stable EUT. It should include typical variations that will occur in normal testing. This will only need to be considered if the measured result is close to the specification limit to the extent that variations due to the EUT may affect compliance.

RS

dB

REUT

Repeatability of EUT

dB

6406 HDJ

2004 11 30

28

7 Appendix Statistics and uncertainty primer

A stochastic variable or a random variable is a variable that may assume different values, at different trials or outcomes. A stochastic variable, as it may appear in measurements, may be fully characterized by a probability distribution function. A distribution function is a function, f(x) describing the probability of occurrence of the outcome of a random variable. For a continuous distribution, the probability that X falls in the interval x < X < x + dx equals f(x)dx. For a discrete distribution, such as the outcome of a throw with a dice, f(x) is a set of values. It is so that f ( x)dx = 1 .
all x

The expectation value (or mean value) of a distribution function is given by E( X ) =

all x

x f ( x)dx
2

The variance of a distribution function is given by V (X ) =

all x

(x E ( X ) )

f ( x)dx

The GUM method relies on the basic concept: The result is an estimate of the expectation value and the standard uncertainty is the squareroot of an estimate of variance. The Normal distribution or Gaussian distribution has a special significance. It can be shown (the Central Limit Theorem) that a combination of random variables with any well-behaved distribution functions will approach the Normal distribution. Hence, in many physical processes, where sources from the scale of molecular motion and up affect the outcome, quantities follow a Normal distribution. The Normal distribution is described by two parameters, and , and the probability distribution function is given by
1 N ( x; , ) = f ( x) = e 2

( x )2
22

For the Normal distribution, the expectation value is E(X) = and the variance is V(X) = 2.

Combining uncertainties
Given a model function connecting input quantities X 1 , X 2 ,K, X n and an output quantity Y by a mathematical function Y = f ( X 1 , K , X N ) , the propagation of uncertainties follows from a first order Taylor expansion of f: The result is given by y = f ( x1 ,K, xN ) The sensitivity coefficients are ci = The combined uncertainty of y is
6406 HDJ 2004 11 30

f X i

X 1 = x1 ,K, X N = x N

29 u 2 ( y ) = ci u ( xi , x j )c j
i =1 j =1 N N N

= ci2u 2 ( xi ) + 2 ci c j r ( xi , x j )u ( xi )u ( x j )
i =1 i =1 j =i +1

N 1 N

where r(xi, xj) is the correlation coefficient between xi and xj.

Estimates for the Normal distribution

When performing repeated observations of a physical quantity, say {x1 , x2 ,K, xn } , which we may expect is normally distributed, we can from the observations estimate the value of the quantities and of the underlying Normal distribution. It can be shown that the average value x = mental standard deviation s = 1 n xi is the best estimate for and the experin i =1

1 n (xi x )2 is the best estimate for . Hence, from re n 1 i =1 peated observations of the value of a quantity, the GUM method identifies, the result as the average x = x and the standard uncertainty as u(x) = s.

It should be noted that the standard uncertainty assigned is that of a single observation. If the measurand is specified as the average of n samples, this average is itself a model function, y = (x1 + x2 + L + xn ) n , and if the xi are independent, the standard uncertainty of y becomes 1 1 s . u( y) = u ( x1 ) 2 + u ( x2 ) 2 + L + u ( xn ) 2 = n s2 = n n n

Degrees of freedom
The estimate of from an experimental standard deviation based on n observations is itself subject to uncertainty. If the experiment of doing n observations were repeated, another value for s would be obtained. It can be shown, that the relative uncertainty of s with respect to the underlying being estimated by s, ie. u(s)/s, is approximately 1 2(n 1) . Hence, for an estimate based on n = 2 observations, the relative uncertainty to assign to s is 71 %, even for n = 10 observations, the relative uncertainty of s is 24 %. This uncertainty of the uncertainty is reflected in the concept of degrees of freedom. An experimental standard deviation based on n observations is assigned the number of degrees of freedom = n 1.

Cumulative standard uncertainty

The estimate of for the distribution of the value of the quantity may be improved by collecting information on the dispersion from several experiments. For example, under the assumption that the dispersion is the same, but the mean may have changed, the calculated experimental standard deviations may be combined into a pooled standard deviation with a larger number of degrees of freedom, and hence with a higher degree of confidence. The cumulative standard deviation is found from the formula

6406 HDJ

2004 11 30

30

sc =

v s
i =1 n i

2 i

v
i =1

, where the si and vi are experimental standard deviation and degrees of freedom
n

for the i-th experiment. The number of degrees of freedom for sc is vi = vi .

i =1

With a cumulative standard deviation, performing a single observation, again under the assumption that the underlying distribution is the same, may exploit the cumulative standard deviation as an estimate for , thus the standard uncertainty becomes u = sc. with vc degrees of freedom. The standard uncertainty of the average of m (say, a few) repeated observations becomes u = sc m with vc degrees of freedom.

Type-B estimates
If repeated measurements are impractical, impossible or uneconomical, the GUM method allows the use of an assumed probability distribution for the measurement quantity. In this case, only the imagination and reasonable assumptions set the limits for the possible probability distributions to assign the value of a quantity. The assignment of a probability distribution to a particular measurand will take its starting point in the professional experience of the user. What information is available? Are certain limits appropriate? Are some values more likely than others? For some physical situations, e.g. limited resolution of a digital display or oscillatory behaviour with known amplitude, but unknown phase, the distribution to assign is well established (see the table below). When the contribution to uncertainty can be put in probabilistic or combinatory terms, an important tool to derive probability distributions is Bayesian inference. When the probability distribution has been assigned, the value and standard uncertainty to use in the subsequent calculations are again the expectation value and the square-root of the variance. In the table below, the most common probability distributions, the value and standard uncertainty to use and the common use, is listed.

Expanded uncertainty
For a Normal distribution, the interval < x < + covers 68 % of the possible outcomes. This low level of coverage is often unsatisfactory for decision-making, and accreditation bodies have decided that the reported uncertainty should have a coverage probability of approximately 95 %. To reach 95 % coverage probability, a factor, the coverage factor k, should be multiplied to the standard uncertainty, u, to achieve the expanded uncertainty, denoted U. For a Normal distribution, a coverage factor k = 2, achieves 95,45 % coverage probability (the reason for phrase approximately 95 %). The uncertainty of the uncertainty is further reflected when the coverage probability to the uncertainty statement is found. If we use the average value, x = x , as the estimate for the mean , and the experimental standard deviation, s, as the estimate for , then we need the coverage factor k, so that the interval x k s < x < x + k s covers 95 % of the outcomes; note that the degrees of freedom is a parameter for this estimation. This is equivalent to a 95 % xx xx < k . It can be shown, that the quantity follows a coverage probability for k < s s

6406 HDJ

2004 11 30

31 Students t-distribution, and the values for k may be found tabulated versus the degrees of freedom, , and typically for various values of the coverage probability

10

25

50

100

13,97 4,53 3,31 2,87 2,65 2,37 2,28 2,11 2,05 2,025

Table of coverage factors to achieve 95,45 % coverage probability for standard uncertainty with degrees of freedom.

6406 HDJ

2004 11 30

32

7.1 Table of distribution functions often used for type-B evaluation: Distribution name
Rectangular; general

Parameters

E(X), V(X)

Use
A quantity with equal probability between limits a and b. E.g. from known limits of variation (could be by experience), but with no information on a specific probability distribution. A quantity with expectation 0 (and hence no influence on the result), but an estimated possible variation of a. E.g. rounding of a reading, manufacturers specification, estimates of error from experience. As for the rectangular, but where it is assumed more likely that the quantity takes a value in the centre than at the limits of the interval. E.g. markings of volumetric glassware or a difference of two readings. A sinusoidal variation in time between limits a and b. E.g. a temperature being regulated, or a quantity with unknown phase, e.g. the magnitude of a complex quantity. Counts, n, of discrete events or items in a limited sample, e.g. cars passing on a road in one hour, ticks on a Geiger counter per sec, bits in a transmission system.
Given m positives out of n, the proportion x ~ m/n is Beta distributed. E.g. error rates (m bit errors out of n transmitted bits), proportion of defects in a production based on investigation of a number of samples.

(b + a) 2 (b a) u = V (X ) = 12

x = E ( X )=

Rectangular; symmetric

x = E ( X )=0 u = V (X ) =
-a a

a 3

Triangular

x= u= x= u=

(b + a) 2 (b a) 24 (b + a) 2 (b a) 8

U-shaped

Poisson

x= n u= n

Beta

x= u=

1+ m 2+n (1 + m )(1 + n m )

(2 + n )2 (3 + n )

33

8 References
Some relevant sources of more detailed information is listed here: 1. 2. 3. 4. 5. 6. 7. 8. Measurement Uncertainty in Testing, Eurolab, Technical report No. 1/2002 A2LA Guide for the Estimation of Measurement Uncertainty in Testing, by T. Adams, A2LA 2002. EA guidelines on the expression of uncertainty in quantitative testing, EA-4/16, EA 2003. ISO 21748, Guide to the use of repeatability, reproducibility and trueness estimates in measurement uncertainty estimation, ISO/TC 69/SC 6 N456, ISO 2002. ISO 5725, Accuracy (trueness an precision) of measurement methods and results, ISO 2002. ISO Guide to the Expression of uncertainty in Measurement, ISO 1993. W. Press et al., Numerical Recipes, Cambridge University Press, New York (1989). L. Nielsen, Metrologia 35, 115 (1998); L. Nielsen, "Evaluation of measurements by the method of least squares", in Algorithms for Approximation IV, J. Levesley, I.J. Anderson, J.C. Mason (Eds), University of Huddersfield, 2002, 170-186.

6406 HDJ

2004 11 30