Sunteți pe pagina 1din 7

JOURNAL OF CLINICAL MICROBIOLOGY, Mar. 2003, p. 1080–1086

Vol. 41, No. 3

0095-1137/03/$08.00 0

Copyright © 2003, American Society for Microbiology. All Rights Reserved.

DOI: 10.1128/JCM.41.3.1080–1086.2003

International Proficiency Study of a Consensus L1 PCR Assay for the Detection and Typing of Human Papillomavirus DNA: Evaluation of Accuracy and Intralaboratory and Interlaboratory Agreement

Janet R. Kornegay, 1 Michel Roger,

2,3

Philip O. Davies, 4 Amanda P. Shepard,

2,3 *

1 Nayana A. Guerrero, 5

Belen Lloveras, 5 Darren Evans, 4 and Franc¸ois Coutle´e

Roche Molecular Systems Inc., Alameda, California 1 ; De´partement de Microbiologie et Infectiologie, Hoˆpital Notre-Dame du Centre Hospitalier de l’Universite´ de Montre´al, 2 and De´partement de Microbiologie et Immunologie, Universite´ de Montre´al, 3 Montre´al, Que´bec, Canada; BMI, London, United Kingdom 4 ; and Department of Pathology, CSU Bellvitge/Lab. Recerca Translacional, Institut Catala d’Oncologia, Barcelona, Spain 5

Received 24 May 2002/Returned for modification 26 August 2002/Accepted 9 December 2002

The PGMY L1 consensus primer pair combined with the line blot assay allows the detection of 27 genital human papillomavirus (HPV) genotypes. We conducted an intralaboratory and interlaboratory agreement study to assess the accuracy and reproducibility of PCR for HPV DNA detection and typing using the PGMY primers and typing amplicons with the line blot (PGMY-LB) assay. A test panel of 109 samples consisting of 29 HPV-negative (10 buffer controls and 19 genital samples) and 80 HPV-positive samples (60 genital samples and 20 controls with small or large amounts of HPV DNA plasmids) were tested blindly in triplicate by three laboratories. Intralaboratory agreement ranged from 86 to 98% for HPV DNA detection. PGMY-LB assay results for samples with a low copy number of HPV DNA were less reproducible. The rate of intralaboratory agreement excluding negative results for HPV typing ranged from 78 to 96%. Interlaboratory reliability for HPV DNA positivity and HPV typing was very good, with levels of agreement of >95% and kappa values of >0.87. Again, low-copy-number samples were more prone to generating discrepant results. The accuracy varied from 91 to 100% for HPV DNA positivity and from 90 to 100% for HPV typing. HPV testing can thus be accomplished reliably with PCR by using a standardized written protocol and quality-controlled reagents. The use of validated HPV DNA detection and typing assays demonstrating excellent interlaboratory agreement will allow investigators to better compare results between epidemiological studies.

Human papillomavirus (HPV) infection is a strong indepen- dent predictor for squamous intraepithelial lesions (SIL) and invasive cancer of the uterine cervix (10, 34). Women with persistent HPV infection are at highest risk for development of cervical SIL or evolution from low-grade SIL to higher-grade disease of the uterine cervix (8, 24). The causal association between HPV and cervical cancer was demonstrated by epide- miological investigations that used PCR, the most sensitive tool for HPV detection and typing, in cells collected from the uterine cervix (5). Because of the genetic polymorphism of HPV, consensus PCR assays have been utilized to amplify in one reaction the majority of known, as well as novel, anogenital HPV geno- types. The most widely used primer sets, the MY09/MY11/ HMB01, GP5 /GP6 , PGMY09/PGMY11, and SPF1/SPF2 consensus primers, target conserved sequences in the HPV L1 gene (1, 7, 12, 15, 18, 20, 23). To render the consensus PCR assays more feasible for large-scale testing and clinical appli- cation, convenient assays for the detection and typing of HPV have been developed for all primer sets. HPV amplicons gen- erated by PGMY primers can be easily detected and typed by

* Corresponding author. Mailing address: De´partement de Micro- biologie et Infectiologie, Hoˆpital Notre-Dame du Centre Hospitalier de l’Universite´ de Montre´al, 1560 Sherbrooke est, Montre´al, Que´bec H2L 4M1, Canada. Phone: (514) 890-8000, ext. 25162. Fax: (514) 412-7512. E-mail: francois.coutlee@ssss.gouv.qc.ca.

1080

a nonisotopic reverse hybridization assay, the line blot (LB)

assay (4, 13). Recently, a colorimetric microtiter plate-based enzyme immunoassay was also reported for screening of the broad spectrum of HPV amplified by the PGMY primers using

a generic probe mix (21). The proficiency of microbiology laboratory testing is gener- ally monitored by proficiency-testing programs that also allow the determination of test variability between laboratories (32). Implementation of proficiency testing panels is essential for unregulated molecular diagnostic tests. Proficiency panels have

been developed for the molecular diagnosis of several infec- tious agents (25, 32) and in research settings to monitor the performance of molecular virology laboratories (17). Thus far, no proficiency panel has been constructed and made available to the general research community for HPV testing. The va- lidity of HPV DNA detection and typing with PCR assays has not been thoroughly assessed. Epidemiological studies and vaccine clinical trials require the reliable and reproducible identification of genital HPV infection. Several studies have evaluated the intermethod vari- ation of HPV DNA detection (2, 11, 14, 19, 22, 26, 28, 30, 33). However, few studies have evaluated the intralaboratory and interlaboratory reproducibility of L1 consensus PCR assays although these assays have been widely used (6, 16, 19). The latter studies were conducted with in-house reagents and pro- tocols. Although the consensus L1 PCR assays are not com- mercially available, standardized reagents and protocols for

VOL. 41, 2003

INTERLABORATORY STUDY OF A CONSENSUS L1 PCR FOR HPV

1081

the PGMY-LB assay are available from Roche Molecular Sys- tems for research purposes. The use of standardized and re- producible protocols for HPV detection and typing could fa- cilitate the comparison of results between studies on HPV infection. In order to assess the accuracy and reproducibility of the PGMY-LB assay, three laboratories were invited to participate in a collaborative study to compare their abilities to detect the presence of and to genotype HPV DNA in blinded specimens. We report here the estimate of intralaboratory and interlabo- ratory reproducibility of the PGMY-LB assay for HPV DNA detection and typing on 109 specimens tested in triplicate by three independent laboratories under standardized conditions. This information is useful in view of the wider use of the PGMY-LB assay by numerous research groups, as well as its potential application in diagnostic laboratories for HPV detec- tion and typing.

MATERIALS AND METHODS

Selection of samples for the test panel. Roche Molecular Systems established the test panel of samples with the PGMY-LB assay. The quality control set included 30 synthetic samples: negative buffer samples devoid of HPV DNA sequences (n 10), samples containing 0.063 g of human genomic DNA per 5 l (catalog no. 1691112; Roche Molecular Biochemicals, Indianapolis, Ind.) spiked with 4,000 copies of HPV-16 DNA plasmid per 5 l (n 5), human DNA spiked with 400 copies of HPV-16 DNA plasmid per 5 l (n 5), human DNA spiked with 4,000 copies of HPV-45 DNA plasmid per 5 l (n 5), and human DNA spiked with 400 copies of HPV-45 DNA plasmid per 5 l (n 5). The panel also included 79 anonymous cervical samples collected with cytobrushes from 79 women attending clinics at the BMI hospitals in London. The latter samples came from a set of 90 samples that had been screened initially with the PGMY-LB assay at the BMI Health Services Laboratory and stored frozen. Roche Molecular Systems retested with the PGMY-LB assay the 90 clinical samples in duplicate to ensure the presence of HPV genotypes detected initially. Conrmation of initial PCR results was not obtained for 11 of the 90 samples (8 generated discordant results between PCR runs, 2 had been mislabeled, and 1 was only tested once at Roche), leaving 79 clinical samples (19 HPV negative and 60 HPV positive) in the panel. Of the 109 samples included in the prociency-testing panel, 29 were HPV- negative specimens (10 buffer controls and 19 genital samples) and 80 samples contained HPV DNA. The samples containing HPV DNA included 47 genital samples with one HPV type, 10 synthetic samples with low HPV DNA copy numbers, 10 synthetic samples with high HPV DNA copy numbers, 3 synthetic samples containing multiple HPV types, and 10 genital samples containing, in addition to the genotype(s) consistently detected in repeated PGMY-LB assay runs, at least one type that was not consistently detected. The latter 10 samples were tested four times by Roche Molecular Systems: an additional type was detected in one out of four runs (two samples), two out of four runs (seven samples), or three out of four runs (two samples). One sample contained two HPV types that were inconsistently detected in the four runs. HPV genotypes that were inconsistently detected in these 10 samples included types 42 (in three samples), 31 and 58 (in two samples each), and 16, 35, 33, and 84 (in one sample each). In addition to the 10 samples containing HPV-16 plasmids and 10 samples containing HPV-45 plasmids, there were a variety of HPV genotypes represented among the 60 HPV-positive samples. They included types 16 (in 10 samples), 18 and 51 (in 5 samples each), 31, 45, 52, 59, and 66 (in 4 samples each), 33, 39, 42, and 54 (in 3 samples each), 56, 68, 83, and 6 (in 2 samples each), and 35, 53, 58, 82, 84, and 73 (in 1 sample each). Of the three samples with multiple types, one contained types 16 and 31, one contained types 16, 18, 52, and 73, and another one contained types 16, 35, 42, and 82. The results from the detection of -globin were not considered and compared in this work. Study design. Three laboratories with different levels of experience with mo- lecular diagnostic methods participated in the study. Laboratory A had experi- ence in human genetic testing but was unfamiliar with molecular virology tests. Laboratory B was a diagnostic molecular microbiology laboratory specically trained to perform the PGMY-LB assay. Laboratory C was a diagnostic cytopa- thology laboratory familiar with commercialized molecular diagnostic techniques

but unfamiliar with the PGMY-LB assay. Standardized reagents comprising the PCR master mix, LB strips, and reagents were sent to each participating center along with a written standard operating procedure for the PGMY-LB assay.

Each participating center was asked to follow the protocol without modication. Three 20- l aliquots from each panel sample were coded and distributed on dry ice by Roche Molecular Systems to each of the three independent labora- tories. In the test panel, the HPV-negative or buffer controls were distributed randomly among the HPV-positive specimens. Five microliters of each sample

was used for HPV testing with the PGMY-LB assay. Each laboratory tested, in

different PCR runs, each of the 109 samples three times, blinded to the results obtained from Roche Molecular Systems, from the other laboratories, and from previous runs. Laboratories A and B also tested PGMY amplicons with the generic probe microplate assay. The study testing was conducted between June

and August 2000.

PGMY-LB assay. HPV DNA was amplied in each center under standard conditions with the L1 consensus HPV PGMY09/PGMY11 primer set, as pre- viously described (12). The amplication mixture contained 4 mM MgCl 2 , 50

mM KCl, 7.5 U of AmpliTaq Gold DNA polymerase (Applied Biosystems), 200

M concentrations each of dATP, dCTP, and dGTP, 600 M dUTP, 100 pmol of each biotinylated PGMY primer pool, and 5 pmol each of the 5 -biotinylated

-globin primers GH20 and PC04. HPV was amplied with the ultrasensitive prole that consisted of activation of AmpliTaq Gold at 95°C for 9 min, dena- turation for 1 min at 95°C, annealing for 1 min at 55°C, and extension at 72°C for 1 min for a total of 40 cycles. Amplication was followed by a 5-min terminal extension step at 72°C. Laboratory A used a Biometra Uno II thermocycler, laboratory B used a TC 9600 thermocycler, and laboratory C used an MJ Re- search PTC-1 thermocycler. Measures to avoid false-positive reactions due to contamination were strictly adhered to. HPV genotyping was performed with the reverse LB detection system as previously described (13). PCR products were denatured in 0.4 N NaOH and hybridized to an immobilized probe array con- taining probes for 27 HPV genotypes (types 6, 11, 16, 18, 26, 31, 33, 35, 39, 40, 42, 45, 51, 52, 53, 54, 55, 56, 57, 58, 59, 66, 68, 73, 82, 83, and 84). Positive hybridization was detected by streptavidin-horseradish peroxidase-mediated color precipitation on the membrane at the probe line. Generic probe microplate assay. Generic probe detection reactions were per- formed by using reagents from a PCR-ELISA DIG detection kit (Roche Mo- lecular Biochemicals) as described previously (21). Twenty microliters of dena- tured PCR products was added to the streptavidin-coated microtiter wells, followed by the addition of 200 l of hybridization buffer provided by Roche and 20 l of the denatured generic probe pool. Roche synthesized the digoxigenin- labeled generic probe pool by amplication of DNA from HPV types 11, 16, 18,

and 51 as described previously (21). Hybridization was performed at 37°C for 1 h.

Following color development, absorbance was measured at 405 nm and the background, dened as the average value of blank cells containing no PCR product, was subtracted from all values. A specimen was considered positive if

the corrected A 405 was greater than 0.5, negative if the value was less than 0.2,

and indeterminate in the range between 0.2 and 0.499.

Statistical methods. To ensure an independent evaluation of the reproduc- ibility of the assay, statistical analyses were performed by a scientist (F.C.) without nancial interests in Roche Molecular Systems. Results from the three laboratories were imported into a common database for comparison (Microsoft Excel). Results obtained by Roche Molecular Systems (the reference laboratory) were considered the gold standard(HPV DNA reference standard). Agree- ment for overall HPV positivity (HPV DNA positive irrespective of types iden- tied) and for type-specic positivity was calculated as the percentage of runs

with identical results for the presence or absence of HPV. Cohens unweighted

kappa statistic was calculated to adjust for chance agreement between sites or

between sites and HPV DNA reference standard results (9). In general, a kappa value of 0.75 indicates excellent agreement beyond chance, a kappa value between 0.40 and 0.75 indicates fair to good agreement, and a kappa value of 0.40 represents poor agreement. The reproducibility of repeated PCR assays (intralaboratory reproducibility)

was calculated for each site by comparing results from triplicate runs and calcu-

lating the crude percent agreement and the percent agreement considering only HPV-positive results. The intralaboratory reproducibility for the presence or absence of each of 27 genotypes was rst calculated by using crude typing results.

Since 10 samples contained HPV types that were not consistently detected between runs by the reference laboratory, agreement was recalculated by not considering the presence or absence of these additional types (see above). Interlaboratory agreement (interlaboratory reproducibility) was assessed by

pairwise comparisons of test results from the three laboratories calculating the crude percent agreement and the kappa statistic. Results obtained by the refer-

ence laboratory were not considered in these comparisons. To avoid combining

1082 KORNEGAY ET AL.

J. CLIN. MICROBIOL.

TABLE 1. Intralaboratory reproducibility of PGMY-LB for HPV DNA detection and HPV typing on triplicate testing of 109 samples by three laboratories

Test and laboratory

No. of assays with following results in runs 1/2/3 a :

/ /

/ /

/ /

/ /

Agreement (%) b

All results

HPV

results

HPV DNA detection c

A

65

11

4

29

86.2

81.3

B

77

3

0

29

97.3

96.3

C

78

2

0

29

98.2

97.5

HPV typing d

A

69

17 (14) e 5 (3) e 8 (2) e

6 (5) e 3 (1) e 6 (5) e

2,858

99.2 (99.4) e 99.7 (99.9) e 99.5 (99.8) e

75.0 (78.4) e 91.8 (95.8) e 85.7 (92.3) e

B

90

2,845

C

84

2,845

a Triplicate testing of 109 samples was done by each laboratory.

b Two measurements of percent agreement were calculated: one in which HPV-positive and HPV-negative results were considered (all results) and one in which specimens that had negative results in all triplicates were excluded.

c Results for the detection of HPV DNA irrespective of types detected.

d Typing results for 27 genotypes were considered.

e In parentheses are numbers of assays not taking into consideration types that were inconsistently detected by the reference laboratory.

intralaboratory and interlaboratory variability in the comparisons of the three laboratories, two strategies of analysis were considered. Laboratories were rst compared by considering the HPV DNA and typing results obtained on the initial run for each sample in each laboratory. This strategy allows evaluation of interlaboratory reproducibility when samples are only tested once, as is often the case. Laboratories were also compared by using a consensus denition of HPV results: the nal HPV result for a specimen was that obtained from at least two out of three runs for the presence of HPV DNA and HPV typing. This denition favors a higher degree of concordance between laboratories but takes into account triplicate results for each sample. Accuracy of HPV DNA detection and HPV typing was assessed by pairwise comparisons in contingency tables of each laboratory with the HPV DNA ref- erence standard results, using the two-sided McNemar chi-square analysis for matched-pair data. Agreement was determined rst by considering results of the rst aliquot for each sample and then by considering the consensus HPV results for each sample as explained above. Since the reference laboratory did not consistently detect a type in 10 samples, agreement was recalculated by not considering the presence or absence of these additional types as a discordant result. Results from laboratories A and B for the generic probe microplate assay were compared for HPV DNA positivity with results obtained with the LB assay. Indeterminate results were excluded from the calculation of agreement.

RESULTS

Results from triplicate runs with the PGMY-LB assay were compared in each laboratory to rst assess the reproducibility of HPV DNA detection irrespective of the type(s) detected. The intralaboratory agreement for HPV DNA detection of the PGMY-LB assay was high for each site (Table 1). The agree- ment between triplicates for each center was lower when only the HPV-positive results were considered. Two laboratories had intralaboratory levels of agreement of 97% for HPV DNA detection when all results were considered. A signicant difference was found in the number of discordant triplicates for HPV DNA detection among the three participating laborato- ries (Table 1): 15 (13.8%), 3 (2.8%), and 2 (1.8%) of 109 samples generated discordant triplicates for HPV DNA in laboratories A, B, and C, respectively (P 0.005; Pearson chi-square test). When laboratories were compared two by two, signicant differences were found, after the Bonferroni correction for a total of three comparisons, in the proportion of discordant triplicates between laboratories A and B and between laboratories A and C (P 0.009 and 0.003, respec- tively; Pearson chi-square test). The difference in the number

of discordant triplicates between laboratories B and C did not

reach statistical signicance (P 0.65; Pearson chi-square test).

An intralaboratory agreement analysis stratied by the

amount of HPV DNA introduced in the sample for laboratory

A that had the highest number of discordant triplicates re-

vealed that the presence of a low copy number of HPV DNA had an inuence on reproducibility. In this laboratory, 7 (70.0%) of the 10 samples containing small amounts of HPV DNA generated discordant triplicates for HPV DNA detec-

tion, as opposed to 8 of the 70 samples with clear signals in the PGMY-LB assay or high copy numbers (P 0.002; Fishers exact test). In order to estimate the intralaboratory agreement for HPV DNA typing, we compared the results of triplicate testing of 109 samples for 27 genotypes (2,943 type-specic results per center) (Table 1). Although the number of discordant tripli- cates was greater for typing results than HPV DNA detection

in all laboratories, agreement reached at least 99% when all

specimens were considered (Table 1). The high number of HPV type-specic negative results explains the greater intral- aboratory agreement obtained with HPV typing results than with generic HPV DNA detection despite a greater number of discordant triplicates. When concordant negative triplicates were excluded from the analysis, the level of agreement de- creased as shown in Table 1. We then considered in our eval- uation the fact that 10 samples contained HPV types that were not consistently detected between runs by the reference labo- ratory. In laboratory C, 7 (50%) of the 14 discordant triplicates contained HPV types not uniformly detected by the reference laboratory, thus leaving only 7 truly discordant triplicates for HPV typing (values in parentheses in Table 1). In laboratory B, four (50%) of the eight discordant triplicates contained HPV types not uniformly detected by the reference laboratory, leav- ing only four truly discordant triplicates for HPV typing. In laboratory A, 4 (17.3%) of the 23 discordant triplicates con- tained HPV types not uniformly detected by the reference laboratory, leaving 19 truly discordant triplicates for HPV typ- ing. When only truly discordant triplicates were considered, the intralaboratory agreement for HPV typing increased

VOL. 41, 2003

INTERLABORATORY STUDY OF A CONSENSUS L1 PCR FOR HPV

1083

TABLE 2. Interlaboratory agreement of PGMY-LB results for HPV DNA detection and HPV typing obtained on initial testing of 109 samples by three laboratories

Test and

laboratory pair

No. of concordant results

HPV (n 80)

HPV (n 29)

% Agreement

(no. identical/

no. tested)

Kappa

value

HPV DNA

detection

TABLE 3. Interlaboratory agreement of PGMY-LB results for HPV DNA detection and HPV typing obtained in three laboratories on triplicate testings of 109 samples using the consensus denition for HPV results

Test and

laboratory pair

No. of concordant results

HPV (n 80)

HPV (n 29)

% Agreement

(no. identical/

no. tested)

Kappa

value

HPV DNA

A-B

73

29

94 (102/109)

0.85

detection

B-C

79

29

99 (108/109)

0.98

A-B

76

29

96 (105/109)

0.91

A-C

72

29

93 (101/109)

0.83

B-C

80

29

100 (109/109)

1.00

 

A-C

76

29

96 (105/109)

0.91

HPV typing

A-B

68

29

95 (97/109)

0.75

HPV typing

B-C

71

29

93 (101/109)

0.81

A-B

75

29

95 (104/109)

0.87

A-C

69

29

90 (98/109)

0.78

B-C

80

29

100 (109/109)

1.00

 

A-C

76

29

96 (105/109)

0.91

slightly when all results were included in the calculation of agreement and ranged from 78.4 to 95.8% when only HPV- positive results were used (Table 1). When the numbers of truly discordant triplicates for HPV DNA typing were compared between participating sites, sig- nicant differences were found, after the Bonferroni correction for a total of three comparisons, between laboratories A and B and between laboratories A and C (P 0.006 and 0.054, respectively; Pearson chi-square test). Considering typing re- sults from laboratory A that showed the greatest intralabora- tory variability, 8 of 10 samples with low copy numbers of HPV DNA generated discordant triplicates, as opposed to 15 of 70 samples with clear signals or high HPV copy numbers (P 0.005; Fishers exact test). This relationship was not found for laboratories B and C, which had a higher level of agreement between triplicates (data not shown). Excluding specimens with low copy numbers of HPV types 45 and 16, discordant triplicates were distributed equally across several types. Repro- ducibility was perfect for the three samples with multiple HPV type infections for two laboratories (data not shown). One laboratory failed to detect an HPV type in one of the triplicates for two of the latter samples (data not shown). Interlaboratory reproducibility of the PGMY-LB assay was rst assessed by considering only the rst PCR run in each laboratory. Laboratory pairwise comparisons are shown in Ta- ble 2 for HPV DNA detection and typing. Agreement for HPV DNA positivity between laboratory pairs A and B, B and C, and A and C reached 94% (102 of 109 samples, kappa value of 0.85), 99% (108 of 109 samples, kappa value of 0.98), and 93% (101 of 109 samples, kappa value of 0.83), respectively. Agree- ment for HPV typing results between laboratory pairs A and B, B and C, and A and C reached 94% (102 of 109 samples, kappa value of 0.81), 97% (106 of 109 samples, kappa value of 0.88), and 93% (101 of 109 samples, kappa value of 0.80), respec- tively. Interlaboratory reproducibility of the PGMY-LB assay was then assessed by using a consensus denition of HPV positivity (two of three runs with identical results), as explained in Ma- terials and Methods. Laboratory pairwise comparisons are shown in Table 3 for HPV DNA detection and typing. Inter- laboratory reliability for HPV DNA positivity and HPV typing was excellent, with levels of agreement of 95% and kappa

values of 0.87, including perfect kappa values for HPV DNA detection and HPV typing between laboratories B and C. Of the four samples testing negative in laboratory A but positive in the two other laboratories, two contained low copy numbers of HPV-16 or -45. Low-copy-number samples were more prone to generate discordant results in laboratory A, since 2 (20%) of 10 low-copy-number samples versus 2 (3%) of 70 other HPV-positive samples were misclassied by laboratory A (P 0.07; Fishers exact test). The accuracy of the PGMY-LB assay was measured by com- paring results obtained from each laboratory with those ob- tained at Roche Molecular Systems (HPV DNA reference standard) for 109 samples. Additional HPV types inconsis- tently detected by Roche in multiple testing in 10 samples were not considered in this analysis. For the 29 HPV-negative sam- ples, only 2 (0.8%) of 261 assays scored positive for HPV DNA. One specimen that tested falsely positive for HPV-51 was preceded by an HPV-51-positive sample, while HPV-31 was responsible for the other false-positive result. Since sam- ples were tested in triplicate, results from the rst run and from a consensus denition of HPV positivity were compared sep- arately to the HPV DNA reference standard results, as shown in Table 4. The percentages of agreement between each labo- ratory and the HPV DNA reference standard varied from 91 to 100% for HPV DNA positivity and from 90 to 100% for HPV typing. Laboratory B correctly identied all HPV-positive sam- ples using results from the rst assay or the consensus deni- tion of HPV positivity. HPV types were identied correctly in all samples when the consensus denition of HPV positivity was used and in 79 (98.8%) samples when the rst run only was considered. Similarly, laboratory C correctly identied 99% of HPV-positive samples and correctly identied HPV types for nearly all samples except three (3.8%) and one (1.3%) when the rst run and the consensus denition of HPV positivity were used, respectively. When the rst sample was considered, laboratory A failed to identify 7 (9%) of 80 HPV-positive samples and failed in HPV typing for 8 (10%) for 80 HPV- positive samples. Four (57.1%) of the seven HPV-positive samples that were misclassied by laboratory A in the rst run contained low copy numbers of HPV DNA (data not shown). The agreement for HPV-positive samples, which can be con-

1084 KORNEGAY ET AL.

J. CLIN. MICROBIOL.

TABLE 4. Accuracy of HPV DNA detection and typing results obtained with PGMY-LB for 109 samples by three laboratories

Laboratory and HPV standard

No. of correct PGMY-LB results (% accuracy)

No. of samples

HPV DNA detection

HPV typing

 

1st assay

Consensus

1st assay

Consensus

A

 

HPV

positive

80

73 (91)

76 (95)

72 (90)

76 (95)

HPV

negative

29

29 (100)

29 (100)

29 (100)

29 (100)

All samples

109

102 (94)

105 (96)

101 (93)

105 (96)

B

 

HPV

positive

80

80 (100)

80 (100)

79 (99)

80 (100)

HPV

negative

29

29 (100)

29 (100)

29 (100)

29 (100)

All samples

109

109 (100)

109 (100)

108 (99)

109 (100)

C

 

HPV

positive

80

79 (99)

80 (100)

77 (96)

79 (99)

HPV

negative

29

29 (100)

29 (100)

29 (100)

29 (100)

All samples

109

108 (99)

109 (100)

106 (97)

108 (99)

sidered the sensitivity of the PGMY-LB assay compared to the HPV DNA reference standard, ranged from 95 to 100% for HPV DNA detection and typing using the consensus denition of HPV results and from 90 to 100% using results from the rst test (Table 4). Samples containing multiple HPV types or with high copy numbers of HPV DNA were all appropriately clas- sied by each site (data not shown). In laboratories A and B, amplicons generated by PGMY primers were tested with the LB assay and also with the generic probe microplate assay. Since two laboratories tested 109 sam- ples in triplicate, 654 assay results could be compared. Inde- terminate results for the generic probe microplate assay were obtained for 59 samples (30 HPV-positive samples and 29 HPV-negative samples). When these indeterminate results were excluded, 420 assays were HPV positive by both tests, 149 were negative by both tests, 11 were HPV positive with the generic probe microplate assay only, and 15 were HPV-posi- tive with the LB assay only. The agreement between the tests for detection of HPV amplicons was 95.6% (569 of 595 re- sults), for an excellent kappa value of 0.89.

DISCUSSION

Interlaboratory agreement is a concern for any HPV DNA detection method that has not undergone extensive compara- tive eld testing. The HPV detection and typing method also has to be accurate. We evaluated the intralaboratory and in- terlaboratory reproducibility of the PGMY-LB assay per- formed by three laboratories with different levels of experience with PCR testing. Our study demonstrates the high reliability of the PGMY-LB assay when a laboratory is well trained to perform the assay. Laboratories without experience with in- house PCR tests can also use the PGMY-LB assay, since lab- oratory C generated reproducible and accurate results. The use of standardized reagents and clearly written protocols was instrumental in obtaining these excellent results with the PGMY-LB assay. The lowest intralaboratory, interlaboratory, and accuracy levels were found in a laboratory with experience with PCR testing. That laboratory had determined in previous experiments that the PGMY-LB assay with their thermocycler

required higher concentrations of -globin primers in the PCR master mix (data not shown). It was demonstrated previously that coamplication of -globin with HPV DNA can result in competition and impede PGMY-LB assay performance, ex- plaining in part the discordant results obtained with low-copy- number synthetic samples (3, 4). The importance of utilizing the same instruments and parameters set by validation studies of a test is emphasized by these results. Each of the three laboratories used a different make and model of thermocycler, while the PGMY-LB assay protocol was developed and opti- mized for only one, the TC 9600 from Perkin-Elmer. As an alternative explanation, specimen heterogeneity may have ac- counted for the difference in performance between laborato- ries. We demonstrated on synthetic samples that specimens with small amounts of HPV DNA were prone to generate discor- dant results between triplicates. Since synthetic samples do not contain the amplication modier found in clinical specimens, we also included in the prociency panel anonymous cytobrush samples. Ten clinical samples contained at least one HPV type not consistently detected by the reference laboratory. For all of these samples, the type detected intermittently by the refer- ence laboratory was also detected by at least one of the par- ticipating laboratories. This phenomenon could result from the presence of a low copy number of HPV DNA or inhibitors and illustrates the difculty of using clinical samples in a prociency test. The intralaboratory agreement was excellent for all three laboratories. It increased when we did not consider variable HPV results obtained in the certication panel by the refer- ence laboratory. Some groups have reported greater reproduc- ibility of PCR for HPV testing (22, 27). In those studies, du- plicates instead of triplicates were tested, only one laboratory was evaluated, and the panel included more HPV-negative samples than did our study. Our level of intralaboratory agree- ment for HPV DNA detection and typing corresponds to a previous evaluation by Daniel et al. (6). They also reported that the reproducibility of HPV typing with MY09/MY11 in- creased in the presence of greater HPV loads. We found that for the laboratory with several discordant triplicates, low-viral-

VOL. 41, 2003

INTERLABORATORY STUDY OF A CONSENSUS L1 PCR FOR HPV

1085

load synthetic samples were prone to generating discordant results. As reported by others, we found a lower agreement for HPV typing than for HPV DNA detection (6, 19). We calcu- lated not only overall agreement but also agreement with re- gard only to HPV-positive results because of the large number of concordant HPV-negative assays that articially increased the level of agreement for HPV typing. As previously reported, agreement was strong for samples with multiple HPV types, although we only analyzed three such specimens (6, 19). Two studies reported the interlaboratory agreement of PCR for HPV detection using the MY09/MY11 primer pair (16, 22). In one study, 33 samples were tested in two laboratories, while in the other study, 70 anal samples were analyzed in two lab- oratories. Our evaluation of agreement was consistent with the estimates from these previous reports for HPV DNA positivity (83 and 96%) and HPV typing (90 and 88%). The accuracy of the PGMY-LB assay was excellent. How- ever, specimens may better agree with a reference standard when the same test is used to validate and test the panel. The low rate of false-positive results in the 29 HPV-negative sam- ples suggests a low rate of contamination in our study. One of the false-positive specimens was preceded by an HPV-51-pos- itive sample. Contamination may have occurred after ampli- cation during the hybridization of amplicons with the LBs, since the same amplicons hybridized with the generic probe tested negative. Another study evaluated the accuracy of the GP5 /GP6 L1 consensus primer pair and reported excellent accuracy for 50 samples tested in four laboratories, ranging from 92 to 100% for detection of HPV DNA, a rate similar to our results (19). That study also reported that most discrepant results were false-negative results from HPV-positive samples (19). The reproducibility of the PGMY-LB assay in our study was also similar to that of the FDA-approved hybrid capture assay from Digene (29). The accuracy of hybrid capture in detecting high-risk HPV types ranged for three laboratories from 88 to 92%, and interlaboratory agreement reached 90%. Nonetheless, our results should be interpreted cautiously, with the following caveats. Few samples with multiple type infections were included in our evaluation. These samples have been demonstrated to generate intralaboratory variability in one study (6). Future evaluations of the PGMY-LB assay should include multiple type infections with some types in small amounts. Our clinical samples were obtained without knowledge of cytopathological diagnoses. Assay reproducibil- ity and accuracy may differ when samples from different pa- tient populations are selected, especially since viral load is related to cervical disease status and assay reproducibility (31). We demonstrated here that there is a very strong interlabo- ratory agreement when the PGMY-LB assay is used. HPV testing can thus be reliably accomplished with PCR by using a validated and clearly written protocol with quality-controlled reagents. Although the assay procedure is well standardized, an initial certication panel to ensure the reliability of results would be desirable before a laboratory performs the PGMY-LB assay on a regular basis. Our study stresses the importance of using standardized reagents and protocols and the establishment of prociency panels for the comparability of results between studies. More studies on performance stan- dards that include samples with multiple type infections, with inhibitors and low viral load infections, need to be performed.

ACKNOWLEDGMENTS

We thank Diane Gaudreault and France Dion for testing samples with the PGMY-LB assay, Pierre Forest for training participants, and BMI Health Services for the provision of clinical samples. Roche Molecular Systems supplied reagents for the PGMY-LB assay. F.C. and M.R. hold a clinical career award supported by the Fonds de Recherche en Sante´ du Que´bec.

REFERENCES

1. Bauer, H. M., Y. Ting, C. E. Greer, J. C. Chambers, C. J. Tashiro, J. Chi- mera, A. Reingold, and M. M. Manos. 1991. Genital human papillomavirus infection in female university students as determined by a PCR-based method. JAMA 265:472477.

2. Cope, J. U., A. Hildesheim, M. H. Schiffman, M. M. Manos, A. T. Lorincz,

R.

D. Burk, A. G. Glass, C. Greer, J. Buckland, K. Helgesen, D. R. Scott,

M.

E. Sherman, R. J. Kurman, and K. L. Liaw. 1997. Comparison of the

Hybrid Capture tube test and PCR for detection of human papillomavirus DNA in cervical specimens. J. Clin. Microbiol. 35:22622265.

3. Coutle´e, F., P. Gravitt, J. Kornegay, C. Hankins, H. Richardson, N. Lapointe, H. Voyer, The Canadian Women’s HIV study Group, and E. Franco. 2002. Use of PGMY primers in L1 consensus PCR improves detec-

tion of human papillomavirus DNA in genital samples. J. Clin. Microbiol.

40:902907.

4. Coutle´e, F., P. Gravitt, H. Richardson, C. Hankins, E. Franco, N. Lapointe,

H. Voyer, and The Canadian Women’s HIV Study Group. 1999. Nonisotopic

detection and typing of human papillomavirus DNA in genital samples by the line blot assay. J. Clin. Microbiol. 37:18521857.

5. Coutle´e, F., M. H. Mayrand, D. Provencher, and E. Franco. 1997. The future of HPV testing in clinical laboratories and applied virology research. Clin. Diagn. Virol. 8:123141.

6. Daniel, R. W., L. Ahdieh, D. Hayden, S. Cu-Uvin, and K. V. Shah. 2000. Intra-laboratory reproducibility of human papillomavirus identication in

cervical specimens by a polymerase chain reaction-based assay. J. Clin. Virol.

19:187193.

7. de Roda, H., J. M. Walboomers, A. J. van den Brule, C. J. Meijer, and P. J.

Snijders. 1995. The use of general primers GP5 and GP6 elongated at their

3 ends with adjacent highly conserved sequences improves human papillo-

mavirus detection by PCR. J. Gen. Virol. 76:10571062.

8. Ferenczy, A., and E. L. Franco. 2002. Persistent human papillomavirus in- fection and cervical neoplasia. Lancet Oncol. 3:1116.

9. Fleiss, J. L. 1981. Statistical methods for rates and proportions. John Wiley and Sons Inc., New York, N.Y.

10. Franco, E. L. F. 1996. Epidemiology of anogenital warts and cancer, p. 597623. In R. Reid and A. T. Lorincz (ed.), Human papillomavirus I. W. B. Saunders, Philadelphia, Pa.

11. Gravitt, P., A. Hakenewerth, and J. Stoerker. 1991. A direct comparison of methods proposed for use in widespread screening of human papillomavirus

infection. Mol. Cell. Probes 5:6572.

12. Gravitt, P., C. L. Peyton, T. Q. Alessi, C. Wheeler, F. Coutle´e, A. Hildesheim,

M. Schiffman, D. R. Scott, and R. J. Apple. 2000. Improved amplication of

genital human papillomaviruses. J. Clin. Microbiol. 38:357361.

13. Gravitt, P., C. L. Peyton, R. J. Apple, and C. Wheeler. 1998. Genotyping of 27 human papillomavirus types by using L1 consensus PCR products by

single-hybridization, reverse line blot detection method. J. Clin. Microbiol.

36:30203027.

14. Guerrero, E., R. W. Daniel, F. X. Bosch, X. Castellsague´, N. Munoz, M. Gili,

P. Viladu, C. Navarro, M. L. Zubiri, N. Ascunce, L. C. Gonzales, L. Tafur, I.

Izaraugaza, and K. V. Shah. 1992. Comparison of Virapap, Southern blot hybridization, and polymerase chain reaction methods for human papillo-

mavirus identication in an epidemiological investigation of cervical cancer. J. Clin. Microbiol. 30:29512959.

15. Hildesheim, A., M. H. Schiffman, P. E. Gravitt, A. G. Glass, C. E. Greer, T. Zhang, D. R. Scott, B. B. Rush, P. Lawler, M. E. Sherman, R. J. Kurman, and M. M. Manos. 1994. Persistence of type-specic human papillomavirus infection among cytologically normal women. J. Infect. Dis. 169:235240.

16. Hsing, A. W., R. D. Burk, K. L. Liaw, C. J. Chen, T. Zhang, M. Schiffman,

C.

E. Greer, S. L. You, C. Y. Hsieh, T. W. Huang, T. C. Wu, T. J. O’Leary,

J.

D. Seidman, and M. M. Manos. 1996. Interlaboratory agreement in a

polymerase chain reaction-based human papillomavirus DNA assay. Cancer Epidemiol. Biomed. Prev. 5:483484.

17. Jackson, J. B., J. Drew, H. J. Lin, P. Otto, J. W. Bremer, F. B. Hollinger,

S. M. Wolinski, The ACTG PCR Working Group, and The ACTG PCR

Virology Laboratories. 1993. Establishment of a quality assurance program for human immunodeciency virus type 1 DNA polymerase chain reaction

assays by the AIDS clinical trials group. J. Clin. Microbiol. 31:31233128.

18. Jacobs, M. V., P. J. F. Snijders, A. J. C. van den Brule, T. J. M. Helmerhorst,

C. Meijer, and J. Walboomers. 1997. A general primer GP5 /GP6 -medi-

ated PCR-enzyme immunoassay method for rapid detection of 14 high-risk and 6 low-risk human papillomavirus genotypes in cervical scrapings. J. Clin. Microbiol. 35:791795.

1086 KORNEGAY ET AL.

19. Jacobs, M. V., P. J. F. Snijders, F. J. Voorhorst, J. Dillner, O. Forslund, B. Johansson, M. V. K. Doeberitz, C. J. L. M. Meijer, T. Meyer, I. Nindl, H. Pfister, E. Stockfleth, A. Strand, G. Wadell, and J. M. M. Walboomers. 1999. Reliable high risk HPV DNA testing by polymerase chain reaction: an intermethod and intramethod comparison. J. Clin. Pathol. 52:790.

20. Kleter, B., L. J. van Doorn, L. Schrauwen, A. Molijn, S. Sastrowijoto, J. ter Schegget, J. Lindeman, B. ter Harmsel, M. Burger, and W. Quint. 1999. Development and clinical evaluation of a highly sensitive PCR-reverse hy- bridization line probe assay for detection and identication of anogenital human papillomavirus. J. Clin. Microbiol. 37:25082517.

21. Kornegay, J. R., A. P. Shepard, C. Hankins, E. Franco, N. Lapointe, H. Richardson, The Canadian Women’s HIV Study Group, and F. Coutle´e. 2001. Nonisotopic detection of human papillomavirus DNA in clinical spec- imens using a consensus PCR and a generic probe mix in an enzyme-linked immunoassay format. J. Clin. Microbiol. 39:35303536.

22. Kuypers, J. M., C. W. Critchlow, P. E. Gravitt, D. A. Vernon, J. B. Sayer,

M. M. Manos, and N. B. Kiviat. 1993. Comparison of dot lter hybridization,

Southern transfer hybridization, and polymerase chain reaction amplication for diagnosis of anal human papillomavirus infection. J. Clin. Microbiol.

31:10031006.

23. Manos, M. M., Y. Ting, D. K. Wright, A. J. Lewis, T. Broker, and S. M.

Wolinski. 1989. Use of the polymerase chain reaction amplication for the detection of genital human papillomaviruses. Cancer Cells 7:209215.

24. Nobbenhuis, M. A., J. M. Walboomers, T. J. Helmerhorst, L. Rozendaal,

A. J. Remmink, E. K. Risse, H. C. van der Linden, F. J. Voorhorst, P.

Kenemans, and C. J. Meijer. 1999. Relation of human papillomavirus status to cervical lesions and consequences for cervical-cancer screening: a pro- spective study. Lancet 354:2025.

25. Noordhoek, G. T., A. H. Kolk, G. Bjune, D. Catty, J. W. Dale, P. E. Fine, P. Godfrey-Faussett, S. N. Cho, T. Shinnick, and S. B. Svenson. 1994. Sensi-

tivity and specicity of PCR for detection of Mycobacterium tuberculosis: a blind comparison study among seven laboratories. J. Clin. Microbiol. 32:277

284.

J. CLIN. MICROBIOL.

26. Peyton, C. L., M. Schiffman, A. T. Lorincz, W. C. Hunt, I. Mielzynska, C.

Bratti, S. Eaton, A. Hildesheim, L. A. Morera, A. C. Rodriguez, R. Herrero,

M. E. Sherman, and C. M. Wheeler. 1998. Comparison of PCR- and hybrid

capture-based human papillomavirus detection systems using multiple cer- vical specimen collection strategies. J. Clin. Microbiol. 36:32483254.

27. Qu, W., G. Jiang, Y. Cruz, C. J. Chang, G. Y. Ho, R. S. Klein, and R. D. Burk. 1997. PCR detection of human papillomavirus: comparison between MY09/ MY11 and GP5 /GP6 primer systems. J. Clin. Microbiol. 35:13041310.

28. Schiffman, M. H., H. M. Bauer, and A. T. Lorincz. 1991. Comparison of Southern blot hybridization and polymerase chain reaction methods for the detection of human papillomavirus DNA. J. Clin. Microbiol. 29:573577.

29. Schiffman, M. H., N. B. Kiviat, R. D. Burk, K. V. Shah, R. W. Daniel, R. Lewis, J. Kuypers, M. M. Manos, D. R. Scott, M. E. Sherman, R. J. Kurman,

M. H. Stoler, A. G. Glass, B. B. Rush, I. Mielzynska, and A. T. Lorincz. 1995.

Accuracy and interlaboratory reliability of human papillomavirus DNA test- ing by hybrid capture. J. Clin. Microbiol. 33:545550.

30. Smits, H. L., L. J. M. Bollen, S. P. Tjong-A-Hung, J. Vonk, J. van der Vendel,

F. J. W. Ten Kate, J. A. Kaan, B. W. Mol, and J. ter Schegget. 1995.

Intermethod variation in detection of human papillomavirus DNA in cervical smears. J. Clin. Microbiol. 33:26312636.

31. Swan, D. C., R. A. Tucker, G. Tortolero-Luna, M. F. Mitchell, L. Wideroff,

E. R. Unger, R. A. Nisenbaum, W. C. Reeves, and J. P. Icenogle. 1999.

Human papillomavirus (HPV) DNA copy number is dependent on grade of cervical disease and HPV type. J. Clin. Microbiol. 37:10301034.

32. Tang, Y. W., and D. H. Persing. 1999. Molecular detection and identication of microorganisms, p. 215244. In P. R. Murray, E. J. Baron, M. A. Pfaller, F. C. Tenover, and R. H. Yolken (ed.), Manual of clinical microbiology, 7th ed. American Society for Microbiology, Washington, D.C.

33. Vernon, S. D., E. R. Unger, and D. Williams. 2000. Comparison of human papillomavirus detection and typing by cycle sequencing, line blotting, and Hybrid Capture. J. Clin. Microbiol. 38:651655.

34. Villa, L. L. 1997. Human papillomaviruses and cervical cancer. Adv. Cancer Res. 71:321341.