Sunteți pe pagina 1din 20

CUMITECH

CUMULATIVE
IN CLINICAL
TECHNIQUES
MICROBIOLOGY
AND PROCEDURES
31
FEBRUARY 1997

Verification and
Validation of
Procedures in the
Clinical Microbiology
Laboratory
B. LAUREL ELDER, SHARON A. HANSEN. JAMES A. KELLOGG,
FREDERIC J. MARSIK, and RONALD J. ZABRANSKY

COORDINATING EDITOR
BRENDA W. McCURDY

XILlERIC;\iX SOCIETY FOR MICROBIOLOGY


Cumitech 1A l Blood Cultures II l June 1982
Cumitech 2A l Laboratory Diagnosis of Urinary Tract Infections l March 1987
Cumitech 3A l Quality Control and Quality Assurance Practices in Clinical Microbiology l May 1990
Cumitech 4A l Laboratory Diagnosis of Gonorrhea l April 1993
Cumitech 5A l Practical Anaerobic Bacteriology l December 1991
Cumitech 6A l New Developments in Antimicrobial Agent Susceptibility Testing: a Practical Guide l
February 1991
Cumitech 7A l Laboratory Diagnosis of Lower Respiratory Tract Infections l September 1987
Cumitech 8 l Detection of Microbial Antigens by Counterimmunoelectrophoresis l December 1978
Cumitech 9 l Collection and Processing of Bacteriological Specimens l August 1979
Cumitech 10 l Laboratory Diagnosis of Upper Respiratory Tract Infections l December 1979
Cumitech 11 l Practical Methods for Culture and Identification of Fungi in the Clinical Microbiology
Laboratory l August 1980
Cumitech 12A l Laboratory Diagnosis of Bacterial Diarrhea l April 1992
Cumitech 13A l Laboratory Diagnosis of Ocular Infections l September 1994
Cumitech 14A l Laboratory Diagnosis of Central Nervous System Infections l March 1993
Cumitech 15A l Laboratory Diagnosis of Viral Infections l August 1994
Cumitech 16A l Laboratory Diagnosis of the Mycobacterioses l October 1994
Cumitech 17A l Laboratory Diagnosis of Female Genital Tract Infections l June 1993
Cumitech 18 l Laboratory Diagnosis of Hepatitis Viruses l January 1984
Cumitech 19 l Laboratory Diagnosis of Chlamydial and Mycoplasmal Infections l August 1984
Cumitech 20 l Therapeutic Drug Monitoring: Antimicrobial Agents l October 1984
Cumitech 21 l Laboratory Diagnosis of Viral Respiratory Disease l March 1986
Cumitech 22 l Immunoserology of Staphylococcal Disease l August 1987
Cumitech 23 l Infections of the Skin and Subcutaneous Tissues l June 1988
Cumitech 24 l Rapid Detection of Viruses by Immunofluorescence l August 1988
Cumitech 25 l Current Concepts and Approaches to Antimicrobial Agent Susceptibility Testing l
December 1988
Cumitech 26 l Laboratory Diagnosis of Viral Infections Producing Enteritis l September 1989
Cumitech 27 l Laboratory Diagnosis of Zoonotic Infections: Bacterial Infections Obtained from Com-
panion and Laboratory Animals l February 1996
Cumitech 28 l Laboratory Diagnosis of Zoonotic Infections: Chlamydial, Fungal, Viral, and Parasitic
Infections Obtained from Companion and Laboratory Animals l February 1996
Cumitech 29 l Laboratory Safety in Clinical Microbiology l July 1996

Cumitech 30 l Selection and Use of Laboratory Procedures for Diagnosis of Parasitic Infections of the
Gastrointestinal Tract l September 1996
Cumitech 31 l Verification and Validation of Procedures in the Clinical Microbiology Laboratory l
February 1997
Cum/tech should be cited as follows, e g.: Elder, B L., S. A Hansen, J. A. Kellogg, F. J Marslk, and R. J.
Zabransky 1997. Venflcatlon and Valldatlon of Procedures tn the Clinical MIcrobiology Laboratory. Coordinating ed.,
B. W. McCurdy. Amencan Society for Microbiology, Washington, D C.

Editorial Board for ASM Cumitechs: Frederick S Nolte, Charman; Lorraine Clarke, Janet Handler, Brenda W
McCurdy, Alice S. Welssfeld, and Stephen Young

The purpose of the Cumltech series is to provide consensus recommendations by the authors as to appropriate state-of-the-art
operating procedures for cllnlcal mlcroblology laboratones which may lack the facllltles for fully evaluating routine or new methods
The procedures given are not proposed as “standard” methods

CopyrIght 0 1997 American Society for MIcrobIology


1325 Massachusetts Ave N W
Washington, DC 20005
VERIFICATION AND VALIDATION OF PROCEDURES IN
THE CLINICAL MICROBIOLOGY LABORATORY
B. LAUREL ELDER, CompuNet Clinical Laboratories, Moraine, Ohio 45439

SHARON A. HANSEN, Food and DIU~ Admmistration, Rockvdle, Maryland 20850

JAMES A. KELLOGG, Department of Pathology, York Hospital, York, Pennsylvania 174057198

FREDERIC J. MARSIK, Food and Drug’ Admmistration, Rockvllle, Malyland 20857

RONALD J. ZABRANSKY, Pathology and Laboratov Medicine, VA Medical Center, Cleveland, Ohio 44106

COORDINATING EDITOR

BRENDA WV. McCURDY, VA Medical Center, Detroit, Michigan 48201-l 932

The key desirable attribute of a laboratory test reagent tests and the most complex of instruments
is its ability to produce accurate and precise generating a variety of analytic results and inter-
results consistently over extended periods of time pretations.
with an appropriately rapid turnaround time so The guidelines and suggestions found in this
that the test results are of clinical utility. Accuracy Cumitech offer information that users of tests may
and precision are readily measurable and have consider in their efforts to improve the general
become the hallmarks for comparing tests and operation or quality of their laboratory services.
laboratories, and they have even been used as These guidelines should not be considered mini-
criteria for granting certification, licensure, or mal criteria or policy which regulatory, accredit-
accreditation. Furthermore, when a new labora- ing, licensing, or standard-setting agencies use in
tory test is introduced, these factors may be assessing either the compliance of a laboratory or
weighed with other issues, such as clinical rele- the accuracy of individual tests or instruments that
vance, cost, instrumentation, and ease of perfor- may be considered for acceptance by laboratory
mance. The laboratory must have in place a personnel.
detailed method for analysis of new tests to eval- The information in this Cumitech is not all-
uate these parameters before the tests are intro- inclusive. Much information can be found in the
duced as new offerings or as replacements for literature regarding the verification and validation
of test methods; more often, however, this infor-
older procedures (verification). In addition, the
laboratory must have a systematic approach for mation applies to quantitative analyses, as are
commonly found in the chemistry section of the
the ongoing review of established tests which
laboratory. When this process is applied to mi-
demonstrates that the testing process and the
crobiology, where qualitative results are more
consistency of results have not changed and that
common, where subjective interpretations are re-
testing personnel remain competent to perform quired, or where the results include the identifi-
tests and report results (validation). cation of microorganisms with biological varia-
The processes of verification and validation are tion, more flexibility is required. Thus, the focus of
part of the quality assurance program for the this Cumitech is on the common qualitative and
laboratory. Laboratories are required to establish semiquantitative test procedures performed in the
policies and procedures to maintain or improve clinical microbiology laboratory.
the reliability, efficiency, and clinical utility of
laboratory tests. The purpose of this Cumitech is
to provide guidance on the necessary criteria that DEFINITIONS
may be required as new tests are considered for The primary processes addressed in this
clinical use and as old tests are reevaluated for Cumitech are the verification of a new test or
their clinical relevance. This document is a guide method prior to introduction into the laboratory
by which a laboratory may verify the performance and the ongoing validation of the performance of
of a new method or undertake the revision of an existing test methodology. However, before these
established method to extend its applicability or processes can be discussed in detail, relevant
improve its performance. The guidelines within definitions must be established.
this Cumitech apply equally to simple single- Accuracy (20,23,24) : Technical accuracy is the
2 ELDER ET AL. CUMITECH 31

nearness of an individual measurement to the true methodology change, or a test performed in-house
value, as determined by a reference method. Cl&z- that was previously performed at a reference
ical accuracy is the overall ability of a test to both laboratory. Such tests include detection or identi-
rule in and rule out an analyte or a specific fication of a totally new analyte, the use of totally
disease. Accuracy is synonymous with test effi- new methodology, a new approach to detecting an
ciency and can be expressed mathematically as a analyte, a change from a manual method to an
percent: automated one, a new application of existing
technology, or the test of a new matrix (old
number of correct results analyte in a different specimen).
x 100 Old test: An old test is any procedure for
total number of results
detection of a disease, analyte, or characteristic
Analyte (20,23): The component of a specimen (e.g., antimicrobial susceptibility) that had been in
or organism which is to be measured or-demon- use prior to September 1, 1992, the effective date
strated. An analyte may be a particular antigen, of the Clinical Laboratory Improvement Amend-
antibody, nucleic acid, organism, enzyme, species, ments of 1988 (CLIA ‘88).
or metabolic product. Precision (20, 23): A measure of the extent to
Gold standard: The best available approxima- which replicate analyses (using identical proce-
tion of the truth (20). It is a commonly used term, dures) of a homogeneous analyte agree with each
generally indicating a test method currently ac- other. Precision implies freedom from inconsis-
cepted as reasonably, but not necessarily 10076, tency and random error but does not guarantee
accurate (8,23). It is used as the reference method accuracy. Precision is synonymous with reproduc-
for assessing the performance characteristics of ibility; however, the use of the term precision is
another test method. When the true disease status generally applied to quantitative assays, while
of a patient is unknown and the disease state is reproducibility is used with qualitative analyses.
being determined by using a test compared with Mathematically, precision can be expressed as a
an imperfect gold standard, the results will be percent:
skewed and errors of the reference method will be
magnified. In this situation, it may be more appro- number of repeated results in agreement
x 100
priate to display the agreement and disagreement total number of results
between the gold standard and new test in graphic
or tabular form. Areas of disagreement may then Predictive value (8, 20, 23): The predictive
be further investigated by other tests or by follow- value of a test is the probability that a positive
ing the patient’s condition to determine if disease result (positive predictive value, or PPV) accu-
develops. In those instances in which it cannot be rately indicates the presence of an analyte or a
determined whether the new test is better than the specific disease or that a negative result (negative
gold standard, a decision to use the new test alone predictive value, or NPV) accurately indicates the
or in combination with the gold standard may be absence of an analyte or a specific disease. PPV is
made by using a cost-benefit analysis (8). Prob- expressed as a percent:
lems associated with use of an imperfect gold number of true-positive results
standard are more fully discussed in the National
Committee for Clinical Laboratory Standards number of true-positive plus false-positive results
(NCCLS) publication I/LA 18-A (23) and in ref-
erences 6 and 11. x 100
Home brew test: A proce dure developed in- NPV is expressed as:
house th at uses reagen s that are either commer-
cially available or produced in-house or any pro- number of true-negative results
cedure that incorporates modifications of the
number of true-negative plus false-negative results
manufacturer’s package insert instructions. Re-
agents provided by companies for either investi- x 100
gative or research use must be used in accordance
with the guidelines described by the company Predictive values can vary significantly with the
and/or the U.S. Food and Drug Administration prevalence of the disease or analyte unless the test
(FDA) and must not be used for in vitro diagnos- is 100% sensitive (for NPV) or specific (for PPV).
tic testing. Since a home brew test is not reviewed The highest predictive values are desired when
by the FDA and since the reagents used therein inappropriate treatment due to false-positive or
cannot be reviewed for their intended use, these -negative results has serious clinical, emotional,
tests require a more extensive performance veri- epidemiological, public health, or economic con-
fication. sequences. Predictive values are most meaningful
New test: A new test includes any test not in evaluating a test’s performance in specific risk
previously offered by a laboratory, a procedure or population groups (see Appendix A).
CUMITECH 31 VERIFICATION AND VALIDATION 3

Prevalence (24): The pretest probability of a detect. Clinical sensitivityis the percent test posi-
particular clinical state in a specifiedpopulation; tivity in a population of affected patients. The
the frequency of a diseasein the population of highest sensitivity is desired when a diseaseis
interest at a given point in time. seriousand treatable and when false-positivere-
Quality assurance (28, 29): A systemfor con- sultswill not lead to seriousclinical or economic
tinuouslyimproving and monitoring the reliability, problems.Mathematically, sensitivity is expressed
efficiency, and clinical utilization of laboratory asa percent:
tests. Quality control, quality improvement, and
method validation are integral components of numberof true-positive results
quality assurance.
numberof true-positive plusfalse-negativeresults
Quality control (29): The processof ongoing
performancechecks,including personnelperfor- x 100
mance, using known organismsor analytes to
ensure on a regular and frequent basisthat a Specificity (8, 20, 23, 24): A measureof a test’s
method which has gone through the verification efficiency in ruling out an analyte or a specific
process(seebelow) and is now part of the labo- disease.Analytic specificity is the ability of an
ratory’s routine test battery is performing as ex- analytical methodto detect only the analyte that it
pected. Quality control systematicallydetects de- wasdesignedto measure.Clinicalspecificityis the
ficienciesin testing by setting limits of acceptable percent of negative test results in a population
performance (accuracy and precision). It thus without the specifieddisease.The highest speci-
allowsdetection, and corrective action where ap- ficity is desiredwhen the diseaseis seriousbut not
propriate, of major problems or errors with test treatable,when diseaseabsencehaseither psycho-
systemsand their performance. Quality control logical or public health value, or when false-
impliesthat there exist standardanalytesthat have
positive results might cause serious clinical or
known reactionsor reactivity. Quality control is an economicproblems.Mathematically, specificity is
integral part of the test validation process. expressedas a percent:
Quality improvement (29): The prevention of
test deficienciesand enhancementof a test’sclin-
ical utility by establishinga thorough understand- numberof true-negativeresults
ing of the test’s capabilities and limitations, as numberof true-negative plusfalse-positiveresults
gathered from experience and observation, and
the subsequentuseof this knowledgeto makeand x 100
verify procedural changesfor improved test per-
formance. Validation: The documentation that a test
Reference method (20): A thoroughly investi- which hasalready been verified is repeatedly giv-
gated method, in which exact and clear descrip- ing the expected resultsas the test is performed
tions of the necessaryconditions and procedures over a period of time. Validation confirmsthat the
are given for the accuratedeterminationof one or test continuesto perform satisfactorily according
more values;the documentedaccuracyand preci- to the laboratory’s requirementsor the manufac-
sion of the method are commensuratewith the turer’s claimsor, for homebrew tests,accordingto
method’suse for assessing the accuracyof other its intendeduse.The requirementsfor test valida-
methodsfor measuringthe sameproperty values tion may include personnel competency assess-
or for assigningreference methodvaluesto refer- ment, quality control, internal and external pro-
encematerials.A currently usedmethod is unac- ficiency testing, and correlation with clinical
ceptable as a reference method unlessthere is findings.Validation thus becomesan integral part
on-site or peer-review journal documentation of of the laboratory’s quality assuranceprogram.
an acceptablelevel of accuracy and precision of Verification (7, 29): The documentation of ei-
the method. ther commercialor home brew test accuracy.For
Reproducibility: SeePrecision. commercially obtained tests, it is the processof
Sensitivity (8, 20, 23, 24): A measureof a test’s examination or evaluation of a test system to
efficiency in detecting an analyte or a specific determine whether the claims stipulated by the
disease.Analytic sensitivitymeasuresthe smallest manufacturer in the packageinsert as they relate
quantity of an analyte that can be reproducibly to the product, the process,the results, or the
distinguishedfrom background levels, or a zero interpretation can be achieved. Verification re-
calibrator, in a given assaysystem; it is usually quires determination or confirmation of the test
defined at the 0.95 confidencelevel (52 standard performance characteristics,including sensitivity,
deviations)and is more appropriately called “de- specificity, and, where appropriate, the predic-
tection limit.” In microbiology,the detection limit tive values, precision, and accuracy of the test.
can be correlated to the number of colonies in Verification is a one-time process,completed be-
culture or the lowestquantity of antigen a test can fore the test or systemis usedfor patient testing.
4 ELDER ET AL. CUMITECH 31

FEDERAL REGULATION: ROLE OF CLIA ‘St.4 oratory Accreditation) must include these re-
On February 28, 1992, the Department of quirements as part of their inspection criteria.
Health and Human Services published the final Since to date the FDA has not implemented the
regulation implementing the Clinical Laboratory process of clearing tests that meet CLIA require-
Improvement Amendments of 1988 (7). CLIA ‘88 ments for general quality control, all tests of high
complexity are subject to verification. Verification
extended federal regulation to cover all laborato-
ries that examine human specimens for the diag- is not required for tests of moderate complexity
unless such tests were developed in-house, are not
nosis, prevention, or treatment of any disease or
cleared by the FDA, or have been modified by the
impairment or for the assessment of the health
laboratory. Validation requirements for tests of
of human beings. Sections 493.1201 through
both moderate and high complexity are delineated
493.1285 of subpart K of the regulation set forth in the quality assurance and specific quality con-
the quality control requirements, which became
trol sections of the regulation.
effective September 1, 1992, for tests of moderate It should be noted that the 1992 regulations
or high complexity (waivered tests are not subject were published as a final rule with comment and,
to quality control). For tests of moderate complex- as of May 1996, are undergoing revision on the
ity, quality control requirements were to be basis of the comments and recommendations from
phased in over a two-year period. For tests of high the Clinical Laboratory Improvement Advisory
complexity, the category of the majority of tests in Committee (CLIAC). Publication of the final reg-
microbiology, the requirements became effective ulations has been projected for 1997, but final
in full as of that date. deliberations are still in process as of this writing.
Section 493.1213 describes the requirement for
the establishment and verification of method per- ROLE OF THE FDA IN REGULATION
formance specifications. The requirement only OF CLINICAL MICROBIOLOGY
applies to instruments, kits, or test systems intro- IN VITRO DEVICES
duced after the effective date of the quality control Considerable confusion often exists concerning
regulations, September 1, 1992. After that date, a the process followed by the manufacturer of a test
laboratory that introduces a new procedure for kit or system to gain FDA clearance for marketing
patient testing which uses a method developed of the product and the relationship of this process
in-house, a modification of the manufacturer’s test to the requirement for later verification in the user
procedure, or a method that has not been cleared laboratory. An understanding of the processes
by the FDA as meeting the CLIA requirements utilized by the FDA can help the end user of a
for general quality control must, prior to reporting product understand the reasons for the regulatory
patient test results, verify or establish for each requirement that the laboratory verify the perfor-
method the performance specifications for the mance of a test approved and/or cleared by the
following performance characteristics, as applica- FDA in its own setting.
ble: The Medical Device Amendments to the Food,
Drug, and Cosmetic Act were enacted in 1976 and
l Accuracy directed the FDA to regulate medical devices
l Precision under the appropriate control levels necessary to
l Analytical sensitivity ensure safety and effectiveness. The Safe Medical
l Analytical specificity, to include interfering Devices Act of 1990, a major revision of the 1976
substances amendments, added new provisions to better en-
l Reportable range of patient test results sure that devices entering the market were safe
l Reference range(s) and effective and provided means for the FDA to
l Any other characteristic required for test per- learn quickly about serious device problems and
formance and interpretation of results remove defective devices from the market. The
Food, Drug, and Cosmetic Act and the Safe
Also, control procedures for patient testing Medical Devices Act are administered by the
must be established on the basis of the verified FDA’s Center for Devices and Radiological
performance specifications. The laboratory is re- Health through the Office of Device Evaluation,
quired to have documentation of the verification which includes the Division of Clinical Laboratory
of manufacturer’s specifications or establishment Devices.
of performance specifications for tests developed A manufacture r can legally place an in vitro
in-house. All laboratory accrediting programs di agnostic device into the market in two main
which have been granted deemed status by the ways. The first is by Premarket Notification
Health Care Financing Administration (HCFA) 510(k), in which a manufacturer demonst rates
(including the College of American Pathologists, that its device is substantially eq uivalent to a
the Joint Commission on Accreditation of Health- preamendment device (a device in use before
care Organizations, and the Commission on Lab- 1976) or a predicate device (a similar legally
CUMITECH 31 VERIFICATION AND VALIDATION 5

marketed device) in terms of both safety and cutoff; results and their interpretation; quality
effectiveness. Typically in a 510(k), the manufac- control recommendations; and specific perfor-
turer must describe the device methodology/tech- mance characteristics. All microbiology products
nology, its intended use, indications for its use, which undergo a scientific evaluation of data to
and its performance characteristics as shown in substantiate product performance claims as stated
labeling (product insert) and in promotional ma- in the product insert (i.e., moderate and high risk
terial and advertisements. The manufacturer must devices) are expected by the FDA to maintain that
also compare and contrast the submitted device performance throughout the life of the product.
with a similarly legally marketed device with sup- Failure to maintain that expected performance
porting data, and, in the case of modified devices could result in compliance or regulatory action.
for which the manufacturer has determined that Promotional and advertising material also falls
the modification could affect the safety and/or under the labeling regulation. Such material can
effectiveness of the device, the manufacturer only be reflective of the information contained in
must provide documentation/data to address the package insert for the device.
that effect(s). The 510(k) review is entirely a Devices which are in the laboratory research
paper review; the FDA does not submit the phase of development may not be represented as
actual products to direct laboratory evaluation. effective diagnostic products, and the statement
The agency therefore has no hands-on experience “For Research Use Only. Not for use in diagnostic
with the vast majority of products under review. If procedures” must be prominently placed in the
the FDA assessment of the 510(k) submission labeling. A product being shipped or delivered for
indicates that it is substantially equivalent to a product testing prior to full commercial marketing
legally marketed device, the device is cleared and must prominently bear the statement “For Inves-
the manufacturer is free to market it in the United tigational Use Only. The performance character-
States. The FDA has granted exemptions from the istics of this product have not been established.”
requirement for 510(k) notification for a variety of Only in vitro devices which have been 510(k)
generic-type devices, including such microbiology cleared or PMA approved by the FDA may legally
products as anaerobic chambers, incubators, gas- have the statement “For In Vitro Diagnostic Use”
generating devices, and most media. as part of their package insert. In the clinical
A second pathway to market is via the premar- laboratory, results from the following must not be
ket approval (PMA) application. In this case, reported without verification: (i) tests or proce-
there is no preamendment device and the manu- dures that have been developed in-house and use
facturer must provide reasonable assurance of class I reagents that do not have an indicated use
safety and effectiveness under conditions of in- for that test or (ii) reagents or tests provided by
tended use(s). Before approval, a PMA receives companies for investigative or research use only
an in-depth scientific review, and the firm must that are not used within the guidelines described
undergo a comprehensive good manufacturing by the company or by the FDA. Reports of such
practice inspection. Finally, the PMA is reviewed results should indicate that the test is not FDA
by an FDA advisory panel of outside experts who cleared or approved but has been developed and
provide recommendations to the FDA for ap- verified in-house. Since home brew tests are not
proval with or without conditions or for dis- reviewed by the FDA and since the reagents used
approval of the application. Examples of micro- therein cannot be reviewed for their use in those
biology devices requiring PMA applications are tests, home brew tests require a more extensive
devices directed at detection or typing of human performance verification. Furthermore, validation
papillomavirus; all hepatitis and human immuno- must be performed at least every six months when
deficiency virus diagnostic, detection, and moni- no commercial quality control reagents or profi-
toring devices; and devices using nucleic acid ciency test samples are available (2, 7).
amplification techniques for direct detection of FDA clearance or approval of a product does
Mvcobacteriumtuberculosisfrom clinical material. not predict how that product will perform in the
The proposed package insert is a part of the end user’s laboratory under actual testing condi-
510(k) or PMA application. After all analytical tions and with the specimen mix encountered in a
and clinical data have been critically reviewed, the particular patient population. Thus, initial clear-
final step of the Microbiology Branch review is to ance by the FDA cannot be used as a substitute
“clear or approve” the product labeling (thus, for verification of test performance by the per-
product clearance by the FDA does not guarantee forming laboratory. In addition, test verification
test performance). The Branch pays particular and validation by clinical laboratories now also
attention to the following components of the play a critical role in ensuring that devices and
package insert, all of which are required by law (21 medical products previously cleared/approved by
CFR 809.10): intended use; specimen collection, the FDA are performing as expected. On Decem-
transport, and storage recommendations; warn- ber 11, 1995, the final rule for mandatory report-
ings and limitations; expected values; validation of ing of medical product-related serious incidents
6 ELDER ET AL. CUMITECH 31

was published (4). The rule requires hospitals and 98%. Confirmatory tests may not be necessary
other health care facilities for the first time to when the screening or diagnostic test has high
report deaths and serious injuries connected with specificity and positive predictive values.
the use of medical products (including in vitro Diagnosis. Diagnosis is used for the evaluation
diagnostic devices) to the manufacturer and of persons suspected of having a given disease
directly to the FDA. The agency evaluates the state or characteristic (e.g., a particular type of
seriousness of the health hazard, takes correc- infection). If the characteristic is important,
tive action, and communicates that action to the either for treatment or for prognostic consid-
health professional community. In addition, erations, sensitivity should be as high as possi-
product problems (e.g., suspected contamina- ble. When diagnostic test results are not con-
tion, questionable stability, defective components, firmed by additional laboratory or clinical data,
poor packaging or labeling, and device malfunc- specificity may also need to be very high (see
tion) should be reported to the manufacturer. Appendix A). However, if an accurate confir-
matory test is readily available, a high degree of
SELECTION OF A LABORATORY METHOD specificity might not be necessary. The majority
Once a laboratory has reached the decision to of clinical tests for infectious diseases are for
offer a new test, the next step in the process is diagnostic use.
generally selection of the method by which the test
will be performed. Few laboratories have the time 2. Decide what analyte (e.g., organism, antigen,
or resources to perform in-house evaluations of nucleic acid, etc.) is to be detected and what the
the large number of available test systems, kits, or reference method or gold standard will be for
methods which may be available to the microbiol- comparison. Note: if the new test is likely to be
ogy laboratory for detection of an organism, anti- more sensitive than the gold standard, then ways
gen, or other analyte of interest. Thus, it becomes to arbitrate discrepant results should be consid-
crucial to approach the selection of a new method ered (e.g., clinical data, other assays, etc.) (6, 11,
in an organized fashion, making use of all avail- 23).
able information to narrow down the selection of 3. In conjunction with the end user of the test
a laboratory method without performing expen- (e.g., the physician) and the information from
sive in-house studies. The following steps are steps 1 and 2, determine the medical usefulness of
designed to serve as a guide for the initial selec- the test (e.g., whether the test will lead to im-
tion of a laboratory method. Although all steps proved patient care and/or shortened hospital
may not be necessary for every test method under stay) and preliminary clinical and/or microbiolog-
consideration by a laboratory, the basic process ical requirements for test sensitivity, specificity,
can be followed for the majority of tests utilized by and predictive values (as appropriate).
the microbiology laboratory. 4. Survey the technical and medical literature
1. Define the purpose for which the method is for performance claims of various methods that
to be used. Common purposes for tests include may indicate that one or more methods will meet
the following (23). the initial requirements for sensitivity, specificity,
etc. When reviewing the literature, confirm that
l Screening. Screening is used for testing large the method described is actually the test (unmod-
populations of patients for the presence of a ified) that is to be evaluated in the laboratory.
disease state or analyte (such as an infectious 5. Determine the characteristics of the meth-
agent). In general, screening tests should have od(s) of interest. The choice of method may also
high (i.e., greater than 95%) clinical sensitivity be based on the following practical parameters (9,
and negative predictive values. In most cases, 14, 27).
the recommended specificity and positive pre-
dictive value can be lower than those of diag- 0 Cost of the method. What are the comparative
nostic and confirmatory tests. Thus, a negative costs for material and labor relative to alterna-
screening test result should indicate that the tives to the test? What is the extent of reim-
person has a high probability of being free of bursement?
the characteristic, whereas a positive test result 0 Practicality in the laboratory setting
might reflect only the need for confirmatory Can the test be performed on all necessary
testing. shifts?
l Confirmation. Confirmation is used after ob- Does the test require special equipment?
taining a positive screening or diagnostic test What is the turnaround time for the test?
result to ensure the accuracy of that initial What are the personnel requirements?
result. Specificity and positive predictive value, Are quality control and proficiency test mate-
rather than sensitivity and negative predictive rials available?
value, are usually the primary considerations What is the extent of quality control that will
for confirmatory tests; specificity should exceed need to be performed?
CUMITECH 31 VERIFICATION AND VALIDATION 7

Is there adequate space in the laboratory to tional (e.g., a medium which is selective for an
perform the test? organism and contains biochemicals that are th .en
Can the test be automated to reduce labor? used to presumptively identify the organism). This
Does the system have an indication for all the would include Campylobacter agar, media for the
uses/organisms that are of interest? selective isolation of the pathogenic Neisseria spp,
Specimen requirements and any other media not listed in Table 3 of
Volume and type of specimen needed reference 17.
Collection requirements In the following sections, suggested methods
Transport requirements are included for verification of many of the com-
Storage requirements monly used tests and test systems found in the
Quality of specimen microbiology laboratory. These suggestions are
Quantities of reagents and controls needed for not meant to be all-inclusive, and alternative
test; storage requirements approaches may be utilized in individual labora-
Shelf-life of reagents and controls before and tories. In addition, it is recognized that the verifi-
after opening cation process can be timely and expensive, often
Availability of supplies, service, and/or techni- complicated by a paucity of specimens or samples
cal support containing or lacking the desired analyte. In some
Possible safety hazards related to performing cases, laboratories will need to make difficult
test choices about the extent of verification that is
Whether a reference range is appropriate for possible, taking into consideration how widely the
the test, and how it will be determined for the test has been used and accepted by the microbi-
institution ology community, the extent and results of pub-
6. Make a preliminary selection of a test method lished evaluations, and the impact of an incorrect
and perform the in-house verification. A brief test result on the patient. In some cases, repeat
outli ne of this process is ou tlined in Appendix B. testing of selected control material near the test
cutoff value(s) may give a level of satisfaction. In
VERIFICATION OF COMMON other cases, laboratories may decide that they are
MICROBIOLOGY TESTS unable to perform a reasonable verification and
Verification of a test’s performance parameters may choose to refer the test to another laboratory.
is accomplished by performing the new or revised Whenever possible, purchase of a new system or
test method in parallel with a reference method test methodology should be made contingent upon
that has an established and satisfactory level of the results of the verification studies. Records of
accuracy. The results of test verification should the actual test verification results must be main-
indicate one of three possibilities: tained at least two years. However, since the
laboratory must be able to provide the ordering
l The test is acceptable for routine use physician with the performance parameters of
l Further verification studies are required each test it performs, good laboratory practice
l Immediate corrective action is required by the would dictate that the records of test verification
manufacturer (if commercially obtained), the
be kept for as long as the test is in use.
user, or both. The test is unsuitable for routine
use until its performance parameters can be
verified. Antimicrobial Susceptibility Systems
Certain commonly used microbiology items are The generation of antimicrobial susceptibility
not considered instruments, kits, or test systems test results is one of the most important functions
under CLIA ‘88 and may not require a complete of the microbiology laboratory, as the results may
verification process prior to use. Items such as directly affect the therapy chosen for treatment of
media or individual reagents used as components an infection. Thus, it is critical that the microbi-
of the identification process (e.g., oxidase, cata- ologist be confident that the system chosen is able
lase) may instead be monitored through the qual- to provide accurate and reliable results in the
ity control protocols of the laboratory. However, user’s own laboratory. Within the past 10 years, a
each laboratory must assess the nature and pur- number of recommendations for verification of
pose of each of these components and may choose susceptibility test systems have been discussed in
to perform more elaborate in-house verification. the literature (5, 10, 16). Although it may be
For example, a decision may reasonably be made difficult for laboratories to perform a rigorous
not to perform verification on common media study of a new system, use of selected control and
with a history of limited failures, such as most of clinical isolates can aid in the effort to verify the
the media listed in Table 3 of NCCLS document claims made in the literature and by the manufac-
M22-A (17), but it should be performed on some turer regarding the accuracy and reproducibility
of the highly complex media which are multifunc- of a system (10).
8 ELDER ET AL. CUMITECH 31

The evaluation must be designed to allow de- criteria for very major errors, a minimum of 65
tection of the foll owing types of errors (10, 16): resistant isolates would need to be tested against
each antibiotic). For antibiotics without an inter-
Very major errors. The system under evalua- mediate interpretive category (where a one-dilu-
tion indicates a “susceptible” response while
tion error in the MIC might spuriously appear as
the reference method indicates a “resistant”
a very major or major error), there should be
response. Clinically, very major errors are the
>90% agreement within one dilution between the
most serious type of error and can only be
two procedures being evaluated (10). Growth
detected by testing organisms resistant to each
failures for susceptibility systems should not ex-
antimicrobial agent. ceed 10% of the total number of isolates tested
Major errors. The system under evaluation
(5). If the chosen limits are exceeded for any of
indicates a “resistant” response while the ref- the error types for any drug-isolate combination,
erence method indicates a “susceptible” re-
the test must be considered “unverified,” and a
sponse. Major errors can only be detected by
corrective action investigation must be under-
testing organisms susceptible to each antimi- taken in conjunction with the manufacturer to
crobial agent.
attempt to resolve the discrepancies. Following
Minor errors. Either the new system or the
corrective action, the new or revised test should be
reference method indicates an “intermediate” run again in parallel with the reference method on
result. The other method indicates either a
a minimum of 20 appropriate isolates (isolates
“susceptible” or “resistant” result.
which will demonstrate that the problem[s] has
Evaluation of susceptibility test methods should been corrected).
be done using a distribution of organisms similar
to those commonly isolated but should include Diagnostic Microbiology Tests
susceptible and resistant isolates for each antibi- Diagnostic microbiology tests (nonculture tests
otic whenever possible. The distribution should for detection of microorganisms, microbial anti-
contain examples of clinically relevant isolates gens, nucleic acids, or antibodies from clinical
appropriate for the class of compounds routinely specimens) may be used for screening, confirma-
tested and should include isolates showing impor- tory’ or diagnostic purposes. The process for
tant resistance mechanisms. For example, methi- verifying a diagnostic microbiology test may be
cillin-resistant Staphylococcusaureusand coagu- relatively straightforward for analytes which are
lase-negative staphylococci should be included in common in the population. When the new test will
the evaluation of antistaphylococcal agents (22). be detecting a relatively uncommon analyte (for
Although it may be desirable to test at least 35 example, a direct test for the detection of Crypto-
resistant isolates for each antibiotic in the test sporidium),assessment of the sensitivity of the test
system (lo), this is not always possible or prac- in the user’s own patient population becomes
tical for laboratories. In these cases, laborato- difficult. In these circumstances, the manufac-
ries should carefully scrutinize the organisms to turer’ other laboratories, or commercial sources
be tested to ensure that they make the best may be able to provide specimens of known
choices possible when selecting isolates for test- content to be used in the evaluation.
ing. It is important to remember that a test may be
Since very major errors can only occur with suitable for one population of patients and not for
isolates that are resistant, very major error rates another. Laboratories must evaluate this individ-
should be calculated using the number of resistant ually and be prepared to provide physicians with
strains as the denominator rather than the total the specifications of the test for individual popu-
number of isol ates tested (5, 10, 16). For the same lations with different disease prevalences.
reason, major error rates should be calculated by
using the total number of susceptible isolates as Verification of commercially-obtained tests
the denominator. Very major errors determined performed exactly as described in the
for a large sample (n = ~35) of known resistant manufacturer’s product insert
isolates should be ~3% (10). The combination of The new or revised test should be performed in
major and minor errors attributable to the new parallel with the existing test or a reference meth-
test should be ~7% when determined for a large od on a minimum of 20 specimens that contain
known-susceptible population or a large unse- each target analyte and a minimum of 50 speci-
lected sample of clinical isolates (a minimum of mens that lack the target analyte. An effort should
100 strains) (10). More stringent criteria are sug- be made to include weakly positive specimens that
gested by NCCLS (22) and required by the FDA will challenge the detection limit of the assay. If
for manufacturers (an acceptable error rate of less these are unavailable, dilution of strongly positive
than 1.5% for very-major errors and less than 3% specimens with an appropriate material (e.g., nor-
for major errors; however, for the laboratory to mal serum) can achieve the same effect. In some
demonstra te that it has met the more stringent circumstances, it may be necessary to test more
CUMITECH 31 VERIFICATION AND VALIDATION 9

specimens to document that the test meets the protocol which has been previously verified. In
required level of sensitivity, specificity, or positive this situation, verification must be performed to
and negative predictive values. An example of this the extent necessary to demonstrate that the
decision-making process can be found in Appen- change has not affected the performance of the
dix A. test, but it may not require the extensive testing
After comparing the results of the new or performed initially. It may be useful to maintain a
revised test with those of the reference method, panel of a limited number of well-characterized
the number of true-positive, true-negative, false- specimens and to only do a complete verification if
positive, and false-negative results obtained with these specimens do not give satisfactory results.
the test to be verified should be documented.
When the reference method is known to be an Microbial or Microbial Product
imperfect standard, an attempt to resolve discrep- Identification Tests
ancies should be made. By using these results, the Microbial or microbial product identification
sensitivity, specificity, predictive values, accuracy, tests include antisera, antigens, chemicals, stains,
and precision of the new or revised test may be instruments, reagents, or kits used to identify mi-
calculated. The calculated predictive values may croorganisms or their products from cuhure.
not be indicative of the performance of the test in
an actual patient population if the prevalence of Verification of commercially obtained tests
the target analyte is different from that used in the performed exactly as described in the
verification study. Remember that a test with a manufacturer’s product insert
specificity of 95% when the prevalence of a target Evaluation of test systems for identification of
disease is only 1% will have a predictive value of a microorganisms to the species level may be struc-
positive result of only 16.7%. Individual predictive tured around one of the following three sugges-
values may be calculated from the sensitivity and tions (13).
specificity data for known different prevalences, as
l Perform at least 1 week of consecutive parallel
seen in the example in Appendix A. The test may testing (minimum of 50 strains) with the exist-
be considered verified if it meets the requirements
ing method. Discrepancies must be arbitrated.
initially established for performance by the users
l Test known representative strains (stock cul-
of the test and if the sensitivity and specificity are tures) of a minimum of 12 to 15 commonly
no lower than 5% below those of the reference isolated species of organisms, for a total of 50
method, those appearing in peer-reviewed jour-
or more tests.
nals, or those claimed by the manufacturer in the l Confirm that 20 to 50 organism identifications
product insert. Whenever possible, data published
(12 to 15 different species) agree in concurrent
in a peer-reviewed journal, rather than the man- testing with the current method (discrepancies
ufacturer’s marketing data, should be used in the
must be arbitrated) or with the results of
evaluation of test kits and reagents (23). Even the
reference laboratory testing of split samples.
results of published studies should be carefully
analyzed for the procedure used (13) . In each case, the appropriate quality control or-
If the sensitivity or specificity of the new or ganisms should also be tested and included in the
revised test does not satisfy the verification re- verification process.
quirements, the test must be considered unverified For identification methods that detect only one
and corrective action must be taken by the man- analyte (e.g., coagulase, indole, oxidase, immuno-
ufacturer, the user, or both. Following corrective fluorescent reagents, etc.), the new or revised test
action, the new or revised test should be run again should be run in parallel with the existing test or a
in parallel with the reference method and inter- reference method on a minimum of 20 microbial
preted as described above. isolates that contain the target analyte and 20
isolates that lack the target analyte.
Verification of home brew diagnostic tests When the results of the verification of tests
The new or revised test should be verified as which identify to the species level are evaluated,
described above, except that a minimum of 50 both the level of agreement between the new and
specimens that contain the target analyte and a reference method and the types of errors or
minimum of 100 specimens that lack the target disagreements should be evaluated. Overall, there
analyte should be studied. If either the sensitivity should be at least 90% agreement with the existing
or specificity is lower than 5% below the values of system or reference method before the new
the reference method, verification tests should be method is considered verified. Certain groups of
repeated, after appropriate corrective action has organisms will commonly be more challenging for
been taken. new systems to identify (e.g., nonfermenters,
In some instances, test verification will be re- corynebacteria, coagulase-negative staphylococ-
quired because of very small modifications (e.g., a ci), and greater flexibility may be necessary in
minimal change in incubation time) of an existing assessing the accuracy of the new method. Other
10 ELDER ET AL. CUMITEEH 31

groups of organisms (e.g., members of the family Blood Culture Systems


Enterobacteriaceae) shouldbe identified with high Meaningful verification of a new blood culture
levels of accuracy, and these test systemsshould systemis one of the most difficult tasksfacing the
be held to a higher (e.g., 95%) requirement for clinical microbiologist. Parallel testing requires
agreement(14). In addition, the typesof disagree- collection of additional blood from each patient
mentsencounteredwith the new systemshouldbe and may not be possible in some patients or
scrutinized. The new systemmay misidentify an institutions. The level of positivity for clinical
organism,may require further testsbefore identi- pathogens (usually in the range of 8 to 11%)
fying an organism,or may give no identification at meansthat most of the specimenscollected will
all. Misidentification is the most seriouserror for not be of value in the comparison(30). In addi-
an identification system,while a laboratory may tion, the incidenceof contamination(usually 1 to
chooseto accepta certain numberof isolateswith 3%) and predominanceof a limited number of
no or partial identification if other factors (e.g., pathogens may result in an evaluation skewed
cost or speed) outweigh the inconvenience of toward only a few of the potentially clinically
further testing. significant organisms.Thus, in-houseverification
When the results of the verification of tests shouldbe designedto answerthe following ques-
which detect a particular analyte are evaluated, tions:
the sensitivity and specificity of the new test
shouldbe calculated.The test may be considered 0 Will the mediausedby the systemsupport the
verified if the sensitivity and specificity are no growth of organisms(including yeasts, anaer-
lower than 5% belowthosevaluesof the reference obes,or fastidiousorganisms,where appropri-
method. ate) commonly seenin the user’spatient pop-
ulation?
Verification of homebrew identification tests 0 Will the instrument (for automated systems)
If the new or revised test identifies isolatesto detect, in a timely fashion,the majority of path-
the specieslevel, it should be tested in parallel ogenic organismsfrom blood cultures which
with the existingtest or another reference method contain thesemicroorganisms?
on a minimumof 200isolates.Whenever possible,
these isolatesshould include all speciesidentifi- Two approachesfor verification of blood culture
ableby the new or revisedtest. The samerequire- systemsare discussedbelow. Laboratories may
mentsfor method agreementdescribedfor com- also chooseto combinethese approachesto take
mercialmethods(at least90% agreementoverall) advantage of the strong points of each (e.g.,
must be met by home brew teststo considerthe perform parallel studiesto assess
the ability of the
method verified. systemto detect commonlyisolatedorganismsand
If the new or revisedtest identifies a particular perform seededblood culturesto assess lesscom-
analyte, it should be tested in parallel with the mon pathogens).
existingtest or a referencemethod on a minimum
of 50 microbial isolatesthat contain the target Parallel blood culture studies
analyte and a minimum of 100 isolatesthat lack Performance of parallel blood cultures allows
the target analyte. After the resultsof the new or the laboratory to evaluate all aspectsof the new
revisedtest are comparedwith thoseof the refer- systemunder actual patient and laboratory condi-
ence method, the number of true-positive, true- tions. When a laboratory choosesto perform
negative, false-positive,and false-negativeresults parallel studiesof commerciallyavailablesystems,
obtained with the test to be verified should be duplicate sets of blood cultures inoculated with
documented. When the reference method is equivalentblood volumesshouldbe obtaineduntil
known to be an imperfect standard,an attempt to one or more isolatesof at least20 different species
resolve discrepanciesshould be made. By using of clinically significantorganismshavebeenrecov-
theseresults,the sensitivity, specificity, predictive ered. The new systemmay be consideredverified
values(if appropriate), accuracy,and precisionof if its sensitivity in detection of clinical pathogensis
the newor revisedtest may be calculated.The test no lower than 5% below that of the reference
may be consideredverified if the sensitivity and method.
specificity are no lower than 5% below those of The method must be consideredunverified if
the reference method or those appearing in the the sensitivity of the new systemis more than 5%
peer-reviewedliterature, whichever is higher. below that of the reference method. Corrective
If the new or revised test does not satisfy the action should be taken by the user (and the
verification requirements,the test mustbe consid- manufacturer where appropriate), and the verifi-
ered unverified and corrective action must be cation study shouldbe repeated.The new system
taken. Following corrective action, the new or may then be consideredverified if its sensitivityfor
revised test should be compared again with the detection of pathogensis no lower than 5% below
reference method asdescribedabove. that of the reference method.
CUMITEEH 31 VERIFICATION AND VALIDATION 11

Seeded blood culture studies organismsin an actual patient setting.This can be


done by performing blind subculturesof all nega-
ASSESSMENT OF THE ABILITY OF THE MEDIUM TO tive cultures prior to discard.For institutionswith
SUPPORTTHE GROWTH OF PATHOGENS a 10% positivity rate, it will be necessaryto
1. Review past records of positive blood cul- subculture at least 500 bottles of each medium
tures to identify those organisms(a minimum of type to confirm that 298% of the positiveshave
20 different species)mostprevalent in the patient been detected.
population servedby the institution. In addition,
significantpathogenswhich are relatively uncom- VALIDATION OF DIAGNOSTIC TESTS USED
mon or fastidious (e.g., Neisseria gonowhoeae or IN CLINICAL MICROBIOLOGY
N. meningitidis, Cryptococcus neoformans, Campy- While verification of a new or revised test
lobacter fetus) should alsobe included. Whenever (commercially obtained or home brew) servesto
possible,actual patient isolatesshould be used establishthat the performanceparametersof the
rather than stock strains. test are satisfactory, it doesnot provide ongoing
2. Make seededblood cultures of isolatesof assurancethat the test is continually performing
each of the above species.To challengethe sys- as expected under routine use over extended
tem, the minimum amount of sterile, antibiotic- periods of time. Test validation is the ongoing
free humanblood recommendedby the manufac- processused by the laboratory to provide this
turer shouldbe placed in eachbottle. In addition, assurance.Although validation as a specific pro-
the numbersof organismsplaced in each bottle cessis not addressedin CLIA ‘88, the components
shouldapproximatethosefound in casesof septi- of the process(quality control, proficiency testing,
cemia(which are often <l CFU per ml of blood). verification of employeecompetency, and instru-
This is best accomplishedby making serial dilu- ment calibration) are all covered. The end result
tions of the organismsprior to inoculation. of validation will indicate one of three possibili-
ties: (i) the test continues to be acceptable for
Dilute each organismin sterile salineor broth
routine use,(ii) further investigationis warranted,
to match a 0.5 McFarland standard. or (iii) immediate corrective action must be un-
Make serial dilutions by transferring 0.1 ml of dertaken by the manufacturer (if commercially
the suspensioninto 10 ml of saline or broth, obtained), the user, or both, and the test mustbe
mixing, and repeating this transfer two addi- consideredunsuitable for continued routine use
tional times (total of three transfers). This will until it can be validated. Lot numbersand expira-
result in a concentration of approximately 5 to
tion datesshouldbe documentedfor all reagents
30 organismsper 0.1 ml in the final dilution and materials used in the validation process.
tube.
Records of validation proceduresshould be re-
Inject 0.1 ml of the diluted organismsuspen- tained for at leasttwo years.
sion into each culture bottle. Also inoculate
two plates of appropriate media with 0.1 ml Components of the Validation Process
from the samedilution tube and spread the The standardcomponentsof a validation pro-
inoculum for confirmation of the quantitation cessinclude the following.
of the organismsinjected into the bottle. 1. Quality control organisms. Quality control
3. Examine the seededbottles, following the organismsshouldbe asstipulatedby the manufac-
manufacturer’sinstructionsfor the system. turer of a commercially available test or instru-
4. The method is considered verified if all ment or chosenby the user. If possible,these
isolateswere detected. Any problemswith detec- shouldbe reference strains(American Type Cul-
tion shouldbe investigatedby repeatingthe tests ture Collection) or have some other recognized
with the samepatient strains.If detection is still source; they should be maintained in a standard
not obtained, corrective action must be taken by manner, so as not to be genetically affected by
the userand/or the manufacturerprior to institut- storage, passage,etc. If nonreference strains are
ing useof the systemin the laboratory. used, the laboratory should have a complete
record of the history of the organism,including
ASSESSMENT OF THE ABILITY OF AN AUTOMATED SYS- characterization, storage,and recovery from stor-
TEM TO DETECT CLINICAL ISOLATES OF ORGANISMS age. The frequency of testing and actions to be
WHICH HAVEGROWN INTHEMEDIA(OPTIONAL) taken after control failures should follow the
Once the initial verification of the systemindi- manufacturers’recommendationsor those of one
catesthat the systemwill support and allow de- or more of the various regulatory or advisory (2,
tection of commonisolatesin a seededtrial, the 17,19,21) agencies.For homebrew tests,positive,
systemmay be usedfor testingof patient samples. negative, and other relevant controls should be
At this time, a concurrent verification may be selectedfrom recommendationsfor the test found
done for automated systemsto ensure that the in referencessuch as appropriate NCCLS guide-
instrument is detecting a minimum of 98% of lines, the current Manual of Clinical Microbiolo~
12 ELDER ET AL. CUMITECH 31

(15), or the Clinical Microbiology Procedures Hand-tion of all quality control, proficiency testing and
book (28). calibration results,aswell asany corrective action
2. Quality control analyte. Quality control ana- taken. Personnel competency is discussedin a
lytes are a metabolic product, nucleic acid, en- separatesectionof this document.
zyme, or antigen usually provided by the manu- When all these componentsare in place, the
facturer of a test or systemfor the routine quality user has assurancethat the test or test system
control of a specificprocedure or instrument. The meetsthe validation requirements.
analyte shouldbe identified with a lot number or
shouldhave other traceableidentification; further Quality Control Reagentsand/or Proficiency
description should indicate concentration, titer Test SamplesNot Available
(where appropriate),use,date of preparation, and There are frequently occurringcircumstancesin
storageconditions. For an analyte introduced by which quality control reagentsor proficiency test
the user laboratory, a defined record of its devel- samplesare not available or in which tests are of
opment and assayshould be available. The fre- such a nature that standard analytes cannot be
quencyof testingshouldfollow the manufacturer’s developed for proficiency test purposes.These
recommendationsor those stipulated by one or situationsinclude tests for unusual or rare ana-
more of the various regulatory or advisory (2, 17, lytes, testsfor labile analytes,relatively new tests,
19, 21) agencies.For home brew tests, positive, tests that are hard to standardize, or tests that
negative, and other relevant controls should be have low sensitivity or specificity. Someexamples
selectedfrom recommendationsfor the test found of thesesituationsinclude isolation of Haemophi-
in appropriate NCCLS guidelines, the current Zus ducreyi from specimens,serum bactericidal
Manual of Clinical Microbiology (15), or the Clin- tests,or direct fluorescent-antibodytestsfor Trep-
ical Microbiology Procedures Handbook (28). onema pallidum. In the absenceof adequatequal-
3. Proficiency test (survey) samples.Proficiency ity control materials or proficiency test samples,
test samplesare provided by CLIA ‘88-approved such tests would fail the College of American
proficiency testing programs(College of Ameri- Pathologistsaccreditationrequirement for valida-
can Pathologists,American Association of Bio- tion.
analysts,states, etc.) when a specific laboratory Several alternative approachesto satisfy the
enrolls in a program provided by that agency. validation requirement include the following.
HCFA reviewsapproved proficiency testing pro- 1. Split the patient sampleand senda portion to
grams on an annual basis. The frequency and a reference laboratory and compare the results.
rotation of the testing of various analytes are This would not haveto be donewith every sample,
usuallydeterminedby the provider but have been but it shouldbe done at least twice a year. If the
approved by HCFA. test volume is solow that even this approachis not
4. Instrument calibration. Certain instruments possible,then the laboratory should reexamine
require that specificcomponentsor internal sys- whether the test shouldbe offered at all, because
temsbe checkedon a regular basis.It is impera- of economics and personnel proficiency. It is
tive that the manufacturer’s directions for the advisableto inform the reference laboratory that
calibration be carried out at the specified time the samplesare to be usedasa meansof validating
intervals. This criterion may not apply to all tests a test procedure. If this approachcan be worked
or test systems. out mutually with another facility, the validation
5. Use of historical data (for blood culture data would serve two laboratories.
systems).Laboratories are encouragedto utilize 2. Split samplesmay be tested asunknownsin
historical data concerningrecovery of pathogens the user’s laboratory by ~2 testing personnel.
in their own populationsas an aid in confirming Again, this would only have to be done periodi-
that the systemis operating asexpected.If signif- cally, and the economics and personnel profi-
icant changesare seenin the distribution and/or ciency of the offering shouldbe reevaluated if the
frequency of recovery of isolatesobtained from test volume is low.
patients over a period of time (for example, a 3. Save known positive and negative samples
substantialreduction in the number of isolatesof and prepare in-housequality control and/or un-
Streptococcuspneumoniae from that obtainedwith known (proficiency testing) samples.This ap-
the old systemduring an equal time period), a proach may work well for highervolume testsand
more intensiveinvestigationinto the ability of the thosethat have a relatively high degreeof positiv-
new systemto support and detect thesespeciesis ity, so that sampleswith various degreesof posi-
warranted. tivity may be utilized.
6. Other. Other componentsof the validation 4. Obtain the analyte from an outside source
processmust include a determination and docu- and useit asa referencestandard.This approach
mentation of competencyof testing personnelin would work for tests or proceduresfor isolating
performing the proceduresand obtaining the cor- unusualorganismsthat might be available from
rect resultson an ongoingbasisand documenta- the American Type Culture Collection.
CUMITEiCH 31 VERIFICATION AND VALIDATION 13

Frequency of Test Validation complete program of employee training, verifica-


Individual laboratories are responsible for en- tion, and competency assessment (12,25,26). The
suring that test validation occurs frequently basic components of this process as it pertains to
enough to assure the continued performance of a incorporating a new procedure into the microbi-
laboratory test. In most cases, following the man- ology laboratory will be discussed in this section.
ufacturer’s guidelines and/or the requirements of
the regulatory or accrediting agencies will provide Writing the procedure(s)
this assurance. For home brew tests, it may be The key to successful training and consistency
necessary to perform test validation more fre- of test performance is a well-written procedure or
quently until it is determined how long the test will procedures which follow the guidelines of the
continue to perform satisfactorily. NCCLS document GP2-A2 (18). One suggested
method for preparing a procedure which lends
PERSONNEL TRAINING AND itself to training and demonstration of compe-
DEMONSTRATION OF COMPETENCY tency is outlined in the NCCLS document GP21-
A new test may not be introduced into the A (25). In this document, the process of procedure
laboratory until adequately trained personnel are writing hinges on identification of the critical steps
available to perform the test and accurately report of the procedure. These can be graphically out-
lined in a flow chart, and the flow chart can be
the results. In the microbiology section of the
clinical laboratory, the ability to recognize and used as a basis for writing the formal procedure
and (if desired) a separate training procedure.
respond to unusual or critical organisms or results
is crucial to good patient service. CLIA ‘88 states Identifying the critical steps also aids in the devel-
opment of learning objectives which focus on the
that the laboratory director must ensure that all
personnel have the appropriate education and key elements of the procedure.
experience, receive the appropriate training for
the type and complexity of the services offered, Preparing a training document
and demonstrate that they can perform all testing In order to assess the ability of testing personnel
operations reliably to provide and report accurate to successfully perform a new procedure, it is
results (7). The director must also ensure that necessary for the individual to clearly understand
policies and procedures for monitoring individuals what is expected of him or her when performing
who conduct preanalytical, analytical, and post- each of the steps of the procedure. Thus, a
analytical phases of testing are established to en- training document differs from a procedure in
sure that they are competent and that they main- several significant ways. First, it needs to clearly
tain their competency to process specimens, per- outline what outcomes are expected once the
form test procedures, report test results promptly individual has successfully completed training.
and proficiently, and, whenever necessary, identify These outcomes need to be stated as specific,
needs for remedial training or continuing educa- measurable learning objectives. Second, the train-
tion to improve skills. Finally, the responsibilities ing methods and required materials to be used
and duties of each person engaged in the perfor- during the training process need to be detailed.
mance of preanalytic, analytic, and postanalytic Training methods might include reading the pro-
phases of testing must be specified in writing, cedure, reading background material, viewing a
identifying which procedures the individual is training videotape or working through a comput-
authorized to perform and whether supervision is er-based training program, observing the proce-
required for specimen processing, test perfor- dure being performed, and performing the proce-
mance, or result reporting and whether director dure. Third, the training document needs to state
review is required prior to reporting patient test what measurement tool or tools will be used to
results. document as objectively as possible that the learn-
Personnel competency for persons performing ing objectives have been met. Suggested tools in
highly complex tests must be verified and docu- the NCCLS GP21-A document (25) include:
mented at least semiannually during the first year
the individual tests patient specimens and at least l testing of blind specimens, proficiency test spe-
annually thereafter, unless there is a change in cimens, or previously analyzed specimens
methodology or instrumentation. In the latter l administration of a written test
circumstance, the employee’s performance using l observation of procedure, process, and out-
the changed procedure must be reverified prior to come
reporting patient test results. If corrective action is l assessment of response to case studies, prob-
required, it should be completed and documented lems, or situations related to the procedure
in a timely fashion (i.e., 30 days) (12). l documentation of response to actual incidents
Several resources are available to assist the which may have occurred during the perfor-
laboratory in establishing and documenting a mance of the procedure (“critical incidents”)
14 ELDER ET AL. CUMITECH 31

l assessment of response to oral queries related laboratory. As stated at the outset, these are
to the procedure guidelines and should not be considered regula-
tory standards. For those responsible for estab-
Finally, if desired, the training document can in- lishing and maintaining standards in clinical
clude an additional section which describes what laboratories, there are many excellent docu-
the trained employee will need to do to demon- ments available, many of which are cited through-
strate ongoing competency in the performance of out this text. Our goal here is to make this in-
the procedure. formation more “microbiology friendly.” Ensuring
One approach to development of a training good laboratory practice, which includes comply-
document would be to include the training infor- ing with regulations from various agencies, can be
mation listed above, as well as a specific training a challenge. The availability of clear and useful
procedure (the complete technical procedure mod- guidelines for performing verification and valida-
ified to include explanatory training notes and tion which specifically address clinical microbiol-
information which will be useful as aids to train- ogy should make the accomplishment of this as-
ing). This training document would not require pect of good laboratory practice easier to achieve.
the individual to refer to the standard procedure
manual during the training process. If this ap-
proach is used, it requires extra diligence to en- APPENDIX A. METHOD SELECTION
sure that the training procedure is updated and EXAMPLE
modified whenever changes are made to the stan- I am considering a methodology change in my
dard laboratory procedure. laboratory. I hope to substitute a DNA probe for
A second approach to the training document Neisseria gonowhoeae (GC) in place of my stan-
would be to use the standard (unmodified) labo- dard GC culture. I will consider the culture the
ratory procedure when training and to limit the reference test in this situation. I know that about
training document to the specific training infor- 10% of my GC cultures are positive. How do I go
mation alone. This approach has the advantage of about verifying the method and seeing if it will
eliminating the effort required to ensure that the satisfy the needs of my clinicians? I begin by ask-
training procedure and the standard procedure ing my Obstetrics and Gynecology (OB-GYN)
are both modified consistently, and it ensures that department to send in duplicate samples. I obtain
the trainee will be using the actual laboratory 100 samples, perform both tests, and get the fol-
procedure when training. It has the disadvantage lowing results:
of not allowing inclusion of explanatory notes or
instructions which may not be part of the standard Culture results
procedure. An example of this type of document is New test results Positive Negative
presented in Appendix C. Positive 9 2
Negative 1 88
Documentation of training
Documentation of training can be accomplished From these data, I can calculate the sensitivity,
with a variety of written or computer-based doc- specificity, and positive and negative predictive
umentation systems. Specific examples of possible values of the new test:
checklists or documentation forms are included in Sensitivity = [9/(9 + l)] X 100 = 90%
the appendices of the NCCLS GP21-A guideline
(25), and these could be modified to suit the needs Specificity = [88/(88 + 2)] X 100 = 97.7%
PPV = [9/(9 + 2)] x 100 = 81.8%
of the laboratory. Procedures may be combined NPV = [88/(88 + 1)] x 100 = 98.9%
on one document to reduce the amount of paper-
work, or individual documents for each employee I now evaluatethe validity of theseresults.Have I
and procedure may be maintained. The American tested enough samplesto feel confident in the
Society of Clinical Pathologists has developed a generatedresults?
computer-based documentation program (ASCP With a prevalence of IO%, I will only expect
Comptec) (1) which may be purchased and used around IO positives out of my sample. Thus, the
to document training and ongoing competency. In closest I am likely to be when calculating sensitivity
addition, many commonly used commercial data- is within 10%. In other words, if I tested 100 samples
base programs may be used to design individual and did not get any false negatives, my sensitivity
laboratory-specific computer-based documenta- would be 100%. If the test failed to detect 1 positive,
tion programs. All documentation should be re- the sensitivity would calculate to 90%. If it failed to
tained for a minimum of two years. detect 2positives, the sensitivity would be 80%. This
is not a very discerning analysis.
SUMMATION How many samples should I test? If I stack the
This document provides guidance in performing deck and save samples and then per$orm the new test
test verification and validation in the microbiology on 100 patients with positive cultures and 100
CUMITECH 31 VERIFICATION AND VALIDATION 15

patients with negative cultures, I will be able to could feel confident about a negative result, but the
calculate sensitivity and specificity to the nearest 1%; positives are much too uncertain (in fact, I couldflip
if I test 50 of each, I can calculate to the nearest 2%. a coin and get as accurate a result).
How exact I want or need to be when I evaluate the So, what kind of sensitivity andspecificitywould
test will depend in large part on how critical the the test have to have at this low prevalence to
result is: whether it is to be used for diagnosis or satisfy theseclinicians?
screening, whether it stands alone or can be vali- Sensitivity will not affect PPK specificity will. So,
dated by other laboratory data, and how well the test
assuming I am testing the same IO,000 samples, 97
has been previously evaluated in laboratories and
will still be true positives (TP) and 3 will still be false
patient populations similar to my own. negatives. Now I can calculate how many false
In this case,assumethat I savesamplesand test positives (FP) I can have to achieve my desired
100 patients with positive GC cultures and 100 PPV
patientswith negativecultures and get the follow-
ing results: 95 = [TP/(TP + FP)] x 100
95 = [97/(97 + x)] x 100
Culture results x=5
New test results Positive Negative
Positive 97 1 Thus, of the 10,000 samples,I can have only 5
Negative 3 99 falsepositivesto achievethis goal, which meansof
my 9,906negativestested, 9895will have to give a
Now I calculatethe sensitivity and specificity. negative result with our new test. Thus, my table
Sensitivity is 97% and specificity is 99%. PPVand will look like this:
NPV are 99% and 97%, respectively, but they are
based on a prevalence of 50% because of the way the Culture results
study was designed.
Next, I find out that my OB-GYN physiciansare New test resu1ts Positive Negative
adamantthat they must have a high level of con- Positive 97 5
fidence in my positive results, becausethey have Negative 3 9895
recently been sued by a patient for pain and
sufferingwho was told that her test was positive Now I cancalculatethe new specificityrequired by
when it wasnot. Thesephysiciansdo mostly pre- the clinician by usingthe data from the table I just
natal screeningof a low-prevalence population filled out. Is the specificity for this test high
(only 1% of their patients are estimatedto have enough?Statistically,can I even answerthis ques-
GC). They would like a positive predictive value tion?
of 95% for their population. CanI offer this test to When I calculate the specificity, I see that at this
them?To makethe math easy,assumethat I can prevalence, to achieve the physician’s desired PPV I
test 10,000samplesin my evaluation; then, on the need a test with a specificity of 99.95%.Since I only
basisof the known sensitivity(97%) and specificity know the new test’s specificity to the nearest 1%
(99%) of this test, I fill out the expectedresultsin (based on the lOOpositives and negatives I originally
the table. tested) I cannot be sure that it will satis& the
clinician’s needs (although by common sense, it
Culture results probably will not!). To know for sure, I would
New test results Positive Negative have to be able to detect 5 of 10,000 (or 1 out of
Positive 97 99 2,000) false-positive specimens. Statistically, this
Negative 3 9801 means I would have to test approximately 6,000
negative specimens to be able to assure the clini-
I then calculatethe PPV and NPV in this case. cian with 95% confidence that this test would
On Jthe basis of the new prevalence, the PPV perform to the desired level of specificity. It is
would be 49% and the NPV would be 99.96%. I probably better to do culture!
16 ELDER ET AL. CUMITECH 31

Estimated Freauencv of Test Orders

-‘1 \
More than one/week L Less than one/week
I
Clinical and/or laboratory benefits I
of test (altered therapy, reduced /
hospital stay, faster reporting,
less expense etc.)

I Test performance corn peer


reviewed publications I

Test verification
(sensitivity, PPV,
etc.)

Implementation of
test is justified
CUMITEEH 31 VERIFICATION AND VALIDATION 17

APPENDIX C. SAMPLE OF LABORATORY Evaluation Criteria for Ongoing Verification of


TRAINING DOCUMENT Competency*
1. Ongoing supervisor review of technologist
Procedure: Urine Culture worksheetsof positive cultures indicates that the
Expected Outcome of Training: The trainee will individual correctly selectsappropriate identifica-
be able to: tion and susceptibilitytests (lessthan two errors
1. Describe the various methods used to collect detected per year).
specimens for urine culture and the advantages 2. Technologist successfully(95%) identifies
and disadvantages of each pathogenic and nonpathogenicflora on colonial
2. List the requirements for specimen storage morphology practical exam.* *
and transport to prevent bacterial growth in the 3. Technologist successfully(95%) identifies
specimen, evaluate individual specimens for ac- potential pathogensand correct workups on a
ceptability on the basis of the criteria, and reject variety of cultures (contaminated, noncontami-
or accept them correctly following laboratory pro- nated, multiple pathogens,etc.) in a written prac-
cedure tical exam.* *
3. Describe the procedures used to process 4. Technologist correctly identifies potential
specimens (clean catch, catheterized, suprapubic problemsin collection and transport of specimens
aspirate, etc.) for quantitative urine culture when presentedwith case histories as part of a
4. After incubation, calculate the original spec- written practical exam.* *
imen concentration of each of the organisms pres- *Note: all evaluations are “open book”; any written resources
ent in a culture normally available to the employee for performance of the job
5. List potential urinary tract pathogens and may be used when demonstrating competency.
recognize which organisms most commonly repre- **Note: the colonial morphology exam and written exam may
be used to evaluate multiple procedures at one time.
sent urethral or vaginal contamination of the spec-
imen
6. Recognize the morphology of potential path- REFERENCES
ogens and nonpathogens on blood agar and Mac- 1,, 1American Society of Clinical Pathologists. 1994. ASCP
/
Comptec: the Computerized Ongoing Monitoring Program to
Conkey agar Evaluate Competency. American Society of Clinical Pathol-
7. Correctly identify which organisms require ogists, Chicago, Ill.
full identification and susceptibility testing on the 2, College of American Pathologists. 1994. Section 1, p. 2, and
basis of their quantities and the presence of other section 4, p. 16-21. In Commission on Laboratory Accredi-
tation Inspection Checklist. College of American Patholo-
organisms in the culture gists, Northfield, Ill.
3. Doer-n, G.V., R. Vautour, M. Gauder, and B. Levy. 1994.
Clinical impact of rapid in vitro susceptibility testing and
Documents and References To Be Used during bacterial identification. J. Clin. Microbial. 32:1757-1762.
Training 4. Federal Register. 1995. Medical device user facilities and
manufacturers; adverse events, reporting requirements, cer-
1. Laboratory Specimen Collection Manual tification, and registration. Fed. Regist. 60:63578- 63606.
2. Bacteriology Procedure Manual 5. Ferraro,
. M. J., and J. H. Jorgensen. 1995. Instrument-based
3. Reference texts (e.g., Manual of Clinical antibacterial
, susceptibility testing, p. 1379-1384. In P. R.
Microbiology, Bailey and Scott, etc.) Murray, E. J. Baron, M. A. Pfaller, F. C. Tenover, and R. H.
Yolken (ed.), Manual of CZinicaZ McrobioZogy, 6th ed. Amer-
ican Society for Microbiology, Washington, D.C.
6. Hadgu, A. 1996. The discrepancy in discrepant analysis.
Training Methods To Be Used Lancet 348:592-593.
1. Readingof procedure in manual 7. Health Care Financing Administration. 1992. Medicare,
Medicaid, and CLIA programs. Regulations implementing
2. Supervised set-up, interpretation, workup, the Clinical Laboratory Improvement Amendments of 1988
and reporting of urine cultures (CLIA). Fed. Regist. 57~7002-7186.
8. Ilstrup, D. 1990. Statistical methods in microbiology. CZin.
Microbial. Rev. 3:219-226.
Evaluation Criteria for SuccessfulTraining* 9. Jenkins, S. G. 1995. Evaluation of new technology in the
1. When presentedwith casehistoriesdescrib- clinical microbiology laboratory. Diagn. Microbial. Infect.
Dis. 23:53-60.
ing collection and storageof urine specimens,the 10. Jorgensen, J. H. 1993. Selection criteria for an antimicrobial
trainee will correctly identify 100%of the time (i) susceptibility testing system. J. CZin. Microbial. 31:2841-2844.
improper procedures,(ii) the likely effect these 11. LeBar, W. D. 1996. Keeping up with new technology: new
approaches to diagnosis of Chlamydia infection. CZin. Chem.
will have had on the culture results,and (iii) the 42:809-812.
correct laboratory procedure to follow in accep- 12. McCaskey, L., and M. LaRocco. 1995. Competency testing in
tance or rejection of the specimen. clinical microbiology. Lab. Med. 26:343-349.
2. The trainee will correctly read, interpret, 13. Miller, J. M. 1991. Evaluating biochemical identification
systems. J. CZin. Microbial. 29: 1559 -1561.
work up, and report at least 100 urine cultures 14. Miller, J. M., and C. M. O’Hara. 1995. Substrate utilization
(minimumof 50 positivespecimens)without error systems for the identification of bacteria and yeasts, p.
or deviation from laboratory protocol. 103-109. In P. R. Murray, E. J. Baron, M. A. Pfaller, F. C.
18 ELDER ET AL. CUMITECH 31

Tenover, and R. H. Yolken (ed.), Manual of Clinical Micro- 1994. Development of In Vitro Susceptibility Testing Criteria
6th ed. American Society for Microbiology,
bioZogy, Wash- and Quality Control Parameters. Approved guideline M23-A.
ington, D.C. National Committee for Clinical Laboratory Standards,
15, Murray, P. R., E. J. Baron, M. A. Pfaller, F. C. Tenover, and Wayne, Pa.
R. H. Yolken (ea.). 1995. Manual of Clinical Microbiology, 23 National Committee for Clinical Laboratory Standards.
6th ed. American Society for Microbiology, Washington, 1994. Specifications for Immunological Testing for Infectious
D.C. Diseases. Approved guideline I/LA 18-A. National Commit-
16. Murray, P. R., A. C. Niles, and R, L. Heeren. 1987. Com- tee for Clinical Laboratory Standards, Wayne, Pa.
parison of a highly automated 5-h susceptibility testing 24 National Committee for Clinical Laboratory Standards.
system, the Cobas-Bact, with two reference methods: Kirby- 1995. Assessment of the Clinical Accuracy of Laboratory Tests
Bauer disk diffusion and broth microdilution. J. Clin. Micro- Using Receiver Operating Characteristic (ROC) Plots. Ap-
biol. 252372-2377. proved guideline GPlO-A. National Committee for Clinical
17. National Committee for Clinical Laboratory Standards. Laboratory Standards, Wayne, Pa.
1996. Quality Assurance for Commercially Prepared Microbi- 25 National Committee for Clinical Laboratory Standards.
ological Culture Media.Approved standard M22-A2. Na- 1995. Training Verification of Laboratory Personnel. Ap-
tional Committee for Clinical Laboratory Standards, Wayne, proved guideline GP21-A. National Committee for Clinical
Pa. Laboratory Standards, Wayne, Pa.
18. National Committee for Clinical Laboratory Standards. 26
1992, Clinical Laboratory Technical Procedure Manuals, 2nd Nevalainen, D. E., and L. M. Berte. 1993. Training Verifica-
ed. Approved guideline GP2A2. National Committee for
tion and Assessment: Keys to Quality Clinical
Management.
Clinical Laboratory Standards, Wayne, Pa. Laboratory Management Association, Paoli, Pa.
19. National Committee for Clinical Laboratory Standards. 27 Radetsky, M., and J. K. Todd. 1984. Criteria for the evalu-
1993. Methods for Dilution Antimicrobial Susceptibility Tests 3. ation of new diagnostic tests. Pediatr. Infect. Dis. 3:461-466.
for Bacteria That Grow Aerobically, 3rd ed. Approved Stan- Lo Sewell, D. L. (ea.). 1992. Quality assurance, quality control,
dard M7-A3. National Committee for Clinical Laboratory laboratory records, and water quality, p. 13.0.1-13.4.11. In H.
Standards, Wayne, Pa. D. Isenberg (ed.), Clinical Microbiology Procedures Hand-
20. National Committee for Clinical Laboratory Standards. book, vol. 2. American Society for Microbiology, Washing-
1993. Nomenclature and Definitions for Use in NRSCL and ton, D.C.
Other NCCLS Documents, 2nd ed. Proposed guideline NR- 29. Sewell, D. L., and R B. Schifinan. 1995. Quality assurance:
SCL 8-P2. National Committee for Clinical Laboratory ,
quality improvement, quality control, and test validation, p.
Standards, Wayne, Pa. 55-66. In P. R. Murray, E. J. Baron, M. A. Pfaller, F. C.
21. National Committee for Clinical Laboratory Standards. Tenover, and R. H. Yolken (ed.), Manual of Clinical Micro-
1993. Performance Standards for Antimicrobial Disk Suscep- bioZogy, 6th ed. American Society for Microbiology, Wash-
tibility Tests, 5th ed. Approved standard M2A5. National ington, D.C.
Committee for Clinical Laboratory Standards, Wayne, Pa. 30. Wilson, M. L. 1994. Blood Cultures-Introduction. Chin.
22. National Committee for Clinical Laboratory Standards. Lab. Med. 14~1-7.

S-ar putea să vă placă și