Reader Criminal Is Tics 2011-2012

Reader Criminalistics 2011-2012
History
1. Inman, K. & Rudin, R. (2002) 'Forensic Science Timeline'.
Evidential evaluation
2. Association of Forensic Science Providers (2009) 'Standards for the formulation of
evaluative forensic science expert opinion', Science & Justice 49, 161-164.
3. Berger C.E.H. (2010) 'Criminalistics is reasoning backwards, Logically correct
reasoning in forensic reports ... and in the courtroom', unofficial translation of:
Nederlands Juristenblad, 784-789.
4. Berger C.E.H. & Aben D.J.C. (2010) 'Evidence and conviction: Rational reasoning
since Aristotle', unofficial translation of: Expertise en Recht, 52-56.
5. Evett I.W. (1998) 'Towards a uniform framework for reporting opinions in forensic
science casework', Science & Justice 38, 198-202.
6. Inman, K. & Rudin, R. (2002) 'The Origin of Evidence', Forensic Science International
126, 11-16.
7. Jackson G., Jones S., Booth G., Champod C., Evett I.W. (2006) 'The nature of
forensic science opinion - a possible framework to guide thinking and practice in
investigations and in court proceedings', Science & Justice 46, 33-44.
8. Robertson B., Vignaux G.A. (1995) 'Interpreting Evidence - Evaluating Forensic
Science in the Courtroom', John Wiley & Sons Inc., Chichester, Chapters 2 & 3.
9. Saini A. (2009) 'Justice you can count on', NewScientist, 43-45.
10. Saks, MJ. & Koehler, J.J. (2005) 'The Coming Paradigm Shift in Forensic
Identification Science' Science 309, 892-895.
11. Stoney, D.A. (1991) 'What made us ever think we could individualize using
statistics?', Journal of the Forensic Science Society 31(2), 197-199.
DNA
12. Dann, M.P., Hans, V.P. & D.H. Kaye (2006) 'Can Jury Trial Innovations Improve
Juror Understanding of DNA Evidence', NIJ Journal 255, 2-8.
13. Jobling, M.A. & P. Gill (2004) 'Encoded Evidence: DNA in Forensic Analysis', Nature
Reviews: Genetics, 5, 739-751.
Fingerprints
14. Egli N.M., Champod C., Margot P. (2007) 'Evidence evaluation in fingerprint
comparison and automated fingerprint identification systemsModelling within finger
variability', Forensic Science International 167, 189-195.
15. Meuwly, D. (2006) 'Forensic Individualisation from Biometric Data', Science & Justice
46, 205-213.
Case related
16. Berger C.E.H., Buckleton J.S., Champod C., Evett I.W., Jackson G. (2011) 'Evidence
evaluation: A response to the court of appeal judgment in R v T', Science & Justice
51, 43-49.
17. Broeders, A.P.A. (2006) 'Of Earprints, Fingerprints, Scent Dogs, Cot Deaths and
Cognitive Contamination - a Brief Look at the Present State of Play in the Forensic
Arena', Forensic Science International 159 (2-3), 148-157.
18. Carpenter, R.G., Waite, A., Coombs, R.C., Daman-Willems, C., McKenzie, A., Huber,
J., & J.L. Emery (2005) 'Repeat sudden unexpected and unexplained infant deaths:
natural or unnatural?', Lancet 365, 29-35.
19. Robertson B., Vignaux G.A., Berger C.E.H. (2011) 'Extending the Confusion About
Bayes', Modern Law Review 74, 444-455.
Contextual bias
20. Dror, I. & Charlton, D. (2006) 'Why experts make errors', Journal of Forensic
Identification 56(4), 600-616.
21. Dror, I., Charlton, D. & A.E. Peron (2006) 'Contextual information renders experts
vulnerable to making erroneous identifications' Forensic Science International 156
(1), 74-78.
22. Saks M.J., Risinger D.M., Rosenthal R., Thompson W.C. (2003) 'Context effects in
forensic science: A review and application of the science of science to crime
laboratory practice in the United States', Science & Justice 43, 77-90.
23. Thompson, W.C. (2009) 'Painting the target around the matching profile: the Texas
sharpshooter fallacy in forensic DNA interpretation', Law, Probability and Risk 8,257-
276.
Criticism
24. Drummer, O., Forrest, A.R., Goldberger, B. & S.B. Karch (2004) 'Forensic Science in
the Dock', British Medical Journal 329, 636-637.
25. Faigman, D.L. (2002) 'Is science different for lawyers?', Science 297, 339-340.
26. Houck, M.M. (2006) 'CSI: Reality', Scientific American, 67-71.
27. Kennedy D. (2003) 'Forensic Science: Oxymoron?', Science 302, 1625.
28. Schweitzer, N.J. and Michael J. Saks (2007), The CSI Effect: Popular Fiction about
Forensic Science Affects the Public's Expectations about Real Forensic Science, 47
Jurimetrics J. 357-364.
The Forensic Science Timeline can also be found as an appendix in our recently published book
Principles and Practice of Forensic Science: The Profession of Forensic Science
http://forensicdna.com/Bookstore/index.html
See also the Forensic Science Bibliography
http://forensicdna.com/Bibliography.html
This is a ~work in progress
Please e-mail comments and suggestions
http://forensicdna.com/~emailforms/emailtimeline.html
Forensic Science Timeline
updated 2/7/02
BCE Evidence of fingerprints in early paintings and rock carvings of prehistoric humans
700s Chinese used fingerprints to establish identity of documents and clay sculpture, but without any formal classification
system.
(1000) Quintilian, an attorney in the Roman courts, showed that bloody palm prints were meant to frame a blind man of his
mothers murder.
1248 A Chinese book, Hsi Duan Yu (the washing away of wrongs), contains a description of how to distinguish drowning
from strangulation. This was the first recorded application of medical knowledge to the solution of crime.
1609 The first treatise on systematic document examination was published by Franois Demelle of France
1686 Marcello Malpighi, a professor of anatomy at the University of Bologna, noted fingerprint characteristics. However,
he made no mention of their value as a tool for individual identification.
1784 In Lancaster, England, 1ohn Toms was convicted of murder on the basis of the torn edge of wad of newspaper in a
pistol matching a remaining piece in his pocket. This was one of the first documented uses of physical matching.
(1800s) Thomas Bewick, an English naturalist, used engravings of his own fingerprints to identify books he published.
1810 Eugne Franois Vidocq, in return for a suspension of arrest and a jail sentence, made a deal with the police to
establish the first detective force, the Sret of Paris.
1810 The first recorded use of question document analysis occurred in Germany. A chemical test for a particular ink dye
was applied to a document known as the Konigin Hanschritt.
1813 Mathiew Orfila, a Spaniard who became professor of medicinal/forensic chemistry at University of Paris, published
Traite des Poisons Tires des Regnes Mineral, Vegetal et Animal, ou Toxicologie General l. Orfila is considered the
father of modern toxicology. He also made significant contributions to the development of tests for the presence of
blood in a forensic context and is credited as the first to attempt the use of a microscope in the assessment of blood and
semen stains.
1823 1ohn Evangelist Purkinji, a professorprofessor of anatomy at the University of Breslau, Czecheslovakia, published
the first paper on the nature of fingerprints and suggested a classification system based on nine major types. However,
he failed to recognize their individualizing potential.
1828 William Nichol invented the polarizing light microscope.
(1830s) Adolphe Quetelet, a Belgian statistician, provided the foundation for Bertillons work by stating his belief that no two
human bodies were exactly alike.
1831 Leuchs first noted amylase activity in human saliva.
1835 Henry Goddard, one of Scotland Yards original Bow Street Runners, first used bullet comparison to catch a
murderer. His comparison was based on a visible flaw in the bullet which was traced back to a mold.
1836 1ames Marsh, an Scottish chemist, was the first to use toxicology (arsenic detection) in a jury trial.
1839 H. Bayard published the first reliable procedures for the microscopic detection of sperm. He also noted the different
microscopic characteristics of various different substrate fabrics.
1851 1ean Servais Stas, a chemistry professorprofessor from Brussels, Belgium, was the first successfully to identify
vegetable poisons in body tissue.
1853 Ludwig Teichmann, in Kracow, Poland, developed the first microscopic crystal test for hemoglobin using hemin
crystals.
1854 An English physician, Maddox, developed dry plate photography, eclipsing M. Daguerre`s wet plate on tin method.
This made practical the photographing of inmates for prison records.
1856 Sir William Herschel, a British officer working for the Indian Civil service, began to use thumbprints on documents
both as a substitute for written signatures for illiterates and to verify document signatures.
1862 The Dutch scientist 1. (Izaak) Van Deen developed a presumptive test for blood using guaiac, a West Indian shrub.
1863 The German scientist Schnbein first discovered the ability of hemoglobin to oxidize hydrogen peroxide making it
foam. This resulted in first presumptive test for blood.
1864 Odelbrecht first advocated the use of photography for the identification of criminals and the documentation of
evidence and crime scenes.
1877 Thomas Taylor, microscopist to U.S. Department of Agriculture suggested that markings of the palms of the hands
and the tips of the fingers could be used for identification in criminal cases. Although reported in the American
Journal of Microscopy and Popular Science and Scientific American, the idea was apparently never pursued from this
source.
1879 Rudolph Virchow, a German pathologist, was one of the first to both study hair and recognize its limitations.
1880 Henry Faulds, a Scottish physician working in Tokyo, published a paper in the journal Nature suggesting that
fingerprints at the scene of a crime could identify the offender. In one of the first recorded uses of fingerprints to solve
a crime, Faulds used fingerprints to eliminate an innocent suspect and indicate a perpetrator in a Tokyo burglary.
1882 Gilbert Thompson, a railroad builder with the U.S Geological Survey in New Mexico, put his own thumbprint on
wage chits to safeguard himself from forgeries.
1883 Alphonse Bertillon, a French police employee, identified the first recidivist based on his invention of anthropometry.
1887 Arthur Conan Doyle published the first Sherlock Holmes story in Beetons Christmas Annual of London.
1889 Alexandre Lacassagne, professorprofessor of forensic medicine at the University of Lyons, France, was the first to
try to individualize bullets to a gun barrel. His comparisons at the time were based simply on the number of lands and
grooves.
1891 Hans Gross, examining magistrate and professor of criminal law at the University of Graz, Austria, published
Criminal Investigation, the first comprehensive description of uses of physical evidence in solving crime. Gross is also
sometimes credited with coining the word criminalistics.
1892 (Sir) Francis Galton published Fingerprints, the first comprehensive book on the nature of fingerprints and their use
in solving crime.
1892 1uan Vucetich, an Argentinean police researcher, developed the fingerprint classification system that would come to
be used in Latin America. After Vucetich implicated a mother in the murder of her own children using her bloody
fingerprints, Argentina was the first country to replace anthropometry with fingerprints.
1894 Alfred Dreyfus of France was convicted of treason based on a mistaken handwriting identification by Bertillon.
1896 Sir Edward Richard Henry developed the print classification system that would come to be used in Europe and
North America. He published Classification and Uses of Finger Prints.
1898 Paul 1esrich, a forensic chemist working in Berlin, Germany, took photomicrographs of two bullets to compare, and
subsequently individualize, the minutiae.
1901 Paul Uhlenhuth, a German immunologist, developed the precipiten test for species. He was also one of the first to
institute standards, controls, and QA/QC procedures. Wassermann (famous for developing a test for syphilis) and
Schtze independently discovered and published the precipiten test, but never received due credit.
1900 Karl Landsteiner first discovered human blood groups and was awarded the Nobel prize for his work in 1930. Max
Richter adapted the technique to type stains. This is one of the first instances of performing validation experiments
specifically to adapt a method for forensic science. Landsteiner's continued work on the detection of blood, its
species, and its type formed the basis of practically all subsequent work.
1901 Sir Edward Richard Henry was appointed head of Scotland Yard and forced the adoption of fingerprint
identification to replace anthropometry.
1901 Henry P. DeForrest pioneered the first systematic use of fingerprints in the United States by the New York Civil
Service Commission.
1902 Professor R.A. Reiss, professor at the University of Lausanne, Switzerland, and a pupil of Bertillon, set up one of the
first academic curricula in forensic science. His forensic photography department grew into Lausanne Institute of
Police Science.
1903 The New York State Prison system began the first systematic use of fingerprints in United States for criminal
identification.
1903 At Leavenworth Federal Penitentiary, Kansas, Will West, a new inmate, was initially confused with a resident convict
William West using anthropometry. They were later (1905) found to be easily differentiated by their fingerprints. For a
historical clarification, please see http://www.scafo.org/library/110105.htm
1904 Oskar and Rudolf Adler developed a presumptive test for blood based on benzidine, a new chemical developed by
Merk.
1905 American President Theodore Roosevelt established Federal Bureau of Investigation (FBI).
1910 Victor Balthazard, professor of forensic medicine at the Sorbonne, with Marcelle Lambert, published the first
comprehensive hair study, Le poil de l'homme et des animaux. In one of the first cases involving hairs, Rosella
Rousseau was convinced to confess to murder of Germaine Bichon. Balthazard also used photographic enlargements
of bullets and cartridge cases to determining weapon type and was among the first to attempt to individualize a bullet
to a weapon.
1910 Edmund Locard, successor to Lacassagne as professor of forensic medicine at the University of Lyons, France,
established the first police crime laboratory.
1910 Albert S. Osborne, an American and arguably the most influential document examiner, published Questioned
Documents.
1912 Masaeo Takayama developed another microscopic crystal test for hemoglobin using hemochromogen crystals.
1913 Victor Balthazard, professor of forensic medicine at the Sorbonne, published the first article on individualizing bullet
markings.
1915 Leone Lattes, professor at the Institute of Forensic Medicine in Turin Italy, developed the first antibody test for ABO
blood groups. He first used the test in casework to resolve a marital dispute. He published LIndividualit del sangue
nella biologia, nella clinica, nella medicina, legale, the first book dealing not only with clinical issues, but heritability,
paternity, and typing of dried stains.
1915 International Association for Criminal Identification, (to become The International Association of Identification
(IAI), was organized in Oakland, California.
1916 Albert Schneider of Berkeley, California first used a vacuum apparatus to collect trace evidence.
1918 Edmond Locard first suggested 12 matching points as a positive fingerprint identification.
1904 Locard published L'enquete criminelle et les methodes scientifique, in which appears a passage that may have given
rise to the forensic precept that Every contact leaves a trace.
1920 Charles E. Waite was the first to catalog manufacturing data about weapons.
1920s Georg Popp pioneered the use of botanical identification in forensic work.
1920s Luke May, one of the first American criminalists, pioneered striation analysis in tool mark comparison, including an
attempt at statistical validation. In 1930 he published The identification of knives, tools and instruments, a positive
science, in The American Journal of Police Science.
(1920s) Calvin Goddard, with Charles Waite, Phillip O. Gravelle, and John H Fisher, perfected the comparison microscope
for use in bullet comparison.
1921 1ohn Larson and Leonard Keeler designed the portable polygraph.
1923 Vittorio Siracusa, working at the Institute of Legal Medicine of the R. University of Messina, Italy, developed the
absorbtion-elution test for ABO blood typing of stains. Along with his mentor, Lattes also performed significant work
on the absorbtion-inhibition technique.
1923 In Frye v. United States, polygraph test results were ruled inadmissible. The federal ruling introduced the concept of
general acceptance and stated that polygraph testing did not meet that criterion.
1924 August Vollmer, as chief of police in Los Angeles, California, implemented the first U.S. police crime laboratory.
1925 Saburo Sirai, a Japanese scientist, is credited with the first recognition of secretion of group-specific antigens into
body fluids other than blood.
1926 The case of Sacco and Vanzetti, which took place in Bridgewater, Massachusetts, was responsible for popularizing
the use of the comparison microscope for bullet comparison. Calvin Goddard`s conclusions were upheld when the
evidence was reexamined in 1961.
1927 Landsteiner and Levine first detected the M, N, and P blood factors leading to development of the MNSs and P
typing systems.
1928 Meller was the first medico-legal investigator to suggest the identification of salivary amlyase as a presumptive test
for salivary stains.
1929 K. I. Yosida , a Japanese scientist, conducted the first comprehensive investigation establishing the existence of
serological isoantibodies in body fluids other than blood.
1929 Calvin Goddard`s work on the St. Valentine`s day massacre led to the founding of the Scientific Crime Detection
Laboratory on the campus of Northwestern University, Evanston, Illinois.
1930 American Journal of Police Science was founded and published by staff of Goddards Scientific Crime Detection
Laboratory in Chicago. In 1932, it was absorbed by Journal of Criminal Law and Criminology, becoming the Journal
of Criminal Law, Criminology and police science.
1931 Franz 1osef Holzer, an Austrian scientist, working at the Institute for Forensic Medicine of the University of
Innsbruck, developed the absorbtion-inhibition ABO typing technique that became the basis of that commonly used in
forensic laboratories. It was based on the prior work of Siracusa and Lattes.
1932 The Federal Bureau of Investigation (FBI) crime laboratory was created.
1935 Frits Zernike, a Dutch physicist, invented the first interference contrast microscope, a phase contrast microscope, an
achievement for which he won the Nobel prize in 1953.
1937 Holzer published the first paper addressing the usefulness of secretor status for forensic applications.
1937 Walter Specht, at the University Institute for Legal Medicine and Scientific Criminalistics in Jena, Germany,
developed the chemiluminescent reagent luminol as a presumptive test for blood.
1937 Paul Kirk assumed leadership of the criminology program at the University of California at Berkeley. In 1945, he
formalized a major in technical criminology.
1938 M. Polonovski and M. 1ayle first identified haptoglobin.
1940 Landsteiner and A.S. Wiener first described Rh blood groups.
1940 Vincent Hnizda, a chemist with the Ethyl Corporation, was probably the first to analyze ignitable fluid. He used a
vacuum distillation apparatus.
1941 Murray Hill of Bell Labs initiated the study voiceprint identification. The technique was refined by L.G. Kersta.
1945 Frank Lundquist, working at the Legal Medicine Unit at the University of Copenhagen, developed the acid
phosphatase test for semen.
1946 Mourant first described the Lewis blood group system.
1946 R.R. Race first described the Kell blood group system
1950 M. Cutbush, and colleagues first described the Duffy blood group system.
1950 August Vollmer, chief of police of Berkeley, California, established the school of criminology at the University of
California at Berkeley. Paul Kirk presided over the major of criminalistics within the school..
1950 Max Frei-Sulzer, founder of the first Swiss criminalistics laboratory, developed the tape lift method of collecting
trace evidence.
1950 The American Academy of Forensic Science (AAFS) was formed in Chicago, Illinois. The group also began
publication of the 1ournal of Forensic Science (1FS).
1951 F. H. Allen and colleagues first described the Kidd blood grouping system.
1953 Kirk published Crime Investigation, one of the first comprehensive criminalistics and crime investigation texts that
encompassed theory in addition to practice.
1954 R. F. Borkenstein, captain of the Indiana State Police, invented the Breathalyzer for field sobriety testing.
1958 A. S. Weiner and colleagues introduced the use of H-lectin to determine positively O blood type.
1959 Hirshfeld first identified the polymorphic nature of group specific component (Gc).
1960 Lucas, in Canada, described the application of gas chromatography (GC) to the identification of petroleum products in
the forensic laboratory and discussed potential limitations in the brand identity of gasoline.
1960s Maurice Muller, a Swiss scientist, adapted the Ouchterlony antibody-antigen diffusion test for precipiten testing to
determine species.
1963 D.A. Hopkinson and colleagues first identified the polymorphic nature of erythrocyte acid phosphatase (EAP).
1964 N. Spencer and colleagues first identified the polymorphic nature of red cell phosphoglucomutase (PGM).
1966 R. A. Fildes and H. Harris first identified the polymorphic nature of red cell adenylate cyclase (AK).
1966 Brian 1. Culliford and Brian Wraxall developed the immunoelectrophoretic technique for haptoglobin typing in
bloodstains.
1967 Culliford, of the British Metropolitan Police Laboratory, initiated the development of gel-based methods to test for
isoenzymes in dried bloodstains. He was also instrumental in the development and dissemination of methods for
testing proteins and isoenzymes in both blood and other body fluids and secretions.
1968 Spencer and colleagues first identified the polymorphic nature of red cell adenosine deaminase (ADA).
1971 Culliford published The Examination and Typing of Bloodstains in the Crime Laboratory, generally accepted as
responsible for disseminating reliable protocols for the typing of polymorphic protein and enzyme markers to the
United States and worldwide.
1973 Hopkinson and colleagues first identified the polymorphic nature of esterase D (ESD).
1974 The detection of gunshot residue (GSR) using scanning electron microscopy with electron dispersive X-rays (SEM-
EDX) technology was developed by 1. E. Wessel, P. F. 1ones, Q. Y. Kwan, R. S. Nesbitt and E. 1. Rattin at
Aerospace Corporation.
1975 1. Kompf and colleagues, working in Germany, first identified the polymorphic nature of red cell glyoxylase (GLO).
1975 The Federal Rules of Evidence, originally promulgated by the U.S. Supreme Court, were enacted as a congressional
statute. They are based on the relevancy standard in which scientific evidence that is deemed more prejudicial than
probative may not be admitted.
1976 Zoro and Hadley in the United Kingdom first evaluated GC-MS for forensic purposes.
1977 Fuseo Matsumur, a trace evidence examiner at the Saga Prefectural Crime Laboratory of the National Police Agency
of Japan, notices his own fingerprints developing on microscope slides while mounting hairs from a taxi driver murder
case. He relates the information to co-worker Masato Soba, a latent print examiner. Soba would later that year be the
first to develop latent prints intentionally by Superglue
fuming.
(1977) The fourier transform infrared spectrophotometer (FTIR) is adapted for use in the forensic laboratory.
(1977) The FBI introduced the beginnings of its Automated Fingerprint Identification System (AFIS) with the first
computerized scans of fingerprints.
1978 Brian Wraxall and Mark Stolorow developed the multisystem method for testing the PGM, ESD, and GLO
isoenzyme systems simultaneously. They also developed methods for typing blood serum proteins such as haptoglobin
and Gc.
1984 (Sir) Alec 1effreys developed the first DNA profiling test. It involved detection of a multilocus RFLP pattern. He
published his findings in Nature in 1985.
1986 In the first use of DNA to solve a crime, 1effreys used DNA profiling to identify Colin Pitchfork as the murderer of
two young girls in the English Midlands. Significantly, in the course of the investigation, DNA was first used to
exonerate an innocent suspect.
1983 The polymerase chain reaction (PCR) was first conceived by Kerry Mullis, while he was working at Cetus
Corporation. The first paper on the technique was not published until 1985.
1986 The human genetics group at Cetus Corporation, led by Henry Erlich, developed the PCR technique for a number of
clinical and forensic applications. This resulted in development of the first commercial PCR typing kit specifically for
forensic use, HLA DQ (DQA1), about 2 years later.
1986 In People v. Pestinikas, Edward Blake first used PCR-based DNA testing (HLA DQ) , to confirm different autopsy
samples to be from the same person. The evidence was accepted by a civil court. This was also the first use of any kind
of DNA testing in the United States
1987 DNA profiling was introduced for the first time in a U.S. criminal court. Based on RFLP analysis performed by
Lifecodes, Tommy Lee Andrews was convicted of a series of sexual assaults in Orlando, Florida.
1987 New York v. Castro was the first case in which the admissibility of DNA was seriously challenged. It set in motion a
string of events that culminated in a call for certification, accreditation, standardization, and quality control guidelines
for both DNA laboratories and the general forensic community.
1988 Lewellen, McCurdy, and Horton, and Asselin, Leslie, and McKinley both publish milestone papers introducing a
novel procedure for the analysis of drugs in whole blood by homogeneous enzyme immunoassay (EMIT).
1990 K. Kasai and colleagues published the first paper suggesting the D1S80 locus (pMCT118) for forensic DNA analysis.
D1S80 was subsequently developed by Cetus (subsequently Roche Molecular Systems) corporation as a commercially
available forensic DNA typing system.
1992 In response to concerns about the practice of forensic DNA analysis and interpretation of the results, the National
Research Council Committee on Forensic DNA (NRC I) published DNA Technology in Forensic Science.
1992 Thomas Caskey, professor at Baylor University in Texas, and colleagues published the first paper suggesting the use
of short tandem repeats for forensic DNA analysis. Promega corporation and Perkin-Elmer corporation in
collaboration with Roche Molecular Systems independently developed commercial kits for forensic DNA STR
typing.
1991 Walsh Automation Inc., in Montreal, launched development of an automated imaging system called the Integrated
Ballistics Identification System, or IBIS, for comparison of the marks left on fired bullets, cartridge cases, and shell
casings. This system was subsequently developed for the U.S. market in collaboration with the Bureau of Alcohol,
Tobacco, and Firearms (ATF).
1992 The FBI contracted with Mnemonic Systems to developed Drugfire, an automated imaging system to compare marks
left on cartridge cases and shell casings. The ability to compare fired bullets was subsequently added.
1993 In Daubert et al. v. Merrell Dow, a U.S. federal court relaxed the Frye standard for admission of scientific evidence
and conferred on the judge a gatekeeping role. The ruling cited Karl Poppers views that scientific theories are
falsifiable as a criterion for whether something is scientific knowledge and should be admissible.
(1994) Roche Molecular Systems (formerly Cetus) released a set of five additional DNA markers (polymarker) to add to
the HLA-DQA1 forensic DNA typing system.
1996 In response to continued concerns about the statistical interpretation of forensic DNA evidence, a second National
Research Council Committee on Forensic DNA (NRC II) was convened and published The Evaluation of Forensic
DNA Evidence.
1996 The FBI introduced computerized searches of the AFIS fingerprint database. Live scan and card scan devices allowed
interdepartmental submissions.
1996 In Tennessee v. Ware, mitochondrial DNA typing was admitted for the first time in a U.S. court.
1998 An FBI DNA database, NIDIS, enabling interstate cooperation in linking crimes, was put into practice.
1999 The FBI upgraded its computerized fingerprint database and implemented the Integrated Automated Fingerprint
Identification System (IAFIS), allowing paperless submission, storage, and search capabilities directly to the national
database maintained at the FBI.
1999 A Memorandum of Understanding is signed between the FBI and ATF, allowing the use of the National Integrated
Ballistics Network (NIBIN), to facilitate exchange of firearms data between Drugfire and IBIS.

REFERENCES
Block, E. B., Science vs. Crime: The Evolution of the Police Lab, Cragmont Publications, 1979.
Dillon D., A History of Criminalistics in the United States 1850-1950, Doctoral Thesis, University of California, Berkeley, 1977.
Else, W. M. and Garrow, 1. M., The Detection of Crime, The Police JournalLondon, 1934.
Gaensslen, R. E., Ed., Sourcebook in Forensic Serology, Unit IX: Translations of Selected Contributions to the Original Literature of Medicolegal Examination of
Blood and Body Fluids, National Institute of Justice, 1983.
Gaensslen, R. E., Sourcebook in Forensic Serology, U.S. Government Printing Office, Washington, D.C., 1983.
Gerber, S. M., Saferstein, R., More Chemistry and Crime, American Chemical Society, 1997.
German, E., Cyanoacrylate (Superglue) Discovery Timeline 1999. http://onin.com/fp/cyanoho.html
German, E., The History of Fingerprints, 1999. http://onin.com/fp/fphistory.html
Kind, S., and Overman, M., Science Against Crime, Aldus Book Limited, Doubleday, US. 1972.
Morland, N., An Outline of Scientific Criminology, Philosophical Library, NY, 1950.
Olsen, R. D., Sr., A Fingerprint Fable: The Will and William West Case, (initially published in: Identification News which became Journal of Forensic Identification,
37: 11, 1987), Kansas Bureau of Investigation. http://www.scafo.org/library/110105.html
Thorwald, 1., Crime and Science, Harcourt, Brace & World, Inc., New York, 1966, Translation, Richard and Clara Winston.
Thorwald, 1., The Century of the Detective, Harcourt, Brace & World, Inc., New York, 1964, Translation, Richard and Clara Winston, 1965.
Standards for the formulation of evaluative forensic science expert opinion
1. Scope
Forensic Expert Opinion formulated in the Evaluative or Evidential
mode across all scientic disciplines.
It does not include investigative opinion or factual and technical
reporting.
2. Denitions
2.1. Evaluative opinion
An opinion of evidential weight (evaluation of a likelihood ratio),
based upon case specic propositions and clear conditioning informa-
tion (framework of circumstances) that is provided for use as evidence
in court.
An evaluative opinion is an opinion based upon the estimation of a
likelihood ratio.
2.2. Investigative opinion
This arises in casework in which explanations are generated to
account for observations (the outcome of analytical tests or visual ex-
aminations). In some circumstances these explanations may be ranked
using estimates of probabilities based upon the knowledge and ex-
perience of the expert and taking into account all uncertainties relating
to the observations and the framework of circumstances. The provision
of an explanation for an observation is termed an investigative opinion.
2.3. Technical (factual) reporting
This is the factual reporting of a test outcome based solely on the
technical competence of the individual. No inferences/explanations
(opinion) are drawn fromthe test results (observations) a technical
interpretation is provided. For examples see Section 7.
3. Guiding principles
The expert will base his opinion upon the four principles: balance,
logic, robustness and transparency. The standards set out in this doc-
ument describe the mechanism by which these principles will be
applied in formulating such opinion.
Balance The expert should address at least one pair of pro-
positions usually one based upon the prosecution issue and one based
upon an alternative (defence issue). If a reasonable alternative cannot
be identied then the expert may address only the one proposition but
will make it clear that he cannot evaluate the strength of the evidence.
Logic The expert will address the probability of the evidence
given the proposition and relevant background information and not
the probability of the proposition given the evidence and background
information.
Robustness The expert will provide opinion that is capable of
scrutiny by other experts and cross-examination. He will base his
opinion upon sound knowledge of the evidence type(s) and use where-
ver possible veried databases. He will be satised that the results of
the tests and examinations upon which he has based his opinion are
themselves robust.
Transparency The expert will be able to demonstrate how he
came to his conclusion. He will set out in the statement or report the
basis of his opinion viz.:
Propositions addressed.
Test or examination results.
The background information he has used in arriving at his conclusion.
He will be able, if required, to provide the data he has used and its
provenance.
It is the duty of an expert to help both prosecution and defence and
the courts as much as possible by using his expert knowledge.
Therefore he should address propositions at activity level wherever
possible.
An expert will not give evaluative opinion on matters outside his
own area of expertise.
An expert will not usually give his opinion on issues that do not
require expert knowledge. However, if asked, he may do so provided it
is made clear that he is not giving expert opinion.
4. Standards
(The superscripts refer to Guidance notes in Section 5)
4.1 The key issue(s) in the case will be established and
agreed by:
Considering all available, relevant information and, where
necessary, requesting additional information.
Discussing with the relevant client e.g. police, CPS/PPS/
COPFS and defence team.
4.2 On the basis of the case circumstances and the agreed key
issue(s) identify:
The relevant prosecution proposition.
1
The relevant defence proposition.
2
4.3 Establish if the scientic examination of items submitted
can assist by carrying out a pre-assessment. This will
involve:
Identifying the potential outcomes of examinations/tests.
These potential outcomes may be grouped into broader
categories where appropriate.
Science and Justice 49 (2009) 161164
1355-0306/$ see front matter 2009 Published by Elsevier Ireland Ltd. on behalf of Forensic Science Society.
doi:10.1016/j.scijus.2009.07.004
Contents lists available at ScienceDirect
Science and Justice
j our nal homepage: www. el sevi er. com/ l ocat e/ sci j us
Assigning estimated probabilities to each outcome given
the propositions set and relevant information.
3,4
Estimating likelihood ratios for each outcome and using
them to inform examination strategy and tests carried
out.
4
Tests and examinations commissioned will be those that
best help to progress the issue in the case whether in
favour of the prosecution or defence taking into account
the views of other experts where appropriate.
4.4 If, as a result of the probability and likelihood ratio as-
sessments, scientic examinations are unlikely to assist
either prosecution or defence the client will be advised
accordingly unless client protocols/agreements dictate
otherwise. Such advice (whether accepted or not) will be
documented on the case le.
4.5 If a client dictates an examination strategy that, in the
opinion of the Expert, is inappropriate then the client
will be advised accordingly and the advice and conver-
sations will be recorded on the case le. Any resulting
limitations on the interpretation[s] will be described in
the statement.
4.6 If access to relevant items, as identied through the
assessment, is denied then the client will be advised as to
the limits of any resulting interpretation. Reference to
this will also be made in the statement.
4.7 Examination of samples and/or items will be carried out
on the assumption that such samples or items have been
collected, packaged, preserved and transported in accor-
dance with accepted protocols or best practice unless
there is good reason to believe otherwise e.g. from the
submission form, sample container or packaging. In such
cases further enquiries will be made to conrm or
otherwise such suggestions and discussions will take
place with the client to agree a way forward. This may
result in the items not being examined or, if they are, the
results will not be the subject of an evaluative opinion.
4.8 Examinations, tests and analyses carried out should
wherever possible be validated and accredited.
4.9 All examinations, tests and analyses will be carried out by
trained and competent personnel.
4.10 If the case requires non-accredited, infrequently used or
improvised tests to be carried out then these will be
subject to QC checks and peer review.
4.11 Propositions set and probabilities assigned will not be
changed at any stage unless:
The key issue in the case changes and/or.
The conditioning information changes.
In some circumstances probabilities may be reassessed in
the light of test results e.g. a rare glass or bre type. Such
reassessments should be robust and justiable.
4.12 The changing of propositions fromactivity to source level
after the examinations and/or test results have been
obtained will not be done (refer ref. [1]).
4.13 The following will be documented clearly on the case le:
The key issue(s) and propositions addressed.
Potential outcomes and assigned probabilities.
Sources of data used in probability assignments.
Examination strategy.
All discussions with clients and other experts including
those working for the defence.
4.14 Statements or reports will include:
Background information used in the assessment/
interpretation.
The propositions addressed.
Relevant items received.
Items examined.
Signicant ndings.
Conclusion(s).
4.15 The conclusion(s) in the report will be related to the pro-
positions addressed and the estimated likelihood ratio.
4.16 The strength of evidence will be expressed either by a
value of the likelihood ratio or using a verbal scale related
to the value of the likelihood ratio. This verbal scale will
be adopted by all AFSP organisations.
An example of a scale currently used by many practitioners is
given in Section 5.
5
The full verbal scale used will be provided in the statement for
reference.
[Note. In relation to DNA evidence the Doheny and Adams ruling
should be considered when using a verbal scale.]
5. Guidance notes
5.1. Guidance note 1
In cases where the Prosecution are unable to clearly identify what
happened in the case the expert, if his expertise can be of help, should
be operating in the investigative mode to assist the police or Crown
investigators and to work with them to identify the key issue(s).
If possible, the expert should establish the defence scenario from
the investigator, prosecution team, defence team, or through the case
managing judge.
Cases where the suspect makes No comment.
The expert is in a dilemma in these situations. There may well be
a reasonable prosecution proposition but in order to operate in
evaluative mode and offer a balanced opinion on the weight of his
ndings, the expert requires an alternative proposition to consider.
There are three options available to the expert in such circumstances:
1. Adopt, on behalf of the defendant, an alternative proposition.
2. Explore a range of alternative explanations for the ndings.
3. Discuss the likelihood of the observations given the prosecution
proposition and stress that no one can evaluate logically the weight
of these observations without an alternative.
There are strengths and weaknesses to all three approaches and it
may be that all have their uses at different stages in the Criminal
Justice process. What is clear is the need for all relevant participants
in the Criminal Justice System to be engaged in deciding the most
appropriate approach in any particular case/case type.
In cases where conditioning Information may be insufcient to
estimate probabilities, the expert may operate in investigative mode
to identify potential scenarios. If then an evaluative statement is
required it should only be provided with a clear statement of the
assumptions made (conditioning information) in identifying proposi-
tions and assessing probabilities.
162 Association of Forensic Science Providers / Science and Justice 49 (2009) 161164
Wherever possible probability estimates should be made. Published
data will be used wherever possible as a basis for these assessments. If
published data is not available then data fromunpublished sources may
be used as long as it has been peer reviewed and documented on le.
Personal data such as experience in similar cases and peer consultations
may be used provided that the practitioner can justify the use of such
data and demonstrate that it is soundly based.
In cases where the specic material or evidence type is not known
to the expert then estimates of the probabilities should be made from
either technical knowledge and/or simulated tests.
Probability assignments may be qualitative. In such cases the like-
lihood ratio may also be expressed in qualitative terms provided this
can be justied.
It is recommended that, wherever possible, a probability and LR
table should be used. The use of probability trees in some cases will be
more appropriate.
The opinion or conclusion should express the degree of support for
one proposition or the other depending upon the value of the like-
lihood ratio (L.R.).
For values of L.R. close to one then the opinion /conclusion should
be to the effect that the ndings do not support one proposition over
the other or the ndings support each proposition equally.
For values of L.R. greater than one then the opinion/conclusion
should be that the ndings provide a degree of support for the pro-
secution's proposition.
For values of L.R. less than one then the opinion/conclusion should
be that the ndings provide a degree of support for the defence
proposition.
The degree of support will relate to the magnitude of the like-
lihood ratio. It is suggested that the scale adopted by a large number of
forensic practitioners is used as set out below.
Value of likelihood ratio Verbal equivalent
N110 Weak support for proposition
10100 Moderate support
1001000 Moderately strong support
100010,000 Strong support
10,0001,000,000 Very strong
N1,000,000 Extremely strong
6. Glossary of terms
6.1. Case assessment
The application of expert judgment to devise an examination stra-
tegy based upon a framework of circumstances in the form of written
submission details, photographs, preview examinations of items, dis-
cussions with submitting ofcers etc.
6.2. Examination strategy
Acourse of action based upon expert assessment, and ideally agreed
with the customer, that addresses the case issue(s) to the maximum
benet to the CJS in terms of time, cost and effectiveness.
In summary an assessment is the thinking bit and the strategy is
the plan for the doing bit. Assistant instructions activate the doing.
6.3. Framework of circumstances
A summary of all of the information known to the scientist about
the crime and suspect(s) that is relevant to the assessment and in-
terpretation of the scientic observations. This framework must always
be regarded as provisional because it depends on the evidence of others
(police, CSM/CSI pathologists, eye witnesses, victim, suspect etc).
6.4. Explanation
This is considered to be a hypothesis (theory) that explains sci-
entic observations. Unlike propositions, explanations can be gener-
ated without mutually exclusive alternatives. The generation
of explanations may be an intermediate stage between collecting
observations and formulating propositions.
6.5. Probability of an explanation
At the investigative stage it is accepted practice for the scientist to
express an opinion in the form of a probability that a particular
explanation is true (i.e. to provide a posterior probability).
6.6. Proposition
Is a formal hypothesis that is generated, in part, from the back-
ground information but may also depend upon the observations that
have been made.
In the context of a criminal trial there will most often be a pair of
propositions one representing the prosecution position, the other
representing the defence's.
Propositions are mutually exclusive (i.e. if one is true then the
other must be false) and exhaustive (i.e. they cover all possibilities
within the framework of circumstances).
6.7. Evidential weight
This is the expression of the extent to which the observations
support one of the two competing propositions. The extent of the
support is expressed to the client in terms of a numerical value of the
likelihood ratio (where sufciently robust data is available) or a verbal
scale related to the magnitude of the likelihood ratio when it is not.
6.8. Likelihood ratio [LR]
This is the ratio of two probabilities; the probability of the evidence
given that the prosecution proposition is true divided by the prob-
ability of the evidence given that the alternative proposition is true.
These probabilities are assigned on the basis of the scientist's
expectation of the outcomes of the examinations given that each of
the propositions is true.
6.9. Probability of a proposition
At the evaluative stage, it is logically incorrect to express an
opinion in the formof a probability that a particular proposition is true
(i.e. to provide a posterior probability). Indeed, within the context
of DNA proling it would normally be interpreted as a manifestation
of the prosecutor's fallacy and could form grounds for appeal. The
scientist should consider the probability of the evidence given each of
the propositions.
7. Examples
Some examples of technical/factual investigative and evaluative
interpretation.
7.1. Technical/factual interpretation
It's amphetamine.
Petrol was detected in the sample of re debris.
The blood sample contains 95 mg % alcohol.
The database says the prole matches Burt.
The gun is subject to control by rearms legislation.
The stain is blood.
The paint has a blue top coat.
7.2. Investigative interpretation
The re was probably started deliberately.
Inmy opinion the patternand distributionof the blood spots suggest
that the assault began in the living room and continued into the
kitchen.
The blue paint came from a 1963 Hillman Imp.
The tyre deated as a result if the impact.
7.3. Evaluative interpretation
My ndings are far more likely if the amphetamine came from the
suspect's stock than if they had come from some other stock.
Myglass ndings provide support for the viewthat the suspect broke
the windowrather than he poked his head through the broken pane
afterwards.
My observations are equally as likely if Ms. X's or Mr. Y's versions of
events were true.
The pattern and distribution of the blood spots are far more likely if
Mr. X had assaulted Mr. Y than if he had been standing close to the
assault when someone else did.
Reference
[1] R. Cook, I.W. Evett, G. Jackson, P.J. Jones, J.A. Lambert, A hierarchy of propositions:
deciding which level to address in casework, Science and Justice 38 (4) (1998)
231239.
Association of Forensic Science Providers
E-mail address: n.nicdaeid@strath.ac.uk.
646 Practice
Criminalistics is
reasoning backwards
Logically correct reasoning in forensic reports
... and in the courtroom
Charles Berger
1

SCIENCE PLAYS AN INCREASING ROLE IN CRIMINAL LAW, AND BOTH ARE RIGHTLY HELD UP TO HIGHER
STANDARDS. THE AWARENESS THAT SCIENTISTS AND LAWYERS WILL NEED TO FIND EACH OTHER
MORE OFTEN TO REACH A HIGHER LEVEL IS ALSO INCREASING. LOGICALLY CORRECT REASONING AND
CONCLUDING ARE INDISPENSABLE AT THAT HIGHER LEVEL, ESPECIALLY WHEN UNCERTAINTY IS
INVOLVED. THIS PAPER DESCRIBES HOW LOGICALLY CORRECT CONCLUSIONS ARE GIVEN IN FORENSIC
REPORTS, AND HOW THE READER CAN DEAL WITH THIS.

We are faced with the big challenge to reason in the most
rational way in criminal cases. Science teaches us that
probability theory is essential to that effort, while for
most lawyers its not their favorite subject. On top of that,
communication between scientists and lawyers is not
always as good as it should be. Even so, awareness that
logically correct reasoning in the presence of uncertainty
is important is growing internationally, among lawyers
and forensic scientists. In the new Handbook experts for
the judge in criminal law
2
for example, the importance
of logic, probability theory and methodology has been
recognized very clearly. It also correctly concludes that a
judge can generally limit himself to these central
elements, and does not need to have intimate knowledge
of all the details of all separate forensic disciplines.
A logically incorrect conclusion
that is understood is no
alternative to a logically correct
conclusion which needs
explanation
The Netherlands Forensic Institute (NFI) is engaged in a
transition to a new logically correct conclusion scale for its
forensic reports. An investigation into how well readers
understand logically correct conclusions without further
explanation
3
shows that they are struggling. So should the
NFI make that transition? The answer to this question is a
resounding yes.
There is a false choice: a logically incorrect
conclusion thats understood is no alternative to a
logically correct conclusion which needs explanation. Of
course the scientist should do his best for his conclusions
not only to be correct, but also written and explained as
clearly as possible. Explanation is also offered in courses
and lectures for the various actors in the criminal justice
system.
For the treatment of logically correct conclusions it
is good to first have a closer look at forensic science.
Forensic science is reasoning backwards from
effect to cause
Forensic science is often simply defined as the application
of science to questions of interest to the court.
Criminalistics
4,5
is the part of forensic science where the
exact sciences are applied. This definition is certainly
practical, but says very little about forensic science from a
scientific standpoint.
784 NEDERLANDS JURISTENBLAD 02 04 2010 AFL. 13

In the last century, a number of so-called principles of
forensic science were proposed. One example is the so-
called Locards principle. This is usually stated as Every
contact leaves a trace, a phrase that was never found in
Locards oeuvre. Apart from the fact that this is not a
principle Locard proposed,
6
it can also hardly act as such.
One could say with the same validity: every contact wipes
a trace.
In the absence of a number of classical principles,
one might wonder what is the binding factor between the
various disciplines in forensic science, apart from their
application to questions of interest to the court. In my
view, forensic science is the science of reasoning
backwards from a known result to a cause, using the
aforementioned three pillars: logic, probability theory, and
methodology. This is also makes the common interest of
forensic scientists and lawyers clear: how do we reason
back from effect to cause, when we are dealing with
uncertainty. Consequences exist for example in the form
of traces at a crime scene or on an object, and possible
causes are presented in the form of hypotheses about how
these traces were created (for example about who left the
trace). The process of reasoning backwards, i.e. the use of
traces (in the most general sense) to find support for these
hypotheses, is a common and binding theme for all
forensic disciplines.
Reasoning is based on logic. If logic is to be applied
to uncertain events, probability theory is required for the
most rational outcome. It is even used as the definition of
probability theory: the extension of logic to uncertain
events.
7
Reasoning in the presence of uncertainty is
something we do intuitively in everyday life, but that does
not mean that our intuition is always right, or that we are
capable of reasoning sufficiently explicit to convince
someone else. To reason in a rational and logically correct
way, we need to know some principles from probability
theory, but that does not mean that a case can be reduced
to a calculation. More importantly, logic tells us which
conclusions are justified by the information we have, and
which are not.
Three forms of reasoning
There are three main forms of reasoning that play an
important role in criminalistics. Abduction
8
is the type of
reasoning by which we creatively generate hypotheses,
which, if true, best explain the initial observations.
Thus, when encountering a person killed in a house, and
observing muddy footprints from the window to the victim,
we can come to the hypothesis of an intruder that entered
through the window. Abduction creates a starting point for
two other forms of reasoning: deduction and induction.
Deduction is a type of reasoning that allows you to
reach hard conclusions (categorical conclusions), which are
necessarily correct. This is the kind of reasoning we use
when we can exclude a hypothesis, such as when our
observations are absolutely impossible if the hypothesis
were correct. One example is a suspect with a watertight
alibi: he could not have been at the crime scene during the
crime. In most cases however, we can not exclude a number
of hypotheses, and for those hypotheses the use of
deduction is unfortunately not an option.
When we make observations that form more support
for one hypothesis than for the other, but exclude neither of
these hypotheses, we use induction for our reasoning. With
this type of reasoning, our observations support a
hypothesis, but they will never prove it with certainty.
Adding more observations can increase our belief further,
but certainty is never achieved. For example, by
continuously observing white swans the belief that all
swans are white will increase, but we will not achieve
certainty until all swans have been observed (even those in
Australia).
The evidential value of an observation
In order to avoid tunnel vision and to make explicit which
hypotheses were or were not considered, at least a second,
alternative hypothesis is required. The two competing
hypotheses that are considered, are to exclude each other:
they can not simultaneously be true.
Our knowledge (or degree of belief) concerning the
hypotheses can be expressed in odds: the probability that
hypothesis H1 is true (all swans are white), divided by the
probability that the alternative hypothesis H2 is true (not
all swans are white).
A relevant new observation will change the odds of
the hypotheses. According to probability theory, the extent
to which an observation changes the odds defines the
evidential value of that observation. The odds of the
hypotheses prior to the observation are called prior odds, the
odds after the observation are called posterior odds. Bayes
theorem
5
shows that the evidential value is a multiplication
factor, and defines it as the likelihood ratio (LR):

Author
1. Dr. ir. C.E.H. Berger is principal scientist at
the Netherlands Forensic Institute, and
member of the Board of the Dutch Forensic
Experts Registry (NRGD).

Notes
2. Handboek deskundigen voor de
strafrechter, Project Deskundigheids-
bevordering Programma Strafsector
2010, 2009.
3. J.W. de Keijser, H. Elffers, R.M. Kok,
M.J. Sjerps, Bijkans begrepen?, Den
Haag: Boom Juridische uitgevers 2009.
This study was commissioned by the NFI,
and carried out by the NSCR.
4. A.P.A. Broeders, Op zoek naar de bron:
Over de grondslagen van de criminalistiek
en de waardering van het forensisch
bewijs, Deventer: Kluwer (juridisch) 2003.
5. B. Robertson, G.A. Vignaux,
Interpreting Evidence: Evaluating Forensic
Science in the Courtroom, New York: Wiley
1995.
6. E. Locard, Manuel de technique policire,
Paris: Payot 1923.
7. E.T. Jaynes, Probability Theory: The Logic of
Science, Cambridge: Cambridge University Press
2003.
8. C.S. Peirce, How to make our ideas clear,
Popular Science Monthly 1878, 12, 286-3.

NEDERLANDS JURISTENBLAD 02 04 2010 AFL. 13 785
Practice

Prior odds likelihood ratio = posterior odds.

The three terms in this equation are defined as:

The evidential value of an observation is equal to the
likelihood ratio: the probability of that observation if
hypothesis H1 is true, divided by the probability of the
same observation when the alternative hypothesis H2 is
true. The theorem shows that the evidential value (LR)
only gives the relative increase in the odds and not the
posterior odds themselves.
The three examples in the boxes can clarify this.
In the box pot belly you will find an anecdotal example
of the likelihood ratio as the evidential value of an
observation.

Pot belly (example 1)
In this example we see an acquaintance that we rarely
meet across the street. The size of her belly suggests that
she is pregnant, but we have not heard about her partner
or pregnancy. Before we decide to congratulate her, the
question arises: is she really pregnant or not? What is the
evidential value of our observation of her belly?
Probability theory tells us that the evidential value is
equal to the ratio of two probabilities: the probability of a
big belly during pregnancy, divided by the probability of a
big belly without a pregnancy. The evidential value
increases as more pregnant women have a big belly, and
less non-pregnant women show that same feature. Thus,
the evidential value of the observation of a big belly is
much greater in a population with only thin women than
in a population where a pot belly is the norm.

A numerical example from the medical world is given in
the box HIV test. The medical example shows that the
diagnostic test for a disease by itself is not enough for a
estimate on the probability that we have the disease.

HIV-test (example 2)
If we test someone for HIV, we consider the hypotheses
has HIV and has no HIV. The so-called enzyme
immunoassay HIV test is known to give a positive result
for 99.7% of all people with HIV. People without HIV have
a probability of only 1.5% to obtain such results.
9

But what is the probability that a person who
tested positive (bad news) actually has HIV? Intuitively,
we might think that that probability is very large, and
especially the 99.7% number appears to play a role. In
reality, we need more information to conclude on the
probability of HIV infection.
But we can report on the evidential value of the
test result: the likelihood ratio is 99.7% / 1.5% = 66. In the
medical world, this is known as the diagnostic value. It is
66 times more likely to test positive when you are HIV
infected, than when you are not.

If we know the prior odds, we can now determine the
posterior odds. The prior odds depend on other evidence
and information. If we know for example that the tested
person is an adult South African, then the prior odds are
known from the prevalence (the relative proportion of
the population that is infected) of HIV among adults in
this country: 0.22.
10
The posterior odds are therefore
0.22 66 = 14.6. For this person, the result means that the
odds are 14.6 against 1 that he has HIV. For an adult
Dutch person, the situation is quite different. In the
Netherlands, the prior odds of HIV infection are around
0.002, and the posterior odds are 0.002 66 = 0.13. So for
the Dutchman the same positive test means that the odds
of infection are 1 to 7.6, much better!
For the probability that the tested person has HIV,
the knowledge prior to the test (prevalence) is as
important as the evidential value of the test.

The same principle applies in forensic science: the
comparison (observation) of trace and reference material
of a suspect is not sufficient to determine the
probability that the traces were left behind by the
suspect (hypothesis), except when you can exclude the
suspect. To determine that probability, prior odds are
required which include all other information (motive,
alibi, etc.) and evidence (e.g. other types of traces).
In the red scarf box, a forensic example is given
from which this is clear (even without using numbers):
the examiner cannot report the probability that the fibers
originated from the red scarf, but he can report the
evidential value of his observations.

Red scarf (example 3)
In this example, red fibers were found on a murder
victim. The police send these fibers and a red scarf to the
NFI, asking whether the fibers originated from the scarf.
If that cannot be determined with certainty, they want
to know how likely it is that the fibers came from the
scarf. This question can not be answered based on the
similarities between the fibers of the victim and the
scarf alone (at least not unless the scarf can easily be
excluded as a source, such as when large differences are
observed). Why not?
The reason is that the NFI examiner does not have
all other information, and so does not know the prior
odds, while those are essential for the probability that the
fibers came from the scarf. This is perhaps easier to see
using the example. We consider two different scenarios
preceding the submission of the fibers and the scarf. In
the first scenario, a suspect was arrested while running
away from the victim, and he had the red scarf around

n observatio after true is H2 that y probabilit
n observatio after true is H1 that y probabilit
true is H2 when n observatio of y probabilit
true is H1 when n observatio of y probabilit
n observatio before true is H2 that y probabilit
n observatio before true is H1 that y probabilit
=

his neck. In the second scenario, the police found the
victim (a student) the next day, and also observed the red
fibers. In the cloakroom of a large school in the
neighborhood a red scarf was found and sent to the NFI.
It is clear that these scenarios will lead to a quite
different probability of the fibers originating from the
scarf. But the NFI examiner has no information on such
scenarios. The examiner cannot report the probability
that the fibers came from the red scarf and has to limit
himself to the evidential value of his observation.

Forensic conclusions
There are different types of conclusions in forensic reports.
For example, dactyloscopists (fingerprint experts)
traditionally report categorical conclusions. This type of
conclusion is scientifically problematic because we have
seen that inductive reasoning cannot lead to a categorical
conclusion (certainty). A positive conclusion explicitly
excludes the possibility that a person exists whose
fingerprint is as close to the trace as the fingerprint of the
current suspect.
A much more common type of conclusion is one
for which a probability is assigned to a hypothesis. For
example, an assessment of the probability that a trace of
the perpetrator and reference material from the suspect
originate from the same source. Because such a
conclusion is not a categorical statement this might seem
quite reasonable, but we have seen that such a
conclusion is not justified. The expert does not have all
information in the case, which is necessary for an
assessment of the prior odds. The judge is the person
who does have all this information available to him, and
he can use it in weighing the evidence.
Even if the examiner would have had all that
information, it is not good to use it for the conclusion. It
would mean he would tread outside his own field of
expertise, and possibly even express opinion on the
ultimate issue. Moreover, experts in various forensic
fields could do so, and by combining the reports in the
courtroom things would be counted multiple times (and
given too much weight). Therefore, the examiner should
limit himself in his conclusion to the evidential value of
his observations.
The way that conclusions are reported in DNA
examination is in line with what logic and probability
theory teach us. The evidential value of a found match is
given by the probability of the match when the suspect
has left the trace, divided by the probability of that
match when a random other person left the trace. The
first probability is one (barring errors), while the second
probability is equal to the frequency with which the
profile appears in the population, and that can be
calculated. The conclusions in DNA examination are thus
logically correct and numerical. For many other areas of
forensic expertise the evidential value can not be
reported numerically, because of a lack of data. Still,
logically correct conclusions can also be reported with a
In his conclusion the forensic
scientist must limit himself to
the evidential value of his
observations.

verbal scale, when numerical data are lacking.

Introducing the new verbal conclusion scale
Currently, the NFI is in a transition to a new verbal
conclusion scale. The logically correct wording chosen to
report the evidential value is:
11

The findings of the examination are ...
about equally likely;
somewhat more likely;
more likely;
much more likely;
very much more likely
... when hypothesis 1 is correct, as / than when
hypothesis 2 is correct.
Of course, this conclusion scale does not apply where only
measurement results are reported, where a categorical
response is justified (as in exclusion), or when the
evidential value can be reported numerically. Where
possible, it is desirable to report numerically. Therefore, a
research and development program has been established
at the NFI, to obtain more empirical data and objective
comparison methods. This is a large but exciting challenge
for criminalistics in the years ahead.
Other verbal conclusion scales are also possible,
but might give rise to more interpretation errors. The
evidential value can e.g. also be expressed in how much
more support the observations give to one hypothesis
relative to the alternative hypothesis. In practice, this is
easily misread as: how much more likely one hypothesis
is than the other. This kind of thinking errors (also called
fallacies) is very common, so we will now look at the
main variants of these fallacies.

Misunderstanding conclusions (fallacies)
The reader of a conclusion of a forensic report is looking
for the answer to his question, and when no such answer is
given, there is a tendency to read the given answer
incorrectly as if it were the desired answer. The logical
errors thus made are named after the party that
supposedly tends to make that error.

Notes
9. R. Chou, L.H. Huffman, R. Fu, A.K.
Smits, P.T. Korthuis, Screening for HIV:
a review of the evidence for the U.S.
Preventive Services Task Force, Ann.
Intern. Med. 2005, 143, 55-73.
10. www.cia.gov, the world factbook,
data from 2007.
11. De reeks waarschijnlijkheidstermen
van het NFI en het Bayesiaanse model
voor interpretatie van bewijs, vakbijlage
NFI 2008.

Practice

ANP Photo Xtra

The prosecutors fallacy is an error of reasoning in which
conditions are transposed (it is also called the transposed
conditional). Thus the conclusion the probability of a
match is 1000 times greater when the accused left the
trace than when a random other person left it can be
incorrectly read as the probability that the accused left
the trace is 1000 times greater than that a random other
person left it, when there is a match. This is a fallacious
transposition of conditions that is more noticeable when
using simpler statements like an animal has four legs
when it is a cow and an animal is a cow when it has four
legs. The desire to know the probability of the hypothesis
leads one to read a logically correct conclusion (in terms
of evidential value) as if it would directly give the
probability of the hypothesis. But as we have seen before,
this also requires the prior odds. By ignoring the prior
odds these are implicitly assumed equal to one (50% /
50%), while in reality they can be much higher or lower.
In the defense fallacy the prior odds are implicitly
assumed to be very small. The above conclusion can for
example be interpreted incorrectly by stating: there are
16,000 people in the Netherlands that would match as
well, which means the probability that the current
suspect left this trace is only 1 in 16,000. Here it is
implicitly (and almost always wrongly) assumed that a
priori all 16 million Dutch people have the same
probability of having left the trace.
This type of reasoning error exists in many
different forms, some more subtle than others. Thus,
even with strong evidence it can suggestively be stated
that the accused at most is not excluded, while when
considering the evidential value it is also crucial how
many others have been excluded.
Concluding on conclusions
Logically correct reasoning and concluding is not an effort
that cant fail when it is based on experience and intuition
only. It is not easy, and logic and probability theory play
an important role. Perhaps from an evolutionary point
of view it is more important to predict the future than
the past, and we are therefore not very good at reasoning
backwards.
Forensic science is in a transition to conclusions
that are logically correct, but not always easy. Sometimes
you cannot make things easier without straying from the
truth. It is clear that a logically incorrect conclusion that is
well understood is no alternative to a logically correct
conclusion that requires some explanation. I hope to have
contributed with this article to the understanding of this
vital matter, but for an improved understanding practice
and repetition are important.
A good conclusion considers at least two competing
hypotheses in order to avoid tunnel vision and make
explicit which hypotheses are considered. If neither of
these hypotheses can be completely excluded, the
conclusion says how much more likely the results of the
examination are under one than under the other
hypothesis. This factor is the evidential value, and it tells
us how much the odds of the hypotheses have increased
because of the results. The final probability that the
hypothesis is true can not be given by the expert, because
that probability is also determined by information that he
does not have, and which often falls outside his expertise.
The combining of all the forensic evidence with the other
information in the case is up to the judge, who has all
information available to him.

Perhaps from an evolutionary point of view it is more important
to predict the future than the past, and we are therefore not very
good at reasoning backwards

So if you read such a logically correct conclusion, keep
in mind that (in simple terms) the expert can usually
not tell if it is so, nor how likely it is, but only what the
evidential value of his observations is. Beware of the
errors of reasoning. If lawyers and scientists
communicate more and common errors of reasoning
can be avoided more often, that can be a very important
advance.

The NFI
The NFI provides forensic services, mostly to requesters
in the criminal justice system, including the prosecution
and the police. The NFI in 2008 provided about 53,500
products, especially in DNA examination.
12
Partly in
response to the recommendations in the report of the
committee Posthumus,
13
the forensic reports have
greatly improved: they are more uniform, more
structured, and equipped with appendices that give
more information on the examination methods used.
The introduction of a uniform, logically correct
conclusion scale is accompanied by many forms of
explanation on this matter, including lectures and
workshops, publications and an e-learning course.
The NFI is internationally at the forefront of this
development. It is in contact with colleagues in
countries that already make use of logically correct
conclusion scales, such as the United Kingdom, Sweden
and Switzerland. The United States still has a long way
to go, partly due to the poor condition of much of the
U.S. forensic system.
14

Notes
12. Dutch Forensic Institute, Annual Report 2008.
13. F. Posthumus, Evaluation in Schiedam parkmoord 2005.
14. Strengthening Forensic Science in the United States: a path forward, National Academies
Press 2009.
52 IxperIIse cn RechI 2U1U-2

Dr. r. C..H. erger
*
nnd mr. D.).C. Aben
**

FvIdence and convIctIon:
RatIonaI reasonIng sInce ArIstotIe

)udgcs soncIIncs sn) Ihc) do IhcIr vorI Inscd on ArIsIoIIc, cxcrIcncc nnd InIuIIIon. CuI ]ccIIng, so Io scnI.
5cIcncc vouId Ihus hnVc IIIIIc Io do vIIh IhIs, nnd hnVc IIIIIc Io o]]cr Io Ihc ]udgc IhnI rcnchcs hIs VcrdIcIs I)
usIng hIs cxcrIcncc. 5cIcncc hovcVcr, hns cVoIVcd sIncc Ihc IIncs o] ArIsIoIIc (34-322 C1. ThIs Is no dI]]crcnI
]or ]orcnsIc scIcncc. Cnn Ihc ]udgc do vIIhouI?

ThIs arIIcIc Is Ihc !IrsI o! a scrIcs o! Ihrcc, In whIch Ihc auIhors dwcII on Ihc possIbIIIIIcs o! usIng a BaycsIan
rcasonIng schcmc In cvaIuaIIng cvIdcncc, and common !aIIacIcs. In IhIs parI Ihc hIsIorIcaI dcvcIopmcnI o! Ihc
rcasonIng schcmc

*OUSPEVDUJPO
1he crImInaI IrIaI Is based on Ihe premIse IhaI Ihe
IruIh noI onIy exIsIs, buI can acIuaIIy be !ound. In any
case II musI be soughI In Ihe IrIaI, and Ihen a !InaI
decIsIon shouId be Iaken. 1hIs means IhaI In Ihe
judIcIaI InvesIIgaIIon reIevanI daIa are coIIecIed, a!Ier
whIch Ihe seIecIIon and evaIuaIIon Is a duIy o! Ihe
courI. IInaIIy, a verdIcI !oIIows In whIch a decIsIon Is
Iaken on Ihe scenarIo IhaI Ihe prosecuIor presenIed Io
Ihe courI In Ihe !orm o! Ihe IndIcImenI. WhaIever Ihe
ouIcome o! Ihe IrIaI wIII be, aII parIIcIpanIs shouId
have had Ihe opporIunIIy Io dIscuss Ihe coIIecIed
In!ormaIIon be!ore a decIsIon Is Iaken.
1o IhaI exIenI, Ihere are sIrIkIng sImIIarIIIes beIween
IhIs judIcIaI process and Ihe pracIIce o! scIence.
ScIenIIsIs aIso coIIecI daIa, expIore IheorIes (scenarIos)
and do noI eschew debaIe.
BuI Ihere are aIso dI!!erences. ScIenIIsIs have IheIr
reservaIIons abouI Ihe somewhaI naIve word 'IruIh',
and Ihey !rown even more aI someIhIng IIke 'Ihe whoIe
IruIh.' ScIenII!Ic research Is based on Ihe coIIecIIon o!
empIrIcaI daIa, and !ormuIaIIng and IesIIng hypoIheses
on Ihe basIs o! such daIa. 1he scIenIIsI does noI have
Ihe Iask Io Iake a decIsIon. Iven I! he chooses !or one
parIIcuIar hypoIhesIs, IhaI choIce wIII be provIsIonaI
and new observaIIons may sIIII requIre hIm Io rejecI
IhaI hypoIhesIs.
|udges however, musI aI some poInI reach a decIsIon
In Ihe !orm o! a verdIcI IhaI Is expressed In absoIuIe
Ierms. 1he crIme Is proven or noI. 1he judge can noI
pronounce In hIs verdIcI IhaI Ihe accused Is IIIcI) or
unIIIcI) Io have commIIIed Ihe crIme. |udges are aIso
noI In a posIIIon Io reconsIder earIIer choIces. 1he
usuaI procedures o! an appeaI and !urIher appeaI are
Ihe onIy way Io accompIIsh IhaI. Moreover, Ihe IrIaI
musI be compIeIed wIIhIn a reasonabIe IIme. 1haI way,
II Is dI!!IcuII Io make use o! years o! advancIng
InsIghI.
T

Iawyers are noI used Io Ihe concepI o! a 'hypoIhesIs'
wIIhIn Ihe scope o! Ihe IruIh-!IndIng process. When
Pro!. Dr. R. MeesIer, pro!essor o! probabIIIIy and
sIaIIsIIcs assocIaIed wIIh Ihe VU unIversIIy o!
AmsIerdam, had dropped IhaI Ierm more Ihan once
durIng hIs quesIIonIng aI a hearIng as an experI, he
was correcIed by Ihe judge: "Mr. MeesIer, we do noI
engage In hypoIhesIzIng, buI In IruIh !IndIng!".
A key poInI Is IhaI In !acI judges do make use o!
hypoIheses and probabIIIIIes In IheIr quesI !or Ihe
IruIh. And II Is a good IhIng IhaI Ihey do so. 1he
hypoIheses IncIude Ihe scenarIos regardIng Ihe o!!ense
IhaI Is Ihe subjecI o! InvesIIgaIIon, and are dIscussed
by Ihe prosecuIIon and Ihe de!ense. Or - o!
consIderabIe ImporIance - Ihe scenarIos IhaI Ihe courI
may consIder IhemseIves. ReasonIng wIIh aII
hypoIheses and condIIIons esIabIIshed, Ihe courI
uIIImaIeIy reaches IIs decIsIon; aI IeasI IhaI Is Ihe
modeI IhaI we envIsage and o! whIch we suspecI IhaI
Ihe judge uses II, wheIher InIenIIonaIIy or noI.
1he judge cannoI avoId reasonIng wIIh In!ormaIIon
whIch Is InherenIIy uncerIaIn. UnIII he decIdes wheIher
Ihe IoIaI o! Ihe daIa Is su!!IcIenI !or a convIcIIon, he
wIII noI escape probabIIIsIIc reasonIng In IhaI sense.
We beIIeve IhaI InsIghIs !rom scIence can be exIremeIy
heIp!uI Io Ihe courI. Irom severaI perspecIIves, we wIII
dIscuss how a BayesIan reasonIng scheme can heIp
IruIh-!IndIng In Ihe judIcIaI process. II Is cIoseIy
reIaIed Io how !orensIc scIenIIsIs shouId do IheIr work,
and Io how Ihey phrase IheIr concIusIons. I! onIy !or
IhaI reason, Ihe crImInaI courIs shouId have some
knowIedge o! IhIs scheme. BuI Ihe reasonIng scheme Is
jusI as ImporIanI Io Ihe reasonIng o! Ihe oIher
parIIcIpanIs In Ihe IegaI process, and - conIrary Io whaI
Is someIImes IhoughI - IIs use does noI depend on Ihe
avaIIabIIIIy o! scIenII!Ic daIa and hard numbers.

*
Dr. Ir. C.F.H. erger Is prIncIpaI scIentIst at the NetherIands ForensIc InstItute, and member oI the oard oI the NetherIands RegIster Ior Court Fxperts (NRCD).
**
Mr. D.}.C. Aben Is Advocate CeneraI at the Supreme Court oI the NetherIands, and edItor oI thIs journaI.

1. Fven the extraordInary remedy oI a revIsIon oIIers IIttIe or no soIace, because changed scIentIIIc InsIghts are dIIIIcuIt to transIate Into a noveIty.

IxperIIse cn RechI 2U1U-2 53
&WJEFODFBOEDPOWJDUJPO3BUJPOBMSFBTPOJOHTJODF"SJTUPUMF

)JTUPSZ
ArIsIoIIe wroIe hIs RheIorIc
2
, a masIerpIece on Ihe arI
o! persuasIon, beIween 360 and 330 BC. He made a
dIsIIncIIon beIween poIIIIcaI, ceremonIaI and !orensIc
rheIorIc. IorensIc rheIorIc was aImed aI convIncIng a
judge IhaI had Io render a judgmenI on pasI evenIs.
ArIsIoIIe reaIIzed IhaI Ihe proposIIIonaI and predIcaIe
IogIc he had worked ouI, whIch !ocused on argumenIs
wIIh hard !acIs, wouId rareIy be use!uI !or IhaI:

'1here are !ew !acIs o! Ihe 'necessary' Iype IhaI can
!orm Ihe basIs o! rheIorIcaI syIIogIsms. MosI o! Ihe
IhIngs abouI whIch we make decIsIons, and InIo whIch
Ihere!ore we InquIre, presenI us wIIh aIIernaIIve
possIbIIIIIes. Ior II Is abouI our acIIons IhaI we
deIIberaIe and InquIre, and aII our acIIons have a
conIIngenI characIer; hardIy any o! Ihem are
deIermIned by necessIIy. AgaIn, concIusIons IhaI sIaIe
whaI Is mereIy usuaI or possIbIe musI be drawn !rom
premIses IhaI do Ihe same, jusI as 'necessary'
concIusIons musI be drawn !rom 'necessary' premIses'.

So ArIsIoIIe reaIIzed - even Ihen - IhaI In addIIIon Io
Ihe IogIc IhaI wIIh proper premIses yIeIds necessarIIy
correcI concIusIons, argumenIs exIsIs wIIh premIses
IhaI are IIkeIy buI noI necessarIIy correcI. DespIIe Ihe
Iack o! necessary correcIness, Ihese argumenIs shouId
In!Iuence Ihe convIcIIon o! a raIIonaI person.

'1he argumenI may, !or InsIance, be IhaI DIonysIus, In
askIng as he does !or a bodyguard, Is schemIng Io make
hImseI! a despoI. "Ior In Ihe pasI PeIsIsIraIus kepI
askIng !or a bodyguard In order Io carry ouI such a
scheme, and dId make hImseI! a despoI as soon as he
goI II; and so dId 1heagenes aI Megara"; and In Ihe
same way aII oIher InsIances known Io Ihe speaker are
made InIo exampIes, In order Io show whaI Is noI yeI
known, IhaI DIonysIus has Ihe same purpose In makIng
Ihe same requesI: aII Ihese beIng InsIances o! Ihe one
generaI prIncIpIe, IhaI a man who asks !or a bodyguard
Is schemIng Io make hImseI! a despoI.
We have now descrIbed Ihe sources o! Ihose means o!
persuasIon whIch are popuIarIy supposed Io be
demonsIraIIve.'

Here, exampIes !rom Ihe pasI - known Io ArIsIoIIe - are
used Io make an asserIIon abouI an unknown, uncerIaIn
!uIure. 1haI asserIIon Is noI necessarIIy correcI buI Ihe
exampIes can In!Iuence Ihe convIcIIon o! a raIIonaI
person. In IIseI! IhIs Is noI sIrange observaIIon. In
everyday II!e, peopIe gaIher In!ormaIIon derIved !rom
observaIIons o! exampIes aII Ihe IIme. AI Ihe IIme
however, IhIs was a groundbreakIng way o! IhInkIng,
whIch was !orgoIIen !or a Iong IIme a!Ier ArIsIoIIe.
In order Io expand hIs IogIc !rom cerIaIn Io uncerIaIn
evenIs, ArIsIoIIe needed probabIIIIy Iheory. BuI Ihe
deveIopmenI o! Ihe concepI o! probabIIIIy wouId Iake

anoIher Iwo mIIIennIa. UnIII Ihen, IogIcaI reasonIng had
Io be IImIIed Io eIIher bIack and whIIe, dIchoIomous
reasonIng, or non-scIenII!Ic reasonIng.
Ior deducIIve reasonIng as wIIh ArIsIoIIe's "syIIogIsm"
Ihere was no probIem: AII men are morIaI and SocraIes Is
a man, Ihere!ore SocraIes Is morIaI. In such an argumenI
Ihe concIusIon !oIIows necessarIIy !rom Ihe prevIous
premIses. 1he IogIc known aI Ihe IIme was su!!IcIenI Io
provIde !or Ihe !ormaIIzaIIon o! IhIs argumenI. BuI Io
come Io new knowIedge more was needed, a dI!!erenI
Iype o! reasonIng: InducIIve reasonIng.
|ohn SIuarI MIII wroIe abouI II In Ihe T9Ih cenIury
(T859) In A s)sIcn o] IogIc, rnIIocInnIIVc nnd InducIIVc
3
:

'In every InducIIon we proceed !rom IruIhs whIch we
knew, Io IruIhs whIch we dId noI know: !rom !acIs
cerII!Ied by observaIIon, Io !acIs whIch we have noI
observed, and even Io !acIs noI capabIe o! beIng now
observed; !uIure !acIs, !or exampIe: buI whIch we do
noI hesIIaIe Io beIIeve upon Ihe soIe evIdence o! Ihe
InducIIon IIseI!.'

InducIIon Is exacIIy Ihe kInd o! reasonIng wIIh whIch
ArIsIoIIe's exampIe o! Ihe guard can be undersIood. II
does noI maIIer wheIher we are "predIcIIng" Ihe
unknown pasI or Ihe !uIure: II Is Ihe evIdence o! our
observaIIons regardIng compeIIng hypoIheses abouI
whaI couId noI be observed dIrecIIy IhaI maIIers.
MIII has descrIbed how Ihe mosI IIkeIy cause o! an
observed e!!ecI (resuII) can be !ound when Ihe
hypoIheses abouI Ihe cause are equaIIy probabIe
be!ore observIng Ihe e!!ecI:

'Common sense and scIence aIIke dIcIaIe IhaI, aII oIher
IhIngs beIng Ihe same, we shouId raIher aIIrIbuIe Ihe
e!!ecI Io a cause whIch I! reaI wouId be very IIkeIy Io
produce II, Ihan Io a cause whIch wouId be very
unIIkeIy Io produce II.'

WIIh IhaI, he wenI much !arIher Ihan ArIsIoIIe, who
spoke onIy on argumenIs abouI cause and e!!ecI IhaI
couId convInce a raIIonaI person. ArIsIoIIe Iacked a
su!!IcIenI concepI o! probabIIIIy Io be abIe Io cIarI!y
why such a argumenI has a cerIaIn persuasIve power.
MIII however, examIned !rom whaI IhIs persuasIve
power arose. He concIuded IhaI Ihe convIcIIon Increa-
ses I! Ihe resuII Is more IIkeIy when one hypoIhesIs Is
Irue, Ihan when Ihe oIher hypoIhesIs Is Irue.
MeanwhIIe, probabIIIIy Iheory had been InIroduced,
InIIIaIIy maInIy !or Ihe anaIysIs o! beIIIng and card
games (see c.g. ChrIsIIaan Huygens
4
(T629-T695) In
T657). ScIenIIsIs such as PIerre SImon de IapIace (T749-
T827) appIIed probabIIIIy Iheory (or "DocIrIne o!
Chances") Io scIenII!Ic and pracIIcaI probIems. In Ihe
InIroducIIon Io hIs ThorIc nnnI)IIquc dcs roInIIIIIs
5

(T8T4) he IeIIs us how !ar-reachIng Ihe ImporIance o!
hIs work Is:

2. Marc Huys, ArsIoIeIes ReIorcn, HIstorIsche UItgeverIj 2004.
3. }ohn Stuart MIII, A sysIem oj Iogc, rnIocnnIve nnd nducIve, beng n connecI ed vew oj Ihe prncpIes oj evdence nnd Ihe meIhods oj scenIjc
nvesIgnIon, Harper & rothers FubIIshers 1859
4. ChrIstIaan Huygens' De RnIocns n Ludo AIe was pubIIshed In LatIn In 1657.
5. LapIace, Thore nnnIyIque des probnbIIs, CourcIer 1814.

'In IhIs InIroducIIon I wIII presenI Ihe prIncIpIes o!
probabIIIIy Iheory, and Ihe resuIIs Io whIch I have come
In IhIs work by appIyIng Ihem Io Ihe key quesIIons o!
II!e, whIch In !acI, are mosIIy probabIIIIy probIems. You
can even say IhaI, sIrIcIIy speakIng, aImosI aII our
knowIedge Is onIy probabIe'

In IhIs work he IndIcaIes somewhaI crypIIcaIIy how !rom
Ihe observaIIon o! an e!!ecI, Ihe probabIIIIy o! possIbIe
causes can be caIcuIaIed:
Any cause Io whIch an observaIIon can be aIIrIbuIed, Is
IndIcaIed wIIh as much more probabIIIIy as Ihe
probabIIIIy o! IhaI observaIIon when Ihe cause Is
supposed Io exIsI;
Noah K. DavIs (T830-T9T0) descrIbes Ihe same a IoI
cIearer In hIs IIemenIs o! InducIIve IogIc (T895)
6
:

'CIven an eecI Io be accounIed !or, and Ihere beIng
severaI causes IhaI mIghI have produced II, buI o! whose
presence In Ihe parIIcuIar case noIhIng Is known; Ihe
probabIIIIy IhaI Ihe eecI was produced by any o! Ihese
causes Is as Ihe anIecedenI probabIIIIy o! Ihe cause,
muIIIpIIed by Ihe probabIIIIy IhaI Ihe cause, I! II exIsIed,
wouId have produced Ihe gIven eecI.'

1he observaIIon o! an e!!ecI Is Ihus evIdence () !or a
hypoIhesIs abouI Ihe cause (H). In maIhemaIIcaI
noIaIIon, IapIace and DavIs say IhaI In IhIs sIIuaIIon:

I(H}) = I(H)I(}H).

In IhIs noIaIIon I(H}) means: Ihe probabIIIIy o!
hypoIhesIs H beIng Irue, gIven Ihe observaIIon o! Ihe
evIdence . And I(}H) means Ihe probabIIIIy o!
observIng , gIven IhaI hypoIhesIs H Is Irue.
7

PoIsson (T78T-T840) wrIIes In T837
8
on IapIace's work:

'1he soIuIIon he gave o! IhIs probIem, one o! Ihe mosI
deIIcaIe o! probabIIIIy Iheory, Is based on Ihe prIncIpIe
IhaI serves Io deIermIne Ihe probabIIIIIes o! dIverse
causes Io whIch one can aIIrIbuIe Ihe observed !acIs; a
prIncIpIe IhaI BIayes [sIc] has presenIed !IrsI In a
sIIghIIy dI!!erenI !orm, and o! whIch IapIace
subsequenIIy has made Ihe besI use, In hIs memoIrs
and hIs IreaIIse, !or caIcuIaIIng Ihe probabIIIIy o!
!uIure evenIs a!Ier Ihe observaIIon o! pasI evenIs'.

PoIsson Ihus re!ers Io Ihe Iheorem named a!Ier an
IngIIsh cIergyman, 1homas Bayes (T702-T76T), whose
work on IhIs subjecI was pubIIshed posIhumousIy In
T763
9
and conIaIned a specIaI case o! IhIs Iheorem. 1he
work commenced by Bayes was conIInued by IapIace.
A very heIp!uI !orm o! Bayes' Iheorem Is expressed In
odds: Ihe probabIIIIy raIIos IhaI we know !rom
gambIIng on c.g. Ihe ouIcome o! horse races. 1hIs
!ormuIa shows how an observaIIon o! an e!!ecI
provIdes evIdence Io heIp us choose beIween Iwo
possIbIe causes (hypoIheses HT and H2):

1 1 1
2 2 2
prior odds likelihood ratio posterior odds
( ) ( , ) ( , )
( ) ( , ) ( , )
3 + 3 ( + 3 + (
3 + 3 ( + 3 + (
u

1he prIor and posIerIor odds are a good measure !or
our degree o! beIIe! In Ihe IruIh o! eIIher hypoIhesIs
be!ore (prIor) and a!Ier (posIerIor) observaIIon. 1he
IIkeIIhood raIIo (IR) Is a measure o! Ihe Increase In IhIs
convIcIIon, and Ihere!ore o! Ihe evIdenIIaI vaIue o! Ihe
observaIIon.
T0
1he courI Is uIIImaIeIy InIeresIed In Ihe
posIerIor odds: Ihe probabIIIIIes o! Ihe hypoIheses
gIven Ihe observaIIon.
In Ihe !ormuIa, we see how Ihe odds are Increased by
Ihe observaIIon (I): by a !acIor (Ihe IR) equaI Io Ihe
probabIIIIy o! IhaI observaIIon when HypoIhesIs T Is
Irue, dIvIded by Ihe probabIIIIy o! IhaI same
observaIIon when HypoIhesIs 2 Is Irue. 1hIs brIngs us
one sIep !urIher Ihan Ihe quaIIIaIIve IndIcaIIon o! |.S.
MIII.
As an exampIe, IeI us appIy IhIs knowIedge Io Ihe case
menIIoned by ArIsIoIIe o! Ihe despoI who asks !or a
bodyguard.
In II, Ihe evIdence Is Ihe observaIIon IhaI DIonysIus
asks !or a bodyguard. WhaI does IhIs observaIIon IeII
us abouI Ihe hypoIheses IhaI are Io be IesIed? 1here
are Iwo hypoIheses:

HT = DIonysIus wIII become a despoI;
H2 = DIonysIus wIII noI become a despoI.

On Ihe basIs o! oIher, prevIousIy esIabIIshed daIa, we
couId esIImaIe Ihe prIor odds: our degree o! beIIe!
prIor Io Ihe observaIIon o! Ihe bodyguard requesI.
Bayes' Iheorem IeIIs us IhaI due Io Ihe observaIIon Ihe
prIor odds Increase by

1
2
( , )
.
( , )
3 ( +
3 ( +

1o esIImaIe Ihe evIdenIIaI vaIue o! Ihe observaIIon, we
musI noI onIy examIne Ihe !requency wIIh whIch
despoIs asked !or Ihe !ormaIIon o! a prIvaIe bodyguard
prIor Io IheIr despoIIsm, buI aIso how o!Ien non-
despoIs dId Ihe same. ArIsIoIIe expIIcIIIy menIIons Iwo
exampIes o! Ihe !IrsI: PeIsIsIraIus and 1heagenes. Ior a
beIIer esIImaIe, we wouId obvIousIy IIke Io know how a
Iarger number o! non-despoIs and despoIs have acIed
prIor Io IheIr regImes, buI we'II have Io do wIIh Ihe
daIa IhaI ArIsIoIIe !urnIshes.

(T) Suppose IhaI prIor Io Ihe bodyguard requesI we
esIImaIe Ihe probabIIIIy IhaI DIonysIus wIII be a despoI
aI 75, based on oIher In!ormaIIon. 1he prIor odds
Ihen are

3/ 4
3
1/ 4

(3 agaInsI T).

6. Noah K. DavIs, IemenIs oj InducIve Logc, Harper & rothers FubIIshers 1895.
7. These two probabIIIty statements shouId not be conIused. ThInk Ior exampIe oI the probabIIIty that a woman Is pregnant. That chance Is not equaI to the
probabIIIty that a pregnant person Is... a woman.
8. SImeonDenIs FoIsson, Recherches sur In FrobnbII des )ugemenIs en MnIre CrmneIIe eI en MnIre CvIe, acheIIer 1837.
9. Thomas ayes, 'An Fssay towards soIvIng a FrobIem In the DoctrIne oI Chances', FhIosophcnI TrnnsncIons oj Ihe RoynI 5oceIy oj London 53 (1763), 370418.
10. ThIs observatIon can be a measurement resuIt, but aIso a judIcIaI observatIon, or an observatIon by a wItness.

IxperIIse cn RechI 2U1U-2 55

(2) Suppose IhaI In Ihe pasI Ihere were Ihree despoIs,
Iwo o! whIch (PeIsIsIraIus and 1heagenes) prevIousIy
asked !or a bodyguard (2 ouI o! 3). O! Ihe sIx non-
despoIs !rom Ihe pasI, Iwo have aIso asked !or
bodyguards (2 ouI o! 6), wIIhouI subsequenIIy
deveIopIng InIo a despoI. Irom Ihose ancIenI
observaIIons In IhIs exampIe, we wIII derIve Ihe
sIrengIh o! Ihe evIdence.

Our prIor odds (o! 3 Io T) Increase due Io Ihose
observaIIons by a !acIor equaI Io

2 / 3
2.
2 / 6

1hIs !acIor o! 2 Is Ihe IIkeIIhood raIIo (IR) In IhIs
exampIe. In a medIcaI conIexI Ihe synonymous Ierm
'dIagnosIIc vaIue' Is o!Ien used. In IhIs case IhIs raIIo
IndIcaIes Ihe evIdenIIaI vaIue o! Ihe observaIIon IhaI
DIonysIus asks !or a bodyguard. 1he worryIng requesI
!or a bodyguard makes Ihe odds !or despoIIsm doubIe
!rom 3 Io 6. 1hose are Ihe posIerIor odds, In whIch Ihe
evIdence Is Iaken InIo accounI. 1hey are 6 agaInsI T
IhaI DIonysIus wIII be despoI. 1he esIImaIIon o! Ihe
probabIIIIy IhaI he wIII become a despoI wIII Ihere!ore
Increase !rom 75 Io 86 (6,7). WIIh IhIs, Ihe mIIIennIa
oId exampIe o! ArIsIoIIe was soIved!
TT

1Ime Io Iake sIock.

In Ihe !IrsI pIace, II's ImporIanI Io reaIIze IhaI Ihe
evIdenIIaI vaIue o! Ihe observaIIon Is reIaIed Io aI IeasI
Iwo hypoIheses. In Ihe second pIace, noIe IhaI Ihe
evIdenIIaI vaIue (IIkeIIhood raIIo) Is reIaIIve: onIy Ihe
Increase (or decrease) o! our degree o! beIIe! Is gIven,
and noI Ihe degree o! beIIe! IIseI!. 1he degree o! beIIe!
gIven Ihe evIdence Is expressed by Ihe posIerIor odds,
!or whIch In addIIIon Io Ihe evIdence, Ihe prIor odds
need Io be Iaken InIo accounI. IeI's re!IecI on IhIs !or a
momenI.
II Is a common mIsIake Io IhInk IhaI Ihe probabIIIIy o!
a hypoIhesIs can be dIrecIIy derIved !rom (Ihe
evIdenIIaI vaIue o!) an observaIIon. A hIsIorIcaI
exampIe Is Ihe mIscarrIage o! jusIIce whIch has become
known as Ihe Drey!us a!!aIr. Among Ihe evIdence
agaInsI Ihe Irench |ewIsh army o!!Icer Drey!us (T859-
T935) were documenIs IhaI were aIIrIbuIed Io hIm -
wrongIy, as II Iurned ouI - and !rom whIch wouId
!oIIow IhaI he was acIIve as a spy !or Ihe Cermans. 1he
promInenI !orensIc experI AIphonse BerIIIIon (T853-
T9T4) commIIIed IhIs mIsIake In Ihe InIerpreIaIIon o!
Ihe handwrIIIng evIdence, a poInI IhaI was made
paIn!uIIy cIear In T906, by Ihe renowned
maIhemaIIcIan HenrI PoIncare (T854-T9T2)
T2
:

'... gIven Ihe ImpossIbIIIIy o! knowIng Ihe prIor
probabIIIIy, we cannoI say: IhIs agreemenI proves IhaI
Ihe raIIo o! Ihe probabIIIIy o! a !orgery Io Ihe Inverse
probabIIIIy has IhIs or IhaI vaIue. We can onIy say, by

observIng Ihe agreemenI: IhaI raIIo becomes IhIs much
Iarger Ihan be!ore Ihe observaIIon.'

WIIhouI knowIng Ihe prIor odds we can'I say anyIhIng
abouI Ihe posIerIor odds. I! we know Ihe IIkeIIhood
raIIo (Ihe evIdenIIaI vaIue), we onIy know by how much
our degree beIIe! shouId Increase or decrease buI noI
whaI IhaI degree o! beIIe! shouId be.
II Is aIso possIbIe IhaI Ihe evIdence Is noI In!ormaIIve
aI aII on Ihe probabIIIIy o! Ihe hypoIheses beIng Irue.
Indeed when Ihe observaIIon Is equaIIy IIkeIy under
boIh hypoIheses, Ihe evIdenIIaI vaIue Is equaI Io T. 1he
observaIIon does noI Increase or decrease our degree
o! beIIe!, and In IhIs case Is IrreIevanI !or Ihe
hypoIheses consIdered.

IeI's go back Io our 'cIassIc' exampIe.

I! In ArIsIoIIe's exampIe PeIsIsIraIus had noI become
despoI, Ihen Ihe IR wouId have been equaI Io

1/ 3
1.
2 / 6

An observaIIon wIIh an evIdenIIaI vaIue o! T does noI
dIsIInguIsh beIween Ihe hypoIheses and Is neuIraI, or
In oIher words, o! no weIghI.

Such a crIIerIon !or reIevancy can be recognIzed In Ihe
U.S. IederaI RuIes o! IvIdence:

'RuIe 40T. De!InIIIon o! "ReIevanI IvIdence"
"ReIevanI evIdence" means evIdence havIng any
Iendency Io make Ihe exIsIence o! any !acI IhaI Is o!
consequence Io Ihe deIermInaIIon o! Ihe acIIon more
probabIe or Iess probabIe Ihan II wouId be wIIhouI Ihe
evIdence.'

In Ihe above we were makIng caIcuIaIIons wIIh
numbers. BuI when no quanIIIaIIve daIa are avaIIabIe,
Ihe same reasonIng Is equaIIy appIIcabIe.

1he core o! Ihe above can be summarIzed as !oIIows:

T. 1he evIdenIIaI vaIue o! observaIIons ('evIdence') Is
reIaIed Io Ihe hypoIheses IesIed. 1hIs concerns aI
IeasI Iwo hypoIheses whose probabIIIIIes are
consIdered reIaIIve Io one anoIher. 1he BayesIan
reasonIng scheme Ihus !orces us Io Iake aIIernaIIves
InIo consIderaIIon, and - In a crImInaI jusIIce
conIexI - noI mereIy Ihe accounI o! evenIs o! c.g.
Ihe prosecuIIon.
2. An observaIIon ('evIdence') poInIs In Ihe dIrecIIon o!
Ihe hypoIhesIs under whIch Ihe observaIIon Is mosI
IIkeIy. 1haI "poInIIng" Is reIaIIve: Ihe raIIo o! Ihe
probabIIIIIes o! Ihe observaIIon under boIh
hypoIheses (Ihe IR) gIves Ihe reIaIIve Increase o!
our degree o! beIIe!, regardIess o! how Iarge or
smaII IhaI beIIe! was be!ore Ihe observaIIon.

11. DIonysIus (c. 430367 C) In reaIIty dd become a despot.
12. Henry Mornard, L'njjnre Dreyjus: In revson du procs de Rennes, LIgue IranaIse pour Ia deIense des droIts de I'homme et du cItoyen 1907, p. 334.


3. 1he evIdenIIaI vaIue In IIseI! says noIhIng abouI Ihe
probabIIIIy o! Ihe hypoIheses wIIhouI knowIng, or
wIIhouI aI IeasI makIng assumpIIons abouI Ihe
probabIIIIy o! Ihose hypoIheses prIor Io knowIng
Ihe evIdence, I.c.: Ihe prIor odds. In one IIne: no
posIerIor odds wIIhouI prIor odds.

1here Is one more poInI Io make. NoI aII knowIedge Is
uncerIaIn and noI aII sIaIemenIs are probabIIIIy
sIaIemenIs. SomeIImes someIhIng Is obvIousIy noI
Irue. 1here are Indeed !acIuaI and IogIcaI
ImpossIbIIIIIes whIch we may sa!eIy assume dId noI
occur. We onIy gIve a reassurIng exampIe: none o! Ihe
auIhors o! IhIs arIIcIe has gIven SocraIes Ihe hemIock.

In Ihe nexI parI o! IhIs IrIpIych, enIIIIed "ReasonIng In
Ihe courIroom", we wIII dIscuss Ihe appIIcaIIon o! Ihe
BayesIan reasonIng scheme In crImInaI jusIIce. 1hereIn
we wIII demonsIraIe Ihe above usIng exampIes. II wIII
noI requIre caIcuIaIIons. 1he core o! our argumenI Is
IhaI Ihe BayesIan reasonIng scheme promoIes InsIghI,
even wIIhouI quanII!yIng Ihe evIdence and Ihe odds.

Thc IrIpIych vIdcncc and convIcIIon" Is abouI raIIonaI rcasonIng !or scIcnIIsIs and Iawycrs. Thc BaycsIan
!ramcwork gIvcs a numbcr o! ruIcs and provIdcs a cIcar vIcw o! Ihc pII!aIIs.
1. vIdcncc and convIcIIon: RaIIonaI rcasonIng sIncc ArIsIoIIc
2. vIdcncc and convIcIIon: RcasonIng In Ihc courIroom
3. vIdcncc and convIcIIon: A cIcar vIcw o! Ihc pII!aIIs
PROCEEDINGS
Towards a uniform framework for reporting
opinions in forensic science casework
IW EVETT
Forensic Science Service, 109 Lambeth Road, London SEI SLP, United Kingdom
Plenary lecture presented at the First Meeting of the
European Academy of Forensic Sciences, Lausanne, Switzerland, 1997
The Bayesian approach to interpreting evidence is now interesting for many reasons, not least being the following
firmly established within forensic science. Because it pro- comparison:
vides a logical framework for every type of scientific evi-
dence it naturally leads to a set of unifying principles.
These, in turn, suggest the manner in which expert opinions
should be communicated to the courts. Weight of evidence
is properly conveyed by the use of the word supports
together with an appropriate qualifying term. The range of
qualifiers that are used can be chosen with reference to the
underlying concept of the likelihood ratio. This philosophy
points the way to uniform standards for reporting strengths
of corroborative opinions throughout the science and in all
languages and disciplines.
Introduction
There have been various attempts to establish conventions
for reporting strength of opinions in forensic science state-
ments and reports, see for example [ I ] and [2]. The case for
harmonising such conventions across all of the disciplines
and also across nations is strong. First, by restricting termi-
nology to a limited range of words, we aim to reduce the
problems of effective communication with laymen. Second,
communication among forensic scientists of differing disci-
plines and languages must be facilitated. Third, it may be
seen as stabilising the foundations of forensic science.
This paper is concerned primarily with the third of these
aims. If we are to be serious about our claim that our pur-
suit is truly scientific, then any reporting convention must
be rooted firmly in logic.
The Starzecpyzel case
The case of United States vs Starzecpyzel [3], although
concerned with handwriting evidence, should be of interest
to all of those who have an interest in determining the
extent to which any branch of forensic science is a 'sci-
ence'. The admissibility of the evidence in the case was
challenged in a lengthy Daubert hearing, during the course
of which both sides called expert testimony to support argu-
ments for and against. The ruling of Judge McKenna is
The court has studied the nature of the skill claimed by
forensic document examiners, and finds it to be closer
to a practical skill, such as piloting a vessel, than to a
scientific skill, such as that which might be developed
by a chemist or a physicist.
The one aspect of this case which is of interest here is that,
in support of the argument that the pursuit was scientific, a
prosecution witness explained that the American Board of
Forensic Document Examiners had adopted a standardised
terminology for reporting conclusions. This nine point scale
had been described by McAlexander et a1 [4] and is incor-
porated in an ASTM standard [5]. It is not necessary to
reproduce the entire scale here, what is of particular interest
are those opinions which invoke the concept of probability.
In particular, the second point in the scale is classed as
strong probability which is described as:
The evidence is very persuasive, yet some critical fea-
ture or quality is missing so that an identification is not
in order; however, the examiner is virtually certain that
the questioned and known writings were written by the
same individual.
Examples: There is strong probability that John Doe
wrote the questioned material, or it is my opinion (or
conclusion or determination) that John Doe very prob-
ably wrote the questioned material.
The third point in the scale is classed as probable and is
described as:
The evidence contained in the handwriting points
rather strongly toward the questioned and known writ-
ings having been written by the same individual; how-
ever, it falls short of the 'virtually certain' degree of
confidence.
Examples: It has been concluded that the John Doe of
the known material probably wrote the questioned
Science & Justice 1998; 38(3): 198-202
material, or it is my opinion (or conclusion or determi-
nation) that the John Doe of the known material prob-
ably wrote the questioned material.
The seventh and eighth points in the scale are defined in
similar manner as the third and second respectively, but
refemng to negative opinions.
Whereas the motivation of US document examiners to
agree a common convention is laudable their execution was
far from satisfactory. If the approach is to be scientific then
it is necessary to ensure that statements which invoke the
concept of probability are rooted in a logical framework.
Unfortunately, the above statements do not satisfy this
requirement.
Opinions of moral certainty
This paper is concerned with reporting in those cases where
the weight of evidence in favour of a given proposition falls
short of what may be termed 'moral certainty': i.e. the case
in which a scientist states a categorical opinion that two
entities have the same source. Stoney [5] provided a valu-
able contrast between fingerprint evidence and DNA profil-
ing evidence and explained how the opinion of certainty
associated with a fingerprint identification involves a 'leap
of faith'. The process by which this leap of faith is made
appears to be incapable of description within a logical
framework. This is not to demean the activity, nor to fail to
recognise its value to criminal justice but there appears to
be nothing intrinsically scientific about a categoric opinion
of identity of source.
The rest of the paper is concerned with those cases where
the expert considers that the weight of evidence falls short
of that needed for the Stoney leap of faith - only corrobo-
rative opinions will be considered.
Probability
Words such as probable, likely, chance and odds are used
Probability is personal
If the question about tomorrow's weather is put to different
people then it will not be surprising if there is a range of
answers. This is because different people will know andlor
assume different things. Because of this it is necessary to
recognise that probability is personal.
True, it is possible to think of trivial situations - such as
those based on tossing a coin or a die - in which all rational
people would tend to give the same answer. But in the more
interesting situations which occur in everyday life and in
the work of forensic scientists then so the scope for
disagreement increases.
Notation
It is necessary to introduce some notation at this point. It is
convenient to employ a single letter, for example H, to
denote a proposition (or hypothesis) about which there is
uncertainty. It will also help to use a single letter, say I, to
summarise everything that we know and/or assume in rela-
tion to H. Then we denote the probability of H by
Pr(HI I ) . The 'Pr' is self-explanatory; H and I have already
been defined; it remains only to explain the meaning of the
vertical line. This is probably the most important symbol in
probability theory: it is shorthand for 'given' or 'if'. So the
notation, if written in full, would read 'the probability that
H is true given that I is true'.
It is not particularly meaningful to consider the truth of a
particular proposition without considering at least one alter-
native proposition. The symbol fi can be used to denote the
alternative proposition to H. Often, it is useful to talk in
terms of odds and the odds in favour of H are simply the
probability of H divided by the probability of H. We can
write:
0 (H 11) =
Pr ( H I I )
P ~ ( H I I )
extensivelv when talking about corroborative evidence: if
u
not in the first instance by the scientist, then later at court
by the judge and advocates. Yet it appears that few forensic
scientists receive training in the meaning of probability.
True, all scientists will have received instruction in basic
statistics as part of their undergraduate education. However,
those courses mostly pay no more than passing reference to
probability and, if then, in terms of tossing coins and deal-
ing playing cards. Probability has deeper and richer mean-
ings than these trivial examples suggest.
Probability is conditional
The answer that a person gives to a simple, everyday ques-
tion such as 'what is the probability of rain tomorrow?',
will depend on many things: location; the time of year; the
person's knowledge; what the weather forecast says; and so
on. It follows that to deal with probability scientifically, it is
necessary to recognise that any probability is conditioned
by what is known andlor assumed.
Science & Justice 1998; 38(3): 198-202
Whereas probabilities can take values only between zero
and one, odds can take any positive value. Odds greater
than one mean that H is more probable than f i ; odds less
than one mean that H is more probable than H.
A logical framework
Probability theory allows the creation of an extensive logi-
cal framework, based on this simple view of probability,
together with three basic laws. Within this framework there
is one result - Bayes' theorem - which is of crucial impor-
tance in our search for a logical foundation for forensic
inference.
The detail of Bayes' theorem is not discussed here. Much
has been written of its application in our field and interest-
ed readers should refer, for example to the books by Evett
and Weir [6], Robertson and Vignaux [7] and Aitken [8].
The central ideas, within the context of forensic science, are
as follows.
199
Framework for reporting opinions in forensic science casework
In any case that comes to court there will be certain non-sci- difference between these two different kinds of question is
entific evidence for the jury to evaluate. This could include essential if one is to claim any kind of scientific approach to
factors such as: opportunity; motive; eyewitness' evidence; the evaluation of evidence. There is no other logical
alibi; and so on. It will be convenient to use C to denote the approach that is available to us.
prosecution proposition that the suspect is the person who
committed the crime (assuming, of course, that the court is
satisfied that a crime has, indeed, been committed). Then
prosecution will urge the jury to consider Pr(CII): the
probability that the suspect committed the crime given the
non-scientific evidence. The court, however, must take a
balanced view and will consider the alternative proposition
C that some other person committed the crime. Then
Pr(C II) is also important and it is convenient to talk in
terms of odds:
0 (C II) =
Pr (C I I)
P ~ ( C I I )
In words, the odds in favour of C given the non-scientific
evidence.
Assume now that the scientist gives evidence about some
examination that has been made: summarise this evidence
by E. Then the conditioning on which the jury must consid-
er its two important probabilities has changed because it
now includes E as well as I. This is a classic example of
updating uncertainty in the light of new information and
Bayes' theorem is a simple model for doing this logically.
In the present context, the theorem provides the following
equation:
This is the fundamental formula of forensic science inter-
pretation and its importance cannot be over-emphasised.
The idea has been around since the early part of the centu-
ry. Taroni et a1 [9] have recently discovered that it con-
tributed to the argument of Poincari and his colleagues in
their report which was instrumental in the exoneration of
Dreyfus. Mosteller and Wallace [lo] used a Bayesian
approach in their analysis of the problem of the authorship
of The Federalist papers and a major step forward in clari-
fying the role of Bayes' theorem in the interpretation of
transfer evidence was taken in 1970 by Finkelstein and
Fairley [ l 11.
The formula can be expressed in words as follows:
Posterior odds = Likelihood ratio x Prior odds
The court is concerned with questions of the kind 'what is
the probability that the defendant committed the crime
given the evidence?' but Bayes' theorem demonstrates that,
for the scientist to assist the court in updating its probabili-
ties helshe must address questions of the kind 'what is the
probability of the evidence given that the defendant com-
mitted the crime?' An understanding of the fundamental
The prosecutor's fallacy
DNA profiling has had a major impact on forensic science
and crime detection. Unlike most other evidence types, the
new techniques have been subjected to intense scrutiny in
the courts and the statistical methods underlying the pre-
sentation of the evidence attracted particular attention.
Alone, of all transfer evidence types, the weight of a DNA
match is usually indicated numerically. The intense scrutiny
of the numbers that have been presented has undoubtedly
been good for the science and it has focused attention on
issues which have implications for all kinds of scientific
evidence. Of these, the one which has attracted most atten-
tion is the so-called 'prosecutor's fallacy' (Thompson and
Schumann [12]).
It is common practice to convey the weight of a DNA match
by giving the denominator of the likelihood ratio, invoking
the implicit assumption that the numerator is one. Thus, the
scientist may say something of the kind 'the probability of
a match if the DNA came from someone else is one in a mil-
lion'. In an endeavour to make this statement more accessi-
ble to a jury, a lawyer may attempt to translate it as 'given
that we have a DNA match, the probability that the DNA
came from someone else is one in a million'. In the notation
introduced earlier, the first sentence is an expression of
P~(EIC,Z) and the second sentence is an expression of
P~(CIE,I). The E and C have been interchanged about the
vertical line and the more general name for this kind of
error is 'transposing the conditional'. The scientist's sen-
tence is correctly worded - the lawyer's re-interpretation of
it is quite incorrect and illogical.
In the above context it is easy to see how the two sentences,
one correct and one incorrect, might be confused. One of
the illustrations that puts the error into starker contrast is the
two sentences: 'If an animal is an elephant then there is a
strong probability that it has four legs'; and 'if an animal
has four legs then there is a strong probability that it is an
elephant'. The first sentence is unexceptionable; the second
is patent nonsense. But this is no more than the illogical
progression between the two DNA statements.
Viewed from this logical framework, the flawed nature of
the ASTM standard can now be seen. Imagine that the
examiner compares the known handwriting of John Doe
with some questioned handwriting and he concludes that
there is a close correspondence between them. Let M denote
the evidence that there is a close correspondence; and let C
denote the proposition that John Doe wrote the questioned
handwriting. Then, in the spirit of the notation used here he
might agree with something to the effect:
Science & Justice 1998; 38(3): 198-202
Pr (M I C) = 'very small'
Or, in words, 'the probability of such a close correspon-
dence if someone other than John Doe wrote the questioned
handwriting is very small'. The prosecutor's fallacy then
leads to:
Pr (C I M ) = 'very small'
Or, 'the probability that someone other than John Doe wrote
the questioned handwriting, given such a close correspon-
dence, is very small'. This clearly demonstrates the rele-
vance of the phrase 'transposed conditional' and is exactly
the same kind of error as in the elephant example. From this
incorrect deduction, the following further deduction can be
made:
Pr (C I M) = 'very strong'
Or, 'the probability that John Doe wrote the questioned
handwriting, given such a close correspondence, is very
strong'.
This third statement, which is similar to the ASTM verbal
convention, follows from the illogical step between the first
and second statements.
The prosecutor's fallacy has been a factor in successful
appeals in the British courts. Here is an extract from an
Appeal Court ruling in one of these cases (R. v. Doheny and
R. v. G.Adams [I9971 1 Cr App Rep 369):
The scientist should not be asked his opinion of the
likelihood that it was the Defendant who left the crime
stain, nor when giving evidence should he use termi-
nology which may lead the Jury to believe that he is
expressing such an opinion.
This is a clear directive which is routinely followed in the
reporting of DNA cases in the UK. A DNA scientist is not
permitted to make a statement of the form 'there is a strong
probability that the DNA profile is that of John Doe'. It is
difficult to see how the use of probability in that way can be
justified in other areas of forensic science.
Principles of statement writing
The Bayesian view of forensic inference presents us with
three principles that should govern, not just how we think
about our evidence, but also how we present it to a jury.
They can simply be expressed as follows:
1. Interpretation of evidence takes place within a frame-
work of circumstances
All of the terms in the statement of Bayes' theorem given
earlier include I in the conditioning. It is necessary that the
scientist should view the evidence that has been found with-
in that framework. Many will see this as an unoriginal state-
ment: most forensic scientists would subscribe to this view
without the need for Bayesian inference. The advantage of
the approach is that the intuitive view has been logically
formalised.
Science & Justice 1998; 38(3): 198-202
Because the scientist's interpretation depends on the cir-
cumstances, it follows that the relevant circumstances, as
they appear to the scientist at the time of examination,
should be recorded in the statement. It is true that, in many
cases, the circumstances will change as the case develops
and so it will be necessary for the scientist to review his
interpretation accordingly.
2. To interpret evidence it is necessary to consider at least
two propositions
For any scientist it must be a guiding precept that it is not
sensible to speculate about the truth of a particular hypoth-
esis without considering at least one alternative hypothesis.
This is as true in forensic science as it is in astrophysics,
anthropology or archaeology. Bayes' theorem formalises
this.
The forensic scientist who works in a busy crime laborato-
ry will often face the temptation to focus on the prosecu-
tor's view of the evidence. However, for a balanced view,
we know that it is necessary to consider the defence posi-
tion. The choice of propositions depends on several factors:
what the prosecution allege; whether the suspect proposes
any particular line of defence; the circumstances of the
incident and the arrest; and the scope of the scientist's
expertise. The choice of propositions has recently been dis-
cussed at length by Cook et a1 [13]. Whatever propositions
are decided on by the scientist, they should be clearly pre-
sented in the statement.
3. It is necessary for the scientist to consider the probabil-
ity of the evidence given each of the stated propositions
It is clear that the ASTM reporting standard does not with-
stand logical scrutiny. It is not meaningful for a forensic sci-
entist to address the probability of the prosecution proposi-
tion. After following the steps in the previous two sections,
the expert should consider the probability of the evidence
given each of the two stated propositions and thus assess the
extent to which one of them is supported by the evidence.
If we can quantify the weight of evidence then the problem
of indicating the amount of support is resolved, but this
only seems to be realistic in DNA cases. In all others, we
must use linguistic qualifiers and this is where the notion of
a convention comes in. The scale in Table 1 could form a
TABLE 1 Scale of likelihood ratios
and strength of verbal support for the
evidence.
Verbal equivalent
1 to 10 Limited support
10 to 100 Moderate support
100 to 1000 Strong support
Over 1000 Very strong support
Framework for reporting opinions in forensic science casework
basis for discussion. Likelihood ratios that are less than one
have a corresponding scale for expressing support for the
alternative proposition.
The Forensic Science Service convention is always under
review, and changes are made from time to time, but the
same broad structure has been in place for nearly ten years.
Reporting convention
The first major conference of ENFSI is an excellent oppor-
tunity to set in train a pan-European move towards a uni-
form reporting standard. Although it is true that forensic
science evidence is often difficult to quantify, the existence
of a numerical scale as a central concept provides us with
the basis for rational and coherent debate about the use of
terminology. The Bayesian framework unifies all kinds of
scientific evidence across all languages and cultures.
ENFSI is the ideal organisation for taking these ideas for-
ward into the 2 1 st Century.
References
1. Brown GA and Cropp PL. Standardised nomenclature in forensic sci-
ence. Journal of the Forensic Science Society 1987; 27: 393-399.
2. Rudram DA. Interpretation of scientific evidence. Science & Justice
1996; 36: 133-138.
3. McAlexander, TV, Beck J and Dick R. The standardization of hand-
writing opinion terminology. Journal of Forensic Sciences 1991;
36: 311-319.
4. Anon. Standard Terminology for Expressing Conclusions of Forensic
Document (sic). Designation: E 1658-96. West Conshohocken,
PA: American Society for Testing and Materials, 1996.
5. Stoney DA. What made us ever think we could individualize using sta-
tistics. Journal of the Forensic Science Society 1991; 31: 197-199.
6. Evett IW and Weir BS. Interpreting DNA Evidence: Statistical
Genetics for Forensic Scientists. Sinauer, Sunderland MA 1998.
7. Robertson B and Vignaux GA. Interpreting Evidence: Evaluating
Forensic Science in the Courtroom. Chichester: John Wiley &
Sons, 1995.
8. Aitken CGG. Statistics and the Evaluation of Evidence for Forensic
Scientists. Chichester: John Wiley & Sons, 1995.
9. Taroni F, Champod C and Margot P. Forerunners of Bayesianism in
early forensic science. Jurimetrics 1998; 38: 183-200.
10. Mosteller F and Wallace DL. Applied Bayesian and Classical
Inference - the Case of the Federalist Papers. New York: Springer-
Verlag, 1964.
11. Finkelstein MO and Fairley WB. A Bayesian approach to identifica-
tion evidence. Harvard Law Review 1970; 83: 489-517.
12. Thompson WC and Schumann EL. Interpretation of statistical evi-
dence in criminal trials - the prosecutor's fallacy and the defence
attorney's fallacy. Law and Human Behavior 1987; 11: 167-187.
13. Cook R, Evett IW, Jackson G, Jones PL and Lambert JA. A model for
case assessment and interpretation. Science & Justice 1998;.38:
151-156.
Science & Justice 1998; 38(3): 198-202
The origin of evidence
K. Inman
a
, N. Rudin
b,*
a
California Department of Justice, 626 Bancroft Way, Berkeley, CA 94710, USA
b
1563 Solano Avenue, PMB 506, Berkeley, CA 94707-2116, USA
Received 16 August 2001; received in revised form 7 January 2002; accepted 16 January 2002
Abstract
Forensic science is an applied science based on the laws of physics and chemistry. Over time, a set of fundamental concepts has
developed that apply specically to a forensic analysis. Traditionally, ve concepts have been articulated: transfer, identication,
individualization, association between source and target, and reconstruction. We suggest that an additional sixth concept, the idea
that matter must divide before it can be transferred, is necessary to complete the paradigm. Divisible matter is particularly useful
in describing physical match evidence. Additionally, we propose a paradigm that logically divides into scientic principles that
govern the generation of evidence, and processes that pertain to the recognition, analysis, and interpretation of evidence. The
principles of divisible matter and transfer pertain to the generation of evidence before and during the crime event; the processes of
identication, classication or individualization, association, and reconstruction describe the practice of forensic science starting
with the recognition of an item as evidence. # 2002 Elsevier Science Ireland Ltd. All rights reserved.
Keywords: Forensic science; Criminalistics; Divisible matter; Identication; Classication; Individualization; Association; Transfer;
Reconstruction; Fundamental principles
1. Introduction
In 1963, Paul Kirk published a short monograph entitled
The ontogeny of criminalistics [1]. In this, he states:
With all of the progress that has been made in this eld,
and on a wide front, careful examination shows that for
the most part, progress has been technical rather than
fundamental, practical rather than theoretical, transient
rather than permanent. Many persons can identify the
particular weapon that red a bullet, but few if any can
state a single fundamental principle of identication of
rearms. Document examiners constantly identify
handwriting, but a class of beginners studying under
these same persons, would nd it difcult indeed to
distinguish the basic principles used. In short, there
exists in the eld of criminalistics a serious deciency
in basic theory and principles, as contrasted with the
large assortment of effective technical procedures.
Remarkably, although Kirk goes on to suggest that
criminalistics is the science of individualization, he fails
to offer us the comprehensive set of fundamental principles
whose absence he deplores.
Over the last several decades, a theoretical framework of
sorts has, in fact, evolved. These fundamental concepts
provide a philosophical and rational framework for the
application of scientic knowledge to the forensic arena.
They are concepts which guide a forensic analysis in a
logical progression, starting with understanding the origin
of evidence, and culminating in a statement of the signi-
cance of an analytical result. Unfortunately, these concepts
have evolved in a fragmented manner and, in fact, no
published record of a comprehensive organized paradigm
exists. Traditionally, forensic science practitioners have
come to understand the major paradigm of their work to
comprise following ve basic concepts [15]:
1. Transfer (Locard exchange principle) [2];
2. Identication (placing objects in a class) [3];
3. Individualization (narrowing the class to one) [1,3,4];
4. Association (linking a person with a crime scene) [5];
5. Reconstruction (understanding the sequence of past
events) [3].
Forensic Science International 126 (2002) 1116
*
Corresponding author. Tel.: 1-510-236-9386;
fax: 1-510-236-1601.
E-mail address: norah@forensicdna.com (N. Rudin).
0379-0738/02/$ see front matter # 2002 Elsevier Science Ireland Ltd. All rights reserved.
PII: S0 3 7 9 - 0 7 3 8 ( 0 2 ) 0 0 0 3 1 - 2
This framework has served the eld well for several
decades, but does not provide a complete picture of the
fundamentals involved. In particular, this model fails to
incorporate aspects of physical match evidence that see-
mingly begin only at the individualization step. The model
also fails to formally consider the origin, change, and
subsequent relationship of physico-chemical traits in the
evidence and reference samples. A basic understanding and
consideration of how material divides and the effect of
division on its physico-chemical properties are necessary
to complete the philosophical framework of the forensic
analysts work. Additionally, we suggest that identication,
individualization, association, and reconstruction are not
really basic scientic principles, but relate more properly
to the recognition, analysis, and interpretation of physical
evidence.
We have proposed two distinct, but interdependent theses.
First, we proffer an as yet unarticulated fundamental prin-
ciple necessary to the understanding of forensic evidence,
the principle of divisible matter. Second, we reorganize the
paradigm into fundamental principles relating to the gen-
eration of evidence and essential processes relating to the
practice of forensic science [6].
2. Divisible mattera fundamental principle in
forensic science
In thinking critically about the generation of evidence, we
were struck by the seemingly obvious fact that matter must
divide before it can be transferred. This suggests that an
additional fundamental principle of forensic science is
necessary to complete any forensic science paradigm, the
principle of divisible matter.
Matter divides into smaller component parts when
sufcient force is applied. The component parts will
acquire characteristics created by the process of divi-
sion itself and retain physico-chemical properties of
the larger piece.
The principle of divisible matter leads directly to three
corollaries with important consequences.
Corollary 1. Some characteristics retained by the smaller
pieces are unique to the original item or to the division
process. These traits are useful for individualizing all pieces
to the original item.
Corollary 2. Some characteristics retained by the smaller
pieces are common to the original item as well as to other
items of similar manufacture. We rely on these traits to
classify the item.
Corollary 3. Some characteristics from the original item
will be lost or changed during or after the moment of division
and subsequent dispersal; this confounds the attempt to infer
a common source.
In particular, the principle of divisible matter and its
logical corollaries have a profound effect on the forensic
process of individualization.
3. Properties useful for source determination
The process of division creates physical traits not present
in the original object, and may serve to associate separated
progeny fragments at a later time. These traits are created at
the boundary of the fracture. Boundary roughness is a natural
consequence of breaking one surface into two [7], and these
new surfaces are closely, but not completely, comple-
mentary. Forensic scientists use complementary edges in
physical match comparisons to infer a common source for
two items. For instance, tearing a piece of paper in half
will create edges that were not present in the original piece
of paper, and juxtaposing the two new edges may convince
the examiner that they were once a contiguous item (Fig. 1).
This is the most immediate consequence of the division
of matter. Also, traits and characteristics present in the
undivided object at the moment of division are carried
with all pieces that originate from it. This includes all
physico-chemical traits present in the undivided object
except those (size and shape) that dene it as intact. Exam-
ples of properties that might be inherited by the progeny
fragments are color, elemental composition, and micro-
crystalline structure.
To some extent, the properties inherited depend on the
scale of homogeneity of the original object as compared to
the scale of fragmentation. If pottery with a characteristic
texture design is broken into several large pieces, the frag-
ments will show the characteristic texture (Fig. 2). This
texture would assist in associating a pottery fragment with
the small class of pottery pieces of similar design. If,
Fig. 1. Physical match evidence: juxtaposing two pieces (A and B)
of torn paper reveals complementary edges. A sufcient length of
such complementarity may convince the examiner that the two
fragments were once one contiguous piece of paper.
12 K. Inman, N. Rudin / Forensic Science International 126 (2002) 1116
however, the pottery is shattered into small particles, it is
unlikely that any one particle will clearly exhibit the char-
acteristic texture. The fragments are likely to be placed in the
larger class of pottery with similar mineral composition
(Fig. 3), rather than the smaller class of items exhibiting
the characteristic texture.
In summary, both the new complementary boundaries
resulting from division, and the physico-chemical traits of
materials, are useful for individualizing two objects, leading
to an inference of common origin.
4. Properties confounding source determination
At the moment of division, the separated fragments
commence to change and become different both from each
other and from the original object. We call this phenomenon
temporal instability to indicate change over time; it is, most
simply, a consequence of increasing entropy. After the
process of division and dispersal, each property of the newly
created items will diverge from the others at some discon-
tinuous rate, k. This temporal instability will affect the
ability to accurately assess the original values at and before
division. A future comparative analysis of both items might
lead to the detection of different values and a potentially
erroneous conclusion of different sources for the items.
Consider again the example of torn paper. As a result of
non-uniform exposure to local environmental factors, the
edges of each piece, as well as the physico-chemical traits,
will begin to diverge from their original values. At some
point, it might become impossible to associate the two paper
fragments by either physical matching of their complemen-
tary edges or by physico-chemical traits.
In addition, the very act of division inevitably results in
the loss of some characteristics that dene the original
material. From a consideration of a fragment of paper,
one cannot infer the number of remaining pieces, nor the
exact size and shape of the original item. One cannot, as
another example, infer a sweater by examining a single ber
divided from it.
In the absence of physical matching between an evidence
and reference item, ambiguity about the original global
character of the intact item leads to the consideration of
more than one possible source for the evidence. No analysis
can determine which source is correct, even if the true source
item is recovered and compared. Even assuming all of the
physical and chemical traits from an evidence/reference pair
correspond, ambiguity will preclude an inference that the
Fig. 2. Pottery: if complementary edges are insufcient to conclude a common origin for evidence and reference samples, other physico-
chemical traits are required to compare the two objects. In this photograph, the texture seen in the two large pieces narrows the possible
sources to those pottery manufacturers creating such a texture.
Fig. 3. Minerals: if gross manufacturing traits are not present in
either the evidence or the reference material, more common
physico-chemical characteristics are compared. In this circum-
stance, only the mineral composition is available for analysis and
comparison. Pottery from many manufacturers may be included as
possible sources for the evidence fragments.
K. Inman, N. Rudin / Forensic Science International 126 (2002) 1116 13
source was this sweater, as opposed to all other sweaters
manufactured at the same time, or any other item made from
the precursor yarn(s).
5. The paradigm
With the introduction of divisible matter, we conclude
that this fundamental concept, along with the previously
understood fundamental concept of transfer, dene scientic
principles that relate to the generation of evidence. We
suggest that the concepts of identication, classication
or individualization, association, and reconstruction are
integral to the practice of forensic science and are processes
we use in our attempt to answer the various investigative
questions: who, what, where, why, when, and how. In the
rest of this paper, we rene our understanding of these
processes and relate them to the generation of evidence
via divisible matter and transfer through the crime event.
Fig. 4 is a pictorial representation of the paradigm as we
understand it. All of the ideas we have discussed are
arranged around a physical and temporal focus, the crime
event. The interactive elements of a scene, a victim, a
suspect, and witnesses are not novel. However, they are
usually depicted as a triangle with the victim, suspect, and
witnesses as apices surrounding the physical scene. We
prefer to think of these elements as overlapping domains.
Regardless, the crime denes the border between the gen-
eration of evidence and the recognition and subsequent
analysis and interpretation of evidence.
6. The principles
Only two of the concepts we have discussed thus far
emerge from the fundamental nature of matter, divisible
matter and transfer. These principles exist independently
of any human intervention, or even recognition; therefore,
we accordthema different status thanthe processes that begin
with the recognition of evidence by human beings. However,
although all matter is constantly dividing and transferring, it
does not become evidence until division and transfer occur in
conjunction with a criminal event. Note that for some types of
evidence, the contact necessary for transfer may be the force
for division. For example, a collision between two vehicles
causes the simultaneous division and transfer of paint.
Because the concept of transfer arose through the study of
dusts and other microscopic material [2], we are used to
Fig. 4. A forensic paradigm: individuals practicing a profession need a common map to guide them through their work. While never fully
articulated in an organized fashion, over the years a map of the discipline of criminalistics has emerged. Numerous workers have contributed to
the conceptual framework presented in this paper. The paradigm includes the principles of evidence formation (the origin of evidence) and the
processes of analysis that describe the profession of criminalistics.
thinking about transfer only on a microscopic scale. In fact,
it is inordinately useful to consider transfer on a macroscopic
scale as well. For instance, the scrap of paper used to write
the ransom note is only one-half of the original intact piece,
the other half of which is still in the kidnappers pocket.
Because it is impossible to draw an arbitrary line between
microscopic evidence and that which is easily visible to the
naked eye, we will take the liberty of creating the term
macro-transfer to describe this situation.
However, divisible matter does not account for a large
category of evidence, that of pattern transfer evidence, such
as prints and impressions. Although small amounts of
physical matter may be transferred,
1
it is the pattern of
transfer that concerns us, not the substance. Therefore,
divisible matter does not apply. The transfer of matter
requires its prior division; the transfer of traits may not.
2
7. The processes
At some point after the commission of a crime, evidence
may be recognized as such and collected. The recognition of
evidence, and the processes that follow in a forensic inves-
tigation, all result from decisions made and actions per-
formed by people. We therefore separate the practice of
forensic science from the fundamental scientic principles
upon which the generation of evidence rests. If the crime is
never discovered or the evidence is never detected, matter
has still divided and transferred, and traits have still trans-
ferred. But it is only by attempting to answer investigative
questions about a crime that the recognized processes of
identication, individualization, association, and reconstruc-
tion are employed. We also suggest a renement in which
identication becomes an end in and of itself, and we accord
the process of classication a formal status distinct from
identication.
7.1. Identication
Kirk and others emphasize the process of individualiza-
tion, the reduction of a class of evidence to one member. It is
useful to take a step back and realize that identication, the
categorization of evidence, can be an end in itself. For some
purposes, for instance the recognition of illegal drugs or the
quantitation of blood alcohol level, the forensic process
stops with identication. The criminal justice system is
not necessarily concerned with the marijuana eld or
methamphetamine lab from which the drugs originated
(although sometimes they may be); simple possession of
the scheduled substance fullls the criteria of illegality.
Similarly, the quantity of alcohol present in an individuals
bloodstream while operating a motor vehicle is the result of
interest, not the vintage of the wine. The process of identi-
cation answers the forensic investigation question: what is
it?
7.2. Individualization
Identication may also occur as a step leading to indivi-
dualization. To distinguish it fromend-point identication as
discussed in the previous section, we will refer to the
intermediate process that may lead to individualization as
classication. Several authors [3,4,8,9] have remarked on
the special meaning of individualization in a forensic context
as a conclusion of common source for two items. Any
forensic analysis that proceeds on the path towards indivi-
dualization relies on a comparison of at least two items.
Physics and logic determine that any individual object is
unique; this is not the question. The forensic question asks
whether items share a common origin. There may be some
disagreement about whether an item must be classied
before it is individualized. We believe that, whether inten-
tionally or not, the analyst will know what the item is by the
time, he concludes a common source. If ambiguity exists
about the classication of an item, the individualization to a
common source is also compromised. The process of indi-
vidualization answers the questions: which one is it? or
whose is it? depending on whether the item is animate or
inanimate; it does this by inferring a common source or
origin.
7.3. Association
Although the word association is used freely in describing
the results of a forensic examination, no clear denition
seems to exist, at least not in published literature. We
propose that association be dened as an inference of contact
between the source of the evidence and a target. Such
an inference is based on the detection of transferred evi-
dence. The source and the target are relative operational
denitions dened by the structure of the case; if transfer is
detected in both directions, for instance, each item is both a
source and a target of evidence.
The association process involves the evaluation of all of
the evidence for and against the inference of common
source; in other words, competing hypotheses are compared.
The probability of the evidence under competing hypotheses
is an expression of the likelihood of the evidence given that
the target and source items were in physical contact, con-
trasted with the likelihood of the evidence given that the
target was in contact with a different unrelated source. This
process requires combining the strength of the evidence
1
For instance, sweat, oils, and even a few cells are transferred in
leaving a fingerprint. We could choose to analyze the substance, for
instance the cells for DNA, rather than the pattern. This evidence
would then be considered a product of divisible matter.
2
It has recently been suggested to us that the transfer of matter
and the transfer of traits might be considered as two ends of a
continuum, effecting a convergence of theory to easily allow
divisible matter to explain the origin of all physical evidence. We
present our original thoughts in this paper and assume that the ideas
will inevitably be rened with time and scrutiny.
K. Inman, N. Rudin / Forensic Science International 126 (2002) 1116 15
established during the individualization process with addi-
tional information (such as may be provided by manufac-
turers of materials and empirical studies), as well as
assumptions made by the analyst. Others have commented
on the complexity of determining the signicance of an
association, including Robertson and Vignaux [10], and
Evett and co-workers [8].
Consider a ber collected from the body of a deceased
individual. The evidence ber from the body and the refer-
ence bers from the van carpet are found to be the same type
and to contain indistinguishable dye components. These
physico-chemical similarities are expected if the van carpet
is the source of the evidence ber, if the ber was transferred
during the crime, and if it persisted on the body until
collected. Next, an evaluation is made of all other possible
sources of bers indistinguishable from the evidence ber,
including all carpets made from such bers and any other
items manufactured from indistinguishable bers. From this
information, the probability of nding the ber on the
deceased if it derived from some other source can be
estimated.
Comparing these alternative explanations allows the ana-
lyst to decide whether the evidence supports an inference of
contact between the deceased individual and the van carpet.
Note the distinction between a conclusion of common source
(the evidence and reference ber are classied or individua-
lized as sharing a common source) and an inference of
contact between a source and a target (the carpet and the
deceased are associated).
7.4. Reconstruction
We consider reconstruction to be the ordering of associa-
tions in space and time. Reconstruction attempts to answer
the questions: where, how, and when. It should be stressed
that the when usually refers to an ordering in relative time
only; was the sweater in contact with the couch before,
during, or after the murder took place? The last remaining
question, why can never, of course, be addressed by a
consideration of physical evidence.
8. Summary
In considering a fundamental scientic basis for the
discipline of forensic science, we have articulated an addi-
tional principle, that of divisible matter. The principle of
divisible matter states that, when sufcient force is applied,
matter divides. In Corollaries 13, we nd that the process of
division creates traits or characteristics that can be used to
individualize two objects in the context of a forensic inves-
tigation. In particular, this principle supplies a novel model
for the generation and understanding of physical match
evidence, a concept that the previous paradigm failed to
address. Divisible matter allows for formal consideration of
the factors affecting change or loss of physical properties
and presages incorporation of those factors into a statement
of signicance at the conclusion of the analysis.
We have also proposed an organization of the forensic
paradigm centered around the crime event. The principles of
divisible matter and transfer interact in the generation of
evidence before and during the crime. The practice of
forensic science begins after the crime event with the
recognition of evidence. Divisible matter and transfer are
the two fundamental principles upon which the forensic
analysis of physical evidence is based. Identication, clas-
sication or individualization, association, and reconstruc-
tion form the infrastructure for the practice of forensic
science.
References
[1] P.L. Kirk, The ontogeny of criminalistics, J. Crim. Law
Criminol. Police Sci. 54 (1963) 235238.
[2] E. Locard, LEnquete criminelle et les Methodes scienti-
ques, Flammarion, Paris, 1920.
[3] P. DeForest, R. Gaensslen, H. Lee, Forensic Science: An
Introduction to Criminalistics, McGraw Hill, New York,
1983.
[4] P.L. Kirk, in: John Thornton (Ed.), Criminal Investigation,
2nd Edition, Wiley, New York, 1974.
[5] J.W. Osterburg, The Crime Laboratory: Case Studies of
Scientic Criminal Investigation, Indiana University Press,
1968.
[6] K. Inman, N. Rudin, Principles and Practice of Criminalis-
tics: The Profession of Forensic Science, CRC Press, Boca
Raton, FL, 2001.
[7] R.K. Kalia, A. Nakano, A. Omeltchenko, K. Tsuruta, P.
Vashishta, Role of ultrane microstructures in dynamic
fracture in nanophase silicon nitride, Phys. Rev. Lett. 78
(1997) 21442147.
[8] R. Cook, I.W. Evett, G. Jackson, P.J. Jones, J.A. Lambert, A
hierarchy of propositions: deciding which level to address in
casework, Sci. Justice 38 (1998) 231239.
[9] H. Tuthill, Individualization: Principles and Procedures in
Criminalistics, Lightening Powder Company, Salem, OR,
1994.
[10] B. Robertson, G.A. Vignaux, Interpreting Evidence, Wiley,
Chichester, 1995.
Scientific and technical
The nature of forensic science opinion-a possible framework
to guide thinking and practice in investigations and in court
proceedings
G Jack~on*, ~
Advance Forensic Science, Suite 121,17 Union Street, Dundee DDI 4BG, UK
e-mail: enquiriesQadvanceforensicscience.com
S Jones
Forensic Science Service, Trident Court, 2920 Solihull Parkway, Birmingham Business Park, Birmingham 837 7YN, UK
G Booth
Forensic Science Service, Usk Road, Chepstow, Gwent NP6 6YE, UK
C Champode
lnstitut de Police Scientifique et de Criminologie, Batiment de Chimie, CH-1015 Lausanne-Dorigny. Switzerland
IW Evett
Forensic Science Service, 109 Lambeth Road, London SEI 7LP, UK
Science &Justice 2006 46 33 - 44 Received 8 March 2005 accepted 21 February 2006
The questions that are asked of forensic scientists during the
course of a criminal investigation, and during subsequent court
proceedings, are of varied form. This paper attempts to place these
questions into broad generic types and explores the difference in
the inferential process that a scientist may employ when forming
opinions that help answer these questions. From this model, a
working definition of different roles and attributes for forensic
scientists is described which may offer greater clarity for both
practitioners and users of forensic science.
Las cuestiones que se plantean a 10s cientificos forenses en el
curso de una investigaci6n criminal y en el subsiguiente pro-
cedimientos judicial son de variadas formas. Este trabajo intenta
colocar estas cuestiones en amplios grupos genCricos y explora
la diferencia en el proceso de inferencias que un cientifico puede
emplear cuando se forma opiniones que ayudan a contestar estas
cuestiones. Se describe a partir de este modelo una definici6n
de trabajo de 10s diferentes roles y atributos de 10s cientificos
forenses de forma que se proporciona mayor claridad tanto para
10s que practican como para 10s usuarios de las ciencias forenses.
Die an forensisch arbeitende Wissenschaftler wiihrend laufender
Ermittlungen und in den nachfolgenden Gerichtsverhandlungen
gerichteten Fragen sind von verschiedenartiger Form. Diese Ar-
beit versucht diese Fragen in weit gefasste Gattungen einzuteilen
und erkundet den Unterschied im Folgerungsprozess, den ein
Wissenschaftler einsetzen kann, um sich eine Meinung zu bilden,
die bei der Beantwortung dieser Fragen helfen kann. Aus diesem
Model1 heraus wird eine Arbeitsdefinition verschiedener Rollen
und Eigenschaften fur forensisch arbeitende Wissenschaftler en-
twickelt, die sowohl den Ausfiihrenden als auch den Nutzern der
Kriminaltechnik mehr Klarheit bringen kann.
Les questions demandkes aux forensiciens durant l'investigation
criminelle, ou durant les dCbats au tribunal, presentent des formes
variCes. Cet article essaie de placer ces questions dans des types
gCnCriques et explore la diffCrence dans le processus infkrentiel
qu'un scientifique peut utiliser lorsqu'il forme ses opinions qui lui
permettent de rkpondre h ces questions. A partir de ce modkle une
dCfinition de travail des differents r6les et attributs du forensicien
est propos6e. Celle-ci peut offrir une plus grande clartC aussi
bien pour les pratiquants que pour les utilisateurs des sciences
forensiques.
*Author for correspondence
aWork carried out whilst employed by the Forensic
Science Service
Q The Forensic Science Society 2006
Key words Investigation, evaluation, opinion, expertise,
evidence, forensic science, probability, likelihood ratio,
Bayes.
science&justice Volume 46 No. 1 (2006) 33 - 44 Page 33
G Jackson, S Jones, G Booth, C Charnpod, and IW Even
The nature of forensic science opinion
Introduction
Previous papers [ 1 4 ] have described a model for Case Assess-
ment and Interpretation (CAI) and the principles on which it is
based. One of the principles is the requirement that a pair of for-
mal propositions be put forward in order for the scientist to offer
an evaluation of the weight of her evidence. One of the papers [3]
developed the notion of explanations and explored the difference
between explanations and propositions. It was suggested that ex-
planations have a role in the investigative stage of an enquiry
whereas logical evaluation of weight of evidence, in a judicial
setting, requires formal propositions based on prosecution and
defence positions.
More recent work by the current authors with practitioners across
a broad spectrum of forensic science disciplines has led to a
deeper understanding of the application of the CAI model. In
particular, the nature of opinion has been explored and a clearer
picture of thinking and practice in the different phases of a case
has emerged.
This paper explores the nature of forensic science opinion and at-
tempts to formalise a distinction between the roles of the scientist
in the investigative and judicial phases of a criminal investigation.
Inference and the nature of opinion
The role of forensic science may be defined as:
The provision of information to help answer questions of im-
portance to investigators and to courts of law
The information provided by forensic scientists may be simply
factual, and therefore non-contentious, or it may take the form
of an opinion. In developing an opinion, the scientist will utilise
some form of inferential process and, if we require fresh insights
into the role of forensic science, understanding the nature of the
inferential process appears to be a necessary prerequisite.
We begin with a consideration of whether or not, when giving an
opinion, a scientist is able to provide robust, reliable and informed
answers to all questions that are asked during a case.
As a practical example, in the field of DNA evidence, a Court of
Appeal judgment (Doheny and Adams [I9971 1 Cr.App.R 369.)
in the English and Welsh jurisdiction has provided guidance on
this very issue. The Court of Appeal considered the scientist's role
during the presentation of DNA evidence in rape cases where the
issue was the source of semen recovered from a victim. For that
specific situation, the Appeal Court said:
bility of the origin of a crime stain, then that view will be limited
by the scientist's inevitably partial knowledge of the totality of
the evidence in a case. Potentially misleading opinion could then
be given.
However, it is notoriously difficult for a scientist to resist of-
fering answers to questions of this type - questions that are
of obvious importance to the court. Much effort goes into the
training of DNA scientists to deal with such questions and to
use a form of words [7] that does not offend the Appeal Court
guidance. In contrast, there is less concentration on this aspect
for scientists who deal with other forms of evidence. The so-
called 'Doheny/Adams' DNA guidance is just as relevant, log-
ically, to any other form of scientific evidence as it is to DNA
evidence. We believe more attention should be given to how
scientists respond to questions that are asked of them during
the various stages of a case as it progresses from investigation
through to a court trial. We hope that, through a consideration
of forensic science reasoning, this paper is a contribution to that
debate.
It should be noted that we do not intend in this paper to consider
the special nature of, and the reasoning process used to arrive at,
categorical opinion as in, for example, 'This (finger)mark was
left by Mr X' or 'This bullet was fired from this gun'. This type
of opinion has been discussed elsewhere [8,9].
Reasoning
Nordby [lo] provides a useful description and examples of rea-
soning in a forensic science context and defines three forms of
inference - deductive, inductive and abductive. For the moment
we will put to one side the process of abduction and concentrate
on deduction and induction.
Deductive reasoning
Deductive reasoning occurs in those situations where a logical
rule can be applied to a particular set of observations. For example,
let us assume that there is a rule of the form 'If A, B and C are
true, then X must be true'. It follows that when we observe A,
B and C, we can deduce that X is true. This form of reasoning
requires complete knowledge and understanding of the world in
which the questioned event exists. Take, for example, a case in
which it is beyond all doubt that a murder was committed by one
of five men (A) and that the offender, whoever it was, had left
his blood at the scene (B). Assume also that each suspect had
a DNA-profile different from that of each of the other four men
(C), then it may be deduced that the man whose profile matched
that at the scene was the murderer (X).
The scientist should not be asked his opinion on the likelihood
Examples of deductive reasoning in forensic science may be
that it was the defendant who left the crime stain, nor when
found in classification systems. Let us assume that we have a
giving evidence should he use terminology which may lead
set of reference samples that represent all possible types of a
the jury to believe that he is expressing such an opinion
class of object or material. Also let us assume that we have a set
The likelihood, or probability, that it was a particular defendant
of rules, or a scheme, based on knowledge of the attributes of
who left a crime stain depends not only on the strength of the
those samples, and that the scheme allows us to discriminate be-
scientific evidence but also on the combined strength of the other
tween each of the samples. Any questioned sample can therefore
non-scientific evidence. If a scientist gives a view on the proba-
be analysed and categorised using deduction to arrive at the class
of the questioned sample.
Page 34 sciencetkjustice Volume 46 No. 1 (2006) 34 - 44
G Jackson, S Jones, G Booth, C Champod, and IW Evett
In our opinion, however, only a limited number of forensic science
questions are amenable to deductive reasoning. Firstly, there are
few instances involving closed, known populations and, secondly,
the knowledge and understanding of forensic scientists in any one
specific area is rarely complete. For those reasons, we believe the
more appropriate form of inference for most forensic science
questions is inductive reasoning.
Inductive reasoning
Induction is the process of reasoning from a set of observa-
tions within a framework of incomplete knowledge. Whereas
deductive reasoning is deterministic, or exact, inductive rea-
soning involves uncertainty. Philosophers have debated the
nature of induction for centuries but Jaynes [ll] provides
a concise account of the problem and its solution through
probability.
Bayes' theorem offers a practical, robust mechanism for inductive
reasoning. The theorem provides a logical framework to appraise
the value of new pieces of information and to update one's uncer-
tainty about a questioned event. It is gaining wide acceptance as
a robust approach to forensic science problems and it is the basis
of the CAI model [I]. Of central importance in the CAI model
is the concept of a likelihood ratio (LR). The LR modifies the
prior odds of an uncertain event into the posterior odds of that
event. In essence, the LR can be likened to a weight that is added,
depending on whether it is greater than or less than 1 in numerical
value, to one or other of the pans of the scales of justice 151. Fol-
lowing the argument of Good [12] that the measure of weight of
evidence should be the logarithm of the likelihood ratio, various
verbal scales have been proposed to express that weight (see [4],
for example).
A key consideration, if Bayesian inference is to be employed in
a forensic science context, is to define whose role it is to:
1. decide on the prior odds
2. evaluate the LR
3. assess the posterior odds
To explore this aspect, we will revisit the concept of the hierar-
chy of propositions [2] and extend that to a notion of a hierarchy
of questions that is its forerunner. This may offer fresh insight
into the types of question that are of importance in forensic sci-
ence and into whether or not scientists provide answers to these
questions.
The hierarchy of questions
The notion of a hierarchy of propositions has been very helpful
in structuring and understanding the issues in a case, particularly
in the judicial setting [2, 3, 131. We do not suggest that all
case situations can be forced into this apparently rigid structure
but we have found that the framework helps clarify the purpose
of the scientific work in this setting. We have found that the
specification of propositions begins with a consideration of the
questions that are of importance. We deal firstly with the types of
questions usually associated with court proceedings. A generic
framework of questions, based on three levels of the hierarchy of
propositions, would be of the form:
Did he commit the offence?
Did he do this activity?
Does the recovered material have the same source as the ref-
erence material?
It is important to note that, with these questions, there is an
implicit assumption that a particular 'activity' and a particular
'offence' have taken place. In addition, the questions are usually
centred on the defendant and/or his property. For convenience,
we will describe these questions as 'judicial' because, in general,
they are of primary importance in a court setting.
It is useful to compare and contrast this generic structure with the
position before there is a suspect. What are the generic questions,
for example, when there are no witnesses to an incident?
From our work with practitioners, we have developed a set of
questions for 'investigative' issues that mirrors the hierarchy for
'judicial' questions and takes the general form:
What offence, if any, has been committed?
What activity happened?
What is the source of this recovered material?
Note that the main differences are, firstly, the removal of any
mention of a suspect or defendant and, secondly, there are no
assumptions about what activity or offence occurred.
It feels appropriate that, with this 'investigative' set of questions,
it is the scientist who is called upon, where she is able, to provide
guidance on the answers to the 'source' and 'activity' types
of questions. In some instances, the scientist may well provide
answers even to the 'offence' questions.
In contrast, the scientist has been guided, at least in DNA cases
in courts of law, not to provide an answer to the source-level
'judicial' question. If a scientist cannot provide an answer to a
source-level question, it follows that she is also unable to provide
an answer to a higher, activity-level question. With very few ex-
ceptions, offence-level questions are the sole province of the jury.
If this model of 'investigative' and 'judicial' questions is a useful
structure for the general types of questions that seem to be of im-
portance in forensic science, how does the scientist address them
and, in particular, are there any differences in the style of operation
for the scientist when dealing with these two types of questions?
Operating in 'Investigative' mode
We mentioned earlier that Nordby [lo] describes a third process
of inferential reasoning called abduction. The process of abduc-
tion, he argues, is at the root of investigation. Abductive reasoning
follows a process of generating likely explanations, testing these
with new observations and eliminating or re-ranking the expla-
nations. In this way, the investigator arrives at her 'best bet' to
explain the observations, continually refining that view as further
observations are made. Nordby suggests (p. 224) that the clas-
sic fictional investigator, Sherlock Holmes, utilises an abductive
method that '. . . follows guesses, or hypotheses, that remain sim-
ple and natural, easy, and cheap to test, and yet that contribute to
explaining all the relevant facts'.
However, it is not clear to us how, in abduction, an investigator
generates the initial, likely explanations.
In an attempt to understand more fully the abductive process, we
have tried to apply a Bayesian framework to abduction and to
describe the logical steps in the inferential process (Table 1). If
this is a useful and valid description of the investigative process,
it should provide pointers to the skills and knowledge required
in good investigators. We explore this in detail for some of the
stages of this model:
Table 1 A description of the 'investigative' process.
Stage
1. Observation
2. Hypothesis generation
3. Prior probabilities
4. Observation likelihoods
5. Observation (second phase)
6. Posterior probabilities
7. Communication
Activities
Stage 2) Hypotheses generation
The list of hypotheses generated by the investigator should be
exhaustive within the framework of circumstances of the case.
In particular, hypotheses with significant priors should not be
omitted. Wide experience, in-depth knowledge and good under-
standing would therefore seem to be key requirements if an in-
vestigator is to generate a useful set of hypotheses in a given case
situation. Inexperienced investigators may not generate a suffi-
ciently wide set of hypotheses and thereby run the risk of missing
plausible explanations. By contrast, some investigators may gen-
erate, maybe through over-active imagination, totally implausible
hypotheses - hypotheses that most rational, experienced people
would agree are 'non-starters'. The risk then is one of mislead-
ing the overall investigation by providing worthless explanations.
There will be, of course, a balance to be struck between the ar-
tificial limits imposed by inexperience and the unending list of
hypotheses generated by over-active imagination. The balance
may need to be towards the latter. %o key attributes at this stage
would therefore seem to be:
-knowledge of all reasonable, potential explanations
- a thinking style that remains open and inventive
Stage 3) Prior probabilities
Having listed all possible hypotheses, the process now requires
probabilities to be assigned for each of the hypotheses. What
factors might influence the scientist's thinking in this respect?
Almost certainly, the scientist will be influenced, consciously
or sub-consciously, by what she has been told about the case
circumstances. There may well also be the influence of her past
experience of similar situations and of the explanations that were
proven to be true, or otherwise, in those cases. This information
and knowledge may provide a valid, robust assessment of the
priors but there is arisk of the scientist being swayed by unreliable
or limited prior information, from witnesses for example, towards
an erroneous high prior for one hypothesis.
Initial observations are made
A set of hypotheses, that are exhaustive in the context of the case, is generated to explain
the observations
Probabilities for each hypothesis are assigned
Probabilities (likelihoods) for the range of possible future observations, given that each of
the hypotheses were true, are assigned
Further searching and observations are made
Posterior probabilities for each hypothesis are evaluated and the hypotheses re-ranked
in order of probability (N.B. if the probability of the observation is zero, given the
hypothesis were true, then that hypothesis must be eliminated)
The opinions are communicated to the user
Page 36 sciencetkjustice Volume 46 NO. I (2006) 36 - 44
A useful tool for this stage would be the availability of databases
of past cases from which to assign prior probabilities, and a key
attribute would be:
- awareness and avoidance of bias
Stage 4) Likelihoods of future observations
How does the scientist assign probabilities (likelihoods) for po-
tential observations?
Experience of similar situations, either through casework or
through the scientist's own practical experimentation, would be
of benefit at this stage. Good memory of the outcomes of these
previous situations and experiments, and good organisation of
that information, would be an essential attribute to help form ro-
bust probabilities. There is a risk that, through poor recall and/or
bias, poor probability judgements may be made, thereby skewing
further actions and appraisal. The availability of published data,
relevant to the case in hand, would greatly assist in the assignment
of subjective probabilities. Key attributes would be:
- extensive knowledge and good recall of outcomes of previous
cases and experiments
- the ability to assign robust probabilities for potential observa-
tions
-for 'source' issues, knowledge of within-sample and between-
sample variability of evidential materials
- for 'activity' issues, additional knowledge of transfer, persis-
tence and detection, together with background occurrence, of
the evidential material
Stage 5) Second-phase observations
It would seem that this stage would require the scientist to have in-
quisitiveness, drive, thoroughness and balance of approach. There
is the risk of bias towards seeking only those signs that point to-
wards a preconditioned, preferred explanation. Once a few of
those signs are found, the scientist may feel no need to continue
the examination. Signs that point away from the preferred option
are not sought or, worse still, if they are found, are ignored [14].
Key attributes:
- inquisitiveness, drive and balance
-ability to keep options open and not to eliminate hypotheses too
quickly
Stage 6) Posterior probabilities
In Bayesian terms, once prior probabilities and likelihoods (prob-
abilities for the observations) have been assigned, the calculation
of posterior probabilities, and hence re-ranking, is a straight-
forward mathematical process. In practical terms, however, it
is difficult to imagine a situation in which there will be an
explicit, mathematical precision to the process. The produc-
tion of priors and likelihoods may occur as a natural but non-
mathematical process in the mind of the better investigative
scientist.
A list of hypotheses with the 'best bets' (the most probable) at
the top of the list will be the result of the re-ranking process.
In the sense that this process is a mental one, rather than an
explicit, recorded process, it is neither transparent nor auditable.
The outcome of the process will not necessarily be robust because
of the possible hidden influence of bias [14]. A key attribute at
this stage would be:
- the ability to integrate prior probabilities with likelihoods
Stage 7) Communication
The interface between the providers and users of forensic sci-
ence information is a critical one and is an aspect that has been
overlooked for too long. How do scientists communicate the ben-
efits and limitations of the information they are providing? Do
scientists, police officers and lay people speak the same lan-
guage and, perhaps more importantly, do they think in the same
way?
It is perhaps natural that people prefer categoric answers to
questions-'just tell me what happened' or 'just tell me where
this has come from'. As a result, scientists may feel obliged to
offer strong, even categoric, opinions in order to provide mean-
ingful help. However, a probabilistic view of the world, and of
dealing with uncertainty, is a more robust approach but it is one
that is difficult to communicate.
General comments on the 'investigative' model
Nordby [lo] proposes that an investigator '. . .. integrates abduc-
tion, deduction and induction into a unified process' but that
'The steps begin with abduction.' De Forest [15] describes an
investigative process called 'hypothesis generation and cogita-
tion'. Jamieson [16] describes a cyclical, investigative process
of 'observation-hypothesis-assessment' that ultimately leads to
the 'most likely' hypothesis. However, we are not convinced that
any of these descriptions provides a clear definition of a logical
process that leads to the most likely explanation. Nor, we should
add, are we convinced that, as the generation of hypotheses is of
an open-ended nature, a Bayesian framework offers a complete
description of the complex, creative thought-processes involved
in investigation. We believe it does offer, however, a fresh per-
spective and may eventually lead to a better description of the
underlying nature of investigation.
We would suggest here that a working description of what may
be called investigative opinion would be:
an expression of hypotheses that have been generated to ex-
plain observations
We have described this type of opinion elsewhere as explanation
[31.
If the scientist goes further and expresses posterior probabili-
ties for these hypotheses, i.e. which of the hypotheses is the
more likely, then there should be a clear expression of the prior
probabilities that were used in arriving at these opinions. We
believe that any expression of posterior probabilities, without
science&justice Volume 46 NO. 1 (2006) 37 - 44 Page 37
making clear the priors, and the information on which the pri-
ors were based, runs a serious risk of providing misleading
opinion.
Recent work [17, 181 has demonstrated the potential of apply-
ing Bayesian reasoning, through the use of Bayesian networks,
to fire investigation. The authors describe how the initial (prior)
probabilities could be assigned through the use of a database.
The availability of databases of information on past cases, on the
results of experimentation and on surveys would be an invalu-
able source of help and guidance in stage 2 through to stage 5
('hypothesis-generation' through to 'second-phase observations')
of the model described in this paper.
Operating in 'Evaluative' mode
The early CAI papers [l-41 concentrated on the scientist's role as
a case progresses towards court and on the evaluation of an LR
for the scientific findings. The Appeal Court guidance mentioned
earlier clearly puts the assessment of prior and posterior odds
during court proceedings in the province of the jury. The jury will
have heard all the non-scientific evidence that has been presented
in the case and will have formed views, albeit maybe unexpressed
or subconscious, on the probability of the various proposed events
and, ultimately, on the defendant's guilt or innocence. If there
are two, mutually exclusive, competing propositions, exhaustive
in the framework of circumstances of the case, then the odds
form of Bayes' theorem can be used. The role of the forensic
scientist could then be defined as that of providing an LR for
the scientific findings. We would suggest then that a working
description of what we could call an evaluative opinion would
be:
an expression of the magnitude of the LR
In order for the scientist to provide useful evaluative opinion,
the propositions that the scientist addresses must be based on the
questions that are of importance to the court. Good communica-
tion between the forensic participants is the key to helping the
scientist appreciate the issues and thereby to provide an effective
service in forensic science.
However, there will be situations that do not fit comfort-
ably with this simple model of evaluation and these require
special consideration. These situations would include those
where:
a. there is uncertainty about the events at the scene of an incident
b. the defendant has given what is called a 'No comment' response
during interviews with the police.
How should a scientist proceed in each of these situations?
a. Uncertainty about the incident
If there are no eyewitnesses to an incident, then inevitably a hy-
pothesis needs to be generated by someone if progress is to be
made. The police (and, in some situations, the prosecuting author-
ities) may well have a strong suspicion about what happened at a
scene, based, for example, on past experiences of similar events.
It would seem that there are two reasonable options available to
the scientist:
- the scientist could adopt a working proposition based on the
police suspicion or suggestion
- if the scientist has, at an earlier stage of the case, formed
an investigative inference about the 'activities' at the scene,
based on her scientific observations and the prior information,
then it is that inference that could be adopted as the working
proposition
Whether or not the first option would be acceptable to the de-
fence, when a case comes eventually to court, is debatable but the
emphasis must be on the proposition being a working hypothesis
that is open to challenge, revision or rejection by the court.
With the second option, it is essential that the scientist explains
clearly how she has arrived at the investigative opinion on which
the proposition is based. It may be that the opinion would be
affected by further information put to the scientist at a later date.
This needs to be made explicit and to be understood by all users
of forensic science evidence.
There may be a third option of addressing a 'source' issue instead
of an 'activity' issue. This may be of help when in investigative
mode but there are real risks of providing misleading evidence
if used in an evaluative context. Scientific findings may provide
very valuable investigative information about potential sources
of recovered material. However, when used to help address an
evaluative issue of whether or not a defendant carried out a par-
ticular activity, or committed a particular offence, that value may
be much reduced [13].
b. 'No comment' responses
The scientist is in a dilemma in these situations. There may well
be a reasonable 'prosecution' proposition but, in order to operate
in evaluative mode and offer a balanced opinion on the weight
of her findings, the scientist requires an altemative proposition to
consider. There are several options available for the scientist:
- the scientist could adopt, on behalf of the defendant, an alter-
native proposition
- the scientist could explore a range of altemative explanations
- the scientist could discuss the likelihood of the observations
given the prosecution proposition
The first option would enable the scientist to assess a likelihood
ratio and thereby to offer an evaluative opinion, according to our
earlier definition.
With the second option, the scientist would be offering what we
have called investigative opinion.
The third option provides the scientist with, we believe, a novel
approach. The scientist would be offering a comment not on a like-
lihood ratio but rather on a single likelihood. We could classify
this sort of opinion as a preliminary form of evaluative opinion.
However, if this were to be offered as an opinion, there must
be strong guidance to all the users of that opinion that no-one
Page 38 science&justice Volume 46 No. 1 (2006) 38 - 44
G Jackson, S Jones, G Booth, C Charnpod, and IW Evett
can evaluate logically the weight of these observations until at
least one aiternative proposition is considered. We suggest, in
this situation, that the scientist declares her willingness to eval-
uate the findings when a defence alternative becomes available.
This approach invites a response from the parties in the Criminal
Justice System and engages the scientist in the process.
There are strengths and weaknesses to all three approaches and
it may be that all three have their uses at different stages in the
Criminal Justice process. What is clear to us is the need for all
relevant participants in the Criminal Justice System to be engaged
fully in deciding which is the most appropriate in any particular
case. It is our intention to discuss and develop these issues in a
further paper.
Summary
We propose three types of forensic science opinion.
Investigative opinions
- explanations, or conjectures, for observations, sometimes
associated with posterior probabilities for the explanations
Preliminary evaluative opinions
- expressions of the likelihoods for the findings, given the
truth of individual propositions
Fully evaluative opinions
-expressions of the magnitude of the likelihood ratio.
Whichever is to be offered by the scientist will depend just as
much on the needs of the customer as it does on the case circum-
stances. Investigative opinions are generally appropriate when
there are only vague, or very limited, details of the circumstances
of an incident. Preliminary evaluative opinions are useful as a
case proceeds towards a court hearing and there is no defence
alternative available. Fully evaluative opinions are of importance
when a case is to be heard at court, particularly when the issues in
question centre on the defendant, when there are two competing
propositions and when 'transposing the conditional' [7] would not
be advisable. Investigative opinions may still be of importance
and relevance to a court hearing, particularly when the issues are
further away from the defendant [7].
Case examples
I . A 'missing person' incident
A scientist has been called by the police to the home of a woman
who has been reported missing for a number of days. When access
is gained, areas of bloodstaining are observed in the house but
there is no body to be found in the premises. The woman lived
alone and there are no witnesses to say what has occurred in the
house. The police requirement is to find out what has happened.
On examination of the scene, the scientist identifies four main
areas of bloodstaining on the living room walls. The staining is
positioned between 50 and 100 cms from the floor and consists
of fine spatter and some larger splashes and smears of blood. On
the floor, between these bloodstained areas, there are round drops
of blood. There are what appear to be drag marks in blood across
the floor towards the rear door from the last area of bloodstaining.
What could be inferred, reasonably and usefully, from the obser-
vations?
Let us assume the scientist offers the opinion that there had been
a sequence of four sites of assault and that the victim had been
dragged from the site of the last attack to the rear door. How has the
scientist arrived at that answer, and with what degree of certainty?
Are there other possible explanations? For example, could one
reasonable alternative explanation be the woman having sustained
a broken nose in a fall? The blood on the wall was a result of her
having sneezed/coughed blood in four different places and then
wiping up the last set of drops with a single sweep of a towel.
The absence of blood elsewhere was a result of her leaving the
premises clutching the towel to her nose.
Another explanation, albeit a highly speculative one in most peo-
ple's view, could be that the house was occupied by a blood
pattern expert who had been experimenting with her own blood.
And why not suggest that there had been an attempt to frame
someone and that the pattern had been constructed to look like a
vicious assault?
Whatever the rights or wrongs, and the plausibility, of the sug-
gested explanations, it can be seen that it is possible to construct
sets of actions that will go some way to explaining the observa-
tions. Following the framework described earlier, a key question
is - how does a scientist, when in investigative mode, construct
these explanations and then give a view on the most likely?
In this case example, there may well be some prior conditioning
of the scientist's thinking. She may well be thinking 'The police
have called me out so there must be something suspicious about
the incident'. The scientist's prior probability on an assault may
well be high. This may increase if, on attending the scene, the
local police officer says, in an aside to the scientist, that the
ex-husband has a history of domestic violence and is a 'strong
suspect'. It is difficult for the scientist to put this to one side and
not to adopt assault as the most likely explanation, i.e. having the
highest prior probability before any scene observations have been
made.
Alternatively, prior information may be available that suggests
the missing person suffered severely from dizziness and loss of
balance and her condition made her highly prone to falling over.
In the scientist's mind, the prior probability on an assault may well
be lower than in the first situation and the prior on an accident
high.
In both these situations, the danger is that the scientist will be
influenced, possibly implicitly, by the prior information. The sci-
entist must be aware of the danger and be very conscious of any
potential influence and therefore bias.
If the scientist were to follow the investigative model described
earlier, she would generate a range of hypotheses based on ex-
perience and knowledge. An element of inventiveness and imag-
ination may well be required to do so. A crucial issue is the
extent to which she has the experience and knowledge to be able
to generate a sufficiently wide range of hypotheses to cover all
reasonable possibilities. Indeed, there is a question of what is to
be defined as 'reasonable'. In blood pattern interpretation, there
is a risk that forensic scientists see only patterns produced as a
result of violence in crime situations. They may see very few
blood patterns generated as a result of medical emergencies and
trauma. In that sense, their capability to generate a plausible set
of potential explanations may be limited.
Given that she is able to generate a good range of hypotheses,
she would assign prior probabilities to the hypotheses and make
those explicit. She would seek further observations on the form
and distribution of bloodstaining that would advance or push back
each hypothesis. She may form the opinion that the observations
would be more likely if there had been an assault than if any of
the other hypotheses were true. If equal priors for the hypotheses
were adopted, the effect of the observations would be to make
the assault-hypothesis the most likely explanation, the best bet,
without eliminating the others. But, critically, this opinion would
depend on the original hypotheses being exhaustive within the
framework of circumstances, on the priors being realistic and on
the competence of the scientist in making the observations and
assessments.
2. A case involvingfirearms discharge
Police have arrested a man on general suspicion of being associ-
ated with offences involving the possession and use of firearms
but they have no specific allegation against him. They take swabs
from his hands and take his clothing. The police require inves-
tigative information to help inform the types of questions to be
put to the suspect. He has not been interviewed and has made no
comments to the police.
The scientist examines the hand swabs and clothing for traces of
firearms discharge residue (FDR) and obtains positive findings
(to help keep this discussion simple, the precise form of these
findings has not been defined). What could be inferred from the
findings and what can be relayed back to the police?
As there is only a vague proposition from the police about in-
volvement with firearms, and there is no alternative proposition,
clearly the police requirement is to obtain answers to questions
about what sort of traces the suspect may have on his hands and
clothing, and about what activities the man may have been doing.
The model we have described would suggest the role of the sci-
entist is to try to provide answers to these investigative questions.
The scientist will be operating with little prior information or
conditioning but she may well be able to classify the type of FDR
trace and to give investigative information on potential 'sources'.
It may also be appropriate, and valuable to the police investigator,
for the scientist to discuss issues about transfer, persistence and
background levels of these traces and to advance the information
given at source-level to potential activity-level explanations for
the findings, such as:
-he has recently fired a gun containing x-type of ammunition
-he has recently handled such a weapon
-he has been near to such a gun being fired
- he has recently shaken hands with someone who has recently
fired a gun
- he has recently handled fireworks/heart tablets/lacquers etc.
-he hasn't fired a gun; the evidence has been 'planted'
It may be tempting to give an opinion on which of these explana-
tions is the most likely. This opinion could, however, be mislead-
ing if, firstly, inappropriate priors were considered and, secondly,
if a lack of knowledge led to poor assessment of the likelihoods of
obtaining the findings. In investigative mode, provided the users
of the opinion are well aware of the limitations, this may not be
disastrous but the consequences of poor opinion could be consid-
erable if and when the scientist enters a judicial setting and offers
opinion on the (posterior) probabilities of the explanations. The
court, as well as the scientist, may be unaware of the influence
exerted by prior information on the scientist's views.
The generation of the propositions to.be considered in evaluative
mode would seem to be best performed through dialogue with all
parties in the Criminal Justice System rather than by the scientist
alone. It may be that police investigators could carry out further
enquiries to help establish the circumstances with more preci-
sion and thereby to help with the generation of the prosecution
propositions; the defence proposition should be built on what the
suspect alleges.
3. A case involving breakage of a glass window
A window at the scene of a burglary has been broken by some
unknown means. There are no witnesses to say how it was broken
but it seems a reasonable assumption that the perpetrator of the
crime gained access to the premises through the broken window.
A suspect is arrested and makes no comments in reply to police
questions. The police wish to know if the man has on him any
glass fragments that match the broken window and, if so, what
that might mean in support of their suspicion that he is the burglar.
In this situation, it seems the scientist would be unable to carry out
any pre-assessment at activity-level because there is a lack of clear
propositions. There would appear to be a possible prosecution
proposition at source-level (i.e. 'Any glass fragments found came
from the window'). This may be a useful issue to address for
the police investigator and prosecutor but whether this would be
useful later for the court is debatable. Assuming the scientist does
address the 'source' issue, there is no defence alternative to enable
an evaluation of an LR. Specification of a pair of propositions
that are relevant to the issues in a case is a prerequisite if an
evaluation of the weight of the findings is required. So, how
could the scientist help in this situation?
The scientist could revert to investigative mode. The clothing
could be examined for glass and, if any is found, comments
could be made on possible mechanisms for how that glass may
have been obtained, irrespective of whether or not it matches the
broken window. Once analytical results for the recovered glass
Page 40 science&justice Volume 46 NO. 1 (2006) 40 - 44
are known, information on possible sources of that glass may be
given, again irrespective of whether or not it matches the broken
window.
The scientist may be able to go a stage further and offer more use-
ful information for the prosecution. If the scientist has knowledge
of the ways in which windows are broken during the commission
of similar offences, she may be able to construct 'activity' level
propositions and to comment on how likely it would be to obtain
the findings, given the truth of the constructed propositions. This
would constitute preliminary evaluative opinion. The limitation
that such an opinion is not an opinion on the weight of the evi-
dence should be stressed. Weight of evidence is assessed through
an evaluation of an LR.
To be in a position to offer afully evaluative opinion in this case,
the scientist requires:
- the prosecution proposition to be realistic and acceptable to all
parties
- an alternative, defence proposition to be proposed
Who provides and agrees these propositions, together with when
they are provided, are general policy issues for all parties in the
Criminal Justice System to debate and agree.
4. A criminal damage case
After a verbal altercation with another driver over an empty space
on a busy supermarket car park, Mr C drives his vehicle into the
empty space before the other driver can do so. Mr C gets out of his
car, locks the doors and walks away toward the store. Before he
enters the store, he looks back towards his car and sees the other
driver walking alongside it and looking down at the side of the
car as he walks by. Feeling that this is suspicious behaviour, Mr
C walks back to his car and, on inspecting the side of his vehicle,
sees that there is a long, deep scratch running the full length of the
vehicle. He immediately suspects that the other driver has caused
the damage.
By coincidence, a police officer drives into the car park at this
time. Mr C explains to the officer what he believes has happened
to his vehicle. They both enter the store and Mr C identifies to
the officer the person he believes is the other driver. On searching
this person (Mr S), the officer recovers a key from the suspect's
coat pocket but Mr S is not in possession of any other items that
could cause scratch damage to car paintwork
The police officer submits the key, paint from the damaged area
and pictures of the damage. How does the scientist proceed?
Following the CAI model, once sufficient, relevant case details
have been obtained, the scientist would consider the questions that
would seem to be of importance in the case. A possible hierarchy
for this case could look like this:
Offence level
- Has Mr S committed criminal damage?
Activity level relating to the suspect
-Has Mr S scratched the side of the vehicle?
Activity level relating to an item
-Has the key been used to scratch the side of the vehicle?
Source level
-Has any paint on the key come from Mr C's vehicle?
If on interview Mr S offers his alternative version of events, then
pairs of propositions could be generated and pre-assessment in
evaluative mode could be attempted.
a) Operating in evaluative mode
Assume that Mr S denies any involvement with the damage on Mr
C's car. The pairs of propositions that could be developed from
the hierarchy of questions could then be as follows:
Offence level:
A - Mr S criminally damaged Mr C's vehicle
A - Mr S did not; someone else is the offender
Activity level:
B1 - Mr S scratched Mr C's vehicle
B1 - Mr S did not; someone else scratched the vehicle
B2 - The key was used to scratch Mr C's vehicle
B2 - The key was not used to scratch the vehicle; it was some
other implement
Source level:
C - Paint on the key came from Mr C's vehicle
e - Paint came from another source(s)
It is important to note that an LR for any source-level pair of
propositions can be evaluated only when the source-level material
has been found. If no paint were to be found on the key, then we
cannot address this pair.
However, let us assume that paint has indeed been found on
the key. The numerator probability for the source-level pair, C
and c, would be informed by knowledge of 'within-sample'
variability of the vehicle's paint. The denominator probability
would be informed by whatever Mr S would say about his key
and why there might be paint on it. If, for example, he says
that any paint had come from a door, unspecified by him, then
knowledge of the variability of door paint would inform the
denominator. If he gives no explanation for paint on his key,
then knowledge of the background types of paint found on
keys would inform the denominator. In both situations, an LR
could be developed and offered as a fully evaluative opinion at
source-level.
sciencetkjustice Volume 46 No. 1 (2006) 41 - 44 Page 41
The nature of forensic science o~inion
Could the activity-level pairs be addressed?
Even if paint had been found on the key, there would be issues over
whether Mr S had the key in his possession at the material time.
It may therefore be that the B1 pair would not be within the remit
of the scientist, unless an assumption on the issue of possession
could be made. On the other hand, it seems the scientist could
address the B2 pair through, firstly, examination for paint deposits
on the key and, secondly, a comparison of the scratch marks with
the key's characteristics.
In relation to paint deposits, the probabilities to be assigned would
be those of obtaining the distribution and physical attributes of
paint deposits (if any) on the key given the two competing propo-
sitions. For the numerator, this would involve consideration of
transfer, persistence and within-sample variability. For the de-
nominator, if Mr S gives no explanation for why paint may be on
his key, the probability would be informed by knowledge of the
background occurrence and types of paint found on keys. An LR
for the results of an examination for paint could then be evaluated.
In relation to scratch marks, an evaluation of the strength of evi-
dence provided by the scratch marks would require the scientist
to assign probabilities for observing the particular form of scratch
marks on the vehicle, given the outcome of examination of the
key for paint and given the two propositions. For the numerator,
the probability would be informed by knowledge of the repro-
ducibility of marks made with the key. For the denominator, the
probability would be informed by knowledge of the variability
of marks found on the sides of cars that had been scratched.
Provided that robust, conditional probabilities could be assigned,
then an LR could be evaluated for the outcome of the examina-
tion of the scratch marks. An expression of the combined value
of the outcomes of both the paint examination and scratch mark
comparison could be conveyed as a fully evaluative opinion.
Could the offence-level pair be addressed?
Even if the B1 pair could be addressed, there are issues over
whether there was any intent to damage that would need to be
considered in order to progress to the offence level pair. This,
patently, is for the court, not the scientist.
b) Operating in investigative mode
In the circumstances of the case as described, a fully evaluative
opinion would appear to meet the needs of the Criminal Justice
System and a competent scientist could provide such an opinion.
However, aspects of the case could be dealt with through inves-
tigative opinions: opinions that could be formed irrespective of
the alleged case circumstances. It is useful to split these investiga-
tive opinions into two groups based on the vehicle and the key:
i) Investigative opinions re the vehicle
There is a hierarchy of investigative questions that relate to the
crime but which are completely detached from the suspect:
- How was the scratch made?
-What is the source of the scratch? (What implement?)
Let us assume that a 'marks' examiner looks at the photographs of
the scratch on the vehicle and sees, within the scratch, a number
of parallel lines between two and four in number. These lines are
spaced between 2 and 5 mm apart. The lines are quite fine but
do, in places, go down to the bare metal of the bodywork. The
examiner may then form an investigative, source-level opinion
that a likely implement would be a key She may also give a view
on the spacing and characteristics of the teeth of that key. How
has she formed that opinion? Clearly, there may be some priors
- she may know that the most likely implement used to scratch
cars on the spur of the moment is a key. As the observations fit
well with an explanation that a key was used, the high likelihood
of the observations increases the probability on it being a key that
was used to cause the damage.
Considering next the activity-level question, then the height of the
scratches, and their slightly wavy, horizontal nature, may fit well
with the scratch having been made by a person walking alongside
the car with a key in hand. Again, there may be a high prior for
this type of action and the effect of the observations would be
to strengthen the probability for that action. The examiner may
then feel able to give an investigative opinion that the most likely
mechanism was through a person walking alongside the vehicle
and using a key to make the scratch.
These inferences seem reasonable in this case but what if the
damage was nothing like that made by a key? Would the examiner
start with the same priors? Given the observations, would the
examiner reduce the probability on it being a key that was used and
increase the probability on it being another implement? Would
the examiner try to generate alternatives other than a key? Or
would she stick with the good prior of a key and try to imagine
what unusual form of key could have made the scratches?
ii) Investigative opinions re the key
Let us assume that a second scientist will examine the key, again
with no prior information whatsoever about the circumstances
of the crime. There is a hierarchy of questions that relate to the
suspect item, completely detached from the crime.
Activity
-What has the key been used for?
Source
-What is the source of any deposited material?
Let us assume that the examiner sees deposits of coloured material
on the teeth of the key. She identifies the material as paint and
states that she can see four distinct layers of paint - clear lacquer,
dark metallic-green, mid-grey and light grey. The examiner has
a comprehensive knowledge of paint and, in particular, she has a
good database of vehicle paint. From that, she forms the opinion
that the paint is from a Ford car with a metallic green topcoat,
Page 42 science&justice Volume 46 No. 1 (2006) 42 - 44
manufactured sometime between 1998 and 2000 (N.B. This is an
entirely hypothetical example).
How has she arrived at this investigative, source-level opinion?
Firstly, she may say that she has only ever seen this structure of
paint when used on vehicles, although she could not rule out other
sources. Secondly, the data collection that she has used comprises
information on the more common vehicles in the UK for only the
years 1998-2003. Hence, she cannot give a categoric, deductive
opinion on the source of the paint. What she can give is her best
guess; the explanation with the best posterior probability based
on her knowledge, the available database and her understanding
of the issues involved in paint examination. The opinion depends
on some prior probability derived from her knowledge of the
frequencies of occurrence of car makes and models in the UK.
At activity-level, the scientist may be able to offer her best expla-
nation for how the key has been used. The scientist may have prior
knowledge of the reasons why paint may be deposited on keys.
Let us assume that this knowledge suggests that scratching of ve-
hicle paint is the most common cause of paint deposition on keys.
The scientist observes the form and distribution of paint on the
key. Her opinion is that this form and distribution is very likely to
be obtained if the key had been used to scratch a painted surface.
She believes the observations would have lower likelihoods given
the key had been used for other activities. The observations, in a
sense, reinforce her prior view on scratching of vehicles - the
observations increase the probability of the key having been used
to scratch vehicle paint. Scratching of vehicle paint remains the
scientist's best explanation but she can now offer greater certainty
for that explanation.
Both the investigative and the evaluative opinions could be set
out in the statements that the examiners could write for court
purposes. However, the form of these two types of opinion, and
the inferential process used in arriving at them, are quite different
and should be reflected in the wording of the statements. Further-
more, there is nothing to preclude one scientist forming all the
opinions in this particular case provided that, when forming fully
evaluative opinion, the scientist has been able to 'detach' herself
from the findings. She would need to ask herself questions of the
type 'How likely is it that I would have obtained these observa-
tions if the propositions were true?' This notion of 'detaching
from the findings' is a key skill that may not come naturally to
investigatively-minded scientists. It is one, however, that must be
acquired if a scientist is to offer a balanced, robust evaluation.
Table 2 Summary of the roles of investigative and evaluative scientists.
lnvestigative
The issues addressed by the scientist tend to be
centred on, or generated by, the incident
The scientist begins the process by making
observations to help answer the key questions
Hypotheses are generated to explain the observations;
explanations may be speculative and/or imaginative
and/or novel
The scientist will seek new observations to
strengthen or weaken belief in the explanations
The scientist may operate from one perspective
If the probability of the observations, given the truth
of the explanation, is very low, the scientist will
seek clarification and re-assessment of the suitability
of the explanation.
Opinions provide direction to the investigation; options
for further investigation may be generated or eliminated
The scientist may provide an opinion on the truth or
otherwise of an uncertain event
Prior probabilities, either generated by the scientist
or given by others, are taken into account explicitly by
the scientist
lnvestigative opinion may be provided by the
scientist as a posterior probability
The scientist is findings-led
Evaluative
The issues addressed by the scientist tend to be
centred on the suspecvdefendant
The scientist begins the process by considering case
circumstances, the allegations and the issues
Propositions are generated generally from
prosecution and defence positions
The scientist makes observations and checks against
her predictions of the outcomes, given that each
of the propositions were true
The scientist is required to be impartial in approach
The scientist is not concerned, generally, about very
low probabilities for the observations. Concern might
be raised if the probability for the observations, given
the truth of EACH of the propositions, is very low
Information is presented to assist the court in
addressing intermediate and ultimate issues
The scientist provides information to help the court to
form an opinion on the probability of an uncertain
event
Prior probabilities are the remit of the court
Evaluative opinion is provided in the form of an
expression either of the LR or of single likelihoods;
posterior probabilities are for the court
The scientist is propositions-led
G Jackson, S Jones, G Booth, C Charnpod, and IW Evett
The nature of forensic science o~i ni on
Conclusion
We offer in Table 2 what we believe are the essential features
that characterise the two roles of investigative and evaluative
scientists, based on the differing nature of the inductive inferential
process used to form opinions in these two roles.
In investigative mode, explanations are generated from the
observations. There should be a clear statement by the scientist
of how the explanations were generated and, if a posterior
probability for one explanation is given, the prior probabilities
that have been used should be made explicit. We believe the
model we have presented is useful in beginning to analyse the
complex mental process of investigation. We believe it begins to
clarify the skills and knowledge needed by investigators to help
them operate in an effective, robust and useful manner and to
offer reliable answers to questions.
In evaluative mode, there should be clear statement of the proposi-
tions that have been addressed. These should be formulated from
the framework of circumstances of the case and through dialogue
between parties in the Criminal Justice System. Generally, prior
probabilities are not in the province of the scientist and should
not be considered by the scientist. The opinion of the scientist
should be based around an assessment of a likelihood ratio; such
an opinion we are calling a fully evaluative opinion. In certain
situations, it may not be possible to offer an opinion based on
an LR, it may only be possible to give a view on the likelihood
of the observations given specific individual propositions. These
opinions we are calling preliminary evaluative opinions.
In reality, it may well be that scientists do not think and operate
with such a clear distinction between the different modes. Indeed,
it may well be that scientists tend, by nature and maybe training, to
be investigative in their thought-processes and, therefore, in their
activities. However, we believe there are advantages in applying
this investigative-evaluative model to analyse customer requests
and to deliver tailored, staged services to customers. We have
seen that these advantages include:
-clarifying for all parties which issues are investigative and which
evaluative at each stage of the case progression
-helping to design an effective examination strategy at each stage
- specifying the nature of the opinions and hence the form of
language to be used in communicating results
- helping to resolve the issue of whether or not scientists should
be answering questions
- defining the scientist's role in the Criminal Justice System.
We believe that forensic scientists need to engage more widely
with the users of forensic science information to ensure that sci-
entists understand their customers' needs. There is also a need to
check that the users understand and are comfortable with this pic-
ture of the scientist's contribution, as we have described. Collab-
oration with the users in clarifying and agreeing that contribution
would be an excellent way forward.
Unless and until all concerned are clear on these issues, there is
a risk that there will be misinterpretation and misuse of opinion.
Acknowledgments
The authors would like to thank the many colleagues in the Foren-
sic Science Service with whom we have had invaluable discus-
sions and, in particular, Jim Larnbert and Didier Meuwly for
providing helpful comments during the drafting of this paper and
Richard Morley for the outline of case study 4.
References
1 Cook R, Evett IW, Jackson G, Jones PJ, and Lambert JA. A model for case assess-
ment and interpretation. Science and Justice 1998: 38: 151-156.
2 Cook R, Evett IW, Jackson G. Jones PJ, and Lambert JA. A hierarchy of propositions:
Deciding which level to address in casework. Science and Justice 1998; 38: 231-239.
3 Evett IW, Jackson G, and Lambert JA. More on the hierarchy of propositions: Exploring
the distinction between explanations and propositions. Science and Justice 2000; 40:
3-10,
4 Evett IW, Jackson G, Lambert JA, and McCrossan S. The impact of the principles
of evidence interpretation on the structure and content of statements. Science and
Justice 2000; 40: 233-239.
5 Jackson G. The scientist and the scales of justice. Science and Justice 2000; 40:
81 -85.
6 Champod C. Evett IW, and Jackson G. Establishing the most appropriate databases
for addressing source level propositions. Science and Justice 2004; 44: 153-164.
7 Evett IW. Avoiding the transposed conditional. Science and Justice 1995; 35: 127-
131.
8 Champod C and Evett IW. A probabilistic approach to fingerprint identification. Journal
of Forensic Identification 2001; 51(2): 101-122.
9 Champod C, Baldwin D, Taroni F, and Buckleton JS. Firearm and Tool Mark Identifica-
tion: the Bayesian Approach. Forensic Science Setvice Report RR830, March 2003.
10 Nordby JJ. Dead Reckoning: The Art of Forensic Detection. CRC. Boca Raton, FL,
2000.
11 Jaynes, ET. Probability Theory: The Logic of Science. Cambridge University Press,
Cambridge, UK, 2003.
12 Good IJ. Weight of evidence and the Bayesian likelihood ratio, in The Use of Statistics
in Forensic Science (Eds: Aitken CGG and Stoney DA). Ellis HOIWOO~, Chichester,
UK, 1991, pp. 85-1 06,
13 Evett IW, Gill PD, Jackson G. Whitaker J, and Champod C. interpreting small
quantities of DNA: The hierarchy of propositions and the use of Bayesian networks.
Journal of Forensic Sciences 2002; 47: 520-530.
14 Saks MJ, Risinger DM, Rosenthal R, and Thompson WC. Context effects in forensic
science: A review and application of the science of science to crime laboratoly
practice in the United States. Science and Justice 2003; 43: 77-90.
15 De Forest P. Recapturing the essence of criminalistics. Science and Justice 1999;
39: 196-208.
16 Jamieson A. A rational approach to the principles and practice of crime scene
investigation: 1. Principles. Science and Justice 2004; 44: 3-7.
17 Bierderman A, Taroni F. Delemont 0. Semandeni C, and Davison AC. The evaluation
of evidence in the forensic investigation of fire incidents (Part I) : An approach using
Bayesian networks. Forensic Science International 2005; 147: 49-57.
18 Bierderman A, Taroni F, Delemont 0, Semandeni C, and Davison AC. The evaluation
of evidence in the forensic investigation of fire incidents (Part 11) : Practical examples
of the use of Bayesian networks. Forensic Science International 2005; 147: 59-69.
Page 44
The Coming Paradigm Shift in
Forensic Identification Science
Michael J. Saks
1
and Jonathan J. Koehler
2
Converging legal and scientific forces are pushing the traditional forensic identification
sciences toward fundamental change. The assumption of discernible uniqueness that re-
sides at the core of these fields is weakened by evidence of errors in proficiency testing and
in actual cases. Changes in the law pertaining to the admissibility of expert evidence in
court, together with the emergence of DNA typing as a model for a scientifically de-
fensible approach to questions of shared identity, are driving the older forensic sciences
toward a new scientific paradigm.
L
ittle more than a decade ago, forensic in-
dividualization scientists compared pairs
of marks (handwriting, fingerprints, tool
marks, hair, tire marks, bite marks, etc.), in-
tuited whether the marks matched, and testified
in court that whoever or whatever made one
made the other. Courts almost never excluded
the testimony. Cross-examination rarely ques-
tioned the foundations of the asserted expertise
or the basis of the analyst_s certainty.
Today, that once-complacent corner of
the law and science interface has begun to
unravelor at least to regroup. The news car-
ries reports of erroneous forensic identifica-
tions of hair, bullets, handwriting, footprints,
bite marks, and even venerated fingerprints.
Scientists have begun to question the core
assumptions of numerous forensic sciences
(16). Federal funding has materialized to sup-
port research that examines long-asserted but
unproven claims. Courts have started taking
challenges to asserted forensic science exper-
tise seriously (1). A dispassionate scientist or
judge reviewing the current state of the tra-
ditional forensic sciences would likely regard
their claims as plausible, underresearched, and
oversold.
The traditional forensic individualiza-
tion sciences rest on a central assumption: that
two indistinguishable marks must have been
produced by a single object. Traditional foren-
sic scientists seek to link crime scene evi-
dence to a single person or object Bto the
exclusion of all others in the world[ (7, 8).
They do so by leaning on the assumption of
discernible uniqueness. According to this as-
sumption, markings produced by different
people or objects are observably different. Thus,
when a pair of markings is not observably
different, criminalists conclude that the marks
were made by the same person or object.
Although lacking theoretical or empirical
foundations, the assumption of discernible
uniqueness offers important practical benefits
to the traditional forensic sciences. It enables
forensic scientists to draw bold, definitive con-
clusions that can make or break cases. It ex-
cuses the forensic sciences from developing
measures of object attributes, collecting popu-
lation data on the frequencies of variations in
those attributes, testing attribute independence,
or calculating and explaining the probability
that different objects share a common set of
observable attributes. Without the discernible
uniqueness assumption, far more scientific work
would be needed, and criminalists would need
to offer more tempered opinions in court.
Legal and scientific forces are converging
to drive an emerging skepticism about the
claims of the traditional forensic individual-
ization sciences. As a result, these sciences
are moving toward a new scientific paradigm.
EWe use the notion of paradigm shift not as
a literal application of Thomas Kuhn_s con-
cept (9), but as a metaphor highlighting the
transformation involved in moving from a pre-
science to an empirically grounded science.^
Two such forces are outgrowths of DNA typ-
ing: the discovery of erroneous convictions and
a model for a scientifically sound identification
science. A third force is the momentous change
in the legal admissibility standards for expert
testimony. A final force grows from studies
of error rates across the forensic sciences.
REVIEW
1
College of Law, Arizona State University, Tempe, AZ
85287, USA. E-mail: saks@asu.edu
2
McCombs School
of Business, University of Texas, Austin, TX 78712,
USA. E-mail: koehler@mail.utexas.edu
Fig. 1. Factors associated with wrongful conviction in 86 DNA exoneration cases, based on case
analysis data provided by the Innocence Project, Cardozo School of Law (New York, NY), and
computed by us. Percentages exceed 100% because more than one factor was found in many
cases. Red bars indicate factors related to forensic science.
5 AUGUST 2005 VOL 309 SCIENCE www.sciencemag.org 892
Post-Conviction DNA Exonerations
During the past decade, scores of people who
were convicted of serious crimesincluding
at least 14 who had been sentenced to death
have been exonerated by DNA analyses of
crime scene evidence that had not been tested
at the time of their trials (10). It was not
surprising to learn that erroneous convictions
sometimes occur, and that new science and
technology can help detect and correct those
mistakes. Nor was it surprising to learn, from
an analysis of 86 such cases (Fig. 1), that
erroneous eyewitness identifications are the
most common contributing factor to wrongful
convictions. What was unexpected is that
erroneous forensic science expert testimony
is the second most common contributing
factor to wrongful convictions, found in 63%
of those cases. These data likely understate the
relative contribution of forensic science expert
testimony to erroneous convictions. Whereas
lawyers, police, and lay witnesses participate
in virtually every criminal case, forensic
science experts participate in a smaller subset
of casesabout 10 to 20% of criminal cases
during the era when these DNA exonerations
were originally tried (11).
Figure 1 also indicates that forensic scien-
tists are the witnesses most likely to present
misleading or fraudulent testimony. Deceitful
forensic scientists are a minor sidelight to this
paper, but a sidelight that underscores cultural
differences between normal science and foren-
sic science (12, 13). In normal science, academ-
ically gifted students receive four or more years
of doctoral training where much of the social-
ization into the culture of science takes place.
This culture emphasizes methodological rigor,
openness, and cautious interpretation of data.
In forensic science, 96% of positions are held
by persons with bachelors degrees (or less),
3% masters degrees, and 1% Ph.D.s (14).
When individuals who are not steeped in the
culture of science work in an adversarial, crime-
fighting culture, there is a substantial risk that a
different set of norms will prevail. As one
former forensic scientist noted, this pressure-
packed environment can lead to data fudging
and fabrication: All [forensic science] experts
are tempted, many times in their careers, to
report positive results when their inquiries
come up inconclusive, or indeed to report a
negative result as positive [(15), p. 17].
DNA Typing as the New Model for
Scientific Forensic Identification
Much of the above criticism does not apply
to the science of DNA typing as practiced
today. Indeed, DNA typing can serve as a
model for the traditional forensic sciences in
three important respects. First, DNA typing
technology was an application of knowledge
derived from core scientific disciplines. This
provided a stable structure for future empirical
work on the technology. Second, the courts
and scientists scrutinized applications of the
technology in individual cases. As a result,
early, unscientific practices were rooted out.
Third, DNA typing offered data-based, prob-
abilistic assessments of the meaning of evi-
dentiary matches. This practice represented
an advance over potentially misleading match/
no-match claims associated with other forensic
identification sciences.
Immediately after DNAs first courtroom
appearance in the 1980s, scientists from dis-
ciplines as varied as statistics, psychology, and
evolutionary biology debated the strengths and
limitations of forensic DNA evidence. Blue-
ribbon panels were convened, conferences were
held, unscientific practices were identi-
fied, data were collected, critical papers
were written, and standards were de-
veloped and implemented. The scientif-
ic debates focused on the adequacy of
DNA databases (16), the computation
of DNA match probabilities (17), the
training of DNA analysts (18), the pre-
sentation of DNA matches in the court-
room (19), and the role of error rates
(20). In some cases, disputants worked
together to find common ground (21).
These matters were not resolved by the
forensic scientists themselves, by fiat, or
by neglect. Most exaggerated claims
and counterclaims about DNA evidence
have been replaced by scientifically
defensible propositions. Although some
disagreement remains (22), the scientif-
ic process worked.
One of the great strengths of DNA
typing is that it uses a statistical ap-
proach based on population genetics
theory and empirical testing. Experts
evaluate matches between suspects and
crime scene DNA evidence in terms
of the probability of random matches
across different reference populations
(e.g., different ethnicities). These prob-
abilities are derived from databases that
identify the frequency with which var-
ious alleles occur at different locations on the
DNA strand. The traditional forensic sciences
could and should emulate this approach (23).
Each subfield must construct databases of
sample characteristics and use these data-
bases to support a probabilistic approach to
identification. Fingerprinting could be one of
the first areas to make the transition to this
approach because large fingerprint databases
already exist. The greatest challenge in this
effort would be to develop measures of the
complex images presented by fingerprints,
tool marks, bite marks, handwriting, etc.
(Figs. 2 and 3). Forensic scientists will need
to work with experts in differential geometry,
topology, or other fields to develop workable
measures.
A second data collection effort that would
strengthen the scientific foundation of the fo-
rensic sciences involves estimating error rates.
Although the theoretical promise of forensic
technology is considerable, the practical value
of any particular technology is limited by the
extent to which potentially important errors
arise. The best way to identify the frequency
with which errors occur is to conduct blind,
external proficiency tests using realistic sam-
ples. A proficiency test requires analysts to
make judgments about samples whose proper-
ties are known. External proficiency tests are
conducted by an agency unaffiliated with the
forensic scientists laboratory. Externality is
important to the integrity of proficiency tests
because laboratories have strong incentives to
Fig. 2. Bite mark evidence exhibit from trial of
Ray Krone, suggesting alignment of a cast of
Krones dentition with bite wounds in victims
flesh [State v. Krone, 182 Ariz. 319 (1995)]. A
forensic odontologist testified that this showed
Krone to be the biter. Krone was convicted of
murder and sentenced to death, but a decade
later he was exonerated by DNA analysis. [Source:
E. Thomas Barham (Los Alamitos, CA) and Alan
Simpson (Phoenix, AZ), attorneys for Krone]
Fig. 3. Image of two bullets viewed through a com-
parison microscope. The bullets were fired from two
consecutively manufactured Smith & Wesson 38 Spe-
cial revolver barrels. Whether fired through the same or
different barrels, numerous matching and nonmatching
striations are engraved onto bullets. To reliably identify
the barrel through which a questioned bullet was fired,
an examiner must distinguish among class, subclass, and
individual characteristics. These two bullets illustrate sub-
class characteristic agreement of striated markings on a
groove impression that could be mistaken for individual
characteristics. Without investigating the potential for
subclass carryover, the examiner could mistake these as
having been fired from the same gun. [Source: Bruce
Moran, firearms examiner with the Sacramento County
(CA) District Attorney, Laboratory of Forensic Services]
R E V I E W
www.sciencemag.org SCIENCE VOL 309 5 AUGUST 2005 893
be perceived as error-free. An even better
test would be a blind proficiency test, in
which the analyst believes the test materials
are part of ordinary case work. Blindness in-
creases the validity of proficiency test results
because it ensures that analysts treat the test
sample as they would other case samples. Al-
though proficiency tests are used in many
forensic sciences, the tests are generally in-
frequent, internal, and unrealistic; blind tests
are practically nonexistent.
Changes in the Law
Until recently, courts assessed expertise by
looking for superficial indicia of validity. In
the 19th century, courts were impressed by
qualifications and success in the market-
place. If the market valued an asserted ex-
pertise or expert, courts generally did, too.
In Frye v. United States [293 F. 1013 (D.C.
Cir. 1923)], a federal appellate court
confronted the question of admissi-
bility of an expertise that had no life in
any commercial marketplace. The
court solved the problem by substitut-
ing an intellectual marketplace. The
court asked whether the proffered
expertise had gained general accept-
ance in the particular field in which it
belongs. Sixty years later, the Frye
test had become the dominant expert
evidence filter in American courts.
In 1993, the law began to catch up
with the scientific method. In Daubert
v. Merrell Dow Pharmaceuticals [509
U.S. 579 (1993)], the U.S. Supreme
Court introduced a new standard for
the admissibility of scientific evi-
dence. Under Daubert, proffered sci-
entific testimony must be shown to
stand on a dependable foundation. The
court suggested that trial judges mak-
ing this determination consider wheth-
er the proffered science has been tested, the
methodological soundness of that testing, and
the results of that testing. The Daubert test in
effect lowers the threshold for admission of
sound cutting-edge science and raises the
threshold for long-asserted expertise that lacks
a scientific foundation. Seriously applied, the
Daubert test subjects the forensic sciences to a
first-principles scientific scrutiny that poses a
profound challenge to fields that lack rigorous
supporting data.
United States v. Starzecpyzel [880 F. Supp.
1027 (S.D.N.Y. 1995)] offered an early in-
dication of how Daubert could change judicial
views. After an extensive hearing on the sound-
ness of asserted handwriting identification ex-
pertise, a federal district court concluded that
the field had no scientific basis: [T]he tes-
timony at the Daubert hearing firmly es-
tablished that forensic document examination,
despite the existence of a certification pro-
gram, professional journals and other trappings
of science, cannot, after Daubert, be regarded
as scientific Iknowledge (p. 1038). How-
ever, the court did not exclude this unscientific
testimony. It reasoned that handwriting identi-
fication did not have to reach the Daubert
standard because Daubert applied only to sci-
entific evidence, and handwriting identification
plainly was not scientific evidence. Thus, when
a forensic science was found to stand on a
weak foundation, the threshold of admission
was lowered to accommodate this weakness.
In Kumho Tire v. Carmichael [526 U.S.
137 (1999)], the Supreme Court directly con-
fronted the question of whether Daubert
applies to nonsciences. A consortium of law
enforcement organizations prepared an amicus
brief urging that Daubert scrutiny not be ex-
tended to the testimony of police agency ex-
pert witnesses. The brief argued that the
great bulk of expert testimony provided by
law enforcement officers does not involve sci-
entific theories, methodologies, techniques, or
data in any respectI. Instead, law enforce-
ment officers testify about such things as
accident reconstruction, fingerprint, footprint
and handprint [identification], handwriting
analysis, firearms markings and toolmarks
and the unique characteristics of guns, bullets,
and shell casings, and bloodstain pattern iden-
tification (24). Ironically, then, fields that
initially gained entry to the courts by declaring
themselves to be sciences now sought to
remain in court by denying any connection
with scientific methods, data, or principles.
Despite efforts to preserve the nonscience
loophole, the Supreme Court doctrinally sealed
it shut when Kumho Tire held that all expert
testimony must pass appropriate tests of
validity to be admissible in court.
Error Rates
Although Dauberts testing recommenda-
tions are familiar to most scientists, there has
been remarkably little research on the ac-
curacy of traditional forensic sciences. Pro-
ficiency tests in some fields offer a step in
the right direction, even though simple tasks
and infrequent peer review limit their value.
Nonetheless, the available data hint that
some forensic sciences are best interpreted in
tandem with error rates estimated from sound
studies.
Unfortunately, forensic scientists often re-
ject error rate estimates in favor of arguments
that theirs is an error-free science. For exam-
ple, an FBI document section chief asserted
that all certified document examiners in the
United States would agree with his conclu-
sions in every case [(25), p. 196]. Likewise,
fingerprint experts commonly claim that all
fingerprint experts would reach the same con-
clusions about every print (2). Such hubris
was on display in spring 2004 when the FBI
declared that a fingerprint recovered from a
suspicious plastic bag near the scene
of a terrorist bombing in Madrid pro-
vided a 100 percent match to an
Oregon attorney (Fig. 4). The FBI
eventually conceded error when Spanish
fingerprint experts linked the print to
someone else (26).
The FBI and other agencies often
seek to preserve the illusion of perfec-
tion after disclosure of such errors by
distinguishing between human errors
(possible) and errors of method
(impossible). A leading FBI scientist
explained the distinction to the court in
United States v. Llera-Plaza I [58 Fed.
R. Evid. Serv. 1 (E.D. Pa. 2002)]: We
have to understand that error rate is a
difficult thing to calculate. I mean, peo-
ple are trying to do this, it shouldnt be
done, it cant be doneI. An error rate
is a wispy thing like smoke, it changes
over timeI. If you made a mistake in
the past, certainly thats valid information I
but to say theres an error rate thats definable
would be a misrepresentationI. Now, error
rate deals with people, you should have a
method that is defined and stays within its
limits, so it doesnt have error at all. So the
method is one thing, people making mistakes
is another issue.
Such claims are problematic. First, the
suggestion that humans err but forensic tech-
niques do not is unfalsifiable. It is impossible
to disentangle method errors from prac-
titioner errors in fields where the method is
primarily the judgment of the examiner. Sec-
ond, even if such disentanglement were pos-
sible, it is a red herring. When fact-finders
hear evidence of a forensic match, a proper
assessment of the probative value of that match
requires awareness of the chance that a mistake
was made. The source of such a mistake is
irrelevant for this purpose. If method errors
could be distinguished from practitioner errors,
Fig. 4. (A) A latent fingerprint believed to belong to a terrorist
involved in train bombings in Madrid, Spain, in March 2004.
(B) A database print belonging to Brandon Mayfield of Port-
land, Oregon. On the basis of these prints (though not neces-
sarily these very images), FBI fingerprint examiners erroneously
identified Mayfield as the bomber (26). [Source: Problem Idents,
onin.com/fp/problemidents.html#madrid]
R E V I E W
5 AUGUST 2005 VOL 309 SCIENCE www.sciencemag.org 894
a 1% method error affects the probative value
of the match in exactly the same way as a 1%
practitioner error. Identifying sources of error
is relevant for improving forensic science prac-
tice, but it plays no role in identifying the
probative importance of a match.
Third, the suggestion that error rates do
not exist because they change over time and
are not specific to the case at hand is a base-
rate fallacy. In this fallacy of reasoning, peo-
ple underuse (or willfully ignore) general
background data in judgment tasks because
they believe the data are irrelevant to the
instant case. However, general background
data (or base rates) are relevant for specific
predictions (27, 28). For example, although
risk estimates for a disease fluctuate and are
developed on patients other than the patient
now seeking medical advice, these estimates
provide information useful for predicting
whether this patient will contract the disease.
A 20% base-rate risk of contracting the dis-
ease makes it more likely that the patient
will get the disease than would a 1% risk.
Likewise, an X% base-rate risk of error in a
given forensic science provides some indica-
tion of the chance that a particular conclusion
is in error (22).
Data from proficiency tests and other ex-
aminations suggest that forensic errors are not
minor imperfections. Spectrographic voice
identification error rates are as high as 63%,
depending on the type of voice sample tested
[(1), chap. 31]. Handwriting error rates aver-
age around 40% and sometimes approach
100% [(1), chap. 28]. False-positive error rates
for bite marks run as high as 64% [(1), chap.
30]. Those for microscopic hair comparisons
are about 12% (using results of mitochondrial
DNA testing as the criterion) (29). Fingerprint
examiners generally fare better, although
data from a well-known forensic testing pro-
gram contradict industry boasts of perfect, or
even near-perfect, agreement (30). Since
1995, about one-fourth of examiners failed
to correctly identify all latent prints in this
test (which includes 9 to 12 latent prints and
palmprints). About 4 to 5% of examiners
committed false-positive errors on at least
one latent. In one test, 20% of examiners
mistook one persons prints for those of his
twin. The editor of the leading fingerprint
journal called this performance unaccept-
able [(31), p. 524]. It is noteworthy that
these misidentifications are not confined to a
single lab, circumstance, or marking. More-
over, the misidentification rates do not show
a clear pattern of improvement (the mis-
identification rates in 2004 were 4 to 6%).
Nor are these errors limited to arguably arti-
ficial testing situations; erroneous fingerprint
identifications have made their way out of
the crime lab and into prosecutions in at least
21 documented cases (32).
Forensic science proficiency tests and ex-
aminations are obviously imperfect indicators
of the rate at which errors occur in practice.
This fact does not justify ignoring the worri-
some data these tests have yielded. Indeed,
these data are probably best regarded as lower-
bound estimates of error rates. Because the
tests are relatively easy (according to test par-
ticipants), and because participants know that
mistakes will be identified and punished, test
error rates (particularly the false-positive error
rate) probably are lower than those in every-
day casework (33, 34).
The studies mentioned above cry out for
attention and follow-up investigations. In light
of the laws growing reluctance to accept
experts personal guarantees in lieu of sci-
entific data, these studies should increase
candor about performance and create pres-
sure for improvement.
The Future
The traditional forensic sciences need look
no further than their newest sister discipline,
DNA typing, for guidance on how to put the
science into forensic identification science.
This effort should begin with adoption of the
basic-research model. Just as DNA scientists
tested the genetic assumptions that undergirded
DNA typing theory (e.g., Hardy-Weinberg equi-
librium), traditional forensic scientists should
design experiments that test the core assump-
tions of their fields. As basic research knowl-
edge grows, experts will be able to inform
courts about the relative strengths and weak-
nesses of their theories and methods, and
suggest how that knowledge applies to indi-
vidual cases.
At the same time, data should be collected
on the frequency with which markings and at-
tribute variations occur in different popula-
tions. In addition to their case-specific benefits,
these data may also facilitate the development
of artificial intelligence or computer-aided pat-
tern recognition programs for the identification
sciences. Forensic scientists might also adopt
protocols, such as blind examinations in com-
bination with realistic samples, that minimize
the risks that their success rates will be inflated
and their conclusions biased by extraneous ev-
idence and assumptions (34). When matches
are identified, forensic scientists in all fields
would compute and report random-match prob-
abilities similar to those used in DNA typing.
These estimatesin combination with error
rate estimates provided by mandatory, well-
constructed proficiency testswould inform
fact-finders about the probative value of the
evidentiary match.
Simply put, we envision a paradigm shift
in the traditional forensic identification sci-
ences in which untested assumptions and semi-
informed guesswork are replaced by a sound
scientific foundation and justifiable protocols.
Although obstacles exist both inside and out-
side forensic science, the time is ripe for the
traditional forensic sciences to replace anti-
quated assumptions of uniqueness and per-
fection with a more defensible empirical and
probabilistic foundation.
References
1. D. L. Faigman, D. Kaye, M. J. Saks, J. Sanders, Modern
Scientific Evidence: The Law and Science of Expert
Testimony (Thompson-West, St. Paul, MN, ed. 2, 2002).
2. R. Epstein, S. Cal. Law Rev. 75, 605 (2002).
3. M. J. Saks, Hastings Law J. 49, 1069 (1998).
4. National Research Council, Forensic Analysis: Weighing
Bullet Lead Evidence (National Academy Press, Wash-
ington, DC, 2004).
5. I. Pretty, D. Sweet, Sci. Justice 41, 85 (2001).
6. C. M. Cooley, Stanford Law Policy Rev. 15, 381 (2004).
7. D. A. Stoney, J. Forensic Sci. Soc. 31, 197 (1991).
8. J. W. Osterburg, J. Crim. Law Criminol. 60, 97 (1969).
9. T. S. Kuhn, The Structure of Scientific Revolutions
(Univ. of Chicago Press, Chicago, 1962).
10. B. Scheck, P. Neufeld, J. Dwyer, Actual Innocence
(Doubleday, New York, 2000).
11. J. Peterson, Forensic Science and the Courts (National
Institute of Justice, Washington, DC, January 1986).
12. P. C. Giannelli, Issues Sci. Tech. 20, 55 (fall 2003).
13. P. C. Giannelli, Va. J. Soc. Policy Law 4, 439 (1997).
14. K. G. Furton, Y. L. Hsu, M. D. Cole, J. Forensic Sci. 44,
128 (1999).
15. A. A. Moenssens, J. Crim. Law Criminol. 84, 1 (1993).
16. Committee on DNA Technology in Forensic Science,
DNA Technology in Forensic Science (National Re-
search Council, Washington, DC, 1992).
17. R. C. Lewontin, D. L. Hartl, Science 254, 1745 (1991).
18. Technical Working Group on DNA Analysis Methods,
Crime Lab. Digest 18, 44 (1991).
19. K. Roeder, Stat. Sci. 9, 263 (1994).
20. J. J. Koehler, A. Chia, J. S. Lindsey, Jurimetrics J. 35,
201 (1995).
21. B. Budowle, E. S. Lander, Nature 371, 735 (1994).
22. J. J. Koehler, Jurimetrics J. 37, 425 (1997).
23. M. J. Saks, J. J. Koehler, Cardozo Law Rev. 13, 361 (1991).
24. Brief Amicus Curiae of Americans for Effective Law En-
forcement et al., submitted in Kumho Tire v. Carmichael,
No. 97-1709, in the Supreme Court of the United
States (October Term, 1997).
25. D. Fisher, Hard Evidence (Simon & Schuster, New
York, 1995).
26. D. Heath, H. Bernton, Portland lawyer released in probe
of Spain bombings, Seattle Times (21 May 2004).
27. D. Kahneman, A. Tversky, Psychol. Rev. 80, 237 (1973).
28. J. J. Koehler, Behav. Brain Sci. 19, 1 (1996).
29. M. M. Houck, B. Budowle, J. Forensic Sci. 47, 964 (2002).
30. Collaborative Testing Services Inc., Forensics Testing
Program: Latent Prints Examination, 19932004.
31. D. Grieve, J. Forensic Ident. 46, 524 (1996).
32. S. Cole, J. Crim. Law Criminol., in press (available at
www.seweb.uci.edu/faculty/cole).
33. J. L. Peterson et al., Forensic Evidence and the Police
(National Institute of Justice, Washington, DC, 1984).
34. M. J. Saks, D. M. Risinger, R. Rosenthal, W. Thompson,
Sci. Justice 43, 77 (2003).
10.1126/science.1111565
R E V I E W
www.sciencemag.org SCIENCE VOL 309 5 AUGUST 2005 895
A
single spot of blood on a pink
windowsill will tell investigators
who broke a windowpane, turned a
lock, and kidnapped 2-year-old Molly Evans
from her bedroom in the middle of the night.
An expert witness will testify that the DNA
profile of the blood evidence recovered from
the windowsill was entered into CODIS,
an electronic database of DNA profiles.
1

That process yielded a hit, identifying
the defendant as the most likely source
of the blood inside Mollys room.
But will jurors be able to understand the
experts intricate analysis and use it to reach
a verdict? And whatif anysteps can be
taken to increase jurors comprehension of
complex DNA evidence?
Questions such as these prompted an
NIJ-funded study on the impact of jury trial
innovations upon mock jurors understanding
of contested mitochondrial DNA (mtDNA)
evidence. (See How Mitochondrial DNA
Compares to Nuclear DNA.) By examining
how jurors in different experimental condi-
tions performed on a Juror Comprehension
Scale both before and after deliberations,
researchers were able to assess whether
four specific innovations improved jurors
understanding of this complex evidence
and identify which innovations worked best.
Trial Innovations Tested
The four innovations used in the experiment
were:
Juror note taking. Mock jurors were given
a steno pad and pen for note taking and
were told that their notes would be avail-
able to them during deliberations.
Questions by jurors. Mock jurors could
submit questions to the presiding judge,
who obtained answers from an offsite
DNA expert.
Can Jury Trial Innovations Improve
Juror Understanding of DNA Evidence?
by B. Michael Dann, Valerie P. Hans, and David H. Kaye
About the Authors
The Honorable B. Michael Dann (Ret.) was a Visiting Fellow
at the National Institute of Justice in 2004; Valerie P. Hans is a
Professor at Cornell Law School; and David H. Kaye is a Regents
Professor at Arizona State University College of Law. University of
Delaware doctoral candidates Stephanie Albertson and Erin Farley
assisted with the research project.
N I J J O U R N A L / I S S U E N O . 2 5 5
3
Mitochondrial DNA (mtDNA) checklists.
This innovation guided jurors through
complex mtDNA evidence by asking
them a series of questions. (See mtDNA
Evidence Checklist.)
Multipurpose juror notebooks. Mock

jurors were given notebooks containing
paper, copies of the two experts slides,
the mtDNA checklist, a glossary of DNA
terms used in the case, and a witness list.
Selecting the Mock Jury
Jurors were selected from jury-eligible
adults called to jury duty in the Superior
Court of New Castle County, Delaware.
Jurors were randomly assigned to 60 eight-
person juries. Each juror filled out an initial
questionnaire that queried his or her views
on the reliability of certain types of scientific
testimony and about science in general.
(See Mock Jurors Attitudes About
Science and DNA.)
Researchers then divided the juries into
groups of 10 and subjected each group
to one of the following conditions:
2

The Mock Trial
The jurors then watched a videotape of an
armed robbery trial. Prosecutors presented
the testimony of bank employees who could
not make a positive identification because
the robber wore a blue hooded sweatshirt
and a partial mask. However, one teller testi-
fied that she saw an unmistakable inch-long,
horizontal scar on the suspects cheek when
he wiped his face with his gloved hand.
Police searched the crime scene immedi-
ately after the robbery and recovered a blue
sweatshirt, a glove, and a small amount of
cash, including some of the bait money.
3

Two human hairs recovered from the
sweatshirt hood were analyzed and found
to match the defendants mtDNA. No
other physical evidence was recovered.
Jurors learned that an anonymous caller
told police the defendant had robbed the
bank. Testimony established that the defen-
dant owned a blue hooded sweatshirt, had
a scar on his cheek, and had recently been
seen flashing a large roll of cash.
The defendant testified in his own defense
and denied committing the robbery. He told
a detective that he had never been in that
bank and that he was at work when the
robbery occurred. He claimed that the
excess cash was from a friends recent
repayment of a loan.
In an attempt to dispute the prosecutions
mtDNA evidence, the defense introduced
evidence that the defendants wayward
half-brother on his fathers side lived in
town at the time of the robbery. This fact,
however, would have been irrelevant to
any juror who understood that mtDNA is
inherited only through the mothers lineage.
Researchers made the rest of the circum-
stantial evidence purposefully ambiguous
so that jurors would feel compelled to
consider the mtDNA identification evidence
Experimental
Condition
Jury Innovations
Condition 1 No innovations (control)
Condition 2 Note taking
Condition 3 Question asking and
note taking
Condition 4 DNA checklist and note
taking
Condition 5 Juror notebook and note
taking
Condition 6 All innovations (note
taking, question asking,
DNA checklist, and juror
notebook)
HOW MITOCHONDRIAL DNA COMPARES TO NUCLEAR DNA
Nuclear DNA, or nDNA, is the genetic material inherited from both
parents (one-half from the mother and one-half from the father). It
is found in the nucleus of each cell and is unique to each individual
(except in cases of identical twins). Nuclear DNA is a powerful
identifier and has been used for forensic purposes for decades.
Mitochondrial DNA (mtDNA)which is found in the mitochondria of
a cell, outside of the nucleus and separate from nDNAis inherited
solely from the mother and is not unique. Everyone in the same
maternal line, for generations, will have the same mtDNA. Its use
as a forensic tool, in narrowing the pool of possible donors of a
sample, is a more recent development.
4
The multipurpose
notebook was the
most popular
innovation:
92 percent of the
jurors said that the
notebooksin
particular, the
experts slides
helped them
to remember
and understand
the case.
MOCK JURORS ATTITUDES ABOUT SCIENCE AND DNA
Researchers found that the demographic
profile (sex, race, and age) of the 480
mock jurors bore striking similarities to
those of the entire pool of jury-eligible
adults. Most mock jurors had some
science or mathematics courses: on
average, most had more than nine such
courses in high school or college. About
half had some job experience involving
science or math.
Almost all (89 percent) of the mock
jurors held positive attitudes about
science. However, a significant minority
expressed reservations about science.
Negative attitudes about the role
of science in their lives were strongly
correlated with the level of formal
education; jurors with less education
tended to express more negative
views.
Before the videotape was presented,
researchers solicited jurors views about
DNA. Two-thirds of mock jurors agreed
that DNA evidence was extremely
reliable. Although half of the partici-
pants had heard about mtDNA before
this trial, most said they had heard only
a small amount about it.
After the trial, however, almost all of
the jurors had a basic understanding
of the mtDNA evidence. Solid majorities
of jurors (ranging from 66 to 90 percent)
exhibited correct understandings of
most of the core knowledge items
about mtDNAe.g., where the mito-
chondria are found in the cell, how
samples are compared and matches
declared, and how mtDNA compares
to nuclear DNA.
Ninety percent of jurors correctly under-
stood that unlike nuclear DNA, mtDNA
is inherited solely from ones mother.
Those jurors rejected the defense sug-
gestion that the crime could have been
committed by the defendants wayward
half-brother on his fathers side, noting
that the relationship would not account
for the presence of the defendants
mtDNA in the hair strands recovered
from the hooded sweatshirt.
On the other hand, some of the
participants showed a susceptibility
to adversarial exaggerations and
misstatements about the scientific
evidence:
A number of jurors were persuaded
by the prosecutors argument that
the likelihood of the defendants inno-
cence was equal to the percentage
of Caucasian males who could not be
excluded as possible contributors of
DNA found on the hooded sweatshirt.
Because the prosecutions expert
estimated that 99.98 percent of
Caucasian males would be excluded
as contributors, prosecutors argued
that there was only a .02 percent
possibility that the defendant did not
commit the crime. This rationale erro-
neously hinged the defendants guilt
on one piece of evidencehair found
on a sweatshirt at the scenewhile
ignoring other circumstantial evidence
that was not directly incriminating.
Some jurors also agreed with the
defense attorneys questionable claim
that the mtDNA evidence was entirely
worthless because people other than
the defendant could have contributed
the hairs.
One-quarter of the mock jurors

thought that sample contamination
was likely despite the absence of
evidence or argument from either side
suggesting contamination of the hair
samples or the mtDNA.
As anticipated, the amount of formal
education, the number of courses in
science and mathematics, and some
job experience involving science and
mathematics positively correlated with
jurors correct understanding of mtDNA.
5
and resolve the issues raised by the
prosecution and defense experts.
Expert Testimony on mtDNA
The prosecutions expert testified that
the mtDNA profiles of hair from the sweat-
shirt and the samples combed from the
defendants head at the time of his police
interview were an exact match. He com-
mented that the profile was rare and had
not been observed in the Federal Bureau
of Investigations (FBIs) mtDNA database
of more than 5,000 samples. He added
that 99.98 percent of all Caucasian males
would be excluded as potential contributors
of the two mtDNA samples. That meant
that in addition to other men in the same
maternal line as the defendant, only 6 males
in a population of 29,000 would have the
same mtDNA profile.
The defense expert agreed that the mtDNA
samples matched, but said that the FBIs
percentage of the population excluded by
the mtDNA evidence was too large because
the FBI failed to properly account for the
possibility of heteroplasmy in human hair.
Heteroplasmy is a condition where some of
a persons mtDNA exhibits a mutation and
thus differs (in at least one base pair) from
the remainder of the persons mtDNA. By
including heteroplasmic individuals as
possible sources of the hairs, the defense
expert reduced the FBIs percentage of
excluded males to 99.80 percent. She
projected that 57 males in the localityas
opposed to the prosecutions estimate of
6could have been the source of the hairs.
After the videotape, jurors completed a
second questionnaire about their uses of
and attitudes toward trial innovations. They
were then allowed to deliberate. Following
the return of a unanimous verdict or the
declaration of a mistrial (hung jury) in
each case, jurors filled out a third and
final questionnaire.
Researchers then coded and analyzed
jurors responses to the questionnaires
and reviewed the jurors written notes,
copies of the checklist, and notebook
materials. Questions posed by jurors
during the trial were also analyzed. All
of the jury deliberations were videotaped,
reviewed, and coded to assess the use
of jury innovations in group deliberations.
Which Innovations Did Jurors Use?
The research showed that jurors used
three of the innovations the mostthe
multipurpose notebook, note taking, and
the mtDNA checklist.
The multipurpose notebook was the most
popular innovation: 92 percent of the jurors
said that the notebooksin particular, the
experts slideshelped them to remember
and understand the case. The second
most used innovation was juror note taking:
88 percent of jurors took notes. Two-thirds
said their notes helped them remember the
evidence. The third most used innovation
was the mtDNA checklist: 85 percent of
jurors allowed to use the checklist said they
reviewed it during deliberations. Most found
that the checklist increased their under-
standing and recall of the evidence. The
least used innovation was jury questioning:
only 22 percent of the jurors allowed to ask
questions actually did.
Which Innovations Enhanced
Juror Understanding?
To see whether innovations improved
juror understanding of mtDNA evidence,
researchers explored how jurors in the
different experimental conditions performed
on Jury Comprehension Scales
4
before
and after their deliberations, controlling
for jurors educational levels.
5
In general,
researchers found that jury deliberations
improved jurors comprehension of mtDNA.
Prior to deliberations, there were no
significant differences in how jurors
who were assigned to the various
conditionsthose who used innovations
and those who did notperformed on the
Juror Comprehension Scale. Even after
deliberations, comparisons of the responses
of jurors given no innovations (control group)
with those who had them still showed no
significant differences in their understanding
of mtDNA evidence.
6
However, when the postdeliberation
responses of jurors allowed to use each
particular innovation were compared with
the responses of jurors not allowed to use
that innovation (both those in the control
group and those assigned another innova-
tion), differences emerged. Under this
analysis, researchers found that jurors
allowed to use juror notebooks performed
significantly better on two aspects of the
comprehension testing (basic and expanded
factual true-false tests) than those not
provided notebooks. Jurors provided with
an mtDNA checklist also performed better
(on an expanded Jury Comprehension Scale)
than those without access to the checklist.
Researchers also examined whether
actual usage of an innovation improved
juror understanding. The results were
mixed. Data showed that jurors who took
advantage of two innovationsnote taking
and question askingdid not have higher
levels of comprehension; however, jurors
who actually used the mtDNA checklist
and the juror notebook significantly outper-
formed jurors who were afforded use of
those innovations but declined to use them.
There was also evidence that use of multiple
innovations improved juror comprehension.
Using the note taking condition as a control,
researchers found that jurors allowed to
take notes and use a juror notebook did
better on the Jury Comprehension Scales
postdeliberation than did those allowed only
to take notes. The same was true for jurors
exposed to all four innovationsthey also
outperformed those jurors who were only
allowed to take notes. Thus, it appears that
additional innovations on top of jury note
taking improves mock jurors comprehen-
sion of scientific mtDNA evidence.
Practical Suggestions for Practitioners
Based on the study, researchers believe
that the use of certain jury innovations has
the potential to improve jurors comprehen-
sion of mtDNA and other scientific evidence.
Methods that provided direct guidance or
additional expert informationsuch as the
mtDNA checklist and the juror notebook
best improved juror understanding. This
suggests that other jury innovations that
mtDNA EVIDENCE CHECKLIST

1. Was the blue hooded sweatshirt found by the police
probably the one worn by the bank robber?
Yes
No Then the FBIs DNA analysis of the hair will
not assist you in identifying the robber.
2. Did the FBI correctly identify the mtDNA sequences of
the suspect (sweatshirt hood) and known (defendants)
samples of hair?
Yes
No The results of the FBIs analysis of the
hairs mtDNA will not assist you in
identifying the robber.
3. Did the FBI correctly conclude that the mtDNA
sequences of the two hair samples matched?
Yes
No The results of the FBIs analysis of the
hairs mtDNA will not assist you in
identifying the robber.
4A. Did the FBI correctly calculate how often the hairs
mtDNA sequence is likely to occur in the Caucasian
population?
Yes 5A. What percent of the Caucasian
population can be excluded as possible
contributors of the mtDNA found on
the sweatshirt hairs?
Answer ________%
No 4B. Did the defendants expert correctly
calculate how often the hairs mtDNA
sequence is likely to occur in the
Caucasian population?
Yes 5B. What percent of the
Caucasian population can
be excluded as possible
contributors of the
mtDNA found on the
sweatshirt hairs?
Answer ________%
No Neither experts testimony
will assist you in identifying
the robber.
6. How many Caucasian males in the Middletown area
could have contributed the hairs found in the sweatshirt
hood? (Check one.)
_____ 6 males (prosecution expert estimate)
_____ 57 males (defense expert estimate)
_____ Other number (your estimate: ____________)
7. How likely is it that the defendant was the source of
the hairs found in the sweatshirt hood? (Check one.)
_____ Extremely likely
_____ Somewhat likely
_____ Dont know
_____ Somewhat unlikely
_____ Extremely unlikely
7
provide a better understanding of expert
evidencesuch as juror tutorials in complex
subjects and court-appointed experts to
discuss the parties often conflicting
scientific evidenceare ripe for evaluation.
The results of the study showed that most
juries are capable of comprehending and
using different forms of DNA evidence at
trial. Nonetheless, researchers acknowl-
edged that some jurors are likely to have
trouble with complex DNA evidence.
Researchers offered five ways to facilitate
juror understanding of DNA evidence:
Distribute juror notebooks that contain
copies of the experts slides, overheads,
and charts; a glossary of technical terms;
a list of the issues presented by the
DNA evidence; and blank paper for
note taking.
Distribute a checklist or inference chart
listing the issues presented by the DNA
evidence and provide a step-by-step
pathway for the jurors resolution of
those issues.
Provide a brief, straightforward explanation
of forensic DNA without burdening jurors
with nonessential technical details about
the analysis. Some deliberating jurors
complained about technical overload
of essentially uncontested matters.
Allay fears of contaminationeven in
cases where there is no evidence it has
occurred. A significant number of jurors
believed sample contamination was a
problem despite the total lack of evidence
or argument by defense counsel to
suggest it occurred.
Encourage jurors to weigh the probative
value of the DNA evidence linking the
defendant to the crime with the value
of other nonscientific evidence. Jurors
attempt to combine both types of informa-
tion to arrive at an opinion regarding guilt,
but are unsure how to do so. Attorneys
and experts should present simple, under-
standable approaches to considering the
value of different types of evidence.
NCJ 215455
For More Information
Dann, B.M., V.P. Hans, and D.H. Kaye,

Testing the Effects of Selected Jury Trial
Innovations on Juror Comprehension of
Contested mtDNA Evidence, final report
submitted to the National Institute of
Justice, Washington, DC: August 2005
(NCJ 211000), available at www.ncjrs.gov/
pdffiles1/nij/grants/211000.pdf.
Notes
1. The Combined DNA Index System (CODIS)
is an electronic database of DNA profiles
administered through the Federal Bureau of
Investigation. The system lets Federal, State,
and local crime labs share and compare DNA
profiles. Through CODIS, investigators match
DNA from crime scenes with convicted
offenders and with other crime scenes
using computer software, just as fingerprints
are matched through automated fingerprint
identification systems. CODIS primarily uses
two indexes: (1) the Convicted Offender Index,
which contains profiles of convicted offend-
ers, and (2) the Forensic Index, which contains
profiles from crime scene evidence. The
strength of CODIS lies in solving cases that
have no suspects. If DNA evidence entered
into CODIS matches someone in the offender
index, a warrant can be obtained authorizing
the collection of a sample from that offender
to confirm the match. If the offenders DNA
is in the Forensic Index, the system allows
investigatorseven in different jurisdictions
to exchange information about their respective
cases.
2. Juror note taking was permitted in all but the
control condition because the more advanced
techniques (such as question asking and juror
notebooks) are unlikely to be offered by a
court without the basic reform of note taking.
3. Bait money is cash that tellers are instructed
to turn over in the event of a robbery. It con-
tains prerecorded serial numbers, enabling
investigators to identify the funds if recovered.
4. Researchers combined eight facts about
mtDNA to develop a Juror Comprehension
Scale that measured jurors understanding
of mtDNA.
5. Researchers also controlled for juror member-
ship on a particular jury by using a nested
analysis. Because mock jurors in the study
deliberated with one another, jurors potentially
influenced one another. A nested analysis
was used because jurors responses post-
deliberation were no longer strictly indepen-
dent observations.
The use of certain
jury innovations
has the potential
to improve jurors
comprehension of
mtDNA and other
scientific evidence.
Forensic science (known in some countries as legal
medicine) is a specialism that aims to help judges and
juries solve legal issues, not only in criminal law but also
in civil cases. The field has great breadth, crossing the
boundaries between biology, chemistry, physics and
mathematics, and including disciplines as varied as
botany and ballistics, and the analysis of fingerprints,
ear-prints, recorded sound and handwriting. Over the
past 20 years, however, one particular biological tool has
revolutionized forensic investigations the analysis of
DNA. As all living things contain DNA, and all DNA
exhibits variability both among and within species, any
biological material associated with a legal case carries in
it information about its source.
In this review, with the twentieth anniversary appr-
oaching of the development of DNA FINGERPRINTING
1,2
the first molecular genetic forensic technique we take

this opportunity to present an overview of the field.
DNA analysis has evolved to become an indispensable
and routine part of modern forensic casework, employ-
ing extremely sensitive PCR-based techniques to analyse
biological material. Suspects can be linked to crime
scenes, or one crime scene to another, using DNA evi-
dence from as little as the saliva on a cigarette butt, skin
cells on a steering wheel or pet hairs on clothing. Large
DNA databases can be rapidly interrogated for matches
to DNA profiles found at the scene of a crime, or even
partial matches to close relatives of a perpetrator.
Undetected cold cases involving sexual assault can be
solved decades after investigations were begun by
analysing degraded DNA from stored swabs or micro-
scope slides. Victims of mass disasters such as air crashes,
where physical identification might be impossible, can
be identified unambiguously in days. However, as pow-
erful as DNA analysis is, it is far from being the sine qua
non of forensic casework. DNA evidence must always
be considered within the framework of other evidence
of many types, and the role of the forensic geneticist is
not to make presumptions of guilt or innocence, but to
provide unbiased information to judge and jury.
We concentrate here on the analysis of human DNA,
including a discussion of recent massive forensic cases
following wars and disasters. However, we also describe
applications of non-human DNA analysis, in particular
the use of animal and plant DNA-typing and the field of
microbial forensics, which has expanded as a response
to the threat of bioterrorism. Finally, we ask what the
future holds for FORENSIC GENETICS, including a considera-
tion of new technological developments and ethical
issues arising from expanding DNA databases. PATERNITY
TESTING (reviewed in REF. 3) forms part of the field of
forensic genetics and is of great importance in civil and
immigration cases, but owing to space restrictions we
cannot discuss it here.
The evolution of forensic genetics
The aim of the forensic geneticist is one of attribution
to identify with as much certainty as possible the ori-
gin of a biological sample. The amount of variation that
is currently accessible in DNA is extremely informative
ENCODED EVIDENCE: DNA IN
FORENSIC ANALYSIS
Mark A. Jobling* and Peter Gill
Abstract | Sherlock Holmes said it has long been an axiom of mine that the little things are
infinitely the most important, but never imagined that such a little thing, the DNA molecule, could
become perhaps the most powerful single tool in the multifaceted fight against crime. Twenty
years after the development of DNA fingerprinting, forensic DNA analysis is key to the conviction
or exoneration of suspects and the identification of victims of crimes, accidents and disasters,
driving the development of innovative methods in molecular genetics, statistics and the use of
massive intelligence databases.
DNA FINGERPRINTING
Generation of a pattern of
bands, by Southern blotting and
hybridization with a multilocus
probe, which is highly
individual-specific.
FORENSIC GENETICS
The application of genetics for
the resolution of legal cases.
PATERNITY TESTING
Determining whether or not a
particular man is the father of a
child, using genetic analysis. This
generally uses similar autosomal
markers to individual
identification work.
NATURE REVIEWS | GENETICS VOLUME 5 | OCTOBER 2004 | 739
*Department of Genetics,
University of Leicester,
University Road,
Leicester LE1 7RH,
United Kingdom.
Forensic Science Service,

Trident Court,
2920 Solihull Parkway,
Birmingham Business Park,
Birmingham B37 7YN,
United Kingdom.
Correspondence to
M.A.J. or P.G.
e-mails: maj4@le.ac.uk;
Peter.Gill@fss.pnn.police.uk
doi:10.1038/nrg1455
R E V I E WS
2004 Nature PublishingGroup
sequences between different minisatellite loci allowed
probes to detect many independent minisatellites simul-
taneously, yielding the hypervariable multi-band
patterns known as DNA fingerprints
2,5
. Using only a
single probe, the match probability was estimated to
be <3 10
11
and two probes together gave a value of
<5 10
19
(REF. 2) so low that the only individuals
sharing DNA fingerprints are monozygotic twins. At the
same time, a method known as DIFFERENTIAL LYSIS was devel-
oped
5
that selectively enriched the sperm concentration in
vaginal fluid/semen mixtures, thereby avoiding the prob-
lem of the victims DNA (which is in great excess) mask-
ing the rapists. This is the only protocol to have remained
unchanged throughout the past 20 years.
Single-locus probes. Although use of DNA fingerprint-
ing persisted for some years in paternity testing, crimi-
nal casework soon concentrated on the use of specific
cloned minisatellites single-locus probes (SLPs)
that each revealed only a single, highly polymorphic,
restriction fragment length polymorphism, therefore
simplifying interpretation. Typically, four SLPs were
used successively to probe a Southern blot, yielding
eight hypervariable fragments per individual.
It was with SLPs that the first DNA-based criminal
investigation was carried out; this case, culminating in
the conviction of Colin Pitchfork for a double rape and
homicide in Leicestershire in 1986, encapsulated many
of the defining characteristics and virtues of DNA
analysis. First, the two killings, spaced three years apart,
were shown to have been committed by the same indi-
vidual, because SLP profiles (and DNA fingerprints)
from the crime scenes matched. Second, a suspect who
had confessed was excluded because his SLP profile and
that found on the victims did not match, demonstrat-
ing the power of DNA to exonerate innocent people.
Third, the first ever mass screen was organized by the
Forensic Science Service, in which all 500 local men not
and the degree of certainty can be correspondingly high.
However, it was not always this straightforward.
Early markers. The evolution of forensic genetics has
been driven by the analysis of human genetic variation,
beginning more than a century ago with Karl
Landsteiners discovery
4
of the human ABO blood
group polymorphisms and his early realization that this
variation was applicable to solving crimes. The TIMELINE
summarizes the important developments that have
occurred since that time. It is noteworthy that even a
simple genetic system such as ABO can be used to show
conclusively that a sample did not come from a specific
person to prove an exclusion. However, showing that
the sample actually did come from another specific per-
son is more difficult and depends on the degree of vari-
ation revealed by the typing system. Until the 1980s,
serological and protein electrophoretic methods were
used to access diversity in blood groups and polymorphic
proteins, but the main drawback of these markers was
that they tended to rapidly degrade or were compromised
by bacterial enzymes. In addition, they showed relatively
low variability and informativeness; when eight systems
were used together to analyse a bloodstain, the probability
of two unrelated people sharing a combination (the
MATCH PROBABILITY, Pm) was ~0.010.001, but for other
body fluids, such as semen, not all markers were present
and so the match probability was greater. Also, contami-
nation of body fluid from one person by that from
another (a mixed stain), such as in rape cases, was diffi-
cult to resolve because the vaginal cellular component
tended to mask the contribution from the sperm.
DNA fingerprinting. The DNA revolution began in 1984
with the discovery, by Alec Jeffreys in Leicester, UK, of
hypervariable loci known as MINISATELLITES
1
. These were
detected by hybridization of probes to Southern blots of
restriction-enzyme-digested genomic DNA. Shared core
MATCH PROBABILITY
The chance of two unrelated
people sharing a DNA profile.
MINISATELLITES
Loci made up of a number
(~101,000) of tandemly
repeated sequences, each
typically 10100 bp in length.
Usually GC-rich and often
hypervariable.
DIFFERENTIAL LYSIS
A method to enrich for sperm
DNA in a mixture of sperm and
epithelial cells by preferentially
lysing the latter using detergent
and protease, so that sperm
nuclei can be recovered by
centrifugation.
740 | OCTOBER 2004 | VOLUME 5 www.nature.com/reviews/genetics
R E V I E WS
First genetic
polymorphism,
human ABO
blood groups,
discovered by
Landsteiner
4
Locard lays
down principle
that every
contact leaves
a trace
First antibody test
for blood group
(ABO) introduced
and used by Lattes
(1920s1950s) Discovery
and use of other blood
groups and serum
proteins (e.g. MNSs
system, Rhesus, Lewis,
Kell, haptoglobin)
132
(1960s1980s)
Multilocus DNA
fingerprinting developed
by Jeffreys
1,2
; followed by
single-locus profiling (SLP)
First use of DNA in
a criminal case
(identification of
double-murderer
Colin Pitchfork, in
Leicestershire, UK)
Discovery and use
of electrophoretic
variants of red blood
cell enzymes (e.g.
phosphoglucomutase,
glyoxylase)
133
First commercial
forensic PCR kit -
detecting SNPs at
the polymorphic
HLA-DQA1 locus
by dot blot and
oligonucleotide
hybridization
First commercial
forensic STR
profiling kits
developed
First Y-STR described, and
used in casework acquittal
of rape (Germany)
135
Publication of National
Research Council
report The Evaluation
of Forensic DNA
evidence
137
First mass
disaster case
(Waco,
Texas)
95,121
First useful
polymorphic
human STRs
characterized
134
First use of mtDNA
in casework (UK)
Publication of
National Research
Council report DNA
technology in
forensic science
136
UK National DNA
Database established
(STR profiles)
DNA profiling from
touched objects
138
and single cells
139
demonstrated
New York vs.
Castro judgment
on admissability
of DNA evidence
led to strict
quality control
1900 1915 1920 1960 1984 1986 1987 1988 1991 1992 1993 1995 1996 1997
Timeline | Developments in forensic genetics
See also REF. 6 for further general background. HLA, human leukocyte antigen; mtDNA, mitochondrial DNA;
STR, short tandem repeat.
R E V I E WS
PCR-based methods. DNA amplification by PCR pro-
vided an enormous increase in sensitivity, allowing
minute amounts of degraded DNA to be analysed, and
now forms the basis of all forensic DNA typing. Early
PCR-based systems targeted a small number of SNPs
in the HLA-DQA1 GENE
7
. Although these systems were use-
ful when the SLP technology failed, discriminating
power was low and mixtures were difficult to interpret.
Consequently, there was a period when both PCR and
SLP tests were done in parallel. It was the discovery of
SHORT TANDEM REPEATS (STRs), discussed in the next sec-
tion, together with the introduction of automated
sequencing technology, that led to the current power-
ful systems for individual identification. Subsequently,
the use of STRs supplanted both the early PCR and
SLP tests worldwide once their advantages of high dis-
criminating power, sensitivity and ability to resolve
simple mixtures were realized. In addition, the time
needed to carry out an analysis was greatly reduced.
Reduction of costs resulting from partial automation
paved the way for the creation of national STR DNA
databases.
eliminated by a preliminary test for conventional protein
markers were recruited for DNA testing. Pitchfork
showed prescience in realizing the power of DNA analy-
sis: he attempted to evade the screen, but his evasion was
detected, and when his profile was shown to match those
from the crime scenes, he pleaded guilty to the killings.
Methods in forensic genetics must surmount several
hurdles before being applied to casework. First, tech-
niques must be adapted to work on samples that are not
pristine and are often limited in quantity. Second, exten-
sive validation is required to demonstrate robustness,
to pass the test of admissibility in court
6
(admissibility
criteria and legislation differ between countries and even
among states in the United States). Finally, quality man-
agement systems must be implemented once processes
are introduced into casework: external accreditation of
forensic laboratories to internationally recognized stan-
dards (such as ISO17025) is a prerequisite. Because of these
constraints, adoption of new technical developments can
sometimes be slow. For example, SLP technology was still
widely in use long after methods based on the polymerase
chain reaction (PCR) were developed in 1988.
ISO17025
A global standard, established by
the International Organization
for Standardization, for the
technical competence of
calibration and testing
laboratories (see Online links
box).
HLA-DQA1 GENE
A polymorphic gene within the
MHC class II region on
chromosome 6, encoding a
human leukocyte antigen cell-
surface protein.
SHORT TANDEM REPEAT
A DNA sequence containing a
variable number (typically 50)
of tandemly repeated short (26
bp) sequences, such as (GATA)
n
;
forensic STRs are usually
tetranucleotide repeats, which
show little PCR stutter.
Mitochondrion
(100s per cell)
Y chromosome
(one copy per
male cell)
Marker
types
Source of
variation
Advantages Disadvantages
STRs Independent
chromosomal
assortment;
recombination;
mutation
Extremely
high
discrimination
power
Very degraded
DNA difficult
to type
STRs Mutation
only
Male-
specificity;
useful in
male-female
mixtures
Relatively low
discrimination
power; sharing
within patriline;
possible population
structure problems
SNPs,
usually
in control
region
Mutation
only
High copy
number,
therefore
good survival
in old/damaged
samples
Heteroplasmy;
low discrimination
power; sharing
within matriline;
possible population
structure problems
SNPs Independent
chromosomal
assortment;
recombination;
mutation -
but low rate
Usable on
very degraded
DNA
Mostly biallelic,
so relatively low
discrimination
power; mixtures
difficult to resolve
mtDNA (several copies
per mitochondrion)
16,569 bp
N
o
n
-
r
e
c
o
m
b
i
n
i
n
g

r
e
g
i
o
n
Pseudoautosomal
region 1(2.6 Mb)
Pseudoautosomal
region 2 (0.32 Mb)
Autosomes
X
Control region
(1.1 kb)
HVS I
(341 bp)
HVS II
D2S1338
TPOX
D3S1358
HUMFIBRA/
FGA
D5S818
CSF1PO
D8S1179
D7S820
HUMTH01
HUMVWA
D13S317
D19S433 D18S51 D16S539
D21S11
AMELX
sex-test
AMELY-sex-test
N
o
n
-
r
e
c
o
m
b
i
n
i
n
g
e
u
c
h
r
o
m
a
t
i
n

(

2
3

M
b
)
DYS19
DYS393
DYS391
DYS389I,II
DYS390
DYS385
DYS392
H
e
t
e
r
o
-
c
h
r
o
m
a
t
i
n
YCAII
SGM Plus
locus
FBI CODIS
locus
locus used
in both
Y-STR
Nucleus
Blood stain
(268 bp)
Figure 1 | Sources of human genetic variation used in forensic analysis. Further details of the properties of different loci can
be found in the text. Heteroplasmy describes the presence of two or more different mitochondrial DNA sequences in the same cell,
or individual. FBI CODIS, US Federal Bureau of Investigation Combined DNA Index System; HVS, hypervariable site; Mb,
megabase; mtDNA, mitochondrial DNA; SGM, second generation multiplex; STR, short tandem repeat.
R E V I E WS
is mostly owing to active collaboration between interna-
tional groups that are coordinated under various acade-
mic and government-sponsored institutions (TABLE 1).
Recommendations on standard practice, quality issues
and collaborative activities are made at a global level.
Autosomal STR profiling. The first widely used multi-
plex (the quadruplex
8
) consisted of four SIMPLE STRS.
However, because it had a high match probability of ~1
in 10,000, the first criminal cases involving autosomal
STR profiling were reported in conjunction with SLP
profiling. Subsequent addition of two highly variable
COMPLEX STRS decreased the match probability to ~1 in
Current methods in human identification
Human forensic casework is now done using commer-
cially developed autosomal STR multiplexes (single-tube
PCR reactions that amplify multiple loci); other sources
of genetic variation that find more specialized uses are
autosomal SNPs, and markers on the Y chromosome
and mitochondrial DNA (mtDNA) (FIG. 1). Differences
in practice between jurisdictions are considerable owing
to historical, social and legal circumstances. Detailing
these is beyond the scope of this article, so we take a pre-
dominantly UK perspective here. However, despite these
differences, the rapid development and universal accep-
tance of new DNA-based technology in forensic genetics
POPULATION STRUCTURE
The absence of random mating
within a population, leading to
allele frequency differences
among subpopulations.
SIMPLE STRS
Short tandem repeat loci
composed of uninterrupted runs
of a single repeat type.
COMPLEX STRS
Short tandem repeat loci
containing more than one run of
repeats that can be of one or
more repeat type.
Box 1 | Evaluating the weight of DNA evidence
The evidential weight of a match between crime stain profile and suspect is quantified by the match probability (Pm);
the chance of two unrelated people sharing a profile. For independently inherited loci, Pmis calculated by multiplying
the individual allele frequencies in the profile in question (the product rule): the greater the number of loci, and the
greater the heterozygosity of each locus, the lower the value of Pm. However, there are a number of situations in which
Pm can be substantially increased:
if the profile is partial because of degradation, reducing the number of informative loci;
if a suspect and a perpetrator share many alleles by descent (for example, are brothers);
if a suspect and a perpetrator originate from the same subpopulation.
POPULATION STRUCTURE can cause frequencies of alleles (and hence profiles) to vary between subpopulations an issue
that caused great controversy in the application of SLP profiling
90
. The debate was resolved by applying guidelines to
ensure match probabilities quoted in court were conservative (that is, favourable to the defendant). Similar conservatism
is now applied to STR profiles
91
.
Despite the high discriminating power of very low Pmvalues, interpretation in the courtroom has not been without
controversy
92
, and this is because of the way that DNA evidence is sometimes presented. A wellknown example is the
prosecutors fallacy, or fallacy of the transposed conditional: suppose a crime is committed in London (population ~7
million) and a crime-scene profile is obtained that has a Pmof 10
6
. The prosecutor, finding that a defendant matches the
profile, might say: The odds are a million to one in favour of the defendant being guilty. But, given the population size,
~7 people in the city are expected to match the profile, so it can then be argued that the odds are actually 7 to 1 in favour
of innocence; however, this defence fallacy unrealistically assumes that each of the 7 people has equal probability of
guilt, which is untrue as DNA evidence is not used in isolation. This problem of logic can be avoided by an approach
based on a likelihood ratio, using conditional probabilities based on prosecution and defence scenarios: the job of the
court, based on an evaluation of both DNA and non-DNA evidence, is to decide the ultimate issue of guilt or innocence,
given all of the evidence. Under complex scenarios with many different variables, Bayesian networks
93,94
intuitive
graphical means to display hypotheses regarding the probabilistic relationships between variables are a powerful aid
to understanding, although they are not intended to supplant the role of the jury.
Table 1 | International coordinating bodies in forensic genetics
Organization/subgroup Purpose Web site
International Society of Forensic Genetics International organization promoting scientific http://www.isfg.org/
(ISFG) knowledge in forensic genetics
DNA Commission Makes recommendations for use of DNA markers http://www.rechtsmedizin.uni-
European DNA Profiling (EDNAP) group Harmonization of European DNA technologies mainz.de/Remedneu/ednap/ednap.htm
Paternity Testing Workshops
European Network of Forensic Science Mainly represents government institutions; www.enfsi.org/
Institutes (ENFSI) coordinates efforts to develop European DNA
databases
American Academy of Forensic Sciences Academic body for north American forensic http://www.aafs.org/
(AAFS) scientists
Federal Bureau of Investigation (FBI) Responsible for setting standards, training and http://www.fbi.gov/publications.htm
Scientific Working Group DNA Analysis development of the national DNA database
Methods (SWGDAM)
National Institute of Science and Supports the forensic community by organizing http://www.nist.gov/
Technology (NIST) collaborative proficiency exercises
STRBase Database giving characteristics of forensically http://www.cstl.nist.gov/biotech/
useful short tandem repeats (STRs) and SNPs strbase/index.htm
R E V I E WS
used in the United States
11
, but to improve efficiency,
new multiplexes that amplify 16 loci in a single reaction
(including amelogenin) have also been introduced
12
. In
Germany, eight loci are used, including the ACTBP2
locus
13
. International collaborations have recommended
core loci to facilitate international data exchange for
example, a set of seven for common use in Europe
14
.
Detailed information on most forensic STR markers
can be found at STRBase (see Online links box) and
in REF. 15, which also describes the technology of STR
typing.
The match probabilities obtained with STR multi-
plexes are so low that their reciprocals vastly exceed the
entire human population. However, although DNA
profiling is often referred to as individual identification
and assessments can be made whether or not a DNA
profile is unique in an unsampled population
16
, it would
50 million. This second-generation multiplex (SGM)
also included a PCR assay targeted at the XY-homolo-
gous amelogenin genes
9
, thereby revealing the sex of a
sample donor. It became clear that STRs were more
sensitive than other methods and allowed unambigu-
ous assignment of alleles, making the method suitable
for the development of databases. In 2000, an addi-
tional four loci were added to the multiplex, which
was renamed SGM Plus
10
, thereby reducing the match
probability to less than 10
13
. The above account tracks
developments in the United Kingdom; events elsewhere
unfolded differently, and globally there are now a num-
ber of different systems that nonetheless have many loci
in common. The US FBI CODIS (Combined DNA
Index System) contains 13 STRs plus the amelogenin sex
test (FIG. 1), with a match probability lower than that of
the UK system. Typically, two separate multiplexes are
Amelogenin
Amelogenin
D8S1179
D8S1179
D21S11
D18S51
D3S1358
HUMVWA D16S539
D2S1338
HUMTHO1 HUMFIBRA/FGA
D19S433
Amelogenin
D21S11
D8S1179
D18S51
F
l
u
o
r
e
s
c
e
n
t

y
i
e
l
d
Molecular weight (bp)
X Y 13 14 16 18 29 30
15 18 6 9.3 14
15 18 18 19 11 12 16 17
80 90 100 110 120 130 140 150 160 170 180 190 200 210 220 230 240 250 260 270 280 290 300 310 320
D21S11 D18S51
a
b
c
X Y 18 10 28 30 13 12
X Y 12 13 15 28 30 31 32.2 14 13
Figure 2 | Electropherograms illustrating autosomal STR profiles. a | An electropherogram of the second-generation multiplex
SGM Plus profile from a male, including X- and Y-specific amelogenin products of 106 and 112 bp, respectively. Most short tandem
repeats (STRs) are heterozygous and the alleles are evenly balanced. Numbers beneath STR peaks indicate allele sizes in repeat
units. The STR profile is displayed in the green, blue and yellow channels of a four-colour fluorescent system, with the red channel
being used for a size marker (not shown). b | A typical mixture from two individuals (green channel only shown). Mixtures can only be
identified if the alleles of the minor component are above the background noise in an electropherogram (in practice a ratio of ~1:10)
and can usually be resolved by inspection
95
. In this example, the contributions are in even proportions for example, D21S11
shows four alleles where the peaks are approximately equal in height, whereas D18S51 shows two peaks in a 3:1 ratio. The X- and
Y-specific amelogenin peaks are of approximately equal height, indicating that this is a mixture from two males. More complex
mixtures might require interpretative methods based on likelihood ratios
96,97
, calculations based on peak area
98
and consideration of
PCR stutter
99
, and have led to the development of computer programs to deduce potential underlying genotypes
100,101
. c | A profile
(green channel only shown) obtained by low copy number (LCN) testing
102
, a method used in the United Kingdom when little DNA
(typically <100 pg or <17 diploid genomes) is available. The method uses an increased number of PCR cycles (for example, 34,
rather than the usual 28), and leads to marked heterozygote imbalance at some loci (here, D21S11 and D18S51) because of
stochastic variation in amplification. Extreme imbalance is drop-out (failure to amplify) of an allele (not shown). Drop-out is a
stochastic effect of the limited number of template molecules, and therefore no two amplifications of the same extract will behave
the in the same way. The LCN protocol requires duplicate PCR of an extract and only those alleles that are observed in both PCRs
are reported. (See also BOX 2.)
R E V I E WS
account fragment size (accurately measured with
respect to internal standard markers and to an ALLELIC
LADDER
19
used to identify alleles), ELECTROPHEROGRAM peak
height and area and an assessment of HETEROZYGOTE
BALANCE, and include automated checks to interpret arte-
facts such as PCR STUTTER
20
. DNA quality from reference
samples (taken from individuals to create databases
see next section) is predictably good and makes
automation of typing and interpretation relatively
straightforward. In the UK Forensic Science Service, if
two expert systems working on different principles
agree on a profile it is accepted, but there are regular
challenges with blind controls. For casework samples,
preliminary assessment is vital to determine the best
method of processing; but automation is more difficult
because DNA quality and quantity are variable and
DNA mixtures are often encountered, complicating
interpretation. Anomalous profiles can also arise for
biological reasons (such as mutation) and if methods
are used that are sensitive enough to detect single DNA
molecules (low copy number), then laboratory-based
contamination of single or multiple alleles is a strong
possibility and interpretation strategies are needed to
deal with this. These situations are discussed in BOX 2.
STR-based forensic DNA analysis has achieved
worldwide public and professional acceptance as a reli-
able means of individual identification and has had a
major impact on criminal justice systems. The increase
in sensitivity of DNA methods has allowed the reopen-
ing and solving of cold cases and has also led to the
exoneration of prisoners (some of whom were awaiting
execution) convicted through miscarriages of justice.
Most of these cases have been in the United States (see
the Innocence Project web site in the Online links box),
where post-conviction testing attracts federal funding,
although one case in the United Kingdom has been
described
21
.
DNA databases of autosomal STR profiles. As well as
matching a crime-scene profile to that from an appre-
hended suspect, a match (hit) can be made to a DNA
database of offender profiles (an intelligence database),
allowing a new suspect to be investigated. The advances
in automation described above have allowed the estab-
lishment of such databases, which are now in use or
under development in many countries. Criteria for
including a profile in a database vary among countries
(TABLE 2). The largest example, the UK National DNA
Database
22,23
, contains (at July 25th 2004) ~2.5 million
reference profiles (mostly from buccal scrapes) and
~200,000 crime-scene profiles. Since 1995, there have
been more than 550,000 matches between reference
profiles and crime-scenes, and more than 30,000
matches between crime scenes. A new method of using
the database, familial searching, has recently been intro-
duced. In 2003, a lorry driver was killed in Surrey, UK,
by a brick thrown through his windscreen. A DNA pro-
file was obtained from the brick, but had no match in
the database. A geographically restricted search of the
database was carried out for potential close relatives of
the perpetrator, under the assumption that close
be a rare case in which everyone on the planet could be
considered as the pool of potential contributors for a
crime-scene sample
17
. Reporting of DNA evidence in
court takes a conservative approach to the low match
probabilities and commonly uses likelihood-based
methods to take account of the factors applying to a
particular case (BOX 1).
Multiplexes are analysed and typed using automated
sequencing equipment. These are typically multi-chan-
nel capillary electrophoresis systems that are used to
detect fluorescently labelled PCR products (FIG. 2a) and
are combined with robotics and laboratory information
management systems, including bar-coding of samples
to reduce operator errors. This automation reduces cost
and increases throughput. Interpretation (defining
the alleles in a profile) is more difficult to automate.
However, there has been progress in converting tradi-
tional subjective expert opinion into programmable
(heuristic) rules for computer programs (known as
expert systems
18
), generally intended to complement,
rather than replace, the human expert. These take into
ALLELIC LADDER
An accurate marker used to
identify alleles at a particular
STR, generated by PCR
amplification of a series of
sequenced alleles from that STR.
ELECTROPHEROGRAM
The graphical output of
electrophoresis devices in STR
and sequencing analysis,
showing fluorescence intensity as
a function of molecular weight;
peak at a particular wavelength
(colour) corresponds to a
specifically labelled molecule
of a particular size.
HETEROZYGOTE BALANCE
The proportion of the two alleles
of a heterozygote, expressed as
the area of the smaller peak
divided by the area of the larger
peak in an electropherogram.
Box 2 | Dealing with anomalous autosomal STR profiles
As shown in FIGURE 2, several factors can complicate the interpretation of DNA profiles.
These include mixed samples and the availability of only small amounts of DNA, which
(in the UK) is analysed by low-copy number (LCN) methods and can lead to allele
drop-out. LCN analysis also increases the probability of contamination (additional
foreign alleles in the profile), despite stringent precautions to prevent it.
Contamination can be gross and lead to full additional profiles, where typically
the negative control will be affected; these can be searched against forensic staff
elimination databases and stored in a separate database to allow detection and
monitoring of problems such as contamination of plasticware during
manufacture
103,104
. Alternatively, contamination can contribute an extra one or two
alleles per DNA profile (allele DROP-IN
102,104
), where the negative control is usually
unaffected. The probabilities of a match, of drop-out and of contamination (based
on computer simulation analysis of negative control data
104
) can be readily
incorporated into the calculation of the likelihood ratio (LR) when the significance
of the DNA evidence is being assessed
102
; calculations are complex but are aided by a
computer program (LoComatioN)
105
that can also combine the results of several
profiles into a single LR. Here, we no longer think of a DNA profile either matching
or not matching a suspect, because the twin effects of drop-in and drop-out will
alter the suspects apparent profile so that it does not match the crime scene sample,
without indicating an exclusion; rather, the probability of the evidence is lowered.
Anomalous profiles can arise from causes other than mixtures but can readily be
resolved by careful analysis:
Mutations in the PCR primer target region can cause allele drop-out or heterozygote
imbalance
106,107
.
An STR can be duplicated and segregated in a normal Mendelian fashion or a somatic
STR mutation occurring early in development can lead to a three-peak profile for the STR,
which might vary between tissues; examples are on STRBase (seeOnline links and REF. 108).
The STR D21S11 can reveal three alleles in trisomy 21 cases (reviewed in REF. 109).
Discordant results in the amelogenin sex test can be observed in rare individuals,
including cases of sex-reversal (XX males and XY females) and males
110112
carrying
deletions that remove AMELY.
A true mixed profile can also originate from a single individual. Analysis of DNA from
people who had undergone successful bone marrow transplants 5 years previously
113
,
showed a mixed profile in buccal and fingernail samples; in blood, the recipients own
profile had been completely replaced by that of the donor, whereas in hair, the recipients
profile remained unmixed.
R E V I E WS
international. However, although large databases with
permissive entry criteria can be powerful, they also raise
ethical questions (BOX 3).
Autosomal SNP typing. Compared to STRs, SNPs have
much lower heterozygosities (a per-SNP maximum of
0.5) and so ~50 SNPs are required to approach the low
match probability of an STR profile
25
(see BOX 1). In
addition, mixtures are especially difficult to resolve for
relatives are more likely to share alleles than unrelated
people (50% for brothers). This highlighted 150 candi-
dates, leading to the identification of a suspect whose
profile matched that on the brick. He was convicted of
manslaughter
24
.
Legal differences between countries make such
speculative searches impossible in some jurisdictions
and can also complicate the international exchange
of data when crime is itself becoming increasingly
PCR STUTTER
A PCR artefact in which, as well
as a band of the expected size, an
additional band is seen which is
typically one repeat unit smaller,
resulting from slippage synthesis
errors by the PCR polymerase
DROP-IN
Addition of (typically) one or
two alleles to a DNA profile,
owing to contamination.
Table 2 | Characteristics of some national DNA databases
Country (Year Reference Crime-scene Suspect to Scene to Entry criteria Entry criteria for Removal criteria
established) profile size sample size scene hits scene hits for suspects convicted offenders
UK 2.5 million 200,000 550,000 30,000 Any recordable Entered as suspect Never removed,
(1995) offence* including suspects
USA 1.52 million 67,000 Figure Figure No suspects entered, Depends on Depends on state
(1994) unavailable unavailable but under revision state law law
Germany 286,840 54,570 13,700 5,500 Offence leading to After court decision After acquittal or 510
(1998) >1 yr in prison years after conviction,
if prognosis is good
Austria 64,740 11,460 3,200 1,350 Any recordable Entered as suspect Only after acquittal
(1997) offence*
New Zealand 44,000 8,000 4,000 2,500 No suspects entered A relevant offence Never removed, unless
(1996) (including 7 yr in prison) conviction quashed
Switzerland 42,530 7,240 4,840 5,540 Any recordable Entered as suspect After acquittal or 530
(2000) offence* years after conviction
France 14,490 1,080 50 70 No suspects entered Sexual assault and 40 years after
(2001) serious crime conviction
Finland 8,170 5,450 2,080 780 Offence leading to Entered as suspect Only after acquittal
(1999) >1 yr in prison
Slovenia 4,820 2,360 370 80 Any recordable Entered as suspect Depends on severity
(1998) offence* of crime
Netherlands 4,260 13,700 2,520 4,260 No suspects entered
Offence leading to >4 yr 2030 years after

(1997) in prison conviction
Sweden 3,980 9,860 2,500 4,750 No suspects entered Offence leading to >2 yr 10 years after release
(2000) in prison from prison
*That leads to a term of imprisonment.
Except when the suspects DNA is tested for the case. Adapted from REF. 140, with additional information from Peter Schneider and
Jill Vintiner (personal communications). See also BOX 3.
Box 3 | DNA databases: ethical issues
DNA databases are seen by some as without fault
114
, but they have not been without their critics
115,116
. Any criminal
whose profile is in a database risks detection in further criminal activity the probability of identifying a suspect when a
crime-scene profile is checked against the UK database is >40%
23
. Culprits are apprehended more quickly and criminals
might be deterred from future offending, but it is unclear whether overall crime rates are reduced. The UK database is
projected to reach 5 million samples
22
, which is ~10% of the population and >30% of 1050-year-old men (those most
likely to offend
30
). This enormous size is a result of the relatively liberal criteria for database entry: most samples are taken
(with or without consent) from individuals arrested for offences that could lead to a prison sentence, whereas in some other
jurisdictions only more serious offences are considered and a specific degree of connection (decided by a court) between the
suspect and the offence is necessary before DNA can be sampled (TABLE 2). One justification given for the United Kingdom
practice is that minor criminals might also be perpetrators of more serious crimes one example is the arrest and
subsequent conviction, in 2001, of a man for a murder committed in 1968, triggered by a match between crime-scene DNA
evidence and a sample taken from the culprit in connection with a motoring offence
114
. Recent legislative changes allow
retention of profiles from exonerated suspects and from individuals profiled during mass screens (albeit with consent).
The use of any database involves a balance between the rights of the individual and the interests of the state; this differs
from country to country, and some believe it is tipped too far towards the state in systems like that of the UK
117
. However, the
discoverer of DNA fingerprinting, Alec Jeffreys
118
, has argued for the UK database to be extended to the entire population,
under the auspices of an independent authority, arguing that the current database is discriminatory because some groups in
the population are probably overrepresented among suspects. James Watson
119
has called for a global database to fight crime
and terrorism an enormously costly and complex endeavour, raising serious issues in ethics and law
117
.
Retention of samples for possible retesting with future technologies (as is done in the UK) is also considered
controversial by some, who argue that it could reveal private genetic information.
R E V I E WS
malefemale body-fluid mixtures where conventional
methods fail to resolve autosomal profiles, Y-STR typing
can give specific information about the male compo-
nent. Although differential lysis often allows autosomal
profiling of a rapist, the vasectomized or naturally
azoospermic rapist leaves no sperm; in such cases
31
,
Y-specific profiling is effective, even in the presence of
a 4,000-fold excess of female DNA
32
. In multiple rape
it might be possible to gain information about the
number of assailants.
There are 219 known useful STRs on the Y chromo-
some
33
, but a set of 9 or 11 loci is commonly typed in
casework, and there is a large collaborative quality-con-
trolled online population database of more than 24,000
9-locus profiles from 200 populations
29,34
. Clearly, the
product rule (see BOX 1) for independently segregating
autosomal STRs cannot be applied to markers on the
non-recombining Y chromosome and haplotype fre-
quencies are instead often determined simply by
counting or by more sophisticated BAYESIAN methods
35
.
Excluding STR mutation, all patrilineal relatives
(brothers, father, sons, paternal uncles and so on) of a
suspect will share his Y haplotype and this needs to
be considered when assessing the strength of the evi-
dence
36
. Furthermore, the potential association of sur-
names inherited through the paternal line
37
with
Y haplotypes has led to suggestions that surname pre-
diction from haplotype might be possible. However, the
complexity of the relationship
38
probably precludes this
approach as an absolute determinant, although it might
be powerful if used in the context of a Bayesian method
binary markers, although this limitation might be
overcome by targeting rare tri-allelic SNPs
26
.
The practical advantage of SNP typing is that DNA
template size can in principle be only as large as a pair of
specific primers; ~50 bp. This is considerably smaller
than the ~300 bp needed for successful STR profiling
(although special STR multiplexes have been developed
that use particularly small amplicons
27
) and makes SNPs
of interest for the analysis of severely degraded material.
The technical challenges of the World Trade Center dis-
aster (see BOX 4) have led to the application of forensic
SNP typing. The European Network of Forensic Science
Institutes (ENFSI) and the US FBI Scientific Working
Group on DNA Analysis Methods (SWGDAM) (TABLE 1)
working groups are assessing potentially useful multi-
plexes and will make recommendations for global stan-
dardization
28
, although it is hard to imagine that SNP
profiling will replace STR-based systems.
Y-chromosomal analysis. Autosomal STR profiles owe
their variability to three processes: independent chro-
mosomal reassortment, recombination and mutation.
On the Y chromosome, mutation alone functions to
diversify STR HAPLOTYPES. These haplotypes are therefore
less diverse than autosomal profiles (genotypes) con-
taining an equivalent number of markers, leading to rel-
atively high average-match probabilities of ~0.003 for
11 Y-STRs
29
. However, Y chromosomes have one crucial
forensically useful property: they are confined to males.
As most serious offences are committed by men
30
, we
expect to find their Y chromosomes at crime scenes; in
HAPLOTYPE
The combination of allelic states
of a set of polymorphic markers
lying on the same DNA
molecule, such as the Y
chromosome or mtDNA
BAYESIAN
Statistical method, based on
Bayes theorem, that allows
inferences to be drawn from
both the data themselves and
any prior information.
Box 4 | The challenge of large human identification cases
DNA analysis has a key role in the identification of victims of accidents, disasters and wars, therefore aiding
emotional closure for bereaved relatives. Reference material often comes from these relatives
3
for example, an
approach used early on in the identification of the skeletal remains of murder victim Karen Price compared STR
alleles with her presumptive parents
120
.
An early example of a mass identification case followed the Waco disaster
95,121
in 1993, in which more than 70 occupants of
a heavily fortified compound in Waco, Texas, perished in a fire following a siege by US law enforcement agencies. About 40
bodies were unidentifiable by conventional means and, of these, 26 could be identified by quadruplex STR profiling.
Reference samples came from living or dead (but positively identified) relatives. For instance, given the alleles present in
parents, the possible genotypes of children could be sought among profiles of victims (taking account of the possibility of
STR mutations between the generations), and the weight of identification evidence considered on a likelihood basis (BOX 1).
Air crashes, such as the August 1996 Spitzbergen disaster
122
or the September 1998 Swissair flight 111 air crash
123
,
present problems because of the level of damage to bodies from fragmentation and burning. However, given an accurate
list of passenger and crew, the site can be considered a closed scene, where there are no unknown victims; families are
often among the dead, so allele sharing is expected. Reference profiles can be obtained from personal effects of victims
such as clothing or toothbrushes
123
, and samples provided by relatives, leading to complete identification.
Without doubt the largest mass identification cases are those arising from wars and genocides, such as that in the
former Yugoslavia, where >30,000 people went missing; in Bosnia-Herzegovina alone >10,000 bodies required
identification (reviewed in REF. 124). On a comparable scale, and with unique challenges, has been the effort to identify the
remains of the estimated 2,819 people who died in the World Trade Center terrorist attacks of September 11
th
, 2001
(reviewed in REF. 125). Physical identification was impossible for most victims and mixed profiles were common
126
.
Technical developments during the identification process have included:
new computer programs to deal with large and complex kinship calculations, involving reference samples from ~6,000
relatives and ~5,000 personal effects
127,128
;
novel DNA extraction procedures
129
;
redesigned STR multiplexes (miniSTRs, based on shorter DNA amplicons
27
) and autosomal SNP multiplexes to allow
analysis of severely degraded DNA
125
.
Despite these efforts, the remains of ~1,000 people might never be identified
125
.
R E V I E WS
basis of highly improbable sequences that are detectable
by phylogenetic analysis
50
.
Heteroplasmy can lead to different sequences being
found between hairs or tissues in a single individual,
and even along the length of a single hair shaft
51
.
Mutation, which distinguishes heteroplasmic types, is
particularly common at some sites (hot spots), but this
can be built into the interpretation using a likelihood
ratio approach
52
. Shared heteroplasmy between two
samples can actually increase the strength of evidence,
as was the case in confirming the matrilineal relation-
ship between the putative Tsar Nicholas II and his
brother Georgij Romanov
53
(BOX 5).
Putting face and place to a DNA profile
When a profile from a crime scene does not find a hit
in an intelligence database, any information that can
be deduced from the DNA about the donor is useful.
A basic piece of information, sex, has already been men-
tioned, but two other areas, population of origin and
phenotypic features, have also been investigated and used
to aid criminal investigations.
Deducing population of origin. Most (~85%; REFS 54,55)
genetic variation is found within human populations.
Nonetheless, individuals from different populations are,
on average, slightly more different from each other than
are individuals from the same population, and this
allows sets of markers to be used to predict population
of origin (reviewed in REF. 56). Similar methods might be
applicable to the analysis of a crime-scene sample.
Forensic STR profiles are very variable among
individuals and so show low inter-population vari-
ance (F
ST
). They are therefore not ideal for predicting
population of origin. The ability of SGM Plus profiles
to classify individuals into one of five police-defined
ethnic groups has been assessed
57
, and showed, for
example, that 67% of profiles known to be from
of analysis. On a larger scale, the Y chromosome shows
particularly strong population structure
39
and the avail-
ability of local population databases is essential. These
difficulties can complicate the evaluation of match sig-
nificance, but exclusion of a suspect remains straightfor-
ward. The use of Y-chromosome analysis will increase,
particularly in rape casework, aided by the availability
of standardized commercial Y-STR
40
and Y-SNP kits
(reviewed in REF. 41).
Mitochondrial DNA. mtDNA shares many of the the-
oretical disadvantages of the Y chromosome: it is
non-recombining, so markers (almost all SNPs, but
including length variation in a run of C nucleotides) do
not segregate independently, thereby reducing diversity;
it is uniparentally inherited (through the mother), so all
members of a matriline share a haplotype; and it shows
marked population structure
42
. Furthermore, there is
the complication of heteroplasmy.
The advantage of mtDNA lies in its copy number,
which is between ~200 and 1,700 per cell (reviewed in
REF. 43); this means that it has a greater probability of
survival than nuclear DNA does. Forensic applica-
tions
43,44
include analysis of samples that are old or
severely damaged, or low in DNA (such as hair shafts),
and include historical criminal cases (see BOX 5).
The normal practice is to sequence two segments of the
CONTROL REGION that are particularly polymorphic,
known as hypervariable segments I and II (HVSI,
HVSII). SNPs outside the hypervariable segments will
increase the power of mtDNA typing
45,46
.
Rather than considering the average-match proba-
bility (which is high
47
, at ~0.0050.025), match sig-
nificance is usually evaluated by the counting
method how many times a specific sequence has
been observed in a population database
48,49
, with a
correction for sampling error. There has been criti-
cism of the quality of some forensic datasets, on the
CONTROL REGION
Part of mitochondrial DNA that
is non-coding and therefore
more able to accumulate
variation than the rest of the
molecule.
Box 5 | Forensic DNA analysis of historical samples the case of the Russian royal family
The first DNA-based historical criminal investigation was carried out in 1994 when bones purported to be those of the
Russian royal family (the Romanovs), executed by a Bolshevik firing squad in 1918, were analysed using a combination
of mitochondrial DNA (mtDNA) sequencing, sex typing, STR analysis and PCR cloning
130
.
The samples were more than 70 years old, yet yielded autosomal STR profiles consistent with the presence of a family
group and mtDNA sequences matching reference sequences from living matrilineal relatives: the sequence obtained
from Prince Philip, Duke of Edinburgh, matched those of the putative Tsarina and her children, whereas those from the
Duke of Fife and Princess Xenia Cheremeteff-Sfiri matched that of the putative Tsar Nicholas II, except for one
discrepant base. After PCR cloning, the putative Tsars mtDNA was shown to comprise two different molecules (one of
which matched the living reference samples) and was concluded to be an example of heteroplasmy, thought at the time
to be rare; this led to speculation about the reliability of the results. However, when an independent analysis was carried
out on the remains of the Tsars brother, Georgij Romanov, by the Armed Forces DNA Identification Laboratory in
Rockville, Maryland
53
, it was discovered that he shared a heteroplasmy at the same position in the mtDNA molecule; this
effectively dispelled any lingering doubts. After a consideration of all of the DNA and non-DNA (anthropological)
evidence, the Russian authorities pronounced the remains to be those of the Romanovs.
The remains of one of the Tsars daughters, Anastasia, were absent from the grave, and controversy surrounded the
claim that she escaped execution and survived, under the identity of Anna Anderson. STR analysis
131
of 20-year-old
paraffin wax embedded samples from Anderson was inconsistent with her being a daughter of the Tsar and Tsarina.
However, the mtDNA sequences matched those of Carl Maucher, a putative maternal relative of a woman named
Franzisca Schankowska. The mtDNA results were confirmed by an independent group from Penn State University, who
concurrently analysed hair shafts purported to have come from Anna Anderson
131
.
R E V I E WS
colour
65
, for which the P gene, the product of which is
involved in melanin production, is a candidate
66
. Two
amino-acid substitutions in the gene are associated with
blue or grey eyes
66
. A broader association study includ-
ing SNPs in several candidate genes
67
has identified 61
SNPs that explain 15% of the variation in eye colour in a
sample, but probably do not provide useful predictive
testing. Work on these and other phenotypes will proba-
bly increase in the future. However, the complexity of
these quantitative traits, coupled with variability intro-
duced by environmental and nutritional differences,
means that even if the genes influencing them were iden-
tified there is no guarantee that simple deterministic tests
would emerge.
Non-human species in forensic genetics
Forensic analysis of animal DNA has been used both
when animal material (usually pet hairs) is found at
crime scenes, and in investigations of the illegal trade in
endangered species. The best-known example of the
former was the matching, using 10 feline-specific dinu-
cleotide STRs, of cat hairs on a bloodstained leather
jacket with a pet cat, known as Snowball, who lived with
the suspect in a murder case
68
. More recently, a com-
mercial kit containing 11 tetranucleotide STRs has been
produced for the individual identification of cats. Work
on canine identification is mostly based on STRs devel-
oped for parentage testing
69
, but also includes mtDNA
profiling
70
. In a recent case, the conviction of a man for
the murder of a seven-year-old girl in California was
supported by mtDNA analysis of dog hairs that
matched a pet belonging to the victim.
In the endangered species field, species-specific
methods target the gene that encodes cytochrome b of
mtDNA
71,72
; examples include tests for tiger-bone DNA
in traditional Chinese medicines (all of which proved to
be cow or pig
73
), and also for rhinoceros horn
74
.
As with animal material, plant material can be asso-
ciated with a crime scene and provide vital evidence.
When morphology is uninformative, DNA could, in
principle, offer species identification or a link to a spe-
cific place. However, in the analysis of plant DNA there
is no easy equivalent of the widely studied animal
mtDNA sequences (although regions of the chloro-
plast genome and the nuclear ribosomal RNA loci
seem promising) and STRs in most species are poorly
characterized. PCR-based fingerprinting methods such
as RANDOM AMPLIFIED POLYMORPHIC DNA (RAPD) can allow
identification of plant strains and have been used in
the analysis of mosses in a murder case
75
, and in civil
disputes over the identity of commercially valuable
cultivars of strawberry
76
and chilli
77
. A species-specific
PCR assay is available for Cannabis sativa
78
and the iso-
lation of a hexanucleotide STR from the same species
provided a marker with some potential to identify the
source of cannabis samples
79
.
Microorganisms can be sources of evidence in situa-
tions such as foodstuff contamination and medical
negligence cases involving infections, such as HIV trans-
mission
80
. However, in October 2001, at least five people
died in the United States from inhalation anthrax after
African-Caribbeans were classified correctly, whereas
the remainder were wrongly assigned to other ethnic
groups. Despite the misclassification, prediction is
useful if it reduces the number of suspect investiga-
tions carried out before the actual perpetrator is
reached. Policemen are not anthropologists and one
problem with interpreting these studies is the over-
simplified way in which populations are defined. The
haploid Y chromosome and mtDNA show strong geo-
graphic differentiation because their small EFFECTIVE
POPULATION SIZE (one quarter of that of any autosome)
leads to enhanced GENETIC DRIFT. Mating practices
might also contribute to inter-population differences.
These markers therefore contain information on pop-
ulation of origin, but, owing to ADMIXTURE, can give
misleading results.
Markers with greater power have emerged from stud-
ies of admixed populations for epidemiological purposes
or for mapping disease genes by LINKAGE DISEQUILIBRIUM.
Autosomal binary or STR loci have been identified
that show large allele frequency differences (3050%)
between parental population groups
58,59
. Multilocus
genotypes based on such ANCESTRY INFORMATIVE MARKERS
(AIMS) can be analysed using model-based cluster-
ing algorithms, yielding individual proportions of
ancestry from a number of populations. Although
forensic evaluation has not yet been carried out, tests
using 175 AIMs are already available commercially
for forensic applications
60
; their use will probably
increase, although it might be limited in admixed
populations.
Phenotypic information. A strong prediction of popula-
tion of origin might indicate some aspects of pheno-
type, such as skin colour. However, direct genetic tests
would be more useful. Many human phenotypes (for
example, stature, facial features and pigmentation) have
a strong genetic component.
The only relevant trait that has undergone serious
investigation is pigmentation. However, although there
are many human genes that when mutated are known
to cause abnormal pigmentation such as albinism
61
,
only a minority appear to influence normal variation.
The best studied is the melanocortin 1 receptor (MC1R)
gene, the gene product of which lies in the cell membrane
of the MELANOCYTE. Binding of -melanocyte stimulating
factor to the receptor leads to production of black/
brown pigments, whereas in the absence of a signal
through MC1R, red/yellow pigments predominate. The
MC1R gene has more than 30 known variant alleles
involving amino-acid substitutions, three of which are
associated with red hair, fair skin and freckling
62,63
.
Population studies
64
show that homozygosity or com-
pound heterozygosity for such a variant gives a >90%
probability of having red hair. This test is therefore use-
ful as an investigative tool in populations such as that of
the United Kingdom where red hair is found at an
appreciable frequency.
Other candidate pigmentation genes have been
investigated, but with less success. Linkage analysis has
identified a locus on chromosome 15 that influences eye
EFFECTIVE POPULATION SIZE
The size of an idealized
population that has the same
properties with respect to
genetic drift as does the actual
population in question.
GENETIC DRIFT
The stochastic fluctuation of
allele frequencies in a population
owing to chance variations in the
contribution of each individual
to the next generation.
ADMIXTURE
The formation of a hybrid
population through the mixing
of two ancestral populations.
LINKAGE DISEQUILIBRIUM
MAPPING
Analysing single nucleotide
polymorphism alleles in
population-based studies to
identify loci that are associated
with a particular disease or
phenotype.
ANCESTRY INFORMATIVE
MARKERS
Markers showing marked allele
frequency differences between
ancestral populations, useful for
determining the probable
ancestry of an individual.
MELANOCYTE
The specialized cell type, lying at
the boundary between the
dermis and epidermis, in which
the pigment melanin is
synthesized.
RANDOM AMPLIFIED
POLYMORPHIC DNA
Polymorphic markers generated
by using short (812 bases long)
primers to amplify random
fragments of DNA.
R E V I E WS
such devices are validated for forensic use. Methods of
whole-genome amplification (in particular, MULTIPLE
DISPLACEMENT AMPLIFICATION
84
) have potential forensic value
when the amount of template DNA is extremely small.
However, allele DROP-OUT and imbalance has been
observed with less than 50 picograms of input DNA
85
,
and further work is needed to determine whether there
are any advantages over conventional low copy number
typing methods.
One of the most difficult problems facing the forensic
biologist is the identification of body fluids. Molecular
biological approaches to the identification of blood,
semen and saliva stains using analysis of specific mRNAs
(which are surprisingly stable) have been described
86
and
will probably increase in use and importance.
As our understanding of the genetic basis of disease
and differences in the response to drugs increases, it will
be increasingly applied to determining the cause of
death forensic molecular pathology. Work has
already been done
87,88
on molecular diagnosis of the
genetic cardiac arrhythmia long QT syndrome, which
can cause sudden unexplained death leaving no trace at
autopsy, and also on the post mortem determination of
functional copy number of a gene (CYP2D6) encoding
a drug metabolizing enzyme
89
, variation in which can
lead to adverse drug effects, including death.
The most important and controversial develop-
ments, however, probably lie in the area of DNA data-
bases (BOX 3) and debates on database issues seem set to
continue well into the second two decades of forensic
DNA analysis.
handling mail deliberately contaminated with spores
of Bacillus anthracis and it is the threat of acts of
bioterrorism like this, as well as potential attacks on
crops and animals, that has led to a surge in interest in
the field of forensic microbiology, renamed microbial
forensics
81
. Its aim is to develop methods to show that
a micro-organism has come from a particular source
and DNA analysis will probably have a major role.
There are formidable problems in the wide range of
possible species that could be encountered, the limited
sequence diversity between strains and sub-strains
and the lack of universally applicable cross-species
methods to detect variation. Microbiologists, epidemi-
ologists and forensic scientists have met to define
problems and make recommendations, many of
which will be expensive to implement. Although the
extent of the bioterrorism threat is unclear, the 2001
attacks showed the major impact that even small-
scale incidents can have; also, developments in this
area will probably have useful spin-offs in tracing
natural disease outbreaks.
Future developments
Forensic genetics will continue to take advantage of
technical developments in DNA analysis. A sci-fi vision
of a hand-held device (the lab on a chip) that would
allow rapid DNA profiling at the crime scene is close to
realization, with developments in microfabrication of
capillary electrophoretic arrays
82
; single integrated plat-
forms that extract, amplify and sequence DNA have
already been developed
83
, but it will be some time before
MULTIPLE DISPLACEMENT
AMPLIFICATION
A method for whole-genome
amplification using a highly
processive polymerase from
bacteriophage 29 and random
primers to synthesize long
molecules from the template.
DROP-OUT
Absence of one or more alleles in
a DNA profile, owing to
stochastic failure of PCR
amplification when the number
of template molecules is small.
1. Jeffreys, A. J., Wilson, V. & Thein, S. L. Hypervariable
minisatellite regions in human DNA. Nature 314, 6773
(1985).
Describes the discovery of hypervariable DNA and a
method of detection that is sensitive enough to allow
analysis of the small amounts of DNA that might be
encountered in casework.
2. Jeffreys, A. J., Wilson, V. & Thein, S. L. Individual-specific
fingerprints of human DNA. Nature 316, 7679 (1985).
3. Primorac, D. & Schanfield, M. S. Application of forensic DNA
testing in the legal system. Croat. Med. J. 41, 3246 (2000).
4. Landsteiner, K. Zur Kenntnis der antifermentativen,
lytisichen und agglutinierenden Wirkungen des
Blutserums und der Lymphe. Zentralbl. Bakteriol. 27,
357362 (1900).
5. Gill, P., Jeffreys, A. J. & Werrett, D. J. Forensic application of
DNA fingerprints. Nature 318, 577579 (1985).
Shows, for the first time, the application of DNA
profiling in forensics, demonstrated by analysing stain
material. Also, the first description of differential lysis
method to separate sperm from vaginal epithelial cells.
6. Rudin, N. & Inman, K. An Introduction to Forensic DNA
Analysis 2nd edn (CRC, Florida, USA, 2002).
7. Helmuth, R. et al. HLA-DQ alpha allele and genotype
frequencies in various human populations, determined by
using enzymatic amplification and oligonucleotide probes.
Am. J. Hum. Genet. 47, 515523 (1990).
8. Kimpton, C. P. et al. Automated DNA profiling employing
multiplex amplification of short tandem repeat loci. PCR
Methods Appl. 3, 1322 (1993).
9. Sullivan, K. M., Mannucci, A., Kimpton, C. P. & Gill, P.
A rapid and quantitative DNA sex test: fluorescence-based
PCR analysis of X-Y homologous gene amelogenin.
Biotechniques 15, 636641 (1993).
10. Cotton, E. A. et al. Validation of the AMPFlSTR SGM plus
system for use in forensic casework. Forensic Sci. Int. 112,
151161 (2000).
11. Moretti, T. R. et al. Validation of short tandem repeats (STRs)
for forensic usage: performance testing of fluorescent
multiplex STR systems and analysis of authentic and
simulated forensic samples. J. Forensic Sci. 46, 647660
(2001).
12. Greenspoon, S. A. et al. Validation and implementation of
the PowerPlex 16 BIO System STR multiplex for forensic
casework. J. Forensic Sci. 49, 7180 (2004).
13. Junge, A., Lederer, T., Braunschweiger, G. & Madea, B.
Validation of the multiplex kit genRESMPX-2 for forensic
casework analysis. Int. J. Legal Med. 117, 317325 (2003).
14. Schneider, P. M. & Martin, P. D. Criminal DNA databases:
the European situation. Forensic Sci. Int. 119, 232238
(2001).
15. Butler, J. M. Forensic DNA Typing: Biology and Technology
Behind STR Markers (Academic Press, 2001).
16. Balding, D. J. When can a DNA profile be regarded as
unique? Sci. Justice 39, 257260 (1999).
17. Budowle, B., Chakraborty, R., Carmody, G. & Monson, K. L.
Source attribution of a forensic DNA profile. Forensic Sci.
Comm. [online] <www.fbi.gov/programs/lab/fsc/backissu/
july2000/source.htm> (2000).
18. Werrett, D., Pinchin, R. & Hale, R. Problem solving: DNA
data acquisition and analysis. Profiles DNA 2, 36 (1998).
19. Gill, P. et al. A new method of STR interpretation using
inferential logic development of a criminal intelligence
database. Int. J. Legal Med. 109, 1422 (1996).
20. Gill, P., Sparkes, R. & Kimpton, C. Development of
guidelines to designate alleles using an STR multiplex
system. Forensic Sci. Int. 89, 185197 (1997).
21. Johnson, P. & Williams, R. Post-conviction DNA testing:
the UKs first exoneration case? Sci. Justice 44, 7782
(2004).
22. Werrett, D. J. The National DNA Database. Forensic Sci. Int.
88, 3342 (1997).
23. Forensic Science Service. The National DNA Database.
annual report 200203. (2003). <http://www.forensic.
gov.uk/forensic/news/press_releases/2003/NDNAD_Annual
_Report_02-03.pdf>
24. Forensic Science Service. First successful prosecution after
use of pioneering DNA technique. <http://www.forensic.
gov.uk/forensic_t/inside/news/list_press_release.php?case
=23&y=2004> (2004).
25. Gill, P. An assessment of the utility of single nucleotide
polymorphisms (SNPs) for forensic purposes. Int. J. Legal
Med. 114, 204210 (2001).
26. Phillips, C., Lareu, V., Sala, A. & Carracedo, A. Nonbinary
single-nucleotide polymorphism markers. Prog. Forensic
Genet. 10, 2729 (2004).
27. Butler, J. M., Shen, Y. & McCord, B. R. The development of
reduced size STR amplicons as tools for analysis of
degraded DNA. J. Forensic Sci. 48, 10541064 (2003).
28. Gill, P., Werrett, D. J., Budowle, B. & Guerrieri, R. An
assessment of whether SNPs will replace STRs in national
DNA databases joint considerations of the DNA working
group of the European Network of Forensic Science
Institutes (ENFSI) and the Scientific Working Group on DNA
Analysis Methods (SWGDAM). Sci. Justice 44, 5153
(2004).
29. Roewer, L. et al. Online reference database of Y-
chromosomal short tandem repeat (STR) haplotypes.
Forensic Sci. Int. 118, 103111 (2001).
Describes the Y-chromosomal DNA reference
database that has become indispensable to the
analysis of Y-STR DNA profiles.
30. Home Office. Criminal Statistics England and Wales 2002
(The Stationery Office, London, 2003).
31. Shewale, J. G., Sikka, S. C., Schneida, E. & Sinha, S. K.
DNA profiling of azoospermic semen samples from
vasectomized males by using Y-PLEX 6 amplification kit. J.
Forensic Sci. 48, 127129 (2003).
32. Prinz, M., Ishii, A., Coleman, A., Baum, H. J. & Shaler, R. C.
Validation and casework application of a Y chromosome
specific STR multiplex. Forensic Sci. Int 120, 177188
(2001).
33. Kayser, M. et al. A comprehensive survey of human Y-
chromosomal microsatellites. Am. J. Hum. Genet. 74,
11831197 (2004).
34. Kayser, M. et al. Online Y-chromosomal short tandem repeat
haplotype reference database (YHRD) for US populations. J.
Forensic Sci. 47, 513519 (2002).
35. Krawczak, M. Forensic evaluation of Y-STR haplotype
matches: a comment. Forensic Sci. Int. 118, 114115
(2001).
36. de Knijff, P. Son, give up your gun: presenting Y-STR results
in court. Profiles DNA 6, 35 (2003).
R E V I E WS
37. Jobling, M. A. In the name of the father: surnames and
genetics. Trends Genet. 17, 353357 (2001).
38. Sykes, B. & Irven, C. Surnames and the Y chromosome.
Am. J. Hum. Genet. 66, 14171419 (2000).
39. Jobling, M. A. & Tyler-Smith, C. The human Y chromosome:
an evolutionary marker comes of age. Nature Rev. Genet. 4,
598612 (2003).
40. Sinha, S. K. et al. Development and validation of a
multiplexed Y-chromosome STR genotyping system, Y-
PLEX 6, for forensic casework. J. Forensic Sci. 48, 93103
(2003).
41. Butler, J. M. Recent developments in Y-Short Tandem
Repeat and Y-Single Nucleotide Polymorphism analysis.
Forensic Sci. Rev. 15, 91111 (2003).
42. Richards, M. & Macaulay, V. The mitochondrial gene tree
comes of age. Am. J. Hum. Genet. 68, 13151320
(2001).
43. Holland, M. M. & Parsons, T. J. Mitochondrial DNA
sequence analysis validation and use for forensic
casework. Forensic Sci. Rev. 11, 2149 (1999).
44. Budowle, B., Allard, M. W., Wilson, M. R. & Chakraborty, R.
Forensics and mitochondrial DNA: applications, debates
and foundations. Annu. Rev. Genomics Hum. Genet. 4,
119141 (2003).
45. Vallone, P. M., Just, R. S., Coble, M. D., Butler, J. M. &
Parsons, T. J. A multiplex allele-specific primer extension
assay for forensically informative SNPs distributed
throughout the mitochondrial genome. Int. J. Legal Med.
118, 147157 (2004).
46. Coble, M. D. et al. Single nucleotide polymorphisms over the
entire mtDNA genome that increase the power of forensic
testing in Caucasians. Int. J. Legal Med. 118, 137146
(2004).
47. Budowle, B. et al. Mitochondrial DNA regions HVI and HVII
population data. Forensic Sci. Int. 103, 2335 (1999).
48. Monson, K. L., Miller, K. W. P., Wilson, M. R., DiZinno, J. A.
& Budowle, B. The mtDNA population database: an
integrated software and database resource of forensic
comparison. Forensic Sci. Comm. 4, [online]
<http://www.fbi.gov/hq/lab/fsc/backissu/april2002/
miller1.htm> (2002).
49. Parson, W. et al. The EDNAP mitochondrial DNA population
database (EMPOP) collaborative exercises: organisation,
results and perspectives. Forensic Sci. Int. 139, 215226
(2004).
50. Yao, Y.-G., Bravi, C. M. & Bandelt, H. J. A call for mtDNA
data quality control in forensic science. Forensic Sci. Int.
141, 16 (2004).
51. Tully, G. et al. Results of a collaborative study of the EDNAP
group regarding mitochondrial DNA heteroplasmy and
segregation in hair shafts. Forensic Sci. Int. 140, 111
(2004).
52. Tully, G. et al. Considerations by the European DNA profiling
(EDNAP) group on the working practices, nomenclature and
interpretation of mitochondrial DNA profiles. Forensic Sci.
Int. 124, 8391 (2001).
53. Ivanov, P. L. et al. Mitochondrial DNA sequence
heteroplasmy in the Grand Duke of Russia Georgij Romanov
establishes the authenticity of the remains of Tsar Nicholas II.
Nature Genet. 12, 417420 (1996).
DNA analysis of the remains of Georgij Romanov
solves one of the biggest historical mysteries of the
twentieth century.
54. Lewontin, R. C. The apportionment of human diversity. Evol.
Biol. 6, 381398 (1972).
55. Barbujani, G., Magagni, A., Minch, E. & Cavalli-Sforza, L. L.
An apportionment of human DNA diversity. Proc. Natl Acad.
Sci. USA 94, 45164519 (1997).
56. Bamshad, M., Wooding, S., Salisbury, B. A. & Stephens, J. C.
Deconstructing the relationship between genetics and race.
Nature Rev. Genet. 5, 598609 (2004).
57. Lowe, A. L., Urquhart, A., Foreman, L. A. & Evett, I. W.
Inferring ethnic origin by means of an STR profile. Forensic
Sci. Int. 119, 1722 (2001).
58. Shriver, M. D. et al. Ethnic-affiliation estimation by use of
population-specific DNA markers. Am. J. Hum. Genet. 60,
957964 (1997).
59. Collins-Schramm, H. E. et al. Ethnic-difference markers for
use in mapping by admixture linkage disequilibrium. Am. J.
Hum. Genet. 70, 737750 (2002).
60. Shriver, M. D. & Kittles, R. A. Genetic ancestry and the
search for personalized genetic histories. Nature Rev. Genet.
5, 611618 (2004).
61. Sturm, R. A., Teasdale, R. D. & Box, N. F. Human
pigmentation genes: identification, structure and
consequences of polymorphic variation. Gene 277, 4962
(2001).
62. Valverde, P., Healy, E., Jackson, I., Rees, J. L. & Thody, A. J.
Variants of the melanocyte-stimulating hormone receptor
gene are associated with red hair and fair skin in humans.
Nature Genet. 11, 328330 (1995).
63. Bastiaens, M. et al. The melanocortin-1 receptor gene is the
major freckle gene. Hum. Mol. Genet. 10, 17011708
(2001).
64. Grimes, E. A., Noake, P. J., Dixon, L. & Urquhart, A.
Sequence polymorphism in the human melanocortin 1
receptor gene as an indicator of the red hair phenotype.
Forensic Sci. Int. 122, 124129 (2001).
65. Eiberg, H. & Mohr, J. Assignment of genes coding for brown
eye colour (BEY2) and brown hair colour (HCL3) on
chromosome 15q. Eur. J. Hum. Genet. 4, 237241 (1996).
66. Rebbeck, T. R. et al. P gene as an inherited biomarker of
human eye color. Cancer Epidemiol. Biomarkers Prev. 11,
782784 (2002).
67. Frudakis, T. et al. Sequences associated with human iris
pigmentation. Genetics 165, 20712083 (2003).
68. Menotti-Raymond, M. A., David, V. A. & OBrien, S. J. Pet
cat hair implicates murder suspect. Nature 386, 774 (1997).
69. DeNise, S. et al. Power of exclusion for parentage
verification and probability of match for identity in American
Kennel Club breeds using 17 canine microsatellite markers.
Anim. Genet. 35, 1417 (2004).
70. Wetton, J. H. et al. Mitochondrial profiling of dog hairs.
Forensic Sci. Int. 133, 235241 (2003).
71. Parson, W., Pegoraro, K., Niederstatter, H., Foger, M. &
Steinlechner, M. Species identification by means of the
cytochrome b gene. Int. J. Legal Med. 114, 2328 (2000).
72. Branicki, W., Kupiec, T. & Pawlowski, R. Validation of
cytochrome b sequence analysis as a method of species
identification. J. Forensic. Sci. 48, 8387 (2003).
73. Wetton, J. H., Tsang, C. S. F., Roney, C. A. & Spriggs, A. C.
An extremely sensitive species-specific ARMs PCR test for
the presence of tiger bone DNA. Forensic Sci. Int. 140,
139145 (2004).
74. Hsieh, H.-M. et al. Species identification of rhinoceros horns
using the cytochrome b gene. Forensic Sci. Int. 136, 111
(2003).
75. Korpelainen, H. & Virtanen, V. DNA fingerprinting of mosses.
J. Forensic Sci. 48, 804807 (2003).
76. Congiu, L., Chicca, M., Cella, R., Rossi, R. & Bernacchia, G.
The use of random amplified polymorphic DNA (RAPD)
markers to identify strawberry varieties: a forensic
application. Mol. Ecol. 9, 229232 (2000).
77. Lekha, Kumar, M., Kathirvel, M., Rao, G. V. & Nagaraju, J.
DNA profiling of disputed chilli samples (Capsicum annum)
using ISSR-PCR and FISSR-PCR marker assays. Forensic
Sci. Int. 116, 6368 (2001).
78. Linacre, A. & Thorpe, J. Detection and identificaion of
cannabis by DNA. Forensic Sci. Int. 91, 7176 (1998).
79. Hsieh, H.-M. et al. A highly polymorphic STR locus in
Cannabis sativa. Forensic Sci. Int. 131, 5358 (2003).
80. Ou, C. Y. et al. Molecular epidemiology of HIV transmission
in a dental practice. Science 256, 11651171 (1992).
81. Keim, P. Microbial Forensics: a scientific assessment
(American Academy of Microbiology, Washington, DC, USA,
2003).
82. Goedecke, N. et al. A high-performance multilane
microdevice system designed for the DNA forensics
laboratory. Electrophoresis 25, 16781686 (2004).
83. Paegel, B. M., Blazej, R. G. & Mathies, R. A. Microfluidic
devices for DNA sequencing: sample preparation and
electrophoretic analysis. Curr. Opin. Biotechnol. 14, 4250
(2003).
84. Dean, F. B. et al. Comprehensive human genome
amplification using multiple displacement amplification.
Proc. Natl Acad. Sci. USA 99, 52615266 (2002).
85. Schneider, P. M. et al. Whole genome amplification the
solution for a common problem in forensic casework? Prog.
Forensic Genet. 10, 2426 (2004).
86. Juusola, J. & Ballantyne, J. Messenger RNA profiling: a
prototype method to supplant conventional methods for
body fluid identification. Forensic Sci. Int. 135, 8596 (2003).
Provides an insight into the potential of mRNA to
analyse the type of body fluid in crime-scene samples.
87. Ackerman, M. J., Tester, D. J. & Driscoll, D. J. Molecular
autopsy of sudden unexplained death in the young. Am. J.
Forensic Med. Pathol. 22, 105111 (2001).
88. Lunetta, P., Levo, A., Mnnikk, A., Penttil, A. & Sajantila, A.
Death in bathtub revisited with molecular genetics: a victim
with suicidal traits and a LQTS mutation. Forensic Sci. Int.
130, 122124 (2002).
89. Levo, A., Koski, A., Ojanpera, I., Vuori, E. & Sajantila, A.
Post-mortem SNP analysis of CYP2D6 gene reveals
correlation between genotype and opioid drug (tramadol)
metabolite ratios in blood. Forensic Sci. Int. 135, 915
(2003).
90. Lander, E. S. & Budowle, B. DNA fingerprinting dispute laid
to rest. Nature 371, 735738 (1994).
91. Foreman, L. A. & Evett, I. W. Statistical analyses to support
forensic interpretation for a new ten-locus STR profiling
system. Int. J. Legal Med. 114, 14755 (2001).
92. Balding, D. J. & Donnelly, P. Inferring identity from DNA
profile evidence. Proc. Natl Acad. Sci. USA 92,
1174111745 (1995).
93. Garbolino, P. & Taroni, F. Evaluation of scientific evidence
using Bayesian networks. Forensic Sci. Int. 125, 149155
(2002).
94. Evett, I. W., Gill, P. D., Jackson, G., Whitaker, J. & Champod,C.
Interpreting small quantities of DNA: the hierarchy of
propositions and the use of Bayesian networks. J. Forensic
Sci. 47, 520530 (2002).
95. Clayton, T. M., Whitaker, J. P., Sparkes, R. L. & Gill, P.
Analysis and intepretation of mixed forensic stains using
DNA STR profiling. Forensic Sci. Int. 91, 5570 (1998).
96. Evett, I. W., Buffery, C., Willot, G. & Stoney, D. A. A guide to
interpreting single locus profiles of DNA mixtures in forensic
cases. J. Forensic Sci. 31, 4147 (1991).
97. Weir, B. S., Triggs, C. M., Starling, L., Stowell, K. A. J. &
Buckleton, J. Interpreting DNA mixtures. J. Forensic Sci. 42,
213222 (1997).
98. Evett, I. W., Gill, P. & Lambert, J. A. Taking account of peak
areas when interpreting mixed DNA profiles. J. Forensic Sci.
43, 6269 (1998).
99. Gill, P., Sparkes, R. & Buckleton, J. S. Interpretation of
simple mixtures when artefacts such as a stutters are
present with special reference to multiplex STRs used by
the Forensic Science Service. Forensic Sci. Int. 95, 213224
(1998).
100. Perlin, M. W. & Szabady, B. Linear mixture analysis: a
mathematical approach to resolving mixed DNA samples.
J. Forensic Sci. 46, 13721378 (2001).
101. Bill, M., Gill, P. & Curran, J. L. PENDULUM a guideline
based approach to the interpretation of STR mixtures.
Forensic Sci. Int. (in the press).
102. Gill, P., Whitaker, J., Flaxman, C., Brown, N. & Buckleton, J.
An investigation of the rigor of interpretation rules for STRs
derived from less than 100 pg of DNA. Forensic Sci. Int.
112, 1740 (2000).
103. Schmidt, T., Hummel, S. & Herrmann, B. Evidence of
contamination in PCR laboratory disposables.
Naturwissenschaften 82, 423431 (1995).
104. Gill, P. & Kirkham, A. Development of a simulation model to
assess the impact of contamination in casework using
STRs. J. Forensic Sci. 49, 485491 (2004).
105. Curran, J. M., Gill, P. & Bill, M. R. Interpretation of repeat
measurement DNA evidence allowing for multiple
contributors and population substructure. Forensic Sci. Int.
(in the press).
106. Clayton, T. M., Hill, S. M., Denton, L. A., Watson, S. K. &
Urquhart, A. J. Primer binding site mutations affecting the
typing of STR loci contained within the AMPFlSTR SGM
Plus kit. Forensic Sci. Int. 139, 255259 (2004).
107. Leibelt, C. et al. Identification of a D8S1179 primer
binding site mutation and the validation of a primer
designed to recover null alleles. Forensic Sci. Int. 133,
220227 (2003).
108. Rolf, B., Wiegand, P. & Brinkmann, B. Somatic mutations at
STR loci a reason for three-allele pattern and mosaicism.
Forensic Sci. Int. 126, 200202 (2002).
109. Gill, P. Role of short tandem repeat DNA in forensic
casework in the UK past, present, and future
perspectives. Biotechniques 32, 366368 (2002).
110. Santos, F. R., Pandya, A. & Tyler-Smith, C. Reliability of
DNA-based sex tests. Nature Genet. 18, 103 (1998).
111. Steinlechner, M., Berger, B., Niederstatter, H. & Parson, W.
Rare failures in the amelogenin sex test. Int. J. Legal. Med.
116, 117120 (2002).
112. Thangaraj, K., Reddy, A. G. & Singh, L. Is the amelogenin
gene reliable for gender identification in forensic casework
and prenatal diagnosis? Int. J. Legal. Med. 116, 121123
(2002).
113. Dauber, E. M. et al. Discrepant results of samples taken from
different tissues of a single individual. Prog. Forensic Genet.
10, 4849 (2004).
114. Linacre, A. The UK National DNA Database. Lancet 361,
18411842 (2003).
115. Pascali, V. L., Lago, G. & Dobosz, M. The dark side
of the UK National DNA Database. Lancet 362, 834
(2003).
116. Johnson, P., Martin, P. & Williams, R. Genetics and
forensics: making the National DNA Database. Sci. Studies
16, 2237 (2003).
117. Guilln, M., Lareu, M. V., Pestoni, C., Salas, S. & Carracedo,
A. Ethical-legal problems of DNA databases in criminal
investigation. J. Med. Ethics 26, 266271 (2000).
118. Newton, G. DNA fingerprinting and national DNA databases
(The Wellcome Trust, 2004).
119. Connor, S. Take everyones DNA fingerprint, says pioneer.
The Independent (London, UK, 3 February 2003).
120. Hagelberg, E., Gray, I. C. & Jeffreys, A. J. Identification of the
skeletal remains of a murder victim by DNA analysis. Nature
352, 427429 (1991).
The first analysis of bone samples to identify a
murder victim, conducted using mitochondrial DNA
analysis.
R E V I E WS
121. Clayton, T. M., Whitaker, J. & Maguire, C. N. Identification of
bodies from the scene of a mass disaster using DNA
amplification of short tandem repeat (STR) loci. Forensic Sci.
Int. 76, 715 (1995).
The first use of STRs on a mass disaster case,
demonstrating their usefulness in identifying highly
degraded remains.
122. Olaisen, B., Stenersen, M. & Mevg, B. Identification by
DNA analysis of the victims of the August 1996
Spitzbergen civil aircraft disaster. Nature Genet. 15,
402405 (1997).
123. Leclair, B., Frgeau, C. J., Bowen, K. L. & Fourney, R. M.
Enhanced kinship analysis and STR-based DNA typing for
human identification in mass fatality incidents: the Swissair
Flight 111 disaster. J. Forensic Sci. 49,
doi:10.1520/JFS2003311 (2004).
124. Huffine, E., Crews, J., Kennedy, B., Bomberger, K. & Zinbo, A.
Mass identification of persons missing from the break-up of
the former Yugoslavia: structure, function, and role of the
International Commission on Missing Persons. Croat. Med. J.
42, 271275 (2001).
125. Marchi, E. Methods developed to identify victims of the
World Trade Center disaster. Am. Labor. 36, 3036 (2004).
126. Budimlija, Z. M. et al. World Trade Center human
identification project: experiences with individual body
identification cases. Croat. Med. J. 44, 259263 (2003).
127. Brenner, C. H. & Weir, B. S. Issues and strategies in the
identification of World Trade Center victims. Theor. Popul.
Biol. 63, 173178 (2003).
Provides a valuable insight into the interpretation
problems that might arise in a major mass disaster,
and how they can be solved.
128. Leclair, B. Large-scale comparative genotyping and
kinship analysis: evolution in its use for human
identification in mass fatality incidents and missing persons
databasing. Prog. Forensic Genet. 10, 4244 (2004).
129. Bille, T. et al. Novel method of DNA extraction from bones
assisted DNA identification of World Trade Center victims.
Prog. Forensic Genet. 10, 553555 (2004).
130. Gill, P. et al. Identification of the remains of the Romanov
family by DNA analysis. Nature Genet. 6, 130135 (1994).
131. Gill, P. et al. Establishing the identity of Anna Anderson
Manahan. Nature Genet. 9, 910 (1995).
132. Mourant, A. E., Kopec, A. C. & DomaniewskaSobczak, K.
The distribution of the human blood groups and other
polymorphisms (Oxford Univ. Press, London, 1976).
133. Harris, H. & Hopkinson, D. A. Handbook of enzyme
electrophoresis in human genetics (North-Holland Pub. Co.,
Amsterdam / New York, 1976).
134. Edwards, A., Civitello, A., Hammond, H. A. & Caskey, C. T.
DNA typing and genetic-mapping with trimeric and tetrameric
tandem repeats. Am. J. Hum. Genet. 49, 746756 (1991).
135. Roewer, L. & Epplen, J. T. Rapid and sensitive typing of
forensic stains by PCR amplification of polymorphic simple
repeat sequences in case work. Forensic Sci. Int. 53,
163171 (1992).
136. National Research Council Committee on DNA Technology
in Forensic Science. DNA Technology in Forensic Science,
(National Academy Press, Washington, DC, USA, 1992).
137. National Research Council Committee on DNA Technology in
Forensic Science. The Evaluation of Forensic DNA Evidence,
(National Academy Press, Washington, DC, USA, 1996).
Many of the recommendations of this NRC report
have been incorporated into casework and it has
provided a reference point for the development of
DNA profiling.
138. van Oorschot, R. A. H. & Jones, M. J. DNA fingerprints from
fingerprints. Nature 387, 767 (1997).
The first demonstration that DNA profiles can be
obtained from fingerprints on touched surfaces. This
rapidly led to the development of low copy number
methods.
139. Findlay, I., Frazier, R., Taylor, A. & Urquhart, A. Single cell
DNA fingerprinting for forensic applications. Nature 389,
555556 (1997).
140. Martin, P. D. National DNA databases practice and
practicability. A forum for discussion. Prog. Forensic Genet.
10, 18 (2004).
Acknowledgements
We thank Alec Jeffreys, Chris Tyler-Smith and four anonymous
reviewers for helpful comments on the manuscript. We apologize to
colleagues whose work we were unable to cite owing to space
restrictions. M.A.J. is supported by a Wellcome Trust Senior
Fellowship in Basic Biomedical Science.
Competing interests statement
The authors declare no competing financial interests.
Online links
DATABASES
STRBase:
http://www.cstl.nist.gov/biotech/strbase/index.htm
The following terms in this article are linked online to:
Entrez: http://www.ncbi.nih.gov/Entrez/
MC1R
OMIM: http://www.ncbi.mlm.nih.gov/Omim/
Long QT syndrome
FURTHER INFORMATION
UK Forensic Science Service:
http://www.forensic.gov.uk/forensic_t/index.htm
Innocence Project: http://www.innocenceproject.org/
International Organization for Standardization:
http://www.iso.org/iso/en/ISOOnline.frontpage
Access to this interactive links box is free online.
Evidence evaluation in ngerprint comparison and automated ngerprint
identication systemsModelling within nger variability
Nicole M. Egli
*
, Christophe Champod, Pierre Margot
Ecole des Sciences Criminelles, Institut de Police Scientique, Batochime, Quartier Sorge,
Universite de Lausanne, CH-1015 Lausanne-Dorigny, Switzerland
Received 9 June 2006; accepted 14 June 2006
Available online 17 August 2006
Abstract
Recent challenges and errors in ngerprint identication have highlighted the need for assessing the information content of a papillary pattern in
a systematic way. In particular, estimation of the statistical uncertainty associated with this type of evidence is more and more called upon.
The approach used in the present study is based on the assessment of likelihood ratios (LRs). This evaluative tool weighs the likelihood of
evidence given two mutually exclusive hypotheses. The computation of likelihood ratios on a database of marks of known sources (matching the
unknown and non-matching the unknown mark) allows an estimation of the evidential contribution of ngerprint evidence.
LRs are computed taking advantage of the scores obtained from an automated ngerprint identication system and hence are based exclusively
on level II features (minutiae). The AFIS system attributes a score to any comparison (ngerprint to ngerprint, mark to mark and mark to
ngerprint), used here as a proximity measure between the respective arrangements of minutiae. The numerator of the LR addresses the within
nger variability and is obtained by comparing the same congurations of minutiae coming from the same source. Only comparisons where the
same minutiae are visible both on the mark and on the print are therefore taken into account. The denominator of the LR is obtained by cross-
comparison with a database of prints originating from non-matching sources. The estimation of the numerator of the LR is much more complex in
terms of specic data requirements than the estimation of the denominator of the LR (that requires only a large database of prints from an non-
associated population). Hence this paper addresses specic issues associated with the numerator or within nger variability.
This study aims at answering the following questions: (1) howa database for modelling within nger variability should be acquired; (2) whether
or not the visualisation technique or the choice of different minutiae arrangements may inuence that modelling and (3) what is the magnitude of
LRs that can be expected from such a model. Results show that within nger variability is affected by the visualisation technique used on the mark,
the number of minutiae and the minutiae conguration. They also show that the rates of misleading evidence in the likelihood ratios obtained for
one of the congurations examined are low.
# 2006 Elsevier Ireland Ltd. All rights reserved.
Keywords: Fingerprint evaluation; Likelihood ratio; Within-variability
1. Introduction
In law enforcement, the ngerprint used to be the most
sought after evidence from crime scene investigation, for
several reasons. Firstly, ngerprints are extremely variable
between individuals, so that even the partial and smudged
impressions lifted from crime scenes can be used to identify the
donor of these marks. Secondly, chemical and physical
detection techniques applied on relevant exhibits have allowed
through their selectivity and sensitivity to increase the number
and quality of marks detected and useful for comparison
purposes. Finally, classication of reference collections of
prints was quite straightforward, and allowed the identication
of known suspects previously ngerprinted and from the 1970s
algorithmic research has brought to the eld efcient search
algorithms that have been implemented in most nations, in the
form of automated ngerprint (or palmprint) identication
systems (AFIS). These systems allow the successful search of
the nger or palm area having left a partial mark among
millions of ten-print cards. AFIS produce a ranked list of best
matches candidates, based on proximity measures called
scores. Their computation is generally based on the positions of
minutiae as well as their directions.
www.elsevier.com/locate/forsciint
* Corresponding author. Tel.: +41 21 6924600; fax: +41 21 6924605.
E-mail address: Nicole.Egli@unil.ch (N.M. Egli).
0379-0738/$ see front matter # 2006 Elsevier Ireland Ltd. All rights reserved.
doi:10.1016/j.forsciint.2006.06.054
These scores will be used in this study. Our system outputs
higher scores indicating a better match (or a smaller
algorithmic distance) between two compared congurations
of minutiae. When AFIS systems are used operationally, the
computed score is only used as a generator of a promising list of
say 15 potential donors. The operator then carries out a usual
ngerprint comparison from this suggested list as if these
potential donors had been suggested through conventional
police inquiry.
Recent cases of misidentication, combined with a more
strident legislative scrutiny on identication evidence elds
(mainly as a consequence of the Daubert admissibility criteria),
have led commentators to call for a more articulated under-
pinning, including from a statistical perspective, of the
identication process [13].
In this paper, we propose to move towards a probabilistic
approach [4] to ngerprint evidence by deriving benet from
the score information from an AFIS system.
Several statistical studies have already been carried out, and
are commented in Ref. [5]. The rst step for modelling
probabilities associated with level II features is to measure
distances (in some sense) between minutiae congurations. For
that purpose, AFIS scores will be used a proxy for the measure
of similarity between two congurations. Secondly, in any
model for ngerprint feature evaluation, the fact that no two
appositions of a given papillary surface are exactly alike needs
to be considered very closely. It is only the consideration of this
within mark variability combined with the probability of a
randomassociation with a unrelated source that allows to assess
the weight of evidence associated with a set of matching level II
features. This is fully achieved when adopting a likelihood ratio
approach such as proposed in recent model development [6].
The evaluative tool investigated here is applicable in a very
precise context: a mark has been compared to a ngerprint, and
no discordance, which should lead to exclusions, has been
noted. In this context, the minutiae, which have been found in
concordance between the mark and the print, are considered as
known, and it is the evaluation of this concordance that can be
carried out by the present approach. The number of minutiae
and their topological conguration is therefore known.
The likelihood ratio (LR) can be expressed by the following
ratio of probabilities:
LR
PEjH; I
PEjH; I
where E is the evidence or ndings, observed between a mark
and a given print; H the proposition that the same nger is at the
origin of the print as well as the mark and; and H is the
alternative, that the mark originates from another unknown
nger than the nger at the source of the print. I is any relevant
background information which may impact on the probability
of the evidence, such as for example information regarding the
gender or ethnicity of the donor of the mark.
In the approach presented here, an AFIS is used in order to
obtain the evidence. The scores employed by the system are
the proximity measure between two minutiae congurations.
The likelihood ratio therefore becomes:
LR
psjH; I
psjH; I
where s is now the score obtained for the comparison between
the mark and the considered print. H, H and I have the same
denition as before. Note here that we are assessing the LR
using a ratio of densities for the score value obtained. The
estimation of these two density functions is at the heart of this
approach and this paper will deal rstly with the numerator
density and secondly with the denominator, starting with a
conguration of 6 minutiae.
This study aims at answering the following questions: (1)
howa database for modelling within nger variability should be
acquired; (2) whether or not the visualisation technique or the
choice of different minutiae arrangements may inuence that
modelling and (3) what is the magnitude of LRs that can be
expected from such a model.
2. Material and methods
To model within nger variability and address question (1) and (2), three
sources of images have been considered:
704 livescan images of a right thumb have been acquired using a Single
Finger Scanner (livescan) ACCO 1394 from Smiths Heimann Biometrics.
These images have been acquired to cover extreme distortion allowed by the
exibility of the skin.
88 pseudo-marks have been acquired of the same right thumb, where 22 were
visualised using each of four frequently used detection methods: DFO (1,8-
diaza-9-uorenone), ninhydrin, cyanoacrylate fuming and powder.
15 ten-print cards of the same donor have been used.
N.M. Egli et al. / Forensic Science International 167 (2007) 189195 190
Fig. 1. Illustration of the 6 peripheral minutiae.
On all these images, 6 minutiae have been selected initially positioned in the
periphery of the ngerprint pattern (Fig. 1).
In a second step, the number of selected minutiae has been increased up to
10 minutiae to verify if this factor affects the scores of within nger variability
(Fig. 2).
Finally, another conguration of 6 minutiae has been chosen (Fig. 3), in
another region of the ngerprint (towards the centre), and similarly, the number
of minutiae has been increased up to 10 (Fig. 4).
To model the variability of the score when a mark is compared to non-
matching sources (denominator of the likelihood ratio), a sample of approxi-
mately 10,000 ten-print cards has also been introduced.
All images used are at a resolution of 500 dpi and in greyscale, in tiff format.
ASagem DMAAFIS has been used for the computation of scores. Minutiae
have been placed automatically for marks as well as for prints, using manual
correction where needed, as in normal operational use. In other words, all
matching minutiae used in this study have been checked manually.
3. Results
An optimized sample size of 66 images from the 704
livescans has been determined using progressive subsampling.
Indeed 66 images, compared to one inked print, correspond to
the minimal number of images per sample, where two
independent samples still show the same shape and estimators.
A model has been estimated from the scores obtained using
these images, and a Weibull distribution has been tted (Fig. 5).
The tails of the distribution do not t well. However, when
comparing repeated samples of livescans in the same way, the
departure is at least as evident, even though they are issued from
one distribution. This shows that these effects in the tails of the
distribution are only due to sample size.
Using the marks instead of livescan images, and the same 6
minutiae, results show that these scores can also be modelled
using a Weibull distribution (Fig. 6). These scores are obtained
by comparing the marks to 15 inked impressions of the same
nger.
The mode of the distributions from livescan images
coincides with the mode of the distribution obtained for
Fig. 2. Illustration of the 10 peripheral minutiae.
Fig. 3. Illustration of the 6 central minutiae.
Fig. 4. Illustration of the 10 central minutiae.
marks. However, a greater variance is observed for the scores
from livescans, which in turn are based on much fewer
observations than the marks, as well as being inuenced by
extreme distortion.
A box plot by visualisation method shows no signicant
differences between visualisation methods and livescans,
except the higher variability, which is to be explained by
acquisition technique (see Fig. 7), a livescan with a donor
attempting to distort his nger. Furthermore, higher variability
may be very much appropriate for the estimation of likelihood
ratios in casework, since the support of the mark is not
necessarily at, nor is there necessarily no distortion present in
the mark itself.
Friedmans test has been used in order to see if there are
signicant differences between marks visualised using different
detection techniques. This hypothesis has not been rejected. It
has also been used to check if there is a difference between the
scores obtained for the different marks when compared to the
different inked prints. Here again, the hypothesis that there is a
difference has not been rejected.
In conclusion, these data provide supporting evidence for
using livescans in order to model within nger variability. For
the study, all detection methods as well as multiple ten-print
cards will be retained.
In a next step, the minutia number has been increased: 7, 8,
9, and nally 10 minutiae have been noted on the marks, using
an incremental process starting from the 6 original minutiae.
Minutiae were chosen according to their proximity to the initial
conguration.
Fig. 8 shows how within nger variability evolves with these
changes for the rst peripheral conguration of minutiae.
As expected, the scores increase as more minutiae are
considered. Another trend in the data is that the variance of
these scores increases with each added minutia. Since more
Fig. 6. Weibull probability plot of the scores obtained using 6 minutiae on
marks visualised using DFO compared to 15 ngerprints of the same nger.
Fig. 7. Boxplots of scores for 6 minutiae on livescans (Lsc) and marks
visualized using four methods (CA, powder, Ninhydrin and DFO) as well as
a random selection of marks visualised using any of the four methods (All).
Fig. 8. Histograms of the scores obtained for 6, 7, 8, 9 and 10 minutiae in a
peripheral region.
Fig. 5. Probability plot between scores obtained from 66 livescans and a
Weibull distribution.
error in the positioning (even manual) of the minutiae is
introduced, this is not surprising.
When the group of minutiae is changed, scores also increase
with the number of minutiae (Fig. 9).
The t to the Weibull distribution, with each added minutiae,
is well supported by the data (see Fig. 10 for 7 minutiae in the
second, central, region).
For 6, 8, 9 and 10 minutiae, results are similar. In the
peripheral region, the t to a Weibull distribution observed is
less optimal. The reason for this is that minutiae were often less
similarly designated between the mark and the print.
The difference between minutiae congurations of a given
number of minutiae in different regions of the ngerprint has
been investigated; formal tests (KolmogorovSmirnoff test for
equal distribution between two samples) have rejected the
hypothesis of an absence of differences for all except the two 6-
minutiae congurations. A difference between distributions of
scores issued from comparisons between minutiae congura-
tions, which have the same number of minutiae, is therefore a
reality. However, since there are some doubts about the origin
of some deviations in the peripheral minutiae conguration,
this will be investigated further once this origin is known.
Tippett plots [7] have been used in order to check if
modelling choices are admissible on the original minutia
conguration. The between-nger variability is modelled using
a lognormal distribution [8] that is tted to the scores of the
mark in question (the latent evaluated for a given likelihood
ratio) when compared with the 100,000 ngerprints from
unrelated sources.
The Tippett plot for 7 minutiae in the peripheral region is
shown in Fig. 11.
Very few LRs above 1 are observed (0.25% of comparisons
between different ngers), on this database when truly the print
is not at the source of the mark. In the majority of cases, the
evidence provides strong to very strong support for the
alternative when truly this alternative is established. Con-
versely, when the proposition is true (in other words when the
mark truly originates from the nger at the source of the print),
the evidence provides almost always evidence for that
proposition. The proportion of cases where the evidence is
supporting the alternative when the matching proposition is true
is limited (1.90%). This allows to conclude that the system is
quite reliable in the given context: within nger variability, in
particular, has been modelled using repeated pseudo-marks
from the same nger as the latent-print comparison being
evaluated.
4. Case example
Although our results constitute only a rst step towards an
operational system, we felt it was important to illustrate the
operational use of such a model using a case example. It will
stress two important points: rst, once the two density functions
available, the computation is straightforward and can be
undertaken in the background and second, the computation
process will not heavily distract ngerprint examiners from
their usual practice. For each comparison a full ACE-Vprotocol
Fig. 9. Histograms of the scores obtained for 6, 7, 8, 9 and 10 minutiae in a
central region.
Fig. 10. Weibull probability plot for 7 minutiae in central region.
Fig. 11. Tippett plot obtained using the scores for 7 minutiae.
will have to be followed forcing the examiner to consider the
concordant features and potential discrepancies. It is only when
at that stage that the hypothesis of common source cannot be
ruled out that a computation of a LR based on the considered
features can be undertaken. For example, let us assume the
following mark to be compared to a print (Fig. 12).
The comparison process has highlighted 10 minutiae in
agreement, and no discordance which would lead to an
exclusion.
Let us assume for the sake of the argument that an
individualisation cannot be achieved due to the limited
information available in terms of quantity and quality
(otherwise, the statistical argument is irrelevant).
In the AFIS system used, the rst nger on the list is that
precise left index, with a score of 4192.
The numerator density (reecting on the within nger
variability) is obtained from the 10 minutiae conguration in
the center discussed above, assuming that the above data offer a
generalisation capability across ngers and individuals. Within
nger variability is a Weibull with parameters (4573.6, 9.8).
The denominator density (reecting between ngers
variability) is obtained by considering all other scores in the
database (from non-matching prints), and is modelled using a
lognormal distribution with parameters (7.3562, 0.1634). The
parameters have been directly estimated from the scores
obtained by comparing the conguration of minutiae on the
mark with all prints on the database. See Fig. 13 for an
illustration of the t of the lognormal distribution to the scores
obtained for this mark, and Fig. 14 for an illustration showing
Fig. 12. The mark (a) and the print (b) considered in the case example.
Fig. 13. Lognormal probability plot for between nger variability on 7 minutiae
in the peripheral region.
within and between nger variability as well as the score
obtained in this comparison.
The likelihood ratio is
LR
psjH; I
psjH; I

6:498 10
4
7:5927 10
12
8:56 10
7
which means that the evidence provides very strong support for
the view that mark and print come from the same source.
5. Discussion and conclusion
The Weibull distribution used for the modelling of within-
nger variability had been mentioned previously in the
literature and discarded in favour of a Gamma distribution
[8]. However, in the present case, it is the model that ts the best
the data, and has therefore been retained.
It has been shown that
(1) Within-nger variability, in the sense used here, i.e., the
variability observed when the same minutiae are used, can
be modelled using a Weibull distribution.
(2) Marks can be replaced by livescans. This has strong
operational impacts. Indeed livescan images are much
easier to obtain that casework realistic marks.
(3) An increase of the minutia number lead to an increase of the
scores, as well as the variance of the numerator distribu-
tions.
(4) A change of the minutia conguration triggers a change of
the scores distribution for a given number of minutiae.
(5) The addition of a minutia does not cause a continuous
increase of the scores of the within nger variability;
it depends on the minutiae present before and the one
added.
An evaluation of the likelihood ratios obtained using the
adopted modelling strategy, using a Tippett plot, indicates a
very good performance of the system. This warrants further
research about variations observed when minutiae are chosen in
the delta area, when the nger, the general pattern and the donor
are changed, as well as an investigation of between-nger
variability. Ideally we would hope that the within nger
distribution has identied and known characteristics across
ngers and areas. Should that not be the case, the promising
results obtained using a livescan device testify to the feasibility
of modelling within nger variability on a case-by-case basis.
Finally, the case example demonstrates a way of applying
this evaluative tool to an actual ngerprint comparison.
References
[1] S.A. Cole, More than zero: accounting for error in latent ngerprint
identication, J. Crim. Law Criminol. 95 (2005) 9851078.
[2] S.A. Cole, Grandfathering evidence: ngerprint admissibility rulings from
Jennings to Llera Plaza and back again, Am. Crim. Law Rev. 41 (2004)
11891276.
[3] M.J. Saks, J.J. Koehler, The coming paradigmshift in forensic identication
science, Science 309 (2005) 892895.
[4] C. Champod, I.W. Evett, Aprobabilistic approach to ngerprint evidence, J.
Forensic Ident. 51 (2001) 101122.
[5] D.A. Stoney, Measurement of ngerprint individuality, in: H.C. Lee, R.E.
Gaensslen (Eds.), Advances in Fingerprint Technology, CRC Press LLC,
Boca Raton, 2001.
[6] C. Neumann, C. Champod, R. Puch-Solis, D. Meuwly, N. Egli, A. Antho-
nioz, A. Bromage-Grifths. Computation of likelihood ratios in ngeprint
identication for congurations of three minutiae, J. Forensic Sci., in press.
[7] I.W. Evett, J.S. Buckleton, Statistical analysis of STR data, in: A.
Carracedo, B. Brinkmann, W. Bar (Eds.), Advances in Forensic Haemo-
genetics, vol. 6, Springer Verlag, Heidelberg, 1996.
[8] L.M. Wein, M. Baveja, Using ngerprint image quality to improve the
identication performance of the U.S. Visitor and Immigrant Status Indi-
cator Technology Program, PNAS 102 (2005) 77727775.
Fig. 14. Within and between nger scores distribution with the score of the
comparison indicated.
This paper deals with the question of inference of identity of
source in forensic science and focuses on the individualisation
process of human beings from biometric data. Its goal is to
explicitly describe the logical mechanisms that underpin the
inference of identity of source in forensic science and to propose
a concrete methodology for the forensic individualisation from
biometric data. It also analyses the limits of several biometric
modalities (fingerprint, face, voice) used for forensic
individualisation.
Firstly, the concept of identity and its application in forensic
science is described. Secondly, the confusion between numerical
and qualitative identity is addressed. Thirdly, the hypothetical-
deductive method is described as a logical approach for forensic
individualisation and the likelihood ratio approach premised on
the Bayes Theorem is proposed as a concrete methodology to
infer identity of human beings from biometric data in forensic
science. Finally a concrete framework is proposed, which
includes hypotheses definition, sources and databases selection,
analysis and comparison of biometric data and their
interpretation.
Este trabajo trata sobre el tema de la inferencia de identidad de
una fuente en ciencia forense y se concentra en el proceso de
individualizacin de seres humanos a partir de datos
biomtricos. Su meta es describir explcitamente los
mecanismos lgicos que subrayan la inferencia de identidad en
ciencia forense y proponer una metodologa concreta para la
individualizacin forense a partir de datos biomtricos. Tambin
analiza los lmites de algunas modalidades biomtricas (huellas
dactilares, rostro, voz) usadas para la individualizacin forense.
In dieser Abhandlung wird die Frage der Quellen-
Identittsinferenz untersucht mit Konzentration auf den
Identifizierungsprozess von Menschen mittels biometrischer
Daten. Das Ziel ist eine detaillierte Beschreibung der logischen
Ablufe, welche die Identittsinferenz der Quellen in der
Forensik sttzen und eine konkrete Methodik fr die forensische
Identifizierung aus biometrischen Daten aufzustellen. Auerdem
werden die Grenzen verschiedener biometrischer Modalitten
(Fingerabdruck, Gesicht, Stimme) fr die forensische
Identifizierung analysiert.
Zunchst wird beschrieben, was man in der Forensik unter
Identitt und deren Anwendung versteht. Zweitens wird die
Verwechslung zwischen numerischer und qualitativer Identitt
besprochen. Drittens wird die hypothetisch deduktive Methode
als logischer Ansatz bei der forensischen Identifizierung
angesprochen sowie die Theorie der Wahrscheinlichkeitsquote
basierend auf dem Bayes Theorem als konkrete Methodik zur
Ableitung der menschlichen Identitt aus biometrischen Daten in
der Forensik vorgeschlagen. Schlielich wird ein konkreter
Rahmen aufgestellt einschlielich der Definition der Hypothese,
Auswahl der Quellen und Datenbanken, Analyse und Vergleich
biometrischer Daten sowie deren Interpretation.
Cet article traite de la question de linfrence de lidentit de la
source en sciences forensiques et se focalise sur le processus
dindividualisation de personnes partir de donnes
biomtriques. Son but est de dcrire les mcanismes qui sous-
tendent linfrence de lidentit de la source en sciences
forensiques et de proposer une mthodologie concrte
lindividualisation forensique partir de donnes biomtriques.
Il analyse galement les limites de plusieurs modalits
biomtriques (traces digitales, vocales et faciales) utilises pour
lindividualisation forensique.
Premirement le concept didentit et son application en
sciences forensiques est dcrit. Deuximement la confusion
entre identit numrique et qualitative est traite. Troisimement
la mthode hypothtico-dductive est dcrite comme une
approche logique pour lindividualisation forensique et
lapproche du rapport de vraisemblance base sur le thorme de
Bayes est propos comme une mthodologie concrte pour
infrer lidentit de personnes partir des donnes biomtriques
en sciences forensiques. Finalement, un cadre concret est
propos, incluant la dfinition des hypothses, la slection des
sources et des bases de donnes, lanalyse et la comparaison de
donnes biomtriques et linterprtation des rsultats.
Forensic Individualisation
from Biometric Data
D. Meuwly*
Institut de Police Scientifique et Criminologie, Faculty of Law, University of Lausanne
CH-1015 Lausanne, Switzerland
Science & Justice 2006 46 205 213 Received: 13/06/2003 Accepted: 10/06/2006
science&justice Volume 46 No.4 (2006) 205 213
Page 205
* Author for correspondence
The Forensic Science Society 2006
Key words Biometrics, Forensic science, Individualisation,
Methodology, Likelihood ratio, Bayes Theorem, Falsifiability
Ambiguity
The term identity has a dual character and this duality gives
rise to ambiguity. When the concept of source is used with
reference to an object of interest to signify a class of individual
entities from which this object could originate, it refers to
qualitative identity. This is due to the fact that a class is defined
by an identity of the properties of its members. The operation in
which a class is determined to be the source is termed
classification or identification in science. When the concept
of source is used to signify one particular individual entity from
which an object originates, it refers to numerical identity. The
operation in which a particular individual entity is determined to
be the source of an object is termed individualisation in
science, but is often wrongly named identification in forensic
science.
Confusion surrounds the terms identity, identify and
identification in forensic science. This is clearly demonstrated
in popular practice, when the perpetrator of an infringement is
said to be identified from her/his fingerprints. The perpetrator
is not identified, but individualised. What is proved by the
fingerprints is individuality [6, 9]. Kirk emphasizes this
confusion as well in [3], but concludes:
The real aim of all forensic science is to establish
individuality or to approach it as closely as the present
state of science allows. Criminalistics is the science of
individualisation. The criminalist is not ultimately
interested in the similarity of two objects but in their
source [3].
Therefore, to individualise a human being on the basis of
biometric data in forensic science ultimately consists in
determining if an individual is the source of the biometric data
considered as forensic traces/marks.
Identity in forensic science
According to Kwan, what is meant by identity of
source is relative to what source signifies. When source
refers to class, identity is akin to qualitative identity
and when source refers to an individual, identity of
source is akin to numerical identity. This distinction
made between the two kinds of identity is consistent
with the classical forms of identity distinguished by
philosophers over the ages. The reason for the
approach from a philosophical perspective is that the
central problem of identity of source comes to be
known [4].
Distinguishing numerical from qualitative identity
Qualitative identity is established when a set of properties agrees
in two objects. Numerical identity is demonstrated by
Introduction
On May 6, federal prosecutors strode into a courthouse in
Portland, Oregon, and claimed the FBI had made a 100 percent
positive identification linking a local lawyer to a fingerprint
found on a bag connected to terrorist bombings in Madrid.
Within weeks, the same prosecutors were forced to return to the
courtroom and admit an international humiliation: the fingerprint
analysis that led to the arrest of Brandon Mayfield was wrong.
But the FBI didnt realize it until Spanish authorities linked the
fingerprint to an Algerian man, Ouhnane Daoud. Not just one but
three FBI analysts, all seasoned veterans, had made the same
mistake. A fourth expert independently appointed by the judge
erred as well when he determined Mayfields prints were a
match [1]. This erroneous fingerprint individualisation in the
Madrid Train bombing case [2] indicates the necessity to revisit
the forensic methodology for the individualisation of human
beings from biometric data collected on crime scenes, and
particularly the fingermarks.
The study of fingerprints as a means of positive identification by
outstanding scientists like Sir William Herschel, Dr Henry
Faulds, Sir Francis Galton, Sir Edward Richard Henry and
Dr Edmond Locard, was the starting point of the science of
individualisation, termed by some criminalistics, by others
forensic science [3].
The first part of this paper is dedicated to the concept of identity
and to its application in forensic science. The content of this
theoretical basis is a brief summary of the unique research effort
on this particular topic, the brilliant but sadly underrated PhD
dissertation of Dr Quon Yin Kwan entitled Inference of identity
of source [4]. The second part of this paper illustrates the
confusion associated with the notion of identity when used for
forensic individualisation from biometric data. The third part is
devoted to the description of the forensic individualisation process
based on the hypothetical-deductive method, while the last part
proposes a concrete framework to design a computer-based
approach for forensic individualisation from biometric data.
Identity
Definition
En police scientifique et en droit, lidentit est
lensemble des caractres par lesquels un homme
dfinit sa personnalit propre et se distingue de tout
autre. Dans ce dernier ordre dides, tablir lidentit
dun individu est lopration policire ou mdico-
lgale appele identification. Un homme peut tre
semblable plusieurs autres, ou un autre, au point
damener des erreurs; il nest jamais identique qu un
seul, lui-mme. Cest discriminer avec soin les
lments de ressemblance des lments didentit que
consiste le problme de lidentification
1
[5].
D Meuwly
Forensic Individualisation from Biometric Data
Page 206
1. Freely translated from the original French citation: In forensic science and law, identity is the set of characteristics in terms of which
a human being defines its own personality and distinguishes itself from all others. In this context, establishing the identity of an individual
is the forensic activity called identification. A human being may be similar to several others or to one other person to the extent that it
causes errors; but it can only be identical to one person, himself/herself. The challenge of identification lies in the careful discrimination
of elements of similarity from elements of true identity.
D Meuwly
Page 207
establishing continuity in time. A well-known example of
numerical identity in forensic science is the chain of evidence, or
chain of custody. To be maintained, the chain of evidence
requires proof that the item is the same individual entity, from
the time of collection on the crime scene to the time of
presentation in court. Because qualitative identity is not
determined by this criterion of continuity, it is relative. The
relativity of qualitative identity is with respect to the observer,
especially with respect to his choice of properties to characterize
objects [4].
The principal distinguishing characteristic between numerical
and qualitative identity is time. Time makes identity of objects
absurd since no two objects can be one object simultaneously.
Identity of source is a relationship that involves time because of
the need to establish the continuity of objects from one source in
time. Qualitative identity, on the other hand, is independent of
time. Time implies that numerical identity is compatible with
change whereas qualitative identity is not.
A thing can be identical only with itself, never with
any other object, since objects in the universe are
unique. If this were not true, there could be no
identification in the sense used by the criminalist [3].
To convince yourself that numerical identity is compatible with
change, take your ten-year-old identity document, compare your
face with the passport photo of this identity document and
observe the changes. This shows that two objects with different
qualities, existing at different times, can be numerically
identical. This finding illustrates that numerical identity does not
entail qualitative identity and, vice versa, qualitative identity
does not entail numerical identity. It demonstrates the absence of
a logical relationship between numerical and qualitative identity.
Forensic individualisation is not a deduction
The discussion on the relationship between qualitative and
numerical identity provides different perspectives on identity. In
his work, Leibniz never explicitly stated a law relating
qualitative and numerical identity, but two different
interpretations of his position on identity have created
considerable confusion. Leibnizs position on identity is
expressed in two formal maxims: the Law of Identicals and the
Law of Identity of Indistinguishables. The former says that what
is featured in one thing is featured in another when the two are
numerically identical simultaneously. There is no problem with
this interpretation although, according to Wittgenstein, it is a
tautology [4]. The latter simply equates qualitative identity with
numerical identity, which is not a valid relationship, as
explained supra. In spite of the questionable validity of this
second interpretation, it has become a fundamental principle for
forensic science.
General principle of uniqueness
Dogma
The idea that identity of source can be deduced on the basis of
qualitative identity and on the assumption of uniqueness of the
source is still widely supported by the forensic community [6]. In
fact, in forensic science the individualisation process is most often
perceived as a process of rigorous deductive reasoning [7], as a
syllogism constituted of a major premise, a minor premise and a
conclusion
2
. The major premise is the general principle of
uniqueness applied to the properties of a source and to the
properties of the object generated by this source, the trace/mark,
the minor premise is the observation of the correspondence of the
observed properties between the source and the trace/mark and the
conclusion is that the source and the trace/mark have a common
origin because of the correspondence of the observed properties.
but shortcomings
However, the general principle of uniqueness assumed in forensic
science must be qualified as a dogma because this principle of
forensic identification is based on inductive reasoning. It is
founded on a line of reasoning that proceeds from particular
statements (based on observation or experience) to universal laws
or theories [8-10]. Both Locard [11] and Eco [12] are conscious
of this misuse of inductive reasoning in forensic science, when
they invoke the eighth rule of syllogism according to Aristotle:
nihil sequitur geminis ex particularibus unquam
3
. This
logical statement was first developed by Sextus Empiricus
(AD 150230) and David Hume (17111776) adapted this
criticism of the application of induction to the notion of causality.
Therefore, induction cannot be considered as rigorous reasoning
for individualisation in forensic science because what is true for
the particular is not necessarily true for the whole. The misuse of
induction provides the illusion that science substantiates
categorical conclusions of identification or exclusion [8].
The criterion of empirical falsifiability
Popper uses the argumentation of Hume but argues that complete
verification of any scientific statement is impossible; therefore,
the complete verification of the general principle of uniqueness
applied to forensic science is not possible. Popper refutes the
criterion of scientific verifiability but proposes the criterion of
empirical falsifiability as demarcation criterion.
The reasoning process of Popper is based on a fundamental
logical consideration: to prove the hypothesis of a general
principle from particular or singular statements is impossible,
but it is possible to falsify it. Falsifying a hypothesis amounts to
proving that the hypothesis of a general principle is false because
one or more cases contradict it: what is false to the particular is
false to the whole. To fail in the attempt to falsify the hypothesis
of uniqueness will never prove that the principle is true but, in
2 Following the first rule of syllogism according to Aristotle Terminus esto triplex: medius, majorque, minorque, a syllogism should be
composed, to be valid, of a general principle, named major premise, a particular observation, called minor premise and a deduction,
called medium term or conclusion.
3 No deduction can follow two minor premises.
D Meuwly
Page 208
standing the tests, the hypothesis can reach a sufficient degree of
corroboration to be used as a basis for practical application,
described as the degree of verisimilitude by Popper [13].
Principle of uniqueness applied to individualisation
from biometric data
Dogma
The uniqueness of the properties used for individualisation from
biometric data is in general assumed without debate, both in
forensic and commercial applications of biometrics [14].
Moreover, in forensic science the principle of uniqueness
assumed for the properties of the source is often extended to the
properties of the trace/mark without question. For example, the
uniqueness is assumed for the fingerprint (source) and the
fingermark (mark), for the face (source) and the still or the live
picture of the face (trace) as well as for the voice (source) and
the speech audio recording (trace).
but shortcomings
The demarcation criterion of empirical falsifiability developed
by Popper in most cases conflicts with the hypothesis of
uniqueness of the properties used for individualisation from
biometric data. The different extent to which various biometric
characteristics are able to meet the demarcation criterion of
falsifiability is experienced in everyday life as well as in forensic
practice, as shown in the following examples.
Assumption of uniqueness for complete rolled inked
fingerprints
Automatic fingerprint identification systems (AFIS) have been
used for 30 years, collecting databases of convicted people and
of asylum seekers in order to detect repeat offenders and prevent
multiple asylum applications. In these two situations, a ten-print
sheet from a suspected person or from an asylum seeker is made,
and two fingers, generally the right index and middle finger, are
searched against the main fingerprint collection. The eight other
fingers are generally not used for the automatic research, but
when a candidate is proposed by the system, they can be used by
the fingerprint examiner to falsify the hypothesis of identity
based on the automatic comparison of the two first fingers. As
far as we know, this procedure of individualisation based on the
comparison of ten inked fingerprints (considered as secondary
sources, the primary sources being the real fingers) is extremely
difficult to falsify. Of course this consideration only takes in
account the output of the automatic process and not the output of
the entire process including human interaction with the AFIS,
even though human errors, due to clerical mistakes or others are
intrinsic to any human activity.
From our point of view, the assumption of uniqueness of the
fingerprint first formulated by Herschel and Galton [15] reaches
a sufficient level of verisimilitude and therefore renders very
valid the assumption that the complete rolled inked fingerprint as
a reliable enough representation of the real finger for human
individualisation.
Assumption of uniqueness of the fingermark
In contrast, forensic practice is punctuated with several cases of
false individualisation of defendants on the basis of the
comparison of fingermarks with inked fingerprints by fingerprint
examiners. Apart from the Mayfield case [1], one can quote the
Scottish case of H.M. Advocate v. Shirley McKie; for the first
time since the adoption of the 16-minuti standard, a full
identification of a latent mark has been challenged in a court of
law in the UK, and that challenge was upheld by a unanimous
verdict [16]. Therefore, from our point of view, the assumption
of uniqueness of the fingermark does not reach a sufficient level
of verisimilitude.
Assumption of uniqueness of the properties of the face
and voice
The uniqueness of properties of the human face is also
questionable, as no idiosyncratic property is known or can be
assumed for this biometric modality. For the voice as well, the
hypothesis that no two humans speak exactly alike is plausible,
but to date no large-scale demonstration of the extent of
idiosyncrasy in a homogeneous community of speakers has been
adduced in support of this hypothesis [17]. Therefore, the
assumption of uniqueness does not reach a sufficient level of
verisimilitude for the human face and voice.
Assumption of uniqueness of the properties of face
and the voice traces
The consideration mentioned supra is also true for the properties
of the face and voice traces. Experiences of everyday life as well
as the findings of applied psychology and artificial perception
confirm the limitations of human beings and machines in their
capacity for human individualisation based on face recognition,
speaker recognition or on those two combined [18, 19]. This
situation is particularly relevant in forensic science, where the
trace is normally of limited quality for voice and face, for
instance a telephone quality audio recording, a CCTV video
recording or a passport photo.
Discussion
This assessment does not mean that fingermarks, face traces and
voice traces cannot be used or are not suitable for
individualisation from biometric data in forensic science, but it
clearly lays down limits on the certainty that can be expected,
depending on the quality of the trace/mark and on the method
chosen for the individualisation process. The lack of constancy
in the face and the voice, as well as in the fingermark, highlights
the existence of a within-source variability besides the between-
source variability, which should also be considered in the
forensic individualisation process.
Since the forensic individualisation process cannot be seen as a
deduction based on qualitative identity and on the general
principle of uniqueness, another approach must be considered to
infer the identity of source.
Forensic individualisation remains an inference
When the concept of source refers to a class of entities from
which an object could originate, the question of how the source
is identified is merely an epistemological question of how
qualitative identity comes to be known, i.e., through a set of
characteristics, as explained supra. When the concept of source
signifies a particular individual entity from which an object
D Meuwly
Page 209
originates, the question of how the source is identified is a much
more complex problem. The general rule to demonstrate
numerical identity is by establishing continuity. But this answer is
impracticable for the forensic scientist, who almost never knows a
priori the actual source nor have seen the source. As there is no
one to ensure unbroken continuity between the source and the
object of interest since the creation of this object, there is no way
of knowing the numerical identity of source [4]. By essence, the
identity of source remains an inference in forensic science.
The hypothetical-deductive method
Principle
The notion of proof is not subordinated by essence to the concept
of deduction [20]. The hypothetical-deductive method, already
described by the ancient Greek philosopher Parmenides, seems
to be a much more practicable approach for inferring identity of
source in forensic science. It can be described as a process of
hypothesizing and testing; a set of hypotheses is formulated
which is tested and modified in a cyclic fashion until a modified
hypothesis is arrived at which is not rejected.
In using the hypothetical-deductive method, one
commences by posing several hypotheses that could
explain a phenomenon which has just been observed.
Hypotheses are generally posed after taking into
account what is generally known about the properties
of the class of phenomena of interest prior
knowledge. One then proceeds to determine which one
hypothesis from the set of all plausible hypotheses best
explains the phenomenon. Deductions are made from
the posed hypotheses and these serve as bases for
proposing experiments. That is, if one can make
predictions from these hypotheses, one can devise
experiments to test the predictions. This is the most
valuable congruent of the hypothetical-deductive
method. If a hypothesis either does not agree with the
body of prior knowledge or if its predictions are
falsified by experimentation, it is rejected. This is done
successively until a single hypothesis remains that
explains the phenomenon which no alternative
hypothesis does [4].
The hypothetical-deductive method was already considered a
relevant reasoning process by Edgar Allan Poe, when he
described the adventures of the detective Dupin, in his Tales
stories. As pointed out by Locard, Poe requires the ideal
detective to combine the imagination of the poet (to generate the
hypotheses) with the method of the mathematician (to test these
hypotheses) [11]. Even Sherlock Holmes repeats often: The old
axiom that when all other contingencies fail, whatever remains,
however improbable, must be the truth [21].
Although the analytical process described supra is first of all an
exercise of logic which is not directly related to reality, the
hypothetical-deductive method requires an empirical validation
of the resulting hypotheses [22].
It is worthwhile emphasizing that to generate hypotheses is a
necessary but not sufficient condition for the reasoning from
hypotheses. In fact, as a hypothesis is intrinsically not
demonstrable, it is necessary to distinguish the plausible
alternative hypotheses from the fanciful ones. It is now
acknowledged that it is impossible to derive criteria to define
ex nihilo the notion of plausible hypothesis, because these
criteria depend dramatically on the problem analysed [20].
For the questions related to the inference of identity of source of
a trace/mark, it is possible to define two mutually exclusive
alternative hypotheses: the prosecution hypothesis (Hp) and the
defence hypothesis (Hd). A hypothesis can be considered as
plausible when it is accepted as a possible explanation by the
fact-finder.
Methods of statistical inference
Methods of statistical inference supplement the hypothetical-
deductive method by assigning weight to predictions of
hypotheses facilitating the selection of the hypothesis that best
explains the source of the trace/mark. But statistics will not lead
to an objective process of absolute identification [23]. One
important assumption that must be heeded is that every
quantitative method chosen for the inference of identity of
source is based upon the premise of qualitative identity. It means
that the features used to characterize an object must be selected
on the criteria of distinguishability, ratio between the within-
source and the between-source variability, stability in time,
standardization and independence [4]. Among the methods of
statistical inference, the likelihood ratio approach premised on
the Bayes Theorem is currently considered as the most logical
framework for the inference of identity of source in forensic
science. This choice was acknowledged as early as the beginning
of the twentieth century in the Dreyfus case [24] and
publications of the past fifteen years illustrate this trend for the
interpretation of many kinds of forensic evidence (e.g.
fingermark, DNA, speaker recognition or earmark) [9, 25-28].
The likelihood ratio approach
premised on the Bayes Theorem
Definition of the hypotheses
The background information (I) on the case and the preliminary
observation of the trace/mark are the necessary information to
define the set of all the plausible sources of the trace/mark,
named the potential population. The background information
also determines which particular source of the potential
population can be focused on and selected as the putative source
of the trace/mark.
The prosecution hypothesis Hp is the one according to which the
putative source is truly the source of the trace/mark. The defence
hypothesis Hd is the one according to which an alternative
plausible source is truly the source of the trace/mark. A logical
constraint necessitates that the two hypotheses are mutually
exclusive, but not necessarily exhaustive.
Test of the hypotheses
The likelihood ratio approach shows how an a priori
probability ratio of the two competitive hypotheses Hp and Hd
can evolve to an a posteriori probability ratio, considering the
background information and the result of comparison of the
D Meuwly
Page 210
putative source with the trace/mark, named the evidence (E).
The likelihood of the evidence is evaluated when the
hypothesis Hp is true on the one hand, and when the
hypothesis Hd is true on the other hand. The ratio between
these two likelihood values, the likelihood ratio (LR), is
defined as the numerical value that allows for revision of the
a priori probability ratio (prior odds), based on the new
information E, to give the a posteriori probability ratio
(posterior odds) of the two hypotheses Hp and Hd:
Framework for forensic individualisation
from biometric data
This section presents a concrete framework based on the
hypothetical-deductive method for the forensic individualisation
from biometric data. This framework, summarized in the scheme
presented in Table 1, includes the definition of the alternative
hypotheses, the selection of the sources and databases, the
analysis and comparison of the biometric properties, and the
interpretation of the evidence using the likelihood ratio
approach.
The trace/mark (X) considered for forensic individualisation
from biometric data can be a fingermark, a photo or a video
recording of a face or a speech audio recording. The set of all the
plausible sources of the trace/mark is defined on the basis of the
background information (I) and on the preliminary observation
of this trace/mark. The background information also determines
which of the plausible sources can be focused on and selected as
the putative source (Y).
The prosecution hypothesis is the hypothesis that the putative
source (Y) is the source of the trace/mark (X). For the clarity of
the scheme, the subset of all the other plausible sources is
considered as one generic alternative source. The defence
hypothesis Hd is the hypothesis that an alternative source is the
source of the trace/mark (X). In reality however, the background
information about the different other plausible sources can vary,
and if so, a distinct defence hypothesis should be considered for
each other plausible source.
Selection of the databases
When applied to forensic individualisation from biometric data,
the likelihood ratio approach needs biometric data to estimate
the within-source variability of the putative source and the
between-source variability of the trace/mark. The data has been
structured in three databases: the potential population database
(P), the putative source reference database (R) and the putative
source control database (C). The content and use of each of these
databases is detailed infra.
Analysis and comparison
Extraction of the characteristics
The importance of selecting characteristics judiciously cannot be
overemphasized. If the characteristics are poorly chosen, no
amount of mathematics can salvage an individualisation scheme
[29]. For the fingerprint, the information is known and structured
in three levels of characteristics: the first level is the pattern type
of the fingerprint (arch, loop and whorl), the minuti or Galton
points (ridge ending, bifurcation and dot) form the second level
and the pores and ridge edges constitute the third one. The current
AFIS systems use mainly the position and angle of the minuti as
well as the skeleton for the analysis, but the use of other
characteristics such as the probability for minuti configuration
on the surface of the finger would contribute to developing a
probabilistic assessment of fingermarks based on statistics [26].
For face and voice, the knowledge of the existence of idiosyncratic
characteristics is hampered by the difficulty to provide a symbolic
description for this information. In such cases, the
individualisation process is supported by the recognition of the
information containing source-dependent characteristics, the
information itself remaining impossible to define [30].
Estimation of the value of the evidence
The evidence is the result of the comparative analysis of the
characteristics (x) extracted from the trace/mark X, with the
characteristics (y) extracted from the putative source Y. In a
computer-based approach of individualisation from biometric
data, the result of the comparison of x and y leads to a
unidimensional or multidimensional numerical value which
estimates the distance or the proximity index between them;
this information represents the evidence E.
Interpretation of the evidence using the likelihood
ratio approach
Estimation of the between-source variability
and calculation of the denominator of the LR
The potential population database (P) is a large-scale database
used to estimate the variability of the sources from the potential
population. Ideally, P is made up of information that contains the
exhaustive characteristics of interest of the alternative sources
from the potential population. For individualisation from
biometric data, this database can be made up of rolled inked
fingerprint pictures, 2D or 3D pictures of the face or spontaneous
speech audio recordings of the alternative sources from the
potential population.
The characteristics concerning the relevant alternative sources
are extracted and compared to the characteristics of the
trace/mark. The result of this comparative analysis is a set of
distance measures (B) used to estimate the between-source
variability given the trace/mark. This amounts to estimating the
distribution of the distance measures that can be obtained when
the trace/mark is compared to the alternative sources of the
potential population database. The between-source variability
matches the relative frequency of the evidence in the potential
population, in the limit of the P database in which it is observed
approaching that of the full population. The calculated between-
source variability is then used to estimate the denominator of the
likelihood ratio P (E | Hd).
p H
p
E, I
( )
p H
d
E, I
( )
p EH
p
, I
( )
p EH
d
, I
( )
p H
p
, I
( )
p H
d
, I
( )
a posteriori
probability ratio
posterior odds
likelihood ratio
a priori
probability ratio
prior odds
= x
= x
D Meuwly
Page 211
Estimation of the within-source variability
and calculation of the numerator of the LR
The putative source reference database (R) is made up of
information that ideally contains the exhaustive characteristics
of interest of the putative source. For individualisation from
biometric data this database can be made up of rolled inked
fingerprints pictures, 2D or 3D pictures of the face or
spontaneous speech audio recordings. The characteristics (y)
extracted from the putative source Y serves to calculate the
evidence (E) when they are compared to the characteristics of
the trace/mark (x).
The putative source control database (C) is made up of
information that is ideally of the same quality as the trace/mark,
but originated from the putative source. For individualisation
from biometric data, it can be fingermarks of the putative source
detected on the same surface as the trace/mark, pictures of the
face of the putative source taken in similar conditions as the
trace/mark or speech audio utterances of the putative source
recorded in similar conditions as the trace/mark. These pseudo-
marks are used to evaluate the within-source variability of the
putative source when the characteristics of the samples of the C
database are compared to the characteristics of the putative
source.
The result of this comparative analysis is a set of distance
measures (W) used to estimate the within-source variability of
the putative source. It involves estimating the distribution of the
distance measures that can be obtained when the putative source
and a pseudo-trace/mark of the same origin are compared. This
Table 1 Framework for forensic individualisation from biometric data
D Meuwly
Page 212
distribution matches the within-source variability of the putative
source, in the limit of the C and R databases in which it is
observed, approaching the real variability of the putative source.
This calculated within-source variability is then used to estimate
the numerator of the likelihood ratio P (E | Hp).
Evaluation of the strength of the evidence
The evaluation of the likelihood ratio of the two hypotheses
results from the calculation of P (E | Hp) / P (E | Hd). The
strength of evidence can be expressed by numerical values,
but it can also be expressed by linguistic qualifiers, which
report the amount of support of the evidence E for the
hypothesis Hp against the hypothesis Hd, following a
qualitative scale of verbal equivalents corresponding to values
of likelihood ratios [25].
Conclusion
In forensic science, the concept of identity is related to the
identity of source. Establishing the identity of a source refers to
an individualisation process, which consists of determining if a
particular individual entity is the source of a trace/mark. As the
continuity in time of a source and a trace cannot be proved in
forensic science, the identity of source is not absolute and can
only be inferred.
The assumption of uniqueness, often considered as a principle
in forensic science, is not satisfied for the traces/marks and
some of the sources considered for forensic individualisation
from biometric data. This assessment does not mean that
fingermarks, face traces and voice traces are not usable for
forensic individualisation from biometric data, but it clearly
lays down limits in the certainty that can be expected,
depending on the nature of the source, on the quality of the
trace/mark and on the method chosen for the analysis. One, in
general, cannot deduce the source of a trace.
The logical consequence is that the individualisation process
used for forensic individualisation from biometric data cannot
be simply considered as a comparative analysis of the
characteristics of a source and a trace/mark followed by a
binary decision of identification. It should be considered
according to the hypothetical-deductive method, whose
highlights are its taking into account of prior knowledge, its
requirement to consider all plausible hypotheses to explain the
source of a trace/mark and its power to test them. The
framework proposed to design a computer-based system for
forensic individualisation from biometric data gives a possible
framework to apply the hypothetical-deductive method.
In practice, probabilistic statements about identity, that can be
substantiated by forensic science, must be distinguished from
categorical statements about identity that can be substantiated
at best by a combination of forensic science and opinion.
Unfortunately the belief that forensic individualisation rests on
a deductive reasoning is still present in the forensic
community, where the misleading terminologies of fingerprint
(instead of fingermark), voiceprint or faceprint are still
commonly used.
Aknowledgments
The author would like to express his gratitude towards Dr Ton
Broeders, Dr Charles Berger and Dr Arent de Jongh from the
Netherlands Forensic Institute (NFI), for their help in reviewing
this document.
References
1 McRoberts FS. Mills S, and Possley M Forensics under the Microscope, Unproven
Techniques Sway Courts, Erode Justice, in Chicago Tribune. 2004: Chicago.
2 Stacey R A report on the Erroneous Fingerprint Individualization in the Madrid Train
Bombing Case. Journal of Forensic Identification, 2004. 54(6): p. 706718.
3 Kirk P The Ontogeny of Criminalistics. The Journal of Criminal Law, Criminology and
Police Science, 1963. 54: p. 235238.
4 Kwan Q Inference of Identity of Source, in Department of Forensic Science. 1977,
University of California: Berkeley, CA. p. 179.
5 Locard E LIdentification des Rcidivistes. 1909, Paris: A. Maloine.
6 Tuthill H Individualization: Principles and Procedures in Criminalistics. 1994, Salem:
Lightning Powder Co.
7 Taroni F La Recherche et la Gestion des Liens dans lInvestigation Criminelle: Une
Etape vers lExploitation Systmatique des Donnes de Police, in Institut de Police
Scientifique et de Criminologie. 1997, Universit de Lausanne: Lausanne. p. 253.
8 Evett I Expert Evidence and Forensic Misconceptions of the Nature of Exact Science.
Science and Justice, 1996. 36(2): p. 118122.
9 Champod C, and Meuwly D The Inference of Identity in Forensic Speaker
Recognition. Speech Communication, 2000. 31(2-3): p. 193203.
10 Champod C, and Evett I A Probabilistic Approach to Fingerprint Evidence. Journal
of Forensic Identification, 2001. 51(2): p. 101122.
11 Locard E Policiers de Roman et Policiers de Laboratoire. 1924, Paris: Payot.
12 Eco U The Name of the Rose. 1st ed. 1983, San Diego: Harcourt Brace Jovanovich.
13 Verdan A Karl Popper ou la Connaissance Sans Certitude. 1991, Lausanne: Presses
polytechniques et universitaires romandes.
14 Doddington G Speaker Recognition - Identify People by their Voices. Proc. IEEE,
1985. 73(11): p. 1651.
15 Bolt R et al Speaker Identification by Speech Spectrograms: A Scientists View of its
Reliability for Legal Purposes. J. Acoustic. Soc. Am., 1970. 47(2): p. 597612.
16 Grieve D Built by many hands. Fingerprint Whorld, 2000. 26(100): p. 5160.
17 Nolan F Forensic Phonetics. Journal of Linguistics, 1991. 27: p. 483493.
18 Clifford B Voice Identification by Human Listeners: on Earwitness Reliability. Law and
human behaviour, 1980. 4(4): p. 373394.
19 Boves L Commercial Applications of Speaker Verification: Overview and Critical
Success Factors. in RLA2C Workshop: Speaker Recognition and its Commercial and
Forensic Applications. 1998. Avignon.
20 Marquis P Sur les Preuves non Dductives en Intelligence Artificielle, in :(Ed.), 1999,
in Le Concept de Preuve la Lumire de lIntelligence Artificielle, S. J and J.
Szczeciniarz Editors. 1999, Presses Universitaires de France: Paris. p. 139158.
21 Conan Doyle A The Sign of Four, in The Complete Sherlock Holmes. 1953,
Doubleday & Company, Inc.: New York. p. 91155.
22 Truzzi M Sherlock Holmes, in The sign of three, Eco U, and Sebeok TA, Editors.
1983, Indiana University Press: Bloomington. p. 5580.
23 Stoney DA What Made Us ever Think We Could Individualize Using Statistics. Journal
of The Forensic Science Society, 1991. 31(2): p. 197199.
24 Taroni F, Champod C, and Margot P Forerunners of Bayesianism in Early Forensic
Science. Jurimetrics Journal, 1998. 38: p. 183200.
25 Evett I Toward a Uniform Framework for Reporting Opinions in Forensic Science
Casework. Science & Justice, 1998. 38(3): p. 198202.
26 Champod C, and Margot P Computer Assisted Analysis of Minuti Occurences on
Fingerprints, in International Symposium on Fingerprint Detection and Identification,
Almog J, and Springer E, Editors. 1995, Israel National Police: Neurim, Israel, 2630
June 1995. p. 305318.
27 Meuwly D Reconnaissance de Locuteurs en Sciences Forensiques: lApport dune
Approche Automatique, in Institut de Police Scientifique et Criminologie. 2001,
Universit de Lausanne: Lausanne. p. 272.
28 Champod C, and Evett I, Earmarks as Evidence: A Critical Review. Journal of
Forensic Sciences, 2001. 46(6): p. 12751284.
29 Bremermann H What Mathematics Can and Cannot Do for Pattern Recognition, in
Pattern Recognition in Biological and Technical Systems, OGrusser, Editor. 1971,
Springer-Verlag: New York.
30 Thvenaz P Rsidu de Prdiction Linaire et Reconnaisance de Locuteurs
Indpendante du Texte. 1993, University of Neuchtel, Switzerland.
D Meuwly
Page 213
Evidence evaluation: A response to the court of appeal judgment in R v T
Charles E.H. Berger
a
, John Buckleton
b
, Christophe Champod
c
, Ian W. Evett
d,
, Graham Jackson
e
a
Netherlands Forensic Institute (NFI), PB 24044, the Hague, the Netherlands
b
Environmental Science & Research, PB 92021, Auckland, New Zealand
c
Ecole des Sciences Criminelles, Institut de Police Scientique, Universit de Lausanne, BCH, 1015 Lausanne-Dorigny, Switzerland
d
Forensic Science Service, London Laboratory, 109 Lambeth Road, London SE1 7LP, United Kingdom
e
University of Abertay Dundee, DD1 1HG, United Kingdom
a b s t r a c t a r t i c l e i n f o
Article history:
Received 11 January 2011
Received in revised form 2 March 2011
Accepted 19 March 2011
Keywords:
Forensic Science
Interpretation
Bayesian inference
Likelihood ratio
Footwear
This is a discussion of a number of issues that arise from the recent judgment in R v T [1]. Although the
judgment concerned with footwear evidence, more general remarks have implications for all disciplines
within forensic science. Our concern is that the judgment will be interpreted as being in opposition to the
principles of logical interpretation of evidence. We re-iterate those principles and then discuss several
extracts from the judgment that may be potentially harmful to the future of forensic science. A position
statement with regard to evidence evaluation, signed by many forensic scientists, statisticians and lawyers,
has appeared in this journal [2] and the present paper expands on the points made in that statement.
2011 Forensic Science Society. Published by Elsevier Ireland Ltd. All rights reserved.
1. Introduction
The judgment in R v T [1] (freely available at www.bailii.org) has
caused considerable concern amongst forensic scientists within the
UK and overseas. Apart fromthose aspects of the judgment in relation
to the case at hand on which we do not comment in any regard
remarks were made of a more general nature and we consider it
essential that we respond to them in the open literature. Para 1 of the
judgment states:
The appeal raised an issue of some importance in relation to the
use of likelihood ratios in the provision of an evaluative opinion
where the statistical data available were uncertain and incom-
plete.
And at para 15:
The real issue in relation to footwear marks was the use of
likelihood ratios in forming an evaluative opinion on the degree of
likelihoodthat amarkhadbeenmadebya particular itemof footwear.
The Court formed the view that there were inadequate data
available for use in forming an evaluative opinion and this led to a
view that seems encapsulated in the following extract from para 86:
An attempt to assess the degrees of probability where footwear
could have made a mark based on gures relating to distribution
is inherently unreliable and gives rise to a verisimilitude of
mathematical probability based on data where it is not possible to
build that data in a way that enables this to be done; We are
satised that in the area of footwear evidence, no attempt can
realistically be made in the generality of cases to calculate the
probabilities. The practice has no sound basis.
And, in para 95:
In our judgment, an expert footwear mark examiner can therefore
in appropriate cases use his experience to express a more denite
evaluative opinion where the conclusion is that the mark could
have been made by the footwear. However no likelihood ratios or
other mathematical formula should be used in reaching that
judgement
1
for the reasons we have given.
Science and Justice 51 (2011) 4349
Corresponding author. Tel.: +44 7552616968.
E-mail address: ianevett@btinternet.com (I.W. Evett).
1
The dictionary offers the two spellings, judgment and judgement. It is
convention to use the former in the context of legal judgment. The latter seems
preferred when talking more generally about human judgement. In quotations, we
have followed the spelling as it occurs in the judgment.
1355-0306/$ see front matter 2011 Forensic Science Society. Published by Elsevier Ireland Ltd. All rights reserved.
doi:10.1016/j.scijus.2011.03.005
Contents lists available at ScienceDirect
Science and Justice
j our nal homepage: www. el sevi er. com/ l ocat e/ sci j us
It is this last sentence that raises our principal concerns: it implies
that a likelihood ratio may not underpin the evaluation of a forensic
scientist unless the two key probabilities can be assigned by purely
statistical methods. In spite of the fact that the judgment states (para
32) that We are solely concerned in this appeal with the use (of
likelihood ratios) in relation to footwear evidence, nevertheless it
seems to us inevitable that it will be cited to support court
submissions in relation to other evidence types.
The reason for this paper is to explain that the evaluation of
evidence for a court of law is not just a matter of using likelihood
ratios but one of working to a set of principles that are founded on
logic. To deny scientists the contemplation of the likelihood ratio
whether quantitative or qualitative is to deny the central element of
this logical structure.
The phrase likelihood ratio and the role of Bayes' theorem in
establishing a logical framework for forensic inference now appear so
widely in the forensic science literature that we do not consider it
necessary to explain the mathematical background. In the next
section, we give only a brief overview of the logical structure and we
urge any reader who is unfamiliar with its basis to refer to standard
texts particularly Robertson and Vignaux [3], Aitken and Taroni [4]
and Aitken, Roberts and Jackson [5]. Whereas we have chosen to
address the readership of Science and Justice, papers dealing with the
same judgment have been submitted to legal journals by Redmayne
et al. [6] and by Robertson, Vignaux and Berger [7].
After outlining the basic principles we briey explain how what
has come to be known as the Bayesian approach has evolved. We
then explain a little of the notion of probability and the relationships
between probability, data, statistics, knowledge and experience. This
leads to a discussion of presenting an evaluation to a court. Finally, we
make remarks that seem relevant to us about related issues such as:
the transposed conditional; the distinction between the Bayesian
approach and Bayes' Theorem; and exposure to wider debate.
2. The logic of evaluating forensic science evidence in court
We emphasise that we are considering the evaluation of evidence
at court; it often happens that a scientist is required to advise on the
inuence that his observations might have on the conduct of a police
investigation. The requirements of this latter investigative function
(described in [8]) are not considered in this paper.
We consider the case where a scientist is required to assist a court
of lawby providing an evaluative opinion with regard to the assistance
that a set of scientic observations might provide in addressing
matters that the court is deliberating. We conne ourselves to criminal
lawbut believe that the principles extend analogously to the civil law.
For the purpose of simplication we consider the situation where a
single defendant is on trial for a stated offence. Then the scientist's
evaluation should be governed by the following principles:
1. Interpretation must be carried out within a framework of
circumstances. It is necessary for the scientist to make clear her
understanding of those aspects of the circumstances that are
relevant to the evaluation.
2. It is necessary for the scientist to consider her observations in the
light of propositions that represent both the prosecution and defence
views. The propositions must be clearly stated and are subject to
change at the direction of prosecution, defence or the court.
3. It is necessary for the scientist to consider the probability that the
observations would have been made given that the prosecution
proposition were true; and the probability that the observations
would have been made given that the defence proposition were
true. The relative magnitude of these two probabilities, known as
the likelihood ratio, determines the assistance provided by the
observations in weighing the two propositions against each other.
The logical reasoning for these principles is founded on Bayes'
theorem. The approach also invokes notions relating to the nature of
probability that we explain in the next section. We note that these
principles are the foundation of the standards for evidence interpre-
tation that have recently been adopted in the UK by the Association of
Forensic Science Providers [9].
In the UK and the USA, the development of Bayesian methods for
interpreting forensic science evidence was initially inspired by a
paper by Finkelstein and Fairley in the Harvard Law Review of 1970
[10]. From the latter half of the 20th century and the rst decade of
the 21st the papers that employ Bayesian reasoning in some form or
another for forensic science problems number in the hundreds.
However, research at the University of Lausanne, published in 1998
[11] demonstrated that this kind of reasoning had been employed
some ninety years earlier. A central element in the notorious
conviction for treason in 1894 of Alfred Dreyfus was a document
that was known as the Bordereau. An opinion with regard to the
authenticity of the Bordereau was given by Bertillon, regarded by
many as one of the founding fathers of forensic science, who
concluded that it was a forgery. As part of a review of the conviction
of Dreyfus, the French Academy of Sciences set up a committee,
composed of Darboux, Appell and Poincar, which concluded in 1908
that the approach which Bertillon followed in forming his opinion
had been inherently awed. Although couched in language that
today appears arcane, it is perfectly clear that their reasoning
was what we would nowcall Bayesian though that termwas not in
use then.
A similar perspective has been provided by the eminent legal
commentators Bender, Nack and Treuer [12] in the German system
of justice. They argue (592) that there are three key questions
that should guide any judge for the assessment of any piece of
evidence:
(i) How common is the evidence if the main proposition were
true?
(ii) How common is the evidence if the alternative proposition
(converse of the main proposition) were true?
(iii) In which of the above cases is the evidence more likely? The
asymmetry between both assessments provides the weight of
evidence.
Although expressed in completely different terms, the connexion
between these three questions and the principles that we stated
earlier should be obvious.
A similar position is adopted in McCormick [13], which explains
(185) how the likelihood ratio indicates the probative value of
circumstantial evidence.
3. Probability
3.1. Aleatory and epistemic probabilities
Perhaps the most important notion to recognise in understanding
probability is that all probabilities are conditional. To assign a
probability for the truth of a proposition, one needs information.
Even if one considers something as simple as the toss of a coin, to
assign a probability to the outcome one needs to know something
about the physical nature of the coin and how it is to be tossed. In
general, the tossing mechanism (such as before the start of a sporting
event betweentwo teams) is chosen so as to maximise the uncertainty
of the outcome. Given such information, most would be content with
the statement that the probability of a head is 0.5. In another kind of
uncertainsituation one might, as the Court was clearly aware, study an
appropriate collection of data to condition a probability. The Court was
content with the notion that this is properly and reliably done in the
case of calculating a DNA match probability.
44 C.E.H. Berger et al. / Science and Justice 51 (2011) 4349
However, there are still further situations where the notionof using
a database appears unrealistic. Consider, for example, the question:
What is the probability that Mary, Queen of Scots,
2
knew of the
plot to murder her husband?
Clearly, to address such a question one would wish to address
historical sources. These may be extremely extensive, complicated
and of varying degrees of veracity. It is not surprising, then, that
different people would give different answers to the question. For the
best answer one would wish to look for the historian who is
considered the most knowledgeable on the life of Mary. Similarly, one
would not be surprised to see different historians having different
views on the matter. It is this kind of example that makes it clear to us
that probability is personal.
Probabilities such as that in the coin tossing example are known as
aleatory: they arise in situations where the uncertainty derives solely
from the randomness of the conditions that pertain. On the other
hand, the latter example is of an epistemic probability: the uncertainty
derives solely from limitations of knowledge.
A DNA match probability, when it is calculated from undisputed
genetic principles and using a database that is indisputably relevant to
the case at hand, could be argued to be a function of purely aleatory
uncertainty (though, as we see in the next section, this is a matter for
debate). On the other hand, when a handwriting expert is considering
the comparison of features between two sets of handwriting then any
probabilities assigned to variations in features are purely epistemic. In
more general usage, it is customary to call aleatory probabilities
objective, in the sense that their magnitudes are independent of
human judgement. Epistemic probabilities, on the other hand, are
called subjective in the sense that they are personal, depending as
they do on human judgement. Any reader who seeks a comprehensive
non-mathematical introduction to the nature of probability could do
no better than the book Understanding Uncertainty by Lindley [14]
who introduces the concept as a measure of degree of belief.
Whereas much of the logic of statistical inference in the 20th
century concerned itself entirely with aleatory probabilities, there
grew an increasingly substantial school of thought that was founded
on the realisation that it was possible to apply all of the richness of
statistical and probabilistic reasoning to epistemic, or subjective,
probabilities. Gradually, and for reasons that are not entirely clear
(see, in particular, Fienberg [15]), this came to be known as the
Bayesian approach.
3.2. Probability and statistics
The Court in R v T appeared to take the view that assigning
probabilities is a matter of statistics alone. For example, the judgment
says, at para 77:
It is clear that in DNA cases, there has been for sometime a
sufcient statistical basis that match probabilities can be given.
And, at para 78:
However, no case was drawn to our attention which suggests that
a mathematical formula is appropriate where it has no proper
statistical basis.
And, at para 80:
An approach based on mathematical calculations is only as good
as the reliability of the data used. The acceptance of a
mathematical approach to the calculation of a match probability
in DNA cases is based on the reliability of the statistical database,
though an element of judgment is required (emphasis added).
These extracts emphasise the notionof a probability being assigned
from data and a statistical calculation of some kind. However, even
here as the Court recognised in the italicised clause in the third
extract there is an essential contribution to be made by personal
judgement. Underlying the calculation of a DNA match probability is a
statistical/population genetic model that cannot be anything other
than an approximation to the complexities of the real world. The
application of the model will incorporate estimates of unknown
parameters that are themselves approximations. Even with DNA the
element of judgment might be wider than the Court appreciated. For
example, if there is a full tenlocus matchbetweenthe DNAprole from
a crime sample and the DNA prole of a known individual, in the UK
the scientist will report a match probability of one in a billion. In the
USAit wouldnot be surprising to ndthe matchprobability, for exactly
the same prole, reported as one in a trillion even if the same
database were used. There is much more to this issue than a simple
statistical calculation, see, for example, Evett et al. [16] and Buckleton
[17]. Indeed, it takes judgement to decide how to model a
phenomenon and individuals may disagree about the appropriateness
of any particular model.
Para 1 of the judgment (see Introduction) refers to data that were
uncertain and incomplete. The Court clearly felt this to be so in
relation to the case at hand, but the essential point to recognise here is
that whenever we are making an inference froma sample the data are
always an incomplete representation of the full picture; furthermore,
their relevance is a matter of judgement and the uncertainty that
concerned the Court is an unavoidable feature of such inference. The
probability that is quoted then will inevitably be a personal
probability and the extent to which the data inuence that probability
will depend on expert judgement. This is not a process that can be
governed strictly by mathematical reasoning but this does not make it
any less scientic: scientists are called on to exercise personal
judgement in all aspects of their several pursuits. See Taroni et al. [18]
for a cogent discussion of this.
In relation to footwear evidence, the judgment says, at para 86:
we have concluded that there is not a sufciently reliable basis
for an expert to be able to express an opinion based on the use of a
mathematical formula.
It is our understanding of the judgment that the formula referred
to here was one that was used in the individual case to calculate a
relative frequency. As we have said, we intend no reference to the
specics of the case itself. More generally, it is our position that the
scientist considers both of the probabilities that constitute the
likelihood ratio and, if data exist that are of some relevance then it
is unreasonable to deny her the facility to take account of that data by
exercising reasonable scientic judgement. This is not a matter of a
mathematical probability (which we take to mean an aleatory
probability) but an epistemic probability that is conditioned by all of
the data and information that are available to the expert. The
judgment in R v Weller [19] is in favour of this kind of view. The
recognition of subjective probabilities as a sound basis for offering
expert opinion should not be misconstrued as being an open door to
unstructured reasoning and ad hoc guesses. Any appraisal offered,
based on the spectrum of knowledge an expert may call upon, should
be transparent and subject to peer review and challenge. We stress
that the probability may be informed not only by systematic research
and data but also by expert judgement, though we recognise that the
notion of what constitutes expert judgement would justify a
separate paper entirely.
2
Readers unfamiliar with this queen will nd a short biography at http://en.
wikipedia.org/wiki/Mary_Queen_of_Scots.
In addition to its position with regard to assigning probabilities by
statistics, the Court expressed concerns about database size. Para 84
says:
Use of the FSS's own database could not have produced reliable
gures as it had only 8122 shoes whereas some 42 million are
sold every year.
Yet the Court was clearly convinced of the robustness of DNA
statistics that yieldmatchprobabilities of the order one ina billionbased
ondatabases that are typically made upfroma fewhundred individuals.
Is a database of 8122shoes toosmall?Wedon't thinksoone thingthat
seems inescapable is that it is larger than a database of zero. Any
knowledge is better than no knowledge and the Court surely could not
be suggesting that the data should be ignored. Of course, it must be
treated with caution and its limitations should be recognised but that
is what scientists are trained to do. Para 86 of the judgment says that in
the area of footwear evidence, no attempt can realistically be made in
the generality of cases to use a formula to calculate the probabilities.
But we hope that we have explained that probabilities are assigned
rather than calculated: that means using whatever data are available
sensibly and exercising judgement with regard to limitations of size and
representativeness.
3.3. The likelihood ratio
We have seen that the Court of Appeal is content with the notion of
likelihood ratios when they perceive them, as with DNA, as deriving
from purely mathematical methods applied to reliable databases.
Although we have explained that this view is illusory, for the sake of
discussion, we will call this Type A scientic evidence: anunattainable
ideal where the probabilities that form the evaluation are aleatory.
At the other extreme is scientic evidence of the kind that the
Court considered in reference to the previous judgment in R v Atkins
and Atkins [20] which concerned facial comparison evidence. We will
call this extreme Type C scientic evidence. Para 93 of the judgment
includes:
The court was again holding that evaluative evidence could be
given; there was no issue as to the use of a likelihood ratio, as it
was clear that there was no statistical database.
The inescapable conclusion to draw from this is that the Court saw
the likelihood ratio as something that derives only from a calculation
of some sort. However, as we have explained, the likelihood ratio is
the ratio of two probabilities and there is nothing in the logical
framework that we have described that demands those two
probabilities to be aleatory. Indeed, the foundation of the Bayesian
paradigm (from which the forensic Bayesian approach derives) is
that the logic works equally well with purely epistemic probabilities
as it does with aleatory probabilities: the depth and richness of the
Bayesian paradigm derives entirely from this profound tenet. This is
what is Bayesian about the Bayesian approach.
With regard to this type C evidence we would still consider it
necessary for the expert to adhere to the logical principles stated in
section1: if she didotherwisethenshe wouldhave abandonedlogic. The
principles lead inevitably to the two key probabilities and we entirely
accept that the expert will not be able to assign precise values to them.
Nevertheless, it is essential for the expert to address qualitatively their
relative magnitudes. Under whichof the two propositions is the evidence
more probable? That is the proposition which is better supported by the
scientic observations. The likelihood ratio is still central, even if the
probabilities can only be qualitatively assessed though it is worth
reiterating that the expert needs to be able to state the basis for such an
assessment. We fully endorse thenotionof transparency inthis regard: it
is not acceptable merely to cite years of service, numbers of previous
cases and to expect the Court to accept ipse dixit.
Betweenthe extremes of types AandCwe haveevidencetypes where
the scientist's opinion will depend in part on data, part on published
material and part onknowledge and judgement. Not surprisingly, we call
this type B and it is to this category that footwear evidence belongs along
with, in particular, bres and glass evidence. In relation to the latter, the
judgment cited R v Abadom [21] and said, at para 99:
The decision of the court in Abadom as long ago as 1982 explained
the importance of referring to all the material so that the cogency
and probative value of the conclusions can be tested and
evaluated by reference to it.
Whereas the judgment is addressing a disclosure issue here, it is a
recognition of the multi-faceted approach to a type B evidential
assessment. However, the judgment also says at para 91:
It is important to note as happened in R v Abadom that an expert
can give an opinion using a statistical database by simply using
that database and expressing an opinion by reference to it,
without recourse to the type of mathematical formula used in this
case or to any form of Bayes theorem.
The mathematical formula that the judgment refers to appears to
be a relative frequency calculation that was put on the le for this
particular case and we repeat that we make no comment on that case.
However, we are not comfortable with the lack of clarity as to howthe
expert might, without performing any calculations, use data to inform
an opinion. The idea that an expert might just look at a database and,
by some undened mental process, form an opinion does not seem
right to us and seems counter to the requirement for transparency
that is rightly called for elsewhere in the judgment. At the very least,
simple calculations of proportions would seem desirable.
4. Presenting an evaluation at court
4.1. The verbal scale
Having followed the principles of evidence evaluation and arrived
at a likelihood ratio, whether qualitative or quantitative, there
remains the question of how this is to be presented to a court. The
notion that evidence might support a particular proposition is
commonly encountered in the courtroom. For example, these are
just a small sample from the Court of Appeal judgments:
In R v Denis John Adams [22]
None of these provides any support for Mr Thwaites' proposition.
In R v Gilfoyle [23]
the material before him demonstrated convincing support for
the deceased having taken her own life.
In R v Puaca [24]
But what they did say, and say most forcefully, was that there was
no pathological evidence to support Dr Heath's view.
If the probability of the observations is greater given the prosecution
proposition than it is given the defence proposition it is intuitively
reasonableandlogically justiable tostatethat the observations support
the former. The notion of something being added to one of the scales of
justice is apt. However, we are well aware that some items of evidence
might contribute to the resolution of the two propositions more than
others: we tend to think of DNA evidence as having the potential to be
extremely strong; whereas evidence from a basic visual comparison of
hairs, for example, might be relatively weak.
In those cases where a quantitative likelihood ratio has been
calculated, the interpretation of R v Doheny and Adams [25] currently
in England and Wales suggests that it is the number alone that should
be put to the jury. However, the solution for all those cases where the
likelihood ratio is qualitative is the notion of using some kind of verbal
qualier to convey the strength of the support that the observations
bring. The approach is endorsed by Kaye et al. [26] (15.6). Forensic
scientists have always recognised the desirability of maintaining some
sort of consistency of language between experts, disciplines and
organisations; this, in turn, inspires the wish to standardise, as far as
reasonable, on a small number of qualiers. This notion of a verbal
scale was recognised as valid in R v Atkins and Atkins [20]. R v T accepts
the principle but denies that it has anything to do with the likelihood
ratio (para 93). On the contrary, it is our position that the concept of
the likelihood ratio should be central to every logical evaluation,
irrespective of the extent to which the expert's judgement has been
informed by data.
A verbal scale has been in use in the Forensic Science Service (FSS)
for many years and the version that was current at the time of the trial
is reproduced in para 31 of the judgment. The expressions of support
range from weak support, through strong support to extremely
strong support. Because of the desire to achieve a measure of
consistency across all evidence types, broad ranges of values of the
likelihood ratio have been assigned to the various expressions. But this
should not be taken to mean (as implied at para 31 of the judgment)
that a numerical value must be derived in a given case in order that a
verbal expression might be selected even if that were preferable
whenever possible. In the case of type C assessments, the verbal scale
restricts the expert to the specied range of qualiers and provides a
rough ordering that, at best, implies that strong is stronger than
moderate but not as strong as very strong. We recognise that there
is much scope for improvement but the limitations derive not from
the verbal scale per se, rather fromthe complexities of the observations
and the lack of data for interpreting them.
Where type B assessments are concerned the extent to which the
likelihood ratio is quantied will vary from case to case, depending
on the expert's assessment of the quality of whatever data are
available. But it is not the verbal scale that is driving things. The
assessment comes rst, the decision about a verbal qualier comes
later.
4.2. The use of could have
The Court's endorsement of the phrase could have runs counter,
in our view, to what is said in R v Puaca [24]. We quote frompara 42 of
that judgment:
Whereas inconsistency is often probative, the fact of consis-
tency is quite often of no probative value at allWe consider that
there is a very real danger in adducing before a jury dealing with a
case such as the present evidence of matters which are
consistent with a conclusion, at least unless it is to be made
very clear to them that such matters do not help them to reach
the conclusion. If it is introduced in evidence, and particularly if it
is given some emphasis, a jury may well think that it assists them
in reaching a conclusion: for why otherwise are they being told
about it?
In our opinion, the same observations may be made in relation to
the use of could have. To say that a shoe could not have made a
mark has clear probative value. To say that a shoe could have made a
mark is unbalanced unless one adds that it is also true that some other
shoe could have made the mark. In effect, one is addressing two
propositions but giving no indication as to which is supported by the
evidence. The judgment says at para 73:
The use of the term could have made is a more precise
statement of the evidence; it enables a jury better to understand
the true nature of the evidence than the more opaque phrase
moderate [scientic] support.
On the contrary, our viewis that could have made is valueless for
expressing evidential weight and this view is shared by others. For
example, Lyon and Koehler advocate the use of a likelihood ratio
instead of poorly informative terms such as consistent with for
expert witnesses offering opinions in sexual abuse cases [27].
In most situations could have is no more than a statement of the
blindingly obvious such as:
The DNA in the crime sample has the same prole as that of the
suspect, therefore it could have come from him.
The latter part of this sentence does not warrant the status of an
expert opinion; whereas the jury will understand it, it does not convey
any appreciation of the weight of evidence that the observations
provide to assist them in their task. Moderate support is an attempt,
however imperfect, to convey some impression of weight of evidence.
In a public inquiry that followed the miscarriage of justice against Guy
Paul Morin in Canada, the Kaufman report [28] revealed major
shortcomings in laboratory work and interaction between experts, police
and judges with regard to hair and bre evidence. The report noted that
the use by experts of vague conclusions such as the hair might come
from, the bres are consistent with or the bres match, which
lend themselves to a range of interpretations, was one of the causes of the
attributionof disproportionate probative value to evidence whichactually
made only a limited contribution to the establishment of the facts.
The recent report of the US National Academy of Sciences [29] also
drewattention to the dangers of using tendentious, or even fallacious,
terminology, to the extent that its authors felt the need to make a
specic recommendation on the subject. The report specically
referred to Aitken and Taroni [4], Evett et al. [30] and Evett [31] as
providing the essential building blocks for the proper assessment and
communication of forensic ndings.
The warmth with which the Court endorsed a could have
statement is in stark contrast with many other rulings in the DNA eld
(e.g. [32,33] Section III and more generally in Kaye et al. ([26] (15.4)).
These rulings hold that a statement of non-exclusion, which we
equate with the could have statement, is not admissible unless a
more precise expression of weight of evidence is available.
4.3. The transposed conditional
The second logical principle, in a footwear case for example, would
direct the scientist to address two questions, one of which would be:
What is the probability that those observations would have been
made if the suspect's shoe made the crime mark?
For an understanding of the logic of evidence evaluation, it is
necessary to recognise that this is not the same as:
What is the probability that the suspect's shoe made the crime
mark given that those observations were made?
Failure to recognise the fundamental difference between those
two probabilities (and the answers that they generate) is known to
statisticians as the fallacy of the transposed conditional. Within the
context of forensic science, it was dubbed the prosecutor's fallacy by
Thompson and Schuman [34]. See also [35].
The dangers of the fallacy have been recognised by the Court of
Appeal, within the context of DNA evidence, in the judgments in R v
Deen [36] and R v Doheny and Adams [25]. However, it appears not to
have been recognised by the Court in R v T as may be seen from the
following example in para 33:
In the present case it was expressed (i.e. the likelihood ratio) as the
probability that the Nike trainers owned by the appellant had
made the marks discovered at the scene divided by the probability
that the Nike trainers had not made the marks.
This is not right. The likelihood ratio in this context is the
probability of the observations given that the Nike trainers owned by
the appellant had made the marks; divided by the probability of the
observations given that the Nike trainers had not made the mark. The
likelihood ratio is always a function of the probabilities of the
observations; the quotation is expressed in terms of the probabilities
of the propositions.
Para 91 of the judgment says:
If there was a sufcient database in footwear cases an expert
might be able to express a view reached through a statistical
calculation of the probability of the mark being made by the
footwear, very much in the same way as in the DNA cases
This is yet another of the many instances in the judgment where a
transposed conditional is implied. The scientist should not address the
probability of the mark being made by the footwear rather,
the probability of the observations given that the mark was made by
the footwear and, of course, the probability of the observations given
the appropriate defence proposition.
4.4. The US examiners verbal scale
The judgment (para 64) cites a verbal scale that is followed by shoe
and toolmark examiners in the USA; this is quite different from that
described in Section 4.1. A report fromthe marks working group of the
European Network of Forensic Science Institutes (ENFSI) [37] includes
a similar scale.
The US examiners' scale embodies phrases and notions that have
been abandoned in the UK because of their potential to confuse. In
particular, one of the phrases is based on the use of could have,
which we have already dealt with in the previous section. Another
category is to say that the shoe probably made the mark. But this is
clearly a statement about the probability of the truth of a proposition.
There are two ways in which a statement can be arrived at. The rst, as
we have seen is to commit the fallacy of the transposed conditional.
The second is to follow a path that has been widespread practice in
paternity testing for many years and that is to assume that, based on
all of the evidence other than the scientic observations, the
prosecution proposition is just as likely to be true as the defence
proposition. The probably made conclusion then follows from an
application of Bayes' Theorem in a manner that was rejected in R v
Denis John Adams [22]. More details on this kind of approach can be
found in [38] and [39].
Indeed, were a statement of the kind probably came from used in
a DNA case, it would violate the judgment in R v Doheny and Adams
[25]. It is, perhaps, surprising that legal practitioners in the USA have
apparently not realised that the probably made style of opinion is
susceptible to challenge either on the basis of logic or on the basis of
inappropriate assumptions about the other evidence in the case.
Another category in the US examiners' scale is a categoric opinion
of the kind this shoe made that mark. Such opinions are, indeed,
expressed by UK examiners, but they do not form part of the verbal
scale, which is designed only for corroborative weight of evidence.
5. Other issues
5.1. The Court of Appeal and Bayesian inference
Para 90 of the judgment includes:
It is quite clear therefore that outside the eld of DNA (and possibly
other areas wherethere is a rmstatistical base), this court has made
it clear that Bayes theoremand likelihood ratios should not be used.
Para 89 explains that this view derives from R v Doheny and G
Adams [25] and R v Denis John Adams [22]. However, we should point
out that in the Doheny and G Adams trials, the weight of the DNA
evidence was conveyed by means of a match probability (the phrase
random occurrence ratio, which appears to have been coined for
reasons that are far from clear has, thankfully, not become general
use) not a likelihood ratio nor was Bayes' theoremused in any sense in
either case.
In R v DJ Adams, defence encouraged the jury to assign probabilities
to each item of non-scientic evidence in the case (geography, eye-
witness, alibi etc.) and to combine them by means of Bayes' theorem.
In no sense was Bayes' theorem used to evaluate the scientic
evidence in the case; nor was a likelihood ratio quoted again the
evaluation of the DNA evidence was based on a match probability.
It is a matter of concern to us that an exercise where Bayes'
theorem was employed for the non-scientic evidence in a trial has
been the cause of a direction against the notion of applying the
principles of logic to the evaluation of scientic evidence.
It may be useful to address a point that may have been a source of
confusion: there is a distinction to be made between using Bayes
theorem and following a Bayesian approach. To do the former, it is
necessary to assign prior probabilities and manipulate them, via the
likelihood ratio, to arrive at posterior probabilities. To follow the
Bayesian approach, however, one only has to follow the three
principles detailed earlier principles that can, in fact, be rationalised
without using Bayes theorem. As we have explained, the adjective
Bayesian has derived from the subjective view of probability, rather
than the mechanics of using Bayes' theorem.
5.2. Exposure to wider debate
Para 52 of the judgment refers to the adoption of the Bayesian
approach by the Association of Forensic Science Providers in the UK,
adding:
Despite inquiries made by us, it is not clear to what extent, if any, it
was subject to wider debate outside the forensic science community.
Apart from the substantial literature that has appeared in journals
in the forensic science, legal and statistical elds, the FSS has carried
out various consultative exercises. Considerable progress was made
towards implementing the principles of the Bayesian approach within
the FSS through the Case Assessment and Interpretation (CAI) project
and the work was described in a series of papers, including, for
example, Evett et al. [30]. As the project developed, the Crown
Prosecution Service (CPS) was consulted across England and Wales
about its implication for casework. The new statement format that is
described in [30] was not nalised until agreement with the CPS.
Starting fromaround 2000 and up to 2006 a lecture, primarily relating
to DNA but from a Bayesian perspective, was given regularly to
Continuing Education Seminars of the Judicial Studies Board. In this
way, several hundred judges from England and Wales were exposed
to the ideas and the feedback was positive.
As we said in the Introduction, papers involving the Bayesian
approachto forensic science number inthe several hundreds. Whereas
most will be in the forensic science literature, others have been
published in legal journals and others in the statistical literature
most notably the journals of the Royal Statistical Society.
Following the appeals in R v Clark [40], the Royal Statistical Society
set up a working group to deliberate on the use of statistics in legal
trials. The group comprises statisticians, forensic scientists, academic
lawyers, barristers and a judge. A report by Aitken et al. [5] explains
the Bayesian approach in some detail.
The eighth International Conference on Forensic Inference and
Statistics is to be held in Seattle, USA, in July 2011. This series, started
in 1990, and alternating its venues between the USA and Europe, aims
to bring together forensic scientists, statisticians and lawyers to
discuss matters relating to the interpretation of evidence, particularly
in criminal proceedings.
6. Conclusion
From the viewpoint of public perception, technological advances
have epitomised the evolution of forensic science over the last 20 years.
However, the development of its logical foundations and their practical
application that have taken place over a comparable period will, we
believe, be of even greater importance. The main thrust of the evolution
of what has come to be known as the Bayesian approach has centred on
the UK, Europe and Australasia. This is a movement that seeks to
contribute toimprovement, innovationanda searchfor howsciencecan
best inform the proceedings of a criminal justice system.
Forensic science is moving forward (in some countries more than
in others) from a craft towards a science, and an increasing level of
professionalism is required of those acting in it. Every claim of
improvement inevitably entails an admission that things were not as
good before. This however should never impede the possibility of
improvement. The Court rightly observed that the adoption of the
ideas of the Bayesian approach and, more specically the CAI model,
has been patchy in those countries that are leading the movement.
Furthermore, practitioners in the USA seem to be largely unaffected
by it. It is encouraging that the Board of ENFSI an organisation with
58 member laboratories in 33 countries has engaged itself to
implement the principles outlined in the position statement [2] in its
member laboratories. It is regrettable that the judgment in R v T will
inevitably be seen to be, and used as, a weapon for defending the
status quo or even a return to pre-scientic notions.
The judgment sees the likelihood ratio as something that inevitably
is founded on statistical calculations and data. We hope that we have
adequately explained that this is not the case. The likelihood ratio is a
central element in a framework for evaluative opinion that is both
logical and balanced. Furthermore, the probabilities that the scientist is
directed to address are always founded (even with DNA) on personal
judgement. This is not a bad thing, it is an inescapable feature of science
and it is why science is still a human endeavour. It is also why there is
always scope for honest disagreement between experts; as we have
seen, probabilities are personal, even when informed by data. The
Bayesian approach does not give the expert the right to make up
numbers as she pleases, but rather the duty to make sure the
probabilities involved obey the laws of probability. Overall, the duty of
transparency prevails and the expert should articulate the basis for her
probabilistic assignments in a given case.
In R v Denis John Adams [22] the Court of Appeal came out strongly
against the use of Bayes' theorem by the jury for evaluating non-
scientic evidence. There is a serious danger that the ruling in R v T
will be seen as a proscription of the Bayesian approach to evaluating
scientic evidence.
Acknowledgement
The authors are grateful to David Kaye and Paul Roberts for
comments on a draft of the paper.
References
[1] R v T, Court of Appeal Criminal Division, EWCA Crim 2439, 2010.
[2] Expressing Evaluative, Opinions: a position statement, Science & Justice 51 (2011)
12.
[3] B. Robertson, G.A. Vignaux, Interpreting Evidence Evaluating Forensic Science in
the Courtroom, John Wiley & Sons Inc., Chichester, 1995.
[4] C.G.G. Aitken, F. Taroni, Statistics and the Evaluation of Evidence for Forensic
Scientists, 2nd edJohn Wiley & Sons Ltd., Chichester, 2004.
[5] C. Aitken, P. Roberts, G. Jackson, Fundamentals of Probability and Statistical
Evidence in Criminal Proceedings Guidance for Judges, Lawyers, Forensic Scient-
ists and Expert Witnesses. London: The Royal Statistical Society, in press. http://
www.rss.org.uk/statsandlaw.
[6] M. Redmayne, P. Roberts, C. G. G. Aitken, G. Jackson, Forensic Science Evidence in
Question. Criminal Law Review (2011) 347356.
[7] B. Robertson, G. A. Vignaux, C. E. H. Berger, Extending the Confusion About Bayes,
Modern Law Review (2011) 444-455.
[8] G. Jackson, S. Jones, G. Booth, C. Champod, I.W. Evett, The nature of forensic sci-
ence opinion a possible framework to guide thinking and practice in investiga-
tions and in court proceedings, Science & Justice 46 (2006) 3344.
[9] Association of Forensic Science Providers, Standards for the formulation of
evaluative forensic science expert opinion, Science & Justice 49 (2009) 161164.
[10] M.O. Finkelstein, W.B. Fairley, A Bayesian approach to identication evidence,
Harvard Law Review 83 (1970) 489517.
[11] F. Taroni, C. Champod, P. Margot, Forerunners of Bayesianism in early forensic
science, Jurimetrics Journal 38 (1998) 183200.
[12] R. Bender, A. Nack, W.-D. Treuer, Tatsachenfeststellung Vor Gericht glaubwr-
digkeits- Und Beweislehre Vernehmungslehre, 3rd ed.Verlag C.H. Beck, Mnchen,
2007.
[13] K.S. Broun, R.P. Mosteller, G.E. Dix, E.F. Roberts, I. E. J, J.W. Strong, D.H. Kaye, E.
Swift, Mccormick on Evidence, Thompson/West Publishing Co., Eagan, MN, 2006.
[14] D.V. Lindley, Understanding Uncertainty, John Wiley & Sons, Inc., Hoboken, 2006.
[15] S.E. Fienberg, When did Bayesian inference become Bayesian? Bayesian Analysis
1 (2006) 140.
[16] I.W. Evett, L.A. Foreman, G. Jackson, J.A. Lambert, DNA proling: a discussion of
issues relating to the reporting of very small match probabilities, Criminal Law
Review (May 2000) 341355.
[17] J.S. Buckleton, Population genetic models, in: S.J. Walsh, C.M. Triggs, J.S. Buckleton
(Eds.), Forensic DNA Evidence Interpretation Methods and Interpretation, CRC
Press, Boca Raton, 2004, pp. 65122.
[18] F. Taroni, C.G.G. Aitken, P. Garbolino, De Finetti's subjectivism, the assessment of
probabilities and the evaluation of evidence: a commentary for forensic scientists,
Science & Justice 41 (2001) 145150.
[19] R. v Peter Weller, UK Court of Appeal, EWCA Crim 1085, 2010.
[20] R. v Dean Atkins, Atkins Michael, Court of Appeal Criminal Division, EWCA Crim
1876, 2009.
[21] R v Abadom, Court of Appeal Criminal Division, [1983] 1 Wlr 126, [1983] 1 All Er
364.
[22] R. v Denis John Adams, Court of Appeal Criminal Division, EWCA Crim 222, 1996.
[23] R. v Norman Edward Gilfoyle, Court of Appeal Criminal Division, EWCA Crim81,
2000.
[24] R. v Steven Puaca, Court of Appeal Criminal Division, EWCA Crim 3001, 2005.
[25] R. v Alan James Doheny, R. v Gary Adams, Court of Appeal Criminal Division,
EWCA Crim 728, 1996.
[26] D.H. Kaye, D.E. Bernstein, J.L. Mnookin, The NewWigmore, a Treatise on Evidence:
Expert Evidence, 2nd ed.Aspen Publishers Inc., New York, 2011.
[27] T.D. Lyon, J.J. Koehler, The relevance ratio: evaluating the probative value of expert
testimony in child sexual abuse cases, Cornell Law Review 82 (1996) 4378.
[28] F. Kaufman, Report of the Kaufman Commission on Proceedings Involving Guy
Paul Morin, Ministry of the Attorney General, Ontario, Toronto, 1998.
[29] National Research Council, Strengthening Forensic Science in the United States: A
Path Forward, The National Academies Press, Washington, D.C., 2009
[30] I.W. Evett, G. Jackson, J.A. Lambert, S. McCrossan, The impact of the principles of
evidence interpretation on the structure and content of statements, Science &
Justice 40 (2000) 233239.
[31] I.W. Evett, The theory of interpreting scientic transfer evidence, Forensic Science
Progress 4 (1990) 141179.
[32] State of Western Australia v Dair, WADC 157, 2006.
[33] W.C. Thompson, Motion to Exclude DNA Evidence in United States V Cuff,37 F.
Supp.2d 279 (S.D.N.Y. 1999), BNA Criminal Practice Guide (January 12, 2000) 14
18.
[34] W.C. Thompson, E.L. Schumann, Interpretation of statistical evidence in criminal
trials: the prosecutor's fallacy and the defence attorney's fallacy, Law and Human
Behavior 11 (1987) 167187.
[35] D.J. Balding, P. Donnelly, The prosecutor's fallacy and DNA evidence, Criminal Law
Review (1994) 711721.
[36] R v Deen, Court of Appeal Criminal Division, The Times, January 10th.
[37] ENFSI Expert Working Group Marks Conclusion Scale Committee, Conclusion
scale for shoeprint and toolmarks examination, Journal of Forensic Identication
56 (2006) 255280.
[38] A. Biedermann, F. Taroni, P. Garbolino, Equal prior probabilities: can one do any
better? Forensic Science International 172 (2007) 8593.
[39] A. Biedermann, F. Taroni, C.G.G. Aitken, Letter to the editor re: conclusion scale
for shoeprint and toolmarks examinations, J. For. Ident. 56 (2) (2006) 6856938
(with response).
[40] R. v Sally Clark, Court of Appeal Criminal Division, EWCA Crim 1020, 2003.
Of earprints, ngerprints, scent dogs, cot deaths
and cognitive contaminationa brief
look at the present state of play
in the forensic arena
A.P.A. Broeders
Department of Criminal Law and Criminology, Leiden University,
Netherlands Forensic Institute, The Hague, Netherlands
Received 26 August 2004; received in revised form 9 November 2004; accepted 9 November 2004
Available online 13 October 2005
Abstract
Over the last decades, the importance of technical and scientic evidence for the criminal justice system has been steadily
increasing. Unfortunately, the weight of forensic evidence is not always easy for the trier of fact to assess, as appears from a brief
discussion of some recent cases in which the weight of expert evidence was either grossly over- or understated. Also, in recent
years, questions surrounding the value of forensic evidence have played a major role in the appeal and revision stages of a
number of highly publicized criminal cases in several countries, including the UK and the Netherlands. Some of the present
confusion is caused by the different ways in which conclusions are formulated by experts working within the traditional
approach to forensic identication, as exemplied by (1) dactyloscopy and (2) the other traditional forensic identication
disciplines like handwriting analysis, rearms analysis and bre analysis, as opposed to those working within the modern
scientic approach used in forensic DNA analysis. Though most clearly expressed in the way conclusions are formulated within
the diverse elds, these differences essentially reect the scientic paradigms underlying the various identication disciplines.
The types of conclusions typically formulated by practitioners of the traditional identication disciplines are seen to be directly
related to the two major principles underpinning traditional identication science, i.e. the uniqueness assumption and the
individualization principle. The latter of these is shown to be particularly problematic, especially when carried to its extreme, as
embodied in the positivity doctrine, which is almost universally embraced by the dactyloscopy profession and allows categorical
identication only. Apart fromissues arising out of the interpretation of otherwise valid expert evidence there is growing concern
over the validity and reliability of the expert evidence submitted to courts. While in various countries including the USA, Canada
and the Netherlands criteria have been introduced which may be used as a form of input or output control on expert evidence, in
England and Wales expert evidence is much less likely to be subject to forms of admissibility or reliability testing. Finally, a
number of measures are proposed which may go some way to address some of the present concerns over the evaluation of
technical and scientic evidence.
# 2005 Published by Elsevier Ireland Ltd.
Keywords: Forensic identication; Individualization; Dactyloscopy; DNA analysis; Interpretation of expert evidence
0379-0738/$ see front matter # 2005 Published by Elsevier Ireland Ltd.
doi:10.1016/j.forsciint.2004.11.028
1. Problems in assessing the weight of forensic
evidence: some examples
1.1. Mitochondrial DNA: incontrovertible evidence
In a recent, highly publicized criminal case in which a
revision was granted by the Dutch Supreme Court, a pro-
secutor insisted that a DNA prole obtained from a pubic
hair found on the victims jersey which matched the prole
of one of the two male suspects constituted damning evi-
dence against this suspect.
1
However, the prole obtained
was a mitochondrial prole. Mitochondrial DNA is passed
on unchanged from mother to child.
2
While this is in itself a
remarkable genetic fact which has led to the claim that all
Caucasians are descended from only a handful of women,
3
its relevance here is that an unknown number of relatives of
the suspect on the suspects mothers side, as well as an
unknown number of unrelated individuals,
4
could have the
same prole. The assessment of the evidential weight of the
DNA-match was therefore clearly erroneous.
1.2. Semen in a rape case: non-perpetrator trace
In the same case, a full nuclear DNA-prole that was
obtained fromsemen found on the thigh of the strangled rape
victim was declared a non-perpetrator trace by the prosecu-
tion when it turned out not to match the proles of the two
main suspects. The two men eventually confessed to raping
and killing the girl, a 23-year-old ight attendant, who was
found dead in her grandmothers house, but only after they
had undergone what later turned out to have been prolonged
and somewhat unorthodox questioning by the police. When
the suspects later retracted their confessions both in front of
the district court as well as the appeal court, police and
prosecution were singularly unimpressed. They explained
the non-match by arguing that the semen found on the
victims thigh originated from an earlier consensual sexual
contact, and had been dragged from the victims vagina to its
position on her thigh as a result of the subsequent involun-
tary sexual intercourse. Largely because it was backed by an
expert opinion proffered by a highly qualied but non-
professional ad hoc forensic expert, an emeritus professor
of gynaecology, this argument, which came to be known as
the drag theory, was accepted by both the district court and
the appeal court. The two men were convicted and sent to
prison. Several years later, when the case came up for review,
the professor retracted his theory, on the grounds that he had
not given due consideration to the fact that there was no
drag trail on the womans leg to mark the route the semen
had travelled. The two men were released in 2002, after
serving 7 years in prison.
1.3. Fingerprints: absolute identication
About 5 years ago, Detective Constable Shirley McKie
was charged with perjury when she denied entering a crime
scene where a ngerprint was found which the Scottish
Criminal Records Ofce claimed was hers. Two years later
some of the worlds leading dactyloscopists pointed out that
the latent print did not match the policewomans reference
ngerprint and could not be hers.
5
Today Ms. McKie is still
ghting for rehabilitation.
6
A similar example involving the FBI occurred in May
2004, whenfollowing the Madrid train bombings on 11
March 200437-year-old US born attorney and Muslim
convert Brandon Mayeld was arrested as a material
witness and spent 2 weeks in solitary connement in a
federal jail. Three FBI dactyloscopists categoricallybut
wronglyidentied a ngermark on a plastic bag contain-
ing detonators found in a van parked near the station from
which three of the four affected trains had departed as his.
7
An independent expert appointed by the judge reached the
same conclusion. In spite of the fact that the Spanish
authorities had advised the FBI that Mayelds reference
print did not match the nger mark even before Mayeld was
arrested, it was only after Mayeld had been detained for 2
weeks and the Spanish authorities had informed the FBI that
the mark in fact originated from an Algerian national that
Mayeld was released.
A.P.A. Broeders / Forensic Science International 159 (2006) 148157 149
1
Court of Appeal (Hof) Leeuwarden 24 April 2002, LJN-number
AE1877.
2
Mutations are thought to occur only once every 6500 years.
3
Sykes, B. (2002) Seven Daughters of Eve: The Science that
Reveals our Genetic Ancestry, W.W. Norton & Co: New York.
4
The fragment at hand is a short hypervariable fragment of the D-
loop and it has been shown that unrelated individuals can share
identical fragments because of recurrent and fast mutation pro-
cesses, leading to so-called homoplasy (Dr. Peter de Knijff: personal
communication).
5
H.M. Advocate v. Detective Constable Shirley McKie (visit:
www.clpex.com/Articles/McKie).
6
See also: Grieve, D.L. (1999) Built by Many Hands, Journal of
Forensic Identication 49(5), 565579; McKie, I.A.J. (2003)
Theres nane ever feard that the truth should be heard but they
whom the truth would indite, Science & Justice 43(3), 161165,
and the recently opened internet site www.ShirleyMcKie.com.
7
On the forensic implications see: Rudin, N. & Inman, K. (2004)
Fingerprints in Print the Apparent Misidentication of a Latent
Print in the Madrid Bombing Case, CACnews, 4, 1421. For a
description of the case by the defence see: Wax, S.T. & Schatz C.J.
(2004) A Multitude of Errors, The Champion, September/October,
6. An international committee of ngerprint experts asked by the
FBI to examine the case concluded that: . . . the failure was in the
application of the ACE-V methodology during this particular exam-
ination. (Robert B. Stacey (2004) Report on the Erroneous Fin-
gerprint Individualization in the Madrid Train Bombing Case
Journal of Forensic Identication 54(6), 706718). The problem
with this analysis, which tries to save the method by blaming the
expert is that the method and the expert cannot be separated because
the expert plays an essential role as judge/measuring instrument in
the ngerprint examination process.
On 23 January 2004, Stephan Cowans was released from
prison in Boston, Massachusetts, after DNA analysis had
demonstrated that biological trace material on the baseball
hat and the sweatshirt of an unknown perpetrator and on a
drinking glass used by this same perpetrator could not be
Cowans. The three DNA-proles obtained from the cell
material on these objects were identical but they did not
match Cowans prole. It subsequently appeared that a
thumb print on the glass which the Boston police had
attributed to Cowans did not originate from him either.
This ngerprint and an identication from a photo line-up
by two eye witnesses, the victim and another witness, had
been the only evidence against him at the time of his
conviction. Cowans is the rst case in which DNA-evidence
has led to the release and subsequent exoneration of a
suspect whose conviction was based on awed ngerprint
evidence. Cowans, who was convicted for the nonfatal
shooting a policeman with his own gun, spent nearly 7
years in prison.
8
1.4. Scent lineup: up in the air
In a second Dutch case in which a revision was granted,
the Deventer murder case, a canine scent identication test
or scent line-up was used to link a knife to a murder
suspect. The positive result that was obtained was appar-
ently taken by both the prosecution and the appeal court to
provide sufcient grounds to establish an association
between the knife and the suspect, as well as to infer that
the knife was the murder weapon.
9
At this stage no
biological material had been found on the knife so that
there was no independent conrmation for the association
in the form of a DNA test. Not only was there no DNA
evidence linking the knife to the suspect, stranger still,
there was no DNA evidence linking the knife to the victim
either, even though she had received several deep stab
wounds and had been bleeding profusely. When the case
came up for review before the Dutch Supreme Court,
meticulous sampling of the knife did produce some cell
material. However, as themixedDNA prole obtained
from this biological material failed to match that of the
victim, the Supreme Court decided to grant revision of the
appeal courts verdict and order a re-trial. At the re-trial in
February 2004, the prosecution dropped the knife as
evidence but maintained the murder charge. Contrary to
expectation, the re-trial ended in a renewed conviction of
the suspect on the basis of new scientic evidence when the
suspects DNA prole was obtained from a small blood
stain and tiny fragments of cell material found on the
female victims blouse.
10
1.5. Cot deaths and statistics
Three further examples are fromEngland. The rst is one
of a series of questionable cot death convictions which may
well herald a much larger miscarriage of justice. It took Sally
Clark, a solicitor from Chester, 5 years to be cleared of the
charge of killing her two baby sons. Forensic experts of
various medical persuasions gured prominently both in the
rst trial and in the later appeal proceedings. What is
particularly worrying is the fact that while expert opinion
was clearly divided this did not stop the jury fromnding her
guilty, or the judge from administering two life sentences, of
which Ms. Clark eventually served almost 3 years.
11
It is
now clear that several women suffered a similar fate and that
women at the centre of recent cases like Trupti Patel and
Angela Cannings only narrowly escaped the same fate.
Following the acquittal of Angela Cannings early in 2004,
a large-scale review of hundreds of cot death convictions
was announced.
1.6. Earprints versus DNA
In the UK, in July 2002, the Court of Appeal quashed the
conviction of Mark Dallagher, who was sentenced to life
imprisonment at Leeds Crown Court in 1998 for the murder
of 94-year-old Dorothy Wood in Hudderseld.
12
A re-trial
was ordered but in the course of the new investigation by the
prosecution a non-matching partial low copy number DNA
prole was obtained from an earprint found on a window at
the crime scene which was originally categorically attributed
to the suspect. Based on this nding, in January 2004, the
prosecution decided to drop the charges against Dallagher,
who was the rst man to be convicted of murder on earprint
evidence. Even though it is not clear whether the DNA
material, or the earprint for that matter, bear any relation to
the crime, it now looks as though several other convictions
involving ear print evidence will be presented for review in
the wake of this decision.
8
Loftus, E.F. & Cole, S.A. (2004) Contaminated Evidence,
Science 304(5673): 959.
9
On the basis of Schoon, G.A.A. (1998) A First Assessment of
the Reliability of an Improved Scent Identication Line-up, Jour-
nal of Forensic Sciences 43(1), 7075. The diagnostic value or
likelihood ratio of a positive identication in a properly executed
scent line-up must be estimated to lie somewhere between 6 and 14.
If the a priori likelihood that the knife was held by the suspect is low,
a positive result is unlikely but if it occurs it is more likely to be a
false alarm (false positive) than a correct identication.
10
LJN-number AO03222 Court of Appeal s-Hertogenbosch, 9
February 2004.
11
R. v. Sally Clark (2003) EWCA Crim 1020; Case No:
200203824 Y3. For a brief discussion of some of the forensic
aspects of the case see Richardson, B.A. (2004). The Sally Clark
Case and its Implications, Interfaces 38, 67. For a full account, see
Batt, J. (2004) Stolen Innocence A Mothers Fight for Justice: The
Authorised Story of Sally Clark, Ebury Press, London.
12
R. v. Dallagher, UK Court of Appeal 2002, EWCA Crim 1903,
July 25.
1.7. Prole mix-up
My nal example involves DNA-evidence. On 14 Feb-
ruary 2003, Peter Hamkin, a 23-year-old barman from
Merseyside, was arrested and taken to London, where he
spent 3 days in custody before narrowly escaping extradition
to Italy. It appeared that his DNA-prole matched that of the
man who shot 24-year-old Italian Annalisa Vincentini after a
failed rape attempt in a pinewood in Tuscany. A second test
showed that his prole differed from that of the cell material
collected at the crime scene. His protestations of inno-
cencehe claimed that he had never set foot in Italy, let
alone Tuscany and that he had 20 witnesses who were
prepared to swear he was stuck behind the beer pumps in
his employers bar at the timeapparently merited more
credence than the British police were initially prepared to
give them.
2. Forensic science under attack
Forensic sciencemore specically, but not exclusively,
forensic identication sciencehas come under erce
attack in recent years. Some of the graver miscarriages of
justice which have come to light in several countries in the
last decades were seen to be at least partly associated with
inadequate standards of forensic expertise. Names of foren-
sic scientists that spring to mind are those of serologist Fred
Zain, odontologist Michael West, serologist Timothy Dixon,
DNA-analyst Jacqueline Blake, footprint specialist dr.
Louise Robbins (of Cinderella Evidence fame),
13
all from
the United States, and to a lesser extent that of serologist dr.
Alan Clift and the SCRO in the United Kingdom. Cur-
iouslybut not necessarily signicantlyall of these seem
to be associated with adversarial type criminal law systems,
as are many of the better-known miscarriages of justice. In
fact, recent British history provides numerous examples of
what must now be believed to be erroneous convictions,
many of them involving expert evidence, like the Birming-
ham Six, the Guildford Four or the case of Stefan Kiszko.
14
More generally, it appears that what were long held to be
tried and trusted forensic identication procedures like
dactyloscopy, questioned document examination or rearm
examination are nowsaid to lack a sound scientic basis, and
the traditional claims of forensic identication science have
come to be dismissed as logically untenable. Studies like
those by Evett and Williams (1996) on ngerprint identi-
cation provide further evidence of the present crisis in the
forensic arena.
3. Forensic DNA analysis as a model
At the same time, forensic science is rapidly expanding.
DNA proling in particular may fairly be said to have
revolutionized forensic science. It not only constitutes a
powerful investigative and evidential tool in its own right, it
is, ironically perhaps, also largely as a result of the growing
familiarity with the scientic paradigmassociated with DNA
evidence that traditional identication science is now lying
so heavily under siege. Indeed, there can be little doubt that
the most signicant forensic development of the 20th cen-
tury has been the introduction of DNA-typing for the
purposes of personal individualization. In the two decades
that have elapsed since DNA-analysis was rst applied in a
criminal case by Sir Alec Jeffreys in the UK, DNA evidence
has not only revolutionized forensic identication science. It
has also been instrumental in resolving countless old and
cold cases by its power to exclude certain individuals as
donors of biological crime scene material and thereby
categorically eliminate them as suspects, and by its power
to provide incriminating evidence against others. It has
proved its investigative and evidential value in solving
violent, high prole crimes as well as in tackling the rather
more common type of transgression of the law that is
nowadays collectively referred to as high volume crime.
Over and above this, it has played an essential role in
bringing to light grave miscarriages of justice in an ever
widening range of countries including those which have long
prided themselves on being among the nations with the best
criminal justice systems in the world.
13
See Gold, A.D. (2003) Expert Evidence in Criminal Law: the
Scientic Approach, Irwin Law: Toronto, 5 ff. on Robbins and other
examples of questionable forensic expertise including Lawrence
Farwells brain ngerprinting, a technique which he claims tells
investigators whether certain information is or is not present in the
suspects brain.
14
In 1976, Kiszko was sentenced to life imprisonment following a
conviction based largely on his own confession of the murder of 11-
year-old Lesley Molseed. Kiszko admitted having masturbated on
the victims underwear, a statement which seemed to be borne out
by serological tests. Fourteen years later it appeared that the perpe-
trators semen contained sperm heads, while Kiszkos reference
sample that was taken at the time of the original investigation did not
contain sperm heads. Although Kiszko, who suffered from hypo-
gonadism (underdevelopment of the testes), was infertile and could
not possibly be the donor of the questioned semen, the defence was
not informed of this fact at the time of the trial. Kiszko was released
in 1992 but died a year later at the age of 44 of a heart attack. The
prosecutor who brought the case against Kiszko and secured Kisz-
kos conviction, was appointed Lord Chief Justice in 1992 (www.u-
sers.bigpond.com/burnside/kiszko.htm). In 1993, this case and
similar cases led to the foundation of INNOCENT, an independent
organization based in Manchester ghting for persons whose con-
victions are believed to amount to miscarriages of justice. The
INNOCENTwebsite, www.innocent.org.uk, features a Hall of Judi-
cial Infamy with the names of scores of people whose innocence has
since been established, albeit in some cases only after their death by
hanging. A similar organization operating under the name of
MOJUK (Miscarriages Of Justice UK; at www.mojuk.org.uk) is
also committed to examining cases of people believed to be wrongly
convicted.
DNA evidence is used not only to bring true perpetrators
to book but also to exonerate innocent victims, through post-
conviction DNA testing. In this sense, forensic DNA ana-
lysis can be said to be ideologically neutral. Perhaps the best
example here is the Innocence Project, set up as a non-prot
legal clinic by Barry C. Scheck and Peter J. Neufeld at
Cardozo Law School in 1992.
15
The Project only handles
cases where post-conviction DNA testing of evidence can
yield conclusive proof of innocence. As a clinic, students
handle the case work while supervised by a teamof attorneys
and clinic staff. To date, the project has produced 153
exonerations. Recent examples of post-conviction exonera-
tions based on DNA evidence from the UK and the Nether-
lands include the exoneration of the Cardiff Three of the
murder of Lynette White, a case which dates from 1988, and
the resolution of the Fitting Room murder (Zaanse paska-
mermoord) in the Netherlands, which dates from 1984. In
both these cases men who were previously released on other
grounds were denitively exonerated as DNA evidence
identied the true perpetrators. The rst UK exoneration
through DNA testing involving an imprisoned offender is
reported in Johnson and Williams (2004).
16
But DNA evidence was not immediately taken on board
by the courts. After a spectacular startthe Pitchfork case in
the UK, in which the rst suspect of the murder of two young
girls was excluded and the perpetrator initially avoided
detection by asking a friend to take his place in the sub-
sequent, rst-ever mass forensic DNA screeningforensic
DNA analysis met with considerable criticism (as it did in
the O.J. Simpson case in the USA). The subsequent devel-
opment of forensic DNA-analysis has taken place before a
broad scientic forum composed of molecular biologists,
population geneticists, methodologists and statisticians. As a
result, it has acquired a rm theoretical basis. In this, it
differs fromvirtually all other types of forensic identication
evidence as well as many types of non-identication evi-
dence, in which interest tended to focus on the potential for
practical application of the method used (i.e., does it help us
to catch criminals?) rather than on aspects like the reliability
or validity of the method, or on a critical scrutiny of the
theoretical principles underlying it.
Apart from being a very effective identication method,
forensic DNA analysis has had an unparalleled effect in
increasing our understanding of the potential and actual
dangers of relying too heavily on the use of eyewitness
testimony and various types of forensic or technical exper-
tise in judicial fact-nding. Last but not least, the rise of
forensic DNA analysis and the accompanying interest in the
probabilistic model associated with it have sparked off a
renewed critical interest in the interpretation and evaluation
of other types of forensic identication evidence which was
long overdue and whose full impact is as yet unclear.
4. Does forensic science qualify as science?
17
Almost inevitably, the brunt of the critical assault was
initially borne by the tried and trusted traditional forensic
identicationdisciplines like handwritinganalysis, morecom-
monly known as questioned document examination, rearm
and tool mark examination, and morphological hair analysis.
More recently however, the very icon of traditional forensic
identication science, the ngerprint, has also come under
attack, culminating in early 2002 in Philadelphia-based Judge
Louis H. Pollak
18
ruling ngerprint evidence inadmissible
only tosomewhat anticlimacticallyreverse himself less
than 2 months later.
19
Even more so than earlier admissibility
decisions like US v Starzecpyzel,
20
in which Judge McKenna
ruled handwritinganalysis not tobe a science but somethingin
the nature of a craft or a technical skill (such as piloting a
vessel) and as such admissible after all, Judge Pollaks
decision sent shock waves through the forensic science com-
munity which extended all the way to the serious press.
At this point, it may be good to pause for a moment to see
what distinguishes forensic identication science from other
forms of forensic science. Essentially, this difference can be
describedinterms of what InmanandRudin
21
have termedthe
Table 1
Different forensic processes
Identication Determination of physical-chemical
composition (e.g., illicit drugs)
Classication Determination of class, type
(e.g., hair, bres, blood type, DNA
22
)
Individualization Determination of unique identity of source
(e.g., ngerprints, handwriting)
Association Determination of contact between two
objects (e.g., through transfer of bres, glass)
Reconstruction Determination of facts of the case:
nature and place of events in time and
space (e.g., murder, explosion)
15
www.innocenceproject.org.
16
Johnson, P &Williams, R. (2004) Post-conviction DNAtesting;
The UKs rst exoneration case, Science & Justice 44(2), 7782.
17
The same question was implicit in the title of Kennedy, D.
(2003) Forensic Science: Oxymoron?, Science 203, 1625. An
unequivocal answer can be found in Saks, M.J. & Koehler, J.J
(2005) The Coming Paradigm Shift in Forensic Identication
Science Science 309, 892895.
18
U.S. v. Plaza 188 F.Supp 2d p. 576 (E.D. Pa. 13 March 2003).
19
On recent criticism of ngerprint evidence see also Cole, S.A.
(2001) Suspect Identities: A History of Fingerprinting and Criminal
Identication, Harvard University Press: Cambridge MA; Epstein,
R. (2002) Fingerprints Meet Daubert: the Myth of Fingerprint
Science is Revealed, Southern California Law Review
75:605658 and Steele, L.J. (2004). The Defense Challenge to
Fingerprints, Criminal Law Bulletin 40(3), 213240.
20
U.S. v. Starzecpyzel, 880 F. Supp. 1027; S.D.N.Y. 1995.
21
Inman, K. & Rudin, R. (2002) The Origin of Evidence,
Forensic Science International 126, 1116.
22
Although the purpose of forensic DNA analysis is individuali-
zation, the forensic process involved is technically one of classica-
tion, based as it is on the use of class characteristics, in the form of
DNA markers or alleles.
various forensic processes, i.e., identication, classication,
individualization, association and reconstruction (Table 1).
It is the individualization process, the inference of iden-
tity of source, that is the ultimate aim of all forensic
identication science.
23
Unfortunately, the various types
of forensic identication science do not use the same
scientic paradigm andconsequentlydo not report their
conclusions in the same format. The situation may be
summed up as given in Table 2.
Largely as a result of the growing prominence of the
scientic paradigm associated with DNA evidence, tradi-
tional identication science, as exemplied by dactyloscopy
and the remaining identication disciplines, is now lying
heavily under siege. What is at issue here is nothing less than
the scientic status of forensic identication procedures.
Critics point to the lack of scientic rigour in the methodol-
ogy applied, to the presence of various types of examiner
bias and to the prosecutorial orientation inherent in many
traditional and well-accepted forensic procedures.
25
Else-
where I have argued that the positivity doctrine
26
sub-
scribed to by the dactyloscopist community, whereby
identications are exclusively reported as absolute under
penalty of excommunication from said community, may
well have served to perpetuate fundamental misunderstand-
ings about the nature of scientic evidence whichcontrary
to widespread beliefis essentially not of a categorical or
deterministic but of a probabilistic nature.
27
As Kwan
28
and others have since pointed out, any form of
individualization is an essentially inductive process. While
forensic identication procedures may lead to categorical
elimination, as again the example of DNA proling demon-
strates, unless the number of potential sources is limited and
known, no forensic identication procedure can lead to a
categorical identication. It is again forensic DNA analysis
that, through the implicit example it sets as a forensic identi-
cation procedure with a solid scientic basis, is increasingly
being referred to as a standard for other forensic identication
procedures, which so far are seen to be failing to reach the
standard of proper scientic procedure set by DNA.
5. The traditional approach in forensic identication
This is not to say that the results reported by traditional
forensic identication scientists are necessarily always or
even frequently wrong. What it does mean is that the
conclusions are not logically tenable, as in the case of
categorical ngerprint identications, and that they typically
have a subjective element, which makes it difcult to assess
their validity in an objective fashion. Traditional forensic
identication science distinguishes itself from other types of
scientic endeavor in that it is almost exclusively practiced
in the context of the criminal law. In so far as it does not meet
the requirements of the scientic method, it may be more
aptly described as a Science Constructed in the Image of the
Criminal Law, as one of its most vociferous critics, Michael
Saks, put it.
29
Because the method used does not meet the
scientic standard, there are no safeguards against the
Table 2
Identication (identity of source), more properly called individualization
Discipline Type of conclusion Example
Dactyloscopy Categorical: yes or no Finger trace does (not) originate from suspect
DNA Probabilistic and quantitative Probability of random match (RMP) of prole
suspect with that of crime scene cell material
Other
a
Probabilistic and verbal
24
Crime scene material (very) probably does (not)
originate from suspect
a
Handwriting, hair, bres, rearms, toolmarks, impression marks.
23
In The Ontogeny of Criminalistics, Journal of Criminal Law,
Criminology and Police Science 54 (1963), P.L. Kirk dened
criminalistics as the science of individualization (p. 235).
24
Increasingly, following the DNA model, these types of evidence
may be reported using a likelihood ratio or a likelihood ratio-based
scale, as proposed by Evett, I.W., Jackson, G., Lambert, J.A. &
McCrossan, S. (2000). The impact of the principles of evidence
interpretation on the structure and content of statements, Science &
Justice 40(4), 233239.
25
Saks, M.J., Risinger, D.M., Rosenthal, R. & Thompson, W.C.
(2003) Context effects in forensic science: A review and applica-
tion of the science of science to crime laboratory practice in the
United States, Science & Justice 43(2), 7790. An almost identical
version of this article appeared earlier as Risinger, D.M., Saks, M.J.,
Thompson, W.C. & Rosenthal, R. (2002). The Daubert/Kumho
Implications of Observer Effects in Forensic Science: Hidden Pro-
blems of Expectation and Suggestion, California Law Review 90,
156.
26
Friction ridge identications are absolute identications. Prob-
able, possible, or likely identication are outside the acceptable
limits of the science of friction ridge identication. McRoberts,
A.L. (2002) Scientic Working Group on Friction Ridge Analysis,
Study and Technology, Journal of Forensic Identication 52, 263
348.
27
Broeders, A.P.A. (2003) Op zoek naar de bron: Over de grond-
slagen van de criminalistiek en de waardering van het forensisch
bewijs, Kluwer: Deventer.
28
Kwan, Q.Y. (1977) Inference of Identity of Source, Ph.D. Thesis,
University of California, Berkeley, CA.
29
Saks, M.J. (1998) Merlin and Solomon: Lessons fromthe Laws
Formative Encounters with Forensic Identication Science, Hast-
ings Law Journal 49(4), 10691141.
potentially pernicious effects of observer bias and cognitive
contamination due to domain-irrelevant information.
30
These are not only known to present powerful threats to
human observation and judgment in general but have now
also been demonstrated to provide some of the contributing
factors to the conditions in which grave miscarriages of
justice may occur.
There are two major principles underpinning classical
forensic identication science. The rst is the principle of
uniqueness, summed up in the phrase Nature never repeats
itself, which is almost invariablybut as far as I have been
able to establish with little justication
31
attributed to the
Belgian astronomer and social statistician Quetelet, and is
echoed in claims like All ngerprints/ears/voices are
unique. The second is the principle of individualization,
which says that every trace can be related to a unique source.
The main problem here lies in the second of these
assumptions. While the rst principle, that every object is
unique, is an unproved assumption whichin a philosophi-
cal but forensically trivial senseis both necessarily logi-
cally true and impossible to prove, it is the second principle
that is largely responsible for the methodological problems
surrounding forensic identication science. The proper
question to ask is not if all physical traces are unique and
therefore theoretically capable of being uniquely identied
with a particular source but whether they can always be so
identied in the forensic context and using the methods and
procedures employed by the forensic scientist. That is also,
or rather should be, the central question in the currently
raging ngerprint debate. What the traditional individuali-
zation procedure entails is formulated by Huber as follows:
When any two items have characteristics in common of
such number and signicance as to preclude their simulta-
neous occurrence by chance, and there are no inexplicable
differences, then it may be concluded that they are the same,
or from the same source.
32
The problem here of course lies in the basis on which
chance occurrence of shared features can be excluded and
differences can be deemed to be explicable. This requires
reference to an indenitely large set of alternative potential
sources whose size cannot normally be dened with any
precision and whose feature composition is not really very
well known. A more authoritative handbook of forensic
science describes the process as follows, invoking the dis-
tinction between class characteristics and individual char-
acteristics that is commonly made in the forensic
identication context:
Class characteristics are general characteristics that sepa-
rate a group of objects froma universe of diverse objects. In a
comparison process, class characteristics serve the very
useful purpose of screening a large number of items by
eliminating from consideration those items that do not share
the characteristics common to all the members of that group.
Class characteristics do not, and cannot establish unique-
ness.
Individual characteristics, on the other hand, are those
exceptional characteristics that may establish the uniqueness
of an object. It should be recognized that an individual
characteristic, taken in isolation, might not in itself be
unique. The uniqueness of an object may be established
by an ensemble of individual characteristics. A scratch on
the surface of a bullet, for example, is not a unique event; it is
the arrangement of the scratches on the bullet that mark it as
unique.
33
While this sounds plausible, there is still a major logical
problem. The denition of individual characteristics is
essentially circular. Individual characteristics are dened
as characteristics that arefrequently in combination
capable of establishing uniqueness. But uniqueness is
dened by an ensemble of individual characteristics.
Whether characteristicseither by themselves or collec-
tivelyare unique is an inductive question, which raises
the classical induction problem: we can never be sure that all
swans are white as long as we have not seen all swans.
Similarly, we can never be sure that a feature or combination
of features is unique, until we have observed all relevant
objects.
What practitioners of traditional forensic identication
sciences really do is perhaps best described by Stoney, who
used the image of the leap of faith as the mechanism
whereby the forensic scientist actually establishes indivi-
dualization:
When more and more corresponding features are found
betweenthe twopatterns scientist and layperson alike become
subjectively certain that the patterns could not possibly be
duplicated by chance. What has happened here is somewhat
analogous to a leap of faith.
34
It is a jump, an extrapolation,
based on the observation of highly variable traits among a few
characteristics, and then considering the case of many char-
acteristics. [] In ngerprint work, we become subjectively
convinced of identity; we do not prove it.
Dactyloscopists, and all other traditional forensic identi-
cation scientist are ultimately making a subjective decision
in reaching a categorical decision about the identity of
30
Saks et al. op. cit.
31
I have been unable to nd it, i.e., its French equivalent) in the
source that is most frequently given, i.e., Quetelet, Ad. (1870)
Anthropometrie ou Mesure des Differentes Facultes de lHomme,
Muquardt: Bruxelles.
32
Huber, R.A. (19591960) Expert Witnesses, Criminal Law
Quarterly 2, 276296.
33
Thornton, J.I. & Peterson, J.L. (2002) The General Assump-
tions and Rationale of Forensic Identication 242.02.1, in D.L.
Faigman, D.H. Kaye, M.J. Saks & J. Sanders (red.) Modern Scien-
tic Evidence: The Law and Science of Expert Testimony, West
Publishing Co.: St. Paul, MINN.
34
Bold type APAB.
source of two traces. They become convinced that the
unknown trace and the reference material have the same
origin. But there is no logical basis for this type of conclu-
sion, as Tuthill points out:
At the present time, in most jurisdictions, an opinion of
certainty is the only acceptable opinion when dealing with
the individualization of a ngerprint impression. This is not a
rule that has been laid down by the courts (who will deal with
any opinion offered) but rather by the ngerprint examiners.
There is no scientic basis for the rule. It is simply one of the
aberrations that have developed within the discipline of
ngerprint identication.
35
Quantication of the frequency of the characteristics
involved is often difcult but even if it is possibleas again
pre-eminently in the case of nuclear DNA typingand no
matter how infrequent we estimate the combined occur-
rence of the characteristics to be, it will not allow us to
individualize, as Stoney so aptly expressed in the title of his
1991 paper, What made us ever think we could individua-
lize using statistics?
36
The estimated frequency of the
prole of the crime scene sample and that of the reference
sample of the suspect may be vanishingly small, there is still
no logical basis on which the forensic scientist could
pronounce them to have the same origin. DNA character-
istics are essentially class characteristics, which, as Thorn-
ton & Peterson put it . . . do not, and cannot establish
uniqueness.
37
6. Rules of expert evidence
Traditionally of course expert evidence in common law
systems is subject to four exclusion rules: the common knowl-
edge rule, the eld of expertise rule, the ultimate issue rule and
the basis rule. These rules can be seen to also play at least an
implicit role in dening expert evidence in Continental sys-
tems. However, while expert evidence in the United States
may be said to be subject to a formof entrance examination or
input control, with the judge acting as gatekeeper for the jury,
the Dutch legal systemmay be said to operate an exit check or
use a form of output control. In general, Continental style
judges are free to accept or reject evidence, and to assess and
evaluate it as they see t, as long as such decisions are duly
motivated. As a result, the need to rely on guidelines by means
of which the question of the admissibility of evidence can be
resolvedof which the Frye test, the Federal Rules of Evi-
dence and the Daubert ruling are examples in the US con-
texthas clearly traditionally been less strongly felt in the
continental European systems, especially those using profes-
sional judges rather than juries. However, in recent years, the
Dutch Supreme Court has made several important decisions
on the reliability of expert evidence of which the most
important is the Shoemaker (stick to thy last opinion)
decision.
38
The Supreme Court ruled that the expert testimony
of an orthopedic shoemaker who identied footwear marks
found at a crime scene as originating fromthe suspects shoes
should not have been used as evidence by the appellate court
because the court had failed to motivate why it used the expert
evidence even though its validity was questioned by the
defence. The Supreme Court ruled that if expert evidence
is challenged it can only be used by a court if the court
indicates (Table 3):
An important milestone in this context in the United
States is the so-called Daubert decision from 1993. In this
civil case, the presiding judge laid down a number of criteria
for scientic evidence to meet if it is to be submitted to the
jury (Table 4).
It appears that the position in the UK is vastly different
form that in the United States or Canada. Ormerod (2002:
774) describes the situation as follows:
English law admits novel scientic techniques as the basis
of expert opinion without any special scrutiny. The attitude is
that there are no closed categories where evidence may be
placed before a jury and that it would be entirely wrong to
deny to the law of evidence the advantages to be gained from
new techniques and new advances in science.
39
The lack of
regulation governing admissibility in England and Wales is
in stark contrast to the position in the USAwhere the much-
discussed Daubert-criteria require judges to consider the
reliability as well as the relevance of the expert evidence. In
particular Federal courts are to have regard to [whether the
method] has been subjected to peer review, to whether the
technique is falsiable and has published error rates as
Table 3
The shoemaker criteria
1. What is the profession, the education and
experience of the expert?
2. Does the expertise relate to the subject on
which the expert is giving an opinion?
3. What method did the expert use?
4. What is the reliability (validity) of the method used?
5. Was the expert able to apply the method in a
competent fashion?
35
Tuthill, H. (1994) Individualization: Principles and Procedures
in Criminalistics, Lightning Powder Company: Salem, Oregon, p.
61.
36
Stoney, D.A. (1991a) What made us ever think we could
individualize using statistics?, Journal of the Forensic Science
Society 31(2), 197199.
37
Thornton, J.I. & Peterson, J.L. (2002) The General Assump-
tions and Rationale of Forensic Identication 242.02.1, in D.L.
Faigman, D.H. Kaye, M.J. Saks & J. Sanders (red.) Modern Scien-
tic Evidence: The Law and Science of Expert Testimony, West
Publishing Co.: St. Paul, MINN.
38
HR 27 January 1998, NJ, 404.
39
Per Steyn L.J. in Clarke (1995) 2 Cr. App. R. 425, 430.
regards its validity and reliability, and to whether the data are
capable of replication.
41
According to Meintjes-van der Walt (2001: 156), in
England/Wales, and in South Africa there are no reported
cases where courts had to decide on the admissibility of
novel forms of scientic expertise.
42
Ormerod (2002: 774)
observes that . . . it is arguable that recent cases signal a
growing judicial unease with the relaxed English approach
. . . towards expert evidence, even though it is impossible to
conrm a denite move towards any particular test. He even
goes on to observe that recent cases . . . demonstrate [] that
the present position is prone to lead to miscarriages of
justice. Gold (2003: 36) strikes a similar chord: . . . the
UK law of expert evidence has demonstrated little move-
ment towards modernization. [] Unlike in the United States
and Canada, it appears that gatekeeping with regard to
reliability is not an aspect of the British judicial role with
regard to expert evidence.
43
Finally, Grant (2002: 121)
points out that [i]n contrast to the American Daubert
approach, the British laws on admissibility recognize the
expert rather than the method and while there are obvious
difculties with this, it might sometimes allow the baby in
with the bathwater.
44
If this analysis is correct, it might in
fact go some way towards explaining the enormous inuence
an expert like Sir Roy Meadow may have had
45
in a eld
where validation of expert evidence is notoriously difcult.
7. Remedies
Although the responsibility for judicial errors tradition-
ally tends to be primarily associated with the police and the
prosecution (Another prosecution blunder in case X), other
players in the criminal justice eld including forensic scien-
tists also have a major part to play in improving the admin-
istration of criminal justice. Miscarriages of justice, whether
they are real or imagined, reect on the credibility of the
entire criminal justice system and may ultimately undermine
its very existence. A major question in this context is that of
the lack of communication and understanding between
the forensic expert on the one hand and the prosecution
and the trier of fact, and to a lesser extent, the defence lawyer
on the other. There is traditionally a wide gulf separating the
natural sciences from the law. As the Dutch legal psychol-
ogist Crombag put it, with the increasing importance of
expert evidence in criminal as well as civil cases, lawyers
can no longer afford to hold empirical science as essentially
a terra incognita, an unknown area that they are determined
never to set foot on if they can avoid it.
46
Lawyers need
experts because they do not possess the required expertise; at
the same time it is the lawyer or the trier of fact who
ultimately has to judge the competence of the expert and
the weight of the expert evidence by actively seeking to
establish its validity and reliability.
A prominent Dutch defense lawyer and professor of law
in the university of Utrecht, Dr. Knoops, has argued that the
criteria for granting revision of a judicial decision or a re-
trial should be brought in line, or synchronized as he put it,
with the rapid advances made in forensic science.
47
Under
current Dutch law, revision is only possible if, after the nal
verdict, a so called novum emerges, i.e., a new circumstance
of a factual nature which gives rise to the grave presumption
that, had this fact it had been known to the trier of fact at the
time of the trial, this would have led to the acquittal of the
suspect. Knoops adduces three arguments in support of his
proposal. He rst observes that in countries like the United
States and Canada the truth of the maxim that a fair trial
does not always guarantee a safe verdict has been more than
convincingly demonstrated by the scores of cases in which
post-conviction DNA testing has exonerated innocent sus-
pects. Knoops second and third arguments, the possibility of
forensic error and the danger of prosecutorial bias in forensic
expertise are also of such grave concern that careful con-
sideration of his proposal seems warranted.
At present, under Dutch law, a revised expert opinion
does not qualify as a novum. This is somewhat difcult to
Table 4
The Daubert criteria
1. Is the underlying theory, technique, methodology or reasoning at issue empirically testable and has it been tested?
2. Has the theory been subjected to peer review and publication?
3. What is the known or potential error rate of the technique in question?
4. Are there standards controlling the techniques operation and are these being maintained?
5. Does the method or technique pass the general acceptance test of the relevant scientic community?
40
40
Daubert v. Merrell Dow Pharmaceuticals Inc. (113 S. Ct. 2786
1993).
41
Ormerod, D. (2002) Sounding Out Expert Voice Identication,
Criminal Law Review, 771790.
42
She does mention two apparent exceptions, both in South
Africa; the last of which dates from 1986 and relates to the
admissabliltyof identication procedures involving scent dogs.
43
Gold, A.D. (2003) Expert Evidence in Criminal Law: the
Scientic Approach, Irwin Law: Toronto.
44
Grant, T. (2002) Review of Donald Foster (2001) Author
Unknown: On the Trail of Anonymous, 2nd edition, London:
Macmillan, Forensic Linguistics 9(1), 119121.
45
As in the Sally Clark case referred to above.
46
Crombag, H.F.M. (2000) Rechters en deskundigen, Neder-
lands Juristenblad 33, 16591665.
47
Knoops, G.G.J. (2002) De Puttense moordzaak: herzienings-
rechtelijke implicaties van de voortschrijdende forensische exper-
tise, Nederlands Juristenblad 29, 14021407.
reconcile with the prevailing view today in which all scien-
tic knowledge is conceived of as inherently non-nal or
provisional in nature: it is presumed to be valid only for as
long as and inasmuch as it has not been proved otherwise.
This means that the possibility that conclusions need to be
revised in the light of advancing knowledge is always open.
In the forensic context, this means that advances in forensic
techniques may produce results which may modify or con-
tradict earlier results and make it necessary to revise earlier
conclusions in old cases. Knoops proposal in response to this
dilemma, the institution of a special Revision Chamber,
composed not only of experienced lawyers but also of
experts from the forensic sciences, is certainly worthy of
serious consideration. In a similar vein, Gold (2003: 231
237) proposes three changes to ensure the reliability and
validity of expert evidence. They are: (1) the use of scientic
methodology as a litmus test for expert evidence, (2) the use
of science as a criterion for scientic evidence, and (3) the
use of a court-appointed scientist in the role of devils
advocate to examine and critique expert evidence to be
presented by parties.
8. Conclusion
Forensic science is going through a period of consider-
able turmoil. What were long held to be incontestable
forensic disciplines like dactyloscopy and handwriting
expertise are increasingly being required to meet the same
standards of scientic procedure as DNA evidence, even
though DNA proling in its present forensic application
essentially amounts to a classication not an individualiza-
tion process. While judges in countries like the USA are
becoming more critical of forensic identication expertise,
they will frequently still tend to plump for what might be
termed the split the baby approach, whereby rather than
throwing out the expert evidence altogether, the judge allows
the expert to discuss similarities and differences but not to
pronounce upon the ultimate issue, i.e., the source of the
trace material. The criticism is beginning to produce fresh
research efforts aimed at improving forensic identication
methods, which are frequently undertaken in an international
context. Harmonization of methods and techniques, as well
as quality control is high on the forensic agenda. A relatively
new development is the growing awareness of the danger of
cognitive contamination or observer effects in traditional
forensic identication disciplines like dactyloscopy, hand-
writing comparison and rearm examination and the need
for blind testing in these areas.
48
To conclude on a positive note, the critical scrutiny of
traditional forensic science procedure is likely to lead to an
improved understanding of the nature of scientic evidence
which can eventually only strengthen its position. Despite or
better precisely because of the current critical climate forensic
science is bound to play an increasingly important role in the
various national criminal justice systems as well as in those
associated with the diverse international courts and tribunals.
Acknowledgements
I wish to thank Dr. Ate Kloosterman and an anonymous
reviewer for their comments on an earlier version of this
paper, which was originally presented at the 8th Cross
Channel Conference of Forensic Sciences at Bruges, Bel-
gium on 2024 April 2004.
48
Saks, M.J., Risinger, D.M., Rosenthal, R. & Thompson, W.C.
(2003) Context effects in forensic science: A review and applica-
tion of the science of science to crime laboratory practice in the
United States, Science & Justice 43(2), 7790.
Articles
Introduction
Recent court cases in the UK have highlighted the need
for epidemiological data on the relative frequency of
natural and unnatural deaths in families who have had
more than one unexpected infant death. In 1956, Barratt
1
introduced the term cot death (crib death) to describe
unexpected infant deaths without obvious explanation.
The term highlights the common feature that many of
these infants are discovered dead in their cots. The term
sudden unexpected death in infancy provides a broader
designation, which includes cot deaths. After thorough
post-mortem enquiries, sudden unexpected deaths in
infancy divide into various explained deaths and a
residual group for which no explanation can be found,
designated sudden infant death syndrome (SIDS).
It has long been recognised that some sudden
unexpected deaths in infancy are not natural.
25
Starting
from the assumptions that SIDS deaths appear to occur
in families at random and that there is no evidence of a
genetic etiology, DiMaio and DiMaio
6
concluded that
Siblings of SIDS have the same risk as the general
population. They therefore implicitly derived the
probability of two or three SIDS deaths occurring in one
family by squaring or cubing the probability of a single
death. This approach led them to conclude that while a
second SIDS with one mother is improbable, it is
possible . . . or . . . remotely possible . . . (2nd edition)
and A third case, in our opinion is not possible and is a
case of homicide. Meadow
7
and the Condential
Enquiry into Stillbirths and Deaths in Infancy (CESDI)
in their report on Sudden Unexpected Death in Infancy
(SUDI)
8
used a similar argument for deducing the
probability of two SIDS deaths. Meadow, apparently
following the DiMaios lead, wrote in the third edition of
the ABC of Child Abuse that two is suspicious and three
murder unless proved otherwise . . . is a sensible
working rule for anyone encountering these tragedies.
9
The assumptions of independence that underlie these
estimates are invalid, because siblings have 25% of
genes in common and experience an essentially similar
domestic environment, which might, however, be
altered by the preceding death. Several studies have
shown that siblings of infants with SIDS are at increased
risk of SIDS.
1012
The best estimate was given by the
Norwegian population-based study, which reported that
the relative risk of recurrence was 58 (95% CI
21132).
13
Such studies of recurrence have been
criticised for not considering the possibility of serial
infanticide.
14
However, a review of the accuracy of the
diagnosis of SIDS in Norway during the period of the
study suggested that SIDS was originally substantially
underdiagnosed.
15
Nevertheless, the risk of recurrence is
sufciently low to exclude the possibility of a simple
inheritance pattern for SIDS.
In consideration of whether repeated unexpected
infant deaths in a family are natural or not, isolated
estimates of the probability of more than one SIDS case
in a family, however derived, are invalid and easily
misinterpreted. Multiple infanticide is also very rare.
What is important is the relative likelihood that the
deaths are natural versus unnatural, given that two or
more deaths have occurred.
16
Lancet 2005; 365: 2935
See Comment page 3
Medical Statistics Unit,
Department of Epidemiology
and Population Health, London
School of Hygiene and Tropical
Medicine, Keppel Street,
London WC1E 7HT, UK
(Prof R G Carpenter PhD,
A McKenzie MSc); Care of Next
Infant (CONI) programme,
Academic Unit of Child Health,
Shefeld Childrens Hospital,
University of Shefeld,
Shefeld, UK (A Waite BN,
Prof J L Emery MD); Neonatal
Intensive Care Unit, Jessop
Wing, Royal Hallamshire
Hospital, Shefeld, UK
(R C Coombs FRCPCH);
University Hospital, Lewisham,
London, UK
(C Daman-Willems FRCPCH); and
University Hospital for Children
and Youth, Utrecht,
Netherlands
(Prof J Huber FRCPC)
Correspondence to:
Prof R G Carpenter
bob.carpenter@lshtm.ac.uk
www.thelancet.com Vol 365 January 1, 2005 29
Repeat sudden unexpected and unexplained infant deaths:
natural or unnatural?
R G Carpenter, A Waite, R C Coombs, C Daman-Willems, A McKenzie, J Huber, J L Emery
Summary
Background There have been suggestions that when two or three unexpected unexplained infant deaths occur within
a family they are more likely to be unnatural than natural. We aimed to estimate the probability that a second infant
death is natural versus unnatural.
Methods The Care of Next Infant programme (CONI) supports parents who have previously had an unexpected and
apparently unexplained infant death and is currently available in over 90% of health districts in England, Wales, and
Northern Ireland. We studied all deaths in 6373 infants who had completed the CONI programme by December,
1999. After a CONI death, we made detailed enquiries into the previous death and the CONI death, including a
family interview, a review of autopsies, and case discussion.
Findings 57 (89 per 1000) CONI infants died. Nine deaths were inevitable, and 48 were unexpected. 44 families lost
one child, and two families lost two children. Of the 46 rst CONI deaths, 40 were natural; the other six were
probable homicides, ve committed by one or both parents (two criminally convicted). The ratio of 40 natural to six
unnatural deaths is 67 (95% CI 28194). Enquiries identied 18 families with two SIDS (sudden infant death
syndrome) deaths and two families with probable covert double homicides (ratio 90 [22 to 800]). There were no
convictions in 13 incomplete cases. Families with three deaths are reported.
Interpretation Repeat unexpected infant deaths are most probably natural.
Articles
Such information can be obtained only from
unselected series of families who have had more than
one unexpected infant death. Emery, who was concerned
about licide and also undetected hereditary disease, in a
personal series of cases
17
reported that licide was
probable in ve of 12 cases of two or more unexpected
deaths in a family. He and colleagues reported similar
results in another ten unselected cases of repeated
unexpected infant deaths.
18
Stanton,
19
in a follow-up
study of unselected cases, has recently reported three
families who each had had two unexpected infant
deaths, two of which raised maltreatment issues. In this
paper we report the proportion of natural and unnatural
infant deaths (ie, deaths before the age of 1 year),
occurring in families enrolled on a support programme
for parents who had previously had a sudden unexpected
and apparently unexplained infant death.
Methods
The Foundation for the Study of Infant Deaths facilitates
a programme of home support for parents with young
babies when there is a history of sudden unexpected and
apparently unexplained infant death in a sibling.
20
The
Care of Next Infant (CONI) scheme, which started in
1988, was available in 75% of health districts in England,
Wales, and Northern Ireland by 1994 and is currently
available in 91%. It is coordinated nationally from
Shefeld and run locally in collaboration with the
hospital and community-health services by a CONI
coordinator. Support measures include symptom
diaries, apnoea monitors, scales, and weekly home visits
by the family health visitor. The general practitioner and
a named hospital or community paediatrician are also
involved. Support lasts for at least 6 months, or
2 months beyond the age of the previous infant death.
Centres aim to offer the programme to any woman
during a subsequent pregnancy, who herself or whose
partner has suffered a cot deathie, an unexpected
and reportedly unexplained infant death. Participation in
the scheme is voluntary. Registration data included the
number of previous livebirths and causes of all previous
infant deaths. Deaths were investigated only in the event
of the death of an infant on the CONI programme.
We use the term SIDS to describe unexpected infant
deaths in which the cause remains unknown despite a
thorough investigation including a review of the clinical
and family history, a detailed post-mortem examination,
and in many cases bacteriological and biochemical
study. As in the CESDI SUDI studies, we have included
in this category some deaths that may be attributed to
accidental suffocation by history but in which other
investigations are inconclusive.
8
Sudden unexpected
deaths in infancy must be referred to the coroner for
certication. The coroner can record an open verdict in
cases in which the cause of death cannot be determined
from the facts, the facts cannot be proved, or there is real
doubt as to the cause of death.
21
Our classication of
these cases in the light of the available information is
reported in the text.
The index death is the infant death that prompted the
family to join the CONI programme. Enrolled infants
are CONI babies, and death of a baby on the
programme is a CONI death. During the study, two
families had a second CONI death.
CONI deaths were notied to the central ofce by local
coordinators. When death was not unexpected, the local
coordinator, local paediatrician, or both, provided
details. For sudden unexpected deaths in infancy,
permission to contact the parents was sought from the
paediatrician and family doctor. To avoid legal
involvement, the enquiry was offered to parents only
after the initial police and legal proceedings had been
completed. (Two cases were reopened after our
enquiries and JLE was involved in the legal proceedings.)
Parents were approached through their health visitor,
who explained the enquiry procedure and completed
consent forms with them. The enquiry consisted of a
home interview with one of us (JLE, RCC, CD-W), a
review of pathology reports and histological sections of
both the index and the CONI death by an experienced
paediatric pathologist (JLE, JH), and a case discussion in
the general practitioners surgery with professionals
involved with the family. Interviews were based on
previous condential enquiries and lasted 23 h.
2224
To
allow the best rapport to be achieved, interviews were
unstructured. They covered family history; background
and childhood of both partners; social circumstances;
child-rearing practices; smoking; and alcohol and drug
use. Life histories were taken of all children of both
partners, including a detailed account of events
surrounding any child death. The concluding case
discussion aimed to establish the likely cause of death, to
identify contributory factors, and to plan care for any
subsequent child. The general practitioner or health
visitor reported the conclusions back to the parents.
Our enquiries differ from those of the CESDI SUDI
studies in that their cases were examined by use of
standard protocols shortly after the death. By contrast,
our interviews and the post-mortem investigations were
inevitably retrospective, but they were carried out by
experts using an informal though structured approach
designed to detect clues as to the causes of the deaths.
The study was part of the continuing audit of the
CONI programme and has ethical approval from the
South Shefeld Research Ethics Committee.
Exact 95% CI (STATA version 5.0) are reported.
Role of the funding source
The sponsors of this study had no role in study design;
collection, analysis, or interpretation of data; or the
writing of this report. The corresponding author had full
access to all the data in the study. The authors were
jointly responsible for the decision to publish and at all
stages in the publication of this article.
30 www.thelancet.com Vol 365 January 1, 2005
Articles
Results
From 1988 to December, 1999, 6373 babies from
5229 families had completed the CONI programme.
1144 (22%) were siblings to an earlier CONI baby. 57
(89 per 1000) CONI babies died under the age of
1 year. Nine deaths were not unexpected. 48 presented
suddenly and unexpectedly; seven of these (110 per
1000 [95% CI 042226]) were classied as probable
homicides and 41 (643 per 1000 [462873]) as
natural sudden unexpected deaths in infancy. Thus, the
proportion of sudden unexpected deaths in infancy was
586 (260155) times greater than the proportion of
probable homicides. Between 1993 and 1999, coroners
in England and Wales certied 113 per
1000 postperinatal infant deaths, almost all natural
sudden unexpected deaths in infancy. Therefore, the
ratio of the rate of natural sudden unexpected death in
infancy among CONI infants to that in the general
population (ie, the relative risk of recurrence) was at
least 571 (410774). Elsewhere, we will show that
the relative risk of recurrence of SIDS was 59
(4181).
There were nine CONI deaths from disorders with
recognised poor prognoses, which include anomalies
and birth-related diseases. Such CONI deaths occur
because programme registration occurs before birth in
most cases.
The remaining 48 unexpected deaths are reported in
the groups shown in tables 1 and 2.
Of the 46 unexpected rst CONI deaths, homicide
accounted for six, four overt and two covert. Of the four
overt homicides, one was due to a skull fracture and
brainstem injury, and three were due to overt non-
accidental injury from which no prosecutions followed.
In two of these three cases, legal investigation failed to
identify which parent was culpable. The third death was
attributed to a juvenile babysitter and, although the
parents were responsible for the choice of babysitter,
they had no active involvement in the death.
One of the two covert homicides was initially classied
as unascertained and the conclusion reached was that
there was insufcient evidence for a criminal conviction.
Seven rib fractures were found in the CONI case but
whether these were due to resuscitation could not be
resolved. After our enquiries, a family court judge found,
on the balance of probabilities, that the father had killed
both the index infant and the CONI infant. In the second
case, the index death and the rst CONI death were both
registered as SIDS, at the ages of 45 weeks and 28 weeks.
Our enquiry after the CONI death failed to discover any
explanation for either death, despite healed rib fractures
in the CONI case, but identied grave concerns about
the father. Subsequently the father had a further child
on the CONI programme with a new partner 16 months
later. He confessed to suffocating this child at 12 weeks
and was convicted of suffocating all three children.
Therefore the rst CONI death was a covert homicide.
Our enquiries were completed for 27 (68%) of the
remaining 40 deaths and only partially completed for
13 (32%). In every case we have used all available
information. Table 2 classies the sudden unexpected
deaths in infancy in the 27 families with completed
enquiries as explained or SIDS. Both deaths were
explained for three CONI infants. In two families both
deaths were attributed to the same disorder (one
deciency of very-long-chain acyl-coA dehydrogenase
[VLCAD], and one long-QT syndrome). Diagnosis was
Final classication by Final allocation
Total homicide Total natural Total
Coroner and CONI panel Coroners court only Family court Criminal court Appeal court Due to parents (licide) Due to others (not family)
First CONI death
Overt homicide .. .. 3 1 .. 3 1 .. 4
Probable covert homicide
Initially SIDS (1) .. .. 1* .. 1 .. .. 1
Initially unascertained .. .. 1 .. .. 1 .. .. 1
Natural (explained/SIDS) 27 .. .. .. .. .. 27 27
Natural (incomplete information) .. 11 .. 1 1 .. 13 13
Second CONI death
Confessed homicide .. .. .. 1 1 .. .. 1
SIDS 1 .. .. .. .. .. 1 1
Total 6 1 41 48
Excludes four families in which either the index death or the CONI death occurred after 1 year of age. *Initially registered as SIDS, but the father was subsequently convicted of three homicides, including this child. Some of the
families were the subject of Social Service case conferences. We have no information that any were the subject of Family Court or criminal proceedings in relation to the death of the CONI baby.
Table 1: A general grouping of the deaths
Index death CONI death Number
Explained Explained* 3 (11%)
Explained SIDS 4 (15%)
SIDS Explained 2 (7%)
SIDS SIDS 18 (67%)
*One death was recorded as an open verdict. Three deaths were recorded as open
verdicts.
Table 2: Classication of deaths in 27 families with completed enquiries
according to explained or SIDS
Articles
assisted or conrmed by the birth of a third child found
to have the same disorder. Rib fractures, attributed to
resuscitation, were found in the CONI infant with
VLCAD. An open verdict was recorded for the child with
a previously identied cardiac arrhythmia. In the third
family, the index and CONI deaths were explained by
different causes. The index death was considered to be
multifactorial: a dysmature baby died during the
invasive phase of an acute lung infection. The CONI
death was attributed to endocardial broelastosis and
elastosis of the lung.
In six families, although one death was attributed to
SIDS, a cause was found for the other childs death (two
bronchopneumonia, two signicant congenital
anomalies, one myocarditis, and one septicaemia).
In 18 families there were two cases of SIDS. All these
families had high frequencies of SIDS risk factors,
including smoking, illicit drug use, and unemployment.
Our enquiries also revealed that many of the families
demonstrated disorganisation and social deprivation. In
at least six families (ve mothers and one partner), there
were mental-health concerns, in most cases depression.
Three fathers had been in prison, and six mothers and
three partners were described as having been in violent
relationships. Another case was an unsupported mother
known to have abused an older child; the CONI infant
who died showed pronounced failure to thrive, and
despite an open verdict and after review, including the
pathology of both cases, we have classied both deaths
as SIDS. The mothers subsequent CONI infant
survived.
Of the 36 deaths in these 18 families, nine of the
babies were in bed with one or both parents (one being
found under the mother), and one was sleeping on a
settee with a parent. Two of these nine infants were
under 10 days old (one CONI death was recorded as an
open verdict). In ve of these cases asphyxia could not be
proven or excluded as the cause of death. In nine further
cases there were pathological ndings compatible with
an asphyxial component to the death.
25,26
A history of apparent life-threatening event was
reported for three of these 18 CONI infants, one
subsequently diagnosed as due to an acute respiratory
infection, but for no index child. In two families both the
mother and her partner were profoundly deaf, one of
these CONI deaths being recorded as an open verdict.
None of the infants of the deaf parents were reported to
be congenitally deaf and one, a CONI death, had been
investigated in hospital for an apparent life-threatening
event or suspected apnoea but nothing was found. Long-
QT syndrome seems unlikely to be the cause of these
deaths.
27
In four of these 18 families, the CONI baby who
died was the second or third CONI baby in the family,
and ve further families have subsequent surviving
CONI infants.
Although the infants in these 18 families categorised
as having had two cases of SIDS were at high risk, in no
case was there evidence of abuse at autopsy, and
enquiries and case discussions gave no grounds for us
to think that any of these deaths were unnatural.
However, in four cases, concern was expressed about
the safety of any subsequent child in the family. Three
of these families had six subsequent children, some
with other partners. Five of them were on the CONI
programme. Concerns were recorded about some, but
none were taken into care while on the CONI
programme and all are known to have survived while on
the programme.
There remain 13 families for which data were
insufcient to show whether the deaths were explained
or due to SIDS. There was partial information on these
families. Eight of these CONI deaths were certied as
SIDS (International Classication of Diseases, 9th revision,
causes 798 or 7999). None were attributed to homicide
or non-accidental injury.
Of the 13 families, enquiries were declined by seven
(14 deaths). One baby had congenital abnormalities. In
one CONI death, two fresh rib fractures and a healing
fractured clavicle were discovered at autopsy. One
mother had had ten livebirths, two previous sudden
unexpected deaths in infancy, and a CONI death. After
the CONI death, all the children living with her were
taken into care. Four of the 14 infants who died in these
seven families died while sleeping with a parent on a
settee and two while sleeping in the parents bed.
The remaining six of the 13 families were lost to
investigation, one because the mother was murdered
(the CONI death was certied as SIDS). In two families
there were extensive legal investigations. In one, the
conclusion reached was that the deaths were
unexplained; in the other, suspected septicaemia could
not be conrmed. Of the other three, two of the CONI
infants died while in bed with their parents. There was
no evidence of injury in any of these three and no
prosecutions have followed despite legal involvement.
However, in these three families, other older children
were taken into care, in two cases because they were
already thought to be at risk at the time of the CONI
death.
Thus, after all investigations, of the 46 rst unexpected
CONI deaths, ve were probably licides, one was a
homicide by a babysitter, and 40 were sudden
unexpected deaths in infancy, none of which were
deemed to have been unnatural. Therefore, the odds that
the rst unexpected death on the CONI programme was
licide versus an unexpected natural death were ve to
40. However, for the total unexpected deaths, the
Group Number Probability (95% CI)
Natural 40 087 (074095)
Unnatural 6 013 (005026)
Table 3: Probability that index death and rst CONI death were both
sudden unexpected deaths in infancy versus one or both were unnatural
Articles
probability that the CONI death was natural versus the
probability that it was unnatural was 013 versus 087
(table 3). The corresponding odds ratio that two
unexpected deaths is natural versus unnatural is 67
(28194).
Two of the rst 46 CONI deaths were covert homicide.
There were 18 families in which both the index death
and rst CONI death were SIDS. Therefore, when both
deaths were unexplained, in 18 (90%) cases both deaths
were SIDS and in only two (10%) cases were one or both
deaths probably unnatural (table 4). The corresponding
odds ratio that two unexplained deaths were both SIDS
versus probably unnatural was 90 (22800).
There were too few cases to indicate whether either of
these ratios was affected by socioeconomic
circumstances.
There were two families who lost two CONI infants in
addition to the index case. In one, there were three
homicides. In the other, twins died 6 weeks apart. The
death of the rst twin was classied as SIDS after
investigations; the respiration monitor of the second
twin failed to ring and the coroner recorded an open
verdict. Another CONI death (enquiries declined)
occurred in a family of several children in which there
had previously been two sudden unexpected deaths in
infancy; all three deaths were registered in categories
usually classied as SIDS. From registration data of the
5229 families, we identied a further six families who
had lost three infants before participating in the CONI
scheme. Of these, one family was reported to have had
three cases of SIDS and the other ve at least one
explained death. All six families had a CONI infant that
survived. In addition, in the period after January, 2000,
another family that was lost to the study has had a
second death on the CONI programme; after extensive
investigation all three deaths remained unexplained.
Thus, in these ten families there was one triple homicide
and four families who all apparently had three cases of
SIDS.
There were two deaths in four other families on the
programme, which have been excluded because either
the index death or the CONI death occurred after the age
of 1 year. In one of these families, both the index death at
13 months and the CONI death had pathology atypical of
SIDS and suggesting metabolic disturbance; a third
sibling developed cardiac arrhythmia and was found to
have a low selenium concentration, as had a fourth
sibling. Two CONI deaths at ages 20 and 21 months
were due to homicide, one being overt. The last family
could not be traced.
Discussion
When considering these and many previous ndings, it
is essential to distinguish between cases in which
homicide was or might have been legally proven and
cases in which homicide appears to be the most
probable explanation.
12,17,18,19
Emerys work on this issue
was primarily concerned with the identication and
protection of subsequent infants of families in which
homicide appeared probable.
17,18,23,24
Similarly, in this
study, one object of our condential enquiries was to
identify cases of probable licide in which the evidence
might be insufcient to convict.
We have classied seven CONI infant deaths as
probably unnatural. Two criminal convictions
accounted for three CONI deaths, one of which was
initially considered natural and was, therefore, a covert
homicide. Three were due to overt injuries in which
there have been no criminal prosecutions. Finally, in
one case the family court concerned with the care of
siblings found on the balance of probabilities that the
father was responsible for both deaths.
The recent High Court judgment relates to the need
for indisputable evidence for criminal convictions and
the associated mandatory life sentence.
28
It may also
have implications for rulings of family courts. Review
of our cases suggests that the conclusions relating to
cases of overt homicide are unlikely to be changed
because all had evidence of violence. The court rulings
relating to the two cases designated covert homicide
may be reviewed, but in both cases licide appears the
most probable explanation.
In addition, in 27 cases we agreed with the
professionals involved that there was no suspicion of
licide, including 18 cases in which both deaths were
unexplained. In many, but not all, of the latter there
may have been less than optimum care. Not
surprisingly, therefore, in four of these 18 cases
concern was expressed about the safety of subsequent
children. None of the subsequent children were taken
into care while on the CONI programme, and we know
that six children in three families survived. But in one
case initially unexplained, the concern expressed about
the father was subsequently justied.
The proportion of homicides in this study appears
much lower than in previous studies of recurrence.
Levine and Bacon
29
have summarised an earlier CONI
report as indicating that 30% of subsequent deaths
were due to homicide and that this proportion was
similar to those in Emerys earlier studies. However,
the gures they used from the earlier CONI report
included seven overt homicides (two over the age of
1 year) and two open verdicts.
30
In this fuller study, we
have counted one as unexplained and the other as
apparently unexplained (both families had subsequent
infants who were not taken into care). Differences of
denition probably account for discrepant ndings of
different studies.
Group Number Probability (95% CI)
Natural 18 090 (068099)
Probably unnatural 2 010 (001032)
Table 4: Probability that index death and rst CONI death were both
SIDS versus one or both were probably covert homicide
Articles
The CESDI SUDI study reports that in 64% (22 in
346) of cases classied as SIDS, maltreatment was
thought to be the main cause of death after review.
8
Our
estimate that 10% (two of 20) of cases of two
unexplained sudden unexpected deaths in infancy were
probably unnatural is higher than their estimate but not
signicantly so (p=0631, Fishers exact test). However,
cases might have been classied differently in these
studies. Also, we cannot exclude the possibility in our
study that some of the 13 cases in which enquiries were
not possible were cases of covert homicide.
As stated, we have counted as SIDS several cases in
which asphyxia might have been a contributory factor.
However, in such cases there is no way to distinguish
reliably between SIDS and unnatural death if there is no
confession or evidence of violence.
3,5
Parents may make a
false confession because of overwhelming feelings of
guilt, not because of what they have actually done.
31
If
any of these infants were suffocated, our enquiries
concluded that the suffocation was unintentional.
In three families (one excluded from the main study),
there was clear evidence of congenital disease that
emerged in both the index death and the CONI death
and in subsequent children who were successfully
treated.
Our ndings can be criticised because participation in
the CONI programme is voluntary, although we know of
only 4% (200 in 5200) who declined involvement.
30
Enquiries suggested that those who did not participate
had either lost a baby a long time ago or wanted to leave
the experience behind them. Parents covertly
responsible for a death could easily opt out. However,
they could, equally, participate to avoid suspicion. Emery
and Meadow have suggested that Munchhausen-by-
proxy syndrome (ie, fabricated or induced illness) is a
factor in unnatural unexpected infant deaths.
7,32
Such
parents might even be attracted to the CONI
programme.
Statements that the risk of recurrence of SIDS is
incredibly small are intended to suggest that the
probability of such deaths being natural is zero. Selective
media reporting can endorse this misconception. Our
data suggest that second deaths are not rare and that the
majority, 8090% (40 in 45; or 18 in 20), are natural.
Families who have experienced three unexpected deaths
also occur. The study included two families in which
there were two CONI deathsone triple SIDS and one
triple licide.
This study is the largest follow-up of families who have
had a sudden unexpected and unexplained infant death.
The CONI programme has been available in over 75 %
of districts in England, Wales, and Northern Ireland
since 1994, and we would know if many eligible mothers
declined to participate. We have therefore probably
included the majority of families in which there have
been two or three sudden and unexpected deaths in
recent years. Consequently, although child abuse is not
uncommon, from the best available data, we believe that
the occurrence of a second or third sudden unexpected
death in infancy within a family, although relatively rare,
is in most cases from natural causes. For a host of
reasons, not the least of which is the protection of
parents from false accusations, it is essential that all
sudden unexpected infant deaths are submitted to a
detailed, expert investigation like this study, which
includes a full family history, clinical history, and
paediatric autopsy. Also, adequate post-mortem material
must be retained from every unexplained infant death
for re-examination in the event of recurrence.
Contributors
J L Emery, who died in May, 2000, was a founding director of the CONI
programme. He was largely responsible for the setting up of this study
and for investigation of the earlier cases. The authors (except J L Emery)
comprise the Steering Committee of the CONI project. In this study
R G Carpenter was responsible for the tabulation and analysis of the
data on the index and CONI deaths. A Waite is the National CONI
Coordinator whose role is to set up and maintain the CONI programme
in each of the health districts; she abstracted the data for this study from
case notes. R C Coombs and C Daman-Willems conducted the
enquiries during the past 5 years and advised on the abstraction of the
data. A McKenzie was responsible for data processing of all CONI
records and for tabulation and analysis of national statistics of infant
mortality from raw ONS data tapes. J Huber reviewed J L Emerys post-
mortem ndings and completed the autopsy reviews of the later cases.
All these authors (except JLE) were involved in drafting this paper.
Conict of interest statement
None of the authors had any legal involvement with any of the 28 cases
classied by the CONI panel as natural deaths. After the reviews were
complete, JLE was legally involved with two cases that were reopened
after our enquiries had been completed and which were nally classied
as covert licide.
The rst was the case of the triple licide. The rst CONI baby died
in April, 1993. JLE conducted enquiries and the autopsy review of the
index and CONI deaths. The concluding case discussion was held in
May the next year, when the deaths were classied as SIDS, although
JLEs notes indicate concern about the father. The fathers third child
was born in the following August and was also on the CONI
programme. Social services held a case conference in September, which
was not attended by JLE. The third baby died in November. The police
asked JLE for a statement the following March and he was called as an
expert witness in September (17 months after the case conference on
the rst CONI death). The following January, the father was convicted
of causing all three deaths.
In the second case, a CONI baby died in December, 1994, and
initially a prosecution seemed likely. The following March a concerned
paediatrician in another district asked JLE to conduct enquires because
the mother was again pregnant. In April, JLE was told that the police
found no evidence that would justify a prosecution, and the CONI death
was certied as unascertained. The family was then interviewed by JLE
as part of our enquiries and the autopsies of both index and CONI
infants were reviewed by him. In July, JLE chaired a case discussion at
which he noted that in 6000 cases he had never previously seen similar
rib fractures as a result of a normal resuscitation procedure. 10 days
later JLE was asked to assist with care proceedings. 10 months later a
judge asked him to coordinate a telephone discussion between expert
witnesses. The next month the judge issued an interim care order
because on the balance of probabilities the father had killed both
children. The case is classied as probable covert licide.
Our case conferences for this study were all complete in August,
2001. However, some of us were legally involved subsequently with
three cases for which enquires could not be conducted. In the rst, JLE
was consulted about the care of an infant born to a mother initially
convicted of killing two previous children. The second occurred in
autumn 2002, when RGC and JH became involved with the defence in
Articles
the trial of a mother who had lost three infants; she was acquitted. The
courts ndings in regard to these cases are included with the incomplete
cases.
After the rst draft of this paper was submitted for publication in
August, 2003, RGC was involved with the defence in an appeal relating
to multiple deaths of infants who were not on the CONI programme.
4 years previously, JLE had been consulted about the care of a surviving
child of this family, who had been on the CONI programme.
Standard legal fees were received by JLE, RGC, and JH for court work
described above. AW, RCC, CD-W, and AM received no fees for work
related to any of the cases described in this report. As paediatric
pathologists, JLE and JH have been involved over the years in cases of
infant death both for the defence and prosecution. In May, 2002, RGC
was rst involved for the defence of a case in Scotland, where the CONI
programme is not available.
Acknowledgments
We thank Anne Hudson for coordinating the studies of the CONI
deaths; Margaret Evans for assistance with the pathology reviews of
some cases; and Chris Bacon for encouragement and helpful
comments. CONI is supported by an annual grant from the Foundation
for the Study of Infant Deaths. We are especially indebted to the
bereaved CONI parents and others involved with these deaths for the
cooperation that has made this study possible.
References
1 Barrett AM. Sudden death in infancy. In: Gairdner D, ed. Recent
advances in paediatrics. London: Churchill, 1954: 30120.
2 Templeman C. Two hundred and fty eight cases of suffocation of
infants. Edinburgh Med J 1892; 38: 32229.
3 Valdes-Depena MA. Sudden and unexpected death in infancy:
a review of the world literature 195466. Pediatrics 1967; 39:
12338.
4 Kukull WA, Peterson DR. Sudden infant death and infanticide.
Am J Epidemiol 1977; 106: 48586.
5 Guntheroth WG. Crib death: the sudden infant death syndrome,
3rd edn. Armonk, NY: Futura, 1995.
6 DiMaio VJ, DiMaio D. Forensic pathology. New York: CRC Press,
1989.
7 Meadow R. Unnatural sudden infant death. Arch Dis Child 1999;
80: 714.
8 Fleming P, Blair P, Bacon C, Berry J. Sudden unexpected deaths in
infancy: the CESDI SUDI Studies 19931996. London: Stationery
Ofce, 2000.
9 Meadow R. Fatal abuse and smothering. In: Meadow R, ed. ABC
of child abuse, 3rd edn. London: BMJ Publishing Group, 1997:
2729.
10 Guntheroth WG, Lohmann R, Speirs PS. Risk of sudden infant
death syndrome in subsequent siblings. J Pediatr 1990; 116: 52024.
11 Peterson DR, Chinn NM, Fisher LD. The sudden infant death
syndrome: repetition in families. J Pediatr 1980; 97: 26367.
12 Beal SM, Blundell HK. Recurrence of incidence of sudden infant
death syndrome. Arch Dis Child 1988; 63: 92430.
13 Oyen N, Skjaerven R, Irgens LM. Population-based recurrence risk
of sudden infant death syndrome compared with other infant and
fetal deaths. Am J Epidemiol 1996; 144: 30005.
14 Firstman R, Talan J. SIDS and infanticide. In: Byard RW,
Krous HF, eds. Sudden infant death syndrome. London: Arnold,
2001: 291300.
15 Vege , Rognum TO, Lberg EM, et al. Diagnosis of sudden infant
death in the Nordic countries since 1970, revised. In: Rognum TO,
ed. Sudden infant death syndrome: new trends in the nineties.
Oslo: Scandinavian University Press, 1995: 6769.
16 The President, Royal Statistical Society. RSS comments on
statistics misuse in courts. RSS News, Dec 2001.
17 Emery JL. Families in which two or more cot deaths have occurred.
Lancet 1986; 1: 31315.
18 Wolkind S, Taylor EM, Waite AJ, Dalton M, Emery JL. Recurrence
of unexpected infant death. Acta Paediatr 1993; 82: 87376.
19 Stanton AN. Sudden unexpected death in infancy associated with
maltreatment: evidence from long term follow up of siblings.
Arch Dis Child 2003; 88: 699701.
20 McKenzie A, Waite AJ. The CONI project: parents evaluation of
the care provided to the rst 2000 babies. Health Visitor 1995; 68:
23940.
21 Levine M. Levine on Coroners Court. London: Sweet and Maxwell,
1999.
22 Carpenter RG, Gardner A, McWeeny PM, Emery JL. Multi-stage
scoring system for identifying infants at risk of unexpected death.
Arch Dis Child 1997; 52: 60612.
23 Taylor EM, Emery JL. Two year study of the causes of postperinatal
deaths classied in terms of preventability. Arch Dis Child 1982; 57:
66873.
24 Taylor EM, Emery JL. Family and community factors associated
with infant deaths that might be preventable. BMJ 1983; 287:
87174.
25 Yukawa N, Carter N, Rutty G, Green MA. Intra-alveolar
haemorrhage in sudden infant death syndrome: a cause for
concern? J Clin Pathol 1999; 52: 58187.
26 Berry PJ. Intra-alveolar haemorrhage in sudden infant death
syndrome: a cause for concern? J Clin Pathol 1999; 52: 55354.
27 Splawski I, Timothy KW, Vincent GM, et al. Molecular basis of the
long-QT syndrome associated with deafness. N Engl J Med 1997;
336: 156267.
28 Lord Justice Judge, Justice Rafferty, Justice Pitchers. Regina -v-
Cannings. Judgement. Neutral Citation No. EWCA Crim. 01 Case
No.: 200201711 D3. London: Royal Courts of Justice, January 2004.
29 Levene S, Bacon CJ. Sudden unexpected death and covert homicide
in infancy. Arch Dis Child 2004; 89: 44347.
30 Waite A, McKenzie A, Carpenter RG, et al. Report on 5000 babies
using the CONI (Care of Next Infant) Programme. London:
Foundation for the Study of Infant Deaths, 1998.
31 Emery JL. Infanticide, licide and cot death. Arch Dis Child 1985;
60: 50507.
32 Emery JL. Child abuse, sudden infant death syndrome and
unexpected infant death. Am J Dis Child 1993; 147: 1097100.
u
n
d
e
r
s
t
o
o
d
e
i
t
h
e
r
a
s
a
f
o
r
m
o
f
s
h
a
r
i
n
g
(
o
f
t
h
e
s
o
r
t
d
e
s
c
r
i
b
e
d
a
b
o
v
e
)
o
r
a
s
a
n
i
n
s
t
a
n
c
e
o
f
c
o
m
p
e
n
s
a
t
i
o
n
,
1
1
6
a
l
i
n
e
o
f
a
r
g
u
m
e
n
t
t
h
a
t
w
a
s
r
e
j
e
c
t
e
d
b
y
t
h
e
m
a
j
o
r
i
t
y
o
n
t
h
e
b
a
s
i
s
t
h
a
t
h
i
s
d
e
c
i
s
i
o
n
t
o
m
o
v
e
t
o
a
c
a
d
e
m
i
a
w
a
s
n
o
t
m
o
t
i
v
a
t
e
d
b
y
t
h
e
f
a
m
i
l
y
s
n
e
e
d
s
b
u
t
w
a
s
s
i
m
p
l
y
M
r
G
r
a
n
a
t
i
n
o
s
o
w
n
c
h
o
i
c
e
.
1
1
7
W
h
a
t
i
s
c
l
e
a
r
i
s
t
h
a
t
t
h
i
s
i
s
s
u
e
w
i
l
l
p
r
o
v
i
d
e
f
e
r
t
i
l
e
g
r
o
u
n
d
f
o
r
a
r
g
u
m
e
n
t
i
n
f
u
t
u
r
e
c
a
s
e
s
,
a
s
s
p
o
u
s
e
s
w
h
o
s
e
a
g
r
e
e
m
e
n
t
s
f
a
l
l
w
i
t
h
i
n
t
h
e
m
a
j
o
r
i
t
y
s
p
r
i
n
c
i
p
l
e
i
n
f
a
v
o
u
r
o
f
u
p
h
o
l
d
i
n
g
p
r
e
-
n
u
p
t
i
a
l
a
g
r
e
e
m
e
n
t
s
,
a
t
l
e
a
s
t
t
o
t
h
e
e
x
t
e
n
t
o
f
p
r
e
c
l
u
d
i
n
g
e
q
u
a
l
s
h
a
r
i
n
g
,
s
e
e
k
t
o
m
a
x
i
m
i
s
e
t
h
e
i
r
r
e
s
i
d
u
a
l
c
l
a
i
m
t
h
a
t
n
e
e
d
-
o
r
c
o
m
p
e
n
s
a
t
i
o
n
-
b
a
s
e
d
p
r
o
v
i
s
i
o
n
w
o
u
l
d
b
e
f
a
i
r
.
D
e
t
e
r
m
i
n
i
n
g
,
a
s
a
m
a
t
t
e
r
o
f
p
r
i
n
c
i
p
l
e
,
h
o
w
b
r
o
a
d
s
u
c
h
r
e
s
i
d
u
a
l
c
l
a
i
m
s
s
h
o
u
l
d
b
e
w
i
l
l
b
e
a
k
e
y
c
o
n
s
i
d
e
r
a
t
i
o
n
f
o
r
t
h
e
L
a
w
C
o
m
m
i
s
-
s
i
o
n
a
s
i
t
d
e
l
i
b
e
r
a
t
e
s
o
n
i
t
s
c
o
n
s
u
l
t
a
t
i
o
n
r
e
s
p
o
n
s
e
s
.
I
n
d
e
e
d
,
t
h
e
L
a
w
C
o
m
m
i
s
s
i
o
n
w
i
l
l
n
e
e
d
t
o
d
e
c
i
d
e
w
h
e
t
h
e
r
t
o
a
d
o
p
t
t
h
a
t
m
o
d
e
l
o
f
r
e
f
o
r
m
a
t
a
l
l
,
o
r
i
n
s
t
e
a
d
t
o
b
a
s
e
t
h
e
c
o
u
r
t
s
p
o
w
e
r
t
o
s
e
t
a
n
a
g
r
e
e
m
e
n
t
a
s
i
d
e
o
n
s
o
m
e
b
r
o
a
d
e
r
b
a
s
i
s
,
s
u
c
h
a
s
s
i
g
n
i
-
c
a
n
t
i
n
j
u
s
t
i
c
e
.
1
1
8
I
n
t
h
e
m
e
a
n
t
i
m
e
,
w
h
a
t
e
v
e
r
i
t
s
s
c
o
p
e
m
a
y
b
e
,
t
h
e
c
o
n
t
i
n
u
e
d
a
v
a
i
l
-
a
b
i
l
i
t
y
o
f
r
e
l
i
e
f
o
n
n
e
e
d
^
o
r
c
o
m
p
e
n
s
a
t
i
o
n
-
b
a
s
e
d
g
r
o
u
n
d
s
^
d
e
s
p
i
t
e
a
n
a
g
r
e
e
m
e
n
t
,
s
e
r
v
e
s
t
o
e
n
s
u
r
e
t
h
a
t
m
a
r
r
i
a
g
e
d
o
e
s
r
e
m
a
i
n
d
i
s
t
i
n
c
t
f
r
o
m
c
o
h
a
b
i
t
a
t
i
o
n
.
M
a
r
r
i
a
g
e
i
s
n
o
t
d
e
a
d
.
A
n
d
t
h
e
r
e
w
a
s
n
o
n
e
e
d
o
n
t
h
e
f
a
c
t
s
o
f
t
h
i
s
c
a
s
e
t
o
m
a
k
e
f
u
r
t
h
e
r
p
r
o
v
i
s
i
o
n
f
o
r
M
r
G
r
a
n
a
t
i
n
o
i
n
o
r
d
e
r
t
o
s
e
c
u
r
e
i
t
s
c
o
n
t
i
n
u
e
d
v
i
t
a
l
i
t
y
.
E
x
t
e
n
d
i
n
g
t
h
e
C
o
n
f
u
s
i
o
n
A
b
o
u
t
B
a
y
e
s
B
e
r
n
a
r
d
R
o
b
e
r
t
s
o
n
n
,
G
.
A
.
V
i
g
n
a
u
x
n
n
a
n
d
C
h
a
r
l
e
s
E
.
H
.
B
e
r
g
e
r
n
n
n
I
n
R
v
T
[
2
0
1
0
]
E
W
C
A
C
r
i
m
2
4
3
9
,
[
2
0
1
1
]
1
C
r
A
p
p
R
e
p
8
5
,
t
h
e
C
o
u
r
t
o
f
A
p
p
e
a
l
i
n
d
i
c
a
t
e
d
t
h
a
t
m
a
t
h
e
m
a
t
i
c
a
l
f
o
r
m
u
l
a
e
,
s
u
c
h
a
s
l
i
k
e
l
i
h
o
o
d
r
a
t
i
o
s
,
s
h
o
u
l
d
n
o
t
b
e
u
s
e
d
b
y
f
o
r
e
n
s
i
c
s
c
i
e
n
t
i
s
t
s
t
o
a
n
a
l
y
s
e
d
a
t
a
w
h
e
r
e
r
m
s
t
a
t
i
s
t
i
c
a
l
e
v
i
d
e
n
c
e
d
i
d
n
o
t
e
x
i
s
t
.
U
n
f
o
r
t
u
n
a
t
e
l
y
,
w
h
e
n
c
o
n
s
i
d
e
r
i
n
g
t
h
e
f
o
r
e
n
s
i
c
s
c
i
e
n
t
i
s
t
s
e
v
i
d
e
n
c
e
,
t
h
e
j
u
d
g
m
e
n
t
c
o
n
s
i
s
t
e
n
t
l
y
c
o
m
m
i
t
s
a
b
a
s
i
c
l
o
g
i
c
a
l
e
r
r
o
r
,
t
h
e
t
r
a
n
s
p
o
-
s
i
t
i
o
n
o
f
t
h
e
c
o
n
d
i
t
i
o
n
a
l
w
h
i
c
h
i
n
d
i
c
a
t
e
s
t
h
a
t
t
h
e
B
a
y
e
s
i
a
n
a
r
g
u
m
e
n
t
h
a
s
n
o
t
b
e
e
n
u
n
d
e
r
s
t
o
o
d
a
n
d
e
x
t
e
n
d
s
t
h
e
c
o
n
f
u
s
i
o
n
s
u
r
r
o
u
n
d
i
n
g
i
t
.
T
h
e
j
u
d
g
m
e
n
t
a
l
s
o
f
a
i
l
s
t
o
d
i
s
t
i
n
g
u
i
s
h
b
e
t
w
e
e
n
t
h
e
v
a
l
i
d
i
t
y
o
f
t
h
e
r
e
l
a
t
i
o
n
s
h
i
p
s
i
n
a
f
o
r
m
u
l
a
a
n
d
t
h
e
p
r
e
c
i
s
i
o
n
o
f
t
h
e
d
a
t
a
.
W
e
e
x
p
l
a
i
n
w
h
y
t
h
e
B
a
y
e
-
s
i
a
n
m
e
t
h
o
d
i
s
t
h
e
c
o
r
r
e
c
t
l
o
g
i
c
a
l
m
e
t
h
o
d
f
o
r
a
n
a
l
y
s
i
n
g
f
o
r
e
n
s
i
c
s
c
i
e
n
t
i
c
e
v
i
d
e
n
c
e
,
h
o
w
i
t
w
o
r
k
s
a
n
d
w
h
y
m
a
t
h
e
m
a
t
i
c
a
l
f
o
r
m
u
l
a
e
c
a
n
b
e
u
s
e
f
u
l
e
v
e
n
w
h
e
r
e
r
m
s
t
a
t
i
s
t
i
c
a
l
d
a
t
a
i
s
l
a
c
k
i
n
g
.
T
h
e
p
r
o
b
a
b
i
l
i
t
y
t
h
a
t
a
n
a
n
i
m
a
l
w
i
t
h
f
o
u
r
l
e
g
s
i
s
a
c
o
w
i
s
n
o
t
t
h
e
p
r
o
b
a
b
i
l
i
t
y
t
h
a
t
a
c
o
w
h
a
s
f
o
u
r
l
e
g
s
.
A
t
t
h
e
s
c
e
n
e
o
f
a
c
r
i
m
e
,
a
f
o
o
t
w
e
a
r
m
a
r
k
i
s
f
o
u
n
d
.
A
s
u
s
p
e
c
t
s
s
h
o
e
s
a
r
e
e
x
a
m
i
n
e
d
.
T
h
e
y
a
r
e
o
f
t
h
e
s
a
m
e
b
r
a
n
d
a
s
t
h
e
f
o
o
t
w
e
a
r
t
h
a
t
l
e
f
t
t
h
e
m
a
r
k
a
n
d
o
n
e
s
h
o
e
h
a
s
1
1
6
L
a
d
y
H
a
l
e
p
r
i
n
c
i
p
a
l
l
y
r
e
f
e
r
s
t
o
n
e
e
d
i
n
t
h
i
s
s
e
c
t
i
o
n
o
f
h
e
r
j
u
d
g
m
e
n
t
,
b
u
t
d
o
e
s
a
l
s
o
t
a
l
k
a
b
o
u
t
c
o
m
p
e
n
s
a
t
i
n
g
t
h
e
h
y
p
o
t
h
e
t
i
c
a
l
p
a
r
e
n
t
,
i
b
i
d
a
t
[
1
9
3
]
.
1
1
7
i
b
i
d
a
t
[
1
2
1
]
.
1
1
8
F
o
r
t
h
e
f
u
l
l
r
a
n
g
e
o
f
o
p
t
i
o
n
s
o
n
o
e
r
,
s
e
e
L
a
w
C
o
m
m
i
s
s
i
o
n
,
n
4
.
a
b
o
v
e
p
a
r
a
8
.
1
9
.
n
B
a
r
r
i
s
t
e
r
,
W
e
l
l
i
n
g
t
o
n
.
n
n
V
i
c
t
o
r
i
a
U
n
i
v
e
r
s
i
t
y
o
f
W
e
l
l
i
n
g
t
o
n
.
n
n
n
N
e
t
h
e
r
l
a
n
d
s
F
o
r
e
n
s
i
c
I
n
s
t
i
t
u
t
e
.
E
x
t
e
n
d
i
n
g
t
h
e
C
o
n
f
u
s
i
o
n
A
b
o
u
t
B
a
y
e
s
4
4
4
r
2
0
1
1
T
h
e
A
u
t
h
o
r
s
.
T
h
e
M
o
d
e
r
n
L
a
w
R
e
v
i
e
w
r
2
0
1
1
T
h
e
M
o
d
e
r
n
L
a
w
R
e
v
i
e
w
L
i
m
i
t
e
d
.
(
2
0
1
1
)
7
4
(
3
)
4
3
0
^
4
5
5
s
o
m
e
w
e
a
r
f
e
a
t
u
r
e
s
w
h
i
c
h
c
o
r
r
e
s
p
o
n
d
w
i
t
h
f
e
a
t
u
r
e
s
i
n
t
h
e
f
o
o
t
w
e
a
r
m
a
r
k
.
W
h
a
t
c
a
n
a
n
e
x
p
e
r
t
s
a
y
a
b
o
u
t
w
h
e
t
h
e
r
t
h
e
s
u
s
p
e
c
t
s
s
h
o
e
l
e
f
t
t
h
e
f
o
o
t
w
e
a
r
m
a
r
k
a
t
t
h
e
s
c
e
n
e
?
T
o
a
n
s
w
e
r
t
h
i
s
q
u
e
s
t
i
o
n
w
e
h
a
v
e
t
o
g
o
b
a
c
k
t
o
b
a
s
i
c
s
.
T
h
e
n
e
e
d
t
o
d
o
s
o
i
s
i
l
l
u
-
s
t
r
a
t
e
d
b
y
t
h
e
j
u
d
g
m
e
n
t
o
f
t
h
e
C
o
u
r
t
o
f
A
p
p
e
a
l
i
n
R
v
T
[
2
0
1
0
]
E
W
C
A
C
r
i
m
2
4
3
9
,
[
2
0
1
1
]
1
C
r
A
p
p
R
e
p
8
5
(
R
v
T
)
w
h
i
c
h
i
s
c
a
u
s
i
n
g
d
i
s
q
u
i
e
t
i
n
t
h
e
f
o
r
e
n
s
i
c
s
c
i
e
n
t
i
c
w
o
r
l
d
.
1
T
h
i
s
j
u
d
g
m
e
n
t
i
s
b
a
s
e
d
o
n
a
d
i
s
t
i
n
c
t
i
o
n
b
e
t
w
e
e
n
e
v
i
d
e
n
c
e
b
a
s
e
d
o
n
q
u
a
n
-
t
i
a
b
l
e
d
a
t
a
a
n
d
l
e
s
s
q
u
a
n
t
i
a
b
l
e
d
a
t
a
a
n
d
o
n
a
v
e
r
s
i
o
n
t
o
t
h
e
u
s
e
o
f
w
h
a
t
t
h
e
C
o
u
r
t
l
a
b
e
l
s
a
m
a
t
h
e
m
a
t
i
c
a
l
f
o
r
m
u
l
a
w
h
e
n
t
h
e
d
a
t
a
i
s
n
o
t
r
e
a
d
i
l
y
q
u
a
n
t
i
a
b
l
e
.
W
H
A
T
C
A
N
T
H
E
E
X
P
E
R
T
S
A
Y
?
T
o
i
l
l
u
s
t
r
a
t
e
w
h
a
t
a
n
e
x
p
e
r
t
c
a
n
s
a
y
a
b
o
u
t
e
v
i
d
e
n
c
e
l
e
t
u
s
u
s
e
t
h
e
e
x
a
m
p
l
e
o
f
p
a
t
e
r
-
n
i
t
y
e
v
i
d
e
n
c
e
.
A
n
e
x
p
e
r
t
t
o
d
a
y
t
a
k
e
s
a
s
a
m
p
l
e
a
n
d
e
x
a
m
i
n
e
s
t
h
e
D
N
A
,
b
u
t
a
g
e
n
-
e
r
a
t
i
o
n
a
g
o
,
t
h
e
b
l
o
o
d
t
y
p
e
w
a
s
e
x
a
m
i
n
e
d
.
T
h
e
r
e
a
r
e
o
n
l
y
t
w
o
t
h
i
n
g
s
t
h
a
t
t
h
e
e
x
p
e
r
t
c
a
n
a
n
d
m
u
s
t
e
s
t
a
b
l
i
s
h
.
T
h
e
r
s
t
i
s
t
h
e
p
r
o
b
a
b
i
l
i
t
y
t
h
a
t
t
h
e
c
h
i
l
d
w
o
u
l
d
h
a
v
e
t
h
e
d
e
t
e
r
m
i
n
e
d
b
l
o
o
d
t
y
p
e
i
f
t
h
e
d
e
f
e
n
d
a
n
t
w
a
s
i
t
s
f
a
t
h
e
r
.
I
n
s
o
m
e
c
a
s
e
s
,
t
h
e
e
v
i
d
e
n
c
e
w
o
u
l
d
e
x
c
l
u
d
e
t
h
e
d
e
f
e
n
d
a
n
t
,
i
n
o
t
h
e
r
w
o
r
d
s
,
t
h
i
s
p
r
o
b
a
b
i
l
i
t
y
w
o
u
l
d
b
e
z
e
r
o
.
I
n
o
t
h
e
r
c
a
s
e
s
,
c
e
r
t
a
i
n
t
y
c
o
u
l
d
n
o
t
b
e
o
b
t
a
i
n
e
d
,
a
n
d
t
h
e
e
s
t
a
b
l
i
s
h
e
d
p
r
o
b
-
a
b
i
l
i
t
y
o
f
t
h
e
e
v
i
d
e
n
c
e
w
o
u
l
d
b
e
l
a
r
g
e
r
t
h
a
n
z
e
r
o
.
T
h
e
s
e
c
o
n
d
t
h
i
n
g
t
h
e
e
x
p
e
r
t
c
a
n
e
s
t
a
b
l
i
s
h
i
s
t
h
e
p
r
o
b
a
b
i
l
i
t
y
t
h
a
t
t
h
e
c
h
i
l
d
w
o
u
l
d
h
a
v
e
t
h
i
s
b
l
o
o
d
t
y
p
e
i
f
s
o
m
e
o
n
e
e
l
s
e
w
a
s
t
h
e
f
a
t
h
e
r
.
T
h
i
s
w
o
u
l
d
d
e
p
e
n
d
o
n
t
h
e
f
r
e
q
u
e
n
c
y
o
f
t
h
e
b
l
o
o
d
t
y
p
e
i
n
t
h
e
p
o
p
u
l
a
t
i
o
n
(
o
f
p
o
s
s
i
b
l
e
f
a
t
h
e
r
s
)
.
W
h
a
t
t
h
e
c
o
u
r
t
h
a
s
t
o
a
s
s
e
s
s
,
c
o
n
s
c
i
o
u
s
l
y
o
r
o
t
h
e
r
w
i
s
e
,
i
s
t
h
e
p
r
o
b
a
b
i
l
i
t
y
t
h
a
t
t
h
e
d
e
f
e
n
d
a
n
t
w
a
s
t
h
e
f
a
t
h
e
r
.
T
h
e
e
x
p
e
r
t
c
a
n
n
o
t
a
s
s
e
s
s
t
h
i
s
p
r
o
b
a
b
i
l
i
t
y
,
s
i
n
c
e
i
t
r
e
q
u
i
r
e
s
t
h
e
c
o
m
b
i
n
a
t
i
o
n
o
f
a
l
l
t
h
e
e
v
i
d
e
n
c
e
,
n
o
t
j
u
s
t
t
h
a
t
o
f
t
h
e
b
l
o
o
d
t
y
p
e
.
T
h
e
p
r
o
b
a
b
i
l
i
t
y
t
h
a
t
t
h
e
e
x
p
e
r
t
c
a
n
a
s
s
e
s
s
i
s
d
i
e
r
e
n
t
:
t
h
e
p
r
o
b
a
b
i
l
i
t
y
o
f
t
h
e
e
v
i
d
e
n
c
e
w
h
e
n
c
o
n
s
i
d
e
r
i
n
g
t
h
e
a
l
l
e
g
e
d
a
n
d
a
l
t
e
r
n
a
t
i
v
e
f
a
t
h
e
r
s
.
I
n
t
u
i
t
i
v
e
l
y
,
w
e
c
a
n
a
c
c
e
p
t
t
h
a
t
i
f
t
h
e
e
v
i
d
e
n
c
e
i
s
m
o
r
e
p
r
o
b
a
b
l
e
i
f
o
n
e
h
y
p
o
t
h
e
s
i
s
i
s
t
r
u
e
t
h
a
n
i
f
t
h
e
o
t
h
e
r
h
y
p
o
t
h
e
s
i
s
i
s
t
r
u
e
,
t
h
e
e
v
i
d
e
n
c
e
m
a
k
e
s
t
h
e
r
s
t
h
y
p
o
t
h
e
s
i
s
m
o
r
e
p
r
o
b
a
b
l
e
a
n
d
s
c
i
e
n
t
i
c
a
l
l
y
t
h
i
s
t
u
r
n
s
o
u
t
t
o
b
e
c
o
r
r
e
c
t
.
T
h
e
d
e
g
r
e
e
t
o
w
h
i
c
h
t
h
e
e
v
i
d
e
n
c
e
s
u
p
p
o
r
t
s
o
n
e
h
y
p
o
t
h
e
s
i
s
r
e
l
a
t
i
v
e
t
o
a
n
o
t
h
e
r
i
s
t
h
e
s
t
r
e
n
g
t
h
o
f
t
h
e
e
v
i
d
e
n
c
e
.
A
f
u
n
d
a
m
e
n
t
a
l
t
h
e
o
r
e
m
d
e
m
o
n
s
t
r
a
t
e
s
t
h
a
t
t
h
i
s
s
t
r
e
n
g
t
h
o
f
t
h
e
e
v
i
d
e
n
c
e
i
s
d
e
t
e
r
m
i
n
e
d
b
y
h
o
w
m
a
n
y
t
i
m
e
s
m
o
r
e
p
r
o
b
a
b
l
e
t
h
e
e
v
i
d
e
n
c
e
i
s
i
f
o
n
e
h
y
p
o
t
h
e
s
i
s
i
s
t
r
u
e
t
h
a
n
i
f
t
h
e
c
o
n
s
i
d
e
r
e
d
a
l
t
e
r
n
a
t
i
v
e
i
s
t
r
u
e
.
2
T
h
i
s
g
i
v
e
s
a
s
i
n
g
l
e
g
u
r
e
,
w
h
i
c
h
i
s
g
e
n
e
r
a
l
l
y
k
n
o
w
n
a
s
t
h
e
l
i
k
e
l
i
h
o
o
d
r
a
t
i
o
(
i
n
p
a
t
e
r
n
i
t
y
c
a
s
e
s
i
t
i
s
a
l
s
o
k
n
o
w
n
a
s
t
h
e
p
a
t
e
r
n
i
t
y
i
n
d
e
x
)
.
I
t
i
s
t
h
e
t
a
s
k
o
f
t
h
e
c
o
u
r
t
,
n
o
t
t
h
e
e
x
p
e
r
t
w
i
t
n
e
s
s
,
t
o
c
o
m
b
i
n
e
t
h
i
s
w
i
t
h
t
h
e
o
t
h
e
r
e
v
i
d
e
n
c
e
i
n
t
h
e
c
a
s
e
.
3
1
E
x
p
r
e
s
s
i
n
g
e
v
a
l
u
a
t
i
v
e
o
p
i
n
i
o
n
s
:
a
p
o
s
i
t
i
o
n
s
t
a
t
e
m
e
n
t
s
i
g
n
e
d
b
y
n
u
m
e
r
o
u
s
l
e
a
d
i
n
g
f
o
r
e
n
s
i
c
s
c
i
e
n
-
t
i
s
t
s
i
n
(
2
0
1
1
)
5
1
S
c
i
&
J
1
.
2
F
o
r
a
h
i
g
h
l
y
a
c
c
e
s
s
i
b
l
e
e
x
p
l
a
n
a
t
i
o
n
s
e
e
:
I
.
W
.
E
v
e
t
t
,
I
n
t
e
r
p
r
e
t
a
t
i
o
n
:
a
p
e
r
s
o
n
a
l
o
d
y
s
s
e
y
i
n
:
C
.
G
.
G
.
A
i
t
k
e
n
a
n
d
A
.
D
.
S
t
o
n
e
y
(
e
d
s
)
,
T
h
e
U
s
e
o
f
S
t
a
t
i
s
t
i
c
s
i
n
F
o
r
e
n
s
i
c
S
c
i
e
n
c
e
(
N
e
w
Y
o
r
k
:
E
l
l
i
s
H
o
r
w
o
o
d
1
9
9
1
)
9
^
2
2
.
3
T
h
e
w
a
y
t
h
i
s
i
s
d
o
n
e
i
s
e
x
p
l
a
i
n
e
d
i
n
:
B
.
R
o
b
e
r
t
s
o
n
a
n
d
G
.
A
.
V
i
g
n
a
u
x
,
I
n
t
e
r
p
r
e
t
i
n
g
E
v
i
d
e
n
c
e
:
E
v
a
l
u
-
a
t
i
n
g
F
o
r
e
n
s
i
c
S
c
i
e
n
c
e
i
n
t
h
e
C
o
u
r
t
r
o
o
m
(
C
h
i
c
h
e
s
t
e
r
;
N
e
w
Y
o
r
k
:
J
o
h
n
W
i
l
e
y
&
S
o
n
s
,
1
s
t
e
d
,
1
9
9
5
)
c
h
5
.
B
e
r
n
a
r
d
R
o
b
e
r
t
s
o
n
,
G
.
A
.
V
i
g
n
a
u
x
a
n
d
C
h
a
r
l
e
s
E
.
H
.
B
e
r
g
e
r
4
4
5
r
2
0
1
1
T
h
e
A
u
t
h
o
r
s
.
T
h
e
M
o
d
e
r
n
L
a
w
R
e
v
i
e
w
r
2
0
1
1
T
h
e
M
o
d
e
r
n
L
a
w
R
e
v
i
e
w
L
i
m
i
t
e
d
.
(
2
0
1
1
)
7
4
(
3
)
4
3
0
^
4
5
5
C
O
N
F
U
S
I
O
N
U
n
f
o
r
t
u
n
a
t
e
l
y
,
f
o
r
r
e
a
s
o
n
s
b
e
y
o
n
d
t
h
e
s
c
o
p
e
o
f
t
h
i
s
a
r
t
i
c
l
e
,
4
s
c
i
e
n
t
i
s
t
s
e
s
t
a
b
l
i
s
h
e
d
a
c
o
n
v
e
n
t
i
o
n
t
h
a
t
t
h
e
y
w
o
u
l
d
g
i
v
e
a
p
r
o
b
a
b
i
l
i
t
y
o
f
p
a
t
e
r
n
i
t
y
t
o
t
h
e
c
o
u
r
t
s
.
T
h
i
s
p
r
o
b
a
b
i
l
i
t
y
w
a
s
c
o
m
p
u
t
e
d
b
y
m
u
l
t
i
p
l
y
i
n
g
t
h
e
p
a
t
e
r
n
i
t
y
i
n
d
e
x
(
t
h
e
s
t
r
e
n
g
t
h
o
f
t
h
e
e
v
i
d
e
n
c
e
)
b
y
s
o
-
c
a
l
l
e
d
n
e
u
t
r
a
l
p
r
i
o
r
o
d
d
s
.
T
h
a
t
i
s
,
w
i
t
h
o
u
t
t
a
k
i
n
g
t
h
e
e
v
i
-
d
e
n
c
e
i
n
t
o
a
c
c
o
u
n
t
,
t
h
e
t
w
o
h
y
p
o
t
h
e
s
e
s
w
e
r
e
a
s
s
u
m
e
d
e
q
u
a
l
l
y
p
r
o
b
a
b
l
e
.
T
h
i
s
a
p
p
e
a
r
s
t
o
a
p
p
l
y
B
a
y
e
s
T
h
e
o
r
e
m
,
w
h
i
c
h
s
t
a
t
e
s
t
h
a
t
t
o
a
r
r
i
v
e
a
t
p
o
s
t
e
r
i
o
r
o
d
d
s
f
o
r
a
n
h
y
p
o
t
h
e
s
i
s
,
o
n
e
m
u
l
t
i
p
l
i
e
s
t
h
e
l
i
k
e
l
i
h
o
o
d
r
a
t
i
o
b
y
p
r
i
o
r
o
d
d
s
w
h
i
c
h
a
r
e
d
e
t
e
r
-
m
i
n
e
d
b
y
a
l
l
t
h
e
o
t
h
e
r
e
v
i
d
e
n
c
e
o
n
e
h
a
s
c
o
n
s
i
d
e
r
e
d
.
T
h
e
p
a
t
e
r
n
i
t
y
e
v
i
d
e
n
c
e
c
o
n
-
v
e
n
t
i
o
n
,
h
o
w
e
v
e
r
,
t
a
k
e
s
t
w
o
i
l
l
e
g
i
t
i
m
a
t
e
s
t
e
p
s
.
F
i
r
s
t
,
i
t
a
r
r
o
g
a
t
e
s
t
o
t
h
e
e
x
p
e
r
t
t
h
e
t
a
s
k
o
f
a
r
r
i
v
i
n
g
a
t
a
n
a
l
p
r
o
b
a
b
i
l
i
t
y
w
h
i
c
h
i
s
t
h
e
t
a
s
k
o
f
t
h
e
c
o
u
r
t
.
S
e
c
o
n
d
l
y
,
t
h
e
c
h
o
i
c
e
o
f
t
h
e
p
r
i
o
r
o
d
d
s
5
w
a
s
a
r
b
i
t
r
a
r
y
w
h
e
n
i
t
s
h
o
u
l
d
h
a
v
e
b
e
e
n
b
a
s
e
d
o
n
c
o
n
-
s
i
d
e
r
i
n
g
t
h
e
o
t
h
e
r
e
v
i
d
e
n
c
e
i
n
t
h
e
c
a
s
e
,
w
h
i
c
h
,
a
g
a
i
n
,
i
s
a
m
a
t
t
e
r
f
o
r
t
h
e
c
o
u
r
t
.
O
n
c
e
t
h
i
s
c
o
n
v
e
n
t
i
o
n
w
a
s
e
s
t
a
b
l
i
s
h
e
d
,
s
c
i
e
n
t
i
s
t
s
r
o
u
t
i
n
e
l
y
g
a
v
e
e
v
i
d
e
n
c
e
o
f
a
p
r
o
b
a
b
i
l
i
t
y
o
f
p
a
t
e
r
n
i
t
y
.
T
h
i
s
n
a
t
u
r
a
l
l
y
c
o
n
f
u
s
e
d
t
h
e
c
o
u
r
t
s
a
s
t
h
e
y
c
o
u
l
d
n
o
t
s
e
e
h
o
w
t
o
c
o
m
b
i
n
e
t
h
i
s
p
r
o
b
a
b
i
l
i
t
y
w
i
t
h
t
h
e
o
t
h
e
r
e
v
i
d
e
n
c
e
i
n
t
h
e
c
a
s
e
^
a
n
d
i
n
d
e
e
d
t
h
e
r
e
i
s
n
o
p
r
o
p
e
r
w
a
y
o
f
d
o
i
n
g
s
o
.
T
h
i
s
d
i
l
e
m
m
a
l
e
d
c
o
u
r
t
s
i
n
t
o
m
a
k
i
n
g
p
u
z
z
l
i
n
g
s
t
a
t
e
m
e
n
t
s
s
u
c
h
a
s
:
T
h
e
c
o
n
c
e
p
t
o
f
p
r
o
b
a
b
i
l
i
t
y
i
n
t
h
e
l
e
g
a
l
s
e
n
s
e
i
s
c
e
r
t
a
i
n
l
y
d
i
e
r
e
n
t
f
r
o
m
t
h
e
m
a
t
h
e
m
a
t
i
c
a
l
c
o
n
c
e
p
t
:
i
n
d
e
e
d
i
t
i
s
r
a
r
e
t
o
n
d
a
s
i
t
u
a
t
i
o
n
i
n
w
h
i
c
h
t
h
e
s
e
t
w
o
u
s
a
g
e
s
c
o
-
e
x
i
s
t
,
a
l
t
h
o
u
g
h
w
h
e
n
t
h
e
y
d
o
,
t
h
e
m
a
t
h
e
m
a
t
i
c
a
l
p
r
o
b
a
b
i
l
i
t
y
h
a
s
t
o
b
e
t
a
k
e
n
i
n
t
o
t
h
e
a
s
s
e
s
s
m
e
n
t
o
f
p
r
o
b
a
b
i
l
i
t
y
i
n
t
h
e
l
e
g
a
l
s
e
n
s
e
a
n
d
g
i
v
e
n
i
t
s
a
p
p
r
o
p
r
i
a
t
e
w
e
i
g
h
t
.
6
T
h
e
f
o
r
e
n
s
i
c
s
c
i
e
n
t
i
s
t
s
g
i
v
i
n
g
t
h
i
s
e
v
i
d
e
n
c
e
w
e
r
e
f
o
l
l
o
w
i
n
g
a
p
r
o
c
e
d
u
r
e
l
a
i
d
d
o
w
n
b
y
c
o
n
v
e
n
t
i
o
n
;
t
h
e
y
c
o
u
l
d
n
o
t
h
a
v
e
j
u
s
t
i
e
d
i
t
o
n
a
n
y
l
o
g
i
c
a
l
g
r
o
u
n
d
s
.
T
h
e
y
w
e
r
e
c
o
m
b
i
n
i
n
g
(
b
y
m
u
l
t
i
p
l
y
i
n
g
)
t
h
e
j
u
d
g
m
e
n
t
t
h
e
y
w
e
r
e
e
n
t
i
t
l
e
d
t
o
m
a
k
e
f
r
o
m
t
h
e
i
r
k
n
o
w
l
e
d
g
e
,
r
e
s
e
a
r
c
h
a
n
d
e
x
p
e
r
i
e
n
c
e
,
o
f
h
o
w
m
u
c
h
m
o
r
e
p
r
o
b
a
b
l
e
t
h
e
e
v
i
d
e
n
c
e
w
a
s
u
n
d
e
r
o
n
e
h
y
p
o
t
h
e
s
i
s
t
h
a
n
u
n
d
e
r
t
h
e
o
t
h
e
r
,
w
i
t
h
a
n
a
r
b
i
t
r
a
r
i
l
y
s
e
l
e
c
t
e
d
p
r
i
o
r
p
r
o
b
a
b
i
l
i
t
y
.
B
y
g
i
v
i
n
g
e
v
i
d
e
n
c
e
o
n
l
y
o
f
a
p
r
o
b
a
b
i
l
i
t
y
o
f
p
a
t
e
r
n
i
t
y
t
h
e
y
w
e
r
e
u
n
w
i
t
t
i
n
g
l
y
c
o
n
c
e
a
l
i
n
g
w
h
a
t
t
h
e
y
w
e
r
e
d
o
i
n
g
.
T
h
e
p
r
o
b
l
e
m
i
d
e
n
t
i
e
d
b
y
O
r
m
-
r
o
d
L
J
,
i
n
t
h
e
q
u
o
t
a
t
i
o
n
a
b
o
v
e
,
w
a
s
n
o
t
d
u
e
t
o
a
n
y
i
n
h
e
r
e
n
t
d
i
s
t
i
n
c
t
i
o
n
b
e
t
w
e
e
n
d
i
e
r
e
n
t
k
i
n
d
s
o
f
p
r
o
b
a
b
i
l
i
t
y
b
u
t
t
o
t
h
e
w
a
y
t
h
e
e
v
i
d
e
n
c
e
w
a
s
g
i
v
e
n
.
7
T
h
i
s
p
r
a
c
-
t
i
c
e
w
a
s
n
o
t
u
n
i
q
u
e
t
o
p
a
t
e
r
n
i
t
y
e
v
i
d
e
n
c
e
b
u
t
b
e
c
a
m
e
t
h
e
u
s
u
a
l
w
a
y
i
n
w
h
i
c
h
s
c
i
e
n
t
i
c
e
v
i
d
e
n
c
e
w
a
s
g
i
v
e
n
u
n
t
i
l
t
h
e
a
d
v
e
n
t
o
f
D
N
A
.
U
n
f
o
r
t
u
n
a
t
e
l
y
,
a
s
w
e
c
a
n
s
e
e
f
r
o
m
R
v
T
,
c
o
u
r
t
s
c
a
m
e
t
o
e
x
p
e
c
t
e
x
p
e
r
t
s
t
o
g
i
v
e
e
v
i
d
e
n
c
e
i
n
t
h
e
f
o
r
m
o
f
a
4
F
o
r
t
h
e
f
u
l
l
s
t
o
r
y
s
e
e
D
.
K
a
y
e
,
T
h
e
p
r
o
b
a
b
i
l
i
t
y
o
f
a
n
u
l
t
i
m
a
t
e
i
s
s
u
e
:
t
h
e
s
t
r
a
n
g
e
c
a
s
e
o
f
p
a
t
e
r
n
i
t
y
t
e
s
t
i
n
g
(
1
9
8
9
)
1
I
o
w
a
L
R
7
5
.
5
F
o
r
a
n
e
x
a
m
p
l
e
i
n
h
a
n
d
w
r
i
t
i
n
g
a
n
a
l
y
s
i
s
,
s
e
e
:
N
.
K
l
l
e
r
e
t
a
l
,
P
r
o
b
a
b
i
l
i
t
y
C
o
n
c
l
u
s
i
o
n
s
i
n
E
x
p
e
r
t
O
p
i
-
n
i
o
n
s
o
n
H
a
n
d
w
r
i
t
i
n
g
(
M
u
n
i
c
h
:
W
o
l
t
e
r
s
K
l
u
w
e
r
,
2
0
0
4
)
.
S
e
e
a
l
s
o
:
I
.
M
.
E
l
l
m
a
n
a
n
d
D
.
K
a
y
e
,
P
r
o
b
-
a
b
i
l
i
t
i
e
s
a
n
d
p
r
o
o
f
:
c
a
n
H
L
A
a
n
d
b
l
o
o
d
g
r
o
u
p
t
e
s
t
i
n
g
p
r
o
v
e
p
a
t
e
r
n
i
t
y
?
(
1
9
7
9
)
5
4
N
Y
U
L
R
1
1
3
1
,
a
n
d
A
.
B
i
e
d
e
r
m
a
n
n
e
t
a
l
,
E
q
u
a
l
p
r
i
o
r
p
r
o
b
a
b
i
l
i
t
i
e
s
:
C
a
n
o
n
e
d
o
a
n
y
b
e
t
t
e
r
?
(
2
0
0
7
)
8
5
F
o
r
S
c
i
I
n
t
n
t
l
1
7
2
.
6
R
e
J
S
(
a
m
i
n
o
r
)
[
1
9
8
1
]
F
a
m
2
2
,
2
9
;
[
1
9
8
0
]
1
A
l
l
E
R
1
0
6
1
p
e
r
O
r
m
r
o
d
L
J
.
7
F
o
r
d
i
s
c
u
s
s
i
o
n
o
f
m
o
r
e
i
s
s
u
e
s
a
r
i
s
i
n
g
f
r
o
m
a
n
a
r
r
o
w
v
i
e
w
o
f
p
r
o
b
a
b
i
l
i
t
y
o
n
t
h
e
p
a
r
t
o
f
m
a
n
y
l
a
w
y
e
r
s
s
e
e
B
.
R
o
b
e
r
t
s
o
n
a
n
d
G
.
A
.
V
i
g
n
a
u
x
,
P
r
o
b
a
b
i
l
i
t
y
-
T
h
e
L
o
g
i
c
o
f
t
h
e
L
a
w
(
1
9
9
3
)
1
3
O
J
L
S
4
5
7
.
E
x
t
e
n
d
i
n
g
t
h
e
C
o
n
f
u
s
i
o
n
A
b
o
u
t
B
a
y
e
s
4
4
6
r
2
0
1
1
T
h
e
A
u
t
h
o
r
s
.
T
h
e
M
o
d
e
r
n
L
a
w
R
e
v
i
e
w
r
2
0
1
1
T
h
e
M
o
d
e
r
n
L
a
w
R
e
v
i
e
w
L
i
m
i
t
e
d
.
(
2
0
1
1
)
7
4
(
3
)
4
3
0
^
4
5
5
c
o
n
c
l
u
s
i
o
n
a
b
o
u
t
w
h
e
t
h
e
r
t
w
o
m
a
r
k
s
h
a
d
a
c
o
m
m
o
n
s
o
u
r
c
e
.
8
A
s
a
r
e
s
u
l
t
t
h
e
c
o
u
r
t
s
a
r
e
n
o
w
c
o
n
f
u
s
e
d
w
h
e
n
t
h
e
e
v
i
d
e
n
c
e
i
s
g
i
v
e
n
i
n
t
h
e
l
o
g
i
c
a
l
m
a
n
n
e
r
.
N
U
M
B
E
R
S
A
N
D
/
O
R
V
E
R
B
A
L
S
C
A
L
E
S
W
h
e
n
D
N
A
p
r
o
l
i
n
g
b
e
c
a
m
e
t
h
e
s
t
a
n
d
a
r
d
m
e
t
h
o
d
o
f
t
e
s
t
i
n
g
f
o
r
p
a
t
e
r
n
i
t
y
(
a
n
d
i
d
e
n
t
i
t
y
)
t
h
e
i
n
t
e
n
s
e
s
c
r
u
t
i
n
y
t
o
w
h
i
c
h
i
t
w
a
s
s
u
b
m
i
t
t
e
d
c
a
u
s
e
d
f
o
r
e
n
s
i
c
s
c
i
e
n
t
i
s
t
s
t
o
r
e
c
o
n
s
i
d
e
r
t
h
e
w
a
y
t
h
e
y
g
a
v
e
e
v
i
d
e
n
c
e
.
I
t
h
a
s
n
o
w
b
e
c
o
m
e
r
o
u
t
i
n
e
f
o
r
f
o
r
e
n
s
i
c
s
c
i
e
n
t
i
s
t
s
t
o
g
i
v
e
t
h
e
i
r
e
v
i
d
e
n
c
e
i
n
t
h
e
w
a
y
w
e
h
a
v
e
e
x
p
l
a
i
n
e
d
.
I
n
p
a
t
e
r
n
i
t
y
c
a
s
e
s
,
t
h
e
y
e
x
p
l
a
i
n
h
o
w
p
r
o
b
a
b
l
e
t
h
e
D
N
A
p
r
o
l
e
f
o
u
n
d
i
s
i
f
t
h
e
d
e
f
e
n
d
a
n
t
w
a
s
t
h
e
f
a
t
h
e
r
a
n
d
h
o
w
p
r
o
b
a
b
l
e
i
t
i
s
i
f
s
o
m
e
o
n
e
e
l
s
e
w
a
s
t
h
e
f
a
t
h
e
r
a
n
d
t
h
e
n
s
t
a
t
e
t
h
e
l
i
k
e
-
l
i
h
o
o
d
r
a
t
i
o
w
h
i
c
h
i
s
a
s
i
m
p
l
e
m
a
t
t
e
r
o
f
d
i
v
i
d
i
n
g
o
n
e
g
u
r
e
b
y
t
h
e
o
t
h
e
r
.
T
h
e
r
e
i
s
s
o
m
e
d
e
b
a
t
e
,
h
o
w
e
v
e
r
,
a
s
t
o
w
h
e
t
h
e
r
t
h
e
e
x
p
e
r
t
s
h
o
u
l
d
e
x
p
l
a
i
n
t
h
e
e
e
c
t
o
f
t
h
e
e
v
i
d
e
n
c
e
b
y
s
o
m
e
v
e
r
b
a
l
c
o
n
v
e
n
t
i
o
n
,
i
n
s
t
e
a
d
o
f
o
r
i
n
a
d
d
i
t
i
o
n
t
o
g
i
v
-
i
n
g
t
h
e
l
i
k
e
l
i
h
o
o
d
r
a
t
i
o
.
T
h
e
d
e
f
e
n
c
e
c
a
n
c
r
o
s
s
-
e
x
a
m
i
n
e
t
h
e
e
x
p
e
r
t
i
f
i
t
i
s
d
e
s
i
r
e
d
t
o
g
o
i
n
t
o
t
h
e
g
u
r
e
s
,
b
u
t
m
a
y
n
o
t
a
l
w
a
y
s
w
i
s
h
t
o
d
o
t
h
i
s
.
T
h
e
e
x
p
e
r
t
m
i
g
h
t
e
x
p
l
a
i
n
t
h
e
e
e
c
t
o
f
t
h
e
e
v
i
d
e
n
c
e
v
e
r
b
a
l
l
y
i
n
w
a
y
s
s
u
c
h
a
s
:
w
h
a
t
e
v
e
r
o
d
d
s
y
o
u
a
s
s
e
s
s
t
h
a
t
t
h
e
d
e
f
e
n
d
a
n
t
i
s
t
h
e
f
a
t
h
e
r
o
n
t
h
e
b
a
s
i
s
o
f
t
h
e
o
t
h
e
r
e
v
i
d
e
n
c
e
i
n
t
h
e
c
a
s
e
,
t
h
i
s
e
v
i
d
e
n
c
e
m
u
l
t
i
p
l
i
e
s
t
h
o
s
e
o
d
d
s
b
y
n
;
o
r
t
h
i
s
e
v
i
d
e
n
c
e
m
o
d
e
r
a
t
e
l
y
/
s
t
r
o
n
g
l
y
/
v
e
r
y
s
t
r
o
n
g
l
y
s
u
p
p
o
r
t
s
t
h
e
h
y
p
o
t
h
e
s
i
s
t
h
a
t
t
h
e
d
e
f
e
n
d
a
n
t
i
s
t
h
e
f
a
t
h
e
r
.
O
n
e
r
e
a
s
o
n
f
o
r
a
d
o
p
t
i
n
g
v
e
r
b
a
l
c
o
n
v
e
n
t
i
o
n
s
i
s
a
p
e
r
c
e
p
t
i
o
n
t
h
a
t
j
u
r
o
r
s
,
a
n
d
f
o
r
t
h
a
t
m
a
t
t
e
r
j
u
d
g
e
s
,
d
o
n
o
t
f
e
e
l
c
o
n
d
e
n
t
i
n
h
a
n
d
l
i
n
g
n
u
m
b
e
r
s
a
n
d
r
e
a
c
t
n
e
g
a
-
t
i
v
e
l
y
t
o
m
a
t
h
e
m
a
t
i
c
a
l
f
o
r
m
u
l
a
e
.
A
n
o
t
h
e
r
i
s
t
h
a
t
p
r
e
c
i
s
e
g
u
r
e
s
m
a
y
n
o
t
b
e
a
v
a
i
l
-
a
b
l
e
a
n
d
t
h
e
v
e
r
b
a
l
c
o
n
v
e
n
t
i
o
n
s
e
n
a
b
l
e
t
h
e
e
x
p
e
r
t
t
o
e
x
p
r
e
s
s
t
h
e
r
e
l
a
t
i
v
e
s
t
r
e
n
g
t
h
o
f
t
h
e
e
v
i
d
e
n
c
e
i
n
w
o
r
d
s
.
I
n
p
a
t
e
r
n
i
t
y
c
a
s
e
s
,
w
e
l
l
-
r
e
s
e
a
r
c
h
e
d
d
a
t
a
i
n
t
h
e
f
o
r
m
o
f
b
l
o
o
d
t
y
p
e
a
n
d
D
N
A
d
a
t
a
b
a
s
e
s
a
r
e
a
v
a
i
l
a
b
l
e
.
B
u
t
t
h
e
s
t
r
u
c
t
u
r
e
o
f
t
h
e
a
r
g
u
m
e
n
t
i
s
d
i
c
t
a
t
e
d
b
y
l
o
g
i
c
n
o
t
b
y
t
h
e
n
a
t
u
r
e
o
f
t
h
e
d
a
t
a
.
T
h
e
v
a
l
u
e
o
f
a
n
y
p
i
e
c
e
o
f
e
v
i
d
e
n
c
e
l
i
e
s
i
n
i
t
s
p
o
w
e
r
t
o
d
i
s
c
r
i
m
i
n
a
t
e
b
e
t
w
e
e
n
h
y
p
o
t
h
e
s
e
s
:
i
n
h
o
w
m
u
c
h
m
o
r
e
p
r
o
b
a
b
l
e
t
h
e
e
v
i
d
e
n
c
e
i
s
i
f
o
n
e
h
y
p
o
t
h
-
e
s
i
s
i
s
t
r
u
e
t
h
a
n
i
f
t
h
e
o
t
h
e
r
h
y
p
o
t
h
e
s
i
s
i
s
t
r
u
e
.
D
I
F
F
E
R
E
N
T
I
M
P
R
E
S
S
I
O
N
S
F
R
O
M
T
H
E
S
A
M
E
S
H
O
E
T
h
i
s
i
s
e
q
u
a
l
l
y
t
r
u
e
w
h
e
n
t
h
e
o
u
t
c
o
m
e
o
f
a
c
o
m
p
a
r
i
s
o
n
i
s
n
o
t
s
i
m
p
l
y
s
a
m
e
o
r
d
i
e
r
e
n
t
.
W
h
e
n
m
a
r
k
s
l
e
f
t
b
y
t
h
e
s
a
m
e
f
o
o
t
w
e
a
r
a
r
e
c
o
m
p
a
r
e
d
,
t
h
e
r
e
i
s
w
i
t
h
i
n
s
o
u
r
c
e
v
a
r
i
a
t
i
o
n
,
t
h
e
y
w
i
l
l
n
e
v
e
r
b
e
i
d
e
n
t
i
c
a
l
.
F
o
r
s
u
c
h
t
y
p
e
s
o
f
e
v
i
d
e
n
c
e
(
c
o
m
-
p
a
r
a
t
i
v
e
e
x
a
m
i
n
a
t
i
o
n
s
)
t
h
e
o
u
t
c
o
m
e
o
f
t
h
e
c
o
m
p
a
r
i
s
o
n
i
s
t
h
e
d
e
g
r
e
e
o
f
s
i
m
i
l
a
r
i
t
y
.
T
h
e
t
a
s
k
f
o
r
t
h
e
e
x
p
e
r
t
,
t
h
e
r
e
f
o
r
e
,
i
s
r
s
t
t
o
c
o
m
p
a
r
e
a
c
r
i
m
e
s
c
e
n
e
m
a
r
k
w
i
t
h
a
r
e
f
e
r
e
n
c
e
p
r
i
n
t
a
n
d
f
o
r
m
a
n
o
p
i
n
i
o
n
a
s
t
o
h
o
w
w
e
l
l
t
h
e
e
v
i
d
e
n
c
e
(
t
h
e
d
e
g
r
e
e
o
f
s
i
m
i
l
a
r
i
t
y
)
s
u
p
p
o
r
t
s
t
h
e
h
y
p
o
t
h
e
s
i
s
t
h
a
t
b
o
t
h
i
m
p
r
e
s
s
i
o
n
s
w
e
r
e
m
a
d
e
b
y
t
h
e
s
a
m
e
8
S
e
e
C
o
r
r
e
s
p
o
n
d
e
n
c
e
(
1
9
9
6
)
3
6
S
c
i
&
J
2
9
0
^
2
9
2
a
n
d
(
1
9
9
7
)
3
7
S
c
i
&
J
6
4
^
6
5
,
f
o
l
l
o
w
i
n
g
D
.
A
.
R
u
d
r
a
m
,
I
n
t
e
r
p
r
e
t
a
t
i
o
n
o
f
s
c
i
e
n
t
i
c
e
v
i
d
e
n
c
e
(
1
9
9
6
)
3
6
S
c
i
&
J
1
3
3
^
1
3
8
.
B
e
r
n
a
r
d
R
o
b
e
r
t
s
o
n
,
G
.
A
.
V
i
g
n
a
u
x
a
n
d
C
h
a
r
l
e
s
E
.
H
.
B
e
r
g
e
r
4
4
7
r
2
0
1
1
T
h
e
A
u
t
h
o
r
s
.
T
h
e
M
o
d
e
r
n
L
a
w
R
e
v
i
e
w
r
2
0
1
1
T
h
e
M
o
d
e
r
n
L
a
w
R
e
v
i
e
w
L
i
m
i
t
e
d
.
(
2
0
1
1
)
7
4
(
3
)
4
3
0
^
4
5
5
s
h
o
e
.
T
h
i
s
m
a
y
b
e
a
e
c
t
e
d
b
y
t
h
e
q
u
a
l
i
t
y
o
f
t
h
e
m
a
r
k
a
t
t
h
e
s
c
e
n
e
a
n
d
t
h
e
p
a
s
s
a
g
e
o
f
t
i
m
e
.
I
t
m
a
y
b
e
p
o
s
s
i
b
l
e
t
o
e
x
c
l
u
d
e
t
h
a
t
h
y
p
o
t
h
e
s
i
s
a
l
t
o
g
e
t
h
e
r
b
e
c
a
u
s
e
t
h
e
b
a
s
i
c
p
a
t
t
e
r
n
s
o
f
t
h
e
m
a
r
k
s
a
r
e
d
i
e
r
e
n
t
,
o
r
b
e
c
a
u
s
e
t
h
e
e
a
r
l
i
e
r
m
a
r
k
h
a
s
s
o
m
e
m
a
j
o
r
s
c
a
r
o
n
i
t
t
h
a
t
i
s
n
o
t
o
n
t
h
e
s
e
c
o
n
d
m
a
r
k
.
I
n
a
l
l
o
t
h
e
r
c
a
s
e
s
,
t
h
e
e
x
p
e
r
t
c
o
u
l
d
s
a
y
s
i
m
p
l
y
t
h
a
t
o
n
e
c
a
n
n
o
t
e
x
c
l
u
d
e
t
h
e
p
o
s
s
i
b
i
l
i
t
y
t
h
a
t
t
h
e
t
w
o
m
a
r
k
s
w
e
r
e
m
a
d
e
b
y
t
h
e
s
a
m
e
s
h
o
e
.
T
h
i
s
h
o
w
e
v
e
r
,
d
o
e
s
n
o
t
c
o
n
v
e
y
a
n
y
e
v
i
d
e
n
t
i
a
l
v
a
l
u
e
.
N
o
r
d
o
e
s
t
h
e
s
t
a
t
e
m
e
n
t
t
h
a
t
t
h
i
s
s
h
o
e
c
o
u
l
d
h
a
v
e
m
a
d
e
t
h
e
m
a
r
k
w
h
i
c
h
i
s
s
i
m
p
l
y
a
n
o
t
h
e
r
w
a
y
o
f
s
a
y
i
n
g
t
h
a
t
i
t
c
a
n
n
o
t
b
e
e
x
c
l
u
d
e
d
.
9
S
e
c
o
n
d
,
t
h
e
e
x
p
e
r
t
c
o
n
s
i
d
e
r
s
t
h
e
p
r
o
b
a
b
i
l
i
t
y
o
f
t
h
e
e
v
i
d
e
n
c
e
i
f
t
h
e
m
a
r
k
s
w
e
r
e
m
a
d
e
b
y
d
i
e
r
e
n
t
s
h
o
e
s
.
T
h
i
s
w
i
l
l
b
e
a
e
c
t
e
d
b
y
t
h
e
p
r
o
p
o
r
t
i
o
n
o
f
s
h
o
e
s
w
i
t
h
t
h
e
o
b
s
e
r
v
e
d
b
a
s
i
c
p
a
t
t
e
r
n
a
n
d
w
e
a
r
m
a
r
k
s
.
T
h
e
e
v
i
d
e
n
t
i
a
l
v
a
l
u
e
o
f
t
h
e
c
o
m
p
a
r
i
s
o
n
i
s
d
e
n
e
d
b
y
h
o
w
m
u
c
h
m
o
r
e
p
r
o
b
a
b
l
e
t
h
e
o
b
s
e
r
v
e
d
d
e
g
r
e
e
o
f
s
i
m
i
l
a
r
i
t
y
i
s
w
h
e
n
t
h
e
m
a
r
k
s
w
e
r
e
l
e
f
t
b
y
t
h
e
s
a
m
e
s
h
o
e
t
h
a
n
w
h
e
n
t
h
e
m
a
r
k
s
w
e
r
e
l
e
f
t
b
y
d
i
e
r
e
n
t
s
h
o
e
s
.
A
s
t
h
e
d
e
g
r
e
e
o
f
s
i
m
i
l
a
r
i
t
y
i
n
c
r
e
a
s
e
s
,
s
o
w
i
l
l
t
h
e
s
u
p
p
o
r
t
f
o
r
t
h
e
h
y
p
o
t
h
e
s
i
s
t
h
a
t
t
h
e
s
a
m
e
s
h
o
e
l
e
f
t
b
o
t
h
m
a
r
k
s
.
S
o
t
h
e
l
o
g
i
c
a
l
s
t
r
u
c
t
u
r
e
i
s
t
h
e
s
a
m
e
a
s
w
i
t
h
D
N
A
,
a
n
d
i
n
d
e
e
d
w
i
t
h
a
l
l
e
v
i
d
e
n
c
e
:
t
h
e
e
x
p
e
r
t
r
s
t
c
o
n
s
i
d
e
r
s
h
o
w
p
r
o
b
a
b
l
e
t
h
e
e
v
i
d
e
n
c
e
i
s
i
f
t
h
e
t
w
o
i
m
p
r
e
s
s
i
o
n
s
c
a
m
e
f
r
o
m
t
h
e
s
a
m
e
s
h
o
e
a
n
d
t
h
e
n
h
o
w
p
r
o
b
-
a
b
l
e
i
t
i
s
i
f
t
h
e
t
w
o
i
m
p
r
e
s
s
i
o
n
s
c
a
m
e
f
r
o
m
d
i
e
r
e
n
t
s
h
o
e
s
.
T
h
i
s
i
s
a
l
w
a
y
s
t
h
e
c
o
r
r
e
c
t
s
t
r
u
c
t
u
r
e
,
e
v
e
n
i
f
i
n
s
o
m
e
c
a
s
e
s
t
h
e
e
v
i
d
e
n
c
e
i
s
s
o
s
t
r
o
n
g
t
h
a
t
i
t
i
s
n
o
t
c
r
e
d
i
b
l
e
t
h
a
t
t
h
e
r
e
c
o
u
l
d
b
e
a
n
o
t
h
e
r
s
h
o
e
t
h
a
t
w
o
u
l
d
l
e
a
v
e
a
n
i
m
p
r
e
s
s
i
o
n
t
h
a
t
i
s
j
u
s
t
a
s
s
i
m
i
l
a
r
.
I
n
p
r
i
n
c
i
p
l
e
,
t
h
e
p
o
i
n
t
s
t
i
l
l
i
s
t
h
a
t
t
h
e
e
v
i
d
e
n
c
e
i
s
s
o
s
t
r
o
n
g
t
h
a
t
i
t
o
v
e
r
w
h
e
l
m
s
a
n
y
o
t
h
e
r
p
o
s
s
i
b
l
e
e
v
i
d
e
n
c
e
i
n
t
h
e
c
a
s
e
,
n
o
t
t
h
a
t
t
h
e
e
x
p
e
r
t
c
a
n
s
a
y
o
n
t
h
e
b
a
s
i
s
o
f
t
h
e
i
m
p
r
e
s
s
i
o
n
s
a
l
o
n
e
t
h
a
t
t
h
e
y
w
e
r
e
d
e
n
i
t
e
l
y
m
a
d
e
b
y
t
h
e
s
a
m
e
s
h
o
e
.
F
I
N
G
E
R
P
R
I
N
T
E
V
I
D
E
N
C
E
T
h
e
d
i
c
u
l
t
y
i
n
e
x
p
l
a
i
n
i
n
g
t
h
i
s
a
l
s
o
s
t
e
m
s
f
r
o
m
t
h
e
w
a
y
n
g
e
r
p
r
i
n
t
e
v
i
d
e
n
c
e
h
a
s
c
l
a
s
s
i
c
a
l
l
y
b
e
e
n
g
i
v
e
n
.
F
i
n
g
e
r
p
r
i
n
t
e
v
i
d
e
n
c
e
h
a
s
s
t
r
o
n
g
s
i
m
i
l
a
r
i
t
i
e
s
t
o
f
o
o
t
w
e
a
r
e
v
i
d
e
n
c
e
i
n
t
h
a
t
t
h
e
i
m
p
r
e
s
s
i
o
n
s
b
e
i
n
g
c
o
m
p
a
r
e
d
w
i
l
l
h
a
v
e
b
e
e
n
m
a
d
e
u
n
d
e
r
d
i
f
-
f
e
r
e
n
t
c
o
n
d
i
t
i
o
n
s
a
n
d
s
o
w
i
l
l
n
o
t
b
e
i
d
e
n
t
i
c
a
l
e
v
e
n
i
f
m
a
d
e
b
y
t
h
e
s
a
m
e
n
g
e
r
.
I
n
r
e
c
e
n
t
r
e
s
e
a
r
c
h
,
1
0
i
t
h
a
s
b
e
e
n
a
r
g
u
e
d
t
h
a
t
n
g
e
r
p
r
i
n
t
e
v
i
d
e
n
c
e
c
a
n
b
e
e
v
a
l
u
a
t
e
d
l
o
g
i
c
a
l
l
y
a
n
d
c
a
n
e
v
e
n
b
e
q
u
a
n
t
i
e
d
i
n
t
e
r
m
s
o
f
l
i
k
e
l
i
h
o
o
d
r
a
t
i
o
s
.
F
i
n
g
e
r
p
r
i
n
t
e
x
a
m
i
n
e
r
s
a
d
o
p
t
e
d
a
c
o
n
v
e
n
t
i
o
n
t
h
a
t
,
i
f
t
h
e
n
g
e
r
p
r
i
n
t
s
d
i
d
n
o
t
e
x
c
l
u
d
e
a
s
u
s
p
e
c
t
,
t
h
e
y
w
o
u
l
d
o
n
l
y
g
i
v
e
e
v
i
d
e
n
c
e
o
f
i
d
e
n
t
i
t
y
w
h
e
n
t
h
e
d
e
g
r
e
e
o
f
s
i
m
i
l
a
r
i
t
y
w
a
s
s
o
g
r
e
a
t
t
h
a
t
t
h
e
r
e
w
a
s
v
i
r
t
u
a
l
l
y
n
o
p
o
s
s
i
b
i
l
i
t
y
o
f
a
n
o
t
h
e
r
n
g
e
r
m
a
k
i
n
g
a
m
a
r
k
j
u
s
t
a
s
s
i
m
i
l
a
r
.
T
h
e
i
r
e
v
i
d
e
n
c
e
w
a
s
i
n
t
h
e
f
o
r
m
t
h
a
t
,
i
n
t
h
e
i
r
j
u
d
g
-
m
e
n
t
t
h
e
t
w
o
i
m
p
r
e
s
s
i
o
n
s
w
e
r
e
m
a
d
e
b
y
t
h
e
s
a
m
e
n
g
e
r
,
b
a
c
k
e
d
u
p
b
y
r
e
f
e
r
e
n
c
e
t
o
t
h
e
s
i
m
i
l
a
r
i
t
i
e
s
.
O
n
e
e
e
c
t
w
a
s
t
h
a
t
i
n
c
a
s
e
s
w
h
e
r
e
t
h
e
s
i
m
i
l
a
r
i
t
i
e
s
s
u
p
p
o
r
t
e
d
c
o
m
-
m
o
n
o
r
i
g
i
n
b
u
t
w
e
r
e
d
e
e
m
e
d
i
n
s
u
c
i
e
n
t
,
c
o
u
r
t
s
w
e
r
e
d
e
p
r
i
v
e
d
o
f
t
h
a
t
e
v
i
d
e
n
c
e
.
1
1
9
P
a
c
e
R
v
T
a
t
[
7
3
]
w
h
e
r
e
t
h
e
C
o
u
r
t
r
e
g
a
r
d
s
t
h
e
s
t
a
t
e
m
e
n
t
c
o
u
l
d
h
a
v
e
m
a
d
e
t
h
e
m
a
r
k
a
s
m
o
r
e
p
r
e
-
c
i
s
e
t
h
a
n
m
o
d
e
r
a
t
e
(
s
c
i
e
n
t
i
c
)
s
u
p
p
o
r
t
.
1
0
S
e
e
e
g
:
C
.
N
e
u
m
a
n
n
e
t
a
l
,
C
o
m
p
u
t
a
t
i
o
n
o
f
L
i
k
e
l
i
h
o
o
d
R
a
t
i
o
s
i
n
F
i
n
g
e
r
p
r
i
n
t
I
d
e
n
t
i
c
a
t
i
o
n
f
o
r
C
o
n
g
u
r
a
t
i
o
n
s
o
f
A
n
y
N
u
m
b
e
r
o
f
M
i
n
u
t
i
a
e
(
2
0
0
7
)
5
2
J
F
S
5
4
.
1
1
A
t
t
h
e
p
r
e
s
e
n
t
t
i
m
e
,
i
n
m
o
s
t
j
u
r
i
s
d
i
c
t
i
o
n
s
,
a
n
o
p
i
n
i
o
n
o
f
c
e
r
t
a
i
n
t
y
i
s
t
h
e
o
n
l
y
a
c
c
e
p
t
a
b
l
e
o
p
i
n
i
o
n
w
h
e
n
d
e
a
l
i
n
g
w
i
t
h
t
h
e
i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n
o
f
a
n
g
e
r
p
r
i
n
t
i
m
p
r
e
s
s
i
o
n
.
T
h
i
s
i
s
n
o
t
a
r
u
l
e
t
h
a
t
h
a
s
E
x
t
e
n
d
i
n
g
t
h
e
C
o
n
f
u
s
i
o
n
A
b
o
u
t
B
a
y
e
s
4
4
8
r
2
0
1
1
T
h
e
A
u
t
h
o
r
s
.
T
h
e
M
o
d
e
r
n
L
a
w
R
e
v
i
e
w
r
2
0
1
1
T
h
e
M
o
d
e
r
n
L
a
w
R
e
v
i
e
w
L
i
m
i
t
e
d
.
(
2
0
1
1
)
7
4
(
3
)
4
3
0
^
4
5
5
T
h
e
I
n
t
e
r
n
a
t
i
o
n
a
l
A
s
s
o
c
i
a
t
i
o
n
f
o
r
I
d
e
n
t
i
c
a
t
i
o
n
(
I
A
I
)
i
n
1
9
7
9
a
d
o
p
t
e
d
a
r
e
s
o
l
u
-
t
i
o
n
i
n
w
h
i
c
h
i
t
s
t
a
t
e
d
t
h
a
t
a
n
y
m
e
m
b
e
r
.
.
.
w
h
o
p
r
o
v
i
d
e
s
o
r
a
l
o
r
w
r
i
t
t
e
n
r
e
p
o
r
t
s
,
o
r
g
i
v
e
s
t
e
s
t
i
m
o
n
y
o
f
p
o
s
s
i
b
l
e
,
p
r
o
b
a
b
l
e
o
r
l
i
k
e
l
y
f
r
i
c
t
i
o
n
r
i
d
g
e
i
d
e
n
t
i
c
a
t
i
o
n
s
h
a
l
l
b
e
d
e
e
m
e
d
t
o
b
e
e
n
g
a
g
e
d
i
n
c
o
n
d
u
c
t
u
n
b
e
c
o
m
i
n
g
s
u
c
h
m
e
m
b
e
r
.
T
h
i
s
r
e
s
o
l
u
t
i
o
n
w
a
s
r
e
s
c
i
n
d
e
d
i
n
2
0
1
0
b
e
c
a
u
s
e
i
t
w
a
s
n
o
t
c
o
n
s
i
s
t
e
n
t
w
i
t
h
a
d
v
a
n
c
e
m
e
n
t
s
i
n
c
e
t
h
e
i
r
p
a
s
s
a
g
e
.
T
h
e
r
e
s
o
l
u
t
i
o
n
f
r
o
m
2
0
1
0
s
t
a
t
e
s
t
h
a
t
t
h
e
u
s
e
o
f
m
a
t
h
e
m
a
t
i
c
a
l
l
y
b
a
s
e
d
m
o
d
e
l
s
t
o
a
s
s
e
s
s
t
h
e
a
s
s
o
c
i
a
t
i
v
e
v
a
l
u
e
o
f
t
h
e
e
v
i
d
e
n
c
e
m
a
y
p
r
o
v
i
d
e
a
s
c
i
e
n
t
i
c
a
l
l
y
s
o
u
n
d
b
a
s
i
s
f
o
r
s
u
p
p
o
r
t
i
n
g
t
h
e
e
x
a
m
i
n
e
r
s
o
p
i
n
i
o
n
.
1
2
U
n
f
o
r
t
u
n
a
t
e
l
y
,
f
a
m
i
l
i
a
r
i
t
y
w
i
t
h
t
h
e
w
a
y
n
g
e
r
p
r
i
n
t
e
v
i
d
e
n
c
e
h
a
s
h
i
s
t
o
r
i
c
a
l
l
y
b
e
e
n
g
i
v
e
n
h
a
s
c
o
n
t
r
i
b
u
t
e
d
t
o
t
h
e
t
e
n
d
e
n
c
y
o
f
t
h
e
c
o
u
r
t
s
t
o
m
i
s
u
n
d
e
r
s
t
a
n
d
t
h
e
l
o
g
i
c
a
l
s
t
r
u
c
t
u
r
e
o
f
f
o
r
e
n
s
i
c
s
c
i
e
n
t
i
c
e
v
i
d
e
n
c
e
a
n
d
e
v
e
n
t
o
m
i
s
u
n
d
e
r
s
t
a
n
d
t
h
e
e
v
i
-
d
e
n
c
e
w
h
e
n
i
t
i
s
g
i
v
e
n
c
o
r
r
e
c
t
l
y
.
T
h
e
m
o
s
t
c
o
m
m
o
n
e
r
r
o
r
f
o
u
n
d
s
t
e
m
s
f
r
o
m
t
h
e
s
o
-
c
a
l
l
e
d
p
r
o
s
e
c
u
t
o
r
s
f
a
l
l
a
c
y
,
w
h
i
c
h
,
i
n
c
i
d
e
n
t
a
l
l
y
,
d
o
e
s
n
o
t
n
e
c
e
s
s
a
r
i
l
y
f
a
v
o
u
r
t
h
e
p
r
o
s
e
c
u
t
i
o
n
.
T
H
E
P
R
O
S
E
C
U
T
O
R
S
F
A
L
L
A
C
Y
T
h
i
s
f
a
l
l
a
c
y
i
s
t
e
c
h
n
i
c
a
l
l
y
t
h
e
t
r
a
n
s
p
o
s
i
t
i
o
n
o
f
t
h
e
c
o
n
d
i
t
i
o
n
a
l
.
1
3
I
t
i
s
i
l
l
u
s
t
r
a
t
e
d
b
y
t
h
e
s
e
n
t
e
n
c
e
h
i
g
h
l
i
g
h
t
e
d
a
t
t
h
e
b
e
g
i
n
n
i
n
g
o
f
t
h
i
s
a
r
t
i
c
l
e
.
W
h
e
n
w
e
a
r
e
d
e
a
l
i
n
g
w
i
t
h
s
o
m
e
t
h
i
n
g
w
i
t
h
w
h
i
c
h
w
e
a
r
e
f
a
m
i
l
i
a
r
,
l
i
k
e
c
o
w
s
,
w
e
c
a
n
e
a
s
i
l
y
s
e
e
t
h
a
t
t
h
e
t
w
o
p
r
o
b
a
b
i
l
i
t
i
e
s
a
r
e
n
o
t
t
h
e
s
a
m
e
.
T
h
e
p
r
o
b
a
b
i
l
i
t
y
t
h
a
t
a
n
a
n
i
m
a
l
h
a
s
f
o
u
r
l
e
g
s
g
i
v
e
n
t
h
a
t
i
t
i
s
a
c
o
w
i
s
v
e
r
y
h
i
g
h
.
C
l
e
a
r
l
y
t
h
i
s
i
s
n
o
t
t
h
e
s
a
m
e
a
s
t
h
e
p
r
o
b
a
b
i
l
i
t
y
t
h
a
t
a
n
a
n
i
m
a
l
i
s
a
c
o
w
g
i
v
e
n
t
h
a
t
i
t
h
a
s
f
o
u
r
l
e
g
s
.
W
h
e
n
w
e
a
r
e
d
e
a
l
i
n
g
w
i
t
h
m
a
t
t
e
r
s
l
e
s
s
i
n
t
u
i
t
i
v
e
l
y
f
a
m
i
l
i
a
r
,
h
o
w
e
v
e
r
,
i
t
i
s
e
a
s
y
f
o
r
a
n
y
o
n
e
t
o
m
a
k
e
t
h
i
s
m
i
s
t
a
k
e
.
T
h
i
s
i
s
i
l
l
u
s
t
r
a
t
e
d
i
n
R
v
T
a
t
[
3
3
]
.
T
h
e
C
o
u
r
t
d
e
s
c
r
i
b
e
s
l
i
k
e
l
i
h
o
o
d
r
a
t
i
o
s
a
s
f
o
l
l
o
w
s
:
(
i
)
T
h
e
r
a
t
i
o
o
f
t
w
o
p
r
o
b
a
b
i
l
i
t
i
e
s
^
t
h
e
p
r
o
b
a
b
i
l
i
t
y
o
f
t
h
e
e
v
i
d
e
n
c
e
g
i
v
e
n
t
h
a
t
a
p
r
o
-
p
o
s
i
t
i
o
n
i
s
t
r
u
e
d
i
v
i
d
e
d
b
y
t
h
e
p
r
o
b
a
b
i
l
i
t
y
o
f
t
h
e
e
v
i
d
e
n
c
e
g
i
v
e
n
t
h
a
t
t
h
e
a
l
t
e
r
n
a
t
i
v
e
p
r
o
p
o
s
i
t
i
o
n
i
s
t
r
u
e
.
.
.
.
I
n
t
h
e
p
r
e
s
e
n
t
c
a
s
e
i
t
w
a
s
e
x
p
r
e
s
s
e
d
a
s
t
h
e
p
r
o
b
a
b
i
l
i
t
y
t
h
a
t
t
h
e
N
i
k
e
t
r
a
i
n
e
r
s
o
w
n
e
d
b
y
t
h
e
a
p
p
e
l
l
a
n
t
h
a
d
m
a
d
e
t
h
e
m
a
r
k
s
d
i
s
c
o
v
e
r
e
d
a
t
t
h
e
s
c
e
n
e
d
i
v
i
d
e
d
b
y
t
h
e
p
r
o
b
a
b
i
l
i
t
y
t
h
a
t
t
h
e
N
i
k
e
t
r
a
i
n
e
r
s
h
a
d
n
o
t
m
a
d
e
t
h
e
m
a
r
k
s
.
W
e
c
a
n
n
o
w
s
e
e
t
h
a
t
t
h
e
l
a
s
t
s
e
n
t
e
n
c
e
d
o
e
s
n
o
t
f
o
l
l
o
w
f
r
o
m
t
h
e
r
s
t
,
i
n
f
a
c
t
i
t
c
o
m
m
i
t
s
t
h
i
s
e
r
r
o
r
o
f
t
r
a
n
s
p
o
s
i
n
g
t
h
e
c
o
n
d
i
t
i
o
n
a
l
.
T
h
e
r
e
i
s
a
f
u
r
t
h
e
r
p
r
o
b
l
e
m
w
i
t
h
b
e
e
n
l
a
i
d
d
o
w
n
b
y
t
h
e
c
o
u
r
t
s
(
w
h
o
w
i
l
l
d
e
a
l
w
i
t
h
a
n
y
o
p
i
n
i
o
n
o
e
r
e
d
)
b
u
t
r
a
t
h
e
r
b
y
n
g
e
r
p
r
i
n
t
e
x
a
m
i
n
e
r
s
.
T
h
e
r
e
i
s
n
o
s
c
i
e
n
t
i
c
b
a
s
i
s
f
o
r
t
h
e
r
u
l
e
.
I
t
i
s
s
i
m
p
l
y
o
n
e
o
f
t
h
e
a
b
e
r
r
a
t
i
o
n
s
t
h
a
t
h
a
v
e
d
e
v
e
l
o
p
e
d
w
i
t
h
i
n
t
h
e
d
i
s
c
i
p
l
i
n
e
o
f
n
g
e
r
p
r
i
n
t
i
d
e
n
t
i
c
a
t
i
o
n
.
H
.
T
u
t
h
i
l
l
,
I
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n
:
P
r
i
n
c
i
-
p
l
e
s
a
n
d
P
r
o
c
e
d
u
r
e
s
i
n
C
r
i
m
i
n
a
l
i
s
t
i
c
s
(
S
a
l
e
m
,
O
r
e
g
o
n
:
L
i
g
h
t
n
i
n
g
P
o
w
d
e
r
C
o
m
p
a
n
y
,
I
n
c
,
1
9
9
4
)
.
1
2
I
n
t
e
r
n
a
t
i
o
n
a
l
A
s
s
o
c
i
a
t
i
o
n
f
o
r
I
d
e
n
t
i
c
a
t
i
o
n
R
e
s
o
l
u
t
i
o
n
2
0
1
0
^
1
8
,
A
d
o
p
t
e
d
J
u
l
y
1
6
,
2
0
1
0
i
n
S
p
o
-
k
a
n
e
.
h
t
t
p
:
/
/
o
n
i
n
.
c
o
m
/
f
p
/
I
A
I
__r
e
s
o
l
u
t
i
o
n
__2
0
1
0
-
1
8
.
p
d
f
(
l
a
s
t
v
i
s
i
t
e
d
2
1
F
e
b
r
u
a
r
y
2
0
1
1
)
.
1
3
W
.
T
h
o
m
p
s
o
n
a
n
d
E
.
S
c
h
u
m
a
n
n
,
I
n
t
e
r
p
r
e
t
a
t
i
o
n
o
f
s
t
a
t
i
s
t
i
c
a
l
e
v
i
d
e
n
c
e
i
n
c
r
i
m
i
n
a
l
t
r
i
a
l
s
:
t
h
e
p
r
o
-
s
e
c
u
t
o
r
s
f
a
l
l
a
c
y
a
n
d
t
h
e
d
e
f
e
n
s
e
a
t
t
o
r
n
e
y
s
f
a
l
l
a
c
y
(
1
9
8
7
)
1
1
L
a
w
a
n
d
H
u
m
a
n
B
e
h
a
v
i
o
u
r
1
6
7
;
P
.
D
i
a
-
c
o
n
i
s
a
n
d
D
.
F
r
e
e
d
m
a
n
,
T
h
e
p
e
r
s
i
s
t
e
n
c
e
o
f
c
o
g
n
i
t
i
v
e
i
l
l
u
s
i
o
n
s
(
1
9
8
1
)
4
T
h
e
B
e
h
a
v
i
o
r
a
l
a
n
d
B
r
a
i
n
S
c
i
e
n
c
e
s
3
3
3
;
I
.
W
.
E
v
e
t
t
,
A
v
o
i
d
i
n
g
t
h
e
t
r
a
n
s
p
o
s
e
d
c
o
n
d
i
t
i
o
n
a
l
(
1
9
9
5
)
3
5
S
c
i
&
J
1
2
7
;
a
n
d
B
.
R
o
b
e
r
t
-
s
o
n
a
n
d
G
.
A
.
V
i
g
n
a
u
x
,
I
n
t
e
r
p
r
e
t
i
n
g
E
v
i
d
e
n
c
e
:
E
v
a
l
u
a
t
i
n
g
F
o
r
e
n
s
i
c
S
c
i
e
n
c
e
i
n
t
h
e
C
o
u
r
t
r
o
o
m
n
3
a
b
o
v
e
c
h
6
.
B
e
r
n
a
r
d
R
o
b
e
r
t
s
o
n
,
G
.
A
.
V
i
g
n
a
u
x
a
n
d
C
h
a
r
l
e
s
E
.
H
.
B
e
r
g
e
r
4
4
9
r
2
0
1
1
T
h
e
A
u
t
h
o
r
s
.
T
h
e
M
o
d
e
r
n
L
a
w
R
e
v
i
e
w
r
2
0
1
1
T
h
e
M
o
d
e
r
n
L
a
w
R
e
v
i
e
w
L
i
m
i
t
e
d
.
(
2
0
1
1
)
7
4
(
3
)
4
3
0
^
4
5
5
t
h
e
n
a
l
c
l
a
u
s
e
,
d
i
s
c
u
s
s
e
d
b
e
l
o
w
,
b
u
t
l
e
a
v
i
n
g
t
h
a
t
f
o
r
t
h
e
m
o
m
e
n
t
,
t
h
i
s
s
e
n
t
e
n
c
e
s
h
o
u
l
d
h
a
v
e
s
a
i
d
:
I
n
t
h
e
p
r
e
s
e
n
t
c
a
s
e
,
i
t
w
a
s
e
x
p
r
e
s
s
e
d
a
s
t
h
e
p
r
o
b
a
b
i
l
i
t
y
t
h
a
t
t
h
e
m
a
r
k
s
d
i
s
c
o
v
e
r
e
d
a
t
t
h
e
s
c
e
n
e
w
o
u
l
d
h
a
v
e
b
e
e
n
o
b
s
e
r
v
e
d
i
f
t
h
e
N
i
k
e
t
r
a
i
n
e
r
s
o
w
n
e
d
b
y
t
h
e
a
p
p
e
l
l
a
n
t
h
a
d
m
a
d
e
t
h
e
m
a
r
k
s
d
i
v
i
d
e
d
b
y
t
h
e
p
r
o
b
a
b
i
l
i
t
y
t
h
a
t
t
h
e
m
a
r
k
s
w
o
u
l
d
h
a
v
e
b
e
e
n
o
b
s
e
r
v
e
d
i
f
t
h
o
s
e
N
i
k
e
t
r
a
i
n
e
r
s
h
a
d
n
o
t
m
a
d
e
t
h
e
m
a
r
k
s
.
T
h
i
s
i
s
o
n
l
y
o
n
e
o
f
a
n
u
m
b
e
r
o
f
e
x
a
m
p
l
e
s
o
f
t
h
i
s
v
e
r
y
e
r
r
o
r
t
h
r
o
u
g
h
o
u
t
t
h
e
j
u
d
g
-
m
e
n
t
i
n
R
v
T
.
I
n
f
a
c
t
t
h
i
s
m
i
s
u
n
d
e
r
s
t
a
n
d
i
n
g
a
p
p
e
a
r
s
i
n
t
h
e
v
e
r
y
r
s
t
u
n
r
e
d
a
c
t
e
d
p
a
r
a
g
r
a
p
h
,
p
a
r
a
g
r
a
p
h
[
1
5
]
w
h
e
r
e
t
h
e
C
o
u
r
t
s
a
y
s
:
T
h
e
r
e
a
l
i
s
s
u
e
.
.
.
w
a
s
t
h
e
u
s
e
o
f
l
i
k
e
l
i
h
o
o
d
r
a
t
i
o
s
i
n
f
o
r
m
i
n
g
a
n
e
v
a
l
u
a
t
i
v
e
o
p
i
n
i
o
n
o
n
t
h
e
d
e
g
r
e
e
o
f
l
i
k
e
l
i
h
o
o
d
t
h
a
t
a
m
a
r
k
h
a
d
b
e
e
n
m
a
d
e
b
y
a
p
a
r
t
i
c
u
l
a
r
i
t
e
m
o
f
f
o
o
t
-
w
e
a
r
.
I
t
i
s
n
o
t
w
h
o
l
l
y
c
l
e
a
r
w
h
o
s
e
e
v
a
l
u
a
t
i
v
e
o
p
i
n
i
o
n
t
h
e
C
o
u
r
t
i
s
r
e
f
e
r
r
i
n
g
t
o
h
e
r
e
.
I
f
i
t
i
s
r
e
f
e
r
r
i
n
g
t
o
t
h
e
c
o
n
c
l
u
s
i
o
n
t
o
w
h
i
c
h
t
h
e
j
u
r
y
m
u
s
t
c
o
m
e
,
t
h
e
n
t
h
i
s
s
t
a
t
e
m
e
n
t
o
f
t
h
e
i
s
s
u
e
i
s
c
o
r
r
e
c
t
.
T
h
i
s
i
s
n
o
t
t
h
e
i
s
s
u
e
w
i
t
h
w
h
i
c
h
t
h
e
C
o
u
r
t
s
u
b
s
e
q
u
e
n
t
l
y
g
r
a
p
-
p
l
e
s
,
h
o
w
e
v
e
r
,
a
n
d
t
h
e
n
a
t
u
r
a
l
r
e
a
d
i
n
g
i
s
t
h
a
t
t
h
e
C
o
u
r
t
i
s
r
e
f
e
r
r
i
n
g
t
o
t
h
e
w
i
t
n
e
s
s
s
e
v
i
d
e
n
c
e
.
T
h
e
w
i
t
n
e
s
s
,
h
o
w
e
v
e
r
,
d
i
d
n
o
t
p
u
r
p
o
r
t
t
o
g
i
v
e
a
n
o
p
i
n
i
o
n
o
n
t
h
e
l
i
k
e
-
l
i
h
o
o
d
t
h
a
t
t
h
e
m
a
r
k
h
a
d
b
e
e
n
m
a
d
e
b
y
a
p
a
r
t
i
c
u
l
a
r
s
h
o
e
b
u
t
o
n
l
y
a
s
t
o
h
o
w
s
t
r
o
n
g
l
y
t
h
e
e
v
i
d
e
n
c
e
s
u
p
p
o
r
t
e
d
t
h
a
t
p
r
o
p
o
s
i
t
i
o
n
.
A
n
o
t
h
e
r
i
m
p
o
r
t
a
n
t
i
n
s
t
a
n
c
e
a
p
p
e
a
r
s
a
t
p
a
r
a
g
r
a
p
h
s
[
6
6
]
a
n
d
[
6
7
]
.
T
h
e
C
o
u
r
t
r
e
f
e
r
r
e
d
t
o
t
h
e
U
S
v
e
r
b
a
l
s
c
a
l
e
,
w
h
i
c
h
u
s
e
s
e
x
p
r
e
s
s
i
o
n
s
s
u
c
h
a
s
p
r
o
b
a
b
l
y
m
a
d
e
c
o
u
l
d
h
a
v
e
m
a
d
e
a
n
d
p
r
o
b
a
b
l
y
d
i
d
n
o
t
m
a
k
e
.
I
t
h
a
d
p
r
e
v
i
o
u
s
l
y
d
i
s
c
u
s
s
e
d
(
a
t
[
3
1
]
)
t
h
e
F
S
S
v
e
r
b
a
l
s
c
a
l
e
w
h
i
c
h
u
s
e
s
e
x
p
r
e
s
s
i
o
n
s
s
u
c
h
a
s
m
o
d
e
r
a
t
e
s
u
p
p
o
r
t
/
s
t
r
o
n
g
s
u
p
p
o
r
t
/
v
e
r
y
s
t
r
o
n
g
s
u
p
p
o
r
t
f
o
r
t
h
e
h
y
p
o
t
h
e
s
i
s
o
f
c
o
m
m
o
n
o
r
i
g
i
n
.
T
h
e
C
o
u
r
t
t
h
e
n
r
e
c
o
r
d
s
t
h
e
w
i
t
n
e
s
s
a
s
s
a
y
i
n
g
t
h
a
t
t
h
e
t
e
r
m
i
n
o
l
o
g
y
u
s
e
d
i
n
t
h
e
U
n
i
t
e
d
S
t
a
t
e
s
a
n
d
t
h
a
t
u
s
e
d
i
n
E
n
g
l
a
n
d
a
n
d
W
a
l
e
s
w
e
r
e
s
i
m
p
l
y
d
i
e
r
e
n
t
m
e
a
n
s
o
f
e
x
p
r
e
s
s
i
n
g
a
v
e
r
b
a
l
s
c
a
l
e
o
f
c
o
n
c
l
u
s
i
o
n
s
.
T
h
i
s
i
s
n
o
t
c
o
r
r
e
c
t
.
T
h
e
y
s
a
y
d
i
e
r
e
n
t
t
h
i
n
g
s
.
T
h
e
E
n
g
-
l
i
s
h
a
n
d
W
e
l
s
h
s
c
a
l
e
e
x
p
r
e
s
s
e
s
h
o
w
s
t
r
o
n
g
l
y
t
h
e
e
v
i
d
e
n
c
e
s
u
p
p
o
r
t
s
t
h
e
h
y
p
o
t
h
e
s
i
s
.
T
h
e
U
S
s
c
a
l
e
p
u
r
p
o
r
t
s
t
o
e
x
p
r
e
s
s
h
o
w
p
r
o
b
a
b
l
e
t
h
e
h
y
p
o
t
h
e
s
i
s
i
s
,
w
h
i
c
h
,
a
s
w
e
h
a
v
e
s
e
e
n
,
t
h
e
e
x
p
e
r
t
c
a
n
n
o
t
a
s
s
e
s
s
.
T
h
i
s
e
r
r
o
r
a
p
p
e
a
r
s
c
o
n
s
i
s
t
e
n
t
l
y
t
h
r
o
u
g
h
t
h
e
j
u
d
g
m
e
n
t
.
1
4
T
o
r
e
p
e
a
t
,
t
h
e
v
a
l
u
e
o
f
a
n
i
t
e
m
o
f
e
v
i
d
e
n
c
e
l
i
e
s
i
n
i
t
s
a
b
i
l
i
t
y
t
o
d
i
s
t
i
n
g
u
i
s
h
b
e
t
w
e
e
n
t
w
o
h
y
p
o
t
h
e
s
e
s
,
o
n
e
o
f
w
h
i
c
h
,
i
n
t
h
i
s
c
a
s
e
,
i
s
t
h
a
t
t
h
e
s
u
s
p
e
c
t
s
s
h
o
e
l
e
f
t
t
h
e
m
a
r
k
.
I
d
e
a
l
l
y
,
t
h
e
a
l
t
e
r
n
a
t
i
v
e
h
y
p
o
t
h
e
s
i
s
s
h
o
u
l
d
b
e
p
r
e
c
i
s
e
l
y
s
t
a
t
e
d
i
n
p
o
s
i
t
i
v
e
r
a
t
h
e
r
t
h
a
n
n
e
g
a
t
i
v
e
t
e
r
m
s
.
I
n
s
t
e
a
d
,
i
n
R
v
T
t
h
e
e
x
p
e
r
t
r
e
f
e
r
r
e
d
t
o
a
n
e
g
a
t
i
v
e
l
y
-
e
x
p
r
e
s
s
e
d
a
l
t
e
r
n
a
t
i
v
e
h
y
p
o
t
h
e
s
i
s
t
h
a
t
t
h
e
s
u
s
p
e
c
t
s
s
h
o
e
d
i
d
n
o
t
m
a
k
e
t
h
e
m
a
r
k
.
T
h
i
s
l
e
d
t
o
t
r
o
u
b
l
e
w
h
e
n
t
h
e
c
o
u
r
t
c
a
m
e
t
o
c
o
n
s
i
d
e
r
t
h
e
a
p
p
r
o
p
r
i
a
t
e
d
a
t
a
b
a
s
e
,
a
s
t
h
a
t
i
s
d
e
t
e
r
m
i
n
e
d
b
y
a
p
o
s
i
t
i
v
e
l
y
-
e
x
p
r
e
s
s
e
d
a
l
t
e
r
n
a
t
i
v
e
h
y
p
o
t
h
e
s
i
s
r
e
l
e
v
a
n
t
t
o
f
a
c
t
s
o
f
t
h
e
c
a
s
e
.
A
t
[
4
2
]
a
n
d
[
4
3
]
,
t
h
e
C
o
u
r
t
c
o
n
s
i
d
e
r
e
d
t
w
o
p
o
s
s
i
b
l
e
d
a
t
a
b
a
s
e
s
.
T
h
e
w
i
t
n
e
s
s
a
p
p
e
a
r
s
t
o
h
a
v
e
u
s
e
d
t
h
e
F
o
r
e
n
s
i
c
S
c
i
e
n
c
e
S
e
r
v
i
c
e
d
a
t
a
b
a
s
e
,
i
n
w
h
i
c
h
t
h
e
s
o
l
e
p
a
t
-
t
e
r
n
c
o
n
c
e
r
n
e
d
w
a
s
r
e
l
a
t
i
v
e
l
y
c
o
m
m
o
n
.
T
h
e
C
o
u
r
t
a
l
s
o
d
i
s
c
u
s
s
e
d
a
d
a
t
a
b
a
s
e
c
o
n
-
1
4
O
t
h
e
r
e
x
a
m
p
l
e
s
a
r
e
a
t
[
3
0
]
,
[
3
1
]
,
[
5
0
]
,
[
6
7
]
a
n
d
[
6
9
]
.
E
x
t
e
n
d
i
n
g
t
h
e
C
o
n
f
u
s
i
o
n
A
b
o
u
t
B
a
y
e
s
4
5
0
r
2
0
1
1
T
h
e
A
u
t
h
o
r
s
.
T
h
e
M
o
d
e
r
n
L
a
w
R
e
v
i
e
w
r
2
0
1
1
T
h
e
M
o
d
e
r
n
L
a
w
R
e
v
i
e
w
L
i
m
i
t
e
d
.
(
2
0
1
1
)
7
4
(
3
)
4
3
0
^
4
5
5
s
i
s
t
i
n
g
o
f
f
r
e
q
u
e
n
c
i
e
s
o
f
t
h
e
d
i
s
t
r
i
b
u
t
i
o
n
o
f
s
h
o
e
s
b
y
m
a
n
u
f
a
c
t
u
r
e
r
s
.
W
h
i
c
h
o
f
t
h
e
s
e
w
o
u
l
d
,
c
o
n
c
e
p
t
u
a
l
l
y
,
b
e
t
h
e
c
o
r
r
e
c
t
d
a
t
a
b
a
s
e
d
e
p
e
n
d
s
o
n
t
h
e
a
l
t
e
r
n
a
t
i
v
e
o
r
d
e
f
e
n
c
e
h
y
p
o
t
h
e
s
i
s
.
P
l
a
u
s
i
b
l
y
,
t
h
i
s
s
h
o
u
l
d
b
e
t
h
a
t
a
n
o
t
h
e
r
p
e
r
s
o
n
l
e
f
t
t
h
e
m
a
r
k
a
n
d
w
a
s
a
c
r
i
m
i
n
a
l
,
r
a
t
h
e
r
t
h
a
n
a
r
a
n
d
o
m
l
y
s
e
l
e
c
t
e
d
m
e
m
b
e
r
o
f
t
h
e
p
o
p
u
l
a
t
i
o
n
.
S
i
n
c
e
t
h
e
s
h
o
e
t
y
p
e
c
o
n
c
e
r
n
e
d
s
e
e
m
s
t
o
b
e
f
a
v
o
u
r
e
d
b
y
s
u
s
p
e
c
t
s
(
v
i
z
i
t
s
r
e
l
a
t
i
v
e
l
y
c
o
m
-
m
o
n
a
p
p
e
a
r
a
n
c
e
i
n
t
h
e
F
S
S
d
a
t
a
b
a
s
e
)
,
t
h
i
s
,
i
n
t
h
e
p
r
e
s
e
n
t
c
a
s
e
,
p
r
o
d
u
c
e
d
e
v
i
d
e
n
c
e
m
o
r
e
f
a
v
o
u
r
a
b
l
e
t
o
t
h
e
a
c
c
u
s
e
d
t
h
a
n
t
h
e
u
s
e
o
f
a
d
a
t
a
b
a
s
e
d
r
a
w
n
f
r
o
m
m
a
n
u
f
a
c
-
t
u
r
e
r
s
g
u
r
e
s
o
n
s
h
o
e
s
s
o
l
d
t
o
t
h
e
g
e
n
e
r
a
l
p
o
p
u
l
a
t
i
o
n
.
T
H
E
C
O
U
R
T
S
E
V
A
L
U
A
T
I
O
N
T
h
e
C
o
u
r
t
i
n
R
v
T
c
o
m
m
e
n
c
e
d
i
t
s
e
v
a
l
u
a
t
i
o
n
b
y
r
e
f
e
r
r
i
n
g
,
a
t
[
4
6
]
,
t
o
c
a
s
e
s
i
n
w
h
i
c
h
i
t
s
a
i
d
t
h
a
t
t
h
e
B
a
y
e
s
i
a
n
a
p
p
r
o
a
c
h
h
a
d
b
e
e
n
r
o
b
u
s
t
l
y
r
e
j
e
c
t
e
d
i
n
r
e
s
p
e
c
t
o
f
n
o
n
-
D
N
A
e
v
i
d
e
n
c
e
.
T
h
e
s
e
c
a
s
e
s
i
n
c
l
u
d
e
R
v
D
e
n
i
s
A
d
a
m
s
[
1
9
9
6
]
2
C
r
A
p
p
R
4
6
7
(
A
d
a
m
s
)
,
R
v
A
d
a
m
s
(
N
o
2
)
[
1
9
9
8
]
1
C
r
A
p
p
R
3
7
7
;
R
v
D
o
h
e
n
y
a
n
d
A
d
a
m
s
[
1
9
9
7
]
1
C
r
A
p
p
R
3
6
9
.
T
h
e
C
o
u
r
t
d
o
e
s
n
o
t
r
e
f
e
r
t
o
R
v
D
e
e
n
C
A
2
1
D
e
c
e
m
b
e
r
1
9
9
3
,
i
n
w
h
i
c
h
L
o
r
d
T
a
y
l
o
r
C
J
a
c
c
e
p
t
e
d
t
h
e
u
s
e
o
f
l
i
k
e
l
i
h
o
o
d
r
a
t
i
o
s
a
n
d
h
e
l
d
t
h
a
t
t
h
e
w
i
t
-
n
e
s
s
h
a
d
b
e
e
n
w
r
o
n
g
t
o
s
t
a
t
e
a
c
o
n
c
l
u
s
i
o
n
t
h
a
t
t
h
e
s
u
s
p
e
c
t
h
a
d
l
e
f
t
t
h
e
m
a
r
k
.
T
h
e
C
o
u
r
t
r
e
f
e
r
s
t
o
o
n
l
y
o
n
e
a
r
t
i
c
l
e
o
n
A
d
a
m
s
(
a
t
[
9
1
]
)
.
1
5
A
d
a
m
s
w
a
s
a
n
u
n
u
s
u
a
l
c
a
s
e
.
D
N
A
e
v
i
d
e
n
c
e
i
m
p
l
i
c
a
t
e
d
t
h
e
a
c
c
u
s
e
d
.
O
t
h
e
r
e
v
i
-
d
e
n
c
e
f
a
v
o
u
r
e
d
t
h
e
a
c
c
u
s
e
d
s
c
a
s
e
.
T
h
e
d
e
f
e
n
c
e
c
a
l
l
e
d
a
p
r
o
f
e
s
s
o
r
o
f
s
t
a
t
i
s
t
i
c
s
t
o
e
x
p
l
a
i
n
i
n
B
a
y
e
s
i
a
n
t
e
r
m
s
h
o
w
t
h
e
j
u
r
y
c
o
u
l
d
c
o
m
b
i
n
e
t
h
e
f
a
v
o
u
r
a
b
l
e
e
v
i
d
e
n
c
e
w
i
t
h
t
h
e
D
N
A
l
i
k
e
l
i
h
o
o
d
r
a
t
i
o
t
o
a
r
r
i
v
e
a
t
a
c
o
m
b
i
n
e
d
g
u
r
e
f
o
r
t
h
e
e
v
i
d
e
n
c
e
a
n
d
t
h
e
n
a
t
a
p
r
o
b
a
b
i
l
i
t
y
f
o
r
t
h
e
p
r
o
s
e
c
u
t
i
o
n
c
a
s
e
w
h
i
c
h
w
o
u
l
d
b
e
,
t
h
e
d
e
f
e
n
c
e
h
o
p
e
d
,
b
e
l
o
w
w
h
a
t
w
a
s
r
e
q
u
i
r
e
d
f
o
r
p
r
o
o
f
b
e
y
o
n
d
r
e
a
s
o
n
a
b
l
e
d
o
u
b
t
.
T
h
e
a
c
c
u
s
e
d
w
a
s
c
o
n
v
i
c
t
e
d
a
n
d
a
p
p
e
a
l
e
d
.
T
h
e
p
r
o
f
e
s
s
o
r
s
e
v
i
d
e
n
c
e
w
a
s
n
o
t
a
m
a
t
t
e
r
r
a
i
s
e
d
i
n
t
h
e
a
p
p
e
a
l
a
n
d
w
a
s
n
o
t
t
h
e
s
u
b
j
e
c
t
o
f
e
x
p
e
r
t
e
v
i
d
e
n
c
e
o
r
a
r
g
u
m
e
n
t
.
N
o
n
e
t
h
e
l
e
s
s
,
t
h
e
C
o
u
r
t
f
e
l
t
a
b
l
e
t
o
m
a
k
e
a
n
u
m
b
e
r
o
f
c
o
m
m
e
n
t
s
a
b
o
u
t
t
h
e
u
s
e
o
f
l
i
k
e
l
i
h
o
o
d
r
a
t
i
o
s
,
a
n
d
t
h
e
s
u
c
c
e
e
d
i
n
g
c
a
s
e
s
s
i
m
p
l
y
p
i
c
k
e
d
u
p
w
h
a
t
w
a
s
s
a
i
d
i
n
A
d
a
m
s
a
n
d
r
e
p
e
a
t
e
d
i
t
.
S
i
n
c
e
t
h
e
r
e
l
e
v
a
n
t
m
a
t
t
e
r
s
w
e
r
e
n
o
t
a
r
g
u
e
d
,
i
t
i
s
r
e
s
p
e
c
t
f
u
l
l
y
s
u
b
m
i
t
t
e
d
t
h
a
t
A
d
a
m
s
i
s
n
o
t
a
u
t
h
o
r
i
t
y
,
a
s
t
h
e
C
o
u
r
t
i
n
R
v
T
s
a
y
s
i
t
i
s
,
f
o
r
t
h
e
r
e
j
e
c
t
i
o
n
o
f
t
h
e
B
a
y
e
s
i
a
n
a
p
p
r
o
a
c
h
.
O
n
e
o
f
t
h
e
c
r
i
t
i
c
i
s
m
s
o
f
A
d
a
m
s
i
s
t
h
a
t
t
h
e
C
o
u
r
t
g
a
v
e
n
o
g
u
i
d
a
n
c
e
a
s
t
o
h
o
w
j
u
r
o
r
s
w
e
r
e
t
o
c
o
m
b
i
n
e
e
v
i
d
e
n
c
e
i
f
t
h
e
y
w
e
r
e
n
o
t
t
o
u
s
e
t
h
e
l
o
g
i
c
a
l
B
a
y
e
s
i
a
n
m
e
t
h
o
d
s
.
I
n
t
h
e
s
a
m
e
w
a
y
,
t
h
e
C
o
u
r
t
i
n
R
v
T
r
e
f
e
r
s
o
p
a
q
u
e
l
y
t
o
e
x
p
e
r
i
e
n
c
e
a
n
d
j
u
d
g
m
e
n
t
.
T
h
e
s
e
a
r
e
b
l
a
c
k
b
o
x
e
s
.
T
h
e
w
i
t
n
e
s
s
i
n
R
v
T
m
u
l
t
i
p
l
i
e
d
t
o
g
e
t
h
e
r
t
h
e
l
i
k
e
l
i
h
o
o
d
r
a
t
i
o
s
h
e
a
s
s
e
s
s
e
d
f
o
r
t
h
e
v
a
r
i
o
u
s
a
s
p
e
c
t
s
o
f
t
h
e
e
v
i
d
e
n
c
e
f
r
o
m
t
h
e
f
o
o
t
-
w
e
a
r
m
a
r
k
t
o
a
r
r
i
v
e
a
t
a
c
o
m
p
o
s
i
t
e
l
i
k
e
l
i
h
o
o
d
r
a
t
i
o
e
x
p
r
e
s
s
i
n
g
t
h
e
s
t
r
e
n
g
t
h
o
f
t
h
e
e
v
i
d
e
n
c
e
f
o
r
t
h
e
f
o
o
t
w
e
a
r
m
a
r
k
a
s
a
w
h
o
l
e
.
P
a
r
t
o
f
w
h
a
t
t
h
e
w
i
t
n
e
s
s
s
e
e
m
s
t
o
h
a
v
e
d
o
n
e
w
r
o
n
g
,
i
n
t
h
e
e
y
e
s
o
f
t
h
e
C
o
u
r
t
,
w
a
s
t
o
e
x
p
o
s
e
t
h
e
r
e
a
s
o
n
i
n
g
t
h
a
t
l
i
e
s
b
e
h
i
n
d
s
u
c
h
e
x
p
e
r
i
e
n
c
e
o
r
j
u
d
g
m
e
n
t
.
I
n
p
a
r
t
i
c
u
l
a
r
,
a
t
[
7
9
]
,
t
h
e
C
o
u
r
t
r
e
f
e
r
s
t
o
t
h
e
v
i
e
w
1
5
T
w
o
o
f
t
h
e
p
r
e
s
e
n
t
a
u
t
h
o
r
s
v
i
g
o
r
o
u
s
l
y
c
r
i
t
i
c
i
s
e
d
A
d
a
m
s
a
t
t
h
e
t
i
m
e
,
s
e
e
R
o
b
e
r
t
s
o
n
a
n
d
V
i
g
n
a
u
x
,
B
a
y
e
s
T
h
e
o
r
e
m
i
n
t
h
e
C
o
u
r
t
o
f
A
p
p
e
a
l
(
1
9
9
7
)
6
9
T
h
e
C
r
i
m
i
n
a
l
L
a
w
y
e
r
4
,
t
e
x
t
a
v
a
i
l
a
b
l
e
a
t
:
t
i
n
y
u
r
l
.
-
c
o
m
/
d
e
n
i
s
a
d
a
m
s
B
e
r
n
a
r
d
R
o
b
e
r
t
s
o
n
,
G
.
A
.
V
i
g
n
a
u
x
a
n
d
C
h
a
r
l
e
s
E
.
H
.
B
e
r
g
e
r
4
5
1
r
2
0
1
1
T
h
e
A
u
t
h
o
r
s
.
T
h
e
M
o
d
e
r
n
L
a
w
R
e
v
i
e
w
r
2
0
1
1
T
h
e
M
o
d
e
r
n
L
a
w
R
e
v
i
e
w
L
i
m
i
t
e
d
.
(
2
0
1
1
)
7
4
(
3
)
4
3
0
^
4
5
5
t
h
a
t
a
l
l
p
r
o
b
a
b
i
l
i
t
y
a
s
s
e
s
s
m
e
n
t
s
h
a
v
e
s
o
m
e
l
e
v
e
l
o
f
u
n
c
e
r
t
a
i
n
t
y
a
n
d
t
h
a
t
t
h
i
s
u
n
c
e
r
-
t
a
i
n
t
y
i
s
a
c
c
o
m
m
o
d
a
t
e
d
i
n
t
h
e
l
i
k
e
l
i
h
o
o
d
r
a
t
i
o
.
T
h
e
C
o
u
r
t
w
e
n
t
o
n
t
o
s
a
y
,
a
t
[
8
0
]
:
W
e
c
a
n
n
o
t
a
g
r
e
e
w
i
t
h
t
h
i
s
i
n
s
o
f
a
r
a
s
i
t
s
u
g
g
e
s
t
s
t
h
a
t
a
m
a
t
h
e
m
a
t
i
c
a
l
f
o
r
m
u
l
a
c
a
n
b
e
u
s
e
d
.
A
n
a
p
p
r
o
a
c
h
b
a
s
e
d
o
n
m
a
t
h
e
m
a
t
i
c
a
l
c
a
l
c
u
l
a
t
i
o
n
s
i
s
o
n
l
y
a
s
g
o
o
d
a
s
t
h
e
r
e
l
i
a
-
b
i
l
i
t
y
o
f
t
h
e
d
a
t
a
u
s
e
d
.
T
h
e
s
e
c
o
n
d
s
e
n
t
e
n
c
e
i
s
t
r
u
i
s
t
i
c
.
T
h
e
r
s
t
s
e
n
t
e
n
c
e
r
e
f
e
r
s
t
o
a
m
a
t
h
e
m
a
t
i
c
a
l
f
o
r
m
u
l
a
b
u
t
,
a
s
w
e
h
a
v
e
s
e
e
n
,
i
t
i
s
a
m
a
t
t
e
r
o
f
f
o
r
m
a
l
l
o
g
i
c
r
a
t
h
e
r
t
h
a
n
m
a
t
h
e
m
a
t
i
c
s
.
T
h
e
c
o
r
r
e
c
t
w
a
y
t
o
a
r
r
i
v
e
a
t
a
c
o
m
p
o
s
i
t
e
l
i
k
e
l
i
h
o
o
d
r
a
t
i
o
i
s
t
o
m
u
l
t
i
p
l
y
t
h
e
l
i
k
e
l
i
h
o
o
d
r
a
t
i
o
s
f
o
r
t
h
e
v
a
r
i
o
u
s
i
t
e
m
s
o
f
e
v
i
d
e
n
c
e
t
o
g
e
t
h
e
r
.
1
6
T
h
i
s
a
p
p
l
i
e
s
w
h
e
t
h
e
r
e
a
c
h
l
i
k
e
-
l
i
h
o
o
d
r
a
t
i
o
i
s
b
a
s
e
d
o
n
p
r
e
c
i
s
e
o
r
i
m
p
r
e
c
i
s
e
d
a
t
a
o
r
,
f
o
r
t
h
a
t
m
a
t
t
e
r
,
o
n
e
x
p
e
r
i
e
n
c
e
a
n
d
j
u
d
g
m
e
n
t
.
S
a
y
i
n
g
t
h
a
t
t
h
e
e
x
p
e
r
t
s
h
o
u
l
d
n
o
t
u
s
e
t
h
i
s
m
a
t
h
e
m
a
t
i
c
a
l
f
o
r
m
u
l
a
t
o
a
s
s
e
s
s
t
h
e
c
o
m
p
o
s
i
t
e
l
i
k
e
l
i
h
o
o
d
r
a
t
i
o
i
s
l
i
k
e
s
a
y
i
n
g
t
h
a
t
i
f
o
n
e
i
s
j
u
s
t
e
s
t
i
m
a
t
i
n
g
b
y
e
y
e
t
h
e
a
r
e
a
o
f
a
e
l
d
,
o
n
e
i
s
n
o
t
a
l
l
o
w
e
d
t
o
m
u
l
t
i
p
l
y
e
s
t
i
m
a
t
e
s
o
f
i
t
s
w
i
d
t
h
a
n
d
l
e
n
g
t
h
t
o
g
e
t
h
e
r
.
C
l
e
a
r
l
y
i
t
i
s
t
h
e
c
o
r
r
e
c
t
p
r
o
c
e
d
u
r
e
:
t
h
e
r
e
i
s
n
o
u
n
c
e
r
t
a
i
n
t
y
i
n
t
h
e
r
e
l
a
t
i
o
n
b
e
t
w
e
e
n
l
e
n
g
t
h
,
w
i
d
t
h
a
n
d
a
r
e
a
,
o
n
l
y
i
n
t
h
e
i
r
v
a
l
u
e
s
.
I
f
t
h
e
C
o
u
r
t
w
e
r
e
t
o
s
a
y
t
h
a
t
t
h
e
e
x
p
e
r
t
w
a
s
n
o
t
t
o
u
s
e
a
l
o
g
i
c
a
l
p
r
o
c
e
d
u
r
e
,
r
a
t
h
e
r
t
h
a
n
a
m
a
t
h
e
m
a
t
i
c
a
l
f
o
r
m
u
l
a
,
t
h
e
a
w
i
n
i
t
s
r
e
a
s
o
n
i
n
g
w
o
u
l
d
b
e
o
b
v
i
o
u
s
.
T
h
e
C
o
u
r
t
i
l
l
u
s
t
r
a
t
e
d
t
h
e
t
r
u
t
h
o
f
t
h
e
s
e
c
o
n
d
s
e
n
t
e
n
c
e
o
f
p
a
r
a
g
r
a
p
h
[
8
0
]
b
y
s
h
o
w
i
n
g
t
h
a
t
d
e
f
e
n
c
e
c
o
u
n
s
e
l
h
a
d
o
b
t
a
i
n
e
d
o
t
h
e
r
g
u
r
e
s
w
h
i
c
h
a
c
t
u
a
l
l
y
t
u
r
n
e
d
o
u
t
t
o
b
e
m
u
c
h
m
o
r
e
d
a
m
n
i
n
g
o
f
t
h
e
a
c
c
u
s
e
d
.
T
h
e
C
o
u
r
t
s
u
b
s
e
q
u
e
n
t
l
y
,
a
t
[
1
0
8
]
,
g
a
v
e
f
o
u
r
r
e
a
s
o
n
s
w
h
y
t
h
e
c
o
n
v
i
c
t
i
o
n
w
a
s
u
n
s
a
f
e
.
T
h
e
s
e
a
r
e
c
o
m
m
e
n
t
e
d
o
n
o
n
e
a
t
a
t
i
m
e
:
i
)
T
h
e
c
a
l
c
u
l
a
t
i
o
n
s
w
h
i
c
h
M
r
R
y
d
e
r
h
a
d
u
s
e
d
i
n
r
e
a
c
h
i
n
g
h
i
s
v
i
e
w
w
e
r
e
n
o
t
b
e
f
o
r
e
t
h
e
j
u
r
y
;
t
h
o
s
e
g
u
r
e
s
w
e
r
e
m
o
r
e
f
a
v
o
u
r
a
b
l
e
t
o
t
h
e
a
p
p
e
l
l
a
n
t
t
h
a
n
t
h
e
g
u
r
e
s
p
u
t
b
e
f
o
r
e
t
h
e
j
u
r
y
.
I
f
e
v
i
d
e
n
c
e
o
f
t
h
e
f
u
l
l
g
u
r
e
s
h
a
d
b
e
e
n
p
u
t
b
e
f
o
r
e
t
h
e
j
u
r
y
t
h
e
n
,
a
p
p
l
y
i
n
g
t
h
e
t
e
s
t
i
n
R
v
P
e
n
d
l
e
t
o
n
[
2
0
0
1
]
1
W
L
R
7
2
,
[
2
0
0
2
]
1
C
r
A
p
p
R
4
4
1
,
i
t
m
i
g
h
t
r
e
a
s
o
n
a
b
l
y
h
a
v
e
a
e
c
t
e
d
t
h
e
d
e
c
i
s
i
o
n
o
f
t
h
e
j
u
r
y
t
o
c
o
n
v
i
c
t
.
W
h
a
t
h
a
p
p
e
n
e
d
(
s
e
e
[
4
2
]
^
[
4
4
]
)
,
w
a
s
t
h
a
t
t
h
e
w
i
t
n
e
s
s
t
e
s
t
i
e
d
t
h
a
t
t
h
e
e
v
i
d
e
n
c
e
g
a
v
e
m
o
d
e
r
a
t
e
s
u
p
p
o
r
t
t
o
t
h
e
h
y
p
o
t
h
e
s
i
s
t
h
a
t
t
h
e
a
c
c
u
s
e
d
s
s
h
o
e
h
a
d
m
a
d
e
t
h
e
m
a
r
k
.
D
e
f
e
n
c
e
c
o
u
n
s
e
l
t
h
e
n
c
r
o
s
s
-
e
x
a
m
i
n
e
d
o
n
t
h
e
b
a
s
i
s
o
f
a
n
o
t
h
e
r
s
e
t
o
f
g
u
r
e
s
f
a
r
m
o
r
e
u
n
f
a
v
o
u
r
a
b
l
e
t
o
t
h
e
a
c
c
u
s
e
d
.
P
r
o
s
e
c
u
t
i
n
g
c
o
u
n
s
e
l
d
e
c
i
d
e
d
n
o
t
t
o
r
e
-
e
x
a
m
i
n
e
a
s
t
h
e
d
e
f
e
n
c
e
h
a
d
n
o
t
u
n
d
e
r
m
i
n
e
d
t
h
e
w
i
t
n
e
s
s
s
e
v
i
d
e
n
c
e
b
u
t
h
a
d
s
t
r
e
n
g
t
h
e
n
e
d
i
t
.
H
a
d
a
n
y
r
e
-
e
x
a
m
i
n
a
t
i
o
n
t
a
k
e
n
p
l
a
c
e
,
t
h
e
w
i
t
n
e
s
s
w
o
u
l
d
,
i
n
a
d
d
i
-
t
i
o
n
t
o
g
i
v
i
n
g
h
i
s
o
w
n
o
r
i
g
i
n
a
l
g
u
r
e
s
,
h
a
v
e
b
e
e
n
b
o
u
n
d
t
o
a
c
c
e
p
t
t
h
a
t
t
h
e
g
u
r
e
s
p
u
t
t
o
h
i
m
h
a
d
b
e
e
n
r
e
a
s
o
n
a
b
l
e
a
n
d
t
h
a
t
h
e
h
a
d
d
e
l
i
b
e
r
a
t
e
l
y
e
r
r
e
d
o
n
t
h
e
s
i
d
e
o
f
c
a
u
t
i
o
n
.
I
t
i
s
h
a
r
d
t
o
s
e
e
t
h
a
t
t
h
i
s
w
o
u
l
d
u
n
d
e
r
m
i
n
e
t
h
e
e
v
i
d
e
n
c
e
i
n
t
h
e
e
y
e
s
o
f
t
h
e
j
u
r
y
.
i
i
)
T
h
e
p
r
o
c
e
s
s
b
y
w
h
i
c
h
t
h
e
e
v
i
d
e
n
c
e
w
a
s
a
d
d
u
c
e
d
l
a
c
k
e
d
t
r
a
n
s
p
a
r
e
n
c
y
.
T
h
i
s
i
s
n
o
p
e
r
s
o
n
a
l
c
r
i
t
i
c
i
s
m
o
f
M
r
R
y
d
e
r
,
a
s
h
e
w
a
s
s
i
m
p
l
y
f
o
l
l
o
w
i
n
g
p
r
a
c
t
i
c
e
.
H
o
w
e
v
e
r
,
i
t
i
s
1
6
A
s
s
u
m
i
n
g
t
h
e
e
v
i
d
e
n
c
e
i
s
i
n
d
e
p
e
n
d
e
n
t
a
n
d
t
h
e
s
a
m
e
h
y
p
o
t
h
e
s
e
s
a
p
p
l
y
.
E
x
t
e
n
d
i
n
g
t
h
e
C
o
n
f
u
s
i
o
n
A
b
o
u
t
B
a
y
e
s
4
5
2
r
2
0
1
1
T
h
e
A
u
t
h
o
r
s
.
T
h
e
M
o
d
e
r
n
L
a
w
R
e
v
i
e
w
r
2
0
1
1
T
h
e
M
o
d
e
r
n
L
a
w
R
e
v
i
e
w
L
i
m
i
t
e
d
.
(
2
0
1
1
)
7
4
(
3
)
4
3
0
^
4
5
5
s
i
m
p
l
y
w
r
o
n
g
i
n
p
r
i
n
c
i
p
l
e
f
o
r
a
n
e
x
p
e
r
t
t
o
f
a
i
l
t
o
s
e
t
o
u
t
t
h
e
w
a
y
i
n
w
h
i
c
h
h
e
h
a
s
r
e
a
c
h
e
d
h
i
s
c
o
n
c
l
u
s
i
o
n
i
n
h
i
s
r
e
p
o
r
t
.
W
e
w
o
u
l
d
a
g
r
e
e
t
h
a
t
e
x
p
e
r
t
w
i
t
n
e
s
s
e
s
s
h
o
u
l
d
i
n
c
l
u
d
e
t
h
e
i
r
m
e
t
h
o
d
o
l
o
g
y
i
n
t
h
e
i
r
r
e
p
o
r
t
s
.
I
t
s
e
e
m
s
,
h
o
w
e
v
e
r
,
t
h
a
t
t
h
e
C
o
u
r
t
w
o
u
l
d
h
a
v
e
b
e
e
n
c
o
n
t
e
n
t
w
i
t
h
o
p
a
q
u
e
r
e
f
e
r
e
n
c
e
s
t
o
e
x
p
e
r
i
e
n
c
e
a
n
d
j
u
d
g
m
e
n
t
.
i
i
i
)
I
n
t
h
e
l
i
g
h
t
o
f
t
h
e
s
t
r
o
n
g
c
r
i
t
i
c
i
s
m
b
y
t
h
i
s
c
o
u
r
t
i
n
t
h
e
1
9
9
0
s
o
f
u
s
i
n
g
B
a
y
e
s
t
h
e
o
r
e
m
b
e
f
o
r
e
t
h
e
j
u
r
y
i
n
c
a
s
e
s
w
h
e
r
e
t
h
e
r
e
w
a
s
n
o
r
e
l
i
a
b
l
e
s
t
a
t
i
s
t
i
c
a
l
e
v
i
d
e
n
c
e
,
t
h
e
p
r
a
c
t
i
c
e
o
f
u
s
i
n
g
a
B
a
y
e
s
i
a
n
a
p
p
r
o
a
c
h
a
n
d
l
i
k
e
l
i
h
o
o
d
r
a
t
i
o
s
t
o
f
o
r
m
u
l
a
t
e
o
p
i
n
i
o
n
s
p
l
a
c
e
d
b
e
f
o
r
e
a
j
u
r
y
w
i
t
h
o
u
t
t
h
a
t
p
r
o
c
e
s
s
b
e
i
n
g
d
i
s
c
l
o
s
e
d
a
n
d
d
e
b
a
t
e
d
i
n
c
o
u
r
t
i
s
c
o
n
t
r
a
r
y
t
o
p
r
i
n
c
i
p
l
e
s
o
f
o
p
e
n
j
u
s
t
i
c
e
.
W
e
h
a
v
e
q
u
e
s
t
i
o
n
e
d
t
h
e
v
a
l
i
d
i
t
y
o
f
t
h
e
c
r
i
t
i
c
i
s
m
o
f
t
h
e
B
a
y
e
s
i
a
n
a
p
p
r
o
a
c
h
i
n
t
h
e
1
9
9
0
s
c
a
s
e
s
.
W
h
a
t
e
v
e
r
g
o
o
d
r
e
a
s
o
n
s
t
h
e
r
e
m
i
g
h
t
b
e
f
o
r
n
o
t
e
x
p
r
e
s
s
l
y
u
r
g
i
n
g
t
h
e
j
u
r
y
t
o
a
p
p
l
y
B
a
y
e
s
T
h
e
o
r
e
m
i
n
t
h
e
i
r
d
e
l
i
b
e
r
a
t
i
o
n
s
,
t
h
e
y
d
o
n
o
t
a
p
p
l
y
t
o
t
h
e
p
r
o
-
c
e
s
s
b
y
w
h
i
c
h
a
s
c
i
e
n
t
i
s
t
a
r
r
i
v
e
s
a
t
a
c
o
n
c
l
u
s
i
o
n
.
A
t
p
a
r
a
g
r
a
p
h
[
9
0
]
t
h
e
C
o
u
r
t
s
a
i
d
t
h
a
t
o
u
t
s
i
d
e
t
h
e
e
l
d
o
f
D
N
A
(
a
n
d
p
o
s
s
i
b
l
y
o
t
h
e
r
a
r
e
a
s
w
h
e
r
e
t
h
e
r
e
i
s
a
r
m
s
t
a
t
i
s
t
i
c
a
l
b
a
s
e
)
.
.
.
B
a
y
e
s
t
h
e
o
r
e
m
a
n
d
l
i
k
e
l
i
h
o
o
d
r
a
t
i
o
s
s
h
o
u
l
d
n
o
t
b
e
u
s
e
d
.
H
e
r
e
,
h
o
w
e
v
e
r
,
t
h
e
C
o
u
r
t
s
t
a
t
e
s
t
h
a
t
i
t
w
a
s
t
h
e
u
s
e
o
f
l
i
k
e
l
i
h
o
o
d
r
a
t
i
o
s
w
i
t
h
o
u
t
t
h
a
t
a
p
p
r
o
a
c
h
b
e
i
n
g
r
e
v
e
a
l
e
d
i
n
o
p
e
n
c
o
u
r
t
t
h
a
t
w
a
s
c
o
n
t
r
a
r
y
t
o
t
h
e
p
r
i
n
c
i
p
l
e
s
o
f
o
p
e
n
j
u
s
t
i
c
e
.
I
f
t
h
a
t
w
a
s
t
h
e
b
a
s
i
s
o
n
w
h
i
c
h
t
h
e
c
o
n
-
v
i
c
t
i
o
n
w
a
s
q
u
a
s
h
e
d
,
t
h
e
n
t
h
e
p
r
o
h
i
b
i
t
i
o
n
o
n
B
a
y
e
s
i
a
n
r
e
a
s
o
n
i
n
g
a
t
[
9
0
]
i
s
o
b
i
t
e
r
,
h
o
w
e
v
e
r
r
m
l
y
i
t
m
a
y
h
a
v
e
b
e
e
n
e
x
p
r
e
s
s
e
d
.
i
v
)
T
h
e
p
r
a
c
t
i
c
e
o
f
u
s
i
n
g
l
i
k
e
l
i
h
o
o
d
r
a
t
i
o
s
w
a
s
j
u
s
t
i
e
d
a
s
p
r
o
d
u
c
i
n
g
b
a
l
a
n
c
e
,
l
o
g
i
c
,
r
o
b
u
s
t
n
e
s
s
a
n
d
t
r
a
n
s
p
a
r
e
n
c
y
,
a
s
w
e
h
a
v
e
s
e
t
o
u
t
a
t
p
a
r
a
g
r
a
p
h
5
4
.
I
n
o
u
r
v
i
e
w
,
t
h
e
i
r
u
s
e
i
n
t
h
i
s
c
a
s
e
w
a
s
p
l
a
i
n
l
y
n
o
t
t
r
a
n
s
p
a
r
e
n
t
.
A
l
t
h
o
u
g
h
i
t
w
a
s
M
r
R
y
d
e
r
s
e
v
i
d
e
n
c
e
(
w
h
i
c
h
w
e
a
c
c
e
p
t
)
,
t
h
a
t
h
e
a
r
r
i
v
e
d
a
t
h
i
s
o
p
i
n
i
o
n
t
h
r
o
u
g
h
e
x
p
e
r
i
e
n
c
e
,
i
t
w
o
u
l
d
b
e
d
i
c
u
l
t
t
o
s
e
e
h
o
w
a
n
o
p
i
n
i
o
n
o
f
f
o
o
t
w
e
a
r
m
a
r
k
s
a
r
r
i
v
e
d
a
t
t
h
r
o
u
g
h
t
h
e
a
p
p
l
i
c
a
t
i
o
n
o
f
a
f
o
r
m
u
l
a
c
o
u
l
d
b
e
d
e
s
c
r
i
b
e
d
a
s
l
o
g
i
c
a
l
,
o
r
b
a
l
a
n
c
e
d
o
r
r
o
b
u
s
t
,
w
h
e
n
t
h
e
d
a
t
a
a
r
e
a
s
u
n
c
e
r
t
a
i
n
a
s
w
e
h
a
v
e
s
e
t
o
u
t
a
n
d
c
o
u
l
d
p
r
o
d
u
c
e
s
u
c
h
d
i
e
r
e
n
t
r
e
s
u
l
t
s
.
I
t
i
s
s
u
r
e
l
y
a
f
e
a
t
u
r
e
o
f
a
l
o
g
i
c
a
l
,
b
a
l
a
n
c
e
d
o
r
r
o
b
u
s
t
a
p
p
r
o
a
c
h
t
h
a
t
i
f
o
n
e
a
p
p
l
i
e
s
i
t
t
o
d
i
e
r
e
n
t
g
u
r
e
s
i
t
w
i
l
l
p
r
o
d
u
c
e
d
i
e
r
e
n
t
r
e
s
u
l
t
s
.
W
h
i
l
e
i
n
t
h
i
s
c
a
s
e
t
h
e
p
r
o
c
e
s
s
m
a
y
n
o
t
h
a
v
e
b
e
e
n
m
a
d
e
t
r
a
n
s
p
a
r
e
n
t
,
t
r
a
n
s
p
a
r
e
n
c
y
i
s
i
n
d
e
e
d
o
n
e
o
f
t
h
e
a
d
v
a
n
-
t
a
g
e
s
o
f
t
h
e
u
s
e
o
f
l
i
k
e
l
i
h
o
o
d
r
a
t
i
o
s
a
s
i
t
e
x
p
o
s
e
s
a
l
l
t
h
e
f
a
c
t
o
r
s
c
o
n
s
i
d
e
r
e
d
,
u
n
l
i
k
e
a
r
e
f
e
r
e
n
c
e
t
o
j
u
d
g
m
e
n
t
o
r
e
x
p
e
r
i
e
n
c
e
.
D
I
S
C
U
S
S
I
O
N
T
h
i
s
e
v
i
d
e
n
c
e
c
o
u
l
d
h
a
v
e
b
e
e
n
b
e
t
t
e
r
g
i
v
e
n
.
T
h
e
a
l
t
e
r
n
a
t
i
v
e
h
y
p
o
t
h
e
s
i
s
s
h
o
u
l
d
h
a
v
e
b
e
e
n
p
o
s
i
t
i
v
e
l
y
e
x
p
r
e
s
s
e
d
,
e
i
t
h
e
r
t
h
a
t
s
o
m
e
u
n
k
n
o
w
n
s
h
o
e
h
a
d
m
a
d
e
t
h
e
m
a
r
k
(
i
n
w
h
i
c
h
c
a
s
e
t
h
e
a
p
p
r
o
p
r
i
a
t
e
d
a
t
a
b
a
s
e
w
o
u
l
d
h
a
v
e
b
e
e
n
t
h
e
m
a
n
u
f
a
c
t
u
r
e
r
s
d
a
t
a
)
o
r
t
h
a
t
t
h
e
s
h
o
e
o
f
s
o
m
e
u
n
k
n
o
w
n
b
u
r
g
l
a
r
m
a
d
e
t
h
e
m
a
r
k
,
i
n
w
h
i
c
h
c
a
s
e
t
h
e
c
r
i
m
e
-
r
e
l
a
t
e
d
F
o
r
e
n
s
i
c
S
c
i
e
n
c
e
S
e
r
v
i
c
e
d
a
t
a
b
a
s
e
w
o
u
l
d
b
e
a
p
p
r
o
p
r
i
a
t
e
.
W
e
w
o
u
l
d
B
e
r
n
a
r
d
R
o
b
e
r
t
s
o
n
,
G
.
A
.
V
i
g
n
a
u
x
a
n
d
C
h
a
r
l
e
s
E
.
H
.
B
e
r
g
e
r
4
5
3
r
2
0
1
1
T
h
e
A
u
t
h
o
r
s
.
T
h
e
M
o
d
e
r
n
L
a
w
R
e
v
i
e
w
r
2
0
1
1
T
h
e
M
o
d
e
r
n
L
a
w
R
e
v
i
e
w
L
i
m
i
t
e
d
.
(
2
0
1
1
)
7
4
(
3
)
4
3
0
^
4
5
5
a
g
r
e
e
t
h
a
t
t
h
e
e
x
p
e
r
t
s
h
o
u
l
d
m
a
k
e
t
h
e
r
e
a
s
o
n
i
n
g
c
l
e
a
r
i
n
t
h
e
r
e
p
o
r
t
s
,
b
u
t
t
h
e
r
e
i
s
n
o
r
e
a
s
o
n
w
h
y
,
i
n
e
x
a
m
i
n
a
t
i
o
n
-
i
n
-
c
h
i
e
f
,
t
h
e
e
x
p
e
r
t
s
h
o
u
l
d
g
o
f
u
r
t
h
e
r
t
h
a
n
e
x
p
r
e
s
-
s
i
n
g
t
h
e
s
t
r
e
n
g
t
h
o
f
t
h
e
e
v
i
d
e
n
c
e
e
i
t
h
e
r
i
n
n
u
m
e
r
i
c
a
l
o
r
v
e
r
b
a
l
t
e
r
m
s
.
I
f
t
h
e
d
e
f
e
n
c
e
c
h
o
o
s
e
s
,
i
t
c
a
n
t
h
e
n
c
r
o
s
s
-
e
x
a
m
i
n
e
o
n
t
h
e
g
u
r
e
s
a
n
d
r
e
a
s
o
n
i
n
g
p
r
o
c
e
s
s
r
e
p
o
r
t
e
d
.
I
t
i
s
u
n
c
l
e
a
r
h
o
w
e
v
e
r
w
h
y
t
h
e
d
e
f
e
n
c
e
s
h
o
u
l
d
c
h
o
o
s
e
t
o
s
h
o
o
t
i
t
s
e
l
f
i
n
t
h
e
f
o
o
t
b
y
c
r
o
s
s
-
e
x
a
m
i
n
i
n
g
o
n
t
h
e
p
r
o
p
o
s
i
t
i
o
n
t
h
a
t
g
u
r
e
s
l
e
s
s
f
a
v
o
u
r
a
b
l
e
t
o
t
h
e
a
c
c
u
s
e
d
c
o
u
l
d
h
a
v
e
b
e
e
n
u
s
e
d
.
A
t
l
e
a
s
t
t
w
o
p
o
i
n
t
s
c
a
n
b
e
d
e
r
i
v
e
d
f
r
o
m
t
h
e
j
u
d
g
m
e
n
t
:
(
i
)
e
v
i
d
e
n
c
e
c
a
n
b
e
g
i
v
e
n
e
v
a
l
u
a
t
i
n
g
t
h
e
s
i
g
n
i
c
a
n
c
e
o
f
f
o
o
t
w
e
a
r
m
a
r
k
e
v
i
d
e
n
c
e
;
b
u
t
(
i
i
)
m
a
t
h
e
m
a
t
i
c
a
l
f
o
r
m
u
l
a
e
a
r
e
n
o
t
t
o
b
e
u
s
e
d
i
n
r
e
a
s
o
n
i
n
g
t
o
a
n
e
v
a
l
u
a
t
i
o
n
o
f
t
h
a
t
s
i
g
n
i
c
a
n
c
e
,
i
t
m
u
s
t
b
e
b
a
s
e
d
o
n
l
y
o
n
j
u
d
g
m
e
n
t
a
n
d
e
x
p
e
r
i
e
n
c
e
.
A
s
t
o
t
h
e
r
s
t
,
i
t
i
s
m
a
d
e
c
l
e
a
r
e
a
r
l
i
e
r
i
n
t
h
e
j
u
d
g
m
e
n
t
t
h
a
t
t
h
e
C
o
u
r
t
f
a
i
l
e
d
t
o
r
e
c
o
g
n
i
s
e
t
h
e
d
i
e
r
e
n
c
e
b
e
t
w
e
e
n
a
s
t
a
t
e
m
e
n
t
t
h
a
t
t
h
e
e
v
i
d
e
n
c
e
s
t
r
o
n
g
l
y
s
u
p
p
o
r
t
s
t
h
e
h
y
p
o
t
h
e
s
i
s
t
h
a
t
t
h
i
s
s
h
o
e
m
a
d
e
t
h
i
s
m
a
r
k
a
n
d
m
y
c
o
n
c
l
u
s
i
o
n
i
s
t
h
a
t
t
h
i
s
s
h
o
e
p
r
o
b
a
b
l
y
m
a
d
e
t
h
i
s
m
a
r
k
.
T
h
e
r
s
t
i
s
a
s
t
a
t
e
m
e
n
t
o
f
t
h
e
s
t
r
e
n
g
t
h
o
f
t
h
e
e
v
i
-
d
e
n
c
e
,
w
h
i
c
h
i
s
w
i
t
h
i
n
t
h
e
p
r
o
v
i
n
c
e
o
f
t
h
e
e
x
p
e
r
t
.
T
h
e
s
e
c
o
n
d
i
s
a
c
o
n
c
l
u
s
i
o
n
w
h
i
c
h
c
a
n
o
n
l
y
v
a
l
i
d
l
y
b
e
m
a
d
e
b
y
a
l
s
o
a
s
s
e
s
s
i
n
g
t
h
e
s
t
r
e
n
g
t
h
o
f
a
l
l
t
h
e
o
t
h
e
r
e
v
i
d
e
n
c
e
i
n
t
h
e
c
a
s
e
,
w
h
i
c
h
i
s
t
h
e
p
r
o
v
i
n
c
e
o
f
t
h
e
j
u
r
y
.
T
h
e
C
o
u
r
t
p
u
r
p
o
r
t
s
t
o
b
e
c
o
n
c
e
r
n
e
d
t
o
k
e
e
p
e
x
p
e
r
t
s
w
i
t
h
i
n
a
p
p
r
o
p
r
i
a
t
e
b
o
u
n
d
s
,
b
u
t
t
h
e
n
s
e
e
m
s
t
o
f
a
i
l
t
o
r
e
a
l
i
s
e
t
h
a
t
i
t
i
s
i
n
v
i
t
i
n
g
e
x
p
e
r
t
s
t
o
d
r
a
w
c
o
n
c
l
u
s
i
o
n
s
t
h
e
y
h
a
v
e
n
o
b
a
s
i
s
f
o
r
d
r
a
w
i
n
g
.
A
s
f
o
r
t
h
e
s
e
c
o
n
d
p
o
i
n
t
,
a
t
[
9
0
]
,
t
h
e
C
o
u
r
t
a
p
p
e
a
r
s
t
o
l
a
y
d
o
w
n
a
p
r
o
h
i
b
i
t
i
o
n
o
n
t
h
e
u
s
e
o
f
l
i
k
e
l
i
h
o
o
d
r
a
t
i
o
s
a
n
d
B
a
y
e
s
T
h
e
o
r
e
m
i
n
a
s
s
e
s
s
i
n
g
e
v
i
d
e
n
c
e
o
u
t
s
i
d
e
D
N
A
(
a
n
d
p
o
s
s
i
b
l
y
o
t
h
e
r
a
r
e
a
s
w
h
e
r
e
t
h
e
r
e
i
s
a
r
m
s
t
a
t
i
s
t
i
c
a
l
b
a
s
e
)
.
O
n
e
o
f
t
h
e
C
o
u
r
t
s
j
u
s
t
i
c
a
t
i
o
n
s
f
o
r
t
h
i
s
i
s
a
l
l
e
g
e
d
d
i
v
i
s
i
o
n
o
f
o
p
i
n
i
o
n
i
n
o
t
h
e
r
j
u
r
i
s
d
i
c
t
i
o
n
s
o
v
e
r
t
h
e
u
s
e
o
f
B
a
y
e
s
i
a
n
t
e
c
h
n
i
q
u
e
s
.
S
i
n
c
e
t
h
e
n
t
h
e
B
o
a
r
d
o
f
t
h
e
E
u
r
o
p
e
a
n
N
e
t
-
w
o
r
k
o
f
F
o
r
e
n
s
i
c
S
c
i
e
n
c
e
I
n
s
t
i
t
u
t
e
s
,
c
o
v
e
r
i
n
g
3
3
c
o
u
n
t
r
i
e
s
,
h
a
s
s
i
g
n
e
d
t
h
e
P
o
s
i
t
i
o
n
S
t
a
t
e
m
e
n
t
a
n
d
e
n
g
a
g
e
s
i
t
s
e
l
f
t
o
w
o
r
k
t
o
w
a
r
d
s
a
f
u
l
l
i
m
p
l
e
m
e
n
t
a
t
i
o
n
[
o
f
t
h
e
B
a
y
e
-
s
i
a
n
a
p
p
r
o
a
c
h
]
w
i
t
h
i
n
t
h
e
E
N
F
S
I
l
a
b
o
r
a
t
o
r
i
e
s
.
1
7
T
h
e
C
o
u
r
t
d
o
e
s
n
o
t
e
x
p
l
a
i
n
w
h
y
t
h
e
l
o
g
i
c
a
l
s
t
r
u
c
t
u
r
e
o
f
a
n
a
r
g
u
m
e
n
t
s
h
o
u
l
d
d
i
e
r
a
c
c
o
r
d
i
n
g
t
o
h
o
w
q
u
a
n
t
i
a
b
l
e
t
h
e
e
v
i
d
e
n
c
e
i
s
a
n
d
t
h
e
r
e
i
s
n
o
r
e
a
s
o
n
w
h
y
i
t
s
h
o
u
l
d
.
T
h
e
C
o
u
r
t
a
l
s
o
a
s
s
u
m
e
s
t
h
a
t
t
h
e
r
e
i
s
a
c
l
e
a
r
d
i
s
t
i
n
c
t
i
o
n
b
e
t
w
e
e
n
c
a
s
e
s
w
h
e
r
e
a
r
m
s
t
a
t
i
s
t
i
c
a
l
b
a
s
e
i
s
a
v
a
i
l
a
b
l
e
a
n
d
w
h
e
r
e
i
t
i
s
n
o
t
.
A
l
l
p
r
o
b
a
b
i
l
i
t
y
a
s
s
e
s
s
-
m
e
n
t
s
i
n
v
o
l
v
e
u
n
c
e
r
t
a
i
n
a
n
d
i
n
c
o
m
p
l
e
t
e
d
a
t
a
;
c
o
n
v
e
r
s
e
l
y
t
h
e
r
e
a
r
e
s
e
v
e
r
a
l
a
r
e
a
s
,
s
u
c
h
a
s
g
l
a
s
s
a
n
d
b
r
e
s
,
w
h
e
r
e
t
h
e
s
t
a
t
i
s
t
i
c
a
l
d
a
t
a
a
v
a
i
l
a
b
l
e
i
s
b
e
t
t
e
r
t
h
a
n
f
o
r
f
o
o
t
-
w
e
a
r
m
a
r
k
s
b
u
t
n
o
t
s
o
c
o
m
p
l
e
t
e
a
s
f
o
r
D
N
A
.
T
h
e
j
u
d
g
m
e
n
t
f
a
i
l
s
t
o
e
x
p
l
a
i
n
h
o
w
r
e
a
s
o
n
e
d
j
u
d
g
m
e
n
t
s
a
r
e
t
o
b
e
a
r
r
i
v
e
d
a
t
w
h
e
n
s
e
v
e
r
a
l
f
a
c
t
o
r
s
a
r
e
i
n
v
o
l
v
e
d
,
o
t
h
e
r
t
h
a
n
b
y
a
p
p
e
a
l
s
t
o
e
x
p
e
r
i
e
n
c
e
a
n
d
j
u
d
g
-
m
e
n
t
^
b
l
a
c
k
b
o
x
e
s
t
h
e
C
o
u
r
t
w
o
u
l
d
a
p
p
a
r
e
n
t
l
y
p
r
e
f
e
r
t
o
r
e
m
a
i
n
u
n
a
n
a
l
y
s
e
d
.
W
e
i
r
d
l
y
,
t
h
e
C
o
u
r
t
t
h
e
n
p
r
o
c
e
e
d
s
t
o
b
e
p
r
e
s
c
r
i
p
t
i
v
e
a
b
o
u
t
h
o
w
e
x
p
e
r
t
s
a
r
e
n
o
t
t
o
a
r
r
i
v
e
a
t
s
u
c
h
j
u
d
g
m
e
n
t
s
.
T
h
e
C
o
u
r
t
i
s
n
o
t
e
v
e
n
r
e
f
e
r
r
i
n
g
t
o
h
o
w
e
v
i
d
e
n
c
e
i
s
t
o
b
e
g
i
v
e
n
i
n
c
o
u
r
t
o
r
t
o
h
o
w
j
u
r
i
e
s
a
r
e
t
o
b
e
d
i
r
e
c
t
e
d
b
u
t
t
o
t
h
e
r
e
a
s
o
n
i
n
g
p
r
o
c
e
s
s
t
o
b
e
f
o
l
l
o
w
e
d
b
y
t
h
e
e
x
p
e
r
t
i
n
a
r
r
i
v
i
n
g
a
t
a
c
o
n
c
l
u
s
i
o
n
.
I
t
s
e
e
m
s
s
t
r
a
n
g
e
t
h
a
t
a
c
o
u
r
t
1
7
P
o
s
i
t
i
o
n
S
t
a
t
e
m
e
n
t
,
n
1
a
b
o
v
e
.
E
x
t
e
n
d
i
n
g
t
h
e
C
o
n
f
u
s
i
o
n
A
b
o
u
t
B
a
y
e
s
4
5
4
r
2
0
1
1
T
h
e
A
u
t
h
o
r
s
.
T
h
e
M
o
d
e
r
n
L
a
w
R
e
v
i
e
w
r
2
0
1
1
T
h
e
M
o
d
e
r
n
L
a
w
R
e
v
i
e
w
L
i
m
i
t
e
d
.
(
2
0
1
1
)
7
4
(
3
)
4
3
0
^
4
5
5
s
h
o
u
l
d
d
i
c
t
a
t
e
t
o
s
c
i
e
n
t
i
s
t
s
h
o
w
t
h
e
y
c
a
r
r
y
o
u
t
t
h
e
i
r
s
c
i
e
n
c
e
a
n
d
b
y
w
h
a
t
r
e
a
s
o
n
i
n
g
p
r
o
c
e
s
s
t
h
e
y
a
r
e
t
o
a
r
r
i
v
e
a
t
t
h
e
i
r
c
o
n
c
l
u
s
i
o
n
s
.
T
h
e
d
a
n
g
e
r
o
f
d
o
i
n
g
s
o
i
s
i
l
l
u
s
t
r
a
t
e
d
b
y
t
h
i
s
v
e
r
y
j
u
d
g
m
e
n
t
i
n
w
h
i
c
h
t
h
e
C
o
u
r
t
h
a
s
p
r
o
c
e
e
d
e
d
f
r
o
m
s
e
v
e
r
a
l
d
e
m
o
n
s
t
r
a
-
b
l
y
f
a
l
s
e
p
r
e
m
i
s
e
s
.
I
n
d
e
e
d
,
i
t
i
s
q
u
i
t
e
p
l
a
u
s
i
b
l
e
t
h
a
t
t
h
e
r
e
a
s
o
n
t
h
a
t
t
h
e
e
x
p
e
r
t
r
e
f
r
a
i
n
e
d
f
r
o
m
e
x
p
o
s
i
n
g
h
i
s
r
e
a
s
o
n
i
n
g
w
a
s
t
h
e
p
r
i
o
r
d
e
c
i
s
i
o
n
s
o
f
t
h
e
C
o
u
r
t
o
f
A
p
p
e
a
l
o
n
t
h
e
B
a
y
e
s
i
a
n
a
p
p
r
o
a
c
h
.
I
f
t
h
e
s
e
c
a
s
e
s
a
r
e
t
o
b
e
c
i
t
e
d
a
s
a
u
t
h
o
r
i
t
y
t
h
e
n
t
h
e
d
e
t
a
i
l
e
d
c
r
i
t
i
c
i
s
m
s
o
f
t
h
e
m
r
e
f
e
r
r
e
d
t
o
a
b
o
v
e
m
u
s
t
b
e
d
i
s
c
u
s
s
e
d
.
C
O
N
C
L
U
S
I
O
N
T
h
e
j
u
d
g
m
e
n
t
m
a
k
e
s
t
w
o
f
u
n
d
a
m
e
n
t
a
l
e
r
r
o
r
s
.
T
h
e
r
s
t
i
s
c
o
n
s
i
s
t
e
n
t
t
r
a
n
s
p
o
s
i
t
i
o
n
o
f
t
h
e
c
o
n
d
i
t
i
o
n
a
l
w
h
i
c
h
i
n
d
i
c
a
t
e
s
t
h
a
t
t
h
e
C
o
u
r
t
h
a
s
n
o
t
u
n
d
e
r
s
t
o
o
d
t
h
e
d
i
e
r
-
e
n
c
e
b
e
t
w
e
e
n
a
s
s
e
s
s
m
e
n
t
s
o
f
t
h
e
p
r
o
b
a
b
i
l
i
t
y
o
f
a
p
r
o
p
o
s
i
t
i
o
n
a
n
d
o
f
t
h
e
s
t
r
e
n
g
t
h
o
f
t
h
e
e
v
i
d
e
n
c
e
f
o
r
t
h
e
p
r
o
p
o
s
i
t
i
o
n
;
t
h
e
s
e
c
o
n
d
i
s
c
o
n
f
u
s
i
o
n
b
e
t
w
e
e
n
u
n
c
e
r
t
a
i
n
t
y
i
n
t
h
e
v
a
l
u
e
s
o
f
t
h
e
v
a
r
i
a
b
l
e
s
a
n
d
u
n
c
e
r
t
a
i
n
t
y
i
n
t
h
e
i
r
r
e
l
a
t
i
o
n
s
h
i
p
i
n
a
m
a
t
h
e
m
a
-
t
i
c
a
l
f
o
r
m
u
l
a
.
T
h
e
f
a
c
t
t
h
a
t
v
a
r
i
a
b
l
e
s
c
a
n
n
o
t
b
e
p
r
e
c
i
s
e
l
y
e
x
p
r
e
s
s
e
d
d
o
e
s
n
o
t
a
e
c
t
t
h
e
v
a
l
i
d
i
t
y
o
f
t
h
e
r
e
l
a
t
i
o
n
s
h
i
p
s
d
e
s
c
r
i
b
e
d
b
y
t
h
e
f
o
r
m
u
l
a
.
W
h
e
r
e
d
o
e
s
t
h
i
s
l
e
a
v
e
t
r
i
a
l
j
u
d
g
e
s
?
T
h
e
N
e
w
S
o
u
t
h
W
a
l
e
s
C
o
u
r
t
o
f
A
p
p
e
a
l
i
n
R
v
S
h
e
p
h
e
r
d
(
1
9
8
8
)
3
9
A
C
r
i
m
R
2
6
6
f
a
c
e
d
a
d
i
l
e
m
m
a
i
n
t
h
a
t
i
t
w
a
s
s
u
p
p
o
s
e
d
l
y
b
o
u
n
d
b
y
a
d
e
c
i
s
i
o
n
o
f
t
h
e
H
i
g
h
C
o
u
r
t
o
f
A
u
s
t
r
a
l
i
a
i
n
R
v
C
h
a
m
b
e
r
l
a
i
n
(
N
o
2
)
(
1
9
8
4
)
1
5
3
C
L
R
5
2
1
w
h
i
c
h
l
a
i
d
d
o
w
n
s
t
r
a
n
g
e
r
u
l
e
s
o
f
r
e
a
s
o
n
i
n
g
.
R
o
d
e
n
J
r
e
s
p
o
n
d
e
d
(
a
t
2
6
9
^
2
7
0
)
:
I
f
t
h
e
p
r
o
p
o
s
i
t
i
o
n
s
v
a
l
i
d
i
t
y
d
e
p
e
n
d
s
u
p
o
n
l
o
g
i
c
,
a
n
d
t
h
e
r
e
s
o
l
u
t
i
o
n
o
f
a
p
h
i
l
o
s
o
p
h
i
-
c
a
l
d
e
b
a
t
e
,
i
t
s
e
e
m
s
t
o
m
e
t
h
a
t
i
t
c
a
n
n
o
t
p
r
o
p
e
r
l
y
b
e
c
h
a
r
a
c
t
e
r
i
s
e
d
a
s
a
r
u
l
e
o
f
l
a
w
.
I
t
s
v
a
l
i
d
i
t
y
c
e
r
t
a
i
n
l
y
d
o
e
s
n
o
t
s
t
e
m
f
r
o
m
a
n
y
s
t
a
t
u
t
o
r
y
p
r
o
v
i
s
i
o
n
;
n
o
r
,
i
n
m
y
v
i
e
w
,
c
a
n
i
t
b
e
a
r
g
u
e
d
t
h
a
t
i
t
s
a
s
s
e
r
t
i
o
n
a
s
a
p
r
o
p
o
s
i
t
i
o
n
o
f
l
o
g
i
c
i
n
o
t
h
e
r
w
i
s
e
b
i
n
d
i
n
g
j
u
d
g
-
m
e
n
t
s
g
i
v
e
s
i
t
t
h
e
f
o
r
c
e
o
f
l
a
w
.
I
f
i
t
w
e
r
e
s
h
o
w
n
t
o
b
e
n
o
t
l
o
g
i
c
a
l
l
y
v
a
l
i
d
,
t
h
e
n
i
t
c
o
u
l
d
n
o
t
a
p
p
l
y
.
T
h
a
t
w
o
u
l
d
n
o
t
b
e
t
h
e
p
o
s
i
t
i
o
n
w
i
t
h
a
r
u
l
e
o
f
l
a
w
.
[
C
]
o
n
c
l
u
s
i
o
n
s
i
n
r
e
l
a
t
i
o
n
t
o
t
h
e
m
a
n
n
e
r
i
n
w
h
i
c
h
e
v
a
l
u
a
t
i
v
e
e
v
i
d
e
n
c
e
s
h
o
u
l
d
b
e
a
p
p
r
o
a
c
h
e
d
b
y
f
o
o
t
w
e
a
r
e
x
a
m
i
n
e
r
s
(
R
v
T
a
t
[
1
6
]
)
c
a
n
n
o
t
p
o
s
s
i
b
l
y
b
e
s
a
i
d
t
o
b
e
r
u
l
e
s
o
f
l
a
w
.
T
h
e
y
a
r
e
i
n
s
t
r
u
c
t
i
o
n
s
t
o
s
c
i
e
n
t
i
s
t
s
a
s
t
o
h
o
w
t
o
t
h
i
n
k
a
b
o
u
t
s
c
i
e
n
t
i
c
m
a
t
t
e
r
s
.
S
u
c
h
i
n
s
t
r
u
c
t
i
o
n
s
n
e
e
d
t
o
c
o
n
f
o
r
m
w
i
t
h
t
h
e
r
e
q
u
i
r
e
m
e
n
t
s
o
f
l
o
g
i
c
a
n
d
t
h
e
C
o
u
r
t
s
c
o
n
c
l
u
s
i
o
n
s
i
n
R
v
T
a
r
e
n
o
t
l
o
g
i
c
a
l
l
y
v
a
l
i
d
.
T
h
i
s
j
u
d
g
m
e
n
t
,
i
f
a
l
l
o
w
e
d
v
a
l
u
e
a
s
a
p
r
e
c
e
d
e
n
t
,
h
a
s
t
h
e
p
o
t
e
n
t
i
a
l
t
o
h
a
l
t
p
r
o
g
r
e
s
s
i
n
f
o
r
e
n
s
i
c
s
c
i
e
n
c
e
a
n
d
t
o
r
e
d
u
c
e
f
o
r
e
n
s
i
c
s
c
i
e
n
t
i
c
e
v
i
d
e
n
c
e
t
o
u
n
a
n
a
l
y
s
a
b
l
e
i
m
p
r
e
s
s
i
o
n
i
s
t
i
c
c
o
n
c
l
u
s
o
r
y
s
t
a
t
e
m
e
n
t
s
b
y
e
x
p
e
r
t
s
.
B
e
r
n
a
r
d
R
o
b
e
r
t
s
o
n
,
G
.
A
.
V
i
g
n
a
u
x
a
n
d
C
h
a
r
l
e
s
E
.
H
.
B
e
r
g
e
r
4
5
5
r
2
0
1
1
T
h
e
A
u
t
h
o
r
s
.
T
h
e
M
o
d
e
r
n
L
a
w
R
e
v
i
e
w
r
2
0
1
1
T
h
e
M
o
d
e
r
n
L
a
w
R
e
v
i
e
w
L
i
m
i
t
e
d
.
(
2
0
1
1
)
7
4
(
3
)
4
3
0
^
4
5
5
J
o
u
r
n
a
l

o
f

F
o
r
e
n
s
i
c

d
e
n
t
i
f
i
c
a
t
i
o
n
6
0
0

/
5
6
(
4
)
,
2
0
0
6
A
r
t
i
c
l
e
W
h
y

E
x
p
e
r
t
s

M
a
k
e

E
r
r
o
r
s
I
t
i
e
l

E
.

D
r
o
r

D
a
v
i
d

C
h
a
r
l
t
o
n
S
c
h
o
o
l

o
f

P
s
y
c
h
o
l
o
g
y

U
n
i
v
e
r
s
i
t
y

o
f

S
o
u
t
h
a
m
p
t
o
n

S
o
u
t
h
a
m
p
t
o
n
,

U
n
i
t
e
d

K
i
n
g
d
o
m
A
b
s
t
r
a
c
t
:

E
x
p
e
r
t

l
a
t
e
n
t

I
i
n
g
e
r
p
r
i
n
t

e
x
a
m
i
n
e
r
s

w
e
r
e

p
r
e
s
e
n
t
e
d

w
i
t
h

I
i
n
g
e
r
p
r
i
n
t
s

t
a
k
e
n

I
r
o
m

r
e
a
l

c
r
i
m
i
n
a
l

c
a
s
e
s
.

H
a
l
I

o
I

t
h
e

p
r
i
n
t
s

h
a
d

b
e
e
n

p
r
e
v
i
o
u
s
l
y

j
u
d
g
e
d

a
s

i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n
s

a
n
d

t
h
e

o
t
h
e
r

h
a
l
I

a
s

e
x
c
l
u
-
s
i
o
n
s
.

W
e

r
e
-
p
r
e
s
e
n
t
e
d

t
h
e

s
a
m
e

p
r
i
n
t
s

t
o

t
h
e

s
a
m
e

e
x
p
e
r
t
s

w
h
o

h
a
d

j
u
d
g
e
d

t
h
e
m

p
r
e
v
i
o
u
s
l
y
,

b
u
t

p
r
o
v
i
d
e
d

b
i
a
s
i
n
g

c
o
n
t
e
x
t
u
a
l

i
n
I
o
r
m
a
t
i
o
n

i
n

b
o
t
h

t
h
e

i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n
s

a
n
d

e
x
c
l
u
s
i
o
n
s
.

A

c
o
n
t
r
o
l

s
e
t

o
I

i
n
d
i
v
i
d
u
-
a
l
i
z
a
t
i
o
n
s

a
n
d

e
x
c
l
u
s
i
o
n
s

w
a
s

a
l
s
o

r
e
-
p
r
e
s
e
n
t
e
d

a
s

p
a
r
t

o
I

t
h
e

s
t
u
d
y
.

T
h
e

c
o
n
t
r
o
l

s
e
t

h
a
d

n
o

b
i
a
s
i
n
g

c
o
n
t
e
x
t
u
a
l

i
n
I
o
r
m
a
t
i
o
n

a
s
s
o
c
i
a
t
e
d

w
i
t
h

i
t
.

E
a
c
h

e
x
p
e
r
t

e
x
a
m
i
n
e
d

a

t
o
t
a
l

o
I

e
i
g
h
t

p
a
s
t

d
e
c
i
s
i
o
n
s
.

T
w
o
-
t
h
i
r
d
s

o
I

t
h
e

e
x
p
e
r
t
s

m
a
d
e

i
n
c
o
n
s
i
s
t
e
n
t

d
e
c
i
s
i
o
n
s
.

T
h
e

I
i
n
d
i
n
g
s

a
r
e

d
i
s
c
u
s
s
e
d

i
n

t
e
r
m
s

o
I

p
s
y
c
h
o
l
o
g
i
c
a
l

a
n
d

c
o
g
n
i
t
i
v
e

v
u
l
n
e
r
a
b
i
l
i
t
i
e
s
.
B
a
c
k
g
r
o
u
n
d
E
x
p
e
r
t

p
e
r
I
o
r
m
a
n
c
e

a
n
d

a
c
c
u
r
a
c
y

i
s

a
n

i
m
p
o
r
t
a
n
t

i
s
s
u
e

i
n

a
l
m
o
s
t

a
l
l

s
p
e
c
i
a
l
i
z
e
d

d
o
m
a
i
n
s
.

I
n

c
o
n
t
r
a
s
t

t
o

n
o
v
i
c
e
s
,

e
x
p
e
r
t
s

p
o
s
s
e
s
s

a
b
i
l
i
t
i
e
s

a
n
d

s
k
i
l
l
s

t
h
a
t

e
n
a
b
l
e

t
h
e
m

t
o

p
e
r
I
o
r
m

c
e
r
t
a
i
n

t
a
s
k
s
,

s
u
c
h

a
s

m
e
d
i
c
a
l

p
r
o
c
e
d
u
r
e
s

o
r

I
l
y
i
n
g

a
i
r
c
r
a
I
t

|
1
|
.

A
n

e
x
p
e
r
t

n
e
e
d
s

n
o
t

o
n
l
y

k
n
o
w
l
e
d
g
e
,

b
u
t

s
k
i
l
l
,

j
u
d
g
e
m
e
n
t
,

a
n
d

e
x
p
e
r
i
e
n
c
e

t
o

e
v
a
l
u
a
t
e

a
n
d

i
n
t
e
r
p
r
e
t

i
n
I
o
r
m
a
t
i
o
n

c
o
r
r
e
c
t
l
y

t
o

m
a
k
e

c
o
r
r
e
c
t

d
e
c
i
s
i
o
n
s
.

H
o
w
e
v
e
r
,

b
e
i
n
g

a
n

e
x
p
e
r
t

d
o
e
s

n
o
t

n
e
c
e
s
s
a
r
i
l
y

m
e
a
n

e
r
r
o
r
-
I
r
e
e

p
e
r
I
o
r
m
a
n
c
e
;

i
n

I
a
c
t
,

a
l
m
o
s
t

e
v
e
r
y

s
p
e
c
i
a
l
i
s
t

d
o
m
a
i
n

i
s

s
u
b
j
e
c
t

t
o

e
r
r
o
r
.

T
h
e

p
i
v
o
t
a
l

q
u
e
s
t
i
o
n

i
s

t
h
e

s
o
u
r
c
e

o
I

t
h
e

e
r
r
o
r
.

E
r
r
o
r
s
,

b
r
o
a
d
l
y

s
p
e
a
k
i
n
g
,

I
a
l
l

i
n
t
o

t
h
r
e
e

c
a
t
e
g
o
r
i
e
s
.
R
e
c
e
i
v
e
d

D
e
c
e
m
b
e
r

9
,

2
0
0
5
;

a
c
c
e
p
t
e
d

F
e
b
r
u
a
r
y

6
,

2
0
0
6
J
o
u
r
n
a
l

o
f

F
o
r
e
n
s
i
c

d
e
n
t
i
f
i
c
a
t
i
o
n
5
6

(
4
)
,

2
0
0
6

\

6
0
1
E
r
r
o
r
s
T
h
e

I
i
r
s
t

c
a
t
e
g
o
r
y

r
e
l
a
t
e
s

t
o

h
u
m
a
n

e
r
r
o
r
.

H
u
m
a
n

e
r
r
o
r
s

c
a
n

b
e

i
n
t
e
n
t
i
o
n
a
l

e
r
r
o
r
s

(
w
h
e
r
e
b
y

e
x
p
e
r
t
s

a
r
e

i
n
v
o
l
v
e
d

i
n

I
r
a
u
d
u
l
e
n
t

b
e
h
a
v
i
o
r
)
,

n
e
g
l
i
g
e
n
t

e
r
r
o
r
s

(
w
h
e
r
e
b
y

e
x
p
e
r
t
s

d
o

n
o
t

p
a
y

a
t
t
e
n
t
i
o
n
,

d
o

n
o
t

I
o
l
l
o
w

p
r
o
c
e
d
u
r
e
s
,

e
t
c
.
)
,

a
n
d

c
o
m
p
e
t
e
n
c
y

e
r
r
o
r
s

(
w
h
e
r
e
b
y

e
x
p
e
r
t
s

a
r
e

u
n
a
b
l
e

t
o

m
a
k
e

c
o
r
r
e
c
t

j
u
d
g
e
m
e
n
t
s

b
e
c
a
u
s
e

o
I

a

l
a
c
k

o
I

a
p
p
r
o
p
r
i
a
t
e

s
k
i
l
l

s
e
t
s
;

t
h
i
s

c
a
n

b
e

d
u
e

t
o

d
e
c
l
i
n
i
n
g

e
y
e
s
i
g
h
t
,

I
a
u
l
t
y

i
n
i
t
i
a
l

s
e
l
e
c
t
i
o
n

t
h
a
t

r
e
s
u
l
t
s

i
n

r
e
c
r
u
i
t
i
n
g

p
e
o
p
l
e

w
h
o

d
o

n
o
t

p
o
s
s
e
s
s

t
h
e

p
r
o
p
e
r

c
o
g
n
i
t
i
v
e

a
b
i
l
i
t
i
e
s

t
h
a
t

a
r
e

n
e
e
d
e
d

t
o

u
n
d
e
r
p
i
n

t
h
e

e
x
p
e
r
t
i
s
e
,

i
n
a
d
e
q
u
a
t
e

t
r
a
i
n
i
n
g
,

a
n
d

s
o

I
o
r
t
h
)
.
T
h
e

s
e
c
o
n
d

c
a
t
e
g
o
r
y

r
e
l
a
t
e
s

t
o

i
n
s
t
r
u
m
e
n
t
a
t
i
o
n

a
n
d

t
e
c
h
n
o
l
o
g
i
c
a
l

e
r
r
o
r
s
.

I
n

t
h
i
s

c
a
t
e
g
o
r
y
,

e
r
r
o
r
s

d
e
r
i
v
e

I
r
o
m

I
a
i
l
u
r
e

a
n
d

b
r
e
a
k
d
o
w
n

o
I

i
n
s
t
r
u
m
e
n
t
s

a
n
d

t
e
c
h
n
o
l
o
g
y
.

T
h
e
s
e

t
y
p
e
s

o
I

e
r
r
o
r
s

a
r
e

r
a
r
e

i
n

t
h
e

I
i
n
g
e
r
p
r
i
n
t

d
o
m
a
i
n
.

T
e
c
h
n
i
c
a
l

m
a
l
I
u
n
c
t
i
o
n

c
e
r
t
a
i
n
l
y

a
c
c
o
u
n
t
s

I
o
r

e
r
r
o
r
s

i
n

o
t
h
e
r

d
o
m
a
i
n
s
,

s
u
c
h

a
s

b
r
e
a
k
d
o
w
n

o
I

e
q
u
i
p
m
e
n
t

o
n

a
i
r
c
r
a
I
t
.

T
h
e

I
i
r
s
t

a
n
d

s
e
c
o
n
d

c
a
t
e
g
o
r
i
e
s

b
o
t
h

r
e
l
a
t
e

t
o

c
h
a
n
c
e

m
a
l
I
u
n
c
t
i
o
n
s

a
n
d

b
r
e
a
k
d
o
w
n
,

e
i
t
h
e
r

h
u
m
a
n

o
r

m
a
c
h
i
n
e
.
T
h
e

t
h
i
r
d

c
a
t
e
g
o
r
y

o
I

e
r
r
o
r

r
e
l
a
t
e
s

t
o

m
o
r
e

I
u
n
d
a
m
e
n
t
a
l

m
e
t
h
o
d
o
l
o
g
i
c
a
l

I
a
c
t
o
r
s

t
h
a
t

a
r
e

i
n
h
e
r
e
n
t

t
o

t
h
e

I
i
e
l
d

i
n

q
u
e
s
t
i
o
n
.

T
h
e
s
e

m
a
y

i
n
c
l
u
d
e

e
r
r
o
r
s

a
s
s
o
c
i
a
t
e
d

w
i
t
h

t
h
e

t
e
c
h
n
o
l
o
g
y

i
n

q
u
e
s
t
i
o
n
,

i
n
s
t
r
u
m
e
n
t
a
t
i
o
n
,

a
n
d

m
e
a
s
u
r
e
m
e
n
t
s
.

I
n

t
h
e

I
i
n
g
e
r
p
r
i
n
t

d
o
m
a
i
n
,

I
o
r

e
x
a
m
p
l
e
,

a

I
a
i
l
u
r
e

o
I

t
h
e

A
u
t
o
m
a
t
e
d

F
i
n
g
e
r
p
r
i
n
t

I
d
e
n
t
i
I
i
c
a
t
i
o
n

S
y
s
t
e
m

a
l
g
o
r
i
t
h
m
s

t
o

p
r
o
v
i
d
e

a

m
a
t
c
h
i
n
g

I
i
l
e

I
r
o
m

t
h
e

d
a
t
a
b
a
s
e

(
a
s
s
u
m
i
n
g

s
u
c
h

a

I
i
l
e

i
s

p
r
e
s
e
n
t

a
n
d

e
n
o
u
g
h

i
n
I
o
r
m
a
t
i
o
n

e
x
i
s
t
s

t
o

m
a
k
e

t
h
e

m
a
t
c
h
)

c
a
n

l
e
a
d

t
o

a
n

e
r
r
o
r
.
1

T
h
i
s
,

i
n

c
o
n
t
r
a
s
t

t
o

c
a
t
e
g
o
r
y

t
w
o
,

r
e
I
l
e
c
t
s

n
o
t

a

m
a
l
I
u
n
c
t
i
o
n

o
r

a

b
u
g

i
n

t
h
e

s
o
I
t
w
a
r
e
,

b
u
t

t
h
e

i
n
h
e
r
e
n
t

i
n
a
c
c
u
r
a
c
y

o
I

t
h
e

a
l
g
o
r
i
t
h
m
.

T
e
c
h
n
o
l
o
g
i
e
s

a
n
d

i
n
s
t
r
u
m
e
n
t
a
t
i
o
n
s

h
a
v
e

t
h
e
i
r

l
i
m
i
t
s
,

r
a
n
g
e

o
I

a
c
c
u
r
a
c
y
,

l
e
v
e
l
s

o
I

p
r
e
c
i
s
i
o
n
,

v
a
r
i
a
t
i
o
n
s
,

a
n
d

s
o

I
o
r
t
h
,

w
h
i
c
h

a
r
e

n
o
t

d
u
e

t
o

t
h
e
i
r

b
r
e
a
k
d
o
w
n

a
n
d

m
a
l
I
u
n
c
t
i
o
n

b
u
t

t
o

t
h
e
i
r

v
e
r
y

n
a
t
u
r
e
.
1

A
n

e
r
r
o
r

i
s

s
o
m
e
t
i
m
e
s

d
e
I
i
n
e
d

o
n
l
y

a
s

a
n

i
n
c
o
r
r
e
c
t

i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n

(
i
.
e
.
,

a

I
a
l
s
e

p
o
s
i
t
i
v
e
,

w
h
a
t

i
s

r
e
I
e
r
r
e
d

t
o

i
n

s
i
g
n
a
l

d
e
t
e
c
t
i
o
n

t
h
e
o
r
y

a
s

a

I
a
l
s
e

a
l
a
r
m
)
.

H
o
w
e
v
e
r
,

s
u
c
h

a

d
e
I
i
n
i
t
i
o
n

i
s

l
i
m
i
t
e
d

b
e
c
a
u
s
e

i
t

d
o
e
s

n
o
t

i
n
c
l
u
d
e

c
a
s
e
s

w
h
e
r
e

t
h
e
r
e

i
s

s
u
I
I
i
c
i
e
n
t

i
n
I
o
r
m
a
t
i
o
n

t
o

m
a
k
e

a

p
o
s
i
t
i
v
e

i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n
,

b
u
t
,

b
e
c
a
u
s
e

o
I

t
h
e

e
r
r
o
r
,

n
o

s
u
c
h

i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n

i
s

m
a
d
e

(
i
.
e
.
,

a

I
a
l
s
e

n
e
g
a
t
i
v
e
,

w
h
a
t

i
s

r
e
I
e
r
r
e
d

t
o

i
n

s
i
g
n
a
l

d
e
t
e
c
t
i
o
n

t
h
e
o
r
y

a
s

a

m
i
s
s
)
.

T
h
e
s
e

t
w
o

t
y
p
e
s

o
I

e
r
r
o
r
s

a
r
e

d
i
I
I
e
r
e
n
t
,

b
u
t

n
e
v
e
r
t
h
e
l
e
s
s
,

b
o
t
h

c
a
n

b
e

r
e
g
a
r
d
e
d

a
s

e
r
r
o
r
s
.
J
o
u
r
n
a
l

o
f

F
o
r
e
n
s
i
c

d
e
n
t
i
f
i
c
a
t
i
o
n
6
0
2

/
5
6
(
4
)
,
2
0
0
6
H
o
w
e
v
e
r
,

s
u
c
h

e
r
r
o
r
s

a
r
e

n
o
t

l
i
m
i
t
e
d

t
o

t
e
c
h
n
o
l
o
g
y

o
r

i
n
s
t
r
u
m
e
n
t
a
t
i
o
n
,

e
s
p
e
c
i
a
l
l
y

i
n

a

d
o
m
a
i
n

l
i
k
e

I
i
n
g
e
r
p
r
i
n
t
s

w
h
e
r
e

m
u
c
h

o
I

t
h
e

i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n

p
r
o
c
e
s
s

I
a
l
l
s

o
n

h
u
m
a
n

e
x
p
e
r
t
s

a
n
d

t
h
e
i
r

i
n
t
e
r
a
c
t
i
o
n

w
i
t
h

t
e
c
h
n
o
l
o
g
y

|
2
,

3
|
.

H
e
r
e
,

t
h
e

e
r
r
o
r

i
s

d
u
e

n
o
t

t
o

t
h
e

n
a
t
u
r
e

o
I

t
h
e

t
e
c
h
n
o
l
o
g
y

a
n
d

i
n
s
t
r
u
m
e
n
t
a
t
i
o
n

b
u
t

t
o

t
h
e

n
a
t
u
r
e

a
n
d

m
e
c
h
a
n
i
s
m
s

o
I

t
h
e

h
u
m
a
n

m
i
n
d

a
n
d

c
o
g
n
i
t
i
o
n
.

T
h
i
s

i
s

p
a
r
t
i
c
u
l
a
r
l
y

n
o
t
i
c
e
a
b
l
e

w
h
e
n

d
e
a
l
i
n
g

w
i
t
h

l
a
t
e
n
t

p
r
i
n
t
s

t
h
a
t

a
r
e

c
o
l
l
e
c
t
e
d

I
r
o
m

c
r
i
m
e

s
c
e
n
e
s

a
n
d

a
r
e

t
h
u
s

d
e
g
r
a
d
e
d
,

c
o
n
t
a
m
i
n
a
t
e
d
,

p
a
r
t
i
a
l
l
y

m
i
s
s
i
n
g
,

a
n
d

d
i
s
t
o
r
t
e
d
.
T
h
i
s

t
h
i
r
d

c
a
t
e
g
o
r
y

t
h
u
s

i
n
c
l
u
d
e
s

e
r
r
o
r
s

t
h
a
t

a
r
e

n
o
t

s
i
m
p
l
e

p
r
a
c
t
i
t
i
o
n
e
r

e
r
r
o
r

t
h
a
t

c
a
n

b
e

a
t
t
r
i
b
u
t
e
d

t
o

t
h
e

s
p
e
c
i
I
i
c

e
x
p
e
r
t

i
n
v
o
l
v
e
d

(
a
s

s
p
e
c
i
I
i
e
d

a
n
d

b
e
l
o
n
g
i
n
g

t
o

c
a
t
e
g
o
r
y

o
n
e
)
.

W
h
e
n

p
r
a
c
t
i
t
i
o
n
e
r
s

a
r
e

c
o
m
p
e
t
e
n
t
,

w
e
l
l

t
r
a
i
n
e
d
,

a
n
d

I
o
l
l
o
w
i
n
g

p
r
o
c
e
d
u
r
e
s
,

w
h
e
n

i
n
s
t
r
u
m
e
n
t
a
t
i
o
n

a
n
d

t
e
c
h
n
o
l
o
g
y

o
p
e
r
a
t
e

p
r
o
p
e
r
l
y
,

a
n
d

e
r
r
o
r
s

h
a
p
p
e
n

n
e
v
e
r
t
h
e
l
e
s
s
,

t
h
e
n

t
h
e
s
e

e
r
r
o
r
s

b
e
l
o
n
g

t
o

c
a
t
e
g
o
r
y

t
h
r
e
e
.

T
h
i
s

p
a
p
e
r

e
x
a
m
i
n
e
s

t
h
e

p
o
s
s
i
b
l
e

r
o
l
e

t
h
a
t

p
s
y
c
h
o
l
o
g
i
c
a
l

a
n
d

c
o
g
n
i
t
i
v
e

I
a
c
t
o
r
s

m
a
y

h
a
v
e

i
n

c
a
u
s
i
n
g

t
h
e
s
e

t
y
p
e
s

o
I

e
r
r
o
r
s
.

W
h
e
n

e
x
p
e
r
t

p
r
a
c
t
i
t
i
o
n
e
r
s

p
e
r
f
o
r
m

w
e
l
l

a
n
d

t
e
c
h
n
o
l
o
g
v

i
s

e
f
f
e
c
t
i
v
e
,

c
a
n

e
r
r
o
r
s

s
t
i
l
l

o
c
c
u
r
?

A
n
d

i
f

s
o
,

w
h
v

a
n
d

h
o
w
?

S
o
m
e

p
e
o
p
l
e

a
t
t
e
m
p
t

t
o

d
i
s
m
i
s
s

t
h
i
s

p
o
s
s
i
b
i
l
i
t
y

a

p
r
i
o
r
i
,

c
l
a
i
m
i
n
g

t
h
a
t

a
n

e
r
r
o
r

r
e
s
u
l
t
s

e
i
t
h
e
r

I
r
o
m

a

p
r
a
c
t
i
t
i
o
n
e
r
`
s

e
r
r
o
r

(
s
u
c
h

a
s

t
h
o
s
e

s
p
e
c
i
I
i
e
d

i
n

c
a
t
e
g
o
r
y

o
n
e
)

o
r

I
r
o
m

t
h
e

l
a
c
k

o
I

s
c
i
e
n
t
i
I
i
c

b
a
s
i
s

I
o
r

t
h
e

d
o
m
a
i
n

(
s
u
c
h

a
s

t
h
e

u
n
i
q
u
e
n
e
s
s

o
I

I
i
n
g
e
r
p
r
i
n
t
s
)
.

T
h
i
s

d
i
c
h
o
t
o
m
i
z
e
d

a
t
t
r
i
b
u
t
i
o
n

o
I

a
n

e
r
r
o
r

a
s

e
i
t
h
e
r

r
e
I
l
e
c
t
i
n
g

a

b
a
s
i
c

s
c
i
e
n
t
i
I
i
c

I
l
a
w

i
n

t
h
e

d
o
m
a
i
n

o
r

a

s
p
e
c
i
I
i
c

p
r
a
c
t
i
t
i
o
n
e
r
`
s

e
r
r
o
r

I
a
i
l
s

t
o

c
o
n
s
i
d
e
r

a

t
h
i
r
d

a
l
t
e
r
n
a
t
i
v
e
:

e
r
r
o
r
s

t
h
a
t

d
e
r
i
v
e

I
r
o
m

p
s
y
c
h
o
l
o
g
i
c
a
l

a
n
d

c
o
g
n
i
t
i
v
e

e
l
e
m
e
n
t
s

i
n
v
o
l
v
e
d

i
n

I
i
n
g
e
r
p
r
i
n
t

i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n
.

T
h
i
s

p
r
o
c
e
s
s

I
a
l
l
s

o
n

h
u
m
a
n

e
x
p
e
r
t
s
,

a
l
l
o
w
i
n
g

t
h
e

p
o
s
s
i
b
i
l
i
t
y

t
h
a
t

e
r
r
o
r
s

m
a
y

r
e
s
u
l
t

I
r
o
m

t
h
e

w
a
y

t
h
e

b
r
a
i
n

p
r
o
c
e
s
s
e
s

i
n
I
o
r
m
a
t
i
o
n

a
n
d

m
a
k
e
s

d
e
c
i
s
i
o
n
s
.

I
I

t
h
e

n
a
t
u
r
e

o
I

t
h
e

m
i
n
d

a
n
d

c
o
g
n
i
t
i
v
e

p
r
o
c
e
s
s
i
n
g

c
a
n

g
i
v
e

r
i
s
e

t
o

e
r
r
o
r

i
n

I
i
n
g
e
r
p
r
i
n
t

i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n
,

t
h
e
n

t
h
e
s
e

e
r
r
o
r
s

a
r
e

i
n
h
e
r
e
n
t

t
o

t
h
e

d
o
m
a
i
n
.

N
e
v
e
r
t
h
e
l
e
s
s
,

t
h
e
y

d
o

n
o
t

r
e
I
l
e
c
t

a

b
a
s
i
c

o
n
t
o
l
o
g
i
c
a
l

s
c
i
e
n
t
i
I
i
c

I
l
a
w

i
n

t
h
e

d
o
m
a
i
n

n
o
r

a
r
e

t
h
e
y

t
h
e

I
a
u
l
t

o
I

a

s
p
e
c
i
I
i
c

p
r
a
c
t
i
t
i
o
n
e
r
.

T
h
e
y

a
r
e
,

i
n

e
s
s
e
n
c
e
,

e
p
i
s
t
e
m
o
l
o
g
i
c
a
l

p
r
o
b
l
e
m
s

t
h
a
t

d
e
r
i
v
e

I
r
o
m

t
h
e

m
e
c
h
a
n
i
s
m
s

o
I

h
u
m
a
n

c
o
g
n
i
t
i
o
n

a
n
d

t
h
e

w
o
r
k
i
n
g
s

o
I

t
h
e

m
i
n
d
.

J
o
u
r
n
a
l

o
f

F
o
r
e
n
s
i
c

d
e
n
t
i
f
i
c
a
t
i
o
n
5
6

(
4
)
,

2
0
0
6

\

6
0
3
A
s

w
i
t
h

t
e
c
h
n
o
l
o
g
i
c
a
l

a
n
d

i
n
s
t
r
u
m
e
n
t
a
t
i
o
n

a
d
v
a
n
c
e
s

t
h
a
t

i
m
p
r
o
v
e

t
h
e
i
r

l
i
m
i
t
s
,

a
c
c
u
r
a
c
y
,

a
n
d

l
e
v
e
l
s

o
I

p
r
e
c
i
s
i
o
n
,

s
o

c
a
n

h
u
m
a
n

p
e
r
I
o
r
m
a
n
c
e

b
e

i
m
p
r
o
v
e
d

w
i
t
h

c
o
r
r
e
c
t

s
e
l
e
c
t
i
o
n
,

t
r
a
i
n
i
n
g
,

a
n
d

p
r
o
c
e
d
u
r
e
s
.

H
o
w
e
v
e
r
,

s
u
c
h

e
n
d
e
a
v
o
r
s

n
e
e
d

t
o

b
e

b
a
s
e
d

o
n

s
y
s
t
e
m
a
t
i
c

a
n
d

s
c
i
e
n
t
i
I
i
c

r
e
s
e
a
r
c
h
,

a
n
d

e
v
e
n

t
h
e
n

t
h
e
y

w
i
l
l

n
o
t

t
o
t
a
l
l
y

e
l
i
m
i
n
a
t
e

h
u
m
a
n

e
r
r
o
r

o
I

c
a
t
e
g
o
r
y

t
h
r
e
e
.

N
e
v
e
r
t
h
e
l
e
s
s
,

w
i
t
h

s
u
c
h

r
e
s
e
a
r
c
h
,

t
h
e
s
e

e
r
r
o
r
s

c
a
n

b
e

d
r
a
s
t
i
c
a
l
l
y

m
i
n
i
m
i
z
e
d
,

s
o

m
i
n
i
m
i
z
e
d

t
h
a
t

a
l
t
h
o
u
g
h

t
h
e
y

a
r
e

t
h
e
o
r
e
t
i
c
a
l
l
y

p
o
s
s
i
b
l
e
,

t
h
e
y

a
r
e

i
n

I
a
c
t

s
o

v
e
r
y

r
a
r
e

t
h
a
t

d
e

I
a
c
t
o

t
h
e
y

d
o

n
o
t

e
x
i
s
t
.
T
h
e

a
b
o
v
e

d
i
v
i
s
i
o
n

i
n
t
o

t
h
r
e
e

c
a
t
e
g
o
r
i
e
s

o
I

e
r
r
o
r

i
s

a

s
i
m
p
l
i
s
t
i
c

m
o
d
e
l

I
o
r

m
e
t
h
o
d
o
l
o
g
i
c
a
l

r
e
a
s
o
n
s
;

r
e
a
l
i
t
y

c
a
n

b
e

m
o
r
e

c
o
m
p
l
e
x
.

F
o
r

e
x
a
m
p
l
e
,

b
e
c
a
u
s
e

p
r
a
c
t
i
t
i
o
n
e
r
s

a
n
d

m
e
t
h
o
d
o
l
o
g
y

a
r
e

s
o

i
n
t
e
r
t
w
i
n
e
d
,

i
t

i
s

d
i
I
I
i
c
u
l
t

(
p
e
r
h
a
p
s

s
o
m
e
t
i
m
e
s

i
m
p
o
s
s
i
b
l
e
)

t
o

s
e
p
a
r
a
t
e

a
n
d

d
i
s
t
i
n
g
u
i
s
h

b
e
t
w
e
e
n

t
h
e

t
w
o

|
2
|
,

t
h
u
s

m
a
k
i
n
g

i
t

p
r
o
b
l
e
m
a
t
i
c

t
o

a
t
t
r
i
b
u
t
e

a
n

e
r
r
o
r

s
o
l
e
l
y

t
o

e
i
t
h
e
r

c
a
t
e
g
o
r
y

o
n
e

o
r

t
h
r
e
e
.

F
o
r

i
n
s
t
a
n
c
e
,

i
I

e
r
r
o
r
s

o
c
c
u
r

b
e
c
a
u
s
e

a

p
r
a
c
t
i
t
i
o
n
e
r

i
s

i
n
c
o
m
p
e
t
e
n
t
,

b
u
t

h
i
s

o
r

h
e
r

i
n
c
o
m
p
e
t
e
n
c
e

i
s

d
u
e

t
o

b
a
s
i
c

I
l
a
w
s

i
n

t
h
e

d
o
m
a
i
n
,

t
h
e
n

t
h
e

e
r
r
o
r
s

a
r
e

n
o
t

p
u
r
e
l
y

i
n

t
h
e

p
r
a
c
t
i
t
i
o
n
e
r

e
r
r
o
r

o
I

c
a
t
e
g
o
r
y

o
n
e
.

P
r
a
c
t
i
t
i
o
n
e
r
s
`

i
n
c
o
m
p
e
t
e
n
c
e

m
a
y

a
r
i
s
e

I
r
o
m

t
h
e

l
a
c
k

o
I

a
p
p
r
o
p
r
i
a
t
e

s
c
i
e
n
t
i
I
i
c
a
l
l
y

b
a
s
e
d

s
c
r
e
e
n
i
n
g

t
e
s
t
s

I
o
r

r
e
c
r
u
i
t
m
e
n
t

a
n
d

c
e
r
t
i
I
i
c
a
t
i
o
n

o
I

I
i
n
g
e
r
p
r
i
n
t

e
x
a
m
i
n
e
r
s

a
n
d

t
h
u
s

m
a
y

r
e
I
l
e
c
t

d
e
e
p
e
r

I
l
a
w
s

i
n

t
h
e

d
o
m
a
i
n
.

T
o

d
a
t
e
,

t
h
e
r
e

i
s

n
o

s
y
s
t
e
m
a
t
i
c

s
c
i
e
n
t
i
I
i
c

r
e
s
e
a
r
c
h

i
n
t
o

t
h
e

p
s
y
c
h
o
l
o
g
i
c
a
l

a
n
d

c
o
g
n
i
t
i
v
e

s
k
i
l
l
s

t
h
a
t

u
n
d
e
r
l
i
e

I
i
n
g
e
r
p
r
i
n
t

e
x
p
e
r
t
i
s
e
.

R
e
s
e
a
r
c
h

i
s

n
e
e
d
e
d

t
o

c
o
n
s
t
r
u
c
t

a
p
p
r
o
p
r
i
a
t
e

t
e
s
t
s

I
o
r

r
e
c
r
u
i
t
m
e
n
t

s
c
r
e
e
n
i
n
g

a
n
d

s
e
l
e
c
t
i
o
n

o
I

I
i
n
g
e
r
p
r
i
n
t

e
x
a
m
i
n
e
r
s
.

T
h
e
r
e
I
o
r
e
,

i
n

s
o
m
e

c
a
s
e
s
,

a
n

e
x
p
e
r
t
`
s

i
n
c
o
m
p
e
t
e
n
c
e

m
a
y

a
l
s
o

b
e

a
t
t
r
i
b
u
t
e
d

t
o

m
o
r
e

b
a
s
i
c

I
l
a
w
s

i
n

t
h
e

d
o
m
a
i
n

r
a
t
h
e
r

t
h
a
n

p
u
r
e
l
y

t
o

t
h
e

i
n
d
i
v
i
d
u
a
l

p
r
a
c
t
i
t
i
o
n
e
r
.

U
n
d
e
r
s
t
a
n
d
i
n
g

t
h
e

s
o
u
r
c
e

o
I

e
r
r
o
r
s

a
n
d

t
h
e
i
r

a
s
s
i
g
n
m
e
n
t

t
o

o
n
e

o
I

t
h
e

c
a
t
e
g
o
r
i
e
s

c
a
n

b
e

h
i
g
h
l
y

i
n
s
i
g
h
t
I
u
l

a
n
d

h
a
v
e

i
m
p
o
r
t
a
n
t

i
m
p
l
i
c
a
t
i
o
n
s
.

F
o
r

e
x
a
m
p
l
e
,

t
h
e

M
a
y
I
i
e
l
d

e
r
r
o
n
e
o
u
s

i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n

|
4
|

r
a
i
s
e
s

i
n
t
e
r
e
s
t
i
n
g

i
s
s
u
e
s
.

W
o
u
l
d

s
u
c
h

a

m
i
s
t
a
k
e

b
e

t
o
t
a
l
l
y

a
t
t
r
i
b
u
t
a
b
l
e

t
o

p
r
a
c
t
i
t
i
o
n
e
r

e
r
r
o
r
?

O
r

s
h
o
u
l
d

t
h
e

e
r
r
o
r

b
e

a
t
t
r
i
b
u
t
e
d

a
l
s
o
,

a
t

l
e
a
s
t

i
n

p
a
r
t
,

t
o

t
h
e

l
a
c
k

o
I

a
p
p
r
o
p
r
i
a
t
e

p
r
o
c
e
d
u
r
e
s
,

t
r
a
i
n
i
n
g
,

a
n
d

q
u
a
l
i
t
y

a
s
s
u
r
a
n
c
e

t
o

a
d
d
r
e
s
s

a
n
d

d
e
a
l

w
i
t
h

t
h
e

c
a
u
s
e
s

o
I

s
u
c
h

e
r
r
o
r
s
?

T
h
e
s
e

t
y
p
e
s

o
I

q
u
e
s
t
i
o
n
s

a
r
e

c
r
i
t
i
c
a
l
l
y

i
m
p
o
r
t
a
n
t

t
o

i
n
v
e
s
t
i
g
a
t
e

t
o

a
l
l
o
w

a
d
v
a
n
c
e
s

i
n

t
h
i
s

a
n
d

o
t
h
e
r

d
o
m
a
i
n
s

o
I

I
o
r
e
n
s
i
c

s
c
i
e
n
c
e
.

J
o
u
r
n
a
l

o
f

F
o
r
e
n
s
i
c

d
e
n
t
i
f
i
c
a
t
i
o
n
6
0
4

/
5
6
(
4
)
,
2
0
0
6
H
o
w
e
v
e
r
,

r
e
s
e
a
r
c
h
i
n
g

a
n
d

d
i
s
c
u
s
s
i
n
g

e
r
r
o
r
s

i
s

a

p
r
o
b
l
e
m
a
t
i
c

a
n
d

c
h
a
l
l
e
n
g
i
n
g

e
n
d
e
a
v
o
r
.

F
i
r
s
t
,

b
y

i
t
s

v
e
r
y

n
a
t
u
r
e
,

'
e
r
r
o
r

i
s

a

s
e
n
s
i
t
i
v
e

i
s
s
u
e

t
h
a
t

o
I
t
e
n

m
e
e
t
s

d
e
I
e
n
s
i
v
e

r
e
s
p
o
n
s
e
s
.

T
h
i
s

i
s

e
s
p
e
c
i
a
l
l
y

t
r
u
e

i
n

t
h
e

c
r
i
m
i
n
a
l

j
u
s
t
i
c
e

s
y
s
t
e
m

w
h
i
c
h

d
e
a
l
s

w
i
t
h

i
n
c
a
r
c
e
r
a
t
i
n
g

a
n
d

e
v
e
n

e
x
e
c
u
t
i
n
g

p
e
o
p
l
e
.

S
e
c
o
n
d
,

t
h
e

I
r
a
m
e
w
o
r
k

o
I

t
h
e

c
r
i
m
i
n
a
l

j
u
s
t
i
c
e

s
y
s
t
e
m

d
o
e
s

n
o
t

e
n
a
b
l
e

t
h
e

'
g
r
o
u
n
d

t
r
u
t
h

t
o

b
e

p
o
s
i
t
i
v
e
l
y

k
n
o
w
n
.

T
h
u
s
,

i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n

c
a
s
e
s

c
a
n

a
l
w
a
y
s

b
e

o
p
e
n

t
o

s
u
s
p
i
c
i
o
n
s

a
s

b
e
i
n
g

e
r
r
o
n
e
o
u
s
.

A

r
e
c
e
n
t

p
a
p
e
r

b
y

C
o
l
e

|
5
|

t
r
i
e
s

t
o

c
a
t
a
l
o
g
u
e

s
u
c
h

s
u
s
p
e
c
t
e
d

e
r
r
o
n
e
o
u
s

i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n
s
.

T
h
i
r
d
,

e
v
e
n

w
h
e
n

e
r
r
o
r
s

a
r
e

d
e
t
e
c
t
e
d

a
n
d

a
c
k
n
o
w
l
e
d
g
e
d
,

t
h
e
i
r

c
l
a
s
s
i
I
i
c
a
t
i
o
n

a
n
d

e
x
a
m
i
n
a
t
i
o
n

a
r
e

c
o
n
s
t
r
u
c
t
e
d

p
o
s
t

h
o
c

i
n

a

h
i
g
h
l
y

p
o
l
i
t
i
c
a
l

a
n
d

p
e
r
s
o
n
a
l

e
n
v
i
r
o
n
m
e
n
t
.
T
h
e

s
t
u
d
y

p
r
e
s
e
n
t
e
d

h
e
r
e

e
x
a
m
i
n
e
s

w
h
e
t
h
e
r

i
n
h
e
r
e
n
t

p
s
y
c
h
o
l
o
g
i
c
a
l

a
n
d

c
o
g
n
i
t
i
v
e

m
e
c
h
a
n
i
s
m
s

p
r
e
d
i
s
p
o
s
e

I
i
n
g
e
r
p
r
i
n
t

a
n
d

o
t
h
e
r

I
o
r
e
n
s
i
c

i
d
e
n
t
i
I
i
c
a
t
i
o
n

e
x
p
e
r
t
s

t
o

c
o
m
m
i
t

c
a
t
e
g
o
r
y

t
h
r
e
e

e
r
r
o
r
s

|
6
,

7
|
.

T
h
i
s

i
m
p
o
r
t
a
n
t

a
r
e
a

o
I

r
e
s
e
a
r
c
h

h
a
s

b
e
e
n

h
i
g
h
l
y

n
e
g
l
e
c
t
e
d
.

S
o
m
e

o
I

t
h
e

p
s
y
c
h
o
l
o
g
i
c
a
l

p
h
e
n
o
m
e
n
a

a
n
d

m
e
c
h
a
n
i
s
m
s

t
h
a
t

m
a
y

u
n
d
e
r
l
i
e

s
u
c
h

e
r
r
o
r
s

a
r
e

d
e
r
i
v
e
d

I
r
o
m

t
h
e

n
a
t
u
r
e

a
n
d

a
r
c
h
i
t
e
c
t
u
r
e

o
I

t
h
e

h
u
m
a
n

m
i
n
d

|
8
|
.

I
t

i
s

i
m
p
e
r
a
t
i
v
e

t
o

c
o
n
d
u
c
t

e
m
p
i
r
i
c
a
l

e
x
p
e
r
i
m
e
n
t
a
l

r
e
s
e
a
r
c
h

t
o

e
x
a
m
i
n
e

w
h
e
t
h
e
r

s
u
c
h

e
r
r
o
r
s

a
c
t
u
a
l
l
y

d
o

e
x
i
s
t

a
n
d
,

i
I

s
o
,

t
o

I
i
n
d

w
a
y
s

t
o

m
i
n
i
m
i
z
e

t
h
e
s
e

e
r
r
o
r
s
.
T
o

e
x
a
m
i
n
e

s
u
c
h

i
s
s
u
e
s
,

i
t

i
s

i
m
p
o
r
t
a
n
t

t
o

c
o
n
d
u
c
t

s
c
i
e
n
t
i
I
i
c

s
t
u
d
i
e
s

w
i
t
h
i
n

a

r
e
a
l

w
o
r
l
d

s
e
t
t
i
n
g
.

T
h
i
s

i
s

c
h
a
l
l
e
n
g
i
n
g

b
e
c
a
u
s
e

w
h
e
n

p
e
o
p
l
e

k
n
o
w

t
h
e
y

a
r
e

b
e
i
n
g

s
t
u
d
i
e
d
,

t
h
e
i
r

b
e
h
a
v
i
o
r

a
n
d

p
e
r
I
o
r
m
a
n
c
e

c
h
a
n
g
e
,

a
n
d

t
h
u
s

p
u
t
s

i
n
t
o

q
u
e
s
t
i
o
n

t
h
e

a
p
p
l
i
c
a
b
i
l
i
t
y

a
n
d

e
c
o
l
o
g
i
c
a
l

v
a
l
i
d
i
t
y

o
I

t
h
e

I
i
n
d
i
n
g
s
.

I
I

y
o
u

w
a
n
t

t
o

k
n
o
w

h
o
w

p
e
o
p
l
e

d
r
i
v
e
,

t
h
e
n

t
h
e
i
r

p
e
r
I
o
r
m
a
n
c
e

d
u
r
i
n
g

a

d
r
i
v
i
n
g

t
e
s
t

i
s

n
o
t

v
e
r
y

i
n
s
i
g
h
t
I
u
l

a
n
d

r
e
v
e
a
l
i
n
g
,

n
e
i
t
h
e
r

i
s

t
h
e
i
r

d
r
i
v
i
n
g

w
h
e
n

t
h
e
y

k
n
o
w

t
h
e
y

a
r
e

n
e
a
r

s
p
e
e
d

c
a
m
e
r
a
s

o
r

r
a
d
a
r
s
.

O
n
e

m
u
s
t

t
r
y

t
o

o
b
s
e
r
v
e

a
n
d

e
x
a
m
i
n
e

p
e
r
I
o
r
m
a
n
c
e

a
s

w
e
l
l

a
s

c
o
l
l
e
c
t

d
a
t
a

i
n

t
h
e

n
o
r
m
a
l

r
o
u
t
i
n
e

s
e
t
t
i
n
g

w
i
t
h

m
i
n
i
m
a
l

(
o
r

n
o
)

k
n
o
w
l
e
d
g
e

o
I

t
h
e

p
e
o
p
l
e

i
n
v
o
l
v
e
d
.

T
h
i
s

o
I

c
o
u
r
s
e

i
I

v
e
r
y

d
i
I
I
i
c
u
l
t

t
o

a
c
h
i
e
v
e

a
n
d

n
e
c
e
s
s
a
r
i
l
y

r
e
s
u
l
t
s

i
n

s
m
a
l
l

d
a
t
a

s
e
t
s
.

H
o
w
e
v
e
r
,

t
h
e
s
e

d
a
t
a

s
e
t
s

a
r
e

s
t
a
t
i
s
t
i
c
a
l
l
y

v
e
r
y

p
o
w
e
r
I
u
l
,

m
e
a
n
i
n
g
I
u
l
,

a
n
d

m
o
r
e

i
n
t
e
r
p
r
e
t
a
b
l
e
.
J
o
u
r
n
a
l

o
f

F
o
r
e
n
s
i
c

d
e
n
t
i
f
i
c
a
t
i
o
n
5
6

(
4
)
,

2
0
0
6

\

6
0
5
P
r
e
v
i
o
u
s

R
e
s
e
a
r
c
h
I
n

o
u
r

e
m
p
i
r
i
c
a
l

s
t
u
d
i
e
s

|
9

a
n
d

1
0
|
,

w
e

h
a
v
e

s
t
a
r
t
e
d

t
o

a
d
d
r
e
s
s

i
s
s
u
e
s

r
e
l
a
t
i
n
g

t
o

e
r
r
o
r
s

t
h
a
t

d
e
r
i
v
e

I
r
o
m

p
s
y
c
h
o
l
o
g
i
c
a
l

a
n
d

c
o
g
n
i
t
i
v
e

i
n
I
l
u
e
n
c
e
s
.

E
m
o
t
i
o
n
a
l

s
t
a
t
e
s

(
F
i
g
u
r
e

1
)

h
a
v
e

b
e
e
n

s
h
o
w
n

t
o

c
a
u
s
e

n
o
n
e
x
p
e
r
t
s

t
o

b
e

m
o
r
e

l
i
k
e
l
y

t
o

'
m
a
t
c
h
.
.
.
a
m
b
i
g
u
o
u
s

p
a
i
r
s

o
I

I
i
n
g
e
r
p
r
i
n
t
s

w
h
e
n

t
h
e
y

p
e
r
I
o
r
m
e
d

t
h
e

c
o
m
p
a
r
i
s
o
n

i
n

a

h
i
g
h
l
y

e
m
o
t
i
o
n
a
l

c
o
n
t
e
x
t

|
9
|
.

F
i
g
u
r
e

1
A
n

e
x
a
m
p
l
e

o
f

a
n

e
m
o
t
i
o
n
a
l

c
o
n
t
e
x
t

u
s
e
d

i
n

t
h
e

p
r
e
v
i
o
u
s

(
n
o
n
e
x
p
e
r
t
)

s
t
u
d
v
.
I
n

a

p
r
e
v
i
o
u
s

s
t
u
d
y

|
1
0
|
,

I
i
v
e

e
x
p
e
r
t
s

w
e
r
e

s
h
o
w
n

I
i
n
g
e
r
p
r
i
n
t
s

a
n
d

w
e
r
e

t
o
l
d

t
h
a
t

t
h
e

p
r
i
n
t
s

w
e
r
e

I
r
o
m

a

h
i
g
h
l
y

p
u
b
l
i
c
i
z
e
d

e
r
r
o
n
e
o
u
s

i
d
e
n
t
i
I
i
c
a
t
i
o
n

|
4
|
,

s
u
g
g
e
s
t
i
n
g

t
h
a
t

t
h
e

I
i
n
g
e
r
p
r
i
n
t
s

i
n

I
r
o
n
t

o
I

t
h
e
m

w
e
r
e

a
n

e
x
c
l
u
s
i
o
n
.

H
o
w
e
v
e
r
,

r
a
t
h
e
r

t
h
a
n

g
i
v
i
n
g

t
h
e
m

p
r
i
n
t
s

t
h
a
t

w
e
r
e

a
n

e
x
c
l
u
s
i
o
n
,

w
e

p
r
e
s
e
n
t
e
d

t
h
e
m

w
i
t
h

I
i
n
g
e
r
p
r
i
n
t
s

t
h
a
t

h
a
d

b
e
e
n

c
o
m
p
a
r
e
d

a
n
d

i
n
d
i
v
i
d
u
a
l
i
z
e
d
.

T
h
e

I
i
n
g
e
r
p
r
i
n
t
s

w
e
r
e

n
o
t

o
n
l
y

i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n
s
,

b
u
t

t
h
e
y

h
a
d

a
l
s
o

b
e
e
n

p
r
e
v
i
o
u
s
l
y

i
n
d
i
v
i
d
u
a
l
i
z
e
d

b
y

t
h
e

s
a
m
e

e
x
p
e
r
t
s

n
o
w

b
e
i
n
g

t
e
s
t
e
d
.

A
l
t
h
o
u
g
h

t
h
e

e
x
p
e
r
t
s

w
e
r
e

i
n
s
t
r
u
c
t
e
d

t
o

'
i
g
n
o
r
e

a
l
l

t
h
e

c
o
n
t
e
x
t
u
a
l

i
n
I
o
r
m
a
t
i
o
n

a
n
d

t
o

I
o
c
u
s

s
o
l
e
l
y

o
n

t
h
e

a
c
t
u
a
l

p
r
i
n
t
s
,

m
o
s
t

o
I

t
h
e

e
x
p
e
r
t
s

(
I
o
u
r

o
I

t
h
e

I
i
v
e
)

w
e
r
e

a
I
I
e
c
t
e
d

b
y

t
h
e

c
o
n
t
e
x
t

a
n
d

m
a
d
e

i
n
c
o
n
s
i
s
t
e
n
t

d
e
c
i
s
i
o
n
s
.

I
n

t
h
e

a
b
o
v
e

s
t
u
d
y
,

w
e

w
e
r
e

a
b
l
e

t
o

d
e
m
o
n
s
t
r
a
t
e

t
h
e

v
u
l
n
e
r
a
b
i
l
i
t
y

o
I

e
x
p
e
r
t
s

t
o

e
x
t
r
a
n
e
o
u
s

c
o
n
t
e
x
t
u
a
l

i
n
I
o
r
m
a
t
i
o
n
,

b
u
t

o
n
l
y

w
h
e
n

w
e

s
u
b
j
e
c
t
e
d

t
h
e
m

t
o

a

r
e
l
a
t
i
v
e
l
y

e
x
t
r
e
m
e

c
o
n
t
e
x
t

a
n
d

w
h
e
n

w
e

p
r
e
s
e
n
t
e
d

t
h
e
m

w
i
t
h

d
i
I
I
i
c
u
l
t

m
a
t
c
h
e
s
.
J
o
u
r
n
a
l

o
f

F
o
r
e
n
s
i
c

d
e
n
t
i
f
i
c
a
t
i
o
n
6
0
6

/
5
6
(
4
)
,
2
0
0
6
C
u
r
r
e
n
t

R
e
s
e
a
r
c
h

T
h
e

s
t
u
d
y

r
e
p
o
r
t
e
d

w
i
t
h
i
n

t
h
i
s

a
r
t
i
c
l
e

I
o
l
l
o
w
s

u
p

o
n

p
r
e
v
i
o
u
s

s
t
u
d
i
e
s
,

I
u
r
t
h
e
r

e
x
a
m
i
n
i
n
g

p
s
y
c
h
o
l
o
g
i
c
a
l

a
n
d

c
o
g
n
i
t
i
v
e

i
n
I
l
u
-
e
n
c
e
s

t
h
a
t

m
a
y

p
l
a
y

a

r
o
l
e

i
n

t
h
e

w
o
r
k

o
I

I
i
n
g
e
r
p
r
i
n
t

e
x
p
e
r
t
s
.

T
h
e

s
p
e
c
i
I
i
c

p
u
r
p
o
s
e
s

o
I

t
h
i
s

c
u
r
r
e
n
t

s
t
u
d
y

w
e
r
e
:

1
.

T
o

d
e
t
e
r
m
i
n
e

w
h
e
t
h
e
r

t
h
e

o
r
i
g
i
n
a
l

I
i
n
d
i
n
g
s

w
o
u
l
d

r
e
p
-
l
i
c
a
t
e

t
o

a
n
o
t
h
e
r

a
n
d

l
a
r
g
e
r

s
e
t

o
I

d
a
t
a
.

2
.

T
o

u
s
e

l
e
s
s

e
x
t
r
e
m
e

a
n
d

m
o
r
e

r
o
u
t
i
n
e

d
a
y
-
t
o
-
d
a
y

c
o
n
-
t
e
x
t
u
a
l

i
n
I
l
u
e
n
c
e
s
.

3
.

T
o

u
s
e

p
a
i
r
s

o
I

p
r
i
n
t
s

o
I

v
a
r
y
i
n
g

l
e
v
e
l
s

o
I

d
i
I
I
i
c
u
l
t
y
.

4
.

T
o

e
x
a
m
i
n
e

t
h
e

p
o
s
s
i
b
l
e

i
n
I
l
u
e
n
c
e

o
I

c
o
n
t
e
x
t
u
a
l

i
n
I
o
r
m
a
-
t
i
o
n

o
n

d
i
I
I
e
r
e
n
t

d
e
c
i
s
i
o
n

t
y
p
e
s

(
t
h
u
s
,

n
o
t

o
n
l
y

w
h
e
t
h
e
r

i
t

c
a
n

c
h
a
n
g
e

a

p
a
s
t

i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n

d
e
c
i
s
i
o
n

t
o

a
n

e
x
c
l
u
s
i
o
n
,

b
u
t

a
l
s
o

t
o

e
x
a
m
i
n
e

w
h
e
t
h
e
r

i
t

c
a
n

c
h
a
n
g
e

a

p
a
s
t

e
x
c
l
u
s
i
o
n

d
e
c
i
s
i
o
n

t
o

a
n

i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n
)
.

5
.

T
o

e
x
a
m
i
n
e

t
h
e

b
a
s
i
c

c
o
n
s
i
s
t
e
n
c
y

o
I

d
e
c
i
s
i
o
n
s

b
y

r
e
-
p
r
e
s
e
n
t
i
n
g

t
o

e
x
p
e
r
t
s

t
h
e

s
a
m
e

I
i
n
g
e
r
p
r
i
n
t
s

t
h
e
y

j
u
d
g
e
d

i
n

t
h
e

p
a
s
t

b
u
t

w
i
t
h
o
u
t

i
n
t
r
o
d
u
c
i
n
g

a
n
y

c
o
n
t
e
x
t
u
a
l

i
n
I
o
r
-
m
a
t
i
o
n

m
a
n
i
p
u
l
a
t
i
o
n
,

t
h
u
s

e
x
a
m
i
n
i
n
g

t
h
e

r
e
l
i
a
b
i
l
i
t
y

o
I

e
x
p
e
r
t
s
.
M
e
t
h
o
d
P
a
r
t
i
c
i
p
a
n
t
s
S
i
x

I
i
n
g
e
r
p
r
i
n
t

e
x
p
e
r
t
s
,

r
e
p
r
e
s
e
n
t
i
n
g

m
o
r
e

t
h
a
n

3
5

y
e
a
r
s

o
I

e
x
p
e
r
i
e
n
c
e

i
n

e
x
a
m
i
n
i
n
g

I
i
n
g
e
r
p
r
i
n
t
s

(
e
a
c
h

w
i
t
h

a

m
i
n
i
m
u
m

o
I

5

y
e
a
r
s
`

e
x
p
e
r
i
e
n
c
e

i
n

l
a
t
e
n
t

p
r
i
n
t
s
)
,

p
a
r
t
i
c
i
p
a
t
e
d
.

T
h
e

e
x
p
e
r
t
s

w
e
r
e

e
x
p
e
r
i
e
n
c
e
d

a
n
d

s
p
e
c
i
a
l
i
z
e
d

i
n

l
a
t
e
n
t

p
r
i
n
t

e
x
a
m
i
n
a
t
i
o
n

a
n
d

w
e
r
e

n
o
t

I
i
e
l
d

o
p
e
r
a
t
i
v
e
s
;

h
e
n
c
e
,

t
h
e
i
r

e
x
p
e
r
i
e
n
c
e

w
a
s

I
u
l
l
-
t
i
m
e

i
n

l
a
t
e
n
t

p
r
i
n
t

c
o
m
p
a
r
i
s
o
n
s
.

E
a
c
h

o
I

t
h
e

e
x
p
e
r
t

p
a
r
t
i
c
i
p
a
n
t
s

w
a
s

n
o
t

o
n
l
y

h
i
g
h
l
y

e
x
p
e
r
i
e
n
c
e
d
,

b
u
t

w
a
s

h
i
g
h
l
y

t
r
a
i
n
e
d
,

c
e
r
t
i
I
i
e
d

b
y

a

n
a
t
i
o
n
a
l
l
y

r
e
c
o
g
n
i
z
e
d

i
n
d
e
p
e
n
d
e
n
t

a
u
t
h
o
r
i
t
y
,

a
n
d

h
a
d

s
u
c
c
e
s
s
-
I
u
l
l
y

c
o
m
p
l
e
t
e
d

p
r
o
I
i
c
i
e
n
c
y

t
e
s
t
i
n
g
.

N
o
n
e

o
I

t
h
e
m

h
a
d

b
e
e
n

t
h
e

s
u
b
j
e
c
t

o
I

a

p
o
o
r

c
o
m
p
e
t
e
n
c
y

r
e
v
i
e
w
,

a
n
d

t
h
e
y

w
e
r
e

a
l
l

c
o
n
s
i
d
-
e
r
e
d

b
y

t
h
e
i
r

r
e
s
p
e
c
t
i
v
e

l
a
b
o
r
a
t
o
r
y

d
i
r
e
c
t
o
r
s

o
r

b
u
r
e
a
u

c
h
i
e
I
s

t
o

b
e

e
I
I
e
c
t
i
v
e

a
n
d

c
o
m
p
e
t
e
n
t
.

T
h
e

p
a
r
t
i
c
i
p
a
n
t
s

w
e
r
e

t
a
k
e
n

I
r
o
m

o
u
r

i
n
t
e
r
n
a
t
i
o
n
a
l

I
i
n
g
e
r
p
r
i
n
t

e
x
p
e
r
t

p
o
o
l

o
I

v
o
l
u
n
t
e
e
r
s
.

T
h
i
s

p
o
o
l

i
n
c
l
u
d
e
s

I
i
n
g
e
r
p
r
i
n
t

e
x
p
e
r
t
s

I
r
o
m

a
c
r
o
s
s

t
h
e

w
o
r
l
d

(
i
n
c
l
u
d
i
n
g

t
h
e

U
S
A
,

U
K
,

I
s
r
a
e
l
,

t
h
e

N
e
t
h
e
r
l
a
n
d
s
,

a
n
d

A
u
s
t
r
a
l
i
a
)
.
J
o
u
r
n
a
l

o
f

F
o
r
e
n
s
i
c

d
e
n
t
i
f
i
c
a
t
i
o
n
5
6

(
4
)
,

2
0
0
6

\

6
0
7
W
e

u
s
e
d

e
x
p
e
r
t
s

w
h
o
s
e

p
a
s
t

w
o
r
k

w
e

c
o
u
l
d

c
o
v
e
r
t
l
y

a
c
c
e
s
s
.

B
e
c
a
u
s
e

t
h
e

c
o
l
l
a
t
i
o
n

o
I

i
n
d
i
v
i
d
u
a
l

c
a
s
e
w
o
r
k

i
s

s
o

d
i
I
I
i
c
u
l
t

i
n

s
u
c
h

c
i
r
c
u
m
s
t
a
n
c
e
s
,

a
n
d

b
e
c
a
u
s
e

o
I

t
h
e

d
i
I
I
i
c
u
l
t
i
e
s

i
n

c
o
v
e
r
t

t
e
s
t
i
n
g
,

t
h
i
s

e
I
I
e
c
t
i
v
e
l
y

l
i
m
i
t
e
d

t
h
e

n
u
m
b
e
r

o
I

p
a
r
t
i
c
i
p
a
n
t
s

w
i
t
h
i
n

t
h
i
s

s
t
u
d
y
.

T
h
i
s

e
n
a
b
l
e
d

u
s
,

h
o
w
e
v
e
r
,

t
o

c
o
l
l
e
c
t

m
o
r
e

m
e
a
n
i
n
g
I
u
l

a
n
d

p
o
w
e
r
I
u
l

d
a
t
a
.

T
h
i
s

d
e
r
i
v
e
s

n
o
t

o
n
l
y

I
r
o
m

t
h
e

c
o
v
e
r
t

n
a
t
u
r
e

o
I

o
u
r

s
t
u
d
y

b
u
t

a
l
s
o

I
r
o
m

t
h
e

u
s
e

o
I

a

w
i
t
h
i
n
-
s
u
b
j
e
c
t

e
x
p
e
r
i
-
m
e
n
t
a
l

d
e
s
i
g
n

i
n

w
h
i
c
h

p
a
r
t
i
c
i
p
a
n
t
s

a
r
e

u
s
e
d

a
s

t
h
e
i
r

o
w
n

c
o
n
t
r
o
l

t
h
r
o
u
g
h

r
e
p
e
a
t
e
d

m
e
a
s
u
r
e
s

|
1
1
|
.

T
h
u
s
,

v
a
r
i
a
t
i
o
n
s

a
r
e

m
o
r
e

e
a
s
i
l
y

a
t
t
r
i
b
u
t
a
b
l
e

t
o

t
h
e

e
x
p
e
r
i
m
e
n
t
a
l

m
a
n
i
p
u
l
a
t
i
o
n
s

a
n
d

c
o
n
d
i
t
i
o
n
s

r
a
t
h
e
r

t
h
a
n

i
n
d
i
v
i
d
u
a
l

d
i
I
I
e
r
e
n
c
e
s

b
e
t
w
e
e
n

p
e
o
p
l
e
.

T
h
i
s

p
r
o
v
i
d
e
s

c
l
e
a
r
e
r

a
n
d

m
o
r
e

i
n
t
e
r
p
r
e
t
a
b
l
e

r
e
s
u
l
t
s
,

a
s

w
e
l
l

a
s

m
o
r
e

s
t
a
t
i
s
t
i
c
a
l

p
o
w
e
r

I
o
r

e
a
c
h

d
a
t
a

p
o
i
n
t

|
1
1
|
.
M
a
t
e
r
i
a
l
s
A

d
i
I
I
e
r
e
n
t

a
n
d

u
n
i
q
u
e

s
e
t

o
I

e
i
g
h
t

p
a
i
r
s

o
I

I
i
n
g
e
r
p
r
i
n
t
s

w
a
s

p
r
e
p
a
r
e
d

a
n
d

t
a
i
l
o
r
e
d

I
o
r

e
a
c
h

o
I

t
h
e

p
a
r
t
i
c
i
p
a
n
t
s
.

E
a
c
h

s
e
t

i
n
c
l
u
d
e
d

I
o
u
r

p
a
i
r
s

o
I

p
r
i
n
t
s

t
h
a
t

t
h
e

s
p
e
c
i
I
i
c

I
i
n
g
e
r
p
r
i
n
t

e
x
p
e
r
t

h
a
d

i
n

t
h
e

p
a
s
t

j
u
d
g
e
d

a
s

i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n
s

a
n
d

I
o
u
r

p
a
i
r
s

o
I

p
r
i
n
t
s

t
h
a
t

h
e

o
r

s
h
e

h
a
d

j
u
d
g
e
d

i
n

t
h
e

p
a
s
t

a
s

e
x
c
l
u
s
i
o
n
s
.

A
l
l

o
I

t
h
e

e
i
g
h
t

p
a
i
r
s

o
I

I
i
n
g
e
r
p
r
i
n
t
s

h
a
d

b
e
e
n

d
e
e
m
e
d

i
n

t
h
e

p
a
s
t

b
y

t
h
e

s
p
e
c
i
I
i
c

p
a
r
t
i
c
i
p
a
n
t

t
o

h
a
v
e

s
u
I
I
i
c
i
e
n
t

i
n
I
o
r
m
a
t
i
o
n

t
o

m
a
k
e

d
e
I
i
n
i
t
e

j
u
d
g
e
m
e
n
t
s
.

W
i
t
h
i
n

e
a
c
h

o
I

t
h
e

I
o
u
r

p
a
s
t

i
n
d
i
v
i
d
u
a
l
i
z
a
-
t
i
o
n
s

a
n
d

t
h
e

I
o
u
r

e
x
c
l
u
s
i
o
n
s
,

t
w
o

p
a
i
r
s

o
I

p
r
i
n
t
s

w
e
r
e

r
e
l
a
t
i
v
e
l
y

d
i
I
I
i
c
u
l
t

t
o

j
u
d
g
e
,

a
n
d

t
h
e

r
e
m
a
i
n
i
n
g

t
w
o

p
a
i
r
s

w
e
r
e

r
e
l
a
t
i
v
e
l
y

n
o
t

d
i
I
I
i
c
u
l
t

t
o

j
u
d
g
e
.

T
h
e

l
a
t
e
n
t

I
i
n
g
e
r
p
r
i
n
t
s

h
a
d

a
l
l

b
e
e
n

o
b
t
a
i
n
e
d

I
r
o
m

r
e
a
l

c
r
i
m
e

s
c
e
n
e
s

a
n
d

w
e
r
e

a
l
l

p
r
e
s
e
n
t
e
d

t
o

t
h
e

p
a
r
t
i
c
i
p
a
n
t
s

i
n

t
h
e
i
r

o
r
i
g
i
n
a
l

I
o
r
m
a
t

I
o
r

c
o
m
p
a
r
i
s
o
n

a
g
a
i
n
s
t

s
u
s
p
e
c
t

t
e
n
p
r
i
n
t

e
x
e
m
p
l
a
r
s
.

I
t

s
h
o
u
l
d

b
e

s
t
r
e
s
s
e
d

t
h
a
t

a
l
l

t
h
e

p
a
i
r
s

o
I

p
r
i
n
t
s

t
h
a
t

w
e

u
s
e
d

w
e
r
e

I
i
n
g
e
r
p
r
i
n
t

c
o
m
p
a
r
i
s
o
n
s

t
h
a
t

w
e

o
b
t
a
i
n
e
d

I
r
o
m

t
h
e

a
r
c
h
i
v
e
s

a
n
d

h
a
d

b
e
e
n

e
v
a
l
u
a
t
e
d

s
o
m
e

y
e
a
r
s

b
e
I
o
r
e

b
y

t
h
e

v
e
r
y

s
a
m
e

e
x
p
e
r
t
s
.

T
h
e

w
i
t
h
i
n
-
s
u
b
j
e
c
t

d
e
s
i
g
n

o
I

t
h
e

e
x
p
e
r
i
m
e
n
t

w
a
s

d
e
e
m
e
d

v
i
t
a
l

t
o

t
h
e

o
v
e
r
a
l
l

r
o
b
u
s
t
n
e
s
s

a
n
d

c
r
e
d
i
b
i
l
i
t
y

o
I

t
h
e

I
i
n
d
i
n
g
s
.

T
w
o

a
d
d
i
t
i
o
n
a
l

e
x
p
e
r
i
e
n
c
e
d

I
i
n
g
e
r
p
r
i
n
t

e
x
p
e
r
t
s

w
h
o

d
i
d

n
o
t

t
a
k
e

p
a
r
t

i
n

t
h
e

s
t
u
d
y

(
e
a
c
h

h
a
d

m
o
r
e

t
h
a
n

2
0

y
e
a
r
s

o
I

e
x
p
e
r
i
e
n
c
e
)

i
n
d
e
p
e
n
d
e
n
t
l
y

c
o
n
I
i
r
m
e
d

a
n
d

v
e
r
i
I
i
e
d

t
h
a
t

a
l
l

t
h
e

p
a
i
r
s

o
I

I
i
n
g
e
r
-
p
r
i
n
t
s

w
e
r
e

i
n
d
e
e
d

e
i
t
h
e
r

c
o
r
r
e
c
t

i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n
s

o
r

c
o
r
r
e
c
t

e
x
c
l
u
s
i
o
n
s
.

T
h
e
y

a
l
s
o

c
h
a
r
a
c
t
e
r
i
z
e
d

t
h
e

p
a
i
r
s

o
I

p
r
i
n
t
s

a
s

e
i
t
h
e
r

r
e
l
a
t
i
v
e
l
y

d
i
I
I
i
c
u
l
t

t
o

e
v
a
l
u
a
t
e

o
r

a
s

r
e
l
a
t
i
v
e
l
y

n
o
t

d
i
I
I
i
c
u
l
t

t
o

e
v
a
l
u
a
t
e
.
J
o
u
r
n
a
l

o
f

F
o
r
e
n
s
i
c

d
e
n
t
i
f
i
c
a
t
i
o
n
6
0
8

/
5
6
(
4
)
,
2
0
0
6
A
n

i
n
s
t
r
u
c
t
i
o
n

a
n
d

r
e
s
p
o
n
s
e

s
h
e
e
t

w
a
s

p
r
e
p
a
r
e
d

I
o
r

e
a
c
h

o
I

t
h
e

e
i
g
h
t

p
a
i
r
s

o
I

I
i
n
g
e
r
p
r
i
n
t
s
.

F
o
u
r

p
a
i
r
s

o
I

p
r
i
n
t
s

w
e
r
e

u
s
e
d

a
s

c
o
n
t
r
o
l
s

a
n
d

p
r
o
v
i
d
e
d

n
o

c
o
n
t
e
x
t
u
a
l

i
n
I
o
r
m
a
t
i
o
n
.

T
h
e
s
e

I
o
u
r

p
a
i
r
s

o
I

c
o
n
t
r
o
l

p
r
i
n
t
s

i
n
c
l
u
d
e
d

t
w
o

p
a
i
r
s

t
h
a
t

h
a
d

b
e
e
n

j
u
d
g
e
d

i
n

t
h
e

p
a
s
t

a
s

i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n
s

a
n
d

t
w
o

p
a
i
r
s

t
h
a
t

h
a
d

b
e
e
n

j
u
d
g
e
d

i
n

t
h
e

p
a
s
t

a
s

e
x
c
l
u
s
i
o
n
s
.

O
I

e
a
c
h

o
I

t
h
e
s
e

t
w
o

p
a
i
r
s
,

o
n
e

w
a
s

r
e
l
a
t
i
v
e
l
y

d
i
I
I
i
c
u
l
t

t
o

j
u
d
g
e

a
n
d

o
n
e

w
a
s

r
e
l
a
t
i
v
e
l
y

n
o
t

d
i
I
I
i
c
u
l
t

t
o

j
u
d
g
e
.

T
h
e

i
n
s
t
r
u
c
t
i
o
n

s
h
e
e
t

t
h
a
t

w
a
s

g
i
v
e
n

w
i
t
h

t
h
e

c
o
n
t
r
o
l

p
a
i
r
s

i
n
c
l
u
d
e
d

m
i
n
i
m
a
l

i
n
s
t
r
u
c
t
i
o
n
s

t
h
a
t

t
o
l
d

t
h
e

e
x
p
e
r
t
s

t
o

e
v
a
l
u
a
t
e

t
h
e

p
r
i
n
t
s
.

A

r
e
s
p
o
n
s
e

s
h
e
e
t

w
a
s

p
r
o
v
i
d
e
d

I
o
r

p
a
r
t
i
c
i
p
a
n
t
s

t
o

w
r
i
t
e

t
h
e
i
r

c
o
n
c
l
u
s
i
o
n
s
.

I
n

t
h
e

r
e
s
p
o
n
s
e

s
h
e
e
t
,

t
h
e

e
x
p
e
r
t
s

w
e
r
e

I
i
r
s
t

a
s
k
e
d

w
h
e
t
h
e
r

t
h
e
r
e

w
a
s

'
s
u
I
I
i
c
i
e
n
t

i
n
I
o
r
m
a
t
i
o
n

w
i
t
h
i
n

t
h
e

p
r
i
n
t
s

t
o

e
i
t
h
e
r

i
d
e
n
t
i
I
y

o
r

e
x
c
l
u
d
e
.

I
I

t
h
e

a
n
s
w
e
r

w
a
s
,

'
n
o
,

t
h
e
n

t
h
e
y

I
i
n
i
s
h
e
d

w
i
t
h

t
h
i
s

p
a
i
r

o
I

p
r
i
n
t
s

a
n
d

m
o
v
e
d

o
n

t
o

t
h
e

n
e
x
t

p
a
i
r
.

I
I

t
h
e

a
n
s
w
e
r

w
a
s
,

'
y
e
s
,

t
h
e
n

t
h
e

e
x
p
e
r
t
s

c
o
n
t
i
n
u
e
d
,

s
t
a
t
e
d

w
h
a
t

t
h
e
i
r

d
e
c
i
s
i
o
n
s

w
e
r
e
,

a
n
d

t
h
e
n

m
o
v
e
d

o
n

t
o

t
h
e

n
e
x
t

p
a
i
r

o
I

p
r
i
n
t
s
.
O
I

t
h
e

r
e
m
a
i
n
i
n
g

I
o
u
r

p
a
i
r
s

o
I

p
r
i
n
t
s
,

t
w
o

p
a
i
r
s

h
a
d

b
e
e
n

j
u
d
g
e
d

i
n

t
h
e

p
a
s
t

a
s

i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n
s

a
n
d

t
w
o

h
a
d

b
e
e
n

j
u
d
g
e
d

i
n

t
h
e

p
a
s
t

a
s

e
x
c
l
u
s
i
o
n
s
.

T
h
e
s
e

I
o
u
r

p
a
i
r
s

o
I

p
r
i
n
t
s

w
e
r
e

p
r
e
s
e
n
t
e
d

w
i
t
h
i
n

c
o
n
t
e
x
t
u
a
l
l
y

b
i
a
s
i
n
g

i
n
I
o
r
m
a
t
i
o
n

t
h
a
t

w
a
s

h
y
p
o
t
h
e
s
i
z
e
d

t
o

i
n
I
l
u
e
n
c
e

t
h
e

c
o
n
c
l
u
s
i
o
n
s

r
e
a
c
h
e
d

b
y

t
h
e

e
x
p
e
r
t
s
.

T
h
e

t
w
o

p
a
i
r
s

o
I

p
r
i
n
t
s

t
h
a
t

h
a
d

b
e
e
n

j
u
d
g
e
d

i
n

t
h
e

p
a
s
t

a
s

i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n
s

w
e
r
e

p
r
e
s
e
n
t
e
d

i
n

a

c
o
n
t
e
x
t

t
h
a
t

s
u
g
g
e
s
t
e
d

t
h
a
t

t
h
e
y

w
e
r
e

e
x
c
l
u
s
i
o
n
s

(
o
n
e

o
I

t
h
e

p
a
i
r
s

w
a
s

r
e
l
a
t
i
v
e
l
y

d
i
I
I
i
c
u
l
t

t
o

i
n
d
i
v
i
d
u
a
l
i
z
e
,

a
n
d

t
h
e

o
t
h
e
r

w
a
s

n
o
t
)
.

S
i
m
i
l
a
r
l
y
,

t
h
e

t
w
o

p
a
i
r
s

o
I

p
r
i
n
t
s

t
h
a
t

h
a
d

b
e
e
n

j
u
d
g
e
d

i
n

t
h
e

p
a
s
t

a
s

e
x
c
l
u
s
i
o
n
s

w
e
r
e

p
r
e
s
e
n
t
e
d

i
n

a

c
o
n
t
e
x
t

t
h
a
t

s
u
g
g
e
s
t
e
d

t
h
a
t

t
h
e
y

w
e
r
e

i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n
s
.

(
A
g
a
i
n
,

o
n
e

o
I

t
h
e

p
a
i
r
s

w
a
s

r
e
l
a
t
i
v
e
l
y

d
i
I
I
i
c
u
l
t

t
o

i
n
d
i
v
i
d
u
a
l
i
z
e
,

a
n
d

t
h
e

o
t
h
e
r

w
a
s

n
o
t
.
)

I
n

c
o
n
t
r
a
s
t

t
o

o
u
r

p
r
e
v
i
o
u
s

s
t
u
d
y

|
1
0
|
,

w
e

n
o
w

u
s
e
d

m
o
r
e

s
u
b
t
l
e
,

r
o
u
t
i
n
e
,

d
a
y
-
t
o
-
d
a
y
,

c
o
n
t
e
x
t
u
a
l
l
y

b
i
a
s
i
n
g

i
n
I
o
r
m
a
t
i
o
n
.

I
n

t
h
e

i
n
s
t
r
u
c
t
i
o
n
s

t
h
e

p
a
r
t
i
c
i
p
a
n
t
s

w
e
r
e

t
o
l
d
,

I
o
r

e
x
a
m
p
l
e
,

t
h
a
t

t
h
e

'
s
u
s
p
e
c
t

c
o
n
I
e
s
s
e
d

t
o

t
h
e

c
r
i
m
e

(
I
o
r

c
o
n
t
e
x
t
u
a
l

i
n
I
o
r
m
a
t
i
o
n

t
h
a
t

t
h
e

p
r
i
n
t
s

w
e
r
e

a
n

i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n
,

w
h
e
n

i
n

I
a
c
t

t
h
e
y

w
e
r
e

n
o
t

a
n
d

h
a
d

b
e
e
n

j
u
d
g
e
d

b
y

t
h
e

s
a
m
e

e
x
p
e
r
t

a
s

a
n

e
x
c
l
u
s
i
o
n

i
n

t
h
e

p
a
s
t
)

o
r

t
h
a
t

t
h
e

'
s
u
s
p
e
c
t

w
a
s

i
n

p
o
l
i
c
e

c
u
s
t
o
d
y

a
t

t
h
e

t
i
m
e

o
I

t
h
e

c
r
i
m
e

(
I
o
r

c
o
n
t
e
x
t
u
a
l

i
n
I
o
r
m
a
t
i
o
n

t
h
a
t

t
h
e

p
r
i
n
t
s

w
e
r
e

n
o
t

a
n

i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n
,

w
h
e
n

i
n

I
a
c
t

t
h
e
y

w
e
r
e

a
n
d

h
a
d

b
e
e
n

j
u
d
g
e
d

b
y

t
h
e

s
a
m
e

e
x
p
e
r
t

a
s

a
n

i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n

i
n

t
h
e

p
a
s
t
)
.
J
o
u
r
n
a
l

o
f

F
o
r
e
n
s
i
c

d
e
n
t
i
f
i
c
a
t
i
o
n
5
6

(
4
)
,

2
0
0
6

\

6
0
9
T
h
u
s

w
e

p
r
e
p
a
r
e
d

a

t
o
t
a
l

o
I

4
8

u
n
i
q
u
e

e
x
p
e
r
i
m
e
n
t
a
l

t
r
i
a
l
s
.

E
a
c
h

o
n
e

c
o
n
s
i
s
t
e
d

o
I

a

l
a
t
e
n
t

p
r
i
n
t

I
r
o
m

a

c
r
i
m
e

s
c
e
n
e

a
n
d

a

s
u
s
p
e
c
t

t
e
n
p
r
i
n
t

e
x
e
m
p
l
a
r
,

a
c
c
o
m
p
a
n
i
e
d

w
i
t
h

t
h
e

p
r
o
p
e
r

i
n
s
t
r
u
c
t
i
o
n
s

a
n
d

r
e
s
p
o
n
s
e

s
h
e
e
t
s
.

F
o
r

e
a
c
h

e
x
p
e
r
t
,

w
e

p
r
e
p
a
r
e
d

a

c
u
s
t
o
m
i
z
e
d

I
o
l
d
e
r

c
o
n
t
a
i
n
i
n
g

t
h
e

e
i
g
h
t

p
a
i
r
s

o
I

I
i
n
g
e
r
p
r
i
n
t
s

t
h
a
t

t
h
e
y

t
h
e
m
s
e
l
v
e
s

h
a
d

j
u
d
g
e
d

i
n

t
h
e

p
a
s
t
.

T
h
e

e
i
g
h
t

p
a
i
r
s

o
I

I
i
n
g
e
r
p
r
i
n
t
s

i
n

e
a
c
h

I
o
l
d
e
r

w
e
r
e

c
o
u
n
t
e
r
b
a
l
a
n
c
e
d
.

C
o
u
n
t
e
r
b
a
l
a
n
c
e

p
r
e
s
e
n
t
a
t
i
o
n
s

a
s
s
u
r
e

t
h
a
t

r
e
s
u
l
t
s

a
r
e

n
o
t

d
u
e

t
o

o
r
d
e
r

a
I
I
e
c
t
s

a
n
d

c
r
o
s
s
-
c
o
n
t
a
m
i
n
a
t
i
o
n

b
e
t
w
e
e
n

t
h
e

d
i
I
I
e
r
e
n
t

c
o
n
d
i
t
i
o
n
s
,

b
e
c
a
u
s
e

t
h
e

o
r
d
e
r

o
I

p
r
e
s
e
n
t
a
t
i
o
n

i
s

s
y
s
t
e
m
a
t
i
c
a
l
l
y

p
e
r
m
u
t
a
t
e
d

a
c
r
o
s
s

p
a
r
t
i
c
i
p
a
n
t
s

|
1
1
|
.
P
r
o
c
e
d
u
r
e
P
a
r
t
i
c
i
p
a
n
t
s

w
e
r
e

a
p
p
r
o
a
c
h
e
d

b
y

t
h
e

d
i
r
e
c
t
o
r

o
r

h
e
a
d

o
I

t
h
e

l
a
b
o
r
a
t
o
r
y

o
r

b
u
r
e
a
u

a
n
d

w
e
r
e

a
s
k
e
d

t
o

p
r
o
v
i
d
e

o
p
i
n
i
o
n
s

o
n

a

v
a
r
i
e
t
y

o
I

l
a
t
e
n
t

p
r
i
n
t
s

a
n
d

t
h
e
i
r

c
o
m
p
a
r
i
s
o
n
s

t
o

t
e
n
p
r
i
n
t

e
x
e
m
p
l
a
r
s
.

T
h
e
y

w
e
r
e

t
o
l
d

t
h
a
t

t
h
e

c
o
n
c
l
u
s
i
o
n
s

t
h
e
y

r
e
a
c
h
e
d

a
I
t
e
r

t
h
e

e
x
a
m
i
n
a
t
i
o
n

w
o
u
l
d

b
e

u
s
e
d

I
o
r

a
n

a
s
s
e
s
s
m
e
n
t

p
r
o
j
e
c
t
.

T
h
e
y

w
e
r
e

I
u
r
t
h
e
r

t
o
l
d

t
h
a
t

t
h
e

p
r
o
j
e
c
t

w
a
s

i
n
t
e
n
d
e
d

t
o

l
o
o
k

a
t

p
r
o
b
l
e
m
a
t
i
c

p
r
i
n
t
s

a
n
d

a
s
s
e
s
s
m
e
n
t
s
.

T
h
e
n
,

t
h
e

I
i
r
s
t

a
s
s
e
s
s
m
e
n
t
,

a
l
o
n
g

w
i
t
h

t
h
e

i
n
s
t
r
u
c
t
i
o
n

a
n
d

r
e
s
p
o
n
s
e

s
h
e
e
t
,

w
a
s

g
i
v
e
n

t
o

t
h
e

p
a
r
t
i
c
i
p
a
n
t
s
.

A
I
t
e
r

t
h
e
y

I
i
n
i
s
h
e
d

t
h
e

c
o
m
p
a
r
i
s
o
n

a
n
d

d
o
c
u
m
e
n
t
e
d

t
h
e
i
r

c
o
n
c
l
u
s
i
o
n
s

o
n

t
h
e

r
e
s
p
o
n
s
e

s
h
e
e
t
,

t
h
e

m
a
t
e
r
i
a
l
s

w
e
r
e

p
u
t

b
a
c
k

i
n
t
o

t
h
e

o
r
i
g
i
n
a
l

I
o
l
d
e
r
,

a
n
d

t
h
e

s
e
c
o
n
d

a
s
s
e
s
s
m
e
n
t

w
a
s

g
i
v
e
n

t
o

t
h
e
m
.

T
h
i
s

c
o
n
t
i
n
u
e
d

u
n
t
i
l

a
l
l

e
i
g
h
t

a
s
s
e
s
s
m
e
n
t
s

w
e
r
e

c
o
m
p
l
e
t
e
d
.
D
u
r
i
n
g

t
h
e

c
o
m
p
a
r
i
s
o
n

p
r
o
c
e
s
s
,

a
l
l

o
I

t
h
e

p
a
r
t
i
c
i
p
a
n
t
s

w
e
r
e

a
l
l
o
w
e
d

t
o

e
v
a
l
u
a
t
e

t
h
e

p
r
i
n
t
s

a
s

t
h
e
y

w
o
u
l
d

d
o

r
o
u
t
i
n
e
l
y

(
h
a
n
d
l
i
n
g

t
h
e

p
r
i
n
t
s
,

u
s
i
n
g

m
a
g
n
i
I
y
i
n
g

a
n
d

l
i
g
h
t
i
n
g

e
q
u
i
p
m
e
n
t
,

a
n
d

s
o

I
o
r
t
h
)
.

T
h
e

p
a
r
t
i
c
i
p
a
n
t
s

w
e
r
e

a
l
l
o
w
e
d

a
n

u
n
l
i
m
i
t
e
d

a
m
o
u
n
t

o
I

t
i
m
e

a
n
d

a
l
l

n
o
r
m
a
l

r
e
s
o
u
r
c
e
s

(
e
.
g
.
,

c
o
m
p
a
r
a
t
o
r
s
)

t
o

m
a
k
e

t
h
e
i
r

e
v
a
l
u
a
t
i
o
n
s
.
J
o
u
r
n
a
l

o
f

F
o
r
e
n
s
i
c

d
e
n
t
i
f
i
c
a
t
i
o
n
6
1
0

/
5
6
(
4
)
,
2
0
0
6
R
e
s
u
l
t
s
O
v
e
r
a
l
l
,

I
r
o
m

4
8

e
x
p
e
r
i
m
e
n
t
a
l

t
r
i
a
l
s
,

t
h
e

I
i
n
g
e
r
p
r
i
n
t

e
x
p
e
r
t
s

c
h
a
n
g
e
d

t
h
e
i
r

p
a
s
t

d
e
c
i
s
i
o
n
s

o
n

s
i
x

p
a
i
r
s

o
I

I
i
n
g
e
r
p
r
i
n
t
s

(
T
a
b
l
e

1
,

F
i
g
u
r
e

2
)
.

T
h
e

s
i
x

i
n
c
o
n
s
i
s
t
e
n
t

d
e
c
i
s
i
o
n
s

(
1
2
)

i
n
c
l
u
d
e
d

t
h
e

2
4

c
o
n
t
r
o
l

t
r
i
a
l
s

t
h
a
t

d
i
d

n
o
t

h
a
v
e

a
n
y

c
o
n
t
e
x
t
u
a
l

m
a
n
i
p
u
l
a
t
i
o
n
.

F
r
o
m

t
h
e

2
4

e
x
p
e
r
i
m
e
n
t
a
l

t
r
i
a
l
s

t
h
a
t

i
n
c
l
u
d
e
d

t
h
e

c
o
n
t
e
x
t
u
a
l

m
a
n
i
p
u
l
a
t
i
o
n
,

t
h
e

I
i
n
g
e
r
p
r
i
n
t

e
x
p
e
r
t
s

c
h
a
n
g
e
d

I
o
u
r

o
I

t
h
e
i
r

p
a
s
t

d
e
c
i
s
i
o
n
s
,

t
h
u
s

m
a
k
i
n
g

1
6
.
6

i
n
c
o
n
s
i
s
t
e
n
t

d
e
c
i
s
i
o
n
s

t
h
a
t

w
e
r
e

d
u
e

t
o

b
i
a
s
i
n
g

c
o
n
t
e
x
t
.

T
h
e

i
n
c
o
n
s
i
s
t
e
n
t

d
e
c
i
s
i
o
n
s

w
e
r
e

s
p
r
e
a
d

b
e
t
w
e
e
n

t
h
e

p
a
r
t
i
c
i
p
a
n
t
s
.

(
T
h
e

i
n
c
o
n
s
i
s
t
e
n
t

d
e
c
i
s
i
o
n
s

w
e
r
e

b
y

I
o
u
r

o
I

t
h
e

s
i
x

e
x
p
e
r
t
s
,

b
u
t

o
n
e

e
x
p
e
r
t

m
a
d
e

t
h
r
e
e

i
n
c
o
n
s
i
s
t
e
n
t

d
e
c
i
s
i
o
n
s

w
h
i
l
e

e
a
c
h

o
I

t
h
e

o
t
h
e
r

t
h
r
e
e

m
a
d
e

o
n
l
y

o
n
e

i
n
c
o
n
s
i
s
t
e
n
t

d
e
c
i
s
i
o
n
.
)

O
n
l
y

o
n
e
-
t
h
i
r
d

o
I

t
h
e

p
a
r
t
i
c
i
p
a
n
t
s

(
t
w
o

o
u
t

o
I

s
i
x
)

r
e
m
a
i
n
e
d

e
n
t
i
r
e
l
y

c
o
n
s
i
s
t
e
n
t

a
c
r
o
s
s

t
h
e

e
i
g
h
t

e
x
p
e
r
i
m
e
n
t
a
l

t
r
i
a
l
s
.

1
2
3
4
5
6
7
8
P
a
s
t

D
e
c
i
s
i
o
n
i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n
i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n
i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n
i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n
e
x
c
l
u
s
i
o
n
e
x
c
l
u
s
i
o
n
e
x
c
l
u
s
i
o
n
e
x
c
l
u
s
i
o
n
L
e
v
e
l

o
I

D
i
I
I
i
c
u
l
t
y
d
i
I
I
i
c
u
l
t
d
i
I
I
i
c
u
l
t
n
o
t

d
i
I
I
i
c
u
l
t
n
o
t

d
i
I
I
i
c
u
l
t
d
i
I
I
i
c
u
l
t
d
i
I
I
i
c
u
l
t
n
o
t

d
i
I
I
i
c
u
l
t
n
o
t

d
i
I
I
i
c
u
l
t
C
o
n
t
e
x
t
u
a
l

I
n
I
o
r
m
a
t
i
o
n

n
o
n
e
s
u
g
g
e
s
t

e
x
c
l
u
s
i
o
n
n
o
n
e
s
u
g
g
e
s
t

e
x
c
l
u
s
i
o
n
n
o
n
e
s
u
g
g
e
s
t
i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n
n
o
n
e
s
u
g
g
e
s
t
i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n
E
x
p
e
r
t

A
c
o
n
s
i
s
t
e
n
t
c
o
n
s
i
s
t
e
n
t
c
o
n
s
i
s
t
e
n
t
c
o
n
s
i
s
t
e
n
t
c
o
n
s
i
s
t
e
n
t
c
o
n
s
i
s
t
e
n
t
c
o
n
s
i
s
t
e
n
t
c
o
n
s
i
s
t
e
n
t
E
x
p
e
r
t

B
c
h
a
n
g
e

t
o

e
x
c
l
u
s
i
o
n
c
o
n
s
i
s
t
e
n
t
c
o
n
s
i
s
t
e
n
t
c
o
n
s
i
s
t
e
n
t
c
o
n
s
i
s
t
e
n
t
c
o
n
s
i
s
t
e
n
t
c
o
n
s
i
s
t
e
n
t
c
o
n
s
i
s
t
e
n
t
E
x
p
e
r
t

C
c
o
n
s
i
s
t
e
n
t
c
h
a
n
g
e

t
o

e
x
c
l
u
s
i
o
n
c
o
n
s
i
s
t
e
n
t
c
o
n
s
i
s
t
e
n
t
c
o
n
s
i
s
t
e
n
t
c
o
n
s
i
s
t
e
n
t
c
o
n
s
i
s
t
e
n
t
c
o
n
s
i
s
t
e
n
t
E
x
p
e
r
t

D
c
o
n
s
i
s
t
e
n
t
c
h
a
n
g
e

t
o

e
x
c
l
u
s
i
o
n
c
o
n
s
i
s
t
e
n
t
c
h
a
n
g
e

t
o

e
x
c
l
u
s
i
o
n
c
h
a
n
g
e

t
o

i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n
c
o
n
s
i
s
t
e
n
t
c
o
n
s
i
s
t
e
n
t
c
o
n
s
i
s
t
e
n
t
E
x
p
e
r
t

E
c
o
n
s
i
s
t
e
n
t
c
h
a
n
g
e

t
o

c
a
n
n
o
t

d
e
c
i
d
e
c
o
n
s
i
s
t
e
n
t
c
o
n
s
i
s
t
e
n
t
c
o
n
s
i
s
t
e
n
t
c
o
n
s
i
s
t
e
n
t
c
o
n
s
i
s
t
e
n
t
c
o
n
s
i
s
t
e
n
t
E
x
p
e
r
t

F
c
o
n
s
i
s
t
e
n
t
c
o
n
s
i
s
t
e
n
t
c
o
n
s
i
s
t
e
n
t
c
o
n
s
i
s
t
e
n
t
c
o
n
s
i
s
t
e
n
t
c
o
n
s
i
s
t
e
n
t
c
o
n
s
i
s
t
e
n
t
c
o
n
s
i
s
t
e
n
t
T
a
b
l
e

1
R
e
s
u
l
t
s

o
f

f
i
n
g
e
r
p
r
i
n
t

e
x
a
m
i
n
a
t
i
o
n
s
.
J
o
u
r
n
a
l

o
f

F
o
r
e
n
s
i
c

d
e
n
t
i
f
i
c
a
t
i
o
n
5
6

(
4
)
,

2
0
0
6

\

6
1
1

C
h
a
n
g
e

t
o

e
x
c
l
u
s
i
o
n

C
h
a
n
g
e

t
o

i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n

C
h
a
n
g
e

t
o

c
a
n
n
o
t

d
e
c
i
d
e
F
i
g
u
r
e

2

B
r
e
a
k
d
o
w
n

o
f

i
n
c
o
n
s
i
s
t
e
n
t

d
e
c
i
s
i
o
n
s
.
A

I
u
r
t
h
e
r

e
x
a
m
i
n
a
t
i
o
n

o
I

t
h
e

i
n
c
o
n
s
i
s
t
e
n
t

d
e
c
i
s
i
o
n
s

r
e
v
e
a
l
e
d

t
h
a
t

t
h
e
y

w
e
r
e

m
o
s
t

c
o
m
m
o
n

i
n

t
h
e

m
o
r
e

d
i
I
I
i
c
u
l
t

d
e
c
i
s
i
o
n
s
.

I
n

m
o
s
t

c
a
s
e
s

(
I
i
v
e

o
u
t

o
I

s
i
x
)
,

t
h
e

i
n
c
o
n
s
i
s
t
e
n
t

d
e
c
i
s
i
o
n
s

w
e
r
e

m
a
d
e

i
n

t
h
e

d
i
I
I
i
c
u
l
t

c
o
m
p
a
r
i
s
o
n
s
.

N
e
v
e
r
t
h
e
l
e
s
s
,

i
n
c
o
n
s
i
s
t
e
n
t

d
e
c
i
s
i
o
n
s

w
e
r
e

n
o
t

t
o
t
a
l
l
y

l
i
m
i
t
e
d

t
o

t
h
e

d
i
I
I
i
c
u
l
t

c
o
m
p
a
r
i
s
o
n
s
;

o
n
e

o
I

t
h
e

s
i
x

w
a
s

i
n

t
h
e

r
e
l
a
t
i
v
e
l
y

n
o
t

d
i
I
I
i
c
u
l
t

c
o
m
p
a
r
i
s
o
n
s
.
T
h
e

i
n
c
o
n
s
i
s
t
e
n
t

d
e
c
i
s
i
o
n
s

w
e
r
e

m
o
s
t

p
r
e
v
a
l
e
n
t

i
n

p
a
s
t

i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n

d
e
c
i
s
i
o
n
s

(
I
i
v
e

o
u
t

o
I

t
h
e

s
i
x

i
n
c
o
n
s
i
s
t
e
n
t

d
e
c
i
s
i
o
n
s
;

t
h
e
y

w
e
r
e

c
h
a
n
g
e
d

t
o

e
i
t
h
e
r

e
x
c
l
u
s
i
o
n
,

i
n

I
o
u
r

c
a
s
e
s
,

o
r

t
o

c
a
n
n
o
t

d
e
c
i
d
e
,

i
n

o
n
e

c
a
s
e
)
.

H
o
w
e
v
e
r
,

a

c
o
n
I
l
i
c
t
i
n
g

j
u
d
g
m
e
n
t

a
l
s
o

o
c
c
u
r
r
e
d

i
n

a

p
a
s
t

e
x
c
l
u
s
i
o
n

d
e
c
i
s
i
o
n

(
n
o
w

c
h
a
n
g
e
d

t
o

a
n

i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n
)
.
F
i
n
a
l
l
y
,

t
w
o

i
n
c
o
n
s
i
s
t
e
n
t

d
e
c
i
s
i
o
n
s

w
e
r
e

p
r
e
s
e
n
t

i
n

t
h
e

c
o
n
t
r
o
l

c
o
n
d
i
t
i
o
n

t
h
a
t

h
a
d

n
o

b
i
a
s
i
n
g

c
o
n
t
e
x
t
u
a
l

i
n
I
o
r
m
a
t
i
o
n
.

T
h
e
s
e

d
e
c
i
s
i
o
n
s

c
o
n
s
t
i
t
u
t
e
d

3
3
.
3

o
I

t
h
e

c
o
n
I
l
i
c
t
i
n
g

d
a
t
a

(
t
w
o

o
u
t

o
I

s
i
x
)

a
n
d

8
.
3

o
I

t
h
e

c
o
n
t
r
o
l

d
a
t
a

(
t
w
o

o
u
t

o
I

2
4
)
.

F
u
r
t
h
e
r

s
t
a
t
i
s
t
i
c
a
l

a
n
a
l
y
s
e
s

a
n
d

p
r
o
c
e
d
u
r
e
s

(
i
n
I
e
r
e
n
t
i
a
l

s
t
a
t
i
s
t
i
c
s
,

e
I
I
e
c
t

s
i
z
e
s

t
h
a
t

d
i
r
e
c
t
l
y

c
o
m
p
u
t
e
d

r
e
t
e
s
t

r
e
l
i
a
b
i
l
i
t
y

r
s
a
m
p
l
e
,

a
n
d

r
e
q
u
i
v
a
l
e
n
t

t
h
a
t

s
p
e
c
i
I
i
c
a
l
l
y

t
a
k
e
s

i
n
t
o

a
c
c
o
u
n
t

t
h
e

s
a
m
p
l
e

s
i
z
e
,

a
s

w
e
l
l

a
s

m
e
t
a
-
a
n
a
l
y
t
i
c

m
e
a
s
u
r
e
s
)

a
r
e

a
l
l

c
o
n
s
i
d
e
r
e
d

i
n

m
o
r
e

s
t
a
t
i
s
t
i
c
a
l

d
e
t
a
i
l

a
n
d

t
e
c
h
n
i
c
a
l

I
a
s
h
i
o
n

e
l
s
e
w
h
e
r
e

|
1
2
|
.

J
o
u
r
n
a
l

o
f

F
o
r
e
n
s
i
c

d
e
n
t
i
f
i
c
a
t
i
o
n
6
1
2

/
5
6
(
4
)
,
2
0
0
6
D
i
s
c
u
s
s
i
o
n
E
x
p
e
r
t
s
,

j
u
s
t

l
i
k
e

a
l
l

h
u
m
a
n
s
,

a
r
e

b
o
u
n
d

b
y

t
h
e

w
a
y

t
h
e

m
i
n
d

a
n
d

t
h
e

b
r
a
i
n

w
o
r
k
.

O
n
e

o
I

t
h
e

m
o
s
t

n
o
t
a
b
l
e

c
h
a
r
a
c
t
e
r
i
s
t
i
c
s

o
I

h
u
m
a
n
s

i
s

t
h
e

a
c
t
i
v
e

a
n
d

d
y
n
a
m
i
c

n
a
t
u
r
e

i
n

w
h
i
c
h

w
e

p
r
o
c
e
s
s

i
n
I
o
r
m
a
t
i
o
n

|
8
|
.

T
h
i
s

I
u
n
d
a
m
e
n
t
a
l

c
o
g
n
i
t
i
v
e

a
r
c
h
i
t
e
c
t
u
r
e

e
n
a
b
l
e
s

u
s

t
o

d
e
a
l

w
i
t
h

v
a
s
t

a
m
o
u
n
t
s

o
I

i
n
I
o
r
m
a
t
i
o
n

a
n
d

i
s

t
h
e

b
a
s
i
s

o
I

h
u
m
a
n

i
n
t
e
l
l
i
g
e
n
c
e
.

H
o
w
e
v
e
r
,

i
t

c
a
n

a
l
s
o

a
I
I
e
c
t

a
n
d

d
i
s
t
o
r
t

w
h
a
t

w
e

s
e
e

(
o
r

w
h
y

a
n
d

w
h
a
t

w
e

d
o

n
o
t

s
e
e
)
,

h
o
w

w
e

a
s
s
e
s
s

a
n
d

e
v
a
l
u
a
t
e

v
i
s
u
a
l

i
n
I
o
r
m
a
t
i
o
n
,

a
n
d

o
u
r

d
e
c
i
s
i
o
n

m
a
k
i
n
g

|
7
,

1
3
|
.
F
i
n
g
e
r
p
r
i
n
t

a
n
d

o
t
h
e
r

I
o
r
e
n
s
i
c

e
x
p
e
r
t
s

a
r
e

n
o
t

i
m
m
u
n
e

t
o

s
u
c
h

p
s
y
c
h
o
l
o
g
i
c
a
l

a
n
d

c
o
g
n
i
t
i
v
e

I
a
c
t
o
r
s
.

R
e
s
e
a
r
c
h
i
n
g

a
n
d

u
n
d
e
r
s
t
a
n
d
i
n
g

t
h
e
m

b
e
t
t
e
r

i
s

a

n
e
c
e
s
s
a
r
y

s
t
e
p

t
o
w
a
r
d

d
e
a
l
i
n
g

w
i
t
h

a
n
d

m
i
n
i
m
i
z
i
n
g

s
u
c
h

i
n
I
l
u
e
n
c
e
s
.

P
r
e
v
i
o
u
s

r
e
s
e
a
r
c
h

h
a
s

d
e
m
o
n
s
t
r
a
t
e
d

t
h
a
t

u
n
d
e
r

c
o
n
d
i
t
i
o
n
s

o
I

r
e
l
a
t
i
v
e
l
y

e
x
t
r
e
m
e

a
n
d

r
a
r
e

e
x
t
r
a
n
e
o
u
s

c
o
n
t
e
x
t
u
a
l

i
n
I
o
r
m
a
t
i
o
n
,

I
i
n
g
e
r
p
r
i
n
t

e
x
p
e
r
t
s

m
a
y

c
h
a
n
g
e

t
h
e

w
a
y

t
h
e
y

c
o
m
p
a
r
e

a
n
d

j
u
d
g
e

I
i
n
g
e
r
p
r
i
n
t
s

|
1
0
|
.

T
h
e

e
x
t
r
a
n
e
o
u
s

c
o
n
t
e
x
t
u
a
l

i
n
I
o
r
m
a
t
i
o
n

e
I
I
e
c
t

c
a
u
s
e
d

s
o
m
e

e
x
p
e
r
t
s

t
o

r
e
a
c
h

d
i
I
I
e
r
e
n
t

a
n
d

i
n
c
o
n
s
i
s
t
e
n
t

d
e
c
i
s
i
o
n
s

t
o

t
h
o
s
e

t
h
e
y

h
a
d

m
a
d
e

i
n

t
h
e

p
a
s
t

o
n

t
h
e

v
e
r
y

s
a
m
e

p
a
i
r

o
I

I
i
n
g
e
r
p
r
i
n
t
s
.

T
h
e

r
e
s
e
a
r
c
h

r
e
p
o
r
t
e
d

i
n

t
h
e

p
r
e
s
e
n
t

s
t
u
d
y

r
e
p
l
i
c
a
t
e
d

a
n
d

e
x
p
a
n
d
e
d

o
u
r

p
r
e
v
i
o
u
s

I
i
n
d
i
n
g
s

a
n
d

I
o
u
n
d

t
h
a
t

a
c
r
o
s
s

e
i
g
h
t

c
o
m
p
a
r
i
s
o
n
s

m
a
d
e

b
y

e
a
c
h

o
I

t
h
e

s
i
x

p
a
r
t
i
c
i
p
a
n
t
s
,

t
w
o

t
h
i
r
d
s

o
I

t
h
e

I
i
n
g
e
r
p
r
i
n
t

e
x
p
e
r
t
s

m
a
d
e

i
n
c
o
n
s
i
s
t
e
n
t

d
e
c
i
s
i
o
n
s

t
o

t
h
o
s
e

t
h
e
y

h
a
d

m
a
d
e

i
n

t
h
e

p
a
s
t

o
n

t
h
e

s
a
m
e

p
a
i
r
s

o
I

p
r
i
n
t
s
.

T
h
e

I
i
n
d
i
n
g
s

o
I

t
h
i
s

s
t
u
d
y

n
o
t

o
n
l
y

I
u
r
t
h
e
r

s
u
b
s
t
a
n
t
i
a
t
e

t
h
e

v
u
l
n
e
r
a
b
i
l
i
t
y

o
I

e
x
p
e
r
t
s

t
o

c
o
n
t
e
x
t
u
a
l

e
I
I
e
c
t
s

w
i
t
h
i
n

a

l
a
r
g
e
r

d
a
t
a

s
e
t
,

b
u
t

t
h
e
y

I
u
r
t
h
e
r

c
o
n
t
r
i
b
u
t
e

t
o

o
u
r

u
n
d
e
r
s
t
a
n
d
i
n
g

o
I

t
h
i
s

p
h
e
n
o
m
e
n
o
n
.
O
u
r

d
a
t
a

d
e
m
o
n
s
t
r
a
t
e

t
h
a
t

I
i
n
g
e
r
p
r
i
n
t

e
x
p
e
r
t
s

w
e
r
e

v
u
l
n
e
r
a
b
l
e

t
o

b
i
a
s
i
n
g

i
n
I
o
r
m
a
t
i
o
n

w
h
e
n

t
h
e
y

w
e
r
e

p
r
e
s
e
n
t
e
d

w
i
t
h
i
n

r
e
l
a
t
i
v
e
l
y

r
o
u
t
i
n
e

d
a
y
-
t
o
-
d
a
y

c
o
n
t
e
x
t
s
,

s
u
c
h

a
s

c
o
r
r
o
b
o
r
a
t
i
v
e

(
o
r

c
o
n
I
l
i
c
t
i
n
g
)

e
v
i
d
e
n
c
e

o
I

c
o
n
I
e
s
s
i
o
n

t
o

t
h
e

c
r
i
m
e
.

T
h
u
s
,

c
o
n
t
e
x
t
u
a
l

i
n
I
o
r
m
a
t
i
o
n

d
o
e
s

n
o
t

n
e
e
d

t
o

b
e

e
x
t
r
e
m
e

a
n
d

u
n
i
q
u
e

t
o

i
n
I
l
u
e
n
c
e

e
x
p
e
r
t
s

i
n

t
h
e
i
r

I
i
n
g
e
r
p
r
i
n
t

e
x
a
m
i
n
a
t
i
o
n

a
n
d

j
u
d
g
e
m
e
n
t
.
V
a
r
y
i
n
g

t
h
e

l
e
v
e
l
s

o
I

d
i
I
I
i
c
u
l
t
y

i
n

c
o
m
p
a
r
i
n
g

t
h
e

p
r
i
n
t
s

d
e
m
o
n
s
t
r
a
t
e
s

t
h
a
t

p
s
y
c
h
o
l
o
g
i
c
a
l

a
n
d

c
o
g
n
i
t
i
v
e

v
u
l
n
e
r
a
b
i
l
i
t
i
e
s

a
r
e

m
o
s
t

p
r
o
n
o
u
n
c
e
d

i
n

t
h
e

d
i
I
I
i
c
u
l
t

c
a
s
e
s
.

H
o
w
e
v
e
r
,

o
u
r

d
a
t
a

a
l
s
o

s
h
o
w

t
h
a
t

s
u
c
h

v
u
l
n
e
r
a
b
i
l
i
t
i
e
s

c
a
n

a
l
s
o

o
c
c
u
r

a
n
d

c
l
o
u
d

j
u
d
g
e
m
e
n
t

i
n

n
o
n
d
i
I
I
i
c
u
l
t

c
a
s
e
s
,

b
e
c
a
u
s
e

o
u
r

c
o
n
t
r
a
d
i
c
t
o
r
y

I
i
n
d
i
n
g
s

w
e
r
e

n
o
t

l
i
m
i
t
e
d

t
o

o
n
l
y

d
i
I
I
i
c
u
l
t

c
o
m
p
a
r
i
s
o
n
s
.

J
o
u
r
n
a
l

o
f

F
o
r
e
n
s
i
c

d
e
n
t
i
f
i
c
a
t
i
o
n
5
6

(
4
)
,

2
0
0
6

\

6
1
3
P
r
e
v
i
o
u
s

r
e
s
e
a
r
c
h

h
a
s

o
n
l
y

e
x
a
m
i
n
e
d

w
h
e
t
h
e
r

p
a
s
t

d
e
c
i
s
i
o
n
s

a
r
e

s
u
s
c
e
p
t
i
b
l
e

t
o

c
h
a
n
g
e

w
h
e
n

t
h
e

p
a
s
t

d
e
c
i
s
i
o
n
s

w
e
r
e

i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n
s

|
1
0
|
.

I
t

s
e
e
m
s

t
h
a
t

t
h
e

t
h
r
e
s
h
o
l
d

t
o

m
a
k
e

a

d
e
c
i
s
i
o
n

o
I

e
x
c
l
u
s
i
o
n

i
s

l
o
w
e
r

t
h
a
n

t
h
a
t

t
o

m
a
k
e

a

d
e
c
i
s
i
o
n

o
I

i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n
.

I
n
d
e
e
d

o
u
r

d
a
t
a

s
u
p
p
o
r
t

t
h
i
s

c
l
a
i
m
,

a
s

r
e
I
l
e
c
t
e
d

b
y

t
h
e

I
a
c
t

t
h
a
t

m
o
s
t

o
I

t
h
e

c
o
n
I
l
i
c
t
i
n
g

d
e
c
i
s
i
o
n
s

w
e
r
e

p
a
s
t

i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n
s
.

W
e

d
i
d
,

h
o
w
e
v
e
r
,

o
b
s
e
r
v
e

a

c
a
s
e

i
n

w
h
i
c
h

a
n

e
x
c
l
u
s
i
o
n

d
e
c
i
s
i
o
n

w
a
s

n
o
w

j
u
d
g
e
d

t
o

b
e

a
n

i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n
.

T
h
i
s

r
e
l
a
t
e
s

t
o

t
h
e

d
e
c
i
s
i
o
n
-
m
a
k
i
n
g

m
o
d
e
l

u
s
e
d

b
y

e
x
p
e
r
t
s

i
n

t
h
e

I
i
n
g
e
r
p
r
i
n
t

d
o
m
a
i
n
.

C
h
a
n
g
e
s

i
n

d
e
c
i
s
i
o
n
s

m
a
y

r
e
I
l
e
c
t

c
h
a
n
g
e
s

i
n

d
e
c
i
s
i
o
n

t
h
r
e
s
h
o
l
d
s

o
r

c
h
a
n
g
e
s

o
I

t
h
e

d
e
c
i
s
i
o
n

s
t
r
a
t
e
g
y

i
t
s
e
l
I
.

T
h
e

I
o
r
m
e
r

r
e
I
l
e
c
t
s

c
h
a
n
g
e
s

i
n

d
e
c
i
s
i
o
n

c
r
i
t
e
r
i
a

w
i
t
h
i
n

a

s
i
n
g
l
e

s
t
r
a
t
e
g
y

w
h
e
r
e
a
s

t
h
e

l
a
t
t
e
r

r
e
I
l
e
c
t
s

m
o
d
i
I
y
i
n
g

(
o
r

t
o
t
a
l
l
y

a
b
a
n
d
o
n
i
n
g
)

t
h
e

d
e
c
i
s
i
o
n

s
t
r
a
t
e
g
y

a
n
d

r
e
p
l
a
c
i
n
g

i
t

w
i
t
h

a

d
i
I
I
e
r
e
n
t

s
t
r
a
t
e
g
y
.

E
x
a
m
i
n
i
n
g

t
h
e

d
e
c
i
s
i
o
n
s

t
h
e
m
s
e
l
v
e
s

a
n
d

h
o
w

t
h
e
y

c
h
a
n
g
e

c
a
n

r
e
v
e
a
l

w
h
i
c
h

o
c
c
u
r
r
e
d
;

h
o
w
e
v
e
r
,

t
h
i
s

n
e
e
d
s

t
o

b
e

d
o
n
e

c
a
r
e
I
u
l
l
y
,

b
e
c
a
u
s
e

c
h
a
n
g
e
s

i
n

t
h
r
e
s
h
o
l
d
s

a
n
d

d
e
c
i
s
i
o
n

s
t
r
a
t
e
g
i
e
s

c
a
n

y
i
e
l
d

s
i
m
i
l
a
r

(
a
n
d

e
v
e
n

i
d
e
n
t
i
c
a
l
)

d
e
c
i
s
i
o
n

c
h
a
n
g
e
s

|
1
3
|
.
F
i
n
a
l
l
y
,

q
u
i
t
e

s
u
r
p
r
i
s
i
n
g

a
n
d

a
l
a
r
m
i
n
g

a
r
e

t
h
e

d
a
t
a

p
o
i
n
t
s

o
I

i
n
c
o
n
s
i
s
t
e
n
t

d
e
c
i
s
i
o
n
s

m
a
d
e

i
n

o
u
r

c
o
n
t
r
o
l

'
c
o
n
t
e
x
t
-
I
r
e
e

c
o
n
d
i
t
i
o
n
.

T
h
i
s

m
a
y

r
e
I
l
e
c
t

t
h
a
t

e
x
p
e
r
t

d
e
c
i
s
i
o
n
s

a
r
e

i
n
c
o
n
s
i
s
t
e
n
t

a
c
r
o
s
s

t
i
m
e

r
e
g
a
r
d
l
e
s
s

o
I

c
o
n
t
e
x
t
.

I
I

t
h
i
s

i
s

i
n
d
e
e
d

t
h
e

c
a
s
e
,

t
h
e
n

I
u
r
t
h
e
r

r
e
s
e
a
r
c
h

n
e
e
d
s

t
o

e
x
a
m
i
n
e

t
h
e

r
o
o
t

o
I

t
h
i
s

o
b
s
e
r
v
a
t
i
o
n
:

I
s

t
h
i
s

d
u
e

t
o

t
h
e

a
c
c
u
m
u
l
a
t
i
o
n

o
I

e
x
p
e
r
t
i
s
e

a
n
d

e
x
p
e
r
i
e
n
c
e

g
a
i
n
e
d

b
e
t
w
e
e
n

t
h
e

I
i
r
s
t

a
n
d

s
e
c
o
n
d

e
x
p
o
s
u
r
e
s

(
a
n
d

t
h
i
s

l
e
d

t
o

c
h
a
n
g
e

i
n

d
e
c
i
s
i
o
n

t
h
r
e
s
h
o
l
d

o
r

i
n

d
e
c
i
s
i
o
n

s
t
r
a
t
e
g
y

i
t
s
e
l
I
)
,

o
r

i
s

s
o
m
e

o
t
h
e
r

e
x
p
l
a
n
a
t
i
o
n

t
h
e

c
a
u
s
e
?

S
u
c
h

i
n
c
o
n
s
i
s
t
e
n
c
y

d
o
e
s

n
o
t

s
u
g
g
e
s
t

t
h
a
t

c
o
n
t
e
x
t
u
a
l

i
n
I
o
r
m
a
t
i
o
n

d
o
e
s

n
o
t

a
I
I
e
c
t

t
h
e

j
u
d
g
e
m
e
n
t

o
I

I
i
n
g
e
r
p
r
i
n
t

e
x
p
e
r
t
s
;

i
t

o
n
l
y

s
u
g
g
e
s
t
s

t
h
a
t

c
o
n
t
e
x
t
u
a
l

i
n
I
o
r
m
a
t
i
o
n

i
s

n
o
t

t
h
e

o
n
l
y

I
a
c
t
o
r

t
h
a
t

m
a
y

a
I
I
e
c
t

I
i
n
g
e
r
p
r
i
n
t

e
x
p
e
r
t
s
.

T
h
e

c
o
n
t
r
o
l

c
o
n
d
i
t
i
o
n

n
e
e
d
s

t
o

b
e

e
v
a
l
u
a
t
e
d

w
i
t
h

a

n
u
m
b
e
r

o
I

r
e
s
e
r
v
a
t
i
o
n
s
:

F
i
r
s
t
,

t
h
e

c
o
n
t
r
o
l

c
o
n
d
i
t
i
o
n

w
a
s

n
o
t

c
o
n
t
e
x
t

I
r
e
e
.

W
e

d
i
d

n
o
t

i
n
c
l
u
d
e

a
n
y

o
I

o
u
r

e
x
t
r
a
n
e
o
u
s

c
o
n
t
e
x
t
u
a
l
l
y

b
i
a
s
i
n
g

i
n
I
o
r
m
a
t
i
o
n
;

h
o
w
e
v
e
r
,

t
h
a
t

d
o
e
s

n
o
t

m
a
k
e

i
t

c
o
n
t
e
x
t

I
r
e
e
.

S
e
c
o
n
d
,

e
v
e
n

i
I

w
e

w
e
r
e

a
b
l
e

t
o

a
c
h
i
e
v
e

t
h
i
s

i
d
e
a
l

n
o
t
i
o
n

o
I

a

c
o
n
t
e
x
t
-
I
r
e
e

e
n
v
i
r
o
n
m
e
n
t
,

t
h
i
s

c
o
n
t
e
x
t

w
o
u
l
d

s
t
i
l
l

b
e

d
i
I
I
e
r
e
n
t

t
h
a
n

t
h
a
t

w
h
i
c
h

w
a
s

p
r
e
s
e
n
t

d
u
r
i
n
g

t
h
e

I
i
r
s
t

e
x
p
o
s
u
r
e

a
n
d

j
u
d
g
e
m
e
n
t

y
e
a
r
s

a
g
o
.
O
u
r

I
i
n
d
i
n
g
s

a
r
e

e
s
p
e
c
i
a
l
l
y

r
o
b
u
s
t

b
e
c
a
u
s
e

w
e

e
m
p
l
o
y
e
d

a

w
i
t
h
i
n
-
s
u
b
j
e
c
t

e
x
p
e
r
i
m
e
n
t
a
l

d
e
s
i
g
n
.

T
h
u
s

o
u
r

I
i
n
d
i
n
g
s

d
o

n
o
t

r
e
I
l
e
c
t

i
n
d
i
v
i
d
u
a
l

d
i
I
I
e
r
e
n
c
e
s

a
m
o
n
g

e
x
p
e
r
t
s
.

W
e

I
u
r
t
h
e
r

I
e
e
l

J
o
u
r
n
a
l

o
f

F
o
r
e
n
s
i
c

d
e
n
t
i
f
i
c
a
t
i
o
n
6
1
4

/
5
6
(
4
)
,
2
0
0
6
c
o
n
I
i
d
e
n
t

i
n

o
u
r

I
i
n
d
i
n
g
s

b
e
c
a
u
s
e

t
h
e
y

s
u
p
p
o
r
t

t
h
e

I
i
n
d
i
n
g
s

o
I

o
u
r

p
r
e
v
i
o
u
s

s
t
u
d
i
e
s

|
9
,

1
0
|

a
n
d

t
h
u
s

t
h
e

e
x
t
r
a

d
a
t
a

I
u
r
t
h
e
r

s
u
b
s
t
a
n
t
i
a
t
e

a
n
d

v
a
l
i
d
a
t
e

o
u
r

c
o
n
c
l
u
s
i
o
n
s
.

T
h
i
s

s
t
u
d
y

w
a
s

c
o
n
d
u
c
t
e
d

c
o
v
e
r
t
l
y
,

w
h
i
c
h

i
s

c
r
i
t
i
c
a
l

I
o
r

t
h
e

c
o
r
r
e
c
t

m
e
a
s
u
r
e

o
I

p
e
r
I
o
r
m
a
n
c
e
.

W
h
e
n

p
a
r
t
i
c
i
p
a
n
t
s

k
n
o
w

t
h
e
y

a
r
e

p
a
r
t
i
c
i
p
a
t
i
n
g

i
n

a

s
t
u
d
y
,

t
h
e
i
r

b
e
h
a
v
i
o
r

c
h
a
n
g
e
s

|
7
|
.

T
h
e
s
e

e
x
p
e
r
i
m
e
n
t
a
l

d
e
s
i
g
n

c
r
i
t
e
r
i
a

m
a
k
e

i
t

h
a
r
d

t
o

c
o
l
l
e
c
t

d
a
t
a

a
n
d

e
n
a
b
l
e

o
n
l
y

s
m
a
l
l

d
a
t
a

s
e
t
s
,

b
u
t

t
h
e

d
a
t
a

a
r
e

m
e
a
n
i
n
g
I
u
l

a
n
d

s
t
a
t
i
s
t
i
c
a
l
l
y

p
o
w
e
r
I
u
l

|
1
1
,

1
2
|
.
T
h
i
s

e
n
t
i
r
e

a
r
e
a

o
I

r
e
s
e
a
r
c
h

i
s

n
e
w

i
n

t
h
e

I
o
r
e
n
s
i
c

s
c
i
e
n
c
e
s

a
n
d

h
a
s

r
a
r
e
l
y

b
e
e
n

c
o
n
s
i
d
e
r
e
d

b
e
I
o
r
e
.

T
h
e
r
e
I
o
r
e
,

s
u
c
h

s
t
u
d
i
e
s

c
o
n
s
t
i
t
u
t
e

t
h
e

i
n
i
t
i
a
t
i
o
n

o
I

a

r
e
s
e
a
r
c
h

p
r
o
g
r
a
m

t
h
a
t

i
s

a
i
m
e
d

a
t

e
x
a
m
i
n
i
n
g

t
h
e

p
s
y
c
h
o
l
o
g
i
c
a
l

a
n
d

c
o
g
n
i
t
i
v
e

e
l
e
m
e
n
t
s

t
h
a
t

a
r
e

i
n
v
o
l
v
e
d

i
n

I
i
n
g
e
r
p
r
i
n
t

a
n
d

o
t
h
e
r

I
o
r
e
n
s
i
c

i
d
e
n
t
i
I
i
c
a
t
i
o
n
s
.

W
e

h
o
p
e

t
h
a
t

t
h
e
s
e

I
i
n
d
i
n
g
s

w
i
l
l

c
o
n
t
r
i
b
u
t
e

t
o

b
e
t
t
e
r

s
e
l
e
c
t
i
o
n
,

b
e
t
t
e
r

t
r
a
i
n
i
n
g
,

a
n
d

b
e
t
t
e
r

p
r
o
c
e
d
u
r
e
s

I
o
r

w
o
r
k

i
n

t
h
i
s

d
o
m
a
i
n
.

H
o
w
e
v
e
r
,

s
u
c
h

a
n

e
n
d
e
a
v
o
r

t
o

d
e
a
l

w
i
t
h

a
n
d

m
i
n
i
m
i
z
e

t
h
e
s
e

v
u
l
n
e
r
a
b
i
l
i
t
i
e
s

i
s

d
e
p
e
n
d
e
n
t

o
n

t
h
e

c
o
o
p
e
r
a
t
i
o
n

o
I

I
i
n
g
e
r
p
r
i
n
t

e
x
p
e
r
t
s

w
o
r
l
d
w
i
d
e
.
F
u
r
t
h
e
r

r
e
s
e
a
r
c
h

i
s

n
e
e
d
e
d

i
n

a

w
i
d
e

r
a
n
g
e

o
I

i
s
s
u
e
s

p
e
r
t
a
i
n
i
n
g

t
o

i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n
.

W
i
t
h
i
n

t
h
e

i
s
s
u
e
s

r
a
i
s
e
d

i
n

t
h
i
s

p
a
p
e
r
,

a
d
d
i
t
i
o
n
a
l

d
a
t
a

m
a
y

s
h
e
d

l
i
g
h
t

o
n

t
h
e

c
h
a
r
a
c
t
e
r
i
s
t
i
c
s

o
I

t
h
e

e
x
p
e
r
t
s

w
h
o

w
e
r
e

i
m
m
u
n
e

t
o

o
u
r

c
o
n
t
e
x
t
u
a
l

m
a
n
i
p
u
l
a
t
i
o
n
s
,

i
s
s
u
e
s

p
e
r
t
a
i
n
i
n
g

t
o

t
h
e

c
i
r
c
u
m
s
t
a
n
c
e
s

i
n

w
h
i
c
h

t
h
e

m
a
n
i
p
u
l
a
t
i
o
n
s

w
e
r
e

m
o
r
e

(
o
r

l
e
s
s
)

e
I
I
e
c
t
i
v
e
,

a
s

w
e
l
l

a
s

a
d
d
i
t
i
o
n
a
l

i
s
s
u
e
s
.

R
e
s
e
a
r
c
h

n
e
e
d
s

t
o

e
x
a
m
i
n
e

p
s
y
c
h
o
l
o
g
i
c
a
l

a
n
d

c
o
g
n
i
t
i
v
e

i
n
I
l
u
e
n
c
e
s

i
n

a
l
l

t
h
e

s
t
a
g
e
s

t
h
a
t

l
e
a
d

t
o

d
e
c
i
s
i
o
n
s

i
n

c
o
m
p
a
r
i
n
g

I
i
n
g
e
r
p
r
i
n
t
s
,

I
r
o
m

I
e
a
t
u
r
e

p
e
r
c
e
p
t
i
o
n

a
n
d

s
e
l
e
c
t
i
o
n
,

t
o

e
v
a
l
u
a
t
i
o
n

a
n
d

c
o
m
p
a
r
i
s
o
n
,

t
o

t
h
e

I
i
n
a
l

i
n
t
e
r
p
r
e
t
a
t
i
o
n

a
n
d

w
e
i
g
h
t
i
n
g

o
I

a
l
t
e
r
n
a
t
i
v
e

c
h
o
i
c
e
s

t
h
a
t

d
e
t
e
r
m
i
n
e

t
h
e

d
e
c
i
s
i
o
n

o
u
t
c
o
m
e
.

A
d
d
i
t
i
o
n
a
l

p
e
r
s
p
e
c
t
i
v
e
s

I
o
r

I
u
t
u
r
e

r
e
s
e
a
r
c
h

i
n

t
h
e

I
i
n
g
e
r
p
r
i
n
t

a
n
d

o
t
h
e
r

I
o
r
e
n
s
i
c

d
o
m
a
i
n
s

r
e
l
a
t
e

t
o

v
e
r
i
I
i
c
a
t
i
o
n
,

s
e
l
e
c
t
i
o
n

a
n
d

t
r
a
i
n
i
n
g

o
I

e
x
p
e
r
t
s
,

a
n
d

i
n
t
e
g
r
a
t
i
o
n

o
I

t
e
c
h
n
o
l
o
g
y
.

T
h
e
r
e

i
s

n
o

p
o
s
s
i
b
i
l
i
t
y

o
I

1
0
0

o
b
j
e
c
t
i
v
i
t
y

|
7
,

8
,

1
4
|
,

b
u
t

t
h
e
r
e

i
s

p
o
t
e
n
t
i
a
l

I
o
r

v
e
r
y

h
i
g
h

l
e
v
e
l
s

o
I

o
b
j
e
c
t
i
v
i
t
y
.

H
o
w

r
a
r
e

a
n
d

u
n
d
e
r

w
h
a
t

c
o
n
d
i
t
i
o
n
s

e
r
r
o
r
s

o
c
c
u
r

a
t

a

p
r
a
c
t
i
c
a
l

l
e
v
e
l

i
s

s
t
i
l
l

u
n
c
l
e
a
r

a
t

t
h
i
s

s
t
a
g
e
.

E
x
p
e
r
t
s
,

a
s

h
u
m
a
n
s
,

a
r
e

p
r
o
n
e

t
o

e
r
r
o
r
s
;

h
o
w
e
v
e
r
,

w
i
t
h

p
r
o
p
e
r

r
e
s
e
a
r
c
h

a
n
d

i
t
s

s
y
s
t
e
m
a
t
i
c

a
p
p
l
i
c
a
t
i
o
n
,

t
h
e
s
e

e
r
r
o
r
s

c
a
n

b
e

r
e
d
u
c
e
d

a
n
d

m
i
n
i
m
i
z
e
d
.
J
o
u
r
n
a
l

o
f

F
o
r
e
n
s
i
c

d
e
n
t
i
f
i
c
a
t
i
o
n
5
6

(
4
)
,

2
0
0
6

\

6
1
5
A
c
k
n
o
w
l
e
d
g
m
e
n
t
s
F
i
r
s
t
,

w
e

w
o
u
l
d

l
i
k
e

t
o

a
c
k
n
o
w
l
e
d
g
e

a
n
d

t
h
a
n
k

o
u
r

c
o
l
l
a
b
o
r
a
t
o
r
s

i
n

t
h
e

I
i
n
g
e
r
p
r
i
n
t

w
o
r
l
d
,

b
o
t
h

t
h
o
s
e

p
a
r
t
i
c
i
p
a
t
i
n
g

i
n

o
u
r

s
t
u
d
i
e
s

a
n
d

t
h
o
s
e

h
e
l
p
i
n
g

u
s

i
n

c
o
n
d
u
c
t
i
n
g

o
u
r

r
e
s
e
a
r
c
h
.

W
e

w
o
u
l
d

a
l
s
o

l
i
k
e

t
o

t
h
a
n
k

A
i
l
s
a

P
e
r
o
n

I
o
r

d
i
s
c
u
s
s
i
n
g

d
i
I
I
e
r
e
n
t

a
s
p
e
c
t
s

o
I

t
h
i
s

w
o
r
k

a
n
d

A
l
a
n

M
c
R
o
b
e
r
t
s
,

R
o
b
e
r
t

R
o
s
e
n
t
h
a
l
,

A
r
i
e

Z
e
e
l
e
n
b
e
r
g
,

C
e
d
r
i
c

N
e
u
m
a
n
n
,

J
o
h
n

V
a
n
d
e
r
k
o
l
k
,

a
n
d

a
n
o
n
y
m
o
u
s

r
e
v
i
e
w
e
r
s

I
o
r

c
o
m
m
e
n
t
s

o
n

e
a
r
l
i
e
r

v
e
r
s
i
o
n
s

o
I

t
h
i
s

p
a
p
e
r
.

F
o
r

I
u
r
t
h
e
r

i
n
I
o
r
m
a
t
i
o
n
,

p
l
e
a
s
e

c
o
n
t
a
c
t
:

D
r
.

I
t
i
e
l

D
r
o
r

S
c
h
o
o
l

o
I

P
s
y
c
h
o
l
o
g
y

U
n
i
v
e
r
s
i
t
y

o
I

S
o
u
t
h
a
m
p
t
o
n

S
o
u
t
h
a
m
p
t
o
n

S
O
1
7

1
B
J

E
n
g
l
a
n
d

U
n
i
t
e
d

K
i
n
g
d
o
m

4
4

2
3
-
8
0
5
9
4
5
1
9

i
d
e
c
s
.
s
o
t
o
n
.
a
c
.
u
k

w
w
w
.
e
c
s
.
s
o
t
o
n
.
a
c
.
u
k
/
~
i
d
/
R
e
f
e
r
e
n
c
e
s
1
.

D
r
o
r
,

I
.

E
.
;

K
o
s
s
l
y
n
,

S
.

M
.
;

W
a
a
g
,

W
.

V
i
s
u
a
l
-
S
p
a
t
i
a
l

A
b
i
l
i
t
i
e
s

o
I

P
i
l
o
t
s
.

J
.

A
p
p
l
.

P
s
v
c
h
o
l
o
g
v

1
9
9
3
,

7
8

(
5
)
,

7
6
3
-
7
7
3
.
2
.

R
u
d
i
n
,

N
.
;

I
n
m
a
n
,

K
.

F
i
n
g
e
r
p
r
i
n
t
s

i
n

P
r
i
n
t
,

t
h
e

S
e
q
u
e
l
.

T
h
e

C
A
C

N
e
w
s
,

2
n
d

Q
u
a
r
t
e
r

2
0
0
5
,

p
p

6
-
1
0
.
3
.

D
r
o
r
,

I
.

E
.

T
e
c
h
n
o
l
o
g
y

a
n
d

H
u
m
a
n

E
x
p
e
r
t
i
s
e
:

S
o
m
e

D
o
`
s

a
n
d

D
o
n
`
t
s
.

B
i
o
m
e
t
r
i
c

T
e
c
h
n
o
l
o
g
v

T
o
d
a
v

2
0
0
5
,

1
3

(
9
)
,

7
-
9
.
4
.

S
t
a
c
e
y

R
.

B
.

R
e
p
o
r
t

o
n

t
h
e

E
r
r
o
n
e
o
u
s

F
i
n
g
e
r
p
r
i
n
t

I
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n

i
n

t
h
e

M
a
d
r
i
d

T
r
a
i
n

B
o
m
b
i
n
g

C
a
s
e
.

J
.

F
o
r
.

I
d
e
n
t
.

2
0
0
4
,

5
4

(
6
)
,

7
0
6
-
7
1
8
.
5
.

C
o
l
e
,

S
.

A
.

M
o
r
e

t
h
a
n

Z
e
r
o
:

A
c
c
o
u
n
t
i
n
g

I
o
r

E
r
r
o
r

i
n

L
a
t
e
n
t

F
i
n
g
e
r
p
r
i
n
t

I
d
e
n
t
i
I
i
c
a
t
i
o
n
.

J
.

C
r
i
m
.

L
a
w

&

C
r
i
m
i
n
o
l
o
g
v

2
0
0
5
,

9
5

(
3
)
,

9
8
5
-
1
0
7
8
.
6
.

R
i
s
i
n
g
e
r
,

D
.

M
.
;

S
a
k
s
,

M
.

J
.
;

T
h
o
m
p
s
o
n
,

W
.

C
.
;

R
o
s
e
n
t
h
a
l
,

R
.

T
h
e

D
a
u
b
e
r
t
/
K
u
m
h
o

I
m
p
l
i
c
a
t
i
o
n
s

o
I

O
b
s
e
r
v
e
r

E
I
I
e
c
t
s

i
n

F
o
r
e
n
s
i
c

S
c
i
e
n
c
e
:

H
i
d
d
e
n

P
r
o
b
l
e
m
s

o
I

E
x
p
e
c
t
a
t
i
o
n

a
n
d

S
u
g
g
e
s
t
i
o
n
.

C
a
l
i
f
.

L
a
w

R
e
v
.

2
0
0
2
,

9
0

(
1
)
,

1
-
5
6
.
7
.

D
r
o
r
,

I
.

E
.

P
e
r
c
e
p
t
u
a
l
,

C
o
g
n
i
t
i
v
e
,

a
n
d

P
s
y
c
h
o
l
o
g
i
c
a
l

E
l
e
m
e
n
t
s

I
n
v
o
l
v
e
d

i
n

E
x
p
e
r
t

I
d
e
n
t
i
I
i
c
a
t
i
o
n
.

F
r
i
c
t
i
o
n

R
i
d
g
e

S
o
u
r
c
e
b
o
o
k
,

i
n

p
r
e
s
s
.
8
.

D
r
o
r
,

I
.

E
.

P
e
r
c
e
p
t
i
o
n

i
s

F
a
r

F
r
o
m

P
e
r
I
e
c
t
i
o
n
:

T
h
e

R
o
l
e

o
I

t
h
e

B
r
a
i
n

a
n
d

M
i
n
d

i
n

C
o
n
s
t
r
u
c
t
i
n
g

R
e
a
l
i
t
i
e
s
.

B
r
a
i
n

a
n
d

B
e
h
a
v
i
o
u
r
a
l

S
c
i
e
n
c
e
s

2
0
0
5
,

2
8

(
6
)
,

7
6
3
.
J
o
u
r
n
a
l

o
f

F
o
r
e
n
s
i
c

d
e
n
t
i
f
i
c
a
t
i
o
n
6
1
6

/
5
6
(
4
)
,
2
0
0
6
9
.

D
r
o
r
,

I
.

E
.
;

P
e
r
o
n
,

A
.
;

H
i
n
d
,

S
.
;

C
h
a
r
l
t
o
n
,

D
.

W
h
e
n

E
m
o
t
i
o
n
s

G
e
t

t
h
e

B
e
t
t
e
r

o
I

U
s
:

T
h
e

E
I
I
e
c
t

o
I

C
o
n
t
e
x
t
u
a
l

T
o
p
-
D
o
w
n

P
r
o
c
e
s
s
i
n
g

o
n

M
a
t
c
h
i
n
g

F
i
n
g
e
r
p
r
i
n
t
s
.

A
p
p
l
.

C
o
g
n
i
t
i
v
e

P
s
v
c
h
o
l
o
g
v

2
0
0
5
,

1
9

(
6
)
,

7
9
9
-
8
0
9
.
1
0
.

D
r
o
r
,

I
.

E
.
;

C
h
a
r
l
t
o
n
,

D
.
;

P
e
r
o
n

A
.

C
o
n
t
e
x
t
u
a
l

I
n
I
o
r
m
a
t
i
o
n

R
e
n
d
e
r
s

E
x
p
e
r
t
s

V
u
l
n
e
r
a
b
l
e

t
o

M
a
k
i
n
g

E
r
r
o
n
e
o
u
s

I
d
e
n
t
i
I
i
c
a
t
i
o
n
s
.

F
o
r
.

S
c
i
.

I
n
t
e
r
.

2
0
0
6
,

1
5
6

(
1
)
,

7
4
-
7
8
.
1
1
.

R
o
s
e
n
t
h
a
l
,

R
.
;

R
o
s
n
o
w
,

R
.

L
.

E
s
s
e
n
t
i
a
l
s

o
f

B
e
h
a
v
i
o
u
r
a
l

R
e
s
e
a
r
c
h
.

M
e
t
h
o
d
s

a
n
d

D
a
t
a

A
n
a
l
v
s
i
s
;

M
c
G
r
a
w
-
H
i
l
l
:

N
e
w

Y
o
r
k
,

1
9
9
1
.
1
2
.

D
r
o
r
,

I
.

E
.
;

R
o
s
e
n
t
h
a
l
,

R
.

M
e
t
a
-
A
n
a
l
y
t
i
c
a
l
l
y

Q
u
a
n
t
i
I
y
i
n
g

t
h
e

R
e
l
i
a
b
i
l
i
t
y

a
n
d

B
i
a
s
a
b
i
l
i
t
y

o
I

F
i
n
g
e
r
p
r
i
n
t

E
x
p
e
r
t
s
`

D
e
c
i
s
i
o
n

M
a
k
i
n
g
.

(
S
u
b
m
i
t
t
e
d

I
o
r

p
u
b
l
i
c
a
t
i
o
n
.
)
1
3
.

D
r
o
r
,

I
.

E
.
;

B
u
s
e
m
e
y
e
r
,

J
.

R
.
;

B
a
s
o
l
a
,

B
.

D
e
c
i
s
i
o
n

M
a
k
i
n
g

U
n
d
e
r

T
i
m
e

P
r
e
s
s
u
r
e
:

A
n

I
n
d
e
p
e
n
d
e
n
t

T
e
s
t

o
I

S
e
q
u
e
n
t
i
a
l

S
a
m
p
l
i
n
g

M
o
d
e
l
s
.

M
e
m
o
r
v

a
n
d

C
o
g
n
i
t
i
o
n

1
9
9
9
,

2
7

(
4
)
,

7
1
3
-
7
2
5
.

1
4
.

H
o
I
s
t
a
d
t
e
r
,

D
.

R
.

G
d
e
l
,

E
s
c
h
e
r
,

B
a
c
h
.

A
n

E
t
e
r
n
a
l

G
o
l
d
e
n

B
r
a
i
d
;

B
a
s
i
c

B
o
o
k
s
:

N
e
w

Y
o
r
k
,

1
9
7
9
.

Preliminary communication
Contextual information renders experts vulnerable
to making erroneous identications
Itiel E. Dror
*
, David Charlton, Ailsa E. Peron
School of Psychology, Faculty of Medicine, Health and Life Sciences,
University of Southampton, Southampton SO17 1BJ, UK
Received 1 June 2005; received in revised form 17 October 2005; accepted 17 October 2005
Available online 1 December 2005
Abstract
We investigated whether experts can objectively focus on feature information in ngerprints without being misled by
extraneous information, such as context. We took ngerprints that have previously been examined and assessed by latent print
experts to make positive identication of suspects. Then we presented these same ngerprints again, to the same experts, but
gave a context that suggested that they were a no-match, and hence the suspects could not be identied. Within this new context,
most of the ngerprint experts made different judgements, thus contradicting their own previous identication decisions.
Cognitive aspects involved in biometric identication can explain why experts are vulnerable to make erroneous identications.
# 2005 Elsevier Ireland Ltd. All rights reserved.
Keywords: Psychology; Cognition; Erroneous identication; Bias; Extraneous information; Contextual inuence; Fingerprints
1. Introduction
Being a scientist or forensic expert is rooted in the ability
to examine evidence reliably and objectively. To do this,
these professionals must be able to dissociate themselves
from extraneous contexts and other inuences that may
interfere with their ability to examine, evaluate, and judge
the relevant information. Their decisions should be based on
the information relevant to the task at hand and its unbiased
interpretation. This involves independent thought that
ignores to a large extent extraneous pressures and inuences.
External pressures and inuences are many and varied.
The history of science is full of examples of extraneous
inuences, and today too, scientists work within, and are
inuenced by, political, economical and other agendas (e.g.,
global warming, genetically modied crops, and measles
mumps rubella vaccine).
Terrorism has brought about a wave of contextual inu-
ences. These include, among others, heightened suspicion of
Muslims, fear, anger, helplessness, as well as pressure on
governments to control (or at least appear to control) such
threats. Such contextual inuences provide strong and ample
opportunities to contaminate objectivity, leading to distor-
tions and errors of judgement beyond the unavoidable.
Indeed, within this context we have witnessed major mis-
evaluations and misjudgements by intelligence experts.
Within a similar extraneous context the United States
Federal Bureau of Investigation (FBI) positively, but erro-
neously, identied a Muslim as the Madrid bomber (see
Fig. 1). This incorrect identication was further veried by a
number of FBI and other ngerprint experts and led to the
arrest of an innocent person. It was only due to rare and
exceptional circumstances that this error was ever revealed
and eventually acknowledged by the FBI [1]. Errors can
occur across forensic science evidence, including DNA [2].
Empirical cognitive research in these areas has been
largely neglected (if not basically ignored), partially because
professional expert assessment of evidence (as in the crim-
inal justice system) is believed to be relatively objective.
* Corresponding author. Tel.: +44 23 80594519;
fax: +44 23 80594518.
E-mail address: id@ecs.soton.ac.uk (I.E. Dror).
URL: www.ecs.soton.ac.uk/$id
0379-0738/$ see front matter # 2005 Elsevier Ireland Ltd. All rights reserved.
doi:10.1016/j.forsciint.2005.10.017
With the growing number of anecdotal cases that question
this belief and suggest that forensic assessment is far from
being as objective as it can and should be, it is important to
conduct cognitive scientic research in this area. Laboratory
experiments performed by our group have already suggested
that emotional context may bias ngerprint identication.
These studies found that university students were more
likely to judge that there was a positive match between
pairs of ngerprints that were presented within an emotional
context than those presented within an emotionally neutral
control context [e.g., 3]. However, this vulnerability was
apparent only when the prints were ambiguous and lacked
clarity. The emotional context had minimal effect when
there was a clearly matching pair (or a clearly non-matching
pair). These studies, however, were based on non-experts
and conducted in a laboratory setting.
The study that we report here presents empirical data on
whether actual ngerprint experts in their normal everyday
working routines and environment are susceptible to extra-
neous contextual inuences. We employed a within-subject
design in which the same experts made judgements on
identical pairs of ngerprints, but in different contexts.
Our aim was to focus on and to examine the contextual
inuences themselves rather than reveal possible individual
differences between experts. Accordingly, we collected and
used pairs of ngerprints from archives that the same experts
had examined and judged approximately 5 years earlier as a
clear and denite match. These previous identication
matches were taken from real criminal investigations.
In this study, we re-presented these very same pairs of
ngerprints to the same experts who had originally evaluated
them as a match, but we now provided them within an
extraneous context that might bias them to evaluate the
prints as a non-match. We wanted to test whether their
decisions were independent and relatively objective, and
thus consistent regardless of extraneous inuences. Alter-
natively, if they contradicted their previous decisions, this
would demonstrate vulnerability to bias.
2. Method
2.1. Participants
Participants were ve ngerprint experts. Together they
represent over 85 years of experience in examining nger-
prints (mean of 17 years). The participants were taken from
our international ngerprint expert pool of volunteers. This
pool of participants includes ngerprint experts from a
variety of Fingerprint Bureaus, Agencies, and Laboratories
from across the world (including the USA, UK, Israel, The
Netherlands and Australia). We only used experts who were
not familiar with Mayelds ngerprint and from whom we
could covertly access past archival identication matches
that they made in the past (see Section 2.3)
2.2. Materials
A different pair of ngerprints was prepared for each of
the expert participants. Each pair of prints had been pre-
viously identied as a match by that same expert in the year
2000, within the normal course of their work. The latent
ngerprints had been obtained from the crime scenes and
were all presented again to the experts in their original
format.
We further established that all of the pairs of ngerprints
were indeed a match by submitting them for verication,
context free to two experienced ngerprint experts who
were not involved in or aware of our study (each had over 20
years of experience). Both experts independently veried
that all ve pairs of ngerprints were indeed matches.
2.3. Procedure
Participants signed a consent form a few months prior
to the experiment. In this form they consented to being
tested sometime within the next 12 months without their
knowledge. Thus, we were able to obtain consent but yet
I.E. Dror et al. / Forensic Science International 156 (2006) 7478 75
Fig. 1. The FBIs erroneous identication of the Madrid bomber. The latent print from the crime scene (left panel) and the ngerprint of the
innocent suspect who was positively identied by a number of ngerprint experts (right panel).
test the experts within their normal working environment
without them knowing that they were in an experimental
situation. We pre-screened our participants and used only
participants that were not familiar with the ngerprint
of Mayeld.
Participants were asked by one of their colleagues to
examine a set of ngerprints, composed of a latent print
(fromthe crime scene) and a print exemplar (a print obtained
froma suspect). They were told that the pair of prints was the
one that was erroneously matched by the FBI as the Madrid
bomber, thus creating an extraneous context that the prints
were a non-match.
The ngerprint experts were asked to decide whether
there was sufcient information available in the pair of prints
to make a denite and sound decision, and if so, what that
judgement was (a match or non-match). They were allowed
to evaluate the prints as they would do routinely: handling of
the prints, magnifying, lighting equipment, and so forth. The
experts were allowed an unlimited amount of time to make
their evaluation. The ngerprint experts were further
instructed to ignore the context and background information,
and to just focus solely on the actual print in their evaluation
and decision-making.
3. Results
Only one participant (20%) judged the prints to be a
match, thus making a consistent identication regardless of
the extraneous context. The other four participants (80%)
changed their identication decision from the original deci-
sion they themselves had made ve years earlier. Three of
these four participants directly contradicted their previous
decision and now judged the ngerprints as denite non-
matches, whereas, the fourth participant now judged that
there was insufcient information to make a denite deci-
sion (either a match or a non-match) (Fig. 2).
4. Discussion
This study shows that ngerprint identication decisions
of experts are vulnerable to irrelevant and misleading con-
textual inuences. Our study specically demonstrates that
the extraneous context in which ngerprint examinations
occur can determine the identication decision. When pre-
sented within a different context four out of ve experts
made different identication decisions. One of the four
decided that there was insufcient information available
in the latent print to make either a match or non-match
decision, whereas, the other three ngerprint experts decided
that the ngerprints were a denite non-match. This is
striking given that all ve experts had seen the identical
ngerprints previously and all had decided that the prints
were a sound and denite match.
This is the rst research study to experimentally examine
the possible impacts of extraneous context in the real world
of biometric and forensic science. One reason for the lack of
research in this area is the difculty in conducting proper
scientic research with experts without their knowledge and
in their real working environment, while obtaining their
consent. We could only use experts for whom we could
covertly access and obtain archival les of their own past
judgements and who were not familiar with the Mayeld
ngerprint. This stipulation further decreased the availabil-
ity of suitable participants, but had the added advantage of
providing a unique opportunity to conduct a within-subject
study. The magnitude of the contextual effect and the fact
that the experts had judged the same ngerprints in the past
enabled the sample to provide clear ndings with a high level
of condence. Furthermore, given that we conducted our
experiment within the real world conditions of the criminal
justice system, even if only one expert out of ve was
susceptible to such effects that in itself would have serious
implications.
Even if we were able increase our sample of expert
participants 10-fold (which is unrealistic, given all the
constraints detailed above) and assuming that none of the
additional participants would have been vulnerable to our
manipulation and changed their judgements (which is sta-
tistically highly unlikely), our data would still demonstrate
that approximately 10% of the experts were susceptible to
misleading extraneous contextual information. Thus, our
results are striking even though we used ve expert parti-
cipants and a strong extraneous context.
The critical question is what do these results reect and
what do they imply. Are the inconsistent ngerprint identi-
cation decisions a reection of practitioners errors? Do
they reveal deeper methodological and procedural problems
in the way that ngerprint experts are trained and identica-
tions are conducted? Or do the results point out basic aws in
the scientic basis and assumptions underlying ngerprint
identication altogether?
The data presented in this study, along with some of the
rare examples where erroneous identications are publicly
Fig. 2. The covert empirical data showing that most of the expert
LPE changed their decisions when the same pair of ngerprints were
presented in a different context.
revealed and acknowledged, do not necessarily indicate
basic aws in the scientic underpinning of ngerprint
identication. The fundamental question as to whether
ngerprint identication is a science is not addressed in this
study, since that raises a different set of issues that pertain to
a variety of sciences [4]. Our results also do not reect or
reveal practitioners errors whereby experts negligence,
carelessness, and personal fault (intentional or not) produce
erroneous identications. Such causes are often used to
deect deeper scrutiny and discussion.
Rather, it seems that our ndings of inconsistent identi-
cation decisions may reect cognitive aws and limitations
in conducting objective and independent processing and
evaluation of the information. It is important to note that
such problems arise mainly in the more difcult and chal-
lenging cases, such as with latent ngerprints collected at
crime scenes that are distorted, partially missing, and con-
taminated. In such cases subjectivity is more pronounced
[3,5].
As extraneous contextual effects are more pronounced,
greater distortions can arise. The sources of such distortions
are many and varied, including emotional context, pressure,
contextual information, group think, biases, hopes and
expectations, self fullling prophecies, and peer pressure.
In this study, we used a strong misleading extraneous con-
textual inuence, but such inuences do occur.
It is important rst to establish empirically that experts
can be inuenced by extraneous contexts. Now that we have
demonstrated such an effect, further research can and should
use different and more subtle manipulations to examine in
greater depth when such factors affect performance and
render the experts vulnerable to misjudgements, and when
such factors are unlikely to affect performance (and we are
currently pursuing such research, see for example [6]).
When vulnerable, these effects can cause a variety of dis-
tortions that arise from ignoring parts of the evident infor-
mation, over-emphasising and over-evaluating other parts of
the information, and changing decision criteria, to name but
a few.
Vulnerabilities in ngerprint identication can be
minimized by better initial selection and screening of
ngerprint experts; appropriate training and professional
development, and the adoption of methodological proce-
dures that adequately address potential pitfalls. Our
results show that even in the face of strong extraneous
contextual information one expert nevertheless did main-
tain their original judgement. That expert was indeed able
to focus objectively and consistently on the data, ignoring
the extraneous misleading contextual information. This
clearly demonstrates that it is possible to be much more
objective, and that some experts may not be optimizing
objectivity.
The reliability and validity of a scientic method such as
ngerprint identication is maintained only when analysis is
relatively objective, and hence consistent, across indivi-
duals, times, and extraneous contexts. For ngerprint exam-
ination to remain a credible forensic science, it must achieve
this level of objectivity of analysis. Our study shows that it is
possible to alter identication decisions on the same nger-
print, solely by presenting it in a different context. This does
not imply that ngerprint and other forensic identications
are not a science, but it does highlight problems of sub-
jectivity, interpretation, and other psychological and cogni-
tive elements that interact and may distort any scientic
inquiries [7].
One of the main sources of weaknesses in biometric and
other forensic sciences is the lack of research, attention, and
application of psychological elements that play a key role in
the identication processes. These range from the ways in
which perceptual factors (such as similarity and orientation)
affect the process of pattern recognition [8] to how we
consider decision alternatives and shift response criteria
[9]. With new and future statistical tools and technologies
the face of ngerprint and biometric identication is chan-
ging; however, psychology and cognitive elements continue
to play a critical role in their implementation and success
[10]. To highlight and address such potential pitfalls, cog-
nitive research needs to be applied systematically to the
world of biometrics and forensics. This is all the more
necessary in view of our ndings that extraneous contextual
information is able to determine experts evaluation of
ngerprints. Given that ngerprint is a well-established
and relatively objective forensic discipline, then distorting
effects are undoubtedly as prevalent, if not more so, in other
biometrics and forensic disciplines [11].
Acknowledgments
We want to thank all the ngerprint experts who are
working together with us and support our efforts to try and
understand the cognitive elements involved in ngerprint
identication, and to thank Arie Zeelenberg, Robert
Rosenthal, and Nick Donnelly for helpful comments on
an earlier version of the manuscript.
References
[1] R.B. Stacey, Report on the erroneous ngerprint individualiza-
tion in the Madrid train bombing case, J. Forensic Identif. 54
(6) (2004) 706718.
[2] W.C. Thompson, Subjective interpretation, laboratory error
and the value of DNAevidence: three case studies, Genetica 96
(1995) 153168.
[3] I.E. Dror, A. Peron, S. Hind, D. Charlton, When emotions get
the better of us: the effect of contextual top-down processing
on matching ngerprints, Appl. Cogn. Psychol. 19 (6) (2005)
799809.
[4] I.E. Dror, R.D. Thomas, The cognitive neuroscience laboratory:
a framework for the science of mind, in: C. Erneling, D. Johnson
(Eds.), The Mind as a Scientic Object: Between Brain and
Culture, Oxford University Press, 2005, pp. 283292.
[5] I.E. Dror, Perception is far fromperfection: the role of the brain
and mind in constructing realities, Brain Behav. Sci. 28 (6)
(2005).
[6] I.E. Dror, D. Charlton, Why are Experts Prone to Error? Centre
of Visual Cognition Technical Report, University of South-
ampton, UK.
[7] I.E. Dror, Perceptual, Cognitive, and Psychological Elements
Involved in Expert Fingerprint Identication, Friction Ridge
Sourcebook, International Association for Identication, in
press.
[8] A.R.S. Ashworth, I.E. Dror, Object identication as a function
of discriminability and learning presentations: the effect
of stimulus similarity and canonical frame alignment on
aircraft identication, J. Exp. Psychol. Appl. 6 (2) (2000)
148157.
[9] I.E. Dror, J.R. Busemeyer, B. Basola, Decision making under
time pressure: an independent test of sequential sampling
models, Memory Cogn. 27 (4) (1999) 713725.
[10] I.E. Dror, Technology and human expertise: Some dos and
donts, Biometric Technol. Today 13 (9) (2005) 79.
[11] D.M. Risinger, M.J. Saks, W.C. Thompson, R. Rosenthal, The
Daubert/Kumho implications of observer effects in forensic
science: hidden problems of expectation and suggestion, Calif.
Law Rev. 90 (1) (2002) 156.
Context effects in forensic science:
A review and application of the science of science
to crime laboratory practice in the United States
MJ Saks*
Professor of Law and Psychology, Arizona State University, PO Box 877906, Tempe, AZ 85287-7906, USA
DM Risinger
Seton Hall University School of Law, Newark, NJ, USA
R Rosenthal
University of California at Riverside, and Haward University, USA
WC Thompson
Department of Criminology, Law and Society, University of California at Irvine, USA
Science & Justice 2003 43 77 - 90 Received 8 August 2002 accepted 27 January 2003
This article discusses the phenomenon of "context effects" by
reviewing the findings and practices of a range of scientific
fields, including astronomy, physics, biology, medicine, and
especially the relevant research and theory from psychology.
Context information, such as expectations about what one is
supposed to see or conclude, has been found to have a small but
relentless impact on human perception, judgment, and decision-
making. The article then considers the vulnerability of forensic
science practice to context effects, and concludes by suggesting
that forensic science adopt practices familiar in other fields of
scientific work, in particular blind or double-blind testing and
also the use of evidence line-ups.
Cet article discute le phCnomkne des effets contextuels en
passant en revue les rksultats et pratiques de plusieurs domaines
scientifiques comprenant : l'astronomie, la physique, la biologie,
la mCdecine et en particulier la thkorie et la recherche relevante
de la psychologie. L'information contextuelle, telle que les
attentes sur ce que l'on est supposk voir ou conclure, a un impact
petit mais permanent sur la perception humaine, le jugement et
la prise de decision. Cet article considkre alors la vulnCrabilitk de
la pratique des sciences forensiques aux effets dfis au contexte et
conclut en suggCrant que les sciences forensiques adoptent des
pratiques usuelles introduites dans d'autres domaines d7activitC
scientifique, en particulier l'utilisation de tests 2 l'aveugle et en
double aveugle, ainsi que l'utilisation de parades d'indices.
In diesem Review wird das Phanomen der ,,Kontext-Effekte"
vor dem Hintergrund der Erkenntnisse und Praktiken einer
Reihe von Wissenschaftsgebieten, u.a. Astronomie, Physik,
Biologie, Medizin und insbesondere der einschlagigen
Forschung und Theorie der Psychologie diskutiert. Kontext-
Informationen, wie zum Beispiel Erwartungen bzgl. der
mutmaRlichen Wahrnehmungen oder Schlussfolgerungen einer
Person, haben nach diesen Erkenntnissen einen kleinen aber
unbarmherzigen Einfluss auf die menschliche Wahrnehmung,
Beurteilung und Entscheidungsfindung. Der Artikel befasst sich
im Anschluss mit der Frage, wie anfallig forensische
Wissenschaften gegeniiber ,,Kontext-Effekten" sind und schlieRt
mit dem Vorschlag, dass sie Praktiken aus anderen
Wissenschaftsbereichen adaptieren sollten, insbesondere Blind-
oder Doppel-Blindtests und auch die Wahlgegeniiberstellung
von Beweismaterial.
Este articulo estudia el fendmeno de 10s efectos de contexto,
revisando 10s hallazgos y 10s mCtodos de una serie de campos
cientificos, que incluyen la astronomia, fisica, biologia,
medicina y especialmente la investigacidn relevante y teorfa de
la psicologia. Se ha visto que la informacidn de contexto, tal
como expectativas sobre lo que uno supone que ve o que
concluye, tiene un pequeiio per0 continuo impacto en la
percepci6n humana, el juicio y la toma de decisiones. El articulo
considera la vulnerabilidad de la prgctica cientifica forense a 10s
efectos de contexto y concluye sugiriendo que la ciencia forense
adopta practicas familiares en otros campos del trabajo cientifico
en particular anhlisis ciego o doble ciego y tambiCn la evidencia
de poner en fila.
*Author for correspondence
O The Forensic Science Society 2003
Key words Forensic science, context effects, observer
effects, expectancy effects, experimenter effects, bias,
evidence line-ups, blind testing.
science&justice Volume 43 No.2 (2003) 77 - 90 Page 77
MJ Saks, DM Risinger, R Rosenthal and WC Thompson
Context effects in forensic science
Introduction
This article reviews what has been learned in a range of
scientific fields about 'context effects' or 'observer effects,'
considers their likely impact on forensic science examinations,
and suggests some procedural remedies for eliminating these
undesirable influences. Context effects reflect the psychological
principle that the desires and expectations with which people
approach the task of observation measurably affect their
perceptions and interpretations of what they observe. The results
of observation are, to a degree, a function not only of the
characteristics of the things being observed but also of the state
of the observer and the context in which the observation takes
place. The knowledge base which informs us about the nature of
these effects, their power, their limits and methods for their
elimination, consists of work in the history of science [e.g.,
1,2,3] and reports of practitioners [e.g., 4,5,6], as well as
systematic empirical studies [7,8,9], often true experiments [e.g.
10,11,12], and meta-analyses [13,14] or other syntheses [15,16]
of those systematic empirical studies. This body of knowledge
has sometimes been referred to as 'the science of science.'
Forensic science is one of only a few fields that has not yet
profited from the science of science, nor has it conducted studies
to investigate these phenomena in forensic practice. But effects
found in every field that has made the effort to look are not
effects from which forensic science, uniquely, is likely to be
immune. The most obvious danger in forensic science is that an
examiner's observations and inferences will be influenced by
extraneous, potentially biasing information which has been
learned by the examiner in the course of working on a case. In
addition, there are other and more subtle error-producing sources
of expectation as well. Where most scientific fields have adopted
a number of strategies for eliminating context effects, most
conspicuously blind testing regimes, forensic scientists have not
progressed beyond trying to will the problem away by directing
themselves not to allow these influences to affect their judgment
[17, Provision II.F.1. Though the problem cannot be willed away,
it is susceptible to procedural solutions.
Sensitivity to the problems of context effects has become
integral to the modem scientific method. Soon after Renaissance
natural philosophers began thinking about the scientific method,
they began paying attention to the problems of observer effects.
Sir Francis Bacon in 1620 suggested that, '[tlhe human
understanding, when any proposition has once been laid down ...
forces everything else to add fresh support and confirmation ....'
p. 110 [18]. Bacon anticipated what modern research has shown
to be the cognitive phenomenon of biased assimilation, the
tendency of observers to accept more readily information that
supports existing beliefs than information that challenges those
beliefs. He also anticipated what modem cognitive scientists
refer to as confirmation bias, the tendency to test a hypothesis by
looking for instances that confirm it rather than by searching for
potentially falsifying instances (even though most scientists and
philosophers of science agree that the better method is to
proceed by seeking falsification).
The first recorded instance of a scientist recognizing that the
accuracy of specific observations was influenced by attributes of
the observer occurred more than 200 years ago. In 1795
Maskelyne, Astronomer Royal at the Greenwich Observatory,
realized that he and his assistant Kinnebrook, using identical
methods, were obtaining different results for the times of stellar
transits [I]. In the 1820s, Bessel, an astronomer at Konigsberg,
studied the problem and found that such differences were not
only common, but in such tasks as astronomical measurement
they reflected predictable individual tendencies. By the 1830s
astronomers had developed a method for calculating 'personal
equations' which enabled them to measure these kinds of
observer error and remove the distorting effects from their
findings.
Not all observer effects can be so easily corrected but, since that
time, scientists of nearly all kinds have grown alert to context
factors that can distort their findings and produce misleading
conclusions. These realizations and attention to them have
evolved into a 'science of science,' the careful study of the
causes of both random and systematic errors of observation and
methods for preventing them [15]. These methods and
procedures can be seen at work in university, medical, and
industry laboratories where most scientists in most fields
routinely carry out their work in ways calculated to minimize the
impact of such distorting effects. Today, awareness of these
problems and their solutions is so widespread that concepts such
as 'double-blind' and 'placebo' have become household words.
What this article is not about
Let us be clear about what problems are not the direct concern of
this article. When we talk about distortions due to extraneous
influences, we are not talking about deliberate falsification,
where forensic scientists report inculpatory results when the
findings actually were exculpatory or inconclusive, or report
inculpatory findings when they have conducted no examinations
at all, of which there are numerous documented examples
[19,20,21,22]. In addition to conscious falsification, there are
other sources of error beyond the scope of this article. We will
not discuss the invention and use of novel but unvalidated
techniques by maverick forensic scientists [19,23]. Nor, finally,
will we consider ordinary incompetence, that is, forensic
scientists who simply do not know how to do their work properly
and as a result unintentionally reach erroneous conclusions.
The focus of this article is on a far more pervasive but generally
unnoticed source of error. Consequently, the problem to which
this article is directed is in some respects more troublesome and
troubling than any of those mentioned above. This is because
context effects permitted to run uncontrolled through forensic
practice can lead competent and honest forensic scientists using
well validated techniques to offer sincere conclusions that are,
nevertheless, distorted and inaccurate. Moreover, such results
may occur in large numbers, completely without examiners'
awareness, much less with any wrongful intent. Indeed, such
distortions will be more ubiquitous and more insidious precisely
because they are not intended and their presence goes unnoticed.
The psychology of context effects
Observer effects
An elementary principle of psychology is that context and
Page 78 science&justice volume 43 No.2 (2003) 77 - 90
expectations influence people's perceptions and interpretations
of what they observe. There are several terms which refer to this
basic phenomenon, or particular aspects of it - observer effects,
context effects, expectancy effects, cueing, top-down
processing, perceptual set, and others - depending upon details
of the process (e.g., whether the impact is occurring during or
after perception), or its setting, or the theoretical model offered
to explain the phenomenon. In this article, we will use the term
'observer effects' to denote the general phenomenon, with other
terms used to elucidate particular aspects of the general
phenomenon.
At the most general level, observer effects are errors of
apprehension, recording, recall, computation, or interpretation
resulting from some trait or state of the observer. A simple
illustration of this phenomenon is provided in Figure 1. Whether
the character in the centre is perceived to be the letter B or the
number 13 depends upon the context in which it is viewed. This
illustrates how information from the context enables an observer
to resolve the ambiguous symbol into one or the other
possibility. Figure 2 presents a somewhat more complex
illustration. What people see in the middle drawings depends
upon the order in which they examine the series of drawings.
People who begin at the right see the middle drawings as a
woman's figure; people who begin at the left see the middle
drawings as a man's face. (Whether that resolution is 'correct' or
not is a separate matter.) Systematic research on such effects in
a wide variety of settings has advanced our understanding of the
phenomenon and led to solutions to some of the problems
created by it.
Very often observer effects result from expectations about what
is to be learned from an observation, and such expectations can
come either from explicit messages or subtle cues about the
thing to be observed. For example, a pathologist who is told she
is being presented with a slide of abnormal cells is more likely
to conclude that she is seeing abnormal cells than one who is told
she is being presented with a slide of normal cells. The term
observer error refers to errors that are randomly distributed, and
Figure 1 Observer effects illustrated: whether the
character in the centre is perceived to be the
letter B or the number 13 depends upon the
context in which it is viewed.
therefore self-canceling over the long run, while the term
observer bias refers to errors that are not random but systematic
[15]. Note, however, that random observer error may be a
serious problem in any process which is not cumulative, but
which acts on each individual result. It is of no comfort to
individual patients if a pathologist 'in the long run' makes as
many errors calling normal cells cancerous as she does calling
cancer cells normal. Indeed, in principle, biased error of a known
and stable amount is actually easier to deal with, since it can be
corrected for, while random error cannot. The personal equations
of 19th century astronomers were mechanisms to correct for
stable biased error.
None of this is to say that people simply 'see what they want to
see.' The cognitive process underlying observer effects is best
understood as a cyclical interplay between pre-existing schemata
and the uptake of new information. Schemata are mental
categories constructed from experience and belief which provide
the framework for perception and reasoning. Without schemata
to organize and order perception and inference, the world of
perception would remain William James's 'bloomin' buzzin'
confusion.' However, schemata not only facilitate meaningful
perception, they also limit it. The cognitive scientist Ulric
Neisser [24] explains this mildly paradoxical aspect of
meaningful perception thus:
Perception does not merely serve to confirm preexisting
assumptions, but to provide organisms with new information.
Although this is true, it is also true that without some
preexisting structure, no information could be acquired at all.
There is a dialectical contradiction between these two
requirements: we cannot perceive unless we anticipate, but
we must not see only what we anticipate. If we were restricted
to separate and isolated glances at the world, this
contradiction would prove fatal. Under such conditions, we
could not consistently disentangle what we see from what we
expect to see, nor distinguish objects from hallucinations.
This dilemma ... can be resolved in the perceptual cycle.
Although a perceiver always has at least some (more or less
specific) anticipation before he begins to pick up information
about a given object, they can be corrected as well as
sharpened in the course of looking.
The upshot of the argument is that perception is directed by
bias and expectation but not completely controlled by them;
it involves the pickup of real information. Schemata [stored
frameworks of knowledge or belief] exert their effects by
selecting some kinds of information rather than others, not by
Figure 2 Observer effects illustrated: whether the middle
drawings are perceived to be a man's head or
the body of a woman depends upon whether
the observer begins viewing the series from the
left or from the right.
manufacturing false percepts or illusions ... If the environment
is rich enough to support more than one alternative view (and
it usually is), expectations can have cumulative effects on
what is perceived that are virtually irreversible ... The
interplay between schema and situation means that neither
determines the course of perception alone. [24, p. 43451
While schemata may be stubbornly fixed in many dimensions in
adults, and voluntarily revisable in those dimensions only with
effort and training, in certain ways they are adjustable right
down to the point of perception. The context of perception,
including such things as emotional involvement and exterior
suggestion, can set and tune by expectation the way in which
schemata are brought to bear, not only on perception, but on the
recall of the events of perception. These processes can occur
before, during, and after observation. As we approach an
occasion for observation we become 'set' for what we are about
to perceive. The flexibility of the human cognitive system
permits us to 'tune' ourselves to perceive some things and ignore
other things, usually so automatically and seamlessly as to not
even realize we are doing it. For example, we can stand in a
crowd of noisy people and shift our focus from listening to one
person to listening to another. (Two other examples are Figures
I and 2, above.) 'The selection process is programmable, within
the parameters of sensory limits.' p. 50 [25]. In an illustration of
'set' effects resulting from initial expectation, experiments have
been conducted in which investigative interviewers were given
pre-interview reasons to believe or to doubt the person being
interviewed. This assignment of expectation had dramatic effects
on the interview structure, the questions posed, and other aspects
of the interviewing behavior of the investigators [16].
Expectation, whatever its source, contributes to the phenomenon
of confirmatory bias, and lays the groundwork for selective
attention to evidence. Often there is too much information for a
human to attend to all of it or to give equal consideration to all
of it. If one has certain expectations about what is happening, or
hypotheses about the cause of something, one tends to draw
selectively from the available evidence (e.g., a physician looking
at signs, symptoms, and test results while trying to diagnose a
complex or subtle case) and focus on those items that confirm
the working hypothesis. Even when there is time to attend to all
the evidence, people tend to give more weight to facts that are
consistent with their expectations and less to contrary evidence
[26,27,28,29,30].
Thus, expectations, among other factors, lead us to more readily
conclude that we have perceived one thing rather than another,
and having done so it becomes more difficult to perceive details
that may cut the other way. These effects can be reinforced as we
establish the initial interpretation of what we have perceived
(constructive effects), and further still when we later try to
remember what we perceived (reconstructive effects). Indeed,
there is evidence that the most powerful effects occur during the
integration and retrieval phases, as the new percepts become part
of the original schema and the schema is used to recall the
perception [31,32]. In light of this, consider the forensic scientist
who takes poor notes during an examination and prepares a
skimpy report, but then goes back to 'spruce them up' shortly
before trial [21]. Even assuming the most benign of intentions,
that examiner is inviting errors to infiltrate his conclusions and
his testimony. Reconstructive errors are given the opportunity to
manifest themselves during the 'spruce-up' stage.
One important area of research deals with how humans perceive
and process information carrying 'signal' stimuli in the presence
of non-signal stimuli (generally referred to as 'noise'). One well
established effect of expectation (however induced or derived) in
the perception tuning process is that decision thresholds shift as
a function of expectations, such that in response to identical
stimuli, a positive decision becomes more likely (and therefore
more likely to be a false positive), or less likely (and therefore
more likely to be a false negative) - purely as a consequence of
decision thresholds that change as expectations change. (For a
more extensive discussion of signal detection theory and
decision-making in forensic science, see Phillips et al. [33].)
Where the external stimuli are clear, the cognitive biases, which
operate on ambiguity, will be least able to affect the observer.
Conversely, observer effects are most potent where ambiguity is
greatest. Then judgment is more likely to succumb to
expectation or preference or utility.
Another line of relevant research on perception and recall tuning
involves what are known as anchoring effects. These are most
readily understood in the context of quantitative estimates, but
operate on any continuum, including the kind of magnitude
judgements in which forensic scientists engage. Anchoring
research shows that the estimates people make are influenced by
positions on that continuum that have been made salient by task-
irrelevant outside influences.
Even when given an anchor that is clearly made to be regarded
as a random number, the estimates people make of something are
systematically increased or decreased by that anchor value [34].
The anchors need not come from the same dimension that the
estimate of interest falls along. Arbitrary anchor values have
been shown to produce large differences in people's estimates of
various quantities. Jacowitz and Kahneman ran a series of tests
in an attempt to examine the breadth of this phenomenon [35].
They used fifteen different tasks in which different anchors were
introduced by asking respondents: Is X (length of the Mississippi
River, population of Chicago, etc., etc.) larger (or longer than or
more) than Y (either a high or low anchor value)? Respondents
were then asked to give their best estimate of the true value of X.
The anchoring effect occurred on 14 of the 15 estimation tasks
tested.
Expertise does not insulate one from the influence of anchoring
effects. For instance, in one study, experienced real estate agents
were asked to appraise a house [36]. They inspected the house
and were given all the information usually used in making such
appraisals (characteristics of the property, recent sale prices of
other houses in the community, etc.). However, they were also
told a current 'listing price' which for half the appraisers was
given as high and for half of them it was given as low. The
evaluations by the agents were strongly affected by the arbitrary
anchor (the 'listing price').
Page 80 science&justice Volume 4 3 No.2 (2003) 77 - 90
One of the most dramatic demonstrations of expert vulnerability
to anchoring effects is a study by Englisch and Mussweiler [37].
In this study, a student from a totally irrelevant field gave an
estimate of how large the solution to a problem should be to an
expert faced with deciding the problem. Some experts received
high estimates and some received low estimates. This
information, though received from a low-credibility amateur
from an irrelevant field, still was sufficient to create an anchor
which had a systematic impact on the estimates made by the
experts.
Once conclusions are reached, they can be remarkably resistant
to disconfirmation. A number of psychological studies have
demonstrated belief perseverance, the tendency for conclusions
about a proposition to persist after the information that gave rise
to them has been discredited [29, Chapter 81. This phenomenon
is another consequence of confirmatory bias. People can find
apparent confirmation for their conclusions, even when their
conclusions are incorrect, because they evaluate supporting
evidence in a biased manner - meaning simply that they give
more weight to confirming than to disconfirming information.
Hence, a conclusion reached on faulty grounds may continue to
seem correct after its initial basis has been discredited because
other evidence, viewed through the confirmatory lens, seems to
support it.
Finally, we might consider research on the tendency to conform
to the perceptions, beliefs, and behavior of others. Research on
conformity shows that people rely on the views of others in order
to reach their own views, sometimes to gain additional
information, other times merely to be in step with their peers
(termed informational and normative influences, respectively).
For example, in a classic demonstration of conformity effects,
people of differing rank were shown a highly ambiguous
stimulus (a point of light) in one another's presence, and asked
to estimate over how far a range it was moving. (This experiment
took advantage of the autokinetic effect: a perceptual illusion
whereby a stationary point of light in a completely darkened
room appears to be moving.) Although each person's perceptions
of motion range were influenced by the announced perceptions
of the others, those of lower rank were more influenced by those
of higher rank [38].
Experimenter effects
The discussion thus far has emphasized the problem of
observing inanimate objects. The objects do not change, the
states of the observers do - and in consequence of that, the
observers' apprehensions, recordings, computations, and
interpretations change. Scientists whose object of study is living
organisms have numerous additional problems. Human and
animal subjects, unlike inanimate objects, perceive the
experimenter's behavior and their own behavior is altered. The
observer unintentionally communicates something which is
responded to by the subject. A famous instance of this is 'Clever
Hans,' the famous counting horse of the early 20th century, who
was finally shown to be responding to subtle and unintentional
cues of those who watched him 'count.' [2,39].
The problem is all the more serious with that most malleable of
animals, the human being. For example, in Winkel and
Koppelaar's study of 'set' effects, pre-interview cues regarding a
suspect affected the expectations of investigators, which affected
their interviewing behavior, which affected the responses of the
interviewees [16]. The beliefs of medical researchers or
physicians or patients in the effectiveness of a treatment can
produce improvements in patients even if the treatment itself is
worthless - the well-known placebo effect [40,41]. When
teachers are told that certain students (who have been selected at
random) will blossom before the school year is over, it affects
the teacher's interactions with those students and as a
consequence the students do improve relative to control
students. In the educational setting this is termed the Pygmalion
effect after the character in the Greek myth and the play by Shaw
[121.
Much research has been directed toward understanding the
processes by which the researcher's expectancies differentially
change the researcher's (or manager's or doctor's or teacher's)
behavior toward different research participants (or workers or
patients or students), and how the participants in turn perceive
the cues and respond to them with their own changed behavior
[summarized in 13,42,43,44].
Initially, it might seem that such 'experimenter effects' would
have little relevance to forensic scientists, whose objects of
study usually are inanimate. A forensic scientist's expectations
cannot actually change the color of a paint chip, shape of a
friction ridge or the number of striations. The expectations and
context of a forensic scientist, it would seem, can change only
the perceptions of the forensic scientist. But if we consider the
larger organizational setting that is a crime laboratory, we may
fruitfully analogize it to an 'experiment,' with police
investigators, prosecutors, lab directors, and perhaps other
forensic scientists as the 'experimenters' and the forensic
examiners as the 'subjects' of the experiment. From that
perspective, the beliefs and expectancies that superiors, co-
workers, and external personnel manifest in turn affect the
behavior (observations, recordings, computations, and
interpretations) of the examiner 'subjects.' From this
perspective, the more complex 'experimenter effect' findings
become relevant to what happens in the forensic science
laboratory.
Other relevant research from the behavioral and social
sciences
Because influences on perception and judgment are phenomena
of human behavior, it is not surprising that some behavioral
scientists have made context effects of various kinds a focus of
their research. Their work has gone beyond demonstrating the
existence of observer effects and proceeded to conducting
systematic experiments designed to better understand the
conditions under which these effects occur and how to tame
them.
Where errors occur, they might be random or might be skewed
in the direction of outcomes expected or favored by the observer,
however diligently the observer is trying to report exactly what
is there. Kennedy and Uphoff studied recording errors in an
experiment on extrasensory perception [9]. In the study, 28
observers recorded 11,125 attempts to detect what another
person was trying to transmit. Of these, 126 (or 1.13%) were
misrecorded. Because 98.87% were accurately recorded, we can
infer that the observers were being honest and conscientious. But
the errors that did occur were not random. Believers in telepathy
made nearly three-quarters more errors increasing the telepathy
scores than did disbelievers, while disbelievers made twice as
many errors decreasing the telepathy scores than did believers.
Note that the items being observed in this experiment (presented
stimuli and detection attempts) are far less ambiguous than those
forensic scientists often are called upon to compare.
Can errors be induced by creating expectations in the minds of
the observers? In one study, Cordaro and Ison had observers
record the head turns and body contractions of planaria
(flatworms) [8]. A group of identical planaria was randomly
divided in half. But, for one of those halves, observers were led
to expect a high incidence of turning and contracting. For the
other half, the same observers were led to expect a low
incidence. Observers led to expect a high rate of turns and
contractions recorded twice as many head turns and three times
as many body contractions.
Rosenthal conducted a meta-analysis of 21 studies in which it
was possible to check the accuracy of the observers' recordings
of data in those studies [45]. The studies involved a range of
subject matter, including: reaction time, person perception,
human and animal learning, task ability, psychophysical
judgments, questionnaire responses, and telepathy. Together, the
21 studies involved over 300 observers making and recording
about 140,000 observations. About 1% of these observations
were recorded incorrectly, and about two-thirds of all recording
errors favored the hypothesis of the observer.
Cahen carried out an experiment in which 256 prospective
school teachers were asked to score exam booklets of children
supposedly being tested for academic readiness [lo]. Each of 30
test items was to be scored on a 4-point scale using a scoring
manual which gave examples of answers of varying quality.
Each test booklet included some 'background' information on
the child, including IQ score, the purpose of which was to create
an expectation in the grader that the child was either above
average, average, or below average in intellectual ability. The
examination scorers gave differing grades to identical
performances, differences that were correlated with the exam
graders' expectations. Note that in many everyday academic test
settings, procedures have been put in place to avoid such biasing
effects, typically blind grading of examinations.
Prior observations can affect what we think we see in subsequent
observations. 'Whenever human judges are used as the
measuring device, their calibration is subject to systematic
unconscious alterations, so that the central tendency of the
stimulus context to which they are adapted comes to appear as
neutral or intermediate, whereas the stimuli that deviate most
from this adaptation level appear most striking [contrast effect].
If in the course of judgments the central tendency of the
presented stimulus shifts, this produces a shift in judgement
standards of which the judge is unaware [adaptation level]. Such
effects have been found for every type of stimulus attribute for
which they have been examined.. .' p. 34 [46].
To this point, we have been discussing the problem of observer
effects mostly in terms of the impact of mere expectations of
what an observation is likely to disclose. There also is a literature
on 'need-determined perception,' that is, how an emotionally
heightened or 'hot' motivational state (as distinct from a 'cool'
cognitive expectation) affects what the observer perceives. If
even the mildest of expectations (laboratory assistants counting
the movements of flatworms, teachers scoring the exam
performance of students they do not know) affect perception,
then it should not be surprising to find that where an observer
has strong motivation to see something, that something has an
increased likelihood of being 'seen.' Observers sometimes have
preferences for one outcome over another in what they observe.
The conclusion from the research on need-determined
perception is that, in general, the world appears somewhat
different to people who have a desire to see it in different ways,
and how different the world appears is related to the intensity of
that desire [47,48,49,50,51].
The examples above reflect that observer effects may be
occurring at the several stages through which data progress from
the thing being observed to the conclusions drawn about what
was observed: (a) Errors can occur at the stage of initial
perception, termed errors of apprehending. (b) Or perception can
be veridical, but errors creep in at the stage where what is
observed is recorded, that is, errors of recording (assuming a
record beyond memory is made). (c) Or error rates can escalate
over time when memory is relied upon, as both desires and the
need for schematic consistency induce errors of memory. (d) Or
correct observations accurately recorded or remembered can be
transformed into incorrect results when calculations are
performed on them, that is, errors of computation. (e) Finally,
even if all else has gone well, examiners can draw incorrect
conclusions from the data, that is, errors of interpretation.
In the case of errors of interpretation, the criteria for the 'true'
values of the underlying observations often are so vague and
ephemeral, and so submerged in the interpretation, that one
cannot speak of accuracy so much as of interpretative
conclusions [IS]. It is exactly where stimuli are most on the
border of accurate perception and classification that conditions
most favor errors of interpretation. The more ambiguous and ill-
defined the stimulus, and the more frustrated or motivated the
observer, the more likely one or more observer effects will occur
[461.
Confidence, accuracy, and amount of information
Finally, research has examined the relationship among
confidence, accuracy, and amount of information. The more
information people have the more confident they are in their
decisions. But accuracy does not increase as a function of
confidence. One example is a study of horse-race handicappers.
They were asked to predict the winner and state their confidence
in the prediction. As they obtained more and more information
about the horse and rider, their confidence in their prediction
Page 82 science&justice Volume 4 3 No.2 (2003) 77 - 90
kept increasing. But their accuracy remained the same [52]. This
was apparently because the new information, though accurate,
added less to accurate decision-making than these experienced
and motivated experts believed it did. The lack of relationship
between substantial additions of information and accuracy of
result under some conditions, the direct relationship between
such information and confidence in one's conclusions, and the
resultant lack of relationship between confidence in one's
conclusion and actual accuracy, is especially troublesome in a
field, like forensic science, where subjective probability
estimates are the major conclusion being reached. Information
can expand and subjective probability estimates will go up, but
the accuracy (the objective probability) may not improve
commensurately. Indeed, if new information is sufficiently over-
valued, confidence could go up at the same time that accuracy
goes down.
Observer effects in other fields of science
Research has established the existence of context effects in
virtually every mature scientific discipline, which has led
investigators to strive to recognize the conditions that give rise
to them and to develop ways to reduce or eliminate them.
Following are illustrations from a variety of fields.
Examples from the physical sciences
One of the most famous examples of observer effects was noted
at the outset of this article, namely, the discovery in astronomy
of differences in astronomers' perceptual judgements. That
discovery, in turn, led to the development of 'personal equations'
which allowed astronomers to adjust for those individual
differences in perception and thereby achieve more accurate
observations.
Observer effects have been found in the reading of scales. That
is, people do not always read dials and other readouts correctly,
and their errors are nonrandom. Certain numbers or patterns are
more likely to be 'read' than others, resulting in systematic errors
drawn from the measuring instruments [53].
Newton failed to see and report absorption lines in the prismatic
solar spectrum, though they would have been visible with the
apparatus he was using. The most likely explanation for his
failure to see them is that he held theoretically-based
expectations that such phenomena did not exist. Because he did
not believe they existed, he failed to see them [54].
While Newton failed to see something that did exist, scientists of
the early 20th Century saw something that did not exist. First
reported by Blondlot in 1903, 'N-rays' appeared to make
reflected light more intense. So long as they were believed to
exist, the effects of N-rays were 'observed' by many scientists.
But once it was determined that N-rays did not exist, the effects
of N-rays ceased to be observed [55].
Examples from the biological sciences
For many years, laboratory technicians who counted blood cells
visually were taught that correct counting would keep blood cell
counts of successive samples within a certain range of variation.
In 1940 a method was discovered to count blood cells much
more accurately. Use of the new technology led to the
inescapable conclusion that technicians had for years been
reporting routine blood counts that were within a narrower band
of variability than could possibly have existed. Observations had
been made consistent with the expectations they had been
taught, but inconsistent with the realities of nature [7].
Mendel's counts of characteristics in pea plants came much
closer to the theoretical predictions than was possible. Mendel or
his assistant either deliberately misreported their observations or
were the victims of observer errors [3].
Medical researchers have found observer errors due to exposure
to extraneous case information in the use of stethoscopes in
cardiac diagnostics [4], in the reading of orthopedic X-rays [56],
and in the interpretation of echocardiographs. Clinical
information in the patients' charts was found to 'bias the
interpretation toward certain diagnoses, increasing the chances
of a false positive.' [57].
A marine biologist, reflecting on the observer effects he had
come upon in research on the octopus, commented that scientists
may 'equate what they think they see, and sometimes what they
want to see, with what actually happens.' p. 85 [58].
The findings and concepts described above are no less relevant
to forensic science than they are to physics, bio-medicine, and
the behavioral and social sciences. In their daily work, forensic
scientists are observers of a wide variety of objects, shapes,
colors, instrumentation, and test results. The observations to be
made present varying degrees of ambiguity. Subjective judgment
and interpretation by the human observer remain the principal
methods of reaching conclusions in most forensic disciplines.
And the working environment of the forensic scientist is not
lacking in sources of expectations or outcome preferences. Such
circumstances facilitate observer effects, particularly when
observers have armed themselves so lightly against these effects.
In what follows, we explore more concretely the environment of
the forensic scientist and the observer effects that are likely to
impinge on the examinations that are taking place.
The proper function of a forensic scientist is to give an answer to
a question appropriate to the discipline by the application of the
methods of that discipline. It is not to give an answer, even an
honest and accurate answer, to that same question by any other
means.
The reason the products of the forensic scientist's efforts are
admissible in court is not because forensic scientists are better at
drawing conclusions about the meaning of normal relevant
evidentiary information than detectives or jurors are. It is
because the law has accepted that, as to a defined area of
specialized knowledge or skill, the products of their practice are
better than the jury could do alone. But when the forensic
scientist relies on, or is influenced by, information outside of that
specialty's domain, then the forensic scientist is exceeding the
bounds authorized by the law. That is because the forensic
scientist's role is not to tell the judge or jury what they could
figure out for themselves - and especially not to hide extra-
domain evidence within what appears to be domain-appropriate
conclusions - but only to give the judge or jury the reliable
product of the forensic scientist's discipline which is beyond
what they could do on their own.
The dangers of the practice of relying on extra-domain
information are easily illustrated. Assume a bitemark on human
skin is shown to a forensic odontologist. Assume further that the
odontologist, based on the mark alone, would conclude that the
mark was a bitemark and that there was insufficient detail to
identify any particular adult as the source of the bite. However,
if the odontologist is told, and takes into account, that the
complaining witness said she was raped by a man who also bit
her, and that the DNA analysis of the sperm recovered from her
identifies the defendant to a very strong random match
probability, then the odontologist can rationally conclude that
the bitemark can be attributed to the defendant to a very high
degree of probability, but not as a result of his expertise in
forensic odontology.
On one level, there is a certain apparent backwardness to the
odontologist's reasoning, since he is using information about the
identity of the attacker to draw a conclusion about the identity of
the source of the bitemark, instead of providing a conclusion
about the source of the bitemark to be used as a basis for
inferring the identity of the attacker. However, directionality is
not the problem. The problem is a confusion of role. Bayes'
Theorem is commutative and non-directional, so that the order
of consideration of probability affecting factors is formally
irrelevant (if one is authorized by one's role to consider all
factors on the issue under examination). If the odontologist's
role were in fact to offer a conclusion about the identity of the
source of the bitemark using all available information, then his
conclusion would be beyond criticism. However, that is not his
role. He is authorized only to provide information derivable
from his discipline. If he testifies to the conclusion as described,
he would be appearing to provide the judge or jury with new and
meaningful information, while in fact all he would be doing was
repeating in a disguised form other information they already had,
resulting in a Bayesian double impact being given to the domain-
extraneous information.
The result can be an investigative echo chamber, in which a few
items of evidence reverberate and seem more numerous and
stronger than they really are. Consider a simple mathematical
illustration of the potential power of such contamination.
Suppose I believe I have seven items of evidence associating a
suspect with a crime scene. Further, assume that each of these
items of evidence is apparently independent of the others, and
each carries with it a random match probability of only .SO. But
because I have seven items of evidence which connect the
suspect to the crime, they combine to be a far more compelling
identification than any one of them alone: .SO raised to the 7th
power equals .0078 - only 78 chances in 10,000 that a person
selected at random from the population would be incriminated to
the same extent as this suspect. Human intuition corresponds at
least roughly with the mathematics, and we have the feeling that,
as each item of evidence accumulates, the likelihood of
erroneously convicting an innocent suspect decreases. But
suppose there is only one independent item of evidence, and that
all the others are the product of cross-contamination, thus: As a
consequence of an interview with a suspect, a detective comes to
have a hunch that the suspect is the perpetrator. Affected by that
officer's hunch, a handwriting examiner concludes that the
ambiguous handwriting evidence matches. An officer
conducting a line-up knows of his colleague's hunch and the
document examiner's match, and inadvertently steers an
eyewitness to select the suspect from a line-up when, had the
witness not been influenced by the officer, a positive
identification would not have been made. The bitemark expert
learns of those three items of evidence and his positive
identification is influenced by it. Before the toolmark expert
completes his examination, he knows of four other items of
'incriminating' evidence. And so on. But if each of these is
largely an echo of the initial item, then the true random match
probability would be closer to .SO than to ,0078, because each
new incriminating item is little more than a reflection of the
initial one.
Though the result is little more than an illusory consensus, it is
nevertheless a potent one. Although each examiner reached his or
her contaminated conclusion in the shadow of expectations of
what the 'correct' outcome was, the knowledge of the other
'corroborating' conclusions will reinforce the subjective
confidence each has in the accuracy of his or her own result, even
though they add nothing to the accuracy of the finding. This will
in turn yield confident testimony from each expert as a witness.
Borrowing from a more pure science context: 'Insofar as
systematic biases have been observed, they are overwhelmingly ...
a tendency to contaminate one's reports in the direction of
agreement with what others are reporting and thus fail to report
what is uniquely available from one's own perspective. In
addition, the agreement achieved represented pseudo-
confirmation. The tremendous literature on conformity and
suggestion shows how strong and persuasive this effect is. It
could scarcely fail to operate among teams of scientists.' [46]
The resulting harm can be viewed from two angles. If the
evidence is sound, and would have stood up if independently
evaluated, then the informational cross-contamination
undermines the true value of the evidence. Conversely, if the
evidence is unsound, then the informational contamination can
create the appearance of guilt out of next to nothing.
The general principle that forensic examiners should be
insulated from all information about an inquiry except necessary,
domain-specific information is not novel. For instance,
regarding handwriting identification, Hagan wrote in 1894: 'The
examiner must depend wholly upon what is seen, leaving out of
consideration all suggestions or hints from interested parties ...
Where the expert has no knowledge of the moral evidence or
aspects of the case in which signatures are a matter of contest,
there is nothing to mislead him ...' p. 82 [59]. In more recent
times, Ashbaugh [60] has stated of fingerprints that, if the
'comparison is objective, others must be capable of seeing the
physical attributes one sees. If one feels his or her objectiveness
has been compromised due to the consultation, one should ask a
Page 84 science&justice volume 43 No.2 (2003) 77 - 90
third party to carry out the verification.' p. 148. Taking a similar
tack, Smithsonian forensic anthropologist Douglas Ubelaker
describes numerous letters of transmittal he received on
submissions forwarded to him in his role as a consultant to the
FBI which routinely included extensive case information.
Appreciating the dangers of bias inherent in this information, he
adopted the practice of reading nothing except the bare
minimum necessary to log in the specimen, at least before doing
his initial examination [6]. Ubelaker lists 'being influenced by
someone else's expectations,' p. 279, as one of the three biggest
dangers in forensic practice.
But the insights just described are not widely shared in forensic
science theory or practice, and certainly few steps are taken to
prevent observer effects. First, the principle of remaining blind
to biasing cues is reflected in few if any modern textbooks for
forensic examiners. Second, the accreditation standards of the
American Society of Crime Laboratory Directors
(ASCLDILAB) do not reflect the principle or require any
specific or general review of steps taken to insure that it is
followed [61]. It is especially noteworthy that the ASCLD
Manual was substantially revised in 2000 after the Department
of Justice Inspector General's Report on FBI laboratory
practices, yet still did not address problems of biasing
information revealed in the Inspector General's Report [21].
Though significant attention is paid to preventing contamination
of evidence, none is paid to preventing contamination of
examiners. Third, there have been no formal studies on the
actual practices in forensic science laboratories which would
document the use of bias-reducing procedures. Finally, the
anecdotal evidence is extensive and uniform in indicating that
extraneous information is rife in much of forensic practice.
Such examples include: Forms for the submission of evidence
for examination at the New Jersey State Police Laboratory which
have a section marked 'comments' in which submitting local
detectives can and generally do include whatever case
information they wish, and which form is passed along to
examiners with the evidence. It is clear from the US Department
of Justice Inspector General's report that domain-irrelevant
information was routinely available to examiners in the FBI
laboratory. The Inspector General claimed to be hopeful that
ASCLD accreditation and ASCLD quality control standards
would solve these problems [21]. Testimony in the (first) World
Trade Center Bombing case by an examiner from the FBI
explosives unit identified the main charge as a urea nitrate bomb,
based not upon residues found at the scene, but 'on speculation
based on evidence linking the defendants to that explosive' [p.
111. 'Williams implicitly accepted as a premise the prosecution's
theory of guilt'[p. 129,211. In another case, another examiner
'acknowledged that his identification of PETN on the tools was
based in part on the fact that stripped detonating cord was found
in the defendant's garbage. In his interview with the Office of the
Inspector General, Rudolph observed that given this
information, he presumed the material on the knife was PETN ...
Rudolph failed to distinguish between the separate and distinct
roles of an investigator and a forensic scientist' [(emphasis
supplied), p. 30, 211. Some forensic scientists actively promote
reaching into improper domains for assistance in making the
determinations they are called upon to make in their own, proper,
domain. The Document Examiner Textbook advises: 'Before an
attempt by the examiner to identify a handwriting, the
investigator should consult and [obtain] as much circumstantial
evidence as possible about the case' [62]. Examples could
multiply.
None of this is to say that it is always an easy or trivial exercise
to formulate standards of domain-relevant information.
Nevertheless, such difficulties hardly justify the complete
absence of standards of practice that now exists. Up to now, there
has been little motivation to develop such standards, but there is
some indication that, just as Daubert v. Merrell Dow
Pharmaceuticals [63] has prompted efforts among forensic
scientists to develop validity data for some subdisciplines [64],
courts may be growing more sensitive to the distortions wrought
by various sources of observer effects, as Kumho Tire Co. v.
Carmichael [65] focuses attention on reliability under the
particular task conditions of the case. Comments to that effect
appear in US v. Fujii [66]. If the courts continue in that direction,
the development by practitioners in each forensic specialty of
appropriate and defensible standards for distinguishing domain
specific from domain irrelevant information may eventually
become a precondition to admissibility.
Sources of induced observer error in forensic science
practice
Direct communication between investigators and examiners
Transmittal letters which accompany a submission of evidence
for examination by a crime laboratory often communicate more
about the case than is required to perform the necessary
examinations. This information sometimes tells examiners about
other inculpatory evidence that has been found in the case, and
what the investigator making the submission expects or hopes
the requested tests will conclude. In one such case, a transmittal
letter led to the conviction of two innocent men who were
imprisoned for eight years until the identity of the real killer was
established by independent evidence [67,68].
Sometimes such communications are in person. Evan Hodge,
former chief of the FBI Firearms and Toolmark Unit, recounts the
occasion when a police inspector brought a Colt Arms 45 calibre
pistol to a firearms examiner at a local laboratory so that its
barrel's rifling could be compared with marks on a murder bullet.
The inspector in effect told the examiner: 'We know this guy shot
the victim and this is the gun he used; all we want you to do is
confirm what we already know so we can get the scumbag off the
street. We will wait. How quick can you do it?' [5].
On occasion, examiners interact more directly and continuously
with investigators as the evidence in a case develops. In one
published account, a serologist spoke with an investigator
'numerous times during the day, sometimes three to five calls per
hour.' 'With each passing day ... she found herself becoming
more deeply involved with the progress of the ... investigation.
Her daily conversations with [the investigator] routinely went
beyond a particular lab-related inquiry, and she found herself
asking him how his leads were developing, whether he had
received any responses to his teletypes, and so on.' p. 137 [69].
Cross-communication among examiners
In some cases, examiners in different sections of a lab speak with
each other about what their respective tests on different aspects
of the evidence in that case are showing. It is clear from the
Inspector General's investigation of practices at the FBI crime
laboratory that examiners in some units knew of the findings of
examiners in other units and tailored their own 'findings' to be
consistent with the other findings [21]. Further, it is clear that the
FBI laboratory contemplates that examiners will meet to arrive
at resolution in the event of conflicting results [21]. The inherent
dangers of such a practice should by now be apparent. Any
process for refining inquiry after the return of apparently
conflicting findings by different examiners must become
sensitive to observer effects.
A special case of mandated cross-communication involves the
'peer review' process followed in most forensic laboratories, and
mandated by ASCLD standard 1.4.2.17. While it is true that such
'peer review' is common, it is unclear what it is supposed to
accomplish. The ASCLD standard indicates that the purpose of
a laboratory's peer review process is 'to ensure that the
conclusions of its examiners are reasonable and within the
constraints of scientific knowledge.' 'Peer reviewers' may or
may not be exposed to the contaminating information that an
initial examiner was exposed to, but the reviewing examiner
typically knows the conclusions of the initial examiner - itself a
strong form of contamination. If the peer reviewer is serving
merely as a process check on the procedures used, and to make
sure the report adequately documents and explains its findings
and conclusions, then the fact that the reviewer knows the
outcome is not only not a problem, it is necessary. But if the peer
reviewer is being used in an effort to provide an independent
confirmation of the correctness of the initial conclusions, then
knowing the initial examiner's results seriously impairs the
reviewer's ability to perform that function.
Reversing one Sfindings in light of domain-irrelevant evidence
Sometimes examiners reach their own independent conclusions,
but later learn of findings regarding other case evidence which is
inconsistent with their own findings. We are not concerned here
with the intentional fraud of examiners who, in light of the other
findings, deliberately alter their own opinions in order to achieve
a false consistency. Of greater interest for the present article are
the examiners who, upon learning of the contrary findings of
other case evidence, begin to rethink and re-perceive and re-
interpret their own findings, coming to sincerely believe their
revised conclusion.
The Lindbergh kidnapping case may provide an example.
According to a report of FBI special agent Thomas Sisk, Albert
D Osborn initially doubted that Bruno Hauptmann's writings
came from the same source as the ransom notes. However,
within an hour after having been informed of the discovery of
the bulk of the ransom money in Hauptmann's garage, both
Albert D and his father Albert S Osborn changed to the
conclusion that Hauptmann did in fact write the ransom notes,
and they later so testified [70].
Selective re-examination of evidence
Sometimes police or prosecutors respond to test results that are
negative or inconclusive by suggesting to forensic scientists
what they should have found, and ask them to test again in hopes
of obtaining a 'better' result. The contamination here can be
quite crude: the investigator or prosecutor might be signaling to
the examiner that a more inculpatory result is desired and the
examiner is being invited to rethink the conclusions with that in
mind [71].
On a more subtle level, the District Attorney of Milwaukee
described a case in which a man charged with the murder of a
police officer claimed that the officer was beating him and that
in self-defense he took the officer's gun and shot the officer [72].
A state medical examiner concluded that the entry wound was in
the officer's back - inconsistent with the defendant's claim of
self-defense, consistent with the prosecution theory that the
officer was shot while trying to run from the defendant. But FBI
examiners concluded that the entry wound was in the officer's
chest - consistent with the shooter's self-defense claim. In an
effort to resolve the discrepancy, the District Attorney contacted
the FBI, pointed out that the state examiner reached the opposite
conclusion, and asked the FBI to double-check their findings,
just to make sure they were correct. The examiner did so and
reversed his conclusions. The choice of whom to call and ask to
conduct a re-examination skews the results. It leaves the
preferred set of conclusions in place while inviting revision of
the non-preferred conclusion.
Indeed, the mere making of a request for reconsideration
conveys information and sets up expectations, so it has to be
done with care if it is not to bias the outcome. Imagine what
result might have emerged if the DA had called the state
examiner and asked him to reconsider his conclusions because
they were in conflict with those of the FBI. Or, if the DA had
called both, merely pointing out the conflict, and not telling
either of them anything about his theory of the case, and so that
neither knew whether the wound was to a police officer or to
someone shot by a police officer.
Another example is provided by United States v. Mitchell, a case
in which the accuracy of fingerprint identification was
challenged [73]. In an effort to prove the claim that all
fingerprint examiners reach the same conclusion on the same
evidence, an FBI supervisory fingerprint specialist sent two
latent prints and a known fingerprint card to crime laboratories
in every state for them to compare the latents and knowns. Of the
39 labs which responded, 30 agreed with the FBI's reading of the
prints. Of the remaining nine, four agreed on one of the two
latents and five concluded that neither of the latents could be
said to match the known prints [74]. In response, the FBI
enlarged the exhibits and annotated the latent prints to indicate
what it believed were the points of similarity on which a
conclusion of identification could rely, and sent those
embellished exhibits back to the experts who had reached
contrary conclusions. Those experts were asked to reconsider
their conclusions. They all acquiesced to the opinion being urged
upon them.
Page 86 science&justice Volume 43 No.2 (2003) 77 - 90
The obviously skewed nature of that process of 'inquiry'
illustrates the biasing effects of selective re-examination.
Suppose, as a control, the FBI had selected a sample of 'non-
errant' examiners and sent them similar exhibits pointing out the
bases on which those who found no match had reached their
conclusions. One then would have been able to assess the extent
to which the reversed opinions were a product of reconsideration
of the actual evidence or submission to the less than subtle cues
being sent. But the manner in which the request for
reconsideration was structured insured that the situation could
only get 'better.'
The bias in selective re-examination is subtle but powerful.
Some findings but not others are being re-examined, freezing the
preferred results in place but inviting change in the non-
preferred results. In addition, the examiner to whom the re-
examination request is made is told that another examiner
reached different (and more pleasing) conclusions.
Conclusions and recommendations
Minimizing context effects in forensic science
As a result of the growing number of DNA exonerations, and the
analyses of those cases to determine what went wrong, it has
grown apparent that the contribution of forensic science to the
problem of convicting the innocent is greater than expected. The
data indicate that forensic science exaggerations and errors are
second only to eyewitness error as the leading cause of
erroneous convictions [75]. We suggest that a large part of that
serious problem consists of errors resulting from context effects.
If that is so, steps should be taken to protect examiners from
being unwilling - or willing - victims of such these effects. The
serious problems of observer effects can be solved or at least
ameliorated by changes in forensic practice which are neither
tremendously complex nor expensive. This is because,
fortunately, many of these problems already have solutions
which are in routine use in most scientific fields, and can be
found in standard research methods textbooks.
The first step is awareness, a step this article was written to
further. Such awareness of the phenomena of observer effects is
a necessary, but entirely inadequate step. Conventional sciences
have recognized the necessity of methodological solutions. 'The
discovery of suggestibility in patients undergoing experimental
treatments necessitated the introduction of the placebo
experiment, and the possibility of similar suggestibility on the
part of experimenters led to the double-blind experiment.' p. 29,
[46]. Forensic scientists have no less need, and no less ability,
than other serious scientists, to institute procedures to protect
their findings against avoidable sources of error. '[Tlhe
psychological fact of an omnipresent tendency toward
motivational bias fully justifies those many aspects of
experimental procedure, objective scoring, instrumentation, and
the like that guard against self-deception.' p. 37, [50].
Preventing distortions due to expectations: blind testing
No serious person can disagree with the US Department of
Justice Inspector General's affirmation that examiners should
not 'base forensic conclusions on unstated assumptions or
information that is collateral.' p. 511 [21]. The simplest, most
powerful, and most useful procedure to protect against the
distorting effects of unstated assumptions, collateral
information, and improper expectations and motivations, is blind
testing. An examiner who has no domain irrelevant information
cannot be influenced by it. An examiner who does not know
what conclusion is hoped for or expected cannot be influenced
by those considerations. Because the objects of forensic science
examination are inanimate, only single-blind, and not double-
blind, procedures will be necessary. In some contexts, crime
investigators appreciate the value of blind testing: 'The advice
given regarding preventing [observer effects] is the same as that
given to unsophisticated law enforcement ... who employ the use
of psychics: don't let anyone who knows anything about the case
near them to prevent information leakage and non-verbal
reinforcement.' [76].
A similar filtration of examination-irrelevant information must
be created for forensic science examiners. That means properly
controlling information flowing to examiners from external
investigators, from laboratory managers, and from fellow
examiners. Blind testing procedures in forensic science will not
always be simple and straightforward. As previously discussed
in regard to defensible criteria of domain-specific information,
sometimes examiners need to know certain details of a crime in
order to develop meaningful hypotheses which will lead to
determining what tests need to be done. The solution, as we have
implied throughout this discussion, is to provide examiners with
the information they need to perform the tests, and only that
information. At times, good practice might require sharing
information in stages - giving examiners certain information
necessary to performing a test, and subsequent to obtaining those
results provide additional information that might lead to further
testing. Doing so protects the soundness of the early testing
without losing the benefit of the later testing. In one DNA
laboratory we know of, testing practices already reflect
procedures of that kind [77].
Such information management can, with some adjustments, be
made to fit in with common administrative structures already in
place in forensic laboratories, such as those which have intake
personnel who log in submissions and route them to unit chiefs
who then assign the submissions to examiners. The most
important change would be to convert the personnel in the
evidence intake unit from fundamentally clerical personnel to
the most highly trained and highly respected personnel in the
laboratory, true 'Evidence Control' and 'Quality Control'
officers. Such officers should be required to have advanced
degrees in an appropriate conventional science discipline, and to
undergo rigorous training that would allow them to implement a
program designed to filter out domain-irrelevant information
from submissions, to formulate the questions to be answered in
the least suggestive way, and to route and coordinate the
submission of the evidence to the appropriate section or sections.
Such evidence and quality control (EQC) officers would be
responsible not only for coordinating work among examiners in
different specialties, but would be the sole contact point between
the entity requesting the test and the laboratory, and would serve
sciencetkjustice Volume 43 No.2 (2003) 77 - 90 Page 87
as the filter between each examiner and any information about
the case, whether it originated from without or from within the
lab. These officers would decide not only generally what kinds
of tests were needed, but would decide what information about
the case was needed by the examiner assigned to do those tests,
and one of their primary duties would be maintaining
appropriate masking between the examiners doing the actual
tests and all sources of domain-irrelevant information. Those
decisions and actions would be documented in the case file.
Nothing in this process prevents the EQC officers from learning
what needs to be learned about the case from police
investigators, nor does it prevent internal consultation among
examiners to make sense of the findings, after the initial round
of tests, and usually there is nothing to preclude additional tests.
All of that would be documented, so that it could be known how
blind or non-blind any given test was. A somewhat similar
process has been under development in the United Kingdom's
Forensic Science Service, based on Bayesian principles,
involving a more formalized process of 'pre-assessment' of
hypotheses and what would be required of the evidence to test
those hypotheses, as well as careful documentation of every step
of the process [78].
The EQC officer would work much like a primary care
physician, who knows all the background details, decides what
tests are needed, arranges for the tests to be conducted, and then
puts all of the information together to make a diagnosis. The
specialists and technicians doing much of the testing (blood
chemistry, radiology, etc.) do not know or need to know most of
the background details. A pathologist who evaluates suspected
skin cancer cells can only be adversely influenced by knowing
whether the patient is 16 or 76. After the histological evaluation
is done, the primary physician (or the specialist), can take into
account the patient's age (and other factors) in deciding whether
to ultimately conclude that cancer is present or not. The EQC
officer knows everything there is to know about a case and the
testing process as the work progresses, and by the end of the
process the information either has been or can be shared with
examiners. Labs which have caseloads which makes it
impractical for senior personnel to be the exclusive points of
contact and case management will need to invent alternative
procedures which seek to achieve the benefits of using
examiners as investigators as well as the benefits of blind testing.
Put simply, good scientific practice is to 'keep the processes of
data collection and analysis as blind as possible for as long as
possible.' [45]. And to accurately document what was done,
making that documentation automatically available to anyone
concerned with the reliability of the test procedures. Such a
regime would be expected to reduce the incidence of false
positives without reducing the incidence of true positives.
Preventing distortions due to assumed base rates: evidence
lineups
The situation of the forensic scientist is unusual in that the job
comes with an almost built-in expectation that tested evidence
will inculpate. This expectation may come from the realization
that any evidence connected with any suspect has some
heightened likelihood of being inculpatory, since investigators
do not select suspects or evidence at random, but only those they
have some reason for thinking were connected to the crime.
Thus, forensic scientists have a continuing expectation in each
case that something inculpatory is likely to be in hand. This
alone would trigger confirmatory bias, selective attention, and
other factors discussed above, which may operate to incline
examiners toward reaching positive conclusions.
In a detailed study of four different crime laboratories, Peterson
et al. [79] found that, on average, fewer than ten percent of all
reports disassociated a suspect from the crime scene or from
connection to the victim. This might be a consequence of police
investigative work that is in fact so excellent that labs rarely are
troubled with evidence that turns out to be exculpatory. More
likely, it reflects expectations on the part of examiners that most
of what is given to them is going to incriminate. Or it reflects
policies or cultures of labs that evidence ought to incriminate, so
that examiners' errors are skewed in the direction of inclusion.
(Support for the latter possibilities comes from Peterson et al.'s
finding that the labs varied in their criteria for inclusions, so that
the same evidence that in one would be reported as 'not sharing
a common origin' was reported by the others to be
'inconclusive.') Whatever the reasons, the inclusion rate is high,
and examiners come to expect it to be high. Blind testing
procedures, while fundamental, cannot remove baserate or
laboratory culture induced expectations that most examinations
will, or ought to, lead to inclusion. As indicated earlier, the more
subjective and less instrumented a forensic technique, the more
subject to expectation induced errors it is.
Such 'built-in' expectations are a problem that requires an
additional solution beyond blind testing. A technique which can
provide such a solution is the evidence lineup [go].
Appropriately designing such lineups and submitting evidence to
examiners in this form would be another responsibility of the
EQC officer. The evidence lineup would perform many of the
functions that an eyewitness lineup or photospread does.
Examinations in forensic science labs currently are the
equivalent of eyewitness 'show-ups.' In both settings, the test is
structured to be single-suspect, implying that the correct suspect
is in hand, and preventing the ultimate decision-makers (the
courts) from evaluating the quality of the test and the likely
validity of its results. A properly constructed and controlled
lineup solves these problems. The US Department of Justice's
recent guidelines on scientifically informed eyewitness
identification procedures discusses these details which, except
for those guidelines pertaining to interviewing, apply equally
well to evidence lineups [81].
The examiner would, of course, be blind to which items of
evidence in the evidence lineup are fillers and which are the true
questioned evidence. For example, a firearms examiner might be
presented with a crime scene bullet and five questioned bullets
labeled merely 'A' through 'E.' Four of those bullets will have
been prepared for examination by having been fired through the
same make and model of firearm as the crime scene bullet and
the suspect bullet had been. The task for the examiner will then
be to choose which, if any, of the questioned bullets was fired
Page 88 science&-justice Volume 43 No.2 (2003) 77 - 90
through the very same weapon as the crime scene bullet had
been.
A properly constructed evidence lineup would accomplish at
least the following. The examiner would know from the sheer
structure of the test situation that most of the questioned
evidence items are not associated with the suspect, and that a
failure to exercise real expertise in an objective fashion is likely
to lead to an incorrect conclusion that would be recognized as an
incorrect conclusion. That would erase much of the impact of
baserate-induced expectations. In contrast to the show-up
situation that today is common practice, examiners could not
opine 'inclusion' or 'inconclusive' to virtually every test. On the
other hand, when an examiner rejects all of the foils and
concludes that the one known evidence item matches the
questioned evidence item, this conclusion will be a more
convincing indication that what has led to the conclusion is
expertise and not expectations. Moreover, the lineup structure,
unlike the customary show-up procedure, allows a direct
calculation of the probability that the examiner would have
reached a correct conclusion by chance.
Good practice would indicate that as a check on merely choosing
the one that is the most similar to the crime scene evidence,
rather than the one that achieves a sufficient degree of similarity
to be declared a match, good evidence lineups, like good
eyewitness lineups, would sometimes be constructed entirely of
foils and would be conducted in a sequential rather than a
simultaneous manner [14,82]. Furthermore, before receiving any
post-test feedback or other extraneous information, the examiner
would record a rating of his or her subjective confidence that the
selected questioned evidence item shares a common origin with
the crime scene evidence. That rating is a check on extraneous
post-examination information creating exaggerated confidence
in an opinion originally reached with less confidence [83].
Proper evidence lineups present some non-trivial problems of
design, requiring the EQC officer both to determine what would
constitute appropriately similar foil specimens, and to obtain
them. This obviously would be easier for some types of
examinations than for others, and it might be most difficult to do
precisely where it is most needed: in those areas with the least
instrumentation and most subjectivity.
The fundamental task of the eyewitness and of the forensic
examiner share notable similarities, suffer from similar sources
of potential systematic error, and enjoy the same potential for
elimination of those problems merely by structuring the task in
an appropriately rigorous fashion.
Additional procedures for controlling observer effects
Blind testing and evidence lineups are two procedures that, in
combination, would solve the majority of the problems resulting
from observer effects that occur when any human being sets
about to make decisions of the sort that occur in forensic
examinations. For those labs that are interested in developing
procedures that go even further, additional techniques might be
explored, among them: cancellation of biases (creating
counterbalanced and mutually self-cancelling expectations),
production of biases (on a periodic experimental basis to monitor
their effects), and the increased development and use of
mechanical or electronic recorders and apprehenders (thereby
reducing the human role in the observation) [15].
In the same way that other fields have learned from the science
of science in order to strengthen the validity of their research and
practice, the opportunity is open to forensic scientists to borrow
from the same learning in order to improve the rigor of their own
procedures.
References
1 Boring EG. A Hlstory of Experimental Psychology, 2nd Edition. New York: Appleton-
Century-Crofts. 1950.
2 Sebeok TA and Rosenthal R, eds. The Clever Hans Phenomenon: Commun~catlon
with Horses, Whales, Apes, and People. New York: New York Academy of Sciences,
1981
3 Fisher RA. Has Mendel's work been rediscovered? Annals of Science 1936; 1:
115-137.
4 Feinstein AR. The stethoscope. a source of diagnostic ald and conceptual errors in
rheumatic heart disease. Journal Chronlc Dlseases 1960; 11 : 91-101
5 Hodge E. Guarding against error Journal of the Association of Firearms & Toolmark
Examiners 1988; 20: 290,292.
6 Ubelaker D and Scammell H. Bones: A Forenslc Detectwe's Casebook. New York:
Edward Burlingame Books, 1992.
7 Berkson J, Magath TB, Hurn M. The error of estimate of the blood cell count as made
with the hemocytometer American Journal of Physiology 1940; 128: 309-323.
8 Cordaro L. and lson JR. Observer blas classical condltloning of the planarian.
Psychological Reports 1963: 13: 787-789.
9 Kennedy JL and Uphoff HF Experiments on the nature of extra-sensoly perception:
Ill. The recording error crlticlsm of extra-chance scores Journal of Parapsychology
1939; 3: 226-245.
10 Cahen LS. An experimental manipulation of the 'halo effect': a study of teacher bias.
Unpublished manuscript, Stanford University, 1965. (Clted In: Rosenthal R.
Experimenter Effects in Behavioral Research. New York: Appleton-Century-Crofts.
1966; enlarged editlon, 1976.)
11 Roethllsberger FJ and Dickson WJ. Management and the Worker. Cambr~dge. MA:
Harvard University Press, 1939.
12 Rosenthal Rand Jacobson L. Pygrnallon in the Classroom. New York: Holt, Rinehart.
&Winston, 1968; expanded edition. 1992.
13 Harris MJ and Rosenthal R. The mediation of interpersonal expectancy effects 31
meta-analyses. Psychological Bulletin 1985; 97: 363-386.
14 Steblay N, Dysart J, Fulero S, Lindsay RCL. Eyewitness accuracy rates tn sequentlal
and simultaneous line-up presentations: a rneta-analytlc comparison. Law & Human
Behavlor 2001; 25: 459-473.
15 Rosenthal R. Experimenter Effects In Behavioral Research. New York: Appieton-
Century-Crofts, 1966; enlarged edltion, 1976.
16 Winkel FW and Koppelaar L. Perceived credibility of the communicator: studies of
perceptual bias in police offlcers conducting rape interviews. In: F. Losel et al., eds.
Psychology and Law: International Perspectives. Berlln & New York: Walter de
Gruyter, 1992: 223-228.
17 California Assoctation of Criminalistics. Code of Ethlcs; 1957, 1958; 1985.
18 Bacon F. Novum Organum. Book I, point 46. Great Books of the Western World,
Volume 30, p. 110. City: Publisher, YEAR (originally published 1620).
19 Glannelll PC. The abuse of scientific evidence in criminal cases: the need for
Independent crlme laboratories. Virginla Journal Social Policy & Law 1997; 4:
439-478.
20 People v. Longtln, 707 N.E.2d 418 (N.Y.Ct.App., 1998).
21 Wlce of the Inspector General. The FBI Laboratory: An Investigation into Laboratory
science&justice volume 43 No.2 (2003) 77 - 90 Page 89
Practices and Alleged Misconduct In Explosives-related and Other Cases.
Washington, DC: US Department of Justice. 1997.
Kelly JF and Wearne PK. Tainting Evidence. New York: Free Press, 1998.
National lnstitute of Justice. Forensic Sciences: Review of Status and Needs.
Washington. DC: US Department of Justice, 1999.
Ne~sser U. Cognltton and Reality: Principles and Implications of Cognitive Psychology.
San Franc~sco: W.H. Freeman. 1976.
Ornste~n RE. The Psychology of Consciousness, 2d Edltlon. New York: Harcourt
Brace, 1977.
Kety SS. Blochemlcal theor~es of schizophrenia. Science 1959; 129: Part I,
1528-1 532; Part 11. 1590-1596.
Wilson EB. An lntroductton to Scientific Research. New York: McGraw-Hill, 1952.
Wood FG. P~tfall. Science 1962; 135: 261
Nisbett RE and Ross L. Human Inference: Strateg~es and Shortcomings of Social
Judgement. Englewood Cliffs, NJ: Prentice-Hall, 1980.
Gilovich T. How We Know What Isn't So: The Fallibility of Human Reason in Everyday
Life. New York: Free Press, 1991.
Bloom CP The role of schemata In memory for text Discourse Processes 1988, 11 :
305-31 8.
Kardash CAM, Royer JM, Greene BA. Effects of schemata on both encoding and
retrieval of Information from prose. Journal Educational Psychology 1988; 80:
324429.
Phillips V, Saks MJ, Peterson J. Signal detection theory and dec~ston-maktng In
forenstc science. Journal Forenstc Science 2001; 46: 294308.
Tversky A and Kahneman D. Judgment under uncertalnty: heurlsttcs and biases.
Science 1974; 185: 1 124-1 131.
Jacowttz JE and Kahneman D. Measures of anchoring In estlmatlon tasks.
Personality and Social Psychology Bullettn 1995; 21 : 11 61-1 166.
Northcraft GB and Neale MA. Experts, amateurs and real estate: an anchoring-and-
adjustment perspective on property prlcing decis~ons. Organizational Behavior and
Human Dec~s~on Processes 1987; 39: 84-97.
Englich B and Mussweiler T. Sentencing under uncertainty: anchoring effects in the
courtroom. 31 Journal Applied Soc~al Psychology 2001 ; 31 : 1535-1 551.
Mausner B. Studles in social interaction: Ill. Effect of variation in one partner's prestige
on the interaction of obselver pairs. Journal Applied Psychology 1953; 37: 391-394.
Rosenthal R, ed. Clever Hans, The Horse of Mr. von Osten. pranslated from the
German by Carl Rahn] New York: Holt, Rinehart and Wlnston. 1965 (first published in
1911).
Beecher HK. Paln: one mystery solved. Science 1966; 151: 840-841.
Rosenthal R. Designing, analyz~ng, Interpreting, and summarizing placebo studies. In:
White L, Tursky B and Schwartz GE, eds. Placebo: Theory, Research, and
Mechanisms. New York: Guilford. 1985.
Blanck RD, ed. lnterpersonal Expectations: Theory, Research, and Applications. New
York: Cambridge University Press, 1993.
Rosnow RL and Rosenthal R. People Studying People: Artifacts and Ethics in
Behaworal Research. New York: Freeman, 1997.
Rosenthal R. lnterpersonal expectancy effects: a 30-year perspectwe. Current
D~rect~ons in Psychological Science 1994; 3: 17&179.
Rosenthal R. How often are our numbers wrong? American Psychologist 1978; 33:
10051 008.
Campbell DT Systematic errors to be expected of the social scientist on the basis of
a general psychology of cognttive bias. In: Blanck PD, Ed, lnterpersonal Expectations:
Theory, Research, and Applications. New York: Cambridge Un~versity Press, 1993.
Postman L. Toward a general theory of cogn~tion. In: Rohrer JA and Sherlf M. Soclal
Psychology at the Crossroads. New York: Harper, 1951.
Postman L. The experimental analysis of motivational factors in perceptton. In: Brown
JS et al., eds. Current Theory and Research in Mot~vation. Lincoln, Nebraska:
University of Nebraska Press, 1953.
Postman Land Bruner JS. Perception under stress. Psychological Review 1948; 55:
314-323.
Dember WN. The Psychology of Perception. New York: Holt. Rinehart and Winston,
1960.
Bruner JS. Personality dynamics and the process of perceiving. In: Blake RS and
Ramsey GV, eds. Percept~on: An Approach to Personality. New York: Ronald, 1951.
Slovic P Toward understanding and lmprovlng decisions. In: Howell WC and
Fle~shman EA, eds. Human Performance and Productivity: Vol. 2: Information
Processing and Decision Making. Hillsdale, NJ: Erlbaum, 1982; 157-183.
Yule GU. On reading a scale. Journal Royal Statlstlcal Soclety 1927; 90: 570-587.
Boring EG. Newton and the spectral I~nes. Science 1962; 136: 800601.
Rostand J. Error and Deception In Sc~ence. New York: Basic Books, 1980.
Tape TG and Panzer RJ. Echocardiography, endocarditis, and clinlcal informatton
bias. Journal General Internal Medicine 1986; 1: 300-310.
Johnson ML. Seelng's bel~ev~ng. New Biology 1953; 15: 60-80.
Lane MI. Kingdom of the Octopus. New York: Sheridan House, 1960.
Hagan WE. Disputed Handwriting. Albany: Banks & Brooks. 1894.
Ashbaugh DR. Quantitative-qualitative Friction Ridge Analysis. Boca Raton, Fla.: CRC
Press. 1999.
American Society of Crime Laboratory Directors. Laboratory Accreditat~on Board.
Manual. Garner, NC: ASCLDILAB, 2000.
Dines JE. Document Examiner Textbook. Ilvine, CA: Pantex International Ltd., 1998.
Daubert v. Merrell Dow Pharmaceuticals. 509 US 579 (1 993).
Kam M, Fielding G, Conn R. Writer ~dentification by professional document
examiners. Journal Forensic Sciences 1997; 42: 778-786.
Kumho Tire Co. v. Carmichael. 526 US 137 (1999).
Un~ted States v. Fujii, 152 F.Supp2d 939 (2000).
Oregon v. Boots, 94 Or. App. 713 (1989).
Oregon v. Proctor, 94 OrApp. 720 (1989).
Mones ' Stalking Justice: The Dramatic True Story of the Detective Who F~rst Used
DNA Testing to Catch a Sertal Killer. New York: Pocket Books, 1995.
Kennedy L. The A~rman and the Carpenter. New York: Viking, 1985.
DeForest P. Proceedings of the Second International Conference on Forensic
Document Examination. Albany, NY, June 14-18, 1999.
McCann EM. Proceedings of the National Conference on Science and the Law.
Washington, DC: National Institute of Justice, 2000: 24G241.
United States v. Byron Mitchell (E.D. Pa., 1998) (Crim. #96-407-1. Judgment entered
Februaly, 2000; a previous trial and conviction resulted in a reversal, reported at 145
F.3d 572 3rd. Cir. (1998).)
United States v. Llera-Plaza, 188 F.Supp.2d 549 (2002).
Scheck B, Neufeld P, Dwyer J. Actual Innocence: Ftve Days to Execution and Other
Dispatches from the Wrongly Convicted. New York: Doubleday, 2000.
76 Turvey BE. [Discussion on Forens-L, a forens~c science online d~scussion 13sew.]
Reproduced at: Sclentlflc Testimony: an Online Journal, w.scientific.org.
77 LaBerge GS. [Discussion on Forens-L, a forenslc sclence online discussion Ilstselv.]
Reproduced at: Scientific Testimony: an Online Journal, w.sclentific.org.
78 Cook R, Even IW, Jackson G, Jones PJ, Lambert JA. A model for case assessment
and interpretation. Science &Justice 1998; 38: 151-156.
79 Peterson JL, Mihajlovic S, Gilliland M. Forensic Evidence and the Police. Washington,
DC: National Institute of Justice, 1984.
80 Schoon GAA. A flrst assessment of the rel~ab~lity of an Improved scent identification
lineup. Journal Forensic Sciences 1998; 43: 7G75.
81 Technical Working Group for Eyewitness Evidence. Eyewitness Ev~dence: A Gu~de for
Law Enforcement. Washington. DC: United States Department of Justice, 1999.
82 Miller L. Procedural bias in forensic science examinations of human hair. 11 Law &
Human Behavior 1987; 11 : 157-1 63.
83 Wells G et al. From the lab to the police station: a successful application of
eyewitness research. American Psychologist 2000; 55: 581-598.
Page 90 science&justice Volume 43 No.2 (2003) 77 - 90
Law, Probability and Risk Page 1 of 20 doi:10.1093/lpr/mgp013
Painting the target around the matching prole: the Texas
sharpshooter fallacy in forensic DNA interpretation
WILLIAM C. THOMPSON
Department of Criminology, Law and Society,
University of California, Irvine, CA 92697, USA
[Received on 17 November 2008; revised on 19 April 2009; accepted on 24 April 2009]
Forensic DNA analysts tend to underestimate the frequency of matching proles (and overestimate
likelihood ratios) by shifting the purported criteria for a match or inclusion after the prole of a
suspect becomes knowna process analogous to the well-known Texas sharpshooter fallacy. Using
examples from casework, informal and naturalistic experiments, and analysts own testimony, this
article demonstrates how post hoc target shifting occurs and how it can distort the frequency and
likelihood ratio statistics used to characterize DNA matches, making matches appear more probative
than they actually are. It concludes by calling for broader adoption of more rigorous analytical pro-
cedures, such as sequential unmasking, that can reduce the sharpshooter fallacy by xing the target
before the shots are taken.
Keywords: DNA evidence; frequency; likelihood ratio; fallacy; sequential unmasking; DNA prole;
bias; error; statistics; ACE-V.
1. Introduction
When evaluating the signicance of scientic data, it is often helpful to calculate the probability that
a particular event occurred by chance. Calculations of this type can be misleading, however, when
they focus too narrowly on a given outcome without considering the broader context. For example,
an epidemiologist who observes a cancer cluster in a particular neighbourhood might compute
the probability that random chance would produce so many cancer cases in that neighbourhood.
This computation would be misleading because random processes are very likely to produce cancer
clusters in some neighbourhoods even though a cluster is unlikely in any particular neighbourhood
(Neutra, 1990; Rothman, 1990; Thomas et al., 1985).
The Texas sharpshooter fallacy is the name epidemiologists have given to the tendency to as-
sign unwarranted signicance to random data by viewing it post hoc in an unduly narrow context
(Gawande, 1999). The name is derived from the story of a legendary Texan who red his rie ran-
domly into the side of a barn and then painted a target around each of the bullet holes. When the
paint dried, he invited his neighbours to see what a great shot he was. The neighbours were im-
pressed: they thought it was extremely improbable that the rieman could have hit every target dead
centre unless he was indeed an extraordinary marksman, and they therefore declared the man to be
the greatest sharpshooter in the state. Of course, their reasoning was fallacious. Because the sharp-
shooter was able to x the targets after taking the shots, the evidence of his accuracy was far less
Email: william.thompson@uci.edu
c The Author [2009]. Published by Oxford University Press. All rights reserved.
Law, Probability and Risk Advance Access published July 28, 2009

b
y

g
u
e
s
t

o
n

N
o
v
e
m
b
e
r

3
,

2
0
1
1
h
t
t
p
:
/
/
l
p
r
.
o
x
f
o
r
d
j
o
u
r
n
a
l
s
.
o
r
g
/
D
o
w
n
l
o
a
d
e
d

f
r
o
m

2 of 20 W. C. THOMPSON
probative than it appeared. The kind of post hoc target xing illustrated by this story has also been
called painting the target around the arrow.
In this article, I will argue that a process analogous to the Texas sharpshooter fallacy sometimes
occurs in the production of forensic DNA evidence. Analysts create the impression that a DNA
match is a very small target that is unlikely to be hit by chance. But the probability of a coincidental
match may actually be much higher than analysts claim because a match is not always a xed
target. Using examples from casework, informal and naturalistic experiments, and analysts own
testimony, I will show that forensic DNA analysts sometimes shift their criteria for a match based
on the DNA prole of the suspectin effect, they move the target after the shots are red. I will
show how this post hoc target shifting occurs and how it can distort the frequency and likelihood
ratio statistics used to characterize DNA matches, making matches appear more probative than they
actually are. I will conclude by calling for broader adoption of more rigorous analytical procedures,
such as sequential unmasking, that can reduce the sharpshooter fallacy by xing the target before
the shots are taken.
2. Hitting the target with DNA evidence: an easy case
The fallacy does not occur in every case. In many cases, DNA test results are clear and easy to
interpret, which reduces opportunities for post hoc target shifting. Figure 1 shows STR test results
for a crime scene sample (top electropherogram) and four suspects (lower four electropherograms)
in what I will call an easy case. Each electropherogram shows the alleles found in the sample at three
STR loci. The position of the peaks on the electropherogram indicates which alleles are present
at each locus. A computer program labels each peak with a number, which identies the allele it
represents. In order for two samples to match, they must have the same alleles at each locus.
Quick examination of these results shows clearly that Suspect 3 matches the target prole and
the other suspects do not. Hence, Suspect 3 is incriminated as a possible source of the bloodstain at
the crime scene. If we think of the alleles in the evidentiary sample as the target, then it is apparent
that Suspect 3 hit the target exactly.
FIG. 1. Electropherograms in an easy-to-interpret case.

b
y

g
u
e
s
t

o
n

N
o
v
e
m
b
e
r

3
,

2
0
1
1
h
t
t
p
:
/
/
l
p
r
.
o
x
f
o
r
d
j
o
u
r
n
a
l
s
.
o
r
g
/
D
o
w
n
l
o
a
d
e
d

f
r
o
m

PAINTING THE TARGET AROUND THE MATCHING PROFILE 3 of 20
To determine the probative value of this evidence for proving that Suspect 3 is the source of
the bloodstain, we must consider the probability of hitting this target by chance. If a match re-
quires an exact one-to-one correspondence between the alleles in the two samples, as appears to
be the case here, the computational procedure is straightforward. The rst step is to determine the
frequency (in some relevant reference population) of the matching alleles. The allele frequencies are
then combined to determine the frequency of the genotype (pair of alleles) observed at each locus.
If the genotype is heterozygous (i.e. the donor inherited a different allele from each parent, hence
two different alleles are detected) then the genotype frequency is typically computed using the for-
mula 2pq, where p and q represent the frequencies of two heterozygous alleles. If the genotype is
homozygous (i.e. the donor inherited the same allele from each parent, hence only one allele is de-
tected), then the genotype frequency is computed using the formula p
2
, where p is the frequency of
the homozygous allele (Butler, 2005, pp. 498500).
1
Once the genotype frequencies for the match-
ing loci are computed, they are multiplied together to obtain the estimated frequency of the matching
prole.
The frequency of the matching prole is then used to characterize the value of the DNA evi-
dence for incriminating the matching suspect. It is often called the random match probability and
is described, to the trier-of-fact, as the probability that a randomly chosen unrelated person who was
not the source of the evidentiary sample would happen to match. One might say that the prole fre-
quency represents the probability of hitting the target by chance, where the target is an incriminating
result.
The Texas sharpshooter fallacy does not appear to be a problem when computing the random
match probability for Suspect 3 (Fig. 1) because the target was xed. There was one and only one
prole that would match and the analyst could not change the requirements for a match to t a
particular suspect.
3. Opportunities for target shifting: a harder case
Not all cases are as clear and easy to interpret as the case shown in Fig. 1. In cases I have reviewed
over the past few years, evidentiary samples from crime scenes often produce incomplete or partial
DNA proles. Limited quantities of DNA, degradation of the sample, or the presence of inhibitors
(contaminants) can make it impossible to determine the genotype at every locus (Butler, 2005,
p. 168; Buckleton and Gill, 2005) In some instances, the test yields no information about the
genotype at a particular locus; in some instances, one of the two alleles at a locus will dropout
(become undetectable). When testing samples with very low quantities of DNA, spurious alleles
(i.e. alleles not associated with the sample) are sometimes detected, a phenomenon known as allelic
drop-in
A further complication is that evidentiary samples are often mixtures of DNA from more than
one person. It can be difcult to tell how many contributors there were to a mixed sample (Paoletti
et al., 2005), and even more difcult to tell which alleles are associated with which contributor,
hence at each locus there may be a number of different genotypes that a contributor could have,
even if one assumes that all alleles have been detected. If, as is often the case, the analyst cannot be
1
To address concerns about population structure and inter-relatedness most laboratories introduce a slight adjustment to
these simple formuli that is known as a theta correction (Butler, 2005, pp. 506507). For ease of exposition, I will ignore the
theta correction as it makes little difference to the issues raised here.

b
y

g
u
e
s
t

o
n

N
o
v
e
m
b
e
r

3
,

2
0
1
1
h
t
t
p
:
/
/
l
p
r
.
o
x
f
o
r
d
j
o
u
r
n
a
l
s
.
o
r
g
/
D
o
w
n
l
o
a
d
e
d

f
r
o
m

FIG. 2. Electropherogram of a saliva sample and four suspect proles.
certain that all alleles of all contributors have been detected, then the number of possible contributor
genotypes is even greater.
To avoid false exclusions, analysts must make allowances for these phenomena when determin-
ing whether a suspect should be included or excluded as a potential contributor. As discussed
below, these allowances not only expand the target, they create the potential for target shifting and
do so in ways that are not always taken into account by the frequencies (and random match probabil-
ities) computed by most forensic laboratories. Consequently, the statistics presented in these cases
often understate the actual probability of a coincidental inclusion. Figure 2 shows STR test results
in what I regard as a harder case to interpret. This electropherogram shows the alleles detected in
an evidentiary sample (assumed to be saliva) that was swabbed from the skin of a sexual assault
victim. Below each allele are two boxes containing numbers. The upper number identies the allele
and the lower number indicates the peak height in relative orescent units. The height of the peaks
corresponds to the quantity of DNA recovered from the sample.
The table at the bottom of Fig. 2 shows the proles of four possible defendants. I will argue that
it is not so clear which of these defendants should be included or excluded as possible contributors.
The peak heights in Fig. 2 are much lower than those in Fig. 1 because a relatively small amount
of DNA was recovered from the skin swab. For a sample of this type, the analyst must consider the
likelihood of allelic dropout and the possibility of allelic drop-in. For example, at locus D3S1358
(hereafter D3), the analyst must determine whether the peak labelled 12 represents a true allele and,
if so, whether it is associated with (i.e. from the same contributor as) allele 17; at locus FGA, the
analyst must determine whether the peak labelled OL Allele?
2
is a true allele or an artefact. To
compute the probability of a coincidental inclusion, the analyst must also consider whether this is a
single source sample or a mixture.
2
OL stands for off-ladder. In STR test kits, the alleles typically observed at each locus are called ladder alleles
because the test incorporates a control, called a ladder, which contains each of these common variants. An off-ladder allele
may represent an unusual genetic variant. Alternatively, it could indicate a problem with the test, such as the presence of a
spurious peak caused by a technical problem. Most experts who have looked at the electropherogram in Fig. 2, believe the OL
allele at locus FGA is an artefacti.e. a spurious result, rather than a true allele. As noted below, however, experts differ
about whether this spurious result might be masking a true allele. And the masking theory tends to be invoked selectively
depending on whether doing so is necessary to include the defendant.

b
y

g
u
e
s
t

o
n

N
o
v
e
m
b
e
r

3
,

2
0
1
1
h
t
t
p
:
/
/
l
p
r
.
o
x
f
o
r
d
j
o
u
r
n
a
l
s
.
o
r
g
/
D
o
w
n
l
o
a
d
e
d

f
r
o
m

In cases I have reviewed in the USA, analysts almost always have had full knowledge of the
proles of the suspect (or suspects) when making determinations of this kind. The widespread failure
to adopt blind procedures for making these determinations sets the stage for target shifting. For
example, if the suspect has the prole of Tom, the analyst might be more inclined to dismiss the 12
peak at locus D3 and the OL peak at locus FGA as artefacts than if the suspect has the prole of Dick.
4. An inadvertent experiment
As evidence for these assertions, let me describe an informal and rather inadvertent experiment that I
have conducted over the past fewyears. This experiment was neither rigorous nor well controlled and
the results were recorded in a manner that was neither blind, nor systematic, nor (I fear) entirely ob-
jective. Nevertheless, I think the experiment is worth describing here because it so nicely illustrates
how something like the Texas sharpshooter fallacy can arise in the context of forensic DNA testing.
3
It began when I was invited to speak to a group of forensic DNA analysts at a meeting of the
California Association of Criminalists. I was talking about the way forensic analysts interpret DNA
evidence in problematic cases and the tone of my comments was rather critical. I showed a slide
in which the evidentiary prole in Fig. 2 was juxtaposed with a prole labelled Defendant. The
defendant prole was that of Tom in Fig. 2.
I suggested that there might be some uncertainty about the inclusion of the defendant because
his prole did not account for all peaks in the evidentiary prole. In particular, neither the defendant
nor the victim had the 12 allele at locus D3 nor the OL allele at locus FGA. How could we be sure,
I asked, that the true contributor did not have genotype 12,17 at locus D3?
At that point, several analysts interrupted my talk to say that I did not know what I was talking
about. One of the analysts stated, with the apparent support and afrmation of many others, that it
would be obvious to any qualied expert that the 12 peak at locus D3 and the OL peak at FGA are
artefacts that should simply be ignored when comparing the evidentiary prole to the suspect. The
12 peak could not represent a true allele, one of them said, because it did not have the morphology
of a true allele.
4
Even if it was a true allele, another said, it could not be from the same person
who contributed the 17 allele because the peak height disparity was too great for the peaks to have
originated from the same individual. The OL Allele at locus FGA was a known artefact that any
competent analyst would recognize and ignore. Although I was a bit doubtful about some of these
pronouncements, I felt that my arguments had been soundly defeated, particularly when a comment
from the audience that characterized my position as slimy and irresponsible invoked enthusiastic
applause.
Later that same month, I was scheduled to again speak about DNA evidence to an organization
that called itself the Skeptics Society. As the date approached, the Skeptics informed me that a group
of DNA analysts from a local crime laboratory and several local prosecutors who were experts on
DNA evidence would be attending my talk and had asked for time on the program to respond to my
comments. This seemed a bit intimidating but it also suggested an idea.
5
3
I did not set out to do an experiment, merely to create illustrations for teaching purposes. Over time I realized, however,
that the reactions of the audience to variations in these presentations was rather telling.
4
The morphology of a true allele entailed certain characteristics of symmetry and heightwidth ratio that were said to
be recognizable to a trained analyst.
5
My approach to the talk was inspired in part by James Randi (The Amazing Randi), who had preceded me as a speaker
at the Skeptics Society. Randi is famous for devising techniques (often involving deception) to expose people who falsely
claim to have supernatural abilities.

b
y

g
u
e
s
t

o
n

N
o
v
e
m
b
e
r

3
,

2
0
1
1
h
t
t
p
:
/
/
l
p
r
.
o
x
f
o
r
d
j
o
u
r
n
a
l
s
.
o
r
g
/
D
o
w
n
l
o
a
d
e
d

f
r
o
m

When I spoke, I again presented the evidentiary prole shown in Fig. 2. This time, however, the
prole labelled Defendant was that of Dick rather than Tom. I suggested that the inclusion of Dick
was rather problematic due to uncertainty about whether the 12 peak at locus D3 was a true allele
and because no 20 peak had been detected at locus FGA. I then invited the experts in the audience
to tell me if I was wrong.
They wasted little time in doing so. One analyst told me that she could tell the 12 peak at D3
was a true allele based on its morphology. The peak height disparity was nothing that would
trouble a knowledgeable analyst, I was told, because discrepancies of that sort are expected with low
quantities of DNA due to stochastic effects. The OL allele at locus FGA was indeed an artefact, but
it could easily have masked an underlying 20 allele. One analyst said the shape of the artefactual OL
peak supported the theory of an underlying 20 allele. I was again condemned as irresponsible for
questioning the validity of this match. I then had the great pleasure of suggesting that the defendants
prole might actually have been that of Tom rather than Dick, at which point my critics became
noticeably less certain of the correctness of their interpretations (and the incorrectness of mine).
Encouraged by this felicitous result, I extended the experiment a few months later while speaking
to an organization called the Association of Forensic DNA Analysts and Administrators. In that
talk, the prole labelled Defendant was that of Harry in Fig. 2. I suggested that the inclusion
of the defendant was problematic because the defendants genotype at D3 was 14,17, while the
evidentiary prole had peaks at 12 and 17. I also noted the failure to detect the defendants 20 peak
at locus FGA. Once again, the DNA analysts in the audience told me they saw no problem at all
with the inclusion of the defendant (Harry). The failure to detect the defendants 14 allele at locus
D3 could easily be due to allelic dropout and there might well be a 20 allele at locus FGA that
was masked by artefact. The peak labelled 12 at locus D3 was an obvious artefact. If I were truly
an expert like themselves, I was told, I would realize that my concerns about the inclusion of the
defendant were groundless.
At this point, I wonder how much I would need to change the defendant prole to get foren-
sic DNA analysts to agree that the defendant should have been excluded. My friend Dan Krane
from Wright State University presented this case to forensic analysts using the defendant prole
labelled Sally. Even with this prole, analysts have still insisted that the defendant cannot be ex-
cluded. To reach this conclusion, they posited that the evidentiary sample consisted of a mixture
of two contributors, one of whom has the 15 allele at locus vWA and the other who has the 17
allele. In other words, a new theory of the evidence (that it is a mixture) was invoked to include
Sally. Interestingly, the mixture theory was never mentioned when the defendant had the other
proles.
5. The absence of formal standards for inclusion
These informal experiments can be criticized on a number of grounds, but the basic point they illus-
trate should be self-evident to anyone who looks closely at the actual practices of forensic laborato-
ries. A key source of problems is the absence of any formal standards for distinguishing inclusions
from exclusions. I have looked carefully at the protocols of a number of laboratories in the USA
and I am yet to see a protocol at any laboratory that sets forth standards that would allow an un-
ambiguous determination of whether Tom, Dick, Harry or Sally should be included or excluded as
contributors. Some protocols specify factors for analysts to consider in making such determinations,
but none set forth objective standards and none require that any guidelines that are mentioned be

b
y

g
u
e
s
t

o
n

N
o
v
e
m
b
e
r

3
,

2
0
1
1
h
t
t
p
:
/
/
l
p
r
.
o
x
f
o
r
d
j
o
u
r
n
a
l
s
.
o
r
g
/
D
o
w
n
l
o
a
d
e
d

f
r
o
m

followed consistently. Whether the comparison of proles results in a nding of inclusion, exclu-
sion or inconclusive is an entirely subjective determination.
In the absence of objective standards for distinguishing inclusions from exclusions, estimates
of the probability of a coincidental inclusion are problematic. How can we estimate the percentage
of the population who would be included if the standards for inclusion are ill-dened and can be
stretched one way or another by the laboratory? Estimating the size of the included population
under these circumstances is analogous to estimating the length of a rubber band. Just as a rubber
band may be longer or shorter, depending on how far one is willing to stretch it, the size of the
included population may be larger or smaller, depending on how leniently or strictly the analyst
denes the criteria for an inclusion. Because it is unclear just how far the laboratory might stretch
to include a suspect, the true size of the included population cannot be determined. This is a
fundamental problem that has not yet been resolved (and is not yet fully recognized) by the DNA
testing community. This problem is particularly important in what I have called the hard cases in
which there are incomplete and mixed proles.
The standards for inclusion and exclusion are not just ill-dened and subjective, they are also
exible. As the Tom, Dick and Harry example suggests, the standards may shift to encompass the
prole of suspects. This process may well occur without analysts being aware of it. Perhaps the pro-
cess of comparing the defendants prole to the evidentiary prole channels analysts thinking about
how the evidentiary prole might have arisen. Psychological studies have shown that the process
of imagining how a particular outcome might occur increases peoples estimates of the probability
that the imagined outcome will (or did) occur (Anderson et al., 1980). Hence, merely thinking about
how the defendants DNA might have produced the observed prole could increase the analysts
condence that the defendant was a contributor. Moreover, while focusing on a defendants prole,
analysts might also ignore, discount or simply fail to imagine other ways in which the same data
might have arisen if the defendant was not a contributor.
6
6. How the target shifts
To further illustrate the process of target shifting and its effects on statistical estimates, let us return
to Tom, Dick, Harry and Sally. Consider rst how an analyst might estimate the probability of an
inclusion at locus D3. When the defendant was Tom, the analysts tended to dismiss the 12 peak as
an artefact, which led them to think that the perpetrator was a homozygous 17 (like Tom). Indeed,
most of the analysts with whom I discussed the case of Tom told me they thought the correct formula
for estimating the probability of a coincidental inclusion at D3 was p
2
, where p is the frequency of
the 17 allele. This would be the correct estimate if it were the case that the laboratory would only
include someone with genotype 17,17 and would exclude anyone with a different genotype. But as
the experiment shows, the target is actually much wider than this. Neither Dick, Harry nor Sally had
genotype 17,17, yet they were still included. So, an analyst who computed the match probability as
p
2
would be underestimating the frequency of a coincidental inclusion; the analyst would, in effect,
have painted the target around the arrow.
6
The notion that interpretation of scientic data can be inuenced by extraneous factors, including the knowledge,
assumptions and expectations of the scientist, is widely accepted (Risinger et al., 2002; Thompson, 2009). It is a major
theme in science and technology studies (Hess, 1997) and the sociology of science (Longino, 1990) and has been extensively
documented in psychological studies (Nisbett and Ross, 1980; Gilovich, 1991).

b
y

g
u
e
s
t

o
n

N
o
v
e
m
b
e
r

3
,

2
0
1
1
h
t
t
p
:
/
/
l
p
r
.
o
x
f
o
r
d
j
o
u
r
n
a
l
s
.
o
r
g
/
D
o
w
n
l
o
a
d
e
d

f
r
o
m

When Dick was the defendant, the analysts concluded that the perpetrator was likely a heterozy-
gous 12,17 (like Dick). This led some of the analysts to say that the correct formula to use for
estimating the probability of a coincidental inclusion was 2pq, where p and q are the frequencies of
the 12 and 17 alleles. This is a slightly wider target than p
2
, but still not wide enough because this
target excludes both Tom and Harry (who were included when they were the Defendant.)
Some laboratories are more cautious and, in a case like this, would estimate the probability of
a coincidental match at locus D3 using the formula 2p, where p is the frequency of the 17 allele.
Under this interpretation, all that is necessary to be included is that the defendant possesses the 17
allele. This creates a more appropriate target because it would include Tom, Dick, Harry and Sally.
Nevertheless, the uncertainty about what is required for an inclusion still leaves room to doubt
whether the conservative 2p formula is conservative enough. Suppose, for example, that we posit a
defendant (let us call her Jane) who has genotypes 12,14 at D3, 15,15 at vWA and 20,25 at FGA.
Is it a certainty that Jane would be excluded? I believe most forensic DNA analysts would conclude
that the evidentiary sample might be a mixture to which Jane could be a secondary contributor
and therefore would not exclude Jane (even though she lacks the 17 allele at locus D3). Hence, I
believe that even the 2p estimate, which forensic analysts regard as extraordinarily conservative,
still underestimates the probability of a coincidental inclusion.
The tendency to underestimate the probability of a coincidental match can occur at every locus
of a case like this, and hence can affect the match probability assigned to the overall prole in a mul-
tiplicative manner. Let us next consider locus vWA. Faced with a suspect like Tom, Dick or Harry,
most laboratories would treat the evidentiary prole as a single-source sample and would therefore
use the formula 2pq to compute the probability of a coincidental match. In other words, they would
assume that a suspect must have both the 15 allele and the 17 allele to be included. But this formula
is underinclusive because it does not include Sally. When faced with a suspect like Sally, the theory
that the evidentiary prole is a mixture is invoked and the target shifts accordingly. In a case where
Sally is the defendant, I believe most laboratories would compute the probability of a coincidental
inclusion at locus vWA using the formula p
2
+2pq+q
2
, which assumes that each of the contributors
had either genotype 15,15 (like Sally) or 15,17 or 17,17. This formula is sometimes called the ran-
dom man not excluded formula and is also touted as extremely conservative. Once again, however, I
believe this conservative approach is not conservative enough in a case like this because it assumes
that all alleles of all contributors have been detected. As the Harry case illustrates, analysts are not
necessarily willing to make this assumption when faced with a suspect who has an allele that was
not detected in the evidentiary sample. Suppose, for example, that Tom had had genotype 15,18 at
locus vWA. Would he have been excluded? I think not. I think most analysts would conclude that
the evidentiary sample might be a mixture to which Tom could be one of the two contributors. The
fact that he has an allele at vWA that was not detected in the mixture would be attributed to allelic
dropout.
7. More evidence of target shifting (and underinclusive statistics)
I realize that, to this point, my argument rests heavily on extrapolations from the informal Tom,
Dick and Harry experiment. Readers might wonder whether interpretation of DNA test results is
really as exible as this experiment suggests and whether laboratories actually underestimate the
probability of a coincidental inclusion as much as the discussion above implies. So, let me turn to a

b
y

g
u
e
s
t

o
n

N
o
v
e
m
b
e
r

3
,

2
0
1
1
h
t
t
p
:
/
/
l
p
r
.
o
x
f
o
r
d
j
o
u
r
n
a
l
s
.
o
r
g
/
D
o
w
n
l
o
a
d
e
d

f
r
o
m

second line of evidence that supports my basic thesisthat target shifting by DNA analysts leads
them to underestimate the true likelihood of a coincidental inclusion.
The second line of evidence comes from cases in which forensic laboratories have computed a
random match probability in the absence of a suspect or before all suspects are presented. In other
words, these are cases in which the laboratory must paint the target before all the shots are red.
If I am correct that laboratories use targets that are too small and engage in target shifting, then
they should occasionally be caught out in such cases. They should occasionally refuse to exclude a
suspect who fails to t the small target that they drew when computing the frequency of included
proles (thereby proving that the target was too small). That is exactly what I have observed in
several cases.
One such case was that of a prison inmate who sought post-conviction DNA testing in an effort
to prove he was innocent of rape. A forensic laboratory tested a semen sample found on the victims
underwear and a reference sample fromthe victim. Based on that evidence, and without having tested
the inmate, the laboratory inferred the prole of the rapist and generated a statistical estimate of the
random match probability. Figure 3 shows the electropherograms of the semen sample (labelled
Panties, sperm). Below each electropherogram is a chart showing the laboratorys conclusions
regarding the alleles in the panties/sperm sample and the prole of the rapist. The chart also lists the
prole of the victim, which was determined in a separate assay. The bottom row in the chart shows
the formula the laboratory used to estimate the random match probability in this case.
As discussed earlier, the formula used to compute the frequency denes the size of the target.
By using the formula 2pq to compute the frequency of matching proles at locus THO1, e.g. the
laboratory is implicitly saying that only persons with a particular heterozygous genotype (7,9) will
be included as potential sources of the male component of the sperm sample and, by implication,
that persons with any other genotype will be excluded. And this assertion is the basis for the random
match probability that will be used to characterize the value of the evidence if a match is found. If,
as I believe, the laboratory is dening the target too narrowly, then it should be possible to produce
a suspect who does not have the obligatory genotype but whom the laboratory nevertheless refuses
to exclude.
That is what happened in this case. The inmate was tested after the laboratory inferred the prole
of the rapist and computed the random match probability. His prole matched the inferred prole
of the rapist at every locus except THO1. At that locus his genotype was 7,7. So what did the
laboratory do? As in the case of Sally discussed above, the laboratory invoked the theory of an
additional unknown contributor. Under this theory, the inmate could have contributed the 7 allele
at THO1 while the 9 allele came from a second unknown contributor.
7
Based on this theory, the
laboratory decided that the inmate could not be excluded and he therefore remains in prison.
8
The failure to exclude the inmate makes it clear that the laboratorys statistical estimates were
underinclusivei.e. that they understated the actual probability of a coincidental inclusion. As with
the Tom, Dick and Harry case, however, it is unclear what the actual random match probability is.
7
The victim, whose genotype is 8,10, could not account for the 9 allele, although her DNA could explain the 10 allele
if we also invoke the theory of allelic dropout to explain the failure to detect her 8 allele.
8
Interestingly, there is no indication in the laboratory notes or the laboratory reports that anyone suspected the sperm
sample contained DNA of three people (the victim and two others) until after the inmate was tested. It seems rather unlikely
that a random individual would have a prole that overlapped sufciently with the victim and inmate that its presence was
betrayed by only a single allele. To avoid excluding the inmate, then, the laboratory invoked a theory that, a priori, seemed
rather unlikely. Like the target-shifting problem that I am discussing here, this practice also reduces the value of a DNA match
in ways that are not reected in standard random match probabilities, but that is a topic for another paper.

b
y

g
u
e
s
t

o
n

N
o
v
e
m
b
e
r

3
,

2
0
1
1
h
t
t
p
:
/
/
l
p
r
.
o
x
f
o
r
d
j
o
u
r
n
a
l
s
.
o
r
g
/
D
o
w
n
l
o
a
d
e
d

f
r
o
m

FIG. 3. Electropherograms of the evidentiary sample in a post-conviction case (with the laboratorys inferences regarding the
rapists prole and formulae for estimating the random match probability).

b
y

g
u
e
s
t

o
n

N
o
v
e
m
b
e
r

3
,

2
0
1
1
h
t
t
p
:
/
/
l
p
r
.
o
x
f
o
r
d
j
o
u
r
n
a
l
s
.
o
r
g
/
D
o
w
n
l
o
a
d
e
d

f
r
o
m

After testing the inmate, the laboratory revised its estimate of the random match probability by
changing its frequency formula, but only for locus THO1. The laboratory determined that at THO1,
a contributor could have any of the following genotypes: 7,9, or 7,?, or 9,?, where the question mark
indicates any possible allele. (In other words, the laboratory assumed that the test may have failed
to detect all of the contributors alleles at this locus.) Accordingly, the laboratory used the formula
2pq + 2p + 2q to compute the frequency of included genotypes at THO1. At every other locus,
however, the laboratory used the same formulae it had used before.
This second statistical estimate is obviously yet another example of painting the target around
the arrow. If allelic dropout is possible at locus THO1, then it is also possible at other loci. Why
then did the laboratory not take the possibility of allelic dropout into account when computing the
frequency of matching genotypes at these other loci? The answer, of course, is that the laboratory
only paints targets where it sees arrows. Would the laboratory really have excluded the inmate if
he had, say, genotype 23,28 at locus FGA or genotype 31,31 at locus D21? I seriously doubt it.
Consequently, I think the revised random match probability computed by the laboratory was still
underinclusive.
How should the laboratory have computed the random match probability? That question is
difcult to answer because, as discussed above, no one knows what the actual criterion for inclusion
might be in a case like this. Unless laboratories can delineate in advance (i.e. without knowing the
prole of suspects) what genotypes are included and excluded at each locus, and stick by those
determinations, their statistical estimates are likely to be underinclusive in many if not most cases.
The degree of the bias introduced by target shifting is difcult to estimate and undoubtedly varies
from case to case. But the bias clearly exists and will continue to do so until laboratories adopt more
rigorous procedures for interpreting test results.
8. DNA and indeterminacy
In the absence of clear standards for inclusion and exclusion, different experts evaluating the
same evidence may reach different conclusions: one may conclude that a particular suspect is in-
cluded, while another concludes that the same suspect is excluded. An interesting example of such
a disagreement arose in the case of Commonwealth v. Leon Winston, which was tried in the state of
Virginia in 2003. Winston was charged with a horrendous double homicide in which a man and a
woman were shot and killed in the presence of the womans young children during a drug-related
robbery. Winston was convicted and sentenced to death.
One piece of evidence used to link Winston to the crime was his inclusion as a possible con-
tributor to DNA on a glove that was found near the murder scene. There was uncontested testi-
mony that a man named Hardy was the owner of the glove and that he had loaned the glove and
its mate to a man named Brown, who allegedly assisted Winston in the crime. The prosecution the-
ory was that Brown gave the gloves to Winston, who wore them while committing the crime, and
discarded them thereafter. A DNA analyst from the state crime laboratory testied for the prose-
cution that the left glove of the pair contained a mixed DNA prole in which Hardy, Brown and
Winston were all included as possible contributors. However, an expert called by the defense, who
reviewed the work of the state laboratory, reached a different conclusion on the same evidence. She
testied that while Hardy and Brown should be included as possible contributors, Winston should be
excluded.

b
y

g
u
e
s
t

o
n

N
o
v
e
m
b
e
r

3
,

2
0
1
1
h
t
t
p
:
/
/
l
p
r
.
o
x
f
o
r
d
j
o
u
r
n
a
l
s
.
o
r
g
/
D
o
w
n
l
o
a
d
e
d

f
r
o
m

FIG. 4. DNA proles used to link Winston to the glove.
Figure 4 shows the DNA proles reported by the state laboratory.
9
The laboratory used gel
electrophoresis with a commercial test kit called Promega PowerPlex that examines 15 STR loci.
However, the test produced results at only 10 of the 15 loci. Close examination of the alleles detected
at those 10 loci, which are listed in Fig. 4, reveals the source of the disagreement between the experts.
Although the mixed prole found on the glove contains a number of alleles consistent with the three
putative contributors, some of their alleles are missing. The bold numbers in Fig. 4 identify alleles
possessed by contributors that were not detected in the sample from the glove.
The missing alleles did not trouble the prosecution expert, who attributed their absence to allelic
dropout. The defense expert agreed that allelic dropout could account for the absence of some of
Browns and Hardys alleles, but apparently concluded that too many of Winstons alleles were miss-
ing for him to be a plausible contributor and therefore decided that Winston should be excluded.
At this point in the history of DNA testing, it is impossible to determine which expert is right and
which expert is wrong in disagreements of this type because, as noted above, there are no standards
for distinguishing inclusions and exclusions. Leading thinkers in the eld have recognized this
problem and have commented on how an expert might approach the analysis of evidence in cases
where allelic dropout might have occurred (Buckleton and Triggs, 2005; Gill et al., 2006). But
these commentaries and suggestions have, to my knowledge, not yet been translated into rules of in-
terpretation that an analyst can easily apply when interpreting evidence like that in the Winston case.
In most laboratories in the USA, such determinations have been and remain a matter of subjective
judgment.
9. Misuse of likelihood ratios
The experts in the Winston case also differed dramatically over the proper statistical characterization
of the evidence linking the defendant to the glove. The states expert presented a likelihood ratio
declaring that:
The DNA prole I obtained from the sample from the glove is 1.8 billion times more
likely if it originated from Leon Winston, Kevin Brown and David Hardy than if it
originated from three unknown individuals in a Caucasian population. Its 1.1 billion
times more likely it originated from these three individuals than if it originated from
three unknown individual [in] the black population; 2.9 billion times more likely to have
originated from these same three individuals than if it originated from three unknown
individuals in the Hispanic population. (Reporters Transcript, Vol X., p. 33536).
9
The analyst chose to list some alleles in parentheses to indicate that the bands indicating those alleles are less intense
than types not in parentheses. Figure 4 replicates the analysts reporting of those alleles.

b
y

g
u
e
s
t

o
n

N
o
v
e
m
b
e
r

3
,

2
0
1
1
h
t
t
p
:
/
/
l
p
r
.
o
x
f
o
r
d
j
o
u
r
n
a
l
s
.
o
r
g
/
D
o
w
n
l
o
a
d
e
d

f
r
o
m

Assuming the testimony was correctly transcribed, the expert appears to have botched the basic
description of the likelihood ratios that she computed for the Black and Hispanic populations. The
likelihood ratios she was attempting to explain describe how much more likely the evidence is under
one hypothesis (Hardy, Brown and Winston were the contributors) than another hypothesis (three
unknown persons were the contributors), not the likelihood that the evidence originated from these
three individuals. After her reference to the Caucasian population, she seems to have inverted the
relevant conditional probabilities.
The prosecutor later made matters worse by mischaracterizing (further) what the expert had said.
He attempted to explain the experts statistical statement about the glove evidence as follows:
[S]he said that the odds of nding a random match unrelated to the defendant, just
nding someone randomly that had those same characteristics and those same loci, is
one in 1.1 billion in the black race. (Reporters Transcript, Vol XI, p. 176)
The prosecutor, it appears, converted the likelihood ratio (for the black race) into a random
match probability.
Beyond these unfortunate mischaracterizations, there are several problems with the likelihood
ratios that the states expert presented. First, the two hypotheses that the expert compared were
neither relevant nor appropriate for the case. As noted above, it was uncontested that Hardy and
Brown had had contact with the gloves. The real issue was whether Winston, rather than some
unknown person, was the third contributor. Hence, the expert should not have asked how much more
likely the observed evidence would be if the DNA on the gloves originated from Hardy, Brown and
Winston than if it originated from three unknown persons; she should have asked how much more
likely the evidence would be if it originated from Hardy, Brown and Winston than if it originated
from Hardy, Brown and a random unknown person. By my calculations, the answer the expert would
have reached to this latter more appropriate question (using data for African Americans) was 789,
rather than 1 100 000 000. This is a big difference, although, as discussed below, I believe even the
more modest likelihood ratio of 789 greatly overstates the value of this evidence for incriminating
Winston.
10. Another example of target shifting
A second problem with the likelihood ratios computed by the states expert in the Winston case is
that they failed to take full account of the evidence. When testing the glove, the expert obtained
reportable results at the 10 loci shown in Fig. 4. However, the expert used data from only ve of
these loci when computing the likelihood ratios. The ve loci she chose (FGA, D8, vWA, D18 and
D5) were those for which all alleles of the three putative contributors had been detected. She elected
to ignore the other ve loci (Penta E, D21, THO1 and D3) where the test had failed to detect one or
more alleles of the putative contributors.
Why did she choose to focus on only 5 of the 10 loci when computing statistics? The defense
lawyer asked her about this during cross-examination. Her answers made it clear that there was
nothing about the test results on glove sample that dictated this choice. She focused on those loci
because those were the ones where she had found a match with the putative contributors. After the
defense lawyer pointed out that neither of Winstons alleles at locus D7 had been detected in the
glove sample, the following exchange occurred:

b
y

g
u
e
s
t

o
n

N
o
v
e
m
b
e
r

3
,

2
0
1
1
h
t
t
p
:
/
/
l
p
r
.
o
x
f
o
r
d
j
o
u
r
n
a
l
s
.
o
r
g
/
D
o
w
n
l
o
a
d
e
d

f
r
o
m

Q: You didnt include [locus D7] in your calculations, did you? A: No, because like
I said before, in order to calculate statistics, I only chose the loci that had the three
individuals I was including intheir complete proles in. (Reporters Transcript, Vol X,
p. 356)
In other words, this analyst only painted targets where she saw arrows. By dredging through her
ndings, the analyst risked ignoring data that tended to disconrm the governments theory of the
case while focusing only on that portion of the data that tended to conrm the governments theory.
This is a cardinal sin of statistical inference.
Interestingly, the defense expert made a similar error. Rather than computing likelihood ratios,
the defense expert computed random man not excluded statistics. She testied that within the
African-American population, the probability of randomly selecting an individual who would be
included in [the mixture found on the glove] at the ve locations that were used in [the prosecution
experts] analysis is 1 in 195. (RT, p. 25) The frequency of 1 in 195 is obviously more modest than
even the likelihood ratio of 789 mentioned above. But is it conservative enough? I think not because,
once again, the analyst painted the target around the arrow.
When computing the random match probability, the defense expert, like the prosecution expert,
focused only on the ve loci where all alleles of the putative contributors had been detected. For
those loci, she computed the random match probability by taking the sum of frequencies of all
included genotypes. For example, at locus vWA, where alleles 16, 17 and 20 were observed on the
glove, the expert computed the sum of the frequencies of genotypes 16,16; 17,17; 20,20; 16,17 and
16,20, on the assumption that anyone with one of those genotypes would be included as a potential
contributor to the glove at that locus (and presumably that anyone with a different genotype would
be excluded).
For the other ve loci (where some alleles of the putative contributors were not detected), the
defense expert set the random match probability to 1. Her rationale was that no one can be excluded
at these loci due to the possibility of allelic dropout (undetected alleles). If no one can be excluded,
then everyone is included, making the probability of a random match 1 (1.0) for that locus. Her
logic seems sound as far as it goes. At locus D7, e.g. neither Winston nor Hardy was excluded even
though they have alleles that were not detected in the mixture. If they could not be excluded, then it
would appear that no one else could be either. No matter what genotype a suspect has, the suspect
cannot be excluded if the analyst assumes the suspects alleles were not detected.
The problem, of course, is that the analysts conclusion about whether all alleles were detected at
a locus appears to have been inuenced by knowledge of the proles of the putative contributors. If
the alleles of Hardy, Brown and Winston were detected at a locus, then she assumed that all alleles of
all potential contributors had been detected and computed the randommatch probability accordingly;
if one or more alleles of a putative contributor were not detected, she assumed that there was allelic
dropout and that no one could be excluded. This appears, then, to be another example of painting the
target around the arrow.
As with the earlier examples, this procedure may have led the analyst to underestimate the fre-
quency of people in the population who would have been included as potential contributors. Suppose,
for example, that Winstons genotype at locus vWA were 16,18 rather than 16,16. Would he have
been excluded? I seriously doubt it. I believe the analyst would simply have assumed that the failure
to detect the 18 allele was another instance of allelic dropout. In other words, I believe the occur-
rence of allelic dropout was inferred from the very data that it was invoked to explain. Hence, the

b
y

g
u
e
s
t

o
n

N
o
v
e
m
b
e
r

3
,

2
0
1
1
h
t
t
p
:
/
/
l
p
r
.
o
x
f
o
r
d
j
o
u
r
n
a
l
s
.
o
r
g
/
D
o
w
n
l
o
a
d
e
d

f
r
o
m

actual probability that a randomly chosen person would be included as a potential contributor was
higher, perhaps much higher, than the defense experts estimate of 1 in 195.
I believe that the only real requirement for including Winston as a contributor was that he pos-
sessed two alleles observed in the mixture that could not have come from Brown or Hardy (allele 22
at locus FGA and allele 15 at locus D18). But approximately half the human population possesses
those two alleles, so the value of this match for incriminating Winston seems rather slight. It is
indeed unfortunate, in a case where the defendants life was at stake, that the prosecutor told the jury
the probability of a random match to an important sample was 1 in 1.1 billion. In my opinion, the
actual random match probability is close to 1 in 2; hence, the number the prosecutor gave the jury
may have understated the true value by approximately nine orders of magnitude.
11. Can likelihood ratios solve the problem?
In 2006, a DNA Commission of the International Society of Forensic Genetics offered a helpful
analysis of mixture interpretation that included extensive discussion of the problem of allelic dropout
(Gill et al., 2006). The Commission suggested that a likelihood ratio be computed for each locus. It
suggested that the possibility of allelic dropout be addressed by incorporating a dropout probability,
designated Pr(D), into the likelihood ratio. For a locus where the suspect has genotype ab, but only
allele a is observed in the evidentiary sample, the Commission recommended a formula initially
proposed by Buckleton and Triggs (2005):
LR = Pr(D)/p
a
( p
a
+2Pr(D)(1 p
a
)),
where p
a
is the frequency of allele a.
Although this formula makes sense conceptually, it is difcult to apply even in simple cases due
to uncertainty about the dropout probability, Pr(D). The Commission noted that many laboratories
have carried out experiments that provide information relevant to Pr(D). In general, the probability
of dropout of one allele in a genotype increases as the peak height of the other allele decreasesin
other words, allelic dropout is more common when peak heights are low due to limited quantities of
DNA. Empirical study may allow determination of a threshold peak height above which Pr(D) 0,
and this could provide a basis for exclusions when the evidentiary sample includes one of the sus-
pects alleles above the threshold but not the other. However, estimation of Pr(D) when the observed
peaks are below this threshold will inevitably require a considerable element of guesswork, as the
relationship between peak height and Pr(D) is unlikely to be linear and Pr(D) may also vary based
on factors other than peak height, such as the nature and condition of the samples or the substrates
on which they are found.
Matters quickly become even more complicated when formulae incorporating Pr(D) are applied
to mixtures. Overlapping and masking alleles may make it difcult if not impossible to determine
whether one allele from a suspect is above or below the level where the other is likely to dropout.
Consider, e.g. locus PentaE in the Winston case. Although a 12 allele consistent with Winston was
found on the glove, the evidence for the presence of this allele could be accounted for, in part if
not in whole, by the 12 allele of Hardy. What then is the probability that Winstons 15 allele would
dropout, if he is indeed a contributor? I suspect that any answer an expert might provide to this
question based on a subjective evaluation would be no better than a wild guess, and that different ex-
perts might (with persuasive sounding justications) provide a variety of different estimates. Hence,
computation of Pr(D) will be problematic in such cases. Even under the best of circumstances, when

b
y

g
u
e
s
t

o
n

N
o
v
e
m
b
e
r

3
,

2
0
1
1
h
t
t
p
:
/
/
l
p
r
.
o
x
f
o
r
d
j
o
u
r
n
a
l
s
.
o
r
g
/
D
o
w
n
l
o
a
d
e
d

f
r
o
m

experts have adequate empirical data about dropout probabilities, the computations would need to
take account of peak heights at multiple loci, whether the evidentiary sample is degraded, whether
different components of the evidentiary sample might be degraded to different degrees and other
variables of this nature. The Commission acknowledged that [e]xpansion of these concepts to mix-
tures is complex and that is why they are not generally used. My concerns about it go beyond mere
complexity. I wonder whether these determinations fall in a realm of indeterminacy, where so many
unmeasured variables can affect the results that dropout probabilities simply cannot be determined
with any degree of accuracy.
12. Automated systems as a possible solution
One possible solution to the problem of target shifting is the use of automated (programmed) sys-
tems for prole interpretation (Gill et al., 2007; Perlin, 2003, 2006). These systems can deal with the
complexity of the computations needed to incorporate dropout probabilities and other such variables
into likelihood ratios. They also have the tremendous advantage of objectivity. Unlike human ana-
lysts, their interpretations of evidentiary proles are not inuenced by the prole of suspects, hence
they resolve the problem of target shifting. Moreover, they may extract information from underlying
genetic data more efciently than human analysts. According to promoters, these systems generally
produce more extreme (and impressive) likelihood ratios when evaluating forensic comparisons be-
cause they make more complete and efcient use of the available data than the simpler computational
methods typically used by laboratories.
If these claims are true, then automated systems may well solve the problems raised in this
paper. Of course, commercial claims of this type need to be carefully evaluated. Before accepting an
automated system for use in criminal cases and before accepting evidence produced by such a system
in court, we should demand to see a careful program of validation that demonstrates the system can
accurately classify mixtures of known samples under conditions comparable to those that arise in
actual forensic cases. The fact that an automated system can produce answers to the questions one
puts to it is no assurance that the answers are correct. While automated systems appear promising,
their ability to handle hard cases like those discussed in this paper remains to be fully evaluated.
It is also important that the operation of these systems be sufciently transparent to allow experts
to evaluate their conclusions. There must be a way to look inside the black box to determine
whether its conclusions in a particular case are well grounded. Because these systems are proprietary,
their underlying code is not readily available for examination. Nevertheless, there should be ways to
interrogate the system about the premises and assumptions underlying the conclusions in a particular
case. To generate a likelihood ratio in the Winston case, e.g. an automated system would need to
assess such issues as the probable number of contributors, the relative amount of DNA contributed
by each, the dropout probability at each locus and the degree of degradation. These intermediate
assessments should be accessible so that other experts can evaluate whether they are reasonable. A
system that works well in general may trip up and reach implausible intermediate conclusions in
some cases. If there is no way to evaluate whether that happened, it will raise concerns about the
suitability of the system for the legal arena.
13. Sequential unmasking as a possible solution
Another way to deal with the problem of target shifting is simply to use more rigorous procedures
for interpretationprocedures that mask or blind analysts to the proles of suspects when they

b
y

g
u
e
s
t

o
n

N
o
v
e
m
b
e
r

3
,

2
0
1
1
h
t
t
p
:
/
/
l
p
r
.
o
x
f
o
r
d
j
o
u
r
n
a
l
s
.
o
r
g
/
D
o
w
n
l
o
a
d
e
d

f
r
o
m

make critical judgments about the proles of evidentiary samples. Krane et al. (2008) recently pro-
posed a procedure for interpretation of DNA proles that they called sequential unmasking. While
the Krane et al. proposal was designed primarily to minimize observer effects in the interpreta-
tion of DNA proles, sequential unmasking would have the additional benet of preventing target
shifting.
The sequential unmasking procedure is simple and could easily be implemented in any foren-
sic laboratory. When interpreting a DNA test, analysts would begin by looking just at evidentiary
samples. In the initial interpretation, the analyst would determine the alleles associated with each
sample, assess the number of contributors and assess the likelihood at each locus that the test proce-
dure failed to detect some of the alleles of contributors (i.e. the dropout probability). At this point,
and before looking at any reference samples, the analyst would determine (and make record of) the
genotypes that would cause a person to be included or excluded at each locus.
After this initial interpretation, the analyst would unmask information about the reference sam-
ples in a sequential manner, beginning with samples from expected contributors, such as the victim
in a sexual assault case. After considering the proles of expected contributors, the analyst would re-
evaluate the evidentiary prole to determine the possible genotypes of unknown contributors. At this
stage, and before looking at the proles of any suspects, the analyst would compute the frequency (in
appropriate reference populations) of individuals who would be included as possible contributors.
By so doing, the analyst would be documenting an unbiased assessment the genotypes of possible
contributors. The analyst would, in effect, be xing the target and computing its size. Only when
these computations were recorded would the analyst undertake the nal step of determining whether
the suspects have the genotypes necessary to hit the target.
By forcing analysts to paint their targets before knowing where the shots hit the barn, sequential
unmasking would prevent target shifting and lead to more accurate and appropriate estimates of
random match probabilities. It is a simple and scientically rigorous procedure that could and should
be adopted by every laboratory.
Unfortunately, most laboratories in the USA have not yet adopted such procedures. Their failure
to do so can be seen in laboratory protocols, in laboratory notes and in the testimony of analysts
who often make it clear, as in the Winston case, that their interpretation of the evidentiary sample
depended, in part, on the prole of suspects. In my view, this is an endemic problem that affects
a signicant percentage of all forensic DNA cases. The problem may well have become worse in
recent years as DNA testing has been used increasingly to analyze marginal samples that produce
incomplete, ambiguous proles like those in the Tom, Dick and Harry case. As this paper shows,
target shifting in such cases can cause analysts to signicantly underestimate random match proba-
bilities and thereby overstate the value of DNA evidence. Forensic DNA laboratories can and should
take more care to avoid this problem.
14. Beyond DNA
Nowthat we have seen howthe Texas sharpshooter fallacy can affect the interpretation and character-
ization of forensic DNA tests, it is worth considering whether this fallacy might also inuence other
forensic disciplines, such as latent print and toolmark examination. While a complete discussion of
this question is beyond the scope of the present article, I will offer a few observations.
First, DNA testing is not the only discipline in which analysts must rely on subjective judgment
when determining whether two samples could have a common source. As a recent report of the

b
y

g
u
e
s
t

o
n

N
o
v
e
m
b
e
r

3
,

2
0
1
1
h
t
t
p
:
/
/
l
p
r
.
o
x
f
o
r
d
j
o
u
r
n
a
l
s
.
o
r
g
/
D
o
w
n
l
o
a
d
e
d

f
r
o
m

National Research Council (2009) has noted, match determinations in the pattern comparison dis-
ciplines generally require a subjective decision based on unarticulated standards. In fact, there may
well be more latitude for subjectivity in pattern matching disciplines than in DNA testing because
DNAanalysts look for the same set of features (alleles) in every sample, while the set of features used
to make pattern matches, such as latent print and toolmark identications, may differ in every case.
Second, in the pattern comparison disciplines, as in DNA testing, analysts typically fail to em-
ploy blind procedures. They know the characteristics of the reference samples and often are privy
to information suggesting which reference samples are likely (or expected) to match particular ev-
identiary samples (Risinger et al., 2002). As noted earlier, the combination of subjective judgment
with the failure to use blind procedures sets the stage for target shifting. In these circumstances,
there is a danger that knowledge of the reference samples will inuence analysts interpretations of
the evidentiary samples and their decisions about which features are relevant and irrelevant to the
comparison. For example, an examiner may decide to disregard features of an evidentiary sample
that fail to match a reference sample when the analyst would credit and use those same features if
they did happen to match.
A possible example of this process arose in the notorious Mayeld case, in which Federal
Bureau of Investigation (FBI) examiners mistakenly identied a latent print associated with a terror-
ist incident in Madrid as having been made by Brandon Mayeld, a lawyer from Portland, Oregon
(United States Department of Justice, Ofce of Inspector General, 2006; Stacey, 2005; Thompson
and Cole, 2005). It was later determined that the source of the print was an Algerian suspect named
Ouhnane Daoud (Thompson and Cole, 2005). There were discrepancies between the latent print and
Mayelds print which the FBI examiners decided to ignore, apparently because they attributed them
to distortion of the latent or the overlay of another print. Yet, these same features were deemed rel-
evant and informative when they matched Daouds print.
10
This example suggests that latent print
examiners can engage in target shifting (perhaps without even realizing it), just as DNA analysts
can. And the false match with Mayeld illustrates how target shifting can expand the match criteria
in ways that create a danger of false matchesways that analysts may not take fully into account
when judging the probability of a coincidental match.
In some disciplines, most notably latent print examination, analysts employ the analysis-
comparison-evaluation-verication (ACE-V) method, which requires identication of relevant fea-
tures in the evidentiary print and an assessment of whether there is sufcient detail to allow a mean-
ingful comparison (analysis) before the analyst examines the reference print to see if it contains
similar features (comparison) (Ashbaugh, 1999, 1991; Haber and Haber, 2008). The ACE-V method
may well prevent analysts from being inuenced by a reference print during the initial evaluation
of the evidentiary print (analysis phase), and hence may accomplish part of what the sequential
masking procedure is designed to accomplish in DNA interpretation. However, ACE-V does little to
prevent observer effects and target shifting once the analyst moves beyond the analysis stage, and
therefore cannot be regarded as comparable to (nor an acceptable substitute for) a more rigorously
blinded procedure like sequential unmasking.
The key innovation of sequential unmasking is a requirement placed on the analyst after analysing
the evidentiary sample and before looking at the reference sample. At this point, the analyst must
specify and record which features of a reference sample will lead to a determination of match
10
The US Department of Justice Report (2006) suggested that the FBI examiners had engaged in circular reasoning,
which is another way of saying that their interpretations of the evidentiary print were inuenced by their knowledge of
Mayelds reference print.

b
y

g
u
e
s
t

o
n

N
o
v
e
m
b
e
r

3
,

2
0
1
1
h
t
t
p
:
/
/
l
p
r
.
o
x
f
o
r
d
j
o
u
r
n
a
l
s
.
o
r
g
/
D
o
w
n
l
o
a
d
e
d

f
r
o
m

(inclusion) and which will lead to a determination of non-match (exclusion). This process xes the
target, thereby precluding the potential for target shifting and the sharpshooter fallacy. There is no
comparable specication and recording of matching criteria in ACE-V. To the extent ACE-V xes
the target, the x exists solely in the mind of the analyst. In light of the extensive evidence that
observer effects can operate without conscious awareness (Risinger et al., 2002; Thompson, 2009;
Krane et al., 2009), the effectiveness of a purely mental x seems doubtful. As the National Research
Council (2009, pp. 512) report notes, ACE-V does not guard against bias; it is too broad to ensure
repeatability and transparency; and does not guarantee that two analyst following it will obtain the
same results.
In sum, target shifting is likely to be a problem in the pattern-matching disciplinesa problem
that can lead analysts to underestimate the likelihood of coincidental matches. ACE-V is not an
adequate solution to this problem. As in the area of DNA analysis, more rigorous interpretive pro-
cedures, such as sequential unmasking, will be needed to prevent examiners from painting targets
around their forensic arrows.
Funding
National Science Foundation (SES-0617672).
REFERENCES
ASHBAUGH, D.R. (1999) QuantitativeQualitative Friction Ridge Analysis, Introduction to Basic Ridgeology.
Royal Canadian Mounted Police.
ASHBAUGH, D.R. (1991) Ridgeology. J. Forens. Ident., 41(1), 2664.
ANDERSON, C.A., LEPPER, M.R. & ROSS, L. (1980) Perseverance of social theories: The role of explanation
in the persistence of discredited information. J. Pers. Soc. Psych., 39, 10371049.
BUCKLETON J. & GILL, P. (2005) Low copy number. In Buckleton, J., Triggs, C.M. & Walsh, S.J. (Editors)
Forensic DNA Evidence Interpretation. CRC Press, 275297.
BUCKLETON, J. & TRIGGS, C. (2005) Is the 2p rule always conservative? Forensic Sci. Intl., 159(23),
206209.
BUTLER, J.M. (2005) Forensic DNA Typing: Biology, Technology and Genetics of STR Markers. Elsevier
Academic Press.
GAWANDE, A. (1999) The cancer-cluster myth. The New Yorker, Feb 8, 1999, 3437.
GILL, P., BRENNER, C.H., BUCKLETON, J.S., CARRACEDO, A., KRAWCZAK, M., MAYR, W.R.,
MORLING, N. PRINZ, M., SCHNEIDER, P.M. & WEIR, B.S. (2006) DNA commission of the Inter-
national Society of Forensic Genetics: Recommendations for the interpretation of mixtures. Forensic Sci.
Intl., 160, 90101.
GILL, P., KIRKHAM, A. & CURRAN, J. (2007) LoComatioN: A software tool for the analysis of low copy
number DNA proles. Forens. Sci, Intl., 166(23), 128138.
GILOVICH, T. (1991) How We Know What Isnt So. Free Press.
HABER, L. & HABER, R.N. (2008) Scientic validation of ngerprint evidence under Daubert. Law,
Probability & Risk, 7(2), 87109.
HESS, D.J. (1997) Science Studies: An Advanced Introduction. New York University Press.
KRANE, D.E., FORD, S., GILDER, J., INMAN, K., JAMIESON, A., KOPPL, R., KORNFIELD, I., RISINGER,
D.M., RUDIN, N., TAYLOR, M.S. & THOMPSON, W.C. (2008) Sequential unmasking: a means of
minimizing observer effects in forensic DNA interpretation. J. Forens. Sci., 53(4), 10061007.

b
y

g
u
e
s
t

o
n

N
o
v
e
m
b
e
r

3
,

2
0
1
1
h
t
t
p
:
/
/
l
p
r
.
o
x
f
o
r
d
j
o
u
r
n
a
l
s
.
o
r
g
/
D
o
w
n
l
o
a
d
e
d

f
r
o
m

KRANE, D.E., FORD, S., GILDER, J., INMAN, K., JAMIESON, A., KOPPL, R., KORNFIELD, I., RISINGER,
D.M., RUDIN, N., TAYLOR, M.S. & THOMPSON, W.C. (2009) Authors response (to commentary on
sequential unmasking). J. Forens. Sci., 54(2), 501.
LONGINO, H.E. (1990) Science as Social Knowledge: Values and Objectivity in Scientic Inquiry. Princeton
University Press.
NATIONAL RESEARCH COUNCIL. (2009) Strengthening Forensic Science in the United States: A Path
Forward. The National Academies Press (prepublication version).
NEUTRA, R.R. (1990) Review and commentary: counterpoint from a cluster buster. Am. J. Epidemiol., 132,
18.
NISBETT, R. & ROSS, L. (1980) Human Inference: Strategies and Shortcomings of Social Judgment. Prentice
Hall.
PAOLETTI, D.R., DOOM T.E., KRANE, C.M., RAYMER, M.L. & KRANE, D.E. (2005) Empirical analysis
of the STR proles resulting from conceptual mixtures. J. Forens. Sci., 50(6) 136166.
PERLIN, M. (2003) Simple reporting of complex DNA evidence: automated computer interpretation. Proceed-
ings of Promegas 14t h International Symposium on Human Identication.
PERLIN, M. (2006) Scientic validation of mixture interpretation methods. Proceedings of Promegas 17th
International Symposium on Human Identication.
RISINGER, D.M., SAKS, M.J., THOMPSON, W.C. & ROSENTHAL, R. (2002) The Daubert/Kumho implica-
tions of observer effects in forensic science: hidden problems of expectation and suggestion. Calif. L. Rev.
90(1), 156.
ROTHMAN, K.J. (1990) A sobering start for the cluster busters conference. Am. J. Epidemiol 132 (Suppl),
S6S13.
STACEY, R.B. (2005) Report on the erroneous ngerprint individualization in the Madrid Train Bombing case.
J. Forens. Ident., 54, 706718.
THOMAS, D.C., SIEMIATYCKI, J., DEWAR, R. (1985) The problem of mulitiple inference in studies designed
to generate hypotheses. Am. J. Epidemiol., 122, 10801095.
THOMPSON, W.C. (2009) Interpretation: Observer Effects. In A. Jamieson (Editor) & A. Moenssens (co-
Editor), Wiley Encyclopedia of Forensic Science. John Wiley & Sons Ltd., Chichester, UK, pp 15751579.
THOMPSON, W.C. & COLE, S.A. (2005) Lessons from the Brandon Mayeld case. The Champion, 29, 3234.
UNITED STATES DEPARTMENT OF JUSTICE, OFFICE OF INSPECTOR GENERAL. (2006) A Review of the FBIs
Handling of the Brandon Mayeld Case. United States Government Printing Ofce.

b
y

g
u
e
s
t

o
n

N
o
v
e
m
b
e
r

3
,

2
0
1
1
h
t
t
p
:
/
/
l
p
r
.
o
x
f
o
r
d
j
o
u
r
n
a
l
s
.
o
r
g
/
D
o
w
n
l
o
a
d
e
d

f
r
o
m

await a transient ischaemic attack
w2
or, even worse, an
acute infarction for which an urgent endarterectomy is
required
w3
is therefore not good advice. However,
others take a contrary view, perhaps because of a lack
of facilities, excessive competition rates owing to poor
selection of candidates, or inept surgery. Moreover, an
attitudinal bias may also exist regarding prevention
among doctors who have been trained to intervene
only if malfunction of an organ becomes symptomatic.
The degree of stenosis is measured by different
methods, and for most specialists 60% stenosis is the
cut-off point for selecting patients for endarterectomy.
This has led to an erroneous concept that a minimumof
60% stenosis of the internal carotid lumen is the essen-
tial criterion.
3
However, other key indicators are
turbulent flow caused by stenosis, sludge due to eddy
currents, particulate microemboli, and wall abnormali-
ties that are resistant to medical management.
4
Screening for asymptomatic carotid atherosclerosis
by using auscultation for bruits and duplex ultra-
sonography is feasible and is currently the best way of
identifying preclinical atherosclerosis.
5
Patients identi-
fied by preliminary screening to determine flow
dynamics, arterial wall characteristics including
stenosis and ulceration, and microemboli, to identify
those for whommedical management is needed and to
assess the effect of medical remediation.
6 w4
If medical
intervention fails, ACST has proved once and for all
that carotid endarterectomy can be worth the risk if
surgical and anaesthetic skills are such that operative
complications are rare.
7
International collaborative studies such as these
require a huge investment of time, skill, and money and
are an endorsement of evidence based medicine first
promulgated by Austin Bradford Hill and Sir Richard
Doll.
8 w5
For the field of stroke, the baseline from which
they evolved were the autopsy findings of Miller
Fisher,
w6
followed by the landmark report by Eastcott,
Pickering, and Robb at St Marys Hospital in London.
9
Carotid endarterectomy has now come full circle, hav-
ing been validated by Halliday, Thomas, and colleagues
of the same institution.
2
Their multinational effort con-
tinues the search for better methods by which to iden-
tify people with atherosclerosis who should be
considered for medical and surgical intervention.
So far, differentiating symptomatic from asympto-
matic stenosis of the carotid artery has traditionally
been the way to decide on treatment. But this requires
a doctor skilled in neurology to make the judgment.
Moreover, the occurrence of transient ischaemic
attacks is not a satisfactory means of categorisation
because they are very seldom witnessed, cannot be
assessed objectively, are confounded by many other
transitory phenomena, and may occur during sleep
when they cause no recognisable phenomena or in
parts of the brain that do not produce symptoms or
signs.
10 11
Moreover, 3-10% of people older than 65
have asymptomatic infarcts visible on brain imaging.
12
Depending on transient ischaemic attacks for
categorising patients is therefore unacceptable as the
sole criterion for choosing treatment, and preclinical
stenosis and unrecognised transient ischaemic attacks
need to be identified by screening.
James F Toole director
Stroke Research Center, Wake Forest University Baptist Medical
Center, Winston-Salem, NC 27157-1068 USA
(jtoole@wfubmc.edu)
Competing interests: None declared.
1 The Executive Committee for the Asymptomatic Carotid Atherosclerosis
Study. Endarterectomy for asymptomatic carotid artery stenosis. JAMA
1995;273:1421-8.
2 MRC Asymptomatic Carotid Surgery Trial (ACST) Collaborative Group.
Prevention of disabling and fatal strokes by successful carotid
endarterectomy in patients without recent neurological symptoms:
randomized controlled trial. Lancet 2004;363:1491-502.
3 Toole JF, Castaldo JE. Accurate measurement of carotid stenosis. Chaos in
methodology. J Neuroimaging 1994;4:222-30.
4 Fisher CF. Transient ischemic attacks. Perspective. N Engl J Med
2002;347:1642-3.
5 Toole JF, Chambless LE, Heiss G, Tyroler HA, Paton CC. Prevalence of
stroke and transient ischemic attacks in the atherosclerosis risk in
communities (ARIC) study. Ann Epidemiol 1993;3:500-3.
6 Chambless LE, Heiss G, Shahar E, Earp MJ, Toole J. Prediction of
ischemic stroke risk in the Atherosclerosis Risk in Communities Study.
Am J Epidemiol 2004;160:259-69.
7 Toole JF. Quality-based medicine. Arch Neurol 1997;54:23.
8 Doll R. Controlled trials: the 1948 watershed. BMJ 1998;317:1217-20.
9 Eascott HHG, Pickering GW, Robb CG. Reconstruction of the internal
carotid artery in a patient with intermittent attacks of hemiplegia. Lancet
1954;264:994-6.
10 Fisher CM. Transient ischemic attacks. Perspective. N Engl J Med
2002;347:1642-3.
11 Toole JF. The Willis Lecture: transient ischemic attacks, scientific method,
and new realities. Stroke;1991;22:99-104.
12 Brott T, Thomsick T, Feinberg W, Johnson C, Biller J, Broderick J, et al for
the Asymptomatic Catorid Atherosclerosis Study Investigators. Baseline
silent cerebral infarction in the asymptomatic carotid atherosclerosis
study. Stroke 1994;25:1122-9.
Forensic science in the dock
Postmortem measurements of drug concentration in blood have little meaning
I
nvestigations into the circumstances surrounding
the death of David Kelly have led to the exchange
of acrimonious views including allegations of
conspiracy and murder. David Kelly, a government
scientist and weapons expert, committed suicide by
cutting his wrist and taking painkillers after he was
identified in newspapers as the man the UK
government believed was the source for a BBC report
on Iraq. Impetus for the debate stems mainly from
conflicting views about the cause of death, including
issues that relate to postmortem toxicology results
and their interpretation. Controversy occurs from the
mistaken notion that postmortem laboratory meas-
urements, taken in isolation, can be interpreted
effectively.
The current controversy illustrates some univer-
sally held, but mistaken, notions about the process of
death investigation in the United Kingdom and
elsewhere. Many assume that forensic pathology is as
evidence based as other branches of medicine. This
assumption is not accurate.
In the course of caring for living patients, doctors
who interpret hospital laboratory tests know, or can
quickly find out, the normal value for any particular
drug. But most doctors (as well as the general public)
would be surprised to learn that there are few if any
Editorials
BMJ 2004;329:6367
636 BMJ VOLUME 329 18 SEPTEMBER 2004 bmj.com
normals in postmortem toxicology. Non-circulating
blood after death is not the same thing as circulating
blood before death, and evidence that the concepts of
normal or therapeutic drug concentrations can be
applied to blood from dead bodies is severely lacking.
Even in living bodies, interpretation of a single
blood concentration measurement is impossible
without considering route of administration, number
of doses taken, and the amount of drug actually in the
body. Such information is almost never available to
investigators, making it impossible to determine the
cause of death solely by comparing a single
postmortem drug concentration measurement with a
range of published values, originally derived from
measurements made in living people. With chronic
use, tolerance occurs, and tolerance cannot be
measured or estimated after death. Healthy patients
enrolled in methadone maintenance programmes, for
example, may have blood methadone concentrations
in excess of other, non-tolerant methadone users
examined on the autopsy table.
1
Similarly, we have long
known that blood sampled from the heart of a dead
person who had been on long term digoxin treatment
may contain a seemingly toxic concentration of
digoxin when, in fact, the actual blood concentration
immediately before death was the appropriate
non-toxic therapeutic concentration.
2
Even if it could be shown that blood concentrations
after death were the same as concentrations at the time
of death, which blood sample should be used? Drug
concentrations are likely to have changed after death.
3
For many drugs, including those found in David Kelly,
concentrations may increase by as much as 10-fold.
4
Furthermore, drug concentrations in blood samples
from cadavers are site dependent, higher in some loca-
tions and lower in others.
5
Should the site yielding the
lowest or highest result be used? Or should an average
value for three sites be used? Nobody knows because
the process has never been studied systematically.
If the blood concentration at the time of death can-
not be known with certainty, then how is it possible to
extrapolate the time and amount of drug ingested
before death? The simple answer is that such extrapo-
lations are prone to considerable error and generally
should be viewed as unreliable and not evidence
based.
6
Despite these limitations, such calculations are
frequently and wrongly produced during court
proceedings, even though the problems we outline
have been widely known for many years.
Postmortem measurements of drug concentration
in blood have scant meaning except in the context of
medical history, the sequence and circumstances
surrounding death, and necropsy findings. The paucity
of evidence based science, coupled with the pretence
that such science exists in regard to postmortem
toxicology, leads to the abuse of process, almost
certainly to the miscarriage of justice, and possibly
even to false perceptions of conspiracy and cover up.
We have written this editorial partly because of the
Kelly matter, where the central issue concerned the
interpretation of the toxicology results. Death investi-
gation and forensic pathology are also not immune to
misinterpretation. Poor or inadequate death investiga-
tion and incomplete or misinterpreted forensic pathol-
ogy studies may also result in wrong conclusions. All
aspects of the medicolegal death investigation triad
investigation (history), pathology, and laboratory
resultsare essential and must be evaluated in context
with one another. We have formed an ad hoc group to
address this issue. A detailed analysis of the problem
with suggestions for reform is in preparation.
Olaf Drummer adjunct professor
Victorian Institute of Forensic Medicine, 57-83 Kavanagh Street,
Southbank, Victoria 3006, Australia
(olaf@vifm.org)
A Robert W Forrest professor of forensic toxicology
University of Sheffield, Sheffield S3 7ES
(robertforrest@mac.com)
Bruce Goldberger associate professor
Department of Pathology, Immunology and Laboratory Medicine,
University of Florida College of Medicine, PO Box 100275,
Gainesville, FL 32610-0275, USA
(bruce-goldberger@ufl.edu)
Steven B Karch assistant medical examiner
PO Box 5139, Berkeley, California 94705-0139, USA
(skarch@sonic.net)
On behalf of the International Toxicology Advisory Group
(ITAG) (see bmj.com for list of members).
Competing interests: appear on bmj.com
1 Karch SB, Stephens BG. Toxicology and pathology of deaths related to
methadone: retrospective review. West J Med 2000;172:11-4.
2 McKercher HG, Mikhael NZ, De Gouffe M, Lukaszewski T, Peel HW.
Digoxin concentrations in postmortem human tissues. Res Commun Chem
Pathol Pharmacol 1986;52:141-4.
3 Pounder DJ, Jones GR. Post-mortem drug redistributiona toxicological
nightmare. Forensic Sci Int 1990;45:253-63.
4 Drummer OH, Gerostamoulos J. Postmortem drug analysis: analytical
and toxicological aspects. Ther Drug Monit 2002;24:199-209.
5 Hearn WL, Keran EE, Wei HA, Hime G. Site-dependent postmortem
changes in blood cocaine concentrations. J Forensic Sci 1991;36:673-84.
6 Cook DS, Braithwaite RA, Hale KA. Estimating antemortem drug
concentrations from postmortem blood samples: the influence of
postmortem redistribution. J Clin Pathol 2000;53:282-5.
Compulsory registration of clinical trials
Will be a requirement before submission to the BMJ from July 2005
T
he case for registering all clinical trials
first advanced a decade ago
1
is now unan-
swerable.
2
Editors of the BMJ and the
Lancet made this statement in 1999. Five years of
industry resistance, government impotence, and public
confusion followed. Medical journals persisted with
noble intentions and wise words but were themselves
in part resistant, impotent, and confused about how to
enforce registration. Some journals, including the BMJ,
tried an amnesty for unpublished trials, with little suc-
cess.
3
The BMJ also considered asking for compulsory
registration, but it seemed to us that trial registries were
too diverse, disorganised, and easily disregarded to
insist on registration before submission. Nor did we
Editorials
BMJ 2004;329:6378
637 BMJ VOLUME 329 18 SEPTEMBER 2004 bmj.com
65
64
63
62
61
60
59
58
57
56
55
54
53
52
51
50
49
48
47
46
45
44
43
42
41
40
39
38
37
36
35
34
33
32
31
30
29
28
27
26
25
24
23
22
21
20
19
18
17
16
15
14
13
12
11
10
9
8
7
6
5
4
3
2
1
www.sciencemag.org SCIENCE VOL 297 19 JULY 2002 339
O
n 7 January of this year, Judge
Lewis Pollak, former dean of the
Yale Law School and a highly re-
spected federal district court judge,
stunned the legal world when he held that
fingerprint experts could not testify that a
latent print found at the scene of the crime
matched the defendants print (1). De-
spite being admitted into courts for nearly
100 years, Judge Pollak found that no one
had bothered to conduct any meaningful
research on the technique. His ruling was
based on the landmark opinion in Daubert
v. Merrell Dow Pharmaceuticals, Inc., in
which the Supreme Court held that trial
court judges must ensure, as gatekeepers,
that proffered scientific evidence is valid
and reliable. On 13 March, however, Judge
Pollak reversed his earlier ruling (2). Upon
reflection, he said, he had come to the re-
alization that fingerprint identification
was not a matter of science, it was a
specialization, and thus need not meet
the rigors of the scientific method to be
admitted in court. This distinction between
science and specialization is premised on a
basic skepticism of the scientific method
and its usefulness to judicial decision-
making. Although this skepticism is not
universally held by judges, it threatens the
fundamental reform wrought by Daubert,
because it is shared by many and is
thought intellectually respectable by most.
In fact, however, this skepticism stems
from ignorance, a condition that can only
be remedied by a sustained effort to edu-
cate these decision-makers about the prac-
tices and culture of hypothesis testing in
science. Scientists should lead this effort.
Shortly after Daubert was decided, ex-
perts in many disciplines that assist the
law looked about and realized that they did
not have the research to support the opin-
ions they routinely offered in court. These
purveyors of science-lite cleverly sought
to avoid the new rule. They claimed that
the tough new gate-keeping standards did
not apply to them because the decision in-
volved only scientific evidence. These
experts, and the lawyers who employ
them, argued that when witnesses are
technical or specialized experts, Daubert
does not apply. All sorts of formerly proud
scientists joined the exodus from science,
including psychologists, engineers, and
medical doctors. To their everlasting
shame, many forensic scientists also dis-
claimed the science mantle. Obviously,
however, any failed science might make
such an argument. Allowing every failed
science to proclaim itself a specialty
would have gutted the new standard. In a
1999 decision, Kumho Tire Co. v.
Carmichael (3), the Supreme Court closed
this loophole. It held that all expert testi-
mony is subject to the requirement that it
be demonstrated to have a valid and reli-
able basis, whether it is accounting or
rocket science.
The challenge that arose after Kumho
Tire, and one foreseen in Justice Breyers
opinion for the Court, was the question, by
what criteria should the multitude of ex-
perts who appear daily in court be evaluat-
ed? This is no easy task, because this mul-
titude includes physicists, biologists, toxi-
cologists, epidemiologists, psychologists,
engineers, medical doctors, historians, ac-
countants, auto mechanics, and the list
goes on almost without end. This extraor-
dinarily broad array of expertise is simply
not susceptible to any one scheme of eval-
uation. In Daubert, the Court had suggest-
ed four criteria that might be used to as-
sess the reliability of scientific opinion: (i)
Is the opinion testable and has it been test-
ed? (ii) Is the error rate associated with the
technique or opinion acceptable? (iii) Has
the basis for the opinion survived peer re-
view and has it been published? And (iv)
Is it generally accepted among scientists in
the pertinent f ield? Justice Breyer in
Kumho Tire, however, did not attempt to
offer similar criteria by which to judge the
many different kinds of experts, and
many different kinds of expertise routine-
ly confronted by judges (4). This monu-
mental task, the Court said, would be left
to the discretion of lower court judges.
In his first opinion, Judge Pollak laid
out the Daubert framework and found that
fingerprinting had not been adequately
tested, had no known error rate, had not
produced research that was published in
peer-reviewed journals and was only gen-
erally accepted among a group of like-
thinking professionals who are all in the
fingerprint identification business. At the
same time, it should be noted, Judge Pol-
lak also held that there was no dispute that
each persons fingerprint was unique so
that identification could be readily accom-
plished when two full prints could be com-
pared. He took judicial notice of this fact.
Moreover, he held that the government ex-
pert would be permitted to introduce ex-
hibits illustrating the similarities between
the partial latent print found at the scene
and the defendants print. These two hold-
ings are not terribly controversial in them-
selves. The holding that ignited the
firestorm was Judge Pollaks order that the
expert would be foreclosed from offering
an opinion on the import of those similari-
ties or that they indicated a match.
In reversing the first decision, Judge
Pollak was obligated to explain how these
factors were now met or why they were no
longer relevant. Remarkably, he stated, I
concluded in the January 7 opinion that
Dauberts testing factor was not met, and I
have found no reason to depart from that
conclusion. (5). Yet, somehow, he now
found that the other three factors men-
tioned in Daubert, error rate, peer review
and publication, and general acceptance,
were satisf ied. How this was possible,
without testing, is a great mystery of the
decision. For him, this mystery was solved
by his observation that fingerprint identi-
fication is not, in my judgment, itself a
science. He likened forensic scientists to
accountants, vocational experts, accident
reconstruction experts, [and] appraisers of
land or of art. (5). Forensic science was a
specialty, not a science.
Judge Pollaks conclusion has been
echoed by a great number of federal
judges. Judge Crow similarly held that fin-
gerprinting, though as yet untested, is ad-
missible in court. He was persuaded by the
technologys success over the past 100
years, in which it has withstood the scruti- C
R
E
D
I
T
:
P
R
E
S
T
O
N

M
O
R
R
I
G
H
A
N
/
S
C
I
E
N
C
E
S C I E N C E S C O M P A S S P OL I CY F ORUM
P OL I CY F ORUM: S CI E NCE AND THE L AW
Is Science Different
for Lawyers?
David L. Faigman
D. L. Faigman is a professor of law at the Hastings Col-
lege of the Law, University of California, San Francisco,
CA 94102, USA. E-mail: faigmand@uchastings.edu
65
64
63
62
61
60
59
58
57
56
55
54
53
52
51
50
49
48
47
46
45
44
43
42
41
40
39
38
37
36
35
34
33
32
31
30
29
28
27
26
25
24
23
22
21
20
19
18
17
16
15
14
13
12
11
10
9
8
7
6
5
4
3
2
1
19 JULY 2002 VOL 297 SCIENCE www.sciencemag.org 340
ny and testing of the adversarial process.
(6). Scientists undoubtedly will find such
an assertion laughable. Judge Crow, how-
ever, answered their snickering as follows:
Those of a scientific bent certainly
can take issue with whether the judges and
lawyers have the education or training to
engage in scientif ic testing and with
whether the courtrooms provide the sterile,
laboratory-like and objective atmosphere
associated with and probably conducive to
scientific analysis. Even so, it seems an
unreasonable stretch simply to discard this
experiential testing as wholly unreliable
and to relegate the testifying opinion of all
these fingerprint examiners to ipse dixit.
Moreover, this court joins others who do
not read Daubert and Kumho as elevating
the scientific method to the touchstone by
which all [expert] evidence is to be
judged. (6).
In doubting the value of the scientific
method as the touchstone by which expert
evidence is to be evaluated, judges like
Pollak and Crow fail to say what should
replace it. Presumably, it is some combina-
tion of years of personal experience and
general acceptance among members of
some well-meaning guild. As a matter of
law, I believe this is an incorrect interpre-
tation of Daubert and Kumho Tire. More
troubling though, it reflects a basic misun-
derstanding of the subject of empirical ex-
pertise. Contrary to Judge Crows belief,
this overreliance on undifferentiated expe-
rience does indeed relegate the opinions of
testifying experts to ipse dixita Latin
phrase that roughly translates as, because
I said so.
Judge Crows statement is remarkable
for both its candor and its utter failure to
appreciate the culture attending scientific
testing of hypotheses. Science does not
exist categorically or in some concrete
encyclopedia of knowledge that passes
muster by, say, some committee of the Na-
tional Academies of Science. Science is a
process or method by which factual state-
ments or predictions about the world are
devised, tested, evaluated, revised, re-
placed, rejected, or accepted. There are as
many methods of testing as there are hy-
pothesesindeed, probably more. Courts
make a fundamental error when they try to
divide the world into science and specialty
categories. In truth, every expert who ap-
pears in court has specialized knowledge
of one sort or another. At best, it is special-
ized knowledge based upon good applied
science; at worst, it is specialized knowl-
edge based upon years of personal expe-
rience. The question is, for all specialized
knowledge proffered in court, how much
and what kind of testing should be neces-
sary before it is admitted? This is a policy
question that should depend on two fac-
tors. The first concerns the difficulties in-
herent in studying the phenomenon. The
second involves the legal stakes present in
cases in which the specialized knowledge
might be employed. In the specialty area
of fingerprinting, both factors indicate that
the courts are getting it wrong.
On the difficulties of studying the phe-
nomenon of fingerprint identification, con-
sider the following hypothesis: There is a
vanishingly small statistical likelihood that
some set of ridge characteristics (say 8, 10,
or 12) on a given fingerprint will be found
in a random sample of the population. This
hypothesis depends on the factually testable
question of what proportion of the popula-
tion (or relevant subpopulations) has partic-
ular ridge characteristics. This is a question
of base-rates. To be admissible, fingerprint
identification need not be powerful enough
to show identity, but the fact-finder should
be given some idea whether one person in
5, or 100, or 1000, could have left the par-
tial print. Of course, other hypotheses could
be imagined, including especially proficien-
cy rates among practitioners of the special-
ty. Proficiency testing would provide data
regarding the accuracy of forensic examin-
ers in applying the technology, with possi-
ble comparisons to laypeople or even com-
puters. But these hypotheses are hardly
daunting and, indeed, a modestly bright
graduate student could design research to
test many of them. Over time, however, as
the science of fingerprint identification pro-
gressed, more difficult and sophisticated
hypotheses might emerge, requiring greater
ability and resources to study. But the most
basic work has yet to be done. The other
forensic sciences, including bite-mark anal-
ysis, handwriting identification, firearms
analysis, and so on, are similarly amenable
to test. Unfortunately, like fingerprints,
most have not been seriously tested.
The second factor concerns what legal
risks are involved in the case, or cases,
presenting the expertise in question. Fin-
gerprint identification is offered by prose-
cutors in thousands of trials each year in
which defendants are in jeopardy of their
liberty and sometimes their lives. In addi-
tion, as a practical matter, prosecutors
have the institutional wherewithal, espe-
cially through the Justice Department, to
invest in the research. It is true that testing
fingerprinting, not to mention the surfeit
of other forensic sciences, is an expensive
proposition. Cost should affect courts ex-
pectations regarding what research is done
and when it gets done. But the government
has so far not claimed that the costs would
be prohibitive in this areaa claim likely
to receive a skeptical response in any
event. Indeed, failure to put the testing
burden on the government creates perverse
incentives. If courts admit untested specu-
lation, what incentive does the Justice De-
partment have to do the research? The
greater the costs in liberty, lives, and prop-
erty, the greater should be the expectation
that good-quality work be done.
In the context of fingerprinting, the
amenability of the subject to test and the
gravity of the legal stakes involved make
it an easy case. Admittedly, other kinds of
specialized knowledge will present harder
cases. Particularly difficult for lawyers
and judges is the first factor, whether an
empirical subject is amenable to test. Sci-
entists can be enormously helpful in ad-
dressing this question in concrete cases
ranging from alcohol and drug testing to
polygraphs. Organizations such as the
National Academies of Science and the
AAAS have already entered partnerships
with legal institutions, such as the Federal
Judicial Center and the National Institute
of Justice (7). These and other science or-
ganizations should be encouraged to do
more. Most judges and lawyers have little
creativity when it comes to conceptualiz-
ing how certain empirical statements
might be examined. Topics such as pre-
dictions of violence, the battered woman
syndrome, post-traumatic stress disorder,
fire and arson investigation, birth defects,
and repressed memories, all challenge the
legal imagination. If lawyers and judges
are going to improve their comprehension
of these and other subjects, bridges must
be built between the legal and scientific
communities.
Daubert initiated a scientific revolution
in the law. Although it has taken more than
200 years, the law is ever so slowly com-
ing to embrace the scientific culture of
empirical testing. Yet some courts remain
in a prescientific age. When Galileo an-
nounced that he saw moons around Jupiter
through his telescope, the Pope declared
that he was mistaken, for the Bible did not
allow it. But the moons are there. Similar-
ly, courts can decree that fingerprinting is
reliable, but this does not make it true. On-
ly testing will tell us whether it is so.
References and Notes
1. United States v. Plaza, 179 F. Supp. 2d 492 (E.D. Pa. 7
January 2002).
2. United States v. Plaza, 188 F. Supp. 2d 549 (E.D. Pa.
13 March 2002) (Plaza II).
3. Kumho Tire Co. v. Carmichael, 526 U.S. 137 (1999).
4. Kumho Tire, 526 U.S., p. 150.
5. Plaza II, 188 F. Supp. 2d, p. 564.
6. United States v. Cline, 188 F. Supp. 2d 1287 (D. Kan.
21 February 2002); p. 1294.
7. The interested reader will find more information on ex-
isting partnerships between scientific and legal groups at
the National Institute of Justice, www.ojp.usdoj.gov/nij;
the Federal Judicial Center, www.fjc.gov; and the Ameri-
can Association for the Advancement of Science,
www.aaas.org/spp/case/case.htm.
S C I E NC E S COMPA S S
www. sci am. com SCI ENT I FI C AMERI C AN 85
on the air since 2000, now demand unreasonable levels of physical
evidence in trials. Whether the CSI effect truly exists as a quanti-
able inuence on courtroom behavior is still a subject of debate. Of
no debate, though, is the effect that the CSI programs have had on
the activities of police, who now collect more pieces of physical
evidence than ever before; in academia, where some forensics pro-
grams are growing exponentially; and in overburdened working
laboratories, which are a far cry from the glitzy, blue-lit analysis
palaces of TV.
The Effect in the Courtroom
i n one of this seasons episodes of CSI, the plot included
a television crew recording the activities of the ctional crime scene
investigators. Lead researcher Gil Grissom rebuffs the TV crews
attempts, saying, Theres too many forensics shows on TV. Nu-
merous attorneys and judges who believe that jurors are aficted
with the CSI effect would agree. But to what extent do CSI and its
relatives inuence the expectations that jurors bring to trials?
The press started to pay attention to the issue in 2003, collect-
ing anecdotes from attorneys and judges about what appeared to
be a change in the behavior of jurors. In 2005 Oregon district at-
torney Josh Marquis, vice president of the National District At-
torneys Association, told CBS News, Jurors now expect us to
have a DNA test for just about every case. They expect us to have
the most advanced technology possible, and they expect it to look
like it does on television. Indeed, jurors in a Los Angeles murder
case complained that a bloody coat had not been tested for DNA,
even though such tests were unnecessary: the defendant had al-
ready admitted to having been at the crime scene. The judge not-
ed that TV had taught jurors about DNA tests but not about when
they should be used. In a study in Delaware of how juries deal
with evidence, one juror tangling with a complex DNA case com-
plained that these kinds of problems did not happen on CSI.
Attorneys blamed the CSI effect when a Baltimore jury acquit-
ted a man of murdertestimony from two eyewitnesses was
trumped by a lack of physical evidence. Ive seen a big change in
jurors and what they expect over the last ve years, defense at-
orensic science
has been the backbone of mystery stories from
Edgar Allan Poes Dupin adventures to Sir Ar-
thur Conan Doyles Sherlock Holmes tales to
Jack Klugmans Quincy television series to to-
days wildly successful forensics shows. Holm-
ess methods presaged many actual techniques
for linking physical evidence to the perpetrator
of a crime, such as blood testing. Forensic science
was codied as a profession in the early 1900s
and exploded into the public consciousness in
the 1990s with the advent of DNA analysis.
Forensics has never been more popular or
popularized: eight crime dramas, including CSI:
Crime Scene Investigation and its sibling pro-
grams, made it into the top 20 shows last Octo-
ber. On one Thursday that month, 27 percent of
all American televisions that were turned on
were tuned to CSI. On cable, CourtTVs Foren-
sic Files, a documentary-style series featuring
real crimes and real scientists, airs four days a
week. Such programs give the impression that
forensic laboratories are fully staffed with high-
ly trained personnel, stocked with a full comple-
ment of state-of-the-art instrumentation and
rolling in the resources to close every case in a
timely fashion.
The gap between public perception and real-
ity, however, is vast. And the popularity of these
shows has led to complaints of a CSI effect: at
least some lawyers and judges have the impres-
sion that jurors schooled on CSI, which has been
Attorneys, investigators and educators
have felt the impact of televisions
popular forensics programs
By Max M. Houck
B
R
A
D

S
W
O
N
E
T
Z

COPYRIGHT 2006 SCIENTIFIC AMERICAN, INC.
86 SCI ENT I FI C AMERI C AN JULY 2006
torney Joseph Levin of Atlantic City,
N.J., told a local newspaper. Jurors can
ask questions of the judge while in de-
liberations, and theyre asking about
what they see as missing evidence. They
want to know where the ngerprints are
or the DNA. If its not there, they want
to know why. In the California murder
trial of actor Robert Blake, prosecutors
tried to persuade the jury by establish-
ing Blakes motive and opportunity, and
they presented witnesses who testied
that Blake asked them to kill his wife.
But no gunshot residue or blood spatter
evidence was presented, and Blake was
acquitted. A juror was quoted as saying
that if the prosecutor had all that in-
formation, that would have meant
[Blake] was guilty. The defeat was the
prosecutors rst in 50 murder cases.
Before CSI became popular, attor-
neys mostly worried about whether a
jury was going to understand the com-
plexity of DNA evidence. Now, though,
many spend time clarifying the differ-
ence between television and realityit is
common for lawyers to ask prospective
jurors about their exposure to forensics-
themed TV programs. And some pros-
ecutors are attempting to preempt any
potential fallout from the CSI effect. In
trials in Arizona, Illinois and California,
they have put so-called negative evidence
witnesses on the stand to alert jurors to
the fact that real-life detectives often fail
to nd physical evidence, such as DNA
or ngerprints, at crime scenes.
Several legal experts have argued,
however, that the CSI effect may be il-
lusory. The newspaper that quoted At-
lantic City lawyer Levin also noted that
Superior Court Judge Albert Garofolo
said, My initial reaction might have
been Yes, there is a CSI effect. But I
think this may be more of a suspicion
than anything else. Theres a feeling this
could be real, but in truth I cant recall
a situation where Ive heard a jury say
they were expecting more.
In 2005 in the Wall Street Journal,
Simon Cole of the department of crimi-
nology, law and society at the Univer-
sity of California, Irvine, and his stu-
dent Rachel Dioso wrote: That televi-
sion might have an effect on courtrooms
is not implausible. . . . But to argue that
C.S.I. and similar shows are actually
raising the number of acquittals is a
staggering claim, and the remarkable
thing is that, speaking forensically, there
is not a shred of evidence to back it up.
There is a robust eld of research on jury
decision-making but no study nding
any C.S.I. effect. There is only anecdot-
al evidence.
What appears to be the rst study of
the CSI effect was published in February
by Kimberlianne Podlas, an attorney
and assistant professor of media law and
ethics at the University of North Caro-
lina at Greensboro. Podlas concluded
that the chances of, and reasoning for,
acquittals were the same for frequent
CSI viewers as for prospective jurors
who did not watch the showshe saw
no CSI effect. Several participants, how-
ever, said that a lack of forensic testing
was an issue, despite the fact that physi-
cal evidence would not have resolved the
hypothetical charges. Studies of real ju-
ries have been advocated, and at least
ve graduate students (three in the U.S.
and two in England) are preparing the-
ses examining the effect.
What Is Real?
whether or not forensics shows
are measurably inuencing the demands
and decisions of juries, television is un-
Q Prosecutors, judges and police ofcers have noted what they believe to be a
so-called CSI effect whereby the popular television forensics programs have
led jurors to have unreasonable expectations for the quality and quantity of
physical evidence.
Q Any CSI effect in courtrooms is still unproved. But the television programs
have led to an increase in the collection of physical evidence, contributing to
issues of storage and personnel shortages.
Q The television shows have also undoubtedly led to an explosion of interest in
forensics evidence on college campuses, where enrollment in forensics
science studies has greatly increased since the CSI series went on the air.
Overview/Science vs. Fiction
PAUCI TY OF PHYSICAL EVIDENCE led to
acquittal of actor Robert Blake (shown kissing
his attorney after the verdict) in the murder of
his wife, Bonny Lee Bakley, in 2001, despite
Blakes having motive and opportunity. His
attorney holds Blakes ankle monitor aloft. In a
subsequent civil case, Blake was found liable
for the wrongful death.
CSI effect: not guilty by reason of TV?
N
I
C
K

U
T

A
P

P
h
o
t
o

questionably giving the public a distort-
ed view of how forensic science is car-
ried out and what it can and cannot do.
The actors playing forensic personnel
portrayed on television, for instance,
are an amalgam of police ofcer/detec-
tive/forensic scientistthis job descrip-
tion does not exist in the real world.
Law enforcement, investigations and fo-
rensic science are each sufciently com-
plex that they demand their own educa-
tion, training and methods. And spe-
cialization within forensic laboratories
has been the norm since the late 1980s.
Every forensic scientist needs to know
the capabilities of the other subdisci-
plines, but no scientist is an expert in
every area of crime scene investigation.
In addition, laboratories frequently
do not perform all types of analyses,
whether because of cost, insufcient re-
sources or rare demand. And television
shows incorrectly portray forensic scien-
tists as having ample time for every case;
several TV detectives, technicians and
scientists often devote their full atten-
tion to one investigation. In reality, indi-
vidual scientists will have many cases
assigned to them. Most forensics labs
nd backlogs to be a major problem, and
dealing with them often accounts for
most requests for bigger budgets.
Fictional forensics programs also di-
verge from the real world in their por-
trayal of scientic techniques: University
of Maryland forensic scientist Thomas
Mauriello estimates that about 40 per-
cent of the forensic science shown on CSI
does not exist. Carol Henderson, direc-
tor of the National Clearinghouse for
Science, Technology and the Law at Stet-
son University College of Law, told a
publication of that institution that jurors
are sometimes disappointed if some of
the new technologies that they think ex-
ist are not used. Similarly, working in-
vestigators cannot be quite as precise as
their counterparts on the screen. A TV
character may analyze an unknown
sample on an instrument with ashing
screens and blinking lights and get the
result Maybelline lipstick, Color 42,
Batch A-439. The same character may
then interrogate a witness and declare,
We know the victim was with you be-
cause we identied her lipstick on your
collar. In real life, answers are seldom
that denite, and the forensic investiga-
tor probably would not confront a sus-
pect directly. This mismatch between
ction and reality can have bizarre con-
sequences: A Knoxville, Tenn., police of-
cer reported, I had a victim of a car
robbery, and he saw a red ber in the back
of his car. He said he wanted me to run
tests to nd out what it was from, what
retail store that object was purchased at,
and what credit card was used.
Groaning under the Load
despi te not havi ng all the tools
of televisions CSI teams, forensic scien-
tists do have advanced technologies that
are getting more sophisticated all the
time. Initial DNA-testing methods in
the late 1980s required samples the size
of a quarter; current methods analyze
nanograms. The news routinely reports
the solution of a cold case, a suspect ex-
cluded or a wrongful conviction over-
turned through advanced forensic tech-
nology. Databases of DNA, ngerprints
and rearms ammunition have become
important resources that can link of-
fenders to multiple crimes.
Nevertheless, far from being freed to
work telegenic miracles, many labs are
struggling under the increasing demands
they face. As police investigators gain ap-
preciation for the advantages of science
and also feel pressure to collect increas-
ing amounts of evidence, they are sub-
mitting more material from more cases
for forensic analysis. Police detectives
who at one time might have gathered ve
pieces of evidence from a crime scene say
they are collecting 50 to 400 today. In
1989 Virginia labs processed only a few
dozen cases. The number of cases being
submitted this year has ballooned into
the thousands. Of course, not every item
at a crime scene can or should be collect-
ed for testing. The remote chance of an
item being signicant has to be weighed
against the burden of backlogged cases.
But social, professional and political pres-
sures based on unrealistic expectations
engendered by television mean that if an
ofcer brings in a bag lled with ciga-
rette butts, fast-food wrappers and other
trash, chances are good that most of the
items will be scheduled for analysis.
And all that work will have to be
done, in many cases, by already over-
loaded staffs. For example, the state of
Massachusetts has 6.3 million people
outside of Boston and eight DNA ana-
lysts for that region. (Boston has three
analysts of its own.) New York City has
eight million people and 80 DNA ana-
lysts. But Massachusetts and New York
City have similar rates of violent crime
(469.4 versus 483.3 per 100,000), which
is the kind of crime most likely to in-
volve DNA evidence. Massachusetts,
STORING AND TRACKING MILLIONS of items of
evidence pose signicant challenges to law-
enforcement agencies and forensic laboratories.
Who will analyze all the evidence?
C
O
U
R
T
E
S
Y

O
F

W
A
S
H
I
N
G
T
O
N

S
T
A
T
E

P
O
L
I
C
E

L
A
B
O
R
A
T
O
R
Y

88 SCI ENT I FI C AMERI C AN JULY 2006
like many other states, thus appears to
be woefully understaffed. Thankfully,
the state has recognized this imbalance
and has authorized the hiring of more
forensic DNA analysts.
A consequence of the new trends,
then, is exacerbation of the already dis-
turbing backlog problem. A study re-
cently published by the Department of
Justices Bureau of Justice Statistics
found that at the end of 2002 (the latest
available data), more than half a million
cases were backlogged in forensic labs,
despite the fact that tests were being pro-
cessed at or above 90 percent of the ex-
pected completion rate. To achieve a 30-
day turnaround time for the requests of
that year, the study estimated a need for
another 1,900 full-time employees. An-
other Justice Department study showed
that the 50 largest forensic laboratories
received more than 1.2 million requests
for services in 2002: the backlog of cases
for these facilities had doubled in the
course of one year. And these increases
have happened even though crime rates
have fallen since 1994.
Another side effect of the increased
gathering of physical evidence is the need
to store it for various lengths of time, de-
pending on local, state or federal laws.
Challenges for storing evidence include
having the computers, software and per-
sonnel to track the evidence; having the
equipment to safely stow biological evi-
dence, such as DNA; and having ade-
quate warehouse space for physical evi-
dence. In many jurisdictions, evidence
held past a certain length of time may be
destroyed or returned. Storage can be a
critical issue in old or cold casesthe In-
nocence Project at the Benjamin N. Car-
dozo Law School in New York City has
found that the evidence no longer exists
in 75 percent of its investigations into po-
tentially wrongful convictions.
Just keeping track of the evidence
that does exist can be problematic: a
2003 study by the American Society of
Crime Laboratory Directors indicated
that more than a quarter of American
forensic laboratories did not have the
computers they needed to track evidence.
Mark Dale, director of the Northeast
Regional Forensic Institute at the Uni-
versity at Albany and former director of
the New York Police Department Labo-
ratory, estimates that more than 10,000
additional forensic scientists will be
needed over the next decade to address
these various issues. In addition, appro-
priate modernization of facilities will
cost $1.3 billion, and new instruments
will require an investment of greater
than $285 million.
The Effect on Campus
on t he posi t i ve si de , through
CSI and its siblings, the public has de-
veloped a fascination with and respect
for science as an exciting and important
MAX M. HOUCK is director of West Virginia Universitys Forensic Science Initiative, a
program that develops research and professional training for forensic scientists. A trace
evidence expert and forensic anthropologist, he was assigned to the Trace Evidence Unit
at the FBI Laboratory from 1992 to 2001. He received his undergraduate degree in an-
thropology and his masters in forensic anthropology, both from Michigan State Univer-
sity. Houck is chair of the Forensic Science Educational Program Accreditation Commis-
sion and serves on the editorial boards of the Journal of Forensic Sciences and the Jour-
nal of Forensic Identification. He is a fellow of the American Academy of Forensic
Sciences and an associate member of the American Society of Crime Laboratory Direc-
tors and the International Association for Identication.
T
H
E

A
U
T
H
O
R
G
I
O
R
G
I
O

B
E
N
V
E
N
U
T
I

E
P
A
/
C
o
r
b
i
s

(
c
r
i
m
e

s
c
e
n
e
)
;

J
E
F
F

S
I
N
E
R

C
o
r
b
i
s
/
S
y
g
m
a

(
b
a
l
l
i
s
t
i
c
s
)
;

S
I
M
O
N

K
W
O
N
G

R
e
u
t
e
r
s
/
C
o
r
b
i
s

(
b
a
l
l
i
s
t
i
c
s

i
n
s
e
t
)
;

C
B
S
/
E
V
E
R
E
T
T

C
O
L
L
E
C
T
I
O
N

(
M
a
r
g

H
e
l
g
e
n
b
e
r
g
e
r
)

Sus
int
Ballistics
Evidence collection
Fictional TV investigators
CSI MULTI TASKER Catherine Willows combines
roles of real-life investigators.
profession unseen since the Apollo space
program. Enrollment in forensic science
educational programs across the U.S. is
exploding. For example, the forensic
program at Honolulus Chaminade Uni-
versity went from 15 students to 100 in
four years. At my institution, West Vir-
ginia University, the forensic and inves-
tigative sciences program has grown
from four graduates in 2000 to current-
ly being the third largest major on cam-
pus, with more than 500 students in the
program.
The growth of existing programs
and the advent of new ones have been
such that the National Institute of Jus-
tice, in collaboration with West Virgin-
ia University, produced a special report,
Education and Training in Forensic Sci-
ence: A Guide for Forensic Science Lab-
oratories, Educational Institutions and
Students. The report formed the basis
for an accreditation commission under
the American Academy of Forensic Sci-
ences. As of this past January, 11 pro-
grams had received provisional, condi-
tional or full accreditation.
CSIs popularity may have also af-
fected the demographics of forensic sci-
ence. In the 1990s women and minori-
ties were underrepresented as leads in
television series with a scientic theme;
the current slate of CSI dramas, howev-
er, has generally improved this represen-
tation. Women are now in the majority
in forensic science educational programs
in the U.S. and in much of the profession.
Two thirds of forensic science laboratory
management personnel are currently
male, a gure sure to decrease as the
newer women workers advance.
The best result of public interest in
forensics, though, would be increased
investment in forensics research. In the
past, most research was conducted in
police laboratories working on specic,
case-related questions. But for technolo-
gies to advance markedly, testing is
needed in the controlled environment of
the academic laboratory. Such labs could
investigate questions that clearly require
more research. For example, recent legal
challenges have called into question the
long-held assumption of the absolute
uniqueness of ngerprints, tool marks,
bite marks, bullet striations and hand-
writing matches.
As forensic science is increasingly re-
lied on, it must become more reliable: a
recent National Institute of Justice re-
port to Congress stated that basic re-
search is needed into the scientic un-
derpinning of impression evidence, such
as tire marks or footprints; standards
for document authentication; and re-
arms and tool-mark examination. The
report also recommended that the fed-
eral government sponsor research to
validate forensic science disciplines, ad-
dressing basic principles, error rates and
standards of procedure. Clearly, more
funding for such research would be ben-
ecial: one must wonder why the U.S.
spent a mere $7 million this scal year
for basic forensic science research
through the National Institute of Justice
when $123 million was spent on alter-
native medicine through the National
Institutes of Health.
One of the most fundamental obli-
gations of any democratic government
to its citizens is to ensure public safety
in a just manner. Forensic science is an
integral and critical part of the criminal
justice process. In the 21st century prop-
erly educated, well-equipped, fully
staffed forensic science laboratories are
essential to the fulllment of that obli-
gation. The popular interest in forensic
science is at an all-time high, as are the
challenges to the veracity of forensic sci-
ence methods and capacities. Even if no
so-called CSI effect exists in the court-
room, the real effect is the realization of
the need for the advancement of forensic
science laboratories and research.
MORE T O E X PLORE
The CSI Effect: Fake TV and Its Impact on Jurors in Criminal Cases. Karin H. Cather in
The Prosecutor, Vol. 38, No. 2; March/April 2004.
Public Forensic Laboratory Budget Issues. Perry M. Koussiafes in Forensic Science
Communications, Vol. 6, No. 3; July 2004. Available at www.fbi.gov
Trace Evidence Analysis: More Cases in Forensic Microscopy and Mute Witnesses.
Max M. Houck. Elsevier/Academic Press, 2004.
Fundamentals of Forensic Science. Max M. Houck and Jay A. Siegel. Elsevier/Academic Press, 2006.
For updates on forensic science legislation, visit: www.crimelabproject.com/
DNA analysis
Chemistry
M
A
T
T

Y
O
R
K

A
P

P
h
o
t
o

(
i
n
t
e
r
r
o
g
a
t
o
r
)
;

S
T
E
V
E

L
I
S
S

C
o
r
b
i
s
/
S
y
g
m
a

(
c
h
e
m
i
s
t
r
y
)
;

A
N
N
A

C
L
O
P
E
T

C
o
r
b
i
s

(
D
N
A

a
n
a
l
y
s
i
s
)
spect
terrogation
often have exper tise in multiple areas of specialization.
S
P
R
I
N
G
2
0
0
7
3
5
7
T
H
E
C
S
I
E
F
F
E
C
T
:
P
O
P
U
L
A
R
F
I
C
T
I
O
N
A
B
O
U
T
F
O
R
E
N
S
I
C
S
C
I
E
N
C
E
A
F
F
E
C
T
S
T
H
E
P
U
B
L
I
C
S
E
X
P
E
C
T
A
T
I
O
N
S
A
B
O
U
T
R
E
A
L
F
O
R
E
N
S
I
C
S
C
I
E
N
C
E
N
.
J
.
S
c
h
w
e
i
t
z
e
r
M
i
c
h
a
e
l
J
.
S
a
k
s
*
A
B
S
T
R
A
C
T
:
T
w
o
o
f
a
n
u
m
b
e
r
o
f
h
y
p
o
t
h
e
s
e
s
l
o
o
s
e
l
y
r
e
f
e
r
r
e
d
t
o
a
s
t
h
e
C
S
I
E
f
f
e
c
t
s
u
g
g
e
s
t
t
h
a
t
t
h
e
t
e
l
e
v
i
s
i
o
n
p
r
o
g
r
a
m
a
n
d
i
t
s
s
p
i
n
-
o
f
f
s
,
w
h
i
c
h
w
i
l
d
l
y
e
x
a
g
g
e
r
a
t
e
a
n
d
g
l
o
r
i
f
y
f
o
r
e
n
s
i
c
s
c
i
e
n
c
e
,
a
f
f
e
c
t
t
h
e
p
u
b
l
i
c
,
a
n
d
i
n
t
u
r
n
a
f
f
e
c
t
t
r
i
a
l
s
e
i
t
h
e
r
b
y
(
a
)
b
u
r
d
e
n
i
n
g
t
h
e
p
r
o
s
e
c
u
t
i
o
n
b
y
c
r
e
a
t
i
n
g
g
r
e
a
t
e
r
e
x
p
e
c
t
a
t
i
o
n
s
a
b
o
u
t
f
o
r
e
n
s
i
c
s
c
i
e
n
c
e
t
h
a
n
c
a
n
b
e
d
e
l
i
v
e
r
e
d
o
r
(
b
)
b
u
r
d
e
n
i
n
g
t
h
e
d
e
f
e
n
s
e
b
y
c
r
e
a
t
i
n
g
e
x
a
g
g
e
r
a
t
e
d
f
a
i
t
h
i
n
t
h
e
c
a
p
a
b
i
l
i
t
i
e
s
a
n
d
r
e
l
i
a
b
i
l
i
t
y
o
f
t
h
e
f
o
r
e
n
s
i
c
s
c
i
e
n
c
e
s
.
T
h
e
p
r
e
s
e
n
t
s
t
u
d
y
t
e
s
t
e
d
t
h
e
s
e
h
y
p
o
t
h
e
s
e
s
b
y
p
r
e
s
e
n
t
i
n
g
t
o
m
o
c
k
j
u
r
o
r
s
a
s
i
m
u
l
a
t
e
d
t
r
i
a
l
t
r
a
n
s
c
r
i
p
t
t
h
a
t
i
n
c
l
u
d
e
d
t
h
e
t
e
s
t
i
m
o
n
y
o
f
a
f
o
r
e
n
s
i
c
s
c
i
e
n
t
i
s
t
.
T
h
e
c
a
s
e
f
o
r
c
o
n
v
i
c
t
i
o
n
w
a
s
r
e
l
a
t
i
v
e
l
y
w
e
a
k
,
u
n
l
e
s
s
t
h
e
e
x
p
e
r
t
t
e
s
t
i
m
o
n
y
c
o
u
l
d
c
a
r
r
y
t
h
e
c
a
s
e
a
c
r
o
s
s
t
h
e
t
h
r
e
s
h
o
l
d
o
f
r
e
a
s
o
n
a
b
l
e
d
o
u
b
t
.
I
n
a
d
d
i
t
i
o
n
t
o
r
e
a
c
t
i
n
g
t
o
t
h
e
t
r
i
a
l
e
v
i
d
e
n
c
e
,
r
e
s
p
o
n
d
e
n
t
s
w
e
r
e
a
s
k
e
d
a
b
o
u
t
t
h
e
i
r
t
e
l
e
v
i
s
i
o
n
v
i
e
w
i
n
g
h
a
b
i
t
s
.
C
o
m
p
a
r
e
d
t
o
n
o
n
-
C
S
I
v
i
e
w
e
r
s
,
C
S
I
v
i
e
w
e
r
s
w
e
r
e
m
o
r
e
c
r
i
t
i
c
a
l
o
f
t
h
e
f
o
r
e
n
s
i
c
e
v
i
d
e
n
c
e
p
r
e
s
e
n
t
e
d
a
t
t
h
e
t
r
i
a
l
,
f
i
n
d
i
n
g
i
t
l
e
s
s
b
e
l
i
e
v
a
b
l
e
.
R
e
g
a
r
d
i
n
g
t
h
e
i
r
v
e
r
d
i
c
t
s
,
2
9
%
o
f
n
o
n
-
C
S
I
v
i
e
w
e
r
s
s
a
i
d
t
h
e
y
w
o
u
l
d
c
o
n
v
i
c
t
,
c
o
m
p
a
r
e
d
t
o
1
8
%
o
f
C
S
I
v
i
e
w
e
r
s
(
n
o
t
a
s
t
a
t
i
s
t
i
c
a
l
l
y
s
i
g
n
i
f
i
c
a
n
t
d
i
f
f
e
r
e
n
c
e
)
.
F
o
r
e
n
s
i
c
s
c
i
e
n
c
e
v
i
e
w
e
r
s
e
x
p
r
e
s
s
e
d
m
o
r
e
c
o
n
f
i
d
e
n
c
e
i
n
t
h
e
i
r
v
e
r
d
i
c
t
s
t
h
a
n
d
i
d
n
o
n
-
v
i
e
w
e
r
s
.
V
i
e
w
e
r
s
o
f
g
e
n
e
r
a
l
c
r
i
m
e
p
r
o
g
r
a
m
s
,
h
o
w
e
v
e
r
,
d
i
d
n
o
t
d
i
f
f
e
r
s
i
g
n
i
f
i
c
a
n
t
l
y
f
r
o
m
t
h
e
i
r
n
o
n
-
v
i
e
w
i
n
g
c
o
u
n
t
e
r
p
a
r
t
s
o
n
a
n
y
o
f
t
h
e
o
t
h
e
r
d
e
p
e
n
d
e
n
t
m
e
a
s
u
r
e
s
,
s
u
g
g
e
s
t
i
n
g
t
h
a
t
s
k
e
p
t
i
c
i
s
m
t
o
w
a
r
d
t
h
e
f
o
r
e
n
s
i
c
s
c
i
e
n
c
e
t
e
s
t
i
m
o
n
y
w
a
s
s
p
e
c
i
f
i
c
t
o
t
h
o
s
e
w
h
o
s
e
d
i
e
t
c
o
n
s
i
s
t
e
d
o
f
h
e
a
v
y
d
o
s
e
s
o
f
f
o
r
e
n
s
i
c
s
c
i
e
n
c
e
t
e
l
e
v
i
s
i
o
n
p
r
o
g
r
a
m
s
.
*
N
.
J
.
S
c
h
w
e
i
t
z
e
r
i
s
a
P
h
.
D
.
c
a
n
d
i
d
a
t
e
,
D
e
p
a
r
t
m
e
n
t
o
f
P
s
y
c
h
o
l
o
g
y
,
A
r
i
z
o
n
a
S
t
a
t
e
U
n
i
v
e
r
s
i
t
y
.
M
i
c
h
a
e
l
J
.
S
a
k
s
i
s
P
r
o
f
e
s
s
o
r
o
f
L
a
w
a
n
d
P
s
y
c
h
o
l
o
g
y
a
n
d
F
a
c
u
l
t
y
F
e
l
l
o
w
,
C
e
n
t
e
r
f
o
r
t
h
e
S
t
u
d
y
o
f
L
a
w
,
S
c
i
e
n
c
e
,
a
n
d
T
e
c
h
n
o
l
o
g
y
,
S
a
n
d
r
a
D
a
y
O
C
o
n
n
o
r
C
o
l
l
e
g
e
o
f
L
a
w
,
A
r
i
z
o
n
a
S
t
a
t
e
U
n
i
v
e
r
s
i
t
y
.
T
h
e
a
u
t
h
o
r
s
w
i
s
h
t
o
t
h
a
n
k
D
a
w
n
M
c
Q
u
i
s
t
o
n
-
S
u
r
r
e
t
t
f
o
r
h
e
r
a
s
s
i
s
t
a
n
c
e
w
i
t
h
t
h
i
s
p
r
o
j
e
c
t
a
n
d
f
o
r
t
h
e
d
e
v
e
l
o
p
m
e
n
t
o
f
t
h
e
e
x
p
e
r
i
m
e
n
t
a
l
m
a
t
e
r
i
a
l
s
.
C
o
r
r
e
s
p
o
n
d
e
n
c
e
c
a
n
b
e
s
e
n
t
t
o
N
.
J
.
S
c
h
w
e
i
t
z
e
r
,
D
e
p
a
r
t
m
e
n
t
o
f
P
s
y
c
h
o
l
o
g
y
,
A
r
i
z
o
n
a
S
t
a
t
e
U
n
i
v
e
r
s
i
t
y
,
T
e
m
p
e
,
A
Z
8
5
2
8
7
-
1
1
0
4
o
r
n
j
s
@
a
s
u
.
e
d
u
.
A
R
T
I
C
L
E
S
c
h
w
e
i
t
z
e
r
a
n
d
S
a
k
s
3
5
8
4
7
J
U
R
I
M
E
T
R
I
C
S
C
I
T
A
T
I
O
N
:
N
.
J
.
S
c
h
w
e
i
t
z
e
r
a
n
d
M
i
c
h
a
e
l
J
.
S
a
k
s
,
T
h
e
C
S
I
E
f
f
e
c
t
:
P
o
p
u
l
a
r
F
i
c
t
i
o
n
A
b
o
u
t
F
o
r
e
n
s
i
c
S
c
i
e
n
c
e
A
f
f
e
c
t
s
t
h
e
P
u
b
l
i
c
s
E
x
p
e
c
t
a
t
i
o
n
s
A
b
o
u
t
R
e
a
l
F
o
r
e
n
s
i
c
S
c
i
e
n
c
e
,
4
7
J
u
r
i
m
e
t
r
i
c
s
J
.
3
5
7
3
6
4
(
2
0
0
7
)
.
I
n
r
e
c
e
n
t
y
e
a
r
s
,
t
h
e
t
e
l
e
v
i
s
i
o
n
p
r
o
g
r
a
m
C
S
I
a
n
d
i
t
s
s
p
i
n
-
o
f
f
s
h
a
v
e
p
o
r
-
t
r
a
y
e
d
f
o
r
e
n
s
i
c
s
c
i
e
n
c
e
a
s
h
i
g
h
-
t
e
c
h
m
a
g
i
c
,
s
o
l
v
i
n
g
c
r
i
m
e
s
q
u
i
c
k
l
y
a
n
d
u
n
e
r
r
i
n
g
l
y
.
O
f
c
o
u
r
s
e
,
C
S
I
i
s
o
n
l
y
f
i
c
t
i
o
n
.
O
n
e
f
o
r
e
n
s
i
c
s
c
i
e
n
t
i
s
t
e
s
t
i
m
a
t
e
s
t
h
a
t
4
0
%
o
f
t
h
e
s
c
i
e
n
c
e
o
n
C
S
I
d
o
e
s
n
o
t
e
x
i
s
t
,
a
n
d
m
o
s
t
o
f
t
h
e
r
e
s
t
i
s
p
e
r
f
o
r
m
e
d
i
n
w
a
y
s
t
h
a
t
c
r
i
m
e
l
a
b
p
e
r
s
o
n
n
e
l
c
a
n
o
n
l
y
d
r
e
a
m
a
b
o
u
t
.
1
T
h
e
g
a
p
b
e
t
w
e
e
n
f
a
c
t
a
n
d
f
i
c
t
i
o
n
h
a
s
l
e
d
C
S
I
t
o
b
e
a
c
c
u
s
e
d
o
f
(
o
r
c
r
e
d
i
t
e
d
w
i
t
h
)
h
a
v
i
n
g
a
v
a
r
i
e
t
y
o
f
e
f
f
e
c
t
s
o
n
A
m
e
r
i
c
a
n
s
o
c
i
e
t
y
.
2
L
e
g
a
l
s
c
h
o
l
a
r
s
,
h
o
w
e
v
e
r
,
w
i
l
l
b
e
m
o
s
t
i
n
t
e
r
e
s
t
e
d
i
n
h
y
p
o
t
h
e
s
i
z
e
d
e
f
f
e
c
t
s
r
e
l
a
t
e
d
t
o
t
r
i
a
l
s
,
w
h
i
c
h
c
o
m
e
i
n
t
w
o
b
a
s
i
c
f
l
a
v
o
r
s
:
o
n
e
p
o
s
i
t
s
t
h
a
t
C
S
I
a
d
d
s
t
o
t
h
e
b
u
r
d
e
n
s
o
f
t
h
e
p
r
o
s
e
c
u
t
i
o
n
,
a
n
d
t
h
e
o
t
h
e
r
a
s
s
e
r
t
s
t
h
a
t
i
t
a
d
d
s
t
o
t
h
e
b
u
r
d
e
n
s
o
f
t
h
e
d
e
f
e
n
s
e
.
3
T
h
e
h
y
p
o
t
h
e
s
i
s
h
e
a
r
d
m
o
s
t
o
f
t
e
n
i
s
t
h
a
t
C
S
I
h
a
s
r
a
i
s
e
d
t
h
e
p
u
b
l
i
c
s
e
x
p
e
c
t
a
t
i
o
n
s
f
o
r
t
h
e
k
i
n
d
o
f
f
o
r
e
n
s
i
c
-
s
c
i
e
n
c
e
e
v
i
d
e
n
c
e
t
h
a
t
c
o
u
l
d
a
n
d
s
h
o
u
l
d
b
e
o
f
f
e
r
e
d
a
t
t
r
i
a
l
s
t
o
s
u
c
h
h
e
i
g
h
t
s
t
h
a
t
j
u
r
o
r
s
a
r
e
d
i
s
a
p
p
o
i
n
t
e
d
b
y
t
h
e
r
e
a
l
e
v
i
d
e
n
c
e
w
i
t
h
w
h
i
c
h
t
h
e
y
a
r
e
p
r
e
s
e
n
t
e
d
.
J
u
r
o
r
s
t
u
t
o
r
e
d
b
y
C
S
I
h
a
v
e
c
o
m
e
t
o
e
x
p
e
c
t
h
i
g
h
-
t
e
c
h
f
o
r
e
n
s
i
c
s
c
i
e
n
c
e
t
o
e
x
i
s
t
f
o
r
a
l
l
k
i
n
d
s
o
f
c
r
i
m
e
s
c
e
n
e
r
e
s
i
d
u
a
a
n
d
t
o
b
e
a
b
l
e
t
o
s
o
l
v
e
a
l
l
k
i
n
d
s
o
f
c
r
i
m
e
s
.
E
v
i
d
e
n
c
e
a
c
t
u
a
l
l
y
o
f
f
e
r
e
d
a
t
t
r
i
a
l
s
d
i
s
a
p
p
o
i
n
t
s
,
b
e
c
a
u
s
e
e
i
t
h
e
r
t
o
o
l
i
t
t
l
e
(
o
r
n
o
)
f
o
r
e
n
s
i
c
s
c
i
e
n
c
e
i
s
p
r
e
s
e
n
t
e
d
o
r
w
h
a
t
i
s
p
r
e
s
e
n
t
e
d
i
s
l
e
s
s
i
m
p
r
e
s
s
i
v
e
t
h
a
n
w
h
a
t
i
s
s
e
e
n
o
n
t
e
l
e
v
i
s
i
o
n
.
E
i
t
h
e
r
w
a
y
,
g
o
e
s
t
h
i
s
t
h
e
o
r
y
,
j
u
r
o
r
s
a
r
e
a
c
q
u
i
t
t
i
n
g
m
o
r
e
d
e
f
e
n
d
a
n
t
s
b
e
c
a
u
s
e
i
n
c
o
u
r
t
t
h
e
y
a
r
e
n
o
t
s
e
e
i
n
g
e
n
o
u
g
h
f
o
r
e
n
s
i
c
s
c
i
e
n
c
e
t
o
p
e
r
s
u
a
d
e
t
h
e
m
o
f
g
u
i
l
t
.
4
A
n
a
l
t
e
r
n
a
t
i
v
e
h
y
p
o
t
h
e
s
i
s
,
w
h
i
c
h
r
u
n
s
i
n
t
h
e
o
p
p
o
s
i
t
e
d
i
r
e
c
t
i
o
n
,
i
s
t
h
a
t
C
S
I
h
a
s
f
o
o
l
e
d
t
h
e
p
u
b
l
i
c
i
n
t
o
t
h
i
n
k
i
n
g
t
h
a
t
f
o
r
e
n
s
i
c
s
c
i
e
n
c
e
i
s
f
a
r
m
o
r
e
e
f
f
e
c
t
i
v
e
a
n
d
a
c
c
u
r
a
t
e
t
h
a
n
i
t
a
c
t
u
a
l
l
y
i
s
.
5
I
f
t
r
u
e
,
j
u
r
o
r
s
m
a
y
b
e
l
i
k
e
l
y
t
o
r
e
a
d
i
l
y
a
c
c
e
p
t
w
h
a
t
e
v
e
r
c
o
n
c
l
u
s
i
o
n
s
f
o
r
e
n
s
i
c
s
c
i
e
n
c
e
w
i
t
n
e
s
s
e
s
p
o
i
n
t
t
h
e
m
t
o
.
T
h
e
t
w
o
h
y
p
o
t
h
e
s
e
s
a
r
e
n
o
t
n
e
c
e
s
s
a
r
i
l
y
a
t
w
a
r
w
i
t
h
e
a
c
h
o
t
h
e
r
.
T
h
e
p
r
o
s
e
c
u
t
i
o
n
m
i
g
h
t
b
e
n
e
f
i
t
w
h
e
n
i
t
p
r
e
s
e
n
t
s
a
n
y
f
o
r
e
n
s
i
c
-
s
c
i
e
n
c
e
e
v
i
d
e
n
c
e
,
a
n
d
t
h
e
d
e
f
e
n
s
e
m
i
g
h
t
b
e
n
e
f
i
t
w
h
e
n
t
h
e
r
e
i
s
n
o
f
o
r
e
n
s
i
c
-
s
c
i
e
n
c
e
e
v
i
d
e
n
c
e
.
W
h
e
t
h
e
r
t
h
e
C
S
I
E
f
f
e
c
t
h
e
l
p
s
t
h
e
p
r
o
s
e
c
u
t
i
o
n
o
r
t
h
e
d
e
f
e
n
s
e
,
t
h
e
c
o
m
m
e
n
t
a
t
o
r
s
s
e
e
m
t
o
a
g
r
e
e
o
n
o
n
e
t
h
i
n
g
:
t
h
a
t
C
S
I
i
s
c
o
n
v
i
n
c
i
n
g
t
h
e
p
u
b
l
i
c
t
h
a
t
f
o
r
e
n
s
i
c
s
c
i
e
n
c
e
n
o
t
o
n
l
y
i
s
s
c
i
e
n
c
e
,
b
u
t
i
t
i
s
s
u
p
e
r
s
c
i
e
n
c
e
.
I
f
t
r
u
e
,
t
h
i
s
i
s
1
.
S
i
m
o
n
C
o
l
e
&
R
a
c
h
e
l
D
i
o
s
o
,
L
a
w
a
n
d
t
h
e
L
a
b
:
D
o
T
V
S
h
o
w
s
R
e
a
l
l
y
A
f
f
e
c
t
H
o
w
J
u
r
i
e
s
V
o
t
e
?
L
e
t
s
L
o
o
k
a
t
t
h
e
E
v
i
d
e
n
c
e
,
W
A
L
L
S
T
.
J
.
,
M
a
y
1
3
,
2
0
0
5
,
a
t
W
1
3
(
q
u
o
t
i
n
g
f
o
r
e
n
s
i
c
s
c
i
e
n
t
i
s
t
T
h
o
m
a
s
M
a
u
r
i
e
l
l
o
)
.
2
.
I
d
.
;
K
i
m
b
e
r
l
i
a
n
n
e
P
o
d
l
a
s
,
T
h
e
C
S
I
E
f
f
e
c
t
:
E
x
p
o
s
i
n
g
t
h
e
M
e
d
i
a
M
y
t
h
,
1
6
F
O
R
D
H
A
M
I
N
T
E
L
L
.
P
R
O
P
.
M
E
D
I
A
&
E
N
T
.
L
.
J
.
4
2
9
(
2
0
0
6
)
.
3
.
O
t
h
e
r
h
y
p
o
t
h
e
s
i
z
e
d
C
S
I
e
f
f
e
c
t
s
a
r
e
t
h
a
t
t
h
e
p
r
o
g
r
a
m
h
a
s
e
d
u
c
a
t
e
d
t
h
e
p
u
b
l
i
c
a
b
o
u
t
f
o
r
e
n
s
i
c
s
c
i
e
n
c
e
,
t
h
a
t
i
t
h
a
s
i
n
c
r
e
a
s
e
d
t
h
e
p
u
b
l
i
c
s
i
n
t
e
r
e
s
t
i
n
f
o
r
e
n
s
i
c
s
c
i
e
n
c
e
,
a
n
d
t
h
a
t
i
t
h
a
s
i
n
c
r
e
a
s
e
d
t
h
e
n
u
m
b
e
r
o
f
s
t
u
d
e
n
t
s
w
h
o
w
a
n
t
c
a
r
e
e
r
s
i
n
f
o
r
e
n
s
i
c
s
c
i
e
n
c
e
.
I
d
.
a
t
4
4
2
.
4
.
I
d
.
a
t
4
3
3
.
5
.
I
d
.
a
t
4
3
7
.
T
h
e
C
S
I
E
f
f
e
c
t
S
P
R
I
N
G
2
0
0
7
3
5
9
o
c
c
u
r
r
i
n
g
a
t
a
n
i
r
o
n
i
c
t
i
m
e
i
n
t
h
e
h
i
s
t
o
r
y
o
f
l
a
w
a
n
d
f
o
r
e
n
s
i
c
s
c
i
e
n
c
e
b
e
c
a
u
s
e
c
o
u
r
t
s
a
n
d
s
c
i
e
n
t
i
s
t
s
a
r
e
b
e
g
i
n
n
i
n
g
t
o
r
e
a
l
i
z
e
t
h
a
t
t
h
e
r
e
i
s
s
u
r
p
r
i
s
i
n
g
l
y
l
i
t
t
l
e
s
c
i
e
n
c
e
i
n
s
o
m
e
o
f
w
h
a
t
i
s
c
a
l
l
e
d
f
o
r
e
n
s
i
c
s
c
i
e
n
c
e
.
I
n
t
h
e
w
o
r
l
d
b
e
y
o
n
d
t
h
e
t
e
l
e
v
i
s
i
o
n
s
c
r
e
e
n
,
D
N
A
e
x
o
n
e
r
a
t
i
o
n
s
s
u
g
g
e
s
t
t
h
a
t
f
o
r
e
n
s
i
c
-
s
c
i
e
n
c
e
e
r
r
o
r
s
a
n
d
f
a
b
r
i
c
a
t
i
o
n
s
a
r
e
a
m
o
n
g
t
h
e
l
e
a
d
i
n
g
c
a
u
s
e
s
o
f
f
a
l
s
e
c
o
n
v
i
c
t
i
o
n
s
.
6
R
e
s
e
a
r
c
h
e
r
s
h
a
v
e
b
e
e
n
d
e
t
e
c
t
i
n
g
a
n
d
a
n
a
l
y
z
i
n
g
f
o
r
e
n
s
i
c
-
s
c
i
e
n
c
e
e
r
r
o
r
s
a
s
w
e
l
l
a
s
c
o
n
c
e
p
-
t
u
a
l
a
n
d
p
r
a
c
t
i
c
a
l
s
h
o
r
t
c
o
m
i
n
g
s
o
f
t
h
e
f
i
e
l
d
s
,
p
a
r
t
i
c
u
l
a
r
l
y
i
n
r
e
g
a
r
d
t
o
t
h
e
i
d
e
n
t
i
f
i
c
a
t
i
o
n
o
r
i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n
a
r
e
a
s
.
7
F
u
r
t
h
e
r
,
a
f
t
e
r
D
a
u
b
e
r
t
v
.
M
e
r
r
e
l
l
D
o
w
P
h
a
r
m
a
c
e
u
t
i
c
a
l
s
,
I
n
c
.
,
8
a
n
d
K
u
m
h
o
T
i
r
e
C
o
.
v
.
C
a
r
m
i
c
h
a
e
l
,
9
c
o
u
r
t
s
h
a
v
e
b
e
g
u
n
t
o
s
c
r
u
t
i
n
i
z
e
s
u
c
h
e
v
i
d
e
n
c
e
w
i
t
h
m
o
r
e
s
k
e
p
t
i
c
i
s
m
a
n
d
h
a
v
e
b
e
e
n
s
u
r
p
r
i
s
e
d
t
o
d
i
s
c
o
v
e
r
h
o
w
s
c
i
e
n
t
i
f
i
c
a
l
l
y
w
e
a
k
s
o
m
e
o
f
t
h
e
f
o
r
e
n
s
i
c
-
s
c
i
e
n
c
e
f
i
e
l
d
s
a
r
e
.
1
0
O
n
t
h
i
s
s
u
b
j
e
c
t
,
t
e
l
e
v
i
s
i
o
n
a
n
d
r
e
a
l
i
t
y
h
a
v
e
b
e
e
n
m
o
v
i
n
g
i
n
o
p
p
o
s
i
t
e
d
i
r
e
c
t
i
o
n
s
.
T
h
a
t
p
u
b
l
i
c
e
x
p
e
c
t
a
t
i
o
n
s
o
f
s
c
i
e
n
c
e
a
r
e
b
o
r
n
o
f
f
i
c
t
i
o
n
a
l
p
o
r
t
r
a
y
a
l
s
o
f
s
c
i
e
n
c
e
,
r
a
t
h
e
r
t
h
a
n
o
f
s
c
i
e
n
t
i
f
i
c
r
e
a
l
i
t
y
,
h
a
s
l
o
n
g
b
e
e
n
t
h
o
u
g
h
t
t
o
b
e
t
r
u
e
o
f
f
o
r
e
n
s
i
c
s
c
i
e
n
c
e
,
w
h
e
r
e
p
u
b
l
i
c
b
e
l
i
e
f
s
h
a
v
e
b
e
e
n
s
h
a
p
e
d
b
y
f
i
c
t
i
o
n
a
t
l
e
a
s
t
s
i
n
c
e
C
o
n
a
n
D
o
y
l
e
p
e
n
n
e
d
S
h
e
r
l
o
c
k
H
o
l
m
e
s
.
1
1
B
u
t
,
a
s
P
r
o
f
e
s
s
o
r
T
o
m
R
.
T
y
l
e
r
h
a
s
c
o
m
m
e
n
t
e
d
r
e
c
e
n
t
l
y
,
[
T
]
h
e
C
S
I
e
f
f
e
c
t
h
a
s
b
e
c
o
m
e
a
n
a
c
c
e
p
t
e
d
r
e
a
l
i
t
y
b
y
v
i
r
t
u
e
o
f
i
t
s
r
e
p
e
a
t
e
d
i
n
v
o
c
a
t
i
o
n
b
y
t
h
e
m
e
d
i
a
,
a
n
d
w
h
i
l
e
i
t
i
s
c
o
n
s
i
s
t
e
n
t
w
i
t
h
e
m
p
i
r
i
c
a
l
f
i
n
d
i
n
g
s
i
n
o
t
h
e
r
a
r
e
a
s
o
f
l
e
g
a
l
p
s
y
c
h
o
l
o
g
y
a
n
d
a
c
c
o
r
d
s
w
i
t
h
t
h
e
i
n
t
u
i
t
i
o
n
s
o
f
p
a
r
t
i
c
i
p
a
n
t
s
i
n
t
h
e
t
r
i
a
l
p
r
o
c
e
s
s
,
n
o
e
x
i
s
t
i
n
g
e
m
p
i
r
i
c
a
l
r
e
s
e
a
r
c
h
6
.
M
i
c
h
a
e
l
J
.
S
a
k
s
&
J
o
n
a
t
h
a
n
J
a
y
K
o
e
h
l
e
r
,
T
h
e
C
o
m
i
n
g
P
a
r
a
d
i
g
m
S
h
i
f
t
i
n
F
o
r
e
n
s
i
c
I
d
e
n
t
i
f
i
c
a
t
i
o
n
S
c
i
e
n
c
e
,
3
0
9
S
C
I
E
N
C
E
8
9
2
,
8
9
2
(
2
0
0
5
)
(
r
e
p
o
r
t
i
n
g
f
r
o
m
a
d
a
t
a
b
a
s
e
o
f
D
N
A
e
x
o
n
e
r
a
t
i
o
n
s
m
a
i
n
t
a
i
n
e
d
b
y
t
h
e
I
n
n
o
c
e
n
c
e
P
r
o
j
e
c
t
a
t
C
a
r
d
o
z
o
L
a
w
S
c
h
o
o
l
,
f
i
n
d
i
n
g
t
h
a
t
t
h
e
t
r
i
a
l
e
v
i
d
e
n
c
e
i
n
6
3
%
o
f
D
N
A
e
x
o
n
e
r
a
t
i
o
n
c
a
s
e
s
c
o
n
t
a
i
n
e
d
e
r
r
o
r
s
i
n
r
e
s
u
l
t
s
o
b
t
a
i
n
e
d
b
y
f
o
r
e
n
s
i
c
-
s
c
i
e
n
c
e
e
x
p
e
r
t
w
i
t
n
e
s
s
e
s
(
a
r
a
t
e
o
f
e
r
r
o
r
t
h
a
t
w
a
s
s
e
c
o
n
d
o
n
l
y
t
o
e
y
e
w
i
t
n
e
s
s
e
s
)
a
n
d
t
h
a
t
t
h
e
t
e
s
t
i
m
o
n
y
o
f
e
x
p
e
r
t
w
i
t
n
e
s
s
e
s
i
n
2
7
%
o
f
D
N
A
e
x
o
n
e
r
a
t
i
o
n
c
a
s
e
s
c
o
n
t
a
i
n
e
d
f
a
b
r
i
c
a
t
e
d
o
r
s
e
r
i
o
u
s
l
y
m
i
s
l
e
a
d
i
n
g
a
s
s
e
r
t
i
o
n
s
(
a
r
a
t
e
h
i
g
h
e
r
t
h
a
n
f
o
r
a
n
y
o
t
h
e
r
c
a
t
e
g
o
r
y
o
f
w
i
t
n
e
s
s
)
)
.
7
.
S
e
e
,
e
.
g
.
;
i
d
.
4
D
A
V
I
D
L
.
F
A
I
G
M
A
N
E
T
A
L
.
,
M
O
D
E
R
N
S
C
I
E
N
T
I
F
I
C
E
V
I
D
E
N
C
E
3
1
:
1
4
3
:
5
7
(
2
0
0
6
2
0
0
7
)
;
J
a
n
e
C
a
m
p
b
e
l
l
M
o
r
i
a
r
t
y
&
M
i
c
h
a
e
l
J
.
S
a
k
s
,
F
o
r
e
n
s
i
c
S
c
i
e
n
c
e
:
G
r
a
n
d
G
o
a
l
s
,
T
r
a
g
i
c
F
l
a
w
s
,
a
n
d
J
u
d
i
c
i
a
l
G
a
t
e
k
e
e
p
i
n
g
,
4
4
J
U
D
G
E
S
J
.
,
F
a
l
l
2
0
0
5
,
a
t
1
6
;
D
.
M
i
c
h
a
e
l
R
i
s
i
n
g
e
r
e
t
a
l
.
,
T
h
e
D
a
u
b
e
r
t
/
K
u
m
h
o
I
m
p
l
i
c
a
t
i
o
n
s
o
f
O
b
s
e
r
v
e
r
E
f
f
e
c
t
s
i
n
F
o
r
e
n
s
i
c
S
c
i
e
n
c
e
:
H
i
d
d
e
n
P
r
o
b
l
e
m
s
o
f
E
x
p
e
c
t
a
t
i
o
n
a
n
d
S
u
g
g
e
s
t
i
o
n
,
9
0
U
.
C
A
L
.
L
.
R
E
V
.
1
,
2
7
4
2
(
2
0
0
2
)
;
M
i
c
h
a
e
l
J
.
S
a
k
s
,
M
e
r
l
i
n
a
n
d
S
o
l
o
m
o
n
:
L
e
s
s
o
n
s
f
r
o
m
t
h
e
L
a
w
s
F
o
r
m
a
t
i
v
e
E
n
c
o
u
n
t
e
r
s
w
i
t
h
F
o
r
e
n
s
i
c
I
d
e
n
t
i
f
i
c
a
t
i
o
n
S
c
i
e
n
c
e
,
4
9
H
A
S
T
I
N
G
S
L
.
J
.
1
0
6
9
,
1
0
8
2
8
6
,
1
0
9
4
1
2
7
(
1
9
9
8
)
.
8
.
5
0
9
U
.
S
.
5
7
9
(
1
9
9
3
)
.
9
.
5
2
6
U
.
S
.
1
3
7
(
1
9
9
9
)
.
1
0
.
S
e
e
,
e
.
g
.
,
U
n
i
t
e
d
S
t
a
t
e
s
v
.
G
r
e
e
n
,
4
0
5
F
.
S
u
p
p
.
2
d
1
0
4
,
1
0
7
(
D
.
M
a
s
s
.
2
0
0
5
)
(
f
i
r
e
a
r
m
s
i
d
e
n
t
i
f
i
c
a
t
i
o
n
)
;
U
n
i
t
e
d
S
t
a
t
e
s
v
.
H
i
n
e
s
,
5
5
F
.
S
u
p
p
.
2
d
6
2
,
7
3
(
D
.
M
a
s
s
.
1
9
9
9
)
(
h
a
n
d
w
r
i
t
i
n
g
i
d
e
n
t
i
f
i
c
a
t
i
o
n
)
;
U
n
i
t
e
d
S
t
a
t
e
s
v
.
L
l
e
r
a
P
l
a
z
a
,
1
8
8
F
.
S
u
p
p
.
2
d
5
4
9
,
5
6
5
(
E
.
D
.
P
a
.
2
0
0
2
)
(
f
i
n
g
e
r
p
r
i
n
t
i
d
e
n
t
i
f
i
c
a
t
i
o
n
)
;
U
n
i
t
e
d
S
t
a
t
e
s
v
.
S
t
a
r
z
e
c
p
y
z
e
l
,
8
8
0
F
.
S
u
p
p
.
1
0
2
7
,
1
0
3
4
(
S
.
D
.
N
.
Y
.
1
9
9
5
)
(
h
a
n
d
w
r
i
t
i
n
g
i
d
e
n
t
i
f
i
c
a
t
i
o
n
)
;
W
i
l
l
i
a
m
s
o
n
v
.
R
e
y
n
o
l
d
s
,
9
0
4
F
.
S
u
p
p
.
1
5
2
9
,
1
5
5
7
(
E
.
D
.
O
k
l
a
.
1
9
9
5
)
(
m
i
c
r
o
s
c
o
p
i
c
h
a
i
r
i
d
e
n
t
i
f
i
c
a
t
i
o
n
)
.
1
1
.
S
e
e
J
o
h
n
I
.
T
h
o
r
n
t
o
n
,
C
r
i
m
i
n
a
l
i
s
t
i
c
s
P
a
s
t
,
P
r
e
s
e
n
t
,
F
u
t
u
r
e
,
1
1
L
E
X
E
T
S
C
I
E
N
T
I
A
1
,
1
6
(
1
9
7
5
)
.
S
c
h
w
e
i
t
z
e
r
a
n
d
S
a
k
s
3
6
0
4
7
J
U
R
I
M
E
T
R
I
C
S
s
h
o
w
s
t
h
a
t
i
t
a
c
t
u
a
l
l
y
o
c
c
u
r
s
.
1
2
T
h
e
s
u
p
p
o
s
e
d
e
f
f
e
c
t
s
h
a
v
e
b
e
e
n
s
u
p
p
o
r
t
e
d
b
y
n
o
t
h
i
n
g
m
o
r
e
t
h
a
n
a
n
e
c
d
o
t
e
s
.
1
3
T
h
e
p
r
e
s
e
n
t
a
r
t
i
c
l
e
r
e
p
o
r
t
s
a
n
e
m
p
i
r
i
c
a
l
s
t
u
d
y
d
e
s
i
g
n
e
d
t
o
t
e
s
t
o
n
e
f
u
n
d
a
m
e
n
t
a
l
e
l
e
m
e
n
t
o
f
t
h
e
C
S
I
E
f
f
e
c
t
w
h
e
t
h
e
r
p
e
o
p
l
e
w
h
o
a
r
e
m
a
r
i
n
a
t
e
d
i
n
f
o
r
e
n
s
i
c
-
s
c
i
e
n
c
e
f
i
c
t
i
o
n
r
e
a
c
t
d
i
f
f
e
r
e
n
t
l
y
t
o
t
h
e
m
o
r
e
c
o
n
v
e
n
t
i
o
n
a
l
k
i
n
d
s
o
f
f
o
r
e
n
s
i
c
s
c
i
e
n
c
e
t
h
a
t
c
o
m
e
t
o
c
o
u
r
t
.
S
p
e
c
i
f
i
c
a
l
l
y
,
d
o
C
S
I
v
i
e
w
e
r
s
g
i
v
e
f
o
r
e
n
s
i
c
s
c
i
e
n
c
e
m
o
r
e
w
e
i
g
h
t
t
h
a
n
i
t
d
e
s
e
r
v
e
s
?
O
r
,
b
y
h
a
v
i
n
g
h
a
d
t
h
e
i
r
e
x
p
e
c
t
a
t
i
o
n
s
r
a
i
s
e
d
,
h
a
v
e
t
h
e
y
b
e
c
o
m
e
m
o
r
e
s
k
e
p
t
i
c
a
l
?
1
2
.
T
o
m
R
.
T
y
l
e
r
,
V
i
e
w
i
n
g
C
S
I
a
n
d
t
h
e
T
h
r
e
s
h
o
l
d
o
f
G
u
i
l
t
:
M
a
n
a
g
i
n
g
T
r
u
t
h
a
n
d
J
u
s
t
i
c
e
i
n
R
e
a
l
i
t
y
a
n
d
F
i
c
t
i
o
n
,
1
1
5
Y
A
L
E
L
.
J
.
1
0
5
0
,
1
0
8
3
(
2
0
0
6
)
.
A
s
o
f
D
e
c
e
m
b
e
r
2
0
0
6
,
n
o
e
m
p
i
r
i
c
a
l
t
e
s
t
s
o
f
a
n
y
o
f
t
h
e
h
y
p
o
t
h
e
s
i
z
e
d
C
S
I
e
f
f
e
c
t
s
c
o
u
l
d
b
e
f
o
u
n
d
.
O
n
e
r
e
p
o
r
t
e
d
e
m
p
i
r
i
c
a
l
s
t
u
d
y
p
u
r
p
o
r
t
s
t
o
b
e
a
t
e
s
t
o
f
t
h
e
C
S
I
e
f
f
e
c
t
,
b
u
t
a
c
a
r
e
f
u
l
e
x
a
m
i
n
a
t
i
o
n
o
f
i
t
s
m
e
t
h
o
d
s
i
n
d
i
c
a
t
e
s
t
h
a
t
i
t
d
o
e
s
n
o
t
a
c
t
u
a
l
l
y
d
o
s
o
.
S
e
e
K
i
m
b
e
r
l
i
a
n
n
e
P
o
d
l
a
s
,
T
h
e
C
S
I
E
f
f
e
c
t
:
E
x
p
o
s
i
n
g
t
h
e
M
e
d
i
a
M
y
t
h
,
1
6
F
O
R
D
H
A
M
I
N
T
E
L
L
.
P
R
O
P
.
M
E
D
I
A
&
E
N
T
.
L
.
J
.
4
2
9
(
2
0
0
6
)
.
P
o
d
l
a
s
p
r
e
s
e
n
t
e
d
a
b
r
i
e
f
c
a
s
e
s
u
m
m
a
r
y
t
o
s
t
u
d
e
n
t
s
a
c
t
i
n
g
a
s
m
o
c
k
j
u
r
o
r
s
,
m
e
a
s
u
r
e
d
t
h
e
f
r
e
q
u
e
n
c
y
w
i
t
h
w
h
i
c
h
t
h
o
s
e
s
t
u
d
e
n
t
s
w
a
t
c
h
e
d
C
S
I
,
a
n
d
a
s
k
e
d
t
h
e
m
t
o
r
e
n
d
e
r
a
v
e
r
d
i
c
t
o
n
t
h
e
c
a
s
e
s
u
m
m
a
r
y
.
I
d
.
a
t
4
5
4
5
6
.
T
h
e
r
e
s
u
l
t
s
s
h
o
w
e
d
n
o
e
f
f
e
c
t
o
n
v
e
r
d
i
c
t
s
o
r
s
e
v
e
r
a
l
o
t
h
e
r
v
a
r
i
a
b
l
e
s
a
s
a
f
u
n
c
t
i
o
n
o
f
f
r
e
q
u
e
n
c
y
o
f
C
S
I
v
i
e
w
i
n
g
,
w
h
i
c
h
i
s
t
h
e
b
a
s
i
s
f
o
r
t
h
e
t
i
t
l
e
s
r
e
f
e
r
e
n
c
e
t
o
a
C
S
I
e
f
f
e
c
t
m
y
t
h
.
I
d
.
a
t
4
5
9
6
1
.
F
o
r
r
e
a
s
o
n
s
t
h
a
t
a
r
e
n
o
t
e
x
p
l
a
i
n
e
d
,
h
o
w
e
v
e
r
,
t
h
e
c
a
s
e
s
u
m
m
a
r
y
p
r
e
s
e
n
t
e
d
t
o
P
o
d
l
a
s
s
m
o
c
k
j
u
r
o
r
s
w
a
s
d
e
l
i
b
e
r
a
t
e
l
y
d
e
s
i
g
n
e
d
t
o
r
e
n
d
e
r
f
o
r
e
n
s
i
c
-
s
c
i
e
n
c
e
e
v
i
d
e
n
c
e
i
r
r
e
l
e
v
a
n
t
t
o
t
h
e
i
s
s
u
e
o
f
g
u
i
l
t
o
r
i
n
n
o
c
e
n
c
e
:
T
h
e
c
r
i
m
i
n
a
l
l
a
w
s
c
e
n
a
r
i
o
r
e
c
o
u
n
t
e
d
a
n
a
l
l
e
g
e
d
r
a
p
e
,
a
c
r
i
m
e
c
o
m
m
o
n
t
o
C
S
I
.
B
e
c
a
u
s
e
t
h
e
s
t
u
d
y
i
n
v
e
s
t
i
g
a
t
e
d
w
h
e
t
h
e
r
f
o
r
e
n
s
i
c
o
r
C
S
I
r
e
a
s
o
n
s
i
m
p
r
o
p
e
r
l
y
i
n
f
l
u
e
n
c
e
d
t
h
e
v
e
r
d
i
c
t
d
e
c
i
s
i
o
n
-
m
a
k
i
n
g
p
r
o
c
e
s
s
,
t
h
e
s
c
e
n
a
r
i
o
p
r
e
s
e
n
t
e
d
n
o
c
r
i
t
i
c
a
l
i
s
s
u
e
s
p
e
r
t
a
i
n
i
n
g
t
o
o
r
t
h
a
t
c
o
u
l
d
b
e
a
s
c
e
r
t
a
i
n
e
d
w
i
t
h
r
e
f
e
r
e
n
c
e
t
o
f
o
r
e
n
s
i
c
s
.
I
n
s
t
e
a
d
,
i
t
p
r
e
s
e
n
t
e
d
o
n
l
y
i
s
s
u
e
s
o
f
w
i
t
n
e
s
s
c
r
e
d
i
b
i
l
i
t
y
.
I
n
o
t
h
e
r
w
o
r
d
s
,
t
h
e
c
a
s
e
w
a
s
n
o
t
a
w
h
o
d
u
n
i
t
b
u
t
a
w
h
a
t
h
a
p
p
e
n
e
d
.
T
h
e
a
l
l
e
g
e
d
v
i
c
t
i
m
c
l
a
i
m
e
d
t
h
a
t
s
h
e
w
a
s
f
o
r
c
e
d
t
o
h
a
v
e
n
o
n
-
c
o
n
s
e
n
s
u
a
l
s
e
x
,
w
h
e
r
e
a
s
t
h
e
d
e
f
e
n
d
a
n
t
c
l
a
i
m
e
d
t
h
a
t
t
h
e
s
e
x
u
a
l
e
n
c
o
u
n
t
e
r
w
a
s
w
h
o
l
l
y
c
o
n
s
e
n
s
u
a
l
.
B
e
c
a
u
s
e
t
h
e
r
e
w
a
s
n
o
q
u
e
s
t
i
o
n
o
f
w
h
e
t
h
e
r
t
h
e
d
e
f
e
n
d
a
n
t
a
n
d
a
l
l
e
g
e
d
v
i
c
t
i
m
h
a
d
e
n
g
a
g
e
d
i
n
s
e
x
u
a
l
i
n
t
e
r
c
o
u
r
s
e
b
u
t
,
r
a
t
h
e
r
,
w
h
e
t
h
e
r
t
h
e
i
n
t
e
r
c
o
u
r
s
e
w
a
s
c
o
n
s
e
n
s
u
a
l
f
o
r
e
n
s
i
c
e
v
i
d
e
n
c
e
c
o
u
l
d
n
o
t
s
h
e
d
l
i
g
h
t
o
n
t
h
e
c
r
i
t
i
c
a
l
i
s
s
u
e
o
f
c
o
n
s
e
n
t
.
T
h
i
s
r
e
n
d
e
r
e
d
a
n
y
f
o
r
e
n
s
i
c
e
v
i
d
e
n
c
e
u
t
t
e
r
l
y
i
r
r
e
l
e
v
a
n
t
t
o
a
c
o
n
c
l
u
s
i
o
n
o
f
n
o
t
g
u
i
l
t
y
.
I
d
.
a
t
4
5
5
.
(
f
o
o
t
n
o
t
e
o
m
i
t
t
e
d
)
.
O
n
l
y
a
s
m
a
l
l
p
e
r
c
e
n
t
a
g
e
o
f
P
o
d
l
a
s
s
s
t
u
d
y
p
a
r
t
i
c
i
p
a
n
t
s
g
a
v
e
i
n
s
u
f
f
i
c
i
e
n
t
f
o
r
e
n
s
i
c
-
s
c
i
e
n
c
e
t
e
s
t
i
n
g
a
s
t
h
e
r
e
a
s
o
n
f
o
r
t
h
e
i
r
v
o
t
e
s
t
o
a
c
q
u
i
t
.
I
d
.
a
t
4
6
0
.
P
o
d
l
a
s
i
n
t
e
r
p
r
e
t
s
t
h
i
s
a
s
s
h
o
w
i
n
g
t
h
a
t
i
n
r
e
n
d
e
r
i
n
g
n
o
t
g
u
i
l
t
y
v
e
r
d
i
c
t
s
,
f
r
e
q
u
e
n
t
v
i
e
w
e
r
s
o
f
C
S
I
a
r
e
n
o
m
o
r
e
i
n
f
l
u
e
n
c
e
d
b
y
C
S
I
f
a
c
t
o
r
s
t
h
a
n
a
r
e
n
o
n
-
f
r
e
q
u
e
n
t
v
i
e
w
e
r
s
.
I
n
f
a
c
t
,
c
o
n
s
i
d
e
r
i
n
g
t
h
e
s
m
a
l
l
m
i
n
o
r
i
t
y
o
f
C
S
I
v
i
e
w
e
r
s
w
h
o
c
o
n
s
i
d
e
r
e
d
C
S
I
f
a
c
t
o
r
s
i
n
t
h
e
i
r
v
e
r
d
i
c
t
s
,
t
h
e
d
a
t
a
s
u
g
g
e
s
t
s
t
h
a
t
t
h
e
y
a
r
e
n
o
t
i
n
f
l
u
e
n
c
e
d
b
y
s
u
c
h
f
a
c
t
o
r
s
,
o
r
c
o
n
s
i
d
e
r
a
n
d
a
r
e
i
n
f
l
u
e
n
c
e
d
b
y
t
h
e
v
e
r
y
s
a
m
e
f
a
c
t
o
r
s
a
s
a
r
e
n
o
n
-
f
r
e
q
u
e
n
t
v
i
e
w
e
r
s
.
C
o
n
s
e
q
u
e
n
t
l
y
,
t
h
e
e
m
p
i
r
i
c
a
l
e
v
i
d
e
n
c
e
d
o
e
s
n
o
t
s
u
p
p
o
r
t
a
n
y
a
n
t
i
-
p
r
o
s
e
c
u
t
i
o
n
C
S
I
E
f
f
e
c
t
.
I
d
.
a
t
4
6
1
.
H
o
w
e
v
e
r
,
b
e
c
a
u
s
e
f
o
r
e
n
s
i
c
-
s
c
i
e
n
c
e
e
v
i
d
e
n
c
e
w
a
s
i
r
r
e
l
e
v
a
n
t
t
o
t
h
e
c
a
s
e
,
t
h
e
m
o
r
e
l
i
k
e
l
y
e
x
p
l
a
n
a
t
i
o
n
i
s
t
h
a
t
t
h
e
g
r
e
a
t
m
a
j
o
r
i
t
y
o
f
p
a
r
t
i
c
i
p
a
n
t
s
s
a
w
t
h
a
t
i
r
r
e
l
e
v
a
n
c
e
.
A
n
y
t
e
s
t
i
n
g
(
s
u
c
h
a
s
f
i
n
g
e
r
p
r
i
n
t
s
o
r
D
N
A
)
t
h
a
t
m
i
g
h
t
s
h
o
w
t
h
e
d
e
f
e
n
d
a
n
t
w
a
s
i
n
t
h
e
c
o
m
p
l
a
i
n
a
n
t
s
b
e
d
r
o
o
m
o
r
h
a
d
s
e
x
w
i
t
h
h
e
r
w
o
u
l
d
o
n
l
y
c
o
n
f
i
r
m
f
a
c
t
s
t
o
w
h
i
c
h
t
h
e
p
a
r
t
i
e
s
a
l
r
e
a
d
y
a
g
r
e
e
d
.
W
h
y
w
o
u
l
d
a
n
y
o
n
e
,
i
n
c
l
u
d
i
n
g
d
i
e
-
h
a
r
d
C
S
I
f
a
n
s
,
f
a
u
l
t
t
h
e
g
o
v
e
r
n
m
e
n
t
f
o
r
n
o
t
c
o
n
d
u
c
t
i
n
g
i
r
r
e
l
e
v
a
n
t
s
c
i
e
n
t
i
f
i
c
t
e
s
t
s
t
h
a
t
w
o
u
l
d
d
o
n
o
t
h
i
n
g
t
o
h
e
l
p
r
e
s
o
l
v
e
t
h
e
d
i
s
p
u
t
e
d
i
s
s
u
e
s
i
n
t
h
e
c
a
s
e
?
1
3
.
E
.
g
.
,
C
o
l
e
&
D
i
o
s
o
,
s
u
p
r
a
n
o
t
e
1
;
J
a
n
i
n
e
R
o
b
b
e
n
,
T
h
e
C
S
I
E
f
f
e
c
t
:
P
o
p
u
l
a
r
C
u
l
t
u
r
e
F
i
n
d
s
t
h
e
J
u
s
t
i
c
e
S
y
s
t
e
m
,
6
6
O
R
.
S
T
.
B
.
B
U
L
L
.
8
(
2
0
0
5
)
;
K
i
t
R
.
R
o
a
n
e
,
T
h
e
C
S
I
E
f
f
e
c
t
,
U
.
S
.
N
E
W
S
&
W
O
R
L
D
R
E
P
.
,
A
p
r
.
2
5
,
2
0
0
5
,
a
t
4
8
;
R
i
c
h
a
r
d
W
i
l
l
i
n
g
,
C
S
I
E
f
f
e
c
t
H
a
s
J
u
r
i
e
s
W
a
n
t
i
n
g
M
o
r
e
E
v
i
d
e
n
c
e
,
U
S
A
T
O
D
A
Y
,
A
u
g
.
5
,
2
0
0
4
,
a
t
1
A
.
T
h
e
C
S
I
E
f
f
e
c
t
S
P
R
I
N
G
2
0
0
7
3
6
1
I
.
T
H
E
S
T
U
D
Y
W
e
p
r
e
p
a
r
e
d
a
b
r
i
e
f
(
1
1
-
p
a
g
e
)
s
i
m
u
l
a
t
e
d
t
r
a
n
s
c
r
i
p
t
o
f
a
c
r
i
m
i
n
a
l
t
r
i
a
l
i
n
w
h
i
c
h
t
h
e
p
r
i
n
c
i
p
a
l
i
t
e
m
o
f
i
n
c
u
l
p
a
t
o
r
y
e
v
i
d
e
n
c
e
w
a
s
h
a
i
r
r
e
c
o
v
e
r
e
d
f
r
o
m
a
s
k
i
m
a
s
k
t
h
a
t
h
a
d
b
e
e
n
l
e
f
t
a
t
t
h
e
c
r
i
m
e
s
c
e
n
e
b
y
t
h
e
p
e
r
p
e
t
r
a
t
o
r
.
D
u
r
i
n
g
t
h
e
t
r
i
a
l
,
a
f
o
r
e
n
s
i
c
s
c
i
e
n
t
i
s
t
t
e
s
t
i
f
i
e
d
t
h
a
t
h
e
h
a
d
c
o
n
d
u
c
t
e
d
a
m
i
c
r
o
s
c
o
p
i
c
a
n
a
l
y
s
i
s
o
f
t
h
e
c
r
i
m
e
s
c
e
n
e
h
a
i
r
a
n
d
t
h
e
d
e
f
e
n
d
a
n
t
s
h
a
i
r
a
n
d
t
h
a
t
i
t
w
a
s
h
i
s
o
p
i
n
i
o
n
t
h
a
t
t
h
e
h
a
i
r
f
r
o
m
t
h
e
c
r
i
m
e
s
c
e
n
e
o
r
i
g
i
n
a
t
e
d
f
r
o
m
t
h
e
d
e
f
e
n
d
a
n
t
.
1
4
T
h
i
s
m
i
c
r
o
-
s
c
o
p
i
c
h
a
i
r
a
n
a
l
y
s
i
s
w
a
s
t
h
e
o
n
l
y
f
o
r
e
n
s
i
c
e
v
i
d
e
n
c
e
o
f
f
e
r
e
d
a
n
d
w
a
s
s
i
m
i
l
a
r
i
n
i
t
s
e
s
s
e
n
t
i
a
l
s
t
o
t
h
e
s
o
r
t
o
f
t
e
s
t
i
m
o
n
y
t
h
a
t
a
m
i
c
r
o
s
c
o
p
i
c
h
a
i
r
i
d
e
n
t
i
f
i
c
a
t
i
o
n
e
x
p
e
r
t
g
i
v
e
s
i
n
t
r
i
a
l
s
w
h
e
n
i
d
e
n
t
i
f
y
i
n
g
a
s
u
s
p
e
c
t
a
s
t
h
e
s
o
u
r
c
e
o
f
q
u
e
s
t
i
o
n
e
d
h
a
i
r
.
1
5
T
h
e
t
r
a
n
s
c
r
i
p
t
w
a
s
p
r
e
s
e
n
t
e
d
t
o
4
8
u
n
i
v
e
r
s
i
t
y
s
t
u
d
e
n
t
s
,
a
l
l
o
f
w
h
o
m
1
4
.
T
h
e
c
a
s
e
i
n
v
o
l
v
e
d
t
h
e
m
u
r
d
e
r
o
f
a
c
o
n
v
e
n
i
e
n
c
e
s
t
o
r
e
c
l
e
r
k
d
u
r
i
n
g
a
n
a
t
t
e
m
p
t
e
d
r
o
b
b
e
r
y
.
T
h
e
i
n
c
u
l
p
a
t
o
r
y
e
v
i
d
e
n
c
e
c
o
n
s
i
s
t
e
d
o
f
(
1
)
a
d
o
u
b
t
f
u
l
e
y
e
w
i
t
n
e
s
s
i
d
e
n
t
i
f
i
c
a
t
i
o
n
(
t
h
e
w
i
t
n
e
s
s
b
a
r
e
l
y
s
a
w
t
h
e
p
e
r
p
e
t
r
a
t
o
r
,
a
t
s
o
m
e
d
i
s
t
a
n
c
e
,
a
t
n
i
g
h
t
)
,
a
n
d
(
2
)
m
i
c
r
o
s
c
o
p
i
c
h
a
i
r
c
o
m
p
a
r
i
s
o
n
.
T
h
e
k
e
y
s
t
a
t
e
m
e
n
t
s
b
y
t
h
e
e
x
a
m
i
n
e
r
w
e
r
e
t
h
e
f
o
l
l
o
w
i
n
g
:
A
:
Y
e
s
.
I
o
b
t
a
i
n
e
d
s
e
v
e
r
a
l
h
a
i
r
s
t
a
k
e
n
f
r
o
m
a
s
k
i
m
a
s
k
k
n
o
w
n
t
o
h
a
v
e
b
e
e
n
w
o
r
n
b
y
t
h
e
p
e
r
p
e
t
r
a
t
o
r
a
t
t
h
e
t
i
m
e
o
f
t
h
e
c
r
i
m
e
.
I
a
l
s
o
r
e
m
o
v
e
d
a
s
a
m
p
l
e
o
f
h
a
i
r
s
f
r
o
m
t
h
e
d
e
f
e
n
d
a
n
t
.
T
h
e
n
I
c
o
m
p
a
r
e
d
t
h
e
t
w
o
s
e
t
s
o
f
h
a
i
r
s
u
n
d
e
r
a
m
i
c
r
o
s
c
o
p
e
.
A
:
T
h
e
t
w
o
s
e
t
s
o
f
h
a
i
r
s
a
p
p
e
a
r
e
d
q
u
i
t
e
s
i
m
i
l
a
r
.
T
h
e
y
w
e
r
e
a
l
i
k
e
i
n
a
l
l
m
i
c
r
o
s
c
o
p
i
c
d
e
t
a
i
l
.
A
:
I
n
m
y
f
i
e
l
d
,
w
e
e
x
a
m
i
n
e
1
6
d
i
f
f
e
r
e
n
t
a
t
t
r
i
b
u
t
e
s
o
f
h
a
i
r
,
f
o
r
e
x
a
m
p
l
e
,
c
o
l
o
r
,
s
t
r
u
c
t
u
r
e
,
a
n
d
t
h
i
c
k
n
e
s
s
w
o
u
l
d
b
e
a
m
o
n
g
t
h
e
m
.
A
s
I
m
a
k
e
m
y
c
o
m
p
a
r
i
s
o
n
s
,
I
t
h
i
n
k
a
b
o
u
t
h
o
w
c
o
m
m
o
n
o
r
u
n
c
o
m
m
o
n
I
b
e
l
i
e
v
e
t
h
e
d
i
f
f
e
r
e
n
t
c
h
a
r
a
c
t
e
r
i
s
t
i
c
s
a
r
e
i
n
t
h
e
p
o
p
u
l
a
t
i
o
n
,
b
a
s
e
d
o
n
o
t
h
e
r
h
a
i
r
I
v
e
e
x
a
m
i
n
e
d
.
T
h
e
m
o
r
e
r
a
r
e
t
h
e
c
h
a
r
a
c
t
e
r
i
s
t
i
c
s
a
r
e
,
t
h
e
m
o
r
e
s
i
g
n
i
f
i
c
a
n
t
i
t
i
s
i
f
t
h
o
s
e
c
h
a
r
a
c
t
e
r
i
s
t
i
c
s
a
r
e
f
o
u
n
d
i
n
b
o
t
h
t
h
e
c
r
i
m
e
s
c
e
n
e
h
a
i
r
a
n
d
t
h
e
d
e
f
e
n
d
a
n
t
s
h
a
i
r
.
A
:
I
t
i
s
m
y
j
u
d
g
m
e
n
t
t
h
a
t
i
f
a
p
e
r
s
o
n
w
e
r
e
s
e
l
e
c
t
e
d
a
t
r
a
n
d
o
m
f
r
o
m
t
h
e
p
o
p
u
l
a
t
i
o
n
,
t
h
e
r
e
i
s
a
v
e
r
y
l
o
w
p
r
o
b
a
b
i
l
i
t
y
t
h
a
t
t
h
e
p
e
r
s
o
n
w
o
u
l
d
h
a
v
e
h
a
i
r
w
i
t
h
t
h
e
e
x
a
c
t
s
a
m
e
a
t
t
r
i
b
u
t
e
s
a
s
t
h
e
h
a
i
r
l
e
f
t
a
t
t
h
e
c
r
i
m
e
s
c
e
n
e
b
y
t
h
e
p
e
r
p
e
t
r
a
t
o
r
o
f
t
h
i
s
c
r
i
m
e
.
T
h
a
t
s
h
o
u
l
d
g
i
v
e
t
h
e
j
u
r
y
a
s
e
n
s
e
o
f
t
h
e
r
a
r
i
t
y
o
f
t
h
e
c
r
i
m
e
s
c
e
n
e
h
a
i
r
a
n
d
t
h
e
i
m
p
r
o
b
a
b
i
l
i
t
y
t
h
a
t
t
h
e
s
i
m
i
l
a
r
i
t
y
b
e
t
w
e
e
n
t
h
e
d
e
f
e
n
d
a
n
t
s
h
a
i
r
a
n
d
t
h
e
p
e
r
p
e
t
r
a
t
o
r
s
i
s
t
h
e
r
e
s
u
l
t
o
f
r
a
n
d
o
m
c
o
i
n
c
i
d
e
n
c
e
.
S
o
,
a
l
t
h
o
u
g
h
w
e
c
a
n
n
o
t
r
u
l
e
o
u
t
t
h
e
p
o
s
s
i
b
i
l
i
t
y
t
h
a
t
t
h
e
h
a
i
r
s
c
a
m
e
f
r
o
m
s
o
m
e
o
n
e
o
t
h
e
r
t
h
a
n
t
h
e
d
e
f
e
n
d
a
n
t
,
t
h
e
p
r
o
b
a
b
i
l
i
t
y
o
f
t
h
a
t
i
s
l
o
w
.
A
:
B
a
s
e
d
o
n
m
y
a
n
a
l
y
s
i
s
,
I
b
e
l
i
e
v
e
t
h
a
t
t
h
e
h
a
i
r
f
o
u
n
d
a
t
t
h
e
c
r
i
m
e
s
c
e
n
e
b
e
l
o
n
g
s
t
o
t
h
e
d
e
f
e
n
d
a
n
t
.
H
a
l
f
o
f
o
u
r
m
o
c
k
j
u
r
o
r
s
w
e
r
e
p
r
e
s
e
n
t
e
d
w
i
t
h
a
l
l
o
f
t
h
e
s
e
s
t
a
t
e
m
e
n
t
s
(
m
e
t
h
o
d
d
e
s
c
r
i
b
e
d
)
;
t
h
e
o
t
h
e
r
h
a
l
f
r
e
c
e
i
v
e
d
o
n
l
y
t
h
e
f
i
n
a
l
s
t
a
t
e
m
e
n
t
(
c
o
n
c
l
u
s
i
o
n
o
n
l
y
)
.
N
o
d
i
f
f
e
r
e
n
c
e
s
i
n
r
e
s
p
o
n
s
e
s
r
e
s
u
l
t
e
d
f
r
o
m
t
h
e
i
r
r
e
c
e
i
v
i
n
g
t
h
e
d
e
s
c
r
i
p
t
i
o
n
o
f
t
h
e
m
e
t
h
o
d
v
e
r
s
u
s
t
h
e
c
o
n
c
l
u
s
i
o
n
a
l
o
n
e
.
1
5
.
A
s
t
u
t
e
r
e
a
d
e
r
s
m
i
g
h
t
n
o
t
e
t
h
a
t
t
h
e
f
i
n
a
l
s
t
a
t
e
m
e
n
t
m
a
d
e
b
y
t
h
e
f
o
r
e
n
s
i
c
s
c
i
e
n
t
i
s
t
i
n
o
u
r
s
i
m
u
l
a
t
e
d
t
r
i
a
l
t
r
a
n
s
c
r
i
p
t
d
e
s
c
r
i
b
e
d
i
n
t
h
e
p
r
e
c
e
d
i
n
g
f
o
o
t
n
o
t
e
i
s
i
n
a
p
p
r
o
p
r
i
a
t
e
f
o
r
t
e
s
t
i
m
o
n
y
a
b
o
u
t
m
i
c
r
o
s
c
o
p
i
c
h
a
i
r
e
x
a
m
i
n
a
t
i
o
n
,
w
h
i
c
h
i
s
c
o
n
s
i
d
e
r
e
d
u
n
a
b
l
e
t
o
z
e
r
o
i
n
o
n
a
s
i
n
g
l
e
i
n
d
i
v
i
d
u
a
l
,
b
u
t
o
n
l
y
t
o
n
a
r
r
o
w
t
h
e
p
o
o
l
o
f
s
u
s
p
e
c
t
s
.
F
i
r
s
t
o
f
a
l
l
,
n
o
t
w
i
t
h
s
t
a
n
d
i
n
g
t
e
x
t
b
o
o
k
a
d
m
o
n
i
t
i
o
n
s
a
b
o
u
t
t
h
e
i
m
p
r
o
p
r
i
e
t
y
o
f
m
i
c
r
o
s
c
o
p
i
c
h
a
i
r
e
x
a
m
i
n
a
t
i
o
n
t
e
s
t
i
m
o
n
y
i
d
e
n
t
i
f
y
i
n
g
a
s
i
n
g
l
e
i
n
d
i
v
i
d
u
a
l
a
s
t
h
e
s
o
u
r
c
e
o
f
t
h
e
c
r
i
m
e
s
c
e
n
e
h
a
i
r
,
i
t
i
s
n
o
t
u
n
u
s
u
a
l
f
o
r
s
u
c
h
e
x
p
e
r
t
t
e
s
t
i
m
o
n
y
t
o
d
o
s
o
.
S
e
e
,
e
.
g
.
,
G
r
e
g
o
r
y
v
.
C
i
t
y
o
f
L
o
u
i
s
v
i
l
l
e
,
4
4
4
F
.
3
d
7
2
5
(
6
t
h
C
i
r
.
2
0
0
6
)
;
W
i
l
l
i
a
m
s
o
n
v
.
R
e
y
n
o
l
d
s
,
9
0
4
F
.
S
u
p
p
.
1
5
2
9
(
E
.
D
.
O
k
l
a
.
1
9
9
5
)
;
J
o
h
n
I
.
T
h
o
r
n
t
o
n
&
J
o
s
e
p
h
L
.
P
e
t
e
r
s
o
n
,
T
h
e
G
e
n
e
r
a
l
A
s
s
u
m
p
t
i
o
n
s
a
n
d
R
a
t
i
o
n
a
l
e
o
f
F
o
r
e
n
s
i
c
I
d
e
n
t
i
f
i
c
a
t
i
o
n
,
i
n
F
A
I
G
M
A
N
E
T
A
L
.
,
s
u
p
r
a
n
o
t
e
7
,
3
1
:
1
3
1
:
4
7
.
M
o
r
e
i
m
p
o
r
t
a
n
t
,
i
n
e
s
s
e
n
t
i
a
l
r
e
s
p
e
c
t
s
t
h
e
e
x
p
e
r
t
s
o
p
i
n
i
o
n
r
e
g
a
r
d
i
n
g
h
a
i
r
i
d
e
n
t
i
f
i
c
a
t
i
o
n
i
n
o
u
r
s
i
m
u
l
a
t
e
d
t
r
i
a
l
t
y
p
i
f
i
e
s
o
t
h
e
r
f
o
r
e
n
s
i
c
-
s
c
i
e
n
c
e
i
d
e
n
t
i
f
i
c
a
t
i
o
n
t
e
s
t
i
m
o
n
y
,
s
u
c
h
a
s
t
h
a
t
g
i
v
e
n
b
y
e
x
a
m
i
n
e
r
s
o
f
h
a
n
d
w
r
i
t
i
n
g
,
f
i
r
e
a
r
m
s
,
f
i
n
g
e
r
p
r
i
n
t
s
,
a
n
d
b
i
t
e
m
a
r
k
s
.
O
u
r
r
e
s
p
o
n
d
e
n
t
s
h
a
v
e
n
o
w
a
y
o
f
k
n
o
w
i
n
g
t
h
a
t
h
a
i
r
i
d
e
n
t
i
f
i
c
a
t
i
o
n
i
s
a
n
e
x
c
e
p
t
i
o
n
(
f
o
r
m
a
l
l
y
,
t
h
o
u
g
h
n
o
t
a
l
w
a
y
s
i
n
p
r
a
c
t
i
c
e
)
.
F
i
n
a
l
l
y
,
o
u
r
f
o
r
e
n
s
i
c
s
c
i
e
n
t
i
s
t
p
e
r
f
o
r
m
e
d
a
n
e
x
a
m
i
n
a
t
i
o
n
t
h
a
t
i
s
t
y
p
i
c
a
l
o
f
o
t
h
e
r
f
i
e
l
d
s
o
f
f
o
r
e
n
s
i
c
i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n
s
:
s
h
e
m
a
k
e
s
a
v
i
s
u
a
l
o
b
s
e
r
v
a
t
i
o
n
a
n
d
d
r
a
w
s
a
s
u
b
j
e
c
t
i
v
e
i
n
f
e
r
e
n
c
e
.
I
t
i
s
t
h
i
s
s
o
r
t
o
f
u
n
s
c
i
e
n
t
i
f
i
c
,
l
o
w
-
t
e
c
h
p
r
o
c
e
d
u
r
e
t
h
a
t
s
t
a
n
d
s
i
n
c
o
n
t
r
a
s
t
t
o
w
h
a
t
i
s
p
o
r
t
r
a
y
e
d
i
n
s
o
m
u
c
h
o
f
C
S
I
.
S
c
h
w
e
i
t
z
e
r
a
n
d
S
a
k
s
3
6
2
4
7
J
U
R
I
M
E
T
R
I
C
S
w
e
r
e
j
u
r
y
-
e
l
i
g
i
b
l
e
(
o
v
e
r
1
8
y
e
a
r
s
o
f
a
g
e
a
n
d
c
i
t
i
z
e
n
s
o
f
t
h
e
U
n
i
t
e
d
S
t
a
t
e
s
)
,
w
h
o
a
s
s
e
m
b
l
e
d
i
n
s
m
a
l
l
g
r
o
u
p
s
f
o
r
t
h
e
p
u
r
p
o
s
e
o
f
p
a
r
t
i
c
i
p
a
t
i
n
g
i
n
t
h
e
s
t
u
d
y
.
1
6
A
f
t
e
r
r
e
a
d
i
n
g
t
h
e
t
r
a
n
s
c
r
i
p
t
,
p
a
r
t
i
c
i
p
a
n
t
s
c
o
m
p
l
e
t
e
d
a
q
u
e
s
t
i
o
n
n
a
i
r
e
t
h
a
t
a
s
s
e
s
s
e
d
t
h
e
i
r
p
e
r
c
e
p
t
i
o
n
s
o
f
b
o
t
h
t
h
e
t
r
i
a
l
a
s
a
w
h
o
l
e
a
n
d
t
h
e
f
o
r
e
n
s
i
c
-
s
c
i
e
n
c
e
e
v
i
d
e
n
c
e
s
p
e
c
i
f
i
c
a
l
l
y
.
M
o
s
t
q
u
e
s
t
i
o
n
s
c
a
l
l
e
d
f
o
r
r
e
s
p
o
n
s
e
s
o
n
s
e
v
e
n
-
p
o
i
n
t
s
c
a
l
e
s
.
F
i
n
a
l
l
y
,
t
h
e
p
a
r
t
i
c
i
p
a
n
t
s
w
e
r
e
a
s
k
e
d
a
b
o
u
t
t
h
e
f
r
e
q
u
e
n
c
y
w
i
t
h
w
h
i
c
h
t
h
e
y
v
i
e
w
e
d
b
o
t
h
f
o
r
e
n
s
i
c
-
s
c
i
e
n
c
e
t
h
e
m
e
d
p
r
o
g
r
a
m
s
(
f
o
r
e
x
a
m
p
l
e
,
C
S
I
,
C
S
I
M
i
a
m
i
,
a
n
d
C
S
I
N
e
w
Y
o
r
k
)
a
n
d
g
e
n
e
r
a
l
c
r
i
m
e
-
t
h
e
m
e
d
p
r
o
g
r
a
m
s
(
f
o
r
e
x
a
m
p
l
e
,
L
a
w
a
n
d
O
r
d
e
r
,
C
o
l
d
C
a
s
e
,
a
n
d
W
i
t
h
o
u
t
a
T
r
a
c
e
)
.
I
I
.
R
E
S
U
L
T
S
P
a
r
t
i
c
i
p
a
n
t
s
w
e
r
e
g
r
o
u
p
e
d
a
c
c
o
r
d
i
n
g
t
o
t
h
e
f
r
e
q
u
e
n
c
y
w
i
t
h
w
h
i
c
h
t
h
e
y
r
e
p
o
r
t
e
d
v
i
e
w
i
n
g
t
h
e
f
o
r
e
n
s
i
c
s
c
i
e
n
c
e
a
n
d
g
e
n
e
r
a
l
c
r
i
m
e
t
e
l
e
v
i
s
i
o
n
p
r
o
g
r
a
m
s
.
T
h
o
s
e
p
a
r
t
i
c
i
p
a
n
t
s
w
h
o
r
e
p
o
r
t
e
d
n
e
v
e
r
w
a
t
c
h
i
n
g
w
e
r
e
c
l
a
s
s
i
f
i
e
d
a
s
n
o
n
v
i
e
w
-
e
r
s
,
a
n
d
t
h
o
s
e
w
h
o
r
e
p
o
r
t
e
d
w
a
t
c
h
i
n
g
o
n
e
o
r
m
o
r
e
s
h
o
w
s
p
e
r
m
o
n
t
h
w
e
r
e
c
l
a
s
s
i
f
i
e
d
a
s
v
i
e
w
e
r
s
.
1
7
P
a
r
t
i
c
i
p
a
n
t
s
w
h
o
w
e
r
e
f
o
r
e
n
s
i
c
-
s
c
i
e
n
c
e
v
i
e
w
e
r
s
r
a
t
e
d
t
h
e
m
s
e
l
v
e
s
a
s
h
a
v
i
n
g
a
b
e
t
t
e
r
u
n
d
e
r
s
t
a
n
d
i
n
g
o
f
t
h
e
t
a
s
k
s
t
h
a
t
f
o
r
e
n
s
i
c
s
c
i
e
n
t
i
s
t
s
p
e
r
f
o
r
m
(
M
=
5
.
0
0
)
t
h
a
n
n
o
n
v
i
e
w
e
r
s
(
M
=
3
.
4
2
)
.
1
8
M
o
r
e
i
m
p
o
r
t
a
n
t
l
y
,
f
o
r
e
n
s
i
c
-
s
c
i
e
n
c
e
v
i
e
w
e
r
s
w
e
r
e
m
o
r
e
c
r
i
t
i
c
a
l
o
f
t
h
e
f
o
r
e
n
s
i
c
e
v
i
d
e
n
c
e
p
r
e
s
e
n
t
e
d
i
n
o
u
r
t
r
i
a
l
.
W
h
e
n
a
s
k
e
d
a
b
o
u
t
t
h
e
e
x
t
e
n
t
t
o
w
h
i
c
h
t
h
e
y
a
c
c
e
p
t
e
d
a
s
t
r
u
e
t
h
e
o
u
t
c
o
m
e
o
f
t
h
e
h
a
i
r
a
n
a
l
y
s
i
s
(
t
h
a
t
t
h
e
h
a
i
r
s
s
h
a
r
e
d
a
c
o
m
m
o
n
s
o
u
r
c
e
)
,
f
o
r
e
n
s
i
c
-
s
c
i
e
n
c
e
v
i
e
w
e
r
s
r
a
t
e
d
t
h
e
e
v
i
d
e
n
c
e
s
i
g
n
i
f
i
c
a
n
t
l
y
l
e
s
s
b
e
l
i
e
v
a
b
l
e
(
M
=
4
.
0
9
)
t
h
a
n
n
o
n
v
i
e
w
e
r
s
d
i
d
(
M
=
4
.
8
6
)
.
1
9
I
n
a
d
d
i
t
i
o
n
,
w
h
i
l
e
n
o
t
s
t
a
t
i
s
t
i
c
a
l
l
y
s
i
g
n
i
f
i
c
a
n
t
,
d
i
f
f
e
r
e
n
c
e
w
a
s
f
o
u
n
d
b
e
t
w
e
e
n
t
h
e
v
e
r
d
i
c
t
s
r
e
n
d
e
r
e
d
(
2
9
%
g
u
i
l
t
y
v
e
r
d
i
c
t
s
b
y
n
o
n
v
i
e
w
e
r
s
c
o
m
p
a
r
e
d
t
o
1
8
%
b
y
v
i
e
w
e
r
s
)
;
f
o
r
e
n
s
i
c
-
s
c
i
e
n
c
e
v
i
e
w
e
r
s
e
x
p
r
e
s
s
e
d
m
a
r
g
i
n
a
l
l
y
m
o
r
e
c
o
n
f
i
d
e
n
c
e
i
n
t
h
e
i
r
v
e
r
d
i
c
t
s
(
M
=
4
.
9
5
)
t
h
a
n
d
i
d
n
o
n
v
i
e
w
e
r
s
(
M
=
4
.
2
1
)
.
2
0
L
i
k
e
t
h
o
s
e
w
h
o
w
a
t
c
h
f
o
r
e
n
s
i
c
-
s
c
i
e
n
c
e
t
e
l
e
v
i
s
i
o
n
p
r
o
g
r
a
m
s
,
v
i
e
w
e
r
s
o
f
g
e
n
e
r
a
l
c
r
i
m
e
s
h
o
w
s
r
e
p
o
r
t
e
d
t
h
a
t
t
h
e
y
h
a
d
a
b
e
t
t
e
r
u
n
d
e
r
s
t
a
n
d
i
n
g
o
f
t
h
e
t
a
s
k
s
o
f
a
f
o
r
e
n
s
i
c
s
c
i
e
n
t
i
s
t
(
M
=
4
.
5
7
)
t
h
a
n
t
h
o
s
e
w
h
o
d
o
n
o
t
w
a
t
c
h
a
n
y
c
r
i
m
e
s
h
o
w
s
(
M
=
3
.
4
2
)
.
2
1
Y
e
t
t
h
o
s
e
w
h
o
r
e
g
u
l
a
r
l
y
v
i
e
w
g
e
n
e
r
a
l
c
r
i
m
e
p
r
o
g
r
a
m
s
d
i
d
1
6
.
W
e
t
r
u
s
t
t
h
a
t
m
o
s
t
J
u
r
i
m
e
t
r
i
c
s
r
e
a
d
e
r
s
a
r
e
s
u
f
f
i
c
i
e
n
t
l
y
f
a
m
i
l
i
a
r
w
i
t
h
i
n
f
e
r
e
n
t
i
a
l
s
t
a
t
i
s
t
i
c
s
t
h
a
t
t
h
e
y
u
n
d
e
r
s
t
a
n
d
t
h
e
r
e
s
u
l
t
s
w
e
r
e
p
o
r
t
i
n
t
h
e
m
a
r
g
i
n
.
F
o
r
t
h
o
s
e
r
e
a
d
e
r
s
w
h
o
d
e
s
i
r
e
m
o
r
e
d
e
t
a
i
l
e
d
e
x
p
l
a
n
a
t
i
o
n
s
,
w
e
r
e
c
o
m
m
e
n
d
t
h
e
f
o
l
l
o
w
i
n
g
t
e
x
t
s
:
W
I
L
L
I
A
M
L
E
E
H
A
Y
S
,
S
T
A
T
I
S
T
I
C
S
(
5
t
h
e
d
.
1
9
9
4
)
;
J
A
M
E
S
J
A
C
C
A
R
D
&
M
I
C
H
A
E
L
B
E
C
K
E
R
,
S
T
A
T
I
S
T
I
C
S
F
O
R
T
H
E
B
E
H
A
V
I
O
R
A
L
S
C
I
E
N
C
E
S
(
4
t
h
e
d
.
2
0
0
1
)
.
P
r
e
l
i
m
i
n
a
r
y
a
n
a
l
y
s
e
s
(
p
o
w
e
r
a
n
a
l
y
s
e
s
)
s
u
g
g
e
s
t
e
d
a
s
a
m
p
l
e
s
i
z
e
o
f
4
8
w
o
u
l
d
b
e
s
u
f
f
i
c
i
e
n
t
t
o
d
e
t
e
c
t
t
h
e
C
S
I
e
f
f
e
c
t
,
i
f
i
t
e
x
i
s
t
e
d
.
I
n
a
d
d
i
t
i
o
n
,
t
h
e
s
u
b
s
e
q
u
e
n
t
s
i
g
n
i
f
i
c
a
n
c
e
t
e
s
t
s
a
d
j
u
s
t
f
o
r
s
a
m
p
l
e
s
i
z
e
b
y
h
o
l
d
i
n
g
s
m
a
l
l
e
r
s
a
m
p
l
e
s
t
o
a
h
i
g
h
e
r
s
t
a
n
d
a
r
d
w
h
e
n
d
e
t
e
r
m
i
n
i
n
g
s
t
a
t
i
s
t
i
c
a
l
s
i
g
n
i
f
i
c
a
n
c
e
.
I
n
o
t
h
e
r
w
o
r
d
s
,
f
i
n
d
i
n
g
t
h
a
t
a
d
i
f
f
e
r
e
n
c
e
i
s
s
t
a
t
i
s
t
i
c
a
l
l
y
s
i
g
n
i
f
i
c
a
n
t
i
s
t
h
e
s
a
m
e
a
s
s
a
y
i
n
g
t
h
a
t
t
h
e
s
a
m
p
l
e
s
i
z
e
w
a
s
o
f
s
u
f
f
i
c
i
e
n
t
s
i
z
e
t
o
t
e
s
t
f
o
r
t
h
e
e
f
f
e
c
t
.
1
7
.
E
l
e
v
e
n
p
a
r
t
i
c
i
p
a
n
t
s
f
e
l
l
i
n
t
o
n
e
i
t
h
e
r
o
f
t
h
e
s
e
g
r
o
u
p
s
.
1
8
.
T
h
e
p
a
r
t
i
c
i
p
a
n
t
s
r
a
t
e
d
t
h
e
m
s
e
l
v
e
s
o
n
a
s
c
a
l
e
o
f
1
7
.
S
t
a
t
i
s
t
i
c
a
l
a
n
a
l
y
s
e
s
w
e
r
e
c
o
n
d
u
c
t
e
d
w
i
t
h
o
n
e
-
w
a
y
a
n
a
l
y
s
e
s
o
f
v
a
r
i
a
n
c
e
,
F
(
1
,
3
5
)
=
1
5
.
2
1
,
p
=
0
.
0
0
1
,
2
=
0
.
3
0
.
1
9
.
F
(
1
,
3
5
)
=
4
.
8
1
,
p
=
0
.
0
3
5
,
2
=
0
.
1
2
.
2
0
.
F
(
1
,
3
4
)
=
2
.
3
6
,
p
=
0
.
1
3
,
2
=
0
.
0
7
.
2
1
.
F
(
1
,
4
0
)
=
6
.
1
0
,
p
=
.
0
1
8
,
2
=
0
.
1
3
.
T
h
e
C
S
I
E
f
f
e
c
t
S
P
R
I
N
G
2
0
0
7
3
6
3
n
o
t
d
i
f
f
e
r
f
r
o
m
t
h
e
i
r
n
o
n
-
v
i
e
w
i
n
g
c
o
u
n
t
e
r
p
a
r
t
s
o
n
a
n
y
o
f
t
h
e
o
t
h
e
r
d
e
p
e
n
d
e
n
t
m
e
a
s
u
r
e
s
.
S
k
e
p
t
i
c
i
s
m
t
o
w
a
r
d
t
h
e
f
o
r
e
n
s
i
c
-
s
c
i
e
n
c
e
t
e
s
t
i
m
o
n
y
w
a
s
l
i
m
i
t
e
d
t
o
t
h
o
s
e
w
h
o
s
e
d
i
e
t
c
o
n
s
i
s
t
e
d
o
f
r
e
g
u
l
a
r
d
o
s
e
s
o
f
f
o
r
e
n
s
i
c
-
s
c
i
e
n
c
e
t
e
l
e
v
i
s
i
o
n
p
r
o
g
r
a
m
s
.
I
I
I
.
D
I
S
C
U
S
S
I
O
N
A
f
t
e
r
r
e
a
d
i
n
g
a
t
r
a
n
s
c
r
i
p
t
o
f
a
c
r
i
m
i
n
a
l
t
r
i
a
l
c
o
n
t
a
i
n
i
n
g
f
o
r
e
n
s
i
c
-
s
c
i
e
n
c
e
e
x
p
e
r
t
t
e
s
t
i
m
o
n
y
,
p
a
r
t
i
c
i
p
a
n
t
s
w
h
o
w
e
r
e
v
i
e
w
e
r
s
o
f
f
o
r
e
n
s
i
c
-
s
c
i
e
n
c
e
t
e
l
e
v
i
s
i
o
n
p
r
o
g
r
a
m
s
c
l
a
i
m
e
d
a
g
r
e
a
t
e
r
u
n
d
e
r
s
t
a
n
d
i
n
g
o
f
f
o
r
e
n
s
i
c
-
s
c
i
e
n
c
e
t
a
s
k
s
,
g
r
e
a
t
e
r
c
o
n
f
i
d
e
n
c
e
i
n
t
h
e
i
r
v
e
r
d
i
c
t
s
,
a
n
d
g
r
e
a
t
e
r
s
k
e
p
t
i
c
i
s
m
a
b
o
u
t
t
h
e
r
e
s
u
l
t
s
o
f
a
f
o
r
e
n
s
i
c
h
a
i
r
a
n
a
l
y
s
i
s
.
T
h
e
s
e
f
i
n
d
i
n
g
s
p
r
o
v
i
d
e
i
n
i
t
i
a
l
s
u
p
p
o
r
t
f
o
r
o
n
e
i
m
p
o
r
t
a
n
t
c
o
m
p
o
n
e
n
t
o
f
t
h
e
h
y
p
o
t
h
e
s
i
z
e
d
C
S
I
E
f
f
e
c
t
,
n
a
m
e
l
y
,
t
h
a
t
p
e
o
p
l
e
w
h
o
w
a
t
c
h
s
u
c
h
t
e
l
e
v
i
s
i
o
n
p
r
o
g
r
a
m
s
r
e
g
u
l
a
r
l
y
e
x
p
e
c
t
b
e
t
t
e
r
s
c
i
e
n
c
e
t
h
a
n
w
h
a
t
t
h
e
y
o
f
t
e
n
a
r
e
p
r
e
s
e
n
t
e
d
w
i
t
h
i
n
c
o
u
r
t
s
(
p
a
r
t
i
c
u
l
a
r
l
y
w
h
e
n
w
h
a
t
t
h
e
g
o
v
e
r
n
m
e
n
t
i
s
o
f
f
e
r
i
n
g
i
s
e
x
p
e
r
t
t
e
s
t
i
m
o
n
y
f
r
o
m
o
n
e
o
f
t
h
e
s
u
b
f
i
e
l
d
s
o
f
t
r
a
d
i
t
i
o
n
a
l
f
o
r
e
n
s
i
c
i
n
d
i
v
i
d
u
a
l
i
z
a
t
i
o
n
s
c
i
e
n
c
e
)
.
2
2
I
n
o
t
h
e
r
w
o
r
d
s
,
C
S
I
l
e
a
d
s
v
i
e
w
e
r
s
t
o
e
x
p
e
c
t
h
i
g
h
-
t
e
c
h
s
c
i
e
n
c
e
a
n
d
s
o
m
e
t
h
i
n
g
m
o
r
e
t
h
a
n
t
h
e
i
n
t
u
i
t
i
o
n
o
f
t
h
e
w
i
t
n
e
s
s
,
s
o
t
h
a
t
w
h
e
n
i
n
c
o
u
r
t
t
h
e
y
a
r
e
p
r
e
s
e
n
t
e
d
w
i
t
h
m
u
c
h
l
o
w
e
r
-
t
e
c
h
s
c
i
e
n
c
e
a
n
d
t
h
e
w
i
t
n
e
s
s
s
s
u
b
j
e
c
t
i
v
e
j
u
d
g
m
e
n
t
,
t
h
e
y
a
r
e
l
i
k
e
l
y
t
o
f
i
n
d
i
t
l
e
s
s
c
o
n
v
i
n
c
i
n
g
t
h
a
n
d
o
n
o
n
-
C
S
I
-
v
i
e
w
e
r
s
.
T
o
t
h
i
s
e
x
t
e
n
t
,
o
u
r
d
a
t
a
s
u
p
p
o
r
t
t
h
e
c
l
a
i
m
s
o
f
t
h
o
s
e
w
h
o
h
a
v
e
a
r
g
u
e
d
t
h
a
t
t
h
e
C
S
I
e
f
f
e
c
t
i
n
c
r
e
a
s
e
s
t
h
e
p
r
o
s
e
c
u
t
i
o
n
s
b
u
r
d
e
n
.
B
y
s
e
p
a
r
a
t
e
l
y
a
n
a
l
y
z
i
n
g
v
i
e
w
e
r
s
w
i
t
h
a
r
e
g
u
l
a
r
d
i
e
t
o
f
f
o
r
e
n
s
i
c
-
s
c
i
e
n
c
e
t
e
l
e
v
i
s
i
o
n
a
n
d
v
i
e
w
e
r
s
w
i
t
h
a
r
e
g
u
l
a
r
d
i
e
t
o
f
n
o
n
-
f
o
r
e
n
s
i
c
-
s
c
i
e
n
c
e
c
r
i
m
e
t
e
l
e
v
i
s
i
o
n
,
a
n
d
f
i
n
d
i
n
g
a
C
S
I
E
f
f
e
c
t
t
o
e
x
i
s
t
o
n
l
y
f
o
r
t
h
e
f
o
r
m
e
r
g
r
o
u
p
,
w
e
c
a
n
i
n
f
e
r
t
h
a
t
t
h
e
e
f
f
e
c
t
i
s
s
p
e
c
i
f
i
c
t
o
e
x
p
o
s
u
r
e
t
o
f
o
r
e
n
s
i
c
-
s
c
i
e
n
c
e
f
i
c
t
i
o
n
,
n
o
t
t
o
j
u
s
t
a
n
y
k
i
n
d
o
f
c
r
i
m
e
f
i
c
t
i
o
n
.
F
u
r
t
h
e
r
m
o
r
e
,
t
h
e
f
a
c
t
t
h
a
t
t
h
o
s
e
p
a
r
t
i
c
i
p
a
n
t
s
w
h
o
m
w
e
r
e
g
a
r
d
e
d
a
s
v
i
e
w
e
r
s
i
n
c
l
u
d
e
d
t
h
o
s
e
w
i
t
h
a
C
S
I
d
i
e
t
o
f
a
s
l
i
t
t
l
e
a
s
o
n
c
e
p
e
r
m
o
n
t
h
s
u
g
g
e
s
t
s
t
h
a
t
r
e
a
l
-
w
o
r
l
d
m
a
n
i
f
e
s
t
a
t
i
o
n
s
o
f
t
h
e
C
S
I
e
f
f
e
c
t
m
a
y
n
o
t
b
e
l
i
m
i
t
e
d
t
o
t
h
e
h
e
a
v
i
e
s
t
c
o
n
s
u
m
e
r
s
o
f
f
o
r
e
n
s
i
c
-
s
c
i
e
n
c
e
t
e
l
e
v
i
s
i
o
n
s
h
o
w
s
.
O
n
t
h
e
o
t
h
e
r
h
a
n
d
,
n
o
i
m
p
a
c
t
w
a
s
f
o
u
n
d
o
n
v
o
t
e
s
t
o
c
o
n
v
i
c
t
o
r
a
c
q
u
i
t
.
T
h
e
e
x
p
l
a
n
a
t
i
o
n
m
i
g
h
t
b
e
m
e
t
h
o
d
o
l
o
g
i
c
a
l
:
m
e
a
s
u
r
e
s
o
f
s
k
e
p
t
i
c
i
s
m
a
b
o
u
t
t
h
e
f
o
r
e
n
s
i
c
-
s
c
i
e
n
c
e
e
v
i
d
e
n
c
e
s
p
e
c
i
f
i
c
a
l
l
y
a
r
e
m
o
r
e
s
e
n
s
i
t
i
v
e
t
h
a
n
t
h
e
v
e
r
d
i
c
t
,
w
h
i
c
h
i
s
a
m
e
a
s
u
r
e
o
f
j
u
d
g
m
e
n
t
s
a
b
o
u
t
t
h
e
w
h
o
l
e
c
a
s
e
.
A
n
o
t
h
e
r
p
o
s
s
i
b
i
l
i
t
y
i
s
t
h
a
t
w
i
t
h
a
l
a
r
g
e
r
s
a
m
p
l
e
a
n
d
t
h
e
r
e
s
u
l
t
i
n
g
i
n
c
r
e
a
s
e
d
s
t
a
t
i
s
t
i
c
a
l
p
o
w
e
r
,
s
i
g
n
i
f
i
c
a
n
t
d
i
f
f
e
r
e
n
c
e
s
i
n
v
e
r
d
i
c
t
s
m
i
g
h
t
b
e
d
e
t
e
c
t
e
d
.
(
N
o
t
e
t
h
a
t
t
h
e
t
r
e
n
d
s
i
n
v
e
r
d
i
c
t
s
w
e
r
e
i
n
t
h
e
h
y
p
o
t
h
e
s
i
z
e
d
d
i
r
e
c
t
i
o
n
.
)
A
m
o
r
e
s
u
b
s
t
a
n
t
i
v
e
p
o
s
s
i
b
i
l
i
t
y
i
s
t
h
a
t
t
h
e
r
e
d
u
c
e
d
w
e
i
g
h
t
g
i
v
e
n
t
h
e
f
o
r
e
n
s
i
c
-
s
c
i
e
n
c
e
e
v
i
d
e
n
c
e
b
y
o
u
r
C
S
I
v
i
e
w
e
r
s
,
a
n
d
t
h
e
g
r
e
a
t
e
r
w
e
i
g
h
t
a
c
c
o
r
d
e
d
b
y
t
h
e
n
o
n
-
C
S
I
v
i
e
w
e
r
s
,
w
a
s
n
o
t
2
2
.
C
S
I
v
i
e
w
e
r
s
s
e
e
m
t
o
h
a
v
e
a
r
e
a
c
t
i
o
n
s
i
m
i
l
a
r
t
o
t
h
a
t
a
r
t
i
c
u
l
a
t
e
d
b
y
t
h
e
f
e
d
e
r
a
l
j
u
d
g
e
i
n
U
n
i
t
e
d
S
t
a
t
e
s
v
.
G
r
e
e
n
,
w
h
o
w
a
s
c
o
n
c
e
r
n
e
d
t
o
d
i
s
c
o
v
e
r
t
h
a
t
t
h
e
f
o
r
e
n
s
i
c
f
i
r
e
a
r
m
s
e
x
p
e
r
t
w
h
o
h
a
d
t
e
s
t
i
f
i
e
d
c
o
n
c
e
d
e
d
,
o
v
e
r
a
n
d
o
v
e
r
a
g
a
i
n
,
t
h
a
t
h
e
r
e
l
i
e
d
m
a
i
n
l
y
o
n
h
i
s
s
u
b
j
e
c
t
i
v
e
j
u
d
g
m
e
n
t
.
T
h
e
r
e
w
e
r
e
n
o
r
e
f
e
r
e
n
c
e
m
a
t
e
r
i
a
l
s
o
f
a
n
y
s
p
e
c
i
f
i
c
i
t
y
,
n
o
n
a
t
i
o
n
a
l
o
r
e
v
e
n
l
o
c
a
l
d
a
t
a
b
a
s
e
o
n
w
h
i
c
h
h
e
r
e
l
i
e
d
.
A
n
d
a
l
t
h
o
u
g
h
h
e
r
e
l
i
e
d
o
n
h
i
s
p
a
s
t
e
x
p
e
r
i
e
n
c
e
w
i
t
h
t
h
e
s
e
w
e
a
p
o
n
s
,
h
e
h
a
d
n
o
n
o
t
e
s
o
r
p
i
c
t
u
r
e
s
m
e
m
o
r
i
a
l
i
z
i
n
g
h
i
s
p
a
s
t
o
b
s
e
r
v
a
t
i
o
n
s
.
4
0
5
F
.
S
u
p
p
.
2
d
1
0
4
,
1
0
7
(
D
.
M
a
s
s
.
2
0
0
5
)
.
S
c
h
w
e
i
t
z
e
r
a
n
d
S
a
k
s
3
6
4
4
7
J
U
R
I
M
E
T
R
I
C
S
e
n
o
u
g
h
o
f
a
d
i
f
f
e
r
e
n
c
e
t
o
a
f
f
e
c
t
v
e
r
d
i
c
t
s
w
h
e
n
t
h
e
y
e
v
a
l
u
a
t
e
d
t
h
e
w
h
o
l
e
c
a
s
e
,
e
v
e
n
i
n
o
u
r
s
i
m
u
l
a
t
e
d
c
a
s
e
w
h
e
r
e
t
h
e
f
o
r
e
n
s
i
c
-
s
c
i
e
n
c
e
e
v
i
d
e
n
c
e
c
o
n
s
t
i
t
u
t
e
d
a
n
u
n
u
s
u
a
l
l
y
l
a
r
g
e
p
o
r
t
i
o
n
o
f
t
h
e
e
v
i
d
e
n
c
e
.
I
f
t
r
u
e
,
t
h
i
s
f
i
n
d
i
n
g
w
o
u
l
d
s
u
g
g
e
s
t
t
h
a
t
a
t
t
h
e
e
n
d
o
f
t
h
e
d
a
y
C
S
I
d
o
e
s
n
o
t
a
l
t
e
r
v
e
r
d
i
c
t
s
a
s
r
e
a
d
i
l
y
a
s
s
o
m
e
c
o
m
m
e
n
-
t
a
t
o
r
s
h
a
v
e
h
y
p
o
t
h
e
s
i
z
e
d
.
O
t
h
e
r
q
u
a
l
i
f
i
c
a
t
i
o
n
s
s
h
o
u
l
d
b
e
k
e
p
t
i
n
m
i
n
d
.
O
u
r
s
t
u
d
y
t
e
s
t
s
o
n
l
y
a
f
e
w
a
s
p
e
c
t
s
o
f
t
h
e
c
o
m
p
l
e
x
o
f
h
y
p
o
t
h
e
s
e
s
t
h
a
t
c
o
n
s
t
i
t
u
t
e
t
h
e
C
S
I
E
f
f
e
c
t
i
n
t
h
e
c
o
n
t
e
x
t
o
f
o
n
e
c
a
s
e
w
i
t
h
i
t
s
p
a
r
t
i
c
u
l
a
r
f
a
c
t
p
a
t
t
e
r
n
.
F
u
r
t
h
e
r
m
o
r
e
,
i
t
u
s
e
s
o
n
l
y
o
n
e
f
o
r
e
n
s
i
c
-
s
c
i
e
n
c
e
t
e
c
h
n
i
q
u
e
a
s
i
t
s
v
e
h
i
c
l
e
f
o
r
t
e
s
t
i
n
g
t
h
o
s
e
p
o
s
s
i
b
l
e
e
f
f
e
c
t
s
.
S
t
u
d
i
e
s
o
f
o
t
h
e
r
a
s
p
e
c
t
s
o
f
t
h
e
C
S
I
E
f
f
e
c
t
m
i
g
h
t
o
b
t
a
i
n
d
i
f
f
e
r
e
n
t
f
i
n
d
i
n
g
s
.
I
n
a
d
d
i
t
i
o
n
,
f
i
n
d
i
n
g
s
m
i
g
h
t
b
e
s
p
e
c
i
a
l
t
y
-
s
p
e
c
i
f
i
c
b
e
c
a
u
s
e
p
e
o
p
l
e
p
r
o
b
a
b
l
y
h
a
v
e
d
i
f
f
e
r
e
n
t
s
t
e
r
e
o
t
y
p
e
s
o
f
d
i
f
f
e
r
e
n
t
a
r
e
a
s
o
f
f
o
r
e
n
s
i
c
s
c
i
e
n
c
e
.
M
o
r
e
o
v
e
r
,
o
u
r
m
o
c
k
j
u
r
o
r
s
m
a
d
e
i
n
d
i
v
i
d
u
a
l
d
e
c
i
s
i
o
n
s
r
a
t
h
e
r
t
h
a
n
d
e
l
i
b
e
r
a
t
i
n
g
a
s
a
j
u
r
y
w
o
u
l
d
.
D
e
l
i
b
e
r
a
t
i
o
n
m
i
g
h
t
b
l
u
n
t
t
h
e
o
b
s
e
r
v
e
d
d
i
f
f
e
r
e
n
c
e
s
(
a
s
t
h
e
p
e
r
c
e
p
t
i
o
n
s
o
f
v
i
e
w
e
r
s
a
n
d
n
o
n
v
i
e
w
e
r
s
a
r
e
c
o
m
p
r
o
m
i
s
e
d
)
o
r
i
t
m
i
g
h
t
m
a
g
n
i
f
y
t
h
e
m
(
i
n
a
v
a
r
i
a
n
t
o
n
t
h
e
p
o
l
a
r
i
z
a
t
i
o
n
e
f
f
e
c
t
c
o
m
m
o
n
t
o
g
r
o
u
p
d
e
c
i
s
i
o
n
m
a
k
i
n
g
)
.
A
s
s
u
m
i
n
g
o
u
r
f
i
n
d
i
n
g
s
w
e
r
e
r
e
p
l
i
c
a
t
e
d
u
s
i
n
g
o
t
h
e
r
t
r
i
a
l
s
c
e
n
a
r
i
o
s
a
n
d
o
t
h
e
r
f
o
r
e
n
s
i
c
-
s
c
i
e
n
c
e
t
e
c
h
n
i
q
u
e
s
,
t
h
e
d
i
r
e
c
t
i
o
n
o
f
t
h
e
c
a
u
s
a
l
a
r
r
o
w
c
a
n
n
o
t
b
e
m
a
d
e
u
n
a
m
b
i
g
u
o
u
s
l
y
c
l
e
a
r
f
r
o
m
t
h
i
s
o
r
a
n
y
c
o
r
r
e
l
a
t
i
o
n
a
l
s
t
u
d
y
.
S
u
c
h
a
d
e
s
i
g
n
c
a
n
n
o
t
r
u
l
e
o
u
t
t
h
e
p
o
s
s
i
b
i
l
i
t
y
t
h
a
t
t
h
e
c
a
u
s
a
l
a
r
r
o
w
g
o
e
s
i
n
t
h
e
o
p
p
o
s
i
t
e
d
i
r
e
c
t
i
o
n
,
n
a
m
e
l
y
,
t
h
a
t
t
h
o
s
e
w
i
t
h
u
n
u
s
u
a
l
l
y
h
i
g
h
h
o
p
e
s
f
o
r
f
o
r
e
n
s
i
c
s
c
i
e
n
c
e
a
r
e
d
r
a
w
n
t
o
f
o
r
e
n
s
i
c
-
s
c
i
e
n
c
e
f
i
c
t
i
o
n
.
T
h
o
u
g
h
t
h
a
t
p
o
s
s
i
b
i
l
i
t
y
m
a
y
s
e
e
m
u
n
l
i
k
e
l
y
,
i
t
n
e
v
e
r
t
h
e
l
e
s
s
m
i
g
h
t
b
e
w
o
r
t
h
t
e
s
t
i
n
g
(
b
y
e
x
p
e
r
i
m
e
n
t
a
l
l
y
e
x
p
o
s
i
n
g
p
e
o
p
l
e
t
o
d
i
f
f
e
r
e
n
t
d
i
e
t
s
o
f
t
e
l
e
v
i
s
i
o
n
s
h
o
w
s
a
n
d
s
e
e
i
n
g
i
f
t
h
e
e
f
f
e
c
t
w
e
h
a
v
e
f
o
u
n
d
p
e
r
s
i
s
t
s
)
.
T
h
e
b
e
s
t
,
a
n
d
p
e
r
h
a
p
s
t
h
e
o
n
l
y
,
c
u
r
e
f
o
r
d
i
s
t
o
r
t
e
d
b
e
l
i
e
f
s
a
b
o
u
t
f
o
r
e
n
s
i
c
s
c
i
e
n
c
e
c
a
u
s
e
d
b
y
C
S
I
i
s
t
h
e
s
a
m
e
a
s
t
h
a
t
f
o
r
r
e
m
e
d
y
i
n
g
d
i
s
t
o
r
t
e
d
i
m
p
r
e
s
s
i
o
n
s
c
o
m
i
n
g
f
r
o
m
o
t
h
e
r
s
o
u
r
c
e
s
:
t
h
a
t
i
s
,
t
o
c
o
u
n
t
e
r
b
a
d
i
n
f
o
r
m
a
t
i
o
n
w
i
t
h
g
o
o
d
i
n
f
o
r
m
a
t
i
o
n
,
w
h
e
t
h
e
r
i
n
p
u
b
l
i
c
d
i
s
c
o
u
r
s
e
o
r
i
n
t
r
i
a
l
s
t
h
e
m
s
e
l
v
e
s
.
T
h
e
l
a
w
c
a
n
d
o
n
o
t
h
i
n
g
t
o
c
o
n
t
r
o
l
w
h
a
t
t
h
e
w
r
i
t
e
r
s
o
f
C
S
I
s
a
y
o
n
t
e
l
e
v
i
s
i
o
n
,
b
u
t
i
t
c
a
n
d
o
a
f
a
r
b
e
t
t
e
r
j
o
b
o
f
c
o
n
t
r
o
l
l
i
n
g
w
h
a
t
f
o
r
e
n
s
i
c
s
c
i
e
n
t
i
s
t
s
s
a
y
o
n
t
h
e
w
i
t
n
e
s
s
s
t
a
n
d
.
2
3
2
3
.
P
o
s
s
i
b
l
e
m
e
t
h
o
d
s
o
f
j
u
d
i
c
i
a
l
g
a
t
e
k
e
e
p
i
n
g
i
n
c
l
u
d
e
l
i
m
i
t
i
n
g
a
d
m
i
s
s
i
o
n
o
f
f
o
r
e
n
s
i
c
e
v
i
d
e
n
c
e
,
e
n
s
u
r
i
n
g
t
h
e
p
r
o
p
o
s
e
d
e
x
p
e
r
t
i
s
e
f
i
t
s
w
i
t
h
t
h
e
i
s
s
u
e
a
t
h
a
n
d
,
l
i
m
i
t
i
n
g
m
i
s
l
e
a
d
i
n
g
t
e
r
m
i
n
o
l
o
g
y
,
a
l
l
o
w
i
n
g
c
o
m
p
e
t
i
n
g
o
p
i
n
i
o
n
s
i
n
t
o
e
v
i
d
e
n
c
e
,
a
n
d
p
r
o
v
i
d
i
n
g
l
i
m
i
t
i
n
g
j
u
r
y
i
n
s
t
r
u
c
t
i
o
n
s
.
M
o
r
i
a
r
t
y
&
S
a
k
s
,
s
u
p
r
a
n
o
t
e
7
,
a
t
2
9
3
0
.

Reader Criminal Is Tics 2011-2012

Încărcat de

Informații document

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Reader Criminal Is Tics 2011-2012

Încărcat de

Drepturi de autor:

Formate disponibile

Reader Criminalistics 2011-2012

Multipurpose juror notebooks. Mock

One-quarter of the mock jurors

Dann, B.M., V.P. Hans, and D.H. Kaye,

the first molecular genetic forensic technique we take

Forensic Science Service,

Offence leading to >4 yr 2030 years after

S-ar putea să vă placă și