AD A003 260 Technical Paper 260 SUBJECTIVE INTERPRETATION OF RELIABILITY AND ACCURACY SCALES FOR EVALUATING MILITARY INTELLIGENCE Michaol G. Samet ‘SYSTEMS INTEGRATION & COMMAND/CONTROL TECHNICAL AREA. U.S. Army Research Institute for the Behavioral and Social Sciences January 1975 Aporoved for pubic release distribution unlimited,U. S. ARMY RESEARCH INSTITUTE FOR THE BEHAVIORAL AND SOCIAL SCIENCES A Field Operating Agency under the Jurisdiction of the Deputy Chief of Staff for Personnel R.A. ROOTH J. E, UHLANER COL, GS Technical Director Commander nomices, DISTRIBUTION: Primary distribution of this report hes bean made by ARI, Please sdress correspondence ‘oncaming distribution of reports to: U. S. Army Research lastitute forthe Behaviors and Social Sciences, ATTN: PERL, 1300 Witton Boulevard, Arlington, Virginie 22209, the U.S. Army Research Instituta forthe Behavioral and Socal Scincer. NOTE: The sinsigs in tht raport are not to bo construed os en officiel Donartmant ofthe Army position, | | | EINAL DISPOSITION: This report may be destroyed when itn lager needed, Please donot return it to | ‘Unless 50 designoted by other authorized documents. | i | | | |Unclassified SccORTY GLASTIEATION OF TuIs PAGE (Wim Date Bnered REPORT DOCUMENTATION PAGE S| [Reronr waster ESGVT RECERTGN WO] EHEIPTEN'E GNTALOG MOWBER Technical Paper 260 TRE fad Bs ore OF REPORT PuRE COVERED SUBJECTIVE INTERPRETATION OF RELTABILITY AND Interin ACCURACY SCATES FOR EVALUATING MILITARY INTELLIGENCE, RRR 7 CORTRIET OR GRANT NOWBERTT Michael 6. Samet U.S. Army Research Institute for the Behavioral and Social Sciences 2o1sa101a754 1300 Wilson Blvd., Arlington, VA 22209 January 1975 U.S. Army Training & Doctrine Command, Ft Monroe, |; yauser sp mser va. I TE WSRTTSRING NSRHEY WAWE © ROOWESSIT TCTaAT Tm CaneaTng OUTEAY | VB SEEURIVY CLAGE, TT TORY Unclassified ERT IOW STATEMENT ToT BIG Ray Approved for public releases distribution unlimited. FEY WORDT (Casas an vas aah W nacatiay Bd TTT iy BE BOSD reliability information processing accuracy Tactical Operations System (0S) scales quantitative intelligence computer, ‘This report is part of a broader effort to research human performance in required judgnental tasks and to determine ways of improving euch performance, particularly as it affects data input to tacticel intelligence information Processing. The research reported here assessed the adequacy of scurce- reltability and information-accuracy rating scales for conveying intelligence information. DD Sans M73 0ITIOW oF tnovas is oBsoLeTe ee, A SeCURTY ECARIFTEATION OF Tas PER (Rian Dos BohraUnclassified SECURITY CLASSIFICATION OF THIS PAGEOIRaH Bate Brim 20. Intelligence officers completed an original set and a replication of Paper and pencil tasks which measured their attitudes toward and knowledge about the scales; recorded their judgments ae to which report in each of 100 pairings of reports with different joint accuracy and reliability ratings was more likely to be true; asked them to estinate the probability that a report with a specific reliability rating would also carry a specific accuracy rating, and vice versa} and had them assign numerical values representing the Probable truth of reports with given levels of reliability, of accuracy, and reliability-acouracy combinations. About one-fourth of the subjects treated reliability and accuracy as independent dimensions; the majority treated reliability as highly correlated with accuracy, and their judgment of a report's truth was influenced more strongly by ite accuracy rating. Numerical (probabilistic) interpretations of scale levels were consistent within individuals but varied widely between then. Development of a new scale is suggested. The scale should require the assignment of a quantitative value which would reflect the likelihood of a report's being true and be based on all available information including the reliability of the source.Technical Paper 260 AD SUBJECTIVE INTERPRETATION OF RELIABILITY AND ACCURACY SCALES FOR EVALUATING MILITARY INTELLIGENCE Michael G. Samet Robert S. Andrews, Supervisory Project Director SYSTEMS INTEGRATION & COMMAND/CONTROL TECHNICAL AREA (Cecil D. Johnson, Chief Submitted By: Approved By: Joseph Zeidner, Director JE. Uhlaner ORGANIZATIONS & SYSTEMS TECHNICAL DIRECTOR RESEARCH LABORATORY U.S. ARMY RESEARCH INSTITUTE FOR THE BEHAVIORAL AND SOCIAL SCIENCES Office, Deputy Chief of Staff for Personnel Dopertment of the Army 1900 Wilson Boulevard, Arlington, Virginia 22209 January 1978 SUE NSraee eaireeeeere Army Number Intelligence Systems ‘20162101754 ‘Aborove for public release: sibution united,ARI Research Reports and Technical Papers are intended for sponsors of R&D tasks and other research and military agencies. Any findings ready for implementation at the time of publication are presented in the latter part of the Brief. Upon completion of a major phase of the task, formal recommen- dations for official action normally are conveyed to appropriate military agencies by briefing or Disposition Form. hAwFOREWORD The Intelligence Systems Work Unit Area within the U.S. Army Research Institute for the Behavioral and Social Sciences (ARI) is concerned with problems of advancing and exploiting manicomputer technology for improved tactical intelligence informetion processing. A mejor objective is to determine basic capabilities and limitations of man as an information processor ‘and to devise complementary and compensatory processing aids and techniques for these capabilities and limitations. A specific requirement under this objective ie to provide research findings whereby human performance in required judgmental tasks ean be enhanced, The entire research effort is responsive to requirements of RDTE Project 20162101A754, "Intelligence Information Processing,” FY 1974 Work Program and to special requiramants of the U.S, Army Training and Doctrine Command and the Project Menaser’s Office, Army Tactical Data Systems, ‘The U.S. Army currently is developing systems intended to provide computer-based support for command and staff functions on the battlefield (e.g., TOS). The effectiveness of such systems for intelligence functions will be determined in part by the characterstis of the input data, The present publication describes one effort which confirmed the need for improved evaluation procedures for intelligence data and identified the direction of follow-on efforts most likely to satisfy this need nue Cilboury J. CUHLANER Technical DirectorSUBJECTIVE INTERPRETATION OF RELIABILITY AND ACCURACY SCALES. FOR EVALUATING MILITARY INTELLIGENCE BRIEF Requirement: To assess the adequacy of and the relationship between the reliability scale (levels A-F} and accuracy scale (levels 1-8) by determining the quantitative meaning attached by intelligence officers to the various rating levels of each scale, Procedure’ ‘Thirty-seven intelligonos officers completed an original and a replication of paper and pencil tasks constructed to measure their interpretation of the reliability and accuracy scales. “Tho tasks measured officers’ attitudes toward and thelr knowledge about the scales; recorded ‘heir judgments as to which report in each of 100 pairings of reports with different joint sccuracy and reliability ratings was more likely to be true; asked them to estimate the probability thet @ report carrying a specific reliabiity-of-source rating would also carry a Specific. accurscy-of information rating, and vice versa; had them assign numerical values ‘opresenting the probable truth of reports with given levels of reliability, given lovels of ‘accuracy, and given relisbility-eccuracy combinations. Findings Approximately one-fourth of the subjects treated reliability and accuracy as independent dimensions; the other threefourths of the subjects trested the reliability reting as highly correlated with the accuracy rating, A subject's estimate of a report's truth. was influenced much mare by its accuracy rating then by its liability rating, Numerical (Le., probabilistic) Interpretations of scales were relatively consistent within individuals, but such interpretations varied widely between them, Group attitude toward the scales as measured on 2 6-point continuum, which ranged from “very adequate” to “very inadequate,” produced a mean rating of only “slightly adequate." Utilization of Findings: “The findings point out several inadequacies in use of the reliability and accuracy scales ‘that appear to result from their qualitative nature and from frequent interdependence. The basis is provided for research to design and properly validate @ new, less ambiguous, moro sensitive system for communicating evaluations of intelligence data. One approach derived from the findings could require that a report have a single quantitative value assigned which reflects its likelihood of being tue. This value would be based on all avaliable information including the ‘empirical reliability of the source. vy