Running head: THE EVALUATION PROCESS 1

The Evaluation Process

Carlos R. Herrera

January 09, 2017

CUR/528 Assessment of Learning

University of Phoenix

Professor Martha Kennedy

 THE EVALUATION PROCESS 2

The Evaluation Process

Part of the blame for the terrorist attacks our nation suffered during September 11, 2001,

rested on the perceived inability of the intelligence community to analyze the collected

intelligence to alert government officials of the threat. One of the agencies responsible for

reporting such intelligence is the United States Air Force (USAF), and the training they provide

to both enlisted and officers servicemen. The purpose of this paper is to describe the process

used to evaluate the results of a needs assessment of the USAF officers training course 14NX,

conducted by the Air Education and Training Command (AETC), at Goodfellow AFB, San

Angelo, Texas.

Program Description

The USAF Intelligence Officer's Need Assessment was prepared as a systematic approach

to identifying and recommending areas for improvement of intelligence officer's skills. These

skills taught in the USAF Intelligence Officer Course 14NX served as the basis for the training

these officers receive throughout their careers. During the development of the assessment, the

evaluators prepared a series of questions for the Intelligence Officers to determine how

successful was the training in the professional development of the officers, and their primary

trade skills of analysis and interpretation.

Description of the Problem

The USAF Intelligence Officers Need Assessment and the evaluation of the results were

prepared to evaluate the training received by Intelligence Officers and to provide data for further

evaluation of the training programs and officers professional development. After the 9/11

attacks, public opinion, and that of officials in Congress perceived that there was a deficiency in
 THE EVALUATION PROCESS 3

the way the collected intelligence was analyzed and processed. The resulting assessment was

developed as a request by the USAF to be part of a comprehensive effort to examine what are the

shortcomings of their Intelligence programs.

Purpose of the Evaluation

The purpose of the of the evaluation was to graphically present the findings of the USAF

Intelligence Officers Need Assessment. The assessment consisted of asking Intelligence

officers a series of questions on their impression of the course in improving their skills and of the

contribution of the course learning objectives towards that goal. The assessment questions were

divided into three areas that exemplify officers grade and the intelligence professionals and

includes; Integrity, Managerial, and Trade or work skills. The officers Career Field Education

and Training Plan (CFETP), served as the primary document for the development of the

assessment. The CFETP provides Career Field Managers (CFM), training managers, supervisors,

and the officers with the guidance and learning objectives to monitor officer's learning

progression (USAF, 2013). In addition to the CFETP, the evaluators also examined the Course

Lesson Plans and other course control documents including, presentations, tests, quizzes and

other training materials. The purpose was to examine all of the documents used to provide the

education to the Intelligence Officers. By following a detailed and comprehensive approach to

the evaluation, the evaluators ensure that process remains true to the customer requirements and

goals.

Data Collection Procedures

To evaluate the results of the needs assessment the evaluation team collected data using

interviews and surveys from intelligence officers at various locations in the continental United
 THE EVALUATION PROCESS 4

States, as well as at overseas locations. The evaluators chose to interview officers in different

grades, as these attended the Intelligence Course in the last seven years. One of the sampled units

included the 71st Intelligence Squadron (IS), from Wright-Patterson Air Force Base in Dayton

Ohio. The evaluators administered the survey to 25 officers in the grades of First Lieutenant to

Major. Before starting the evaluation, the evaluation team met with Colonel James F. Stafford

the Wing Commander for the 71st IS, to present the group and to discuss the purpose of the

assessment. When finished the evaluation, the team met with the 71st Training Managers, and

with their assistance, they coordinated the schedule details for the evaluation. To keep

disruptions to the daily operations to a minimum, the evaluators were assigned to one of three

work shifts to administer the survey and to collect the data. Once collected, the data was

documented using the format requested by the USAF, of a Microsoft Excel spreadsheet, and a

Microsoft Word document which became the summary of the findings, both documents

included as attachments to this paper.

Procedures for Analyzing the Data

Once collected the evaluators used the 29 questions and learning objectives and assigned a

value of one or zero to distinguish between a yes and a no answer. The Officers were asked to

evaluate if the training received helped them in achieving the provided training objectives and

goals. Once gathered, the data was transferred to the spreadsheet, providing an immediate

feedback of how many students answered each particular question, the average of a yes response

and the overall average of a response that leads to indicate that there is a need for additional

reinforcement or training. The findings were presented to the requesting agency, the Air

Education and Training Command (AETC). The report to AETC stressed that the sample of 25
 THE EVALUATION PROCESS 5

officers assessment and evaluation, although it showed a deficiency in training in those officers

evaluated, should not be used as an indicator of similar deficiencies of the entire workforce.

Results/Findings

The results of the evaluation showed that the students responded to having a poor

understanding of the intelligence tradecraft. Of those responding only 12% of the students (three

students) out of 25 who completed the survey said that the course learning objectives were clear.

They agree that the course was successful in providing them with the knowledge they needed to

become better Intelligence Analysts and Officers. A low score was also observed in areas such as

improvement of analytical skills, and a sense that the course did little to improve their

productivity. Table 1, highlights the findings of the evaluation process in the areas of Quality of

Work; Work Consistency; Collection Management; Targeting Process; and Professional

Tradecraft (Skills).

Table 1. Evaluation Highlights

Training Objective and Findings

Goals
Improve leadership skills16
Promote Integrity
Collection Management
Targeting Process
Professional Tradecraft
44% of the individuals evaluated improved their leadership
skills.
56% of the students said that the course inspires them to
display more integrity as professionals.
40% of the students, demonstrated a better understanding of
the collection management process.
32% of those evaluated, increased their knowledge of
targeting procedures.
12% of the students found that the course improved their
professional skills and knowledge as intelligence
professionals.
 THE EVALUATION PROCESS 6

The Needs Assessment document along with the evaluation findings and the associated

evaluation documents form part of a group of studies that AETC requested to evaluate the

intelligence training programs.

Conclusion

The Needs Assessment Evaluation served as a tool to present AETC with the results of the

assessment. During the evaluation 25 Intelligence officers from the 71 Intelligence Squadron at

Wright-Patterson Ohio, answered a to a series of statements based on the goals and learning

objectives of the Intelligence Course 14NX. These were divided into three broad areas that

included Managerial, Officers Integrity, and Professional Skills. Once the data was transferred to

a Microsoft Excel spreadsheet, the evaluators observed that overall, the students had a poor

understanding of the intelligence tradecraft as well as those areas related to analytical skills and a

lack of confidence that taking the course did little to improve their overall productivity as

Intelligence Officers.
 THE EVALUATION PROCESS 7

Reference
USAF. (2013). AFSC 14NX: Intelligence officer. Retrieved from http://static.e-

publishing.af.mil/production/1/af_a2/publication/cfetp14nx/cfetp14nx.pdf