Program Evaluation: FRIT 8435

Fall 2009

Meribeth Fell, Melissa Merritt, Tammy Story

Introduction

The Center for Online Learning at Georgia Southern University was

developed to “manage, enhance, and develop strategies and technologies

for the online delivery of courses”

(http://academics.georgiasouthern.edu/col/about.php). One of the ways they

attempt to fulfill their mission is by offering course design training for online

faculty. However, the increase in student demand for more online courses

has arrived at a time when the current state budget issues make all

programs and departments susceptible to review or termination. During FY

2008-2009, all University System of Georgia colleges and universities were

required to reduce their general operating budgets by at least 10%. This

year, all schools have already made at least a 6% cut and have recently

been asked to make another 2% (totaling 8%), but Regent James Bishop

anticipates an ultimate 10-12% cut for FY 2009-2010. But Bishop also worries

that the projected number of incoming college students will increase by

100,000 in the next ten years. The current budget issues have led the

university administration to examine all service-oriented departments and

programs. Additionally, they are looking more closely at how to serve their

increasing student body more efficiently and more effectively.

 Online learning addresses both issues – budget reduction issues and

increasing student population – but if online learning options are increased

without paying proper attention to support and training of faculty, the online

programs could ultimately fail. Although faculty who have experienced

training through the COL as well as students who have participated in online

classes offered by trained faculty claim that these classes “are better than

some of the other classes,” anecdotal evidence is not sufficient evidence for

continued program support.

In an effort to provide more substantial evidence of the benefits of

participating in COL training, a more formal evaluation is required. This

evaluation will attempt to address the following questions:

• Question 1: Is the COL’s faculty development program in alignment

with the program’s goal to “manage, enhance, and develop strategies
and technologies for the online delivery of courses?”
• Question 2: Is the COL program providing faculty with adequate
support services related to teaching online?
• Question 3: Is the COL providing sufficient training for faculty who use
technology to teach?
• Question 4: Is the COL program providing faculty with adequate
equipment, software and communications for interactions with
students, institutions and other faculty?
• Question 5: Has the COL program been effective in training faculty to
teach online?
• Question 6: Has the COL program helped to the change the way faculty
think about online instruction in general?

These questions were addressed using a variety of qualitative and

quantitative techniques (see Section ? “Brief Overview of the Evaluation Plan

and Procedures”).
 Many of Georgia Southern University’s online offerings are graduate

level classes taught by highly experienced professors, and the ability of the

students, partially because they are at a graduate level but also because of

the more prestigious nature of a large university versus a small state or two-

year college, greatly increases student perception and student performance

of online learning. In spite of these possible limitations, this evaluation

should be an effective tool for the primary purposes of this evaluation.

The target audience for this evaluation is the COL staff, university

administrators, faculty, staff, and students as well as constituents of the

University System of Georgia, and the purpose of this evaluation is to

evaluate continued feasibility of the COL training program and the current

training needs in an effort to identify potential problem areas as online

learning grows in popularity and in numbers.

Brief overview of Evaluation Plans and Procedures

Because the Center for Online Learning exists to provide support and
training for faculty, the information for the evaluation came from surveying
faculty members and students through questionnaires to determine their
feelings about the Center for Online Learning. We also performed case
studies to ascertain data about the effectiveness of the Center for Online
Learning.

Presentation of evaluation results

A. Summary of evaluation findings

 In this study, 100 faculty members and 100 students at Georgia
Southern University were randomly sampled and surveyed and studied to
determine whether or not the Center for Online Learning has met its
objectives.

The following evaluation questions/ criteria /standards were used and

the results follow the questions.

1. Is the COL’s faculty development program in alignment with the

program’s goal to “manage, enhance, and develop strategies and
technologies for the online delivery of courses?”

Findings: 85% of faculty agreed that the standard is being met.

2. Is the COL program providing faculty with adequate support

services related to teaching online?

Findings: 80% of faculty agreed that the standard is being met.

3. Is the COL providing sufficient training for faculty who use

technology to teach?

Findings: 80% of faculty agreed that the standard is being met.

4. Is the COL program providing faculty with adequate equipment,

software, and communications for interactions with students,
institutions, and other faculty?

Findings: 95% of faculty agreed that this standard is being met.

 5. Has the COL program been effective in training faculty to teach
online?

Findings: 85% of students agreed that this standard is being met.

6. Has the COL program helped to change the way faculty think about
online instruction in general?

Findings: 80% of faculty agreed that this standard is being met.

B. Interpretation of Evaluation Findings.

The data indicates that the Center for Online Learning is meeting the
objectives of the Principles of Good Practice. With each question, the
percentage of those surveyed who answered favorably about the program
was commensurate to the standard we put forth in our evaluation plan.

Conclusions and recommendations

Based upon the data shared in the evaluation program report the following
was concluded:

A. Criteria and standards used to judge evaluation object

The criteria and the standards utilized to evaluate the College of Online
Learning [COL] is based upon the NETS for Technology Facilitators and
Leaders which states, “To meet the growing need for highly qualified
educational technologists, ISTE has worked with the National Council for
Accreditation of Teacher Education (NCATE) to develop a set of performance
assessment standards for initial and advanced endorsements in the areas of
Technology Facilitation and Technology Leadership.” (ISTE, 1997 - 2009)
 Based upon the summary of evaluation findings in order to discover if the
College of Online Learning has met the criteria and the standards developed
by the NETS for Technology Facilitators and Leaders.

Therefore the following was concluded:

80% of faculty surveyed agreed that the COL has complied with the NETS criteria,
TL-VIII Leadership and Vision: “Educational technology leaders will facilitate
development of a shared vision for comprehensive integration of technology and
foster an environment and culture conducive to the realization of the vision.”
(ISTE, 1997 - 2009)

80% of faculty surveyed agreed that the COL has complied with the NETS criteria,
TL-VII Procedures, Policies, Planning, and Budgeting for Technology Environments:
“Educational technology leaders coordinate development and direct
implementation or technology infrastructure procedures, policies, plans and
budgets.” (ISTE, 1997 - 2009)

80% of faculty surveyed agreed that the COL has complied with the NETS criteria,
TL-III Teaching, Learning, and the Curriculum: “Educational technology leaders
model, design, and disseminate plans that include methods and strategies for
applying technology to maximize student learning.” (ISTE, 1997 - 2009)

80% of faculty surveyed agreed that the COL has complied with the NETS criteria,
TF-VII: Procedures, Policies, Planning, and Budgeting for Technology Environments:
“Educational technology facilitators promote the development and implementation
of technology infrastructure, procedures, policies, plans, and budgets.” (ISTE, 1997
- 2009)

70% or 75% students surveyed agreed that the COL has complied with the criteria,
TF-I Technology Operations and Concepts: “Educational technology facilitators
demonstrate an in-depth understanding of technology operations and concepts.”
(ISTE, 1997 - 2009)

80% of faculty surveyed agreed that the COL has complied with the NETS criteria,
TF-I Technology Operations and Concepts: “Educational technology facilitators
demonstrate an in-depth understanding of technology operations and concepts.”
(ISTE, 1997 - 2009)

B. Judgments about evaluation objects (strengths and weakness)

 Based upon the summary of evaluation findings, the following
strengths and weaknesses were concluded about the College of Online
Learning’s program:

Program Strengths:

• The capability to “manage, enhance, and develop strategies and

technologies” for the online delivery of courses at Georgia
Southern University.

• The capacity to provide faculty with adequate support services to

aid in the area of teaching online.

• The knowledge and the skills to provide sufficient training for

faculty who use technology to teach.

• The ability to provide faculty with adequate equipment, software,

and communications in order to interact with students,
institutions, and other faculty.

• The capacity, knowledge and skills to provide effective training

for faculty to teach online.

Program Weakness:

• Reduction in operating budgets but an increase in student

population and the popularity of online learning.

• Technical issues such as the LAN or WAN network having

technical difficulties.

• Small staff

C. Recommendations
After reviewing the overall data in the evaluation report the following
recommendations are suggested:

• Research grants to cover the reduction cost of the operating

budget.

• Increase staff size by enlisting the help of volunteers who are

certified and have training in the area of online learning and tech
support.
 APPENDICES

Checklist for Determining When to Conduct and Evaluation

Faculty Survey – Part A

Faculty Survey-Part B
Evaluation Questions/Criteria/Standards and Findings

Evaluation Plan
Checklist for Determining When to Conduct and Evaluation

[Steps one through ten] Based on Figure 10.1 [186]

1. Is there a legal requirement to evaluate the COL? No,
there isn't a legal requirement, but the timing is good
because the COL is undergoing some philosophical
changes in splitting IT from ID (Instructional design).
Prior to these changes, IT and ID were housed under the
same umbrella COL and worked collaboratively to ensure
the development and support for Georgia Southern's
online classes, including the GOML programs offered
through Georgia Southern.
2. Does the object of the evaluation, which is the COL, have
enough impact or importance to warrant formal
evaluation? Yes, the COL impacts the learning of students
by providing support for faculty teaching online.
3. Is there sufficient agreement among the stakeholders on
the model for the program? Its goals and objectives? Yes,
there seems to be agreement among the stakeholders on
the model for the program, in spite of some differences in
the "how to" aspect of the program. This evaluation is
part of a formative evaluation effort to either continue
the program as is or to make changes to the program
based on this evaluation. There is also some interest in
attracting more faculty to the COL services.
4. If the program has started, are the actions of the COL
consistent with the program’s model? Is the achievement
of the COL’s goals feasible? Yes, to date the actions of
the COL are consistent with the program's goal to "to
manage, enhance, and develop strategies and
 technologies for the online delivery of courses"
(http://academics.georgiasouthern.edu/col/). Some
actions of the program include an online training
development course designed to place faculty in the role
of student and work on enhancing online course
strategies. Other actions of the COL include various
online resources on technology use and online course
design.
5. Is the proposed evaluation of the COL feasible given
existing human and fiscal resources and data availability?
Yes, the proposed evaluation of the COL is feasible. Our
role as evaluators would be a Cost free approach -
formative recommendations that improve productivity
and product quality. Current data will be available
through surveys for all stakeholders --
faculty/participants and program administrators.
6. Do current stakeholders agree on the intended use of the
evaluation? Yes, all stakeholders view this evaluation as a
formative evaluation that will help improve learning
through improving faculty skills in the online
environment.
7. Are current stakeholders in a position to use the
information productively? Yes, most stakeholders in the
COL program have direct input into the COL program and
work collectively to provide services.
8. Will the decisions of stakeholders be made exclusively on
other bases and be uninfluenced by the evaluation data?
No, the client has agreed to allow us to perform this
evaluation for formative purposes and plans to use the
data. However, political and financial influences always
play some role in program evaluation.
9. Is it likely that the evaluation will provide dependable
 information? Yes, the data collected will be from
participants in the COL program and from the program
administrators, and with proper evaluation standards, the
evaluation data will be dependable.
10. Is the evaluation of the COL likely to meet
acceptable standards of propriety? Yes, the "evaluation
will be conducted legally, ethically, and with due regard
for the welfare of those involved in the evaluation as well
as those affected by its results" (p. 446).

Based on steps 1 – 10, an evaluation should be conducted because

of the following reasons:
1. To evaluate if the COL’s faculty development program is align
with the program's goal to "to manage, enhance, and develop
strategies and technologies for the online delivery of courses"
(http://academics.georgiasouthern.edu/col/).

2. To evaluate if the COL program is providing faculty with

adequate support services related to teaching online.
http://www.ecinitiatives.org/publications/principles.asp

3. To evaluate if the COL is providing sufficient training for

faculty who use technology to teach.
http://www.ecinitiatives.org/publications/principles.asp

4. To evaluate if the COL program is providing faculty with

adequate equipment, software and communications for
interaction with students, institutions and other faculty.
http://www.ecinitiatives.org/publications/principles.asp
 Faculty Survey – Part A
Professional Area of Instruction:

Number of years in profession:

Is the COL’s faculty development program in alignment with the

program’s goal to “manage, enhance, and develop strategies and
technologies for the online delivery of courses?”

Yes

No

If no, please explain.

Is the COL program providing faculty with adequate support

services related to teaching online?

Yes

No

If no, please explain.

Is the COL providing sufficient training for faculty who use

technology to teach?

Yes

No

If no, please explain.

Is the COL program providing faculty with adequate equipment,

software, and communications for interactions with students,
institutions, and other faculty?
Yes

No

If no, please explain.

Has the COL program been effective in training faculty to teach

online?

Yes

No

If no, please explain.

Has the COL program helped to change the way faculty think about
online instruction in general?

Yes

No

If no, please explain.

 Faculty Survey – Part B
Professional Area of Instruction:

Number of years in profession:

The following survey questions are being asked to discover if the

College of Online Learning has complied with the NETS criteria for
Technology Facilitators and Leaders which states, “To meet the growing
need for highly qualified educational technologists, ISTE has worked with
the National Council for Accreditation of Teacher Education (NCATE) to
develop a set of performance assessment standards for initial and
advanced endorsements in the areas of Technology Facilitation and
Technology Leadership.” (ISTE, 1997 - 2009)
Do you agree that the COL has complied with the NETS criteria, TL-VIII
Leadership and Vision: “Educational technology leaders will facilitate
development of a shared vision for comprehensive integration of
technology and foster an environment and culture conducive to the
realization of the vision.”
(ISTE, 1997 - 2009)

Yes

No

If no, please explain.

Do you agree that the COL has complied with the NETS criteria, TL-VII
Procedures, Policies, Planning, and Budgeting for Technology
Environments: “Educational technology leaders coordinate development
and direct implementation or technology infrastructure procedures,
policies, plans and budgets.” (ISTE, 1997 - 2009)
Yes

No

If no, please explain.

Do you agree that the COL has complied with the NETS criteria, TL-III
Teaching, Learning, and the Curriculum: “Educational technology leaders
model, design, and disseminate plans that include methods and strategies
for applying technology to maximize student learning.” (ISTE, 1997 - 2009)
Yes

No

If no, please explain.

Do you agree that the COL has complied with the NETS criteria, TF-VII:
Procedures, Policies, Planning, and Budgeting for Technology
Environments: “Educational technology facilitators promote the
development and implementation of technology infrastructure,
procedures, policies, plans, and budgets.” (ISTE, 1997 - 2009)
Yes

No

If no, please explain.

Do you agree that the COL has complied with the NETS criteria, TF-I
Technology Operations and Concepts: “Educational technology facilitators
demonstrate an in-depth understanding of technology operations and
concepts.” (ISTE, 1997 - 2009)
Yes

No

If no, please explain.

Evaluation Questions/Criteria/Standards/Findings

Question 1: Is the COL’s faculty development program in align with the

program’s goal to “manage, enhance, and develop strategies and
technologies for the online delivery of courses?”
Criteria 1: TL-VIII Leadership and Vision: “Educational technology leaders
will facilitate development of a shared vision for comprehensive
integration of technology and foster an environment and culture
conducive to the realization of the vision.”
(ISTE, 1997 - 2009)
Standard: 80% of faculty surveyed agrees that the COL complies with the
criteria.

Question 2: Is the COL program providing faculty with adequate support

services related to teaching online?
Criteria 2: TL-VII Procedures, Policies, Planning, and Budgeting for
Technology Environments: “Educational technology leaders coordinate
development and direct implementation or technology infrastructure
procedures, policies, plans and budgets.” (ISTE, 1997 - 2009)
Standard: 80% of faculty surveyed agrees that the COL complies with the
criteria.

Question 3: Is the COL providing sufficient training for faculty who use
technology to teach?
Criteria 3: TL-III Teaching, Learning, and the Curriculum: “Educational
technology leaders model, design, and disseminate plans that include
methods and strategies for applying technology to maximize student
learning.” (ISTE, 1997 - 2009)
Standard: 80% of faculty surveyed agrees that the COL complies with the
criteria.

Question 4: Is the COL program providing faculty with adequate

equipment, software and communications for interactions with students,
institutions and other faculty?
Criteria 4: TF-VII: Procedures, Policies, Planning, and Budgeting for
Technology Environments: “Educational technology facilitators promote
the development and implementation of technology infrastructure,
procedures, policies, plans, and budgets.” (ISTE, 1997 - 2009)
Standard: 80% of faculty surveyed agrees that the COL complies with the
criteria.

Question 5: Has the COL program been effective in training faculty to

teach online?
Criteria 5: TF-I Technology Operations and Concepts: “Educational
technology facilitators demonstrate an in-depth understanding of
technology operations and concepts.” (ISTE, 1997 - 2009)
Standard: 70% or 75% students surveyed agree that the COL complies
with the criteria.

Question 6: Has the COL program helped to the change the way faculty
think about online instruction in general?
Criteria 6: TF-I Technology Operations and Concepts: “Educational
technology facilitators demonstrate an in-depth understanding of
technology operations and concepts.” (ISTE, 1997 - 2009)
Standard: 80% of faculty surveyed agrees that the COL complies with the
criteria.

Criteria is based upon the NETS for Technology Facilitators and Leaders
which states, “To meet the growing need for highly qualified educational
technologists, ISTE has worked with the National Council for Accreditation
of Teacher Education (NCATE) to develop a set of performance assessment
standards for initial and advanced endorsements in the areas of
Technology Facilitation and Technology Leadership.” (ISTE, 1997 - 2009)
 Evaluation Plan
Evaluation Information Design Information Method for
Question Required Source Collecting
Information
Question 1: Is the COL’s Programs Descriptiv Program Interviews;
faculty development offered; e, participant case
program in alignment with targeted Cross- s (faculty); studies;
the program’s goal to participants sectional program survey
“manage, enhance, and ; common staff;
develop strategies and strategies faculty
technologies for the online and supervisors
delivery of courses?” technologie
s;
Question 2: Is the COL Hours Descriptiv Program Interviews;
program providing faculty available; e, participant case
with adequate support services Cross- s (faculty); studies;
services related to teaching available; sectional, program survey
online? number of Case staff;
staff / staff study faculty
members supervisors
available;
number of
“contacts”
by faculty;
satisfaction
of faculty

Question 3: Is the COL Available Descriptiv Program Interviews;

providing sufficient training training; e, participant case
for faculty who use “expertise” Cross- s (faculty); studies;
technology to teach? of faculty of sectional program survey
all faculty staff;
who use faculty
technology supervisors
to teach;
faculty
satisfaction
Question 4: Is the COL Equipment, Descriptiv Program Interviews;
program providing faculty software e, participant case
with adequate equipment, and Cross- s (faculty); studies;
software and communicat sectional program survey
communications for ions staff;
interactions with students, available; faculty
institutions and other equipment, supervisors
faculty? software ; students,
and inter-
communicat institution
ions contacts;
available at
comparable
institutions;
faculty,
students,
and inter-
institution
satisfaction
;
Question 5: Has the COL Faculty Descriptiv Program Interviews;
program been effective in satisfaction e, participant case
training faculty to teach ; faculty Cross- s (faculty); studies;
online? supervisor sectional program survey
satisfaction staff;
(post- faculty
training); supervisors
student ; students
evaluations who take
(pre- courses
training) / offered by
(post- participatin
training); g faculty
course
design
evaluation
(by
supervisor
or
instructiona
l designer –
if available
pre and
post
training)
Question 6: Has the COL Pre-existing Descriptiv Program Interviews;
program helped to the survey data e, participant case
change the way faculty if available; Case- s (faculty); studies;
think about online post- study, survey
instruction in general? training Cross-
survey sectional,
data; Time
pre/post Series
review of
online
courses
taught by
same
faculty;

Sampling Information- Schedule Analysis Procedures

Collection
Procedures
Program Information Data should be collected Summarize the data
coordinator, collected via from previous collected from the
instructional an online participants participants in order to
technologist survey tool immediately. analyze quantitative and
/ designer, (surveymon qualitative information.
Information key); Data from case study
Systems group of faculty Data will be analyzed by
Coordinator, Interviews; participants, evaluators; each
Systems case studies supervisors, and evaluator will analyze a
Support of faculty students planning to specific portion of the
Specialist III; who go enroll in courses created data collected from the
through the by faculty enrolled in the case study group and will
Cohort of online faculty development meet together to discuss
faculty who training workshop should be their individual findings;
volunteered workshop. collected before evaluators will next
to training, at the end of summarize the overall
participate in the training (not data collected from case
training students), and at the study group.
programs end of the first semester
offered by of teaching post-
the COL. training.

Students in Week 1: Distribute

programs surveys, initiate contact
who plan to with case study
enroll in participants,
courses
offered by Week 2-4: Observe and
faculty post- interview,
training.
Week 5: Survey, and
All available follow-up with case
students, study participants and
faculty, and others through survey –
other
participants Week 6: Data Analysis
who = approximately 6
previously weeks from beginning to
participated end
in or were
affected by
faculty
training from
the COL.

Reporting Procedures
Interpretatio Audience(s) Content Format Schedule
n
Procedures
“The criteria Program Help answer Technical report to COL Meetings to
and coordinator, the to discuss results; discuss
standards instructional question: several meetings with results –
developed technologist Are the project administrators January;
will serve as / designer, criteria and and deliverers to release
a guide to Information standards discuss their report-
the Systems being met interpretation of results mid-
interpretatio Coordinator, according and possible changes; February
ns.” (273) Systems the NETS? meeting with clients to
Support report results and
Evaluators Specialist receive their input
will utilize III;
the NETS for
Technology University
Facilitators President;
and Leaders Deans
to serve as a (Academic
guide to the Affairs,
interpretatio schools);
n of data. Chairs;
faculty;
students;
other
institutions;
Foundation
members ($
providers);
grant
providers;
USG GOML
constituents
.