S t u d i e s in

Educational
Evaluation
Studies in Educational Evaluation 32 (2006) 83-99
ELSEVIER www.elsevier.com/stueduc

A SYSTEMATIC METHOD OF STATING LEARNING OBJECTIVES FOR

AN AMBITIO US SCIENCE PROGRAM: THE A P Q U A SCIENCE PROJECT

Samira El Boudamoussi*, Magda Medir*, Robert Gilabert** and Bonifacio Jim(~nez*

*Faculty of Education Sciences and Psychology, University Rovira i Virgili, Catalonia, Spain
**Department of Electrical and Mechanical Engineering, University Rovira i Virgili, Catalonia, Spain

Abstract

The educational goals and objectives provide a reference for curriculum designers,
teachers, and educators, and they are required for evaluation or accreditation
processes. This article presents a systematic method to state the learning objectives
and analyze the consistency of a secondary school science program developed in
Spain. The method is developed through seven steps, starting with data collection
and organization, followed by induction and categorization of the objectives, and
finishing with an internal consistency analysis between stated objectives and the
program's learning activities and an external consistency analysis between these
objectives and program designers and users' perceptions.

Introduction

Over the last three decades, tendencies in science teaching and in development and
implementation of new educational programs have led to an increasing interest in
evaluation research and, more specifically, in educational evaluation. Also, the
responsibilities and resources allocated by society to educators have increased the need for
accountability.

0191-491X/04/$ - see front matter © 2006 Published b y Elsevier Science Ltd.

doi: 10.1016(i.stueduc.2006.04.002
84 S. E1 Boudamoussi et aL/ Studies' in Educational Evaluation 32 (2006) 83-99

As part of evaluation processes, objectives generally need to be identified and stated

(Tymitz-Wolf, 1982) and they are often essential requirements for accreditation (Micek,
1979; Wittig, 1992). Goals and objectives are also considered key elements in designing
instructional modules (Hashim, 1999). Therefore, it is important for programs and
institutions to have clearly defined goals and objectives (Thier & Daviss, 2002; Wittig,
1992).
Defining goals and objectives is one of the main tasks that helps establish a
reference for designers and define the methods that will or will not be used to achieve those
goals (Thier & Daviss, 2002). It is also a reference for teachers to help them plan courses
and assess students' progress and performance (Bloom 1979; Stufflebeam & Webster,
2000; Tyler, 1998).
Unfortunately, the process of identifying outcomes or objectives involves many
problems and difficulties. As Micek (1979) mentions, the "specific intended outcomes of
programs are difficult to identify and agree upon". For Miceck, "even if a consensus is
reached", there is "a paucity of adequate data collection procedures" and even "in cases
where outcomes information has been obtained, major problems are often encountered in
the interpretation and use of the information".
Furthermore, in practice, most educational programs are based on contents that are
structured and developed in relation to a given discipline rather than a given goal. Also,
some of these programs are designed, developed, and field-tested with students.
The present study investigates the APQUA School Program 12-16, which is part of
the APQUA Project (Medir & Abell6, 1996). APQUA, the Catalan acronym for Learning
about Chemicals', their Uses and Applications, has been implemented in Catalonia since
1989 as the adaptation of SEPUP (Science Education for Public Understanding Program),
which started in 1981 at the Lawrence Hall of Science of the University of California at
Berkeley (Thier, 1985).

The APQUA Project

APQUA is an educational project that develops materials for the community and
schools with the financial support of industry as well as private and governmental
foundations. It has been widely diffused in Spain, reaching more than 3000 teachers and
150,000 pupils in about 900 schools, and more than 7,500 adults (APQUA, 2004).
The main goal of APQUA is to develop greater awareness, knowledge, and
understanding about chemicals and their interaction with people's lives. The project intends
to provide people with the knowledge and tools that help them learn to obtain information
about chemicals, to make effective individual decisions, and to participate actively as
members of a democratic society. APQUA, thus, also aims to promote the use of scientific
principles, processes, and evidence in public decision-making (Medir & Abell6, 1996).
Figure 1 shows the organization of the APQUA project, which consists of the
APQUA Community Program for adults, the APQUA School Program 10-12 for primary
schools and the APQUA School Program 12-16 for secondary schools.
 S. E1Boudamoussi et al./ Studies in Educational Evaluation 32 (2006) 83-99 85

ro ram&
APQUA PROJECT
./
PnblicPro
ram >

Figure 1: Structure of the APQUA Project

The APQUA School Program 12-16, which is the subject of this study, is made up
of modules. Each module contains several units, and every unit integrates several activities.
A list of modules is included in Table 1 (the ones selected for the present study are marked
with an asterisk). All modules deal with two main themes: risk and chemicals. They may
tackle different contents and different specific objectives, however, they all aim at
achieving the goals of the APQUA project (APQUA, 2003). In order to distinguish
between different levels of learning objectives, we will use in this study the words "goals"
for the project, "general objectives" for the programs of the project, "specific objectives"
for the modules of a program and "concrete objectives" for the units of a module.

Table 1: Modules of the APQUA School Program 12-16

Modules of the APQUA School Program 12-16 Abbreviation

Solutions and Pollution* DC

Contamination of Groundwater in Vallfrondosa* CV
Solid Waste Management* GR
Plastics in our Lives* PL
What is a Chemical? PQ
Risk Comparison RV
Hazardous Waste Treatment TRE
Toxicology VU

* Modules included in the sample

The modules' activities propose simulated situations and experiments to teach

concepts and procedures as well as attitudes, norms, and values. They also contain open-
ended questions to help teachers conduct discussions and enhance learners' reflection.
Students make decisions using the new knowledge and considering the tradeoffs involved
with each option (El Boudamoussi, Medir, Jimin~z, & Gilabert, 2001).
86 S. E1 Boudamoussi et al./ Studies in Educational Evaluation 32 (2006) 83-99

Although APQUA is designed and developed from clearly defined goals, the general
objectives of the APQUA School Program 12-16, the specific objectives of its modules and
the concrete objectives of the units were either partially stated or not stated as shown in
Figure 2. This is so because for adapting the SEPUP materials to the context of Catalan
society and schools, the APQUA Project team field-tests various versions of the APQUA
modules before a final one is edited and distributed. Therefore, the learning activities are
modified many times and the specific and concrete objectives initially targeted may
become incomplete, non-structured or roughly stated.

APQUA PROJECT _.~ Goals o f the project

completely stated

~ h o o l Program 12-16~
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

r--- Modules
"........................... Units
iI Module I . objectives of

1 Module2 [] [] [] partially

il
1
I
I :
Module3
i [] [] [] stated

,' ~ Specific objectives o f

I
i......... "~................. modules not stated
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

General objectives of the

program not stated

Figure 2: Diagnosis of Objectives in the APQUA School Program 12-16

Research Purpose

This study develops a systematic method to evaluate the APQUA School Program
12-16 by stating the learning objectives (of the units, the modules and the program) and
analyzing the internal and external consistency of the program. The stages of this method
are built up, step by step, following a systematic and structured process, which can be
described as follows:

Statement of the specific objectives of a representative sample of modules of the

APQUA School Program 12-16
Statement of the general objectives of the program based on the specific objectives
of a sample's modules
Analysis of the program's internal consistency making sure that the objectives stated
are actually linked to learning activities in the modules of the program.
Analysis of the program's external consistency by involving a sample of users and
developers in the process of stating the learning objectives.
 S. E1 Boudamoussi et aL/ Studies' in Educational Evaluation 32 (2006) 83-99 87

A detailed description of the method's development will be published in a future

article.

Program Evaluation

The main goal of program evaluation is to improve a new program or activity

(Anderson, 1994). However, there are as many definitions as authors and models for this
concept (Cronbach, 2000; Eisner, 2000; McDonald, 1989; Parlett & Hamilton, 1989;
Scriven, 2000; Stake, 2000; Stufflebeam & Webster, 2000; Tyler, 2000). An analysis
carried out by E1 Boudamoussi (2002) defines evaluation as a process of collecting
relevant, valid and reliable data, comparing it with established criteria to make evidence-
based decisions and, sometimes, a value judgment.
The method of stating learning objectives for the APQUA School Program 12-16
can be considered as an evaluation process since it takes into account all the aspects of the
above mentioned definition. The method is based on collecting, organizing, analyzing, and
restructuring data, and although it does not include comparing information with established
criteria, it uses an internal and external analysis of consistency.

Methodology

The research methodology followed consists of four steps stating the learning
objectives for the APQUA School Program 12-16:

Sample definition
Information gathering
Objectives statement
Internal and external consistency analyses

Sample Definition

The particular structure of the APQUA School Program 12-16, organized into
modules and units (Figure 2) has shaped, in general, the method developed and, in
particular, the sample selection.
Four of the eight existing modules were selected: Solutions and Pollution,
Contamination of Groundwater in Vallfrondosa, Solid Waste Management, and Plastics in
our Lives (Table 1). The criteria followed included the topic's relevance, the reflection of
the program's educational approach, and the non-similarity between learning activities in
the selected modules.
The modules excluded were: What is a Chemical?, Risk Comparison, Hazardous
Waste Treatment, and Toxicology.

Data Gathering

The data were basically obtained using the following procedure:

88 S. E1Boudamoussi et al./ Studies in Educational Evaluation 32 (2006) 83-99

. Analysis of documents provided by the APQUA project team such as teachers

guides, student guides, brochures, annual reports and other publications on the
project.
. Analysis of documents produced throughout this study. These documents presented
more and more structured information and became more and more useful as the
work progressed.

Statement o f Objectives

The process of stating the general objectives was the inverted path of the program
design. It started from listing the learning activities presented in the units of the modules.
Then, it identified the educational contents (knowledge, abilities, values, and attitudes)
tackled in these units and induced the concrete objectives of the units and the specific
objectives of the modules. Finally, it ended up at inducing the general objectives of the
program (Figure 3).

General
objective

Learning
activities
Concrete Specific
ontents objectives objectives

Statement of Objectives

Consistency Analysis

Figure 3: The Process of Stating Learning Objectives for the Program

The educational character of the program suggested arranging the educational

contents according to the classification of the secondary science curriculum of Catalonia
(Department of Education, 1993). In this educational district, where the APQUA program
is developed, three main types of contents are emphasized:

Facts, concepts and conceptual systems

Processes
Values, attitudes and norms

Consistency Analysis

The internal and external analyses of consistency were carried out using
specification tables (Roegiers, 2000) and expert moderation (Thier & Daviss, 2001),
respectively.
 ~5. L I H o u a a m o u s s t e t a l . / ~51uates m L a u c a t t o n a l Lvaluatton 3 Z (ZUUO) ~3-!~!~ 39

The s p e c i f i c a t i o n tables are g e n e r a l l y used in p r o g r a m d e s i g n to h i g h l i g h t the links

b e t w e e n the g o a l s and the learning activities. In this study, a s p e c i f i c a t i o n table was built
for e a c h m o d u l e o f the s a m p l e in o r d e r to c h e c k the c o h e r e n c e b e t w e e n e a c h s p e c i f i c
o b j e c t i v e stated for the m o d u l e and the c o r r e s p o n d i n g learning activities that led to stating
it. A n e x a m p l e o f a s p e c i f i c a t i o n table is s h o w n in Table 2.

Table 2: Example of a Specification Table

Concrete and common

Specific obiectives of units Educational contents Learning activities
objectives Concrete Concrete
objectives objectives
common to all
units
Concrete All educational All learning activities that
objective 1 contents that correspond to educational content
of unit 1 correspond to concrete 1.1.1 in unit 1
objective 1 of unit 1 All learning activities that
correspond to educational content
1.1.2 in unit 1
Etc.
Common Concrete All educational All learning activities that
objective 1 objective 1 contents that correspond to educational content
of unit 2 correspond to concrete 1.2.1 in unit 2
objective 1 of unit 2 All learning activities that
correspond to educational content
1.2.2 in unit 2
Etc.
Etc. Etc. Etc.
Specific Concrete All educational All learning activities that
objective Common objective 2 contents that correspond to educational content
1 objective 2 of unit 1 correspond to concrete 2.1.1 in unit 1
objective 2 of unit 1 All learning activities that
correspond to educational content
2.1.2 in unit 1
Etc.
Concrete All educational All learning activities that
objective 2 contents that correspond to educational content
of unit 2 correspond to concrete 2.2.1 in unit 2
objective 2 of unit 2 All learning activities that
correspond to educational content
2.2.2 in unit 2
Etc.
Etc. Etc. Etc.
90 S. E1 Boudamoussi et aL/ Studies' in Educational Evaluation 32 (2006) 83-99

A general table (see, for an example, Table 3), was built to show how the general
objectives of the program were generated from the specific objectives of the sample's
modules. This general table, together with Table 2, allowed checking the coherence
between every general objective and the learning activities.

Table 3: Example of a General Table of Consistency

General objectives Specific objectives of modules (S.O)

of the program
(G.O) Module 1 Module 2 Module 3 Module 4
G.O.I S.O I.l.1 S.O 1.2.1 S.O 1.3.1 S.O 1.4.1
S.O 1.1.2 S.O 1.2.2 S.O 1.3.2 S.O 1.4.2

Etc. Etc. Etc. Etc.

G.O. II S.O II.l.1 S.O II.2.1 S.O II.3.1 S.O II.4.1
S.O II.1.2 S.O II.2.2 S.O II.3.2 S.O II.4.2
Etc. Etc. Etc. Etc.

The expert moderation involved 20 secondary school teachers and 6 project team
members in the process of stating the objectives. The secondary school teachers were
selected according to their experience in using the program, in general, and the modules of
the sample, in particular, and the project team members according to their involvement in
the program's development, in general, and the modules of the sample, in particular. The
experts' opinions and suggestions were obtained through questionnaires and interviews.
Five questionnaires were designed: four of them were related to the specific objectives of
the modules and one was related to the general objectives of the program. The experts who
answered at least one questionnaire were interviewed.
A group of three judges intervened throughout the whole research process in order
to evaluate and seek consensus for each of the results obtained.

Results and Discussion

The method developed (shown in Figure 4) consists of seven stages, each of which
has several steps and generates various versions of the program's objectives following a
spiral process. The seven stages of the method include data collection; data organization;
categorization; internal consistency analysis (using specification tables); application and
transfer; generalization, and external consistency analysis (using expert moderation).
 S. E1 Boudamoussi et al./ Studies in Educational Evaluation 32 (2006) 83-99 91

Stage 1. Data Collection

Step 1.1. Select a sample of program's modules
Step 1.2. Read and analyze documents
Step 1.3. List learning activities for the units of the first module of ~-[ Learning activities 1
the sample (M 1)

Stage 2. Data Organization

I
r
I
r
, ........
[
* ......
Version2
~e;si;~i-
ii
I Educational contents I
Step 2.1. State contents for the units o f M 1 . . . . . . . . . . .

Step 2.2. Induce and state concrete objectives for the units o f M 1 I : * I
Step 2.3. Identify concrete objectives common to all units o f M 1 ...........i'"'"'i'""~ C . . . . . teand . . . . . . I I
I I t 1 objectives of units I
Step 2.4. Induce and state specific objectives for M 1 ~__4____1_ ...... i I
4,

i
Stage 3. Categorization
Step 3.1. Define the main categories of s?ecific objectives stated
for M 1 and assemble them into these categories
i I ~- .................... i
. . . . . . . . . .

! i Version : i I
Stage 4. Internal Consistency (of modules) • "...L ................................................. ~ I
L 2 __ "~ - -
Step 4.1. Encode the learning activities, the contents, the concrete
and common objectives of the units and specific
objectives o f M 1
Version3
i
Step 4.2. Build a specification table forM 1
•I .ff Version 4

Step 4.3. Analyze quantitatively and qualitatively the

specification table o f M 1
I
Step 4.4.Restate the specific objectives o f M 1 J

Stage 5. Application and Transfer

Step 5.1. Apply stages 1 and 2 and transfer relevant results to other
modules of the sample
Step 5.2. Transfer categories and encoding system defined for M 1
to other modules of the sample
Step 5.3. Apply sta~e 3 and sta~e 4 to other modules of the sample
4,
Stage 6. Generalization
Step 6.1. Identify specific objectives common to all the modules of
the sample
Step 6.2. Induce general objectives for the program from the
I ....... ,; .........
specific objectives of the sample's modules (Induction) /t I o ..... lobjecti.... f : ,
Step 6.3. Arrange induced objectives into synthesized objectives
(Synthesis) l,
Step 6.4. State general objectives for the program based on _Lie ............. -__j,
synthesized objectives (Version 1)
Step 6.5. Extrapolate stage 4 "Internal Consistency" to analyze the ,
general internal consistency of the program

Stage 7. External Consistency (using Expert Moderation)

Step 7.1. Define a sample of moderation experts
Step 7.2. Design and apply expert moderation instruments
Step 7.3. Analyze expert moderation results
Step 7.4. Restate tile specific objectives of tile sample's modules
and the general objectives of the program

F i g u r e 4: Method of Stating Learning Objectives for the APQUA School Program 12-16
92 S. E1 Boudamoussi et aL/ Studies' in Educational Evaluation 32 (2006) 83-99

The first stage, Data Collection (Stage 1 in Figure 4), includes three steps which
consist of selecting a representative sample of modules (indicated on Tablel), reading and
analyzing documents of the program, and listing the learning activities for the first sample's
module (M1). An example of learning activities is shown in Table 4.

Table 4: Examples o f Learning Activities Listed for Module M1

Examples of learning activities from module M 1

Teacher explains the importance of water as a solvent.
Students read the worksheet.
Teacher asks students whether commercial ammonia is an acid or base; the students answer.
Students who play the role of presenters in the simulation of a public meeting make their presentations.

The second stage, Data Organization (Stage 2 in Figure 4), consisting of four steps,
leads to stating the first version of the specific objectives for module M1 (Table 8). This
stage starts by identifying the educational contents tackled in the learning activities (an
example is shown in Table 5) and induces from them the concrete objectives of the units
(an example is shown in Table 6). Then, it identifies the concrete objectives common to all
units (an example is shown in Table 7) and, finally, based on the concrete and common
objectives, Stage 2 generates the first version of the specific objectives for module M1 (an
example is shown in Table 8).
The concrete objectives common to all units are very important to decide which of
the concrete objectives of the units will contribute to the statement of the specific
objectives of the module. However, the statement of a specific objective rather uses the
information provided in the concrete objectives of the units. As shown in Table 7, only part
of this information is present in the common objectives.
The first version of the educational contents and concrete and common objectives of
module M1 is generated at this Stage 2 (Figure 4).

Table 5: Example o f an Educational Content Resulting from a Set o f Learning Activities in

Module M1

Example of a set of learning activities Resulting educational

content
A student reads' the introduction in a worksheet
A student reads' the experimentalprocedure described in worksheet.
In groups o f two, students' carry out the experiments' described in the Reading and interpreting a
worksheet and they record their observations in a table. worksheet, and describing an
The students' design and carry out new tests using various' experimental procedure.
combinations o f water solutions.
 S. E1 Boudamoussi et aL/ Studies in Educational Evaluation 32 (2006) 83-99 93

Table 6: Example of a Concrete Objective Resulting from a Set of Educational Contents in

Module M1

Example of a set of contents Resulting concrete obiective

Solution, solvent and solute
Preparing solutions To build an operational
Preparing a diluted solution f?om a concentrated solution and vice versa definition o f solution, solvent
Preparing a saturated solution f?om a concentrated solution and so lute

Table 7: Example of a C o m m o n Objective Resulting from Recurring Concrete Objectives in

Module M1

Example of recurring concrete objectives Resulting common objective

Unit 1: To record experimental data obtained f?om observing the
experiments' o f dissolving diffbrent substances in water.
Unit 2: To observe and record quantitative and qualitative data about
the concentration and color o f every solution obtainedJ?om successive
dilutions' o f a colored solution with a known concentration. To observe and record
Unit 3: To observe and record, in a table, experimental data obtained experimental data.
f?om a systematic observation ofan indicator's' behavior with different
solutions and their mixtures'.
Unit 4: To observe and record, in a table, experimental data obtained
f?om observing the colors' o f an indicator in the solutions obtained
f?om successive dilutions ofan acid or base.
Unit 5: To observe the reproducibility o f experimental results' by
iterating the experiment ofneutralizing an acid or basic solution.
Unit 6: To observe and record the results' obtained f?om neutralizing
an acid "A" with a base "B" as well as' with ammonia.
Unit 6: To observe and record the results obtained f?om neutralizing a
base "B" with an acid "A" as' well as' with vinegar.
Unit 7: To observe and experience the fbrmation o f a salt when an acid
(or base) is' neutralized with a base (or an acid).

Table 8: Example of a Specific Objective of Module M1 Resulting from a Set of Concrete

Objectives o f Units

Example of a set of concrete objectives of units Resulting specific obiective of module M 1

To communicate the results' obtained f?om the
observation o f an indicator's' behavior with different
solutions and mixtures o f acids and bases.
To express and accept opinions on using dilution to
To communicate and discuss the results'
solve the problem o f water pollution with acids' or
obtained f?om using successive dilutions
bases.
and acid-base neutralization to solve the
To communicate and discuss the results of the universal problem o f water pollution with acids' or
indicator's, behavior with successive dilutions ofan acid bases.
or base.
To communicate and discuss the results' obtained f?om
the neutralization ofan acid or basic solution.
To express and accept opinions on using neutralization
to solve the problem o f water pollution with acids' or
bases.
94 S. E1 Boudamoussi et aL/ Studies' in Educational Evaluation 32 (2006) 83-99

The third stage, Categorization (Stage 3 in Figure 4), clusters the specific objectives
of module M1 into main categories (Table 9). This categorization leads to the second
version of the specific objectives for this module.

Table 9: Categoriesand Examples of Specific Objectives in Module M1

I. Concepts

E.g.: To build operational definitions of basic concepts related to solutions and neutralization.

II. Scientific methodology, its transfer and application to real life and daily issues

E.g.: To observe and register, in an organized and systematic way, quantitative and qualitative data
concerning the color and concentration of prepared solutions, or solutions obtained from successive
dilutions and from acid-base neutralization.

III. Awareness and responsibility

E.g.: To be aware of cleaning difficulties related to contaminated-natural-water-resource.

The fourth stage, Internal Consistency (Stage 4 in Figure 4), analyzes the
consistency between the specific objectives stated for module M1 and its learning
activities. A specification table, already illustrated in the methodology in Table 2,
establishes the links between each specific objective and the corresponding learning
activities. This stage results in a third version of the specific objectives of module M1.
The fifth stage, Application and TransJer (Stage 5 in Figure 4), applies the four
previous stages to the other sample's modules (M2, M3, and M4) and transfers some of the
relevant results to these modules. A list of the learning activities, and a first version of the
educational contents, concrete and common objectives and specific objectives of these
modules are obtained. The categories for the specific objectives are transferred from
module M1 as well as the form of the statements used to express the concrete and common
objectives and the specific objectives.
The sixth stage, Generalization (Stage 6 in Figure 4), states the general objectives of
the program based on the specific objectives of the sample's modules. It starts by
identifying the specific objectives that are similar or common to all the sample's modules
and induces from them a draft of induced general objectives (see statements inside the
circles on Figure 5). These are then assembled into categories of synthesized objectives
(see statements inside the rectangles on Figure 5) that lead to the statement of a first version
of the general objectives of the program.
 S. E1Boudamoussi et aL/ Studies' in Educational Evaluation 32 (2006) 83-99 95

< OGI.1 To build operational definitions of basic

concepts related to chemicals, their properties,
applications and their interactions with people
and the environment as well as the risk they
Dose for both.
OG.I.1 To build
operational definitions

+
~~I
G.II.I To i n t e r p ~
sheet and to carry out an ~
erimental p . . . . dure

/ OGSH~ To emphasize the accurate use of

~

I +
OG.II.6 To work / experimental material and respect for how it OG.II.1 To carry out
in a group \ sl~ould be used and maintained to carry out experimental
. . . . . iments I p . . . . d. . . .

~ ttitude towards the roles assumed OG.II.6 To work in a group /

\ within a group ~ /
~ and d eba t ~ o l g Svienti tic ~ J OG.II.4+ To develop

Q~ ~ ' i e ~ f i c I ...... ication abilities

+ 1 ~ T o b e c o ~
OG ".II :7 To. develop
. [ ~ f / of the problems g . . . . . ted by waste, the " ~ +
crltlcalthlnklng r / d ~ e s ° V I 2 d e c - ° n t a m i n ~ t ~ nd-the~need I OG.III.3 Tob . . . . . . . . . .
I ( for information I of the environmentalimpact
/ i ~~x, .OG\III.3 To b . . . . . . . . . . . fthe ] of human activities
~, . . . . . .
/~./U.ll./1o acquire a N . ~ environmentalimpact of human activities I
/ critical thinking ~ ~ n~
To observe and record i
+ / v _ ..L~. . . . . . . . N an organised and systematic way
I / evidence-based ~ . . . . ' I
OG.II.10 To make ~ decisions ",~uantltatlve and qualitative data I
evidence-based decisions ] OGII.11 To integrate information about ~ /
] different options in an evidence-l~ased \'.OG •II •3 To analyze
. and /
] decision making process / interpret experimental /
~ ~ data andgraphics /
\ OGIII.7 To emphasize the adwantag . . . . d / /
~ disadvantages, the import . . . . . fth . . . . . ic /
~ ' ~ c t . . . . d the.trade-o ffs. i. . . . . id . . . . . based/./
~ making p r o ~ ~

Figure 5: Assembling the Induced General Objectives into Synthesized Objectives

A synthesized objective consists of a short sentence that refers to a group of general

induced objectives but which does not provide all the information contained in these
objectives or in the specific objectives of the modules. The synthesized objectives turned
96 X E1 Boudamoussi et aL/ Studies' in Educational Evaluation 32 (2006) 83-99

out to be a helpful step in the procedure of stating the general objectives of the program.
They offered an efficient way of gaining an idea about how the general objectives would
look and served as a basis on which the statement of each general objective was built.
The second version of the general objectives is obtained after extrapolating, to the
program, the internal analysis of consistency (step 6.5) previously applied to the sample's
modules. This procedure, which correlates the general objectives stated in the first version
to the learning activities of the modules, via the specific objectives, has already been
illustrated in Tables 2 and 3.
Finally, the seventh and last stage, External consistency (using expert moderation)
(Stage 7 in Figure 4), considers the opinions of two groups of experts about the last stated
versions of the specific and general objectives. These experts are the users and developers
of the program. The qualification of the objectives by the experts from the two groups as
well as their comments and suggestions lead to stating the fourth and thus final version of
the specific objectives of the modules, and the third and thus last version of the general
objectives of the program.

Conclusions

We developed a systematic method of stating learning objectives for the APQUA

School Program 12-16. For its development, this method takes into account both the
modular structure of the program and the basic aspects of an evaluation process.
The method consists of seven stages, five of them applied to the individual modules,
one to the whole program, and one applied to both of them. The stages applied to the
individual modules are data collection, organization, and categorization as well as an
internal consistency analysis. A generalization stage is applied to all the sample's modules
and an external consistency analysis is applied to both the program and the sample's
modules.
Several versions of specific objectives of the modules and general objectives of the
program are generated throughout the method. The definitive versions are the ones
obtained from the external consistency analysis after applying the expert moderation steps.
The method produces four versions of the specific objectives of the modules and
three versions of the general objectives of the program. The final version always results
from the external analysis of consistency.
The systematic method of stating learning objectives is mainly based on the modular
structure of the APQUA School Program 12-16 and the possibility of inducing its general
objectives from the specific objectives of a sample's modules. Thus, the method could be
applied to any program that has a modular structure with educational contents (concepts,
skills and attitudes) repeated in more than one module.

References

Anderson, G. (1994). Fundamentals' of educational research. [3rd Ed.] Basingstoke:Falmer.

APQUA (2003). Proyecto APQUA. Aprendizqje de los Productos Quimicos, sus Usos y
Aplicaciones [APQUAproject. Learningabout chemicals,their uses and applications]. UniversitatRovira i
Virgili.
 S. E1 Boudamoussi et aL/ Studies' in Educational Evaluation 32 (2006) 83-99 97

APQUA (2004). Proyecto APQUA. Injbrme 2004 [APQUA project. Report 2004]. Tarragona.
Departamento de Ingenieria Quimica. Universitat Rovira i Virgili.

Bloom, B.S. (1979). Taxonomia de los ol)jetivos de la educaci6n. Clasificaci6n de las Metas
Educativas [Taxonomy of educational objectives. The classification of educational goals]. (Acarreta
Arnedo, I., Trans. 3rd ed.). Alcoy : Marfil.

Cronbach, L.J. (2000). Course improvement through evaluation. In G.F. Madaus, M.S. Scriven, &
D.L. Stufflebeam (Eds.), Evaluation models'. Viewpoints' on educational and human services evaluation
(13th ed., pp. 101-115). Boston: Kluwer-Nijhoff.

Department of Education of Catalonia (1993). Curriculum. Educaci6 Secundhria Obligat6ria. ~irea

de ci~ncies experimentals [Curriculum. Compulsory secondary education. Section of natural sciences].
Barcelona,: Generalitat de Catalunya.

Eisner, E.W. (2000). Educational connoisseurship and criticism: their form and functions in
educational evaluation, In G.F. Madaus, M.S. Scriven, & D.L. Stufflebeam (Eds.). Evaluation models'.
Viewpoints' on educational and human services evaluation (13th ed., pp. 335-347). Boston: Kluwer-
Nijhoft:

E1 Boudamoussi, S., Medir, M., Jimin6z, B., Gilabert, R.M. (2001). Estudio comparativo entre el
modelo te6rico de un programa educativo de ciencias y su desarrollo en el aula [Curriculum. Compulsory
secondary education. Section of natural sciences]. VI Congreso Intemacional sobre Investigaci6n en la
Didfictica de las Ciencias: Retos de la Ensefianza de las Ciencias en el Siglo XXI. V 1, (pp. 229-230).
Barcelona.

E1 Boudamoussi, S. (2002). Evaluation des ol)jectijg, du Programme APQUA Scolaire 12-16.

Formulation et analyse de coherence [Evaluation of the objectives of the APQUA School Program 12-16.
Statement and coherence analysis]. Unpublished Ph.D. thesis, Tarragona: Universitat Rovira i Virgili.
Retrieved from: <http://www.tdx.cesca.es/TESIS URV/AVAILABLE/TDX-0122103-132206>

Hashim, Y. (1999). Are instructional design elements being used in module writing? British Journal
o f Educational Technology. 30 (4), 341-358.

McDonald, B. (1989). La evaluaci6n y el control de la educaci6n [Evaluation and the control of

education]. In S.J. Gimeno & G. Per6z La enseYmnza. Su teoria y su practica [Teaching. Its' theory and
practice] (3rd ed., pp. 467-478). Madrid: Akal.

Medir, M., & Abell6, M. (1996). APQUA: A program on sciences' and societal issues. Proceedings
of the 2nd scientific conference on: "The Future of Science and Mathematics Teaching and the Needs of
Arab Society" (pp 500-506). Tunis: Arab Development Institute.

Micek S.S. (1979). Identifying, measuring and evaluating educational outcomes. North Central
Association Quarterly, 53 (4), 408-419.

Parlett, M., & Hamilton, D. (1989). La evaluacidn como iluminacidn [Evaluation as enlightenment].
In S.J. Gimeno & G.A. P6rez La ense~anza. Su teoriay supractica [Teaching. Its theory and practice] (pp.
450-466). Madrid: Akal.
98 S. E1 Boudamoussi et aL/ Studies' in Educational Evaluation 32 (2006) 83-99

Roegiers, X. (2000). Une p~dagogie de l'int~gration. Comp~tences et integration des acquis dans
l'enseignement [A pedagogy of integration. Competences and integration of knowledge in teaching].
Brussels: De Boeck Universit6.

Scriven, M.S. (2000). Evaluation ideologies. In G.F. Madaus, M.S. Scriven, & D.L. Stufflebeam
(Eds.). Evaluation models'. Viewpoints' on educational and human services evaluation (13th ed., pp. 229-
260). Boston: Kluwer-NijhofI:

Stake, R.E. (2000). Program evaluation, particularly responsive evaluation. In G.F. Madaus, M.S.
Scriven, & D.L. Stufflebeam (Eds.). Evaluation models'. Viewpoints' on educational and human services'
evaluation (13th ed., pp. 287-310,) Boston: Kluwer-NijhofI:

Stufflebeam, D.L., & Webster, W. J. (2000). An analysis of alternative approaches to evaluation, In

G.F. Madaus, M.S. Scriven, & D.L. Stufflebeam (Eds.). Evaluation models'. Viewpoints' on educational
and human services' evaluation (13th ed., pp. 23-43). Boston: Kluwer-Nijhoff.

Thier, Herbert D. (1985). Societal issues and concerns: A new emphasis for science education.
Science and Education, 69 (20), 155-162.

Thier, H.D., & Daviss, B. (2001). Developing inquiry-based science materials'. Guide jbr educators'.
A project for the Lawrence Hall of Science Center for Curriculum Innovation. New York: Teachers College
Press.

Tyler, R.W. (1998). Principios bdsicos del curriculo [Basic principles of the curriculum]. Buenos
Aires: Troquel.

Tyler, R.W. (2000). A rationale for program evaluation. In G.F. Madaus, M.S. Scriven, & D.L.
Stufflebeam (Eds.). Evaluation models'. Viewpoints' on educational and human services evaluation (13th
ed., pp. 67-78). Boston: Kluwer-Nijhoff.

Tymitz-Wolf, B. (1982). Guidelines for assessing individualized education program goals and
objectives. Teaching Exceptional Children, 14 (5), 198-201.

wittig, G.R. (1992). Making use of goals and objectives for internal program evaluation. Journal oj
Education jb r Library and Injbrmatio n Science. 33 (2), 129-140.

Acknowledgements

To the memory of the late Dr. Bonifacio Jim6nez who made a great contribution to this study. The
authors would moreover like to thank the APQUA project's members, the SEPUP members, and the
teachers who answered the questionnaires and participated to the interviews. They also thank Dr. Herbert
Thief and Marlene Thier for their many and relevant suggestions to an earlier version of the paper.

The A u t h o r s

S A M I R A EL B O U D A M O U S S I is a postdoc researcher at the F a c u l t y o f E d u c a t i o n a n d

P s y c h o l o g y o f the U n i v e r s i t y R o v i r a i Virgili in Tarragona, Spain. She is a c h e m i c a l
 S. E1 Boudamoussi et aL/ Studies' in Educational Evaluation 32 (2006) 83-99 99

engineer from the National School of Mines and Industry at Rabat (Morocco) and gained a
Ph.D. with the European Label of the University Rovira i Virgili.

MAGDA MEDIR is a professor at the Department of Chemical Engineering, member of

the School of Chemical Engineering and of the Faculty of Education and Psychology,
University Rovira i Virgili at Tarragona, Catalonia, Spain. She has been the Director, since
1988, of the APQUA Project "Learning about Chemicals, its Uses and Applications" of the
Chemical Engineering Department of the University Rovira i Virgili, and scientific
collaborator in the SEPUP Project (Science Education for Public Understanding Program)
of the Lawrence Hall of Science, University of California at Berkeley, USA. Her main field
of research is the development and assessment of science and technology educational
programs for the schools and the community.

ROBERT M. GILABERT is a professor at the Department of Mechanical Engineering,

member of the School of Chemical Engineering of the University Rovira i Virgili at
Tarragona, Catalonia, Spain. He is also a member and scientific collaborator of the
APQUA Project "Learning about Chemicals, its Uses and Applications", and a researcher
in the assessment of science and technology educational programs for the schools and the
community.

BONIFACIO JIMENEZ was a professor at the Department of Pedagogy of the Faculty of

Education and Psychology, University Rovira i Virgili at Tarragona, Catalonia, Spain.
Author and co-author of various books and publications on educational evaluation;
curriculum design, development, and evaluation; and educational innovation. Dr. Bonifacio
Jim6nez died in August 2003.

Correspondence: <seboudam@urv.net>