Documente Academic
Documente Profesional
Documente Cultură
To cite this Article Janssen, Rianne and Crauwels, Marion(2011) 'Content and student factors in mastering environmental
studies - nature in primary education: evidence from a national assessment in Flanders (Belgium)', Journal of Biological
Education, 45: 1, 20 28
To link to this Article: DOI: 10.1080/00219266.2011.537836
URL: http://dx.doi.org/10.1080/00219266.2011.537836
The publisher does not give any warranty express or implied or make any representation that the contents
will be complete or accurate or up to date. The accuracy of any instructions, formulae and drug doses
should be independently verified with primary sources. The publisher shall not be liable for any loss,
actions, claims, proceedings, demand or costs or damages whatsoever or howsoever caused arising directly
or indirectly in connection with or arising out of the use of this material.
20 JOURNAL OF BIOLOGICAL EDUCATION, VOLUME 45, NUMBER 1, MARCH 2011
Research paper
Content and student factors in
mastering environmental studies
nature in primary education: evidence
from a national assessment in
Flanders (Belgium)
Rianne Janssena and Marion Crauwelsb
a
Center for Educational Effectiveness and Evaluation and Research Group Quantitative Methods and
Individual Differences, University of Leuven, Leuven, Belgium; bDepartment of Biology, University of
Downloaded By: [informa internal users] At: 15:44 25 February 2011
A large-scale paper-and-pencil assessment of the attainment targets of environmental studies with a focus on the
subject area nature was held in primary education in Flanders (Belgium). The tests on different subfields of nature,
i.e. the human body, healthcare, organisms, ecosystems, environmental care and non-living nature, were adminis-
tered to a representative sample of 4556 pupils of the sixth grade from 145 different schools. The percentage of
students mastering the attainment targets differed clearly across the tested subfields. Moreover, gender differences
were found. Multilevel analyses revealed that 13% of the total variance of the performance on an overall scale could
be attributed to differences between schools. For 82%, these school differences were accounted for by factors
mainly referring to the language spoken at home and to the socio-economic background of the pupils. Differences
among pupils within schools were explained only to a small extent.
Keywords: large-scale assessment; environmental studies; nature; primary education; multilevel analyses
Introduction
Since 1998, the Flemish Parliament has issued attain- tion about the actual performance of Flemish students
ment targets for primary education (for pupils from 6 was only available through participation in international
to 12 years) and secondary education (for students studies, like Trends in International Mathematics and
from 12 to 18 years). Attainment targets are minimum Science Study (TIMSS) and Programme for Interna-
educational objectives with regard to knowledge, tional Student Assessment (PISA), but these studies are
insight, skills and attitudes, which the society and the neither specifically directed at the Flemish curriculum,
educational authorities regard as necessary and attain- nor at the population of children leaving a certain level
able at the end of a certain level of education. of education. A second purpose of the national assess-
As part of a system of quality control of Flemish ments is to investigate whether there are systematic
education, the Flemish ministry of education started a differences among schools in test performance, and
programme of national assessments of educational whether characteristics at the level of students, class-
progress in 2002. The purpose of these national assess- rooms, or schools co-vary with these differences.
ments is twofold. First, an assessment has to give infor- It is the purpose of the present paper to report the
mation about the percentage of pupils that actually reach results of a Flemish large-scale assessment on environ-
the different attainment targets of the domain that is mental studies nature in primary education. Usually,
being tested. Before the national assessments, informa- mathematics and languages receive more attention in
Corresponding author: M. Crauwels, Department of Biology, University of Leuven (Belgium), Naamsestraat 61 (PB 2102),
B-3000 Leuven, Belgium. Email marion.crauwels@bio.kuleuven.be, tel 32 16 32 42 90, fax 32 16 32 42 58
Journal of Biological Education ISSN 00219266 print/ISSN 21576009 online 2011 Society of Biology
http://www.informaworld.com
DOI: 10.1080/00219266.2011.537836
RESEARCH PAPER 21
Levelb
Attainment targetsa K C A
Downloaded By: [informa internal users] At: 15:44 25 February 2011
MAN
Human body: structure and functioning, sexuality and development
6. The pupils are able to describe the function of the most important organs involved in mans
life processes and the function of the senses, the skeleton and the muscles.
7. The pupils are able to recognise physical changes which they observe in themselves and
their peers as normal aspects of their development.
Health care
14. The pupils are able to relate healthy habits to what they know about the functions of their
own body and are aware that certain symptoms of disease or handicaps cannot always be
avoided.
15. The pupils are aware that taking precautions can reduce or exclude the chance of infectious
diseases, parasites and fungal infections.c
NATURE AROUND US
Organisms
1. The pupils are able to discover similarities and differences in a limited collection of humans,
animals and plants, and are able to impose their own order and explain it on the basis of at
least one criterion.
3. The pupils are able to point out the characteristics of organisms, revealing how they are
adapted to their diet, protection against enemies and environmental influences.
Ecosystems
2. The pupils are familiar with a few biotopes in their environment and are able to identify and
name a number of common animals and plants in this.
5. The pupils are able to illustrate the law of eating and being eaten on the basis of the food
chain.
Environmental care
4. The pupils are able to illustrate that man influences the presence of plants and animals in his
environment.
19. The pupils are able to give concrete examples in their environment to illustrate how
peoples approach to the environment can be negative or positive, and that environmental
problems are often caused by conflicting interests.
NON-LIVING NATURE
8. The pupils are able to measure and describe the weather conditions at a particular moment
and over a limited period.
9. The pupils are able to illustrate the link between peoples habits and the climate they live in.
10. The pupils are able to demonstrate how the earth rotates around itself, and how the earth,
the sun and the moon move in relation to each other.
11. The pupils are able to indicate what materials and raw materials common objects in their
environment are made of.
Notes: a The numbers refer to the original numbering of the attainment targets in the official publications; b K = Knowledge, C = Comprehension, A = Application;
c
As explained in the text, this attainment target is officially also an attitude.
22 R. JANSSEN AND M. CRAUWELS
who should develop basic knowledge and insights (2) Are there systematic differences in scores between
through discovery and inquiry. schools and classes within schools and which factors
contribute to these differences? Are there certain
groups of children (e.g. groups based on gender or
Design of the national assessment on language spoken at home) that systematically
The national assessment consisted of two parts. In the perform lower than other children? Detailed anal-
first part, all pupils participated in paper-and-pencil yses of the results may result in pupil characteristics
tests on the six subfields of the knowledge-oriented and educational factors that affect the test perfor-
attainment targets. Each test consisted of about 25 mance in environmental studies nature across
items with a short-answer format. Of the items, 95% pupils, classes, and schools.
were multiple-choice. When designing the test instru-
ment for the national assessment, three levels of
Blooms (1968) taxonomy of educational goals were
Method
taken into account: knowledge, comprehension and
application. The last three columns of Table 1 indicate
Test development
which level applies to every attainment target. Table 2 The tests used in the national assessment were
gives an example of an item for every test. In a second constructed in a previous research project, which was
part, a random subsample of pupils from each school also commissioned by the Department of Education of
participated in a practical assessment on the attainment the Flemish Government. Test construction consisted
targets on research skills. The results of the practical of several phases: (1) description of the test design; (2)
Downloaded By: [informa internal users] At: 15:44 25 February 2011
assessments are discussed in a separate article (Crauwels formulation of the items while taking into account
et al. in preparation). both attainment targets and their operationalisation in
the curricula as defined by the educational authorities;
(3) piloting of the items with small samples of pupils;
Research questions (4) evaluation of the constructed items by experts; and
In line with the two purposes of a national assessment, (5) a large-scale calibration study, in which for each of
the following research questions were asked: the six tests a scale was constructed on the basis of item
response theory (IRT). IRT models are commonly
(1) How many Flemish pupils master the attainment used in national assessment programmes, such as in the
targets on environmental studies nature and are American National Assessment of Educational
there differences in performance among the six Progress (e.g. Mislevy et al. 1992), the Australian Basic
subfields? Detailed analyses of the results may Skills Testing programme (Masters et al. 1989), or the
result in content factors that influence the mastery Dutch National Assessment Programme (e.g. Thijssen
and comprehension of biology by 12-year-olds. et al. 2001). In order to determine the minimal test
Table 2. Example of an item situated below the standard for each scale
Downloaded By: [informa internal users] At: 15:44 25 February 2011
Table 2.
(Continued)
RESEARCH PAPER
23
24 R. JANSSEN AND M. CRAUWELS
Table 2. (Continued)
Downloaded By: [informa internal users] At: 15:44 25 February 2011
Notes: The percentage of students having chosen each response alternative is given in italics. For each item, the percentage of students who did not respond to it
varied between 1 and 2%.
performance necessary to reach the attainment targets, included 145 schools and 4556 children. Schools
a standard-setting study was carried out according to participated anonymously and on a voluntary basis. In
the Bookmark procedure (Mitzel et al. 2001). exchange for their participation, schools received
feedback about the performance of their pupils.
Table 3. Design of the test administration Table 4. Percentage of pupils reaching the
with the number of students from the total standard for each test as a function of
sample (N = 4556) responding to each gender
booklet and to each test
Girls Boys
Test booklet
Test % N % N 2 df p-value
Test 1 2 3 4 5 6 N
Human body 82.0 1546 77.0 1499 11.85 1 0.001
Non-living nature 3022 Health care 84.4 1535 77.3 1522 24.91 1 <0.001
Human body 3046 Organisms 52.4 1533 55.7 1501 3.36 1 0.067
Health care 3058 Ecosystems 82.5 1526 81.8 1506 0.25 1 0.616
Organisms 3034 Environmental 64.9 1505 65.7 1526 0.22 1 0.639
Ecosystems 3032 care
Environmental care 3032 Non-living nature 53.4 1523 58.2 1498 7.14 1 0.008
N 758 766 768 742 756 766
Results
Performance on the different Multilevel analyses
attainment targets When analysing the performance of the students on
Figure 1 shows the percentage of students reaching the the common IRT scale, it was found that schools
attainment targets of the six subfields. The best perfor- differed in their average pupil performance. The
mances were found on the tests of ecosystems, health- school level accounted for 12% of the total variance.
care and human body. For each of these subfields, The classroom level accounted for only 2%. In fact,
about 8 out of 10 pupils mastered the attainment the latter variance was not significantly different from
targets. The test measuring environmental care was zero, implying that there were no systematic differ-
mastered by 65% of the students. The tests measuring ences between the classes within schools. The remain-
non-living nature and organisms were mastered the ing 87% of the variance referred to differences among
worst, respectively, 56 and 54%. pupils within schools.
Table 5 presents the results of the final multilevel
Figure 1. Percentage of pupils reaching the standard for each of the six scales
Figure 1. Percentage of pupils reaching the standard for each of the six scales
26 R. JANSSEN AND M. CRAUWELS
with their mother score even lower. For the latter Discussion
group of children, the probability to solve an average Content factors in mastery of the
item lowers 21% in comparison with a reference pupil different attainment targets
who speaks Dutch with his or her mother. The pupil
Results of the national assessment showed that Flemish
variable age showed that success probability on an
pupils do not master the attainment targets in the differ-
average item diminished by about 10% for pupils that
ent subfields of environmental studies nature to the
were older than their class mates. Finally, pupils with
same extent. The attainment targets on man were
dyscalculia or with behavioural problems scored lower
mastered very well, both for the human body and for
than pupils without learning disabilities. The effect size
health care. The Flemish pupils performed very well on
on the success probability was 8%. Note that pupils
items testing basic knowledge of the human body. Pupils
with dyslexia did not score differently from pupils
knew the different organs of the human body and how
with no learning disabilities. No gender differences
each of them functions. However, items that required
were found on the overall scale. The children from the
higher levels of abstraction were more difficult. For
classes with a computer scored on average a little
example, performance was less for items about the organs
higher on the overall scale in comparison with chil-
involved in transport of oxygen or in the digestion of
dren from classes without a computer. Pupils from
food. This effect of level of abstraction was not found for
municipal schools scored on average somewhat higher
items about the development of the human body during
than pupils from private subsidised schools, while
puberty, which were mastered well. The high perfor-
pupils from Flemish community schools scored on
mance on the subfield of health care is probably related to
average somewhat lower. Finally, it was found that
Downloaded By: [informa internal users] At: 15:44 25 February 2011
Table 5. Results of the final multilevel model on the overall scale (N=3319)a
Notes: aThe final model was estimated on 73% of the total sample as only for these pupils information on all variables was available. Happily, several analyses on
subsamples did not reveal a selection bias. * p < .05; ** p < .01; *** p < .001. bFor a percentage equal to 50%. cDifferent categories of the categorical variables.
d
Percentage of pupils for whom the reported effect holds; one category is chosen as the reference category. eRegression weight; in comparison with a reference
category. fStandard error of estimation. gLevel of significance. hEffect size; a pupil who is in the reference category for all variables is taken as a reference and has a
probability of 0.76 to answer an item of medium difficulty correctly.
RESEARCH PAPER 27
The results on the tests about nature around us were chips can be made of potatoes. Children had little diffi-
mixed. The performance of the Flemish pupils on the culty with the items relating the lifestyle and habits of a
test on organisms was low. Pupils were able to system- community of people to the climate of their habitat.
atically order organisms on the basis of given features,
but systematically ordering organisms on the basis of a
self-chosen feature was one step too far. The items Gender differences
referring to the relations between features of organisms The reported gender differences on the different
and the appearance of these organisms in certain subfields are in line with previous research. First, in
biotopes were mastered worst. A possible explanation the present assessment girls outperformed boys on the
may be that students learn about organisms as separate tests on the human body and on health education.
entities, apart from their biotopes. In a Dutch national Studies in secondary education (Tunnicliffe 1997;
assessment of biology in primary education (Thijssen Uitto et al. 2006) showed that boys were more inter-
et al. 2003), it was also found that pupils had difficulty ested in basic processes in biology, while girls found
in seeing the interrelationships among organisms and human biology and health education more interesting.
between an organism and its habitat. Second, the boys performed better on the test on non-
Flemish pupils performed best on the test on ecosys- living nature, the attainment targets for which are
tems. This indicates that they are well acquainted with related to physics and chemistry. This is in line with
common plants and animals in their environment. the study by Baram-Tsabari and Yarden (2008). Third,
This is not in line with the aforementioned Dutch in the present assessment there was a slight advantage
national assessment, where pupils were not able to for the boys on the test on organisms. Huxham et al.
Downloaded By: [informa internal users] At: 15:44 25 February 2011
name common animals and plants. However, one (2006) reported that in primary education boys had
difference may be that in the Flemish assessment, the greater wildlife knowledge than girls. Finally, as in
pupils were asked to name certain plants and animals TIMSS or PISA, no gender differences were found on
on a figure. They did not have to name a particular the overall scale. Apparently, the gender differences in
animal or plant, as in the Dutch and other studies. different subfields cancel out when looking at an over-
Flemish pupils also master the law of eating and being all science performance.
eaten very well and they can also apply their knowl-
edge to new situations. Apparently, this topic is quite
comprehensible for children of this age. Student factors in mastery of the
Flemish pupils performed poorly on the subfield of attainment targets on environmental
environmental care, despite the attention this topic studies nature
currently gets in our society. However, the present The largest part of the differences in test performance
attainment targets are not just about discerning the on the overall scale refers to differences among pupils.
influence of man on nature. They also require that chil- A much smaller amount refers to differences between
dren have insight into relationships (cause and conse- schools. The differences among pupils can be
quence, method and goal), that they can take different explained only to a limited extent by the available
perspectives on a given environmental problem, and background variables. The language spoken with the
that they can discuss issues about policy-making. This mother and the age of the pupil have the largest
requires an integrated approach, including an overview impact. The effect of language at home is commonly
of all factors and actors involved and the relationships found in international studies. The role of language in
between these factors and actors, when thinking about learning biology has also been demonstrated in several
the problem. As Hipkins et al. (2008) showed, devel- studies (Gilmour and MacGregor 1985; Pallincsar
oping an understanding of biological and environmental 1989; Duran et al. 1998; Cakici 2005; Dimopoulos
issues requires children to have opportunities to learn et al. 2005; Brown and Ryoo 2008; Vikstrom 2008).
about systems interactions and impacts. The socio-economic background of the pupils was
The test on non-living nature was composed of a not measured at the individual level, but it did have an
diverse set of attainment targets. The weak perfor- impact at the school level. Schools with more children
mance for this subfield was largely due to the items with low socio-economic status (SES) have more diffi-
related to the movement of the earth, sun and moon. culties in reaching the attainment targets on nature.
Additionally, items that relied on the understanding of Other school variables that were significant predictors
a combination of weather elements were more diffi- were the educational network and the presence of a
cult. The items related to the identification of materials computer in the classroom. The effect of educational
used in objects were mastered better than the items network is partly an effect of the different composition
assessing the identification of raw materials to make of the school population among the three educational
objects. For example, pupils knew that tyres are made networks as Flemish community schools (and munici-
of rubber or that the leather from a handbag is derived pal schools to a lower extent) attract more pupils from
from animals, but did not know that in order to lower socio-economic backgrounds. It is difficult to
produce glass the raw material sand is needed, or that interpret the positive effect of the presence of a
28 R. JANSSEN AND M. CRAUWELS