Applied Linguistics 2011: 32/2: 236–241 ß Oxford University Press 2011

doi:10.1093/applin/amr007 Advance Access published on 2 March 2011

FORUM

Downloaded from https://academic.oup.com/applij/article-abstract/32/2/236/166388 by University of California, Irvine user on 02 September 2019

Are the differences between CLIL and
non-CLIL groups in Andalusia due to CLIL?
A reply to Lorenzo, Casal and Moore (2010)

ANTHONY BRUTON
University of Seville, Spain
E-mail: ab@us.es

Lorenzo et al. (2010) attribute some quite astounding average FL language score
differences between Content and Language Integrated Learning (CLIL) and
non-CLIL groups in Andalusia, Spain, precisely favoring the CLIL initiative.
However, there are a number of methodological research questions that jeop-
ardize any verifiable conclusions that can be drawn from the study as described
by the authors, beginning with the interpretation of the scores. Three other
significant factors are considered, before a plea is made for more disinterested
research, which does not ignore the less privileged.

THE CONTENTIOUS ISSUES

Content and Language Integrated Learning (CLIL) is not only gaining ground
in certain European educational systems, it is also becoming more prominent
in theoretical and applied debates on foreign language (FL) learning. In Spain,
a number of autonomous administrations have adopted CLIL initiatives of
various kinds, and there has recently been a proliferation of research publica-
tions on these cases, mostly demonstrating the beneficial effects of CLIL. If the
supposed language gain scores of the Lorenzo et al. (2010) results are to be
believed, CLIL can result in quite spectacular relative FL learning gains, more
than 20 per cent points over non-CLIL comparison groups, in a very short
time, one and a half academic years (p. 427). The trouble is that there are
some serious limitations in the research design, implementation, and interpret-
ation of this investigation. I will limit my comments to these major questions:
(i) the overall language scores; (ii) the characteristics of the control and ex-
perimental groups; (iii) extra support for CLIL groups; and (iv) the question-
naires on classroom FL use.

THE OVERALL SCORES

This research project was very ambitious in overall scope and extension and
the global scores seemingly impressive. These overall average language scores
 FORUM 237

are in percentage terms and compare both primary and secondary students

Downloaded from https://academic.oup.com/applij/article-abstract/32/2/236/166388 by University of California, Irvine user on 02 September 2019

from non-CLIL English FL streams with CLIL streams—referred to as ‘bilingual
sections’, in which some content subjects are (partly) taught in one of three
FLs, French, German, but mainly English. The authors state the following:
‘When the results of the linguistic evaluation had been compiled, it emerged
that the CLIL learners were clearly outperforming their mainstream peers.
Global average scores were 62.1 per cent for the bilingual groups in compari-
son with 38 per cent for the control groups’ (p. 426). This may be true, as far as
it goes. However, it actually does not say very much, except that the CLIL
sample outperformed the non-CLIL sample, on average. The trouble is there
are no pretest scores. When conducting research into any form of develop-
ment, it is necessary to establish a benchmark. Without any pretest scores, it is
not possible to assess any form of change since there is no point of departure.
As a result, there is no assurance that there was improvement, and there may
even have been deterioration, perhaps, with the so-called control groups. And
in order to make comparisons, pretest scores are again fundamental. Unless the
initial scores are available, there is no assurance that the two groups did not
begin with initial average score differences. So, without average pretest scores,
it is not possible to assess either absolute or relative gains, or losses. In other
words, it is perfectly possible that the control non-CLIL groups started off with
much lower average proficiency than the CLIL groups, and even deteriorated
thereafter. This takes us on to point two.

THE CONTROL GROUPS

Pretest scores could have shown whether the English non-CLIL controls had
similar initial pretest global language ability scores to the CLIL groups, and
across the four language skills. According to Casal and Moore (2009), the
CLIL groups are not compared to non-CLIL groups in other schools, but
rather with the non-CLIL groups in the same schools. Given the lack of pretest
scores, the authors imply that the groups are similar since the choice of enter-
ing a CLIL stream is optional: ‘. . . admission to bilingual sections is open to
everyone – there is no pretest or screening’ (p. 422). However, they then state
the following: ‘Bilingual sections are, therefore, essentially egalitarian (al-
though the possibilities of corollaries between social class and parental
choice cannot be ignored)’ (p. 423). This admission is something of a contra-
diction: choice supposes equality in theory, but in reality it might not. In a
mini-conference on this CLIL project in Andalusia hosted by Lorenzo (2008)
and the sponsoring Andalusian educational authorities, a consensus voiced by
numerous CLIL teachers was that it was generally the children of higher
socio-economic-status parents who chose the CLIL option in their schools.
The regional education authority cannot admit this, since streaming is not
permitted. This ‘natural’ selection is corroborated in another study in Spain
by Alonso et al. (2008), for example, in which 65 per cent of the CLIL students’
parents had university education. Furthermore, parents and children who
238 FORUM

choose this option will generally be the more motivated, as well. CLIL group

Downloaded from https://academic.oup.com/applij/article-abstract/32/2/236/166388 by University of California, Irvine user on 02 September 2019

students are also very likely to take extra English classes outside school
(Ruiz de Zarobe and Lasagabaster 2010)—78 per cent of the CLIL students
against none in the non-CLIL group in the Villarreal Olaizola and Garcı́a
Mayo (2009) study, also in Spain. Other studies in Spain show that the CLIL
groups tend to be of higher proficiency than the non-CLIL groups in the FL
(Alonso et al. 2008; San Isidro 2010), and that they tend to be more motivated
in the FL (Lasagabaster and Sierra 2009; Ruiz de Zarobe and Lasagabaster
2010; San Isidro 2010). This means that the non-CLIL, supposedly control,
groups are actually those students’ who, along with their parents, elected
not to choose the CLIL streams. These groups in Spain would also include
repeat students, since the educational system requires students with a certain
number of failed subjects to repeat the year. In other words, the control groups
are not students of similar characteristics taken from schools with no CLIL
streams so much as students who on average would have lower initial lan-
guage proficiency scores, lower motivational levels, and probably lower con-
tent subject scores as well, in the same schools. In which case, without any
pretest scores, the only fact that the posttest scores show is that the CLIL
groups score higher on average than the non-CLIL groups in the same schools,
which would be expected, with or without CLIL. In a study in Catalonia by
Navés and Victori (2010), what actually appears to happen is that the CLIL
groups start with a proficiency advantage which they maintain subsequently,
but do not increase. In passing, Lorenzo et al. state ‘that later start learners are
demonstrating competences comparable with early start learners’ (p. 436). This
claim is also put into doubt by the previous arguments.

EXTRA CLIL SUPPORT

There are other additional factors that can help explain the huge discrepancy
in the results. The first fact is that the CLIL groups have coordinators who have
their timetables reduced to dedicate to the CLIL program. Not only does this
mean that they have the disposition and the time to make the CLIL courses
more coherent, but also preparation time for materials. But perhaps more
importantly in terms of language scores, the CLIL groups have native-speaker
language assistants. These are crucial variables. In terms of composition writ-
ing, for example, it would mean that the language assistants could respond to
student writing much more quickly and easily than non-native-speakers and
reduce the load of the language teachers considerably. This might suppose that
the CLIL students actually write more than the non-CLIL students, since in
many typical EFL classes in the Spanish state secondary system the students
rarely write compositions—it might explain why the non-CLIL writing scores
are so low in Table A2 (p. 437). For speaking, of course, the language assistants
make the use of English much more logical or authentic since they do not have
the same L1 as the students. The presence of language assistants, it has to be
 FORUM 239

admitted, has nothing to do with CLIL, but is an enormous advantage for CLIL

Downloaded from https://academic.oup.com/applij/article-abstract/32/2/236/166388 by University of California, Irvine user on 02 September 2019

groups, which is not fully recognized by the authors.

FL USE IN CLASS
This takes us on to the question of the use of the FL in class. The authors,
unfortunately, rely on questionnaire data for quantifying the use of the FL in
class, by teacher type and by function. However, as anyone familiar with this
type of research knows, questionnaire data on the frequency of classroom
activity, especially on the part of teachers, is notoriously unreliable
(Karavas-Doukas 1996; Jarvis and Atsilarat 2004; Hawkey 2006;
Dalton-Puffer 2008). Even so, it is clear that the native speaker assistants
use the target language much more than the other teachers. It is possible,
and even probable, that they use the target language even more in relative
terms than the amount suggested by the authors’ data. In other studies, the
amount of L1 use is much higher among state FL teachers (e.g. Carless 2004;
Dalton-Puffer 2007; Orafi and Borg 2009) than these data suggest. If the rela-
tive difference is that great, it would of course not be the English used by the
CLIL teachers that would have made the difference so much as the English of
the assistants. It might of course be that the English of the content teachers is
counterproductive, if it is limited or substandard, but this is compensated for by
the language assistants. Since there are no observational data, no evidence can
be consulted, even though the authors throughout the article make various
claims about the discourse features of CLIL classes. The fact is that, so far, the
possible positive effects of CLIL per se have not been demonstrated.

ALTERNATIVE PERSPECTIVES
One of the issues raised here is that when CLIL-oriented programs are actually
implemented they are likely to include non-CLIL factors, and a major concern
of the evaluation may be the consequences of the particular initiative, rather
than whether it is more or less pure CLIL. This is a very valid position to adopt.
If the outcomes may be at least partly attributable to non-CLIL-specific factors,
then it should be recognized that they are due to a combination of factors
particular to the initiative, one or some of which characterize CLIL. Such a
perspective would require monitoring the actual practices, rather than any
predetermined set of practices, since the research would be exploratory in
attempting to discover what changes occur, if any, and perhaps typify them,
with their consequences. There have been numerous studies of this kind on
the implementation of the communicative approach or task-based language
teaching, for example by David Carless (2004, 2007) in Hong Kong, the
discussion of which lies outside the scope of this critique.
However, if a CLIL–non-CLIL comparison is to be made, certain distinguish-
ing variables of what CLIL is assumed to be would have to be defined, for
example whether the content and the FL are actually learnt together, or if
240 FORUM

one is prepared before the other, and the amount of FL exposure time.

Downloaded from https://academic.oup.com/applij/article-abstract/32/2/236/166388 by University of California, Irvine user on 02 September 2019

Thereafter, the essential characteristics would need to be monitored, in limited
representative samples, minimally requiring pre–post testing and some
ongoing interim observation, and either selective CLIL would have to be com-
pared to selective non-CLIL, or non-selective CLIL with non-selective
non-CLIL. In this case, in real school circumstances, it would be impossible
and probably undesirable to standardize actual classroom practice, but it would
nevertheless be necessary to characterize representative pedagogical practices
for either both conditions, or each condition separately.

CONCLUSIONS
I have not considered the testing instruments and procedures, since they
would divert attention from the main argument, which is that the research
conducted here does not show conclusively that CLIL is either positive or
negative for FL development, in this case mainly English. Rather more worry-
ing is that there is a growing body of research into CLIL that is being conducted
by investigators who seem to want to demonstrate that CLIL is necessarily a
positive route to raising the standards of FL learning at primary and secondary
levels in Europe. Unfortunately, there is a dearth of research into comparisons
between CLIL and comparable non-CLIL groups, either selected or unselected,
and there is a lack of interest in what happens to the non-CLIL students. The
latter might just be paying the price of their counterparts’ success, if it is ac-
tually genuine success, rather than just disguised selection. Hopefully, future
(disinterested) research into CLIL, especially in Spain, will take heed of some of
the serious research limitations raised here, and not ignore the broader edu-
cational implications of such initiatives.

REFERENCES
Alonso, E., J. Grisaleña, and A. Campo. 2008.
‘Plurilingual education in secondary schools:
Analysis of results,’ International CLIL Journal
1/1: 36–49.
Carless, D. 2004. ‘Issues in teachers’ reinterpret-
ation of a task-based innovation in primary
schools,’ TESOL Quarterly 38: 639–62.
Carless, D. 2007. ‘The suitability of task-based
approaches for secondary schools: Perspectives
from Hong Kong,’ System 35: 595–608.
Casal, S. and P. Moore. 2009. ‘The Andalusian
Bilingual Sections Scheme: Evaluation and
consultancy,’ International CLIL Research
Journal 1/2: 36–46.
Dalton-Puffer, C. 2007. Discourse in Content and
Language Integrated (CLIL) Classrooms. John
Benjamins.
Dalton-Puffer, C. 2008. ‘Outcomes and pro-
cesses in content and language integrated
(CLIL) learning: current research in Europe’
in W. Delanoy and L. Volkmann (eds): Future
Perspectives in English Language Teaching. Carl
Winter, pp. 7–23.
Hawkey, R. 2006. ‘Teacher and learner percep-
tions of language learning activity,’ ELT
Journal 60: 242–52.
Jarvis, H. and S. Atsilarat. 2004. ‘Shifting para-
digms: from communicative to a context-based
approach,’ Asian EFL Journal 6/4. Retrieved
from http://asian-efl-journal.com.
Karavas-Doukas, E. 1996. ‘Using attitude
scales to investigate teachers’ attitudes to the
communicative approach,’ ELT Journal 50:
187–98.
 FORUM 241

Lasagabaster, D. and J. M. Sierra. 2009. Ruiz de Zarobe, Y. and D. Lasagabaster.

Downloaded from https://academic.oup.com/applij/article-abstract/32/2/236/166388 by University of California, Irvine user on 02 September 2019

‘Language attitudes in CLIL and traditional EFL 2010. ‘CLIL in a bilingual community:
classes,’ International CLIL Research 1/2: 4–17. The Basque autonomous community’
Lorenzo, F. 2008. ‘Evaluación de competencias in D. Lasagabaster and Y. Ruiz de Zarobe
lingüı́sticas en centros bilingües. Datos de inci- (eds): CLIL in Spain: Implementation, results
dencia,’ Jornadas sobre Competencia en comunica- and teacher training. Cambridge Scholars
ción lingüı́stica: El curriculum integrado de Publishing, pp. 12–29.
lenguas27–28 November, 2008 San Isidro, X. 2010. ‘An insight into Galician
Lorenzo, F., S. Casal, and P. Moore. 2010. ‘The CLIL: Provision and results’
effects of content and language integrated in D. Lasagabaster and Y. Ruiz de Zarobe
learning in European education: key findings (eds): CLIL in Spain: Implementation, results and
from the Andalusian sections evaluation pro- teacher training. Cambridge Scholars Publishing,
ject,’ Applied Linguistics 31: 418–42. pp. 55–78.
Navés, T. and M. Victori. 2010. ‘CLIL in Villarreal Olaizola, I. and M. P. Garcı́a
Catalonia: an overview of research studies’ Mayo. 2009. ‘Tense and agreement
in D. Lasagabaster and Y. Ruiz de Zarobe morphology in the interlanguage of
(eds): CLIL in Spain: Implementation, results and Basque/ Spanish bilinguals: CLIL versus
teacher training. Cambridge Scholars Publishing, non-CLIL’ in Y. Ruiz de Zarobe and
pp. 30–54. R. M. Jiménez Catalán (eds): Content
Orafi, S. M. S. and S. Borg. 2009. ‘Intentions and Language Integrated Learning: Evidence
and realities in implementing communicative from research in Europe. Multilingual
curriculum reform,’ System 37: 243–53. Matters, pp. 157–75.