Computers & Education 81 (2015) 281e295

Contents lists available at ScienceDirect

Computers & Education

journal homepage: www.elsevier.com/locate/compedu

Virtual CEOs: A blended approach to digital gaming for enhancing

higher order thinking and academic achievement among vocational
high school students
Ya-Ting Carolyn Yang*
Institute of Education & Center for Teacher Education, National Cheng Kung University, No.1, University Road, Tainan City, 701 Taiwan, China

a r t i c l e i n f o

a b s t r a c t

Article history:
Received 23 May 2014
Received in revised form
3 September 2014
Accepted 7 October 2014
Available online 16 October 2014

The requirements of a contemporary workplace include the ability to think critically and creatively in
order to solve problems and respond to changes in economic and social conditions. Unfortunately,
vocational education often fails to prepare graduates for this environment due to limited resources, low
student motivation, or the reliance upon outdated instructional strategies. The use of digital game-based
learning (DGBL) for vocational education has been proposed, but has yet to be effectively implemented,
particularly in terms of the promotion of higher order thinking skills (HOTS). Data from 68 eleventh
grade vocational high school students were evaluated after a quasi-experimental, 27 week intervention.
Pretest and posttest results were evaluated by MANCOVA and demonstrated that the experimental group
(blended DGBL incorporating integrative HOTS activities) outperformed the comparison group (technology enhanced learning) in terms of creative thinking, critical thinking, problem solving, and academic
achievement, with signicant improvements on all four measures. While technology-enhanced learning
was effective in promoting academic achievement and creative thinking, the DGBL condition was
deemed most effective in providing an authentic context for developing employment-related skills and
knowledge. Based on these results, a blended approach for DGBL, which incorporates instructor
orchestration and scaffolding, provision of learning aids, and the use of collaborative learning, is recommended, particularly for vocational learners. This paper provides examples of a concrete model of
DGBL instruction that was veried empirically as successful in signicantly improving all three higher
order thinking skills, including creative thinking, critical thinking, and problem solving.
2014 Published by Elsevier Ltd.

Keywords:
Applications in subject areas
Cooperative/collaborative learning
Improving classroom teaching
Secondary education
Simulations

1. Introduction
Current workplace environments, information-intensive and continually evolving, require employees to continually acquire and transfer
skills and competencies to changing conditions and economic demands (Boahin & Hofman, 2013). The demand for general employability
skills and the ability to develop professional competence highlights the importance of creative thinking, critical thinking, and problemsolving, as well as communication skills, information and communication technology (ICT) skills, and collaboration (Partnership for 21st
Century Skills, 2009). Research has shown that fostering these employability skills and competences benets from a student-centered
approach to problem-based learning which adopts authentic employment-based learning and involves teamwork (Boahin & Hofman, 2013).
The exible and relevant skills and knowledge necessary for preparing students for employment in real world businesses cannot be
developed through traditional instruction that lacks authenticity and ignores the complex realities of the marketplace (Lainema & Nurmi,
2006). On the other hand, simulations or games can provide a) hands-on and relevant experiences with virtual business or employment
settings (Siewiorek, Saarinen, Lainema, & Lehtinen, 2012; Williams, 2008), and b) develop higher order thinking skills (HOTS) when
adopting social constructivist principles, such as active knowledge construction, the provision of support tools, and interaction among
participants (Kiili, 2005; Li, Cheng, & Liu, 2013).

* Tel.: 886 6 2757575x56230; fax: 886 6 2766493.

E-mail address: yangyt@mail.ncku.edu.tw.
http://dx.doi.org/10.1016/j.compedu.2014.10.004
0360-1315/ 2014 Published by Elsevier Ltd.

282

Y.-T.C. Yang / Computers & Education 81 (2015) 281e295

While DGBL has been used for decades, the adoption of simulation games for developing work-related knowledge while promoting HOTS
for vocational students has yet to be explored. In fact, many scholars suggest that rigorous empirical research demonstrating the effectiveness of digital games in learning is lacking (Bourgonjon et al., 2013; Erhel & Jamet, 2013). As Papastergiou (2009) noted, four important
gaps in the literature concerning DGBL should be explored: the lack of emphasis on learning outcomes within authentic classroom environments; the tendency of studies to emphasize traditional subjects, such as mathematics; a lack of studies with adolescent students; and
the problematic use of digital games as an alternative rather than a complement to classroom teaching. In order to address these
outstanding research gaps, this study adopts an authentic blended learning context, evaluating learning outcomes from adolescents in a
vocational education context, and exploring the impact of DBGL on higher order thinking by adopting a research design that integrates game
play with classroom teaching rather than viewing game play as an alternative to classroom instruction.
1.1. Vocational education in Taiwan
Vocational Education and Training, concerned with the development and training of technical and work-related skills, is critical for
increasing employment and ensuring economic stabilization (Inayat, ul Amin, Inayat, & Salim, 2013). As such, most countries rely upon some
form of vocational education to develop a competent workforce and provide an alternative to higher education for learners who seek
employment immediately following graduation. However, two main challenges face vocational educators and students: students' low
engagement and motivation and outdated approaches to instruction that fail to meet the demands of the marketplace.
In Taiwan, students often enroll in vocational schools because of low academic achievement and have, as a result, lower motivation,
condence, and attitudes toward learning (Lee, Shen, & Tsai, 2010). Taiwan's vocational schools also face the difculties of low enrollment,
little public investment, and students who may be distracted by part-time work and lack engagement in the classroom (Lee, Shen, & Tsai,
2008). Additionally, vocational schools are often private and must compete for students, accepting applicants with low levels of academic
achievement (Tsai, Lee, & Shen, 2013). Along with the resulting lack of achievement, many students also lack the problem-solving and
critical thinking skills which are highly valued by employers.
In terms of educational approaches to vocational education, a current report on the state of technical and vocational education in Taiwan
(Taiwan Ministry of Education, 2013) noted that vocational schools often lack an emphasis on job training and are increasingly adopting
exam-oriented approaches to both instruction and evaluation, prompting students to pursue vocational college after high school rather than
enter the marketplace, since vocational high school graduates are unprepared and, as a result, under-paid. The report recommends an
employment-oriented approach to vocational education, facilitating the transition from vocational school to the workplace through the
provision of diverse and exible curricula, hands-on experiences, experimentation, and training in skills relevant to specic industries in
which graduates might seek employment.
Solutions to these challenges may be provided by technology-enhanced learning (TEL). In fact, some studies (i.e., Lee, et al., 2010) have
suggested emphasizing authentic learning experiences with the assistance of e-learning in order to promote problem-solving and selfregulation among Taiwanese vocational students. However, the design of Lee et al.'s experiment relied upon applications of technology
(such as spreadsheet and word processing programs) which did not adequately simulate the complexities and realities of the workplace. As
well, student involvement, rather than actual problem-solving skills, was used as a dependent measure. Therefore, in this study, the
integration of digital games with employment-oriented instruction will be empirically evaluated.
1.2. Digital Game-Based Learning (DGBL)
As a form of TEL, digital games are believed to offer several advantages to learners, some of which may be particularly useful for
vocational learners. Given the changing demands of marketplaces within knowledge-based economies, scholars have argued that contextbased learning with realistic or simulated learning activities, and augmented by information and communications technology, will provide
learners with much needed vocational skills and experiences.
Game-based learning is grounded in the notion of play as an intrinsically motivating experience and, as an extension of Problem Based
Learning (PBL), as a constructivist principle guiding learning through active learning (Wu, Chiou, Kao, Hu, & Huang, 2012). In terms of game
play, Malone (1982), in an early analysis of learner' intrinsic interest in computer games, concluded that three main guidelines inuenced
learner enjoyment: challenge, including the provision of clear goals along with a range of uncertain and adaptive outcomes; fantasy, which
involves an appeal to players' emotions; and curiosity, which is elicited through humor and visual effects and which balances players'
background knowledge with opportunities for acquiring new information. The adoption of a PBL approach to DGBL was explored in detail by
Warren, Dondlinger, McLeod, and Bigenho (2012) who proposed that PBL allows for an authentic learning environment which engages
learners in story-like problem-based tasks.
While digital games have been used over the past few decades, games are increasingly focused on learner-centered approaches and
many have begun including instructional supports, such as clues, feedback, or explanations, to assist learning (Kebritchi & Hirumi, 2008). As
a result, opportunities for collaboration and problem-solving are plentiful in digital games, particularly for simulation games which allow
learners to explore and test hypotheses. Kiili's (2005) model of experiential gaming emphasizes learner engagement, requiring the provision
of immediate feedback, and the use of clear and appropriate challenges which take into consideration players' skills and prior knowledge.
Additionally, cognitive load is a critical factor in successful digital game-based learning. As such, the use of a blended learning environment
was adapted for this study in order to reduce the cognitive load which might arise from the presence of multiple, simultaneous challenges
during game play.
While few studies in the area of digital game-based learning emphasize the area of business management, business simulation games
(BSGs) can provide more authentic and practical opportunities for exploring realistic work environments and engaging in the decisionmaking necessary for success in the marketplace (Chang, Chen, Yang & Chao, 2009; Lin & Tu, 2012). Simulations have even been adopted for developing leadership skills in business education at the graduate school level (Siewiorek et al., 2012) and for professional development with construction professionals (Wall & Ahmed, 2008). The appropriateness of digital games for vocational learners at the high

Y.-T.C. Yang / Computers & Education 81 (2015) 281e295

283

school level will be evaluated by this study. In particular, we will focus on two principle learning outcomes: the acquisition of HOTS and
academic achievement in a course focused on employment-oriented knowledge and skills.
1.3. DGBL and higher order thinking skills
Since employers demand workers who have the ability to think critically and solve problems, classes which rely upon lower-level
thinking and fail to provide opportunities for critical thinking will jeopardize the employability of graduates (Kiener, Ahuna, & Tinnesz,
2014). Thus, HOTS, including creativity, critical thinking, and problem solving, are crucial for vocational students whose training is oriented toward future employability and the development of transferable skills which will lead to a professional career.
Recent papers in the eld of secondary education have highlighted the potential of digital games for fostering 21st century skills,
including HOTS as well as soft skills such as collaboration and global awareness (Ray, Faure, & Kelle, 2013). Well-designed digital games
cannot only assist learners in knowledge construction, but develop HOTS which extend beyond the game to students' personal lives (Kiili,
2005). However, many such studies do not empirically evaluate the impact of digital games on HOTS. For example, in a recent study, Kiener,
et al. (2014) adopted an action research approach to evaluate the role of critical thinking in developing a professional disposition and the
higher order thinking necessary for employability, offering classes which combined lecture components with discussion and case study
analysis. Their ndings indicated that an emphasis on critical thinking skills was an important contributor to students' development of a
professional perspective.
In the context of business or vocational education, simulation offers several advantages, including linking learning with authentic duties,
offering opportunities for collaboration and sharing of experiences, and introducing students to the complexities of the real world, allowing
them to experiment and take risks in a safe context (Siewiorek et al., 2012). Problem-based approaches to games or simulations provide
learners with opportunities to explore and experience realistic scenarios in which all learning tasks are linked to a larger, authentic, and
complex problem in which players can test alternative ideas, take ownership for the consequences, and receive feedback and opportunities
for reection in order to develop HOTS (Warren et al., 2012).
This type of problem-based gaming ts with what Kiili (2007) describes as the three fundamental DGBL characteristics for effectively
developing problem-solving skills: authentic and realistic simulations which leads to learning that can be transferred to the real world;
cooperative team play and knowledge sharing; and learning through active, hands-on participation. One important factor in Kiili's (2007)
problem-based gaming study was an emphasis on critical reection, necessary for problem-solving, planning, and adaption during game
play. Based on these ndings, the present study adopts an authentic simulation game with collaborative small groups for game play.
Additionally, support is provided through integrative learning activities which foster student-centered, active learning, engagement with
the learning topic, support of game play, and sufcient time and scaffolding for critical reection.
1.4. DGBL and academic achievement
Digital games can be successful in addressing the real-world needs of vocational students, creating opportunities for learning trans m
ferable skills and rehearsing workplace-specic knowledge (Ha
al
ainen, Oksanen, & H
akkinen, 2008). It is expected that digital games will
improve student attention and motivation while providing a problem-based learning environment which promotes the rehearsal and
application of curricular content, resulting in improved academic performance. In fact, according to Ebner and Holzinger (2007), digital
games provide opportunities for in-depth learning of subject matter by simulating authentic experiences through realistic settings. The
advantage of simulation games in terms of decision-making and problem-solving was explored by Pasin and Giroux (2011) through the
examination of student errors over the course of simulation play, nding that students benetted from the simulation's realistic context,
emphasis on the need for continual adjustment, and monitoring and responding to ongoing changes and consequences. Thus, it appears that
learners can develop the knowledge and skills necessary to succeed academically, when provided with appropriate and relevant digital
gaming experiences.
However, due to mixed empirical results, Erhel and Jamet (2013) suggest that it is difcult to conclude decisively that digital games and
simulations have a positive impact on learning and motivational outcomes. As such, the authors advocate a value-added approach, which
explores the features of digital games which can enhance the quality of learning, nding that the nature of game instructions and feedback
provided can inuence the development of deep learning. In extending the evaluation of digital game based learning to a blended learning
environment, the current study designs supplemental and scaffolding activities for integrating HOTS with curricular objectives and game
play.
1.5. Suggestions for the effective integration of DGBL
Based on a review of the literature, certain key recommendations and suggestions were discovered which relate to the more effective use
of DGBL. These recommendations concern the use of collaboration, provision of learning aids, scaffolding of learning and instructor
orchestration, and a blended learning environment.
1.5.1. Collaboration
First, collaboration is essential for online learning environments to be successful in promoting learning and higher order thinking, such as
critical reection, questioning, reasoning, and problem-solving. In order to design a computer-supported collaborative learning environ m
ment (CSCL) for vocational students, Ha
al
ainen, et al. (2008) adopted collaboration scripts within the game to prompt learners to interact,
thereby enhancing their learning of authentic, transferable skills which would have been difcult to achieve in traditional classroom environments. Student collaboration in online vocational learning environments was also explored by Inayat et al. (2013) in four vocational
courses, in which it was found that group interaction (among students and between students and instructors), instructor feedback, and the
provision of support materials were signicant contributors to learning.

284

Y.-T.C. Yang / Computers & Education 81 (2015) 281e295

1.5.2. Learning aids

Digital games are inherently interesting and challenging, but learning aids may sometimes be required, particularly for simulations.
Certain elements of commercial computer games have been reported by students to enhance their learning experience, such as audio-visual
elements and the presence of a story or challenge; while games or simulations best incorporate the use of skills such as logical thinking,
problem-solving, and memory (Amory, Naicker, Vincent, & Adams, 1999). While these elements were found to be present in some commercial games, students tended to dislike the simulation game, leading the authors to suggest that learning tools based on games themselves may be the key to motivating learners and providing a superior virtual environment.
1.5.3. Scaffolding and teacher orchestration
Scaffolding of learning is another key recommendation for effective DGBL. From a constructionist perspective, Li, et al. (2013) propose
that digital game learning systems should adopt three principles: having a construction or authentic outcome as the primary goal; ensuring
ease of use while allowing high levels of freedom in playing the game, such that learners of all skill levels can participate; and adopting a
simulation approach to allow learners to practice trial-and-error and evaluate and correct their actions based on game outcomes. Li et al.
(2013) further suggest that a form of debrieng is important for simulations which involve problem solving. Finding that some learners
lacked sufcient self-regulation to complete problem-based simulation games, Warren et al. (2012) also recommend that sufcient scaffolds
be provided to learners in solving game tasks, such as organizing and planning their work, receiving assistance, and recording and selfevaluating their progress.
When adopting computer simulations for solving problems, students require additional support, particularly from instructors, in order to
ensure that they can acquire the background knowledge and skills necessary to successfully achieve their learning goals (Liu, Cheng, &
ma
la
inen
Huang, 2011). Several challenges face instructors seeking to foster collaboration in virtual learning environments, leading Ha
and Oksanen (2012) to consider the active role of teacher orchestration, design, and monitoring of learning activities in which students
collaboratively construct knowledge in CSCL learning environments. Although their study incorporated a small number of students and
involved a short period of data collection, the authors found that teacher orchestration was successful in increasing the amount of on-topic
information provided among participants, concluding that the role of the teacher needs to highlight teachers' possibilities to orchestrate
learning and empower knowledge construction processes (p. 291).
1.5.4. Blended learning environment
ma
la
inen and Oksanen (2012), such as variations in the quality of individual contributions,
Furthermore, certain factors highlighted by Ha
difculties in self-regulation of learning, lack of direction or teacher intervention, and poor teamwork or group interaction, also suggest that
a purely virtual environment for collaborative learning is insufcient, and that a blended learning environment may be more appropriate.
1.6. Identication of a research gap
While the use of commercial games for fostering learning outcomes such as motivation, academic achievement, and thinking skills has
been proposed on the basis of several theoretical approaches, many contemporary scholars (i.e., Bourgonjon, et al., 2013) call for more
rigorous empirical evidence of these claims. While some authors have explored the role of simulation business games in fostering problemsolving (eg., Kiili, 2007; Lainema & Nurmi, 2006; Lin & Tu, 2012), these studies have largely been qualitative in nature and have not provided
convincing empirical evidence of how digital games can promote learning and higher order thinking. This study, therefore, adopts a quasiexperimental design with a two-semester time frame to rigorously and empirically evaluate several key learning outcomes.
Furthermore, there is a relative lack of empirical evidence to demonstrate that CSCL is effective in vocational learning contexts
ma
l
(Ha
ainen & Oksanen, 2012). Therefore, this study emphasizes vocational education as an increasingly important setting for developing
HOTS and promoting the acquisition of work-related knowledge and skills.
Finally, in keeping with the recommendations of several authors, DGBL in this study will integrate collaborative game play in a blended
learning context which emphasizes scaffolding or learning activities, teacher orchestration, and the use of learning aids. Specically,
integrative HOTS learning activities which link course content to game play will be provided for increased motivation as well as opportunities for expanding upon the logical, problem-solving, and memory skills which are rehearsed through game play. Thus, one contribution
of this study is to provide an example of a successful integration of DGBL in a blended learning environment which treats digital games and
classroom instruction as complementary (Papastergiou, 2009).
2. Purpose of the study
Vocational learners, particularly in the Taiwanese system, suffer from low academic achievement (Lee et al., 2010) and a lack of engaging
learning activities (Lee et al., 2008), which must be remedied by a relevant, hands-on, and engaging curriculum (Taiwan Ministry of
Education, 2013). A review of the literature has found that technology-enhanced learning (TEL), and digital game-based learning in
particular, may provide the best context for promoting higher order thinking skills (HOTS) and academic achievement for vocational
learners. However, in a systematic review of the literature on computer games in education, Connolly, Boyle, MacArthur, Hainey, and Boyle
(2012) found that in terms of promoting HOTS the few papers which provided high quality evidence to support these claims presented
qualitative rather than quantitative analysis (p. 672). Thus, the current study attempts to evaluate the use of digital games for promoting
HOTS, simultaneously evaluating creative thinking, critical thinking, and problem solving skills, as well as academic achievement. In order to
fairly evaluate the effects of DGBL on HOTS and academic achievement, a group adopting technology-enhanced learning strategies will be
used for comparison, particularly since TEL is recommended by Taiwanese researchers for fostering higher order thinking in Taiwanese
vocational contexts (Lee, et al., 2010). Thus, this study adopts a quasi-experimental design to explore the following research questions:
 Will participants who learn through DGBL improve more signicantly in terms of creative thinking, as compared to those who learn
through other forms of TEL (comparison group)?

Y.-T.C. Yang / Computers & Education 81 (2015) 281e295

285

 Will participants who learn through DGBL improve more signicantly in terms of critical thinking, as compared to those who learn
through other forms of TEL (comparison group)?
 Will participants who learn through DGBL improve more signicantly in terms of problem solving, as compared to those who learn
through other forms of TEL (comparison group)?
 Will participants who learn through DGBL improve more signicantly in terms of academic achievement, as compared to those who
learn through other forms of TEL (comparison group)?
3. Methods
In order to evaluate the above research questions, a 27 week quasi-experimental design was adopted. Fig. 1 illustrates the overall design
of the experiment, including both independent and dependent variables.
3.1. Participants
Participants included seventy-seven students (16e17 years old) who were recruited from and completed two sections of an 11th grade
Store Planning and Management class at a vocational senior high school in Taiwan. Both sections were taught by the same instructor, using
the same textbook, schedule, and evaluations but were taught using two different instructional strategies. One class, with 41 students (12
males and 29 females), was taught with TEL activities and served as the comparison group, while the other class, with 36 students (7 males
and 29 females), was taught using DGBL and served as the experimental group.
3.2. Independent variable
The independent variable was instructional strategy: TEL vs. DGBL. The comparison group learned through classroom activities that
featured the integration of technology for enhancing learning (such as the use of an interactive whiteboard, multimedia presentations, and
software for spreadsheets, word processing, and desktop publishing), and included class discussions, debates, and student-centered
learning activities which were designed to enhance both student's higher order thinking, as well as academic achievement. Meanwhile,
students in the experimental group engaged in DGBL activities, including both tutorials and game challenges, supplemented by activities
aimed at cohesively integrating higher order thinking with digital game play. The class activities and allocation of time for the two groups
are shown in Table 1.
3.3. Dependent variables
The four dependent variables evaluated in this study were students' HOTS (including creative thinking, critical thinking, and problem
solving) and academic achievement in Store Planning and Management. These variables were measured by the following instruments.
3.3.1. Creative thinking skills
Creative thinking was measured by the Torrance Test of Creative Thinking with Pictures (Torrance, 1998) which evaluates creativity in
ve dimensions: uency, originality, elaboration, resistance to premature closure, and abstractness of titles. The test form consists of three
parts: construction and naming of a large abstract picture; completion and labeling a set of incomplete gures; and using a series of
repeating shapes (such as circles or parallel lines) in order to create as many gures as possible. Ten minutes is provided for each part. In
addition, bonus points are awarded for thirteen creative strengths, such as expressiveness, storytelling, and movement and action. The
reliability of the measure has been reported as Cronbach's a .81 (Torrance, 1981).
In the case of uency, one point is allotted for each unique and meaningful gure which is drawn or completed. To evaluate originality,
the scoring manual provides a list of responses which, based on many thousands of tests being evaluated, are considered too common to be
original. Figures which are drawn that are not listed in the scoring manual are statistically uncommon and, therefore, awarded points for

Fig. 1. Research design.

286

Y.-T.C. Yang / Computers & Education 81 (2015) 281e295

Table 1
Summary of the two research groups including time allocated for learning tasks.
Instructional activities

Comparison group: technology-enhanced learning

Experimental group: digital game-based learning

Content delivery
Student-centered learning tasks (targeting HOTS
and academic achievement)

PPT-assisted instruction (20%)

Case studies and blog reviews (15%)
Online information searches (15%)
Group discussions and debates (20%)
Multimedia to illustrate concepts (10%)
Group presentations (15%)
Quizzes (5%)

PPT-assisted instruction (15%)

Preview of gaming tasks (5%)
Game tutorials (10%)
Game play (25%)
Reporting on game play (10%) HOTS integrated tasks (20%)
Group presentations (10%)
Peer review (5%)

Assessment and reection

originality. Elaboration is evaluated by the degree to which of meaningful elements added which go beyond the minimum required, such as
color, shading, or additional details. Resistance to premature closure relates to Activity Two, and evaluates the test-taker's resistance to
making the shortest or easiest possible completion of an incomplete gure. Abstractness of title is also evaluated by a rubric, which considers the amount of description, particularly non-concrete, which is used to describe or name gures. Other creative strengths, such as
humor add bonus points to the overall score.
3.3.2. Critical thinking skills
Critical thinking skills were measured by the Critical Thinking Test-Level I (Yeh, 2003), which includes a total of ve subscales: recognition of assumptions (identifying implicit claims), induction (inferences based on a set of known facts), deduction (drawing a rational
conclusion on the basis of given premises), interpretation (identifying causal relationships implied by statements), and evaluation of arguments (assessing the strength of an argument). Each subscale includes 5 multiple choice questions worth one point each, with a
maximum score for the test of 25 points. The overall Cronbach's a of the CTT-I is .76.
Two examples will be provided to illustrate the subscales of the CTT-I. For example, one question from the recognition of assumptions
subscale asks students to consider the assumptions behind the statement I have to check if there is a warm coat in my suitcase, since I am
going on a trip to Yangmingshan National Park tomorrow. Three choices are provided: a) Yangmingshan National Park will be cold
tomorrow; b) There is already a warm coat in the suitcase; and c) People should bring a coat when going to Yangmingshan. Of these options,
a) is the correct answer. For the subscale of deduction, one question provides students with two premises and asks them to identify the
correct conclusion: The amusement park is crowded on holidays; tourists will be in a bad mood when it is crowded. The three options are:
a) Tourists who go to the amusement park on holidays will be in a bad mood; b) The reason why people have a bad temper is because of
crowded spaces; and c) People have a bad temper on holidays. Of these options, a) is the correct answer.
3.3.3. Problem solving skills
Problem solving was measured by the New Test of Problem Solving (Chan & Wu, 2007) which includes three subscales: nding causes,
nding solutions, and avoiding problems. This test proposes several scenarios, to which students can provide a maximum of three responses
in terms of the cause, solution, and future avoidance of problems. This measure evaluates both the exibility (i.e., the variety and
uniqueness) and effectiveness (i.e., the practicality and usefulness) of the solution in computing a nal score. The test-retest reliability of the
test is .86 with a Cronbachs a of .91. A sample question for the nding solutions subscale of the NTPS is provided as follows: On your way
to school, the tire of your bicycle went at. You worry that you cannot arrive at school on time. What will you do now?
Students' answers are rst identied according to certain categories of response, which are provided in the NTPS scoring manual. For this
question, answers are organized into four categories: A (xing the bicycle by yourself), B (asking somebody to give you a ride), C (going to
school on your own), and D (informing your teacher of the situation). A rubric in the NTPS manual provides sample items for evaluating
effectiveness scores for each response, ranging from a score of 0 (non-effective solution) to 2 (full points). For exibility, one point is awarded
for each answer that is effective (i.e., received a raw score greater than zero) with a maximum of one point awarded for each category
answer. Effectiveness is calculated as the sum of the raw scores for each response to the question. Using effectiveness and exibility scores,
an overall problem solving score can be computed. As noted above, this score is equal to (exibility  2) effectiveness.
3.3.4. Academic achievement
To evaluate students' academic achievement in Store Planning and Management, a posttest was developed based on the national
vocational education curriculum. The pretest scores of both comparison and experimental groups were obtained from students' nal exams
of a business class from their previous semester. Content/expert validity was achieved through the cooperation of two experienced
vocational high school teachers with approximately 10 years of teaching experience. The academic achievement test consisted of four
categories: principles of store planning and management (four questions), business processes (four questions), inventory management (four
questions), and marketing strategies (eight questions). Each of the 20 multiple choice questions was worth 5 points for a maximum score of
100 points.
3.4. Research procedures
The duration of the quasi-experiment was 27 weeks, including two weeks for pretests and two weeks for posttests. The remaining 23
weeks of the experiment was divided into two semesters, as shown in Table 2.
An elective Store Planning and Management class was selected as the setting for this study, since it was deemed most appropriate for
the integration of digital games or simulations targeting employment-related skills and knowledge. During the rst semester, following two
weeks of pretests, one 50 min class per week was taught for each section (Weeks 3 through 12). However, based on a design-based research
orientation, teacher input was welcomed for the modication of the instructional design. Thus, during the course of the intervention, the

Y.-T.C. Yang / Computers & Education 81 (2015) 281e295

287

Table 2
Experimental procedure.
Research group

Pretests

Semester 1

Semester 2

Posttests

Week
Comparison
Experimental

1e2
O1
O1

3e12
X1
X2

13e24
X1
X2

25e27
O2
O2

Notes. X1: Technology-enhanced learning (comparison group). X2: Digital game-based learning (experimental group).

classroom teacher determined that the class time was too short for both groups and used exible study hours to provide an additional
50 min per week for both groups during the second semester of the course (Weeks 13 through 24), in order to ensure that the effectiveness
of the two approaches could be reasonably compared. This coincides with Taiwan Ministry of Education's (2013) suggestion that diverse and
exible curricula be provided for learners. The study concluded with two weeks of posttests.
For both groups, the rst semester of both courses included the topics of business processes and inventory management, while the
second semester focused on marketing, market segmentation, and market strategies. For both groups, reviews and reports were administered at the end of the rst semester, while debates and group presentations were used at the end of the second semester. A summary of
the two instructional strategies is provided in the following sub-sections, while Tables 3 and 4 provide details on the instructional content
for both groups.
3.4.1. Experimental group: DGBL
The experimental group played the Mandarin version of two commercially available digital games, Chief Executive Ofcer (CEO) produced
by T-Time Technology, and Capitalism 2 by Enlight. As shown in Table 1, approximately 15% of class time was allocated to the presentation of
textbook content, with another 15% used for previewing of games and game tutorials. Once students were familiar with the games, around
35% of the class time was used for game play and reection on the game tasks. Furthermore, in order to integrate course content with game
play, integrated learning tasks were provided, requiring around 20% of the class time. These activities were designed by the researcher and

Table 3
Weekly activities for comparison and experimental groups (semester 1).
Week Instructional content
(for both groups)
1e3
4

Pretest
Introduction

 Multimedia presentations
- Introduce and compare  Tabulate and compare
types of processes
processes
Business Processes
 Internet search
- selection factors
 Downloading of related
- product life cycle
content
Inventory management
 Multimedia
- dene stock and
 Group brainstorming:
inventory types
importance of inventory
- the function of
management
inventory
Inventory management
 Internet search
- performance evaluation  Downloading of related
- classication
content
Case study 1:

Case study 2:

- the manufacturing and

sale of a wristwatch

11

12

 Introduction to the
course and grouping

Business processes

- the manufacturing and

sale of a wristwatch

10

Comparison group activities

Group reports

Review week

Experimental group activities

Game play and tutorials

HOTS integrative activities

 Introduction to the
course and grouping

Creative thinking Group motivation

 4: Produce aluminum

activity: select
team name
Creative thinking Design three
potential team
logos
Creative thinking Create an original
team chant

Tutorials: manufacturing 5: Process

Critical thinking Introduction to

Tutorials: production 1: Construct a mill

 2: Produce lumber
Tutorials: processing 3: Produce paper

materials to
manufacture
desks.
 6: Retail the desks for prot
Game play: CEO

Critical thinking

 Groups complete six game tasks which

match the six tutorials

 Groups access internet in

preparation for a report on
the company

Game play: CEO

 Groups share reports using

multimedia and discussion
via PowerCam

Game play: CEO

 Groups share reports using

multimedia and discussion
via PowerCam

Reporting on game play

Critical thinking

 Groups compete, using the free game

function of CEO to earn 100 million NTD

Problem solving

 Groups respond to consumer demands

and produce a different product

Problem solving

 Groups report their results and reect

on game play

 Review of the content before  Groups report results and reect on

nal exams
game play

Problem solving

critical thinking:
sample questions
based on Store
Management
Induction and
deduction: evaluate
statements about
management
Interpretation,
Evaluation,
Recognition of
Assumptions
activities
Dening problems
and causes
associated with
failing to earn
100 million
Preventing future
problems: solutions
for improving future
performance
SWOT analysis of
production strategies

288

Y.-T.C. Yang / Computers & Education 81 (2015) 281e295

Table 4
Weekly activities for comparison and experimental groups (Semester 2).
Week

Instructional content
(for both groups)

Comparison group activities

13

Introduction

14

Marketing I:

- producer-oriented
marketing
management
15

Marketing II:

- producer and
consumer oriented
marketing management

16

Marketing III:

- social marketing
- evolution of
marketing
17

Market Segmentation I:

- geographic and
demographic
variables

18

Market Segmentation II:

- requirements

19

Marketing Strategies I:

- denition and
types of marketing
strategies

20

Marketing Strategies II:

- functions

21

Marketing Strategies III:

- international and
internet marketing

22

Debate

- goals and rules

23e24

Group presentations

25e27

Posttests

Experimental group activities

Game play and tutorials

HOTS integrative activities

 Introduction to the
course, grouping,
vote for group leader

 Introduction to the
course, grouping, and
voting for group leader
(mayor)

Creative
thinking Revise team

 Students go online to
read a blog
 Class discussion with
SMART board
 Design a table to
contrast producer
and consumer
oriented approaches
 Online search

Tutorials: marketing in
capitalism 2 1: Basics of retail

Critical
thinking Induction:

 Case study analysis

and discussion on
week 15 s web search
 Review quiz for unit
 Advertising strategies
multimedia
 Comparison of
advertising media

Tutorials: marketing in
capitalism 2 2: Retailing: part 2

Problem
solving Dening

 3: Manufacturing

causes
 Preventing problems

Tutorials: marketing in
capitalism 2 4: Advertising

Creative
thinking Evaluate

 Review online blog

 Review quiz for
Marketing unit
 Case study: Market
strategies of the
cellphone industry
 Class discussion
using PowerCam





Suppliers and products

Supply and demand
Game map/functions

Game challenge I: basic

retail strategy Earn 100

million NTD

and Brand
Game challenge II: advertising

 Groups compete, using an

advertising strategy to reach
customers

 Debate which is
more important for
marketing: corporate
branding or product
quality
 Group presentations
on marketing
strategies (topics
approved by the
instructor) and
student assessment

patterns of
consumption
 Evaluate and
explain prot goal
progress
 Suggest reasons for
the results of game
play

the meaning
and creativity
of an award logo
 Logo redesign
Problem
solving Evaluating

attributes of top
Taiwan brands
Game challenge III: emerging
markets Boomtown (single-player):

Facing new markets, players

develop strategies to reach
customers, trying to earn
300 million yearly within
30 years

 Desktop publishing
to creating an
advertisement
 Multimedia of
international and
internet strategies
 Review quiz for
Strategies unit

name, logo,
and chant

Critical
thinking Evaluate a

mobile
marketing
strategy (4Ps)
 Evaluate brand loyalty
claims
Creative
thinking Design a

mobile phone
(creativity criteria)
Game challenge IV: management
team contest Boomtown

(multi-player):
Groups compete online
to earn a prot

Problem
Solving SWOT analysis

of marketing
strategies to
compete in the
Team Contest
Critical
thinking Assessment

rubric

classroom teacher in order to link textbook content with game play while fostering higher order thinking. It should be noted that while
game play itself, as well as reporting on game play and other class activities, were designed to promote higher order thinking and academic
achievement, the highlight of the HOTS-integrated learning tasks was to synthesize game play elements with course content in order to
provide scaffolding for game play as well as hands-on, blended, learning activities. This was deemed important in order to allow DGBL
learners with equal opportunities to engage with course-relevant material outside of the context of game play, in order to create a truly
blended learning environment that balanced the strengths of both digital and face-to-face environments.

Y.-T.C. Yang / Computers & Education 81 (2015) 281e295

289

In keeping with the principles of digital game play covered in the introduction, the experimental group included the following
characteristics.
3.4.1.1. Collaboration. Students formed collaborative teams of six groups of six students each. While a smaller group number was desired,
limits on available computers necessitated larger groups. At the beginning of both semesters, activities were aimed at fostering collaboration
and a sense of group identity, including creating group names, logos, and chants.
3.4.1.2. Learning aids. Following the recommendation of previous studies, clear learning aids were provided. Game tutorials, based on
examples provided by the game publisher or created by the instructor, were assigned in order to ensure learner's familiarity with the games'
functions and tasks and to avoid frustration. Furthermore, the HOTS integrative activities were designed to lead students to reect on their
game play experience in order to effectively integrate textbook knowledge with the games' simulation activities.
3.4.1.3. Scaffolding and teacher orchestration. Given the blended nature of the learning environment, discussed below, the teacher's presence during game play was ensured, and explicit orchestration and regulation of game play was provided in cases where students were
unsure how to proceed independently. Instructional scaffolding was provided both through the aforementioned learning aids, as well as
through modeling of game play, the provision of advice and suggestions during game play, and the weekly integrative HOTS tasks.
3.4.1.4. Blended learning environment. While a certain amount of time was required for face-to-face instruction (including the introduction
of course content and class discussion), a signicant portion of the class time was allocated for autonomous learning through computermediated gaming activities. As such, the advantages of online multi-player gaming and self-paced game play were balanced with access
to teacher support and periods of group reection and discussion.
3.4.2. Comparison group: TEL
While the experimental group used digital games to apply course content to simulated employment-oriented activities, learners in the
comparison group relied upon other forms of TEL in addition to direct instruction, which constituted approximately 20% of class time.
This design was adopted in order to provide a fair comparison between DGBL and an alternative, state-of-the-art instructional strategy.
Activities which utilized the benet of instructional technology for fostering higher order thinking and academic achievement
(approximately 60% of the class time) included inquiry-based learning tasks using internet searches and blog reviews, the adoption of an
interactive whiteboard with videoconferencing capabilities for centralizing classroom discussions by constructing and documenting
knowledge collaboratively, and the integration of interactive multimedia for illustrating principles related to the course content. As Lee,
et al. (2010) have suggested, the use of TEL is expected to bring about improvements to higher order thinking skills, such as problem
solving. Thus, the activities for the comparison group were designed to foster higher order thinking, based more directly on the content of
the course textbook, such as debates and discussions (emphasizing critical thinking skills), case studies and blog reviews (focusing on
problem solving), and students' use of multimedia and design of presentation materials (highlighting creative thinking skills). While
these traditional TEL activities were designed to foster higher order thinking, they did not integrate elements of digital game play and,
thus, were focused on the content of the textbook and supplemental materials. Details of these instructional methods are included in
Tables 3 and 4
Principles of TEL for the comparison group also mirror many of the improvements and innovations adopted for DGBL in this study. For
example, collaboration was emphasized as students also engaged in group work for each of the TEL elements of the course. Furthermore,
scaffolding and teacher orchestration is typical in technology-enhanced classes, and learning aids (in the form of multimedia presentations,
templates for organizing online research, and prociency-leveled blog readings) were provided for comparison group learners. In terms of
the learning environment, both the comparison and experimental group were blended in the sense that they involved both face-to-face and
online elements. However, the TEL group did not employ digital games or simulations and relied upon alternative learning activities for
promoting higher order thinking and employment-related skills, such as searching and evaluating authentic online sources, organizing
information in the form of reports, and engaging in discussions and debates.
3.5. Data analyses
Quantitative data were collected from pretests and posttests for each of the dependent variables. Before analysis was conducted, outliers
in the data were identied by interquartile range (Q1  1.5  IQR and Q3 1.5  IQR) and removed in order to avoid experimental error
(Tukey, 1977). Boxplots for data from the four measures (TTCT, CTT-I, PS and AA) were constructed to identify and remove nine outliers. Each
of these outliers was from the comparison group, and all of them demonstrated abnormally low posttest results. Then, descriptive statistics
were employed to describe and compare the means and standard deviations for the four tests between the two groups. Based on signicant
Pearson's correlations among dependent variables (r between .27 and .52, p < .05) and acceptable homogeneity of covariance (insignicant
Box's M test .16), multivariate analysis of covariance (MANCOVA) was selected for analyzing the results (Tabachnick & Fidell, 2012).
4. Results
Complete and useable data were collected from 68 participants in the quasi-experiment. Descriptive statistics (shown in Table 4) are
provided for the dependent variables. One-way MANCOVA, using the four dependent variables' pretests as covariates, was adopted to
analyze the difference between pretest and posttest scores for all of the dependent variables.
The results reveal that the posttest scores for academic achievement, creative thinking, critical thinking and problem solving differed
signicantly between the two research groups, Wilks' L .37, F 25.07, p .00. Thus, univariate F tests were conducted as follow-ups for
creative thinking, critical thinking, problem solving and academic achievement, as shown in Table 5.

290

Y.-T.C. Yang / Computers & Education 81 (2015) 281e295

Table 5
Descriptive statistics for creative thinking (TTCT), critical thinking (CTT-I), problem solving (PS) and academic achievement (AA).
Comparison group (n 32)
Pretest

TTCT
CTT-I
PS
AA

Experimental group (n 36)

Posttest

Pretest

Posttest

SD

SD

Adj. M

SD

SD

Adj. M

432.41
17.91
108.22
43.91

105.30
3.04
13.53
10.68

486.59
17.19
103.03
61.41

47.55
3.16
14.55
11.93

482.71
17.12
102.89
60.73

443.00
17.61
101.39
31.58

71.66
3.33
17.73
11.51

565.58
19.56
117.67
86.25

56.08
2.25
9.49
11.24

569.04
19.62
117.80
86.85

4.1. Creative thinking skills

As shown in Table 5, the results of the univariate F tests indicate a signicant difference between groups for the creative thinking posttest
(p .00, h2 .38). Creative thinking scores were signicantly higher for the experimental group on the posttest (M 565.58, SD 56.08) as
compared to the comparison group (M 486.59, SD 47.55). According to Cohen's (1988) guidelines, an effect size of .38 is considered large.
Thus, the statistical results demonstrate that participants in the DGBL group showed signicantly better creative thinking following the
intervention than the comparison group.
4.2. Critical thinking skills
As seen in Table 5, the results of univariate F tests show a signicant difference between groups on the posttest (p .00, h2 .18). Critical
thinking scores were signicantly higher for the experimental group on the critical thinking posttest (M 19.56, SD 2.25) as compared to
the comparison group (M 17.19, SD 3.16), with a large effect size. These results demonstrate that DGBL participants achieved signicantly
greater improvements in critical thinking than comparison group participants.
4.3. Problem solving skills
As shown in Table 5, the results of univariate F tests indicate a signicant difference between groups on the posttest (p .00, h2 .24).
Problem solving scores were signicantly higher for the experimental group on the problem solving posttest (M 117.67, SD 9.49) as
compared to the comparison group (M 103.03, SD 14.55), with a large effect size. These results demonstrate that DGBL participants had
signicantly greater improvements in problem solving than comparison group participants.
4.4. Academic achievement
As listed in Table 5, the results of univariate F tests display a signicant difference between groups on the academic achievement posttest
(p .00, h2 .50). Academic achievement scores were signicantly higher for the experimental group on the posttest (M 86.25,
SD 11.24) as compared to the comparison group (M 61.41, SD 11.93), with a large effect size. These results demonstrate that DGBL
participants had signicantly greater improvements in academic achievement than comparison group participants.
5. Discussion
Due to relatively low engagement of vocational learners and the often stagnant instructional approaches adopted by vocational schools
in Taiwan (Lee et al., 2008), this study has proposed and evaluated a model of DGBL which integrates HOTS through a blended learning
environment with enhanced instructor support and scaffolding, provision of appropriate learning aids, and the use of collaborative learning.
These key factors address the gaps in the current system noted by recent studies and publications in Taiwan (Lee et al., 2010; Taiwan
Ministry of Education, 2013; Tsai et al., 2013) which emphasize the need for providing authentic, employment-related learning experiences. The signicant results of this study are promising and offer an alternative DGBL approach that can simultaneously foster all three
HOTS (creative thinking, critical thinking, and problem solving), while positively impacting academic achievement. Reasons for this
demonstrated effectiveness lie in the synergy created by blending digital game play with collaborative and constructivist activities focused
on integrating HOTS principles with course content for developing employment-related skills for vocational students. HOTS, while evaluated by total scores, are also discussed in terms of their specic subscales where appropriate.
5.1. Impact on creative thinking
Since opportunities for creative expression during commercial simulation games is limited, the effectiveness of the proposed approach to
DGBL on creative thinking can be explained by the opportunities for authentic, collaborative creativity provided by the integrative activities.
These activities scaffolded learners in both creating content for use in the game (i.e., through the design of a team logo or a mobile phone
product) and in evaluating genuine examples of creative works used in the marketplace (i.e., in evaluating the Taiwan Excellence Awards
logo). Thus, learners were able to integrate the course content, such as concepts from marketing or advertising, with game play while
constructing a creative product which would be applicable to employment-related contexts as well (Taiwan Ministry of Education, 2013).
Fig. 2 provides an example of a logo design activity which was intended to build team cohesion during the process of game play which
involved the determination of a team name, and the selection of a company type and location. Thus, the creative decisions being made in the
game were supported by classroom activities which allowed for the exploration of multiple options (uency subscale) for determining

Y.-T.C. Yang / Computers & Education 81 (2015) 281e295

291

Fig. 2. Creative thinking activity: Illustration of game play and integrated HOTS activity for Week 5.

unique, team-specic products (originality subscale) such as a logo and chant. Other creative dimensions, such as elaboration, abstractness
of title, and creative strengths, such as expressiveness and storytelling, were included as factors for group members to consider in determining the chant, motto, or logo selected for their team. These activities also served to motivate students, which is particularly important
since Taiwanese vocational learners often suffer from low motivation (Lee et al., 2010).
Since the comparison group participants explored multimedia and collaboratively designed presentations, created advertisements, and
organized materials digitally, an improvement in creative thinking was observed between the pretest and posttest. However, this
improvement was not signicant, suggesting that these activities did not have the same sense of authenticity and practicality as the
experimental group activities, which were directly linked to the challenge, fantasy, and curiosity evoked by the digital games (Malone, 1982).
Although game play itself likely provided some advantages over TEL in terms of opportunities for creativity, the use of a blended game-based
learning environment with HOTS integrative activities as aids and scaffolds for higher order thinking offered the opportunities necessary to
think creatively for and about game play while emphasizing the application of employment-oriented skills and knowledge.
5.2. Impact on critical thinking
Opportunities for critical thinking were provided by digital game play, which required players to evaluate the relative impact of market
conditions, competition, and their own decisions and strategies on game outcomes (induction and deduction subscales). Since vocational
learners often lack motivation and engagement, this study adopted several support mechanisms for learning: a) a blended environment
which offered personalization and feedback, b) scaffolding and teacher orchestration of activities, c) the provision of learning aids to assist in
acquiring and applying the course content, and d) peer support through a collaborative learning context. As such, groups of learners were
provided with the tools and support necessary to think critically about their decisions in the game, and reect upon their progress and any
difculties they faced, critical thinking factors which are important for employability (Kiener et al., 2014).
One example of an activity which supported learners in critically evaluating their decisions in the simulation is shown in Fig. 3. When
presented with a balance sheet and income statement for each of the different products selected by the group, the HOTS integrative activity
guided learners in evaluating how uctuations in cost associated with inventory and manufacturing variables impacted their protability
(interpretation subscale). Further, this form required learners to evaluate the most probable reason for cost uctuations before making a
nal decision as to which product to manufacture in the next stage of game play (evaluation of arguments subscale). In preparation for the
competition of the following week, this handout provided an external learning aid which enhanced learners' ability to remember and record
key information in order to make logical decisions (Amory et al., 1999). In other activities, the learners were prompted to consider a
consumer-oriented as compared to a producer-oriented approach when evaluating the role of the market in supply and demand (recognition of assumptions subscale).

292

Y.-T.C. Yang / Computers & Education 81 (2015) 281e295

Fig. 3. Critical thinking activity: Illustration of game play and integrated HOTS activity for Week 9.

The fact that critical thinking was not enhanced through TEL for the comparison group can be explained by the fact that fostering critical
thinking generally requires a fair amount of explicit instruction and practice in developing these skills. TEL activities did offer learners the
opportunity to engage in debate, compare and contrast alternative perspectives, and evaluate authentic case studies, however these activities were not translated into improved performance the CTT-I test of critical thinking. It appears that clearly labeling critical thinking
elements on the integrative learning activities used for the experimental group, in conjunction with the intrinsic challenges of digital game
competition, offered experimental group participants a signicant advantage in logically and reectively forming judgments. As Kiili (2005)
states, the challenging task of educational game designers is to develop the sort of game worlds that support reective thinking in the
private world (p. 22). The results of this study suggest that, for vocational students who are facing entry into the workplace, business
simulation games can provide this type of game environment, with the augmentation of teacher support and learning aids, in order to
address real-world needs and impact learners personally.
5.3. Impact on problem solving
Problem solving is the higher order thinking skill most directly associated with DGBL and most targeted by game designers (Li et al.,
2013). Opportunities for problem solving were plentiful in both CEO and Capitalism 2. Both gamers required learners to identify problems (whether in terms of demographic and geographic variables, market demands, resource prices, or competition) and solve these
problems through adjusting their gaming strategies. In addition, by repeating certain game levels, learners were required to prevent or avoid
future problems by relying upon information obtained from past decisions.
However, while these opportunities for problem solving during game play allowed for direct feedback in terms of the effectiveness of the
solution (i.e., the effectiveness category evaluated by the NTPS), learners' exibility in problem solving required prompting from integrative
activities. One example of such an activity was the SWOT analysis activity implemented during week 12 of the experiment. This activity,
illustrated in Fig. 4, asked learners to list as many potential strengths, weaknesses, opportunities, and threats identied during that semester
of game play. Thus, through the team rankings and reports provided by the game, players were further instructed to consider multiple
potential factors which led to their relative success or failure during the competition portion of the game.
The TEL condition also emphasized elements of problem solving through the exploration of case studies, the extensive use of discussion
and debate on issues related to store planning and management, and the requirement that learners collaborate in all learning tasks.
However, these tasks were not effective in promoting the development of effective and exible solutions to problems, since they were not

Y.-T.C. Yang / Computers & Education 81 (2015) 281e295

293

Fig. 4. Problem solving activity: Illustration of game play and integrated HOTS activity for Week 12.

grounded in an ongoing and sustained context, but were focused on textbook content which drew from multiple sources and topics.
Conversely, the digital games served as sustained narratives for contextualizing problems (Warren et al., 2012), causally linking game
strategies with game outcomes, and ensuring cognitive and affective engagement with the problem-solving tasks. Thus, in this study, the
use of DGBL as the principle instructional strategy provided an environment for simulating actual employment-related problems (Siewiorek
et al., 2012), which proved effective for promoting problem solving in vocational learners. This is signicant, since problem solving is a key
skill required by employers, and can be relevant to industries when students are seeking employment (Taiwan Ministry of Education, 2013).

5.4. Impact on academic achievement

The signicant and substantial improvement demonstrated by the experimental group in terms of academic achievement exceeded the
expectations of the researcher. While the pretest scores were very low for both groups, which is often typical for vocational students in
Taiwan (Lee et al., 2008; Tsai et al., 2013), it should be emphasized that these pretest scores reect student achievement under typical,
lecture-based instruction during the previous semester. It is noteworthy that with the introduction of both TEL and DGBL, student scores
improved signicantly, suggesting that innovative approaches to instruction are essential to vocational education reform. However, the
DGBL condition was signicantly more effective in promoting learner achievement and, by implication, the development of employmentrelated skills and knowledge. This nding conrms the suggestion by Ebner and Holzinger (2007) and Siewiorek et al. (2012) that digital
games provide greater opportunities for in-depth learning of subject matter by simulating authentic experiences through realistic settings
that allow for exploration of solutions without the risk of real life consequences. Of particular importance for vocational learners is the fact
that the setting of the game was relevant to potential employment opportunities that they would be seeking (i.e., retail and sales jobs).
The effectiveness of the specic HOTS-integrated approach to DGBL adopted by this study may also be attributable to the provision of
learning aids, teacher scaffolding and orchestration, and opportunities for collaborative learning ensured by the blended environment for
game play. However, the fact that the comparison group adopted TEL with similar instructional elements suggests that game play itself is
integral to the success of the intervention. Thus, while the integrative activities were useful in linking specic HOTS to game play and course
content, the role of the digital game as a form of simulation served as a principle mechanism for rehearsing and applying course-specic
knowledge. Future studies should compare other types of DGBL with HOTS-integrated DGBL in order to evaluate the additional benet
provided by the use of teacher facilitation, scaffolding, and learning aids, since these factors are likely important for relieving the anxiety of
vocational learners and fostering their motivation and self-regulation.

294

Y.-T.C. Yang / Computers & Education 81 (2015) 281e295

Table 6
Summary of one-way MANCOVA results for creative thinking (TTCT), critical thinking (CTT-I), problem solving (PS) and academic achievement (AA).
DV

SV

SS

df

MS

Partial h2

TTCT
CTT-I
PS
AA

Group

91,640.46
76.79
2734.21
8389.02

1
1
1
1

91,640.46
76.79
2734.21
8389.02

38.22
13.85
19.93
61.95

.00*
.00*
.00*
.00*

.38
.18
.24
.50

*p < .01.

6. Conclusions
The results of this study demonstrate how a blended approach to DGBL which emphasizes teacher scaffolding and orchestration, provision of learning aids, and use of collaboration can empower vocational learners with improved higher order thinking and greater academic
achievement. While TEL also provided tangible benets for learners, who would otherwise have been taught with traditional, lecture-type
instruction, the synergy of DGBL with integrative activities for emphasizing HOTS was deemed more effective. Implications for both teachers
and researchers are provided in the following subsections (Table 6).
6.1. Implications for teachers
The effectiveness of the proposed approach to DGBL offers several suggestions for classroom teachers, particularly in light of the mixed
m
results for other types of DGBL approaches overall (Connolly et al., 2012) and specically, in terms of vocational education (Ha
al
ainen &
Oksanen, 2012). A review of the literature led to certain key suggestions related to the implementation of DGBL for fostering higher order
thinking and deep learning, which are summarized in the Methods section and linked to specic outcomes in the Discussion section. The
implication for instructors is that commercially available digital games can be an effective platform for fostering learner thinking and academic achievement, but that the implementation of DGBL in a blended environment, focused on scaffolding learners over an extended
period of time, offers greater possibilities for learning. Simple, integrative activities which challenge learners to think creatively, critically, or
with a problem-solving perspective can enhance the digital game experience. By blending digital game play with collaborative in-class
group activities, specic skills and knowledge (in this case HOTS) can be targeted. The fact that all three HOTS were signicantly
improved also suggests that a holistic approach to higher order thinking can be benecial, as the impact of DGBL on one thinking skill may
translate to improvements in terms of other thinking skills, when opportunities are presented for their application.
6.2. Limitations and implications for researchers
Suggestions for future research include the evaluation of traditional DGBL as compared to blended approaches, the adoption of more
precise and appropriate measures of employment-related skills, and the extension of these results to other contexts. First, while this study
demonstrates the effectiveness of the proposed model of DGBL, with signicant results for all three HOTS, the comparison group utilized TEL
rather than traditional DGBL. Therefore, the additive effect of the integrative learning activities over other forms of DGBL cannot be
quantied. Future studies adopting alternative forms of DGBL could provide further evidence of which features of the blended learning
environment are most benecial.
Additionally, while the purpose of this research was to foster learners' employability, this goal was only partially achieved. That is, while
HOTS are important to a learner's future employment, no direct measures of employability were available. Future studies can evaluate the
effects of DGBL on vocational learners' performance in actual workplace settings, possibly by including a form of practicum in the research
design or performing follow-up evaluation of graduates' employment status, salary, or employer evaluation. Likewise, the extension of this
type of DGBL to non-vocational contexts or vocational higher education could further the literature in this area of research.
Acknowledgments
The funding for this research was provided by the National Science Council of Taiwan, under grant NSC 102-2511-S-006 -004 -MY3.
References
Amory, A., Naicker, K., Vincent, J., & Adams, C. (1999). The use of computer games as an educational tool: Identication of appropriate game types and game elements. British
Journal of Educational Technology, 30(4), 311e321.
Boahin, P., & Hofman, A. (2013). A disciplinary perspective of competency-based training on the acquisition of employability skills. Journal of Vocational Education and Training,
65(3), 385e401.
Bourgonjon, J., De Grove, F., De Smet, C., Van Looy, J., Soetaert, R., & Valcke, M. (2013). Acceptance of game-based learning by secondary teachers. Computers & Education, 67,
21e35.
Chan, S. M., & Wu, W. T. (2007). New problem solving ability test. Taipei, TW: Psychological Publishing Press.
Chang, Y. C., Chen, W. C., Yang, Y. N., & Chao, H. C. (2009). A exible web-based simulation game for production and logistics management courses. Simulation Modelling
Practice and Theory, 17, 1241e1253.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum Associates.
Connolly, T. M., Boyle, E. A., MacArthur, E., Hainey, T., & Boyle, J. M. (2012). A systematic literature review of empirical evidence on computer games and serious games.
Computers & Education, 59(2), 661e686.
Ebner, M., & Holzinger, A. (2007). Successful implementation of user-centered game based learning in higher education: an example from civil engineering. Computers &
Education, 49(3), 873e890.
Erhel, S., & Jamet, E. (2013). Digital game-based learning: Impact of instructions and feedback on motivation and learning effectiveness. Computers & Education, 67, 156e167.
ma
l
Ha
ainen, R., & Oksanen, K. (2012). Challenge of supporting vocational learning: Empowering collaboration in a scripted 3D game e how does teacher' real-time
orchestration make a difference? Computers & Education, 59(2), 281e293.

Y.-T.C. Yang / Computers & Education 81 (2015) 281e295

295

ma
la
inen, R., Oksanen, K., & Ha
kkinen, P. (2008). Designing and analyzing collaboration in a scripted game for vocational education. Computers in Human Behavior, 24(6),
Ha
2496e2506.
Inayat, I., ul Amin, R., Inayat, Z., & Salim, S. S. (2013). Effects of collaborative web based Vocation Education and Training (VET) on learning outcomes. Computers & Education,
68(10), 153e166.
Kebritchi, M., & Hirumi, A. (2008). Examining the pedagogical foundations of modern educational computer games. Computers & Education, 51(4), 1729e1743.
Kiener, M., Ahuna, K. H., & Tinnesz, C. G. (2014). Documenting critical thinking in a capstone course: Moving students towards a professional disposition. Educational Action
Research, 22(1), 109e121.
Kiili, K. (2005). Digital game-based learning: towards an experiential gaming model. Internet and Higher Education, 8, 13e24.
Kiili, K. (2007). Foundation for problem-based gaming. British Journal of Educational Technology, 38(3), 394e404.
Lainema, T., & Nurmi, S. (2006). Applying an authentic, dynamic learning environment in real world business. Computers & Education, 47(1), 94e115.
Lee, T. H., Shen, P. D., & Tsai, C. W. (2008). Applying web-enabled problem-based learning and self-regulated learning to add value to computing education in Taiwan's
vocational schools. Educational Technology and Society, 11(3), 13e25.
Lee, T. H., Shen, P. D., & Tsai, C. W. (2010). Enhance low-achieving students' learning involvement in Taiwan's higher education: an approach via e-learning with problembased learning and self-regulated learning. Teaching in Higher Education, 15(5), 553e565.
Li, Z. Z., Cheng, Y. B., & Liu, C. C. (2013). A constructionism framework for designing game-like learning systems: Its effect on different learners. British Journal of Educational
Technology, 44(2), 208e224.
Lin, Y. L., & Tu, Y. Z. (2012). The values of college students in business simulation game: a means-end chain approach. Computers & Education, 58(4), 1160e1170.
Liu, C. C., Cheng, Y. B., & Huang, C. W. (2011). The effect of simulation games on the learning of computational problem solving. Computers & Education, 57(3), 1907e1918.
Malone, T. W. (1982). Heuristics for designing enjoyable user interfaces: Lessons from computer games. In Proceedings of the 1982 Conference on Human Factors in Computing
Systems (pp. 63e68). ACM.
Papastergiou, M. (2009). Digital Game-Based Learning in high school Computer Science education: Impact on educational effectiveness and student motivation. Computers &
Education, 52(1), 1e12.
Partnership for 21st Century Skills. (2009). Framework for 21st century learning. Retrieved from http://www.p21.org/our-work/p21-framework.
Pasin, F., & Giroux, H. (2011). The impact of a simulation game on operations management education. Computers & Education, 57(1), 1240e1254.
Ray, B., Faure, C., & Kelle, F. (2013). Using Social Impact Games (SIGs) to support constructivist learning: creating a foundation for effective use in the secondary social studies
education. American Secondary Education, 41(2), 60e70.
Siewiorek, A., Saarinen, E., Lainema, T., & Lehtinen, E. (2012). Learning leadership skills in a simulated business environment. Computers & Education, 58(1), 121e135.
Tabachnick, B. G., & Fidell, L. S. (2012). Using multivariate statistics (6th ed.). Boston, MA: Pearson.
Taiwan Ministry of Education. (2013). Reconstruction project for technical and vocational education: Second phase (Report No. 1020052561) Retrieved from http://www.ey.gov.
tw/News_Content.aspx.
Tsai, C. W., Lee, T. H., & Shen, P. D. (2013). Developing long-term computing skills among low-achieving students via web-enabled problem-based learning and self-regulated
learning. Innovations in Education and Teaching International, 50(2), 121e132.
Torrance, E. P. (1981). Predicting the creativity of elementary school children (1958e80) and the teacher who made a difference. Gifted Child Quarterly, 25, 55e62.
Torrance, E. P. (1998). The Torrance tests of creative thinking normsdtechnical manual gural (streamlined) forms A & B. Bensenville, IL: Scholastic Testing Service, Inc.
Tukey, J. W. (1977). Exploratory data analysis. Reading, MA: Addison-Wesley Publishing Company.
Wall, J., & Ahmed, V. (2008). Use of a simulation game in delivering blended lifelong learning in the construction industry e opportunities and challenges. Computers &
Education, 50(4), 1383e1393.
Warren, S. J., Dondlinger, M. J., McLeod, J., & Bigenho, C. (2012). Opening the Door: an evaluation of the efcacy of a problem-based learning game. Computers & Education,
58(1), 397e412.
Williams, P. (2008). Assessing context-based learning: not only rigorous but also relevant. Assessment and Evaluation in Higher Learning, 33(4), 395e408.
Wu, W. H., Chiou, W. B., Kao, H. Y., Hu, C. H. A., & Huang, S. H. (2012). Re-exploring game-assisted learning research: the perspective of learning theoretical bases. Computers &
Education, 59(3), 1153e1161.
Yeh, Y. C. (2003). Critical thinking test-level I (CTT-I). Taipei, Taiwan: Psychological Publishing.