Journal of Information Technology and Applications Vol. 2, No. 2, pp.

69-79, 2007

The Effect of Computer Simulation Instruction on Student Learning: A Meta-analysis of Studies in Taiwan
Yuen-kuang Liao* Department of Education, National Taiwan Normal University yliao@ntnu.edu.tw Yu-wen Chen Teacher, Ching Shin Elementary School, Taipei e0141@chjhs.tp.edu.tw

Abstract
A meta-analysis was performed to synthesize existing research comparing the effects of computer simulation instruction (CSI) versus traditional instruction (TI) on students achievement in Taiwan. Twenty-nine studies were located from four sources, and their quantitative data was transformed into Effect Size (ES). The overall grand mean of the study-weighted ES for all 29 studies was 0.54. The results suggest that CSI is more effective than TI in Taiwan. However, only 1 (reliability of measure) of the 17 variables had significant main effect on mean ES. The results of this study suggest that CSI clearly has a more positive effect on students learning than TI. The results also shed light on the debate between Clark and Kozma regarding learning from the media. Keywords: computer simulation, virtual reality, CAI, CAL, achievement, meta-analysis involvement in the learning process, and facilitated their practice and mastery of concepts and principles; clearly computer simulation helped students to meet their learning objectives or goals. Michael [7] pointed out that simulation programs such as Electronic Workbench, LegoCAD, and Car Builder are helping students learn about events, processes, and activities that either replicate or mimic the real world. According to Michael[7], computer simulation can afford learners numerous advantages. For example, computer simulations can (1) provide the students with the opportunity to engage in activities that may otherwise be unattainable, (2) enhance academic performance and the learning achievement levels of students, and (3) be equally as effective as real-life hands-on laboratory experiences. Chou [8], and Serpell[9] also noted the significantly greater effectiveness of computer simulation instruction as compared to traditional instruction. Slack & Stewartv[10], Johnson & Stewart [11], and Collins & Morrison [12] reported that by using genetics construction kits as part of a strategic computer simulation, undergraduate and high school students learned to solve genetics programs and to build accurate and rich mental models of genetic knowledge. However, Parker [13] and Tannehill [14] have found no significant differences between computer simulation instruction and traditional instruction. Hopkins [15], Hummel & Batty [16], and Tylinski [17] even reported an opposite finding: the significantly greater effectiveness of traditional instruction. Two review studies regarding the effectiveness of 69

1.

Introduction

Computer technology has been widely used in education for more than forty years. More specifically, computer simulation as an instructional technology has been commonly used in education [1], [2]. A study by Heinich, Molenda & Russell [3] reported that computer simulations were extensively used for job training in 95% of the Colleges of Management in the USA. In the colleges using CSI, 1/6 of the faculty and 1/4 of the total instructional time were given to computer-simulation-related activities. Faria [4] also found that more than 1700 business schools in the USA used computer simulation software for instruction; more than 200 different types of software were used for this purpose. Many potential benefits have been claimed for the use of computer simulation in teaching. For instance, Huppert, Yaakobi, & Lazarovvitz [5] have noted that In computer simulations, students have opportunities to receive supplemental contact with the variables tested in real experiences or dangerous ones. Students can be active during the simulated experiments by identifying the study problem, writing in their notebooks their hypotheses, planning and performing the simulated experiments, gathering results, collecting data in their notebooks, plotting these data back in the computer, and using the data for drawing tables and graphs. (p.232) Rivers and Vockel [6] also found in their study that computer simulations enhanced students active * Corresponding author.

Journal of Information Technology and Applications Vol. 2, No. 2, pp. 69-79, 2007 computer simulation have also been analyzed. A review conducted by Deckkers & Donatti [18] concluded that although simulations were more effective in the development of attitudes than lectures, it appeared that the claims of improved cognitive development and learning retention were not readily supported. Lees [19] review used a meta-analytic approach: it collected 19 studies and yielded 51 effect sizes. The study found that the overall mean effect size for academic achievement was .41, meaning about 66% of the students in computer simulation classes outperformed the average students from the control groups. On the other hand, the overall effect size for attitude was -.04, meaning control groups performed slightly better than computer simulation classes. Obviously, the results of these two reviews were inconsistent. It is assumed that these conflicting results may be due to the different sources used by the two review studies or their different methodologies. Computer simulations have been defined in different ways by different researchers. According to Alessi & Trollip [20], simulation is just one type among many of computer assisted instruction (CAI). Lee [19] defined simulation in a broad sense as a computer program which temporarily creates a set of images (items, objects) and connects them through cause-and-effect relations. Thomas & Hooper [21] defined computer-based instructional simulation as a computer program containing a manipulatable model of a real theoretical system. The program then enables the student to change the model from a given state to a specified goal-state by directing it through a number of intermediate states. Thus, the program accepts commands from the user, alters the state of the model, and when appropriate displays the new state (p.498). Virtual reality (VR) is a computer technology which combines computer graphics, computer simulation, and human-computer interfaces [22]. In one sense, VR shares some characteristics of computer simulation, such as the mimicking of real life and user-driven control. However, it is not the intention of this study to discuss the different definitions of computer simulation. For the purposes of the present meta-analysis, studies employing computer simulations or VRs as delivery systems for instruction were considered to be types of computer simulation given a broader definition of the term, and were thus included in the group of studies analyzed. simulation (CS) in Taiwan began with system/tool design. Two earliest studies (i.e., [24], [25]) with regard to CS were published in 1980s. However, most of these studies were published after 1996. The total number of studies counted for this category is 41, and the subject areas of these CS systems are diverse, including Physics, Chemistry, Mathematics, Erath Science, Geography, Statistics, Computer Science, Health Education, Physical Education, Architectures, etc. The earliest empirical study of CS was published in 1993 [26], while more than 80% of these studies were published after 2000. The grade levels of students participated in these studies ranged from elementary to graduate, and the subject areas studied are also wide-ranging.

1.2 Purposes of this Study

In spite of the many claims for the potential benefits of using computer simulation in education, the results of research comparing the effects of computer simulation instruction and those of traditional instruction in Taiwan are conflicting. For example, Chao[27], Chuang [28], Huang [29], Lin [30], Nein [31], and Su [32] all reported significant gains for CSI as compared with traditional instruction. On the other side, Chen [33], Jao [26], Tseng [34], and Yu [35] found no significant differences in the effectiveness of using CSI and traditional methods of instruction. Owing to the contradictory evidence provided by existing research in the area, and the fact that very little, if any, thorough quantitative synthesis of computer simulation instruction in Taiwan has been done, the present author thought it important to conduct a meta-analysis in order to clarify the above-mentioned research findings. Moreover, since 2 meta-analyses of CSI have been published in the USA [18],[19], and since this is the first meta-analysis of CSI to be conducted in Taiwan, the synthesis of previous research undertaken here not only examines the accumulated research-based effects of computer simulation on students' learning efficiency in Taiwan, but also provides a comparative view of meta-analyses of CSI in Taiwan and the USA.

2. Procedures
The research method used in this study is a meta-analytic approach similar to that suggested by Kulik, Kulik, & Bangert-Drowns[36]. Their approach requires a reviewer to (a) locate studies through objective and replicable searches; (b) code the studies for salient features; (c) describe outcomes on a common scale; and (d) use statistical methods to relate study features to outcomes [36]. Their method differs from that of Glass, McGaw, & Smith [37]: in Kulik et als approach each individual study, defined as the set of results from a single publication, was weighted equally with all the other studies, so that the problem 70

1.1 The Development of Computer Simulation in Taiwan

The development of CAI in Taiwan has moved from the development of traditional courseware for mainframe computers to Windows-based CAI, then to multimedia CAI, and finally to web-based CAI (see details in [23]). As part of the process of development of CAI in Taiwan, the development of computer

Journal of Information Technology and Applications Vol. 2, No. 2, pp. 69-79, 2007 of aggregate multiple effect sizes from a single study could be avoided. The purpose of the present study was then to synthesize and analyze the existing research on the effects of these two instructional approaches. It was necessary to define these approaches precisely in order to ensure the proper selection of appropriate studies. Computer Simulation Instruction (CSI) was taken to refer to classes using computer simulation as a replacement for or supplement to traditional instruction in order to teach students. Traditional Instruction (TI) was taken to refer to classes using traditional methods of instruction, that is, non-computer-based methods, to teach students. analysis, outcomes from a variety of different studies using a variety of different instruments had to be expressed on a common scale. The transformation used for this purpose was the one recommended by Glass et al [37] and modified by others (e.g., Hunter, Schmidt, and Jackson [38]). To reduce measurements to a common scale each outcome was coded as an Effect Size (ES), which is defined as the difference between the mean scores of two groups divided by the standard deviation of the control group. For those studies that did not report means and standard deviations, F values or t values were used to estimate the ES; these formulas are presented in Table 1. Also, with studies that employed a one-group pretest-posttest design, in which there was no control group, an alternative approach suggested by Andrews, Guitar, and Howie [39] was used. In their approach the ES is estimated by comparing the post-treatment mean with the pre-treatment mean, and then dividing by the pre-treatment standard deviation. Table 1: Formula Used in Calculating Effect Size ___________________________________________ Mean and SD ES =MxMc/ SDc t - value ES = t 1 / N1 + 1 / N 2 F - Value ES = F 1 / N1 + 1 / N 2
Note. ES = Effect size. Mx = mean for the experimental group. Mc = mean for the control group. SDc = standard deviation of the control group. N1 = number of subjects in the experimental group. N2 = number of subjects in the control group.

2.1 Data Sources

The studies considered for use in this meta-analysis came from four sources. One large group of studies came from computer searches of the Chinese Periodical Index. A second group of studies came from the Dissertation and Thesis Abstract System of Taiwan. A third group of studies was retrieved from the Government Research Bulletin (GRB) of Taiwan. The last group of studies was retrieved via the bibliographies of the documents located. Twenty-nine studies were located through these search procedures: 25 came from the Dissertation and Thesis Abstract System, 2 from National Science Council (NSC) research projects, and the other 2 were retrieved from published journals and proceedings. Several criteria were established for inclusion of studies in the present analysis. 1. Studies had to compare the effects of CSI versus TI on students achievement. 2. Studies had to provide quantitative results from both CSI and TI classes. 3. Studies had to be retrievable from university or college libraries by interlibrary loan, from the GRB, or from Taiwans Dissertation and Thesis Abstract System. 4. Studies had to use Taiwans students as subjects. There were also several criteria for eliminating studies or reports cited by other reviews: (a) studies did not report sufficient quantitative data in order to estimate Effect Sizes; (b) studies reported only correlation coefficients - the r value or Chi-square value; (c) studies could not be obtained through interlibrary loans or from standard clearinghouses.

In most cases, the application of the formula given by Glass and his colleagues was quite straightforward. But in some cases, more than one value was available for use in the ES formula. Thus, when some studies reported differences in both posttest measures and pre-post gains, and some studies reported both raw-score differences and covariance-adjusted differences between groups, the pre-post gains and covariance-adjusted differences were selected for estimating ES. In addition, several subscales and subgroups were used in measuring a single outcome: for example, those that reported separate data by gender or grade. In such cases, each comparison was weighted in inverse proportion to the number of comparisons within the study (i.e. 1/n, where n = the number of comparisons in the study) so that the overweighting of the ES in a given study could be avoided (see, for example, [40], p. 230).

2.2 Outcome Measures

The instructional outcome measured most often in the 29 studies was student learning, as indicated on standard or researcher-developed achievement tests at the end of the instructional program. For statistical 71

2.3 Variables Studied

Seventeen variables were coded for each study in the present synthesis. These variables are listed in Table 2. Each of these variables was placed in one of the following sets of characteristics: (a) study

Journal of Information Technology and Applications Vol. 2, No. 2, pp. 69-79, 2007 characteristics, (b) methodological characteristics, and (c) design characteristics. The first 3 variables in the set of study characteristics were coded so that potentially different effects for subjects with different backgrounds could be detected. The other 2 variables (i.e., type of publication and year of publication) in the set of study characteristics were coded because it is important to know how effects are related to sources of information over time. The 6 variables placed in the set of methodological characteristics were coded so that effects related to the characteristics of research procedures could be detected. The last 6 variables in the set of design characteristics were coded because it is critical to know how effects are related to the nature and design of the primary research. Each variable was employed as a factor in an analysis of variance (ANOVA), used to investigate whether there were significant differences within each variable on the ES. Table 2: The Assignments of Studied Variables in Each Characteristic ___________________________________________ Characteristics Variables ___________________________________________ Study Characteristics Grade Level Location of School Subject Area Type of Publication Year of Publication Methodological Characteristics Instructor Bias Instrumentation Reliability of Measure Sample Size Selection Bias Type of Research Design Design Characteristics Comparison group Duration of Treatment Implementation of Innovation Type of Instruction for Treatment Type of Innovation Visual Presentation .

3. Results
The number of comparisons and the study-weighted ESs (defined as the overall ES for a single study) are reported in Table 3. Of the 29 studies included in the present synthesis, 27 (93%) of the study-weighted ESs were positive and favored the CSI group, while 2 (7%) of them were negative and favored the TI group. The range of the study-weighted ESs was from -0.197 to 2.67. The overall grand mean for all 29 study-weighted ESs was 0.537. When this mean ES was converted to percentiles, the percentiles indicating student achievement were 70 for the CSI group and 50 for the TI group. The overall grand median for all 29 study-weighted ESs was 0.373, suggesting that percentiles indicating student achievement were 64 for the CSI group and 50 for the TI group. The standard deviation of 0.573 reflects the mild variability in ESs across studies. Table 3: Number of Comparisons and Study-weighted Effect Sizes ________________________________________ Author(s) Year N of ES Comparison Chang, H. P. [41] 2001 2 0.273 Chao, J. H. [27] 2001 1 0.645 Chao, J. T. [42] 1999 10 0.322 Chen, C. H. [33] 2002 3 -0.083 Chen, C. [43] 2003 1 0.262 Chen, H. [44] 2005 2 0.207 Chen, T. [45] 1998 1 0.373 Chen, Y. S. [46] 2002 2 0.238 Chuang, C. F. [28] 2000 1 1.964 Guan, H. D. [47] 1999 1 0.428 Hsu, Y. S. et al [48] 2001 1 0.302 Hsu, Y. S. [49] 2002 1 0.370 Huang, C. K. [50] 2002 3 0.433 Huang, J. C. [29] 2002 1 1.165 Jao, Y. H. [26] 1993 2 0.144 Lai, Y. J. [51] 2002 1 -0.197 Li, M. [52] 2005 2 0.837 Lin, C. [53]] 2004 1 0.410 Lin, U. C. [30] 2002 1 2.670 Lin, Y. [54] 2005 1 0.849 Nein, J. S. [31] 2002 4 0.531 Shen, S. [55] 2005 1 0.650 Su, C. Y. [56] 2000 1 0.335 Su, J. S. [32] 2002 15 0.511 Tseng, C. [34] 2002 2 0.084 Wang, C. [57] 2005 1 0.716 Wang, K. K. [58] 1994 3 0.290 Wang, T. [59] 2005 1 0.543 Yu, J. [35] 2002 1 0.144 Overall grand mean 0.537 Overall grand SD 0.573 Overall grand median 0.373 95% Confidence interval 0.319~0.755 72

2.4 Coder Reliability

To obtain more reliable outcomes from coding, the author of this study and 2 research assistants coded the studies. Each of the 2 research assistants coded half of the studies on each of the independent variables. To check for accuracy, the author coded each of the studies independently. The inter-coder agreement for the studies coded by coders was 85%. In addition, the different codings (i.e. inter-coder differences) in studies handled by two coders were discussed. Final agreement had to be reached after discussion.

Journal of Information Technology and Applications Vol. 2, No. 2, pp. 69-79, 2007 Range -0.197~2.670 Positive ES (%) 27(93%) Negative ES (%) 2(7%) Note. Total N = 2729. Total N of studies = 29. Total N of comparisons = 67 Among the 67 ESs included in the present synthesis, 59 (88%) were positive and favored the CSI group, while 8 (12%) were negative and favored the TI group. The range of ESs was from -0.691 to 2.67. The ESs for the 67 comparisons are displayed in a scatter diagram in Figure 1. This diagram shows that despite several large effects, most of the ESs were small to moderate in magnitude. About 66% of the ESs lie between -0.5 and 0.5, while less than 31% of the ESs were greater than 0.5. Table 4 lists the F values for the 17 variables for all study-weighted ESs in the study. Descriptive statistics for the 17 variables are presented in Table 5. Among the 17 variables, only 1 variable (reliability of measure), showed statistically significant impact. The post hoc test (Scheffe), F(2,26)=3.88, p<.05, showed that the mean comparison of studies in which the reliability of measure was coded as adequate indication was higher than the studies coded as actual reliability figure or unspecified. There were no significant differences found between the mean comparison of studies coded as actual reliability figure or unspecified. However, since there were only 1 studies that coded as adequate indication, this result may be considered tentative. Methodological Characteristics Instructor bias 3,25 Instrumentation 2,26 Reliability of measure 2,26 Selection bias 2,26 Sample size 2,26 Type of research design 3,25 Design Characteristics Comparison group Duration of treatment Implementation of innovation Type of instruction for treatment Type of innovation Visual presentation *p<.05 0.842 0.484 0.281 0.757 3.880 0.034* 0.648 0.532 0.466 0.633 0.248 0.862

1,27 3,25 1,27 4,24 1,27 2,26

0.018 0.255 0.118 0.352 0.125 0.298

0.895 0.857 0.733 0.840 0.726 0.745

Table 5: Means and Standard Deviations of Study-weighted ESs for Coded Variables ___________________________________________ Variables N % ES SD Study Characteristics Grade Level 1st - 6 th 13 44.8 7th - 9th 5 17.2 10th 12 th 8 27.6 College 3 10.3 Location of School North of Taiwan 9 31.0 Center of Taiwan 5 17.2 South of Taiwan 11 37.9 East of Taiwan 1 3.4 Unspecified 3 10.3 Subject Area Science 13 44.8 Physics & Chemistry 3 10.3 Engineering 5 17.2 Math 1 3.4 Others 7 24.1 Type of Publication Dissertation 25 86.2 Others 4 13.8 Year of Publication 1993~1999 5 17.2 2000~2002 16 55.2 2003~2005 8 27.6 Methodological Characteristics Instructor Bias Same 13 44.8 Different 2 6.9 No Instructor 4 13.8 Unspecified 10 34.5 Instrumentation Local 25 86.2 73

0.555 0.493 0.411 0.873 0.337 0.791 0.619 0.207 0.419 0.559 0.505 0.559 0.084 0.561 0.506 0.731 0.311 0.597 0.559

0.689 0.385 0.286 0.962 0.148 0.670 0.740 N/A 0.690 0.685 0.627 0.217 N/A 0.629 0.540 0.823 0.107 0.749 0.248

3.0 2.5 2.0 1.5 effect size 1.0 0.5 0.0 0 -0.5 -1.0 case 10 20 30 40 50 60 70 80

Figure 1: The Scatter Diagram of Effect Size Table 4: Results of ANOVAs for Coded Variables Variables Study Characteristics Grade level Location of school Subject area Type of publication Year of publication df 3,25 4,24 4,24 1,27 2,26 F 0.459 0.566 0.148 0.523 0.464 p 0.713 0.690 0.962 0.476 0.634

0.474 0.209 0.369 0.753 0.570

0.319 0.091 0.054 0.889 0.612

Journal of Information Technology and Applications Vol. 2, No. 2, pp. 69-79, 2007 Standardized 1 Unspecified 3 Reliability of Measure Actual reliability figure 15 Adequate Indication 1 Unspecified or inadequate 13 Sample Size 0~60 10 61~120 12 Over 121 7 Selection Bias Adequately minimized 11 Probably threat 14 Unspecified 4 Type of Research Design One group repeated measure 5 Posttest only control group 1 Nonequivalent control group 20 Static group comparison 3 Design Characteristics Comparison Group Traditional instruction 22 No comparison group 7 Duration of Treatment 0~6 Hours 16 6~24 Hours 4 24~96 Hours 6 Unspecified 3 Implementation of Innovation Replacement for usual instruction 24 Supplement to instruction 5 Type of Innovation Computer Simulation 19 Virtual Reality 10 Type of Instruction for Treatment Large group 3 Small group (less than 5 persons4 In a group) Individual 10 Mixed 1 Unspecified 11 Visual Presentation 3.4 10.3 0.373 0.321 N/A 0.044 2-D 3-D Unspecified 8 13 8 27.6 44.8 27.6 0.404 0.606 0.559 0.386 0.680 0.584

51.7 3.4 44.8 34.5 41.4 24.1

0.73 1.964 0.503 0.496 0.654 0.397

0.639 N/A 0.337 0.219 0.859 0.236

4. Discussion
The results of this meta-analysis indicate that CSI has moderately positive effects on students achievement in comparison with the effects of TI. An effect is said to be medium when ES = 0.5 and large when ES = 0.8 [60]. The effectiveness of CSI is also confirmed by 93% of the positive study-weighted ES values and 88% of the positive ESs overall. These results are consistent with Lees [19] meta-analysis. The moderateness of the effect must be kept in mind, however; the overall study-weighted mean ES of 0.537 only indicates 20 percentile scores higher than those for the TI group. The percentile scores for the overall grand mean and median were 70 and 64, respectively. The difference of 6 percentile points may possibly be attributed to the mild overall grand standard deviation (0.573). This analysis of the variables suggests some interesting trends in the accumulated research base and is discussed in the following sections.

37.9 48.3 13.8

0.469 0.654 0.317

0.754 0.486 0.044

17.2 3.4 69.0 10.3

0.426 0.144 0.586 0.529

0.134 N/A 0.677 0.272

4.1 Study Characteristics

For the grade-level variable, there was no significant difference in mean ES. However, the smallness of the ES associated with high school subjects (10th 12th graders) is probably due to the fact that these students have to study very hard for a nationwide college entrance examination in Taiwan and using CSI may not be the most sensible approach for achieving the primary goal, which is passing this examination. It is also possible that different instructional approaches were used for these students as compared to the other students. More studies need to be conducted to clarify this variable. CSI studies conducted to measure students achievement tend to focus on specific subject areas. Studies included in the present meta-analysis were spread over a wide range of subject areas. The fact that more than 60% of the studies examined the effects of CSI in the teaching of science (including Physics & Chemistry) or Engineering suggest that CSI is more often used for teaching these two subject areas in Taiwan. In addition, the various subject areas examined also suggest that CSI may potentially be implemented in many different subject areas. However, one subject area (math) showed the smallest ES (0.084), suggesting that the effects of CSI may vary over different subject areas. The source of studies in a meta-analysis is always an important factor to be examined [37], [61]. The fact that approximately 85% of the studies were located in dissertations/theses and only 14% were from journals and research projects was predictable for Taiwan. It is 74

75.9 24.1 55.2 13.8 20.7 10.3

0.546 0.512 0.520 0.506 0.699 0.348

0.640 0.312 0.647 0.233 0.640 0.463

82.8 17.2

0.554 0.456

0.627 0.158

65.5 34.5

0.510 0.590

0.454 0.777

10.3 13.8 34.5 3.4 37.9

0.505 0.268 0.682 0.543 0.512

0.627 0.145 0.510 N/A 0.742

Journal of Information Technology and Applications Vol. 2, No. 2, pp. 69-79, 2007 probably due to the limitation on length for papers published in most journals; also, a large number of educational journal articles in Taiwan do not report detailed quantitative data, which restrains them from being included in a meta-analysis. The larger ES associated with published articles is typical in meta-analysis [37], p. 227. The year-of-publication variable in the meta-analysis allows an assessment of the effect of CSI over time. More than half of the studies located were published between 2000 and 2002, suggesting that CSI studies have just become more popular in the past seven years in Taiwan; it is expected that more studies will be published soon. As for comparison groups, the mean ES for TI studies was slightly higher than for studies with no comparison group, (e.g. studies in which the same group is repeatedly measured). This result suggests the greater effectiveness of CSI as compared with TI. In one sense, this is unusual, because students in TI classes do have certain learning activities, but in No Instruction classes there is no learning activity for students. Why No Instruction had better academic outcomes than TI, when both were compared with CSI, poses a serious question for researchers in Taiwan. Duration of treatment is usually a critical variable in meta-analysis. Clark [62], after reviewing several meta-analyses of CAI, suggested that the effects of new media to instruction were due to a novelty effect, because the ES was reduced when treatment lasted for longer period of time. Liao and Bright [63] also reported this novelty effect in their meta-analysis of the effects of programming on students cognitive abilities. Although no significant difference was found for this variable, the results of this synthesis do not quite support the previous viewpoint regarding the novelty effect. The largest mean ES was associated with studies lasting 24 96 hours, while the smallest mean ES was connected with studies lasting 6 -24 hours. There may be some unknown effects related to the duration of treatment that influence students CSI outcomes in Taiwan and the US, and these may lead to the distinctly different outcomes of their respective meta-analyses. More cross-national comparative studies need to be done to clarify this issue. It is usually difficult for teachers to decide whether to use a new innovation as a replacement for their usual means of instruction or as a supplement to their instruction. In most cases, the decision depends on which approach can provide more effective outcomes. The difference in mean ES between studies coded as replacements and those coded as supplements was quite small, suggesting that in Taiwan replacement CSI and supplemental CSI are equally effective means of enhancing student achievement. As for the type of innovation, the difference in mean ES between computer simulation and virtual reality (VR) was trivial, suggesting that computer simulation and VR are equally effective means of improving students learning outcomes. Of the 29 studies included, 10 (35%) of them employed individual instruction for the CSI classes, 4 (14%) were for small group instruction, and only 3 (10%) studies were for large group instruction. Although no significant difference was found for this variable, the differences in mean ES among studies using individual (ES = 0.682), small group (ES = 0.268), and large group (ES = 0.505) instruction were noticeable. These results suggest that CSI may be more effective if it is implemented in individual or large group settings in Taiwan. However, given that over 1/3 of the studies were coded as unspecified, this 75

4.2 Methodological Characteristics

After reviewing several meta-analyses of media comparison research, Clark [62] suggested that the positive effects of media seemed to be the uncontrolled effects of instructional method or content differences between treatments that were compared; he concluded that effects more or less disappeared when the same instructor delivered all treatments. For the present synthesis, although no significant differences in ES were found, the results show that studies using the same instructors for treatments had visibly higher ES than those using different instructors. This finding clearly indicates that the positive effects of CSI as compared with TI should not be confused with the uncontrolled effects of instructional method noted by Clark. However, more than 1/3 of the studies were coded as unspecified, the results were considered tentative. As for instrumentation, more than 85% of the studies used local (researcher-developed) instruments, while less than 5% used standardized instruments. This is possibly because CSI is a new field in educational research in Taiwan and there are not many published instruments available. The sample size for a study may significantly affect the statistical power of the study; in general, the larger the sample size, the better the statistical power. For the present meta-analysis, 41% of studies and the largest mean ES were associated with studies with medium sample size (61-120). This seems to suggest that the effects of CSI on students achievement may work better for medium sample size in Taiwan. As for research design type, 69% of the studies employed a nonequivalent control group design and its mean ES was higher than that for other types of designs. This result suggests that nonequivalent control group design is a more preferable design for CSI study in Taiwan, and studies using this design also produced results showing higher levels of academic achievement.

4.3 Design Characteristics

Journal of Information Technology and Applications Vol. 2, No. 2, pp. 69-79, 2007 result was considered tentative. As for visual presentation, the mean ES for studies coded as 3-D (3-dimensional) was mildly higher than that for studies coded as 2-D. This suggests that 3-D visual presentation may be more effective than 2-D for students learning. In general, 3-D approximates the real world more closely than 2-D; thus it may tend to provide learners with more authentic feelings with regard to the subject area(s). An opportunity we are missing, Journal of Research on Computing in Education, vol. 23, no. 4, pp. 497-513. Heinich, R., Molenda, M. and Russell, J. D., 1985, Instructional media and the new technologies of instruction, Canada: John Wiley & Sons Press. Faria, A.J. and Nulsen, R.O., 1995, The COMPETE saga: Or 25 years of writing and administering simulation games, Simulation & Gaming, vol. 26, no. 4, pp. 439-448. Huppert, J., Yaakobi, J. and Lazarovvitz, R., 1998, Learning microbiology with computer simulations: Students academic achievement by methor and gender, Research in Science & Technological Education, vol. 16, no.2, 231246. River, R-H. and Vockell, E., 1987, Computer simulations to stimulate scientific problem solving, Journal of Research in Science Teaching, vol. 24, no. 5, pp. 403 416. Michael, K. Y., 2000, Comparison of students product creativity using a computer simulation activity versus a hands-on activity in technology education, Unpublished doctoral dissertation, Virginia Polytechnic Institute and State University, VA. Chou, C. H., 1998, The effectiveness of using multimedia computer simulations coupled with social constructivist pedagogy in a college introductory physics classroom, Unpublished doctoral dissertation, Columbia University Teachers College, NY. Serpell, Z. N., 2002, Ethnicity and tool type as they relate to problem-solving, transfer and proxemic behavior in a communal learning context, Unpublished doctoral dissertation, Howard University, WA. Slack, S. and Stewart, J., 1989, Improving student problem solving in genetics, Journal of Biological Education, vol. 23, pp. 308 312. Johnson, S. K. and Stewart, J., 1990, Using philosophy of science curriculum development: An example from high school genetics, International Journal of Science Education, vol. 12, pp. 297 307. Collins, A. and Morrison, D., 1992, Strategic simulations in undergraduate biology: An opportunity for instruction, The 65th Annual Meeting of the National Association for Research in Science Teaching (NARST), Boston, MA, USA. Parker, Z. J., 1995, An exploratory study of experiential hands-on learning vs. computer simulation in a high school biology class, Unpublished doctoral dissertation, The Pennsylvania State University, PA. Tannehill, D. E., 1998, The use of a

[3]

[4]

5. Conclusion
The results of this study suggest that the effects of CSI are positive as compared with those of TI in Taiwan. While many educators have put and are putting tremendous effort into devising new ways of using computer technology in the classroom, with the clear expectation that such technology will dramatically increase students academic achievement, the results of this study provide can classroom teachers with a cumulative bank of research-based evidence for the positive effects of CSI on student learning. The analyses of the studied variables, however, provided no consistent evidence to support Clarks claim that there are no learning benefits to be gained from employing different media in instruction [62], [64], [65], [66], [67]. From an instructional medium designer's point of view, Clark might overlook the fact that certain media attributes make certain methods possible, particularly when new technology, such as computer simulation, is used as the instructional innovation. Left unanswered is the question as to which factors most clearly contribute to positive outcomes. Future research attempting to answer this question will require further clarification of the exact relationship between computer simulation and learning. This meta-analysis shows only that CSI tends to improve students academic achievement. That information by itself is useful.

[5]

[6]

[7]

[8]

[9]

[10]

Acknowledgements
This research project was supported by a grant from the National Science Council, Taiwan, Republic of China, under the contract NSC 922520S003 017. Our gratitude also goes to the Academic Page Editing Clinic, National Taiwan Normal University.

[11]

[12]

References
References marked with an asterisk indicate studies included in the meta-analysis [1] Sheingold, K. and Hadley, M., 1990, Accomplished teachers integrating computers into classroom practice, New York: Bank Street College of Education, Center for Technology in Education. [2] Thomas, R. and Hooper, E., 1991, Simulation: 76

[13]

[14]

Journal of Information Technology and Applications Vol. 2, No. 2, pp. 69-79, 2007 low-fidelity computer simulation in teaching the diagnosis of electronic automotive systems, Unpublished doctoral dissertation, University of Missouri Columbia, MO. Hopkins, K. S., 2001, The effects of computer simulation versus hands-on dissection and the placement of computer simulation within the learning cycle on student achievement and attitude, Unpublished doctoral dissertation, Baylor University, TX. Hummel, T. J. and Batty, C. M., 1989, A comparison of computer simulation and video-taped roleplays as instructional methods in the teaching of specific interviewing skills, The Annual Meeting of the American Educational Research Association, San Francisco, CA, USA. Tylinski, D. J., 1994, The effect of a computer simulation on junior high students' understanding of the physiological systems of an earthworm, Unpublished doctoral dissertation, Indiana University of Pennsylvania, PA. Dekkers, J. and Domatti, S., 1981, The integration of research studies on the use of simulation as an instructional strategy, Journal of Educational Research, vol. 74, no. 6, pp. 424-427. Lee, J., 1999, Effectiveness of Computer-based instruction simulation: A meta-analysis, International Journal of Instructional Media, vol. 26, no. 1, pp. 71-85. Alessi, S. M. and Trollip, S. R., 2001, Multimedia for learning : methods and development, Boston: Allyn and Bacon. Thomas, R. and Hooper, E., 1991, Simulation: An opportunity we are missing, Journal of Research on Computing in Education, vol. 23, no. 4, pp. 497-513. McLellan, H., 1996, Virtual realities, In Jonassen D. H. (Eds.), Handbook of Research for Educational Communications and Technology (p.457-487). New York: Simon & Schuster Macmillan. Liao, Y. C, 2007, Effects of computer assisted instruction on students achievement in Taiwan: A meta-analysis, Computers and Education, vol. 48 , no. 2, pp. 216-233. Feng, D., 1984, A computer simulation on corn-dried system, Unpublished masters dissertation, National Taiwan Unicersity, Taipei, Taiwan. Tao, J., 1999, A computer simulation for prelaplacian-of gaussian operation, Unpublished masters dissertation, National Taiwan Unicersity, Taipei, Taiwan. Jao, Y. H., 1993, An empirical study of the learning effects of a computer simulation, 77 Unpublished master dissertation, Tamkang University, Taipei, Taiwan. Chao, J. H., 2001, A study for learning effectiveness of Electronic Circuit practial course with computer-simulation teaching strategy on vocational senior High school level, Unpublished master dissertation, National Changhua University of Education, Changhua, Taiwan. Chuang, C. F., 2000, The efficacy of computer-simulated courseware in teaching selected technological concepts, Proceedings of the National Science Council (Part D), vol. 10, no. 2, pp. 51-60. Huang, J. C., 2002), The effect of applying multimedia animation resources on the teaching of experiments of physics and chemistry in junior high schools, Unpublished master dissertation, National Kaohsiung Normal University, Kaohsiung, Taiwan. Lin, U. C., 2002, A study of virtual-lab on WWW affects students learning achievement of natural science learning at elementary schools, Unpublished masters dissertation, National Tainan Teachers College, Tainan, Taiwan. Nein J. S., 2002, A study of learning effectiveness of digital logic course with computer-simulation teaching method at vocation senior high school level, Unpublished master dissertation, National Changhua University of Education, Changhua, Taiwan. Su, J. S., 2002, The study on the learning effect of algorithms with Internet hypermedia and interaction algorithm animations, Unpublished master dissertation, Ming Chuan University, Taipei. Taiwan. Chen, C. H., 2002, The effects of multimedia simulation on senior high school students physics achievement, Unpublished master dissertation, National Kaohsiung Normal University, Kaohsiung, Taiwan. Tseng C., 2002, The effects of advanced organizers and computer visual simulated environment on primary school level's concept of probability, Unpublished masters dissertation, National Tainan Teachers College, Tainan, Taiwan. Yu, J., 2002, Applying artificial neural network on the study of the learning style in a web-based virtual science lab, Unpublished masters dissertation, National Tainan Teachers College, Tainan, Taiwan. Kulik, J., Kulik, C-L. and Bangert-Drowns, R. L., 1985, Effectiveness of computer-based education in elementary pupils, Computers in Human Behavior, vol. 1, pp. 59-74.

[27]

[15]

[28]

[16]

[29]

[17]

[30]

[18]

[31]

[19]

[20]

[32]

[21]

[33]

[22]

[34]

[23]

[24]

[35]

[25]

[36]

[26]

Journal of Information Technology and Applications Vol. 2, No. 2, pp. 69-79, 2007 [37] Glass, G. V., McGaw, B. and Smith, M. L., 1981, Meta-analysis in social research, Beverly Hills, CA: Sage Publications. Hunter, J. E., Schmidt, F. L. and Jackson, G. B., 1982, Meta-analysis: Cumulating research findings across studies, Beverly Hills, CA: Sage. Andrews, G., Guitar, B. and Howie, P., 1980, Meta-analysis of the effects of stuttering treatment, Journal of Speech and Hearing Disorders, vol. 45, pp. 287-307. Waxman, H. C., Wang, M. C., Anderson, K. A. and Walberg, H. J., 1985, Adaptive education and student outcomes: A Quantitative synthesis, Journal of Educational Research, vol. 78, no. 4, pp. 228-236. Chang, H. P., 2001, A study for learning effectiveness of computer simulation program used in practical course Basic Electricity for vocational high school students in division of refrigerating and air-conditioning, Unpublished master dissertation National Changhua University of Education, Changhua, Taiwan. Chao, J. T., 1999, Computerized project-based learning for elementary science education, Unpublished master dissertation, National Chengchi University, Taipei. Taiwan. Chen, C., 2003, Study on merging information technology into Nine-year coherent curriculum on earth science creative teaching-- An example to 921 earthquake virtual reality education, Unpublished masters dissertation, National Tainan Teachers College, Tainan, Taiwan. Chen, H., 2005, Application of Easy Java Simulation to the Science Course in Junior High School, Unpublished master dissertation, National Changhua University of Education, Changhua, Taiwan. Chen, T., 1998, The study of using virtual reality to enhance the learning of spatial ability, Unpublished master dissertation, National Chengchi University, Taipei. Taiwan. Chen, Y. S., 2002, A study of the effects of pneumatics with computer simulation instruction on vocational education, Unpublished master dissertation, National Changhua University of Education, Changhua, Taiwan. Guan, H. D., 1999, The impacts of teaching for conceptual change strategies enhanced with computer simulation on students learning about density related concepts, Unpublished masters dissertation, National Hualien Teachers College, Hualien, Taiwan. Hsu, Y. S. and Chiu, M. H., 2001, The use of multiple representations in a web-based and 78 situated learning environment, (NSC89-2511-S003-133). Hsu, Y. S., 2002, The effects of virtual learning environment with multiple representations on science learning (III), (NSC90-2520-S003-010). Huang, C. K., 2002, The efficiency of applying animation multimedia for assistance of teaching of physics and chemistry in junior high school, Unpublished master dissertation, National Kaohsiung Normal University, Kaohsiung, Taiwan. Lai, Y. J., 2002, A study of learning styles on natural science virtual-lab by using fuzzy inference rules, Unpublished masters dissertation, National Tainan Teachers College, Tainan, Taiwan. Li, M., 2005, Development and effect analysis and of a VR-based diagnosis training system for a gasoline-injection engine, Unpublished masters dissertation, National Kaohsiung First University of Science and Technology, Kaohsiung, Taiwan. Lin, C., 2004, The effect on fifth-grade students concept of earth motion after implementing information technology integrated into teaching and assessment, Unpublished masters dissertation, National PingTung Teachers College, PingTung, Taiwan. Lin, Y., 2002, The effects of integrating information technology into teaching on elementary school student Astronomy achievement A case of moon, Unpublished masters dissertation, National PingTung Teachers College, PingTung, Taiwan. *Shen, S. (2005). An investigation of using virtual reality to develop earth motion curriculum in elementary school, Unpublished master dissertation, National Chun Cheng University, Chia Yi, Taiwan. Su, C. Y., 2000, How to help students learning "The Stars" by using computer simulation software, Unpublished master dissertation, Taipei Municipal Teachers College. Taipei. Taiwan. *Wang, C., 2005, The effects of integrating information technology into teaching on elementary school student Astronomy achievement A case of star, Unpublished masters dissertation, National PingTung Teachers College, PingTung, Taiwan. Wang, K. K., 1994, Middle school students' decision-making on solid waste management in Taiwan, The Annual Meeting of the National Association for Research in Science Teaching, Anaheim, CA, USA.

[49]

[38]

[50]

[39]

[40]

[51]

[41]

[52]

[53]

[42]

[43]

[54]

[44]

[55]

[45]

[56]

[46]

[57]

[47]

[58]

[48]

Journal of Information Technology and Applications Vol. 2, No. 2, pp. 69-79, 2007 [59] Wang, T., 2005, A Study of problem-based learning applied in the interactive on-line simulation teaching An example for the Digital Logic Lab. course at vocational high school, Unpublished master dissertation, National Changhua University of Education, Changhua, Taiwan. Cohen, J., 1977, Statistical power analysis for the behavioral science (Revised Edition), New York: Academic Press. Christman, E., Badgett, J. and Lucking, R., 1997, Progressive comparison of the effects of computer-assisted instruction on the academic achievement of secondary students, Journal of Research on Computing in Education, vol. 29, no. 4, pp. 325 337. Clark, R. E., 1983, Reconsidering research on learning from media, Review of Educational Research, vol. 53, no. 4, pp. 445-459. Liao, Y. C. and Bright, G. W., 1991, Effects of computer programming on cognitive outcomes: A meta-analysis, Journal of Educational Computing Research, vol. 7, no. 3, pp. 251-268. Clark, R. E., 1985, Computer Research Confounding, The annual meeting of the American Educational Research Association, Chicago, IL, USA. Clark, R. E., 1991, When researchers swim upstream: Reflections on an unpopular argument about learning from media, Educational Technology, vol. 31, no. 2, pp. 34 40. Clark, R. E., 1994, Media will never influence learning, Educational Technology Research and Development, vol. 42, no. 2, pp. 21 29. Clark, R. E. and Craig, T. G., 1992, Research on multimedia learning effects, In Giardina, M. (Ed.), Interactive multimedia learning environments. New York: Springer Verlag.

Biography
Yuen-kuang Liao received his BA degree from National Taiwan Normal University in 1980, M.ed and Ed.D degrees from University of Houston, Houston, Texas, in 1986 and 1990, respectively. Currently, he is a professor in the Department of Education, National Taiwan Normal University (NTNU). He also serves as the director of the Center for Educational Research in NTNU. His current research interests include meta-analysis in teaching and learning, technology and gender issue, game-based learning, and attitudes toward Internet.

[60]

[61]

[62]

[63]

[64]

[65]

[66]

[67]

79