Computer Assisted Language Learning Vol. 22, No.

1, February 2009, 5772

Web-based essay critiquing system and EFL students writing: a quantitative and qualitative investigation
Cynthia Leea*, Kelvin C.K. Wongb, William K. Cheungb and Fion S.L. Leeb
Language Centre, Hong Kong Baptist University, Hong Kong; bDepartment of Computer Science, Hong Kong Baptist University, Hong Kong (Received 9 March 2008; nal version received 12 December 2008) The paper rst describes a web-based essay critiquing system developed by the authors using latent semantic analysis (LSA), an automatic text analysis technique, to provide students with immediate feedback on content and organisation for revision whenever there is an internet connection. It reports on its eectiveness in enhancing adult EFL students writing quantitatively and qualitatively. Finally it suggests ways to use the system eectively. An experimental study was conducted. Twenty-seven student writers, who were assigned to the control and experimental groups randomly, wrote a 300-word argumentative essay, a genre usually tested in local and public examinations, over an 80-minute duration. The experimental group received two modes of feedback from the system whereas the control group typed and revised their essays on the computer in the traditional pen and paper way only. Their nal submissions were co-marked by two raters, and comparison on essay length and nal scores between the two groups was made. Although it was found that there was no statistically signicant dierence between the two groups in these two aspects, the experimental groups writing strategies shed some light on pedagogy. Keywords: second language writing; latent semantic analysis (LSA); adult learners; essay critiquing
a

Introduction Essay writing is a required task for academic communication. Students have to do much of their thinking through writing and produce a fair amount of formal essays at university. It is, however, a stressful task for some English as a Foreign Language (EFL) students, because they may have diculty in generating ideas to discuss the essay topic, or have inadequate time to organise their language and thoughts within the given period of time. It is also a demanding and time-consuming task for some EFL teachers because they may not be able to give customised, immediate content-related feedback to multiple submissions if it is required in an in-class or take-home exercise. Advancement in computer technologies has presented a signicant possibility to solve the teaching and learning problems in EFL writing. In view of this, we developed a web-based essay critiquing system (ECS), using an automatic text analysis technique known as latent semantic analysis (LSA), which has been used to provide content feedback and evaluate

*Corresponding author. Email: cfklee@hkbu.edu.hk

ISSN 0958-8221 print/ISSN 1744-3210 online 2009 Taylor & Francis DOI: 10.1080/09588220802613807 http://www.informaworld.com

58

C. Lee et al.

essays in educational computing research. In this initiative, we make use of LSA to give immediate feedback to help EFL students reect on content and their organisation of ideas, and revise drafts independently using the internet. In this paper, we rst describe the rationale and the system, then present and discuss the ndings of an experimental study on adult EFL students writing. Essay writing in a computer context The invention of the computer has brought important changes to all walks of life, one of which is education. In second language teaching, computers have revolutionised the manner of teaching, especially the teaching of writing. The paradigm shift from exploring the role of word-processors in writing in the late 1980s, to researching the best ways to support or assess student writing through computers (Foltz, Gilliam, & Kendall, 2000; Chapelle, 2001) has evidenced the burgeoning role of computers in language teaching. The functions of computers have been moved from a writing tool in the traditional classroom, to developing computer assisted language learning (CALL) activities in language laboratories, using synchronous and asynchronous networked tools such as local area networks (LAN), wide area networks (WAN), emails, the world-wide web and ICQ (Pennington, 2004), and developing educational software based on such techniques as LSA (e.g. Kintsch et al., 2000; Foltz et al., 2000) to mark language learners discipline essays beyond classroom boundaries. Technological changes, therefore, have generated new literacies, genres, identities and pedagogies (Warschauer, 2004). Writing in a computer context is thought to be more benecial to students than writing in a pen-andpaper context due to its automation, exibility and cognitive demands (Pennington, 2003, p. 289). EFL teachers, in particular, are called upon to be aware of the capabilities of technology in teaching, and to examine dierent options and their applications (Franklin, 1999, cited in Chapelle & Hegelheimer, 2004). However, such optimism needs caution, because research on writing produced in a computer context so far has mixed results in writing quality, and such factors as individual students writing ability, pedagogy and the selected software are inuential. Recent research on EFL writing in a computer context focuses on two main areas. The rst area examines the extent to which networked environments can promote writing quality. Ghaleb (1993, cited in Braine, 2004) and Braine (1997, 2001, 2004), who compared the quality of writing produced in the LAN context with that produced in traditional writing classes, found that nal submissions in traditional writing classes were of higher quality than those in the LAN context. In contrast, Sullivan and Pratts study (1996) produced the opposite results. Their subjects discourse and feedback in a computerassisted classroom were more thought-provoking than those in an oral discussion classroom, and the tool generated better writing quality. The second area uses educational software to evaluate writing with a view to enhancing writing quality. In Kozna and Johnstons study (1991), the quality of writing was not improved when two computerbased tools outliners and organisers were in use; rather it depended on the students initial writing ability. Novice student writers had more diculties revising compositions with the tools than advanced student writers did. Likewise, in Chen and Chengs study (2006), students were not satised with the grade and supercial and inaccurate feedback on both the content and the rhetorical aspects of their writings given by the automated essay grading program MyAccess, although the program has been reported as being useful on some occasions. Nevertheless, the educational software or system that uses the LSA technique has generated satisfactory results in terms of accuracy regarding the content,

Computer Assisted Language Learning

59

outline and coherence of the essay. Studies of Kintsch et al. (2000), Lemaire and Dessus (2001) and Foltz et al. (2000) found that their educational software systems, such as Summary Sheet and Apex, which evaluated a students essay content in comparison with the important source content of a subject stored in the system, had a signicant correlation between human grades and the automated essay grader. Given the encouraging results on using the LSA technique to provide immediate feedback on content and coherence, and considering the trend of teaching writing in a networked environment but with mixed results, we developed an essay critiquing system (ECS) that uses the technique to help students write essays that are required in our teaching context. The following section describes our teaching context and the system design, followed by a report on our experimental study and pedagogical implications.

An essay critiquing system (ECS) Background information Teaching context The essay writing skill is an essential academic communication skill at university, and is tested in international and local examinations such as the Test of English as Foreign Language (TOEFL),1 International English Language Testing System (IELTS),2 Use of English in Hong Kong Advanced Level Examination (HKALE)3 and English courses at our university4 for EFL learners with a designated time and length, ranging from 250 to 500 words in 30 to 80 minutes respectively. Over 90% of our rst year university students, who are EFL learners who have been studying English for over 13 years and have passed local or international examinations, are also required to take two compulsory English courses in which essay writing skills are taught, practised and tested in each course. They learn the writing skills in lectures and are asked to write their essays at home. They can discuss the essays with teachers (but very few do it), and write and revise their essays outside classrooms prior to the submission deadline. The argumentative genre, however, is demanding for our students, because it requires not only their understanding of its structure, but also the ability to plan, express, develop an argument and revise intended meaning, as well as the mastery of rhetorical devices (e.g. organisation, cohesion and coherence). They need practice and appropriate feedback or comments from teachers is deemed necessary. Since it is dicult for teachers always to provide detailed comments, we developed a web-based essay critiquing system, using latent semantic analysis (LSA), an automatic text analysis technique, to provide students with immediate feedback on ideas for revision and use with no teacher presence and whenever there is an internet connection (Cheung et al., 2007; Wong et al., 2007). ECS design ECS is a system that has incorporated LSA to provide immediate feedback on writing ideas to students on an essay topic by comparing student and model essays. LSA is a mathematical technique for computing the semantic similarity between pieces of textual information (for example, sentences, paragraphs or essays) with the help of a large corpus (Landauer & Laham, 1998; Landauer & Psotka, 2000) on a domain of students essays. It has been used for writing summaries (Kintsch et al., 2000) and grading essays (Foltz et al., 2000; Lemaire & Dessus, 2001).

60

C. Lee et al.

The system consists of eight components: teacher input; student input; a database that stores student answers and reference materials from external sources; a text segmentation and a pre-processing engine; an LSA engine; and a semantic matcher for providing critical feedback to students (Figure 1). The system allows a teacher to decide on an essay topic for students, and to make use of some relevant background materials on the topic, such as articles collected from internet resources, students sample essays or textbooks to build a corpus. These materials are rst broken down into sentences and pre-processed by removing stop-words (i.e. removing prepositions, articles etc.) and stemming (i.e. converting verbs and plural nouns into their basic forms) by the text segmentation and pre-processing engine. Next, the LSA engine computes the word-segment association matrix. The matrix is then decomposed using the Singular Value Decomposition (SVD) technique (Strang, 1980) to form a semantic space in which words that frequently occur in the same segments are positioned near one another (Steinhart, 2001). For example, words that do not co-occur (bicycle and bike) in a context, but occur in similar contexts will also be grouped together in the semantic space. In the semantic space, each sub-theme will be represented as a feature vector. After constructing the semantic space, semantic similarity between two text segments can then be computed by calculating the geometric cosine value between the corresponding feature vector in the semantic space. The range of the cosine value is from zero to one, indicating the degree of similarity. System features and corresponding functions The system provides two modes of feedback. The rst mode is on content (i.e. covered and missing, see Figures 2 and 3) sub-themes, and the second one is on organisation (i.e. distribution of sub-themes on the text and degree of relevance, see Figure 3). When a student submits his or her essay to the critiquing system for feedback, the semantic

Figure 1.

Essay critiquing system architecture.

Computer Assisted Language Learning

61

Figure 2.

Student interface: display of missing sub-themes.

Figure 3.

A revised version.

similarity between all possible pairs one from the students essay and the other from the systems sub-theme list is calculated. When the process is completed, the sub-themes on the list that are missing in the students essay will be identied. The missing sub-themes will be reported to the student for his or her consideration when revising their essay. The system is also able to identify text segments in the students essay that match with the subthemes in the sub-theme list, and highlight them in dierent colours according to the degree of matching. More importantly, the system is a web-based system that is available for use at the learners convenience whenever there is an internet connection. The characteristic features

62

C. Lee et al.

and functions are illustrated in Figures 2 and 3. The two gures also illustrate how the system provides feedback, and how a student revised his essay after receiving feedback (in Figure 3). The student interface (Figure 2) is divided into left and right columns. The right column displays the uploaded student essay, while the left column provides missing subthemes, sub-themes that have been covered in the text and the degree of relevance of the text to the matched sub-themes indicated using three colours (Figure 3). After uploading the rst submission, the student will see his or her essay and the suggested sub-themes. When they click a covered sub-theme, they will visualise the part of the text that is relevant to the highlighted sub-theme, ranging from highly related (bold) to related (italic) to slightly related (underline), and the distribution of the sub-theme in the text (Figure 3). The two modes of feedback aim to stimulate the student to think about the development and organisation of arguments in the text, which involves rhetorical features such as cohesion and coherence. The impact of the two modes of feedback can be seen in Figure 3. As shown from the screenshot, the student integrated the second sub-theme health problems/diseases into the rst paragraph, and developed it by elaborating on the harm of fast food and undesirable frying eects. The text segment Eating trans fat will increase the risk of coronary heart disease, which is being assessed as slightly related to the systems sub-theme list, might prompt the student to revise their text to include it, or give a second thought to it. The experimental study Research questions Our study examined the impact of ECS on our students writing in both content and rhetorical aspects, as well as the way in which the system was used, by comparing the performance of students who used and who did not use the system in an experimental context. Research questions were: (1) What is the writing performance of the experimental and control groups, in terms of their essay length, submission rate, amount of arguments, holistic essay scores, content and organisation? (2) How does the system help the experimental group to revise its essays in both content and rhetorical aspects (e.g. organisation of ideas, coherence and cohesion)? (3) What implications does the study have for using computers for writing? Design of the study and procedures An experimental study was designed and conducted in a computer room. The subjects in the study were year 1 and year 2 undergraduates at our university who had passed their Use of English in the Hong Kong Advanced Level of Examination.5 They were assigned to the treatment and the control groups randomly. Each group consisted of 14 subjects, and each of them was asked to write a 300-word argumentative essay within 80 minutes. The experimental group was allowed to use the essay critiquing system for feedback, and to visualise the organisation of their current sub-themes. They were allowed to submit for system feedback as many times as they preferred. The computer recorded their submissions every time they asked for feedback. Before the start of the study, students in the treatment group were briefed about the rationale of the study and on the system features and corresponding functions on the student interface. Submission procedures for system feedback and teacher grading

Computer Assisted Language Learning

63

were demonstrated. In contrast, the control group was asked to type and revise their essays on the computer only with no feedback on content, and to submit the nal submissions when the time was up. Both groups were allowed to use paper dictionaries and/or online dictionaries. The nal submissions of both groups were scored by two raters in accordance with the marking criteria developed exclusively to reect the systems functions and modes of feedback6 (refer to the paragraph on raters and marking criteria for more information). Writing task and time Since writing argumentative essays is a required task both in English courses at university and public examinations, the experimental and the control group subjects were asked to write an argumentative essay on the topic: Should there be rules against fast food in schools? Express your view and support your view with examples in 80 minutes of about 300 words in front of a computer, as if they were doing a written assignment at home, or sitting for the English prociency course examination at the university. Sub-theme list The system generated the feature vectors representing the sub-themes of the topic based on LSA. There were 11 sub-themes for thought, namely: (1) fast food ingredients; (2) illness/ diseases caused by fast food; (3) harmful eects; (4) childrens addiction to fast food; (5) childrens eating habits; (6) childrens amount of exercise; (7) the chance of eating fast food in schools after banning; (8) the importance of public education; (9) schools responsibility in providing clear guidance; (10) schools responsibility in providing healthy food; and (11) parents responsibility. Sub-themes that had been covered or were missing from a students text were displayed on the students interface, prompting the student to think and say more about a sub-theme. Rater and marking criteria Essays from both the control and treatment groups were scored by two raters who were experienced English teachers at the university. Since the focus of the system was to help students enrich their content7 and improve organisation and text development, the raters were instructed to give a mark on these two aspects (see Appendix 1). These two areas are also assessed in two important local public examinations8 and our university English courses. Each category carried a maximum of 10 marks, and the marking scheme was the one that is currently used in the universitys English courses and with which the two raters are familiar, capturing both content and rhetorical features. The raters were told to write down comments either at the end of the essay or to write text-specic comments along the margin as they wished (ibid). Appendix 1 lists the description of each scale. Quantitative and qualitative analyses Mean scores and standard deviations were computed. The T-test was used to calculate the statistical signicance of the two items between the two groups.9 Qualitative analysis was carried out based on the written discourse of the students rst and nal submissions.

64

C. Lee et al.

Results and discussion Comparison between the experimental and control groups writing performance Dierences in length of essay Essay length is one variable that indicates whether students have sucient ideas to full the word requirement set by the teacher. Table 1 shows that the average length of the experimental groups nal submissions, as measured by the number of words, was 26.30 words more than that of the control group. However, according to the T-test, their dierence was not statistically signicant at the 0.05 level. Dierence in amount of arguments Counting of arguments was based on both the systems records and manual counterchecking. Regarding the amount of arguments, it was found that the average number of sub-themes in the experimental groups essays outnumbered those in the control groups. The experimental group had 6.79 arguments on average, while the control group had only four. The average number of arguments of the experimental group increased gradually, from 3.71 in their rst submissions to an average of 6.79, which was an increase of 48% after receiving feedback from the system. Dierences in holistic essay score and performance in content and organisation The mean scores of the two groups on content and organisation were very close, but the T-test showed that the dierence was not statistically signicant at the 0.05 level (Table 3). To summarise, under our current experimental setting, we found that the use of the critiquing system seems to be eective in helping students enrich their essays with more ideas. However, immediate improvement regarding the essay length and the holistic essay score has not been observed. The potential reason for no immediate improvement in both areas is discussed in the next section.

Table 1. Group

Descriptive statistics on essay length. Mean (N 14) 387.86 361.57 374.71 S.D. (N 14) 85.06 67.30 76.44

Experimental Control Overall

Table 2.

Average number of sub-themes in the rst and nal submissions. Control Group (N 14) Number of sub-themes in the nal submission Experimental Group (N 14) Number of sub-themes in the rst submission 3.71 Number of sub-themes in the nal submission 6.79

Student No./Group

Average

Computer Assisted Language Learning Comparison within the experimental groups writing performance Submission rate, amount of arguments and writing performance

65

The experimental group had three submission rates: (1) three to four times; (2) ve to six times; and (3) eight times, corresponding to the dierent amount of arguments and scores in content and organisation given by the raters. Four students fell into the rst submission rate with an average of seven arguments and 6.81 marks; six into the second one with an average of six arguments and 6.32 marks; and four into the third one with an average of 10 arguments but 5.5 marks only. Submission rate and amount of arguments are, however, inversely proportionate to the score in content and organisation (Table 4). That is, a student who had a high submission rate (in this case, eight times) and incorporated all subthemes suggested by the system did not necessarily score high in content and organisation. According to the raters written comments on students essays, the reason for good or poor performance seems to hinge on the development and organisation of arguments. Poor development of arguments is the main rhetorical feature commented on by both raters on students essays in the second and third submission rate groups. Both of them were concerned with a lack of supporting examples for the sub-themes. For instance:
The ideas are generally clear to the reader. Perhaps you can include more details or examples about the eating habits of children. The ideas are related to the topic, but they could be more organised. Some ideas have been repeated. More details should be given to the second point. More elaboration on the reasons, please! You have many ideas but they are not elaborated fully. You talk a lot about fast food, but not enough in the second situation fast food in schools.

Writing strategies and writing performance Three types of writing strategies have been identied, each type reecting the way in which ECS and its feedback are handled. Similar to the experimental-control comparison results,
Table 3. Group Experimental Control Descriptive statistics on content and organisation score. Mean (N 14) 6.86 6.88 S.D. (N 14) 1.55 1.56

Table 4. Relationship among submission rate, amount of arguments and scores in content and organisation. Number of students 4 6 4 Submission rate 34 times 56 times 8 times Average amount of arguments 7 7 10 Average score in content & organisation 6.81 6.32 5.5

Group Experimental

66

C. Lee et al.

there is not much dierence in the content and organisation mean scores among the three types of strategies (see Table 5). The rst group of students strategy was that they submitted for feedback a few times at short intervals when the essay was nearly complete, and this was followed by minor surface-level cosmetic revision by adding some words taken from the system to support the existing argument. The subjects essays were already structured with an introduction, body and a conclusion in the rst submission. Six subjects (42.86%) adopted this pattern, and they spent over half of the designated time thinking, revising (some did it on a paper and some on the computer) and nally seeking feedback several times at short intervals. Their rst submission took place at 49 minutes on average, and then they moved from an interval of nine minutes for the second submission down to 0.36 minutes for the fth submission. About one to two additional arguments were nally generated. The length and rhetorical features of their rst and nal submissions were almost identical. This group of students did not seem to gain optimal benets from the system. Example 1 illustrates the rst pattern and underlines the revision by comparing the introduction and part of the essay body in the rst and nal submissions. The student writer merely added

Table 5.

Average mean scores of the three groups of students. Number of students 5 6 3 Average mean score in content and organisation 6.4 6.5 5.9

Experimental Pattern 1 Pattern 2 Pattern 3

Example 1 First submission (Introduction) The ourish of the fast food shops, especially MacDonalds has been accused of worsening the problem of obesity amongst Hong Kong children. Giving the extraordinary amount of salt and avouring adding to the fast food, there are voices calling for setting rules against fast food in schools. But, can the problem really be solved with such ease? (Body Paragraph 1) Undoubtedly, it is not uncommon to nd that having too much fast food will result in obesity. From the movie Super Size Me, we truly know that the thirst for fast food in the American children has brought the gure of obesity to a new height, and has seriously aected the health of the new generation. Therefore, setting rules against fast food seems inevitable. Final submission (Introduction) The ourish of the fast food shops, especially MacDonalds has been accused of worsening the problem of obesity amongst Hong Kong children. Giving the extraordinary amount of salt and avouring adding to the fast food, there are voices calling for setting rules against fast food in schools. But, can the problem really be solved with such ease? (Body Paragraph 1) Undoubtedly, it is not uncommon to nd that having too much fast food will result in obesity because of (cohesive device) the high fat value (cause of obesity). From the movie Super Size Me, we truly know that the thirst for fast food in the American children has brought the gure of obesity to a new height, and has seriously aected the health of the new generation. Therefore, setting rules against fast food seems inevitable.

Note. The underlined phrase is a new point/ argument suggested by the system.

Computer Assisted Language Learning

67

the phrase the high fat value, a phrase provided in a sub-theme, to the rst paragraph in the nal submission only, and linked it up with the rst sentence with the cohesive device because of. The second groups strategy was to submit for feedback when the essay was nearly complete, and this was followed by continuous and rigorous revision. Five subjects (35.71%) adopted this approach, and on average 1.5 additional sub-themes were generated in the end. Their two submissions, however, looked dierent, and their nal revisions were more dynamic than the previous groups mainly in organisation, cohesion and coherence. They took a missing sub-theme from the system, developed it and linked it up with the original text in an appropriate place. The generation of this new text also stimulated the subjects to revise the other parts continuously. There were some dierences in the rhetorical features of the rst and nal submissions, as shown in the underlined phrases or sentences in Example 2. In Example 2, the student writer incorporated the sub-theme health problem harmful eects to the rst submission with reference to the health professionals warning, reorganised the sequence of discourse and wrote a concluding sentence. A a result the text ended up with a comparison between childrens eating habits now and then, followed by the health professionals warning. The third groups strategy was to write for a very short period of time and to seek feedback from the system immediately. Only three subjects (21.43%) adopted this pattern, with an increase of six sub-themes and 5.9 score on average in the end. Their rst

Example 2 First submission (Introduction) Nowadays, many children in Hong Kong are obese. It is a trend that there is a growth of obesity among Hong Kong children. Many health professionals claim that the reason for the growth of obesity is the increase in fast food. Health professionals point out that fast food contains a lot of unsaturated fat and causes children to be obese. (Body Paragraph 1) In the past, children did not have many choices on food. They only had their meals at home. With the balanced diet, there was nearly no obesity problem. But, due to the popularity of fast food, children have a lot of choices on food. They seldom have their meals at home but have their dinners, lunches at MacDonalds etc. Eating the high calorie food, they are getting fatter and fatter. Final submission (Introduction) Nowadays, many children in Hong Kong are obese. It is a trend that there is a growth of obesity among Hong Kong children. Many health professionals claim that the reason for the growth of obesity is the increase in fast food. Health professionals point out that fast food contains a lot of unsaturated fat and causes children to be obese. (Body Paragraph 1) In the past, with the balanced diet, children had no obesity problem. But, nowadays, children love eating high calorie food and they are getting fatter and fatter (reorganisation of idea). Health professionals have pointed out that obesity can aect the health of children, and can cause many serious diseases such as heart disease. (elaboration on health problems) It is necessary to stop the popularity of fast food among children. (concluding sentence)

Notes. The underlined phrases/sentences are the missing sub-themes suggested by the system. The italicised phrases are changes made by the student writer.

68

C. Lee et al.

submissions took place at 18 minutes or less. Their second submissions took a slightly longer time than the other two groups, mainly because they had more to add to the submissions. Although the average score was not high, their revisions were the most rigorous, from sentence to discourse level for text development, following the genre of an argumentative writing. The length and organisation of the rst and nal submissions were greatly improved. For instance, in Example 3, the rst submission of an introduction from a student (same as Figure 1) consisted of one 20-word paragraph only. After receiving the two modes of feedback, the student revised some wordings, elaborated topic-related subthemes, described the situation, identied the problems, supported arguments with examples and developed the text coherently in the subsequent submissions. At the end of his sixth revision, his nal submission rose to 348 words. As mentioned previously, the mean score dierences among the three patterns were very close, between 0.1 and 0.6. The mean score of patterns 1, 2 and 3 were 6.4, 6.5 and 5.9, respectively (Table 5). In summary, the treatment group that received two modes of feedback from ECS did not outperform the control group, which received no feedback. It seems that the two modes of system feedback could not guarantee better performance. Nevertheless, the students written discourse seems to provide a hint: the crucial factor appears to hinge on the way in which the students utilised the system feedback appropriately according to the analysis of the three identied patterns and their corresponding revision strategies. Therefore, we feel that appropriate amount of a number of factors would help students to benet from the technology, including teacher input on the system and demonstration, discussions about various writing strategies and a recommended approach before and after using the system. In other words, the teachers role is indispensable and the computer system may not aect writing quality unless it is used in conjunction with appropriate pedagogy. The following are some suggested teaching ideas for teachers consideration.

Example 3 First submission (Introduction) Nowadays, the popularity of MacDonalds among children is still growth although more and more people realise that MacDonalds is harmful to themselves. (20 words) Second submission (Introduction) Nowadays, the popularity of fast food (revision of words), for example, the MacDonalds (elaboration on fast food) amongst children is still growth although more and more people realise MacDonalds harm to themselves. The ingredients of French fries are our (describing the fact). However, after being fried, the amount of trans fat, which is a type of unsaturated fat, exceeded one can absorb for one day. Eating trans fat will increase the risk of coronary heart disease. (Stating the problem) (66 words) Final submission (Introduction) Nowadays, the popularity of MacDonalds among children is still growth although more and more people realise that MacDonalds harm to themselves. (Describing the phenomenon) Due to the high fat and oil contained in fast food, such as hamburgers and French fries, the rate of obesity among Hong Kong children is getting more and more serious, and has become a social problem. (Stating the problem) Therefore, there should be rules against selling fast food in schools before all is too late. (Stating the stance) (74 words)

Note. The italicised phrases indicate the changes and rhetorical features.

Computer Assisted Language Learning Using ECS Providing appropriate amount of input before using the system Before using the system, the teacher could hold a brief session or spare a lesson to:

69

(1) introduce the system features and functions; (Rationale: to familiarise students with the system features) (2) demonstrate the steps to upload an assignment to ECS for feedback, revise and submit to the teacher; (Rationale: to familiarise students with the operation) (3) introduce the three writing strategies illustrated with samples; (Rationale: to alert students to the various possible writing strategies) (4) discuss the strengths and shortcomings of each strategy, and (Rationale: to raise students awareness of how various strategies impact system use and writing quality) (5) recommend an approach a blend of Pattern 2 and Pattern 3 (Rationale: to feed research into teaching. Blending Patterns 2 and 3 means encouraging students to obtain feedback from the system after writing for a short period of time, followed by continuous and rigorous revision in content and organisation of ideas) Based on the recommended approach, the teacher could x a time for students to submit their rst drafts and obtain system feedback. Providing appropriate feedback after using the system After using the system, the teacher could: (1) nd out students submission rates and examine the extent of revision (content and rhetorical features) in each draft or between the rst and the nal drafts; (Rationale: enables the teacher to gain a better understanding of the way their students use the system, and how the students revise their essays.) (2) provide feedback to students in the subsequent lesson (in class verbally, or written comments on individual essays along the margin as usual) on: Content (a) The number of sub-themes they have incorporated into the essay; and (b) The way in which a sub-theme or an argument is developed. Organisation (c) How well dierent sub-themes/arguments are linked and developed with supporting examples or illustrations (coherence and cohesion); and (d) How well the sub-themes/arguments have been reorganised or revised, with follow-up discussions whenever necessary. (Rationale: enables students to reect on their writing strategies and the best way to use the system) Feedback from the teacher on how well students utilise the system to improve their writing is denitely essential, though the teacher needs to learn how to give feedback from

70

C. Lee et al.

experience, and invest time in reading and comparing students multiple drafts. Additional steps that the teacher feels necessary for his or her own students could be added, and pedagogical eectiveness is subject to further classroom research. Conclusion Discourse analysis of the experimental groups essays has been insightful, although the control-experiment comparison in several aspects does not show any great quantitative dierences. Since the study is an experimental one that aims to elicit results within a xed period of time, it is worth adopting a longitudinal approach to investigate the impact of the critiquing system after integrating it into EFL writing classrooms for learners of dierent prociency levels on a larger scale. More work on the system should be done by developing its corpus, such as including and oering more essay topics10 for students, and generating some analysis reports for teachers to understand the performance of their students. The study has shown that the critiquing system is a useful tool to facilitate EFL learners writings, and we believe its teaching eectiveness can be further enhanced if it is supported with appropriate pedagogy.11 Acknowledgements
We would like to thank Man-Kam Chan and Patricia Wong for their assistance in handling various technical and editorial work of the project. The project is a Teaching Development Grant Project (TDG/0405/I/03) funded by Hong Kong Baptist University.

Notes
1. TOEFL requires candidates to complete an integrated writing task and an independent writing task. The independent writing task resembles an argumentative essay, giving opinions on an issue or expressing personal preference, and candidates are required to write 300 words in 30 minutes (Rogers, 2007). IELTS requires candidates to complete a graph description task and an argumentative writing task. The argumentative writing task expects candidates to organise answers clearly, giving examples to support their points. Candidates are expected to write at least 250 words in approximately 40 minutes (Jakeman & McDowell, 1999). In the writing paper of HKALE, candidates are required to write a minimum of 500 words on a single topic on expository writing e.g. persuasion, argument, reporting, development of hypotheses etc. in 1.25 hours (retrieved December 1, 2008 from the Hong Kong Examinations and Assessment Authority website at http://www.hkeaa.edu.hk/DocLibrary/HKALE/Subject_ and_Syllabuses/2008as(e)_ue.pdf). In our university, students who take the English prociency courses are required to write an argumentative essay of about 400 words as an assignment and in their nal examination, in approximately 60 minutes. The Hong Kong Advanced Level of Examination (HKALE) is regarded as a university entrance examination. The marking criteria are adapted from the marking criteria used in the English courses at our university. In the study, content was glossed as the writers intended meaning about the topic expressed in the text (i.e. ideas or arguments, in laymans term) rather than the amount of ideas. Rhetorical features such as coherence, cohesion and development of ideas in or across paragraphs were important (Goldstein, 2005). The two important local public examinations are the Hong Kong Certicate of Education Examination (HKCEE) for Form Five school leavers and the Hong Kong Advanced Level Examination (HKALE) for university entrants. In both examinations, it is stated that in marking writing papers, attention will be paid both to language accuracy and to the organisation and coherence of the argument presented (Retrieved December 2008 from Hong Kong Examinations and Assessment Authority website at http://www.hkeaa.edu.hk/ DocLibrary/HKALE/Subject_and_Syllabuses/2008as(e)_ue.pdf/).

2.

3.

4. 5. 6. 7.

8.

Computer Assisted Language Learning

9. 10. 11.

71

The T-test is also used with a small sample size (range from 13 to 77 subjects) in such disciplines as social work (e.g., Palihawadana & Holmes, 1999; Davis, 2003; Quinn 2006) and IT in education (e.g., Schou, 2007). The team has further developed four topics. The team is investigating the eectiveness of the pedagogy suggested in the paper in a Hong Kong secondary school.

Notes on contributors
Cynthia Lee is an associate professor in the Language Centre, Hong Kong Baptist University. Her research areas are second language teaching and learning, interlanguage pragmatics, IT and language learning. Kelvin Chi Kuen Wong is an assistant professor in the Department of Computer Science, Hong Kong Baptist University. He received his PhD from the University of Nottingham. His research interests include e-learning, IT in education, Web-based assessments and applications of latent semantic analysis. William K. Cheung is an associate professor in the Department of Computer Science, Hong Kong Baptist University. He received his PhD degree in computer science from the Hong Kong University of Science and Technology. His research interests include articial intelligence and machine learning, as well as their applications to collaborative ltering, Web mining, distributed data mining, and Web/grid service management. Fion S.L. Lee is a lecturer in the Department of Computer Science, Hong Kong Baptist University. Her research interests include behaviour in online communities, online learning, electronic business, and information technology strategy.

References
Braine, G. (1997). Beyond word processing: Networked computers in ESL writing classes. Computers and Composition, 14(1), 4558. Braine, G. (2001). A study of English as a foreign language (EFL) writers on a local-area network (LAN) and in traditional classes. Computers and Composition, 18(3), 275292. Braine, G. (2004). Teaching second and foreign language writing on LANs. In S. Fotos & C. Browne (Eds.), New perspectives on CALL for second language classrooms (pp. 93108). Mahwah, New Jersey, London: Lawrence Erlbaum Associates. Chapelle, C.A. (2001). Computer applications in second language acquisition: Foundations for teaching, testing and research. Cambridge: Cambridge University Press. Chapelle, C.A., & Hegelheimer, V. (2004). The language teacher in the 21st century. In S. Fotos & C. Browne (Eds.), New perspectives on CALL for second language classrooms (pp. 299316). Mahwah, New Jersey, London: Lawrence Erlbaum Associates. Chen, Q.F., & Cheng, W.Y. (2006, May). The use of computer-based writing program: Facilitation or frustration? Paper presented at the 23rd International Conference on English Teaching and Learning in the Republic of China. Retrieved December 1 2008 from http://eric.ed.gov/ ERICDocs/data/ericdocs2sql/content_storage_01/00000196/80/1b/d7/26.pdf. Cheung, W., Mrch, A.I., Wong, K., Lee, C., Liu, J.M., & Lam, M.H. (2007). Grounding collaborative learning in semantics-based critiquing. International Journal of Distance Education Technologies, 5(2), 4055. Davis, S. (2003). Statistic anxiety among female African American graduate-level social work students. Journal of Teaching in Social Work, 23(3/4), 143158. Foltz, P.W., Gilliam, S., & Kendall, S. (2000). Supporting content-based feedback in on-line writing evaluation with LSA. Interactive Learning Environment, 8(2), 111127. Goldstein, L.M. (2005). Teacher written commentary in second language writing classrooms. Ann Arbor: The University of Michigan Press. Jakeman, V., & McDowell, C. (1999). Insights into IELTS. Cambridge: Cambridge University Press. Kintsch, E., Steinhart, D., Stahl, G., & research group (2000). Developing summarization skills through the use of LSA-based feedback. Interactive Learning Environments, 8(2), 87109. Konza, R.B., & Johnston, J. (1991). The technological revolution comes to the classroom. Change, 23(1), 1023. Retrieved December 1 2008 from http://eric.ed.gov/ERICWebPortal/custom/ portlets/recordDetails/detailmini.jsp?_nfpbtrue&_&ERICExtSearch_SearchValue_0EJ423195 &ERICExtSearch_SearchType_0no&accnoEJ423195.

72

C. Lee et al.

Landauer, T.K., Foltz, P.W., & Laham, D. (1998). An introduction to latent semantic analysis. Discourse Processes, 25(2&3), 259284. Landauer, T.K., & Psotka, J. (2000). Simulating text understanding for educational applications with latent semantic analysis: Introduction to LSA. Interactive Learning Environments, 8(2), 7386. Lemaire, B., & Dessus, P. (2001). A system to assess the semantic content of student essays. Journal of Educational Computing Research, 24(3), 305320. Palihawadana, D., & Holmes, G. (1999). Modelling module evaluation in marketing education. Quality Assurance in Education, 7(1), 4146. Pennington, M.C. (2003). The impact of the computer in second language writing. In B. Kroll (Ed.), Exploring the dynamics of second language writing (pp. 287310). Cambridge, UK, New York: Cambridge University Press. Pennington, M.C. (2004). Electronic media in second language writing: An overview of tools and research ndings. In S. Fotos & C. Browne (Eds.), New perspectives on CALL for second language classrooms (pp. 4568). Mahwah, New Jersey, London: Lawrence Erlbaum Associates. Quinn, A. (2006). Reducing social work students statistics anxiety. Academic Exchange Quarterly, June 2006. Retrieved from http://www.thefreelibrary.com/Reducingsocialworkstudents statisticsanxiety-a0149613325. Rogers, B. (2007). The complete guide to the TOEFL test (IBT edition). Boston, Mass: Thomson. Schou, S.B. (2007). A study of student attitudes and performance in an online introductory business statistics class. Electronic Journal for the Integration of Technology in Education, 6, 7178. Retrieved December 11, 2008 from http://ejite.isu.edu/Volume6/Schou.pdf. Steinhart, D.J. (2001). Summary street: An intelligent tutoring system for improving student writing through the use of latent semantic analysis. Unpublished PhD thesis, Department of Psychology, University of Colorado, Boulder. Strang, G. (1980). Linear algebra and its applications (2nd edition.). London: Academic Press. Sullivan, N., & Pratt, E. (1996). A comparative study of two ESL writing environments: A computer-assisted classroom and a traditional oral classroom. System, 29(4), 491501. Warschauer, M. (2004). Technological change and the future of CALL. In S. Fotos & C. Browne (Eds.), New perspectives on CALL for second language classrooms (pp. 1526). Mahwah, New Jersey: Lawrence Erlbaum Associates. Wong, K., Lee, F., Lee, C., Cheung, W., Morch, A.I., & Liu, J.M. (2007, August). A pilot study on the impact of the web-based essay critiquing system on writing at the tertiary level. Proceedings of the 2007 International Joint Conference on e-Commerce, e-Administration, e-Society and e-Education, Hong Kong. Appendix 1. Marking criteria Content and Organisation (110) Rich and relevant supporting evidence/materials are given. The ideas in the text are clearly organised. A clear standpoint is provided. Supporting details/materials are given to a good extent. The ideas in the text are fairly well organised and coherent. A clear standpoint is provided. Some supporting details/ materials are given. The ideas in the text are organised and coherent. A clear standpoint is provided. Brief and general supporting details/materials are given. The ideas in the text are poorly organised and barely coherent. Standpoint is unclear. Little or no supporting details/materials are given. The ideas in the text are very disorganised and not coherent. Standpoint is unclear. 910

78

56

34

12

Total score ________ (out of 10)