Shehu & Tidwell 1 Lina Shehu Jenna Tidwell EDUC 8540 Dr.

Jean Turner 14 May 2013 Original Test Project China International Publishing Group Original Test Specifications With the purpose of using this assessment as a part of the curriculum of a cooperative exchange program here at the Monterey Institute of International Studies, our test will be created with the students participating in the China International Publishing Group as the primary target audience. These students are adult Chinese nationals who currently work for the China International Publishing Group, hereafter referred to as CIPG, and will be participating in an exchange program which will allow them to live in Monterey and attend courses at MIIS during the fall semester of 2013. This program is a cooperative venture between CIPG and MIIS, which aims to enhance the students' editing, writing, translating, listening, reading, and speaking skills in the English language. Yet, to be chosen to participate in this program, these students must already have a high command of the English level, and most of the participants enter the program with a high to advanced level of proficiency in the target language. This particular assessment will be given at the beginning of the program and will act as a pretest that allows instructors to gather information concerning what tasks the students can perform without difficulty and to gauge the level of English that the learners possess at the beginning of the program in order to better tailor the lessons and coursework to the needs of these individuals. This test will measure two constructs: the students' reading and writing skills in the English language. There will also be a self-assessment portion of the test that will further enhance the instructors' abilities to tailor the

Shehu & Tidwell 2 coursework to these particular students. Therefore, this test will separate consist of three sections: the reading section, the writing section, and the self-assessment section. Each of these sections will be scored and evaluated independently due to the fact that they will be used as a part of a needs assessment. Due to the fact that these students are currently employed by a publishing company, this test aims to assess skills that could be considered quite important in the world of publishing, both reading comprehension and writing in a foreign language. After reviewing previous student evaluations from the years 2010 and 2012, it was concluded that CIPG students are generally quite interested in traveling and American culture. Therefore, the authentic materials utilized in this assessment will include travel based content. Reading Sections The reading sections of this test will aim to evaluate how well the students participating in the CIPG program are able to read and comprehend a short passage in English. This task seems appropriate for these particular learners due to the fact that they are expected to have an advanced level of English and that they will need to be able to comprehend passages in order to correctly translate them into their native language of Chinese. Continuing with the theme of traveling, the passages that the students are asked to read will be authentic and will come directly from Sunset Magazine, which features articles concerning Wine Country in California. An example of one of the passages that may appear on the test would be as follows: The sky is dark, the kind of inky black that only exists without a single streetlight. We bounce down the washboard dirt road and, when it forks, turn toward our destination: the areas best new restaurant. Or is it the other way? The road keeps twisting and branching off. My husband, Todd, grips the official map of Baja Californias main wine region, Valle de Guadalupe. All day we used this map, with its cartoonish purple icons of grape bunches indicating wineries, to sip our way among the valleys 50 wineries. Visiting them often means venturing off the areas three paved roads onto rutted dirt ones, but Todd laughs now at our assumption that the straight brown dotted line we took to be a shortcut would translate to a straight road. He tosses the map to the floor, I pull a U-turn, and we carefully retrace our path back to the highwaya paved two-laner that links the port city of

Shehu & Tidwell 3 Ensenada, 20 miles to the west, to the border town of Tecate. This is a booming wine country, with new tasting rooms, a couple of fine restaurants, and hotels with high-thread-count sheets, but it still has a wild side. And for a moment, the utter darkness gives me a familiar feeling, like the one I get hiking in the wilderness or paddling my surfboard into a wave. The natural world is powerful. After reading a passage such as this one, students will be asked to answer eight multiplechoice questions, which have four optional answers each: A, B, C, and D. Students will then repeat this process and will read a second passage and answer eight more multiple-choice questions. Therefore, in total, the students will answer a total of sixteen multiple-choice questions that will all be scored objectively. An example of a multiple-choice question that may appear on the final version of the test would be as follows: The main theme of this article is to: A. talk about how easy it is to get lost in California. B. introduce one of Californias wineries. C. express how difficult it is to travel with a spouse. D. none of the above An answer key will be created that instructors will be able to use to score this portion of the test. Each multiple-choice question will be worth one point. A student will earn one point if he or she answers the question correctly but will not be penalized for answering a question incorrectly. If a student answers 0-4 questions correctly, he or she will be considered to have a low level of reading comprehension in English. Answering 5-8 questions correctly will be considered as an intermediate level, answering 9-12 questions will be considered as a high level, and answering 13-16 questions correctly will be considered as an advanced level. Writing Sections For the writing portion of this assessment, the theme of traveling will continue, but students will then be asked to produce an original text. This portion of the test will also be split into two sections, and the students will be asked to create two original short stories. The genre of short story was chosen

Shehu & Tidwell 4 due to the fact that the students are working in a publishing house and also in hopes that this task would be fun and engaging for the students. Writing prompts are often open-ended, and this test will follow that example. The students will be provided with two travel related photographs and asked to compose a short story of no more than 150 words. An example of a photograph that might appear on the final version of the test could look as follows:

2012 COND NAST. ALL RIGHTS RESERVED.

The stories that the students compose will be subjectively scored based on a rubric used by ACTFL, which scores students based on criteria in six different sections: task completion, comprehensibility, level of discourse, vocabulary, language control, and mechanics. Students will be awarded a score between 0 and 6 based on the quality of their composition, with a perfect score being 36. Self-assessment Section For the self-assessment portion of this test, students will be asked to rate their own abilities based on how well they feel they can accomplish a certain task. For example, students will be presented with a statement such as, I can understand native speakers of English at the grocery store, and will then be asked to rate themselves on a likert scale with one being I strongly agree, two being I agree, three being I disagree, and four being I strongly disagree. The statements that will be presented to the

Shehu & Tidwell 5 students in the self-assessment portion of the test will be based on the curricular goals of the CIPG exchange program. For example, students will be asked to rate their own speaking, listening, reading, and writing skills in English due to the fact that this program aims to improve these skills, and this information could greatly influence and aid instructors and program directors in the subsequent design of the coursework and lesson plans. The scores that students receive on this portion of the test will be interpreted only in the sense that they will affect the needs assessment process of the instructors of the program. Development of the Test We decided to assess students reading and writing skills in English, because CIPG students will have to read and comprehend various texts in their classes, and will additionally be required to keep a blog about their classes, tasks, and extracurricular activities while at MIIS. The decision to create a diagnostic test was made because we wanted to design a curriculum that is appropriate to their level of English. The CIPG program coordinator here at MIIS, Ms. Heather Kokesch, informed us that the majority of students from CIPG are highly proficient in English. Therefore we chose texts that are slightly challenging, Flesch-Kincaid Reading Ease of 60 and 59.4. but appropriate to an advanced speaker of English. The objectively scored portion of the text features two texts, the first with 7 multiple choice items and the second with 8 multiple choice items. The subjectively scored portion calls for a minimum of one written paragraph. Originally, we only had one text in the objectively scored portion, but decided to include another text for several reasons. First, it was almost impossible to have 15 multiple choice items on a text that has a readability score of 60. The questions would have simply become redundant. Second, we wanted to include two different genres on travel. Our multiple choice item questions were revised several times because the goal was to create a challenging task for our students

Shehu & Tidwell 6 but not overwhelming. Furthermore, the wrong answers were revised so that they are not immediately obvious to be the wrong answer. Original Test China International Publishing Group Diagnostic Test Directions: Please read the passage below and answer the following multiple choice questions by circling the letter of the correct answer. The Road to Anatolia Yugoslav-Greek frontier When you leave Yugoslavia for Greece, the blue-the color of the Balkans- follows you, but its nature changes; you move from a slightly muted night-blue to an intensely gay sea-blue, which affects the nervous system like caffeine. That's just as well, as the rhythm of conversations and exchanges speeds up considerably. You've got into the habit of explaining things slowly-usually twice rather than onceand of pausing over words for comprehension to catch up... Constantinople The very morning we arrived, we ferried the car across to the Asian shore and were prowling through the little streets of the Moda neighborhood in search of a lodging that would beckon to us, when a frail but imperious voice calling out in French made us turn round. it belonged to a large woman with snowwhite hair, wearing a heavy amethyst brooch and elegant mourning. From the top of her steps she gazed thoughtfully at our bags, as if they reminded her of something, and asked what we were looking for. We explained. 'My season ended last week, but I've kept on my servants and I rather like travelers. You can stay here.' With her cigarette-holder, she indicated a little gold plate above the door: Moda-Palas. In silence our bags were taken across a sombre Victorian dining room. On the sideboard, a mustard cat was asleep between ornate Christofle teapots. The room gave on to a faded garden, and had a light but distinct odor of polish and mildew. Except for a chambermaid, and madame Wanda, the proprietress, the hotel was deserted and, with its shutters closed, more intimidating than a tomb. 1. The sentence an intensely gay-sea blue, which affects the nervous system like caffeine suggests that the author is: A. negatively stimulated B. positively stimulated C. not stimulated 2. What does and of pausing over words for comprehension to catch up mean? A. The author believes one must speak slowly in order to be understood. B. The author is making fun of the people with whom he is speaking. C. The author conceives of comprehension as a person that is physically catching up.

Shehu & Tidwell 7 3. The sentence we ferried the car across to the Asian shore means: A. The author drove really fast across a bridge to the Asian shore. B. The author drove a stolen car across to the Asian shore. C. The author drove his car onto a large boat that carried it across to the Asian shore. 4. What does in search of a lodging that would beckon to us mean? A. The author is searching for lodging that appeals to him. B. The author means to say that he is unable to find lodging. C. The author is commenting on the neighborhoods shabby lodgings. 5. When the French woman says my season ended last week, she means: A. the time for renting out her lodgings is past B. She is complaining about the change of seasons. C. She is implying that she will charge them double for lodging. 6. Why does the author compare the hotel to a tomb? A. because it smells like polish and mildew B. because the hotel is quiet and dark C. the hotel is built on an old cemetery 7. What is the genre of this passage? A. It is an excerpt from a major newspaper. B. It is an excerpt from a travelogue. C. It is an excerpt from a complaint letter.

Directions: Please read the passage below and answer the following multiple choice questions by circling the letter of the correct answer. Explore Valle de Guadalupe Wine Country The sky is dark, the kind of inky black that only exists without a single streetlight.We bounce down the washboard dirt road and, when it forks, turn toward our destination: the areas best new restaurant. Or is it the other way? The road keeps twisting and branching off. My husband, Todd, grips the official map of Baja Californias main wine region, Valle de Guadalupe. All day we used this map, with its cartoonish purple icons of grape bunches indicating wineries, to sip our way among the valleys 50 wineries. Visiting them often means venturing off the areas three paved roads onto rutted dirt ones, but Todd laughs now at our assumption that the straight brown dotted line we took to be a shortcut would translate to a straight road. He tosses the map to the floor, I pull a Uturn, and we carefully retrace our path back to the highwaya paved two-laner that links the port city of Ensenada, 20 miles to the west, to the border town of Tecate. This is a booming wine country, with new tasting rooms, a couple of fine restaurants, and hotels with

Shehu & Tidwell 8 high-thread-count sheets, but it still has a wild side. And for a moment, the utter darkness gives me a familiar feeling, like the one I get hiking in the wilderness or paddling my surfboard into a wave. The natural world is powerful. Only a few minutes later, we walk through the door of Corazn de Tierra. The sleek one-room restaurant is a Venice-modern oasis, with a large garden off the back deck. Diners can either sit facing rows of beets, basil, and arugula or watching chef Diego Hernandez and his team grilling, searing, and assembling food in the open kitchen. There are no menus. Its nature that decides, says Hernandez, 29, who opened Corazn de Tierra two years ago and cooks and talks about food with equal gusto. Here in Baja, we have a palette of flavors, like a painter. Each dish that comes is a surprise: a pesto of beet greens served with a thick hunk of rosemary bread thats been both baked and smoked. A salad of broccoli, radishes, and Ramonetti (a soft, local cheese akin to a blend of brie and manchego) with an orange calendula flower and parsnip pure artfully arranged on a slab of slate. Then its a piece of perfectly seared fresh tuna bathed in a tangy sour-milk and mint-oil broth, served in a warm stone bowl. After that, roasted partridge with carrots, roasted pumpkin pure, and beef sauce, paired with a spicy Vena Cava Tempranillo from the winery at Villa del Valle, the B&B where Corazn de Tierra is located. Finally, its Hernandezs dessert oeuvre, each tasting portion made from fresh oranges: ice cream, panna cotta, marmalade cake, and a cookie. Hernandez is building on the regions farm-to-table legacy established by the valleys first and most renowned farm-to-table restaurant, Laja, where during a delicious, epic meal the night before, I learned that the restaurants refrigerators and freezers often sit emptythe fresh ingredients never make it there. We grew up eating sashimi and ceviches, says Hernandez of his childhood in Ensenada. Now he delivers his own sophisticated takea mix of Mexican, Mediterranean, and Asianhoned at top restaurants in Mexico City and Tijuana. Hernandez says that visitors who come expecting only traditional Mexican cuisine are surprised. The truth is, this is the other Mexico. 1. Why does the author use the metaphor inky black to describe the sky? A. to convey the darkness of the sky to the reader B. to explain the need for streetlights C. to lament getting lost D. to show off her writing skills 2. The sentence the road keeps twisting and branching off is used to convey: A. the road looks like a tree B. the dark and labyrinthine setting C. the authors hatred of rural areas D. the need for a map 3. The second paragraph is about: A. the difficulty of traveling with a spouse B. some of Californias roads C. some of Californias wineries in the Baja region D. the reasons why the author and her husband got lost

Shehu & Tidwell 9 4. The word booming can be understood as: A. extraordinarily loud B. quickly prospering C. slowly developing D. exceptionally beautiful 5. Why does the author say the natural world is powerful? A. because she is frightened of it B. because despite the darkness in the wine country, she can still see its beauty C. because she loves hiking and surfboarding D. because despite the fine restaurants and hotels, the area has a wild side 6. What does Chef Hernandez mean when he says its nature that decides? A. He is referring to the power of the natural world. B. He is referring to his gardens abundance. C. He is referring to the vegetables that are in season. D. He is referring to the wild atmosphere in his restaurant. 7. Is the authors view of the restaurant that of: A. appreciation B. dismay C. shock D. disappointment 8. In this context what does farm-to-table mean? A. The restaurant, Laja, is the only one who does it. B. The ingredients used at the restaurant are very fresh. C. The majority of Mexican restaurants are expensive. D. The food at Corazon de Tierra is quite unusual.

Shehu & Tidwell 10 Directions: Look at the picture below. Imagine that the two men in the picture are traveling in California and write a brief (one paragraph) short story about their adventure.

Shehu & Tidwell 11 Directions: Considering your own abilities, answer the following questions by circling True or False. 1. I can read and understand a magazine in English. True 2. I can read and understand academic texts in English. True 3. Writing fiction short stories in English is difficult for me. True 4. When I write in English, I wish that my vocabulary was larger. True 5. The questions about the reading passage were easy. True False False False False False

6. I had to reread the text many times to answer the questions about the reading passage. True Scoring Key for the Objectively Scored Section First Text 1. B 2. A 3. C 4. A 5. A 6. B 7. B Second Text 1. A 2. B 3. C 4. B 5. D 6. C 7. A 8. B False

Shehu & Tidwell 12 Information on the scoring protocol for the subjectively scored section can be found in the revised test specifications section in the form of a rubric used as a part of the TOEFL PBT Writing and Structure Section. Guide to Test Administration In order to administer this test, instructors involved in the CIPG program will not need to undergo extensive training. Test administrators will be in charge of transporting paper versions of the test to the testing site, along with extra writing utensils in case of test taker emergency. Test administrators will control the classroom and ensure that students are seated and remain silent throughout the duration of the test. Students will be allotted the time necessary to complete the test, and they will not be allowed to use any outside sources of aide such as a dictionary or the internet. Test administrators will pass out the tests to the students, remain in the classroom as the test is being given, collect the tests as students complete them, and transport the tests away from the test site once each student has finished. Test administrators will also not be allowed to answer any questions concerning the content of the test before or during the actual test administration to ensure that students complete the test without assistance. Actual Test Administration Due to time constraints and a lack of eager and willing participants, the actual trial administration of this test was not performed in a controlled setting as it idealistically would be when functioning as a part of the CIPG program. The test designers recruited a total of eight nonnative English speakers to complete the test, many of whom were students at the Monterey Institute of International Studies. The test takers were recruited on the basis that they were, first and foremost, nonnative English speakers, and secondly, that they were able to communicate with ease with native speakers of English. These requirements were set in hopes of emulating the levels of English of the China International Publishing Group students.

Shehu & Tidwell 13 The recruited test takers were allowed to take their tests home with them and return them to the test designers at a later time. Yet, the test takers were instructed to refrain from using any type of dictionary or outside aide while they completed the test, although there was no set time limit for completing the test. While there is no certainty that the test takers did not consult a dictionary or another source of aide while completing the test, the test designers found that allowing participants to take their tests home was the only option to successfully recruit enough participants. All of the tests were given to the test takers in a paper format; however, some test takers chose to scan their tests after completion and returned them electronically. Revised Test Specifications With the purpose of using this assessment as a part of the needs assessment process for a cooperative exchange program here at the Monterey Institute of International Studies, our test will be created with the students participating in the China International Publishing Group as the primary target audience. These students are adult Chinese nationals who currently work for the China International Publishing Group, hereafter referred to as CIPG, and will be participating in an exchange program which will allow them to live in Monterey and attend courses at MIIS during the fall semester of 2013. This program is a cooperative venture between CIPG and MIIS, which aims to enhance the students' editing, writing, translating, listening, reading, and speaking skills in the English language. Yet, to be chosen to participate in this program, these students must already have a high command of the English level, and most of the participants enter the program with a high to advanced level of proficiency in the target language. This particular assessment, which will be diagnostic in nature, will be given at the beginning of the program and will act as a pretest that allows instructors to gather information concerning what tasks the students can perform without difficulty and to gauge the level of English that the learners possess at the beginning of the program in order to better tailor the lessons and coursework to the needs of these

Shehu & Tidwell 14 individuals. This test will measure two constructs: the students' reading and writing skills in the English language. There will also be a self-assessment portion of the test that will further enhance the instructors' abilities to tailor the coursework to these particular students. Therefore, this test will separate consist of three sections: the reading section, the writing section, and the self-assessment section. Each of these sections will be scored and evaluated independently due to the fact that they will be used as a part of a needs assessment, and the designers of this test would like to know where each student stands in the areas that are being assessed. By having the students complete a reading comprehension section, generate a written text, and assess their own ability levels, the instructors of the course will be able to better understand the levels of each student in order to better tailor the course to their specific needs. Due to the fact that these students are currently employed by a publishing company, this test aims to assess skills that could be considered quite important in the world of publishing, both reading comprehension and writing in a foreign language. After reviewing previous student evaluations from the years 2010 and 2012, it was concluded that CIPG students are generally quite interested in traveling and American culture. Therefore, the authentic materials utilized in this assessment will include travel based content. Reading Sections The reading sections of this test will aim to evaluate how well the students participating in the CIPG program are able to read and comprehend a short passage in English. This task seems appropriate for these particular learners due to the fact that they are expected to have an advanced level of English and that they will need to be able to comprehend passages in order to correctly translate them into their native language of Chinese. Continuing with the theme of traveling, the passages that the students are asked to read will be authentic and will come directly from Sunset Magazine, which features articles concerning Wine Country in California. In order to make the course engaging and interesting for the

Shehu & Tidwell 15 CIPG students, the designers of the curriculum have decided to create a content-based curriculum focused on traveling. Therefore, the designers of this test made the decision to also use travel-based content in this diagnostic test. An example of one of the passages that may appear on the test would be as follows: The Road to Anatolia Yugoslav-Greek frontier When you leave Yugoslavia for Greece, the blue-the color of the Balkans- follows you, but its nature changes; you move from a slightly muted night-blue to an intensely gay sea-blue, which affects the nervous system like caffeine. That's just as well, as the rhythm of conversations and exchanges speeds up considerably. You've got into the habit of explaining things slowly-usually twice rather than onceand of pausing over words for comprehension to catch up... Constantinople The very morning we arrived, we ferried the car across to the Asian shore and were prowling through the little streets of the Moda neighborhood in search of a lodging that would beckon to us, when a frail but imperious voice calling out in French made us turn round. it belonged to a large woman with snow-white hair, wearing a heavy amethyst brooch and elegant mourning. From the top of her steps she gazed thoughtfully at our bags, as if they reminded her of something, and asked what we were looking for. We explained. 'My season ended last week, but I've kept on my servants and I rather like travelers. You can stay here.' With her cigarette-holder, she indicated a little gold plate above the door: M oda-Palas. In silence our bags were taken across a sombre Victorian dining room. On the sideboard, a mustard cat was asleep between ornate Christofle teapots. The room gave on to a faded garden, and had a light but distinct odor of polish and mildew. Except for a chambermaid, and madame Wanda, the proprietress, the hotel was deserted and, with its shutters closed, more intimidating than a tomb. The following data were retrieved from Readability-Score.com. The first text, The Road to Anatolia has a Flesch-Kincaid Reading Ease of 60. The second text, Explore Valle de Guadalupe Wine Country, has a Flesch-Kincaid Reading Ease of 59.4. Since both texts have a very similar score, the data below apply to both texts. The Flesch-Kincaid Scores range from 0-100. Higher scores indicate easier to read material. Scores over 22 should generally be taken to mean graduate level text. Scores ranging from 60-70 should be understood by 13 to 15 year old students. The Gunning fog index for our texts was 11.7. Texts that are accessible for a wide audience generally need a fox index less than 12. The Average Grade Level for our texts was 10.4. That means the texts are comprehensible for high

Shehu & Tidwell 16 school sophomores. The multiple choice items on our objectively scored portion of the test, were designed to measure students comprehension of a passage from two different genres: a travelogue, and a magazine article. The CIPG students for whom this test was designed will be reading similar texts in their English classes at MIIS. We wanted to assess their reading comprehension so that we may develop a curriculum applicable to their skill and level of English. Furthermore the multiple choice items measure students knowledge of English metaphors and idiomatic expressions. We believe both are important since they add to a students communicative competence. In addition we purposefully chose texts about travel, because our curriculum will include travel as complementary to students studies and we wanted to familiarize them with different genres on travel. After reading a passage such as this one, students will be asked to answer eight multiplechoice questions, which have four optional answers each: A, B, C, and D. Students will then repeat this process and will read a second passage and answer eight more multiple-choice questions. Therefore, in total, the students will answer a total of sixteen multiple-choice questions that will all be scored objectively. An example of a multiple-choice question that may appear on the final version of the test would be as follows: 2. What does and of pausing over words for comprehension to catch up mean? A. The author believes one must speak slowly in order to be understood. B. The author is making fun of the people with whom he is speaking. C. The author conceives of comprehension as a person that is physically catching up. An answer key will be created that instructors will be able to use to score this portion of the test. Each multiple-choice question will be worth one point. A student will earn one point if he or she answers the question correctly but will not be penalized for answering a question incorrectly. The designers of this test have created their own scoring guide and will interpret the levels of the students based on how many correct responses they provide, which therefore implies how much of the text a

Shehu & Tidwell 17 student understood. Students will be evaluated on their comprehension of the text on four levels. If a student answers 0-4 questions correctly, he or she will be considered to have understood 25% or less of the text, therefore having a low level of reading comprehension in English. Answering 5-8 questions correctly will imply that a student understood 50% or less of the text, and he or she will be considered to have an intermediate level. Answering 9-12 questions will be considered as a high level due to the student being able to understand 51-75% of the text, and answering 13-16 questions correctly will be considered as an advanced level, as a student will have been able to comprehend 76-100% of the given text. Writing Sections For the writing portion of this assessment, the theme of traveling will continue, but students will then be asked to produce an original text. This portion of the test will also be split into two sections, and the students will be asked to create two original short stories. The genre of short story was chosen due to the fact that the students are working in a publishing house and also in hopes that this task would be fun and engaging for the students. Writing prompts are often open-ended, and this test will follow that example. The students will be provided with two travel related photographs and asked to compose a short story of no more than 150 words. An example of a photograph that might appear on the final version of the test could look as follows:

Shehu & Tidwell 18

The stories that the students compose will be subjectively scored based on a rubric used to score the TOEFL PBT, which can be seen below.
Writing Scoring Guide The following scoring guidelines relate to the TOEFL PBT Test Writing & Structure section. Score of Six An essay at this level: shows effective writing skills is well organized and well developed uses details clearly and properly to support a thesis or illustrate ideas displays consistent ability in the use of language demonstrates variety in sentence structure and proper word choice Score of Five An essay at this level: may address some parts of the task more effectively than others is generally well organized and developed uses details to support a thesis or illustrate an idea displays ability in the use of the language shows some variety in sentence structure and range of vocabulary Score of Four An essay at this level:

Shehu & Tidwell 19

addresses the writing topic adequately but does not meet all of the goals of the task is adequately organized and developed uses some details to support a thesis or illustrate an idea shows adequate but possibly inconsistent ability with sentence structure may contain some usage errors that make the meaning unclear Score of Three An essay at this level may reveal one or more of the following weaknesses: inadequate organization or development poor choice of details or does not provide enough details to support or illustrate generalizations a noticeably improper choice of words or word forms numerous errors in sentence structure and/or usage Score of Two An essay at this level is seriously flawed by one or more of the following weaknesses: serious disorganization or underdevelopment little or no detail, or irrelevant specifics serious and frequent errors in sentence structure or usage serious problems with focus Score of One An essay at this level: may be incoherent may be undeveloped may contain severe and persistent writing errors Score of 0 An essay will be rated 0 if it: contains no response merely copies the topic is off-topic, written in a foreign language or consists only of keystroke characters

Self-assessment Section For the self-assessment portion of this test, students will be asked to rate their own abilities based on how well they feel they can accomplish a certain task. For example, students will be presented with a statement such as, Writing fiction short stories in English is difficult for me, and will then be asked to rate themselves by answering either true or false in response to the given statement. The

Shehu & Tidwell 20 statements that will be presented to the students in the self-assessment portion of the test will be based on the curricular goals of the CIPG exchange program. For example, students will be asked to rate their own reading and writing skills in English due to the fact that this program aims to improve these skills, and this information could greatly influence and aid instructors and program directors in the subsequent design of the coursework and lesson plans. The scores that students receive on this portion of the test will be interpreted only in the sense that they will affect the needs assessment process of the instructors of the program. Therefore, the activities and lessons planned for the course may be subject to change after the instructors obtain this information and are aware of what areas in which students believe they need help. Descriptive Statistics For the subjectively scored portion of the test, the test takers were scored using a rubric with a scale of 1 to 6. We entered the test takers scores on the computer program known as R in order to calculate the mean and the standard deviation (sd). Two different scorers scored the subjectively scored portion of the assessment. The first set of scores was: 5, 4, 4, 4, 4, 6, 6, 3. The second set of scores was: 4, 4, 4, 4, 5, 6, 6, 2. The mean for the first set was: 4.5, and the mean for the second set was: 4.375, which shows that the first rater gave slightly higher scores than the second. The mode for the scores of the subjectively scored portion of the exam was 4, which appeared four times in both sets of scores. The median of these scores was also 4. The sd for the first set was: 1.069045, and the sd for the second set was: 1.30247. As for the objectively scored section, there were 15 multiple-choice questions that were worth one point each, making the section worth a total of 15 points. The scores for this section were: 4, 9, 11, 11, 11, 11, 12, and 13. The mode for this section of the test was 11, which was the median score as well. The mean of these scores was 10.25. The standard deviation for the scores in this section was 2.56.

Shehu & Tidwell 21 Internal Consistency

In order to calculate the internal consistency of the objectively scored items, the test designers entered the scores into the reliability calculator and found that the estimated internal consistency was 0.91 using the Spearman-Brown Prophecy formula. This number was much larger than anticipated and shows that this test has a high rate of internal consistency, which speaks favorably of the design of this assessment. Item Difficulty According to the chart below, questions 3, 7, and 14 were too easy for test takers because every single tester answered it correctly, whereas question 10 was too difficult because no one answered correctly. Questions 4, 5, 8, 13 and 15 were also a little easy since the majority of test takers answered correctly, whereas question 2 was quite difficult since relatively few people responded with the correct answer.

Shehu & Tidwell 22 Question Number 1 2 3 4 5 6 7 8 9 10 11 12 13 Item Facility .5 .25 1 .875 .875 .75 1 .875 .375 0 .75 .375 .875

Shehu & Tidwell 23 Item Discrimination Question Number 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 High 1 0.5 1 1 1 1 1 0.5 0.5 0 1 1 1 1 1 Low 0 0.5 1 0.5 0.5 0.5 1 0.5 0 0 0 0 0.5 1 0.5 Item Discrimination 1 0 0 0.5 0.5 0.5 0 0 0.5 0 1 1 0.5 0 0.5

The test designers found that item 10 did not discriminate well due to the fact that none of the test takers answered this item correctly. Similarly, since all of the examinees answered items 3, 7, and 14 correctly, it was determined that these questions did not discriminate well either. Therefore, these questions could possibly be deemed as too easy for the test takers, just as item 10 could be viewed as too difficult. Distractor Content Analysis We took great care in creating multiple choice items answers, which were challenging but not overwhelming. Obviously, some were simply too challenging and others were too easy, nonetheless we feel that we inserted distractor content in our multiple choice item answers that made our test interesting and creative. For example: on question number 1, The sentence an intensely gay sea-blue,

Shehu & Tidwell 24 which affects the nervous system like caffeine suggests that the author is a. positively stimulated, b. negatively stimulated and c. not stimulated, we thought that since many people are negatively affected by caffeine they may be tempted to just answer b, even though the sentence in its context suggests that the author was positively stimulated. For question number 2 we wanted to test knowledge of idiomatic expression thus: what does pausing over words for comprehension to catch-up mean? one of the distractors we provided was c. the author conceives of comprehension as a person that is physically catching up. This idiomatic expression, however, means to be understood. On question 6 we asked: why does the author compare the hotel to a tomb? and one of the distractors was answer c. because it was built on an old cemetery. Subjectively Scored Writing Section Interrater Reliability For the subjectively scored writing section of the test, both of the test designers used the TOEFL PBT Writing Rubric, as mentioned in the revised test specifications, to score the narratives produced by the test takers. After both test designers had independently scored all of the tests, the interrater reliability was calculated using Spearman rho with the aide of the computer program known as R. Through the use of this program, the interrater reliability was found to be .83, which was quite high for this type of diagnostic assessment. This number also came as a shock to the test designers since neither had participated in any formal training on how to properly use the writing rubric for scoring the tests, and the expectation was that the interrater reliability would be quite for two novice test designers and scorers. However, inferences could be made that the rubric used was very clear and self-explanatory due to the fact that the scorers were able to achieve such a high rate of interrater reliability without any formal training. Content Analysis The content of the narratives that were written by the test takers for the subjectively scored

Shehu & Tidwell 25 section of the assessment was also quite surprising for the test designers. Overall, the narratives that were written were well-organized, creative, quite detailed, and relatively free of major grammatical errors. All of the narratives were one paragraph in length, as specified in the directions, and some even contained extensive dialogue. This section of the test was deemed successful as little to no changes or revisions were needed due to the fact that the test takers successfully followed all of the directions, a substantial amount of ratable text was produced, and future instructors of the CIPG program would be able to easily decipher the levels of incoming students' written English based on their completion of this task. Self-Assessment Section After analyzing the answers provided in the self-assessment portion of the exam, it was concluded that many of the test takers found this assessment to be difficult. Many of the students circled the word false in response to the statement, The questions about the reading passage were easy. Also, many of the students claimed that they were forced to reread the passage several times in order to answer the multiple-choice questions with accuracy. In addition, many of the test takers claimed that they wished that their vocabularies in English were larger and that they found it difficult to write short fiction narratives in English. Therefore, these answers could help the instructors working in the CIPG program choose what to focus their lessons on as they begin preparing for the upcoming course. Yet, there were portions of the self-assessment section that could and should be improved upon before the test is given again. When the test was first given, the self-assessment section asked the test takers to circle either true or false in response to a given statement. However, after analyzing the data, the test designers concluded that the responses to these statement could have been more useful if the test takers were required to rate themselves on a Likert scale. With this revision, test takers would respond to the same statements, but instead of circling true or false, they would be asked to circle a

Shehu & Tidwell 26 number between one and four, with one being strongly disagree, two being disagree, three being agree, and four being strongly agree. This type of scale was mentioned in the original test specifications, and the test designers agree that these types of answers could have provided them with more specific information about how the students rate their own abilities in both reading comprehension and writing in English. Revisions to the Objectively Scored Section After analyzing item facility we have decided to revise the following questions: 2 and 10, because very few people answered 2 correctly, and no one answered 10 correctly. Below are the original questions followed by the revisions. 2. What does and of pausing over words for comprehension to catch up mean? A. The author believes one must speak slowly in order to be understood. B. The author is making fun of people with whom he is speaking. C. The author conceives of comprehension as a person that is physically catching up. Many test takers answered c, so we would revise c into: the author is irritated that they dont speak his native language. 10. The second paragraph is about: A. the difficulty of traveling with a spouse B. some of Californias roads C. some of Californias Wineries in the Baja region D. the reason why the author and her husband got lost None of the test takers answered question 10 correctly and we believe that is because the second paragraph mentions roads and getting lost, in addition to Californias Wineries. We would revise the second question into: Reread the second paragraph. What do you think is the authors main purpose for this trip? 10. What is the main idea of the second paragraph? A. The difficulty of traveling with a spouse. B. They are going on an adventure with no destination in mind. C. They want to visit Californias Wineries in the Baja region. D. It is a complaint about Californias highways.

Shehu & Tidwell 27 Revised Test China International Publishing Group Diagnostic Test Directions: Please read the passage below and answer the following multiple choice questions by circling the letter of the correct answer. The Road to Anatolia Yugoslav-Greek frontier When you leave Yugoslavia for Greece, the blue-the color of the Balkans- follows you, but its nature changes; you move from a slightly muted night-blue to an intensely gay sea-blue, which affects the nervous system like caffeine. That's just as well, as the rhythm of conversations and exchanges speeds up considerably. You've got into the habit of explaining things slowly-usually twice rather than onceand of pausing over words for comprehension to catch up... Constantinople The very morning we arrived, we ferried the car across to the Asian shore and were prowling through the little streets of the Moda neighborhood in search of a lodging that would beckon to us, when a frail but imperious voice calling out in French made us turn round. it belonged to a large woman with snowwhite hair, wearing a heavy amethyst brooch and elegant mourning. From the top of her steps she gazed thoughtfully at our bags, as if they reminded her of something, and asked what we were looking for. We explained. 'My season ended last week, but I've kept on my servants and I rather like travelers. You can stay here.' With her cigarette-holder, she indicated a little gold plate above the door: Moda-Palas. In silence our bags were taken across a sombre Victorian dining room. On the sideboard, a mustard cat was asleep between ornate Christofle teapots. The room gave on to a faded garden, and had a light but distinct odor of polish and mildew. Except for a chambermaid, and madame Wanda, the proprietress, the hotel was deserted and, with its shutters closed, more intimidating than a tomb. 1. The sentence an intensely gay-sea blue, which affects the nervous system like caffeine suggests that the author is: A. negatively stimulated B. positively stimulated C. not stimulated 2. What does and of pausing over words for comprehension to catch up mean? A. The author believes one must speak slowly in order to be understood. B. The author is making fun of people with whom he is speaking. C. The author conceives of comprehension as a person that is physically catching up. 3. The sentence we ferried the car across to the Asian shore means: A. The author drove really fast across a bridge to the Asian shore. B. The author drove a stolen car across to the Asian shore. C. The author drove his car onto a large boat that carried it across to the Asian shore.

Shehu & Tidwell 28 4. What does in search of a lodging that would beckon to us mean? A. The author is searching for lodging that appeals to him. B. The author means to say that he is unable to find lodging. C. The author is commenting on the neighborhoods shabby lodgings. 5. When the French woman says my season ended last week, she means: A. the time for renting out her lodgings is past B. She is complaining about the change of seasons. C. She is implying that she will charge them double for lodging. 6. Why does the author compare the hotel to a tomb? A. because it smells like polish and mildew B. because the hotel is quiet and dark C. the hotel is built on an old cemetery 7. What is the genre of this passage? A. It is an excerpt from a major newspaper. B. It is an excerpt from a travelogue. C. It is an excerpt from a complaint letter.

Directions: Please read the passage below and answer the following multiple choice questions by circling the letter of the correct answer. Explore Valle de Guadalupe Wine Country The sky is dark, the kind of inky black that only exists without a single streetlight.We bounce down the washboard dirt road and, when it forks, turn toward our destination: the areas best new restaurant. Or is it the other way? The road keeps twisting and branching off. My husband, Todd, grips the official map of Baja Californias main wine region, Valle de Guadalupe. All day we used this map, with its cartoonish purple icons of grape bunches indicating wineries, to sip our way among the valleys 50 wineries. Visiting them often means venturing off the areas three paved roads onto rutted dirt ones, but Todd laughs now at our assumption that the straight brown dotted line we took to be a shortcut would translate to a straight road. He tosses the map to the floor, I pull a Uturn, and we carefully retrace our path back to the highwaya paved two-laner that links the port city of Ensenada, 20 miles to the west, to the border town of Tecate. This is a booming wine country, with new tasting rooms, a couple of fine restaurants, and hotels with high-thread-count sheets, but it still has a wild side. And for a moment, the utter darkness gives me a familiar feeling, like the one I get hiking in the wilderness or paddling my surfboard into a wave. The natural world is powerful. Only a few minutes later, we walk through the door of Corazn de Tierra. The sleek one-room

Shehu & Tidwell 29 restaurant is a Venice-modern oasis, with a large garden off the back deck. Diners can either sit facing rows of beets, basil, and arugula or watching chef Diego Hernandez and his team grilling, searing, and assembling food in the open kitchen. There are no menus. Its nature that decides, says Hernandez, 29, who opened Corazn de Tierra two years ago and cooks and talks about food with equal gusto. Here in Baja, we have a palette of flavors, like a painter. Each dish that comes is a surprise: a pesto of beet greens served with a thick hunk of rosemary bread thats been both baked and smoked. A salad of broccoli, radishes, and Ramonetti (a soft, local cheese akin to a blend of brie and manchego) with an orange calendula flower and parsnip pure artfully arranged on a slab of slate. Then its a piece of perfectly seared fresh tuna bathed in a tangy sour-milk and mint-oil broth, served in a warm stone bowl. After that, roasted partridge with carrots, roasted pumpkin pure, and beef sauce, paired with a spicy Vena Cava Tempranillo from the winery at Villa del Valle, the B&B where Corazn de Tierra is located. Finally, its Hernandezs dessert oeuvre, each tasting portion made from fresh oranges: ice cream, panna cotta, marmalade cake, and a cookie. Hernandez is building on the regions farm-to-table legacy established by the valleys first and most renowned farm-to-table restaurant, Laja, where during a delicious, epic meal the night before, I learned that the restaurants refrigerators and freezers often sit emptythe fresh ingredients never make it there. We grew up eating sashimi and ceviches, says Hernandez of his childhood in Ensenada. Now he delivers his own sophisticated takea mix of Mexican, Mediterranean, and Asianhoned at top restaurants in Mexico City and Tijuana. Hernandez says that visitors who come expecting only traditional Mexican cuisine are surprised. The truth is, this is the other Mexico. 1. Why does the author use the metaphor inky black to describe the sky? A. to convey the darkness of the sky to the reader B. to explain the need for streetlights C. to lament getting lost D. to show off her writing skills 2. The sentence the road keeps twisting and branching off is used to convey: A. the road looks like a tree B. the dark and labyrinthine setting C. the authors hatred of rural areas D. the need for a map 3. What is the main idea of the second paragraph? A. The difficulty of traveling with a spouse. B. They are going on an adventure with no destination in mind. C. They want to visit Californias Wineries in the Baja region. D. It is a complaint about Californias highways.

Shehu & Tidwell 30 4. The word booming can be understood as: A. extraordinarily loud B. quickly prospering C. slowly developing D. exceptionally beautiful 5. Why does the author say the natural world is powerful? A. because she is frightened of it B. because despite the darkness in the wine country, she can still see its beauty C. because she loves hiking and surfboarding D. because despite the fine restaurants and hotels, the area has a wild side 6. What does Chef Hernandez mean when he says its nature that decides? A. He is referring to the power of the natural world. B. He is referring to his gardens abundance. C. He is referring to the vegetables that are in season. D. He is referring to the wild atmosphere in his restaurant. 7. Is the authors view of the restaurant that of: A. appreciation B. dismay C. shock D. disappointment 8. In this context what does farm-to-table mean? A. The restaurant, Laja, is the only one who does it. B. The ingredients used at the restaurant are very fresh. C. The majority of Mexican restaurants are expensive. D. The food at Corazon de Tierra is quite unusual.

Shehu & Tidwell 31 Directions: Look at the picture below. Imagine that the two men in the picture are traveling in California and write a brief (one paragraph) short story about their adventure.

Shehu & Tidwell 32 Directions: Considering your own abilities, answer the following questions by circling 1, 2, 3, or 4, with one (1) being strongly disagree, two (2) being disagree, three (3) being agree, and four (4) being strongly agree. 1. I can read and understand a magazine in English. 1 2. I can read and understand academic texts in English. 1 3. Writing fiction short stories in English is difficult for me. 1 4. When I write in English, I wish that my vocabulary was larger. 1 5. The questions about the reading passage were easy. 1 2 3 4 2 3 4 2 3 4 2 3 4 2 3 4

6. I had to reread the text many times to answer the questions about the reading passage. 1 2 3 4

Shehu & Tidwell 33 References Bouvier, Nicolas. (1985). The Road to Anatolia. The Way of the World. New York Review of Books: New York, NY. Educational Testing Service (2012). TOEFL Writing Score Guide. Retrieved from http://www.ets.org/toefl/pbt/scores/writing_score_guide/ Martin, C. (2003). Explore Valle de Guadalupe wine country. Sunset Magazine http://www.sunset.com/travel/hawaii-mexico/valle-de-guadalupe-00418000082058/