Sunteți pe pagina 1din 9

Reading study questions for Testing for language teachers (Hughes, 2003)

Hughes, Ch.1

1. What is backwash? What is an example of positive or beneficial backwash? What


is an example of negative or harmful backwash?
2. According to Hughes, what are the two main sources of inaccuracy in tests?
Explain.
3. What is the difference between formative and summative assessment?
4. According to Hughes, what three contributions can teachers make to the
improvement of testing?
5. According to Hughes, why are tests necessary? Can you imagine any language
learning situation in which tests might be unnecessary? Explain.
6. Think about and be prepared to discuss Hughess questions on page 6.

Hughes, Ch.2

1. According to Hughes, Each testing situation is unique and sets a particular


testing problem (p. 8). Therefore, what are three goals towards which we should
aim when creating tests?
2. What is the first thing that testers have to be clear about before creating a test?

Hughes, Ch.3

1. How are proficiency tests and achievement tests related? What is the main
difference between proficiency tests and achievement tests?
2. Explain the following: final achievement tests, the syllabus-content approach to
achievement tests, the course-objectives approach to achievement tests, progress
achievement tests.
3. Why is it difficult to obtain a detailed analysis of a students command of specific
grammatical structures (see Diagnostic tests)?
4. According to Hughes, what type of placement tests are usually the most
successful? Why?
5. What is the difference between direct and indirect testing? Hughes gives a
number of reasons why direct testing is generally to be preferred. What are some
of these reasons?
6. What is the difference between discrete point testing and integrative testing? Give
examples.
7. Norm-based testing relates test takers performance on the test to _____.
8. What is the purpose of criterion-referenced tests? How are they different from
norm-based tests?
9. What is the difference between objective and subjective testing? Can you think of
an example of each?
10. Think about and be prepared to discuss Hughess questions on pages 23-24.

Hughes, Ch.4Validity
Reading study questions for Testing for language teachers (Hughes, 2003)

1. Until now we have talked about validity in general. In recent years, Hughes says that
this has come to be called what? Why/how did this happen? (p. 26)

2. What are the different types of validity discussed? How are they related to each other?

3. A test is said to have content validity if it (p. 26)

4. In order to judge whether or not a test has content validity, we need (p. 26)

5. Content validity is important because (p. 27)

6. Criterion-related validity is (p. 27)

7. Briefly, what is the difference between the 2 types of criterion-related validity


(concurrent and predictive validity)? (pp. 27-30)

8. Youre trying to figure out how valid your new placement test is by measuring its
concurrent validity with another, established test that you know is good. You compare the
scores and come up with a correlation coefficient (or validity coefficient in this case), and
get .85. On this basis, what percentage of the score on one of the tests is predicted by the
score on the other test? Are you happy with this level of validity?

9. Hughes believes that, in terms of validity, the direct testing of general, commonsense
constructs like reading ability and writing ability (p. 31) is
a. unproblematic b. problematic

10. What is one example of invalid scoring that Hughes gives? (pp. 32-33)

11. Explain face validity.

12. What are the 4 suggestions Hughes gives for making teacher-made tests more valid?
(pp. 33-34)

Hughes, Ch.5Reliability

1. Youre reading a research article in the journal Applied Linguistics about the effect of
extraversion on learners L2 proficiency. In describing the instrument (test) used to
measure extraversion, the researcher reports reliability coefficient of .3 using the split
half method. How reliable is this test?

2. What is meant by true score? (p. 40)

3. Answer question #2 on p.51.

4. What suggestions does Hughes offer for improving reliability? (pp. 44-50)
Reading study questions for Testing for language teachers (Hughes, 2003)

5. Be able to discuss the relationship between reliability and validity.

Hughes, Ch.6Achieving beneficial backwash

1. What suggestions does Hughes give for helping to achieve beneficial backwash?

Hughes, Chapter 7Stages of test development

In this chapter, Hughes suggests 10 steps for developing tests.

1. In stage 1, stating the problem, what are some things that Hughes suggests taking into
consideration? (p. 59)

2. In stage 2, writing specifications for the test, what sorts of things does Hughes suggest
that we should consider including in each of these sub-areas? Do we always need to
include all of these? (pp.59-61)

a. Content
b. Structure, timing, medium/channel and techniques
c. Criterial levels of performance
d. Scoring procedures

3. Stage 3, writing and moderating items (pp. 62-63):

a. How does Hughes suggest dealing with the fact that we may not be able to cover
ALL of the things we wrote in the content section of the test specifications (see
sampling)?
b. What are some things Hughes suggests keeping in mind when writing test items?
c. What does Hughes mean by moderating items? How does he suggest doing
this?
d. What do you think of the table on p. 64? Would you add/subtract/change
anything?

4. According to Hughes, in stage 4, informal trialling of items on native speakers (pp. 63-
64), what kind of native speakers are preferable when trialling (piloting) items? What
should be done with items that are difficult for native speakers according to Hughes?

5. When, according to Hughes, might stage 5 not be possible? (pp. 64-65)

6. In stage 6, analysis of results of the trial; making of any necessary changes, what two
kinds of analysis should be carried out, according to Hughes? (p. 65)

7. How can scales be calibrated? Why is this useful? (p. 65)


Reading study questions for Testing for language teachers (Hughes, 2003)

8. When is validation absolutely necessary? How were the tests described in the test
development examples validated (p. 66, 69, 72)

9. What kinds of information might be included in the handbooks discussed in stage 9?


(p. 66)

Hughes, Ch.8Common test techniques

1. Look at the following multiple choice question. Identify the stem, options, distractors,
and the key (H, p. 75-76; AND Bailey, p. 131).

A: Wheres the book?


B: I put it ___ the table.

a. at b. over c. under d. on

2. Was it possible to identify the key in the multiple choice question in #1? Whats wrong
with this question?

3. What are the advantages of multiple choice tests, according to Hughes? (p. 76)

4. What are the drawbacks of multiple choice tests, according to Hughes? (pp. 76-78)

5. What are the advantages/disadvantages of YES/NO or TRUE/FALSE tests, according


to Hughes? (p. 79)

6. What are the advantages/disadvantages of short-answer items, according to Hughes?


(pp. 79-80)

7. What are the advantages/disadvantages of gap-filling items (or fill-in-the-blank


items), according to Hughes? (pp. 80-81)

Hughes, Chapter 9Testing writing

1. What fundamental assumption does Hughes make about assessing writing? What
reasons does he give to justify making this assumption? (p. 83)

2. What are the three basic considerations for assessing writing that Hughes outlines at
the beginning of this chapter? (p. 89)

3. In the process of setting representative tasks for the assessment of writing, Hughes
suggests that we should (1) specify all possible content and (2) include a
representative sample of the specified content in our assessment. What do these steps
entail? What two examples situations does Hughes describe? (pp. 83-89)
Reading study questions for Testing for language teachers (Hughes, 2003)

4. Which of these two example situations (from #3 above) is more similar to your
anticipated future teaching situation? Could you use or modify either of the two
assessments he describes to fit your situation?

5. What steps does Hughes recommend to help elicit a valid sample of writing ability
that you want to assess? (pp. 89-93)

6. What do you think of the following passage from Hughes? Do you agree/disagree?

In language testing we are not normally interested in knowing whether students


are creative, imaginative, or even intelligent, have widespread knowledge, or have
good reasons for the opinions they happen to hold. Therefore, for the sake of
validity, we should not set tasks which measure these abilities (p. 90).

7. What do you think of the four writing assessment tasks Hughes describes on p.90?

8. What are some ways that Hughes suggests limiting the effects of reading ability on the
assessment of writing? (pp. 90-93).

9. How and why does Hughes suggest restricting candidates in their performance of
assessment tasks? (p. 93)

10. What are four suggestions Hughes gives to assist in setting tasks that can be reliably
scored? (p. 94)

11. What are the two different types of scoring scales Hughes describes? (pp. 94-105)
What are some advantages and disadvantages of each (pp. 95, 102, 103, 105)?

12. What does Hughes mean by calibrate the scale? (p. 106)
13. What are the last two steps Hughes suggests to ensure valid and reliable scoring? (pp.
106-108)

14. You may wish to try Hughes activity #2 on p. 109.

Hughes, Chapter 10Testing oral ability

1. What two assumptions does Hughes make about assessing oral ability? (p. 113)

2. Hughes says that the three basic considerations for assessing oral ability are very
similar to those for assessing writing. What are these three considerations? (p. 113)

3. Look at the lists of operations/skills specified on pp. 114-115. Think about your
anticipated future teaching situation. What skills/operations do you imagine you will
want to assess? Are there any operations/skills that you might want to assess that are not
listed on pp. 114-115? What?
Reading study questions for Testing for language teachers (Hughes, 2003)

4. Finish this sentence: The greater the detail in the specification of content, the more
(p. 116)

5. What two important points does Hughes make about setting tasks for oral assessment
interviews? (p. 117)

6. What is one disadvantage of the interview format that Hughes describes on p. 119?

7. List the various interview techniques Hughes describes. (pp. 119-121)

8. List the various interaction with fellow candidates techniques Hughes describes. (pp.
121-122). According to Hughes, what are the advantages and disadvantages of assessing
oral ability by having candidates interact with fellow candidates? (p. 121)

9. List the various responses to audio- or video-recording techniques Hughes describes.


(pp. 122-124). According to Hughes, what are the advantages and disadvantages of this
format? (p. 121)

10. What are ten suggestions Hughes gives regarding the creation of oral assessments on
pp. 124-126?

11. What are the two different types of scoring scales Hughes describes? (pp. 127) What
are some advantages and disadvantages of each (pp. 95, 102, 103, 105)?

12. What is one thing that Hughes emphasizes must be avoided in scoring oral
assessments? (pp. 133-134)

Hughes, Ch. 11Testing reading

1. According to Hughes, what are some of the difficulties of assessing receptive skills?
(p. 136)

2. Hughes divides reading operations into two large sub-categories. What are these
categories and what kinds of operations does he include in each category? (pp. 137, 138-
139)

3. According to Hughes, how should we decide what reading skills/operations to include


in our assessments? (p. 137)

4. Can you make a connection between Hughes discussion of pragmatic inference on the
bottom of p. 139 and Baileys discussion of the role of background knowledge (chapter
4)?
Reading study questions for Testing for language teachers (Hughes, 2003)

5. Other than operations, what other factors does Hughes suggest considering when
specifying what candidates should be able to do? (pp. 140-141)

6. Under Setting the tasks Hughes gives 10 guidelines for selecting texts. What are
these guidelines? Were any of them things you had not considered before? (pp. 142-143)

7. What are some of the possible techniques for testing reading that Hughes describes?
(pp. 143-153) Which technique(s) does he suggest for assessing learners understanding
of the organization of a passage? (pp. 145-146)

8. What is a summary cloze? (pp. 149-151) Why does Hughes not include regular cloze
tests or c-tests in this chapter? (p. 153)

9. Which of the techniques Hughes suggests might be useful in your anticipated future
teaching situation?

10. On p. 153, Hughes suggests a procedure for writing items. What are the steps in this
procedure?

11. Were any of Hughes tips on writing items things you hadnt thought of before?
Which ones? (pp. 154-155)

12. What do you think of the passage from Hughes at the bottom of p. 155 (A note on
scoring)?

Hughes, Ch. 12Testing listening

1. According to Hughes, what are three justifications for assessing listening separately
from oral ability? (p. 160)

2. How is assessing listening similar to assessing reading? What difficulties arise in both
situations? (pp. 136, 160)

3. Hughes divides listening operations into global and more specific skills. What are
examples of global skills and specific skills? What else does Hughes suggest should be
specified when it comes to listening operations? What are examples of lower level
listening skills that might be tested for diagnostic reasons? (pp. 161-162)

4. In his discussion of Texts, Hughes gives some of the average speech rates for
different types of spoken English. What is the average speech rate for radio monologues?
Conversations? Interviews? Lectures to non-native speakers of English? (p. 162)

5. What tips does Hughes give about selecting samples of speech to use? (p. 164)

6. When writing items to test extended listening skills (such as a lecture), what steps does
Hughes suggest? (pp. 164-165)
Reading study questions for Testing for language teachers (Hughes, 2003)

7. List the various techniques for assessing listening that Hughes describes (pp. 165-169).
What are the advantages/disadvantages of each? Are any of these techniques new or
especially appealing to you?

Hughes Ch. 13 (pp. 172-185)

1. What are some of the justifications for testing grammar that Hughes gives? (pp.
172-173)
2. When writing specification for grammar achievement tests, if the course syllabus
lists grammar objectives, specifying the content of the test is straightforward.
How does Hughes recommend going about specifying content? (p. 173)
3. When writing grammar items, Hughes emphasizes that it is important for the text
to be written in ___ and ___ language (p. 174). He recommends using ___ to
avoid using unnatural language.
4. Hughes describes several techniques for testing grammar: Gap filling, paraphrase,
completion, and multiple choice. You should be able to explain each of these
techniques. Also, how are multiple-choice items different from the other types of
items? (pp. 174-178)
5. In what situation(s) can multiple-choice items be particularly useful, according to
Hughes? (p. 178)
6. What tips does Hughes offer about scoring grammar tests? (pp. 178-179)
7. What are some of the justifications Hughes offers for testing vocabulary that
Hughes gives? (p. 179)
8. When writing specifications for a vocabulary test, Hughes recommends grouping
words according to their ___ and ___ (p. 180).
9. Hughes divides vocabulary testing into Testing recognition ability and Testing
production ability. What techniques does he suggest using for each of these
focuses? (pp. 180-184)

Hughes, Ch. 15Tests for young learners

1. How does Hughes feel about testing young learners? What example does he give of
successful language instruction for young learners that does not involve testing? How
does Hughes justify including this chapter in his book? (p. 199)

2. What 3 recommendations does Hughes make regarding the testing of young learners?
(p. 200)

3. What 6 points does Hughes make about the characteristics of young learners that affect
how they should be tested/assessed?

4. What final general recommendation does Hughes make about testing children on p.
202? Do you think this should only apply to children or to all learners?
Reading study questions for Testing for language teachers (Hughes, 2003)

5. For the rest of the chapter, Hughes offers suggestions about how to assess listening,
reading, writing, and oral ability of young learners. Which of these do you find
particularly interesting? Could you use any of these techniques with adult learners?

Hughes, Ch. 16Test administration

Many of the guidelines given in this chapter are especially applicable to large-scale
testing, but consider:

1. Have you ever been/do you anticipate being involved in such large-scale testing? If so,
what problems have you encountered/do you imagine might arise?

2. If you dont expect to be involved in large-scale testing, which of the guidelines given
in this chapter might still be useful for you to consider when organizing assessments in
your classes?

3. Is there anything you would add to this list?

Hughes, Appendices 1&2The Statistical analysis of test data; Item banking

1. What is one way to compute IF values for questions that have more than 1 point (for
example questions that are worth 5 points)? (pp. 225-226)

2. If we investigate the reliability of a test and find that it is rather low, what steps does
Hughes suggest we follow to try to fix the problem? (pp. 227-228)

3. Dont worry too much about IRT/Rasch analysis. Its a little complicated to compute
and you need rather large sample sizes so unless you get really involved in testing, you
are unlikely to use these tools. However, its worth reading and trying to understand
Hughes explanation and example on pp. 222-232.

4. What is an item bank? (pp. 234-235)

S-ar putea să vă placă și