Concurrent Validity and Stability of The Maze Task in A Sample of College Students

IDL - International Digital Library Of
Management & Research

Volume 1, Issue 3, Mar 2017 Available at: www.dbpublications.org
International e-Journal For Management And Research-2017
Concurrent Validity and Stability of the Maze

Task in a Sample of College Students
James M. Kuterbach
Dept. of Human Development and Family Studies
Penn State DuBois
DuBois, PA, USA
jmk110@psu.edu
Abstract: This paper examines the concurrent comprehension curriculum-based assessment tool, in a
sample of college students.
validity and test-retest reliability of the maze task in a
sample of college students. The maze task is a form of
Curriculum Based Measurement that is typically used The disability category with the largest proportion of
to assess reading comprehension in elementary and students continuing on for a postsecondary education
secondary students, but has not been used with college is students with a learning disability (Horn, Berktold,
students. Three visions of the maze task (one-, two-, & Bobbitt, 1999). Students with a learning disability
and three-minute probes) were created and compared also make up the fastest growing population of college
to student scores on the Nelson-Denny Reading Test students with a disability (Henderson, 2001), with
(NDRT), students self-report of GPA, and scores on more than twenty-seven percent of high school
the SAT-Reading, -Math, and Writing tests. The students diagnosed as having a specific learning
one-minute probe was found to have the best disability continuing on for postsecondary education
psychometric properties, with high correlations with (Wagner, Newman, Cameto, Garza, & Levine, 2005).
the NDRT, GPA, and SAT-Reading, as well as This increase in students with a learning disability at
divergent validity with the SAT-Writing test. the postsecondary level is creating new challenges for
Implications for use with college students is discussed. educators, evaluators, and administrators.
Keywords: Reading assessment, maze task, college
students, learning disabilities College disability services administrators report that
students with a learning disability tend to want the
same accommodations at the postsecondary level that
1. INTRODUCTION they received in high school (e.g., oral essay exams,
no foreign language, extended time on tests), even
With the large influx of students with a learning when the documentation of their disability maybe
disability into colleges and universities, administrators shaky, at best (McGuire, 2000). One author reports
and evaluators need tools that will aid them in that the number of overall requests for specific
determining the need for special accommodations for accommodations had risen by more than 160%
student populations. While current assessments do a between the years 2000 and 2001 (Ofiesh, Mather, &
good job, they are generally lengthy and may be a Russell, 2005). The required type and
deterrent to students seeking out assistance. What is comprehensiveness of assessments at the secondary
needed is a brief screening tool that is both a valid and and postsecondary levels, as well as the documentation
reliable measure of a students academic skills. The needed to demonstrate a history of services varies
purpose of this study is to evaluate the reliability and greatly from institution to institution (Gregg,
validity of the maze task, a commonly used reading Coleman, Davis, Lindstrom, & Hartwig, 2006). Some
students are required to produce documentation on
IDL - International Digital Library 1|P a g e Copyright@IDL-2017


their own, while others are required to complete new Daneman, 2006; Wood, Nemeth, & Brooks, 1985),
evaluations. With comprehensive evaluation lasting memory, and cognitive processes (Carver, 1992;
six to eight hours some students find this to be Davis, Bardos, & Woodward, 2006; Millis, Magiliano,
daunting (Canter, 2004). & Todaro, 2006). However, the NDRT takes a total of
35 minutes to administer, with the Comprehension
The largest segment of students with a learning section taking 20 minutes.
disability is those with a reading disorder (Lorry,
2000). Both typically achieving students and learning An alternative to norm-based testing is curriculum-
disabled students are often overwhelmed by the based measurements (CBM). Curriculum-based
amount of reading that is needed at the college level measurement has been defined by Deno (1987) as
(Du Boulay, 1999). Even though reading ability has any set of measurement procedures that use direct
been found to be a significant predictor of college observation and recording of a students performance
freshman grades (Wood, 1982), reading is generally as a basis for gathering information to make
not directly assessed beyond the secondary level. instructional decisions (in Marston, 1989, p. 62).
While reading is not directly assessed, the results of Curriculum-based measurement has a long history in
reading are indirectly assessed for college students the research literature (Shapiro, Keller, Lutz, Santoro,
throughout their college career (Du Boulay, 1999). & Hintze, 2006). Test-retest reliability coefficients for
Lack of reading skills, even for students without a reading CBM probes range from .82 to .96, with
diagnosed disability, is one of the biggest problems in interrater reliability of .99 and reliability coefficients
postsecondary education, but problems in reading at for parallel forms ranging from .84 to .96 (Marston,
the postsecondary level are generally not identified 1989). Validity studies for CBM in reading have found
until the problem manifests itself in the classroom (Du correlation coefficients ranging from .73 to .91, with
Boulay, 1999). Reading disabilities are also the most most coefficients above .80 (Marston, 1989).
likely of all learning disabilities to serve as a basis for
an accommodations claim in higher education (Lorry, One example of a reading CBM is the maze task. The
2000). Educators and administrators working with maze task is a multiple-choice cloze task that students
postsecondary students with reading problems need a complete while silently reading a short reading
quick, easy method to assess a students reading passage that is rated at their grade level (Fuchs &
ability. Fuchs, 1992). The first sentence of the passage is left
unchanged, but thereafter every seventh word of the
A test currently in use in colleges around the country passage is replaced with a three-word forced choice
is the Nelson-Denny Reading Test (NDRT; Brown, inside of parenthesis. Two of the words are distractors
Fishco, & Hanna, 1993). The NDRT is a widely used and one is the correct word for the sentence. The
test of reading, which includes a vocabulary and a number of correct words chosen during the testing
comprehension section, as well as a reading fluency time is the students score. The maze task is used to
measure. The NDRT has been used with college determine a students functioning level, but it can also
students in research on reading comprehension be used as a progress-monitoring tool because it can
(Nicaise & Gettinger, 1995; Bell & Perfetti, 1994; produce multiple data points in order to chart growth
Onwuegbuzie & Collins, 2002), as well as for across the school year. Another advantage that the
diagnostic and decision-making purposes (Norman, maze task has is that it can be given to several students
Kemper, & Kynette, 1992). The NDRT has been at the same time (Madelaine & Wheldall, 2004).
found to be a good predictor of college freshman
grades (Wood, 1982) and has been used as a criterion
measure for other tests of reading (Hannon &


Timed tests have been used to distinguish between and 125 female participants, between the ages 18 and
college students with a learning disability from those 50 with a median age of 21. Of those participants
who are typically achieving (Ofiesh, Mather, & reporting a race 78 percent were white, 7.8 percent
Russell, 2005) and another study found similar results were African-American, 2.1 percent were Asian-
using adults in a clinic setting (Lesaux, Pearson, & American, 2.1 percent were Latino, and 3.7 percent
Siegel, 2006). Lack of reading skills is one of the indicated they were multi-racial. Participants semester
biggest problems in postsecondary education, but standing ranged from first-semester to ninth-semester,
problems in reading at the postsecondary level are with a median semester standing of seventh-semester.
generally not identified until the problem manifests Seventy-seven percent of participants were Human
itself in the classroom (Du Boulay, 1999). Students Development and Family Studies majors, with the
are often overwhelmed by the amount of reading that remaining participants majoring in other social
is needed at the college level (Du Boulay, 1999). sciences, including Psychology and Communication
Some researchers feel that reading is the most Sciences and Disorders. Four participants (2.8 percent)
important skill in college (Onwuegbuzie & Collins, reported having received special education services at
2002). some time during their education.
2. OBJECTIVES Materials
The Nelson-Denny Reading Test. The Nelson-Denny

The purpose of this study is to evaluate the reliability Reading Test Form G (NDRT; Brown, Fishco, &
and validity of the maze task, a commonly used reading Hanna, 1993) is a group administered standardized
comprehension curriculum-based assessment tool, in a reading test that assesses a students vocabulary,
sample of college students. The maze test is a quick, reading comprehension, and reading rate. The NDRT
easy reading comprehension test that can be used for is a commonly used measure of silent reading
both determining a students functioning level, as well comprehension (Cirino, Israelian, Morris, & Morris,
as for progress monitoring. While the maze task is 2005), with reported test-retest reliability scores
commonly used in both primary and secondary schools ranging from .76 to .81 (Brown et at., 1993), and has
it has not been used with college students. The established correlations between .60 and .69 with other
Standards for Educational and Psychological Testing measures of reading (Murphy, 1995). The NDRT
(AERA, APA, & NCME, 1999) dictate that when a test Reading Comprehension section consists of seven
is used in a way in which it has not been validated new reading passages and 38 multiple-choice
evidence should be collected in order to justify this new comprehension questions. Administration time-limit
use. Consequently, this study will investigate the for the NDRT Reading Comprehension section is 20
concurrent validity of the maze task in a college sample. minutes (Brown et al., 1993).
3. METHODOLOGY The maze task. The maze task is a commonly used

curriculum-based measurement (CBM) used to assess
student reading comprehension in the elementary and
Participants secondary levels of education. Fuchs and Fuchs (1992)
reported correlations between the maze task and the
Participants were 141 undergraduate college students Reading Comprehension subtest of the Stanford
who were enrolled in two upper-level human Achievement Test of .77 in elementary students, while
development classes. There were 16 male participants Espin and Foegen (1996) report correlations between
the maze task and three different measures of


comprehension to be between .56 and .62 in secondary =0.05; however the Bonferroni correction for
students. The creation and administration of the maze multiple comparisons was employed.
task followed the procedure described by Fuchs and
Fuchs (1992). The maze task requires the participant
4. OUTCOMES
to read a passage which has been previously prepared
such that, following the first sentence, every seventh
Independent samples t-test showed no significant
word has been replaced with a forced-choice of three
difference between male and female participants in
possible words (Espin et al., 2001). This study used
terms of their scores on the NDRT, one-minute, two-
three different passages to create probes of 1-minute,
minute, or three-minute maze task probes. Correlations
2-minutes, and 3-minutes in length. The passages used
above .50 are considered large (Cohen, 1988). Table 1
in the study were taken from a textbook used in a
displays the correlations between the maze task probes
survey of Human Development course (Dacey &
and the criterion measures. All correlations between
Travers, 2004) and each had a Flesch-Kincaid Grade
maze tasks and the NDRT were significant at the
Level of 12.0.
=.003 level, with the strongest correlation found
between the one-minute probe and the NDRT (r =
Procedure
.606). In addition, both the one-minute and two-
minute probes correlated moderately with participant
Participants completed a questionnaire that included
recollection of their GPA (r = .390 and .341,
demographic information, education experiences, and
respectively). Finally, only the one-minute maze task
recollection of grade point average (GPA) and scores
probe correlated significantly with participant
of the Scholastic Aptitude Test (SAT). Both the maze
recollection of their SAT reading score (r = .502).
task and the NDRT were administered in a group
None of the maze task probes correlated significantly
setting. Participants were given the maze task probes
with participant recollection of SAT mathematics
first and were timed by the investigator for one-
scores or SAT writing scores.
minute, two-minutes, and three-minutes. The NDRT
Reading Comprehension section was then
administered, with a 20-minute time-limit.
Approximately three weeks later students from one
class (68 participants) completed a retest involving the
three maze task probes.
Analysis
Pearson Product Moment Correlations were performed

between the three maze task probes and the NDRT. In
addition, the maze task probes were correlated with
participants recollection of their GPA and SAT
scores. Finally, test-retest reliability was evaluated by
correlating the one-minute probe, two-minute probe, Test-retest reliabilities can be found in table 2. All
and three-minute probe maze tasks from the initial three maze task probes had good test-retest reliability.
administration with those from the retest The one-minute probe showed the highest reliability,
administration. Significance level was initially set a with a correlation of .952, which is suitable reliability


for making diagnostic decisions (Salvia & Ysseldyke, Carver, R. P. (1992). Reliability and validity of the
2003). speed of thinking test. Educational and
Psychological Measurement, 52, 125-134.
Cirino, P. T., Israelian, M. K., Morris, M. K., &
Morris, R. D. (2005). Evaluation of the double-
deficit hypothesis in college students referred for
learning difficulties. Journal of Learning
Disabilities, 38, 29-44.
Cohen, J. (1988). Statistical power analysis for the
behavioral sciences (2nd ed.). Hillsdale, NJ:
Erlaum.
CONCLUSION Davis, A. S., Bardos, A. N., & Woodward, K. M.
(2006). Concurrent validity of the general ability
measure for adults (GAMA) with sudden-onset
This research demonstrates that the maze task has the
neurological impairment. International Journal of
psychometric properties necessary to be used to
Neuroscience, 116, 1215-1221.
determine reading comprehension ability in college
Du Boulay, D. (1999). Argument in reading: What
students. This study examined one-minute, two-
does it involve and how can students become better
minute, and three-minute probes and found that the
critical readers? Teaching in Higher Education, 4,
one-minute probe had the best psychometric
147-162.
properties, with the highest test-retest reliability,
Fuchs, L.S., & Fuchs, D. (1992). Identifying a
highest correlation with the NDRT, self-reported
measure for monitoring student reading progress.
GPA, and SAT-Reading scores, and good divergent
School Psychology Review, 21, 45-58.
validity with SAT-Writing scores. While the
Greg, N., Coleman, C., Davis, M., Lindstrom, W., &
divergent validity between the maze task and the SAT-
Hartwig, J. (2006). Critical issues for the diagnosis
Math score was not as robust, the correlation would
of learning disabilities in the adult population.
not be described as high (Cohen, 1988). These results
Psychology in the Schools, 43, 889-899.
give evidence to the idea that the maze task could be
Hannon, B. & Daneman, M. (2006). What do test of
used for determining reading comprehension levels in
reading comprehension ability such as the VSAT
college students, as well as for progress monitoring of
really measure?: A componential analysis. In A. V.
college students with a reading disability.
Mittel (Ed.) Focus on educational psychology (pp.
105-146). Hauppauge, NY: Nova Science
REFERENCES Publishers.
Bell, L. C. & Perfetti, C. A. (1994). Reading skill: Henderson, C. (2001). College freshman with
Some adult comparisons. Journal of Educational disabilities: A statistical profile. HEATH Resource
Psychology, 86, 244-255. Center, American Council on Education, U.S.
Brown, J. I., Fishco, V. V., & Hanna, G. (1993). Department of Education.
Nelson-Denny Reading Test: Manual for scoring Horn, L., Berktold, J. & Bobbitt, L. (1999). Students
and interpretation. Chicago, IL: Riverside. with disabilities in postsecondary education: A
Canter, A. (2004). A problem-solving model for profile of preparation, participation, and
improving student achievement. Principal outcomes. Retrieved from
Leadership Magazine, 5. Retrieved from http://nces.ed.gov/pubs99/1999187.pdf
http://www.naspcenter.org/principals/nassp_probso Lesaux, N. K., Pearson, M. R., & Siegel, L. S. (2006).
lve.html The effects of timed and untimed testing


conditions on the reading comprehension measures to determine the need for extended test
performance of adults with reading disabilities. time among university students with learning
Reading and Writing, 19, 21-48. disabilities. Journal of Psychoeducational
Lorry, B. J. (2000). Language-based learning Assessment, 23, 35-52.
disabilties. In M. Gordon and S. Keiser (Eds.) Onwuegbuzie, A. J. & Collins, K. M. (2002). Reading
Accommodations in higher education under the comprehension among graduate students.
Americans with disabilities act (ADA): A no- Psychological Reports, 90, 879-882.
nonsense guide for clinicians, educators, Salvia, J. & Ysseldyke, J. E. (2003). Assessment in
administrators, and lawyers. (pp. 20-45). New special and inclusive education (9th edition).
York, NY: The Guilford Press. Boston, MA: Houghton Mifflin Company.
Madelaine, A. & Wheldall, K. (2004). Curriculum- Shapiro, E. S., Keller, M. A., Lutz, J. G., Santoro,
based measurement of reading: Recent advances. L.E., & Hintze, J. M. (2006). Curriculum-based
International Journal of Disability, Development measures and performance on state assessment and
and Education, 51, 57-82. standardized tests: Reading and math performance
Marston, D. B. (1989). A curriculum-based in Pennsylvania. Journal of Psychoeducational
measurement approach to assessing academic Assessment, 24, 19-35.
performance: What is it and why do it. In M. R. Wagner, M., Newman, L., Cameto, R., Garza, N., &
Shinn (Ed.) Curriculum-based measurement: Levine, P. (2005). After high school: A first look at
Assessing Special Children. (pp. 18-78). New the postschool experiences of youth with
York, NY: The Guilford Press. disabilities. A report from the National
McGuire, J. (2000). Educational accommodations: A Longitudinal Transition Study-2. Menlo Park, CA:
university administrators view. In M. Gordon and SRI International.
S. Keiser (Eds.) Accommodations in higher Wood, P. H. (1982). The Nelson-Denny Reading Test
education under the Americans with disabilities act as a predictor of college freshman grades.
(ADA): A no-nonsense guide for clinicians, Educational and Psychological Measurement, 42,
educators, administrators, and lawyers. (pp. 20- 575-583.
45). New York, NY: The Guilford Press. Wood, P. H., Nemeth, J. S., & Brooks, C. C. (1985).
Millis, K., Magliano, J., & Todaro, S. (2006). Criterion-related validity of the Degrees of
Measuring discourse-level processes with verbal Reading Power Test (Form CP-1A). Educational
protocols and latent semantic analysis. Scientific and Psychological Measurement, 45, 965-969.
Studies of Reading, 10, 225-240.
Murphy, S. (1995). An analysis of the construct and
predictive validity of the CPT-R and Nelson
Denny tests. Unpublished manuscript, Rose State
College, Midwest City, Oklahoma.
Nicaise, M. & Gettinger, M. (1995). Fostering reading
comprehension in college students. Reading
Psychology, 16, 283-337.
Norman, S., Kemper, S., & Kynette, D. (1992).
Adults reading comprehension: Effects of
syntactic complexity and working memory.
Journal of Gerontology, 47, 258-265.
Ofiesh, N., Mather, N., & Russell, A. (2005). Using
speeded cognitive, reading, and academic

Concurrent Validity and Stability of The Maze Task in A Sample of College Students

Încărcat de

Informații document

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Concurrent Validity and Stability of The Maze Task in A Sample of College Students

Încărcat de

Drepturi de autor:

Formate disponibile

IDL - International Digital Library Of

Management & Research

International e-Journal For Management And Research-2017