Reliability of Rating Visible Landscape Qualities

James F. Palmer

James E Palmer is Associate Profes-
sor and Undergraduate Curriculum
Director of the Landscape Architec-
ture Program at the State University
of New York’s College of Environmen-
tal Science and Forestry in Syracuse,
New York. He holds an M.L.A. degree
and Ph.D. degree from the University
of Massachusetts in Amherst. His
research focuses on perceptions of
visible landscape qualities. He is par-
ticularly interested in developing GIS
models to predict landscape percep-
tions and the interaction between
landscape change and changes in per-
ceptions. He also hosts LArch-L, the
electronic discussion group for land-
scape architecture. He may be
reached by email at
zooey@mailbox.syr.edu.
Abstract: Reliability is measured by whether an investigation will obtain similar results
when it is repeated by another party. It is argued that reliability is important at the level of
individuals. While all landscape assessments are based on individual judgments, they are fre-
quently aggregated to form co~npositejudgments. The use of inter-group, intra-gwup and inter-
rater measures of reliability in the landscape perception literature is reviewed. This paper inves-
tigates the reliability of assessing various visible landscape qualities using data primarily from
previously published studies. The results indicate that there is reason for concern about the reli-
ability of rating scales used in thisJ~eld, and suggest actions for both research and practice.
Landscape studies have joined the shifting sands of scholarly inquiry. The empirical pos-
itivist tradition continues to hold that there is an objective landscape that can be studied
through the careful application of scientific principles. Post-positivists, among whom I include
myself, stand back from the stranglehold that science recently held on legitimate knowledge.
They acknowledge other ways of knowing, but favor empirical methods of discovery. In response
to positivism, post-modern landscape scholars with diverse viewpoints are investigating new
ways to explore the personal meaning that landscapes hold for us. For instance, Norberg-Schulz
(1979) has contributed to the rising importance of genius loci, Potteiger and Purinton (1998)
are exploring landscape narratives, and Brook (1998) is adapting Goethe’s approach to scien-
tific inquiry through direct experience. In this flurry of activity some foundational attributes of
applied everyday experience are lost or ignored. One such attribute is reliability, which is the
subject of this paper.

Reliability and the Human Condition The poor boy hopes to recover This bud of love, by summer’s
from his blunder: ripening breath,

T o illustrate the tension

between our desire to cap-
ture the uniqueness of the moment
and the need for stability in our lives,
I call upon one of the greatest chroni-
clers of the human condition. Quoting
from the balcony scene in Shake-
speare’s Romeo and Juliet act 2, scene 2:
Romeo. Lady, by yonder blessed
moon I swear
That tips with silver all these fruit-
tree tops ....
In the middle of Romeo’s poetic
attempt to swayJuliet’s heart, she
cuts him offl
Juliet. O! swear not by the moon,
the inconstant moon,
The monthly changes in her circled
orb,
Lost that thy love prove likewise
variable.
She will have none of it (this
night at least). She makes it clear
that she wants a constant and
dependable love:
Romeo. What shall I swear by?
Juliet. Do not swear at all;
Or, if thou wilt, swear by thy gra-
cious self,
Which is the god of my idolatry,
And I’ll believe thee.
She embraces the substance of
his desire (their love); it is only the
metaphor for his method that is in
question (the unique moment).
Romeo. If my heart’s dear love ....
He has not learned, and begins
another uniquely poetic expression,
only to be cut off once more:
Juliet. Well, do not swear. Although
I joy in thee,
I have no joy of this contract to-
night:
It is too rash, too unadvised, too
sudden;
Too like the lightning, which doth
cease to be
Ere one can say it lightens. Sweet,
good night!
May prove a beauteous flower when
next we meet.
Good night, good night! as sweet
repose and rest
Come to thy heart as that within
my breast!
He has twice failed, and she
grows weary. However, she leaves him
with an indication of what she
seeks--reliability. What she wants is
not rash declarations, but steadfast
devotion; not unadvised promises, but
considered pronouncements; not sud-
den like a flash of lightning, but a
constant and unfailing partnership in
love.Juliet’s response represents the
need for reliability in our most impor-
tant experiences as a fundamental
condition in our lives.
Reliability in the (Post-)Positivist Paradigm
The National Environmental
Policy Act of 1969 has particular
importance for those of us who con-
duct landscape assessments. It is
NEPA that declared it national policy

166 Landscape Journal

and the "continuing responsibility of
the Federal Government to use all
practicable means to... assure for all
Americans... aesthetically.., pleas-
ing surroundings." In particulm;
there is a responsibility to "identify
and develop methods and procedures
¯.. which will insure that presently
unquantiffed environmental ameni-
ties and values may be given appro-
priate consideration in decision mak-
ing." As one example, the goals of the
Forest Service’s new Scenery Manage-
ment System are to (1) inventory and
analyze scenery; (2) assist in estab-
lishing overall resource goals and
objectives; (3) monitor the scenic
resource; and (4) ensure high-quality
scenery for future generations
(USDA 1995). Reliability is the
implicit cornerstone upon which
these goals can be achieved.
Though speaking about unquan-
tiffed values and amenities, the lan-
guage of NEPA clearly takes a decid-
edly positivist tone. Nunnally (1978,
p. 191) describes the meaning and
importance of reliability to science:
Reliability concerns the extent to
which measurements are repeat-
able--when different persons make
the measurements, on different
occasions, with supposedly alterna-
tive instruments for measuring the
same thing and when there are
small variations in circumstances
for making measurements that are
not intended to influence results.
¯.. Measurement reliability repre-
sents a classic issue in scientific
generalization.
The measurement of reliability
is relatively straightforward.~ "The
average correlation among the items
can be used to obtain an accurate
estimate of reliability" for independ-
ently made assessments (Nunnally
1978, p. 227). There are even gener-
ally accepted standards for reliability
among psychometricians (Nunnally
1978, p. 245).
What a satisfactory level of reliabil-
ity is depends on how a measure is
being used. In the early stages of
research.., one saves time and
energy by working with instruments
that have only modest reliability,
for which purposes reliabilities of
.70 or higher will suffice .... For
basic research, it can be argued
that increasing reliabilities much
beyond .80 is often wasteful of time
and funds .... In many applied
problems, a great deal hinges on
the exact score made by a
person .... In those applied settings
where important decisions are
made with respect to specific test
scores, a reliability of .90 is the
minimum that should be tolerated,
and a reliability of .95 should be
considered the desirable standard.
Since there are few problems
that impact us as directly as decisions
about our common landscape, the
fundamental importance of using
reliable landscape assessment meth-
ods should be readily apparent.
Reliability in Visual Landscape
Assessments
It is surprising that relatively
little attention is paid by most
researchers and practitioners to the
reliability of landscape assessment
methods--and there is a lot on which
to focus¯ For instance, how many pho-
tos are needed to reliably represent
different landscapes (Daniel et al.
1977; Hoffman 1997)? When a long
series of landscape scenes is being
evaluated, are the same standards
being reliably applied throughout the
sequence (Palmer 1998)? Are land-
scape evaluations stable after a year
or two (Hull and Buhyoff 1984); how
about after ten years (Palmer 1997)?
While these and other questions
of reliability are all important, this
paper focuses on the reliability of
raters. Three levels of rater reliability
are distinguished: inter-group, intra-
group, and inter-rater. Inter-group
reliability compares the mean ratings
assigned by different groups. The
intent is to establish whether differ-
ent groups give similar ratings. Intra-
group reliability establishes the relia-
bility of a composite or mean rating
from a particular group. Inter-rater
reliability establishes the expected
reliability for a single rater’s assess-
ment, based on the ratings from a
group. The landscape assessment lit-
erature concerning inter-group, intra-
group, and inter-rater reliability is
reviewed. The analyses of raw
datasets, primarily from previously
published studies, are presented to
illustrate the difference between
inter- and intra-group reliabilities for
scenic value ratings. Then three new
analyses are presented investigating
the reliability of three types of land-
scape descriptors: landscape dimen-
sions (development, naturalism, pref-
erence, and spaciousness), informa-
tional content (coherence,
complexity, legibility, and mystery),
and compositional elements (color,
form, line, and texture). The conclud-
ing discussion makes recommenda-
tions to those in research and prac-
tice concerning the reliability of land-
scape assessment.
Inter-group reliability. At the Our
National Landscape conference,
Craik and Feimer (1979) made a plea
to establish technical standards for
the reliability, validity, generality, and
utility of observer-based landscape
assessments. At this time, most stud-
ies that include reliability estimates
use inter-group measures that com-
pare the mean assessments made by
groups of students, lay public, or pro-
fessionals. Interpretation of inter-
group reliabilities may easily fall prey
to the ecological fallacy (Robinson
1950) of making generalizations
about individuals based on correla-
tions among groups. Establishing
similar rating patterns between
divergent groups, for example profes-
sionals and the public, does not indi-
cate that there is wide agreement
among individuals within or across
these groups. The following examples
illustrate the use of inter-group relia-
bility.
In a study of Southern Con-
necticut River Valley landscapes,
Zube et al. (1974) reported the corre-
lations among mean ratings for thir-
teen groups on eighteen scales.
Eighty-five percent of the correla-
tions among these groups were above
.83. All of the correlations below .83
involved a single group of center-city
Palmer 167
residents, suggesting that they "may
in fact have different perceptions of
scenic quality in the rural land-
scape." The need for further study is
indicated.
Buhyoff and his colleagues
(1979) compare the perception of
insect damage to forest scenes by
foresters, environmentalists, and the
public. There was an overall similar-
ity among the groups, but ratings
were affected by awareness of the dis-
ease. Groat (1982) compared percep-
tions of modern and post-modern
architecture by accountants and
architects. She found that the
accountants were not sensitive to
post-modern principles.
Some of these and other studies
find evidence that there is a strong
similarity among ratings by the pub-
lic, students, and professionals in
many instances. In other studies,
expert knowledge appears to change
the ratings by professionals and stu-
dents, sometimes quite significantly.
However, none of these group com-
parisons sheds light on the reliability
of ratings by individuals, whether
they are environmental professionals
or members of the public.
Intra-group reliability. Most
reports of rater reliability are for
group-mean ratings rather than for
the average reliability for an individ-
Ual rater within a group. For instance,
Gobster and Chenoweth (1989) iden-
tified thirty-four of the most com-
monly used landscape descriptors
representing physical, psychological
and artistic attributes. In two experi-
ments, using groups of thirty and
twenty-two raters, they report relia-
bilities of .64 to .99 for group-mean
ratings.
Extensive research on forest
landscape aesthetics has developed
during the past thirty years. Much of
this research uses the scenic beauty
estimation (SBE) method (Daniel
and Boster 1976), a statistical tech-
nique that standardizes each rater’s
scores in order to control the varia-
tion in how a rater "anchors" the rat-
ing scale. The reliability of group-
mean ratings is frequently reported,
168 Landscape Journal
but not the average reliability for
individuals (Brown et al. 1988; Brown
and Daniel 1987; Daniel et al. 1989;
Hetherington et al. 1993; Rudis et al.
1988). Typically these reliabilities are
above .90 for scenic beauty.
Reports of the intra-group relia-
bility of ratings other than scenic
beauty or preference are less com-
mon. Herzog (1987) used ratings of
identifiability, coherence, spacious-
ness, complexity, mystery, texture,
and preference to characterize sev-
enty natural mountainous, canyon,
and desert scenes. Using groups of
from thirteen to twenty-six students,
he obtained intra-group reliabilities
for mean scores that ranged from .69
for coherence to .97 for preference.
Intra-group reliabilities are
generally high, and the more raters
there are in a group, the greater will
be the group-mean reliability. The
reliability of group-means is impor-
tant if they are going to be used as
variables in predictive research mod-
els or for making environmental deci-
sions. However, they do not reflect
the reliability of individual raters.
Inter-rater reliability. In practice,
landscape assessments conducted in
the field rarely involve judgments by
more than one or perhaps two trained
professionals. Assessing slides or
other representations makes it more
convenient to use a panel of evalua-
tors. However, it is still unusual for
more than a few professionals to
apply any of the agency evaluations
systems to a series of slides (Smardon
et al. 1988; USDA 1995; USDI 1980).
The only common occurrence of a
large group evaluating landscape
scenes is to create mean ratings for a
single attribute (e.g., scenic beauty or
preference), normally for research
purposes.
Schroeder (1984) reports the
inter-rater reliability of attractive-
ness or scenic beauty from ten stud-
ies, as well as one study of visual air
quality, two studies of enjoyableness,
and another of safety for urban parks.
The attractiveness studies have a
mean reliability of .530. Patsfall and
his colleagues (1984) report single-
rater reliabilities of.23 for SBE rat-
ings of vistas along the Blue Ridge
Parkway. Anderson (1976) extended
Zube’s (1974) study and found an
inter-rater reliability for individuals of
.69 for scenic value judgments of 212
scenes. He also used the Spearman-
Brown prophecy formula to deter-
mine that a group of four raters
would achieve a composite reliability
for scenic value of approximately .90.
Craik’s research group used
four scenes to evaluate inter-rater
reliabilities for fifteen attribute rat-
ings, primarily those used in the
Bureau of Land Management’s visual
resource assessment procedure
(Feimer et al. 1979). Even after
extending the analysis to 19 scenes,
they found that "the reliability of rat-
ings tends to be low for single
observers and hence it is advisable to
use composite judgments of panels of
independent observers" (Feimer et al.
1981, p. 16). In a later report on this
same work, Craik (1983, p. 72) rec-
ommends that "given present rating
systems, panels of at least five mem-
bers, rendering independent judg-
ments, are required to achieve ade-
quate levels of composite reliability."
The basis of this recommendation
comes from the application of the
Spearman-Brown prophecy formula
to determine the composite reliability
for a group of any size based on the
mean intra-group reliability. Kopka
and Ross (1984) also evaluated the
inter-rater reliability of BLM’s "level
of influence" procedure applied to
existing scenes. The findings from
these two independent studies con-
cerning the assessment reliability of
formalistic design qualities in the
landscape are summarized in Table 1.
They fall substantially short of
 Table 1. Inter-rater reliability for BLM’s level of influence variables.
Variable
Form
Line
Color
Texture
acceptable levels, even for
exploratory research, let alone for
professional assessments.
Craik (1972; Brush 1976) sug-
gests that a distinction be made
between evaluative appraisals and
preferential judgments. Appraisals
are based on a widely-accepted and
culturally-determined standard.
"Esthetic appraisals reflect the lay-
man’s attempt to employ a commonly
but imperfectly understood external
standard" (Craik 1972, p. 257). In
contrast, personal judgments are
more specialized assessments from a
particular perspective and for a par-
Kopka & Ross 1984 Feimer et al. 1981
.54 .45
.63 .19
.25 .13
.53 .41
ticular purpose. "Preferential ratings
[judgments] reflect all sorts of indi-
vidual and subgroup tastes, inclina-
tions, and dispositions" (Craik 1972,
p. 257). High inter-rater reliability
may be an indication that an attrib-
ute assessment is understood to be an
appraisal using common culturally
accepted criteria, while low reliability
may indicate a more personal judg-
ment (Brush 1976). For instance,
Coughlin and Goldstein (1970)
obtained higher inter-rater reliabili-
ties for attractiveness than for resi-
dential or sightseeing preference rat-
ings of a series of scenes.
Inter-Group and Inter-Rater Reliability of
Scenic Quality
Most landscape assessment
research has sought to explain why
people like certain landscapes more
than others (Zube, Sell and Taylor
1982). The term used to describe this
liking has varied among authors.
Zube (1974) refers to scenic value,
the Kaplans (1989, 1998) use prefer-
ence, and Daniel and Boster (1976)
choose scenic beauty. These terms
have been found to be essentially the
same (Zube et al. 1974). Because sce-
nic quality, preference or beauty is so
widely measured, it is possible to use
this measure to compare the relative
size of inter- and intra-group reliabili-
ties. It also creates a standard for
comparison to other attributes:
Methods. The inter-group and
inter-rater reliability of landscape
preference is evaluated based on data
from four previous research efforts.
Palmer and Smardon (1989) surveyed

Table 2. Study datasets used to compare inter-group and inter-rater reliability.

Sites Represented Participants
Location Media Year n Format Type Selection Year n Study Citation

Juneau, AK B/W offset press ’86 16 Rating-9 Residents Random ’86 406 Palmer & Smardon 1989
Juneau, AK B/W offset press ’86 t6 Rating-9 Public meeting Random ’86 41 ibid.
Dennis, MA Color photos ’76 56 Q-sort-7 Registered voter Random ’76 68 Palmer 1983
Dennis, MA B/W offset press ’76 56 Q-sort-7 Town list Random ’87 34 Palmer 1997
Dennis, MA Color slides ’76 56 Rating-10 Resident Available ’96 31 unpublished
Dennis, MA B/W offset press ’76 56 Q-sort-7 Env. Prof. Selected ’83 5t Palmer t985
Dennis, MA B/W offset press ’76 56 Q-sort-7 Env. Prof. Selected ’85 67 Palmer 1985
White Mtn, NH Color web press ’92 64 Rating-10 Residents Random ’95 77 Palmer 1998
White Mtn, NH Color web press ’92 64 Rating-10 USFS prof. Random ’95 205 ibid.
White Mtn, Nit Color web press ’92 64 Rating-10 Opinion leaders Census ’94/5 97 ibid.
US impact pmrs Color web press n/a 32 Compare-100 Austrian students Available ’90 59 Palmer et al. t990
US ~mpact pmrs Color web press n/a 32 Compare-100 French students Available ’87 99 ibid.
US ampact pmrs Color web press n/a 32 Compare-100 German students Available ’88/9 47 ibid.
US tmpact pmrs Color web press n/a 32 Compare-100 Hong Kong stds. Available ’90 53 ibid.
US ~mpact pmrs Color web press n/a 32 Compare-100 Italian students Available ’87 26 ibid.
US ~mpact pmrs Color web press n/a 32 Compare-100 Japanese students Available ’87 52 ibid.
US ampact pmrs Color web press n/a 32 Compare-100 Korean students Available ’87 128 ibid.
US ampact pmrs Color web press n/a 32 Compare-100 Puerto Rican stds. Available ’87 14 ibid.
US ~mpact pmrs Color webpress n/a 32 Compare-100 Spanish students Available ’87 100 ibid.
US ampact pmrs Color web press n/a 32 Compare-100 Utah students Available ’88 40 ibid.
US ~mpact pmrs Color webpress n/a 32 Compare-100 Yugoslav students Available ’87 47 ibi&
US ~mpact pmrs Color web press n/a 32 Compare-100 Central NY stds. Available ’87 59 ibid.

Palmer 169
 Table 3. Inter-group reliability of scenic ratings for Dennis, Massachusetts.
Citizen ’76 Citizen ’87 Citizen ’96 Env. Prof. ’83 Env. Prof. ’85

Citizens ’76 -- -0.947 0.904 0.955 0.952

Citizens ’87 0.947 -- -0.941 0.940 0.942
Citizens ’96 0.904 0.941 -- -0.899 0.895
Env. Prof. ’83 0.955 0.940 0.899 -- -0.992
Env. Prof. ’85 0.952 0.942 0.895 0.992 --

a random sample of residents and
attendees at a public workshop to
study the human-use values of wet-
lands in Juneau, Alaska. The survey
included sixteen photos representing
the range of local wetland types and
conditions. The second study began
as part of a community effort to
develop a comprehensive plan for
Dennis, Massachusetts. In 1976, a
random sample of registered voters
evaluated fifty-six photographs repre-
senting the town (Palmer 1983). Resi-
dents evaluated the same scenes in
1987 and 1996 (Palmer 1997). These
ratings are compared to those from
employees of the U.S. Army Corps of
Engineers which were gathered in
preparation for a training course in
landscape aesthetics (Palmer 1985).
The third study evaluated simula-
tions of different harvesting intensi-
ties, patterns, and patch sizes of
clearcuts in the White Mountain
National Forest (Palmer 1998).
Respondents included a random sam-
ple of regional.residents, opinion lead-
ers in the management of the area’s
forests, and environmental profes-
sionals stationed in National Forests
in the northeastern quarter of the
United States. The final study
involves twelve groups of college stu-
dents from around the world (Palmer
et al. 1990). They evaluated sixteen
matched simulations from the north-
eastern and southwestern United
States portraying pre- and post-
impact conditions. Citations for these
data-sets and the general characteris-
tics of the respondents and simulation
media are summarized in Table 2.
Results. The correlation between
the mean scenic ratings of Juneau
residents and workshop attendees is
.971 for scenic ratings of sixteen
diverSe wetlands. Table 3 shows the
correlations among groups evaluating
the Dennis scenes. The average cor-
relation among the five groups is
.937. The highest correlation is .992
between the two groups of environ-
mental professionals, and the lowest
is .895 between 1996 citizens and
1985 professionals.
In the White Mountain
clearcutting study, the inter-group
correlation of citizens with opinion
leaders is .980, and with Forest Ser-
vice professionals it is .978. The cor-
relation between opinion leaders and
Forest Service employees is .974. The
correlations among the four groups of
opinion leaders are shown in Table 4,
with an average correlation of .894.
Table 5 shows the correlations among
the seven groups of Forest Service
environmental professionals. Their
average correlation is .971.
In Table 6 are the correlations
between twelve student groups from
around the world. Even with such
diverse respondent groups, the aver-
age inter-group correlation is .804.
The lowest correlation is .496
between students from Japan and
Germany, while the highest is .969
between the Austrian and German
students. The inter-group correla-
tions from these four studies indicate
why the quality of landscape assess-
ments enjoys such a high reputa-
tion-most measures of reliability
meet the highest standards, and all
but a very few meet standards of
acceptability.

Table 4. Inter-group reliability among opinion leaders’ scenic ratings for clearcutting alternatives in the
White Mountains, New Hampshire.
Appalachian Forest Resources North Country Roundtable on
Trail Council Steering Committee Council Forest Law
Appalachian Trail Council -- -0.934 0.890 0.881
Forest Resources Steering Com. 0.934 -- -0.902 0.890
North Country Council 0.890 0.902 -- -0.868
Roundtable on Forest Law 0.881 0.890 0.868 --

170 Landscape Journal

 Table 5. Inter-group reliability among USFS employees’ scenic ratings for clearcutting alternatives in the
White Mountains, New Hampshir,e.
Archaeol. Engineer Forester Land Arch Manager Rec Spec Wild,Bio.
Archaeologist -- -0.985 0.968 0.937 0.948 0.973 0.977
Engineer 0.985 -- -0.973 0.947 0.950 0.975 0.980
Forester 0.968 0.973 -- -0.970 0.984 0.995 0.989
Landscape Arch 0.937 0.947 0.970 -- -0.986 0.975 0.945
Management 0.948 0.950 0.984 0.986 -- -0.987 0.960
Recreation Spec 0.973 0.975 0.995 0.975 0.987 -- -0.984
Wildlife Biologist 0.977 0.980 0,989 0.945 0.960 0.984 --

However, the inter-rater relia-
bilities in Table 7 are much less
encouraging. The average inter-rater
correlation is .307 for Juneau resi-
dents, and .355 for workshop atten-
dees. That is approximately one-third
the inter-rater correlation between
the two groups. The average of the
intra-group correlations for the five
Dennis study groups is .608. Again,
this is a substantial drop in reliability
from the average inter-group correla-
tion of .937. The average inter-rater
correlation is .554 among the three
major groups in the White Mountain
study. This is down from an average
inter-group correlation of .977. The
average of the twelve inter-rater cor-
relations from the international study
is .427, down from an average inter-
group correlation of .804.
If landscape assessments are
made primarily by individuals and not
by large panels of evaluators, then
these results indicate that the relia-
bility of scenic assessments is unlikely
to be reaching acceptable levels. The
next sections will consider the relia-
bility of other visual qualities.
Inter-group and Inter-rater Reliability of
Landscape Dimensions
Zube (1974) defines landscape
dimensions "as physical characteris-
tics or attributes of the landscape
which can be measured using either
normal ratio scales or psychometric
scaling." Examples of such dimen-
sions include: percent tree or water
cover, length or area of the view, rela-
tive elevation change, and various
edge and contrast indices. These
dimensions bear a remarkable resem-
blance to those employed by quantita-
tive landscape ecologists today
(Tnrner and Gardner 1991). Zube
investigated the relationship between
twenty-three landscape dimensions
and scenic quality. His landscape
dimensions were all measured from
USGS 1:24,000 topography or Massa-
chusetts MapDown land use maps.
Palmer (1996) used a similar
approach to validate a GIS model of
spaciousness. A regression analysis
found landscape dimensions
explained approximately half of the
variation in Zube’s perceived scenic
value and Palmer’s perceived spa-
ciousness.
Shafer (1969) employed a dif-
ferent approach to measure land-
scape dimensions. He divided an eye-
level photograph into foreground,
middle ground and background. Then
the area and perimeter of content
areas were measured in each zone.
Examples of content include water,
trees, buildings, ground cover, and
pavement. This approach to measur-
ing landscape dimensions also
accounts for approximately half the
variation in visual preference.

Table 6. Inter-group reliability in a multi-national study of visual impact perceptions.

Au CNY Fr Gr HK It Ja Ko PR Sp Ut Yu
Austrian -- -0.958 0.952 0.969 0.681 0.863 0.604 0.723 0.802 0.934 0.891 0.906
Central NY 0.958 -- -0.928 0.952 0.709 0.881 0.641 0.750 0.813 0.945 0.914 0.923
French 0.952 0.928 -- -0.958 0.631 0.784 0.599 0.712 0.712 0.902 0.870 0.887
German 0.969 0.952 0.958 -- -0.587 0.813 0.496 0.661 0.731 0.880 0.832 0.853
Hong Kong 0.681 0.709 0.631 0.587 -- -0.800 0.791 0.704 0.765 0.790 0.855 0.821
Italian 0.863 0.881 0.784 0.813 0.800 -- -0.714 0.731 0.853 0.733 0.783 0.796
Japanese 0.604 0.641 0.599 0.496 0.791 0.714 -- -0.843 0.619 0.733 0.783 0.844
Korean 0.723 0.750 0.712 0.661 0.704 0.731 0.843 -- -0.663 0.821 0.800 0.844
Puerto Rican 0.802 0.813 0.712 0.731 0.765 0.853 0.619 0.663 -- -0.782 0.767 0.855
Spanish 0.934 0.945 0.902 0.880 0.790 0.855 0.733 0.821 0.782 -- -0.963 0.933
Utah 0.891 0.914 0.870 0.832 0.855 0.847 0.783 0.800 0.767 0.963 -- -0.916
Yugoslav 0.906 0.923 0.887 0.853 0.821 0.902 0.796 0.844 0.855 0.933 0.916 --

Palmer 171
 Table 7. Inter-Rater Reliability of Scenic Ratings from Four Studies.
Location Respondents n Mean 95%-ile Median 5%-ile

Juneau, AK Residents 406 .307 .793 .424 -.249

Public meeting attendees 41 .355 .824 .490 -.400
Dennis, MA Registered voter 68 .603 .824 .636 .272
Town list 34 .563 .837 .606 .076
Residents 31 .539 .853 .655 -. 198
Env. Prof. in Corps of Engineers 51 .635 .818 .674 .315
Env. prof. in Corps of Engineers. 67 .701 .971 .807 .089
White Mtn, NH Citizens 69 .512 .800 .593 -.067
Opinion leaders 95 .532 .812 .630 -.147
Appalachian Trail Council 24 .574 .840 .709 -.645
Forest Res. Steering Committee 18 .344 .737 .441 -.321
North Country Council 43 .653 .833 .701 .275
Roundtable on Forest Law 10 .554 .815 .676 -.095
USFS employees 205 .619 .837 .672 .215
Archaeologist 9 .690 .820 .697 .543
Engineer 13 .537 .814 .598 .122
Forester 87 .601 .830 .658 .798
Landscape Architect 10 .684 .882 .735 .434
Management 30 .584 .838 .651 .002
Recreation Specialist 20 .687 .848 .706 .457
Wildlife Biologist 36 .666 .839 .717 .216
US impact pairs Austrian students 59 .602 .827 .646 .176
French students 29 .451 .774 .461 .090
German students 47 .543 .844 .598 .002
Hong Kong students 53 .344 .675 .383 -.154
Italian students 26 .314 .699 .322 -. 147
Japanese students 52 .347 .704 .390 -.180
Korean students 128 .248 .606 .251 -.121
Puerto Rican students 14 .273 .685 .364 -.274
Spanish students 100 .505 .785 .528 .149
Utah State students 40 .469 .805 .497 .049
Yugoslav students 47 .450 .74 t .479 .061
SUNY ESF students 59 .583 .812 .624 .226

The approaches developed by
Zube and Shafer use physical tools to
measure the landscape’s dimensions.
Human judgment can also be used to
estimate these measurements, for
instance the relative area of a view
covered by forest. However, when
people are used as the measuring
device, more complicated constructs
can also be measured, such as natu-
ralism or spaciousness. It is the relia-
bility of using people to measure
landscape dimensions that is tested in
this section.
Methods. Respondents are thirty
advanced landscape architecture or
environmental science students at
State University of New York’s Col-
lege of Environmental Science and
Forestry in 1997 and 1998. They eval-
uated offset printed photographs of
Dennis, Massachusetts taken in 1976.
They were instructed to identify the
highest, lowest, and intermediate
quality scenes and describe the crite-
ria for their decisions. Using these
scenes as anchor points on a seven-
point scale, they sorted the remaining
scenes among the seven rating levels.
Each quality was evaluated on a dif-
ferent day. The four landscape dimen-
sions were described as follows:
Naturalism refers to aspects of the
landscape that could exist without
human care. Nature is an expres-
sion of how much vegetation is in a
view, how organic are its elements
and patterns, and how uncontrolled
are the natural processes.
Development refers to aspects of
the landscape that are human cre-
ations. Development is an expres-
sion of human control over natural
processes or patterns, and the dom-
inance of structures, such as build-
ings, roads, or dams.

172 Landscape Journal

 Spaciousness is the landscape’s
enclosure or expansiveness. It
describes how much room there is
to wander in the view, or how far
you could go before you reach the
boundaries.
Preference is how much you like or
dislike a landscape. It is also called
scenic quality, attractiveness, or
beauty. People’s preferences are
descriptions of their personal expe-
rience.
Results. The inter-rater correla-
tions reported in Table 8 show that
acceptable levels of reliability are
achieved for perceived naturalism
(.796), development (.762), and spa-
ciousness (.715). The response pat-
terns for naturalism and development
are nearly mirror images of each
other. The reliability of development
is brought down somewhat by one
student whose responses correlate
negatively with the other evaluators.
The reliability of the preference
ratings (.582) in Table 8 is compara-
ble to that for scenic preference from
the other studies presented in this
paper. However, it is substantially
lower than for the three landscape
dimensions.
Table 8. Inter-rater Reliability for Perceived Landscape Dimensions in Dennis, MA.
Attribute n Mean
Naturalism 30
Development 30
Spaciousness 30
Preference 30
Inter-group and Inter-rater Reliability of
Informational Content
The Kaplans (1982, 1989, 1998)
have proposed an elegant theory that
relates the informational content of a
scene to its preference. It is related to
Appleton’s (1975) prospect-refuge
theory of environmental preference¯
They both refer to adaptive pressures
during human evolution to lend
authority to their theories¯ Their
hypothesis is that humans have
evolved to seek and understand infor-
mation in a particular type of land-
scape, namely the savanna (Kaplan
and Kaplan 1982, p. 75-77). As such,
the preference for visual conditions
that enhance the acquisition of infor-
mation in savanna-like landscapes is
in the genes, so to speak.
The Kaplans posit a framework
that characterizes information from
two perspectives. First, information
contributes to either understanding
or exploration. "Understanding
refers to the desire people have to
make sense of their world, to compre-
hend what goes on around them.
Understanding provides a sense of
security.... People want to explore,
to expand their horizons and find out
what lies ahead. They seek more
information and look for new chal-
lenges" (Kaplan, Kaplan & Ryan
1998, p. 10)¯ Second, visual informa-
tion is presented in two forms: two-
dimensional and three-dimensional.
Two-dimensional information
"involves the direct perception of the
elements in the scene in terms of
their number, grouping, and place-
ment .... When viewing scenes, peo-
ple not only infer a third dimension,
95%-ile
.796 .916
.762 .9 l0
.715 .859
.582 .823
but imagine themselves in the scene.
¯.. involve the inference of what
being in the pictured space would
entail" (Kaplan et al. 1998, p. 13).
Four information concepts are
derived from this framework. Two-
dimensional understanding is called
coherence, and two-dimensional
exploration is complexity. Three-
dimensional understanding is legibil-
ity, and three-dimensional explo-
ration is mystery. The Kaplans call
this two-by-two classification grid the
Preference Matrix. It is thought that
people prefer landscapes that are
coherent, complex, legible, and mys-
terious.
It is suggested that the land-
scape dimensions are descriptions of
physical condition. However, the four
Preference Matrix variables describe
our experience and interpretation of
information in the landscape. In a
sense they are more removed from an
objective physical condition and
closer to a psychological outcome.
Methods. The respondents, photo-
graphs, and procedures are the same
as those used for the landscape dimen-
sions ratings. The students were given
a reading assignment (Kaplan and
Kaplan 1989, p. 49-58) in preparation
for a thirty minute lecture about the
Kaplan’s information framework. As
before, students rated the attributes
on different days. The four informa-
tional attributes were described in the
instructions as follows:
Coherence is the landscape’s order-
liness or confusion. It describes how
well a view "hangs together," or
how easy it is to understand what
you see. It is enhanced by anything
that helps organize the patterns of
light, size, texture, or other ele-
ments into a few major units.
Complexity is the landscape’s
intricacy or simplicity. It describes
how much is going on in a view;
Median 5%-ile
.826 .466
.828 -.046
.728 .514
.613 .226
how many eletnents of different
kinds it contains. It is the promise
of further information, if only there
were more time to look at it from
the present vantage point.
Legibility is whether a landscape
is memorable or indistinguishable.
It describes how easy it would be to
find one’s way around the view; how
easy it would be to figure out where
one is at any given moment or to
find one’s way back to any given
place.
Palmer 173
 Mystery in the landscape is the
result of incomplete perception. It
describes the extent to which fur-
ther information is promised to the
obsetwer if she were to walk deeper
into the scene. This is not a prom-
ise of surprise, but of information
that has continuity with what is
already available.
These definitions are based on
descriptions given by the Kaplans and
their graduate students (Kaplan and
Kaplan 1989; Herzog 1989; Herzog,
Kaplan and Kaplan 1982).
Results. Table 9 lists the reliabil-
ities obtained for the informational
attributes. These reliabilities are
highest for the exploratory variables
complexity (.315) and mystery (.262).
The three-dimensional variables,
legibility (.214) and ~nystery (.26),
are higher than the two-dimensional
variables. The lowest reliability was
found for the two-dimensional under-
standing variable, coherence(. 186).
However, all of these intra-group reli-
abilities are unacceptably low.
Inter-Group and Inter-Rater Reliability of
Compositional Elements
A common approach to assess-
ing visual impacts involves evaluating
the amount of change in the scene’s
visual composition. Litton (1968;
USDA 1973) initiated what became
Table 9. Inter-rater reliability for informational attributes of the Dennis, MA landscape.
Attribute n Mean
Coherence 28
Complexity 28
Legibility 28
Mystery 28
the common use of form, line, color,
and texture as the attributes most
commonly used to describe landscape
character and change. In particular;
procedures have been developed that
evaluate changes in contrast associ-
ated with forIn, line, color, texture, as
well as scale and spatial dominance
174 Landscape Journal
(USDI 1980; Smardon et al. 1988).
More recent manuals developed in
Great Britain build on the work of
Dame Sylvia Crowe and demonstrate
how a more complete palette of aes-
thetic factors can be used to describe
and evaluate landscapes (Lucas 1990;
Bell 1993).
Methods. Respondents were
twenty-five professionals in a two-day
visual assessment continuing educa-
tion course taught in early-December
1997 in Albany, New York. The indi-
vidual contrast ratings are the maxi-
mum contrast from the land/water,
vegetation, or structures components
of the landscape. The ratings involve
two visual principles, contrast and
dominance. Visual elements are the
source of visual contrast in the land-
scape, creating the patterns that we
see. An object may differ from its set-
ting or other objects in one or more
element. When there is significant
contrast in one or more of the ele-
ments, one object may dominate
other parts of the landscape. The con-
trast or dominance of the following
six visual elements are evaluated:
Color is the major visual property
of surfaces attributed to reflected
light of a particular intensity and
wavelength. Described by its hue
(tint or wavelength), value (light or
95%-ile
.186 .518
.315 .727
.214 .630
.262 .581
darkness), and chroma (saturation
or brilliance). Lighter, warmer,
brighter colors tend to "advance",
while darker, cooler, duller colors
tend to "retreat" in a scene. Dark
next to light tends to attract the
eye and this contrast becomes a
visual focal point.
Form is the mass or shape of an
object or objects which appear uni-
fied. Forms are described by their
geometry, complexity, and orienta-
tion in the landscape. Forms that
are bold, regular, solid, or vertical
tend to be dominant in the land-
scape.
Line is a path, real or imagined,
that the eye follows when perceiv-
ing abrupt differences in color or
texture, or when objects are
aligned in a one-ditnensional
sequence, described by its boldness,
complexity, and orientation. It is
usually evident as the edge of forms
in the landscape. Bold vertical lines
which interrupt the skyline tend to
dominate weak horizontal lines.
Texture is small forms or color
mixtures aggregated into a contin-
uous surface pattern. The aggrega-
tion is sufficient that the parts do
not appear as discrete objects in
the composition of the scene. Tex-
tures are described by their grain,
density, regularity, and internal
contrast. Coarse and high-contrast
textures tend to dominate fine-
grained textures of low internal
contrast.
Scale is the relative size of an
object in relation to its surrounding
landscape. The scale may be in
relation to the landscape setting as
a whole, the proportion of field-of-
view, or other distinct objects.
Large, heavy, massive objects
within a confined space dominate
small, light, delicate objects in
more expansive settings.
Space is the three-dimensional
arrangement of objects and voids.
Compositions are described as
panoramic, enclosed, feature, focal,
or canopied. Position of objects or
Median 5%-ile
.227 -.205
.350 -.358
.224 -.246
.278 -.081
view in the landscape is relative to
topography. Backdrop is the sky,
water, or land background against
which objects are seen. Objects
which occupy vulnerable positions
within spatial compositions, which
are high in the landscape, and/or
which are seen against the sky dom-
inate in the scene. The sum of the
contrast and dominance ratings is
used to create an index of visual
impact severity.
 The evaluation forms are
adapted from those prepared by
Smardon and his colleagues (1981)
for the Bureau of Land Management.
These judgments were made for five
sets of pre- and post-impact slide
pairs. The impacts evaluated were
eleven wind turbines installed in Ver-
mont as seen from 1.25 and 4.0 miles,
a forest view 15 percent of which is
harvested in four or five acre
clearcuts and another in twenty-five
to thirty acre clearcuts in the White
Mountains, and a new on-ramp to a
limited access highway near Bing-
hamton, New York.
Results. The reliability of the
compositional attributes is shown in
Table 10. The average inter-rater
reliability of scenic value ratings for
the nine slides was .539, which is
comparable, though slightly lower
than the scenic value ratings reported
above. There is a range of reliability
for the contrast ratings. Unaccept-
ably low contrast reliabilities are
found for scale (.280) and line (.423).
The contrast reliabilities for color
Table 10. Inter-rater Reliability of Landscape Composition Attributes.
Rating n Mean
Scenic Value 25
Contrast (Sum) 25
Color Contrast 25
Form Contrast 25
Line Contrast 25
Scale Contrast 25
Texture Contrast 25
Scale Dominance 25
Spatial Dominance 25
Visual Impact Severity 25
(.503), and form (.563) are minimal.
Only texture contrast (.620) has a
minimally acceptable reliability.
These results are comparable to
those from the studies summarized in
Table 1. The reliability of scale domi-
nance (.561) and spatial dominance
(.376) is also low.
The sum of these five contrast
ratings forms a contrast index that is
more reliable (.630) than any of its
individual components. The reliabil-
ity of the index of visual impact
severity is even more reliable (.664).
Reliability of an index is normally
higher than the reliability of its com-
ponents (Nunnally 1978). However,
these levels still fall well below pro-
fessionally acceptable levels.
Discussion
In general, landscape attributes
that have a more denotative charac-
ter seem to have greater inter-rater
reliability than those with a more
connotative charactex: Denotative
attributes provide clear designation
or referential meaning. These are
generally agreed upon "objective"
facts. They are particularly appropri-
ate for evaluative appraisals. Conno-
tative attributes provide emotive or
metaphorical meaning. They include
any suggestion or implication beyond
denotative meaning. This more per-
sonal attribution suggests that conno-
tative meaning is largely a matter of
preferential judgments. Naturalism,
development, and spaciousness seem
to be examples of denotative attrib-
95%-ile
.539 .912
.630 .950
.503 .952
.563 .923
.423 .922
.280 .885
.620 .948
.561 .923
.376 .896
.664 .960
utes. Landscape preference and the
compositional elements form, line,
colo~; and texture may be in a gray
area between connotative and deno-
tative meaning. The reliability of
coherence, complexity, mystery, and
legibility is sufficiently low to suggest
that they have highly connotative
meaning.
The relatively poor reliability of
the information variables may con-
tribute to the discussion of nature
versus nurture, whether there is a
genetically based or human evolu-
tionary significance in their meaning.
Perhaps their relevance to our every-
day survival has changed as our cul-
tural and environmental conditions
have changed. Another possibility is
that static photographs are inade-
quate to effectively trigger our infor-
mation-seeking instincts. Perhaps
more attention needs to be paid to
how" movement through the land-
scape influences both landscape pref-
erence and information-seeking
behavior.
At least with regard to scenic
quality, these results also indicate
that evaluations from landscape pro-
fessionals are more reliable than
from public individuals. Table 11
compares the average individual (i.e.,
inter-rater) reliability for profes-
sional and public respondents to the
Dennis, Massachusetts and White
Mountains studies. This is one aspect
of Carlson’s (1977) argument that
landscape evaluations should be left
to specially trained environmental
professionals. However; the difference
Median 5%-ile
.592 -.005
.681 .193
.583 -.272
.664 -. 144
.497 -.392
.395 -.608
.716 -. 154
.620 .000
.463 -.453
.742 .006
is not so great as to nullify the useful-
ness of evaluations by random sam-
ples of the public, nor can it be used
to justify evaluations by only one or
two professionals.
There are several possible rea-
sons that occur to the author for the
generally poor results reported here.
One possibility is that photographs
Palmer 175

Table 11. Inter-rater reliability of scenic ratings by professionals and the public.
Dennis, MA
Professionals .668
Public .568
provide insufficient information or
that different people fill in missing
information differently. This is a
question of the validity of photo-
graphic representations. Many stud-
ies have reported that photographs
appear to be valid representations,
but recent work by Hoffman (1997;
Hoffman and Palmer 1994) suggests
that there are serious validity con-
cerns under some circumstances.
Another possible explanation is that
the instructions describing the land-
scape attributes to be evaluated were
White Mtns.
.619
.512
For some time, the practice of
landscape assessment has been domi-
nated by methods that use rating
scales or checklists. The mixed find-
ings reported here suggest that it
may be appropriate to investigate the
reliability of other landscape assess-
ment methods. Those who used rat-
ing scales did so because they sought
reliable results; they never made
claims to uncover deep meaning. In
contrast, other researchers chose to
develop qualitative methods to search
for deeper meaning and to gain a
scape assessments are most com-
monly made by a single professional.
Published evaluations of the reliabil-
ity of a single evaluator (i.e., intra-
group reliability) for various land-
scape qualities do not ever meet pro-
fessional standards (i.e., greater than
.9) and normally fall below minimally
acceptable levels (i.e., .7). Table 9
shows the number of evaluators
needed for each of the landscape
qualities considered in this paper.
The size of these evaluation panels is
determined by applying the Spear-
man-Brown Prophecy Formula (Nun-
nally 1978; Feimer et al. 1979; Ander-
son et al. 1976).
Several recommendations seem
appropriate given these findings.
Research involving landscape assess-
ments by the public or professionals
must include: (1) field validations of

Table 12. The number of assessors needed to obtain minimally and professionally reliable ratings of landscape attributes.
Number of assessors to reach:
Attribute Tested reliability .9 reliability .7 reliability
Scenic value .582 7 2
Naturalism .796 3 l
Development .762 3 1
Spaciousness .715 4 1
Complexity .315 20 6
Mystery .262 25 7
Legibility .214 33 9
Coherence .186 40 11
Color Contrast .503 9 3
Form Contrast .563 7 2
Line Contrast .423 13 4
Scale Contrast .280 24 6
Texture Contrast .620 6 2
Scale Dominance .561 7 2
Spatial Dominance .376 15 4

not understood or were inadequate in richer understanding from the land-

other ways. For instance, only written
or oral instructions are typically given
to respondents. It seems reasonable
that some form ofvisuat instructions
may be necessary to provide more
reliable evaluations of visible attrib-
utes. Finally, low retiabilities may be
an indication of low salience. Some of
these landscape attributes may be
fuzzy concepts or constructs that
need further development to reach
professional standards of reliable
assessment.
scape. Perhaps it is time to also inves-
tigate and refine the reliability of
these methods. They may be just as
reliable as some of the rating scales!
Conclusions
The results presented here give
reason for some concern in the way
landscape assessments are conducted
in both research and practice. Land-
photographic representations when
possible; (2) validation of the evalua-
tion instructions; and (3) use of pho-
tographs or images to help explain
the landscape attributes being evalu-
ated. The professional application of
landscape assessments must include:
(1) multiple trait~ed evaluators; (2) a
reliability assessment of the evalua-
tions; and (3) field validation of photo-
graphic representations when possible.

176 Landscape Journal


Acknowledgments
Previously unpublished data used for this
research were funded by the North Central
Forest Research Station, Chicago, Illinois.
Slides for the landscape compositional assess-
ment were provided by Vermont Environmen-
tal Research Associates, Waterbury Center,
Vermont and Integrated Site, Syracuse, New
York.
Note
1. The intraclass correlation (ICC) is an alter-
native approach to estimating reliability that
normally gives roughly comparable values
(Ebel 1951;Jones et al. 1983). The ICC is cal-
culated using variance components from an
ANOVA table. There is an extensive literature
on the appropriate valance components to
include in an ICC (e.g., Shrout & Fleiss 1979).
Burry-Stock et al. (1996) considers ICC calcu-
lations "beyond the statistical expertise of the
average observer using observation or perform-
ance data" in their research.
References
Anderson, T. W., E. H. Zube and W. P. Mac-
Connell. 1976. "Predicting scenic
Resource Values." In Studies in Landscape
Perception. Amherst: Institute for Man
and Environment, University of Massa-
chusetts.
Appleton, J. 1975. The Experience of Landscape.
London: Wiley.
--. 1984. "Prospects and Refuges Re-
visited." Landscape Journal 3 (XX): 91-103.
Bell, S. 1993. Elements of Visual Design in the
Landscape. London: E & FN Spon.
Brook, I. 1998. "Goethean Science as a Way to
Read Landscape." Landscape Research
23(1): 51-69.
Brown, T. C., and T. C. Daniel. 1987. "Context
Effects in Perceived Environmental
Quality Assessment: Scene Selection
and Landscape Ratings."Journal of Envi-
ronmental Psychology 7(3): 233-250.
Brown, T. C., T. C. Daniel, M. T. Richards and
D. A. King. 1988. "Recreation Participa-
tion and the Validity of Photo-based
Preference Judgements."Journal of
Leisure Research 20(4): 40-60.
Brush, R. O. 1976. "Perceived Quality of Scenic
and Recreational Environments." In Per-
ceiving Environmental Quality: Research and
Applications. New York: Plenum Press.
Bnhyoff, G.J., W. A. Leuschner andJ. D. Well-
man. 1979. "Aesthetic Impacts of South-
ern Pine Beetle Damage."Journal of
Environmental Management 8(3): 261-267.
Burry-Stock,J. A., D. G. Shaw, C. Laurire and
B. S. Chissom. 1996. "Rater Agreement
Indexes for Performance Assessment."
Educational and Psychological Measurement
56(2): 25t-262.
Carlson, A. A. 1977. "On the possibility of
quantifying scenic beauty." Landscape
Planning 4(2): 131-172.
Coughlin, R. E. and K. A. Goldstein. 1970.
"The Extent of Agreement Among
Observers on Environmental Attractive-
Ness." Discussion Paper 37. Philadel-
phia: Regional Science Research Insti-
tute.
Craik, C. H. 1972. "Psychological Factors in
Landscape Appraisal." Environment and
Behavior 4(3): 255-266.
. 1983. "The psychology of large scale
environments. In Environmental Psychol-
ogy: Directions and Perspectives. New York:
Praeger Publications.
-. and N. R. Feimer. 1979. "Setting Tech-
nical Standards for Visual Assessment
Procedures." In Our National Landscape.
General Technical Report. PSW-35.
Berkeley, CA: USDA Forest Service,
Pacific Southwest Forest and Range
Experiment Station.
Daniel, T. C., L. M. Anderson, H. W. Schroeder
and L. Wheeler III. 1977. "Mapping the
Scenic Beauty of Forest Landscapes."
Leisure Sciences 1 (1): 335-52.
., and R. S. Boster. 1976. "Measuring
Landscape Esthetics: the Scenic Beauty
Estimation Method." Research Paper
RM-167. Fort Collins, CO: USDA Forest
Service, Rocky Mountain Forest and
Range Experiment Station.
., T. C. Brown, D. A. King, M. T.
Richards, and W. P. Stewart. 1989. "Per-
ceived Scenic Beauty and Contingent
Valuation of Forest Campgrounds." For-
est Science, 35(1): 76-90.
Ebel, R. L. 1951. "Estimation of the reliability
of ratings." Psychometrica 16 (4): 407-424.
Feimer, N. R., K. H. Craik, R. C. Smardon, and
S. R.J. Sheppard. 1979. Evaluating the
Effiectiveness of Observer Based Visual
Resource and hnapact Assessment
Methods." In Our National Landscape.
General Technical Report. PSW-35.
Berkeley, CA.: USDA Forest Service,
Pacific Southwest Forest and Range
Experiment Station.
, R. C. Smardon, and K. H. Craik. 1981.
"Evaluating the effectiveness of
observer based visual resource and
impact assessment methods." Landscape
Research 6(1): 12-16.
Gobster, P. H., and R. E. Chenoweth. 1989.
"The dimensions of aesthetic prefer-
ence: a quantitative analysis."Journal of
Environmental Management 29 (1): 47-72.
Great Literature: Personal Library Series. 1992.
(CD-ROM) Parsippany, NJ: Bureau
Development, Inc.
Groat, L. 1982. "Meaning in post-modern
architecture: an examination using the
multiple sorting task."Journal of Environ-
mentalPsychology 2(1): 3-22.
Guilford,J. P. 1954. Psychometric Methods. New
York: McGraw-Hill.
Herzog, T. R. 1987. "A cognitive analysis of
preference for natural environments:
mountains, canyons, and deserts." Land-
scape Journal 6(2): 140-152.
¯ 1989. "A cognitive analysis of prefer-
ence for urban nature."Journal of Envi-
ronmentalPsychology 9(1): 27-43.
, S. Kaplan and R. Kaplan 1982. "The
prediction of preference for unfamiliar
urban places." Population and Environment
5(1): 43-59.
Hetherington,J., T.C. Daniel and T.C. Brown.
1993. "Is motion more important than it
sounds?: the medium of presentation in
environmental perception research."
Journal of Environmental Psychology 13 (4):
283-291.
Hoffrnan, R.E. 1997. Testing the validity and
reliability of slides as representations of
northern hardwood forest conditions.
(Ph.D. dissertation). SUNY College of
Environmental Science and Forestry.
., andJ. F. Palmer. 1994. "Validity of
using photographs to represent visible
qualities of forest environments." In
History and Culture: Proceedings of the Coun-
cil of Educators in Landscape Architecture
1994 Coherence. Washington, D. C.:
Landscape Architecture
Foundation/Council of Educators in
Landscape Architecture.
Hull, R. B., IV and G.J. Buhyoft: 1984. "Indi-
vidual and group reliability of landscape
assessments." Landscape Planning 11 (!):
67-71.
Jones, A. P., L. A.Johnson, M. C. Butler and
D. S. Main. 1983. "Apples and oranges:
an emperical comparison of commonly
used indices of interrater agreement."
Academy of Management Journal 26(3):
507-519.
Kaplan, R., and S. Kaplan. 1989. The Experience of
Nature: A Psychological Perspective. New
York: Cambridge University Press.
., S. Kaplan and R. L. Ryan. t998. With
People in Mind. Washington, D. C: Island
Press.
Kaplan, S., and R. Kaplan. 1982. Cognition and
Environment. New York: Cambridge Uni-
versity Press.
Kopka, S., and M. Ross. 1984. "A study of the
reliability of the Bureau of Land Man-
agement visual resource assessment
scheme." Landscape Planning 11 (2):
161-166.
Litton, B: R.,Jr. 1968. "Forest landscape
description and inventories--a basis for
land planning and design." Research
Paper PSW-49. Berkeley, CA: SUDA
Forest Service, Pacific Southwest Forest
and Range Experiment Station.
Lucas, O. W. R. 1991. The Design of Forest Land-
scapes. New York: Oxford University
Press.
Palmer 177
Norberg-Schulz, C. 1979. Genius Loci: Towards
a Phenomenology of Architecture. New York:
Rizzoli International Publications.
Nunnally, J. C. 1978. Psychometric Theory. New
York: McGraw-Hill.
Palmer,J. F. 1983. "Assessment of coastaI wet-
lands in Dennis, Massachusetts." In The
Future of Wetlands: Assessing Visual-Cul-
tural Values of Wetlands. Montclair, New
Jersey: Allanheld, Osmun Co.
¯ 1985. "The perception of landscape
visual quality by environmental profes-
sionals and local citizens." Syracuse:
Faculty of Landscape Architecture,
SUNY College of Environmental Sci-
ence and Forestry.
--. 1996. Modeling Spaciousness in the Dutch
Landscape. Report 119. Wageningen, The
Netherlands: Agricultural Research
Department, Winand Staring Centre.
--. 1997. "Stability of landscape percep-
tions in the face of landscape change."
Landscape and Urban Planning 37(1/2):
109-113.
--. 1998. Cleareutting in the White Mountains:
Perceptions of Citizens, Opinion Leaders and
U.S. Forest Service Employees. [NYCFRD
98-0i] Syracuse, NY: New York Center
for Forestry Research and Develop-
ment, SUNY College of Environmental
Science and Forestry.
--., S. Alonso, K. Dong-hee,J. Gury, Y.
Hernandez, R. Ohno, G. Oneto, A.
Pogacnik, and R. Smardon. 1990. "A
multi-national study assessing perceived
visual i~npacts." Impact Assessment Bulletin
8(4): 31-48.
--, and R.C. Smardon. 1989. "Measuring
human values associated with wet-
lands." In Intractable ConJlicts and their
Transformation. Syracuse, NY: Syracuse
University Press.
178 Landscape Journal
Patsfall, M. R., N. R. Feimer, G.J. Buhyoff, and
J. D. Wellamn. 1984. "The prediction of
scenic beauty from landscape content
and composition."J0urna/ofEnvironmen-
talPsychology 4(1): 7-26.
Potteiger, M. andJ. Purinton. 1998. Landscape
Narratives. New York:John Wiley &
Sons.
Robinson, W. S. 1950. "Ecological correlations
and the behavior of individuals." Ameri-
can Sociological Review 15(3): 351-357.
Rudis, V. A.,J. H. Gramann, E.J. Ruddell, and
J. M. Westphal. 1988. "Forest inventory
and management-based visual prefer-
ence models of southern pine stands."
Forest Science 34(4): 846-863.
Schroeder, H.W. 1984. "Environmental percep-
tion rating scales: a case for simple
methods of analysis." Environment and
Behavior 16(5): 573-598.
Shafer, E. L.,J. E. Hamilton and E. A. Schmidt.
1969. "Natural landscape preferences: a
predictive lnodel.’Journal of Leisure
Research 1(1): 1-19.
Shakespeare, W. 1992. Romeo and Juliet. In Great
Literature: Personal Library Series. 1992.
(CD-ROM) Parsippany, NJ: Bureau
Development, Inc.
Shrout, P. E. andJ. L. Fleiss. 1979. "Intraclass
correlations: uses in assessing rater reli-
ability." Psychological Bulletin 86 (2):
420-428.
Smardon, R. C., S. R.J. Sheppard and S. New-
man. 1981. Prototype Visual hnpact Assess-
ment Manual. Syracuse: SUNY College of
Environmental Science and Forestry.
--,J. F. Palmer, A. Knopf, K. Grinde,J. E.
Henderson and L. D. Peyman-Dove.
1988. Visual Resources Assessment Procedure
for US Army Corps of Engineers. Instruction
Report EL-88-1. Vicksburg, Mississippi:
U.S. Army Engineer Waterways Experi-
ment Station.
Turner, M. G., and R. H. Gardner (eds.). 1991.
Quantitative methods in landscape ecology: the
analysis and interpretation of landscape het-
erogeneity. New York: Springer-Verlag.
U.S. Department of Agriculture, Forest Ser-
vice. 1973. National forest management. Vol.
1. Agriculture Handbook No. 434.
Washington, D. C.: U.S. Government
Printing Office.
--. 1995. Landscape Aesthetics: A Handbook for
Scenery Management. Agriculture Hand-
book No. 701. Washington, D. C.: USDA
Forest Service.
U.S. Department of Interior, Bureau of Land
Management. 1980. VisualResource
Management Program. Washington, D. C.:
U.S. Government Printing Office.
Zube, E. H., D. G. Pitt and T. W. Anderson.
1974. Perception and Measurement of Scenic
Resources in the Southern Connecticut River
Valley. Pub. No. R-74-1. Amherst: Insti-
tute for Man and His Environment,
University of Massachusetts.
,J. L. Sell andJ. G. Taylor. 1982. "Land-
scape perception, research, application
and theory." Landscape Planning 9(1):
1-35.