Journal of Applied Psychology

1977, Vol. 62, No. 2, 184-188

An Intraorganizational Analysis of the Roberts and O'Reilly

Organizational Communication Questionnaire
Paul M. Muchinsky
Iowa State University
This study assesses the reliability and validity of a questionnaire developed
by Roberts and O'Reilly to measure organizational communication. The ques-
tionnaire was administered to two samples of employees, both in the home
office (« = 270) and branch offices (n = 404) of a large public utility. The
questionnaire items were factor analyzed for each sample. The results re-
vealed a high degree of similarity between the samples in terms of scale
reliabilities and factor structure, Similarities and differences between the find-
ings in -this study and those reported by Roberts and O'Reilly are discussed.

Roberts and O'Reilly (1974b) commented
that although organizational communication
has been the subject of considerable research,
"there has been no systematic development
of instrumentation to measure communica-
tion variables in organizations" (p. 321).
Roberts and O'Reilly then reported their
progress in developing a "standardized ques-
tionnaire that can be used to compare com-
munication across organizations" (p. 321).
The present study is an assessment of
the reliability and validity of the Roberts
and O'Reilly Organizational Communication
Questionnaire.
The questionnaire consists of 36 items mea-
suring 16 dimensions of organizational com-
munication. Eight of the dimensions consist
of multi-item scales scored on a 7-point
Likert format: Trust (3 items); Influence
(3 items); Mobility (2 items); Desire for
Interaction (3 items); Accuracy (3 items);
Summarization (3 items); Gatekeeping (3
items); and Overload (2 items). 1 Three of
the dimensions consist of multi-item scales
requiring the respondent to indicate per-
centages of time; Directionality—Upward (3
items); Directionality—Downward (3 items);
and Directionality—Lateral (3 items). Four
of the dimensions are single-item scales re-
quiring the respondent to indicate the fre-
quency (percentage of time) various modes
of communication are used: Written, Face-
to-Face, Telephone, and Other. The last di-
mension consists of a single-item scale scored
on a 7-point Likert format indicating the
degree of satisfaction with communication in
the organization. This dimension is a varia-
tion of the General Motors faces scale
(Kunin, 19SS).
Three of the dimensions (trust in superior,
perceived influence of the superior, and mo-
bility aspirations of the respondent) are
considered noncommunication variables by
Roberts and O'Reilly, but they were included
in the questionnaire because they have been
repeatedly shown to influence individual com-
munication in organizations (e.g., Cohen,
1965; O'Reilly & Roberts, 1974; Read, 1962;
Roberts & O'Reilly, 1974a).
Roberts and O'Reilly arrived at their ques-
tionnaire by successively reducing the number
of items from a much larger pool through
cluster-analytic techniques. In developing
their instrument, Roberts and O'Reilly tested
the questionnaire on samples of mental health
workers, military personnel, bank employees,
hospital personnel, and college students.
Based upon their results, Roberts and O'Reilly

1
This study was made possible by a grant from The version of the questionnaire reported by
the Graduate College of Iowa State University. Roberts and O'Reilly (1974b) consisted of 35 items,
Requests for reprints should be sent to Paul M. with only 1 item on information overload. The
Muchinsky, Department of Psychology, Old Botany questionnaire used in the present study contained
Hall, Iowa State University, Ames, Iowa S0011. 36 items, with 2 items composing the Overload scale.
184
 QUESTIONNAIRE ANALYSIS 185

concluded that their instrument was useful Table 1

to discriminate among organizations. Because Coefficient Alpha Reliability for the 11
of the potential utility of a psychometrically Multi-Item Scales
sound instrument in measuring organizational
Sample
communication, the present study was under-
taken to assess the reliability and validity of Home Branch
office offices
the questionnaire based upon a large sample Scale R&O (n=270) (»=4<H)
of employees holding very diverse positions Trust .68 .84 .82
within a major organization. It was assumed Influence .69 .67 .70
Mobility .82 .93 .93
that an analysis of the instrument based upon Desire for Interaction .68 .62 .62
such a sample would probably offer a more Directionality —
Upward .65 .64 .54
rigorous test of the organizational communi- Directionality —
cation questionnaire than would analyses Downward .84 .87 .83
Directionality —
based on small and homogeneous samples Lateral .71 .76 .76
(e.g., college students and managers). Accuracy .62 .61 .46
Summarization .73 .82 .74
Gatekeeping .53 .40 .51
Overload — .64 .70

Method
Sample
The sample consisted of employees of a large
public utility. Respondents covered a broad spec-
trum of occupations, including various levels of
management, telephone operators, telephone service
repairmen, PBX installers, technical, craft, and cler-
ical personnel. The total size of the organization
(state-wide) was approximately 8,OOC employees. A
random sample of 1,160 was drawn by selecting
every seventh employee from a computerized listing
of all employees. The Roberts and O'Reilly Orga-
nizational Communication Questionnaire (among
others) was mailed to the home of each employee.
Subjects were instructed that the study was a uni-
versity study and not a company study, and that
their responses were completely confidential. After
one follow-up letter, 674 usable questionnaires were
returned. For cross-validation purposes, the sample
was split into two groups, those who worked in the
company's home office (n = 210) and those who
worked in branch offices (n — 404).
Statistical Analyses
The reliability of the a priori multi-item scales
for both groups of employees was determined by
computing Cronbach's coefficient alpha. Separate
factor analyses were performed on the correlation
matrix of the 36 questionnaire items for both
samples. A principal factor analysis was employed
that initially uses squared multiple correlation values
as estimates of communality and then iterates until
the communalities are stable. Orthogonal rotations
(utilizing varimax method) were completed, rotating
from two to nine factors on the data from each
sample. After careful examination of the resultant
factor structures, the factor structure that best
represented the data from the questionnaire for
each sample was selected.
Note. R & O = seven samples from the Roberts and O'Reilly
(1974b) questionnaire.
8
Median value reported across seven samples; total N = 1,218.
Results
Roberts and O'Reilly computed the inter-
nal consistency reliability (coefficient alpha)
for the multi-item scales for each of seven
samples. Table 1 shows the median reliabil-
ity coefficients from the seven samples in
their study, as well as the internal consist-
ency reliability of the multi-item scales from
both samples in the present study. The re-
sults of the two studies appear similar. Both
studies identified the scales Mobility and
Directionality—Downward as having the two
highest reliabilities, and both studies found
the scales Accuracy and Gatekeeping to have
the lowest reliabilities. Also within the present
study, the results from the two samples are
highly similar.
Table 2 shows the results from the factor
analysis of the 36-item questionnaire for the
home office and branch offices samples. The
factor structure that best represented the
data from the home office sample was the
eight-factor solution, whereas the seven-factor
solution was most representative of the data
from the branch offices sample. Factor load-
ings and item communalities are shown in
the table.
As can be seen in Table 2, the results from
the two samples are highly similar. The first
six factors from both samples are composed
of identical items. Factor 7 (Directionality—•
186 PAUL M. MUCHINSKY

Table 2
Factor Loadings and Item Communalities from Factor Analysis of the Roberts and O'Reilly
Organizational Communication Questionnaire for Home and Branch Office Samples
Item 1 2 3 4 5 6 7 8

Trust— 1 .73 (.67) .55 (.46)

Trust— 2 .55 (.65) .35 (.43)
Trust— 3 .80 (.79) .69 (.66)
Influence— 1 .75 (.75) .58 (.59)
Influence— 2 .48 (.62) ,62 (.48)
Influence— 3 .39 (.49) .56 (.39)
Accuracy — Superior .66 (.67) .47 (.50)
Desire for Interaction — Superior .46 (.49) .40 (.38)
Satisfaction with Communication .60 (.58) .37 (.40)
Mobility— 1 .91 (.88) .84 (.80)
Mobility— 2 .87 (.89) .79 (.83)
Interaction — Superior .53 (.39) .29 (.20)
Receiving — Superior .58 (.82) .60 (.70)
Sending— Superior .72 (.63) .57 (.44)
Face-to-Face Modality -.79 ( .87) .72 (.91)
Telephone Modality .83 (-.86) .74 (.81)
Overload — 1 .56 (.37) .34 (.20)
Overload — 2 .55 (.42) .34 (.24)
Summarization — Superior .76 (.67) .59 (.47)
Summarization —Subordinates .74 (.65) .64 (.50)
Summarization — Peers ,76 (.74) .59 (.42)
Interaction — Subordinates .82 ( ,82) .77 (.78)
Receiving — Subordinates .67 ( .76) .58 (.67)
Sending —Subordinates .75 ( .71) .76 (.60)
Desire for Interaction — Subordinates .33 ( .26) .38 (.44)
Gatekeeping — Subordinates .56 .46
Gatekeeping — Peers (-.35) (.20)
Interaction — Peers -.78 (-.72) .66 (.59)
Receiving — Peers -.49 (-.60) .59 (.71)
Sending — Peers -.73 (-.63) .66 (.64)
Desire for Interaction — Peers -.22 (-.32) .32 (.25)
Accuracy —Subordinates .52 .35
Accuracy — Peers .52 .31

ic, Factor
1'ciLi.ui i1 —
= Motivation
iviuLiviiviuii lu
to Interact
iiiLuiLtcL with Superior ; riti-iui f, ~ iviuuinty, i UULOI o — Directionality—Upward; Factor 4
\viLii ou]jenui
Communication Modalities; Factor 5 = Overload; Factor 6 = Summarization; Factor 7 = Directionality—Downward/

factor in the branch offices sample.

Downward/Lateral) is nearly identical for quite supportive of the research reported by

both groups, with the exception that the
item Datekeeping'—Subordinates is included
in the home office sample, whereas the item
Gatekeeping—Peers is included in the branch
offices sample. The major difference between
the two factor solutions is that an eighth
factor (Accuracy) emerged from the home
office sample, whereas no such comparable fac-
tor emerged from the branch offices sample.
Discussion
The results from the factor analyses re-
vealed mixed support for the Roberts and
O'Reilly Organizational Communication Ques-
tionnaire. Several of the derived factors
replicated the original scales (Mobility,
Directionality—Upward, Overload, and Sum-
marization). In general, these results are
Roberts and O'Reilly (1974b), However, two
other factors are broad both in terms of the
numbers of items comprising the factors and
the divergence of their content. Factor 1
(Motivation to Interact with Superior) is
composed of items from the original Trust
and Influence scales and a variety of other
items, all of which seem to reflect employee
desire or willingness to interact with supe-
riors. Because the factor includes so many
items from both the communication and non-
communication scales, it is heterogeneous in
conceptual content and does not confirm
Roberts and O'Reilly's findings.
Factor 7 (Directionality—Downward/Lat-
eral) is also a broad factor, composed of
items from four different a priori scales.
Although the Directionality—Upward scale
 QUESTIONNAIRE ANALYSIS
broke out as a distinct factor (Factor 3), the
lateral and downward directionalities merged
into a common dimension. However, some
caution must be exercised in interpreting this
factor, as the signs of the factor loadings
(Table 2) differ between the lateral and
downward directions. Nevertheless, the data
suggest that there are two dimensions to the
direction of employee communication: up-
ward (superior) and other (peers, subordi-
nates). This particular finding does not sup-
port a result reported by O'Reilly and
Roberts (1974), who found qualitative dif-
ferences between horizontal (peers) and ver-
tical (superior, subordinates) communica-
tion. Their study addressed specifically infor-
mation filtration, whereas the present study
addressed the broad patterns of day-to-day
communication. The dimensions of communi-
cation directionality may well be contingent
upon such factors as communication content
and purpose, a contention supported by the
research of Ackoff (19S7) and Eilon (1968).
The difference in findings between the
present study and the original investigation
by Roberts and O'Reilly (1974b)—specif-
ically involving Factors 1 and 7—might be
attributed to differences in analytic tech-
niques. The present study employed factor
analysis, whereas Roberts and O'Reilly em-
ployed cluster analysis (see Tryon & Bailey,
1970). Blashfield (1976) reported that the
Tryon and Bailey method of cluster analysis
is conceptually and operationally more like
factor analysis than other cluster-analytic
techniques. Thus although it is conceivable
that the difference in statistical analyses be-
tween the two studies may account for the
differences in findings, it must be kept in
rnind that (a) the similarities between the
two studies were greater than the differences
and (b) that the data do not suggest that
the original type of analysis performed by
Roberts and O'Reilly would produce results
other than what was presently obtained.
For the most part, the home office and
branch offices samples revealed highly similar
results in terms of scale reliabilities and fac-
tor strucures. On an empirical basis, the re-
sults manifest cross-validity. However, what
do the results suggest about the concept of
187
"organizational communication?" Farace and
MacDonald (1974) stipulated four units of
analysis for organizational communication:
individual, dyad, work group, and organiza-
tion. Missing from the Roberts and O'Reilly
questionnaire are items dealing with com-
munication between departments within an
organization, communication between the
branch offices and the home office of the same
organization, etc. In using the term organi-
zational communication, it seems that the
unit of analysis justifiably could be struc-
tural organization elements (e.g., depart-
ments or units) as well as, or in addition to,
individuals. Perhaps a more appropriate name
for the Roberts and O'Reilly questionnaire
would be "Individual Communication in
Organizations" rather than "Organizational
Communication."
An additional area of concern is the need
to include other variables in the question-
naire. Communication purpose and content
are examples of such variables. Roberts and
O'Reilly stated that these variables have
been frequently mentioned in organizational
literature but were discarded in the question-
naire development because of the difficulty
subjects had in responding to them. The
difficulty not withstanding, these items may
be vital to understanding organizational com-
munication. The potential importance of these
items is accentuated by the fact that Roberts
and O'Reilly stated the need to relate orga-
nizational communication to job performance
criteria as a step in developing organizational
communication models. It seems conceivable
that two employees may engage in equal
amounts of "upward communication," for ex-
ample, but for different reasons, One employee
may be highly effective serving in a consulta-
tive capacity, giving advice and service to his
or her supervisor, whereas the second em-
ployee may be highly ineffective, requiring
continual monitoring, guidance, and direction
by his or her supervisor. Although the
amount or frequency of upward communica-
tion (interacting, sending, and receiving in-
formation) may be the same for both em-
ployees, the communication would obviously
differ in terms of content or purpose. In such
a case, using the Roberts and O'Reilly com-
188 PAUL M. MUCHINSKY
munication questionnaire in its present form,
there would appear to be no relationship be-
tween upward communication and job per-
formance. It is this type of refinement that
is needed in the questionnaire before com-
munication can be meaningfully related to
other variables. In summary, although the
Roberts and O'Reilly questionnaire deals
more precisely with individual communication
in organizations than with organizational
communication, with some additional vari-
ables and continued fine tuning the instru-
ment should prove useful in the measurement
of a very elusive concept.
References
Ackoft', R. L. Towards a behavioral theory of com-
munication. Management Science, 1957, 4, 218-234.
Blashficld, R. K. Mixture model tests of cluster
analysis: Accuracy of four agglomerative hier-
archical methods. Psychological Bulletin, 1976, 83,
377-388.
Cohen, A. Upward communication in experimentally
created hierarchies. Sociological Review, 1965,
IS, 342-351.
Eilon, S. Taxonomy of communications. Adminis-
trative Science Quarterly, 1968, 13, 266-288.
Farace, R. V., & MacDonald, D. New directions in
the study of organizational communication. Per-
sonnel Psychology, 1974, 27, 1-19.
Kunin, R. The construction of a new type of atti-
tude measure. Personnel Psychology, 1955, 8,
65-78.
O'Reilly, C. A., Ill, & Roberts, K. H. Information
nitration in organizations: Three experiments.
Organizational Behavior and Human Performance,
1974, 11, 253-265.
Read, W. Upward communication in industrial
hierarchies. Human Relations, 1962, 15, 3-16.
Roberts, K. H., & O'Reilly, C. A., III. Failures in
upward communication in organizations: Three
possible culprits. Academy oj Management Jour-
nal, 1974, 17, 205-215. (a)
Roberts, K. H., & O'Reilly, C. A., III. Measuring
organizational communication. Journal of Applied
Psychology, 1974, 59, 321-326. (b)
Tryon, R. C., & Bailey, D. E. Cluster analysis.
New York: McGraw-Hill, 1970.
Received January 23, 1976 B