Factor Analysis & Scale Revision

Summary

Personality evaluation is popularly used as it is important develop theories and test various
psychological methods. The article by Steven P. Reise & Niels G. Waller focuses on methodological
problems that emerge in the application if Exploratory Factor Analysis (EFA), particularly to the scale
revision and refinement process.

The 2 topics which has been addressed in this article are the Hierarchical Nature of
psychological constructs and the impetus of revising a scale. The objective of scales revision is to
improve the psychometric properties and validate individual differences. Validity in this article suggest
that a measure a) has item content and a corresponding factor structure that is representative of a
consistent with what is already known regarding a construct – b) has a factor structure that is reproduced
and generalizable across relevant populations – c) has a clearly interpretable and relatively precise
scaling of individuals along one or more common dimensions.

The Construct Hierarchy

There is an infinite number of psychological constructs that can be proposed and assessed. The
level of conceptual breadth distinction has implications for the predictive ability of factor – analytic –
based measures (Mershon & Gorsuch, 1988; Ozer & Reise, 1994). The construct hierarchy also has
profound ramifications for the application of EFA to scale revision.

Motivations for Scale Revision

A primary reason for scale revision is that the scale’s psychometric properties are deemed
lacking. The second reason for scale revision is research may demonstrate that an existing measure has
a factor structure that is not generalizable across different samples or has a factor structure that is “not
as advertise” by the original authors. The third reason why the scales are revised is due to inadequate
construct representation (West & Finch, 1997). They are two distinct problems why an existing measure
can isolate. The author suggest that it may recognize that a scale is missing an important aspect of a
construct. The second problem is that research may demonstrate that a measure is not discovering a
researcher’s intended dimension.

Selection of Variables (Items)

It is beneficial for the researcher to plan and anticipate the final factor and solutions will look
like (Comrey, 1978). For constructing new item pools, the authors offer suggestion to 1) the new item
pool should be substantively over inclusive (Loevinger, 1957). 2) For each construct researchers might
consider writing multiple sets of content homogenous items that tap different aspects of the target
construct. The creation of an over-inclusive item pool with items tapping different aspects of a construct
will achieve two objectives. First, subsequent factor analyses will be better able to provide empirical
evidence of what type of item content belongs in the construct and what belongs somewhere else
(Tellegen & Waller, in press). An over-inclusive item pool with multiple items representing different
aspects of constructs also achieves second goal, which is the creation of facet scales, item parcels (Catell
& Burdsal, 1975, Kishton & Widaman, 1994), homogenous item composites (Hogan, 1983) or factoral
homogenous item dimensions (FHID; Comrey, 1984, 1988).

Facets are often conceptualized as being at the lower end of the hierarchy. Researchers who are
primarily interested in higher order constructs should not neglect the use of facets because facets can
serve as the building blocks from which higher order dimensions may emerge in EFA. It is well known
that dichotomously scored items cannot meet these conditions. Furthermore, the use of dichotomous
item response formats can cause serious distortions in the correlation matrix. Even if two items measure
the same construct, the phi coefficient may be low if the response proportion differ markedly (Lord &
Novick, 1968). Because EFA is often based on correlations, any distortions in the correlations can result
in misleading EFA findings (Comrey, 1978). With these circumstances taken into considerations, many
authors suggest the test developers create multipoint rating scales. Two potential advantages to
multipoint ratings are better psychometric properties of the resulting scale and the avoidance of
problems that are inherent in the factor analysis of dichotomous items (Comrey, 1988; Comrey &
Montag, 1982).

Selection of the Sample Respondents.

Two issues that requires attention when it comes to considering sample characteristics of a
factor analytic study. The first issue concerns sample size. Numerous rules of thumb for the minimum
sample size needed to obtain a robust factor solution are offered in research articles and textbooks
(Goldberg & Digman, 1994). Second issue is sample heterogeneity. In terms of identifying replicable
factors, researchers should assemble samples with sufficient examinee representation at all levels of the
trait dimensions. This means, there should be many examinees trait at all levels in order to accurately
estimate the population item correlations.

Factor Extraction.

The goal to factor extraction is to identify the number of the latent dimensions (factors) needed
to accurately account for the common variance among the item. The issue of how many factors to extract
and retain the rotation has been a source of contention for years. If two few factors are extracted, a
researcher may miss important distinctions among the items and the subsequently rotated solution may
be distorted in non-systematic ways (see Comrey, 1978). However, if too many dimensions are retained,
some rotated factors may be ill defined with only one or two salient loadings. However, studies (Fava
& Velicer, 1992a; Wood et al., 1996) that have empirically compared the effects of under and over-
extraction on the factor recovery of known population structures generally agree that it is preferable to
extract too many factors rather than too few.

Rotation of Factors.

The rotation algorithms that are most often used in psychology (e.g. Varimax, Promax, and
Oblimin) attempt to orient the factors to maximize criterion known as simple structure (Thurstone,
1947). Rotated simple structure solutions are often easy to interpret, whereas the originally extracted
(unrotated) factors are often difficult to interpret. Simple structure rotations such as Varimax, however,
are not guaranteed to find the most psychologically defensible placement of factors. There are at least
five reasons to consider oblique rotations. First, if the different factors are postulated to be aspects of a
more general construct then lower order factors can themselves be factored to obtain a higher order
general factor. Second, oblique rotations will always meet the simple structure criterion better than
orthogonal rotations. Third, some research supports a slight superiority of oblique rotations in terms of
factor replicability (Dielman, Catell & Wagner, 1972). Fourth, it might be unreasonable to assume that
any set of psychological variables are truly uncorrelated and thus oblique rotations may represent a
more realistic modeling of psychological phenomena. Lastly, the use of an oblique rotation means that
the correlations among the factors will also be estimated rather than fixed to zero as in orthogonal
rotation. In turn, the factor correlations may be providing substantively valuable information.

Evaluating the Rotation.

Smith and McCarthy (1995) provided a summary of fundamental psychometric criteria that
should be considered at all stages of the scale revision process. These criteria include (a) recognizing a
scale’s hierarchical structure, (b) establishing internal consistency reliability when appropriate, (c)
testing of content homogeneity of the facets and ensuring that different aspects of the construct are
equally represented in a scale, (d) ensuring that the items discriminate between respondents at the
appropriate level of trait intensity and (e) replication of factor structure across independent samples.

Goldberg and Digman (1994) best represent a prevailing view in the literature. If the data are
well structured, it makes no difference whether a factor or a components analysis is used.
 Factor Analysis and Scale Revision

The article, “Factor Analysis and Scale Revision”, by Comrey L. Andrew et.al discussed on
several issues related to the methodological approaches in the application of the explanatory factor
analysis (EFA) to scale revision and refinement. First, the author stated that there are two main topics
which influencing the decision for revising the scale which are the hierarchal nature of physiological
constructs and the motivations. Specifically, the author himself seeks to addresses issues that arise prior
to data collection, technical aspects of factor analysis, and procedures used to evaluate the outcome of
the scale revision. Furthermore, the author also conclude additional facts to relate on the findings of the
Factor Analysis and Scale Revision by highlighting two additional topics which are dimensionality and
scale score interpretation, and principal components versus factor analysis. These additional facts may
provide some useful evidence of comparison in addressing the methodological issues for the
Explanatory Factor Analysis (EFA) for the individual who are seeks to get more understanding on the
discussed journal herein.

The authors has stated their intentions clearly that the issues need to be addressed is simply
based on the topic discussed here that are primarily concerned with EFA, and they do not provide a
thorough consideration of alternative multivariate models that may be helpful during scale revision,
such as principal-components analysis or multidimensional scaling. Secondly, they do emphasizing on
the use of confirmatory factor analysis (CFA) in scale revision only sparingly because more extended
treatments of this topic are available elsewhere (see Finch & West, 1997, or Floyd & Widaman, 1995).
They author do have their own point of views and assumptions that any measurement use for the
constructive approach are solely for a factor analytic representation.

The purpose of carrying out scale revision had been stated clearly by the authors and what are
the factors that may influences on the scale revision itself. Here, the purpose is stated clearly that is to
improve the psychometric properties and therefore directly improve the validity—of individual-
differences measures. The authors stressed on term validity for the reader to imply that a measure should
have following criteria which has item content and a corresponding factor structure that is representative
of and consistent with what is currently known regarding a construct. Next it must has a factor structure
that is replicable and generalizable across relevant populations, and also has a clearly interpretable and
relatively precise scaling of individuals along one or more common dimensions. Therefore, the author
has diligently highlighted three objectives for measurements of validity clearly here and at the same
time do acknowledged of being overreliance on literature review and past studies been done related to
the topics especially on the default options been offer here. For example, the author stated that beyond
common observed design problems such as the use of idiosyncratic or small samples, the use of mixed
populations (e.g., men and women, psychotic and nonpsychotics) in a single analysis, and one-shot
scale revision or validity studies has been influencing on poorly application of the Factor Analysis itself.

The authors did proposed the issue on the construction hierarchy and the reason why they
choose five-factor model as their main tools to determine best methodological approach for the factor
analysis to the scale revision and refinement. He did suggest that the five-factor model not only because
it can revealed truth or the best map of the normal range personality domain but rather because it so
clearly makes explicit the situation researchers face when proposing new constructs and trying to
develop factor-analytic-based measures of them. Author did correlating with past studies had been done
to understand the concepts of construction hierarchy here. Specifically, psychological constructs have
a hierarchical structure such that different constructs have different levels of conceptual breadth
(Comrey, 1988; Guilford, 1975; John, 1990; West & Finch, 1997). Therefore, It enable the researcher
to select definite choices regarding the type of items that are written, how many items of each type are
included on the revised scale, how many factors will be extracted, how those factors will be rotated,
and ultimately how the resulting factor solution will be judged and interpreted.

Motivation for scale revisions

The purpose of selecting the scale revision over scale psychometrics properties is also one of
the main issue also been emphasized by the authors themselves as their ideal choice in forming a
methodological approach to be applied on the factor analysis. The authors clearly stand the benefits of
the scale revision for different motivational reasons due to the psychometric properties are deemed
inadequate. For example, a scale may not provide scores that have an appropriate degree of internal
consistency reliability. Next, the author stressed on research may demonstrate that an existing measure
has a factor structure that is not generalizable across different samples (e.g., men and women) or has a
factor structure that is "not as advertised" by the original authors (e.g., a purported three-dimensional
measure is really four dimensional). Lastly it is due to the inadequate construct representation in scale
psychometrics properties itself.

Next, the authors did provide good guidelines principle shown in the journal here, emphasizing
on how to make proper planning stage on carrying out the scale revision. The journal had been indicated
clearly on what items need to be consider for the scale revisions as writing a new items is highly
influenced by the particular motivations underlying the revision. First, the author stated that a clearly
articulated plan should be develop regarding to the need for the revised instrument. There are few
questions been raised by the authors to accommodate what need to be consider in order to carry out
proper plan such as the constructivism of the scale itself in term of measurement, the construction
methods, the stage at which the construct hierarchy being measured and lastly the differences in
comparison between the measurement from others. Secondly, a systematic series of studies by using
large samples of respondents should conducted by the scale researcher. The principles had been
supported clearly with past research statements at which Factor-analytic-based scale revision should be
an iterative process where data inform construct definition and refinement.

Selection of Variables (Items)

Based on the supporting findings, the authors had pointed out clearly that the selection of items
should be guided by theory and the findings from past research as there are abundance of items that are
remain unclears and unclassified. The explanations for the selection of variables are more stated in
terms of guidance on what and how the researchers needs to do in constructing new items pool which
is very essential. Firstly, the new item pool should be substantively over inclusive (Loevinger, 1957).
Secondly, for each construct researchers might consider writing multiple sets of content homogenous
items that tap different aspects of the target construct. Furthermore, the chronological selection of
items had been discussed here clearly together with past literature reviews, limitations, data
expectations based on the selections, preventions and measurements need to be done in order to get
more refining categorized items.

Selection of the sample of respondents

The authors had brought up on particular attentions to the reader that there are several issues
need to be concerned when considering sample characteristics which are the sample size and sample
heterogeneity. Sample which including numerous rule of thumb for the minimum sample size is much
needed in order to obtain a robust factor solution are offered in research articles and textbooks.
Nevertheless on the past finding stated in this journal, the author did not able to prove the reader on the
consistency of the sample sizes itself and confusion may keep arising from this point. Another issues
been discussed here is sample heterogeneity. The author did mentioned on the importance of assembling
samples with sufficient examinee representation at all levels of the trait dimensions when comes to the
part of identifying replicable factors. However, the authors did provide a reminder about the limitation
on using this rule which been influenced by group category mostly on undergraduate at which using the
standard pool of undergraduates may be suitable when undergraduates manifest sufficient heterogeneity
with respect to trait standing. The approach is not stated clearly for any other reasonable methods such
as uni-polar depression or psychotic ideation and is not suitable for the undergraduates to accurately
map the factor space of clinical assessment scales.

Factor extraction

The author stated clearly on the purpose of the factor extraction in the application of factor
analysis itself which is to identify the number of latent dimensions (factors) needed to accurately
account for the common variance among the items. However, the issue for the factor extraction remains
unclear as the past research remain in contention over the years in terms of the validity itself. The author
did provide some clues based on the past findings study that if too few factors are extracted, a researcher
may miss important distinctions among the items, and the subsequently rotated solution may be
distorted in nonsystematic ways (see Convey, 1978). However, if too many dimensions are retained,
some rotated factors may be ill defined with only one or two salient loadings. Even though with the
uncertainty of methods itself, the journal had indicated few techniques us to extracted the factors for
the researcher own preferences and suitability. For example, some researchers prefer to extract as many
factors as necessary for the factors to account for a certain percentage of total (e.g., 50%; Streiner, 1994)
or common (e.g., 80%; Floyd & Widaman, 1995) variance. The explanation of the self-residual analysis
may provide researcher with some clearer ideas on the method of factor of extraction itself. The relevant
purpose provided of the method also stated here in regards of determining the number of factors is to
compute the difference between the elements in the original correlation matrix and the model
reproduced correlation matrix given a certain number of extracted factors.

Rotation of Factors

A comprehensive elaboration had been stated here after the initial extraction to produces
orthogonal variables that are often not readily interpretable before. Hence, the researchers typically
rotate the factor pattern to a psychologically interpretable position. One of the method is the simple
structure which is a tool of algorithms that are most often used in psychology (e.g., Varimax, Promax,
and Oblimin) attempt to orient the factors to maximize a criterion known as (Thurstone, 1947). The
author did explain on the beneficiaries of each methods and its limitation been discussed fairly. In an
orthogonal rotation, such as Varimax, the factors are not allowed to correlate in contrast to the oblique
rotations, such as Promax or Oblimin at which the factors are allowed to correlate. Supporting
literature reviews had been discussed in this journal where many researchers prefer orthogonal
rotations because of the simplicity of interpretation. There are at least five additional reasons to
consider oblique rotations as emphasized by the authors. First, if the different factors are postulated to
be aspects of a more general construct, then lower order factors can themselves be factored to obtain a
higher order general factor. Second, oblique rotations will always meet the simple structure criterion
better than orthogonal rotations. Third, some research supports a slight superiority of oblique rotations
in terms of factor replicability (Dielman, Cattell, & Wagner, 1972; Gorsuch, 1970). Fourth, it might be
unreasonable to assume that any set of psychological variables are truly uncorrelated, and thus oblique
rotations may represent a more realistic modeling of psychological phenomena (see Loo, 1979). Finally,
the use of an oblique rotation means that the correlations among the factors will also be estimated rather
than fixed to zero as in an orthogonal rotation. The authors did seek addressed few issues may arise on
these outcomes that none of the simple structure rotation methods will be able to identify an
interpretable simple structure if many variables are complex (i.e., tap into more than a single trait. There
are several fair advices had been taken to ensure researchers make every effort to design instruments
where each variable will load highly on a single dimension. For example, Comrey (1978) advised,
"Ideally, a variable for a factor analytic study should measure one and only one factor in the domain to
any substantial degree" (p. 650). Besides that, few inputs of recommendation have been implied for the
researchers to ensure routinely inspect factor plots for obvious departures from simple structure. Each
of the implication using different method have been discussed here as not depends solely on the integrity
of the methods itself.
Reference.

Cattell, R. B., & Burdsal, C. A. (1975). The radial parcel double factoring design: A solution to the
item-vs-parcel controversy. Multivariate Behavioral Research, 10, 165-179.

Comrey, A. L. (1978). Common methodological problems in factor analytic studies. Journal of

Consulting and Clinical Psychology, 46, 648-659.


Comrey, A. L. (1988). Factor-analytic methods of scale development in personality and clinical

psychology. Journal of Consulting and Clinical Psychology, 56, 754-761.

Comrey, A. L., & Montag, I. (1982). Comparison of factor analytic results with two-choice and
seven-choice personality item formats. Applied Psychological Measurement, 6, 285-289.


Dielman, T. E., Cattell, R. B., & Wagner, A. (1972). Evidence on the simple structure and
factor invariance achieved by five rotational methods on four types of data. Multivariate
Behavioral Research, 7, 223- 242.

Fava, J. L., & Velicer, W. F. (1992a). The effects of overextraction on factor and component
analysis. Multivariate Behavioral Research, 27, 387-415

Goldberg, L. W., & Digman, J. M. (1994). Revealing structure in the data: Principles of exploratory
factor analysis. In S. Strack & M. Lorr (Eds.), Differentiating normal and abnormal
personality (pp. 216—242). New York: Springer.

Hogan, R. T. (1983). A socioanalytic theory of personality. In M. Page (Ed.), 7982 Nebraska

Symposium on Motivation (pp. 55-89). Lincoln: University of Nebraska Press.


Kishton, J. M., & Widaman, K. F. (1994). Unidimensional versus domain representative parceling of
questionnaire items: An empirical example. Educational and Psychological Measurement, 54,
757-765.

Loevinger, J. (1957). Objective tests as instruments of psychological theory. Psychological

Reports, 3, 635-694.


Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA:
Addison-Wesley.

Mershon, B., & Gorsuch, R. L. (1988). Number of factors in the personality sphere: Does increase in
factors increase predictability of real-life criteria? Journal of Personality and Social
Psychology, 55, 675-680.

Ozer, D. J., & Reise, S. P. (1994). Personality assessment. Annual Review of Psychology, 45,
357-388.


Smith, G. T., & McCarthy, D. M. (1995). Methodological considerations in the refinement of

clinical assessment instruments. Psychological Assessment, 7, 300-308.


Tellegen, A., & Waller, N. G. (in press). Exploring personality through test construction:
Development of the Multidimensional Personality Questionnaire. In S. R. Briggs & J. M.
 Cheek (Eds.), Personality measures: Development and evaluation (Vol. 1). Greenwich,
CT: JAI Press.

Thurstone, L. L. (1947). Multiple factor analysis. Chicago: University of Chicago Press.

West, S. G., & Finch, J. F. (1997). Personality measurement: Reliability and validity issues. In R.
Hogan, J. Johnson, & S. Briggs, Handbook of personality psychology (pp. 143-164). San
Diego, CA: Academic Press.

Wood, J. M., Tataryn, D. J., & Gorsuch, R. L. (1996). Effects of under and overextraction on
principal axis factor analysis with varimax rotation. Psychological Methods, 1, 354-365.