Creating Valid and Effective

Measures
Using optifact- to Create Better
Summated Rating Scales
North American Stata Users' Group Meetings
Boston, July 11-12, 2005

Are measures important?

Measures are how theory is translated into

data for modeling

Unlike art, where we want to engage
individual interpretation as a major
component of the work
Measures need to be unambiguously
interpreted in precisely the same way by all

Why Scales?
Combining the results of several questions

is more reliable and precise, and reduces

measurement error (Spector, 1992)

What makes a good scale?

It is reliable
The same measurement, produces similar results across time
and persons
The questions or items that make up the scale are consistent
with one another

It is valid
It measures the concept that it is intended to measure

It measures one concept

It is regularly distributed

Reliability
Internal Reliability
Internal consistency: the items should intercorrelate (Cronbachs Alpha)

Test-Retest Reliability
Same measure taken again should produce
consistent results

Validity
Rationale
The items must be consistent with theory and the
concept under consideration

Criterion Validity
The items should vary in the same way to other
variables that are theoretically related to the
concept
Canonical correlation

Uni-dimensional, Efficient
Scales that measure a single concept are

more useful in establishing causal

relationships
Efficient scales use only enough items to
reliably measure a concept
Regular distributions allow standard
modeling techniques

Research Question
If my scales are not uni-dimensional, are

there sub-scales which are and meet the

criteria associated with good scales?
Investigated using data gathered for the
Canadian National Longitudinal Survey of
Children and Youth by Statistics Canada.

Data and Method

The NLSCY cycles 1, 2 and 3
Measures on parenting and child behaviour taken
every two years (allows the investigations of testretest reliability) N > 13,000.

Common factor analysis

Dimensions, alpha

Canonical correlation
Criterion validity

Problems
There are a lot of candidate scales
2k 1, if scales of all sizes are considered

The testing procedure is lengthy

Factor analysis (dimensionality)
Alpha (internal reliability)
Canonical Correlation (external validity)
Re-test (all of the above, twice)
More than 13 scales

Solution: optifact
A Stata software component to analyze a

list of candidate items for the creation of a

summated rating scale

-optifact- Specification
optifact varlist [weight] [if exp] [in range]
[, top(#) smallest(#) largest(#) maxfact(#) minslope(#)]

Varlist: the list of candidate items (numeric)

Options:
Top: the number of candidate scales to list (10)
Smallest: the smallest k allowed (3)
Largest: the smallest k allowed (all items)
Maxfact: the maximum number of factors allowed (1)

-optifact- Output
optifact abecq6b abecq6n abecq6w abecq6p abecq6i abecq6qq abecq6hh abecq6s
[aweight=awtcw01c], top(20) criteria(childsex ammcq01)
8 items will be processed
There are 219 potential scales
56 combinations of
Top 5 scales using
Avg.
K Alpha Cov.
- ----- ----1 3 0.774 0.201
2 3 0.769 0.203
3 3 0.744 0.231
4 3 0.742 0.169
5 3 0.734 0.213

3 items had
3 items
Can
Chk Items
--- ----Yes abecq6p
Yes abecq6p
Yes abecq6b
Yes abecq6p
Yes abecq6n

one factor, 0 had more than one factor

55 combinations of
Top 5 scales using
Avg.
K Alpha Cov.
- ----- ----1 4 0.802 0.185
2 4 0.784 0.199
3 4 0.776 0.191
4 4 0.775 0.198
5 4 0.768 0.172

4 items had
4 items
Can
Chk Items
--- ----Yes abecq6p
Yes abecq6n
Yes abecq6n
Yes abecq6b
Yes abecq6p

one factor, 15 had more than one factor

abecq6i abecq6qq
abecq6i abecq6hh
abecq6n abecq6hh
abecq6qq abecq6hh
abecq6p abecq6i

abecq6i
abecq6p
abecq6p
abecq6p
abecq6i

abecq6qq abecq6hh
abecq6i abecq6hh
abecq6i abecq6qq
abecq6i abecq6hh
abecq6qq abecq6s

Behaviour Scales
Variable

Concepts

Alpha

Hyperactivity
(becs06)

0.84

Prosocial Behaviour
(becs07)

10

0.76

Emotional Disorder /
Anxiety (becs07)

0.78

Aggression
(becs09

0.78

Indirect Aggression
(becs10)

0.78

Property Offences
(becs11)

0.62

Parenting Scales
Variable

Concepts

Alpha

Positive interaction (0-23

months) (prcs01)

0.73

Ineffective parenting style

(0-23 months) (prcs02)

0.37

Positive interaction (2-11

years) (prcs03)

0.81

Ineffective parenting style

(2-11 years) (prcs04)

0.70

Consistency (2-11 years)

(prcs05)

0.68

Rational parenting style

(2-11 years) (prcs06)

0.62

PMK Depression
(dpps01)

12

0.82

Results
Of these 13 scales only one was uni-

dimensional
Most had larger number of items than
desirable for the value of alpha

New Behaviour Scales

Variable

Concepts

Alpha

Concepts

Alpha

Hyperactivity
(becs06)

0.84

0.80

Prosocial Behaviour
(becs07)

10

0.76

0.77

Emotional Disorder /
Anxiety (becs07)

0.78

0.62

Aggression
(becs09)

0.78

0.67

Indirect Aggression
(becs10)

0.78

0.70

Property Offences
(becs11)

0.62

0.63

New Parenting Scales

Variable

Concepts

Alpha

Concepts

Alpha

Positive interaction (0-23

months) (prcs01)

0.73

0.63

Ineffective parenting style

(0-23 months) (prcs02)

0.37

Positive interaction (2-11

years) (prcs03)

0.81

0.78

Ineffective parenting style

(2-11 years) (prcs04)

0.70

Consistency (2-11 years)

(prcs05)

0.68

0.66

Rational parenting style (2-11

years) (prcs06)

0.62

PMK Depression (dpps01)

12

0.82

0.75

Implications
These scales are part of a large survey

conducted every 2 years in Canada on

more than 20,000 children
The survey has completed 6 cycles of data
collection
Each question has been asked more than
120,000 times

Implications (Contd)
The scales required 85 questions
The revised scales require 30 questions
More than 55,000,000 questions

Caveats
Most scales in the NLSCY are not the best

that can be made from the available data

Some of these scales should not be used
The NLSCY is an excellent survey,
conducted by competent statisticians
Similar results might be expected in other
major surveys

Conclusions
-optifact- can help find better measures
Uni-dimensional
Valid
Reliable

-optifact- can reduce costs

Equivalent or better measure for same money