RESEARCH AND SURVEY STATISTICS – STA3022F

SOLUTION TO TUTORIAL #3
Week 4 2008

ITEM RELIABILITY ANALYSIS

REPORT TO RESEARCHER – Reliability of constructs underlying attitudes towards women in

senior managerial positions

Aims:
1. To assess whether the sets of questions designed to test the three constructs underlying attitudes towards
women in senior management positions are internally consistent (i.e. are reliable indicators of the
underlying construct)
2. To assess whether any of the questions should be deleted in order to increase the reliability of the
scales.

Methodology:
Initially, an exploratory factor analysis of all questions was performed to identify, without imposing any
structure on the data, which questions loaded together on a factor. If responses to the researcher’s questions
are consistent with the hypothesised constructs, one would expect 3 factors to be present, with questions
loading on factors in accordance with the specified relationships (i.e. one factor comprising A4, A5, A8,
A10; another factor comprising A1, A2, A3; and another factor comprising A6, A7, A9)

Then, an item-reliability analysis was conducted on each of the three underlying constructs. For each of the
constructs, the overall Cronbach alpha and the average item-total correlation was examined to assess the
overall construct reliability. Then, for each of the questions within a particular scale, the Alpha if deleted
and item-total correlations were examined to assess whether any particular questions should be deleted in
order to increase the reliability of the measurement scale.

Results:
The initial factor analysis shows three significant factors, in agreement with the researcher’s expectation.
The factor loadings show some (but not total) consistency with the hypothesised construct structure, with
the following factors emerging:

Factor 1: A1, A2, A3 (Hypothesised: Decision making skills)

Factor 2: A5, A8, A10 (Hypothesised: Interpersonal skills)
Factor 3: A6, A9 (Hypothesised: Conflict resolution skills)

Only the decision making skills shows total consistency with the data. Question A4 and A7 do not load on
any one factor in particular. Question A4 loads most strongly on factor 2, in accordance with the
hypothesised structure, but the loading is fairly weak at 0.56. Question A7 is even more unclear, and loads
on factor 1 and factor 2 with roughly equal strength.

Construct 1: Decision making skills (TABLE 3)

Overall Cronbach alpha = 0.6286

Average item-total correlation = 0.3744

Both the overall Cronbach alpha of 0.63 and the average item-total correlation of 0.37 suggest that the
overall internal consistency of the 3 questions designed to measure decision making skills is moderate but
not high. Although the overall reliability fails the traditional cut-off of 0.7, the difference is fairly small, and
the questions could probably be considered to adequately represent the underlying construct. From table 3,
there are no improvements in reliability that would result from deleting any of the questions measuring the
decision making skills construct.

1
Construct 2: Interpersonal skills (TABLE 4)

Overall Cronbach alpha = 0.65067

Average item-total correlation = 0.3378

The interpretation is very similar to the previous construct. Both the overall Cronbach alpha of 0.65 and the
average item-total correlation of 0.33 suggest that the overall internal consistency of the 3 questions
designed to measure interpersonal skills is moderate but not high. Although the overall reliability fails the
traditional cut-off of 0.7, the difference is fairly small, and the questions could probably be considered to
adequately represent the underlying construct. From table 4, there are no improvements in reliability that
would result from deleting any of the questions measuring the interpersonal skills construct.

Construct 3: Conflict resolution skills (TABLE 5)

Overall Cronbach alpha = 0.210124

Average item-total correlation = 0.0635

The overall Cronbach alpha of 0.21 and the average item-total correlation of 0.06 suggest extremely low
internal consistency between the 3 questions designed to measure conflict resolution skills. Although table 5
shows that there is a substantial improvement in reliability that would result from deleting question A7 from
the interpersonal skills construct, the overall reliability of the remaining two questions (A6 and A9) would
still only be 0.40. Thus the questions designed by the researcher to measure conflict resolution skills appear
not to be measuring the same underlying construct.

Conclusions and recommendations:

The aim of the analysis was to identify whether the questions in the pilot questionnaire adequately captured
the three hypothesised constructs underlying attitudes towards women in senior management positions. The
decision making skills and interpersonal skills constructs can be considered to be reliably measured by the
sets of questions hypothesised by the researcher. However, the conflict resolution skills construct is
extremely unreliable, with very poor internal consistency. Removing question A7 substantially improves the
reliability, and is recommended, but even then the reliability remains low. The research should consider
changing the questions designed to measure conflict resolution skills before proceeding with the main study,
and possibly rerun the pilot study to ensure that the new questions are more reliable. If the same questions
are used, any results based on the conflict resolution skills construct should be interpreted with caution.

2
QUESTION 2

Q2.1
Construct 1:
– Composed of Q1, Q2, Q3, Q4, Q5
– Results: alpha = 0.80, avg inter-item correlation = 0.45
– Assessment: Reliable scale; could remove Q5 to increase the alpha to 0.82, but
improvement is only marginal so might retain Q5 anyway (either outcome is fine but must
have mentioned possibility of removal)

Construct 2:
– Composed of Q6, Q7, Q8
– Results: alpha = 0.84, avg inter-item correlation = 0.64
– Assessment: Reliable scale; don’t remove any questions

Construct 3:
– Composed of Q9, Q10
– Results: alpha = 0.76, avg inter-item correlation = 0.52
– Assessment: Reliable scale; don’t remove any questions

To explain apparent differences between factor and item reliability analysis results: mention that
item-reliability analysis only looks at the correlations between indicated questions, and not at
correlations between those indicated questions and any questions outside the indicated construct
(Key idea: correlations between questions inside/outside a user-specified construct).

3
QUESTION 3

Q3.1
For two attributes,
var[Q1 + Q2] = var[Q1] + var[Q2] + 2 cov[Q1;Q2]

Thus rewrite alpha as

k 2 cov(Q1, Q 2)
α=
( k − 1) var(Q1 + Q 2)

Now:
• Variances are always non-negative, so for Q1 and Q2 positively correlated, Cronbach alpha must be
greater than 0 and increase with increasing covariance relative to variance (stronger association)
• Cronbach alpha could be negative if items were negatively correlated

Q3.2
var[Q1 + Q2 + Q3] = 5.452 = 29.7
var[Q1] + var[Q2] + var[Q3] = 5.9 + 5.1 + 6.3 = 17.3
k=3
Alpha = (3/2)(29.7-17.3)/29.3 = 0.68

Items are not internally-consistent according as the alpha is below the rule-of-thumb cutoff of 0.70.
However, they are only very slightly below the cutoff and their reliability is thus not too bad.

Q3.3
Recommend removing item Q3. By doing so, the internal-consistency reliability rises greatly from 0.68 to
0.95 which (1) pushes the “unreliable” construct over the 0.70 benchmark, and (2) is a substantial increase
in reliability (0.95 – 0.68 = 0.27)