1

Critique of a Program Evaluation: Evaluation of a Canadian Brain and Spinal Cord Injury

Prevention Video

EPID 806

Sarah Kelly

10187241
2

Summary of Program Evaluation

The program being evaluated is an educational video aimed at preventing concussion and

spinal cord injury among hockey players age 11 and 12 (Cook et al., 2003). The video was

presented to the target population in the form of a one hour-long television program entitled

Smart Hockey (Cook et al., 2003). ThinkFirst Canada created the video, which featured famous

professional hockey players; team doctors, coaches, and trainers; as well as referees (Cook et al.,

2003). The video was comprised of personal statements, educational material, and important

medical guidance about concussions and spinal cord injuries (Cook et al., 2003). Information in

the video was presented via the reading of facts, the repetition of key points, and visual

demonstration through the portrayal of possible injury scenarios (Cook et al., 2003).

The type of evaluation conducted was an outcome evaluation. The outcome measures of

coach interviews and surveys, player questions, number and nature of penalties, and league

standings were all measured after exposure to the video occurred (Cook et al., 2003). These

outcome measures were used to indicate whether or not the goal of the program, to prevent

concussion and spinal cord injury, was being achieved. The study design was a randomized

controlled trial (RCT), which was community-based and clustered (Cook et al., 2003). An RCT

is classified as a true experimental design (Boonmak, 2020).

The study found that the teams did not perform significantly better or worse after having

seen the video (Cook et al., 2003). Knowledge about concussion symptoms and causes improved

in the group that was exposed to the video, and this knowledge was sustained after 3 months

(Cook et al., 2003). The group that was not exposed to the video saw no improvement in this

area of knowledge (Cook et al., 2003). Total penalties remained the same for teams before and

after viewing the video, although one type of penalty, presented as dangerous in the video,
3

decreased in players who were exposed to the video (Cook et al., 2003). Coaches who refused to

show the video perceived that reducing rough play would reduce quality of play and in turn game

wins (Cook et al., 2003). Negative attitudes about prevention efforts by coaches was an immense

barrier to the success of the program (Cook et al., 2003). Finally, the study found that coaches

who participated in the program had some type of history with concussions (Cook et al., 2003).

Critique of Evaluation

It is evident that communication theory, for behavioural change, was used to ground the

evaluation, although there was no explicit mention of theory. The evaluation was consistent in

emphasizing that behavioural change was more likely to occur if messaging was linked to the

specific outcomes of concussion and spinal injury (Center for Research in Implementation

Science and Prevention [CRISP], 2020). The evaluation also indicated that messaging was loss

framed, with emphasis on the negative outcomes of behaviours that may lead to concussion or

spinal injury (CRISP, 2020). The main goal of the evaluation was to determine if targeted group

messaging, which is a key aspect of communication theory, would have an effect on behaviour

change of the target population (Slater, 2020b). Communication theory is logical for this

evaluation, because it is centered around targeted group messaging, which is the basis of the

program being evaluated (Slater, 2020b). Other successful prevention program videos have used

communication theory, such as the targeted and loss framed messaging used in drunk driving

prevention commercials. Communication theory was appropriate for this evaluation because it is

logical, it has been used in other successful prevention programs, and it is grounded in behaviour

change research (Slater, 2020a).

4

The evaluation makes no mention of the use of a logic model. They describe program

outcomes in detail, but they do not indicate inputs, outputs, assumptions, or external factors. It is

thus difficult to observe a link between the logic model and evaluation, due to the absence of an

obvious logic model. Based on the evaluation, inputs for a logic model may include researchers,

relevant materials, time, and money. Outputs could include video showings, surveys, interviews,

players, and coaches. Assumptions may include that children pay close attention during the

video, and that they will not be exposed to the video again before they are evaluated. External

factors could include negative attitudes and perspectives that children may be exposed to

surrounding injury prevention in contact sports, or their previous perceptions of the risk of

engaging in the target behaviours.

RCTs are commonly known as the gold standard when evaluating the effectiveness of a

program (Hariton & Locascio, 2018). By randomizing the exposure, there is much less room for

bias than with many other study designs (Hariton & Locascio, 2018). A true experimental

design, such as this one, has the highest capability of any other study design to produce evidence

that can be deemed as causal (Boonmak, 2020). Additionally, the researcher synthesizing the

interview and survey data was blinded (Cook et al., 2003). RCTs may reduce the phenomenon of

information bias, known as misclassification, when researchers are blinded (Boonmak, 2020).

RCTs are also able control for confounding variables (Boonmak, 2020). The RCT design

reduced some potential bias, and it was the most appropriate and effective choice of study design

for this evaluation.

Outcomes identified and measured were coach interviews and surveys, player questions,

number and nature of penalties, and league standings (Cook et al., 2003). All outcomes chosen

were important, reasonable, and realistic (Slater, 2020a). Outcomes were important; reasons for
5

coach resistance to program needed to be identified as it is critical to the program, player

knowledge needed to be assessed in order to demonstrate the effectiveness of the program,

penalties needed to be assessed in order to determine if high risk behaviour had changed, and

league standings would demonstrate whether or not safer playing affected the success of the

team. Outcomes were reasonable and logical for the evaluation. Outcomes were realistic, which

was demonstrated in the evaluation through the statistically significant results. Additionally,

outcomes were specific, measurable, attainable, results-oriented, and timed (Slater, 2020a). The

outcomes were extremely specific to the program evaluation. The outcomes were measurable

and reasonably attainable through minimally invasive methods such as surveys, interviews, and

quantitative analysis. Outcomes chosen were results-oriented, which is demonstrated through the

significant results achieved in the evaluation. And finally, the critical outcome measure of player

concussion knowledge was timed; at mid hockey season before viewing the video, five minutes

after the viewing the video, and three months after viewing the video (Cook et al., 2003).

The way in which the outcomes were measured was thorough and effective. Researchers

were blinded when synthesizing coach interviews and player survey results (Cook et al., 2003).

Player questions were measured at specific points to enable a timed analysis. Player questions

measured knowledge transfer with regard to the symptoms and onset of concussion (Cook et al.,

2003). Number and nature of penalties, indicating risky contact, were effectively measured

quantitatively. League standings were also measured quantitatively. Surveys and interviews were

effectively used to measure coach concerns about the video as well as reasons for accepting it

(Cook et al., 2003). All outcome measurements were appropriate and adequate.

All studies, including RCTs, can be susceptible to bias. Sampling bias may be an issue in

this study design, limiting the findings of the study (Cook et al., 2003). Just 15% (5 teams)
6

agreed to take part in the study (Cook et al., 2003). Additionally, only 24% of coaches who

refused to show the video were interviewed (Cook et al., 2003). Due to an extremely small

sample size, the findings may not be externally valid (Cook et al., 2003). Randomization of the

teams was done prior to the agreement of the teams to participate, meaning that certain teams

may only have participated based on the awareness of their exposure, or lack thereof, to the

video (Cook et al., 2003). This introduced sampling bias, which may have skewed the results and

rendered the study internally invalid (Cook et al., 2003). Using the results of this study, future

studies should try to increase sample size and minimize sampling bias to increase the internal

and external validity of the results.

7

References

Boonmak, P. (2020). Quantitative methods in program planning and evaluation [PowerPoint

Slides]. https://onq.queensu.ca/d2l/le/content/364561/viewContent/2237862/View

Center for Research in Implementation Science and Prevention. (2020). Develop a

communication strategy use communication theory to target and tailor messages.

http://www.ucdenver.edu/academics/colleges/medicalschool/programs/crisp/training/toolki

ts/textingtoolkit/CommunicationStrategy/Pages/TargetandTailorMessages.aspx

Cook, D. J., Cusimano, M. D., Tator, C. H., & Chipman, M. L. (2003). Evaluation of the

ThinkFirst Canada, Smart Hockey, brain and spinal cord injury prevention video. Injury

Prevention, 9(4), 361-366.

Hariton, E., & Locascio, J. J. (2018). Randomised controlled trials - the gold standard for

effectiveness research: Study design: randomised controlled trials. BJOG : an

international journal of obstetrics and gynaecology, 125(13), 1716.

https://doi.org/10.1111/1471-0528.15199

Slater, M. (2020a). Program theory and logic models [PowerPoint Slides].

https://onq.queensu.ca/d2l/le/content/364561/viewContent/2219187/View

Slater, M. (2020b). Theories of change slides [PowerPoint Slides].

https://onq.queensu.ca/d2l/le/content/364561/viewContent/2224866/View