Critical appraisal for speech and language therapists: intervention study

The purpose of this tool is to provide a structured framework for reading and appraising reports of intervention or treatment studies. The gold standard design for an intervention study is the randomised controlled trial (RCT). The tool can also be used for other types of intervention study e.g. exploratory, preliminary or pilot-study designs. However, there is a separate tool for appraising reports of single-case study and case series designs. Use the following questions to assess the extent to which you feel the study was well-designed, apt and relevant, how reliable (replicable) and valid (true) the results are and whether there are implications for your service. There is space for you to add your own notes. The prompts are there to highlight why an issue is important but they shouldnt limit your thinking. There is quite a lot of overlap between questions.

Study reference:

Important clinical question?

Did the study ask a clearly focused question, in terms of: the Population studied

the Intervention given

the Comparator/control(s) used

the Outcomes considered

Is the question important for your clinical practice?

Design

Was this a randomised controlled trial (RCT) and if so, was it appropriately so? Why was this study carried out as an RCT? Was this the right research approach for the question being asked?

Adapted from Public Health Resource Unit, England (2006) Critical Appraisal Skills Programme by Jennifer Reid, Speech and Language Therapy Service, NHS Fife. Cartoons by Fran commissioned by Speech and Language Therapy in Practice. Accompanies Reid, J. (2011) Journal club 4: intervention studies, Speech & Language Therapy in Practice Summer, pp.18-21. Please attribute these sources if you use, distribute or adapt any of this material.

Allocation

Were participants appropriately allocated to intervention and control groups? Was the method of allocation adequately described?

Was the process truly random?

Was a method used to balance the randomisation, e.g. stratification?

How was the randomisation schedule generated and how was a participant allocated to a study group?

Were the groups well balanced? Are any differences between the groups at entry to the trial reported?

Were there differences reported that might have explained any outcome(s) (confounding)?

Blinding

Were participants, staff and study personnel blind to participants study group? Consider: the fact that blinding is not always possible if every effort was made to achieve blinding if you think it matters in this study that we are looking for observer bias.

Adapted from Public Health Resource Unit, England (2006) Critical Appraisal Skills Programme by Jennifer Reid, Speech and Language Therapy Service, NHS Fife. Cartoons by Fran commissioned by Speech and Language Therapy in Practice. Accompanies Reid, J. (2011) Journal club 4: intervention studies, Speech & Language Therapy in Practice Summer, pp.18-21. Please attribute these sources if you use, distribute or adapt any of this material.

Setting

Can the results be applied to the local population? Could the population sample used be different from your population in ways that would produce different results?

Does your local setting differ much from that of the study?

Could you provide the same intervention in your setting?

Participants

Were the participants in all groups followed up and data collected in the same way? Were there any differences that might introduce performance bias, e.g. time intervals between measures or the amount of attention received from researchers, therapists or other practitioners?

Did the study have enough participants to minimise the play of chance? Look for report of a power calculation. This will estimate how many participants are needed to be reasonably sure of finding something important (if it really exists and for a given level of uncertainty about the final result).

Adapted from Public Health Resource Unit, England (2006) Critical Appraisal Skills Programme by Jennifer Reid, Speech and Language Therapy Service, NHS Fife. Cartoons by Fran commissioned by Speech and Language Therapy in Practice. Accompanies Reid, J. (2011) Journal club 4: intervention studies, Speech & Language Therapy in Practice Summer, pp.18-21. Please attribute these sources if you use, distribute or adapt any of this material.

Intervention

Is the intervention described in sufficient detail? Consider: Intensity frequency and length of sessions

Dosage how much, for how long and according to what sort of schedule (e.g. distributed vs. massed learning)

Content targets, success criteria, resources

Role and/or responses of therapist or other practitioner i.e. how changes in behaviour are facilitated.

Could you replicate it on the basis of the information given?

Does the intervention have a track record? What prior support is there for its efficacy (delivery under ideal conditions) or effectiveness (delivery in real world settings)?

Is it consistent with known facts, previous research, and theory?

Attrition

Were all of the participants who entered the trial accounted for at its conclusion? Did any intervention-group participants get a control-group option or vice versa?

Were all participants followed up in each study group (was there loss-to-follow-up?)

Were all the participants outcomes analysed by the groups to which they were originally allocated (intention-to-treat analysis)

What additional information would you have liked to see to make you feel more confident about this study?

Adapted from Public Health Resource Unit, England (2006) Critical Appraisal Skills Programme by Jennifer Reid, Speech and Language Therapy Service, NHS Fife. Cartoons by Fran commissioned by Speech and Language Therapy in Practice. Accompanies Reid, J. (2011) Journal club 4: intervention studies, Speech & Language Therapy in Practice Summer, pp.18-21. Please attribute these sources if you use, distribute or adapt any of this material.

Results

What are the main results, and how are they presented? Are results presented as e.g. proportion of participants with a given outcome, or as a measurement such as mean or median differences?

How large and how meaningful is this result?

Can you sum up the bottom-line results of the study in one sentence?

How precise are these results? Is the result precise enough to make a decision for your own practice or that of the wider service?

Is a confidence interval or a p-value reported?

Outcomes

Were all important outcomes considered so the results can be applied? Could the people included in the study be different from your population in ways that would produce different results?

Does your local setting differ much from that of the trial?

Could you provide the same treatment in your setting?

Consider outcomes from the point of view of the: individual

policy makers and professionals

family/carers

wider community

Adapted from Public Health Resource Unit, England (2006) Critical Appraisal Skills Programme by Jennifer Reid, Speech and Language Therapy Service, NHS Fife. Cartoons by Fran commissioned by Speech and Language Therapy in Practice. Accompanies Reid, J. (2011) Journal club 4: intervention studies, Speech & Language Therapy in Practice Summer, pp.18-21. Please attribute these sources if you use, distribute or adapt any of this material.

Impact Summary

Should policy or practice change as a result of the evidence contained in this review?

Consider whether any benefit reported outweighs any harm and/or cost. If this information is not reported, can it be filled in from elsewhere?

Adapted from Public Health Resource Unit, England (2006) Critical Appraisal Skills Programme by Jennifer Reid, Speech and Language Therapy Service, NHS Fife. Cartoons by Fran commissioned by Speech and Language Therapy in Practice. Accompanies Reid, J. (2011) Journal club 4: intervention studies, Speech & Language Therapy in Practice Summer, pp.18-21. Please attribute these sources if you use, distribute or adapt any of this material.