Sunteți pe pagina 1din 9

J. Behav. Ther. & Exp. Psychiat. 48 (2015) 66e74

J. Behav. Ther. & Exp. Psychiat. 48 (2015) 66 e 74 Contents lists available at ScienceDirect

Contents lists available at ScienceDirect

Journal of Behavior Therapy and Experimental Psychiatry

journal homepage: www.elsevier.com /locate/jbtep

journal homepage: www.elsevier.com /locate/jbtep The utility of case formulation in treatment decision

The utility of case formulation in treatment decision making; the effect of experience and expertise

Robert Dudley a , b , * , Barry Ingham b , Katy Sowerby a , Mark Freeston b , c

a Doctorate of Clinical Psychology, Newcastle University, UK

b Northumberland Tyne and Wear NHS Trust, UK

c Institute of Neuroscience, Newcastle University, UK

UK c Institute of Neuroscience, Newcastle University, UK article info Article history: Received 7 July 2014

article info

Article history:

Received 7 July 2014 Received in revised form 17 January 2015 Accepted 22 January 2015 Available online 18 February 2015

Keywords:

Cognitive therapy

Formulation

Treatment

Expertise

abstract

Objectives: We examined whether case formulation guides the endorsement of appropriate treatment strategies. We also considered whether experience and training led to more effective treatment de- cisions. To examine these questions two related studies were conducted both of which used a novel paradigm using clinically relevant decision-making tasks with multiple sources of information. Methods: Study one examined how clinicians utilised a pre-constructed CBT case formulation to plan treatment. Study two utilised a clinician-generated formulation to further examine the process of formulation development and the impact on treatment planning. Both studies considered the effect of therapist experience. Results: Both studies indicated that clinicians used the case formulation to select treatment choices that were highly matched to the case as described in the vignette. However, differences between experts and novice clinicians were only demonstrated when clinicians developed their own formulations of case material. When they developed their own formulations the experts' formulations were more parsimo- nious, internally consistent, and contained fewer errors and the experts were less swayed by irrelevant treatment options. Limitations: The nature of the experimental task, involving ratings of suitability of possible treatment options suggested for the case, limits the interpretation that formulation directs the development or generation of the clinician's treatment plan. In study two the task may still have limited the capacity to demonstrate further differences between expert and novice therapists. Conclusions: Formulation helps guide certain aspects of effective treatment decision making. When asked to generate a formulation clinicians with greater experience and expertise do this more effectively. Crown Copyright © 2015 Published by Elsevier Ltd. All rights reserved.

Case formulation is the process of blending the theoretical framework and scienti c knowledge the clinician brings with the unique experience of the client to help understand the presenting issues, and to select the optimal treatment ( Kuyken, Padesky, & Dudley, 2009; Mumma & Mooney, 2007 ). Consequently, formula- tion is considered to be at the heart of effective Cognitive Behav- ioural Therapy (CBT, Butler, 1998 ). However, the status afforded case formulation is somewhat mismatched with the relatively scant evidence base ( Bieling & Kuyken, 2003 ).

* Corresponding author. Doctorate of Clinical Psychology, Ridley Building, New-

castle University, Newcastle Upon Tyne, England, NE1 7RU, UK. Tel.: þ 44 191 222

7925.

E-mail address: r.e.j.dudley@ncl.ac.uk (R. Dudley).

http://dx.doi.org/10.1016/j.jbtep.2015.01.009

0005-7916/Crown Copyright © 2015 Published by Elsevier Ltd. All rights reserved.

Most of the previous research on formulation has examined the extent to which clinicians agree with each other or with an expert's formulation (e.g. Dudley, Park, James, & Dodgson, 2010; Kuyken, Fothergill, Musa, & Chadwick, 2005 ). Clinicians demonstrate modest levels of agreement (see for example Dudley et al., 2010 ; that demonstrated most elements of a formulation were not agreed on by over 70% of clinicians); although this is better for more overt presenting issues (where agreement was often over 70%). Also, clinicians with advanced training, more clinical experience, and accreditation as a CBT therapist, produce more reliable formula- tions ( Kuyken et al., 2005; Persons & Bertagnolli, 1999 ). Research considering validity has shown that therapist exper- tise ( Kuyken et al., 2005 ) and training in case formulation ( Kendjelic & Eells, 2007 ) improve the quality of case formulations

R. Dudley et al. / J. Behav. Ther. & Exp. Psychiat. 48 (2015) 66e 74

67

produced and lead to higher quality treatment plans ( Eells & Lombart; 2003; Eells, Lombart, Kendjelic, Turner, & Lucas, 2005 ). Research into the utility of formulation has considered the relationship between CBT case formulation and therapeutic outcome and has produced mixed results ( Chadwick, Williams, & Mackenzie, 2003; Ghaderi, 2006; Schulte, Kunzel, Pepping, & Schulte-Behrenburg, 1992 ). These studies have a number of meth- odological limitations one of which is that the quality of the formulation has generally not been evaluated ( Mumma, 2011 ). So whilst formulation is afforded a central role in CBT the evidence for its reliability, validity and utility is mixed and at present it is not clear how formulation affects treatment outcome ( Kuyken, Beshai, Dudley, Abel, Gorg, Gower et al., 2015 ). Of course, multiple factors potentially impact on treatment outcome. Owing to the complexity of linking any one aspect of therapy to outcome it may be more helpful to investigate the impact of formulation on an intermediate feature such as treatment planning ( Eells et al., 2005 ). The focus of the two studies reported here is whether formulation guides certain aspects of treatment decision making and planning. There were two aims to this research. The rst was to establish whether formulation guides the ratings of treatment interventions that do or do not t the formulation. The second was to consider whether greater training and experience leads to more effective use of CBT case formulations when making these ratings of treatment relevance. To examine these questions two related studies were conducted both of which used a novel paradigm using clinically relevant decision-making tasks with multiple sources of information. The rst study examined how clinicians utilised a pre-constructed CBT formulation to make treatment decisions. The second study utilised a clinician-generated formu- lation, rather than a pre-constructed one, to examine the process of formulation development and the impact on treatment plan- ning and decision making. Both studies considered the effect of therapist experience.

1. Study one

If formulation guides treatment, then formulations that differ in key features should lead to different interventions ( Butler, 1998 ). Beck, Epstein, and Harrison (1983) identi ed two dimensions important in the development of depression: autonomy (i.e. stressing the importance of independence and freedom of choice) and sociotropy (i.e. stressing the importance of af liations and interpersonal relations). Therefore, a formulation of a person with a sociotropic depression would differ from that of someone with an autonomous depression, even though both people would meet the criteria for major depressive disorder. Previous research has indicated that greater experience in CBT leads to better performance on formulation tasks ( Kuyken, Padesky, & Dudley, 2008; Mumma & Mooney, 2007). Hence, greater expe- rience in CBT should increase recognition of such features (i.e. au- tonomy and sociotropy) and guide clinicians towards treatment focussed on the key dimension rather than less pertinent features. Conversely, less experienced clinicians may be more distracted by less pertinent, or plausible but actually unhelpful or irrelevant treatment options. The primary hypothesis is that the content of a CBT case formulation would help the therapist rate as more appropriate formulation matching interventions rather than less pertinent or mismatched interventions. The secondary hypothesis was that more experienced clinicians would endorse more formulation matched interventions than the novice clinicians and endorse less the less pertinent or mismatched treatment options than novice clinicians.

2. Method

2.1. Design

A mixed (between-within) design was used. The within- subjects (formulation type with two levels, sociotropy or auton- omy) manipulation was used to test the main hypothesis that CBT case formulations have an effect on ratings of treatment options. The between-subject manipulation (experience; two levels, novice or experienced) tested the secondary hypothesis that differences in clinician experience would account for variance in performance.

2.2. Participants

Two groups (labelled as novice and experienced practitioners) were recruited. The novice group consisted of 23 clinicians who had an introductory training in CBT and a limited amount of practice in the use of CBT (e.g. rst year trainee clinical psychologists). The experienced group included 20 clinicians with extensive experience and training in CBT. The sample came from a variety of backgrounds but were clinicians who had supervised others in training and practice of CBT (e.g. experienced clinical psychologists, psychiatrists, nurse specialists in CBT). The demographic informa- tion for the participants is shown in Table 1.

2.3. Measures

A novel task was developed to assess use of CBT case formula- tions to plan treatment. Participants were presented with two prepared case formulation vignettes followed by multiple-choice options of potential CBT treatments. Participants made judge- ments as to which treatment planning options were the best t for the presented case formulations. Information about the case was presented sequentially to the participant. This consisted of; a referral letter from the client's GP relevant history from childhood to present day including early experiences and problem history, the results of assessment mea- sures relating to emotional distress, a completed thought record and a completed activity schedule. Then an initial psychological conceptualisation (described as developed after two assessment sessions) outlined an early working hypothesis for the develop- ment and maintenance of dif culties. Finally, a full formulation was provided. The model for the CBT case conceptualisation diagram (see Fig. 1 for an example) was based on the cognitive model of depression ( Beck, Rush, Shaw, & Emery, 1979 ) and featured a dia- grammatical representation of the longitudinal (which in- corporates information about early experience and how this predisposes a person to emotional problems owing to their core beliefs, rules and assumptions) and cross-sectional (also described

Table 1 Demographic information (including professional experience, CBT-related quali - cations, further/higher education) within clinician experience groups.

 

Novice (n ¼ 23)

Experienced (n ¼ 20)

Gender (n, % female) Diploma in CBT trained BABCP accredited Supervised on Diploma level CBT training

N ¼ 17, 74% N ¼ 0, 0% N ¼ 0, 0% N ¼ 0, 0%

N ¼ 11, 55% N ¼ 15, 75% N ¼ 10, 50% N ¼ 18, 90%

Mean

Sd

Mean

Sd

Age (yrs) Further/higher education (yrs) Months of clinical experience Months quali ed in profession

31.1

8.3

41.8

6.5

4.1

1.4

8

2.4

14.3

14.3

117.2

64.2

41.8

16.2

175.4

86.4

68

R. Dudley et al. / J. Behav. Ther. & Exp. Psychiat. 48 (2015) 66e 74

/ J. Behav. Ther. & Exp. Psychiat. 48 (2015) 66 e 74 Fig. 1. CBT case
/ J. Behav. Ther. & Exp. Psychiat. 48 (2015) 66 e 74 Fig. 1. CBT case
/ J. Behav. Ther. & Exp. Psychiat. 48 (2015) 66 e 74 Fig. 1. CBT case

Fig. 1. CBT case conceptualisation of Jess's dif culties.

as a maintenance formulation which emphasises what factors lead to the perpetuation of distress, such as the Padesky ve factor maintenance model, Greenberger & Padesky, 1995 ) case formula- tions (see Dudley & Kuyken, 2013; Kuyken et al., 2008 ). To establish the extent to which the formulation in uences judgments of relevance of treatment options the two vignettes differed in content. The rst formulation vignette (Jess) was based on a published case of depression ( Blackburn & Twaddle, 1996 ). The second vignette (Gerald) was based on the same conceptualisation model, but differed on a key theoretical dimension. Hence, Jess's vignette was high autonomy and low sociotropy, and Gerald's was high sociotropy and low autonomy. Information consistent with sociotropy or autonomy featured at the longitudinal (e.g. in the early experience and core beliefs) and cross-sectional (e.g. forming the basis for maintenance cycles) systems of the CBT case formulation.

The two cases (and their formulations) had to be equivalent in their accuracy, complexity, coherence, parsimony and overall clin- ical credibility. If they differed except on the key dimension of sociotropy and autonomy, then it would not be possible to demonstrate that different performance was not owing to some confounding variable. To help ensure these criteria were met, one of the cases was derived from a published case study and served as a template for the second case that was based on the clinical work undertaken by the authors. 1 Then, three experienced CBT clinicians acted as an expert reference group and completed a rating sheet

1 The rst case material was derived from a published chapter. The second was an amalgamation of several cases the authors had worked with. In addition, the cases were anonymised with identi able information removed, and or disguised to ensure that the person/people they were based on where not identi able.

R. Dudley et al. / J. Behav. Ther. & Exp. Psychiat. 48 (2015) 66e 74
R. Dudley et al. / J. Behav. Ther. & Exp. Psychiat. 48 (2015) 66e 74

69

Fig. 2. An example treatment planning question following from Jess's formulation.

that provided feedback on the consistency and coherency between and within vignettes. The experts were asked to assess each set of materials for i) the coherence of the assessment information and conceptualisation; ii) the quality of the case conceptualisation; iii) how realistic the cases were; and iv) the coherency of the treatment planning options in relation to the case. Each was rated on a scale from 1 to 10 (with 10 being high quality, coherence etc.). The ex- perts endorsed all the items above 7. Hence, the case materials were considered to be equivalent. The formulations were also endorsed as differing on the key dimension of sociotropy or autonomy. The reviewers also completed a quality of case formulation measure ( Kuyken et al., 2005 ) and both of the experimenter provided for- mulations were rated good enough on the quality of case formu- lation measure. Two sets of CBT based treatment planning tasks (with 14 questions in each set) were developed that followed from Jess and Gerald's formulation vignettes respectively. Participants had to judge how good a t those options were with the formulations (see Fig. 2 for an example of a treatment planning question). For each of these questions, three options were presented. Each option was either a good match (pertinent), mismatched (not pertinent) or a plausible but irrelevant option (a red herring ). The treatment planning questions were taken from CBT manuals (e.g. Beck, 1995 ) and included problem list development (1), goal setting (2), behavioural experiments (2), use of thought records (2), activity scheduling (1), role plays (2), continuum method (2), relapse

prevention planning (1) and identi cation of potential therapeutic barriers/problems (1). For each of the two vignettes (Gerald and Jessica) the questions were equivalent (e.g. the same number of behavioural experiment, or cognitive restructuring interventions were presented). Within each question for each of the options participants were asked to rate (on a scale of 0 e10, where 0 ¼ doesn't t and 10 ¼ de nitely ts) how good a t that option was to the CBT case formulation. 2 It was important that the treatment options really were regar- ded as matched, mismatched or as irrelevant. The match treatment options were taken from the case study as published for Jess, and for Gerald were based on a round of feedback and discussion with a group of experienced therapists before being presented to an expert in CBT. Then the materials were piloted with participants representative of the potential sample (i.e. novice [n ¼ 3] and experienced [n ¼ 3] clinicians). A further round of feedback from an expert clinician was then completed prior to full experimental testing. In keeping with the rst hypothesis it was anticipated that all the participants would rate the match option higher than the mismatch and irrelevant options. Consistent with the second hy- pothesis the experienced group was expected to endorse the match

2 Copies of all the materials including treatment options are available from the corresponding author on request.

70

R. Dudley et al. / J. Behav. Ther. & Exp. Psychiat. 48 (2015) 66e 74

option more than the novice group. The novice group was expected to less strongly endorse appropriate treatment options and more strongly endorse the red herring and irrelevant options.

2.4. Procedure

Participants were recruited from local psychology and CBT training courses and were provided with an information sheet, and signed a consent form. The participants were told to imagine that they were the therapist and that they need to plan treatment based on the assessment information and the case conceptualisation. They were instructed they needed to decide how well the treat- ment options t in relation to the case. An example unrelated to either of the two subsequently presented cases was provided to help familiarise them with the task. Each participant was presented with both of the case vignettes. The order of presentation was counterbalanced. After reading through the case information and formulations, participants worked through the treatment planning task. The order of the treatment planning questions and options was also counterbalanced throughout.

2.5. Review procedures and ethics

The work was subject to independent peer review, was regis- tered with the Research and Development Department of the local NHS trust and received a favourable opinion from a Local NHS Research Ethics Committee.

3. Results

Preliminary analysis revealed no missing data. Extreme re- sponses were identi ed by boxplot and winsorised. Data analysis was completed using SPSS 20 for Windows ( SPSS, 2012 ). Ratings on the response to treatment planning tasks are out- lined in Table 2 that shows the ratings across the two groups and by vignette. A mixed ANOVA with a between subject variable ( experience

with two levels [novice/experienced) and two within subject var- iables ( vignette type with two levels [Jess/Gerald], and response type with three levels [match/irrelevant/mismatch]) examined differences between the ratings for responses to treatment plan- ning tasks. The analysis was also run with Order as a between subjects variable. However, it had no bearing on the results and is not reported further. The main effect for vignette type across

¼ 1.26,

treatment planning options was not signi cant (F

p ¼ 0.27, h 2 ¼ .03, CI ¼ .00, .18). There was a main effect for response type (F (2,40) ¼ 388.01, p < 0.001 h 2 ¼ .90, CI ¼ .84, .93). Mauchley's test of sphericity was signi cant so the lower-bound Epsilon correction was used when determining F values. Planned simple contrasts showed that match responses were signi cantly higher in rated t to the vignette and provided case formulations than both

irrelevant (F (1,41) ¼ 221.85, p < 0.001 h 2 ¼ .84, CI ¼ .74, .89) and mismatch (F (1,41) ¼ 516.85, p < 0.001, h 2 ¼ .93, CI ¼ .88, .95), and

(1, 41)

irrelevant responses were signi cantly higher than mismatch response types (F (1,41) ¼ 365.62, p < 0.001, h 2 ¼ .90, CI ¼ .83, .93). This is consistent with the rst hypothesis.

Within the above ANOVA, there was no effect for experience (F

(1,

41) ¼ 1.35, p ¼ 0.19, h 2 ¼ .04, CI ¼ .00, .19). There was also no sig- nicant interaction effect between experience and response type (F (2, 40) ¼ .10, p ¼ 0.76, h 2 ¼ .00, CI ¼ .00, .09). Experience did not have an effect on treatment decision making hence hypothesis two was not supported.

4. Discussion

This study considered whether a predetermined formulation guided treatment planning decisions and whether CBT experience effected these decisions. As expected, participants rated match responses signi cantly higher on t to the formulation than the other options. This supports previous case formulation research proposing a link between formulation and treatment plans ( Butler, 1998 ). However, note that the match option was not endorsed at ceiling, and mismatched and red herring were not at oor, so overall the treatment plans, which clinicians may develop, would include a proportion of matched elements they would also include less than optimal elements. There was a second hypothesis that clinician experience would effect how CBT case formulation was used to inform treatment decisions. It was predicted that novice clinicians would more often endorse the red-herring/irrelevant and mismatch options. How- ever, there was no difference between the novice and experienced practitioners. This may be owing to limitations within this study outlined below and that are addressed in the subsequent study. This study indicates that clinicians who are provided with a case formulation and a list of treatment planning options are able to choose options that t with the formulation, regardless of level of training. Perhaps knowledge of CBT techniques provided through basic CBT training is suf cient, at least in this type of task to plan treatment. Previous studies have shown that more experienced and expert clinicians construct higher quality and more reliable CBT case formulations ( Persons & Bertagnolli, 1999 ). Hence, the differ- ence in experience may not be in the use of the formulation for treatment planning but in the development of the formulation. A number of potential limitations need to be considered when interpreting the ndings. First, in this study clinicians rate pre- determined treatment options from a number of possible options. Essentially, this was a recognition task. This is easier than when the clinician generates their own treatment option, and may help ac- count for the lack of differences between the experienced and novice clinicians. Second, the task involved using a constrained, prepared formulation and this may have reduced the potential for difference between the two groups. If clinicians had a greater role in constructing a formulation then this may lead to differences in the formulations produced and increase the likelihood of detecting differences in the resultant treatment plans based on these differing formulations.

Table 2 Mean ratings for response type across the two clinician experience levels and vignette.

 

Jess

Gerald

Match

Irrelevant

Mismatch

Match

Irrelevant

Mismatch

Novice

Mean

7.8

5.3

2.7

8.0

4.7

2.5

Std. Deviation

0.8

1.2

1.6

1.0

1.3

1.3

Experienced

Mean

7.6

4.7

2.1

7.5

4.2

2.5

Std. Deviation

0.9

1.5

1.3

0.7

1.4

1.3

Total

Mean

7.7

4.8

2.2

7.9

4.3

2.4

Std. Deviation

0.9

1.5

1.4

0.9

1.3

1.2

R. Dudley et al. / J. Behav. Ther. & Exp. Psychiat. 48 (2015) 66e 74

71

Third, the inclusion criteria used to distinguish between the novice and experienced groups may not have been suf cient to distinguish the two. Often samples have been composed of expe- rienced rather than expert clinicians, which may make the groups too similar to be contrasted ( Skovholt, Ronnestad, & Jennings, 1997 ). Furthermore, the tasks used in the research may have been too simple to differentiate between groups ( Skovholt et al., 1997 ). These criticisms may apply to study one and may be a reason for a failure to detect difference between experienced and novice ther- apists. Study two addresses some of these possible limitations.

5. Study two

This study investigated the role of formulation in treatment planning and drew on and adapted the method in study one, and utilised Ericsson and Smith's (1991) Expert Performance Approach as a model to increase the difference between novices and experts. This three-stage descriptive and inductive framework is proposed to underpin the empirical analysis of expert performance; arguing for i) ecologically valid tasks, ii) process-tracing methods to measure processes underpinning performance, and iii) a detailed understanding of the individual's background to aid classi cation of expertise. Following this framework led to three key changes to the task outlined in study one. First, in order to increase the ecological validity, therapists were asked to generate their own formulation when treatment planning. This change increased the potential differences between the participants. Experts have been shown to generate better quality ( Kuyken et al., 2005 ) formulations than novices. Hence, it was predicted that the formulations of experts would be of a higher quality than those of novices. Speci cally, it was hypothesised that expert formulations would contain more essential (and less extraneous) content from the vignette and thus, they would be more parsimonious and more internally consistent than those of novice clinicians. Second, in this study an attempt was made to explore or trace the processes underpinning performance during each stage of the formulation generation ( Bennett, 2008; 2010; Checkel, 2008; Einhorn & Hogarth, 1981; Einhorn, Kleinmuntz, Klein & Muntz, 1979 ). In other domains, experts, as compared to novices, have been shown to perform faster, whilst making fewer errors, demonstrating frequent self-monitoring and adjustments of their work ( Glaser & Chi, 1988 ). Hence, it was predicted that experts would make fewer errors than novices. Previous research has indicated that novices are often unaware of their lack of skill and hence are overcon dent in their estima- tions of task performance ( Davis et al., 2006; Kruger & Dunning, 1999 ). Hence it was predicted that novices would report a higher level of con dence in key aspects of the formulation than the experts. Third, in this study particular effort was made to recruit expert rather than experienced therapists. As with study one, it was hypothesised that experts would choose better treatment planning options (endorsing more the match option, and rating as less relevant to the formulation the mismatched, and irrelevant op- tions) than novice clinicians as it was assumed that the expert produced formulations were of a higher quality than the novices and would better direct treatment planning.

6. Method

6.1. Design

A mixed between groups (novice versus expert) with within subject (three levels treatment option) design was utilised.

Table 3 Demographic information for each group.

 

Novice (n ¼ 31)

Expert (n ¼ 16)

Gender (n, % female) Diploma in CBT trained BABCP accredited Supervised on Diploma level CBT training

N ¼ 31, 94% N ¼ 0, 0% N ¼ 0, 0% N ¼ 0, 0%

N ¼ 16, 56% N ¼ 2, 13% N ¼ 8, 50% N ¼ 1, 6%

Mean

Sd

Mean

Sd

Age (yrs) Months of clinical experience Months quali ed in profession Research publications Number of workshops delivered Number of CBT cases seen

27

2.7

49

11.4

28.7

17.5

271.2

63.5

0

0

231.7

64.2

0

0

12.3

18.9

0

0

20.8

19.7

4.1

4.9

251.9

154.0

6.2. Participants

Two groups (different to those in study one) were recruited. One group consisted of 31 novice therapists all in their rst year of clinical psychology training. The second group consisted of 15 expert CBT therapists who met at least two of three criteria: a minimum of ten years of experience; evidence of continuous re ection regarding formulation and/or CBT techniques through research, publications/conference appearances on related topics, supervision of other CBT therapists; or evidence of commitment through delivery of training and continuing professional develop- ment (CPD) events on the topic of formulation ( Skovholt et al.,1997 ). Table 3 indicates that the expert group were very highly expe- rienced (with a mean of 271 months of experience). Whilst these participants were more experienced than the group of experienced therapists in study one they were less likely to have completed a diploma or equivalent in CBT, or to supervise on diploma level training but were just as likely to be BABCP accredited. However, in terms of publications, books, and other esteem indicators the ex- perts differed substantially. 3

6.3. Measures

The assessment and formulation materials (Jess) developed in study one were utilised within study two. However, participants were not provided with the provisional or nal completed formu- lation, instead they were asked to generate their own which was recorded on to a blank formulation template. They then used this to answer the treatment planning questions. A manual 4 used to score the quality of the participants formu- lations was utilised in this study. It was based on the manuals developed by Kuyken et al., (2005) and Eells et al. (2005) . Four components were chosen as important measures for the current study namely; parsimony (correct items divided by total items entered), internal consistency or coherence of the formulation (which was scored out of a maximum of 11), exibility (number of changes made as new information was provided) and errors (number of mistakes in either providing material that experts had not seen as appropriate in the formulation or entering the correct information in the wrong section of the formulation). The quality manual was developed and tested on the formula- tions produced by a separate group of 30 novice, and experienced but not expert clinicians (this data was not used within the main analyses as it was used for manual development) who undertook the same task as the novice and experts in this study. Initially ve of

3 Further information about the expert participants is available on request from the corresponding author.

4 A copy of this is available from the corresponding author.

72

R. Dudley et al. / J. Behav. Ther. & Exp. Psychiat. 48 (2015) 66e 74

these formulations were used to generate content examples for the manual. Two members of the research team then independently scored a further ve formulations, both were blind to the experi- ence level of the participants. At this point the two researchers achieved 67.3% consistency in scoring, indicating the usability of the manual. Discrepancies were discussed and resolved and the manual was updated. A further ve formulations were scored in light of these revisions. Eighty-two percent agreement was ach- ieved. The two researchers then scored the 15 remaining formu- lations and agreement increased to 86%. The manual was used to score the expert and novice formulations by the same two raters who were blind to group membership with a similarly high rate of agreement between raters (Cohen's kappa greater than 0.8 for ex- perts and novice ratings and overall).

6.4. Procedure

The novices were recruited via Clinical Psychology training courses. Experts were identi ed if they published on the topic of formulation, and or were regular contributors to national and in- ternational conferences in CBT. They were invited by email to participate. Individuals were provided with the task instructions; a complete set of assessment materials from the case vignette from study one (referral letter, relevant history, assessment scores, thought record, and activity schedule); a blank formulation tem- plate based on the format in Fig. 1 ; four extra maintenance cycle templates (as in the lower part of Fig. 1 ); treatment planning questions; and a set of coloured pens.

6.5. The formulation

The task was structured to allow a formulation to be built up step-by-step from the sequential presentation of the ve pieces of assessment material. Each piece of assessment material was colour coded and for each a corresponding coloured pen was provided. Individuals were asked to start with the referral letter and using the corresponding coloured pen note down emerging ideas about the case on the blank formulation template. Every time new informa- tion was added therapists were asked to rate their certainty (0 e100% certainty) regarding the contribution of it to the emer- gence or maintenance of Jess's dif culties. Therapists were asked to follow the same procedure for each piece of assessment material. In order to explore the way in which judgments and decisions change as more information comes to light therapists were asked to neatly score through information they later believed to be irrelevant or erroneous, including changing con dence ratings.

6.6. Treatment planning

Once all ve pieces of assessment material had been seen and the formulation template was completed participants were asked to consider the treatment planning questions (taken from study 1) in light of their own clinician-generated formulation.

6.7. Review procedures and ethics

The work was subject to the same review processes as study one.

7. Results

7.1. Formulation quality

The Quality of formulations was scored using the manual. Con dence intervals, means and standard deviations for parsimony

and consistency are summarised in Table 4 . The experts were more parsimonious in that they reported more correct information

relative to the irrelevant information ( t (44) ¼ 7.7, p < 0.01, Cohen's

d ¼ 2.61), and produced formulations that were more internally

consistent and coherent ( t (44) ¼ 3.1, p < 0.01, Cohen's d ¼ 0.73) than the novices.

7.2. Process measures

It was not possible to record how quickly each individual per- formed the task as some of the testing sessions were undertaken in

groups, so a proxy measure of speed of processing was to record the stage at which participants detected the key dimension within their formulation, and the stage at which they were at least 70% certain that this contributed to, or maintained, the client's dif - culties. Errors, changes made and legitimacy of changes were also recorded. Fifty-seven percent of novices identi ed the key dimension of autonomy by stage one, 32.1% by stage two, and the remaining 10.9% between stages three and ve. Similarly 35.7% of experts identi ed it in stage one, 14.3% in stage two and the rest between three and ve. More novices (33.3%) than experts (18.2%) were con dent about the importance of the key theme by stage one, most experts reached the con dence threshold by stage two or three. None of these trends were signi cant. However, they imply that the experts were more cautious than the novices in the early stages of formulation. This would be consistent with the hypothesis that novices are relatively overcon dent. Experts made fewer changes ( t (44) ¼ 3.02, p < 0.005, Cohen's

d ¼ 1.1, M ¼ .4, sd ¼ 0.6) than novices (M ¼ 3.45, sd ¼ 3.75) and

fewer errors, with novices making 35 errors between them and experts making only two. This supports the hypothesis that experts would make fewer errors. Many of the errors made by the novices involved either placing incorrect material in the formulation (i.e. core beliefs that were not related to the client formulation) or placing information in the wrong part of the formulation (so labelling core beliefs as rules or assumptions or as thoughts, or perhaps placing trigger/precipitant information in the early expe- rience part of the formulation). Overall it appeared that the experts waited until certain and then did not need to revise their

formulations.

7.3. Treatment planning

Preliminary analysis on the dataset identi ed four missing values within the treatment planning questions. Data was replaced with the mean of the person's rating on the other items. Parametric assumptions were met. Subsequently, a mixed ANOVA with a be- tween subjects variable of Group (2 levels) and within subjects of

Table 4 Parsimony and coherence of the expert and novice generated formulations.

Quality component

Novice (n ¼ 31)

Expert (n ¼ 15)

Parsimony (%) Mean (SD) Con dence intervals Coherence/Consistency a Mean (SD) Con dence intervals Flexibility (changes made) Mean (sd) Con dence intervals Errors made Total

81.7 (5.2) 79.8 e 83.6

93.0 (3.2) 91.3 e 94.8

7.9 (.9) 7.6 e 8.3

8.7 (1.3) 7.9 e 9.4

3.4 (3.8) 2.1 e 4.8

0.4 (0.6) 0.01 e 0.7

35

2

a Coherence is scored out of a maximum of 11.

R. Dudley et al. / J. Behav. Ther. & Exp. Psychiat. 48 (2015) 66e 74

73

Table 5 Mean ratings for response type across the two clinician experience levels.

 

Match

Irrelevant

Mismatch

Novice

Mean

8.9

6.9

4.4

Std. Deviation

1.5

1.6

2.0

Expert

Mean

7.9

4.5

1.9

Std. Deviation

2.9

1.3

1.2

response type (3 levels) was undertaken with planned comparisons

to compare responses to the treatment planning tasks between the

groups. Ratings on overall response to therapeutic prediction tasks are

outlined in Table 5 , these provide the summed responses from the task. As in the rst study, these were either match, irrelevant, or mismatch. An ANOVA comparing experts versus novice clinicians on response type (match, mismatch, irrelevant) demonstrated a sig-

42) ¼ 137.92, p < 0.001,

ni cant main effect of response type (F

h 2 ¼ 0.76). Mauchly's test was violated ( c 2 (2) ¼ 21.61, p < 0.001)

and hence the Greenhouse-Geisser correction was applied. Planned simple contrasts showed that match (M ¼ 8.46, sd ¼ 2.25) was signi cantly higher than irrelevant (5.77, sd ¼ 1.45) and mismatch (3.19, sd ¼ 1.6), and that irrelevant/red herring was signi cantly higher than mismatch (all p values < .001). There was a main effect

of group (F (1, 43) ¼ 12.1, p ¼ 0.001, h 2 ¼ 0.23) with the expert group

(2,

(M

¼ 5.27, sd ¼ 0.42) endorsing all options less than the novices

(M

¼ 6.73, sd ¼ 0.28). There was no signi cant interaction between

the response patterns of the two groups (F

¼ 3.43, p ¼ 0.05,

h 2 ¼ 0.07), although it was approaching a signi cant value. Given

the predicted differences in performance by the groups on the task these were explored as these pertained to the hypotheses. There was no signi cant difference between experts and novice clini- cians' ratings on match questions ( t (44) ¼ 1.52, p ¼ 0.14,

CI ¼ 3.27 e23.22, Cohen's d ¼ 0.43). Experts rated the irrelevant or

red herring treatment planning questions as a lower t to the

formulation than the novices (t(44) ¼ 4.91, p < 0.001,

CI ¼ 14.14 e 33.85, Cohen's d ¼ 1.65). Also, the experts rated the

mismatch option as less relevant to the formulation than the novices (t(44) ¼ 4.18, p < 0.001, CI ¼ 12.79 e 36.61, Cohen's d ¼ 1.52). Whilst these were planned analyses owing to the lack of interaction effect in the omnibus test we applied a Bonferroni correction and the latter two analyses remained signi cant.

(2,42)

8. Discussion

As predicted Experts generated higher quality formulations than novices. Expert clinicians included less inappropriate or super cial information, and provided more internally consistent and coherent formulations than the novices. Both groups identi ed the impor- tant features, however, the experts seemed to be able to do so more elegantly. A second hypothesis was that novices would be overcon dent in their formulations relative to experts. Novices reported the key theme earlier and felt more con dent in it earlier than expert therapists but not to a signi cant degree. As predicted, experts were shown to make far fewer errors than novices. In general the experts demonstrated a more cautious approach. This ts with Glaser and Chi's assertion that experts require more information than novices before making a decision. It was further hypothesised that clinician experience would affect treatment planning choices. Experts rated inappropriate treatment plans as less relevant than novices, although there was no signi cant difference in ratings of the matched treatment op- tions. It seems therefore that both experts and novices can identify

important treatment options, but experts are less likely to rate or endorse less-relevant or less-appropriate interventions. Previous studies have also suggested that experience and expertise leads to differences in treatment planning from a formulation ( Eells & Lombart, 2003; Eells et al., 2005 ). It is plausible that as individuals move towards expertise they become better at seeing the key themes, or patterns, ltering out the less salient information when formulating, and choosing the best of all available treatment options, in turn becoming more ef cient and hopefully effective. Certainly, experts have been shown to notice meaningful patterns more readily than novices ( Glaser & Chi, 1988 ). However, based on Glaser and Chi's (1988) assertion that experts are more able to self-monitor their performance and adapt accordingly, it was surprising that experts made fewer changes than novices when presented with new information. One plausible explanation (given the small number of errors made by experts) was that the task was not dif cult enough to challenge experts. Whilst the task may have been too easy to challenge experts, it did draw out a number of crucial differences between experts and novices in line with previous research (e.g. Eells et al., 2005 ). In study one a comprehensive formulation was provided and there were no differences in ratings of t treatment options, however when individuals were asked to generate their own formulation signi cant differences in treatment choices were evident between novice and expert clinicians. Of course, it is important to remember that both studies were essentially a recognition task which sub- stantially reduces the ecological validity of the tasks. The experts were selected based on a broad range of criteria shown to be fundamental to the development of expertise (such as involvement in supervising others, researching in the area) and differed from novices on a range of key variables taken to be evi- dence of expertise (such as years of experience, evidence of re ection and/or evidence of commitment to Continuing Profes- sional Development). In this way the study moved towards providing a more comprehensive operationalisation of expertise.

9. General discussion

Study one demonstrated that providing a comprehensive formulation enabled clinicians of all levels to make similar treat- ment decisions. In study two, when participants were asked to generate their own formulations, it resulted not only in novices generating less parsimonious formulations than experts, but also negatively impacted upon the appropriateness of their treatment choices. This has important clinical implications. Butler (1998) proposed that the formulation provides a plausible explanation for a person's symptoms and is of central importance as it is thought to instil hope in the client as well as enhancing the alliance, and provide opportunities for intervention. A poorer quality formulation as generated by the novice clinicians is less likely to ful l these aims and may even lead to therapeutic ruptures. The importance of expert supervision therefore, especially at the formulation stage is crucial, given the nding in study one that given a comprehensive formulation all groups can plan treatment equally well, receipt of expert supervision, especially during the formulation stage, may lead to better treatment plans. However, it is important to note that experts and novices did not differ on the correct information within the formulation, or indeed on ratings of appropriate treatment plans. This indicates that all clinicians were able to identify important aspects of a client's presentation on the case formulation generation task as was used here, but differ solely on the focus and clarity with which this is developed and used. Future research may consider manipulating the content of the cases so that there are simple and more complex presentations

74

R. Dudley et al. / J. Behav. Ther. & Exp. Psychiat. 48 (2015) 66e 74

( Dudley, Kuyken, & Padesky, 2011 ). Such material may challenge novice clinicians and better reveal the value that expertise brings to the process of formulation. Finally, using think out aloud methods may provide a means of understanding the process of formulation rather than just seeing the outcome of the formulation process.

Acknowledgements

We would like to thank Peter Armstrong, Stephen Barton, Ivy Blackburn and Willem Kuyken who kindly acted as the expert re- viewers of the materials used in these studies.

References

Beck, J. S. (1995). Cognitive therapy: Basics and beyond . New York: Guilford Press. Beck, A. T., Epstein, N., & Harrison, R. P. (1983). Cognitions, attitudes and personality dimensions in depression. British Journal of Cognitive Psychotherapy, 1, 1 e16 . Beck, A. T., Rush, A. J., Shaw, B. F., & Emery, G. (1979). Cognitive therapy of depression . New York: Guilford Press. Bennett, A. (2008). Process-tracing: a Bayesian Perspective. In J. M. Box-Steffen- smeier, H. E. Brady, & D. Collier (Eds.), The Oxford Handbook of Political Meth- odology (pp. 702 e721). Oxford: Oxford University Press . Bennett, A. (2010). Process tracing and causal inference. In H. E. Brady, & D. Collier (Eds.), Rethinking Social Inquiry: Diverse Tools Shared standards (2nd ed.). (pp. 207 e220). Lanham: Rowman and Little eld Publishers Inc . Bieling, P. J., & Kuyken, W. (2003). Is cognitive case formulation science or science ction? Clinical Psychology-Science and Practice, 10 , 52 e 69 . Blackburn, I., & Twaddle, V. (1996). Cognitive therapy in action: A practitioners' case book . Butler, G. (1998). Clinical formulation. In A. S. Bellack, & M. Hersen (Eds.), Comprehensive clinical psychology (pp. 1 e24). New York: Pergammon Press. Chadwick, P., Williams, C., & Mackenzie, J. (2003). Impact of case formulation in cognitive behaviour therapy for psychosis. Behaviour Research and Therapy, 41, 671 e680 . Checkel, J. T. (2008). Tracing Causal Mechanisms. International Studies Review, 8 (2), 362 e 370 . Davis, D. A., Mazmanian, P. E., Fordis, M., Van Harrison, R., Thorpe, K. E., & Perrier, L. (2006). Accuracy of physician self-assessment compared with observed mea- sures of competence: a systematic review. Journal of the American Medical As- sociation, 296 (9), 1094 e1102 . Dudley, R. E. J., & Kuyken, W. (2013). Case formulation in cognitive behavioural therapy: a principle driven approach. In L. Johnstone, & R. Dallos (Eds.), Formulation in psychology and psychotherapy (2 nd ed.). Brunner Routledge . Dudley, R., Kuyken, W., & Padesky, C. A. (2011). Disorder speci c and trans- diagnostic case conceptualisation. Clinical Psychology Review, 31, 213 e 224 . Dudley, R., Park, I., James, I., & Dodgson, G. (2010). Rate of agreement between clinicians on the content of a cognitive formulation of delusional beliefs: the effect of quali cations and experience. Behavioural and Cognitive Psychotherapy, 38 , 185 e 200 . Eells, T. D., & Lombart, K. G. (2003). Case formulation and treatment concepts among novice, experienced, and expert cognitive-behavioural and psychody- namic therapists. Psychotherapy Research, 13 , 187 e204 .

Eells, T. D., Lombart, K. G., Kendjelic, E. M., Turner, L. C., & Lucas, C. (2005). The quality of case formulations: a comparison of expert, experienced and novice cognitive-behavioural and psychodynamic therapists. Journal of Consulting and Clinical Psychology, 73 (4), 579 e 589 . Einhorn, H. J., & Hogarth, R. M. (1981). Behavioral decision Theory: processes of judgment and choice. Annual Review of Psychology, 32 , 53 e 88 . Einhorn, H. J., Kleinmuntz, D. N., Klein, & Muntz, B. (1979). Linear regression and process-tracing models of judgment. Psychology Review, 86 , 465 e 485 . Ericsson, K. A., & Smith, J. (1991). Prospects and limits in the empirical study of expertise: an introduction. In KA, J. Ericsson, & Smith (Eds.), Toward a general Theory of Expertise: Prospects and limits (pp. 1 e38). Cambridge: Cambridge University Press. Ghaderi, A. (2006). Does individualization matter? A randomized trial of stan- dardized (focused) versus individualized (broad) cognitive behavior therapy for bulimia nervosa. Behaviour Research and Therapy, 44 , 273 e 288 . Glaser, R., & Chi, M. T. H. (1988). Overview. In M. T. H. Chi, R. Glaser, & M. J. Farr (Eds.), The nature of expertise . Hillsdale, NJ: Lawrence Erlbaum . Greenberger, D., & Padesky, C. A. (1995). Mind over mood: Change how you feel by changing the way you think . New York: Guilford . Kendjelic, E. M., & Eells, T. D. (2007). Generic psychotherapy case formulation training improves formulation quality. Psychotherapy, 44 , 66 e77. Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: how dif culties in recognizing one's own incompetence Lead to in ated self-assessments. Journal of Personality and Social Psychology, 77 (6), 1121 e1134 . Kuyken, W., Fothergill, C. D., Musa, M., & Chadwick, P. (2005). The reliability and quality of cognitive case formulation. Behaviour Research and Therapy, 43 ,

1187 e1201.

Kuyken, W., Padesky, C., & Dudley, R. (2008). The science and practice of case conceptualisation. Behavioral and Cognitive Psychotherapy, 36 (Special Issue 06),

757 e768 .

Kuyken, W., Padesky, C., & Dudley, R. (2009). Collaborative case conceptualisation. Working effectively with clients in cognitive behavioural therapy. Guildford Press. Kuyken, W., Beshai, S., Dudley, R., Abel, A., Gorg, N., Gower, P., McManus, F., & Padesky, C. Assessing competence in collaborative case conceptualization,:

development and preliminary psychometric properties of the collaborative case conceptualization rating scale (CCC-RS), Behavioural and Cognitive Psychother- apy, published online 28/01/15. Mumma, G. H. (2011). Validity issues in cognitive-behavioral case formulation. European Journal of Psychological Assessment, 27 (1), 29 e 49. http://dx.doi.org/10. 1027/1015-5759/a000054 . Mumma, G., & Mooney, K. (2007). Comparing the validity of alternative cognitive case formulations: a latent variable, multivariate time series approach. Cognitive Therapy and Research, 31 (4), 451 e481. Persons, J. B., & Bertagnolli, A. (1999). Inter-rater reliability of cognitive-behavioral case formulations of depression: a replication. Cognitive Therapy and Research, 23 , 271 e283 . Schulte, D., Kunzel, R., Pepping, G., & Shulte-Bahrenberg, T. (1992). Tailor-made versus standardized therapy of phobic patients. Advances in Behaviour Research and Therapy, 14 , 67 e 92 . Skovholt, R., Ronnestad, M. H., & Jennings, L. (1997). The search for expertise in counselling, psychotherapy and professional psychology. Educational Psychology Review, 9 , 361 e369 . Statistical Packages for Social Sciences (SPSS). (2012). SPSS 20.0 for Windows. Chi- cago: SPSS inc .