Sunteți pe pagina 1din 12

Toate pasajele albastre din text sunt linkuri pe care se poate da click (și, Doamne-ferește, citi și în plus)

Probabil cea mai controversată declarație a ultimului deceniu din psihiatrie îi aparține lui Thomas Insel, care,
în 2017, făcea un comentariu care avea să facă rapid înconjurul Internetului și avea să dea muniție interminabilă
anti-psihiatriei, în anii următori. Insel, care condusese timp de 13 ani National Institues of Mental Health, cea
mai puternică instituție de cercetare în psihiatrie din lume, declara atunci că institutul cheltuise nu mai puțin
de 20 de miliarde de dolari pe cercetare neurobiologică în psihiatrie fără a fi reușit să ofere niciun tratament
nou și nici măcar vreo îmbunătățire a tratamentelor existente deja. Preluată rapid de publicul de psihologi
extrem de interesați de propaganda dualistă conform căreia mintea e separată de creier și necesită un
tratament complet ne-medicalizat, declarația a fost într-adevăr un indiciu important al unei înfrângeri cel
puțin temporare. Aparent.

În realitate, dansul victoriei era pregătit de mult timp și fusese alimentat de multă frustrare, parțial întemeiată,
din partea comunității psihologice. Deși absolvent al unui rezidențiat de psihiatrie, Insel fusese întotdeauna
un outsider în psihiatria clinică și chiar în cea universitară. Își petrecuse două decenii din carieră făcând
cercetare fundamentală neuroștiințifică explorând rolul oxitocinei în comportamentele sociale la mamifere.
În anii ’90, avea să colaboreze cu Paul MacLean la National Institues of Mental Health, înainte de a pleca la
Universitatea Emory, în Atlanta, unde avea să lucreze la încercarea de a dezvolta un vaccin împotriva HIV, iar
la finalul deceniului, avea să conducă nou înființatul centru pentru știință și tehnologie din cadrul National
Science Foundation, unde avea să exploreze rolul oxitocinei și al vasopresinei în tratamentul autismului. În
2002, Insel avea să surprindă lumea academică a psihiatriei americane atunci când revenea la National
Institutes of Mental Health pentru a-l înlocui pe Steven Hyman la șefia institutului. Mutarea era neobișnuită.
Hyman fusese clinician și cercetător în psihiatrie, precum și profesor de psihiatrie la Harvard, o carieră similară
cu majoritatea șefilor NIMH de până atunci, unde outsiderii fuseseră o raritate. În 1950, la înființarea
institutului, un alt neurocercetător celebru, însă fără legătură directă cu lumea psihiatriei, Seymour Kety,
fusese numit director la insistențele autorităților, foarte sceptice cu privire la abordarea psihanalitică,
neinteresată de cercetare, a psihiatriei americane a epocii. Kety avea să reziste însă numai un an la șefia
institutului, dat fiind că interesul lui pentru cercetare neuroștiințifică fundamentală era prea puțin interesant
pentru psihiatria universitară din epocă.

Insel își începuse mandatul într-o manieră autoritară, declarând explicit toate patologiile psihiatrice ca fiind
patologii neurologice și insistând ca politica de cercetare și finanțare a institutului să se orienteze cât mai strict
către neuroimagistică și genetică, reducând masiv finanțarea pentru psihologie clinică și psihoterapie, spre
nemulțumirea comunității psihologice. Cele 20 de miliarde de dolari aveau să producă o lungă serie de studii
publicate în jurnalele de top ale lumii științifice, în încercarea de a aduce nou apăruta medicină de precizie în
psihiatria care încă nu se hotărâse întru totul ce patologii tratează și cum. Politica NIMH avea să producă
încercarea de a influența profund Asociația Psihiatrică Americană, care în anii 2000 lucra la reconstrucția
manualului DSM, în speranța de a transforma psihiatria într-o disciplină medicală similară cu neurologia,
reformulând masiv diagnosticele psihiatrice în manieră bottom-up, dinspre cercetarea fundamentală către
clinică, la fel ca în restul medicinii, nu invers, cum fusese construită psihiatria de-a lungul secolului 20. Una
dintre puținele victorii are cercetării din neuroștiință avea să fie înființarea unei categorii de sine stătătoare a
tulburării obsesiv compulsive, grupată până atunci în grupul tulburărilor de anxietate, având în vedere că
studiile de neuroimagistică dezvăluiseră anomaliile cortico-striate ca fiind caracteristice tulburării obsesiv-
compulsive, iar tulburările de anxietate erau caracterizate mai degrabă prin anomalii fronto-temporo-limbice,
anxietatea din tulburarea obsesiv-compulsivă fiind mai degrabă un produs al compulsiilor decât un simptom
central.
Cu toate acestea, desprinderea tulburării obsesiv-compulsive ca și categorie distinctă de diagnostic n-a avut
niciun impact major în practica clinică, ci a fost relevantă numai pentru înțelegerea fundamentală a
neurobiologiei din spatele celor două tipuri de categorii nosologice. În plus, Asociația Psihiatrică Americană s-
a arătat rezistentă și puțin interesată de fundamentarea DSM-ului 5 pe baze neurobiologice, anunțând că
manualul avea să fie în continuare construit în mare parte în funcție de criterii comportamentale, stabilite prin
consensul clinicienilor. Ca urmare, înainte de lansarea DSM-5, Insel avea să publice un comunicat de presă în
care anunța încheierea colaborării dintre APA și NIMH, acesta din urmă renunțând definitiv la criteriile DSM
pentru includerea pacienților în studii de genetică și neuroimagistică, având în vedere că acestea se potriveau
foarte puțin cu maniera de a trasa diagnostice în manieră comportamentală a psihiatriei clinice și psihologiei.
Prin urmare, 11 din cei 13 ani din mandatul lui Insel la NIMH fuseseră în realitate petrecuți cheltuind sume
importante în încercarea de a corela criteriile comportamentale din DSM cu neuroimagistica și genetica, în
speranța că acestea se vor potrivi destul încât să ducă la o reformă “soft” a psihiatriei, care să necesite
schimbări moderate în DSM, nu dramatice, cum s-a dovedit în cele din urmă. Însă, dovada putea veni numai
prin încercările repetate, care aveau să coste mult mai mult decât studiile de psihologie, dat fiind că
neuroimagistica și genetica nu se mărginesc să aplice chestionare și să ruleze corelații și regresii pe datele
astfel obținute.

Pe de altă parte, puțină lume remarcase contextul în care Insel făcuse remarca legată de fondurile cheltuite
aparent fără succes de NIMH. Plecat din 2015 de la NIMH, Insel conducea la vremea respectivă Mindstrong,
un start-up privat preocupat de cercetare pentru a dezvolta o aplicație mobilă care-și propunea
monitorizarea stării de sănătate mentală prin măsurarea modificărilor vitezei cu care utilizatorii cu
probleme psihiatrice dădeau scroll pe ecranele telefoanelor. De fapt, reportajul Wire care conține citatul era,
în mare parte, o reclamă la start-up-ul condus de Insel, despre care cititorii erau asigurați că e viitorul în
materie de sănătate mentală și cercetare. După plecarea de la NIMH, palmaresul de publicații academice al
lui Insel avea să scadă brusc și considerabil, iar aplicația revoluționară de la Mindstrong n-a mai apărut nici
până astăzi. În realitate, ideea că doar 13 ani de cercetare fundamentală ar fi putut vreodată revoluționa o
practică de 100 de ani guvernată de un manual care aduce APA cel puțin 5 milioane de dolari pe an era
absurdă. Un review din 2013 concluzionase că majoritatea studiilor din neuroștiință au o putere statistică
inutil de mică, iar un al doilea, axat doar pe cercetarea din RMNf, ajunsese la concluzii similare. De fapt,
creierul rămâne în cea mai mare parte ultima frontieră a științei contemporane, încă bine ascuns după o cutie
craniană la fel de greu de trecut ca și bariera hematoencefalică după care e izolat. Pentru psihologia
clinică, ideea că miliardele de dolari de eforturi neuroimagistice și genetice puteau fi înlocuite cu chestionare
și interviuri clinice a continuat, în ciuda bunului simț elementar, să pară plauzibilă. Domeniul a aplaudat
declarația lui Insel ca dovadă clară că patologiile psihiatrice sunt constructe sociale, fără nicio legătură
cu neurofiziologia și a continuat să-și vadă de chestionare.

Cu toate acestea, deși adesea vândute (fals) ca fiind lipsite de efecte secundare, cercetarea tratamentelor
psihologice suferă din plin de aceleași probleme care parazitează și neuroștiința: putere statistică scăzută,
QRP-uri masive care forțează rezultate aproape exclusiv confirmatorii, iar finanțarea e irosită pe studii cu
concluzii trase încă de dinaintea colectării datelor. Deși încă nu există estimări similare pentru psihologia
clinică, Russell T. Warne estima în 2020 că numai pe domeniul eșuat sistematic la replicare al amenințării
stereotipurilor din psihologia socială se irosiseră nu mai puțin de 67.5 milioane de dolari. Departe de cele 20 de
miliarde ale NIMH, însă Ioana Cristea și Florian Naudet fac o trecere extinsă în revistă a practicilor care
produc pierderi care ar putea rivaliza cel puțin cu cei 67 de milioane estimați de Warne pentru
amenințarea stereotipurilor, un comentariu util pentru entuziaștii care au concluzionat absurd și hazardat că
eșecul neuroștiinței înseamnă pe cale de consecință succesul psihoterapiei.
Behaviour Research and Therapy 123 (2019) 103479

Contents lists available at ScienceDirect

Behaviour Research and Therapy


journal homepage: www.elsevier.com/locate/brat

Increase value and reduce waste in research on psychological therapies T


a,b,∗ c
Ioana A. Cristea , Florian Naudet
a
Department of Clinical Psychology and Psychotherapy, Babes-Bolyai University, Republicii 37 Street, Cluj-Napoca, Romania
b
Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA 94305, California, USA
c
University Rennes, CHU Rennes, Inserm, CIC 1414 (Centre d’Investigation Clinique de Rennes), F- 35000, Rennes, France

ARTICLE INFO ABSTRACT

Keywords: A seminal Lancet series focused on increasing value and reducing waste in biomedical research, providing a
Research waste transferrable template to diagnose problems in research. Our goal was to document how some of these sources of
Open science waste apply to mental health and particularly psychological treatments research. We synthesize and critically
Reproducibility evaluate empirical findings in relation to four major sources: i) defining research priorities; ii) research design,
Transparency
methods and analysis; iii) accessibility of research information; iv) accuracy and usability of research reports. We
Psychological treatment
Psychotherapy
demonstrate that each source of waste considered is well-represented and amply documented within this field.
meta-Research We describe hype and insufficient consideration of what is known in defining research priorities, persistent risk
Risk of bias of bias, particularly due to selective outcome reporting, for psychotherapy trials across mental disorders, in-
Publication bias tellectual and financial biases, direct and indirect evidence of publication bias, largely inexistent adoption of
Data sharing data sharing, issues of multiplicity and fragmentation of data and findings, and insufficient adoption of reporting
guidelines.
We expand on a few general solutions, including supporting meta-research, properly testing interventions to
increase research quality, placing open science at the center of psychological treatment research and remaining
vigilant particularly regarding the strains of research currently prioritized, such as experimental psycho-
pathology.

In 2014 a seminal Lancet series focused on increasing value and management is efficient (Salman et al., 2014); iv) research information
reducing waste in biomedical research. In the prefacing commentary, is fully accessible (Chan et al., 2014) and finally v) research reports are
M. R. Macleod et al. (2014) start from a case example both straight- unbiased and useable (Glasziou et al., 2014). Each report delineates
forward and striking: out of 1575 reports about cancer prognostic various steps and initiatives for reducing waste and increasing value,
markers published in 2005, 96% claimed to have discovered at least with several of these subsequently expanded in similar contributions,
one significant prognostic marker (Kyzas, Denaxa-Kyza, & Ioannidis, such as the “Manifesto for reproducible science” (Munafò et al., 2017).
2007). Yet only a minority of markers were replicated in subsequent Beyond the content of the included papers, the Lancet series sparked
research, with adoption in clinical practice similarly limited (Anderson, a movement for reducing waste and increasing value in research. This is
2010). M. R. Macleod et al. (2014) maintain that such a pattern of in- reflected in initiatives like the REWARD Alliance (http://
itially promising findings that nonetheless fail to engender tangible rewardalliance.net/), a dedicated platform gathering documentation,
healthcare improvements is the rule rather than the exception across information, resources and a string of initiatives, such as working
biomedical research. Avoidable waste in the production and reporting groups, prizes, blogs and other tools. More generally, the Lancet series
of biomedical research had previously been estimated at a staggering papers coalesce in an exportable framework for diagnosing problems
85% (Chalmers & Glasziou, 2009). The papers in the Lancet series ex- with research conducted in other healthcare domains. Though examples
plore possible reasons for the failure to produce transformative and in the reports occasionally touch on mental health or psychotherapy
useful research, with each examining a potential source of “research research, their main emphasis rests with biomedical research and
waste”. The reports discuss whether i) decisions about what research to treatments. Still, a closer examination of specific subfields allows de-
fund are based on questions relevant to users of research (Chalmers tecting particular problems and suggesting tailored solutions.
et al., 2014); ii) appropriate research designs, methods and analyses are Therefore, in this opinion piece, our goal is to document how some of
being employed (Ioannidis et al., 2014); iii) research regulation and the sources of waste identified apply to research in mental health, and


Corresponding author. Department of Clinical Psychology and Psychotherapy, Babes-Bolyai University, Republicii 37 Street, 400015, Cluj-Napoca, Romania.
E-mail addresses: ioana.alina.cristea@gmail.com (I.A. Cristea), floriannaudet@gmail.com (F. Naudet).

https://doi.org/10.1016/j.brat.2019.103479
Received 23 February 2019; Received in revised form 25 July 2019; Accepted 9 September 2019
Available online 12 September 2019
0005-7967/ © 2019 Elsevier Ltd. All rights reserved.
I.A. Cristea and F. Naudet Behaviour Research and Therapy 123 (2019) 103479

in particular on psychological treatments. in non-laboratory settings, the very contexts most relevant for transla-
tion to practice. Ironically, not only did translation from the experi-
1. Are research decisions based on questions relevant to users of mental studies to treatments useful to patients prove largely a failure,
research? but even the notion of surpassing the proof-of-concept stage has re-
cently been questioned. One meta-analysis failed to find evidence that
Though any research aimed at finding, testing or understanding the clinically anxious individuals enrolled in ABM trials displayed atten-
nature and treatment of mental disorders is potentially valuable, lim- tional bias to begin with (Kruijt, Parsons, & Fox, 2019).
ited resources set constraints and impose defining priorities. Mental Another illustrative example is memory reconsolidation, the notion
health funding is usually limited (Hazo et al., 2016), though with sig- that postretrieval learning can be employed to modify or even erase
nificant discrepancies across countries (Hazo et al., 2017). Funding calls existing memory traces, effectively affording “rewriting” of existing
and reports make it clear certain topics are prioritized. For instance, the memories. This theory stems from basic research claiming to demon-
MQ report on mental research funding in the UK showed that about a strate reconsolidation effects, first in animals (Nader, Schafe, & Le
quarter (23.7%) of the annual mental health funding goes into “basic Doux, 2000), and subsequently in humans (Schiller et al., 2009; Walker,
psychology and neuroscience” and about half goes into research on Brakefield, Hobson, & Stickgold, 2003). Should the initial laboratory
underpinning (i.e., brain structure and functioning in healthy in- findings prove reproducible and translatable, treatment implications
dividuals) and aetiology (MQ, 2019). The Wellcome Trust, one of the would be immense, particularly in disorders where intrusive memories
largest non-profit funders, recently announced a commitment of £200 play a central part, like post-traumatic stress disorder (PTSD).
million for research on discovering and mobilizing treatment mechan- Yet both reproducibility and translation are doubtful. The initial
isms (Thielking, 2019). experimental results demonstrating memory reconsolidation effects in
Similar research priorities were described in a Lancet Psychiatry humans failed both procedural and conceptual replication by in-
Commission (Holmes et al., 2018), which delineated an agenda for dependent groups (Hardwicke, Taqi, & Shanks, 2016; Kindt & Soeter,
research on psychological treatments. The fundamental premise of this 2013). In terms of translation, a recent randomized trial attempted to
programmatic document was that psychological treatment research had mitigate future post-traumatic symptoms in motor vehicle collision
strayed away from the mechanistically informed approaches that were survivors in an emergency setting (Iyadurai et al., 2018). The inter-
the foundation for some of the current effective treatments, (e.g., ex- vention, founded on a theoretical mix between memory consolidation
posure and response prevention for obsessive-compulsive disorder), to and reconsolidation, consisted of a brief trauma reminder followed by
instead reorient itself towards modifying or adapting existing treat- playing the computer game Tetris. It had already been tested in the
ments. Owing to this shift, the authors contend, even if effective psy- laboratory on healthy participants who were induced symptoms labeled
chological treatments are available across a variety of settings and as trauma-analogue, by means of watching a traumatic film, and it had
populations, there is little knowledge about their mechanisms of action. showed promising effects in reducing memory flashbacks over the
In response, the authors of the Commission champion the burgeoning course of a week (Holmes, James, Kilford, & Deeprose, 2010). The
field of experimental psychopathology, broadly encompassing studies subsequent trial on trauma victims showed a similar reduction in
in which the experimental manipulation of a purported mechanism of flashbacks over 1 week, on the basis of which the authors declared the
treatment leads to symptom change. This field promises to identify “key intervention a “compelling translation of previous laboratory findings”
processes that maintain or change aspects of psychopathology” (Holmes (p.680) and likened it to a “cognitive therapeutic vaccine”. In a com-
et al., 2018) (p. 244), as well as to determine which processes are mentary, we disputed this characterization, questioning the magnitude
modifiable and could consequently represent appropriate treatment and stability of the observed effects (Cristea, Naudet, Shanks, &
targets. Hardwicke, 2018), which were largely influenced by outliers, reduced
Similarly to research on prognostic markers for various cancers, in a more robust analysis, and, critically, not observed at 1-month
experimental psychopathology unlocks a multitude of exciting possi- follow-up. Furthermore, the stakes of a successful translation would
bilities. Still, analogously to prognostic markers, many modest findings involve not just reducing post-trauma flashbacks, but delaying or
coming from experimental psychopathology, with as yet minimal avoiding the onset of PTSD altogether . To this purpose, a larger, phase
treatment implications, have been replaced by hype. Several of the most III trial would be necessary, employing a heterogeneous sample of
touted and oft-cited developments, mentioned in the Commission as trauma victims, pre-screened for having a high level of post-traumatic
prototypical research in this field, have not been replicated in sub- flashbacks, with reliably measured primary outcomes and longer
sequent, independently conducted, studies or had little impact for follow-up durations, including for monitoring adverse effects. It is
translation to clinical practice. One revealing example is cognitive bias currently unclear whether conducting such a costly and logistically
modification, a class of interventions developed in the laboratory and complex trial is a judicious investment. Its foundation would consist of
broadly encompassing techniques for the direct manipulation of a non-replicated experimental findings, favorable results observed in
target cognitive bias with the goal of obtaining symptom change. healthy participants in a laboratory context that has little in common
Attention bias modification (ABM) interventions, the most prominent with real trauma, and transient effects of disputable magnitude in one
example in this class, are premised on experimental studies supporting proof-of-concept trial on victims of one type of trauma. Given its con-
the existence of a causal relationship between attentional bias and siderable risk of producing null results and not meaningfully informing
anxiety (C. MacLeod & Mathews, 2012; Van Bockstaele et al., 2014). clinical practice, such a trial would rather amount to research waste.
Based on these largely laboratory-based studies and early small trials, Similar doubts apply to other promising experimental psychopathology
ABM interventions were hailed as “having passed the proof-of-concept findings.
stage” (C. MacLeod, 2012) (p.118), effective in reducing dysfunctional When confronted with replication or translation failures, re-
anxiety (C. MacLeod & Mathews, 2012) and nothing short of a “cognitive searchers often hypothesize potential moderators or boundary condi-
vaccine” for depression (Browning, Holmes, Charles, Cowen, & Harmer, tions to account for small, elusive or inconsistently observed effects.
2012). Yet in parallel with growing excitement, larger clinical trials This is certainly a distinct possibility in some cases, but in others the
accrued and mostly produced negative results. Subsequent meta-ana- original results were probably artefacts. Still, moderators or boundary
lyses showed that ABM and other bias modification approaches had conditions are usually inferred post-hoc and would also need to be
limited, if any, effects, for anxiety and depression outcomes, in both examined empirically and prospectively (Cristea, 2018a; Hardwicke &
adult (Cristea, Kok, & Cuijpers, 2015) and young populations (Cristea, Shanks, 2016). More importantly, it remains questionable whether
Mogoase, David, & Cuijpers, 2015). Moreover, effects were particularly priorities in research on psychological interventions should coalesce
small in clinical populations and when the intervention was delivered around effects of this size and stability. Of course, it is impossible to

2
I.A. Cristea and F. Naudet Behaviour Research and Therapy 123 (2019) 103479

know a priori whether a research idea will lead to massive progress or to psychological interventions. We described three categories of beha-
fleeting, non-replicable, or difficult to translate effects. As is often viours, ranging from those clearly inadequate to those problematic
stated, absence of evidence is not evidence of absence. We are not de- under certain circumstances. The first one involved exploiting the de-
nying the importance of attempting to identify treatment mechanisms, sign, implementation and reporting of a trial so as to “prettify” the
nor are we arguing that exploring novel ideas should not be encouraged results and make an intervention appear more effective. Examples in-
or is in itself a source of waste. However, we are concerned waste might cluded: i) not concealing the allocation of the participants from the
result from two other sources. First, the unjustified hype that usually research staff who could, unwittingly or not, use this information to
accompanies research in areas like experimental psychopathology influence the delivery of the intervention; ii) not using blind assessors
might be counterproductive. As with the drug development pipeline, of outcome, hence allowing them to be influenced by knowledge of
many of the exciting pre-clinical or early clinical findings prove to be assignment and possibly judge participants in the intervention group as
dead-ends (Contopoulos-Ioannidis, Ntzani, & Ioannidis, 2003). If they more improved; iii) including a stack of outcome measures and selec-
are conveyed with forceful and often grandiose language, like by ana- tively reporting the outcomes with positive findings; iv) reporting
logy with vaccines, high and unwarranted expectations are set up in analysis on completers instead of all randomized participants (i.e., in-
other interested researchers and the general public. The untoward tent-to-treat), since completers are most likely to have benefited from
consequences might include reticence in researchers, reviewers and treatment and therefore remained in the study; and v) holding back the
journal editors to credit and publish negative findings, as well as pre- publication of negative results. The second category encompassed
mature implementation of an insufficiently tested intervention, which choices in research design that can be appropriate in the early stages of
may be ineffective or even harmful. Second, continued efforts to research, but become questionable when studies with these character-
“rescue” a hypothesis in the face of accumulating negative results, istics dominate the evidence base, such as: i) using waiting list controls;
simply because initial findings were promising, is also counter- ii) employing small samples; iii) not comparing the intervention with
productive. Instead, resource limitations would dictate reorienting ef- others known to be effective. Finally, the third category involved be-
forts to other ideas. haviours in themselves not inadequate, and which many researchers
Another source of waste in setting research priorities is ignoring view as the mark of competence and commitment in developing and
what is already known or being researched (Chalmers et al., 2014). administering interventions. These include various ways of enhancing
Often, research questions are grounded in a narrative synthesis, which users' expectations, like expressing your own confidence in the inter-
facilitates cherry-picking ideas to support preferred hypotheses. One vention, using a variety of methods to convey trust and enthusiasm,
possible preventing strategy is requiring that a systematic review and, if such as writing targeted books, giving lectures at conferences and
feasible, a meta-analysis preclude any investment or funding decisions, events with professionals, giving interviews in the media, or proposing
so as to ensure that the question has not already been answered sa- anecdotical evidence like patient testimonials and success stories.
tisfactorily. Determining which questions have value is more challen- Though not intrinsically inappropriate, the intellectual investment in an
ging, since allocating resources to any priority comes at the cost of approach could function as a bias, reducing vigilance in detecting in-
divesting them from alternatives. For example, the focus on mechanistic appropriate choices in design, analysis and reporting, particularly in the
approaches to psychological treatments could distract from other, presence of findings favorable for the intervention. Along the same
pervasive, determinants of mental disorders, like poverty (Deighton lines, Meichenbaum and Lilienfeld (2018) proposed a 19-item “psy-
et al., 2019), income inequality (Patel et al., 2018) or living in a conflict chotherapy hype checklist”, describing warning signs that an inter-
zone (Jayasuriya, Jayasuriya, Tay, & Silove, 2016). Specifically, pro- vention's efficacy has been significantly exaggerated, grouped under
posing a simple behavioral intervention like the Tetris game to tackle promotion /marketing, and research evidence.
intrusive memories in refugees from warzones like Syria (Holmes et al., Are these counter-suggestions illustrative of the reality “on the
2017) might deflect from the disabling and enduring consequences of ground” in research on psychological interventions? Meta-epidemiolo-
war-related trauma (Silove et al., 2014). gical investigations that systematically look at patterns in collections of
Waste in setting research priorities also ensues when users of re- studies, like systematic reviews, surveys, meta-analyses or umbrella
search are not involved (Chalmers et al., 2014). Patient input is crucial reviews (i.e., systematic reviews of systematic reviews) appear to sup-
regarding key outcomes, type of treatment preferred (i.e., psycholo- port this thesis, indicating endemic problems. For clinical trials, po-
gical, pharmacological), or the design (i.e., superiority, equivalence, tential markers of questionable practices in the design, methods or
non-inferiority) that answers the most important question from a pa- analysis are represented by the risk of bias (RoB) domains (Higgins
tient perspective (Holmes et al., 2018). Pragmatically, there are direct et al., 2011). These include: (i) adequate sequence generation; (ii)
and immediate benefits of patient involvement in trials. For example, a concealment of allocation; (iii) blinding of outcome assessors; (iv)
meta-analysis showed that patient and public involvement, especially dealing with incomplete data; and (v) selective reporting. Assessing
for people with lived experience, improved participant enrolment trial risk of bias is standard practice in a meta-analysis, offerring us a
(Crocker et al., 2018). Patient preferences have still not been suffi- snapshot of the state of the field. Across meta-analyses, a consistent
ciently probed in psychological treatment research. Most analyses so far finding is that there are few trials of psychological interventions with
have focused on defining target outcomes that matter to patients low RoB across all domains. High or uncertain RoB seems to be ubi-
(Cuijpers, 2019). But whether patients favour many of the currently quitous across the whole range of treatment approaches and disorders,
prioritized mental health topics remains an open question. For instance, such as cognitive behaviour therapy (CBT) for major depression and
how do deeply traumatized refugees feel about being offered a simple anxiety disorders (Cuijpers, Cristea, Karyotaki, Reijnders, & Huibers,
behavioural intervention like the Tetris game? What we are advocating 2016), interpersonal therapy for mental disorders (Cuijpers, Donker,
for is a vision of research as an enterprise that advances collaboratively Weissman, Ravitz, & Cristea, 2016), virtual reality-based therapies
with patients instead of the traditional paternalistic view of research (Fodor et al., 2018), or psychological treatments in general for de-
done for patients. pression (Cuijpers, Karyotaki, Reijnders, & Ebert, 2019), social anxiety
(Mayo-Wilson et al., 2014), panic disorder (Pompoli et al., 2016), ob-
2. Appropriate research design, methods, and analysis? sessive-compulsive disorder (Skapinakis et al., 2016), PTSD (Bisson,
Roberts, Andrew, Cooper, & Lewis, 2013), schizophrenia (Bighelli et al.,
A few years ago, we provocatively proposed a set of guidelines on 2018), bipolar disorder (Chatterton et al., 2017), borderline personality
how to prove a psychological treatment is effective even when in reality disorder (Cristea et al., 2017) or eating disorders (Slade et al., 2018).
it is not (Cuijpers & Cristea, 2016). We had intended these as a tongue- An inverse relationship between study quality, conceptualized as low
in-cheek recipe of how not to design, conduct and report trials of overall RoB, and the magnitude of effects was consistently shown either

3
I.A. Cristea and F. Naudet Behaviour Research and Therapy 123 (2019) 103479

as a direct relationship in meta-regression analyses (Barth et al., 2013; specific or minimal contact interventions are more appropriate control
Cristea et al., 2017), or as a considerable effect size reduction in ana- options for newly developed psychological treatments.
lyses restricted to studies with low RoB (Cuijpers, Cristea, et al., 2016; Small study effects, the tendency of interventions to appear more
Cuijpers et al., 2019). beneficial in smaller studies (Higgins & Green, 2011), are also pervasive
A particularly critical and difficult to assess bias is selective outcome in research on psychological interventions. An umbrella review of meta-
reporting, which cannot be detected in the absence of complete and analyses of randomized controlled trials of psychotherapy for a variety
accurate information about what investigators planned to do and why. of outcomes including symptoms of mental disorders, distress and
Prospective registration of trial protocols in publicly accessible re- physical outcomes, found evidence of small study effects for about
gistries is therefore an indispensable prerequisite. Selective reporting 30%–40% of the included meta-analyses, depending on the method
bias is closely linked with another source of waste, namely the usability used (Dragioti, Karathanos, Gerdle, & Evangelou, 2017). Conversely, an
of research reports and protocols, discussed in a further section. It en- umbrella review of psychological interventions for pain reduction
compasses an array of inappropriate reporting behaviours, ranging across a range of conditions, most of them physical, did not report
from not reporting certain planned outcomes at all, to selecting which evidence of small study effects (Markozannes et al., 2017).
of the measurements of an outcome to report, to modifying the prior- Risk of bias, weak control groups and small study effects are not
itization of planned outcomes or measurements (i.e., downgrading a circumscribed to a few trials and probably act synergistically in dis-
primary outcome to secondary or upgrading a secondary outcome to torting estimates. In an interesting analysis, Cuijpers et al. (2019) tried
primary). Crucially, reporting decisions are usually based on the di- to estimate the “bias-free” effectiveness of psychological treatments for
rection, magnitude or significance of an effect estimate, so as to elevate depression, by sequentially eliminating each source of bias and re-
positive results, and minimize or eclipse negative ones. As a con- calculating the pooled treatment effect. They started from an initial
sequence, the intervention appears more effective or less harmful than unadjusted estimate (Hedges’ g) of 0.63 (325 comparisons), which
it really is. Selectively reported trials are included in research synthesis dropped to 0.51 (179 comparisons) when comparisons with the waiting
and have a lasting distorting effect on the evidence base. This perni- list were excluded, 0.38 (71 comparisons) when only studies with low
cious bias seems to be rampant in research on psychological interven- risk of bias across domains were included, and respectively 0.31 (84
tions. A study of selective reporting in depression trials found that only comparisons) when a method was used to attempt to correct the pooled
20% of the CBT trials had been properly registered, and only around effect for the presence of small study effects.
14% both properly registered and reported (Shinohara et al., 2015). Other biases are also prevalent, though their distorting influence of
Another investigation of psychotherapy trials published in five top-tier outcomes is difficult to ascertain. Contamination by researcher alle-
dedicated journals in clinical psychology showed that under a quarter giance, the belief of the investigators in the superiority of one of the
of the surveyed trials had been prospectively registered and that this treatments studied, has long been debated (Munder, Fluckiger, Gerger,
proportion dropped to just under 12% when the accuracy of registration Wampold, & Barth, 2012). Over 60% of the trials included in meta-
(i.e., no ambiguities regarding the primary outcome measure or the analyses of psychotherapy were appraised as allegiant in a systematic
timeframe of assessment) was taken into account (Bradley, Rucklidge, & evaluation (Dragioti, Dimoliatis, & Evangelou, 2015). However, de-
Mulder, 2017). Under 5% of trials were judged free of selective out- fining and reliably assessing which researcher behaviors constitute al-
come reporting, i.e., had no discrepancies between the registration and legiance remain challenging aspects (Yoder, Karyotaki, Cristea, van
the report. Duin, & Cuijpers, 2019). Still, some meta-analyses of psychological
Another ubiquitous and often inappropriate research design choice interventions, using diverse assessment procedures, reported an asso-
is resorting to weak control groups, such as the waiting list. Based ciation between the presence of allegiant investigators and outcomes
largely on comparisons to this condition, estimates of effects for psy- (Munder, Gerger, Trelle, & Barth, 2011; D. T. Turner, van der Gaag,
chological interventions can appear very large. For instance, a meta- Karyotaki, & Cuijpers, 2014). A survey of meta-analyses of RCTs of
analysis of CBT (Cuijpers, Cristea, et al., 2016) found very large effects psychotherapy revealed that effects were larger by almost 30% for trials
when compared to waiting list controls, ranging from 0.85 for gen- involving allegiant investigators (Dragioti, Dimoliatis, Fountoulakis, &
eralized anxiety disorder to 0.96 for panic disorder, and 0.98 for social Evangelou, 2015). However, allegiance effects are observational and
anxiety disorder and, major depressive disorder. For depression, meta- might also reflect reverse causality: investigators become allegiant be-
analyses indicated that participants assigned to the waiting list showed cause they are exposed to treatments that are truly more effective
diminished response in terms of symptom reduction, and that this (Wilson, Wilfley, Agras, & Bryson, 2011). Furthermore, other variables
condition was significantly inferior to other types of control, like such as therapist competence or treatment integrity could confound the
treatment as usual or placebo (Barth et al., 2013; Furukawa et al., 2014; relation between allegiance and outcomes. For instance, trials with al-
Khan, Faucett, Lichtenberg, Kirsch, & Brown, 2012). Some researchers legiant investigators did not result in more favorable outcomes when
even likened the waiting list to a nocebo (Furukawa et al., 2014). For treatment integrity was assessed in a survey of meta-analyses (Dragioti,
social anxiety disorder, a meta-analysis (Steinert, Stadter, Stark, & et al., 2015). Similarly, a randomized trial comparing an IPT and a CBT
Leichsenring, 2017) showed a pooled within group effect (Hedges' g) of intervention for binge eating disorders found no evidence of allegiance
0.13 for the waiting list arms, considerably smaller than that of active bias though ostensibly allegiant investigators were involved (Wilson
treatments (Hedges' g = 0.88). In another meta-analysis (Fodor et al., et al., 2011), a result the authors partly attribute to high treatment
2018) of virtual reality-based intervention for anxiety outcomes across integrity. Finally, investigator background, such as specialization or
multiple anxiety disorders, the effects versus waiting list controls were affiliation (i.e., a biomedical or psychosocial concentration), might also
very large (Hedges' g = 0.90). We argued that the waiting list is an engender an intellectual bias, though we did not find corroborating
inadequate benchmark for estimating the effectiveness of psy- evidence in a meta-analysis of psychotherapy and pharmacotherapy for
chotherapy, at least for depression (Cristea, 2018b). Due to the dis- depression (Cristea, Gentili, Pietrini, & Cuijpers, 2017a).
advantages this type of control carries in terms of possible harm to Beyond the intellectual biases, more direct, financial conflicts of
patients and an overestimation of an intervention's effects, we contend interests (COIs) in relationship to psychological interventions may have
it should be used sparingly (Cristea, 2018b), possibly as part of designs remained largely under the radar (Cristea & Ioannidis, 2018). We
that attempt to maximize patients' time while waiting for therapy charted a variety of possibly applicable financial COIs, ranging from
(Lovell et al., 2017) or that allow them to refuse waiting list assignment commercial companies developing and testing psychological interven-
and still remain in the trial (i.e., equipoise-stratified). At least for the tions, a situation akin to the pharmaceutical industry, to less straight-
more prevalent mental disorders discussed above (depression, anxiety), forward cases of potential financial gain, such as involvement with
there is an excess of trials with waiting list arms. Supportive, non- academic spin-offs, temporary participations in commercial ventures,

4
I.A. Cristea and F. Naudet Behaviour Research and Therapy 123 (2019) 103479

for-profit support activities like training, and involvement with foun- Linardatos, Tell, and Rosenthal (2008) showed that among 74 anti-
dations or non-profits. There is a dearth of systematic evaluations as to depressant trials, 31% remained unpublished. Separate meta-analyses
whether these strains of financial COIs distort effectiveness estimates of the FDA and journal datasets showed publication bias accounted for
for psychological interventions. a relative effect size increase of 32%. Thirty-seven out of the 38 studies
Another related bias, sponsorship bias, refers to the notion that in- characterized by the FDA as having positive results were published. In
dustry funding of trials is associated with findings favoring the sponsor's contrast, except for 3 studies, trials viewed by the FDA as having ne-
product. The study of this bias was traditionally confined to trials of gative or questionable results were mostly not published (22 studies) or
pharmacological treatments, but we reported corroborating evidence in published in a way that conveyed a positive outcome (11 studies).
head-to-head RCTs of psychotherapy versus pharmacotherapy (Cristea, An authority similar to the FDA does not exist for psychological
Gentili, Pietrini, & Cuijpers, 2017b). Industry-funded depression trials interventions, making direct assessments of publication bias more
tended to report more favorable results for pharmacotherapy versus complex and often unfeasible. In one of the rare evaluations, Driessen,
psychotherapy, a pattern not present in non-industry funded trials. Hollon, Bockting, Cuijpers, and Turner (2015) identified US National
Trials where authors had financial COIs linked to the pharmaceutical Institutes of Health grants awarded for randomized trials of psy-
industry showed a similar pattern (Cristea et al., 2017a; 2017b). chotherapy for depression and matched grants with published reports.
For grants that had not resulted in publications, data was requested
3. Fully accessible research information? from investigators and both published and unpublished findings were
meta-analysed. About a quarter of the publicly funded grants did not
The lack of full and accurate information about a completed study, result in publications, an estimate of the extent of publication bias.
ranging from its planning and protocol to its full set of results, is an- Unpublished studies resulted in a small and non-significant effect size
other critical source of research waste. Two major categories of pro- (Hedges’ g) of 0.20, and their addition to the published studies reduced
blems refer to (1) findings that are never published, mostly because the summary effect by 25%, to g = 0.39. Notably, this investigation
they would reflect badly on an intervention, its developers, sponsors or closely parallels previously described findings regarding the selective
sometimes an entire research field (i.e., publication bias) and (2) data publication of antidepressants trials (E. H. Turner et al., 2008).
underlying reported findings not being made accessible to other re- The landmark work of E. H. Turner et al. (2008) also demonstrates
searchers or the public (i.e., data sharing). how the conclusions of published papers can deviate substantially from
Most evidence of publication bias in research on psychological in- what the data really show. Yet without access to datasets, deviations are
terventions is indirect and the result of an extrapolation of observed impossible to detect. An example perhaps even more dramatic example
small study effects described in the previous section. For instance, for of how lack of access to the complete trial data can entirely alter the
trials of psychological therapies compared to control conditions for interpretation of the findings is provided by the re-analysis of
adult depression, Cuijpers, Smit, Bohlmeijer, Hollon, and Andersson SmithKline Beecham's Study 329, comparing paroxetine, imipramine
(2010) used a series of tests to document strong and significant in- and placebo for adolescent unipolar depression (Le Noury et al., 2015).
dication of small study effects, which they attributed to publication This re-analysis relied on clinical study reports previously not publicly
bias. Analyses restricted to trials of CBT, which represented the largest available and showed that none of the drugs outperformed placebo, but
share of the data, pointed to a similar pattern. Another possible proxy both were also associated with significant increases in serious harms,
for publication bias is the test of excess significance, which determines starkly contradicting the published report (Keller et al., 2001). In-
whether the number of statistically significant results in a pool of stu- accessibility of complete individual patient data (IPD) can thus have
dies is too high, based on some plausible assumptions about the mag- catastrophic consequences.
nitude of the true effect (Ioannidis & Trikalinos, 2007). Using this Therefore, IPD underlying reported findings are another crucial
method, an excess of significant findings was reported for trials of class of information. Data sharing could potentially fulfil a multitude of
psychotherapy for depression, and the pattern was maintained in sen- key functions, with far-reaching implications for patients, clinicians,
sitivity analyses restricted to trials with a CBT arm (Flint, Cuijpers, researchers and other users of scientific literature. It would presumably
Horder, Koole, & Munafò, 2015). More broadly, an umbrella review of contribute to resolving the above-mentioned flaws in the current system
meta-analyses of psychotherapy for various symptom, distress and of communicating research results, such as selective reporting, distor-
physical outcomes reported excess significance in around 40% of the tion of evidence and lack of reproducibility. In addition, research value
included meta-analyses (Dragioti et al., 2017). However, another um- increases with greater transparency and the opportunity for external
brella review of psychological interventions for pain found very limited researchers to re-analyse (Ebrahim et al., 2014; Naudet et al., 2018),
evidence of excess significance, in just 7% of the included meta-ana- synthesize (e.g., IPD meta-analyses) (Tannenbaum et al., 2018), or
lyses (Markozannes et al., 2017). build upon previous data (including secondary analyses and methodo-
Nevertheless, both small study effects and excess significance offer logical work). To take just one example: our own commentary (Cristea
only indirect and speculative evidence toward publication bias and et al., 2018) of the Iyadurai et al. (2018) Tetris trial was possible be-
have their own limitations. Though publication bias is often the most cause the authors shared their data, thus allowing for an independent
plausible cause, small study effects can also result from other factors, analysis and, based on it, a divergent interpretation. Not only does this
such as clinical heterogeneity between patients in large and small trials, practice stimulate vigorous scientific debate, a cornerstone of scientific
a mathematical artefact or mere coincidence (Rücker, Carpenter, & progress, but it also prevents the waste of collecting new data to explore
Schwarzer, 2011). The excess significance test relies on assumptions the same question.
about the magnitude of the true effect, which in practice is often Yet, for all of its expected benefits and despite recent re-
equated with the effect from the largest study. However, there are few commendations from the International Committee of Medical Journal
large trials of psychological interventions. Moreover, a large study can Editors (ICMJE. Clinical Trials- Data sharing, 2019), the adoption of
have design, implementation or reporting flaws, and hence be at high data sharing practices remains very low, by both authors and journals.
risk of bias. The summary effect of a meta-analysis is another possible Cross-sectional analyses of the biomedical literature showed derisory
estimation of the true effect; yet, if there is indeed excess significance, it rates of data sharing, ranging from 0 (Iqbal, Wallach, Khoury, Schully,
too would be affected. & Ioannidis, 2016) to 11% (Wallach, Boyack, & Ioannidis, 2018). For
Publication bias can only be directly and unequivocally determined clinical psychology specifically, an audit of the highest-ranking journals
if access is ensured to all conducted studies, for example all trials (Nutu, Gentili, Naudet, & Cristea, 2019) found that even if two thirds of
submitted to a regulatory authority. In a seminal report using the Food the journals recommended data sharing, only one had a mandatory
and Drug Administration (FDA) database, E. H. Turner, Matthews, policy in place. Compliance with the recommended policies in a sample

5
I.A. Cristea and F. Naudet Behaviour Research and Therapy 123 (2019) 103479

of recently published articles was extremely low (2%), in line with re- know what was planned. Hence, it is also critical that the study protocol
ports from the biomedical literature. Of course, barriers need to be and statistical analysis plan are unbiased and useable. Complete and
acknowledged, and a systematic review of data sharing in public health transparent prospective registration of the study plan in a public re-
(van Panhuis et al., 2014) identified several real or potential ones, pository and the publication of the study protocol preceding its im-
ranging from technical (e.g., lack of availability of technical solutions plementation are potential ways to ensure this. These measures would
and of metadata and standards), motivational (e.g., lack of incentives, also be key in minimizing bias due to selective outcome reporting,
disagreement on data use), economic (e.g., lack of resources), political hence limiting research waste due to inappropriate methodological
(e.g., lack of trust or guidelines), legal (e.g., ownership and copyright, choices, described in the previous section. Unfortunately, prospective
protection of privacy) and ethical (e.g., lack of reciprocity). An im- registration is still underutilized in research on psychological inter-
portant element in overcoming barriers is crediting and rewarding in- ventions. Meta-epidemiological studies show low rates of compliance:
vestigators for sharing data, along with other artefacts such as materials 25% in a survey of 25 highest-ranking clinical psychology journals that
or code. Citation is one basic form of awarding credit (Alter & Gonzalez, published randomized controlled trials (Cybulski, Mayo-Wilson, &
2018) and datasets can become citable by receiving a persistent iden- Grant, 2016), and 50% in one of trials published in 2013–2014 in the
tifier, such as a digital object identifier (DOI), through platforms like Journal of Consulting and Clinical Psychology, the American Psycholo-
DataCite (https://datacite.org/). Looking forward, owing to its con- gical Association’ primary outlet for this study type (Azar, Riehm,
spicuous societal value, sharing of data (but also of protocols, materials, McKay, & Thombs, 2015). To map the expanse of the problem, we also
code etc) could be rewarded and incentivized, by incorporating it in looked upstream at what journals are requiring from their authors
assessments of scientists for hiring, promotion and tenure (Moher et al., (Nutu, Gentili, Naudet, & Cristea, 2019). Under 40% of the highest-
2018). Another important issue is that trials with small samples, fre- ranking journals in clinical psychology mentioned registration in their
quent in psychotherapy research, pose a higher risk of partial de- instructions to authors, and only 25% specified it needed to be pro-
identification, particularly for under-represented patient categories. In spective (i.e., before the enrolment of the 1st participant). Compliance
other cases, trials have been underway for a long time, and consent with registration requirements in a sample of articles published in these
forms might have not explicitly asked about sharing data. Yet, it is journals in 2017 was extremely low: 3% if only prospective registration
important to underscore that patients do not seem to share these wor- was considered, and 8% for both prospective and retrospective regis-
ries: only 8% of clinical trial participants in a recent survey (Mello, tration.
Lieou, & Goodman, 2018) felt that potentially harmful consequences of Finally, another source of waste is fragmentation, which can take
data sharing outweighed the benefits and around 26% were concerned several guises: 1) “salami-slicing”, partitioning findings from the same
about being identified. study into a number of thin “slices” or least publishable units (Martin,
2013); 2) reporting results across a multiplicity of sources (e.g., journal
4. Unbiased and useable research reports? articles, trial registries, clinical study reports) (Li, Mayo-Wilson, Fusco,
Hong, & Dickersin, 2018); or 3) running many parallel non-collabora-
Research reports should clearly and transparently delineate “what tive efforts. Though we could not identify investigations of fragmen-
questions were addressed and why, what was done, what was shown and tation and multiplicity in psychological interventions research, an
what the findings mean” (Glasziou et al., 2014) (p.267). Reporting analysis of trials for the drugs gabapentin for neuropathic pain and
guidelines, which originated with the Consolidated Standards for Re- quetiapine for bipolar depression showed that information about trial
porting Trials (CONSORT) statement (Schulz, Altman, & Moher, 2010), design and risk of bias frequently diverged in completeness across re-
are one effective way to ensure this happens. These offer guidance and ports. Different reports of the same data often conflicted, in ways that
structured forms on how to report research methods and findings were difficult to disentangle (Mayo-Wilson et al., 2017). A particularly
(Blanco et al., 2017), specific to different study types ("Library for serious problem occurs when fragmentation of results also comprises
health research reporting,"). The EQUATOR Network (https://www. non-public sources, such as clinical study reports. The re-analysis of
equator-network.org/about-us/), a portal dedicated to augmenting re- SmithKline Beecham's Study 329 (Le Noury et al., 2015) is a revealing
search value by promoting the use of reporting guidelines, lists over example, because it was based on clinical study reports previously not
400 of these, tailored for a diversity of research designs, populations publicly accessible. Multiplicity also enhances the hazardous effects of
and interventions. Though their consistent utilization would reduce the selective reporting beyond single trials to meta-analyses, by allowing
danger of incomplete, misleading or otherwise unusable reports, researchers to cherry-pick outcomes and results and conduct many
adoption from journals and compliance from researchers are both still “possible” and even conflicting meta-analyses (Mayo-Wilson et al.,
wanting. An early systematic review (Grant, Mayo-Wilson, Melendez- 2017).
Torres, & Montgomery, 2013), which surveyed the adoption of re- Overlapping meta-analyses on the same topic are a good example of
porting guidelines in high-impact journals in several fields publishing waste due to fragmentation of efforts. Meta-analyses are expected to
research on psychosocial interventions, found that around a quarter of reduce waste by helping delineate research questions worth exploring
the surveyed journals mentioned guidelines in their instructions to and by delivering a reproducible and quantitative synthesis on a given
authors. For a sample of trials published in these journals, compliance question. However, their number has raised to epidemic proportions in
to reporting standards was around 42%. In a more recent audit of top- the past decade, possibly because they amount to easily produced
tier journals in clinical psychology (Nutu, Gentili, Naudet, & Cristea, publishable units or powerful marketing tools (Ebrahim, Bance, Athale,
2019), approximately half of the surveyed journals explicitly endorsed Malachowski, & Ioannidis, 2016). For instance, the database of trials of
reporting guidelines in their instructions to author. However, the more psychotherapy for depression (Cuijpers, van Straten, Warmerdam, &
stringent requirement of making use of guidelines mandatory was in Andersson, 2008) (http://www.evidencebasedpsychotherapies.org/)
place at 30% of the journals for randomized controlled trials, and at included, at the last publicly available update to the website in 2014,
25% of them for systematic reviews and meta-analyses. An analysis of a 352 trials and 53 published meta-analyses of these, a rate of approxi-
cross-sectional sample of articles recently published in the surveyed mately 1 meta-analysis for every 7 trials. Though widely viewed as a
journals showed breaches in adoption still remained, even when higher level of evidence and as the authoritative final word in a debate,
guidelines were mandatory as per journal policy: 3/14 randomized meta-analyses are often suboptimal, overlapping and sometimes in
trials and 7/24 systematic reviews and meta-analyses were rated as contradiction (Ioannidis, 2016a). Although the basic principles of a
non-compliant. meta-analysis are simple, researchers can make many choices along the
Glasziou et al. (2014) warn that waste is not restricted to study way that can significantly alter the results and sometimes lead to op-
reports. In order to be able to judge what was done, it is essential to posing conclusions. In an egregious simulation, Palpacuer et al., (2019)

6
I.A. Cristea and F. Naudet Behaviour Research and Therapy 123 (2019) 103479

demonstrated that varying methodological criteria and analytical in biased or unusable reports, with findings or data not fully accessible,
models resulted into 9216 possible different and contradictory meta- is at best inconsequential and at worst harmful to patients and others
analyses of indirect comparisons, for 60 primary trials of nalmefene and studying or employing psychological treatments. Secondary/meta-re-
naltrexone in alcohol disorders. search is indispensable in surveilling the integrity and transparency of
Similarly, among actually published meta-analyses, the effective- the research process and in detecting systematic biases. This field in-
ness of long-term psychodynamic psychotherapy for mental disorders cludes research synthesis, meta-analysis and various other types of
was estimated in one meta-analysis with a risk difference of 0 for the meta-epidemiological assessment, which critically examine the ways in
primary outcome of recovery at the longest follow-up and with non- which studies are planned, conducted or reported, or that seek to detect
significant effect sizes for other outcomes (Smit et al., 2012), while in the influences of systematic biases in large collections of studies.
another meta-analysis (Leichsenring & Rabung, 2011) the same treat- Funders, academic institutions and journal editors need to recognize
ment produced significant and moderate effect sizes (Hedges' g) for the role of and encourage meta-research. Yet many meta-research in-
overall effectiveness, as well as for other outcomes. In another example, vestigations themselves suffer from the same waste-generating issues
two meta-analyses of acceptance and commitment therapy (ACT) re- we identified, such as duplicating what is already known or researched,
sulted into largely similar effect sizes for psychiatric and somatic dis- not being prospectively registered, or not sharing their underlying data.
orders (differences in Hedges’ g between the two meta-analyses ranged Interventions intended to improve research quality should be planned
from 0.10 to 0.15), but reached widely divergent conclusions: sup- and tested in adequately conducted and reported randomized trials
porting the use of the therapy in one (A. Tjak et al., 2015), and ques- (Bravo, Grimaldo, Lopez-Inesta, Mehmani, & Squazzoni, 2019); other-
tioning its status as a “well-established” treatment in another (Ost, wise, they risk adding to the waste. For instance, only a handful of
2014). In both examples, meta-analyses published within a year of each interventions to improve the quality of peer-review have been assessed
other diverged in the magnitude or even significance of the effect, and, in randomized trials, with mixed results (Bruce, Chauvin, Trinquart,
even more concerningly, supported markedly discrepant conclusions Ravaud, & Boutron, 2016).
about treatment effectiveness. Though IPD meta-analysis are generally Furthermore, open science principles, such as transparency, en-
seen as powerful tools in identifying individual-level variables that suring that studies are accurately and fully described across all stages,
could act as effect modifiers (Riley, Lambert, & Abo-Zaid, 2010), only a from planning to the interpretation of findings, accessibility of data
small fraction (under 10%) of them uncover statistically significant underlying results, reproducibility checks and independent replication
interactions for treatment effects, translating into a modest potential to at the very least of findings lined up for clinical translation, need to be
personalize treatment (Schuit, Li, & Ioannidis, 2018). placed at the core of psychological treatment research. It was somewhat
surprising that the programmatic Lancet Psychiatry Commission on
5. Solutions for increasing value psychological treatments (Holmes et al., 2018) did not attach greater
importance and a more central part to open science (Cristea, 2018a). As
We used the Lancet series on reducing waste and increasing value as we amply documented in this comment, deviations from open science
a template to discuss the state of research in the field of clinical psy- principles and practices underlie most failures to engender transfor-
chology with a particular focus on psychological therapies. The litera- mative and useful research. Funders, journal editors and academic in-
ture we reviewed demonstrates that all sources of waste considered are stitutions could test the usefulness of the Transparency and Openness
well-represented and amply documented within this field. We did not Promotion (TOP) guidelines (Nosek et al., 2015) to incentivize the
discuss waste related to research regulation and management (Salman principles of transparency, openness and reproducibility in planning,
et al., 2014), because, in contrast to biomedicine, research on psycho- conducting and reporting research.
logical treatments is neither formally regulated, nor does it have to Finally, we need to remain vigilant particularly of the strains of
comply with the requirements of regulatory bodies like the FDA or the research currently prioritized, such as experimental psychopathology.
European Medicine Agency. Our goal was to exemplify varieties of The hype that often accompanies findings in this field is most likely
waste and as such our analysis was neither systematic, nor exhaustive. detrimental and should be contained. The fact that a particular result
Unlike Chalmers and Glasziou (2009), we did not attempt to quantify could have major treatments implications should not distract us from
the proportion of avoidable waste from research investment. Whether ensuring that the methods by which it was obtained are sound, its re-
the 85% estimated for biomedicine is an under- or an overestimation porting complete and transparent, and independent replication does
remains an open question. Still, we hope our evaluation can serve as a occur. The hypothetical promise of a particular type of theory or model
warning call to all those involved with psychological treatments, from cannot supplant modest or fleeting effects, which are unlikely to en-
funders to researchers and patients, to stimulate reflection and, when gender any tangible benefit to patients. Furthermore, if our aim is for
possible, reforms to increase value in research on psychological inter- psychological interventions to be developed mechanistically, following
ventions. a process similar to the development and testing of drugs, it must also
In response to the problems diagnosed and extending the solutions be acknowledged that, as with most initially encouraging drug targets,
suggested in the Lancet series, several measures were proposed to op- many originally promising hypotheses stemming from the laboratory
timize key elements of the scientific process, such as methods, reporting will need to be abandoned when they fail replication, negative findings
and dissemination, reproducibility, evaluation and initiatives (Munafò accumulate, or when independent, well-conducted meta-analyses show
et al., 2017). A discussion of these is beyond the scope of this paper, but small and unreliable effects, lack of effects in clinical populations, or
most, if not all, could be successfully tested with the goal of improving critical issues in the planning, conducting and reporting of studies. Even
research on psychological interventions. Similarly, Tackett et al. (2017) if all these conditions were met, translation from to proof-of-concept
scrutinized possible reasons why the conversation around reproduci- trials might result in weak, unstable or non-durable effects which might
bility has remained peripheral in clinical psychology, such as a greater not justify the immense investment of a phase III trial.
appreciation of complex and imperfect data or more variable sampling In a thought-provoking comment, Ioannidis (2016b) argued that
procedures. The authors also suggest challenges and recommendations since most psychological interventions probably do not work, they
in reference to reproducibility concerns specific to clinical psychology. should be examined in “carefully properly biased” (p.437) studies, where
We further expand on a few general suggestions. First, the emerging investigators give their best shot to augment the effectiveness of the
field of meta-research (“research on research”) is essential for mon- intervention, by increasing expectations, boosting any placebo effects,
itoring and diagnosing the “health” of psychological treatment re- using waiting list control arms and small samples. The phase would
search. Though science is often equated with primary research, in- function as a pre-screening and only the interventions attaining a rea-
adequately planned, conducted or reported primary research, described sonable effect under these very favorable conditions would move to the

7
I.A. Cristea and F. Naudet Behaviour Research and Therapy 123 (2019) 103479

phase of testing in a larger trial. While appealing, this system could Cristea, I. A. (2018b). The waiting list is an inadequate benchmark for estimating the
never work if other sources of waste are not tackled: studies poorly effectiveness of psychotherapy for depression. Epidemiology and Psychiatric
Sciences, 1–2.
planned and implemented, reports unusable and incomplete, negative Cristea, I. A., Gentili, C., Cotet, C. D., Palomba, D., Barbui, C., & Cuijpers, P. (2017).
findings stashed away, or data not accessible. Ultimately, if research Efficacy of psychotherapies for borderline personality disorder: A systematic review
and meta-research on psychological interventions aims to become more and meta-analysis. JAMA Psychiatry, 74, 319–328.
Cristea, I. A., Gentili, C., Pietrini, P., & Cuijpers, P. (2017a). Is investigator background
mechanistic and science-based, it should heed Douglas Altman's famous related to outcome in head to head trials of psychotherapy and pharmacotherapy for
proclamation “We need less research, better research, and research adult depression? A systematic review and meta-analysis. PLoS One, 12, e0171654.
done for the right reasons” (Altman, 1994) (p.308). Cristea, I. A., Gentili, C., Pietrini, P., & Cuijpers, P. (2017b). Sponsorship bias in the
comparative efficacy of psychotherapy and pharmacotherapy for adult depression:
Meta-analysis. British Journal of Psychiatry, 210, 16–23.
Acknowledgements Cristea, I. A., & Ioannidis, J. P. A. (2018). Improving disclosure of financial conflicts of
interest for research on psychosocial interventions. JAMA Psychiatry, 75, 541–542.
Cristea, I. A., Kok, R. N., & Cuijpers, P. (2015). Efficacy of cognitive bias modification
Financial support
interventions in anxiety and depression: meta-analysis. British Journal of Psychiatry,
206, 7–16.
Ioana A. Cristea was supported by a grant from the Romanian Cristea, I. A., Mogoase, C., David, D., & Cuijpers, P. (2015). Practitioner review: Cognitive
Ministery of Research and Innovation, CNCS – UEFISCDI (project bias modification for mental health problems in children and adolescents: A meta-
analysis. Journal of Child Psychology and Psychiatry, 56, 723–734.
number PN-III-P1-1.1-TE-2016-1054). The funder had no role in the Cristea, I. A., Naudet, F., Shanks, D. R., & Hardwicke, T. E. (2018). Post-retrieval Tetris
design of the study, collection, analysis and interpretation of data, and should not be likened to a ‘cognitive vaccine’. Molecular Psychiatry, 23, 1972–1973.
the decision to approve publication. Crocker, J. C., Ricci-Cabello, I., Parker, A., Hirst, J. A., Chant, A., Petit-Zeman, S., et al.
(2018). Impact of patient and public involvement on enrolment and retention in
We would like to thank Dr. Tom E. Hardwicke for comments on an clinical trials: Systematic review and meta-analysis. BMJ, 363, k4738.
early draft of the manuscript. Cuijpers, P. (2019). Targets and outcomes of psychotherapies for mental disorders: An
overview. World Psychiatry, 18(3), 276–285. https://doi.org/10.1002/wps.20661.
Cuijpers, P., & Cristea, I. A. (2016). How to prove that your therapy is effective, even
Conflict of interest disclosure when it is not: A guideline. Epidemiology and Psychiatric Sciences, 25, 428–435.
Cuijpers, P., Cristea, I. A., Karyotaki, E., Reijnders, M., & Huibers, M. J. (2016). How
All authors have completed the ICMJE uniform disclosure form at effective are cognitive behavior therapies for major depression and anxiety disorders?
A meta-analytic update of the evidence. World Psychiatry, 15, 245–258.
http://www.icmje.org/coi_disclosure.pdf (available upon request from Cuijpers, P., Donker, T., Weissman, M. M., Ravitz, P., & Cristea, I. A. (2016). Interpersonal
the corresponding author) and declare no conflict of interest. psychotherapy for mental health problems: A comprehensive meta-analysis. American
Journal of Psychiatry, 173, 680–687.
Cuijpers, P., Karyotaki, E., Reijnders, M., & Ebert, D. D. (2019). Was eysenck right after
References
all? A reassessment of the effects of psychotherapy for adult depression. Epidemiology
and Psychiatric Sciences, 28, 21–30.
Alter, G., & Gonzalez, R. (2018). Responsible practices for data sharing. American Cuijpers, P., Smit, F., Bohlmeijer, E., Hollon, S. D., & Andersson, G. (2010). Efficacy of
Psychologist, 73, 146–156. cognitive-behavioural therapy and other psychological treatments for adult depres-
Altman, D. G. (1994). The scandal of poor medical research. BMJ, 308, 283–284. sion: meta-analytic study of publication bias. British Journal of Psychiatry, 196,
Anderson, N. L. (2010). The clinical plasma proteome: A survey of clinical assays for 173–178.
proteins in plasma and serum. Clinical Chemistry, 56, 177–185. Cuijpers, P., van Straten, A., Warmerdam, L., & Andersson, G. (2008). Psychological
Azar, M., Riehm, K. E., McKay, D., & Thombs, B. D. (2015). Transparency of outcome treatment of depression: A meta-analytic database of randomized studies. BMC
reporting and trial registration of randomized controlled trials published in the Psychiatry, 8, 36.
journal of consulting and clinical psychology. PLoS One, 10, e0142894. Cybulski, L., Mayo-Wilson, E., & Grant, S. (2016). Improving transparency and re-
Barth, J., Munder, T., Gerger, H., Nuesch, E., Trelle, S., Znoj, H., et al. (2013). producibility through registration: The status of intervention trials published in
Comparative efficacy of seven psychotherapeutic interventions for patients with de- clinical psychology journals. Journal of Consulting and Clinical Psychology, 84,
pression: A network meta-analysis. PLoS Medicine, 10, e1001454. 753–767.
Bighelli, I., Salanti, G., Huhn, M., Schneider-Thoma, J., Krause, M., Reitmeir, C., et al. Deighton, J., Lereya, S. T., Casey, P., Patalay, P., Humphrey, N., & Wolpert, M. (2019).
(2018). Psychological interventions to reduce positive symptoms in schizophrenia: Prevalence of mental health problems in schools: Poverty and other risk factors
Systematic review and network meta-analysis. World Psychiatry, 17, 316–329. among 28 000 adolescents in england. British Journal of Psychiatry, 1–3.
Bisson, J. I., Roberts, N. P., Andrew, M., Cooper, R., & Lewis, C. (2013). Psychological Dragioti, E., Dimoliatis, I., & Evangelou, E. (2015). Disclosure of researcher allegiance in
therapies for chronic post-traumatic stress disorder (PTSD) in adults. Cochrane Database meta-analyses and randomised controlled trials of psychotherapy: A systematic ap-
Syst Rev Cd003388. praisal. BMJ Open, 5, e007206.
Blanco, D., Kirkham, J. J., Altman, D. G., Moher, D., Boutron, I., & Cobo, E. (2017). Dragioti, E., Dimoliatis, I., Fountoulakis, K. N., & Evangelou, E. (2015). A systematic
Interventions to improve adherence to reporting guidelines in health research: A appraisal of allegiance effect in randomized controlled trials of psychotherapy. Annals
scoping review protocol. BMJ Open, 7. of General Psychiatry, 14, 25.
Bradley, H. A., Rucklidge, J. J., & Mulder, R. T. (2017). A systematic review of trial Dragioti, E., Karathanos, V., Gerdle, B., & Evangelou, E. (2017). Does psychotherapy
registration and selective outcome reporting in psychotherapy randomized controlled work? An umbrella review of meta-analyses of randomized controlled trials. Acta
trials. Acta Psychiatrica Scandinavica, 135, 65–77. Psychiatrica Scandinavica, 136, 236–246.
Bravo, G., Grimaldo, F., Lopez-Inesta, E., Mehmani, B., & Squazzoni, F. (2019). The effect Driessen, E., Hollon, S. D., Bockting, C. L., Cuijpers, P., & Turner, E. H. (2015). Does
of publishing peer review reports on referee behavior in five scholarly journals. publication bias inflate the apparent efficacy of psychological treatment for major
Nature Communications, 10, 322. depressive disorder? A systematic review and meta-analysis of US national Institutes
Browning, M., Holmes, E. A., Charles, M., Cowen, P. J., & Harmer, C. J. (2012). Using of health-funded trials. PLoS One, 10, e0137864.
attentional bias modification as a cognitive vaccine against depression. Biological Ebrahim, S., Bance, S., Athale, A., Malachowski, C., & Ioannidis, J. P. (2016). Meta-
Psychiatry, 72, 572–579. analyses with industry involvement are massively published and report no caveats for
Bruce, R., Chauvin, A., Trinquart, L., Ravaud, P., & Boutron, I. (2016). Impact of inter- antidepressants. Journal of Clinical Epidemiology, 70, 155–163.
ventions to improve the quality of peer review of biomedical journals: A systematic Ebrahim, S., Sohani, Z. N., Montoya, L., Agarwal, A., Thorlund, K., Mills, E. J., et al.
review and meta-analysis. BMC Medicine, 14, 85. (2014). Reanalyses of randomized clinical trial data. Jama, 312, 1024–1032.
Chalmers, I., Bracken, M. B., Djulbegovic, B., Garattini, S., Grant, J., Gülmezoglu, A. M., Flint, J., Cuijpers, P., Horder, J., Koole, S. L., & Munafò, M. R. (2015). Is there an excess of
et al. (2014). How to increase value and reduce waste when research priorities are significant findings in published studies of psychotherapy for depression?
set. The Lancet, 383, 156–165. Psychological Medicine, 45, 439–446.
Chalmers, I., & Glasziou, P. (2009). Avoidable waste in the production and reporting of Fodor, L. A., Cotet, C. D., Cuijpers, P., Szamoskozi, S., David, D., & Cristea, I. A. (2018).
research evidence. The Lancet, 374, 86–89. The effectiveness of virtual reality based interventions for symptoms of anxiety and
Chan, A.-W., Song, F., Vickers, A., Jefferson, T., Dickersin, K., Gøtzsche, P. C., et al. depression: A meta-analysis. Scientific Reports, 8, 10323.
(2014). Increasing value and reducing waste: Addressing inaccessible research. The Furukawa, T. A., Noma, H., Caldwell, D. M., Honyashiki, M., Shinohara, K., Imai, H., et al.
Lancet, 383, 257–266. (2014). Waiting list may be a nocebo condition in psychotherapy trials: A contribu-
Chatterton, M. L., Stockings, E., Berk, M., Barendregt, J. J., Carter, R., & Mihalopoulos, C. tion from network meta-analysis. Acta Psychiatrica Scandinavica, 130, 181–192.
(2017). Psychosocial therapies for the adjunctive treatment of bipolar disorder in Glasziou, P., Altman, D. G., Bossuyt, P., Boutron, I., Clarke, M., Julious, S., et al. (2014).
adults: Network meta-analysis. British Journal of Psychiatry, 210, 333–341. Reducing waste from incomplete or unusable reports of biomedical research. The
Contopoulos-Ioannidis, D. G., Ntzani, E. E., & Ioannidis, J. P. A. (2003). Translation of Lancet, 383, 267–276.
highly promising basic science research into clinical applications. The American Grant, S. P., Mayo-Wilson, E., Melendez-Torres, G. J., & Montgomery, P. (2013).
Journal of Medicine, 114, 477–484. Reporting quality of social and psychological intervention trials: A systematic review
Cristea, I. A. (2018a). Experimental psychopathology should not be examined with rose- of reporting guidelines and trial publications. PLoS One, 8, e65442.
tinted glasses. The Lancet Psychiatry, 5, 465–466. Hardwicke, T. E., & Shanks, D. R. (2016). Reply to Walker and Stickgold: Proposed

8
I.A. Cristea and F. Naudet Behaviour Research and Therapy 123 (2019) 103479

boundary conditions on memory reconsolidation will require empirical verification. Markozannes, G., Aretouli, E., Rintou, E., Dragioti, E., Damigos, D., Ntzani, E., et al.
Proceedings of the National Academy of Sciences of the United States of America, 113, (2017). An umbrella review of the literature on the effectiveness of psychological
E3993–E3994. interventions for pain reduction. BMC Psychol, 5, 31.
Hardwicke, T. E., Taqi, M., & Shanks, D. R. (2016). Postretrieval new learning does not Martin, B. R. (2013). Whither research integrity? Plagiarism, self-plagiarism and coercive
reliably induce human memory updating via reconsolidation. Proceedings of the citation in an age of research assessment. Research Policy, 42, 1005–1014.
National Academy of Sciences of the United States of America, 113, 5206–5211. Mayo-Wilson, E., Dias, S., Mavranezouli, I., Kew, K., Clark, D. M., Ades, A. E., et al.
Hazo, J. B., Gandre, C., Leboyer, M., Obradors-Tarrago, C., Belli, S., McDaid, D., et al. (2014). Psychological and pharmacological interventions for social anxiety disorder
(2017). National funding for mental health research in Finland, France, Spain and the in adults: A systematic review and network meta-analysis. The Lancet Psychiatry, 1,
United Kingdom. European Neuropsychopharmacology, 27, 892–899. 368–376.
Hazo, J. B., Gervaix, J., Gandre, C., Brunn, M., Leboyer, M., & Chevreul, K. (2016). Mayo-Wilson, E., Li, T., Fusco, N., Bertizzolo, L., Canner, J. K., Cowley, T., et al. (2017).
European Union investment and countries' involvement in mental health research Cherry-picking by trialists and meta-analysts can drive conclusions about interven-
between 2007 and 2013. Acta Psychiatrica Scandinavica, 134, 138–149. tion efficacy. Journal of Clinical Epidemiology, 91, 95–110.
Higgins, J. P. T., Altman, D. G., Gøtzsche, P. C., Jüni, P., Moher, D., Oxman, A. D., et al. Meichenbaum, D., & Lilienfeld, S. O. (2018). How to spot hype in the field of psy-
(2011). The Cochrane Collaboration's tool for assessing risk of bias in randomised chotherapy: A 19-item checklist. Professional Psychology: Research and Practice, 49,
trials. BMJ, 343, d5928. 22–30.
Higgins, J. P. T., & Green, S. (2011). Cochrane handbook for systematic reviews of inter- Mello, M. M., Lieou, V., & Goodman, S. N. (2018). Clinical trial participants' views of the
ventions. The Cochrane Collaboration Version 5.1.0. ed. risks and benefits of data sharing. New England Journal of Medicine, 378, 2202–2211.
Holmes, E. A., Ghaderi, A., Eriksson, E., Lauri, K. O., Kukacka, O. M., Mamish, M., et al. Moher, D., Naudet, F., Cristea, I. A., Miedema, F., Ioannidis, J. P. A., & Goodman, S. N.
(2017). 'I can't concentrate': A feasibility study with young refugees in Sweden on (2018). Assessing scientists for hiring, promotion, and tenure. PLoS Biology, 16,
developing science-driven interventions for intrusive memories related to trauma. e2004089.
Behavioural and Cognitive Psychotherapy, 45, 97–109. MQ (2019). UK mental health research funding 2014–2017. MQ: Transforming mental
Holmes, E. A., Ghaderi, A., Harmer, C. J., Ramchandani, P. G., Cuijpers, P., Morrison, A. health.
P., et al. (2018). The Lancet Psychiatry Commission on psychological treatments re- Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du
search in tomorrow's science. The Lancet Psychiatry, 5, 237–286. Sert, N., et al. (2017). A manifesto for reproducible science. Nature Human Behaviour,
Holmes, E. A., James, E. L., Kilford, E. J., & Deeprose, C. (2010). Key steps in developing a 1 0021.
cognitive vaccine against traumatic flashbacks: Visuospatial Tetris versus verbal pub Munder, T., Fluckiger, C., Gerger, H., Wampold, B. E., & Barth, J. (2012). Is the allegiance
quiz. PLoS One, 5, e13706. effect an epiphenomenon of true efficacy differences between treatments? A meta-
ICMJE. Clinical Trials- Data sharing. In (Vol. 2019). analysis. Journal of Counseling Psychology, 59, 631–637.
Ioannidis, J. P. A. (2016a). The mass production of redundant, misleading, and conflicted Munder, T., Gerger, H., Trelle, S., & Barth, J. (2011). Testing the allegiance bias hy-
systematic reviews and meta-analyses. The Milbank Quarterly, 94, 485–514. pothesis: A meta-analysis. Psychotherapy Research, 21, 670–684.
Ioannidis, J. P. A. (2016b). Most psychotherapies do not really work, but those that might Nader, K., Schafe, G. E., & Le Doux, J. E. (2000). Fear memories require protein synthesis
work should be assessed in biased studies. Epidemiology and Psychiatric Sciences, 25, in the amygdala for reconsolidation after retrieval. Nature, 406, 722–726.
436–438. Naudet, F., Sakarovitch, C., Janiaud, P., Cristea, I., Fanelli, D., Moher, D., et al. (2018).
Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., Data sharing and reanalysis of randomized controlled trials in leading biomedical
et al. (2014). Increasing value and reducing waste in research design, conduct, and journals with a full data sharing policy: Survey of studies published in < em > The
analysis. The Lancet, 383, 166–175. BMJ < /em > and < em > PLOS medicine < /em > BMJ, 360.
Ioannidis, J. P. A., & Trikalinos, T. A. (2007). An exploratory test for an excess of sig- Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., et al.
nificant findings. Clinical Trials, 4, 245–253. (2015). Promoting an open research culture. Science, 348, 1422–1425.
Iqbal, S. A., Wallach, J. D., Khoury, M. J., Schully, S. D., & Ioannidis, J. P. A. (2016). Nutu, D., Gentili, C., Naudet, F., & Cristea, I. A. (2019). Open science practices in clinical
Reproducible research practices and transparency across the biomedical literature. psychology journals: An audit study. Journal of Abnormal Psychology, 128(6),
PLoS Biology, 14, e1002333. 510–516. http://dx.doi.org/10.1037/abn0000414.
Iyadurai, L., Blackwell, S. E., Meiser-Stedman, R., Watson, P. C., Bonsall, M. B., Geddes, J. Ost, L. G. (2014). The efficacy of acceptance and commitment therapy: An updated sys-
R., et al. (2018). Preventing intrusive memories after trauma via a brief intervention tematic review and meta-analysis. Behaviour Research and Therapy, 61, 105–121.
involving Tetris computer game play in the emergency department: A proof-of-con- Palpacuer, C., Hammas, K., Duprez, R., Laviolle, B., Ioannidis, J. P. A. I., & Naudet, F.
cept randomized controlled trial. Molecular Psychiatry, 23, 674–682. (under review). Vibration of effects from diverse inclusion/exclusion criteria and
Jayasuriya, D., Jayasuriya, R., Tay, A. K., & Silove, D. (2016). Associations of mental analytical choices: 9,216 different ways to perform an indirect comparison meta-
distress with residency in conflict zones, ethnic minority status, and potentially analysis. BMC Medicine.
modifiable social factors following conflict in Sri Lanka: A nationwide cross-sectional van Panhuis, W. G., Paul, P., Emerson, C., Grefenstette, J., Wilder, R., Herbst, A. J., et al.
study. The Lancet Psychiatry, 3, 145–153. (2014). A systematic review of barriers to data sharing in public health. BMC Public
Keller, M. B., Ryan, N. D., Strober, M., Klein, R. G., Kutcher, S. P., Birmaher, B., et al. Health, 14, 1144.
(2001). Efficacy of paroxetine in the treatment of adolescent major depression: A Patel, V., Burns, J. K., Dhingra, M., Tarver, L., Kohrt, B. A., & Lund, C. (2018). Income
randomized, controlled trial. Journal of the American Academy of Child & Adolescent inequality and depression: A systematic review and meta-analysis of the association
Psychiatry, 40, 762–772. and a scoping review of mechanisms. World Psychiatry, 17, 76–89.
Khan, A., Faucett, J., Lichtenberg, P., Kirsch, I., & Brown, W. A. (2012). A systematic Pompoli, A., Furukawa, T. A., Imai, H., Tajika, A., Efthimiou, O., & Salanti, G. (2016).
review of comparative efficacy of treatments and controls for depression. PLoS One, 7, Psychological therapies for panic disorder with or without agoraphobia in adults: A
e41778. network meta-analysis. Cochrane Database of Systematic Reviews, 13(4), https://doi.
Kindt, M., & Soeter, M. (2013). Reconsolidation in a human fear conditioning study: A test org/10.1002/14651858.CD011004.pub2.
of extinction as updating mechanism. Biological Psychology, 92, 43–50. Riley, R. D., Lambert, P. C., & Abo-Zaid, G. (2010). Meta-analysis of individual participant
Kruijt, A.-W., Parsons, S., & Fox, E. (2019). A meta-analysis of bias at baseline in RCTs of data: Rationale, conduct, and reporting. BMJ, 340, c221.
attention bias modification: No evidence for dot-probe bias towards threat in clinical Rücker, G., Carpenter, J. R., & Schwarzer, G. (2011). Detecting and adjusting for small-
anxiety and PTSD. Journal of Abnormal Psychology, 128(6), 563–573. http://dx.doi. study effects in meta-analysis. Biometrical Journal, 53, 351–368.
org/10.1037/abn0000406. Salman, R. A.-S., Beller, E., Kagan, J., Hemminki, E., Phillips, R. S., Savulescu, J., et al.
Kyzas, P. A., Denaxa-Kyza, D., & Ioannidis, J. P. A. (2007). Almost all articles on cancer (2014). Increasing value and reducing waste in biomedical research regulation and
prognostic markers report statistically significant results. European Journal of Cancer, management. The Lancet, 383, 176–185.
43, 2559–2579. Schiller, D., Monfils, M.-H., Raio, C. M., Johnson, D. C., LeDoux, J. E., & Phelps, E. A.
Le Noury, J., Nardo, J. M., Healy, D., Jureidini, J., Raven, M., Tufanaru, C., et al. (2015). (2009). Preventing the return of fear in humans using reconsolidation update me-
Restoring study 329: Efficacy and harms of paroxetine and imipramine in treatment chanisms. Nature, 463, 49.
of major depression in adolescence. BMJ, 351, h4320. Schuit, E., Li, A. H., & Ioannidis, J. P. A. (2018). How often can meta-analyses of in-
Leichsenring, F., & Rabung, S. (2011). Long-term psychodynamic psychotherapy in dividual-level data individualize treatment? A meta-epidemiologic study.
complex mental disorders: Update of a meta-analysis. British Journal of Psychiatry, International Journal of Epidemiology, 48, 596–608.
199, 15–22. Schulz, K. F., Altman, D. G., & Moher, D. (2010). CONSORT 2010 statement: Updated
Library for health research reporting EQUATOR Network. http://www.equator-network. guidelines for reporting parallel group randomized trials. Annals of Internal Medicine,
org/reporting-guidelines/, Accessed date: 24 July 2019. 152, 726–732.
Li, T., Mayo-Wilson, E., Fusco, N., Hong, H., & Dickersin, K. (2018). Caveat emptor: The Shinohara, K., Tajika, A., Imai, H., Takeshima, N., Hayasaka, Y., & Furukawa, T. A.
combined effects of multiplicity and selective reporting. Trials, 19, 497. (2015). Protocol registration and selective outcome reporting in recent psychiatry
Lovell, K., Bower, P., Gellatly, J., Byford, S., Bee, P., McMillan, D., et al. (2017). Low- trials: New antidepressants and cognitive behavioural therapies. Acta Psychiatrica
intensity cognitive-behaviour therapy interventions for obsessive-compulsive dis- Scandinavica, 132, 489–498.
order compared to waiting list for therapist-led cognitive-behaviour therapy: 3-arm Silove, D., Liddell, B., Rees, S., Chey, T., Nickerson, A., Tam, N., et al. (2014). Effects of
randomised controlled trial of clinical effectiveness. PLoS Medicine, 14, e1002337. recurrent violence on post-traumatic stress disorder and severe distress in conflict-
MacLeod, C. (2012). Cognitive bias modification procedures in the management of affected timor-leste: A 6-year longitudinal study. Lancet Glob Health, 2, e293–300.
mental disorders. Current Opinion in Psychiatry, 25, 114–120. Skapinakis, P., Caldwell, D. M., Hollingworth, W., Bryden, P., Fineberg, N. A., Salkovskis,
MacLeod, C., & Mathews, A. (2012). Cognitive bias modification approaches to anxiety. P., et al. (2016). Pharmacological and psychotherapeutic interventions for manage-
Annual Review of Clinical Psychology, 8, 189–217. ment of obsessive-compulsive disorder in adults: A systematic review and network
Macleod, M. R., Michie, S., Roberts, I., Dirnagl, U., Chalmers, I., Ioannidis, J. P. A., et al. meta-analysis. The Lancet Psychiatry, 3, 730–739.
(2014). Biomedical research: Increasing value, reducing waste. The Lancet, 383, Slade, E., Keeney, E., Mavranezouli, I., Dias, S., Fou, L., Stockton, S., et al. (2018).
101–104. Treatments for bulimia nervosa: A network meta-analysis. Psychological Medicine, 48,

9
I.A. Cristea and F. Naudet Behaviour Research and Therapy 123 (2019) 103479

2629–2636. Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., & Rosenthal, R. (2008).
Smit, Y., Huibers, M. J., Ioannidis, J. P. A., van Dyck, R., van Tilburg, W., & Arntz, A. Selective publication of antidepressant trials and its influence on apparent efficacy.
(2012). The effectiveness of long-term psychoanalytic psychotherapy–a meta-analysis New England Journal of Medicine, 358, 252–260.
of randomized controlled trials. Clinical Psychology Review, 32, 81–92. Turner, D. T., van der Gaag, M., Karyotaki, E., & Cuijpers, P. (2014). Psychological in-
Steinert, C., Stadter, K., Stark, R., & Leichsenring, F. (2017). The effects of waiting for terventions for psychosis: A meta-analysis of comparative outcome studies. American
treatment: A meta-analysis of waitlist control groups in randomized controlled trials Journal of Psychiatry, 171, 523–538.
for social anxiety disorder. Clinical Psychology & Psychotherapy, 24, 649–660. Van Bockstaele, B., Verschuere, B., Tibboel, H., De Houwer, J., Crombez, G., & Koster, E.
Tackett, J. L., Lilienfeld, S. O., Patrick, C. J., Johnson, S. L., Krueger, R. F., Miller, J. D., H. (2014). A review of current evidence for the causal impact of attentional bias on
et al. (2017). It's time to broaden the replicability conversation: Thoughts for and fear and anxiety. Psychological Bulletin, 140, 682–721.
from clinical psychological science. Perspectives on Psychological Science, 12, 742–756. Walker, M. P., Brakefield, T., Hobson, J. A., & Stickgold, R. (2003). Dissociable stages of
Tannenbaum, S., Ross, J. S., Krumholz, H. M., Desai, N. R., Ritchie, J. D., Lehman, R., human memory consolidation and reconsolidation. Nature, 425, 616–620.
et al. (2018). Early experiences with journal data sharing policies: A survey of pub- Wallach, J. D., Boyack, K. W., & Ioannidis, J. P. A. (2018). Reproducible research prac-
lished clinical trial investigators. Annals of Internal Medicine, 169(8), 586–588. tices, transparency, and open access data in the biomedical literature, 2015–2017.
https://doi.org/10.7326/M18-0723. PLoS Biology, 16, e2006930.
Thielking, M. (2019). To improve mental health treatments, scientists try to dissect the Wilson, G. T., Wilfley, D. E., Agras, W. S., & Bryson, S. W. (2011). Allegiance bias and
pieces that make them work. STAT. https://www.statnews.com/2019/02/01/back- therapist effects: Results of a randomized controlled trial of binge eating disorder.
translation-mental-health-treatments/, Accessed date: 24 July 2019. Clinical Psychologist, 18, 119–125.
A-Tjak, J., Davis, M. L., Morina, N., Powers, M. B., Smits, J. A., et al. (2015). A meta- Yoder, W. R., Karyotaki, E., Cristea, I.-A., van Duin, D., & Cuijpers, P. (2019). Researcher
analysis of the efficacy of acceptance and commitment therapy for clinically relevant allegiance in research on psychosocial interventions: meta-research study protocol
mental and physical health problems. Psychotherapy and Psychosomatics, 84, 30–36. and pilot study. BMJ Open, 9 bmjopen-2018-024622.

10

S-ar putea să vă placă și