Journal of Affective Disorders 263 (2020) 413–419

Contents lists available at ScienceDirect

Journal of Affective Disorders

journal homepage: www.elsevier.com/locate/jad

Review article

Dropout rates in clinical trials of smartphone apps for depressive symptoms: T

A systematic review and meta-analysis
John Torousa, Jessica Lipschitzb, Michelle Nga, Joseph Firthc,d,
⁎

a
Department of Psychiatry, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA, United States
b
Department of Psychiatry, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, United States
c
Division of Psychology and Mental Health, Faculty of Biology, Medicine and Health, University of Manchester, Manchester, United Kingdom
d
NICM Health Research Institute, Western Sydney University, Westmead, Australia

ABSTRACT

Background: Low engagement and attrition from app interventions is an increasingly recognized challenge for interpreting and translating the findings from digital
health research. Focusing on randomized controlled trials (RCTs) of smartphone apps for depressive symptoms, we aimed to establish overall dropout rates, and how
this differed between different types of apps.
Methods: A systematic review of RCTs of apps targeting depressive symptoms in adults was conducted in May 2019. Random-effects meta-analysis were applied to
calculate the pooled dropout rates in intervention and control conditions. Trim-and-fill analyses were used to adjust estimates after accounting for publication bias.
Results: The systematic search retrieved 2,326 results. 18 independent studies were eligible for inclusion, using data from 3,336 participants randomized to either
smartphone interventions for depression (n = 1,786) or control conditions (n = 1,550). The pooled dropout rate was 26.2%. This increased to 47.8% when adjusting
for publication bias. Study retention rates did not differ between depression vs. placebo apps, clinically-diagnosed vs. self-reported depression, paid vs. unpaid
assessments, CBT vs. non-CBT apps, or mindfulness vs. non-mindfulness app studies. Dropout rates were higher in studies with large samples, but lower in studies
offering human feedback and in-app mood monitoring.
Discussion: High dropout rates present a threat to the validity of RCTs of mental health apps. Strategies to improve retention may include providing human feedback,
and enabling in-app mood monitoring. However, it critical to consider bias when interpreting results of apps for depressive symptoms, especially given the strong
indication of publication bias, and the higher attrition in larger studies.

1. Introduction
The global unmet need for mental health services combined with
the proliferation of smartphones has created unprecedented interest in
digital mental health. Depression is the leading cause of disability
worldwide (Depression 2019), yet fewer than 50% of people around the
world have access to the necessary in-person treatment. The possibility
of delivering effective care to the palm of their hand via a smartphone
presents a path to scalable and accessible mental health care. This po-
tential of smartphone apps to screen, monitor, and even augment
treatment for mood disorders generates enthusiasm among not only
patients and clinicians, but also technology companies, investors, and
healthcare regulators. Already, depression apps are one of the most
downloaded categories of health apps by the public (Krebs and
Duncan, 2015).
A number of studies have evaluated the effectiveness of mood apps
with positive results. A 2017 meta-analysis of randomized controlled
trials found that smartphone apps targeting depressive symptoms can
be efficacious, although the exact effect size is impacted by the com-
parison group (Firth et al., 2017). Systematic reviews from 2019 also
support positive research results of smartphone apps for depression
(Kerst et al., 2019). Despite the accessibility and supporting evidence of
mood apps, however, research studies and implementation efforts de-
monstrate the difficulties of realizing the potential of digital mental
health. High rates of dropout and lack of engagement with apps are two
widely acknowledged barriers, although the mechanisms and reasons
for such remain unclear. A 2019 study purchased independently col-
lected real world app use data suggested that the median rate of daily
engagement, as measured by opening a mental health app, is 4%
(Kerst et al., 2019). Engagement and dropout issues are also present in
research studies and in this paper we focus on that data as it is available
for analysis unlike with commercial app data or expensive samples from
analytics firms (Baumel et al., 2019).
Low engagement with digital health tools is not unique to mental
health, impacting conditions as varied as asthma (Chan et al., 2017)
and diabetes (Rossi and Bigi, 2017). In a review of digital self-help tools

⁎
Corresponding author at: Division of Psychology and Mental Health, Faculty of Biology, Medicine and Health, University of Manchester, Manchester, United
Kingdom.
E-mail address: joseph.firth@manchester.ac.uk (J. Firth).

https://doi.org/10.1016/j.jad.2019.11.167
Received 25 September 2019; Received in revised form 13 November 2019; Accepted 30 November 2019
Available online 03 December 2019
0165-0327/ © 2019 Elsevier B.V. All rights reserved.
J. Torous, et al.
Fig. 1. PRISMA Search diagram.
— including apps for depression, low mood, and anxiety — the authors
reported that completion rates for tools in clinical studies ranged from
44 to 99%, and completion rates for the same tools in real-world set-
tings ranged from 1–28% (Fleming et al., 2018). While user experience
is often cited as a source of high dropout rates, many other potential
causes range from lack of perceived value (Torous et al., 2018) to
technological privacy concerns (Huckvale et al., 2019).
Understanding the factors that lead to poor engagement from
mental health apps is hindered by lack of standardized reporting and
lack of data availability. For example, app companies do not make their
engagement or dropout data publicly available. While these phenomena
can be explored through analyses of published research studies, the
research literature also present challenges in understanding use pat-
terns. A 2019 review of mental health apps’ usability and engagement
reported that every single included study claimed high rates of en-
gagement for their app, despite each study using a unique outcome
variable or scale to make that claim (Ng et al., 2019). The lack of
standardized reporting on user engagement impedes researchers’ ability
to learn from what has and has not worked for others, including the
factors that drive high rates of dropout.
Theoretical models like the non-adoption, abandonment, scale-up,
spread, and sustainability (NAASSS) (Greenhalgh et al., 2017) scale
explore potential causes for dropout across several domains, such as
whether the technology is challenging to use, does not offer value/
useful resources to users, and is not adopted by clinicians and patients
outside of studies. Each of these factors in the NAASSS can be informed
from clinical studies data: challenges to use could be explained if stu-
dies are conducted with non-representative populations or additional
technology support not available in routine practice; challenges of
value/utility could be explained if studies are offering interventions
without control groups or confounding by additional clinical support;
and challenges of adoption could be explained if studies are inherently
biased or non-translatable in their design. Conceptual models from
Mohr et al. (2017) also underscore the importance of conducting digital
mental health research in the actual setting where the technology is to
be clinically deployed and understanding how trial results may not
414
Journal of Affective Disorders 263 (2020) 413–419
always inform real world results {Mohr, 2017 #13,797}.
Study dropout in existing mental health app trials offers a stan-
dardized and practical proxy for beginning to better understand clinical
engagement. In this meta-analysis, we explore factors associated with
dropout from RCTs of smartphone apps targeting depressive symptoms.
Our aim was to evaluate the degree to which study sample, study design
and aspects of apps being studied impacted participants likelihood of
completing post-treatment assessments. The primary goal was to build
insight into variables that may impact user engagement. Additionally,
this analysis offers benchmarks on what can be expected in terms of
study retention in mental health mobile app trials to help guide in-
vestigators’ in conducting a priori power analyses and set a standard
upon which to improve as research practices in this nascent field con-
tinue to develop.
2. Methods
This meta-analysis followed the PRISMA statement for transparent,
comprehensive reporting of methodology and results (Firth et al., 2017)
and adhered to a strict protocol registered in PROSPERO
(CRD42019146179). A systematic search was performed of Ovid
MEDLINE, Embase, Cochrane Central Register of Controlled Trials,
Allied and Complementary Medicine, (AMED), Health Management
Information Consortium (HMIC), Health Technology Assessment
(HTA), and PsycINFO on 25th May 2019, using the keyword search
algorithm: “depression” OR “depressive” OR “mental illness” OR
“mental health” OR “mood disorder” OR “affective disorder” OR “an-
xiety” OR “panic disorder” OR “phobia” OR “bipolar” OR “psychosis”
OR “schizophr*” AND “smartphone*” OR “mobile phone*” OR “cell
phone*” OR “iphone” OR “android” OR “mhealth” OR “mobile appli-
cation” OR “phone application” AND “randomised” OR “randomized”
OR “randomly” OR “random assignment” OR “controlled trial” OR
“clinical trial” OR “control group” OR “intervention”. The search details
are presented in Fig. 1.
J. Torous, et al.
2.1. Eligibility criteria
Only articles published in English-language, peer-reviewed journals
were included. Since we sought to quantify the dropout from smart-
phone apps targeting depressive symptoms (regardless of clinical
status), all clinical and non-clinical populations, including those with
depression (e.g. major depressive disorder [MDD]), other psychiatric
conditions, and physical illnesses, as well as samples that did not reach
clinical thresholds, were eligible for inclusion.
Eligible studies were randomized controlled trials (RCTs) of inter-
ventions targeting adult (18+) individuals that (1) required use of a
mobile device app as part of the experimental condition in which de-
pressive symptoms were a primary outcome of interest and (2) reported
retention in post-treatment assessments. All smartphone-based inter-
ventions were eligible, including self-help apps, mindfulness apps,
smartphone platforms for connecting individuals with health services,
therapy apps, and mood tracking apps, provided that the software and
clinical study sought to target symptoms of depression. Smartphone
interventions focusing on multiple aspects of mental health were only
eligible if depression was maintained as a primary outcome. Studies
that specified other health concerns (e.g. anxiety, stress) as the primary
outcome, while featuring a measure of depressive symptoms only as a
secondary outcome, were excluded from this review.
Studies using all types of control groups were eligible for inclusion.
For the purpose of comparing dropout rates in app intervention con-
ditions to control conditions, the control groups in included studies
were categorized as (a) ‘Waitlist controls’ for studies in which partici-
pants assigned to the control condition did not receive any study-de-
livered treatment during the intervention period; (b) ‘Placebo app
controls’ for studies which assigned participants to either the ‘active’
app condition (i.e. that targeted depression) or an alternative control
app condition, which was not designed to treat depressive symptoms in
any meaningful way; and (c) ‘non-app controls’ for studies that included
a control condition that offered other forms of activities, such as edu-
cational resources, in-person interventions, or ‘time and attention’
matched patient contact.
All studies matching the above criteria and reporting enrollment
and retention rates in sufficient detail for meta-analysis were included.
Two independent investigators judged article eligibility (JF and JL),
with any disagreements resolved through mediation with a third in-
vestigator (JT).
2.2. Data extraction
A systematic extraction process was applied to retrieve the fol-
lowing data from each included study:
(i) Sample information: total size (n), gender distribution (% males),
mean participant age (in years), population sample details (in-
cluding diagnostic information, or health-related inclusion cri-
teria).
(ii) Intervention details: primary aspects of the smartphone app being
tested (e.g. whether it delivered aspects of cognitive-behavioral
therapy, cognitive training, mindfulness, mood monitoring, clin-
ician feedback, psychoeducation on physical health concerns and/
or aspects of gamification), control conditions (waitlist, placebo
app, or non-app comparator, as specified above), and intervention
length (in weeks).
(iii) Dropout rates: the number of participants in each group (inter-
vention or control) who completed the end-of-intervention as-
sessments, divided by the total number randomized to that con-
dition, were calculated for each arm of the study. For studies which
used >1 active intervention condition (i.e. two or more arms
which tested different smartphone apps for depression), the
dropout rates for each group were calculated individually (i.e.
treated as independent groups).
415
Journal of Affective Disorders 263 (2020) 413–419
2.3. Statistical analyses
Meta-analyses were conducted with Comprehensive Meta-Analysis
2.0, using a random-effects model to account for the expected hetero-
geneity between studies. The total dropout rates were calculated for
each study as the number of participants ‘completing’ the study (de-
fined as participation in intervention endpoint data collection), divided
by the number randomized to the study. Pooled dropout rates were
calculated for each arm of included studies separately to determine the
overall dropouts in app-based interventions for depressive symptoms
and control conditions as separate percentages figures (with 95% con-
fidence intervals). After overall dropout rates were calculated in app
interventions, risk of publication bias was examined by applying
Eggers’ regression to all aforementioned analyses. Furthermore, a Duval
and Tweedie's ‘trim-and-fill’ analysis (Duval and Tweedie, 2000) was
applied to the random-effects models, in order to re-calculate the
pooled dropout rates after statistically accounting for any studies that
may introduce publication bias. For all analyses, the degree of statistical
heterogeneity was quantified using I² values, with >25% representing
low heterogeneity, 25–50% for moderate heterogeneity, and >50% for
high heterogeneity.
Subgroup analyses were also conducted to examine how dropout
rates in RCTs of smartphone apps for depression differed between: (i)
studies using psychiatric vs. non-psychiatric samples (psychiatric sam-
ples were defined as those with participants with clinically diagnosed
mental disorders and/or undergoing psychiatric treatment), (ii) studies
applying inclusion criteria to ensure recruited participants had clini-
cally determined depressive symptoms vs. those recruiting with self-
report from the general population, (iii) studies specifying the use of
financial incentives for participants to complete end-of-intervention
assessments vs. those that did not, and (iv) app-based interventions
which incorporated elements of CBT, mindfulness, mood monitoring, or
human feedback within their platforms. Furthermore, a range of meta-
regression analyses were used to examine the putative study/partici-
pant characteristics which may affect dropout rates from the inter-
ventions. Specifically, a ‘logit event rate’ was computed for each study
as the logarithm of the proportional number of drop-outs in that study,
from the initially recruited sample. A total of four independent mixed-
effects regression models were then ran to examine how study char-
acteristics were related to drop-out rates. In each model, the logit event
rate (representing drop-out) was set as the response variable, and pre-
dictor variables were continuous values representing gender distribu-
tion of the study (as% of male participants), sample age (years, mean
average), sample size of the study (total n), and intervention length
(number of weeks).
3. Results
3.1. Included studies
The full systematic search retrieved a total of 2326 results.
Following the removal of duplicate articles from various electronic
databases, 1278 articles were screened at the title-and-abstract phase.
This identified 94 articles as potentially eligible, which were subse-
quently screened in full. Full text screening resulted in the exclusion of
76 for reasons specified in Fig. 1, which details the full PRISMA search
and screening process.
Therefore, 18 independent studies (Arean et al., 2016; Birney et al.,
2016; Cox et al., 2018; Depp et al., 2015; Faurholt-Jepsen et al., 2015;
Fitzpatrick et al., 2017; Flett et al., 2019; Howells et al., 2016;
Lüdtke et al., 2018; Ly et al., 2015; Ly et al., 2014; Mantani et al., 2017;
O'Toole et al., 2019; Proudfoot et al., 2013; Reid et al., 2011;
Roepke et al., 2015; Stiles-Shields, 2018; Watts et al., 2013) were eli-
gible for inclusion with dropout data from 3336 participants rando-
mized to either smartphone interventions for depression (n = 1786) or
control conditions (n = 1550). The mean sample ages ranged from 19 –
J. Torous, et al.
50 years old (median = 39 years) and intervention ranged from 10 days
to 6 months in length. The majority of studies (N = 14) used eligibility
criteria to require enrolled participants to be experiencing symptoms of
depression. However, only 8 of the 18 studies used psychiatric popu-
lations for the studies (i.e. current diagnoses or treatment for psychia-
tric conditions (Depp et al., 2015; Faurholt-Jepsen et al., 2015; Ly et al.,
2015; Ly et al., 2014; Mantani et al., 2017; O'Toole et al., 2019;
Reid et al., 2011; Watts et al., 2013). Across the 18 studies, a total of 22
different apps were used in active treatment conditions (with only 2
studies using the same app, Headspace). Interventions were compared
to waitlist control (Birney et al., 2016; Cox et al., 2018;
Fitzpatrick et al., 2017; Lüdtke et al., 2018), non-app control inter-
ventions (Depp et al., 2015; Ly et al., 2015; Mantani et al., 2017;
O'Toole et al., 2019; Proudfoot et al., 2013; Reid et al., 2011;
Roepke et al., 2015; Stiles-Shields, 2018; Watts et al., 2013), or placebo
app control conditions (Arean et al., 2016; Faurholt-Jepsen et al., 2015;
Flett et al., 2019; Howells et al., 2016).
3.2. Meta-Analysis of dropout rates from apps for depression
The results of primary and subgroup analyses on dropout rates in
Fig. 2. Random effects meta-analyses of dropout rates in RCTs of smartphone apps for depression, comparing intervention arms to various control conditions.
416
Journal of Affective Disorders 263 (2020) 413–419
RCTs of smartphone apps for depression are displayed in Fig. 2. Across
18 studies and 23 app-based interventions for depression (n = 1786),
the pooled dropout rate in intervention arms was 26.2% (95%
C.I. = 18.12% to 36.34%), with significant heterogeneity (I2 = 93.4)
and a strong indication of publication bias (Eggers t = 3.9, p<0.001). A
trim-and-fill analysis identified 9 studies as potentially affecting the
findings due to publication bias and produced an adjusted estimate of
47.8% dropout rate in RCTs of smartphone apps for depression (95%
C.I. = 35.8% to 60.0%).
3.3. Comparative meta-analyses of dropout rates in control conditions
The dropout rates observed in the app intervention conditions did
not significantly differ from dropout rates in placebo app conditions
(n = 422, k = 4, Est. 25.1%, 95% C.I. = 11.34% to 46.75%, I2 = 93.1)
or waitlist control conditions (n = 346, k = 4, Est. 20.45%, 95%
C.I. = 5.14% to 54.9%, I2 = 95.6), with between-group p-values ex-
ceeding 0.50. However, the dropout rates reported for app intervention
conditions were notably higher than the dropout rates reported in 11
‘non-app control intervention’ arms (i.e. comparator treatments which
did not involve app interventions), where there was only 14.2%
J. Torous, et al. Journal of Affective Disorders 263 (2020) 413–419

Table 1
Dropout rates in RCTs of smartphone apps for depression.
] Total N= Studies Dropout Est. (%) 95% Confidence Intervals I2

Overall dropout rates from Smartphone Apps for Depression

Unadjusted Dropout 1786 23 26.22 18.12 36.34 93.4
Trim Fill Adjusted – – 47.8 35.8 60.0 –
Dropout rates in Control Conditions Between-group comparisons (a)
p-value
Placebo Apps 422 4 25.095 11.337 46.748 93.1 0.914
Non-App Control 782 11 14.208 8.236 23.406 83.9 0.053
Waitlist Control 346 4 20.452 5.143 54.937 95.6 0.696
Subgroup comparisons for dropouts for smartphone intervention conditions Between-group comparisons (b)
Inclusion Criteria p-value
Symptomatic 1489 18 30 20.23 42 93.6
Not required 297 5 15.81 6.219 34.71 89.0 0.17
Sample type
Non-psychiatric 1359 14 31.86 20.722 45.56 94.5
Psychiatric 427 9 18.76 9.76 33.03 86.9 0.15
Paid for Assessments
None specified 1017 14 26.30 15.4 41.2 93.8
Yes 969 9 25.20 13.6 41.7 93.5 0.904
App types
Not CBT 1120 14 27.3 17.362 40.15 93.6
CBT 666 9 23.06 10.12 44.38 94.0 .698
Not Mindfulness 1037 13 24.85 16.281 35.99 90.3
Mindfulness 749 10 29.32 13.91 51.57 95.7 .681
No mood monitor 879 10 37.88 23.109 55.31 94.4
Mood monitor 907 13 18.42 10.92 29.38 90.3 0.037
Not person feedback 1513 16 33.96 23.532 46.22 94.2
Real-person feedback 273 7 11.74 6.05 21.56 62.2 0.003

Footnote: Pooled estimates of drop-out rates are presented as% values, with 95% confidence intervals. Between study heterogeneity for pooled estimates are
presented as I2 values. P-values for Between Group comparisons represent the statistical significance of difference in pooled drop-out rates between (a) different types
of control groups in comparison to the ‘active’ app conditions, and (b) different sub-sets of ‘active’ apps, with regards the app components listed in far-left column.

dropout (n = 782, 95% C.I. = 8.236 to 23.406, I2 = 83.9) bordering
toward statistically significantly less than the app intervention arms
(p = 0.053 between-groups). Comparisons of dropout rates between
intervention apps, control apps, and other control groups are shown in
Fig. 2.
3.4. Subgroup comparisons of dropout rates across sample, study design and
intervention types
Subgroup analyses were performed to examine how sample char-
acteristics and intervention types impacted dropout rates from smart-
phone apps for depression. All results are presented in Table 1. Dropout
rates did not vary significantly between studies using non-psychiatric
(n = 1359, k = 14, Est. 31.86%, 95% C.I. = 20.7% to 45.6%,
I2 = 94.5) and psychiatric samples (n = 427, k = 9 Est. 18.76%, 95%
C.I. = 9.76% to 33.03% I2 = 86.9). Similarly, dropouts did not differ
between studies using eligibility criteria to ensure participants had
depressive symptoms (n = 1489, k = 18, Est. 30.0%, 95%
C.I. = 20.23% to 42%, I2 = 93.6) and those recruiting from the general
population without symptomatic inclusion criteria (n = 297, k = 5,
Est. 15.81%, 95% C.I. = 6.22% to 34.7%, I2 = 89.0). Finally, dropout
from interventions in studies that specified the use of financial in-
centives for completing assessments (k = 9, n = 769, Est. 25.2%, 95
C.I. = 13.6% to 41.7%, I2 = 93.5) were no different from studies that
did not (k = 14, n = 1017, Est. 26.3%, 95% C.I. = 15.4% to 41.2%,
I2 = 93.8).
Dropout from smartphone interventions that employed aspects of
CBT (n = 666, k = 9, Est. 23.06%, 95% C.I. = 10.12% to 44.38%
I2 = 94.0) or mindfulness (n = 749, k = 10 Est. 29.32%, 95%
C.I. = 13.9% to 51.6%, I2 = 95.7) within their treatments did not
differ significantly from those that did not. However, studies testing
apps that integrated mood monitoring into their platforms (k = 13,
n = 907) had significantly lower dropout rates (Est. 18.42%, 95%
C.I. = 10.92% to 29.38%, I2 = 90.3) than those that did not (n = 879,
k = 10, Est. 37.88%, 95% C.I. = 23.109 to 55.31, I2 = 94.4), with
between-group comparison of p = 0.037. Additionally, studies of 7
apps that involved human feedback to the users (n = 273) had sig-
nificantly lower dropout rates (Est. 11.74%, 95% C.I. = 6.05% to
21.6%, I2 = 62.2) compared to those without human feedback
(n = 1513 k = 16, Est. 33.96%, 95% C.I. = 23.53% to 46.22%,
I2 = 94.2, between-groups comparison p = 0.003).
Meta-regressions of study characteristics and dropout rates: Meta-re-
gression analyses found no relationships between dropout rates with
study length in weeks (B = 0.00569, S.E.=0.05874, Z = 0.0970,
p = 0.923), mean sample age (B = 0.0512, S.E.=0.0338, Z = 1.51,
p = 0.130) and% male gender distribution of sample (B=−0.0262,
S.E.=0.0271, Z=−0.969, p = 0.333). However, as shown in Fig. 3,
there was a statistically significant relationship between enrolled
sample size and dropout rates from app-based interventions for de-
pression, with studies using larger sample sizes having greater dropout
rates (B = 0.00826, S.E.=0.00415, Z = 1.993, p = 0.046).
4. Discussion
In this meta-analysis of dropout from RCTs of smartphone apps
targeting depressive symptoms, we found a mean dropout rate of 26.2%
that increased to 47.8% when accounting for publication bias. Study
retention rates did not differ between depression vs. placebo apps,
clinically diagnosed vs. self-reported depression participants, paid vs.
unpaid assessments, CBT vs. non-CBT app studies, or mindfulness vs.
non-mindfulness app studies. Retention rates, however, were higher in
studies of apps offering human feedback and mood monitoring as well
as in non-app-based control conditions. Larger studies had a higher
dropout.
These findings have clear implications for powering, designing, and
interpreting app-based studies. The unadjusted dropout rate of 26.2%
and the trim-and-fill adjusted rate of 47.8% should be used when de-
termining sample size for RCTs of depression apps. Additionally, these
attrition rates indicate significant threat to the external and internal
validity of studies evaluating apps and even the apps themselves

417
J. Torous, et al.
Fig. 3. Random effects meta-regression analyses, showing the relationship between total sample size (N, x-axis) with dropout rates (logit event rate, y-axis).
(Roepke et al., 2015). The magnitude of this bias is apparent when
considering a recent meta-analysis of dropout from exercise studies
targeting depression. In that study, the authors adjusted dropout rates
from 15.2% to 18.1% to account for publication bias — a difference of
2.9% (Stubbs et al., 2016), compared to our adjustment of 21.6% for
smartphone apps targeting depression. While high, these dropouts are
in line with other mental health treatments, which include dropout
rates of approximately 30% in treatment retention studies of anti-
depressant medications (Lurie and Levine, 2010) and 47% from psy-
chotherapy (Wierzbicki and Pekarik, 1993).
Despite high dropout rates, our results also suggest actionable mi-
tigation strategies. Two aspects of the findings from this meta-analysis
suggest that integrating a human component into treatment reduces
study attrition. Firstly, non-app, non-waitlist control arms showed
lower study attrition. These study arms heavily consisted of human
support. Secondly, study dropout was lower when the depression app
involved human feedback. While adding human support may boost
engagement, it could potentially reduce scalability and challenge the
popular notion of apps as panaceas for low resource settings. Taken
together, these findings indicate that studies of standalone apps that do
not involve any human feedback can expect higher study dropout rates.
Results from real world market analytical research on mental health
app engagement that suggest rates of 17% for peer support apps com-
pared to the mean of 4% (Baumel et al., 2019) offer further support for
this conclusion.
These findings also have clinical implications. The findings that
dropout from treatment apps was equivalent to that of placebo apps or
waitlist control arms mirror clinical evidence of challenges engaging
patients with apps. Since research participants are often more moti-
vated, this lack of retention beyond placebo and waitlist underscores
the formidable barrier in translating app studies into the clinical set-
ting. Clinicians should anticipate high dropout when using apps and
thus introduce them as part of a comprehensive plan with mechanisms
built in to increase engagement, such as offering feedback and mood
monitoring. This finding aligns with prior meta-analyses of depressive
symptoms apps suggesting that feedback may be the single most im-
pactful factor when considering efficacy (Firth et al., 2017).
Another notable finding is that contrary to expectations, dropout
was not driven by assessment compensation. Said another way, mobile
app intervention study retention was not a function of compensation. If
compensation does not drive study retention, the question is what does?
Time required to complete assessments is one plausible hypothesis as is
418
Journal of Affective Disorders 263 (2020) 413–419
clinical/personal utility of the intervention being tested. To the extent
that the latter is true, study dropout may reflect how engaged partici-
pants are with the intervention being tested. Beyond the above hy-
pothesis, this finding provides evidence against the argument of in-
corporating payment into clinical applications of mobile apps.
While study retention rates suggest several research and clinical
implications, other implications for clinical care are more difficult to
interpret. For example, the lack of difference in dropout with depres-
sion for clinical vs. non-clinical settings as well as symptomatic vs. non-
symptomatic populations could have a number of interpretations.
Probably the most parsimonious is that these apps can be engaging for
diverse populations from cases of clinical depression to cases of well-
ness enhancement. The conclusion then is that these apps may appro-
priately be positioned as direct-to-consumer tools for optimizing health
or as clinical tools for use in treatment based on their specific content
and design.
Similarly, the finding that drop out was lower in smaller studies is
difficult to interpret. One conjecture, which was not evaluated in this
meta-analysis is that smaller studies lent themselves to more in-
dividualized attention to each participant. In the future, it will be useful
for mobile app researchers to carefully track and publish amount of
human interaction time (including technical support) for each partici-
pant to better understand how hands on these interventions really are.
Such implementation outcomes are key to the ultimate scalability of
mobile app interventions.
Strengths of our analysis include strict adherence to a registered
protocol, recency of the search, and screening and evaluation of re-
sulting research studies by two clinician researchers familiar with both
using and studying these apps. Limitations include the inclusion of only
randomized controlled trials as well the heterogeneity of apps, study
designs, clinical approaches, and participant populations analyzed.
Furthermore, the lack of standardized reporting of treatment retention
forced us to limit our meta-analysis to study retention rather than
treatment retention (Ng et al., 2019). While findings related to study
retention are useful in informing research and can serve as a basis for
preliminary hypotheses of what may drive engagement, our findings do
not directly evaluate patient engagement with apps. More research on
treatment retention is essential to the growth of this field. One solution
to accelerate that research may be industry-academic partnerships
where the companies behind successful apps work with academic teams
to report on successful strategies to boost engagement and reduce
dropout.
J. Torous, et al.
5. Conclusions
The potential of digital mental health apps to increase access to care
and deliver evidence-based interventions has fueled global interest,
investment, and research. Our results suggest that realizing this promise
must be considered through the potential lens of strong publication bias
and the underlying realities of dropout inherent in clinical intervention
studies. However, the digital health field is uniquely suited to rapid
adaptations and adjustments, meaning that progress and new solutions
are always on the horizon.
Author disclosure
Disclosures: JT reports unrelated funding for a digital health project
from Otsuka.
Role of the funding source
NA
Author statement
All authors contributed equally
Acknowledgements
JT is supported by a career development award from the NIMH:
1K23MH116130-01.
Supplementary materials
Supplementary material associated with this article can be found, in
the online version, at doi:10.1016/j.jad.2019.11.167.
References
Depression, 2019. Key Facts. World Health Organization. https://www.who.int/en/
news-room/fact-sheets/detail/depression.
Krebs, P., Duncan, D.T, 2015. Health app use among US mobile phone owners: a national
survey. JMIR mHealth uHealth 3 (4), e101.
Firth, J., Torous, J., Nicholas, J., Carney, R., Pratap, A., Rosenbaum, S., Sarris, J, 2017.
The efficacy of smartphone‐based mental health interventions for depressive symp-
toms: a meta‐analysis of randomized controlled trials. World Psychiatry 16 (3),
287–298.
Kerst, A., Zielasek, J., Gaebel, W, 2019 Jan 3. Smartphone applications for depression: a
systematic literature review and a survey of health care professionals’ attitudes to-
wards their use in clinical practice. Eur. Arch. Psychiatry Clin. Neurosci. 1–4.
Baumel, A., Muench, F., Edan, S., Kane, J.M, 2019. Objective user engagement with
mental health apps: systematic search and panel-based usage analysis. J. Med.
Internet Res. 21 (9), e14567.
Chan, Y.F., Wang, P., Rogers, L., Tignor, N., Zweig, M., Hershman, S.G., Genes, N., Scott,
E.R., Krock, E., Badgeley, M., Edgar, R, 2017. The asthma mobile health study, a
large-scale clinical observational study using ResearchKit. Nat. Biotechnol. 35 (4),
354.
Rossi, M.G., Bigi, S, 2017. mHealth for diabetes support: a systematic review of apps
available on the Italian market. Mhealth 3, 16.
Fleming, T., Bavin, L., Lucassen, M., Stasiak, K., Hopkins, S., Merry, S, 2018. Beyond the
trial: systematic review of real-world uptake and engagement with digital self-help
interventions for depression, low mood, or anxiety. J. Med. Internet Res. 20 (6), e199.
Torous, J., Nicholas, J., Larsen, M.E., Firth, J., Christensen, H, 2018. Clinical review of
user engagement with mental health smartphone apps: evidence, theory and im-
provements. Evid. Based Ment. Health 21 (3), 116–119.
Huckvale, K., Torous, J., Larsen, M.E, 2019. Assessment of the data sharing and privacy
practices of smartphone apps for depression and smoking cessation. JAMA Netw.
Open 2 (4), e192542.
Ng, M.M., Firth, J., Minen, M., Torous, J, 2019. User engagement in mental health apps: a
419
Journal of Affective Disorders 263 (2020) 413–419
review of measurement, reporting, and validity. Psychiatric Serv. 70, 538–544
appi-ps.
Greenhalgh, T., Wherton, J., Papoutsi, C., Lynch, J., Hughes, G., Hinder, S., Fahy, N.,
Procter, R., Shaw, S, 2017. Beyond adoption: a new framework for theorizing and
evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and
sustainability of health and care technologies. J. Med. Internet Res. 19 (11), e367.
Mohr, D.C., Weingardt, K.R., Reddy, M., Schueller, S.M, 2017. Three problems with
current digital mental health research... and three things we can do about them.
Psychiatric Serv. 68 (5), 427–429.
Duval, S., Tweedie, R, 2000 Jun. Trim and fill: a simple funnel‐plot–based method of
testing and adjusting for publication bias in meta‐analysis. Biometrics 56 (2),
455–463.
Arean, P.A., Hallgren, K.A., Jordan, J.T., et al., 2016. The use and effectiveness of mobile
apps for depression: results from a fully remote clinical trial. J. Med. Internet Res. 18,
e330.
Birney, A.J., Gunn, R., Russell, J.K., et al., 2016. MoodHacker mobile web app with email
for adults to self-manage mild-to-moderate depression: randomized controlled trial.
JMIR mHealth uHealth 4, e8.
Cox, C.E., Hough, C.L., Jones, D.M., et al., 2018. Effects of mindfulness training pro-
grammes delivered by a self-directed mobile app and by telephone compared with an
education programme for survivors of critical illness: a pilot randomised clinical trial.
Thorax 23.
Depp, C.A., Ceglowski, J., Wang, V.C., et al., 2015. Augmenting psychoeducation with a
mobile intervention for bipolar disorder: a randomized controlled trial. J. Affect
Disord. 174, 23–30.
Faurholt-Jepsen, M., Frost, M., Ritz, C., et al., 2015. Daily electronic self-monitoring in
bipolar disorder using smartphones - the MONARCA I trial: a randomized, placebo-
controlled, single-blind, parallel group trial. Psychol. Med. 45, 2691–2704.
Fitzpatrick, K.K., Darcy, A., Vierhile, M., 2017. Delivering cognitive behavior therapy to
young adults with symptoms of depression and anxiety using a fully automated
conversational agent (Woebot): a randomized controlled trial. JMIR Mental Health 4,
e19.
Flett, J.A., Hayne, H., Riordan, B.C., et al., 2019. Mobile mindfulness meditation: a
randomised controlled trial of the effect of two popular apps on mental health.
Mindfulness 10, 863–876.
Howells, A., Ivtzan, I., Eiroa-Orosa, F.J., 2016. Putting the 'app' in happiness: a rando-
mised controlled trial of a smartphone-based mindfulness intervention to enhance
wellbeing. J. Happiness Stud. 17, 163–185.
Lüdtke, T., Pult, L.K., Schröder, J., et al., 2018. A randomized controlled trial on a
smartphone self-help application (Be good to yourself) to reduce depressive symp-
toms. Psychiatry Res. 269, 753–762.
Ly, K.H., Topooco, N., Cederlund, H., et al., 2015. Smartphone-Supported versus full
behavioural activation for depression: a randomised controlled trial. PLoS ONE 10,
e0126559.
Ly, K.H., Truschel, A., Jarl, L., et al., 2014. Behavioural activation versus mindfulness-
based guided self-help treatment administered through a smartphone application: a
randomised controlled trial. BMJ Open 4, e003440.
Mantani, A., Kato, T., Furukawa, T.A., et al., 2017. Smartphone cognitive behavioral
therapy as an adjunct to pharmacotherapy for refractory depression: randomized
controlled trial. J. Med. Internet Res. 19, e373.
O'Toole, M.S., Arendt, M.B., Pedersen, C.M., 2019. Testing an app-assisted treatment for
suicide prevention in a randomized controlled trial: effects on suicide risk and de-
pression. Behav. Ther. 50, 421–429.
Proudfoot, J., Clarke, J., Birch, M.R., et al., 2013. Impact of a mobile phone and web
program on symptom and functional outcomes for people with mild-to-moderate
depression, anxiety and stress: a randomised controlled trial. BMC Psychiatry 13,
132.
Reid, S., Kauer, S., Patton, G., 2011. Using mobile phones to detect, treat, and manage
adolescent mental halth: a randomised controlled trial of the mobiletype program in
rural and metro primary care. Eur. Psychiatry 12, 131 Conference: 19th European
Congress of Psychiatry, EPA 26.
Roepke, A.M., Jaffee, S.R., Riffle, O.M., et al., 2015. Randomized controlled trial of su-
perbetter, a smartphone-based/internet-based self-help tool to reduce depressive
symptoms. Games for Health 4, 235–246.
Stiles-Shields, C, Montague, E, Kwasny, MJ, Mohr, DC, 2018. Behavioral and cognitive
intervention strategies delivered via coached apps for depression: Pilot trial. Psychol
Serv 16 (2), 233–238.
Watts, S., Mackenzie, A., Thomas, C., et al., 2013. CBT for depression: a pilot RCT
comparing mobile phone vs. computer. BMC Psychiatry 13.
Stubbs, B., Vancampfort, D., Rosenbaum, S., Ward, P.B., Richards, J., Soundy, A.,
Veronese, N., Solmi, M., Schuch, F.B, 2016. Dropout from exercise randomized
controlled trials among people with depression: a meta-analysis and meta regression.
J. Affect. Disord. 190, 457–466.
Lurie, I., Levine, S.Z, 2010. Meta-analysis of dropout rates in SSRIs versus placebo in
randomized clinical trials of PTSD. J. Nerv. Ment. Dis. 198 (2), 116–124.
Wierzbicki, M., Pekarik, G, 1993. A meta-analysis of psychotherapy dropout. Profess.
Psychol. Res. Pract. 24 (2), 190.