ADHD Attention Deficit and Hyperactivity Disorders

https://doi.org/10.1007/s12402-018-0256-y

ORIGINAL ARTICLE

Performance‑based measures and behavioral ratings of executive

function in diagnosing attention‑deficit/hyperactivity disorder
in children
Alexander Tan1,2   · Lauren Delgaty1,3 · Kayla Steward1,4 · Melissa Bunner1

Received: 16 June 2017 / Accepted: 9 April 2018

© Springer-Verlag GmbH Austria, part of Springer Nature 2018

Abstract
Deficits in real-world executive functioning (EF) are a frequent characteristic of attention-deficit/hyperactivity disorder
(ADHD). However, the predictive value of using performance-based and behavioral rating measures of EF when diagnos-
ing ADHD remains unclear. The current study investigates the use of performance-based EF measures and a parent-report
questionnaire with established ecological validity and clinical utility when diagnosing ADHD. Participants included 21
healthy controls, 21 ADHD—primary inattentive, and 21 ADHD—combined type subjects aged 6–15 years. A brief neu-
ropsychological battery was administered to each subject including common EF assessment measures. Significant differ-
ences were not found between groups on most performance-based EF measures, whereas significant differences (p < 0.05)
were found on most parent-report behavioral rating scales. Furthermore, performance-based measures did not predict group
membership above chance levels. Results further support differences in predictive value of EF performance-based measures
compared to parent-report questionnaires when diagnosing ADHD. Further research must investigate the relationship between
performance-based and behavioral rating measures when assessing EF in ADHD.

Keywords  ADHD · Executive function · BRIEF · Ecological validity

Introduction has established that one of the most prominent deficits in

Attention-deficit/hyperactivity disorder (ADHD), charac-
terized by developmentally inappropriate levels of inat-
tention, hyperactivity, and impulsivity, is one of the most
commonly diagnosed disorders in childhood. Much research
Alexander Tan and Lauren Delgaty have contributed equally to
this work.
* Alexander Tan
alexander.tan@utsouthwestern.edu
1
Austin Neuropsychology, PLLC, 711 W. 38th St. F‑2,
Austin, TX 78705, USA
2
Department of Psychiatry, University of Texas Southwestern
Medical Center, 5323 Harry Hines Blvd, Dallas, TX 75390,
USA
3 There has been ongoing interest in the research and clini-
Pinehurst Neuropsychology, 45 Aviemore Dr., Pinehurst,
NC 28374, USA
4 including both performance-based tasks and behavioral rat-
Department of Psychology, University of Alabama
at Birmingham, 1530 3rd Avenue South, Birmingham,
AL 35294, USA
ADHD is impairment in executive functioning (EF) (Bar-
kley 1997; Thaler et al. 2013; Willcutt et al. 2005). EF is a
complex cognitive process that is necessary for efficacious
goal-oriented behavior and includes a set of separate but
related cognitive abilities, such as attentional control, cog-
nitive flexibility, information processing, and goal-setting
(Anderson 2002; Di Trani et al. 2011; Gau and Shang 2010;
Johnson et al. 2001; Oosterlaan et al. 1998). In children with
ADHD, the most notable areas of EF deficits are often plan-
ning, working memory, response inhibition, and set shift-
ing (Gau and Shang 2010; Pennington and Ozonoff 1996).
These deficits may become more pronounced when the task
becomes more complex (Gau et al. 2009; Rommelse et al.
2007). Deficits in EF can have a considerable impact on
daily functioning across academic, social, emotional, and
behavioral domains.
cal utility of EF measures in diagnosing ADHD in children,
ings. Although deficits in real-world EF are often a primary
concern for children with ADHD, the ability to measure these

13
Vol.:(0123456789)

deficits through performance-based neuropsychological
assessments has been inconsistent. One problem with assess-
ing EF is the challenge in separating domain-specific functions
(e.g., memory, social–emotional, language) from actual EF
abilities (Gioia and Isquith 2004). However, Burgess (1997)
suggests that attempts to assess EF apart from any domain-
specific functions result in inadequate assessment of daily
functioning because they separate integrated abilities into
individual segments. As real-life situations are variable and
complex, approaching them successfully typically requires the
cognitive flexibility to adapt the skills of EF in an appropri-
ate manner. Others have suggested that real-life deficits often
remain undetected in performance-based assessments due to
the organization, structure, monitoring, and guidance provided
by the examiner, essentially becoming the examinee’s execu-
tive system during the task (Torralva et al. 2013). Due to these
constraints, neuropsychologists have encountered difficulties
designing performance-based measures of EF that demonstrate
ecological validity and clinical utility.
While some studies have demonstrated relatively robust
relationships between performance-based measures and eve-
ryday abilities in children, others have failed to find significant
relationships (Biederman et al. 2006; Jonsdottir et al. 2006).
Alternatively, parent-report questionnaires have demonstrated
clinical utility in assessing EF deficits with respect to the pres-
ence of ADHD. In particular, the Behavioral Rating Inven-
tory of Executive Function (BRIEF) has significant research
support for its ability to diagnose ADHD (Gioia et al. 2000;
McCandless and O’Laughlin 2007; Riccio et al. 2006). Find-
ings regarding the predictive value of performance-based
measures of EF versus parent-report questionnaires have been
inconsistent (Barkley and Murphy 2010; Biederman et al.
2008; Toplak et al. 2009). These variable results demonstrate
a need for further investigation into the clinical utility of EF
measures in individuals with ADHD.
To advance understanding of the relationship between per-
formance-based measures and behavioral ratings of EF and the
clinical utility of these measures in diagnosing ADHD, this
study sought to further investigate whether performance-based
EF tasks and the parent-report BRIEF demonstrated the same
results in distinguishing between control, ADHD—primary
inattentive (ADHD-PI), and ADHD—combined type (ADHD-
CT) groups. The hypothesis of this study was that both types
of measures would correctly identify impairments in EF where
they exist; however, it was thought that the performance-based
EF tasks would be less sensitive to these differences.
13
A. Tan et al.
Methods
Participants
This study was approved by the Institutional Review
Board (IRB) at Austin Neuropsychology, PLLC, and writ-
ten informed consent was obtained from all participants.
The clinical group was taken from archival data from a
private neuropsychology clinic. This group consisted of
42 consecutive children (57% male) aged 6–15 years who
underwent a comprehensive diagnostic neuropsychologi-
cal evaluation at the private neuropsychology clinic by
a board-certified neuropsychologist, which included par-
ent and patient interviews, multiple behavioral ratings,
and extensive neuropsychological testing. All measures,
including performance-based tests and parent behavioral
ratings, were individually scored and interpreted. Based
on results from the evaluation, participants were given a
Diagnostic and Statistical Manual of Mental Disorders
(4th ed.; DSM-IV; American Psychiatric Association
[APA] 1994) diagnosis of ADHD-PI (n = 21) or ADHD-
CT (n = 21). Nineteen percent of the sample were pre-
scribed medication to address attention symptoms at the
time of their evaluation; however, they were asked not
to take the medication on the day of assessment. While
most research protocol measures (outlined below) were
included in the initial neuropsychological evaluation, if a
participant did not complete one or more research protocol
measures, they were invited back to the clinic to complete
the missing measures within 6 months of their evaluation
so they could be included in the study. The parents of all
participants completed the BRIEF as part of their initial
neuropsychological evaluation.
The control group of this study included 21 HC (57%
male), aged 6–15 years, who were recruited by conveni-
ence sampling through the general community and archival
data from the private neuropsychological clinic. Control
group participants from the clinic underwent a compre-
hensive clinical neuropsychological evaluation but did not
meet criteria for ADHD. Control group participants from
the general community underwent a telephone screening
and were given the Swanson, Nolan, and Pelham (SNAP-
IV) 26-Item Parent Rating Scale to rule out significant
attentional difficulties or previous ADHD diagnosis. Fol-
lowing the phone screening, written informed consent was
obtained, parents completed the BRIEF, and participants
were administered research protocol measures.
Participants were excluded from experimental and con-
trol groups if they had a General Ability Index (GAI) of
less than 90 as determined by the Wechsler Intelligence
Scale for Children, 4th edition (WISC-IV), or the Wechsler
Abbreviated Intelligence Scale, 2nd Edition (WASI-II).
Performance‑based measures and behavioral ratings of executive function in diagnosing…
Participants were also excluded if they had any neurologi-
cal disease (e.g., epilepsy, traumatic brain injury), major
psychiatric illness (e.g., depression, anxiety, bipolar disor-
der), developmental disorder (e.g., autism, mental retarda-
tion), adjustment disorder, or oppositional defiant disorder.
Demographic and descriptive characteristics of clinical
and non-clinical children are presented in Table 1. Group
differences in the demographic variables were examined
with non-parametric Chi-square or a one-way analysis of
variance (ANOVA). No significant differences were found
in gender, age, or GAI between groups.
Materials
As described below, various parent-rated behavioral rat-
ings and performance-based measures of executive func-
tion were used in this study. Because the study used archi-
val data as the primary source for recruitment, the specific
performance-based measures used to determine EF deficits
were chosen due to consistency of use across evaluations.
Therefore, while behavioral ratings were able to capture
more complex aspects of executive functioning (e.g., plan-
ning/problem-solving, set shifting, and self-regulation), the
researchers were limited to performance-based EF measures
that required less processing demands (e.g., working mem-
ory, processing speed, impulsivity, and sustained attention).
SNAP‑IV 26‑Item Parent Rating Scale (Bussing et al. 2008)
The scale includes items from the DSM-IV criteria for
ADHD in two subsets: inattention (9 items) and hyperactiv-
ity/impulsivity (9 items). The scale also includes 8 items
from the DSM-IV criteria for Oppositional Defiant Disorder,
as this is a common comorbid diagnosis in children with
ADHD. The SNAP-IV is based on a 4-point scale related to
frequency of occurrence (Not at all, Just a little, Often, and
Very often). Attention symptoms were considered significant
if four or more items were rated as Often or Very Often in
any category. This measure was used to rule out attentional
concerns in the control group; if symptoms were significant,
the participant was excluded from the study.
Table 1  Participants’
demographic characteristics
Gender (male/female)
Age (year)
WISC-IV/WASI-II GAI
Data are mean ± standard deviation
ADHD-PI attention-deficit/hyperactivity disorder—primary inattentive, ADHD-CT attention-deficit/hyper-
activity disorder—combined type, HC healthy control, WISC-IV Wechsler Intelligence Scale for Chil-
dren—Fourth Edition, WASI-II Wechsler Abbreviated Scale of Intelligence—Second Edition, and GAI
General Ability Index
Behavior Rating Inventory of Executive Functioning (BRIEF):
Parent Form (Gioia et al. 2000)
The BRIEF was created to assess an individual’s EF dif-
ficulties through rating everyday behaviors at home, at
school, and in the community (Gioia and Isquith 2004).
The BRIEF is an 86-item rating scale that is completed
by the parent based on the child’s behavior over the last
6 months. Parents are asked to respond to each statement
regarding behavior using a frequency rating of never,
sometimes, or always. The items are divided into eight
subscales related to executive function: Inhibit (resisting
impulses), Shift (thinking flexibly and alternating atten-
tion), Emotional Control (modulate emotional responses),
Initiate (beginning tasks), Working Memory (holding
information in the mind for manipulation during a task),
Plan/Organization (anticipate future events and create
appropriate steps to meet a goal), Organization of Materi-
als (managing properties), and Monitor (self-check pro-
gression through tasks). The scored report consists of T
scores for each subscale as well as a Global Executive
Composite (all eight subscales), a Behavioral Regula-
tion Index (BRI: Inhibit, Shift, and Emotional Control),
a Metacognition Index (MI: Initiate, Working Memory,
Plan/Organize, Organization of Materials, and Moni-
tor), and two validity scales (Inconsistency and Negativ-
ity). Lower T scores indicate better performance on each
subscale.
Wechsler Abbreviated Scale of intelligence, 2nd edition
(WASI‑II) (Wechsler 2011)
This test was developed to accurately determine intelli-
gence in individuals ranging from 6 to 90 years of age
(McCrimmon and Smith 2013). The WASI-II was used to
determine the General Ability Index (GAI) for the con-
trol subjects in an efficient manner. The Vocabulary and
Matrix Reasoning subtest scores were used to determine
each subject’s GAI score.
ADHD-PI (n = 21) ADHD-CT (n = 21) HC (n = 21)
12/9 12/9 12/9
9.48 ± 2.56 9.48 ± 2.54 9.43 ± 2.48
104.29 ± 13.24 104.76 ± 11.04 104.70 ± 9.96
13

Wechsler Intelligence Scale for children, 4th edition
(WISC‑IV) (Wechsler 2003)
This measure is used to determine intelligence in children
who are between 6 and 16 years of age. It consists of ten
core subtests, which combine into four factor scores and a
Full-Scale Intelligence Quotient (FSIQ). The Verbal Com-
prehension Index (VCI) includes the Similarities, Vocabu-
lary, and Comprehension subtests; the Perceptual Reasoning
Index (PRI) includes the Block Design, Picture Concepts,
and Matrix Reasoning subtests; the Working Memory
Index (WMI) includes the Digit Span and Letter–Number
Sequencing subtests; and the Processing Speed Index (PSI)
includes the Coding and Symbol Search subtests. This study
utilized the WISC-IV to determine intellectual functioning
of subjects and ensure that they met the inclusion criteria of
a GAI above 90. The GAI is based on scores from the VCI
and PRI and is useful in assessing populations with ADHD,
learning disabilities, or other disorders who have deficits in
working memory and/or processing speed which lower their
FSIQ (Lanfranchi 2013).
Digit Span from WISC‑IV
This subtest was used to assess the working memory com-
ponent of EF and requires the participant to repeat digits
read aloud by the examiner in forward and reverse order.
Outcome was the number of correctly executed trials, and
equivalent standard scores were obtained.
Letter–Number Sequencing from WISC‑IV
This subtest was also used to assess the working memory
component of EF and requires the participant to repeat
strings of numbers and letters in combinations. Outcome
was the number of correctly executed trials, and equivalent
standard scores were obtained.
Coding from WISC‑IV
This subtest was used to assess the processing speed com-
ponent of EF and requires the participant to quickly respond
to a stimulus by copying a corresponding figure. Outcome is
the number of correct items, and equivalent standard scores
were obtained.
Symbol Search from WISC‑IV
This subtest was used to assess the processing speed com-
ponent of EF and requires the participant to quickly identify
whether a group of figures contains a stimulus figure. Out-
come is the number of correct items minus the number of
errors, and equivalent standard scores were obtained.
13
A. Tan et al.
Conners’ Continuous Performance Test, 2nd edition, version
5 (CPT‑II) (Conners 2004)
This measure is used to assess impulsivity, sustained, and
selective attention. The CPT-II was administered using com-
puter software. Respondents are asked to watch the screen
as letters are displayed for 250 ms in 1-, 2-, and 4-second
intervals. Administration time is 14 min. They are asked to
press the space bar when any letter is presented except the
letter ‘X’. Instead, when the letter ‘X’ is displayed, they are
to refrain from pressing the space bar. T scores based on age
and gender norms were obtained from the computer-gener-
ated scoring report. This study utilized the omission errors
(number of times the target was presented and the subject
did not respond), commission errors (number of times the
subject responded when a target was not present), and vari-
ability in response rate (based on the time it takes for the
subject to react to the stimulus).
Data analyses
ADHD-PI, ADHD-CT, and HC groups were equal in size
and matched by age and gender for the analyses. There were
no statistical outliers or missing data. Next, the difference
between performance-based measures and parent-report
behavioral ratings in assessing EF deficits in ADHD was
assessed. First, an ANOVA was conducted to compare dif-
ferences in performance on EF tasks and BRIEF results
across groups. Partial eta square (ηρ2) was calculated as an
estimate of effect size (Cohen 1988). Furthermore, the pre-
dictive value of the performance-based measures and BRIEF
data was assessed using a discriminant function analysis
(DFA) to predict group membership in both subtypes of
ADHD and non-ADHD groups. In accordance with Tabach-
nick and Fidell (2013), DFA was preferred over a logistic
regression due to the goal of predicting group membership,
the dependent variable being nominal and more than two
groups, and the absence of categorical predictors. Because
sample sizes produced at least 20 degrees of freedom, sam-
ple sizes were equal, and two-tailed tests were used, the DFA
was determined robust with respect to multivariate normal-
ity. Additionally, sample sizes were determined adequate, as
they exceeded the number of predictor variables (Tabachnick
and Fidell 2013). Statistical analyses were performed using
both an alpha level of 0.05 and a Bonferonni-corrected alpha
level of 0.0025 to correct for multiple comparisons.
Results
It was hypothesized that both performance-based measures
and questionnaire data would accurately identify differ-
ences in EF among ADHD-PI, ADHD-CT, and control
Performance‑based measures and behavioral ratings of executive function in diagnosing…

subjects; however, the performance-based EF measures
were expected to be less predictive of these differences.
An ANOVA was first used to assess this sensitivity dif-
ference; results are displayed in Table 2. Out of all the
performance-based task variables, the CPT variability
score was the only outcome where a significant difference
was found (p < 0.05). A post hoc test revealed ADHD-CT
subjects performed significantly worse on this measure.
On the other hand, significant differences (p < 0.05) were
found on six of the eight BRIEF scales (Inhibit, Initiate,
Working Memory, Plan and Organize, Organization of
Materials, Monitor) and all indices (BRI, MI, GEC). Post
hoc tests revealed that HCs had significantly lower rat-
ings compared to both ADHD groups on all significantly
different BRIEF variables except the Inhibit scale, which
was found to be significantly higher in the ADHD-CT
group compared to the ADHD-PI and HC groups. When
using a Bonferonni-corrected alpha level (p < 0.0025), no
significant group differences were found on performance-
based task variables, and Organization of Materials and
BRI were no longer significant on the BRIEF. Results sug-
gested that overall, the performance-based measures were
less able to differentiate differences in EF between HCs
and those diagnosed with ADHD compared to the BRIEF
in this sample.
To further assess differences in predictive value between
the BRIEF and performance-based EF measures in diagnos-
ing ADHD, a discriminant function analysis (DFA) was used
(results presented in Table 3). All variables were entered
using a step-wise procedure. None of the performance-based
EF variables predicted group membership above chance
levels. Working Memory, Inhibit, Organization of Materi-
als, and the MI and GEC scales on the BRIEF all predicted
group membership above chance levels. Cross-validated
classification results revealed that 76.2% of the original
group was classified correctly as ADHD-PI, ADHD-CT, or
non-ADHD. Results suggest that the BRIEF has significant
predictive value in eventual ADHD diagnosis, while pre-
dictive value of performance-based EF measures requires
additional investigation.

Table 2  ANOVA results examining the effects of ADHD on EF measures

ADHD-PI ADHD-CT HC F value p value Effect size

WISC-IV
Digit Span Backward SS 95.23 ± 19.07 98.57 ± 10.85 89.75 ± 14.99 1.719 0.188 0.055
Letter–Number Sequencing SS 96.19 ± 16.03 101.19 ± 13.59 99.25 ± 12.80 0.658 0.522 0.022
Symbol Search SS 98.33 ± 9.26 99.76 ± 12.09 99.75 ± 14.82 0.094 0.910 0.003
Coding SS 88.10 ± 16.11 96.19 ± 11.39 91.50 ± 11.59 1.670 0.197 0.054
Working Memory Index SS 95.38 ± 17.46 100.19 ± 12.67 96.45 ± 12.91 0.631 0.536 0.021
Processing Speed Index SS 92.81 ± 13.12 97.66 ± 11.05 91.10 ± 23.95 0.843 0.436 0.028
CPT
Omission T score 55.32 ± 17.67 64.78 ± 25.80 54.81 ± 26.71 1.167 0.318 0.038
Commission T score 52.12 ± 10.76 49.78 ± 13.77 49.54 ± 12.26 0.278 0.759 0.009
Variability T score 52.46 ± 9.80 59.58 ± 12.54 49.81 ± 12.26 4.657 0.013* 0.136
BRIEF
Inhibit T score 55.76 ± 11.01 69.85 ± 11.24 50.10 ± 9.40 19.002 < 0.001** 0.392
Shift T score 59.00 ± 12.45 54.81 ± 10.81 50.95 ± 12.53 2.327 0.106 0.073
Emotional Control T score 55.57 ± 10.87 57.09 ± 11.54 52.00 ± 12.15 1.050 0.356 0.034
Initiate T score 63.52 ± 10.83 59.52 ± 11.61 50.50 ± 8.53 8.328 < 0.001** 0.220
Working Memory T score 71.23 ± 10.18 70.61 ± 7.72 52.60 ± 9.47 27.009 < 0.001** 0.478
Plan and Organize T score 65.85 ± 8.83 64.04 ± 10.51 49.75 ± 9.04 17.516 <0.001** 0.373
Organize Materials T score 62.04 ± 8.51 58.33 ± 9.28 53.70 ± 10.97 3.865 0.026* 0.116
Monitor T score 62.71 ± 10.92 65.66 ± 8.07 47.65 ± 10.88 18.858 < 0.001** 0.390
BRI T score 57.57 ± 11.02 62.81 ± 10.88 51.30 ± 10.72 5.736 0.005* 0.163
MI T score 68.09 ± 8.88 66.85 ± 8.41 50.70 ± 8.96 24.977 < 0.001** 0.458
GEC T score 65.23 ± 8.80 66.04 ± 7.89 51.15 ± 9.34 18.888 < 0.001** 0.390

Score data are mean ± standard deviation

*Significant at p < 0.05; **significant at Bonferroni-corrected p < 0.0025
ANOVA analysis of variance, ADHD-PI attention-deficit/hyperactivity disorder—primary inattentive, ADHD-CT attention-deficit/hyperactivity
disorder—combined type, HC healthy control, SS standard score, WISC-IV Wechsler Intelligence Scale for Children—Fourth Edition, CPT Con-
ners’ Continuous Performance Test, BRIEF Behavior Rating Inventory of Executive Functioning, BRI Behavioral Regulation Index, MI Meta-
cognition Index, and GEC Global Executive Composite

13

Table 3  DFA classification: ADHD-PI versus ADHD-CT versus HC
Actual groups Number of Predicted group membership
cases
ADHD-PI ADHD-CT HC
ADHD-PI 21 14 5 2
66.7% 23.8% 9.5%
ADHD-CT 21 2 18 1 (Doyle et al. 2005). The utility of the measures used in diag-
9.5% 85.7% 4.8%
HC 21 3 2 16
14.3% 9.5% 76.2%
Percentage of originally grouped cases correctly classified: 76.2%
DFA discriminant function analysis, ADHD-PI attention-deficit/
hyperactivity disorder—primary inattentive, ADHD-CT attention-def-
icit/hyperactivity disorder—combined type and HC healthy control
Discussion
These results are consistent with previous research that
has demonstrated differences between performance-based
measures and behavioral rating questionnaire-based data
of EF deficits in ADHD individuals. Significantly higher
EF deficits were found in ADHD participants regardless
of subtype on most BRIEF scales compared to HC, with
most scales belonging to the Metacognition Index. The
exception was the Inhibit scale, a measure of inhibition
of impulses and regulation of behavior, which was sig-
nificantly higher in the ADHD-CT group compared to
ADHD-PI and HC groups. These findings are consistent
with the literature, which has demonstrated that symptoms
of hyperactivity/impulsivity are significantly associated
with the Inhibit subscale, while symptoms of inattention
are significantly associated with Metacognition Index
subscales (Gioia et al. 2002; McCandless and O’Laughlin
2007). However, contrary to expectation, differences in
EF deficits noted on behavioral ratings were not found on
most performance-based tasks in this sample. EF difficul-
ties described on the BRIEF such as inhibition, monitor-
ing, initiation, and working memory might be expected on
the CPT and WISC-IV PSI and WMI. One notable excep-
tion was CPT variability, which was significantly elevated
in the ADHD-CT group. This outcome is a measure of
response speed consistency reflecting variable attention
and processing efficiency throughout the administration,
and this finding may be reflective of higher sensitivity of
this variable to difficulties with impulsivity in ADHD-CT.
Additionally, when all variables were entered into a DFA,
BRIEF scales (Working Memory, Inhibit, Organization of
Materials) and indexes (MI and GEC) were significantly
better at predicting eventual ADHD diagnosis (even by
subtype) compared to all performance-based EF measures
in this sample. Thus, present results provide support for
more predictive value of behavioral rating data compared
13
A. Tan et al.
to performance-based EF test measures when diagnosing
ADHD.
The results of this study highlight past claims that reli-
ability, sensitivity, and the ecological validity of EF tasks
cannot be taken for granted and greater attention to psy-
chometric properties of task-based EF measures is needed
nosing ADHD is constrained by its test–retest reliability.
Although published clinical measures generally provide evi-
dence of reasonable test–retest reliability in their manuals,
few objective studies have formally assessed the reliability
of EF measures. Construct validity must also be considered;
deficits in other domains may impact EF test scores even
though the impairments are not truly caused by EF deficits.
For some individuals, impairments in EF could be caused
by lower intelligence, visuospatial deficits, or symptoms
of other comorbid disorders. An EF measure’s ability to
capture subtle impairments must also be considered. For
example, the systematized testing environment that is used
in the administration of clinical EF measures may reduce the
severity of deficits. Finally, EF measures may be designed to
measure simple components of EF compared to behavioral
ratings that account for observation of the complexity of
EF, which may lead to lower sensitivity of EF measures in
capturing real-word EF deficits. Neuropsychologists must
confirm that tests are measuring what they are intended to
measure in a reliable, valid, and sensitive way. Increasing
the clinical utility, construct validity, and sensitivity of EF
measures will contribute to detection of real-world deficits in
clinical settings, in addition to validation of parent concerns.
Another consideration is the role of EF deficits in ADHD.
Research has shown that a substantial percentage of youth
with ADHD perform within the normal range on measures
of EF upon formal testing (Doyle et al. 2005). The literature
remains inconsistent regarding this topic, and clarification is
an important step toward understanding why performance-
based EF tests may not always capture the underlying reason
for behavioral deficits in ADHD children. Normal perfor-
mance may be explained by differences in severity of impair-
ment, the use of compensatory neurocognitive mechanisms,
or the possibility that additional mechanisms underlie the
neurocognitive impairments in ADHD cases. For example,
recent studies have suggested that EF deficits may be only
one neuropsychological subtype of ADHD among others
such as delay aversion and state regulation impairments,
supporting that multiple pathways of ADHD exist (Lambek
et al. 2010). Others have proposed that ADHD and EF defi-
cits may co-occur so frequently due to referral bias. Perhaps
the majority of subjects in neuropsychological studies of
ADHD are represented by individuals exhibiting ADHD
symptoms and EF impairments simultaneously because
these children display more behavior problems and are there-
fore more likely to be referred for treatment than children
Performance‑based measures and behavioral ratings of executive function in diagnosing…
without deficits in EF (Pennington and Ozonoff 1996). Thus,
future studies should further investigate underlying deficits
in ADHD.
Several limitations to this study must be noted. In using a
sample from clinic archival data, results of the present study
may not generalize to a larger population. The majority of
the sample was Caucasian, and differences were not found
between ADHD and non-ADHD participants in grade level
or GAI. Results of the present study may not generalize to
lower-functioning or academically underachieving ADHD
patients, which is more common in populations with lower
socioeconomic status. Additionally, some members of the
control group were necessarily obtained from the same
source of clinic archival data. Difficulties can arise in dis-
cerning group differences within a clinic-referred sample, as
some children referred for suspicions of neuropsychologi-
cal impairment might also display problems related to EF.
Attempts were made to lower the effect of this limitation
by including recruits from the general community for the
control group and requiring a telephone screening to rule
out suspicions of ADHD and other exclusionary data prior
to inclusion. Overall sample size was an additional limita-
tion, and larger samples may have led to increased power in
detecting differences between groups. Future studies should
investigate these questions in a larger and more heterogene-
ous sample.
Additionally, the comorbidity of ADHD with learning
disorders in our sample is a potential limitation, as emer-
gent research has found differences in cognitive function in
children with ADHD and learning difficulties (Huang et al.
2016). While exclusionary criteria related to comorbid dis-
orders were quite stringent, comorbid learning disorders lim-
ited to dyslexia, dysgraphia, dyscalculia, and learning disor-
ders—not otherwise specified (LD-NOS) were accepted to
obtain sufficient sample size. Furthermore, research suggests
that gender may be associated with differences in ADHD
symptom presentation (Quinn and Madhoo 2014). While
both genders were accepted to obtain sufficient sample size,
participant groups were matched by gender to address this
limitation. Future studies should analyze the effects that
comorbid learning disorders, and gender might have on EF
deficits and further elucidate the role of EF in complex pres-
entations of ADHD.
Finally, because the study used archival data as the
primary source for recruitment, the specific performance-
based measures used to determine EF deficits were chosen
due to consistency of use across evaluations. Therefore,
the researchers were limited to EF measures that required
less processing demands such as information processing,
impulsivity, and working memory. Alternatively, behavio-
ral ratings were able to assess additional higher-order EF
skills including planning/problem-solving, set shifting, and
self-regulation. Thus, this difference may have contributed
to findings regarding their respective value in diagnosing
ADHD. Future research on the sensitivity of EF measures
may want to consider using performance-based EF tasks
that assess higher-order EF skills. Additionally, compari-
sons were not made between particular BRIEF scales and
neuropsychological measures designed to reflect specifically
corresponding neuropsychological processes. Future studies
investigating the relationship between performance-based
EF tasks and behavioral rating EF data should consider
selecting clinical EF measures that correspond to specific
EF rating subscales.
The results of the current study suggest that the BRIEF
scales are capturing EF deficits and predicting ADHD status
more accurately compared to performance-based measures.
These findings were clear in a well-selected sample of HCs,
ADHD-PI, and ADHD-CT children matched for gender,
age, and IQ. While these results point to differences between
performance-based measures and behavioral ratings in cap-
turing EF deficits in ADHD individuals, the underlying rea-
son for these differences must continue to be investigated
(Doyle et al. 2005). Continued development of ecologically
valid measures of EF will be an important supplement to
neuropsychological assessment in this population, as rat-
ing scales cannot be used as a proxy for performance-based
neuropsychological measures (Salimpoor et al. 2000).
Compliance with ethical standards 
Conflict of interest  The authors declare that they have no conflict of
interest.
References
American Psychiatric Association (1994) Diagnostic and statistical
manual of mental disorders, 4th edn. Author, Washington
Anderson P (2002) Assessment and development of executive function
(EF) during childhood. Child Neuropsychol 8(2):71–82
Barkley RA (1997) Behavioral inhibition, sustained attention, and
executive functions: constructing a unifying theory of ADHD.
Psychol Bull 121(1):65–94
Barkley RA, Murphy KR (2010) Impairment in occupational func-
tioning and adult ADHD: the predictive utility of executive
function (EF) ratings versus EF tests. Arch Clin Neuropsychol
25(3):157–173
Biederman J, Petty C, Fried R, Fontanella J, Doyle AE, Seidman LJ,
Faraone SV (2006) Impact of psychometrically defined deficits of
executive functioning in adults with attention deficit hyperactivity
disorder. Am J Psychiatry 163(10):1730–1738
Biederman J, Petty CR, Fried R, Black S, Faneuil A, Doyle AE et al
(2008) Discordance between psychometric testing and question-
naire-based definitions of executive function deficits in individuals
with ADHD. J Atten Disord 12(1):92–102
Burgess PW (1997) Theory and methodology in executive function and
research. In: Rabbitt P (ed) Methodology of frontal and executive
function. Psychology, Hove, pp 81–116
Bussing R, Fernandez M, Harwood M, Wei H, Garvan CW, Eyberg
SM, Swanson JM (2008) Parent and teacher SNAP-IV ratings of
13

attention deficit hyperactivity disorder symptoms: psychometric
properties and normative ratings from a school district sample.
Assessment 15(3):317–328
Cohen J (1988) Statistical power analysis for the behavioral sciences,
2nd edn. Lawrence Earlbaum Associates, Hillsdale
Conners CK (2004) Conners’ CPT II continuous performance test II.
Multi Health Systems, North Tonawanda
Di Trani M, Casini M, Capuzzo F, Gentile S, Bianco G, Menghini D,
Vicari S (2011) Executive and intellectual functions in attention-
deficit/hyperactivity disorder with and without comorbidity. Brain
Dev 33(6):462–469
Doyle AE, Faraone SV, Seidman LJ, Willcutt EG, Nigg JT, Waldman
ID et al (2005) Are endophenotypes based on measures of execu-
tive functions useful for molecular genetic studies of ADHD? J
Child Psychol Psychiatry 46(7):778–803
Gau S, Shang C (2010) Executive functions as endophenotypes in
ADHD: evidence from the Cambridge Neuropsychological Test
Battery (CANTAB). J Child Psychol Psychiatry 51(7):838–849
Gau S, Chiu C, Shang C, Cheng A, Soong W (2009) Executive function
in adolescence among children with attention-deficit/hyperactivity
disorder in Taiwan. J Dev Behav Pediatr 30(6):525–534
Gioia GA, Isquith PK (2004) Ecological assessment of executive func-
tion in traumatic brain injury. Dev Neuropsychol 25(1–2):135–158
Gioia GA, Isquith PK, Guy SC, Kenworthy L (2000) Behavior rat-
ing inventory of executive function. Psychological Assessment
Resources, Odessa
Gioia GA, Isquith PK, Kenworthy L, Barton RM (2002) Profiles of
everyday executive function in acquired and developmental dis-
orders. Child Neuropsychol 8:121–137
Huang F, Sun L, Qian Y, Liu L, Ma QG, Yang L et al (2016) Cogni-
tive function of children and adolescents with attention deficit
hyperactivity disorder and learning difficulties: a developmental
perspective. Chin Med J 129(16):1922–1928
Johnson DE, Epstein JN, Waid L, Latham PK, Voronin KE, Anton RF
(2001) Neuropsychological performance deficits in adults with
attention-deficit/hyperactivity disorder. Arch Clin Neuropsychol
16(6):587–604
Jonsdottir S, Bouma A, Sergeant JA, Scherder EA (2006) Relationships
between neuropsychological measures of executive function and
behavioral measures of ADHD symptoms and comorbid behavior.
Arch Clin Neuropsychol 21(5):383–394
Lambek R, Tannock R, Dalsgaard S, Trillingsgaard A, Damm D,
Thomsen P (2010) Validating neuropsychological subtypes of
ADHD: how do children with and without an executive function
deficit differ? J Child Psychol Psychiatry 51(8):895–904
Lanfranchi S (2013) Is the WISC-IV General Ability Index a useful
tool for identifying intellectual disability? Dev Med Child Neurol
55(9):782–783
McCandless S, O’Laughlin L (2007) Utility of the behavior rating
inventory of executive function (BRIEF) in the diagnosis of
ADHD. J Atten Disord 10(4):381–389
13
A. Tan et al.
McCrimmon AW, Smith AD (2013) Wechsler Abbreviated Scale of
Intelligence, second edition (WASI-II). J Psychoeduc Assess
31(3):337–341
Oosterlaan J, Logan GD, Sergeant JA (1998) Response inhibition in
AD/HD, CD, comorbid AD/HD+ CD, anxious, and control chil-
dren: a meta-analysis of studies with the stop task. J Child Psychol
Psychiatry 39(3):411–425
Pennington BF, Ozonoff S (1996) Executive functions and develop-
mental psychopathology. Child Psychol Psychiatry Allied Discip
37(1):51–87
Quinn PO, Madhoo M (2014) A review of attention-deficit/hyperactiv-
ity disorder in women and girls: uncovering this hidden diagnosis.
Prim Care Companion CNS Disord 16(3). https:​ //doi.org/10.4088/
PCC.13r01​596
Riccio CA, Homack S, Jarratt KP, Wolfe ME (2006) Differences in
academic and executive function domains among children with
ADHD predominantly inattentive and combined types. Arch Clin
Neuropsychol 21:657–667
Rommelse NJ, Oosterlaan J, Buitelaar J, Faraone SV, Sergeant JA
(2007) Time reproduction in children with ADHD and their
nonaffected siblings. J Am Acad Child Adolesc Psychiatry
46(5):582–590
Salimpoor R, Mota VL, Logan GD, Tannock R, Klim P (2000) Con-
firmation of an inhibition deficit in attention-deficit/hyperactivity
disorder. J Abnorm Child Psychol 28:227–235
Tabachnick BG, Fidell LS (2013) Profile analysis: the multivariate
approach to repeated measures. In: Using multivariate statistics,
6th edn. Pearson, Boston
Thaler NS, Bello DT, Etcoff LM (2013) WISC-IV profiles are associ-
ated with differences in symptomatology and outcome in children
with ADHD. J Atten Disord 17(4):291–301
Toplak ME, Bucciarelli SM, Jain U, Tannock R (2009) Executive
functions: performance-based measures and the Behavior Rating
Inventory of Executive Function (BRIEF) in Adolescents with
Attention Deficit/Hyperactivity Disorder (ADHD). Child Neu-
ropsychol 15:53–72
Torralva T, Gleichgerrcht E, Lischinsky A, Roca M, Manes F (2013)
‘Ecological’ and highly demanding executive tasks detect real-life
deficits in high-functioning adult ADHD patients. J Atten Disord
17(1):11–19
Wechsler D (2003) Wechsler Intelligence Scale for Children—Fourth
Edition. Psychological Corporation, San Antonio
Wechsler D (2011) Wechsler Abbreviated Scale of Intelligence—Sec-
ond Edition. NCS Pearson, San Antonio
Willcutt EG, Doyle AE, Nigg JT, Faraone SV, Pennington BF (2005)
Validity of the executive function theory of attention-deficit/
hyperactivity disorder: a meta-analytic review. Biol Psychiatry
57(11):1336–1346