Copyright © 1998 Elsevier Science Ltd. All rights reserved.

2.11
Defining Constructs Underlying
the Practice of Psychology in the
United States and Canada
I. LEON SMITH and SANDRA GREENBERG
Professional Examination Service, New York, NY, USA

2.11.1 INTRODUCTION 122

2.11.2 METHOD 123
2.11.2.1 Committee Structure 123
2.11.2.2 Structure and Development of the Draft Delineations of the Practice of Psychology 123
2.11.2.3 Refining the Delineations of the Practice of Psychology 124
2.11.2.4 Development of a Survey of Practice 124
2.11.2.5 Sampling Plan 125
2.11.2.6 Conduct of the Survey 125
2.11.3 SUMMARY OF RESULTS 126
2.11.3.1 Survey Response Rate 126
2.11.3.2 Professional Background and Demographic Descriptions 126
2.11.3.3 Results Related to the Delineations 126
2.11.3.3.1 Process-based delineation 127
2.11.3.3.2 Content-based delineation 127
2.11.3.4 Qualitative Results 128
2.11.3.5 Process- and Content-based Profiles of Practice 129
2.11.3.6 Development of Test Specifications 129
2.11.4 CONCLUSIONS 129
2.11.5 APPENDIX 130
2.11.5.1 Roles and Definitions 130
2.11.5.1.1 Responsibilities related to direct service 130
2.11.5.1.2 Responsibilities related to outreach and consultation 130
2.11.5.1.3 Responsibilities related to academic preparation and professional development 131
2.11.5.1.4 Responsibilities related to research and evaluation 131
2.11.5.2 Content Areas and Definitions 131
2.11.5.2.1 Knowledge statements related to biological bases of behavior 132
2.11.5.2.2 Knowledge statements related to cognitive-affective bases of behavior 132
2.11.5.2.3 Knowledge statements related to social and multicultural bases of behavior 132
2.11.5.2.4 Knowledge statements related to growth and lifespan development 133
2.11.5.2.5 Knowledge statements related to assessment and diagnosis 133
2.11.5.2.6 Knowledge statements related to treatment/intervention 134
2.11.5.2.7 Knowledge statements related to research and evaluation 134
2.11.5.2.8 Knowledge statements related to ethical/legal/professional issues 134
2.11.6 REFERENCES 134

121
122 Defining Constructs Underlying the Practice of Psychology in the US and Canada
2.11.1 INTRODUCTION
The process of establishing content validity,
as described in Chapter 11 of the 1985 revision
of the Standards for Educational and Psycholo-
gical Testing (the Standards) published by the
American Educational Research Association
(AERA), the American Psychological Associa-
tion (APA), and the National Council on
Measurement in Education (NCME), empha-
sizes the need to conduct a practice analysis to
ensure that the knowledge, skills, or abilities
assessed in credentialing initiatives are limited
to those required for competent performance
and that they serve a public protection function.
According to contemporary theorists like Mes-
sick (1989,1995), content validity is one aspect
of a comprehensive theory of construct validity.
From Messick's perspective, the meaning and
interpretation of test scores is established on the
basis of many different types of empirical
investigations, of which a practice analysis is
but one example. This view is consistent with the
thinking of Schoon and Smith (1996), who
suggest that a practice analysis is the second of
six stages in the development, use, and evalua-
tion of licensure examination programs, and
immediately follows the conduct of a strategic
credentialing analysis which is addressed by
Schoon in Chapter 10, this volume.
Although there is no standard way to conduct
a practice analysis, common strategies used to
define the critical performance areas and the
underlying knowledge, skills, or abilities include
the use of a committee of subject-matter experts,
the compilation of worker logs, observations of
practice, and interviews with practitioners
(Knapp & Knapp, 1995). In general, it is best
to collect practice data from more than one
source, as a way of confirming or supplementing
the data obtained from other sources. The
resulting definition of practice can then be
validated by collecting ratings from a repre-
sentative sample of practitioners. The validation
process generates data on the criticality,
importance, and frequency of the elements in
the practice definition which can be used to
evaluate the completeness and appropriateness
of the performance areas and the associated
knowledge, skills, or abilities. Analysis of the
validation data also leads to the development of
test specifications supporting examination de-
velopment activities and to the preparation of
instructions for developing the test questions.
The outcomes of a practice analysis, then,
become the primary basis by which a sponsor of
a national licensure program such as the
Association of State and Provincial Psychology
Boards (ASPPB), formerly the American Asso-
ciation of State Psychology Boards, establishes
and defends the content validity of its creden-
tialing initiative.
ASPPB maintains a research program on the
Examination for Professional Practice in Psy-
chology (EPPP) consistent with the procedures
described above and with the content validity
emphasis articulated in the Standards. For more
information on ASPPB's research program, See
Rosen (1991). For a brief history of the origins
of the psychology licensing program, the reader
is referred to Chapter 10, this volume. Finally,
information on EPPP test performance, candi-
date background information, pass/fail statis-
tics, and the impact of test preparation services
on candidate performance is provided by Rosen
in Chapter 12, this volume.
ASPPB's research includes three steps: (i) the
conduct of a study of professional practice, (ii)
the development of revised test specifications
based on the study of practice, and (iii) the
independent verification that the items and/or
forms of the EPPP reflect the test specifications
in a representative and fair manner and are
consistent with the generic philosophy under-
lying the EPPP.
Since the mid-1970s, ASPPB has conducted
two sets of investigations, each associated with
these three steps. ASPPB contracted with
Professional Examination Service (PES) to
initiate the first set of studies (Richman,
1982). PES conducted an investigation aimed
at clarifying the content most appropriate for
the EPPP. On the basis of the study, ASPPB
developed a new EPPP content outline. All
items in the ASPPB item bank were recorded to
incorporate the clarified information on the role
and knowledge statements. An additional
content validation study was conducted under
contract to PES (Smith, 1984) to check on the
clarity of the new content outline, the quality of
the items, and the correspondence between the
content categories in the test outline and the
items.
ASPPB contracted with the Educational
Testing Service to initiate the second set of
studies (Rosenfield, Shimberg, & Thornton,
1983). Using existing resources and expert
judgments, Rosenfeld et al. created an inventory
consisting of 59 responsibilities, 61 procedures,
techniques, and resources, and 49 knowledge
areas, including ethical/legal considerations and
defining the work of professional psychologists.
Licensed psychologists in the USA and Canada
rated the inventory in terms of time spent,
importance, level of application, and level of
judgment exercised.
Subsequently, revised test specifications were
developed under contract to PES (Rosen &
Mirone, 1986). The procedures called for an
integration of the results of the job analysis
 Method
conducted by Rosenfeld et al. (1983) with the
findings of the earlier study completed by
Richman (1982). All items in the ASPPB item
bank were reclassified on the basis of the
categories in the new test specifications. ASPPB
contracted with PES (Hambleton & Smith,
1988) to conduct a content validity study to
provide an independent check that the first two
forms of the EPPP based on the new test
specifications reflected these specifications, and
that the items were free of any unintended
content bias with regard to ethnicity, gender,
and age. Finally, ASPPB funded PES (Smith &
Greenberg, 1991) to address selected recom-
mendations made as part of an audit of the
EPPP examination program conducted by the
State of California (Werner, 1989). The knowl-
edge, skill, and application statements in the test
specifications were augmented with specific
topics or elements of practice, the interpretation
of the responsibilities was clarified, and guide-
lines were prepared to facilitate development
and classification of items.
The practice analysis study reported herein
was conducted by PES under contract with
ASPPB and initiated the third comprehensive
set of investigations to examine what licensed
psychologists in the USA and Canada do and
know in order to ensure the continued content
validity of the EPPP (Greenberg, Smith, &
Muenzen, 1996).
2.11.2 METHOD
2.11.2.1 Committee Structure
The practice analysis study was conducted in
collaboration with a five-member Practice
Analysis Advisory Committee (PAAC) and a
12-member task force of subject-matter experts
appointed by the PAAC on the basis of
nominations received from ASPPB's Board of
Directors, the Examination Development Com-
mittee, and the PAAC. The members of the
PAAC represented key decision-makers in
ASPPB; in part, they were responsible for
guiding aspects of the licensure program,
including examination development and valida-
tion efforts. The members of the task force
represented psychologists in different major
areas of practice, including new and emerging
specialties.
2.11.2.2 Structure and Development of the Draft
Delineations of the Practice of
Psychology
Process- and content-based approaches were
implemented to study the practice of licensed
psychologists in the USA and Canada. Process-
123
and content-based approaches to practice
analysis are appropriate for professions such
as psychology in which the primary professional
behaviors are cognitive in nature (Schoon,
1985).
A process-based approach was used because
it provides a structure for describing contem-
porary practiceÐthat is, for identifying what
psychologists doÐand because it facilitates the
development of examination items in a practice-
related framework. The process-based ap-
proach comprises the delineation of roles and
associated responsibilities performed by psy-
chologists.
Roles represent constructs described by
major categories of activities. For example,
the role of Direct Service was identified and
defined as the provision and/or administration
of psychological services to clients, patients,
and/or organizations in the areas of problem
definition, need assessment, and diagnosis; and
the design, implementation, and evaluation of
interventions.
Responsibilities represent the specific activ-
ities that psychologists perform within each
role. For example, to observe, interview, and
gather information from the patient/client/
organization and related sources (e.g., relevant
others, written records, referral source) in order
to identify the problems/needs and their con-
texts, is one responsibility performed in con-
nection with the Direct Service role.
A content-based approach was used because
it provides a structure for identifying a
comprehensive listing of the critical knowledge
that psychologists need in order to perform the
responsibilities identified as part of professional
practice. The content-based approach provides
a user-friendly template for giving feedback to
candidates and communicating summary re-
sults of EPPP performance to universities,
professional schools, and training programs.
The content-based approach comprises the
delineation of content areas and associated
knowledge statements.
Content areas represent constructs described
by categories of knowledge used by psycholo-
gists in the performance of responsibilities. For
example, Biological Bases of Behavior a content
area encompassing knowledge of neuroscience,
the physiological bases of behavior and illness,
and psychopharmacology.
Knowledge statements represent organized
bodies of information associated with specific
content areas and are necessary in the per-
formance of responsibilities. For example,
knowledge of basic neuroscience (e.g., neuroa-
natomy, neurophysiology, neurochemistry) and
clinical neuroscience (e.g., brain±behavior re-
lationships, neurological syndromes and their
124 Defining Constructs Underlying the Practice of Psychology in the US and Canada
contribution to cognitive and emotional status
and behavior), is a knowledge statement that
describes information subsumed in the Biolo-
gical Bases of Behavior content area.
2.11.2.3 Refining the Delineations of the
Practice of Psychology
The following data methods were implemen-
ted to supplement and evaluate the structure
and content of the draft delineations initially
developed by the task force: (i) a sample of 25
licensed psychologists completed a practice log
designed to elicit a list of professional respon-
sibilities performed by psychologists; (ii) a
sample of 40 licensed psychologists participated
in a critical incidents interview in which
participants were asked to reflect upon incidents
in which they felt particularly effective and
particularly ineffective in their practice and to
identify the knowledge which made a difference
to the outcome; (iii) job descriptions and
supervisor's rating forms were collected from
members of the PAAC, the task force, and the
participants in both the practice log recording
and the critical incidents interviews; and (iv) a
sample of 29 licensed psychologists completed
an independent review of the draft delineations.
The task force and the PAAC used the
information in preparing the final delineations.
Table 1 presents the roles and the number of
associated responsibilities in the process-based
delineation, and the content areas and the
number of associated knowledge statements in
the content-based delineation. A complete copy
of the delineations is given in the appendix to
this chapter (see Section 2.11.5).
2.11.2.4 Development of a Survey of Practice
In order to validate the process- and content-
based delineations, a survey of the practice of
psychology was developed for dissemination to
licensed psychologists in the USA and Canada.
Rating scales were designed to collect data on
the roles and responsibilities performed by
licensed psychologists and the content areas
and knowledge required for practice. A two-
part pilot test of the survey was conducted to
identify question defects such as ambiguous
items and cognitive difficulties in forming
answers to questions. First, a sample of 38
licensed psychologists completed the survey via
a mail review and evaluated each section to
identify any unclear elements. Then, a second
sample of four licensed psychologists completed
a mail review of the survey and were debriefed in
an in-depth telephone interview regarding the
cognitive distinctions they made among the
rating scales and the scale points associated with
the rating scales (Bolton, 1993).
The results of the two-part pilot test were used
to refine the process- and content-based
delineations, the instructions, and the rating
scales. The final survey document, entitled the
ªSurvey of the Practice of Psychologyº (the
Survey), included six sections. Section One used
a set of three rating scales to elicit information
about the roles performed in the practice of
psychology. Survey respondents were asked to
rate the roles in regard to three questions:
(i) How important was the role to your
practice as a psychologist during the past year?
(ii) What percentage of your work time was
devoted to each role during the past year?
(iii) How critical is performing the role
protecting the patient/client/public from harm?
Section Two used a similar set of rating scales
to elicit information about the responsibilities in
the practice of psychology associated with each
role. Survey respondents were asked to rate the
responsibilities in regard to three questions:
(i) How important was the performance of
the responsibility to your practice as a psychol-
ogist during the past year?
(ii) How frequently have you performed this
responsibility during the past year?
(iii) How critical is performing the responsi-
bility to protecting the patient/client/public from
harm?
Section Three elicited information about the
content areasÐor categories of knowledgeÐ
that psychologists use in practice. Survey re-
spondents were asked to rate the content areas
in regard to three questions:
(i) How important was the content area to
your practice as a psychologist during the past
year?
(ii) How frequently have you called upon
knowledge from the content area in your
practice during the past year?
(iii) How critical is the content area to
protecting the patient/client/public from harm?
Section Four elicited information about the
knowledge that is associated with the content
areas and is needed to perform the responsi-
bilities. Survey respondents were asked to rate
the knowledge statements in regard to three
questions:
(i) What level best represents your use of this
knowledge in your practice?
(ii) At what point should the knowledge be
acquired by psychologists?
(iii) How critical is possessing the knowledge
to protecting the patient/client/public from
harm?
Section Five elicited information about the
demographic and professional background of
respondents with respect to their work in
 Method
Table 1 Process- and content-based delineations of the practice of psychology.
Role
Direct Service
Outreach and Consultation
Academic Preparation and Professional Development
Research and Evaluation
Content area
Biological Bases of Behavior
Cognitive-Affective Bases of Behavior
Social and Multicultural Bases of Behavior
Growth and Lifespan Development
Assessment and Diagnosis
Treatment/Intervention
Research and Evaluation
Ethical/Legal/Professional Issues
psychology, such as primary and secondary
employment setting, practice area, theoretical
orientation, specialization or respecialization,
and specialty certification; and with respect to
demographic variables, such as education, sex,
disability, and race or ethnicity.
Section Six solicited qualitative comments
wherein respondents were requested to describe
any professional responsibilities they perform
or knowledge they call upon that was not
included in the survey. Most importantly,
respondents were asked to comment upon what
they believe to be the long-range changes
occurring in the practice of psychology.
2.11.2.5 Sampling Plan
The sampling plan for the dissemination of
the Survey was designed to (i) generate a sample
of 7500 licensed psychologists from those
jurisdictions in the USA and Canada that
employed the EPPP as an element in the
credentialing process, and (ii) ensure the
representation of licensed psychologists at or
near the entry level of the profession. A
minimum of 60 licensed psychologists were
sampled from each jurisdiction using the EPPP,
regardless of the size of the jurisdiction, and
recently licensed psychologists, operationally
defined as having been licensed in 1989 or later,
were oversampled in every jurisdiction.
The sampling plan included three different
data sources: (i) ASPPB-member boards, (ii) the
APA Membership Database, and (iii) the
Canadian Register of Health Service Providers
in Psychology (Canadian Register). In general,
the majority of the sample of recent licensees
125
Number of responsibilities
11
10
8
10
Number of knowledge
statements
6
6
11
8
14
11
5
5
was drawn directly from the ASPPB-member
boards using a systematic random sampling
technique, and the majority of the sample of
less-recent licensees was drawn from the
remaining two sources using a systematic
random sampling technique.
2.11.2.6 Conduct of the Survey
The conduct of the Survey was designed to
yield a high rate of return. Accordingly, the plan
involved three carefully timed sequential mail-
ings to the survey recipients as well as one
telephone follow-up to a sample of nonrespon-
dents (Dillman, 1978).
(i) Potential participants received a letter
describing the nature and scope of the project
and inviting their participation in the data
collection efforts. Recipients who declined to
participate or who were no longer active in the
profession were requested to call collect so as to
be eliminated from the sample. Potential survey
recipients in the three Canadian provinces
where the French-language version of the EPPP
was offered received an English/French bilin-
gual version of the letter and a postage-paid
return postcard for requesting a French-lan-
guage version of the survey.
(ii) Approximately two weeks later, all re-
maining potential members of the sample
received an English- or French-language survey
packet, as appropriate. The survey packet
contained: a covering letter reviewing the pur-
pose and importance of the data collection, and
assuring the recipients about the confidentiality
of the responses; a survey and a postage-paid
return envelope; and a postage-paid tracking
126 Defining Constructs Underlying the Practice of Psychology in the US and Canada
postcard to be returned, indicating return or
nonreturn of the completed survey. As a follow-
up, each member of the sample received an
English- or French-language reminder/thank-
you postcard, as appropriate.
(iii) Subsequently, each member of the sam-
ple not returning a tracking postcard received a
second and then a third survey packet. The
packets were identical to the first except that the
covering letter identified the recipient as not
previously returning a completed survey. In
addition, the letter informed recipients of ways
to obtain the results of the study. The letter in
the third packet urged the recipients to respond
so that their own practice might be reflected in
ASPPB's picture of contemporary practice.
Follow-up reminder/thank-you postcards were
sent to all members of the sample who received
the second and third survey packets.
(iv) Approximately four weeks after the final
mailing, a telephone/mail survey of 250 non-
responders was conducted to estimate the
potential impact of nonresponse bias.
2.11.3 SUMMARY OF RESULTS
2.11.3.1 Survey Response Rate
The return rate for completed Surveys from
eligible participants was 60%, which is con-
sidered excellent for surveys of this type (Knapp
& Knapp, 1995). Consistent with the sampling
plan, every ASPPB-member jurisdiction was
represented in the returns, and recently licensed
psychologists were overrepresented in propor-
tion to their representation in the population.
2.11.3.2 Professional Background and
Demographic Descriptions
(i) More than two-thirds of the US and
Canadian respondents indicated that their pri-
mary employment setting fell within the general
category of human-service settings (including
individual independent practice and group
psychological practice), whereas about 20%
indicated that their primary employment setting
was an educational institution/school system.
(ii) US respondents were most likely to have
earned a doctoral degree, including the docto-
rate of psychology degree, as their highest
degree in psychology. Canadian respondents
were equally likely to have earned either a
master's degree or a doctoral degree as their
highest degree in psychology.
(iii) Recently licensed US respondents were
more likely to have earned a degree from a free-
standing school of professional psychology
than were less recently licensed respondents.
Few Canadian respondents earned degrees
from free-standing schools of professional
psychology.
(iv) More than one half of the US and
Canadian respondents indicated clinical psy-
chology as their major area of training; counsel-
ing was reported as the major area of training by
about 20% of the US respondents and 15% of
the Canadian respondents.
(v) In both the USA and Canada, more
respondents indicated clinical child psychology
and clinical neuropsychology as their current
major area of practice than had trained in those
areas.
(vi) In both the USA and Canada, about
one-half of respondents described their primary
theoretical orientation as cognitive/behavioral
or behavioral. In the USA, about one-quarter of
the respondents described their primary theo-
retical orientation as psychodynamic, and no
more than 8% identified any one other primary
theoretical orientation. In Canada, approxi-
mately equal numbers described their primary
theoretical orientation as psychodynamic and
existential/humanistic (both 18%), and no more
than 6% identified any one other primary
theoretical orientation.
(vii) About one-quarter of the US respon-
dents and approximately 15% of the Canadian
respondents indicated that they had partici-
pated in a postdoctoral specialization or respe-
cialization program.
(viii) More than one-half of US and Cana-
dian recently licensed respondents were female,
compared to about 40% of the less recently
licensed US and Canadian respondents.
(ix) About 93% of the US respondents and
89% of the Canadian respondents indicated
they were Caucasian/White. American-Indian
or Alaskan Native, Asian or Pacific Islander,
Black or African-American, and Hispanic/La-
tino psychologists were represented among the
respondents.
(x) Selected comparisons of professional
background and demographic variables indi-
cated that the current sample was representative
of the populations of licensed psychologists
included in the APA Membership Database
and the Canadian Register.
(xi) The results of the respondent/nonre-
spondent survey revealed few differences in
key background and demographic variables.
2.11.3.3 Results Related to the Delineations
Preliminary statistical analyses of the Im-
portance, Time/Frequency, and Criticality rat-
ings indicated that there were few differences in
the results of the respondent/nonrespondent
 Summary of Results
survey, or in the ratings of recently vs. less
recently licensed respondents and US versus
Canadian respondents. Accordingly, the ratings
of all the respondents on the roles and
responsibilities, and on the content areas and
knowledge statements were combined for final
data analysis in connection with the develop-
ment of profiles of practice and revised test
specifications for the EPPP. Readers are
encouraged to review the appendix to this
chapter (see Section 2.11.5) for a complete
description of the roles and responsibilities, and
the content areas and knowledge statements.
These elements comprise the components of the
test specifications.
2.11.3.3.1 Process-based delineation
(i) Roles
The Direct Service role was rated most
important to the practice of the respondents
and most critical to the protection of the
patient/client/public from harm. Respondents
spent about 70% of their time in that role.
(a) The Direct Service role was rated as
moderately to very important to the practice
of the respondents; in fact, nearly 90% of the
ratings were at scale point three, indicating that
the role was very important to the practice of
the respondents, and the associated standard
deviation was the smallest of any role and in
connection with any rating scale. The remaining
rolesÐOutreach and Consultation, Academic/
Professional Development, Research and
EvaluationÐwere rated as minimally to mod-
erately important to their practice.
(b) The mean Time ratings indicated that
respondents spent two-thirds of their time in
the Direct Service role, and one-third of their
time in all other roles. However, the magnitude
of the standard deviations associated with the
mean Time ratings indicated a great deal of
variability in the practice patterns of the
respondents.
(c) All four delineated roles were rated as
moderately to highly critical to the protection of
the patient/client/public from harm.
(d) In general, the Importance and Time
ratings on the roles varied more than the
Criticality ratings; respondents consistently
rated the roles as moderately to highly critical
to the protection of the patient/client/public,
regardless of their own professional practice
patterns.
(e) The ratings of the roles by respondents
representing various major areas of practice
demonstrated some similarities and differences.
For example, in terms of the Importance rating
scale, the ratings were generally similar in
127
regard to the Direct Service role, but somewhat
dissimilar in regard to the Academic Prepara-
tion and Professional Development role and the
Research and Evaluation role.
(ii) Responsibilities
The Importance, Frequency, and Criticality
ratings of the responsibilities were generally
consistent with the Importance, Time, and
Criticality ratings of the associated roles; thus,
the respondents rated the responsibilities asso-
ciated with the Direct Service role as most
important to their practice, most frequently
performed, and most critical to the protection of
the patient/client/public from harm. In general,
the Importance and Frequency ratings on the
responsibilities varied more than the Criticality
ratings.
(a) Respondents rated virtually all the re-
sponsibilities as moderately to highly critical to
the protection of the patient/client/public from
harm, regardless of their own professional
practice patterns.
(b) One responsibility associated with the
Direct Service roleÐto provide direct service in
a manner consistent with professional and
ethical standards and guidelines, and state/
provincial and national laws and regulationsÐ
was rated highest of any responsibility on the
Importance, Frequency, and Criticality rating
scales. Within the Outreach and Consultation
role, a similar responsibilityÐto provide educa-
tion and consultation in a manner consistent
with current professional and ethical standards
and guidelines, and state/provincial and na-
tional laws and regulationsÐwas rated highest
on the same three rating scales.
(c) The ratings of the responsibilities by
respondents representing various major prac-
tice areas demonstrated similarities and differ-
ences. Respondents in various major areas of
practice displayed a greater degree of consensus
in their Criticality ratings than in their Impor-
tance and Frequency ratings.
2.11.3.3.2 Content-based delineation
(i) Content areas
The Treatment/Intervention and Ethical/
Legal/Professional Issues content areas were
rated most important to the practice of the
respondents and most critical to the protection
of the patient/client/public from harm; the
knowledge base associated with the Treat-
ment/Intervention content area was called upon
most frequently by the respondents.
(a) The ratings of the content areas indicated
that all areasÐexcept Research and
128 Defining Constructs Underlying the Practice of Psychology in the US and Canada
EvaluationÐwere moderately to very important
to the practice of the respondents and were called
upon moderately to very frequently.
(b) All content areasÐincluding Research
and EvaluationÐwere rated as moderately to
highly critical to the protection of the patient/
client/public from harm.
(c) In general, the Criticality ratings of the
respondents were somewhat higher than the
Importance ratings.
(d) The ratings of the content areas by
respondents in different major areas of practice
were generally similar. The respondents in
different major areas of practice displayed a
great degree of similarity in what content areas
they considered critical to the protection of the
patient/client/public from harm, even though
they varied considerably in their ratings of what
content areas are important to and frequently
performed in their own practice.
(ii) Knowledge statements
About 85% of the respondents used 51 of the
66 knowledge statements either at the recogni-
tion/recall or the apply/interpret/integrate level,
and about the same percentage of respondents
rated 61 of the 66 knowledge statements as
necessary to be acquired at some point in
timeÐeither primarily before or after licensure.
Finally, the Criticality ratings indicated that 43
of the 66 knowledge statements were rated
moderately to highly critical to the protection of
the patient/client/public from harm. Other key
findings were as follows:
(a) Of the 15 knowledge statements rated by
a majority of the respondents as not used or not
supported for acquisition, 12 were in two
content areasÐAssessment and Diagnosis,
and Treatment/Intervention.
(b) With few exceptions, the ratings of the
knowledge statements by respondents repre-
senting various major areas of practice were
similar. However, respondents in various major
areas of practice were least similar in their
ratings of knowledge statements associated with
the two content areas, Assessment and Diag-
nosis and Treatment/Intervention.
2.11.3.4 Qualitative Results
About two-thirds of the respondents identi-
fied long-range changes occurring in the
practice of the profession, especially those that
may affect the knowledge required of licensed
psychologists in the future. The following
summary characterizes the unique and the
common areas of change identified by the US
and Canadian respondents:
(i) More than one-quarter of the US re-
spondents commented on the impact of the
shift from independent practice to managed
care. Respondents described their concerns
regarding the constraints of managed care,
for example, the focus on brief, solution-
oriented therapy; the ethical dilemmas posed
around issues such as confidentiality, privacy,
the termination of therapy, and the move to
outpatient rather than inpatient treatment; the
reliance on computer-based assessment and
diagnosis; the requirements for quality assur-
ance and outcomes-based assessment; and the
focus on the business aspects of practice,
including the entrepreneurial management
and administration of practice.
(ii) Nearly one-quarter of the US respon-
dents and somewhat fewer of the Canadian
respondents commented on the increased em-
phasis on the biological bases of behavior in
clinical practice.
(iii) More than 10% of the US respondents
and somewhat fewer of the Canadian respon-
dents commented on the redefinition of the
practitioner as a primary healthcare provider or
as a case manager supervising other allied health
practitioners responsible for direct services.
(iv) Nearly 10% of the US and Canadian
respondents commented on one or more of the
following shifts toward (a) a more multicultu-
rally diverse patient/client population, (b) a
larger geriatric patient/client population, and
(c) a more severely disturbed (and violent)
society. Each of these trends is associated with
a specific knowledge base.
(v) Approximately 10% of the Canadian
respondents described the shift from employ-
ment within the government-sponsored health-
care system to independent practice and/or
employee-assistance programs, in contrast to
the move in the US from independent practice
to managed care.
Both US and Canadian respondents identi-
fied the following types of knowledge and skills
as needed in the future:
(i) Knowledge regarding the biological bases
of behavior, brief treatments and/or solution-
oriented therapeutic techniques, hypnotherapy,
forensic psychology, employee-assistance pro-
grams, quality assurance procedures, outcomes-
based assessment, and specialties in psychology.
(ii) Professional skills associated with con-
sulting and/or working in interdisciplinary
collaborative teams, procedures to integrate
aspects of spirituality into therapy; sensitivity
skills to handle the increasingly diverse patient/
client population, and self-assessment skills to
recognize one's strengths and limits.
(iii) Generic skills, such as computer skills,
for managerial and professional aspects of
 Conclusions
practice, business skills for developing and
administering a practice, and marketing skills
for surviving in practice.
2.11.3.5 Process- and Content-based Profiles of
Practice
Preliminary statistical analyses revealed that
among the Importance, Time/Frequency, and
Criticality scales, the latter two ratings con-
tributed maximum independent information to
a description or profile of practice. Accordingly,
a process-based profile was developed by
combining the Criticality and Time/Frequency
ratings associated with the roles and responsi-
bilities, and a content-based profile was devel-
oped by combining the Criticality and
Frequency ratings associated with the content
areas, as well as the Criticality and Acquisition
ratings associated with the knowledge state-
ments which were rank-ordered from most
critical and appropriate to be acquired before
licensure, to least critical and not necessary at
any point in practice.
In the case of the process-based profile, the
largest element related to the Direct Service role
(about 70%); the remaining three roles con-
tributed less than 30% to the overall profile of
the critical activities performed by licensed
psychologists. With respect to the content-
based profile, seven of eight content areas each
contributed between 11% and 16% to the
profile, while one content areaÐResearch and
EvaluationÐmade a smaller contribution (6%)
to the profile. The Ethical, Legal, and Profes-
sional Issues content area included three of the
10 highest-ranked knowledge statements; the
Growth and Lifespan Development, Assess-
ment and Diagnosis, and Treatment/Interven-
tion content areas each included two of the 10
highest-ranked knowledge statements; and the
Biological Bases of Behavior content area
included one of the 10 highest-ranked knowl-
edge statements.
2.11.3.6 Development of Test Specifications
A systematic review of the results related to
the process- and content-based profiles of
practice was conducted by a specially consti-
tuted ASPPB Test Specifications Panel. The
following recommendations were prepared to
ensure that the EPPP would reflect the
responsibilities performed by licensed psychol-
ogists making the greatest contribution to the
protection of the patient/client/public from
harm; emphasize the knowledge needed by
licensed psychologists that serves the public
protection function of regulation; and address
129
the issue of developing a generic examination
for psychologists.
(i) The use of the content areas and knowl-
edge statements as the primary organizing
structure for the test specifications and feed-
back to the candidates.
(ii) The adoption of percentage weights for
the eight content areas.
(iii) The use of a hierarchical weighting for
each knowledge statement, indicating the de-
gree to which related questions might be
included in each version of the EPPP. In order
of priority, the system incorporated the Criti-
cality ratings and the Acquisition ratings for the
knowledge statements.
(iv) The revision of eight knowledge state-
ments, including additions, deletions, and mod-
ifications of the examples included as part of the
knowledge statements; and the elimination of
three knowledge statements via the incorpora-
tion of the related examples into other knowl-
edge statements.
(v) The use of the roles and responsibilities as
an organizing structure for item-writing initia-
tives and examination assembly.
(vi) The adoption of percentage weights for
the four roles and associated responsibilities;
(vii) The use of a classification system by
which: (a) test items would be identified by role,
responsibility, content area, and knowledge
statement; and (b) forms of the EPPP would
be constructed so as to reflect the content- and
process-based weights in the test specifications.
2.11.4 CONCLUSIONS
The analysis of the practice of psychology
reported in this chapter provides a picture of
what licensed psychologists in the USA and
Canada know and do. However, it is apparent
from the descriptions of the short- and long-
range changes occurring in the practice of the
profession that frequent reviews and updates
are needed to ensure that the picture of
practiceÐand the content of the EPPPÐ
continue to reflect contemporary practice.
Rapid changes in the delivery of healthcare
services mandated by the move to managed care
in the USA and to nongovernment-sponsored
employment in Canada may require an ex-
panded knowledge base. Similarly, the changing
gender base of the profession and the shifts in
the client base, along with the move to increased
interdisciplinary collaboration may impact the
required knowledge base.
Accordingly, future practice analysis initia-
tives may require: (i) the periodic, but frequent
monitoring of specific settings and/or specialties
undergoing rapid changes in practice, (ii) the
130 Defining Constructs Underlying the Practice of Psychology in the US and Canada
study of professional relationships among intra-
and interdisciplinary team members, and (iii)
qualitative and quantitative outcome studies of
practice as well as investigations of the percep-
tions of employers and the public regarding the
demands of practice and the value and meaning
of the licensure credential in psychology. The
outcomes of these kinds of studies will be very
useful in the continuous process of re-examining
and re-evaluating the role and function of
regulation in our society (Interprofessional
Workgroup on Health Professions Regulation,
1996; Pew Health Professions Commission,
1995).
The analysis of practice provides a database
for policy-making in regard to education,
certification, and accreditation initiatives in
professional psychology. The results of the
analysis should be reviewed by those responsible
for: college, university, and professional school
curricula; training programs; continuing educa-
tion, in-service education; the development and
implementation of specialty certification initia-
tives; and the accreditation of school, intern-
ship, and residency training programs.
Finally, the results of the current analysis
should be viewed in the context of practice in
other nations and/or political entities. The
implementation and/or emergence of the North
Atlantic Free Trade Agreement, the General
Agreement on Trade and Tariffs, and the
European Union indicates the appropriateness
of a global perspective and the need to under-
stand the commonalities and differences in
practice and in the regulation of psychology
around the world that could be provided by the
conduct of international studies of practice
(ASPPB, 1996).
2.11.5 APPENDIX
2.11.5.1 Roles and Definitions
Direct Service. The provision and/or adminis-
tration of psychological services to clients,
patients, and/or organizations in the areas of
problem definition, need assessment, and diag-
nosis; and the design, implementation, and
evaluation of interventions.
Outreach and Consultation. The preparation,
presentation, and coordination of educational
programs, and/or the dissemination of informa-
tion or the provision of expertise to a variety of
audiences.
Academic Preparation and Professional Devel-
opment. The development, implementation, and
administration of education programs for
psychologists, including teaching, supervision,
and curricula.
Research and Evaluation. The development and/
or participation in any investigation and/or the
use of results to expand or refine knowledge or
to improve programs and services.
2.11.5.1.1 Responsibilities related to direct
service
(i) Make and/or receive referrals.
(ii) Coordinate service delivery with other
psychologists and professionals (e.g., health
professionals, managed care systems, organiza-
tional personnel, schools, community groups,
and other outside agencies).
(iii) Observe, interview, and gather informa-
tion from patient/client/organization and re-
lated sources (e.g., relevant others, written
records, referral source) to identify the pro-
blems/needs and their contexts.
(iv) Develop assessment procedures and/or
instruments (e.g., behavioral analyses, struc-
tured interviews, work samples, performance
tests) for the assessment of relevant character-
istics of individuals, groups, jobs, organiza-
tions, educational and social institutions, and/
or environments
(v) Select, administer, and score norm-refer-
enced, standardized, or other instruments for
the assessment of relevant characteristics
of individuals, groups, jobs, organizations,
educational and social institutions, and/or
environments.
(vi) Evaluate and integrate results of
information-gathering and assessment pro-
cesses with scientific/professional knowledge
to formulate/reformulate working hypotheses,
diagnoses, and intervention recommendations.
(vii) Plan, design, and implement interven-
tion programs (e.g., define goals and objectives,
identify appropriate intervention targets and
strategies).
(viii) Monitor and evaluate efficacy of inter-
ventions/programs, and modify as appropriate.
(ix) Document and/or communicate assess-
ment results, intervention recommendations,
progress, and outcomes.
(x) In administering a professional
practiceÐdesign, implement, and monitor qual-
ity assurance, quality control, risk management,
and/or other procedures.
(xi) Provide direct service in a manner con-
sistent with current professional and ethical
standards and guidelines, and state/provincial
and national laws and regulations.
2.11.5.1.2 Responsibilities related to outreach
and consultation
(i) Prepare/present/coordinate health pro-
motion programs or workshops (e.g., smoking
 Appendix
cessation, parenting, anger control manage-
ment, informational programs on community
psychological services/resources).
(ii) Prepare/present/coordinate prevention
and/or early intervention programs for at-risk
populations (e.g., substance abuse prevention,
HIV±AIDS education, community programs
for the elderly).
(iii) Prepare/present/coordinate classes,
seminars, or workshops for clients, family
and significant others, personnel in school
systems, medical and allied healthcare person-
nel, human resource personnel, and/or the
general public.
(iv) Provide expertise to and/or serve on
local/state/provincial/federal agencies (e.g.,
community outreach program, jurisdictional
licensing board, legal system).
(v) Provide expertise to and/or serve on
local/state/provincial, national or international
professional psychology organizations (e.g.,
APA and Canadian Psychological Association
(CPA), American Psychological Society,
ASPPBs, Society for Industrial and Organiza-
tional Psychology).
(vi) Disseminate knowledge of psychology
and its value to the general public.
(vii) Provide consultation regarding design,
methodology, statistical analysis, and/or sig-
nificance of reported data and conclusions of a
research or evaluation study.
(viii) Provide expertise to and/or serve on
site-specific patient-care, education-related, or
research-related committees (e.g., program ac-
creditation, Institutional Review Board, Qual-
ity Assurance Committee).
(ix) Establish and maintain intra- and inter-
disciplinary collaborative relationships, within
institutional settings and with other profes-
sionals.
(x) Provide education and consultation in
a manner consistent with current professional
and ethical standards and guidelines, and state/
provincial and national laws and regulations.
2.11.5.1.3 Responsibilities related to academic
preparation and professional
development
(i) Prepare/present/coordinate classes, semi-
nars, or workshops for undergraduates, pre-
and postdoctoral students, and professional
psychologists.
(ii) Develop/administer/coordinate pre- and
postdoctoral practicum, internship, and fellow-
ship programs in human services settings.
(iii) Develop/administer/coordinate under-
graduate, pre- and postdoctoral, and continu-
ing education programs in professional
psychology.
131
(iv) Supervise pre- and postdoctoral students
and professional practitioners to enhance ser-
vice delivery.
(v) Supervise and advise undergraduates and
pre- and postdoctoral students on research/
evaluation (e.g., honors thesis, dissertation).
(vi) Provide mentoring for undergraduates,
pre- and postdoctoral students, and/or profes-
sional psychologists.
(vii) Provide training in a manner consistent
with current professional and ethical standards
and guidelines, and state/provincial and na-
tional laws and regulations.
(viii) Participate in professional self-devel-
opment and continuing education designed to
enhance professional and personal knowledge
and skills.
2.11.5.1.4 Responsibilities related to research
and evaluation
(i) Critically review and appraise existing
literature with regard to study design, metho-
dology, method of analysis, and generalizability
of results and conclusions.
(ii) Use the existing knowledge base to for-
mulate clear research/evaluation questions or
to guide intervention or program development.
(iii) Engage in research in a manner that
ensures protection of human and/or animal
rights, adhering to current professional and
ethical standards/guidelines, and jurisdictional
and national laws/regulations.
(iv) Formulate research/program evaluation
hypotheses, and design appropriate methods to
conduct the study.
(v) Collect and analyze data using appro-
priate methods of analysis (e.g., qualitative,
quantitative).
(vi) Report research findings and implica-
tions according to professionally accepted stan-
dards.
(vii) Submit research findings to peer review
for publication and/or presentation.
(viii) Apply research findings in practice,
with awareness of strengths and limitations of
application.
(ix) Prepare proposals to funding agencies.
(x) Provide expertise to and/or serve in an
editorial capacity on professional journals or
other refereed publications, or review proposals
to funding agencies.
2.11.5.2 Content Areas and Definitions
Biological Bases of Behavior. Knowledge of
neuroscience, the physiological bases of beha-
vior and illness, and psychopharmacology.
Cognitive-Affective Bases of Behavior. Knowl-
edge of cognitive science; theories of learning,
132 Defining Constructs Underlying the Practice of Psychology in the US and Canada
memory, motivation, and emotion; and factors
that influence an individual's cognitive perfor-
mance and/or emotional experience.
Social and Multicultural Bases of Behavior.
Knowledge of social cognition, social interac-
tion processes, and organizational dynamics;
theories of personality; and issues in diversity
(multiethnic, multicultural, gender, sexual or-
ientation, and disability).
Growth and Lifespan Development. Knowledge
of age-appropriate child, adolescent, and adult
development; atypical patterns of development;
and the protective and risk factors that influ-
ence developmental outcomes for individuals.
Assessment and Diagnosis. Knowledge of psy-
chometrics, assessment models, methods for
assessment of individuals and organizations/
systems, and diagnostic classification systems
and issues.
Treatment/Intervention. Knowledge of indivi-
dual, group, or organizational interventions for
specific concerns/disorders; treatment theories;
and consultation models and processes.
Research and Evaluation. Knowledge of re-
search design, methodology, and program eva-
luation; statistical procedures; and criteria for
accurate interpretation of research findings.
Ethical/Legal/Professional Issues. Knowledge
of the ethical code, professional standards for
practice, legal mandates, guidelines for ethical
decision-making, and professional training and
supervision.
2.11.5.2.1 Knowledge statements related to
biological bases of behavior
(i) Basic neuroscience (e.g., neuroanatomy,
neurophysiology, neurochemistry) and clinical
neuroscience (e.g., brain±behavior relation-
ships, neurological syndromes and their con-
tribution to cognitive and emotional status and
behavior).
(ii) Physiological correlates/determinants of
behavior and affect (e.g., symptoms of common
psychophysiologic reactions and syndromes,
such as hyperventilation, anxiety disorders,
depressive disorders, stress reactions, head-
aches, irritable bowel syndrome).
(iii) Biological bases of the behavior and
affect associated with acute and chronic illness
(e.g., post-stroke depression, diabetes, AIDS,
asthma, chemotherapy, fibromyalgia, hypogly-
cemia, schizophrenia). Includes knowledge of
psychoneuroimmunology.
(iv) Basic psychopharmacology (e.g., medi-
cation effects, side-effects, and interactions).
Includes knowledge of drug metabolism, drug
categories (e.g., anxiolytics, antidepressants,
antipsychotics, anticonvulsants), and addic-
tive/dependency potential.
(v) Genetic transmission (e.g., the relation-
ship of dominant and recessive genes) and its
role in understanding disorders and their beha-
vioral, emotional, and psychosocial manifesta-
tions (e.g., Duchenne's muscular dystrophy,
Huntington's disease, Down's syndrome).
(vi) Relationship of stress to biological and
psychological functioning, with particular re-
ference to lifestyle and lifestyle modification
(e.g., cardiac rehabilitation, smoking cessation).
2.11.5.2.2 Knowledge statements related to
cognitive-affective bases of behavior
(i) Cognitive science (e.g., sensation and
perception, attention, memory, language and
spatial skills, intelligence, information proces-
sing, problem-solving, strategies for organizing
information).
(ii) Theories and principles of learning (e.g.,
social learning, classical and operant condition-
ing, primacy/recency effects).
(iii) Theories of motivation (e.g., need/value
approaches, cognitive choice approaches, self-
regulation).
(iv) Theories of emotions.
(v) Reciprocal interrelationships among cog-
nitions/beliefs, behavior, affect, temperament,
and mood (e.g., healthy functioning, perfor-
mance anxiety, performance enhancement, job
satisfaction, depression).
(vi) Influence of psychosocial factors (e.g.,
sex differences, family styles and characteristics,
academic/occupational success) on beliefs/cog-
nitions and behaviors.
2.11.5.2.3 Knowledge statements related to
social and multicultural bases of
behavior
(i) Social cognition and perception (e.g.,
attribution theory and biases, information in-
tegration, confirmation bias, person perception,
development of stereotypes, racism).
(ii) Social interaction (e.g., interpersonal re-
lationships, aggression, altruism, attraction).
(iii) Group dynamics and organizational
structures (e.g., school systems, gang behavior,
family systems, group thinking, cultural beha-
vior, conformity, compliance, obedience, per-
suasion) and social influences on individual
functioning.
(iv) Environmental/ecological psychology
(e.g., person±environment fit, crowding, pollu-
tion, noise).
(v) Theories of personality that describe
behavior and the etiology of atypical behavior.
Includes knowledge of limitations in existing
theories for understanding the effect of diversity
(e.g., age, ethnicity, gender).
 Appendix
(vi) Multicultural and multiethnic diversity
(e.g., racial/ethnic minorities, gender, age, dis-
ability, sexual orientation, religious groups,
between- and within-group differences).
(vii) Theories of identity development of
multicultural/multiethnic groups (e.g., accul-
turation theories, racial/ethnic identity).
(viii) Role that race, ethnicity, gender, sexual
orientation, disability, and other cultural
differences play in the psychosocial, political,
and economic development of individuals/
groups.
(ix) Sexual orientation issues (e.g., sexual
identity, gay/lesbian/bisexual, family issues).
(x) Psychology of gender (e.g., psychology of
women, psychology of men, gender identity
development).
(xi) Disability and rehabilitation issues (e.g.,
inclusion, psychological impact of disability).
2.11.5.2.4 Knowledge statements related to
growth and lifespan development
(i) Normal growth and development (cogni-
tive, social, personality, moral, emotional, and
physical) from conception through old age.
(ii) Normative or age-expected behaviors
(e.g., normal age range, individual differences)
and how the definition of normative behavior is
influenced by culture.
(iii) Risk factors which predict an atypical
developmental course (e.g., nutritional deficien-
cies; healthcare, including prenatal care; avail-
ability of social support; adequacy of income
and housing; poverty; parental alcohol/drug
abuse).
(iv) Interventions to reduce risk factors
(e.g., poor healthcare, nutritional deficiencies,
violence) and to increase resilience (e.g., pro-
tective factors such as caregiving, increased
social support) and competence (e.g., skill
building) of individuals living in at-risk envir-
onments.
(v) Life-event changes that can alter the
normal course of development (e.g., injury,
trauma, illness, onset of chronic disease or
disorder in self or parent, death, divorce).
(vi) Theories of development (e.g., construc-
tivist theory, social learning theory, ecological
theory).
(vii) How development is influenced by the
organism±environment interaction over time
(e.g., understanding the relationship between
the behavior of the individual and the social,
academic, or work environment).
(viii) Family systems functioning and family
stages in life and how these impact on indivi-
duals (e.g., family life cycle, parent±adolescent
communication, birth of child).
133
2.11.5.2.5 Knowledge statements related to
assessment and diagnosis
(i) Psychometric theory and concepts (e.g.,
measurement, reliability, validity, item charac-
teristics, test fairness, standardization, norms),
and test validation procedures (e.g., criterion,
predictive, construct, and content strategies;
appropriate measurement standards and legal
regulations).
(ii) Assessment models (e.g., psychometric,
behavioral, neuropsychological, ecological).
(iii) Tests for the measurement of character-
istics of individuals (e.g., social, emotional, and
behavioral functioning; cognitive; achievement;
aptitude; personality; neuropsychological; vo-
cational interest), and the adaptation of these
tests for use with special populations.
(iv) Techniques other than tests (e.g., inter-
views, surveys, naturalistic and structured be-
havioral observations, physical status, history/
biographical data) for the measurement of
characteristics of individuals.
(v) Instruments and methods for the mea-
surement of characteristics of jobs, organiza-
tions, educational and other social institutions
(e.g., job analysis, job evaluation, need assess-
ment, organizational diagnosis, ecological as-
sessment).
(vi) Methods for evaluating environmental/
ecological influences on individuals, groups, or
organizations (e.g., organizational frameworks,
functional analysis of behavior).
(vii) Criteria for selecting assessment de-
vices/approaches (e.g., cultural appropriate-
ness, utility analysis and cost effectiveness,
relevance to referral concern).
(viii) Various classification systems (e.g.,
DSM-IV (Diagnostic and statistical manual of
mental disorders, 4th edition), AAMR (Amer-
ican Academy of Mental Retardation), ICD
(International classification of diseases)) for
diagnosing client functioning.
(ix) Epidemiology of behavioral disorders,
base rates of disorders in clinical or demo-
graphic populations, comorbidity among beha-
vioral disorders and with medical disorders.
(x) Theory and techniques for the measure-
ment of client changes (e.g., client tracking,
formative and summative evaluation, program
evaluation).
(xi) Human imaging principles, uses, and
clinical implications (e.g., magnetic resonance
imaging, computed tomography scanning,
positron emission tomography scanning, elec-
troencephalography, SPECT, single photo-
emission computed tomography).
(xii) Human laboratory principles, uses, and
clinical implications (e.g., drug screens, screen-
ing for genetic disorders).
134 Defining Constructs Underlying the Practice of Psychology in the US and Canada
(xiii) Use of computers and related technol-
ogy in implementing tests, surveys, and other
forms of assessment.
(xiv) Quality assurance measurement techni-
ques.
2.11.5.2.6 Knowledge statements related to
treatment/intervention
(i) Treatment planning process, including
differential diagnosis and efficacy and outcome
data.
(ii) Theories of treatment (e.g., behavioral,
cognitive, and cognitive-behavioral ap-
proaches; psychodynamic approaches; sys-
tems/ecological approaches; humanistic
approaches).
(iii) Treatment techniques/interventions for
specific concerns or specific populations (e.g.,
marital and family, group therapy, crisis inter-
vention, play therapy, feminist therapy,
approaches to stress management, psychoedu-
cational, time-limited/brief therapy, compen-
sation strategies, culturally appropriate
treatments and interventions).
(iv) System theories and system interventions
(e.g., change of environment, school system,
community interventions, family, job and
equipment design, consultation).
(v) Organizational interventions (e.g., orga-
nizational development, organizational change,
performance enhancement/management).
(vi) Consultation models (e.g., mental
health, behavioral, instructional, organiza-
tional) and processes (e.g., stages, communica-
tion skills).
(vii) Human resource management interven-
tions (e.g., selection, performance appraisal,
training).
(viii) Theories of career development and
counseling (e.g., career assessment, career coun-
seling techniques).
(ix) Adjunctive and alternative interventions
and appropriate referral (e.g., physicians, 12-
step programs, psychopharmacology, inpatient
or partial hospitalization, support groups).
(x) Service delivery systems (e.g., education,
health, mental health, social services, forensics,
business, and industry), including the roles of
other professionals.
(xi) Quality assurance measurement techni-
ques.
2.11.5.2.7 Knowledge statements related to
research and evaluation
(i) Research methods (e.g., sampling, instru-
mentation, data collection procedures).
(ii) Research design (e.g., hypothesis genera-
tion, experimental, quasi-experimental, natur-
alistic inquiry, group designs, and single-case
research).
(iii) Appropriate analytical methods (e.g.,
qualitative, quantitative, descriptive, inferen-
tial; univariate, bivariate, and multivariate;
parametric and nonparametric) and interpreta-
tion (e.g., causal versus correlational; degree
and nature of generalizability).
(iv) Criteria for critical appraisal and utiliza-
tion of research (e.g., technical adequacy;
limitations to generalizations; threats to inter-
nal, external, and construct validity).
(v) Program planning and evaluation strate-
gies and techniques (e.g., need assessment,
process/implementation evaluation, outcome
evaluation).
2.11.5.2.8 Knowledge statements related to
ethical/legal/professional issues
(i) APA Ethical Principles of Psychologists
and Code of Conduct and/or Canadian Code of
Ethics for Psychologists (e.g., confidentiality,
research, dual relationships, limits of compe-
tence, advertising practices, informed consent,
record-keeping).
(ii) Professional standards and guidelines for
the practice of psychology (e.g., APA/CPA
Standards for Providers of Psychological Ser-
vices, AERA/APA/NCME Standards for Edu-
cational and Psychological Testing. ASPPB
Code of Conduct, ASPPB Model Licensure
Act, credentialing requirements for advanced
specialties and proficiencies, other published
guidelines for special populations such as wo-
men and minorities).
(iii) Pertinent federal, state and/or provincial
laws/statutes that affect psychological practice
(e.g., laws and regulations relating to family and
child protection, education, disabilities, discri-
mination, duty to warn and privileged commu-
nication, commitment and least restrictive care,
continuing education requirements, practice
regulations, licensure regulations).
(iv) Ethical decision-making process (e.g.,
balancing professionalism with entrepreneur-
ship, integration of ethical principles and legal/
regulatory standards).
(v) Models and approaches for training and
supervision of self and others (e.g., methods for
developing and enhancing knowledge and pro-
ficiency, continuing education, professional
self-management, clinical supervision, peer con-
sultation and supervision).
2.11.6 REFERENCES
American Educational Research Association, American
Psychological Association, & National Council on
Measurement in Education (1985). Standards for educa-
 References
tional and psychological testing. Washington, DC:
American Psychological Association.
Association of State and Provincial Psychology Boards
(1996). Regulation of professional psychology: An inter-
national resource directory. Montgomery, AL: Author.
Bolton, R. (1993). Pretesting questionnaires: Content
analyses of respondents' concurrent verbal protocols.
Marketing Science, 12, 280±303.
Dillman, D. A. (1978). Mail and telephone surveys: The
total design method. New York: Wiley.
Greenberg, S., Smith, I. L., & Muenzen, P. M. (1996).
Study of the practice of licensed psychologist in the United
States and Canada. Montgomery, AL: Association of
State and Provincial Psychology Boards.
Hambleton, R. K., & Smith, I. L. (1988). Content validity
and fairness review of the 1987 forms of the Examination
for Professional Practice in Psychology (EPPP) (final
report). New York: Professional Examination Service.
Interprofessional Workgroup on Health Professions Reg-
ulation (1996, November). Views of the licensure and
regulation of healthcare professionals. Unpublished
manuscript.
Knapp, J. E., & Knapp, L. G. (1995). Practice analysis:
Building the foundation for validity. In J. C. Impara
(Ed.), Licensure testing: Purposes, procedures, and
practices (pp. 93±116). Lincoln, NE: Buros Institute of
Mental Measurements.
Messick, S. (1989). Validity. In R. L. Linn (Ed.),
Educational measurement (3rd ed., pp. 13±103). New
York: Macmillan.
Messick, S. (1995). Validity of psychological assessment:
Validation of inferences from persons' responses and
performances as scientific inquiry into score meaning.
American Psychologist, 50, 741±749.
135
Pew Health Professions Commission (1995, December).
Reforming healthcare workforce regulation: Policy con-
siderations for the 21st century. San Francisco, CA:
Author.
Richman, S. (1982). The role delineation study for the
Examination for Professional Practice in Psychology
(final report). New York: Professional Examination
Service.
Rosen, G. A. (1991). Research digest: the Examination for
Professional Practice in Psychology. Montgomery, AL:
American Association of State Psychology Boards.
Rosen, G. A., & Mirone, J. A. (1986). The test specification
revision project for the Examination for Professional
Practice in Psychology (final report). New York:
Professional Examination Service.
Rosenfeld, M., Shimberg, B., & Thornton, R. F. (1983).
Job analysis of licensed psychologists in the United States
and Canada. Princeton, NJ: The Educational Testing
Service.
Schoon, C. G. (1985). Methods for defining and assessing
professional competence. Professional Practice of Psy-
chology, 6(1), 144±155.
Schoon, C. G., & Smith, I. L. (1996, Summer). Guidelines
for credentialing programs. Professions Education Re-
search Quarterly, 17(4), 6±9.
Smith, I. L. (1984). Content validity study of the EPPP item
bank. New York: Professional Examination Service.
Smith, I. L., & Greenberg, S. (1991). Final report for the
EPPP test specifications enhancement project. New York:
Professional Examination Service.
Werner, E. (1989). Analysis of the validity of the Examina-
tion for Professional Practice of Psychology. Sacramento,
CA: California Department of Consumer Affairs.