Canadian Journal of

School Psychology
http://cjs.sagepub.com/

The Assessment of School Psychologists in Practice Through Multisource

Feedback
Jac J. W. Andrews and Claudio Violato
Canadian Journal of School Psychology 2010 25: 328 originally published online 15
June 2010
DOI: 10.1177/0829573510373585
The online version of this article can be found at:
http://cjs.sagepub.com/content/25/4/328

Published by:
http://www.sagepublications.com

On behalf of:

Canadian Association of School Psychologists

Additional services and information for Canadian Journal of School Psychology can be found at:
Email Alerts: http://cjs.sagepub.com/cgi/alerts
Subscriptions: http://cjs.sagepub.com/subscriptions
Reprints: http://www.sagepub.com/journalsReprints.nav
Permissions: http://www.sagepub.com/journalsPermissions.nav
Citations: http://cjs.sagepub.com/content/25/4/328.refs.html
Downloaded from cjs.sagepub.com by Widiastuti Pajarini on April 29, 2012

>> Version of Record - Nov 23, 2010

OnlineFirst Version of Record - Jun 15, 2010
What is This?

Downloaded from cjs.sagepub.com by Widiastuti Pajarini on April 29, 2012

The Assessment
of School Psychologists
in Practice Through
Multisource Feedback

Canadian Journal of School Psychology

25(4) 328346
2010 SAGE Publications
Reprints and permission: http://www.
sagepub.com/journalsPermissions.nav
DOI: 10.1177/0829573510373585
http://cjs.sagepub.com

Jac J. W. Andrews1 and Claudio Violato1

Abstract
In this article we provide an overview of the nature and scope of multisource feedback
(MSF) and provide empirical evidence of its reliability, validity, and feasibility in one of the
health professions. The overall internal consistency reliability (Cronbach alpha) of MSF
instruments is generally greater than .96 for self and informants such as patients, coworker,
and colleague surveys. Generalizability coefficients for the assessors across persons are
approximately 0.80. There is also substantial evidence of content, criterion-related and
some evidence of construct validity of the MSF instruments applied in the health professions.
Based on these findings, we recommend the development and use of a MSF system for
practicing school psychologists, present information about how MSF instruments can be
constructed, and provide examples of what these instruments could look like.
Rsum
Cet article vous offre un aperu de la nature et de la porte des multisource feedback
(MSF) et vous fournit des preuves empiriques de sa fiabilit, de sa validit et la faisabilit
dans une des professions de la sant. La fiabilit de la cohrence interne globale
(Cronbach alpha) des outils de MSF est gnralement plus grande que, 96 pour des
sondages effectus sur soi-mme, sur des individus tels que des patients, des collgues
de travail ou collaborateurs. Les coefficients de gnralisibilit pour les valuateurs
travers les personnes sont denviron 0,80. Il y a aussi des preuves substantielles du
contenu, les critres lis et certains lments de preuve de la validit des outils
MSF appliqus dans les professions de la sant. Sur la base de ces rsultats, nous
recommandons llaboration et lutilisation dun systme de MSF pour la pratique de la
psychologie en milieu scolaire; prsenter des informations sur la faon dont les outils
MSF peuvent tre dvelopps et fournir des exemples de ce que ces outils peuvent tre.
1

University of Calgary, Calgary, Alberta

Corresponding Author:
Jac J. W. Andrews, Division of Applied Psychology, University of Calgary,
500 University Drive N. W., Calgary, AB T2N1N4, Canada
Email: jandrews@ucalgary.ca
Downloaded from cjs.sagepub.com by Widiastuti Pajarini on April 29, 2012

Andrews and Violato

329

Keywords
multisource feedback, school psychologist assessment, 360-degree evaluation,
professional development

Introduction
Although it has been recommended that multisources of information (360-degree feedback) be utilized in the assessment of client outcomes from therapy (Cone, 2001) and
that 360-degree feedback be applied to psychology training (Falender & Shafranske,
2004), a multisource feedback (MSF) system has yet to be adopted as a systematic way
to assess the practice of psychologists in Canada. MSF systems have been widely used
both in the health professions and industry for assessing professionals (Lockyer &
Clyman, 2008). The major purpose of the present article is to describe the nature and
scope of MSF instruments that have been developed and utilized by some of the health
professions in Canada and to exemplify some empirical information with respect to the
use of these instruments in one of the health professions. A secondary purpose of this
article is to recommend the development and use of an MSF for practicing school
psychologists, present information about how MSF instruments can be constructed,
and provide examples of what these instruments could look like.

Nature and Scope of Multisource Feedback

MSF (sometimes referred to as 360-degree assessment) involves the creation of instruments to assess and compare observable behaviours of professionals. Typically, this
involves a self-assessment and an assessment from others (e.g., colleagues, coworkers,
clients). The use of multiple sources of data to evaluate performance has been considered highly desirable in business (Sala & Dwight, 2002) as well as in medicine
(Lockyer, Violato, & Fidler, 2003; Norcini, 2003; Ramsey et al., 1993) and other health
professions (e.g., occupational therapy; Violato, Worsfold, & Polgar, 2009). It has been
widely adopted in industry as a way of providing feedback to employees to guide selfdirected learning and improve workplace performance (Sala & Dwight, 2002). In
industrial settings, supervisors, peers, and occasionally clients provide feedback. MSF
is frequently used in settings in which the staff person works in a team and/or cannot be
directly and easily supervised by managers (Church, 1997). In medical application of
MSF, physicians complete a self-assessment instrument and receive feedback from
medical colleagues (peers), nonmedical coworkers (e.g., nurses, psychologists, pharmacists), and patients (Violato, Lockyer, & Fidler, 2008a).
Violato and his colleagues began the development and testing of MSF instruments
in Canada in 1996 (Hall et al., 1999; Violato, Marini, Toews, Lockyer, & Fidler, 1997).
At that time, the instruments were developed to assess family physicians, obstetricians, internists, and paediatricians. This work led to the adoption of MSF as a mandatory requirement for continued licensure in the province of Alberta and the Physician
Assessment Program (PAR) of the College of Physicians and Surgeons of Alberta (see

Downloaded from cjs.sagepub.com by Widiastuti Pajarini on April 29, 2012

330

Canadian Journal of School Psychology 25(4)

www.par-program.org). PAR assesses the performance of physicians and is intended

for quality improvement in medical practice by providing structured feedback and
identifying physicians for whom a more detailed assessment of practice performance
is needed. Subsequently, instruments have been developed for assessing surgeons,
medical, paediatric and psychiatry specialists (Lockyer & Violato, 2004; Violato,
Lockyer, & Fidler, 2003) as well as anaesthesiology and episodic care physicians.
These instruments assess the areas that both American and Canadian specialty organizations have identified as critical areas for attention (American Council Graduate
Medical Education, 2004; Levine, 2002; Societal Needs Working Group, 1996).
MSF systems have been used for both formative and summative assessment of
persons and as a way to promote change of practice of persons assessed (Lockyer &
Clyman, 2008). Typically, the assessed person as well as his or her colleagues, peers,
and clients complete questionnaires examining core competencies. This feedback system provides a more global perspective that can be provided by one or a few sources
alone (Bracken, Timmreck, & Church, 2001). MSF systems can assess important
characteristics of health professionals such as their clinical skills, personal communication, and client management as well as facilitate changes in their practice. Fidler,
Lockyer, Toews, and Violato (1999) in a study of family physicians, for example,
reported that 66% made at least one change in their communication with patients and
support of patients based on multisource feedback.
An important issue with the use of a questionnaire-based MSF system is whether or
not it can provide adequate reliability, evidence of validity, and demonstrate utility or
feasibility. Some evidence of these psychometric properties has been reported over the
past decade with instruments developed and analyzed for a number of health professionals, for example, family doctors (Violato et al., 2008b), psychiatrists (Violato et al.,
2008a), and pathologists and laboratory medicine physicians (Lockyer, Violato, & Fidler,
2008; Lockyer, Violato, Fidler, & Alakjia, 2009). As an example, in a study that focused
on the development and psychometric assessment of a questionnairebased MSF for
quality improvement for occupational therapists, Violato et al. (2009) developed surveys
for the assessment of 238 occupational therapists by clients, coworkers, and themselves,
respectively, using 5-point Likert-type scales and an unable to assess category. The competencies assessed by these surveys included professional responsibilities, practice
knowledge, critical thinking, communication, professional development, and management of the practice environment. The number of respondents for the coworker questionnaire was 2,621, and for the client questionnaire it was 2,881. The mean ratings ranged
from 4 to 5 for each item on each scale. All of the instruments full scales had very high
Cronbachs alphas > .92. The factor analysis revealed a 7-factor solution (66.3% of the
total variance) for the coworker survey, and a 4-factor solution for the client questionnaire (73.2% of the variance).The factors (professional responsibilities, practice
knowledge, utilizes practice process, critical thinking, communication, professional
development, manages practice environment) were found to be theoretically meaningful
and cohesive, providing some evidence of the construct validity of the instruments.
On a follow-up survey with the occupational therapists, 90% of those surveyed
found the self, coworker, and client instruments easily readable and easy to understand.

Downloaded from cjs.sagepub.com by Widiastuti Pajarini on April 29, 2012

Andrews and Violato

331

Moreover, the occupational therapists found it was easy to participate in the system
and recruit the required number of informants (coworkers and clients). In summary,
the results demonstrated that the developed instruments and procedures had high reliability, validity, and feasibility.

Assessment of Competence in Psychology

Over the past few years, there has been support for a shift to the assessment of professional competence in professional psychology that involves a continual and systematic
competency-based assessment approach from training to retirement (Kaslow et al., 2007).
There has been long discussion, however, and a lack of consensus in the field about how
to define and measure competence in professional psychology (Fantuzzo, Sisemore, &
Spradlin, 1983; Kaslow, 2004; Shaw & Dobson, 1988) throughout the past years. Coincidentally, licensing and regulatory bodies as well as graduate training programs also vary
in their definitions, procedures, and instruments for assessing competence in the profession of psychology (Roberts, Borden, Christiansen, & Lopez, 2005).
Currently, there is some consensus among many of the leaders in the field of
professional psychology that assessment of competence become more internalized
and institutionalized along the career path of the professional psychologist (Roberts
et al., 2005). Thus it has been recommended that assessment processes should be
developed and implemented that result in feedback to assist professionals in continual learning throughout their careers (Roberts et al., 2005), that assessment of competence include multiple perspectives, for example, by way of MSF that integrates
input from multiple sources (i.e., self, colleagues, clients; Atkins & Wood, 2002;
Fletcher & Bailey, 2003; Maurer, Mitchell, & Barbeite, 2002), that methods of
assessing competency meet criteria for validity, reliability, and feasibility (Roberts
et al., 2005), and that, among other considerations, core competencies be identified
and benchmarks for performance be established (Kaslow et al., 2007). For core
competencies, domains key to effective functioning have been by and large identified and agreed on (Rodolfa, 2005; Rodolfa, Bent, Eisman, Nelson, & Ritchie, 2005;
Stern, 2006). Kaslow et al. (2007), for example, have identified foundational
domains (e.g., professionalism, reflective practice/self-assessment, scientific knowledge and methods, relationships, ethical and legal practice, individual and cultural
diversity, interdisciplinary systems) and functional domains (e.g., assessment,
diagnosis, and conceptualization, intervention, consultation, research, assessment,
supervision, training management, administration).
It is our contention that an MSF system can be developed to assess the core competencies (such as those noted above) of school psychologists that is reliable, valid,
and feasible. Moreover, we suggest that such a system could not only identify strengths
and weaknesses of school psychologists of their core competencies but also provide
useful information and guidance for their professional development. In this regard, we
are not suggesting that MSF is specific to school psychologists. Indeed, MSF can be
used with counselling psychologists, clinical psychologists, as well as with a plethora
of other health professionals (see, for example, Connors & Munro, 2001; Griffin,

Downloaded from cjs.sagepub.com by Widiastuti Pajarini on April 29, 2012

332

Canadian Journal of School Psychology 25(4)

Sanders, Craven, & King, 2000; Violato et al., 2008a, as viable applications of MSF
with other health professionals).

Competence-based Assessment for School Psychologists

A competency-based assessment system for school psychologists can potentially provide feedback to psychologists about their performance and improve their practice as
evidenced with other health professionals. The development and implementation of
MSF instruments designed to assess school psychologist in practice would seem to be
a timely and valuable undertaking within and across the provinces and territories of
Canada. A MSF system that incorporates self-assessment along with peer, coworker,
and client assessment could provide information that is not only useful to practicing
psychologists but also useful for national and provincial psychology associations and
provincial and territorial psychology regulatory boards relative to their oversight and
governance of psychology. In this regard, MSF has been and is currently used, by a
wide variety of regulatory bodies in the health professions. Some of these regulatory
bodies include the College of Physicians and Surgeons of Alberta, College of Physicians and Surgeons of Ontario, College of Occupational Therapists of Ontario, and
the British Columbia College of Nurses (Violato et al., 2003; Lockyer, Violato, &
Fidler, 2003). Medical regulatory bodies have also implemented MSF in the United
Kingdom (e.g., Archer, Norcini, & Davies, 2005) and the United States (e.g., Bennett,
Gatrell, & Packham, 2004). Importantly, the purpose of MSF is to provide systematic
feedback to practitioners for formative purposesto monitor and improve their practice. Regulatory bodies typically have the time, resources, moral obligation, and legal
requirements to conduct ongoing professional review of their members of practice as
part of their quality assurance program and this is detailed in provincial legislation
across Canada (see, for example, Bill 25, Health Professions [Regulatory Reform]
Amendment Act 2008 and Health Professions Act 2009).
To systematically develop and implement MSF instruments for use with school
psychologists, a number of steps need to be undertaken: (a) formation of a MSF advisory group, (b) development of a table of specification for the MSF instruments, (c) creation of items, and (d) formatting and pilot testing the instruments for evidence of
reliability, validity, and feasibility of instruments with school psychologists.

Formation of a Multisource Feedback

System Advisory Group
The first step in developing a MSF system for school psychologists is to get the support
of one or more of the provincial psychology associations and provincial psychology
regulatory bodies within the Provinces of Canada for the development and trial of
instruments. Once support of the project is given by a psychology association and regulatory body, members for an advisory group need to be selected. Participants of the
advisory group would include members of the psychology association and regulatory
body as well as members deemed as experts in the practice of school psychology as

Downloaded from cjs.sagepub.com by Widiastuti Pajarini on April 29, 2012

Andrews and Violato

333

well as experts in the development and use of MSF systems. The roles and responsibilities of the advisory group would include (a) recruiting members for the advisory group
who could provide further content expertise for the instruments, (b) identifying roles
and competencies as integral to the professional practice of school psychologists,
(c) participating in the development of the instruments, (d) providing information to
school psychologists in the province(s) regarding the development and use of the
instruments, and (e) overseeing the distribution and return of instruments from participating school psychologists involved in the trial and follow-up use of the instruments.

Development of a Table of Specifications

for the Multisource Feedback Instruments
There are different ways of specifying what MSF instruments should measure. A twoway chart called a table of specifications is typically used to organize and make
decisions about the nature and scope of the instrument content, however. The content
information within the table of specifications is typically created from reviewing the
relevant literature and from the expertise of the advisory group members. Essentially,
the preparation of the two-way chart serves as a guide for item construction.

Creating Items
Each member of the advisory group would be responsible (under the guidance of
experts in the construction of MSF systems) to create items that reflect the core competencies of school psychologists. After members are given time to create items, they
would be reviewed, modified, and/or replaced. Decisions regarding the final content
of each item are typically arrived at by using a consensus procedure. Once all of the
items are developed for the instruments and judged satisfactory by the advisory
group, it is necessary to design the format of the instruments and pilot them.

Formatting and Pilot Testing the Instruments

Typically, a Likert-type scale response format is recommended for MSF instruments
(for example, 1 = strongly agree, 2 = agree, 3 = neither agree nor disagree, 4 = disagree,
5 = strongly disagree, 6 = unable to assess/does not apply). Once the formatting of the
instrument is complete, it is typically reviewed and potentially modified by members of
the Advisory group. After the formatting is approved it is ready to be pilot tested. The
instruments would be distributed to a representative sample of school psychologists
who would typically complete the self-assessment instrument and distribute respective
instruments to a predetermined number of clients, colleagues, and coworkers. Once all
of the instruments are returned, they would be analyzed for reliability, validity, and feasibility. Often, based on the results of the first pilot test, instruments are revised and sent
out for a second pilot test to confirm and better evaluate their reliability and validity. In
addition, norm tables can be constructed to compare future respondents results with the
original sample and make further adjustments if necessary. Norm tables allow for

Downloaded from cjs.sagepub.com by Widiastuti Pajarini on April 29, 2012

334

Canadian Journal of School Psychology 25(4)

standard scores to be created and used to interpret and compare the performance of a
persons scores from the instruments. From this information, reports can be created to
provide feedback to school psychologists regarding their performance according to their
self-assessment and in relation to others judgment of their performance.

Example of Multisource Feedback

Instruments for School Psychologists
The following examples of instruments have been modified from instruments developed
and used to assess the performance of medical doctors (i.e., PAR). These modified
instruments only serve as an example of what they might look like for use with school
psychologists. They have not been constructed or reviewed by representatives of practicing school psychologists or by members of provincial psychology associations or
regulatory boards. They have been created by the authors of this manuscript for the
purpose of showing the readers what they might generally look like.
(Examples of the surveys are given in Appendices A-D)

Conclusion
By using the expertise of a broad range of psychological association members, and
regulatory groups, the profession of school psychology can develop a broadly applicable MSF system that can meet the current assessment standards of reliability, validity,
and feasibility. This system could be developed relative to the functioning of school
psychologists in their numerous roles. The profession of school psychology has an obligation to ensure the public that its members are competent and be able to offer evidence
of this competence through proper assessment procedures. We suggest that this can be
done by the development and use of MSF instruments that involves a self-assessment
and an assessment from others (e.g., colleagues, coworkers, clients). Finally, although
our examples of instruments can be informally used/modified by school psychologists
to aid in their self-evaluations or be used/modified by supervisors of school psychology
students in training, we believe that MSF would be much better utilized as a systematic
and standardized procedure in the assessment of school psychologists in practice.

Appendix A
Psychologists Assessment of Performance
Self-Assessment Questionnaire
Name _____________________
How to Answer:
Beside each question there is a row of numbers ranging from (1) to (6)
(continued)
Downloaded from cjs.sagepub.com by Widiastuti Pajarini on April 29, 2012

335

Andrews and Violato

Appendix A (continued)
Make your response by circling the number that is closest to the way you want to
respond
Compared to other psychologists you know, rate your performance for each
statement
1. Among the worst
2. Bottom half
3. Average
4. Top half
5. Among the best
6. Unable to assess
Example:
I communicate effectively with clients.
If you rate your performance among the worst, circle
1

If you rate your performance in the Top Half, circle

2

Thank you for your cooperation

RESPONSE CODE
1

Among the worst

Bottom half

Average

Top half

Among the best

Unable to
assess

Answer all questions by circling one of the numbers in the right hand
column referring to the response code at the bottom of the page
1. I communicate effectively with clients
2. I communicate effectively with clients families
3. I communicate effectively with other health care professionals

1
1
1

2
2
2

3
3
3

4
4
4

5
5
5

6
6
6

(continued)

Downloaded from cjs.sagepub.com by Widiastuti Pajarini on April 29, 2012

336

Canadian Journal of School Psychology 25(4)

Appendix A (continued)
4. Within the range of services provided by me, I perform technical
procedures skilfully
5. Within the range of services provided by me, I demonstrate
appropriate judgment
6. I select psychoeducational tests appropriately
7. I critically assess diagnostic information
8. I make the correct diagnosis in a timely fashion
9. I select the appropriate recommendations
10. I maintain quality psychological records
11. I obtain informed consent for my services
12. I inform clients of the limits of confidentiality
13. I report any circumstances that affect the reliability, validity, or
interpretation of assessment results
14. I limit my practice to areas of competence in which proficiency
has been gained through education and training
15. I refer clients in an appropriate manner
16. I provide clear understanding about who is responsible for
continuing care of the client
17. I communicate referral information to clients
18. I recognize biological aspects of psychological disorder
19. I make appropriate use of community resources for biological
aspects of care
20. I manage clients with complex psycho-educational problems
21. I coordinate care effectively for clients with other health
professionals
22. I show compassion for clients and their families
23. I respect the rights of clients
24. I collaborate with other psychologists
25. I am involved with professional development
26. I accept responsibility for my professional action
27. I manage psychological care resources efficiently
28. I manage personal stress
29. I am aware of my own shortcomings

1
1
1
1
1
1
1
1

2
2
2
2
2
2
2
2

3
3
3
3
3
3
3
3

4
4
4
4
4
4
4
4

5
5
5
5
5
5
5
5

6
6
6
6
6
6
6
6

1
1

2
2

3
3

4
4

5
5

6
6

1
1
1

2
2
2

3
3
3

4
4
4

5
5
5

6
6
6

1
1

2
2

3
3

4
4

5
5

6
6

1
1
1
1
1
1
1
1

2
2
2
2
2
2
2
2

3
3
3
3
3
3
3
3

4
4
4
4
4
4
4
4

5
5
5
5
5
5
5
5

6
6
6
6
6
6
6
6

Appendix B
Psychologists Assessment of Performance
Psychologist Colleague Questionnaire
Assessed psychologists Name_______________________
Your name _____________________
(continued)

Downloaded from cjs.sagepub.com by Widiastuti Pajarini on April 29, 2012

337

Andrews and Violato

Appendix B (continued)
How would you describe your professional relationship to the psychologist (select one)?
1. Peer (similar practice)
2. Consultant
3. Referring psychologist
How well do you know this psychologist (mark one)?
1. Not at all
2. Not well
3. Somewhat
4. Well
5. Very well
Interpretation of the Rating Scale
Rate your colleague on the performance statements according to the following
guidelines and scale
How to Answer:
Beside each question there is a row of numbers ranging from (1) to (6)
Make your response by circling the number that is closest to the way you want to
respond
Compared to other psychologists I know, this one is:
1. Among the worst
2. Bottom half
3. Average
4. Top half
5. Among the best
6. Unable to assess
Example:
Communicates effectively with clients.
If you rate the psychologists performance among the worst, circle
1

6
(continued)

Downloaded from cjs.sagepub.com by Widiastuti Pajarini on April 29, 2012

338

Canadian Journal of School Psychology 25(4)

Appendix B (continued)
Answer all questions by circling one of the numbers in the right hand
column referring to the response code at the bottom of the page
If you rate the psychologists performance in the Top half, circle
1

Thank you for your cooperation

RESPONSE CODE
1
Among the
worst

Bottom half

Average

Top half

Among the best

Unable to assess

1. Communicates effectively with clients

2. Communicates effectively with clients families
3. Communicates effectively with other health care professionals
4. Within the range of services provided by this psychologist, he or
she performs technical procedures skilfully
5. Within the range of services provided by this psychologist, he or
she demonstrates appropriate judgment
6. Selects psychoeducational tests appropriately
7. Critically assesses diagnostic information
8. Makes the correct diagnosis in a timely fashion
9. Selects the appropriate recommendations
10. Maintains quality psychological records
11. Obtains informed consent for services
12. Informs clients of the limits of confidentiality
13. Reports any circumstances that affect the reliability, validity, or
interpretation of assessment results
14. Limits his or her practice to areas of competence in which
proficiency has been gained through education and training
15. Refers clients in an appropriate manner
16. Provides clear understanding about who is responsible for
continuing care of the client
17. Communicates referral information to clients
18. Recognizes biological aspects of psychological disorder
19. Makes appropriate use of community resources for biological
aspects of care

1
1
1
1

2
2
2
2

3
3
3
3

4
4
4
4

5
5
5
5

6
6
6
6

1
1
1
1
1
1
1
1

2
2
2
2
2
2
2
2

3
3
3
3
3
3
3
3

4
4
4
4
4
4
4
4

5
5
5
5
5
5
5
5

6
6
6
6
6
6
6
6

1
1

2
2

3
3

4
4

5
5

6
6

1
1
1

2
2
2

3
3
3

4
4
4

5
5
5

6
6
6

(continued)

Downloaded from cjs.sagepub.com by Widiastuti Pajarini on April 29, 2012

339

Andrews and Violato

Appendix B (continued)
20. Manages clients with complex psychoeducational problems
21. Coordinates care effectively for clients with other health
professionals
22. Shows compassion for clients and their families
23. Respects the rights of clients
24. Collaborates with other psychologists
25. Is involved with professional development
26. Accepts responsibility for his or her professional action
27. Manages psychological care resources efficiently
28. Manages personal stress
29. Is aware of his or her own shortcomings

1
1

2
2

3
3

4
4

5
5

6
6

1
1
1
1
1
1
1
1

2
2
2
2
2
2
2
2

3
3
3
3
3
3
3
3

4
4
4
4
4
4
4
4

5
5
5
5
5
5
5
5

6
6
6
6
6
6
6
6

Appendix C
Psychologists Assessment of Performance
Family Questionnaire
Sex

Male

Female

Age of child who received psychological services

1. 5 or under
2. 6 to 10
3. 11 to 15
4. 16 and above
Over the last 3 years how often have you seen this psychologist?
1. Once
2. 2 to 3 times
3. More than 3 times
Your most recent visit was mainly for:
1. New concern
2. Ongoing concern
Psychologists name__________________________________
Interpretation of the rating scale
(continued)

Downloaded from cjs.sagepub.com by Widiastuti Pajarini on April 29, 2012

340

Canadian Journal of School Psychology 25(4)

Appendix C (continued)
Answer questions about this psychologist according to the following guidelines and
scale
How to answer:
Beside each question there is a row of numbers ranging from (1) to (6)
Make your response by circling the number that is closest to the way you want to
respond
1. Strongly disagree
2. Disagree
3. Neutral
4. Agree
5. Strongly agree
6. Not applicable (NA)
Example:
The psychologist explained your childs problems to you thoroughly
If you strongly disagree with this statement, circle
Answer all questions by circling one of the numbers in the right hand
column referring to the response code at the bottom of the page
1

If you agree with this statement, circle

1

RESPONSE CODE
1
Strongly disagree

Disagree

Neutral

Agree

Strongly agree

6
Not applicable
(NA)

(continued)
Downloaded from cjs.sagepub.com by Widiastuti Pajarini on April 29, 2012

341

Andrews and Violato

Appendix C (continued)
Based on the MOST RECENT VISIT with the Psychologist
1. The psychologist explained your childs problems to you thoroughly
2. The psychologist adequately explained your childs treatment
choices
3. The psychologist adequately explained how treatment was going to
be monitored
Based on ALL OF YOUR VISITS with this psychologist, how do you
feel about the psychologists attitude and behaviour toward you?
This psychologist:
4. Spends enough time with me
5. Shows interest in my childs problems
6. Answers my questions well
7. Treats me with respect
8. Talks to me about my childs needs
9. Selects appropriate recommendations
10. Informs me about the limits of confidentiality
11. Obtains informed consent for services
12. Informs me of his or her qualifications
13. Explains all test results on my child well
14. Demonstrates good knowledge and skill
15. Arranges appointments with specialists when necessary
General
16. The psychologist follows up with me about my child
17. I would go back to this psychologist
18. I would send a friend to this psychologist

1
1

2
2

3
3

4
4

5
5

6
6

1
1
1
1
1
1
1
1
1
1
1
1

2
2
2
2
2
2
2
2
2
2
2
2

3
3
3
3
3
3
3
3
3
3
3
3

4
4
4
4
4
4
4
4
4
4
4
4

5
5
5
5
5
5
5
5
5
5
5
5

6
6
6
6
6
6
6
6
6
6
6
6

1
1
1

2
2
2

3
3
3

4
4
4

5
5
5

6
6
6

RESPONSE CODE
1
Among the worst

Bottom half

Average

Top half

Among the best

6
Unable to
assess

Appendix D
Psychologists Assessment of Performance
Coworker Questionnaire
Assessed psychologists Name________________________
(continued)
Downloaded from cjs.sagepub.com by Widiastuti Pajarini on April 29, 2012

342

Canadian Journal of School Psychology 25(4)

Appendix D (continued)
Your name _____________________
Interpretation of the rating scale
Rate this psychologist on the following performance statements according to the following guidelines and scale.
This form is used by a variety of psychologists coworkers (e.g., teachers, educational
specialists, therapists)
How well do you know this psychologist (mark one)?
1. Not at all
2. Not well
3. Somewhat
4. Well
5. Very well
How to answer:
Beside each question there is a row of numbers ranging from (1) to (6)
Make your response by circling the number that is closest to the way you want to
respond
Compared to psychologists I know, this one is:
1. Among the worst
2. Bottom half
3. Average
4. Top half
5. Among the best
6. Unable to assess
Example:
Communicates effectively with clients.
If you rate the psychologists performance among the worst, circle
1

6
(continued)

Downloaded from cjs.sagepub.com by Widiastuti Pajarini on April 29, 2012

343

Andrews and Violato

Appendix D (continued)
If you rate the psychologists performance in the Top half, circle
Answer all questions by circling one of the numbers in the right hand
column referring to the response code at the bottom of the page
1

RESPONSE CODE
1
Among the worst

Bottom half

Average

Top half

Among the best

Unable to
assess

1. Communicates effectively with clients

2. Communicates effectively with clients families
3.Verbal communication with other health care professionals is effective
4. Written communication with other health professionals is effective
5. Is courteous to coworkers
6. Respects the personal knowledge and skills of coworkers
7. Makes appropriate use of community resources for psychoeducational
problems
8. Shows compassion to clients and their families
9. Is nonjudgmental of clients and their families
10. Is courteous to clients and their families
11. Accepts responsibility for professional actions
12. Accepts responsibility for client care
13. Collaborates well with coworkers
14. Is available to clients and their families
15. Maintains confidentiality of clients
16. Is available for consultation about mutual clients

1
1
1
1
1
1
1

2
2
2
2
2
2
2

3
3
3
3
3
3
3

4
4
4
4
4
4
4

5
5
5
5
5
5
5

6
6
6
6
6
6
6

1
1
1
1
1
1
1
1
1

2
2
2
2
2
2
2
2
2

3
3
3
3
3
3
3
3
3

4
4
4
4
4
4
4
4
4

5
5
5
5
5
5
5
5
5

6
6
6
6
6
6
6
6
6

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the authorship and/or
publication of this article.

Funding
The author(s) received no financial support for the research and/or authorship of this
article.

Downloaded from cjs.sagepub.com by Widiastuti Pajarini on April 29, 2012

344

Canadian Journal of School Psychology 25(4)

References
Accreditation Council for Graduate Medical Education Outcome Project. (2004). Accreditation Council for Graduate Medical Education web site. Retrieved November 2, 2004, from
http://www.acgme.org/Outcome/
Archer, J., Norcini, J., & Davies, H. A. (2005). Peer review of paediatricians in training using
SPRAT, British Medical Journal, 330, 1251-1253.
Atkins, P. W. B., & Wood, R. E. (2002). Self- versus others ratings as predictors of assessment
center ratings: Validation evidence for 360-degree feedback programs. Personnel Psychology, 55, 871-904.
Bracken, D. W., Timmreck, C. W., & Church, A. H. (Eds.). (2001). The handbook of multisource
feedback: The comprehensive resource for designing and implementing MFS processes. San
Franscisco: Jossey-Bass.
Bill 25, Health Professions (Regulatory Reform) Amendment Act 2008, 4th Session, 38th
Parliament of the Legislative Assembly, British Columbia, Third Reading (2008, May 27).
Church, A. H. (1997). Do you see what I see? An exploration of congruence in ratings from
multiple perspectives. Journal of Applied Social Psychology, 27, 983-1020.
Cone, J. J. (2001). Evaluating outcomes: Empirical tools for effective practice. Washington,
DC: American Psychological Association.
Connors G. P., & Munro, T. W. (2001). 360-degree physician evaluations. Healthcare Executive, 27(5), 58-59.
Falender, C. A., & Shafranske, E. P. (2004). 360-degree Assessment applied to psychology in
training. Manuscript in preparation.
Fantuzzo, J. W., Sisemore, T. A., & Spradlin, W. H. (1983). A competency-based model for
teaching skills in the administration of intelligence tests. Professional Psychology: Research
and Practice, 14, 224-231.
Fidler, H., Lockyer, J., Toews, J., & Violato, C. (1999). Changing physicians practices: The
effect of individual feedback. Academic Medicine, 74, 702-714.
Fletcher, C., & Bailey, C. (2003). Assessing self-awareness: Some issues and methods. Journal
of Managerial Psychology, 18, 395-404.
Griffin, E., Sanders, G., Craven, D., & King, J. (2000). A computerized 360 feedback tool for personal and organizational development in general practice. Health Informatics Journal, 6, 71-80.
Hall, W., Violato, C., Lewkonia, R., Lockyer, J., Fidler, H., & Toews, J. (1999). Assessment of
Physician performance in Alberta: The physician achievement review project. Canadian
Medical Association Journal, 161, 52-57.
Health Professions Act 2009, Alberta Legislature, Alberta Queens Printer.
Kaslow, N. J., Rubin, N. J., Forrest, L., Elman, N. S., Van Horne, B. A., Jacobs, S. C., et al.
(2007). Recognizing, assessing, and intervening with problems of professional competence.
Professional Psychology: Research and Practice, 38, 479-492.
Kaslow, N. J. (2004). Competencies in professional psychology. American Psychologist, 59,
774-781.
Levine, A. M. (2002). Medical professionalism in the new millennium: A physician charter.
Annals of Internal Medicine, 136, 243-246

Downloaded from cjs.sagepub.com by Widiastuti Pajarini on April 29, 2012

Andrews and Violato

345

Lockyer, J., Violato, C., Fidler, H., & Alakija, P. (2009). The assessment of pathologists/Laboratory medicine physicians through a multisource feedback tool. Archives of Pathology and
Laboratory Medicine, 133, 1301-1308.
Lockyer, J., & Clyman, G. S. (2008). Multisource feedback (360-degree Assessment). In E.S.
Holmboe, & R. E. Hawkins (Eds.), Practical guide to the Assessment of clinical competence
(pp 75-85). Philadelphia: Mosby.
Lockyer, J., Violato, C., & Fidler, H. (2008). Assessment of radiology physicians by regulatory
Authority. Radiology, 247, 771-778.
Lockyer, J., Violato, C., & Fidler, H. (2003). Likelihood of change: A study assessing surgeon
use of multi-source feedback data. Teaching, Learning, and Medicine, 15, 168-174.
Lockyer, J., & Violato, C. (2004). An examination of the appropriateness of using a common
peer Assessment instrument to assess physician skills across specialties. Academic Medicine, 79, 5-8.
Maurer, T. J., Mitchell, D. R. D., & Barbeite, F. G. (2002). Predictors of attitudes toward a
360-degree feedback system and involvement in post-feedback management development
activity. Journal of Occupational and Organizational Psychology, 75, 87-107.
Norcini, J. J. (2003). Peer assessment of competence. Medical Education, 37, 539-543.
Rodolfa, E. E. (Ed.). (2005). Competency education-competent professionals-ethical practice
[Special Issue]. Professional Psychology: Research and Practice, 36, 347-375.
Rodolfa, E. R., Bent, R. J., Eisman, E., Nelson, P. D., & Ritchie, P. (2005). A cube model for
Competency development: Implications for psychology educators and regulators. Professional Psychology: Research and Practice, 36, 347-354.
Ramsey, P. G., Wenrich, M. D., Carline, J. D., Inui, T. S., Larson, E., & Logerfo, J. P. (1993).
Use of peer ratings to evaluate physician performance. Journal of American Medical Association, 269, 1655-1660.
Roberts, M., Borden, K., Christiansen, M., & Lopez, S. (2005). Fostering a culture shift: Assessment of competence in the education and careers of professional psychologists. Professional
Psychology: Research and Practice, 36, 355-361.
Sala, F., & Dwight, S. A. (2002). Predicting executive performance with multirater surveys:
Whom you ask makes a difference. Consulting Psychology Journal: Practice and Research,
54, 166-172.
Shaw, B. F., & Dobson, K. S. (1988). Competency judgments in the training and Assessment of
psychotherapists. Journal of Consulting and Clinical Psychology, 56, 666-672.
Societal Needs Working Group Report. (1996). Skills for the New Millennium: CanMEDS
2000 Project. Annals of Royal College of Physicians and Surgeons of Canada, 29, 206-221
Stern, D. T. (Ed.). (2006). Measuring medical professionalism. Oxford, UK: Oxford University
Press.
Violato, C., Worsfold, L., & Polgar, J. M. (2009). Multisource feedback systems for quality
improvement in the health professions: Assessing occupational therapists in practice. Journal of Continuing Education in the Health Professions, 29(2), 111-118.
Violato, C., Lockyer, J., & Fidler, H. (2003). Multisource feedback: A method of assessing surgical practice. British Medical Journal, 326, 546-548.

Downloaded from cjs.sagepub.com by Widiastuti Pajarini on April 29, 2012

346

Canadian Journal of School Psychology 25(4)

Violato, C., Lockyer, J., & Fidler, H. (2008a). Assessment of psychiatrists with multisource
feedback. Canadian Journal of Psychiatry, 53, 525-533
Violato, C., Lockyer, J., & Fidler, H. (2008b). Changes in performance: A 5-year longitudinal
study of participants in a multi-source feedback program. Medical Education, 42, 1007-1013.
Violato, C., Marini, A., Toews, J., Lockyer, J., & Fidler, H. (1997). Feasibility and psychometric
properties of using peers, consulting physicians, co-workers, and patients to assess physicians. Academic Medicine, 72, 82-84.

Bios
Jac J. W. Andrews is a professor in the Division of Applied Psychology at the University of
Calgary. His current research interests are in the areas of childhood psychopathology, psychological assessment, and clinical supervision. He has published many scientific articles in journals
such as Adolescence, Journal of Psychoeducational Assessment, British Journal of Educational
Psychology, and Canadian Journal of Counselling.
Claudio Violato is a professor in the Faculty of Medicine at the University of Calgary. A research
focus is the development and validation of performance measures for health professionals. He
has published many scientific and technical articles and reports in journals such as the Canadian
Journal of Surgery, Educational and Psychological Measurement, Academic Medicine, Medical
Education, Canadian Journal of Psychiatry, and British Medical Journal and Pediatrics.

Downloaded from cjs.sagepub.com by Widiastuti Pajarini on April 29, 2012