112 Canadian Journal o f C o u n s e l l i n g / R e v u e canadienne de counseling/1997, Vol.

31:2

Integrating Evaluation Into Counselling Practice:

Accountability and Evaluation Intertwined
Bryan Hiebert
University of Calgary

Abstract
Evaluation needs to be seen as an integral part of counselling—an activity that is interwoven
with intervention rather than an added extra. This will require developing new ways to gather
evidence attesting to client change that document the processes in which counsellors and
clients engage and the outcomes associated with those processes. It will also necessitate
developing an expanded, collaborative model for evaluation that involves working with stake-
holders to determine what evidence will be appropriate, the evaluation methods and timelines,
and ways of disseminating the evaluation results.

Résumé
O n doit c o n s i d é r e r l'évaluation comme une partie i n t é g r a n t e du counseling—une activité qui
s'entrelace avec l'intervention, p l u t ô t qu'une activité s u p p l é m e n t a i r e . Ceci demande qu'on
é l a b o r e de nouvelles façons de recueillir les preuves d é m o n t r a n t qu'un client a effectué des
changements; ces façons devraient d é c r i r e les processus d'interaction entre les conseillers et les
clients, ainsi que les résultats associés à ces processus. Cela nécessite é g a l e m e n t l'élaboration
d ' u n m o d è l e collaboratif d'évaluation qui soit plus élargi; selon ce m o d è l e , on travaillerait avec
les parties intéressées pour d é t e r m i n e r les preuves a p p r o p r i é e s , ainsi que les m é t h o d e s d'éva-
luation et les é c h é a n c i e r s , et les moyens de diffuser les résultats de l'évaluation.

E v a l u a t i o n is a " h o t t o p i c " i n c o u n s e l l i n g c i r c l e s these days. L a s t year, t h e

Canadian Journal of Counselling d e v o t e d a s p e c i a l issue to e v a l u a t i n g
c a r e e r a n d e m p l o y m e n t c o u n s e l l i n g , t h e Canadian Journal of Education
p u b l i s h e d a s p e c i a l issue d e a l i n g w i t h a c c o u n t a b i l i t y , E R I C / C A S S h o s t e d
a s p e c i a l c o n f e r e n c e o n assessment i n c o u n s e l l i n g , a n d s e v e r a l p r o v i n c e s
o f f i c i a l l y a d o p t e d a n o u t c o m e - b a s e d a p p r o a c h to g u i d a n c e a n d c o u n s e l -
l i n g i n s c h o o l s . T h u s , i n t e r e s t i n c o u n s e l l i n g e v a l u a t i o n s e e m s to b e
increasing.
A t first g l a n c e , it m i g h t s e e m t h a t this i n c r e a s e d i n t e r e s t i n e v a l u a t i o n
p l a c e s t h e c o u n s e l l i n g p r o f e s s i o n i n a n i d e a l p o s i t i o n to d e a l w i t h the
a c c o u n t a b i l i t y c o n c e r n s a n d t h e e m p h a s i s o n results t h a t have b e e n
p r o m p t e d b y fiscal r e s t r a i n t . H o w e v e r , t h e r e is o n e f u n d a m e n t a l p r o b l e m
w i t h the way m o s t c o u n s e l l o r s a d d r e s s e v a l u a t i o n t h a t n e e d s to b e rec-
t i f i e d i f c o u n s e l l i n g is g o i n g to s u c c e s s f u l l y m e e t the a c c o u n t a b i l i t y
c h a l l e n g e . T h e p r o b l e m is t h a t i n m o s t cases, e v a l u a t i o n is t r e a t e d as a n
after t h o u g h t , " b o l t e d o n to the s i d e " ' o f w h a t c o u n s e l l o r s d o , r a t h e r t h a n
a n i n t e g r a l p a r t o f t h e c o u n s e l l i n g e n d e a v o u r . I f c o u n s e l l i n g is to s u r v i v e
i n t o the n e x t c e n t u r y , e v a l u a t i o n w i l l n e e d to b e c o m e m o r e c o m p l e t e l y
i n t e g r a t e d i n t o t h e c o u n s e l l i n g p r o c e s s . F o r this is to h a p p e n , c o u n s e l l o r s
w i l l n e e d to d e v e l o p n e w ways o f c o n c e p t u a l i z i n g e v a l u a t i o n a n d n e w
Integrating Evaluation 113

ways o f g a t h e r i n g e v i d e n c e a t t e s t i n g to the success o f c o u n s e l l i n g . T h i s

p a p e r d e a l s w i t h s o m e o f the issues i n v o l v e d i n m a k i n g t h a t t r a n s i t i o n a n d
offers a n s t i m u l u s f o r c o m m e n c i n g the p r o c e s s o f d e v e l o p i n g n e w ap-
p r o a c h e s f o r e v a l u a t i n g the effectiveness o f w h a t c o u n s e l l o r s a n d c l i e n t s
d o together.

UNBOLTING EVALUATION

I n m o s t m o d e l s o f c o u n s e l l i n g (e.g., Ivey, 1994; M a r t i n & H i e b e r t , 1 9 8 5 ) ,

p r o b l e m solving, ( D ' Z u r i l l a & G o l d f r i e d , 1971), p r o g r a m d e v e l o p m e n t
(e.g., P o s a v a c 8c C a r e y , 1 9 8 9 ) , a n d e d u c a t i o n (e.g., B r i g g s , G u s t a f s o n , &
T i l l m a n , 1 9 9 1 ) , e v a l u a t i o n is d e p i c t e d as o c c u r r i n g at t h e e n d o f a
p r o c e s s . It is t h e last step, h a p p e n i n g after a r e l a t i o n s h i p is d e v e l o p e d ,
s o m e i n t e r v e n t i o n is a t t e m p t e d , a p r o g r a m is i m p l e m e n t e d , o r a p o t e n -
t i a l s o l u t i o n to a p r o b l e m is c h o s e n . T h i s c r e a t e s t h e i m p r e s s i o n t h a t
p r o f e s s i o n a l s d o s o m e t h i n g first, t h e n t h e y see h o w w e l l it w o r k e d .
H o w e v e r , i n reality, p e o p l e t y p i c a l l y b e g i n e v a l u a t i n g a c h o s e n c o u r s e o f
a c t i o n as s o o n as it b e g i n s to h a p p e n . I n fact, m o s t p e o p l e h a v e s o m e i d e a
o f h o w t h e y w i l l j u d g e the success o f a n e n d e a v o u r b e f o r e t h e y e v e r
e m b a r k o n it. T h e y have a n i d e a o f w h e n t h e p r o c e s s is u n f o l d i n g as
e x p e c t e d a n d w h e t h e r the o u t c o m e s t h e y h o p e d f o r a r e h a p p e n i n g . T h e
c r i t e r i a f o r success s o m e t i m e s a r e i m p l i c i t , b u t m o s t o f t e n t h e y exist.
T h e r e f o r e , it is p e c u l i a r to d e s c r i b e p r o f e s s i o n a l i n t e r v e n t i o n as i f e v a l u a -
t i o n o n l y h a p p e n s w h e n it is a l l over. E v a l u a t i o n n e e d s to b e s e e n as
c e n t r a l , r a t h e r t h a n e x t r a n e o u s to the c o u n s e l l i n g e n d e a v o u r .
It a p p e a r s t h a t c o u n s e l l o r s s e l d o m f o r m a l l y e v a l u a t e t h e i r w o r k w i t h
c l i e n t s . C o n g e r , H i e b e r t , a n d H o n g - F a r r e l l (1993) f o u n d t h a t f e w e r t h a n
10% o f c o u n s e l l o r s r e p o r t e d s y s t e m a t i c a l l y e v a l u a t i n g t h e i r w o r k w i t h
c l i e n t s a n d i n s o m e sectors 4 0 % r e p o r t e d never e v a l u a t i n g t h e i r w o r k .
W h e n c o u n s e l l o r s d i d get f e e d b a c k f r o m t h e i r c l i e n t s r e g a r d i n g the
effectiveness o f t h e i r w o r k , it t e n d e d to b e v e r b a l , i n the s e s s i o n w i t h the
c l i e n t , p r e s u m a b l y by a s k i n g the c l i e n t i f t h e y f o u n d t h e s e s s i o n h e l p f u l .
T h i s finding p r o v i d e s a d d i t i o n a l s u p p o r t f o r t h e c o n t e n t i o n t h a t e v a l u a -
t i o n is n o t s e e n as a n i n t e g r a l p a r t o f t h e c o u n s e l l i n g e n d e a v o u r .
F u r t h e r e v i d e n c e t h a t e v a l u a t i o n is s e e n as e x t r a n e o u s to c o u n s e l l i n g
lies i n the way t h a t p r o g r a m e v a l u a t i o n is c o n d u c t e d i n c o u n s e l l i n g
p r o g r a m s a n d a g e n c i e s . T h e u s u a l p r a c t i c e is t h a t a g e n c i e s a r e f u n d e d to
d e l i v e r p a r t i c u l a r p r o g r a m s o r services. A t s o m e p o i n t , a n e v a l u a t i o n is
c o n d u c t e d , w h e r e u s u a l l y a n e x t e r n a l e v a l u a t o r is b r o u g h t i n to e x a m i n e
the o p e r a t i o n a n d pass j u d g m e n t o n the q u a l i t y o f s e r v i c e b e i n g d e l i v -
e r e d . W h e n this a p p r o a c h is t a k e n , it says e x p l i c i t l y t h a t e v a l u a t i o n is
s e p a r a t e f r o m t h e s e r v i c e b e i n g d e l i v e r e d . It also says i m p l i c i t l y , t h a t the
a g e n c y staff a r e n o t c a p a b l e o f e v a l u a t i n g t h e i r o w n w o r k , o r t h a t at-
t e m p t s b y t h e a g e n c y to p r o v i d e e v i d e n c e o f its success w i l l b e b i a s e d o r
untrustworthy.
114 Bryan Hiebert

I see a n a p p r o a c h t h a t is m o r e c o l l a b o r a t i v e a n d p r o a c t i v e as b e i n g
m o r e u s e f u l to t h e agency, m o r e i n v i t i n g f o r the s e r v i c e p r o v i d e r s ,
a n d m o r e informative for the funders. T h i s m o r e collaborative a n d
proactive a p p r o a c h w o u l d involve all vested parties (funders, special
i n t e r e s t g r o u p s , m a n a g e r s , c o u n s e l l o r s , c l i e n t s ) d e c i d i n g t o g e t h e r , at the
outset, o n the m e t h o d o f e v a l u a t i o n , t h e e v i d e n c e t h a t w o u l d d e m o n -
strate success, a n d t h e m a n n e r i n w h i c h t h e e v i d e n c e w o u l d b e g a t h e r e d
(see M c L e a n - S t e r n s & H i e b e r t , 1 9 9 5 ; R i d d l e & H i e b e r t , 1 9 9 5 ) . T h i s
m e a n s a d d r e s s i n g , at t h e t i m e o f i n t e r v e n t i o n p l a n n i n g , h o w c o u n s e l l o r s
a n d c l i e n t s w i l l d e t e r m i n e t h e success o f the i n t e r v e n t i o n . It m e a n s
i n c o r p o r a t i n g i n t o t h e i n t e r v e n t i o n the m e t h o d s t h a t w i l l p r o v i d e evi-
d e n c e o f success. I n fact, it m e a n s t h a t t h e e v a l u a t i o n p r o c e d u r e s b e c o m e
t h e m s e l v e s p a r t o f t h e i n t e r v e n t i o n p a c k a g e , u t i l i z i n g the m o t i v a t i n g
effects o f e x p l i c i t f e e d b a c k to assist c l i e n t s i n t h e i r q u e s t f o r c h a n g e . S u c h
p r o c e d u r e s a r e b e g i n n i n g to b e u s e d i n s o m e a p p r o a c h e s to c o u n s e l l i n g ,
m o s t n o t a b l y i n s o l u t i o n - f o c u s e d a p p r o a c h e s , b u t t h e y still a r e n o t w i d e l y
a c c e p t e d . T h e m o s t t y p i c a l a p p r o a c h s e e m s to b e , " F i r s t we d o it, t h e n we
find o u t h o w w e l l it w o r k e d . " A f o u n d a t i o n t e n e t i n this p a p e r is to a n s w e r
t h e q u e s t i o n " H o w w i l l I k n o w h o w w e l l t h e i n t e r v e n t i o n w o r k e d ? " at t h e
t i m e t h e i n t e r v e n t i o n is b e i n g p l a n n e d .

RE-THINKING ACCEPTABLE EVIDENCE

C o u n s e l l o r s a n d c l i e n t s a r e c o n t i n u o u s l y e v a l u a t i n g w h a t is h a p p e n i n g i n
c o u n s e l l i n g . C o u n s e l l o r s s e e m to b e h a v e i n t e n t i o n a l l y i n t h e i r i n t e r a c -
tions with clients ( M a r t i n & S t e l m a c z o n e k , 1988). T h e y watch for the
effect o f t h e i r a c t i o n s o n c l i e n t s , a n d they adjust w h a t t h e y d o i n c o u n s e l -
l i n g to f a c i l i t a t e r e a c h i n g c o u n s e l l i n g g o a l s . L i k e w i s e , c l i e n t s e v a l u a t e
w h a t is g o i n g o n : t h e y j u d g e t h e c o u n s e l l o r i n t e n t i o n , f o l l o w t h r o u g h o r
n o t d e p e n d i n g o n t h e i r v i e w o f t h e l i k e l i h o o d f o r success, a n d f o r m
a n i m p r e s s i o n o f t h e d e g r e e to w h i c h c o u n s e l l i n g s e e m s to b e w o r k i n g
f o r t h e m . M u c h o f these sorts o f e v a l u a t i o n s h a p p e n i m p l i c i t l y . N e v e r -
the-less, the j u d g m e n t s h a v e g r e a t i m p a c t o n the u l t i m a t e success o f
counselling.
W h a t is n e e d e d is a n e v a l u a t i o n system t h a t has a b r o a d e r d e f i n i t i o n o f
acceptable evidence a n d a m o d e l that p r o m o t e s all players b e i n g i n -
volved i n m a k i n g the decisions about what evidence will be acceptable
a n d w h a t w i l l n o t . I d e l i b e r a t e l y use the w o r d " e v i d e n c e " r a t h e r t h a n
"data," to e m p h a s i z e t h a t the basis o f c o u n s e l l o r a n d c l i e n t p e r c e p t i o n s
o f c o u n s e l l i n g success a r e o f t e n , p e r h a p s e v e n m o s t o f t e n , n o t b a s e d o n
test scores. T h e y a r e b a s e d o n i n f o r m a l e v i d e n c e . M o r e o v e r , it f r e q u e n t l y
is t h e case t h a t s t a n d a r d i z e d test scores d o n o t d e p i c t c h a n g e , e v e n
t h o u g h t e a c h e r s a n d c o u n s e l l o r s r e p o r t s u b s t a n t i a l c h a n g e as h a v i n g
taken place. F o r e x a m p l e , C a m p b e l l (1995) f o u n d that t r a d i t i o n a l p a p e r
a n d p e n c i l m e a s u r e s d i d n o t s h o w m u c h c h a n g e i n s t u d e n t s o c i a l skills,
Integrating Evaluation 115

a n d l i k e l y w o u l d n o t h a v e s u p p o r t e d c o n t i n u i n g h i s p r o g r a m , i n spite o f
very p o s i t i v e r e p o r t s o f t e a c h e r s , c o u n s e l l o r s , a n d t h e s t u d e n t s t h e m -
selves. H o w e v e r , two c h e c k l i s t s t h a t c l o s e l y r e f l e c t e d t h e g o a l s o f the
p r o g r a m (e.g., K e n d a l l & W i l c o x , 1979) b o t h s h o w e d s u b s t a n t i a l c h a n g e ,
l i k e l y b e c a u s e t h e y w e r e d i r e c t l y r e l a t e d to t h e g o a l s o f t h e p r o g r a m .
A n i m p o r t a n t step i n b u i l d i n g a l t e r n a t e a p p r o a c h e s to e v a l u a t i o n is to
b r o a d e n the r a n g e o f a c c e p t a b l e e v i d e n c e to i n c l u d e s u c h i n f o r m a l
m e a s u r e s , t h e o n e s t h a t tap t h e sorts o f o b s e r v a t i o n s t h a t c o u n s e l l o r s a n d
c l i e n t s use to t e l l t h e m s e l v e s h o w successful a n i n t e r v e n t i o n has b e e n .

Types of Informal Evidence

T o e x p a n d t h e b o u n d a r i e s o f a c c e p t a b l e e v i d e n c e to i n c l u d e i n f o r m a l
m e a s u r e s w i l l n o t i n v o l v e " s t a r t i n g f r o m s c r a t c h " to d e v e l o p n e w assess-
m e n t t e c h n i q u e s . I n r e c e n t years, m a n y p r o c e d u r e s have b e e n d e v e l o p e d
t h a t l e n d t h e m s e l v e s to d o c u m e n t i n g c l i e n t c h a n g e (see H i e b e r t , 1996;
Peavy, 1 9 9 6 ) . S o m e o f the m o r e p r o m i n e n t o f t h e m are l i s t e d b e l o w .

Checklists. O f t e n , a n i n d i c a t i o n o f c l i e n t c h a n g e c a n b e o b t a i n e d f r o m a
s i m p l e c h e c k l i s t o n w h i c h t h e c o u n s e l l o r o r c l i e n t has d o c u m e n t e d
c e r t a i n aspects o f c o u n s e l l i n g . F o r e x a m p l e , Sills (1995) d e s c r i b e s a
simple checklist for tracking student academic a n d social behaviour. H e
also p r o v i d e s a u s e f u l way f o r q u a n t i f y i n g t h e results, w h i c h c a n b e
integrated i n t o the i n t e r v e n t i o n that the counsellor, teacher, a n d student
w o r k o u t c o l l a b o r a t i v e l y . H i e b e r t ( 1 9 9 4 b , 1995) a n d R i d d l e a n d H i e b e r t
(1995) p r o v i d e a w i d e array o f s a m p l e c h e c k l i s t s a p p r o p r i a t e f o r t r a c k i n g
a b r o a d r a n g e o f c o u n s e l l i n g v a r i a b l e s s u c h as: a n g e r m a n a g e m e n t skills,
r e l a x a t i o n t r a i n i n g , i n t e r v i e w i n g skills, c o u n s e l l o r a n d c l i e n t i n v o l v e -
m e n t i n the c o u n s e l l i n g p r o c e s s , m o t i v a t i o n , n a t u r e o f service r e c e i v e d ,
a n d referral criteria. S u c h checklists can provide c o n v i n c i n g evidence o f
c o u n s e l l o r , c l i e n t , a n d a g e n c y v a r i a b l e s i n f l u e n c i n g c o u n s e l l i n g success.

The Life Line. T o h e l p c l i e n t s i d e n t i f y a n d r e f l e c t o n i m p o r t a n t past

events w h i c h h a v e i n f l u e n c e d t h e m , G o l d m a n (1992) d e v e l o p e d " T h e
Life L i n e . " Clients draw a h o r i z o n t a l t i m e l i n e o n a b l a n k sheet o f p a p e r
a n d t h e n p l o t o n this t i m e l i n e s i g n i f i c a n t e x p e r i e n c e s , r e l a t i o n s h i p s ,
events, o r a s p i r a t i o n s w h i c h h a v e i n f l u e n c e d t h e i r lives. T h i s p r o v i d e s
c o u n s e l l o r s a n d c l i e n t s w i t h a g r a p h i c m e a n s o f assessing p o t e n t i a l fac-
tors to a d d r e s s i n c o u n s e l l i n g a n d c a n be u s e d to i n d i c a t e shifts i n c l i e n t
a t t r i b u t i o n s d u r i n g c o u n s e l l i n g . It is also p o s s i b l e to q u a n t i f y c l i e n t
c h a n g e b y s i m p l y c o u n t i n g s u c h t h i n g s as t h e n u m b e r o f i t e m s t h a t
c h a n g e across t i m e o r the n u m b e r o f i t e m s t h a t p e r t a i n to a c e r t a i n
t h e m e , e.g., c l i e n t " o w n e r s h i p . "

Portfolios. P o r t f o l i o s have b e e n u s e d f r e q u e n t l y b y w r i t e r s a n d artists to

p r o v i d e s a m p l e s o f t h e i r w o r k . R e c e n t l y , it has b e g u n to b e u s e d i n
c o u n s e l l i n g settings to h e l p c l i e n t s d o c u m e n t c h a n g e s i n t h e i r lives. F o r
116 Bryan Hiebert

e x a m p l e , S o r m u n e n (1994) describes the role p o r t f o l i o s c a n play i n

h e l p i n g d o c u m e n t s t u d e n t s k i l l a c q u i s i t i o n a n d e a s i n g the s c h o o l - t o -
w o r k t r a n s i t i o n . N e l s o n C a n a d a r e c e n t l y has p u b l i s h e d a p o r t f o l i o g u i d e
f o r use i n c a r e e r p l a n n i n g ( C a l g a r y E d u c a t i o n a l P a r t n e r s h i p F o u n d a -
t i o n , 1 9 9 6 ) . F i e l d test results ( H i e b e r t & T a n n e r , 1995) i n d i c a t e t h a t it
h a d a noteworthy i m p a c t o n substantial n u m b e r s o f students, h e l p i n g
t h e m b e m o r e f o c u s e d a n d m o r e a b l e to r e c o g n i z e t h e i r s t r e n g t h s . T h i s
provides a g o o d e x a m p l e o f h o w a single evaluation tool can simul-
t a n e o u s l y serve as a n e v a l u a t i o n v e h i c l e a n d a n i n t e r v e n t i o n to h e l p
p r o m o t e c h a n g e . T o satisfy c o n c e r n s t h a t p o r t f o l i o s are t o o soft f o r
r i g o r o u s e v a l u a t i o n , S t r o e b l e (1993) d e s c r i b e s ways i n w h i c h state as-
s e s s m e n t s t a n d a r d s c a n b e m e t t h r o u g h p o r t f o l i o assessment a n d p r o -
v i d e s s a m p l e assessment c r i t e r i a f o r p o r t f o l i o s . H a y e s a n d K r e t s c h m a n n
(1993) p r o v i d e a u s e f u l a n n o t a t e d b i b l i o g r a p h y f o r t h o s e w i s h i n g m o r e
d e t a i l e d i n f o r m a t i o n o n t h e use o f p o r t f o l i o s i n c o u n s e l l i n g .

Observation forms. F o r m s f o r t r a c k i n g v a r i o u s c l i e n t b e h a v i o u r s have

b e e n a v a i l a b l e f o r m a n y y e a r s (e.g., C a u t e l l a , 1 9 7 7 ) , t y p i c a l l y b e i n g u s e d
to g a t h e r c l i e n t b a s e l i n e i n f o r m a t i o n o r p r o v i d e i n f o r m a l e v i d e n c e o f
client progress. However, observation forms can p r o v i d e useful evidence
o f c l i e n t c h a n g e across t i m e a n d t h e r e f o r e c a n b e i m p o r t a n t e v a l u a t i o n
resources. In early studies o f classroom m a n a g e m e n t , observations o f
s t u d e n t b e h a v i o u r p r o v i d e d d a t a a t t e s t i n g to t h e success o f v a r i o u s
i n t e r v e n t i o n p r o c e d u r e s . H a v i n g d e m o n s t r a t e d success i n o t h e r settings,
s i m i l a r o b s e r v a t i o n p r o c e d u r e s c a n b e u s e d effectively to e v a l u a t e c o u n -
s e l l i n g i n t e r v e n t i o n s . V a r i o u s o b s e r v a t i o n f o r m s have b e e n d e v e l o p e d to
track a variety o f c l i e n t p r e s e n t i n g p r o b l e m s , i n c l u d i n g headaches a n d
o t h e r types o f c h r o n i c p a i n , c o m m u n i c a t i o n skills, j o b s a t i s f a c t i o n , a n d
a n g e r c o n t r o l . R i d d l e a n d H i e b e r t (1995) p r o v i d e s e v e r a l u s e f u l e x a m -
p l e s t h a t c a n serve as a s t a r t i n g p o i n t f o r c o u n s e l l o r s w i s h i n g to use these
procedures with their clients.

Cognitive mapping. T h o u g h t L i s t i n g a n d C o g n i t i v e M a p p i n g are two

recently d e v e l o p e d techniques for p r o v i d i n g a r e c o r d o f a client's think-
i n g pattern a n d h o w a client's thoughts change over time. In T h o u g h t
L i s t i n g , p e o p l e are a s k e d to l i s t a l l t h e t h o u g h t s t h a t c o m e to t h e i r m i n d
i n r e s p o n s e to a q u e s t i o n s u c h as: " W h a t a r e t h e c r i t i c a l e l e m e n t s c o n t r i b -
u t i n g to y o u r p r o b l e m ? , " " W h a t are t h e m o s t i m p o r t a n t e l e m e n t s c o n t r i b -
u t i n g to y o u r i n a b i l i t y to find a j o b ? , " " W h a t w o u l d n e e d to h a p p e n so t h a t
y o u felt i n c o n t r o l o f y o u r l i f e ? " o r " W h a t are the m o s t i m p o r t a n t c h a r a c -
teristics o f a g o o d c o u n s e l l o r ? " T h e p r o c e d u r e c a n b e u s e d to t r a c k
aspects o f a c l i e n t ' s p r o b l e m s i t u a t i o n o r k e y e l e m e n t s o f a c o u n s e l l o r ' s
p r o f e s s i o n a l d e v e l o p m e n t . I n e i t h e r case, the p e r s o n is s i m p l y a s k e d to
list a l l the t h o u g h t s t h a t c o m e to t h e i r m i n d i n r e s p o n s e to t h e p r o b e
q u e s t i o n . I n C o g n i t i v e M a p p i n g , t h e e x e r c i s e is e x t e n d e d b y h a v i n g t h e
Integrating Evaluation 117

c l i e n t t r a n s f e r the results o f t h e t h o u g h t l i s t i n g t o s m a l l post-it n o t e s a n d

t h e n a r r a n g i n g t h e post-it n o t e s o n a s h e e t o f g r a p h p a p e r so t h a t they
p r o v i d e a p i c t u r e o f h o w the c o n c e p t s are r e l a t e d i n the person's think-
i n g . T h e s e p r o c e d u r e s c a n b e u s e d as i n t e r v e n t i o n s , to h e l p c l i e n t s
b e c o m e m o r e f o c u s e d i n c o u n s e l l i n g . T h e y also c a n b e u s e d to e v a l u a t e
c l i e n t c h a n g e i n e i t h e r a q u a n t i t a t i v e (e.g., M a r t i n , S l e m o n , H i e b e r t ,
H a l l b e r g , & C u m m i n g s , 1989) o r a q u a l i t a t i v e (e.g., C u m m i n g s , H a l -
l b e r g , M a r t i n , S l e m o n , 8c H i e b e r t , 1990) m a n n e r . T h e s e p r o c e d u r e s have
b e e n u s e d to t r a c k c l i e n t c h a n g e ( M a r t i n , 1 9 8 5 ) , c h a n g e i n c l i e n t asser-
tive t h i n k i n g ( C o m e a u 8c H i e b e r t , 1 9 9 1 ) , c h a n g e i n c o u n s e l l o r d e v e l o p -
ment (Hiebert & N o o r t , 1988), a n d similarities between novice a n d
e x p e r i e n c e d c o u n s e l l o r s ( C u m m i n g s et a l . , 1 9 9 0 ; M a r t i n et a l . , 1 9 8 9 ) .

Self-monitored data. S e l f - m o n i t o r i n g has d e m o n s t r a t e d success as a n

i n t e r v e n t i o n (see H i e b e r t & F o x , 1 9 8 1 ; K a n f e r , 1 9 9 1 ) , b u t it a l s o c a n
p r o v i d e a u s e f u l s o u r c e o f d a t a a t t e s t i n g to c l i e n t c h a n g e across t i m e .
S e l f - m o n i t o r i n g c a n s p a n a w i d e r a n g e o f c l i e n t f a c t o r s , i n c l u d i n g sub-
j e c t v a r i a b l e s s u c h as p a i n l e v e l , success i n a j o b i n t e r v i e w , m o t i v a -
t i o n l e v e l , n a t u r e o f self-talk, etc. R i d d l e a n d H i e b e r t ( 1 9 9 5 ) p r o v i d e
e x p l i c i t i n s t r u c t i o n s f o r e s t a b l i s h i n g systematic p r o c e d u r e s f o r u s i n g self-
m o n i t o r i n g with clients. S u c h self-monitored d a t a have b e e n d e m o n -
s t r a t e d to b e as t r u s t w o r t h y as t h i r d p a r t y o b s e r v a t i o n d a t a , e v e n w h e n
d e a l i n g with very explicit physiological variables o r low-cognitive func-
t i o n i n g clients (Hiebert, C a r d i n a l , & D u m k a , 1983; H i e b e r t , & M a l c o l m ,
1988; M a l c o l m & H i e b e r t , 1 9 8 6 ) .

Authentic assessment. I n the c l a s s r o o m i n s t r u c t i o n l i t e r a t u r e , t h e r e have

b e e n r e c e n t c r i e s f o r " a u t h e n t i c assessment." T h e m a i n t h r e a d o f the
a r g u m e n t is t h a t t o d a y ' s e d u c a t i o n a l g o a l s a r e t o o v a r i e d to b e a d e q u a t e l y
e v a l u a t e d b y c o n v e n t i o n a l p e n c i l a n d p a p e r tests ( L i n e s , 1 9 9 4 ) . W h a t is
n e e d e d i n s t e a d , is a m e a n s f o r m a k i n g a s s e s s m e n t p r a c t i c e s m o r e c l o s e l y
m a t c h instructional practices, w h e r e assessment b e c o m e s an o n g o i n g
p r o c e s s , i n t e g r a t e d w i t h i n s t r u c t i o n , i n v o l v i n g s t u d e n t s fully, a n d u s i n g
v a r i o u s strategies a n d t o o l s to d e v e l o p a m u l t i d i m e n s i o n a l p i c t u r e o f
l e a r n e r p r o g r e s s ( P o w e l l , 1 9 9 3 ) . M o s t o f t e n , this i n v o l v e s assessing a c t u a l
s t u d e n t p e r f o r m a n c e i n a subject a r e a , r a t h e r t h a n r e l y i n g o n e x a m i n a -
t i o n s . T y p i c a l l y t h e results o f a u t h e n t i c a s s e s s m e n t c a n b e s u m m a r i z e d
n u m e r i c a l l y o r p u t o n a scale to p r o v i d e q u a n t i t a t i v e d a t a a t t e s t i n g to
learner performance.

Performance assessment. P e r f o r m a n c e a s s e s s m e n t p r o v i d e s a g o o d e x a m -
p l e o f h o w a u t h e n t i c i n s t r u c t i o n c a n w o r k i n c o u n s e l l i n g settings. F o r
e x a m p l e , H u t c h i n s o n a n d F r e e m a n (1994) d e s c r i b e a d e t a i l e d p r o -
c e d u r e f o r u s i n g a u t h e n t i c i n s t r u c t i o n to t e a c h i n t e r v i e w i n g skills a n d
p e r f o r m a n c e assessment to d e t e r m i n e c l i e n t m a s t e r y o f skills. U s e d i n
this way, p e r f o r m a n c e assessment p r o v i d e s c o n v i n c i n g e v i d e n c e o f c l i e n t
1 1H Bryan Hiebert

c h a n g e a c r o s s t i m e (cf. H u t c h i n s o n , 1994; H u t c h i n s o n , 1 9 9 6 ) . F o r r e a d -
ers w i s h i n g m o r e i n f o r m a t i o n a b o u t p e r f o r m a n c e assessment, R u d n e r
a n d B o s t o n ( 1 9 9 4 ) p r o v i d e a n e x c e l l e n t r e v i e w a n d a how-to-do-it m a n -
u a l is a v a i l a b l e f r o m t h e E d u c a t i o n a l T e s t i n g S e r v i c e ( 1 9 9 3 ) . M e s s i c k
(1994) o u t l i n e s g u i d e l i n e s for g e n e r a t i n g sources o f evidence w h e n
d e v e l o p i n g a n d u s i n g h i g h q u a l i t y p e r f o r m a n c e assessments.

Summary

I n f o r m a l assessment p r o c e d u r e s , s u c h as t h o s e o u t l i n e d a b o v e , o f f e r
m a n y benefits for counsellors. T h e y c a n p r o v i d e a c o n v e n i e n t alternative
to s t a n d a r d i z e d t e s t i n g , t h e y d o n o t c r e a t e e x t r a w o r k f o r c o u n s e l l o r s ,
a n d they often e n h a n c e the i n t e r v e n t i o n b e i n g used. H o w e v e r , critics
p o i n t o u t t h a t r e l i a b i l i t y a n d v a l i d i t y c h e c k s have b e e n n o t b e e n c o n -
d u c t e d o n m o s t i n f o r m a l p r o c e d u r e s . T h e r e f o r e , i t is u n c e r t a i n h o w
stable t h e m e a s u r e m e n t s a r e a n d h o w a c c u r a t e l y they r e f l e c t the c o n -
struct u n d e r e x a m i n a t i o n . O n t h e o t h e r h a n d , p r o p o n e n t s p o i n t o u t t h a t
these sorts o f e v i d e n c e a r e i n t u i t i v e l y v a l i d b e c a u s e t h e y c o n n e c t d i -
r e c t l y to the c o n s t r u c t u n d e r c o n s i d e r a t i o n , at least i n the m i n d s o f
c l i e n t s . F u r t h e r m o r e , b e c a u s e t h e l e g i t i m a c y o f i n f o r m a l m e a s u r e s is j u s t
b e g i n n i n g to b e a c k n o w l e d g e d , t h e r e has n o t yet b e e n e n o u g h t i m e to
e s t a b l i s h a t r a c k r e c o r d a t t e s t i n g to t h e i r r e l i a b i l i t y a n d validity. A s
p r a c t i t i o n e r s a n d r e s e a r c h e r s b e g i n to u t i l i z e i n f o r m a l m e a s u r e s m o r e
f r e q u e n t l y , t h e m e a s u r e s w i l l b e c o m e m o r e fine-tuned a n d p s y c h o m e t r i c
s u p p o r t f o r t h e i r use w i l l b e g i n to e m e r g e .
M o s t p r o p o n e n t s for n o n - t r a d i t i o n a l e v i d e n c e g a t h e r i n g m e t h o d s are
firm i n t h e i r p r o c l a m a t i o n t h a t i n f o r m a l m e a s u r e m e n t t e c h n i q u e s c a n
be rigorous, especially w h e n used i n c o m b i n a t i o n . F o r e x a m p l e , M o n s o n
a n d M o n s o n ( 1 9 9 3 ) u s e d p o r t f o l i o assessment to r e f l e c t i n d i v i d u a l d e v e l -
o p m e n t a n d p e r f o r m a n c e - b a s e d assessment to r e f l e c t s k i l l mastery. T h e y
also p r o v i d e e x a m p l e s o f h o w e v a l u a t i o n n e e d n o t c r e a t e m o r e w o r k f o r
counsellors o r clients, i n that evaluation becomes an integrated part o f
w h a t is h a p p e n i n g i n c o u n s e l l i n g . T o e v a l u a t e , c o u n s e l l o r s ( o r c l i e n t s )
s i m p l y d o c u m e n t w h a t is t a k i n g p l a c e , f o c u s i n g o n t h e i n f o r m a l e v a l u a -
t i o n p r o c e d u r e s t h a t are i n t e r w o v e n i n t o the c o u n s e l l i n g e n t e r p r i s e ,
r a t h e r t h a n v i e w i n g e v a l u a t i o n as a n a d d - o n at t h e e n d . I n s t e a d o f
t r e a t i n g t h e c o u n s e l l i n g i n t e r a c t i o n as a n i n t e r e s t i n g o b s e r v a t i o n , c o u n -
s e l l o r s c a n t r e a t it as a s o u r c e o f e v i d e n c e a t t e s t i n g to c l i e n t c h a n g e . B u t
i n o r d e r to d o that, c o u n s e l l o r s n e e d to see i n f o r m a l e v i d e n c e as b e i n g
a c c e p t a b l e d e m o n s t r a t i o n s o f c o u n s e l l i n g effectiveness.

EVALUATION A N D ACCOUNTABILITY! A WINNING COMBINATION

S e v e r a l r e c e n t r e v i e w s o f the a c c o u n t a b i l i t y l i t e r a t u r e exist (e.g.,

M c E w e n , 1 9 9 5 ) . T h e r e f o r e , t h e p u r p o s e h e r e is n o t to r e p e a t o r synthe-
Integrating Evaluation 119

size t h e a c c o u n t a b i l i t y l i t e r a t u r e , b u t to h i g h l i g h t s o m e p o i n t s t h a t have a
particular bearing o n evaluating counselling.
Perhaps o n e o f the most centrally i m p o r t a n t points i n b o t h accounta-
b i l i t y a n d e v a l u a t i o n is t h a t t h e r e n e e d s to b e a g r e e m e n t o n the n a t u r e o f
the s e r v i c e b e i n g o f f e r e d , w h a t e v i d e n c e w i l l b e a c c e p t a b l e , a n d w h a t
p e r f o r m a n c e levels w i l l b e c o n s i d e r e d as s u c c e s s f u l . T h i s a g r e e m e n t
n e e d s to i n v o l v e a l l t h e p e o p l e t h a t are p o t e n t i a l l y a f f e c t e d by the
e v a l u a t i o n results, o r w h o h a v e a stake i n results. T h i s w i l l i n c l u d e
funders, special interest groups, clients, significant others, counsellors,
s u p e r v i s o r s , m a n a g e r s , a n d c o o r d i n a t o r s . A l l stake h o l d e r s n e e d to b e
i n v o l v e d i n d e t e r m i n i n g t h e n a t u r e o f s e r v i c e p r o v i d e d , t h e a p p r o a c h to
e v a l u a t i o n , a n d t h e e v i d e n c e t h a t w i l l i n d i c a t e success ( F r e n c h , H i e b e r t ,
& B e z a n s o n , 1994; R i d d l e & H i e b e r t , 1 9 9 5 ) .
A s B r o w n l e e (1995) p o i n t s out, the p r i m e p u r p o s e o f m e a s u r e m e n t
( a n d a c c o u n t a b i l i t y ) is to c o n f i r m t h a t c u s t o m e r s a t i s f a c t i o n o r dissat-
i s f a c t i o n is j u s t i f i e d . T h e r e f o r e , a c c o u n t a b i l i t y n e e d s to b e s e e n as b e g i n -
n i n g w i t h the c o n s e n s u s d e s c r i b e d i n t h e p r e c e d i n g p a r a g r a p h a n d
e n d i n g w i t h m a r k e t i n g the r e s u l t s to t h e v a r i o u s p u b l i c s t h a t a n a g e n c y
serves i n o r d e r t h a t t h e y m i g h t see t h e effectiveness o f t h e a g e n c y
( M c E w e n , 1 9 9 5 ) . H o w e v e r , i n o r d e r to c o n d u c t t h a t m a r k e t i n g suc-
cessfully, c o u n s e l l o r s n e e d to h a v e s o m e t h i n g to m a r k e t ! T h i s i m p l i e s the
n e e d f o r a d r a m a t i c c h a n g e i n t h e type o f e v i d e n c e t h a t is u s e d to s u p p o r t
a c c o u n t a b i l i t y a r g u m e n t s . T h e types o f i n f o r m a l e v i d e n c e o u t l i n e d ear-
l i e r i n this p a p e r w i l l b e c o m e i n c r e a s i n g l y i m p o r t a n t i n d e m o n s t r a t i n g
v a l u e to s t a k e h o l d e r s .
A n e x a m i n a t i o n o f a c c o u n t a b i l i t y p r a c t i c e s to d a t e reveals t h a t
" t h e r e has b e e n a p r e o c c u p a t i o n w i t h a c t i v i t y a n d c o l u m n c o u n t s . "
( M a c D o n a l d , 1993, p . 1 8 1 ) . T o m a k e e v a l u a t i o n a n d a c c o u n t a b i l i t y m o r e
r e l e v a n t , t h e r e n e e d s to b e a n a c c o u n t a b i l i t y f r a m e w o r k i n p l a c e t h a t
a l l o w s c o u n s e l l o r s to o b t a i n f e e d b a c k , adjust t h e i r i n t e r v e n t i o n to m a x i -
m i z e success, a n d a l l o w f o r r e - e n t r y a n d r e f e r r a l o f c l i e n t s w h o s e n e e d s
w e r e n o t m e t i n i t i a l l y . S u c h a f r a m e w o r k w i l l h e l p to p r o v i d e a c o n s t a n t l y
i m p r o v i n g a n d v a l u e d s e r v i c e ( M a c D o n a l d , 1993, p . 1 8 2 ) . T h u s , the
s e p a r a t i o n b e t w e e n f o r m a t i v e a n d s u m m a t i v e e v a l u a t i o n b e g i n s to b l u r ,
for the s a m e type o f e v i d e n c e c a n b e u s e d f o r b o t h . E v a l u a t i o n b e c o m e s
e m b e d d e d i n p r a c t i c e w h i c h c r e a t e s the o p p o r t u n i t y f o r p r a c t i c e to b e
i n f o r m e d a n d sculpted by evaluation (formative considerations). A t
s o m e p o i n t i n t i m e , t h e e v i d e n c e is g a t h e r e d t o g e t h e r a n d p r e s e n t e d i n a
way t h a t s t a k e h o l d e r s c a n u n d e r s t a n d ( s u m m a t i v e c o n s i d e r a t i o n s ) . S u c h
a n a p p r o a c h w i l l g o a l o n g w a y to m a k i n g e v a l u a t i o n a n d a c c o u n t a b i l i t y
m o r e i m p o r t a n t a n d relevant enterprises i n c o u n s e l l i n g endeavours.
A n o t h e r p o i n t o f convergence between the perspective o n evaluation
p r e s e n t e d e a r l i e r i n this p a p e r a n d p e r s p e c t i v e s i n t h e a c c o u n t a b i l i t y
literature centres a r o u n d the i m p o r t a n c e a n d usefulness o f b u i l d i n g
120 Bryan Hiebert

assessment i n t o t h e i n t e r v e n t i o n p r o c e s s . A l t h o u g h assessment p r o c e -
d u r e s c a n be c o m p l i c a t e d a n d take u p c o n s i d e r a b l e t i m e , m a n y t e a c h e r s
see t h e m as g o o d i n s t r u c t i o n . S i m i l a r l y , m a n y c o u n s e l l o r s see h a v i n g
e x p l i c i t assessment p r o c e d u r e s as b e i n g p a r t o f g o o d c o u n s e l l i n g i n t e r a c -
t i o n . M o r e o v e r , assessment a c t i v i t i e s c a n b e w o r t h y i n a n d o f t h e m s e l v e s .
T h a t is to say, t h e i n s i g h t a n d m o t i v a t i o n t h a t c l i e n t s c a n get f r o m
r e l e v a n t assessment a r e w o r t h the t i m e t h a t t h e y take ( E a r l , 1 9 9 5 ) . T h i s
can be particularly i m p o r t a n t i n d e t e r m i n i n g reasonable time expecta-
t i o n s f o r c h a n g e to o c c u r . F o r e x a m p l e , C a m p b e l l (1995) f o u n d t h a t
p a r t i c i p a n t c h a n g e d i d n o t take p l a c e i n a l i n e a r f a s h i o n : l i t t l e c h a n g e
t o o k p l a c e o v e r t h e first 6-9 w e e k s , w h i l e the largest a m o u n t o f c h a n g e
t o o k p l a c e i n w e e k s 9.12. T h i s i l l u s t r a t e s t h e i m p o r t a n c e o f h a v i n g
e v i d e n c e t h a t c l o s e l y m a t c h e s t h e p r o g r a m g o a l s , t h a t tracks t h e p r o g r e s s
o f c l i e n t i n t h e p r o g r a m , a n d y i e l d s s o m e way o f k n o w i n g h o w m u c h t i m e
it takes to m a s t e r t h e skills the p r o g r a m is t r y i n g to t e a c h .

A MODEL FOR INTEGRATION

F i g u r e 1 presents a m o d e l for c o m b i n i n g m u c h o f the f o r g o i n g discus-

s i o n . P a r a m o u n t i n F i g u r e 1 is the i m p o r t a n c e o f a s o u n d p o l i c y to g u i d e
e v a l u a t i o n a n d a c c o u n t a b i l i t y a c t i v i t i e s . T h e p o l i c y n e e d s to o u t l i n e the
e v a l u a t i o n p r a c t i c e s o f t h e a g e n c y a n d d e s c r i b e the p l a c e o f e v a l u a t i o n i n
the a g e n c y m a n d a t e . A d i s c u s s i o n I h a d w i t h a r e c e n t g r a d u a t e o f o u r
c o u n s e l l o r t r a i n i n g p r o g r a m i l l u s t r a t e s t h e i m p o r t a n c e o f this step. T h i s
c o u n s e l l o r w o r k e d i n a n agency w h e r e the m a n a g e r b e l i e v e d that the
p r i m a r y m a n d a t e o f t h e a g e n c y was to s e r v i c e c l i e n t s a n d t h e r e f o r e ,
c o u n s e l l o r s w e r e n o t to s p e n d t i m e e v a l u a t i n g t h e i r w o r k . Q u i t e p r e d i c t a -
bly, this a g e n c y was d o w n s i z e d to t h e p o i n t w h e r e it is n o l o n g e r a b l e to
serve a n y c l i e n t s . H o w e v e r , c o u n s e l l o r s i n this a g e n c y w h o f o u n d ways to
"get a r o u n d " t h e i r m a n g e r ' s e d i c t a n d s p e n d s o m e t i m e e v a l u a t i n g t h e i r
w o r k w i t h c l i e n t s , e n d e d u p finding n e w p o s i t i o n s i n o t h e r p a r t s o f t h e
o r g a n i z a t i o n w h e r e d o w n s i z i n g was less severe. T h e m o r a l o f this story f o r
m e is t h a t i n s i g h t f u l m a n g e r s w i l l r e c o g n i z e the i m p o r t a n c e o f e v a l u a t i o n
i n e x t e n d i n g the life e x p e c t a n c y o f t h e i r agency a n d c o u n s e l l o r s w h o
w a n t to k e e p e m p l o y m e n t o p t i o n s f l e x i b l e w i l l find ways o f w o r k i n g
evaluation activities into their c l i e n t interventions.
A w o r k a b l e e v a l u a t i o n p o l i c y n e e d s to a d d r e s s several key factors. It
n e e d s to c o n t a i n a s t a t e m e n t o f w h o t h e p o t e n t i a l s t a k e h o l d e r s m i g h t b e ,
how other stakeholders will be identified, a n d h o w consultation a n d
c o l l a b o r a t i o n w i l l take p l a c e . It n e e d s to d e s c r i b e the r o l e s a n d r e s p o n -
s i b i l i t i e s o f a l l p a r t i e s t h a t have a r o l e to p l a y i n e v a l u a t i o n , i n c l u d i n g
c o u n s e l l o r s , s u p e r v i s o r s , m a n a g e r s , o f f i c e staff, c l i e n t s , f u n d e r s , a n d so
o n . It n e e d s to o u t l i n e t h e types o f e v i d e n c e t h a t w i l l be c o n s i d e r e d as
legitimate i n d i c a t i o n s o f w o r t h a n d the time frames for each e l e m e n t i n
evaluation practices.
Integrating Evaluation 121

Time Nature o f Roles &

Stakeholders
frames evidence responsibilities

Evaluation Policy

I n t e r v e n t i o n factors A g e n c y factors Communication

Process O u t c o m e Input Service

Audiences Messages
evidence evidence evidence evidence

FIGURE 1
An Integrated Counselling Evaluation Model

F i g u r e 1 shows t h a t e v a l u a t i o n p r a c t i c e s c a n be g r o u p e d i n t o t h r e e
m a i n c l u s t e r s : i n t e r v e n t i o n o r p r o g r a m factors, a g e n c y factors, a n d c o m -
m u n i c a t i o n factors.

Intervention Factors

I have w r i t t e n at l e n g t h o n t h e factors t h a t are i m p o r t a n t to a d d r e s s w h e n

e v a l u a t i n g c o u n s e l l i n g i n t e r v e n t i o n ( H i e b e r t 1984, 1 9 8 9 , 1 9 9 4 a , 1 9 9 4 b ,
1 9 9 5 , 1 9 9 6 ; R i d d l e & H i e b e r t , 1 9 9 5 ) . T h e r e f o r e , the d i s c u s s i o n h e r e w i l l
b e b r i e f . T h e g e n e r a l g o a l w h e n e v a l u a t i n g i n t e r v e n t i o n factors is to
d e m o n s t r a t e t h a t c o u n s e l l i n g is r e s p o n s i b l e f o r c l i e n t c h a n g e . T h e r e -
f o r e , e v i d e n c e n e e d s to b e o b t a i n e d o n b o t h p r o c e s s a n d o u t c o m e
v a r i a b l e s . R e g a r d i n g t h e o u t c o m e s , it is i m p o r t a n t to d o c u m e n t w h a t
c h a n g e has o c c u r r e d i n t h e c l i e n t p r e s e n t i n g p r o b l e m (e.g., a r e d u c t i o n
i n v i o l e n c e ) a n d also t h e c h a n g e i n c l i e n t s k i l l , k n o w l e d g e , a n d a t t i t u d e
(e.g., a n g e r m a n a g e m e n t s k i l l , k n o w l e d g e a n d s k i l l a b o u t h o w to n e g o t i -
ate c o m p r o m i s e s , a n d a m o r e t o l e r a n t a t t i t u d e ) t h a t is r e s p o n s i b l e f o r
t h a t c h a n g e . W i t h r e s p e c t to i n t e r v e n t i o n p r o c e s s , t h e r e n e e d s to b e
e v i d e n c e t h a t t h e c o u n s e l l i n g p r o c e s s ( a n d n o t o t h e r factors) is r e s p o n -
s i b l e f o r t h o s e c h a n g e s . T h i s c a n b e d o n e by d o c u m e n t i n g t h a t the
c o u n s e l l o r has b e e n f o l l o w i n g a n a c c e p t e d i n t e r v e n t i o n p r o c e d u r e (e.g.,
a n a c c e p t a b l e a n g e r m a n a g e m e n t p r o t o c o l ) a n d t h a t t h e c l i e n t has b e e n
e n g a g e d i n t h e p r o c e s s (e.g., a t t e n d i n g sessions, a t t e m p t i n g i n - s e s s i o n
s k i l l p r a c t i c e , c o m p l e t i n g h o m e w o r k ) . T h e r o l e i n this p r o c e s s o f the
i n f o r m a l evaluation p r o c e d u r e s discussed earlier s h o u l d be obvious.
W h e n these types o f e v i d e n c e are c o m p i l e d , it is easy to m a k e t h e l i n k
between i n t e r v e n t i o n a n d o u t c o m e a n d thereby, s u p p o r t the c l a i m that
c l i e n t c h a n g e t o o k p l a c e as a r e s u l t o f t h e c o u n s e l l i n g i n t e r v e n t i o n .
122 Bryan Hiebert

Agency Factors

A g e n c y f a c t o r s d o n o t a d d r e s s c o u n s e l l i n g o u t c o m e s p e r se, h o w e v e r ,
t h e y h a v e a d i r e c t b e a r i n g o n the success o f a n agency. I n p u t factors r e f e r
to t h e r e s o u r c e s t h a t a n a g e n c y c a n b r i n g to b e a r o n the c o u n s e l l i n g
e n t e r p r i s e . I f a n a g e n c y is u n d e r r e s o u r c e d , o r i f its r e s o u r c e s a r e dis-
p e r s e d i n t o o m a n y d i r e c t i o n s , it w i l l be d i f f i c u l t to a c h i e v e a c c e p t a b l e
results. S e r v i c e f a c t o r s r e f e r to the g e n e r a l factors t h a t affect c l i e n t
p e r c e p t i o n o f q u a l i t y o f s e r v i c e . T h i s m i g h t i n c l u d e s u c h i t e m s as: h o w
m a n y t i m e s t h e t e l e p h o n e r i n g s b e f o r e it is a n s w e r e d , h o w l o n g c l i e n t s
h a v e to w a i t f o r a n a p p o i n t m e n t , h o w m u c h r e s p e c t is s h o w n by a l l a g e n c y
staff, a n d t h e e x t e n t to w h i c h staff m e m b e r s f o l l o w t h r o u g h o n t h e i r
p r o m i s e s to r e t u r n t e l e p h o n e calls o r a d v o c a t e w i t h t h i r d p a r t i e s .

Communication Factors

I n o r d e r to b e a n effective a l l y o f a c o u n s e l l i n g agency, e v a l u a t i o n r e s u l t s
n e e d to be c o m m u n i c a t e d to e a c h s t a k e h o l d e r g r o u p , r e c o g n i z i n g t h a t
s t a k e h o l d e r s d o n o t a l l n e e d o r w a n t the s a m e i n f o r m a t i o n , n o r d o t h e y
n e e d o r w a n t it p r e s e n t e d i n the s a m e way. T h u s , it is i m p o r t a n t to m a k e
s u r e t h a t the m e s s a g e is t a i l o r e d to t h e u n i q u e i n f o r m a t i o n n e e d s o f e a c h
s t a k e h o l d e r . T h i s m i g h t i n v o l v e g a t h e r i n g d i f f e r e n t types o f e v i d e n c e f o r
different stakeholders, for e x a m p l e , some stakeholders prefer n u m b e r s
w h i l e o t h e r s find t e s t i m o n i a l s m o r e u s e f u l . It m i g h t a l s o i n v o l v e u s i n g the
s a m e e v i d e n c e a n d p r e s e n t i n g it i n d i f f e r e n t ways, p i c t o r i a l l y f o r s o m e , i n
t a b u l a r f o r m f o r o t h e r s . A s p a r t o f t h e i n i t i a l c o l l a b o r a t i o n , it is i m p o r t a n t
for the a g e n c y a n d its s t a k e h o l d e r s to a g r e e o n w h a t type o f e v i d e n c e w i l l
b e c o l l e c t e d , a n d h o w it w i l l b e p r e s e n t e d to the s t a k e h o l d e r g r o u p s .

Summary

I n p r a c t i c e , a d y n a m i c i n t e r - r e l a t i o n s h i p exists b e t w e e n a l l the v a r i a b l e s
d e p i c t e d i n F i g u r e 1. A l t h o u g h a g e n c y p o l i c y is d e p i c t e d as t h e d r i v i n g
force for e v a l u a t i o n p r a c t i c e s , t h e p o l i c i e s t h e m s e l v e s a r e i n f o r m e d
by the n a t u r e o f t h e i n t e r v e n t i o n s i m p l e m e n t e d , t h e r e s o u r c e s o f t h e
agency, a n d t h e s t a k e h o l d e r s , a l l o f w h i c h r e f l e c t the m a n d a t e o f the
agency a n d the clients b e i n g served. E v e n the s u b c o m p o n e n t s i n the
F i g u r e a r e d y n a m i c a l l y c o n n e c t e d . M o s t c o u n s e l l o r s a r e u s e d to t h i n k i n g
a b o u t t h e p r o c e s s e s i n v o l v e d i n d e l i v e r i n g a n i n t e r v e n t i o n , b u t to d e m o n -
strate w o r t h , t h o s e p r o c e s s m u s t r e s u l t i n i d e n t i f i a b l e o u t c o m e s . T h e
o u t c o m e s b e i n g a t t e m p t e d , i n t u r n , h a v e a d e t e r m i n i n g effect o n t h e
p r o c e s s e n g a g e d i n . C o u n s e l l i n g p r o c e s s a n d o u t c o m e n e e d to b e s e e n as
d y n a m i c a l l y a n d r e c i p r o c a l l y c o n n e c t e d . F u r t h e r m o r e , i n o r d e r for des-
i g n a t e d i n t e r v e n t i o n s to b e effective, a n a g e n c y m u s t h a v e a d e q u a t e
r e s o u r c e s . T h e n a t u r e o f t h e r e s o u r c e s n e e d e d is d e t e r m i n e d , i n p a r t , b y
t h e types o f i n t e r v e n t i o n s b e i n g o f f e r e d — t h e success o f t h e i n t e r v e n -
Integrating Evaluation 123

t i o n s is a f f e c t e d by the r e s o u r c e s t h a t a n a g e n c y has at its d i s p o s a l . It is

i m p o r t a n t t h a t a g e n c i e s a r e n o t e x p e c t e d to " d e l i v e r the m o o n " w h e n
they are o n l y g i v e n t h e r e s o u r c e s to get to "the h o u s e n e x t d o o r " ! F i n a l l y ,
the m o s t successful agency, d e l i v e r i n g t h e m o s t effective i n t e r v e n t i o n s , is
l i k e l y to b e i n t r o u b l e i f t h e r e l e v a n t p e o p l e d o n o t k n o w a b o u t the
success. T h e r e f o r e , c o m m u n i c a t i n g e v a l u a t i o n results is a n i m p o r t a n t
c o m p o n e n t o f the evaluation process. F u r t h e r m o r e , because different
p e o p l e n e e d , a n d c a n r e l a t e to, d i f f e r e n t types o f messages, b o t h the
a u d i e n c e a n d the n a t u r e o f the message a r e i m p o r t a n t c o n s i d e r a t i o n s .
A l l o f these i n g r e d i e n t s are i m p o r t a n t w h e n d e s i g n i n g a n d c a r r y i n g o u t
an evaluation plan.

CONCLUSION

W e l i v e i n c h a l l e n g i n g t i m e s , m a d e e v e n m o r e c h a l l e n g i n g b y t h e par-
a n o i a t h a t s e r v i c e p r o v i d e r s d e m o n s t r a t e a r o u n d e v a l u a t i o n a n d ac-
c o u n t a b i l i t y c o n c e r n s . T h i s is n o t a c r i t i c i s m o f c o u n s e l l o r s ( o r o t h e r
service p r o v i d e r s ) f o r t h e y a r e p r o d u c t s o f t h e t r a i n i n g p r o g r a m s t h e y
c o m p l e t e d — t r a i n i n g p r o g r a m s that for the most part place little e m -
p h a s i s o n e v a l u a t i o n . M o s t t r a i n i n g p r o g r a m s e m p h a s i z e p r o c e s s as b e i n g
the c e n t r a l c o m p o n e n t i n c o u n s e l l i n g . L i t t l e e m p h a s i s is p l a c e d o n
o u t c o m e , a n d l i t t l e e m p h a s i s is p l a c e d o n e v a l u a t i o n . B y c o n t r a s t , o n e o f
the m a i n t h e m e s u n d e r l y i n g this p a p e r has b e e n t h a t c o u n s e l l i n g n e e d s
to b e s e e n as a p a r t n e r s h i p b e t w e e n p r o c e s s a n d o u t c o m e . O n e i m p o r -
tant f a c t o r t h a t s e p a r a t e s c o u n s e l l i n g f r o m t a l k i n g to a f r i e n d is the
e x p e c t a t i o n f o r c h a n g e o n t h e p a r t o f the c l i e n t . V i e w i n g c l i e n t c h a n g e as
a n i n t e g r a l p a r t o f the c o u n s e l l i n g e n d e a v o u r p r o v i d e s a c o n t e x t f o r
e m b e d d i n g evaluation i n counselling intervention a n d provides an im-
portant f o u n d a t i o n for addressing accountability concerns.
T h e v i e w t h a t c o u n s e l l i n g is c o m p o s e d o f B O T H p r o c e s s a n d o u t c o m e ,
a n d t h a t B O T H p r o c e s s a n d o u t c o m e n e e d to b e p r e s e n t i f o r d e r f o r
c o u n s e l l i n g to b e c o n s i d e r e d s u c c e s s f u l , is c o n s i d e r e d as r a d i c a l b y m a n y
o f m y c o l l e a g u e s . H o w e v e r , it is a n i m p o r t a n t p e r s p e c t i v e to c o n s i d e r . I f
c o u n s e l l i n g a g e n c i e s a r e g o i n g to d e m o n s t r a t e success to t h e i r stake-
h o l d e r s , t h e y n e e d to h a v e s o m e t h i n g t a n g i b l e to show. It w i l l b e d i f f i c u l t
to o b t a i n t a n g i b l e e v i d e n c e u n l e s s t h e o u t c o m e s o f c o u n s e l l i n g are
c o n s i d e r e d co-equal w i t h the processes.
A m a i n t h e m e i n this p a p e r has b e e n t h a t the r o l e o f e v a l u a t i o n i n
c o u n s e l l i n g n e e d s to b e i n t e g r a t e d w i t h i n t e r v e n t i o n , r a t h e r t h a n " b o l t e d
o n t o t h e s i d e " o f t h e i n t e r v e n t i o n p r o c e s s . C o u n s e l l o r s n e e d to h a v e the
m i n d - s e t t h a t w o u l d m a k e it u n t h i n k a b l e to e v e n c o n t e m p l a t e i n t e r v e n -
t i o n w i t h o u t h a v i n g a c l e a r i d e a at t h e outset, h o w t h e success o f the
i n t e r v e n t i o n w o u l d be d e t e r m i n e d .
Years a g o , C a r l R o g e r s a d v o c a t e d t h a t c l i e n t c o n c e r n s s h o u l d b e c e n -
t r a l i n t h e c o u n s e l i n g p r o c e s s . A f t e r m a n y years, a n d c o u n t l e s s r e m i n d -
124 Bryan Hiebert

ers, c o u n s e l l o r s a r e b e g i n n i n g t o u n d e r s t a n d t h e i m p o r t a n c e o f t h a t
a d m o n i t i o n . W e a r e n o w at a s i m i l a r c r o s s - r o a d s r e g a r d i n g t h e r o l e o f
e v a l u a t i o n i n c o u n s e l l i n g . E v a l u a t i o n n e e d s t o b e s e e n as p a r t a n d p a r c e l
o f t h e c o u n s e l l i n g e n d e a v o u r . B e i n g a b l e t o see t h e r e s u l t s , a t a n g i b l e
i n d i c a t o r o f s u c c e s s , serves as i m p o r t a n t i n c e n t i v e f o r c l i e n t s ( a n d l i k e l y
c o u n s e l l o r s a l s o ) t o w o r k e v e n h a r d e r to i n c r e m e n t c h a n g e i n t h e i r l i v e s .
W h e n c o u n s e l l o r s h a v e w i t n e s s e d t h e p o w e r f u l effect t h a t k n o w l e d g e o f
results has p l a y e d i n clients' c h a n g e p r o g r a m s , they have b e c o m e s t r o n g
advocates o f i n t e g r a t i n g evaluation i n t o c l i e n t i n t e r v e n t i o n p r o g r a m s . In
s u c h cases, p r o c e s s a n d o u t c o m e h a v e b e c o m e p a r t n e r s i n t h e client
c h a n g e p r o c e s s . V i e w i n g p r o c e s s a n d o u t c o m e s as p a r t n e r s i n t h e c o u n -
s e l l i n g e n d e a v o u r , a n d m a k i n g s u r e t h a t e v a l u a t i o n is f u l l y i n t e g r a t e d
i n t o t h e c o u n s e l l i n g p r o c e s s , a r e i m p o r t a n t c o n s i d e r a t i o n s i f w e a r e to
increase the l i k e l i h o o d o f c o u n s e l l i n g surviving into the 2 1 " century.

Note
1
I want to thank Dr. Nancy Hutchinson, Queen's University, for initially sharing this metaphor
with me. I think it aptly illustrates many current approaches to evaluation and also paints a
vision for how things should be different.

References
Briggs, L . J., Gustafson, K . L . , & T i l l m a n , M . H . (1991). Instructional design: Principles and
applications. Englewood Cliffs, NJ: Educational Technology Publications.
Brownlee, W. T. (1995). Accountability initiatives: Necessary or contrived? Canadian Journal of
Education, 20, 80-84.
Calgary Educational Partnership Foundation. (1996). Employability skills: Creating my future.
Scarborough, O N : Nelson Canada.
Campbell, D. S. (1995). Evaluating an intervention for at-risk adolescents: T h e case o f The
BreakAway Company. Canadian Journal of Education, 20, 437-54.
Cautella.J. R. (1977). Behavior analysis forms for clinical intervention. Champaign, IL: Research
Press.
Conger, D . S., Hiebert, B., & Hong-Farrell, E . (1993). Career and employment counsellingin Canada.
Ottawa, O N : Canadian Labour Force Development Board.
Comeau, N . , & Hiebert, B . (1991). Conceptualizations of Assertiveness Trainees. Canadian
Journal of Counselling, 25, 56-72.
Cummings, A . L . , Hallberg, E . T., Martin, J . , Slemon, A . , & Hiebert, B. (1990). Implications of
counselor conceptualizations for counselor education. Counselor Education and Supervision,
30, 120-34.
D'Zurilla, T. J., & Goldfried, M . R. (1971). Problem solving and behavior modification./ournaiqf
Abnormal Psychology, 78, 107-26.
Earl, L . M . (1995). Assessment and accountability in education in Ontario. Canadian Journal of
Education, 20, 45-55.
Educational Testing Service. (1993). Performance assessment sampler: A workbook. Princeton, NJ:
Policy Information Center, Educational Testing Service. E R I C Document Reproduction
Service N o . ED361371.
French, F., Hiebert, B., & Bezanson, L . (1994). Issues and solutions for evaluating career
development. Canadian Journal of Counselling, 28, 261-69.
G o l d m a n , L . (1992). Qualitative assessment: A n approach for counselors. Journal of Counseling
&f Development, 70,616-31.
Integrating Evaluation 125

Hayes, B., & Kretschmann, K . J . (1993). Portfolio assessment: An annotated bibliography. Madison,
WI: Wisconsin State Board of Vocational, Technical, and A d u l t Education. E R I C / C A S S
document number ED362731.
Hiebert, B. (1984). Counsellor effectiveness: A n instructional approach. Personnel and Guidance
Journal, 62, 597-601.
Hiebert, B. (1989). Meeting the accountability challenge. In B. O ' B y m e , (Ed.). Natcon 15
(pp. 239-55). Toronto, O N : Ontario College Counsellor's Association.
Hiebert, B . (1994a). A framework for quality control, accountability, and evaluation: Being
clear about the legitimate outcomes of career counselling. Canadian Journal of Counselling,
28, 334-45.
Hiebert, B. (1994b). Moving to the Future: Outcome Based School Counselling in Alberta Schools.
E d m o n t o n , A B : Alberta Education.
Hiebert, B . (1995). Evaluating career counselling: Proactive accountability. In M . Van Norman
(Ed.). Natcon 21 (pp. 297-305). Toronto, O N : OISE Press.
Hiebert, B. (1996). Using Informal Methods T o Assess Client Change. Guidance, and Counselling,
//(4),3-12.
Hiebert, B . , Cardinal, J., & D u m k a , L . (1983). Self-monitored indices of relaxation. Canadian
Counsellor, 18, 47-48.
Hiebert, B. A . , & Fox, E . E. (1981). The reactive effects of self-monitoring anxiety. Journal of
Counselling Psychology, 28, 187-93.
Hiebert, B., Sc Malcolm, D . (1988). Cognitive strategies for mentally handicapped clients. In B.
O'Byrne, (Ed.). Natcon-14 (pp. 289-98). Toronto, O n : Ontario College Counsellor's Associa-
tion. (Reprinted in Resources InHducalion, June 1989, E R I C Document Reproduction Service
N o . E d 302800.)
Hiebert, B., & Noort, D . (1988). Changes in the conceptualizations and skills of counselling practicum
students. Paper presented to the annual meeting of the American Educational Research
Association. (ERIC Document Reproduction Service N o . E d 296267)
Hiebert, B., & Tanner, G . (1995). Student employability skills portfolio project: F.valuation report.
Report presented to the Calgary Educational Partnership Foundation, Calgary, A B .
Hutchinson, N . L . (1994). Evaluating career development in school-based programs; Perform-
ance assessments. Canadian Journal of Counselling, 28, 326-33.
Hutchinson, N . L . (1996). Using performance assessments to evaluate career development
programs. Guidance and Counselling, 11(3), 3-7.
Hutchinson, N . L . , & Freeman, J . G . (1994). Pathways: Succeeding with the interview. Scarborough,
O N : Nelson Canada.
Ivey, A . ( 1994). Intentional interviewing and counseling (3rd ed.). Pacific Grove, C A : B r o o k s / C o l e .
Kanfer, F. H . (1991). F. H . Kanfer & A . P. Goldstein. Helping people change: A textbook of methods
(4th ed.). New York: Pergamon.
Kendall, P. C , & Wilcox, L . (1979). Self-control in children: Development of a rating scale.
Journal of Consulting and Clinical Psychology, 47, 1020-29.
Lines, C . (1994). Authentic assessment at the middle level. Middle School Journal, 25(4), 39-41.
MacDonald, D . (1993). Counselor training in Canada: A n Alberta approach. Journal of Employ-
ment counselling, 30, 174-84.
Malcolm, D . , & Hiebert, B. (1986). Cognitive Stress-Inoculation training for anger outbursts
with a 30 year o l d mentally retarded residential patient: A case study. British Columbia Journal
of Spedai Education, 10, 139-46.
M a r t i n , J . (1985). Measuring client's cognitive competence in research on counselling. Journal
of Counseling and Development, 63, 556-60.
Martin, J., & Hiebert, B. (1985). Instructional counselling: A method for counselors. Pittsburgh, PA:
University of Pittsburgh Press.
Martin, J., Slemon, A . , Hiebert, B., Hallberg, E. T., & Cummings, A . L . (1989). Conceptualiza-
tions of novice and experienced counselors. Journal of Counseling Psychology, 36, 395-400.
Martin, J . , & Stelmaczonek, K . (1988). Participants' identification and recall of important
events in counseling. Journal of Counseling Psychology, 35, 385-90.
126 Bryan Hiebert

McEwen, N . ( 1995). Accountability in education in Canada. Canadian Journal of Education, 20,

1-17.
McLean-Steams, C , Sc Hiebert, B. ( 1995). Building healthy schools: A final report from Pannen for
Healthy Living. Calgary, A B : Calgary Board of Education.
Messick, S. (1994). T h e interplay of evidence and consequences in the validation of perform-
ance assessments. Educational Researcher, 23(2), 13-23.
Monson, M . P., & M o n s o n , R . J . (1993). Exploring alternatives in student assessment: Shifting
the focus to student learning in the middle school. Middle School Journal, 25(2), 46-50.
Peavy, R. V. ( 1996). Constructivist career counselling and assessment. Cuidance and Counselling,
/7(3),8-14.
Posavac, E . J , & Carey, R. G . (1989). Program evaluation: Methods and case studies. Englewood
Cliffs, NJ: Prentice H a l l .
Powell, J . C. (1993). What does it mean to have authentic assessment? Middle .School Journal,
25(2), 36-42.
Riddle, D., & Hiebert, B. (T995). Demonstrating value: A career services evaluation workbook. Ottawa,
O N : Canadian Guidance and Counselling Foundation.
Rudner, L . M . , & Boston, C . (1994). Performance assessment. ERIC Review, 3(1), 2-12.
Sills, A . C. (1995) Checklist for change. The School Counselor. 43. 85-88.
Sormunen, C. (1994). Portfolios: A n assessment tool for school-to-work transition. Business
Education Forum, 48(4) 8-10.
Stroeble, E. J. (1993). Kentucky student portfolios: Expectations of success. Equity and Excellence
in Education, 26(3), 54-59.

About the Author

Bryan Hiebert is a Full Professor and Chair of the Professional Practice in the Department of
Educational Psychology at the University of Calgary. H e has published extensively in the areas
of career counselling, multi-faceted approaches to stress control, ways to enhance the counsel-
lor training process, and evaluating counselling effects. In 1994, Dr. Hiebert was co-chair
person of the first Canadian National Symposium on Evaluation in Career and Employment
Counselling, and subsequently he co-edited a special issue of the Canadian Journal of Counsel-
ling devoted to that topic.

Address correspondence to: Dr. Bryan Hiebert, Department of Educational Psychology, Univer-
sity of Calgary, Calgary, Alberta, T 2 N 1N4.