Active Qualitative Evaluation

Active Qualitative Evaluation
Core Elements And Procedures

HERMAN KUI PERS
Royal Netherlands Military Academy, Breda and Technical University, Eindhoven
RUDY RI CHARDS ON
Royal Netherlands Military Academy, Breda
This article discusses active qualitative evaluation and modern sociotechnical
design as parts of an approach for the evaluation of organizational change.
Intervention in interactions is the most important similarity between these
parts when considering the evaluation of organizational change.The two
parts are discussed in terms of their theoretical core elements, practical
consequences for research and dilemmas. Because, in practice, intervening in
a changing organization is always accompanied by some important
paradoxes, the different parts of the approach are presented as
supplementary in practical evaluation of organizational change.The leitmotif
on the basis of the case of the Royal Netherlands Military Academy here is
that a paradoxical situation such as intervening in a changing organization
needs a paradoxical intervention strategy.
Introduction
In this article we would like to highlight some starting-points and procedures used
during evaluation research at the Royal Netherlands Military Academy
(RNLMA). In particular, we would like to focus on a central aspect of our evalu-
ation, the combination of different procedures combining various pre-existing
methodological and theoretical orientations. This led to an evaluation approach
and corresponding procedures: an active form of qualitative evaluation, guided by
sociotechnical design-theory. We discuss the approach and its procedures as
follows: rst, we elaborate on our approach by describing some core elements of
the three parts, viz. qualitative evaluation, action research and sociotechnical
design. Then, we highlight some differences and similarities between the three
parts of the approach. Our next step is to elucidate the procedures involved as
used during a study we are conducting at the RNLMA. Finally, we discuss some
dilemmas and paradoxes that derive from our approach and its procedures in an
active research setting.
Evaluation
Copyright 1999
SAGE Publications (London,
Thousand Oaks and New Delhi)
[13563890 (199901)5:1; 6179; 008196]
Vol 5(1): 6179
61
05kuipers(ds) 11/2/99 3:45 pm Page 61
Since the changes in the international political and societal situation, the
Dutch Armed Forces have reorganized and changed rapidly (Soeters and
Richardson, 1996; Richardson et al., 1996). Organizational units in the Armed
Forces, Army as well as Air Force and Navy, are being transformed into so-
called Result Responsible Units (RRU): smaller and more exible units,
managed in a more economic, business-like and performance-based manner.
These units are in the process of formulating Mission Statements, introducing
contract management to arrange the vertical and horizontal relations within the
organization, decentralizing the personnel function and delegating responsi-
bilities. This process of change was launched at the strategic level of the Dutch
Armed Forces by the introduction of the policy of Improvement of Conduct of
Business (Richardson et al., 1996). The most important features of this policy
are similar to the aspects of New Public Management as described by Pollitt
(1995). In practice, these changes are not without problems (Soeters and
Richardson, 1996). One of the major problems is that strategic concepts are not
taken for granted at the operational level. At this level, strategic concepts
undergo a process of transformation and modication. The most difcult ques-
tions at the operational level are:
how to shape strategic concepts in an organization, given the organizational
context?
what are the consequences of this shaping process for organizational
change? and,
how can we organize support for organizational change?
It is to answer these three questions that systematic and thorough evaluation of
organizational change in the Dutch Armed Forces is needed. Our goal is to learn
from several change initiatives in order to decide how to deal with change in
future. To this end, we launched an active evaluation approach, that was combined
with ideas about organizational design. One of our cases is the RNLMA. We will
focus on this case in this article and at the same time discuss our overall approach.
Evaluation of Organizational Change
Before starting we formulated a research design for the evaluation (see Richard-
son et al., 1996). This design was based on ideas of constructivist evaluation, on
which we will elaborate later in this article. During the evaluation, the members
of the evaluation team discovered synergy between different approaches in this
evaluation project: a synergy between an active form of qualitative evaluation
and design. By qualitative evaluation we refer to evaluation based on open
qualitative strategies and methods in order to obtain integral knowledge of the
organization as a unique system. By active we mean the constant dialogue with
participants in the process of change in order to transform and modify ideas about
organizational change, based on the results of interim evaluations. By design we
refer to modern sociotechnical system theory as a tool to analyse and implement
ideas on organization change. Therefore, our emerging knowledge of and our
interventions in the organization are not only based on the interaction of several
Evaluation 5(1)
62
participants in the organization, but are also inspired by the principles of
sociotechnical design as a normative framework. This does not, however, imply
that we wanted to take over the position of the designers in this project, but we
did wish to evaluate the design effort from the normative and analytical per-
spective of sociotechnics. We consider qualitative evaluation and action research
to be parts of the qualitative framework and we consider design as a specication
of the modern sociotechnical framework. We discuss the parts of the framework
as core elements of our approach, starting with qualitative evaluation and action
research as specications of the qualitative framework.
There are those who consider change and intervention to be a total contrast to
evaluation with its more hands-off quality. We see this as part of a broader set of
distinctions. In our opinion evaluation research can be classied along three inter-
related dimensions:
the degree to which evaluators distance themselves or become involved;
the degree to which evaluators use a neutral or a normative analytical frame-
work of interpretation;
the degree to which evaluators passively observe instead of using the
(interim) results to actively inuence the proceedings, as well as to nd out
more about the system.
The more the evaluator is involved, uses an explicitly normative analytical frame-
work for the evaluation, the more actively evaluators participate in the discus-
sions concerning what is to be done with (interim) results, the more possible it
becomes, in our opinion, to speak of active qualitative evaluation (AQE).
Involvement is demonstrated through the participative and interactive
manner of evaluation. The evaluation is then regarded as the mutual product
of the evaluator and those being evaluated with the goal being understand-
ing the complex world of lived experience from the point of those who live
it.
The normative and analytical framework becomes clear when the results of
the qualitative evaluation are positioned in a more general framework. Select-
ing that framework is determined by the goal that has to be attained: in this
case the evaluation of structural organizational change. The added value of
the evaluator is then determined by the link that can be established between
the world of lived experience and more general design and development prin-
ciples (in this case what light can the evaluator shed on the RNLMAs unique
project of change using a sociotechnical framework and what conclusions can
be drawn concerning possible changes in the process of change?).
Activeness is expressed by the deliberate use of intervention techniques by
the evaluator in role. The evaluator intervenes deliberately in order to maxi-
mize the impact of evaluation results. In practice this has meant that, on a
number of occasions, use was made of conference techniques with the
(interim) evaluation results providing the input. The conferences were set
up according to familiar principles from the literature on organizational
design (e.g. Weisborn, 1991). What remains to be resolved is whether such
Kuipers and Richardson: Active Qualitative Evaluation
63
interventions, which are designed to enable a discussion of evaluation results
and to enhance their impact, still fall within the domain of the evaluation
itself. We believe that they do and that an active, professional transfer of
results is part of the evaluation process. In addition, such conferences yield
evaluation data. In the RNLMA case the conferences uncovered a multi-
tude of opinions on what responsibility would mean for the Academy and
the stance of some of the stakeholders. In this article we particularly discuss
this active side. The reader, however, must keep in mind that this is done
from the perspective of the evaluator and not from that of the designer or
change agent though now and again we could not resist the temptation to
look at matters from their perspectives.
Our choice of AQE framework is largely determined by the chosen object of
evaluation, i.e. the evaluation of organizational change. We believe that a distant,
neutral and passive description would contribute little. A factual representation
of the proceedings in the change project is also of little value if the evaluator is
unable to provide a link with more general insights in the eld of organizational
change and design. For this we have used the sociotechnical framework.
Although each process of change is unique and can only be understood from the
world of experience of the participants, the added value of evaluation, we believe,
is determined by its capacity to interpret this unique process within a more
general analytical and normative framework.
In the case of organizational change and design, sociotechnics provides such a
framework. The RNLMA was given the assignment to take the concept of Result
Responsibility and translate it into a exible organization with responsibilities
placed at the lowest levels possible. Sociotechnics is a theory which is ideally
suited to enable just that. It can be employed in two ways:
as an evaluative and diagnostic framework for the analysis and explanation
of structural dysfunctions in existing organizational processes;
as a design methodology to nd structural solutions for such dysfunctions
which will suit the specic organization.
In our role as evaluation researchers we employed sociotechnics mainly as an
evaluative framework to start a dialogue with the organization on the basis of
those evaluations in order to nd possible solutions. This was consistent with our
appointment as evaluation researchers and not as designers or change agents.
Our role was to critically evaluate the design team that was to implement the
Result Responsible Unit.
Characteristic of our approach to evaluation research is that we devote atten-
tion to the process as well as to the content. Both are crucial: an evaluation of
only content would disregard the fact that the process of evaluation is essential
for the acceptance of change and for mobilizing the knowledge and experience
necessary to nd the right solutions.
Evaluation 5(1)
64
Qualitative Evaluation and Action Research as Parts of the Quali-
tative Framework
Qualitative methodology, research and methods have made tremendous progress
during the last three decades. We think it is justied to mention Glaser and
Strauss (1967), Denzin (1989), Glaser (1978), Lincoln and Guba (1985), Ham-
mersley and Atkinson (1983), Miles and Huberman (1984), Strauss and Corbin
(1990), Tesch (1990), Denzin and Lincoln (1994) and Stake (1995). Qualitative
research methods and methodology have grown into a complex spectrum of more
or less elaborated approaches, strategies, methods and techniques. However, this
spectrum can be traced back to no more than three major theoretical perspec-
tives, viz. constructivism, interpretism and human inquiry (Schwandt, 1994). The
purpose of these perspectives is a common one: understanding the complex
world of lived experience from the point of view of those who live it (Schwandt,
1994: 118). The theoretical basis of this general framework is that the qualitative
researcher has to grasp the actors denition of the situation in order to under-
stand the complex world of lived experience. This denition of the situation is
made by actors in order to give meaning to events, phenomena and physical
objects in their environment. Such meanings are derived from the process of
social interaction with others and it is the structured meanings actors compose
and use in everyday life that constitute the objective world. Society, from this
perspective, takes on the form of a meaning structure. It is based on the de-
nitions which meaningful others have in interaction with the actor. Social order
in this context is the result of the agreement between actors on the denition of
the situation. In order to understand meaning, the qualitative researcher must
elucidate the process of meaning construction and clarify what and how meaning
are embodied in the language and actions of social actors (Schwandt, 1994: 118).
This elucidation itself is an interpretative process in which the researcher makes
second order constructs of the construction made by actors.
Of course, there are differences between the various qualitative perspectives
within the overall paradigm, as between relativist and realist perspectives, more
action oriented and more distant perspectives, pure qualitative strategies and
mixed quantitative and qualitative strategies, etc. (see Schwandt, 1994). We will
not, however, discuss these differences. Rather, we will focus on the combination
of qualitative evaluation and action research (Schwandt, 1994; Lincoln and Guba,
1985). In this perspective, evaluation is part of a discourse and action with par-
ticipants in a specic context, striving for democratic and educative goals. Eval-
uating changing social networks, whether organizations or society as a whole, is
not a distant, intellectual business of one or more so-called scientic experts.
Evaluating these changing networks is an (inter)active interchange between (the
knowledge of) the researched and the researcher, in order to bring forward
fundamental change and to increase the democratic and educative possibilities of
actors in their social network. In this view, there is no strict division between
changing by acting and having scientic knowledge, rather, there is mutual
learning by participants as well as the researcher (Boog et al., 1996). The pro-
cesses and outcomes of evaluation can be seen as interactive constructions of the
65
meaning both researcher and researched give to intentions, rules, norms, values
and actions and of the results of these constructions (see Lincoln and Guba, 1985;
Patton, 1987, 1990; Chen, 1990; Scriven, 1991; Schwandt, 1994). The most impor-
tant consequences for evaluation practised in the way described are:
(a) that the aim evaluation is change, i.e. we want to communicate and gener-
ate knowledge in and about a changing setting in which actors, goals and
processes are continuously changing, as are the aims and procedures of
the evaluation itself;
(b) that the aim is to communicate and generate integral knowledge, i.e.
knowledge in the sense of broad, complete knowledge of the setting or
the settings processes in their context (holistic approach);
(c) that evaluation itself is messy, i.e. interwoven with political processes;
1
(d) that as a result of (c), evaluation is a process of negotiation and inter-
action;
(e) that the object to be studied is a unique object, i.e. the knowledge we
generate in the evaluation is time- and place-limited;
(f) that the evaluation is formative, i.e. the purpose is to improve (the
development of) a program and to make participants aware of its goals,
developments and bottlenecks;
(g) that as a result of (f) relevance, acceptability and understandability of
(evaluation) processes and their outcomes become important;
(h) that starting points of evaluation are sensitizing concepts, i.e. prior
implicit or explicit theoretical notions on goals, (side) effects, processes,
contextual inuences, etc.; and,
(i) that a combination of research strategies, methods and techniques, quali-
tative as well as quantitative, can be used in the evaluation.
In our view, there are three dilemmas to be faced in this active evaluation
approach. The rst dilemma is between participation (openness) and direction
(closeness). Most researchers want to bring about change but they do not want
to (and cannot) dictate this change. They try to solve this dilemma by acting as
non-directive facilitators, i.e. the change agent as non-directive orchestrator of
the democratic dialogue. However, here the researcher is confronted with the
paradox: how to change a fundamentally undemocratic system by democratic
means. The second dilemma of this approach is the double role of the researcher:
as change agent as well as (qualitative) researcher. Both roles can interfere with
each other and the borderline between both roles may be fuzzy. Apart from that,
the role of change agent is often very demanding. One possibility is to divide the
roles between different persons, but is that sufcient?
The last dilemma we want to raise is how to deal with scientic theory and
theory of practice. Is it important or even possible to use or develop theoretical
insights in this kind of research? If we use theory, how can we prot most from
it? If we develop theory, what is the (scientic) status of this theory?
As we will discuss below we see an important contribution for theoretical
insights into active evaluation to cope with all these dilemmas.
Evaluation 5(1)
66
The Core of the Modern Sociotechnical Framework
Having discussed qualitative evaluation and action research we want to focus on
the design part of our approach. More specically, by design we refer to
Modern Sociotechnical Design (MSD), which was selected because of its use as
a normative and analytical framework. We would like to emphasize once more
that we did not use this framework as designers but as evaluators of the designs
and the design process. We have diagnosed and interpreted actual developments
against the norms and procedures of this theory. The Dutch version of sociotech-
nical design is an applied theory, concerning the integral (re-)design of produc-
tion processes, including the wider organization. This theory of practice is
founded in a more fundamental theory of social systems (Luhman, 1984; Weick,
1969; De Sitter, 1973; 1993, 1994). Theory and practice are discussed in the work
of Kuipers and Van Amelsoort (1990) and, in greater detail and sophistication, in
the work of De Sitter (1994), the founder of the Dutch approach of sociotechni-
cal design. MSD has its roots in the rst generation sociotechnical approach of
the Tavistock Institute in the UK (see for a recent overview Passmore, 1995). In
the Dutch approach, however, some fundamental theoretical problems are
addressed on that basis, a rened design theory has been developed (De Sitter,
1994). In this article we only discuss a few of the basic characteristics of this
approach.
A rst characteristic is a fundamental view that people labour in a framework
of social exchange and interact with the other players in the organization as part
of a social network. Every workplace is a junction of exchange-relations within
such a network of interactions. Norms, motives, aspirations and needs are devel-
oped in this social interaction. People are conceived as social referring, sense-
making systems. There is no inborn norm that guides human behaviour. Norms
are always produced and reproduced in social networks operating in changing
environments, not inside people.
A second characteristic of practical design theory is the contrast made between
classical and exible organizations. Classically designed organizations are based
on simple tasks, multiple rules and complex formalized networks. The junctions
(the workstations) in such mechanized networks offer very little space for the
development of norms, sense-making or the solving of unexpected problems. In
addition, the load of coordination and the loss of regulation capacity in such
complex networks increase exponentially with the degree of the division of
labour. The incapacity of the nodes in the classic networks to regulate and the
heavy weight of coordination explain a lot of serious dysfunctions within classic
organizations especially in circumstances of continuous change. The inability to
handle change is an inherent characteristic of what we call classic design. These
dysfunctions are precisely analysed in terms of the characteristics of different
network structures in Dutch sociotechnical theory. It is a truism to say that these
days organizations have to cope with a turbulent environment. Far more is
demanded of most organizations in terms of exibility, innovation, quality, deliv-
ery times, reduction of stocks, quality of work, etc. A central point of departure
of sociotechnical design is that radical redesign of classic network structures is
67
needed to handle these new demands connected with change and turbulence.
New network structures are based on complex tasks, few rules and simple net-
works. The junctions in such new networks offer space, i.e. regulation capacity
for norm-making, sense-making, problem-solving and commitment in interaction
with other network-nodes, so that the burden of coordination can be drastically
reduced.
A third characteristic of this approach is the idea of redesign. A key idea of
sociotechnical redesign is reduction of structural complexity, so that many forms
of new networks become possible, depending on local conditions. Mixed self-
directed teams are chosen most of the time at the micro-level and self-supporting
strategic units at the meso-level. Drastic reduction of staff departments often
results from redesign and these departments may be regrouped in mixed teams for
integral support. Dutch sociotechnical theory has a well-developed methodology
at its disposal, helping organizations to nd suitable solutions for regrouping and
recoupling network structures. Much experience has been gained with the trans-
formation process itself. The sociotechnical designer is confronted with a paradox
of change, which is the counterpart of the earlier mentioned paradox: how to
arrive at a participative or democratic organization through essentially directive
or undemocratic means. On the one hand, top management need a clear and con-
sistent change policy, on the other hand everyone in the organization must par-
ticipate in making new sense of the organization. This should lead to a process in
which a meaningful structure is the product of sense-making and interaction within
the social network: redesign is self-design. The actors in the social network have
to produce their own redesign based on knowledge of design principles.
The central question MSD tries to answer is how to design and develop
network structures which improve the quality of work and organization, especi-
ally in uncertain and dynamic conditions. A central indicator of both the quality
of work and organization is the self-regulating and self-learning capacity of
people, units and the organization as a whole. In certain networks (bureaucratic,
centralized, functionally specialized) the opportunities for selective and meaning-
ful interaction are few and in other networks (organic, decentralized, function-
ally unspecialized) they are many. MSD as a design-theory is focused on the
precise architecture of organizations as networks depending on the characteristics
of order-ows and the uncertainty of circumstances. MSD is strongly inuenced
by interactionism. Of course, there are differences between active evaluation and
MSD, as we will discuss below. Nevertheless, there is also a clear congeniality of
spirit. Both approach change from an active, integral and holistic orientation.
Also the uniqueness, internal validity and meaningfulness of relations and inter-
actions are stressed in all the approaches referred to. They have much in common
in their scientic philosophy and they all deviate clearly from mainstream posi-
tivist scientic approaches.
Active Evaluation and Sociotechnical Design
Dutch modern sociotechnics offers a general design and implementation theory,
without denying the uniqueness of each organization. The core laws of
Evaluation 5(1)
68
sociotechnical design, which are analytical in character and therefore true by de-
nition, should be used as a general guide to nd suitable local solutions. The
sociotechnical designer will teach organization members the essence of design
theory so that they can participate in the design and development of their own
organization. In this sense, the sociotechnical designer is a teacher and an
expert who teaches the members of the organization how to change and
redesign their own network structures, i.e. their own organization. Designers are
directive in the sense that they teach fundamental design principles to all
members of the organization, and non-directive in the sense that the members
have to transform their own network structures. Organization members have to
select and interpret sociotechnical design principles in the light of local conditions
and constraints. The sociotechnical designer may support the organization in this
transformation process in different ways. Initial power differences in the existing
network are accepted as a fact of life. Paradoxically, however, the power of the
top level has to be used to transform the hierarchical and rigid network structures
and to strive for power equalization. Dutch experience has taught us that absol-
ute commitment of one or more top managers to the transformation project is
the most critical pre-condition for success. An important condition for cooper-
ation between managers and designers is that top managers must subscribe to the
values of MSD and they must be able to face the consequences of their actions.
Mostly, this will, however, only be clear during implementation. In this way, the
designer also has a conicting role. The modern sociotechnical framework con-
trasts with the active evaluation on directivity. The active evaluator is not in a
position to design in a directive way, but he or she is in a position to operate as a
sounding-board inspired by the sociotechnical framework. Furthermore, notions
on the direction of change are mostly absent in active evaluation. Most qualitative
evaluation researchers argue that the inquirer should not take the lead in the
implementation of the results of evaluation to improve the organization. This
view is most obvious in the new realist evaluation approach (Pawson and Tilley,
1997).
Another difference lies in the notion of power which is best described within a
qualitative framework in constructivist and stakeholder evaluation (Lincoln and
Guba, 1985; De Bourgoyne, 1994; Greene, 1994). Here, power is seen as a given
and neutral even though major aspect or research topic in the evaluation. Power
differences are to be ascertained and analysed in networks and are not, as in the
modern sociotechnical framework, to be accepted and strategically used by the
researcher. At best, even the activist inquirer who aims at revitalization, renewal
and increasing the effectiveness of an organization is no more than an organizer
of the democratic dialogue
2
. The statement evaluation is politics (Greene,
1994) therefore has no validity for the researchers commitment to lower or
higher level members in the network structure. In the modern sociotechnical
framework, as we have shown, it means a strategic choice for commitment of
higher level members as a vital factor to break through the existing situation.
There is also a difference in theoretical orientation between active qualitative
evaluation and sociotechnical design. As we have shown above, MSD has its
theoretical roots in system thinking. In both active qualitative evaluation and
69
modern sociotechnics, actors are seen as social-referring and sense-making. In
both paradigms social norms, order, structure and meaning are produced in the
interactive process. But active qualitative evaluation and the Dutch version of
system thinking differ on some major points. In modern sociotechnics, for
example, structure is not only a product of interaction, but also a determinant.
Thus, people run the risk of becoming the prisoners of rigid, unproductive or
chaotic networks created and formulated by themselves or by elites without their
consent. Organizations can behave for long periods of time as if they were closed
systems (Weick, 1969). In Weicks terms, we can speak of a real structure of the
organization. Usually qualitative perspectives are inclined to stress that struc-
tures are the ever-changing products of spontaneous and creative interaction
processes. In the most active qualitative evaluation, objectivity in terms of the
world of facts and what we take as objective knowledge and truth is mostly
absent. Reality is pluralistic: people construct their understandings from experi-
ence and from being told what the world is, not by discovering it whirling there
untouched by experience (Stake, 1995: 100). Even in symbolic interactionism
(social) structure and social culture are abstract systems that only exist for and
have inuence on people by their sense-making. Structure can only be dened
and seen through the eyes of actors that construct and preserve that structure
(Wester, 1992). Modern sociotechnics, by contrast, even speaks of core laws of
structure design, which are analytical in character and therefore true by de-
nition. For example: if an individual or a group is given responsibility for a com-
plete product instead of only part of it, a structural basis is created, by denition,
for intrinsic involvement.
These differences in theoretical orientation did not prevent us combining
frameworks, especially in research practice. On the contrary, we felt that both
frameworks are supplementary as we will describe in our approach below. It has
to be mentioned that in this context sociotechnical design is not used as a design
methodology, but as a normative perspective to analyse, judge and inuence ini-
tiatives of organizational change.
Active Qualitative Evaluation and Design in the Case of the Royal
Netherlands Military Academy
One of the units in the Dutch Armed Forces that has to change into a Result
Responsible Unit is the Royal Netherlands Military Academy (RNLMA). We use
the RNLMA to develop, test and rene our approach. The RNLMA was selected
for evaluation on the basis of accessibility and the progress of change at the time
(Richardson et al., 1996). The RNLMA trains and educates ofcers for the Dutch
Army as well as the Air Force. The Academy has existed for 165 years, and
focuses on military training as well as scientic education and research. Scientic
education and research deal with several aspects of Management Science
(materiel, logistics, economics, organization and personnel) as taught at regular
universities. In 1992 the Academy was reduced in size and its educational system
was reorganized. Scientic education in the academy was broken up into two
phases: RNLMA I and RNLMA II, with two years of military service outside the
Evaluation 5(1)
70
Academy in between. RNLMA I is divided into twelve modules of nine weeks
each: three modules of military training and nine modules of scientic education.
RNLMA II, the period after a two-year military service period,
3
contains ve
modules of scientic education. Having completed the RNLMA I course, cadets
become Lieutenants in the Army or Air Force, and are ready to function at
middle level (Captain, Major) and higher level (Lieutenant-Colonel, Colonel,
Brigadier, Major-General, Lieutenant-General and General) management func-
tions in the Dutch Armed Forces.
Since January 1996 the RNLMA has had the status of Result Responsible Unit.
This means that the Commandant of the RNLMA has to reorganize it along the
lines of a small, exible, economical and performance-based unit. Specications
of control of resources within the unit, between the unit and other units and
between the unit and the parent organization must be formulated in a manage-
ment contract with the Commander-in-Chief of the Dutch Armed Forces
(Richardson et al., 1996). This contract came into force in November 1996. This
process of organizational change in the RNLMA was prepared by a special
internal project group, set up by the Commandant of the RNLMA in January
1995: the Project Group Performance Accountability. This group consisted of ve
persons: a professor in information science (the chairman), the chief of the
department of planning of the RNLMA, the controller of the RNLMA (chief
department of nances) and two advisors (nancial and organizational) from the
Department of Defence in The Hague. This group had the task to investigate the
consequences of performance accountability for the internal structure and func-
tioning of the RNLMA and to prepare its internal implementation. The project
group had six work groups, covering the most important elements of preparing a
new organization (budgeting, processes/product description, management
system, information system, implementation and performance indicators).
During this process of preparation, a small evaluation team was installed by the
Commandant of the RNLMA, alongside the project group, consisting of three
members of the RNLMA Military Management Faculty. It is this team that we
are part of. The installation of the evaluation team rst became an item when the
faculty discussed its new research program for 1995/1996. One of the projects,
called Evaluation of Initiatives in Organizational Change in the Dutch Armed
Forces, was formulated in terms of qualitative evaluation and in-depth case
studies (see: Richardson et al., 1996). The board of project-leaders of the facultys
research program decided to investigate the process of change at the RNLMA.
The evaluation team started this research in 1995. As described here, we speak
of an internal evaluation (Richardson et al., 1996).
Procedures in Active Qualitative Evaluation and Design
It is important to stress that we had no inuence in designing the process of
organizational change as described above and initiated by the Commandant of
the RNLMA. Also it is important to stress that this design was not explicitly based
on sociotechnical principles. When we were installed as an evaluation team, we
decided to use these principles as a normative mirror to evaluate the process of
71
change so far and to inuence this process. The Commandant and the project
team were not familiar with our evaluation approach. In the following we will
describe which procedures we used, concentrating on the active side of the evalu-
ation process. However, the reader should bear in mind that this was always based
on (the interpretations of) evaluative data about the change project and the
design.
Preliminary Moves
The rst step (January 1995) was to make clear the outlines of our approach and
research questions to both Commandant and project group. After discussion, the
following research questions were formulated:
(a) to what extent and in which way will the ideas of the strategic level be
transformed, (re)shaped and modied at the RNLMA?
(b) which interpretations and mechanisms play a central role in this trans-
formation process?
(c) to what extent can the process and possible outcomes of this transform-
ation process at the RNLMA be improved?
(d) how can the outcomes be implemented in the existing organization?
(e) to which kind of organization will this process of implementation lead?
After agreeing on these research questions, it was decided that we attend the
(two-weekly) meetings of the project group. In these meetings, the interim results
of the several work groups were discussed and future steps for the preparation
and implementation of the new organization were formulated. During these
meetings our role was to register the discussions and other interesting events such
as the standpoints of certain participants, the way these were legitimized and the
way coalitions were formed. Apart from the meetings, we gathered documents
dealing with the ideas of the strategic level of the Armed Forces on (aspects of)
conduct of business (see also Richardson et al., 1996). These documents can be
seen as the strategic sensitizing concepts, which the project group has to trans-
late, rene and elaborate on the operational level of the RNLMA. We stored all
these data in a database for qualitative analysis, called KWALITAN (Peters et
al., 1989) and analysed them using qualitative coding techniques (Wester, 1992;
Miles and Huberman, 1984). We discussed the results of the analyses weekly in
our evaluation team.
The Miniconference
During the analyses and discussions in the evaluation team, we discovered some
major and regular points of discussion within the project group. Approaching the
summer of 1995, we decided to organize a discussion of these and other topics
with the members of the project group and the chairs of the work groups. The
goal of the evaluation team when organizing this discussion was to generate some
views and concepts of project group and work group members on these topics as
well as to give feedback from a sociotechnical point of view. The discussions took
place in July 1995 in the form of a half-day miniconference. Here we see clearly
the active participation of the evaluation researcher in the discussion of
Evaluation 5(1)
72
organizational change at the RNLMA. The following four main topics were
brought forward at the conference and will be discussed briey below.
i) The composition of the project group and its work groups. The work groups
consisted solely of members from the four departments of the academy. Each
work group had to deal with one aspect of the reorganization: one with per-
formance indicators, another with budgeting, yet another with the denition of
processes and products, again another with the management system. The func-
tion-based work groups were to produce their reports independent of each other
after which these reports were integrated. The discussion centred around whether
this method was adequate to nally arrive at an integrated picture of the new
organization. In the view of some participants it was not. The method would lead
to a differentiated, fragmented picture of the organization, leading to much more
confusion rather than to an integrated picture and more clarity. The members of
the work groups were hardly committed to the whole task. The project organiz-
ation was based on a functional division of labour and the dysfunctions men-
tioned could easily be predicted from a sociotechnical perspective.
ii) The nature of the documents of the strategic level of the Armed Forces. As
pointed out earlier, these documents are meant to be dealt with as sensitizing
concepts for reorganization. Two different opinions on the status of these docu-
ments emerged. One camp was convinced that reorganization should take place
exactly as dictated by the strategic level in terms of contents and path, while
another camp wanted to leave as much room as possible for their own organiz-
ation to transform and modify these concepts. From our sociotechnical point of
view, transparency and agreement concerning strategy are absolute prerequisites
for internal (re)design.
iii) The contents of these documents. The importance of embedding responsibility
and performance accountability as deep as possible in the organization is stressed
constantly in these documents. However, the development of structural con-
ditions for the organization to be able to implement these ambitions had not yet
been thought of. A central design-rule of sociotechnical design is that certain pre-
conditions in the production structure should be fullled before responsibilities
can be decentralized.
iv) The (lack of) methodology of change. From the start, there was much con-
fusion in the project group on the time path of their work. Should their nal docu-
ment be denitive in January 1996, when the RNLMA was ofcially registered as
a Result Responsible Unit? Or was that the date for the presentation of a rough
starting framework, which would be rened during 1996 and become denitive
in January 1997? Another bottleneck was the lack of agreement on the composi-
tion of the new organization according to the principles of result responsibility.
What is the core of result responsibility? What does the organization want to
achieve? What is needed to achieve a minimal level of result responsibility? How
are these ideas to be implemented in the future?
73
Our conclusion at the end of the conference was that the conference was very
helpful in understanding the (dys)functioning of the changing organization.
Informing the Management Team
The next step for the evaluation team was to analyse outputs and discussions of
the miniconference, in terms of future actions. For the evaluation team, these
outputs were the immediate input for the presentation of the progress of our
research to our principals, the Commandant of the RNLMA and his manage-
ment team. We decided to try and t the results of the miniconference into a
single perspective. This was intended to improve the quality of the discussions
on (aspects of) organizational change and to initiate a strategy of implemen-
tation. The perspective was mainly inspired by the Modern Sociotechnical
Design described earlier. We were convinced that this perspective would give an
integrated view of the elaboration and implementation of organizational change
at the RNLMA to date. The perspective would in our view also offer the struc-
tural pre-conditions within which several initiatives for the organization of
responsibilities and performance accountability could optimally be developed.
It is clear that the design part of our approach is put on stage here. The presen-
tation of our perspective in the management team took place on three occasions
in OctoberNovember 1995. The three main parts of this presentation were as
follows.
Firstly, we presented our evaluation of organizational change and result
responsibility at the RNLMA. Some global but important starting points were
stressed. For example, we stressed that the introduction of result responsibility in
an organization is much more than merely introducing (technical) systems,
bureaucratic rules and norms, and a punishment/reward system to obtain such
responsibility. We stressed that structural conditions must rst be developed in
order to create room for the denition of responsibilities and tasks at each level
of the organization (Kuipers and Van Amelsfoort, 1990; De Sitter, 1994). At the
same time, a cultural change can be initiated and made visible through the
development of new forms of management and leadership, new forms of com-
munication and through new initiatives. We stressed that such a strategy leads to
more conciseness, involvement and initiative in the organization.
Secondly, a possible strategy for elaboration and implementation of the
changing process at the RNLMA was presented. This strategy was intended to
solve the problems in the existing strategy as they had surfaced in the evaluation,
and was inspired by sociotechnical notions. The strategy started with the
organization of a broad strategic management conference of RNLMA staff and
line managers and their immediate subordinates. A rst goal of this strategic
conference was to dene and stress the importance of the global guidelines and
starting points of the strategy for change. This was to be mirrored in the environ-
ment of the RNLMA following on from a Strengths, Weaknesses, Opportunities
and Threats analysis (SWOT). Another goal was to translate global guidelines
into specic principles of result responsibility and to dene future actions for
each part of the organization. The conference took place over two days in Feb-
ruary 1996. The evaluation team participated in this event: one member as
Evaluation 5(1)
74
facilitator and one member as active stakeholder. The results were described in
a substantial report which contains a SWOT-analysis, an outline business model
and an outline management contract. As a result of the conference, the pro-
ject group was disbanded and the responsibilities for further elaboration and
implementation of the change process were brought back into the line
organization.
Thirdly, our presentation contained a possible strategy for implementing the
process of change. This proposal was based on the observation that the top-down
implementation strategy was rather impotent. The members of the project team
and the top management complained that they got no response. The main prin-
ciple of our proposal was the organization of bottom-up development of ideas on
further aspects and principles of result responsibility: what does it mean for our
line organization? This is further described below.
Bottom-up Discussion and Integration of the Result Responsibility
Principles
Having presented a strategy for the elaboration and implementation of the global
guidelines and starting points of change, this strategy was approved during the
conference. The idea is to organize bottom-up discussions to give concrete form
to the guidelines of result responsibility in each part of the organization. This
started around November 1996. Within the global guidelines and starting points,
the different departments are formulating their view on result responsibility and
organizational change. One integration team per organization line is responsible
for the integration of their discussion. The goal is to complete the discussion in
each division with a memorandum of standpoints of that specic organizational
division. The role of the evaluation team was at rst facilitative by formulating
the rules of the game.
4
Some of these rules are:
(a) the different departments per division create their own view on result
responsibility and organizational change as input for discussion to the
integration team;
(b) the integration team per division must include at least one member of
each department who must be acceptable to their department;
(c) the members of the integration team must have some expertise on result
responsibility and organizational change;
(d) the members of the teams are responsible for the feedback to their
department and should constantly put forward the standpoints of that
department;
(e) the means (time, budget, expertise etc.) for those participating in the
teams must be available.
Apart from these rules some cultural norms for the support of the process in the
teams were also articulated. These included open discussion, tolerance for mis-
takes, expertise is more important than hierarchy, teamwork, constant awareness
of the situational context and active involvement.
Furthermore, one member of the evaluation team was to participate in the
75
discussions of each integration-team. While participating in the discussion the
theoretical insights on organizational change of members of the evaluation team
were put forward. This is the point in time that the process of organizational
change and result responsibility at the RNLMA and the accompanying work of
the evaluation team have reached.
Lessons Learned
The main purpose of this project was to learn how to deal with fundamental
organizational change as an active evaluator. The active evaluator aims at maxi-
mizing the effect of his/her evaluative efforts in support of a sound process of
change, with regard to contents as well as form. The criteria for what is sound are
derived from insights from organizational design and development. In particular,
we have made use of insights from sociotechnics. From the above description it
is apparent that whilst the active form of evaluation, based on a framework of
sociotechnical concepts, has been very useful, it has not been a guarantee of
success.
Intervention techniques for organizational design proved to be useful to actively
disseminate the results of the evaluations. The conferences themselves were also
a means to gather information about the change project. The sociotechnical frame-
work proved to be a potent means to interpret the dysfunctions regarding the
change process at the RNLMA in a normative-analytical way. It also allowed
structural remedies to be identied in such a way as to give the Result Respons-
ible Unit (RRU) concept more weight within the RNLMA. However, this did not
imply that these structural remedies were introduced without resistance. The gap
between the professed RRU ambitions and actual practice clearly manifested
itself. In practice the RRU concept was seen mainly as an administrative, bureau-
cratic intervention, shifting hitherto central responsibilities to the Commandant of
the RNLMA without many consequences for internal management. Ofcially, the
RRU philosophy meant that the awareness of result responsibility was to be
mobilized down to the level of people carrying out the work and that the con-
ditions for this were to be created in the organization.
The normative evaluation also helped isolate empty rhetoric within the RRU
philosophy. By empty rhetoric we mean that people say one thing and do the
other. On the one hand they commit themselves to structural innovation within
the RRU framework an innovation aimed at dening the right conditions for
the mobilization of self-responsibility in an atmosphere of mutual trust. On the
other hand the actual routine of the organization is strongly dominated by deeply
ingrained bureaucratic mechanisms, such as micro-management, the drive for
promotion, risk avoidance, and a need for regulation and control. Such mechan-
isms can be perfectly explained from a sociotechnical point of view as belonging
to the pre-existing structure. Fundamental innovation means sticking out your
neck and learning from mistakes. But avoiding real risks and preventing mistakes
is still the best guarantee for promotion in the old structure; and promotion is
sacred. Although people have mastered the rhetoric of modern management, it
still goes against deeply rooted bureaucratic principles. Consequently, in spite of
Evaluation 5(1)
76
the professed support, the RRU project has ultimately yielded little in terms of
innovation, and mainly adopted the guise of administrative, bureaucratic regu-
lation.
All in all, we have concluded that active qualitative evaluation (AQE) proved
to be a useful means to put the RRU project in a broader and more principled
framework and to make the implications of a consistent translation down to the
basic operations of the organization clearly visible. However, this has not led to
a broad implementation. Only in one part of the organization the faculty was
a more or less serious attempt made to translate the RRU philosophy down to
the level of people carrying out the work. This has resulted in an increased
responsibility of the departments, which now manage their own budgets. The sub-
culture in the faculty proved to be a relatively favourable breeding ground for
RRU ideas. Insight into the ultimate consequences of RRU may have frightened
off certain key gures in other parts of the RNLMA.
Of course, AQE in itself is not and cannot be a guarantee for organizational
innovation. But it can support it. In our experience successful innovation projects
in organizations have one thing in common: there is always one central gure who
leads the project, and who inspires it to break through the vicious circle of bureau-
cracy. AQE cannot supersede such gures, but in organizations where they are
present AQE can full an important role. It can act as a sounding board, allow-
ing them to broaden their vision and stimulating dialogue, whilst helping form the
process of participation necessary to mobilize members of the organization and
exploit present know-how to the benet of the design process.
Is this a reason only to embark on a project with organizations in which the
potential for change is present in the gure of an inspiring change manager who
guarantees a serious attempt at innovation? We think not. At the start of this kind
of project, it is very seldom clear whether such persons are there in the organiz-
ation. AQE can make a contribution towards mobilizing and developing poten-
tial talent. It may make potential change managers aware of the structural causes
of problems in their organization and the possible benets of fundamental inno-
vation; though this can never be fully guaranteed. Those engaged in AQE have
to take the risk, to have been a Don Quixote, in hindsight. Without trying in this
active way you can be certain of making no impact as an evaluator of organiz-
ational change. Once again, it is necessary to emphasize here that our role in this
project is essentially different from that of a sociotechnical designer. As (active)
evaluators we did not, on principle, become designers ourselves, but we acted as
a sounding-board for an organization that was supposed to produce its own re-
design. That, on rare occasions, we could not resist the temptation to climb into
the chair of the designer does not detract from this principle and is partly under-
standable in terms of the paradoxes that accompany fundamental organizational
change.
Notes
The authors would like to thank Sjaak Rovers and Harry Kirkels for their help in trans-
lating this article.
77
1. Antonio (1995: 21) denes evaluation as a common process of searching and negoti-
ating for shared meaning and values that connect each other between different stake-
holders.
2. For instance, this was the case in the Swedish LOM project.
3. This does not apply to cadets who become pilots. They will have four or ve years of
instruction and training in the Netherlands as well as in the United States. Cadet-
engineers also receive longer training.
4. In this case the role of (active) evaluator changed into that of process facilitator. With
hindsight it may not have been a wise choice because explicit responsibility was now
taken for the process of change itself, mixing evaluation and process responsibility.
References
Antonio, S. (1994) Theory-driven Evaluations. Verslag van een Workshop van Prof. H. T.
Chen [Theory-driven Evaluations: Report of a Workshop with Prof. H. T. Chen], Belei-
dsanalyse 94(4): 1622.
Boog, B., L. Keune, H. Coene and R. Lammerts (1996) Theory and Practice of Action
Research. Tilburg: Tilburg University Press.
Chen, H. T. (1990) Theory-driven Evaluation. Newbury Park, CA: Sage.
De Bourgoyne, J. G. (1994) Stakeholder Analysis, in C. Cassell and G. Symon (eds)
Qualitative Methods in Organizational Research, pp. 187207. London: Sage.
De Sitter, L. U. (1973) A System-theoretical Paradigm of Social Interaction; Towards a
New Approach to Qualitative System Dynamics, Annals of System Research (3):
10940.
De Sitter, L. U. (1993) A Sociotechnical Perspective, in F. van Eynatten The Paradigm
That Changed The Workplace. Stockholm/Assen: The Swedish Centre of Working Life,
Van Gorcum.
De Sitter, L. U. (1994) Synergetisch Produceren [Synergetic Producing]. Assen: Van
Gorcum.
Denzin, N. K. (1989) The Research Act, 3rd edn. Englewood Cliffs, NJ: Prentice-Hall.
Denzin, N. K. and Y. S. Lincoln (eds) (1994) Handbook of Qualitative Research. Thousand
Oaks, CA: Sage.
Glaser, B. (1978) Theoretical Sensitivity: Advances in the Methodology of Grounded
Theory. Mill Valley, CA: Sociology Press.
Glaser, B. and A. L. Strauss (1967) The Discovery of Grounded Theory: Strategies for
Qualitative Research. Chicago, IL: Aldine.
Greene, J. C. (1994) Qualitative Program Evaluation, in N. K. Denzin and Y. S. Lincoln
(eds) Handbook of Qualitative Research, pp. 53044. Thousand Oaks, CA: Sage.
Hammersley, M. and P. Atkinson (1983) Ethnography: Principles in Practice. London:
Tavistock Publications.
Kuipers, H. and P. van Amelsfoort (1990) Slagvaardig Organiseren [Alert Organizing].
Deventer: Kluwer.
Lincoln, Y. and E. Guba (1985) Naturalistic Inquiry. Beverly Hills, CA: Sage Publications.
Luhman, N. (1984) Soziale Systeme: Grundriss einer Allgemeinen Theorie [Social Systems:
Basics for a General Theory]. Frankfurt am Main: Suhrkamp.
Miles, M. B. and A. M. Huberman (1984) Qualitative Data Analysis: A Method to the
Madness. London: Sage.
Passmore, W. (1995) Social Science Transformed: The Social Technical Perspective,
Human Relations 48(1): 121.
Evaluation 5(1)
78
Patton, M. Q. (1987) How to Use Qualitative Methods in Evaluation. Newbury Park, CA:
Sage.
Patton, M. Q. (1990) Qualitative Evaluation and Research Methods. Newbury Park, CA:
Sage.
Pawson, R. and N. Tilley (1997) Realistic Evaluation. London: Sage.
Peters, V., F. Wester and R. Richardson (1989) Kwalitatieve Analyse in de Praktijk en Han-
dleiding bij KWALITAN versie 2.1 [Qualitative Analysis in Practice and Manual of
KWALITAN version 2.1]. Nijmegen: SWI-reeks.
Pollitt, C. (1995) Justication by Works or by Faith? Evaluating the New Public Manage-
ment, Evaluation 1(2): 13354.
Richardson, R., J. Soeters and H. Kuipers (1996) Evaluation of Organizational Change in
the Dutch Armed Forces, Evaluation (2)1: 722.
Schwandt, T. (1994) Constructivist, Interpretivist Approaches to Human Inquiry, in N.
K. Denzin and Y. S. Lincoln (eds) Handbook of Qualitative Research, pp. 11837. Thou-
sand Oaks, CA: Sage.
Scriven, M. (1991) Evaluation Thesaurus, 4th edn. Newbury Park, CA: Sage.
Soeters, J. and R. Richardson (1996) Van Macro- Naar Micro-Evolutionaire Veranderin-
gen in de Krijgsmacht [From Macro- to Micro-Evolutionary Changes in the Armed
Forces], Bedrijfskunde 68(4): 5160.
Stake, R. (1994) Case Studies, in N. K. Denzin and Y. S. Lincoln (eds) Handbook of
Qualitative Research, pp. 23647. Thousand Oaks, CA: Sage.
Stake, R. (1995) The Art of Case Study Research. Thousand Oaks, CA: Sage.
Strauss, A. L. and J. Corbin (1990) Basics of Qualitative Research: Grounded Theory Pro-
cedures and Techniques. Newbury Park, CA: Sage.
Tesch, R. (1990) Qualitative Research: Analysis Types and Software Tools. New York:
Falmer Press.
Weick, K. E. (1969) The Social Psychology of Organizing. Reading, MA: Addison Wesley.
Weisborn, M. R. (1991) Productive Workplaces. San Francisco, CA: Jossey-Bass.
Wester, F. (1992) Strategien Voor Kwalitatief Onderzoek [Strategies for Qualitative
Research]. Muiderberg: Coutinho.
HERMAN KUI PERS is Professor of Psychology at the Department of Social
Sciences and Philosophy at the Royal Netherlands Military Academy, Breda,The
Netherlands. He is also Professor of Sociotechnical Design at the Technical
University, Eindhoven. His current research interests include (evaluation) research
into organizational and strategic changes in civic and military organizations.
RUDY RI CHARDSON is Assistant Professor of Methodology at the
Department of Social Sciences and Philosophy at the Royal Netherlands Military
Academy, Breda,The Netherlands. His current research interests include
(evaluation) research into organizational and strategic changes in civic and military
organizations.
79

Active Qualitative Evaluation

Încărcat de

Informații document

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Active Qualitative Evaluation

Încărcat de

Drepturi de autor:

Formate disponibile

Active Qualitative Evaluation

Core Elements And Procedures

S-ar putea să vă placă și