Received: 12 July 2017 Revised: 8 June 2018 Accepted: 5 July 2018

DOI: 10.1111/faam.12182

ORIGINAL ARTICLE

The impact of supreme audit institutions and

ombudsmen in Belgium and The Netherlands
Wouter van Acker Geert Bouckaert

KU Leuven Public Governance Institute, Leuven,

Belgium
Correspondence
Wouter van Acker, KU Leuven Public Governance
Institute, Parkstraat 45, Bus 3609, 3000 Leuven,
Belgium.
Email: wouter.vanacker@kuleuven.be
Funding information
European Union Seventh Framework Programme,
Grant/Award Number: 320090
Abstract
This article investigates the impact of the audit and ombud process
on the implementation of recommendations made by supreme audit
institutions and ombudsmen in Belgium and the Netherlands. Based
on previous findings, the focus lies on influential internal and exter-
nal factors in determining the effectiveness of the audit and ombud
investigations. The article concludes that the impact of the process
and factors often mentioned in the literature are less significant than
found in other research. Political factors, such as coalition agree-
ments and politicians trying to save face, seem to be much more influ-
ential in explaining the implementation rate.

KEYWORDS
accountability, audit offices, ombudsman, performance audits

1 INTRODUCTION

Audit and ombudsman institutions are democratic tools in order to check the executive branch of government,
but also management tools in providing advice on customer relations, citizen rights, efficiency, and effectiveness
(Kostadinova, 2015; Pollitt et al., 1999). These roles go hand in hand, ever since performance, instead of a singular
focus on accounting, has taken a prime role in the work of auditors (Arnaboldi & Lapsley, 2008; Arnaboldi, Lapsley, &
Steccolini, 2015). A more efficient, effective, and just government is expected to be more legitimate, and deserves its
citizens’ trust, and thus strengthens democracy. The impact of public accountability, including supreme audit institu-
tions (SAIs) and ombudsmen, has been a research topic for several decades.
However, the effectiveness of audits should not be taken for granted (Morin, 2001). One of the key quests of the
scholars focused on SAIs and ombudsmen is thus to find out how effective these institutions are. Do their opinions
and recommendations really have an effect? How can they be as effective as possible? Which factors are key in making
sure that governmental organizations actually learn from their findings and reports? However, in comparison to perfor-
mance management in general, the amount of research on accountability effectiveness is still somewhat meager (Buck,
Kirkham, & Thompson, 2011; Reichborn-Kjennerud & Vabo, 2017; Van Loocke & Put, 2011). This is especially the case
with regard to the effects of ombudsman investigations (with a few exceptions, differing in degree of similarity to this
article: Gill, 2011, 2012; Hertogh, 2001; Kostadinova, 2015; Wilkins, 2012).
These issues bring social factors and matters of power into the game of accountability, making performance audit-
ing and ombudsman investigations into much more than a mere technical issue (Modell, 2009). These social and
power-related factors come to the fore in the relationship between the actors (organizations under scrutiny) and the

Financial Acc & Man. 2019;35:55–71. wileyonlinelibrary.com/journal/faam 

c 2018 John Wiley & Sons Ltd 55
56 VAN ACKER AND BOUCKAERT

TA B L E 1 Selected cases

BE forum Actor Topic Year of audit

SAI 1 Flemish Employment Bureau HRM Policy 2011
SAI 2 Tax Inspection Bureau Organization and 2010
Functioning
SAI 3 Federal Ministry of Justice HRM Policy 2010
and Penitentiary
Institutions
SAI 4 Agency for the European Use of Resources 2010
Social Fund
Federal Ombudsman Federal Ministry of Justice Complaint Reports on 2009–2012
and Penitentiary Prisoner's Rights
Institutions
Flemish Ombudsman 1 Flemish Tax Collecting General Annual Complaints 2006–2013
Agency Report
Flemish Ombudsman 2 Flemish Public Transport General Annual Complaints 2009–2012
Company (De Lijn) Report
Flemish Ombudsman 3 Flemish Agency for Housing General Annual Complaints 2010–2011
Report
NL forum Actor Topic Year of audit
SAI 1 DNB: National Bank Stability of Banks 2011
SAI 2 Ministry of Economic Affairs European Procurements 2012
SAI 3 Ministry of Economic Affairs Tariff Regulations for the 2009
Energy Market
SAI 4 Ministry of Finance Evaluation of Subsidies 2011
SAI 5 ProRail (Railway Use of Public Funds 2011
Infrastructure)
SAI 6 Ministry for Health and Online Medical Care 2009
Sports

forum (the ombudsman or SAI; Bovens, 2005). Do they trust each other? Is the forum seen as an expert on the issue
at hand? Is the forum perceived as a controller or as an advisor? Are the final recommendations explained properly
to the actor? Each of the answers to these and other questions could have conceivable influence on the effectiveness
and implementation rate of the audit and ombudsman recommendations. Over the past years researchers have come
up with a large number of factors that are claimed and/or found to be influential (see Van Loocke & Put [2011] for an
overview). In this article, several of these factors are investigated for their influence in determining the effectiveness of
six Dutch and eight Belgian cases of public accountability. These cases are shown hereunder, in table 1. We will inves-
tigate how the processes of the Dutch and Belgian SAI, the Belgian federal ombudsman, and the Flemish ombudsman
have influenced the implementation rate of the policy recommendations. The focus lies on the role of the media, parlia-
ment, trust and collaboration, the perception of each other's role and expertise, exit meetings, and to what extent the
SAIs’ and ombudsmen's criteria are appropriate and known to all participating actors.
The research presented in this article is novel in the field for a number of reasons. First, it focuses on some scarcely
researched factors: the dilemma between the role of SAIs and ombudsman as watchdog/controller on the one side, and
advisor on the other, as well as the influence of parliament and the media on the implementation of recommendations.
Second, it integrates ombudsmen into the research on public accountability. The impact of ombudsmen's recommen-
dations and the process of ombudsmen's investigations have been investigated only rarely.
To best of the authors’ knowledge, ombudsman institutions and SAIs have not been investigated in combination in
terms of the effectiveness of their investigations and audits. More has been written from a legal perspective on SAIs
and ombudsmen as being part of a larger accountability regime (McMillan, 2010; Wettenhall, 2012; Wilkins, 2015;
VAN ACKER AND BOUCKAERT 57

FIGURE 1 Venn diagram of ombudsman and SAI activities [Colour figure can be viewed at wileyonlinelibrary.com]

Wilkins, Phillimore, & Gilchrist, 2017). Both institutions share important similarities, but also display important dif-
ferences. Although ombudsmen are more focused on individual complaints, the rights of citizens in dealing with the
government, and resolving conflicts between citizens and the government, SAIs focus on the control and performance
of the public sector at a more aggregate level (Wilkins, 2015). Research has also shown the possibility of both to have
different internal cultures, and can at times even clash (Wilkins, 2010, 2012). However, there are some important over-
laps as well. First, both are concerned with public accountability. They are seen as the two traditional, quintessential
watchdogs for the public sector (Gay & Winetrobe, 2008). Second, when ombudsmen start investigations into govern-
ment policies and government conduct on a more aggregate level (often after they have found trends in the complaints
they receive from citizens), they enter the arena of the SAIs. In those cases, they take on more systematic investigations,
more similar to the type of work the SAIs do as well. Simplified, Figure 1 displays the differences and overlap in ombuds-
man and SAI activities. For the purpose of the investigation presented in this article, we only focus on cases where the
two overlap. This study is therefore also exploratory in comparing these two different types of organizations. Besides
one study by Wilkins (2012), the authors were unable to find other investigations into the effects and implementation
rates of ombudsman investigations and SAI audits. In the conclusion, we reflect on ways in which future research could
delve deeper into this comparison.
First, these factors of influence are introduced. A little further on, in Section 3, some methodological choices are
discussed, with specific attention to our case selection, and the cases that were eventually investigated. Results are
described and critically discussed in Section 4, after which the article concludes with several suggestions for further
research and recommendations for practice.

2 LITERATURE ON INFLUENTIAL FACTORS

There are numerous factors that can potentially be of influence on how the audit/ombudsman process fosters or ham-
pers the implementation of policy recommendations. The focus of this article lies on social and political factors, fol-
lowing the growing literature on (neo-)institutional approaches to audit and ombudsman research (Modell, 2009; Van
Helden, Johnsen, & Vakkuri, 2006). The issues that were picked to focus on in this article came from a thorough read-
ing of the current literature. Although other issues could be investigated (e.g., timing or quality of recommendation) as
well, practical matters such as interview lengths and research capacity led the authors to focus on seven factors.

• Exit meetings
• Collaboration and trust
• Perception on the forum's expertise1
58 VAN ACKER AND BOUCKAERT

FIGURE 2 Summary of influential factors [Colour figure can be viewed at wileyonlinelibrary.com]

• Perception on the forum's role

• Appropriateness of the forum's criteria

• Influence of parliament
• Influence of the media

We summarize these factors in Figure 2. Except for the actor's views on the appropriateness of the forum's criteria,
we compare the view of both parties. In the following sections, each of these factors is discussed.

2.1 Exit meetings

What happens after the audit or investigation has been concluded can be of great importance for the eventual imple-
mentation of recommendations. As a matter of fact, it can be argued that this is still an integral part of the audit and
ombudsman process. One of the issues in the finalization of the process, how audit offices and ombudsmen commu-
nicate their findings (e.g., through exit meetings), has received little to no attention in the literature (González, López,
& García, 2008). This article focuses on whether there was an exit meeting, which would be an opportunity where the
auditor or ombudsman presents his or her findings and recommendations, puts them into context, and answers any
questions the actor might have. As words on paper always fall short in explaining things, such a meeting might help
increase the impact of the audit or investigation and the recommendations. The literature surrounding discursive insti-
tutionalism has shown how change might be explained through deliberation and argumentation (Mansbridge et al.,
2010; Reichborn-Kjennerud, 2014; Schmidt, 2010). This further emphasizes the importance of studying communica-
tion in the audit and ombudsman investigation process. Exit meetings are obviously only one possible instance of com-
munication between forum and actor. Before and during the investigation/audit process, it is likely that more than a few
meetings take place between the forum and actor. The effect of this collaboration, and the nature of this collaboration,
is the topic of the next section.

2.2 Collaboration and trust

Although there is a certain asymmetricity in the actor–forum relationship, it is still to a large part dependent on coop-
eration from both sides. In such relationships, trust, suspicion, risk, and control play an important role in how a pro-
cess between actors can play out (Kastberg, 2016; Oomsels & Bouckaert, 2014). It is important to note that actors
usually have certain specific persons assigned to dealing with external audit and ombudsman investigations. This means
VAN ACKER AND BOUCKAERT 59

that the experiences from previous processes can be transferred to other investigations and audits in the future, if this
specific person stays in place (Justesen & Skærbæk, 2010; Reichborn-Kjennerud, 2015). Whether that relationship is
characterized by confrontation and distrust, or cooperation and trust, can have a major influence on the effectiveness
of the recommendations. If the relationship turns confrontational and distrustful, the recommendations will be solely
produced by the forum. If done in cooperation, the outcomes will follow from open communication, negotiation, and
consultation between the forum and actor (Gill, 2011; Hertogh, 2001; Morin, 2001; Van Loocke & Put, 2011). In that
case, the possibility for implementation of the results is much higher than in the case where the recommendations are
drawn up unilaterally by the forum (Reichborn-Kjennerud, 2014).

2.3 Perception on forum's expertise

The characteristics of the forum, actual and perceived, might to a great extent influence the way in which the actor per-
ceives its reports and overall feedback. Hence, their credibility and reputation in the eye of the actor plays an important
role (Morin, 2001; Van Loocke & Put, 2011). A particularly important aspect of reputation and credibility (besides, e.g.,
impartiality) is that of expertise. This issue has received very little attention in the literature. Does the forum, accord-
ing to the actor, have enough expertise to critically assess the information and/or to convert its judgment into powerful
lessons for the administration (Bovens, Schillemans, & ’t Hart, 2008)?

2.4 Perception on forum's role

SAIs are seen to have a dual role, as “being a source of discipline for auditees and a catalyst for the stimulation of
improvement” (Pollitt & Summa, 1997, cited in Morin & Hazgui, 2016, p. 571). The same can be said of ombudsmen.
Different roles can be present at the same time, but their prominence may change between times, topics, and contexts
(Morin, 2011; Pollitt et al., 1999). Different role-expectations might create very different attitudes toward the oppos-
ing side during the audit or investigation process (Morin & Hazgui, 2016). For example: if the actor sees the forum as a
watchdog, it might act defensive and closed-off. On the other side, if it sees the forum as “only” an advisor, it may not
attach a lot of weight to the opinion of the forum (Bovens et al., 2008; Gill, 2011; Lonsdale, 1999; Van Loocke & Put,
2011). Finally, if the actor and forum have different perceptions of the forum's role, this discrepancy may lead to further
miscommunication and a problematic process.

2.5 Appropriateness of criteria

Criteria are central to any audit or investigation, if a certain judgment is going to be rendered about the actor's policies
or functioning. “Evidence does not speak for itself; it acquires meaning only when gauged against a criterion” (Pollitt
et al., 1999, p. 79). So, criteria connect what is, to what should be (Put, 2011). Insight on the actor's side into the criteria
that are used by the forum will help the actor to understand the recommendations that may follow. Similarly, it can
aid the discussion between forum and actor when the criteria are well understood. If both argue on the basis of other
criteria without being aware of it, this can seriously distort an otherwise open and effective dialogue. Put (2011) found
five different norms used by SAIs, but also found that different norms are used in different reports. It might therefore
not always be clear for actors which criteria are used, or where they come from (Put, 2011). If the actor does not know
the origin or details of a criteria, it is difficult to understand where the forum's report and conclusions are derived from,
perhaps inhibiting the implementation of recommendations. Investigating the appropriateness and transparency of the
used norms is thus worthwhile when researching ombudsman and SAI effectiveness.

2.6 Influence of parliament

Additionally, there are also factors at play outside of the forum, the actor, their relationship, and the reporting that
is being done. “The relation between two parties might be affected by a third or fourth party in a way that has
60 VAN ACKER AND BOUCKAERT

consequences both for the levels of trust and for the choice of control mechanisms” (Kastberg, 2016, p. 34, citing
Tomkins, 2001). The actor in particular can feel pressure from the media and parliament as third actors to implement
or not implement certain recommendations from the forum (Gill, 2011; Lonsdale, 1999; Vanlandingham, 2006).
Parliamentary influence can thus form a catalyst or hindrance in the effectiveness of its reporting. Justesen and
Skærbæk (2010) show how the interplay between the process, politics, and media can have a profound impact on the
actor's identity after the process has been completed.

2.7 Influence of the media

The relationship between audit offices and ombudsmen on the one hand, and the effect of the media coverage of
the audit's or investigation's outcomes on the other hand, seems important as well, if only intuitively. It has however
received surprisingly little attention in the literature (Bringselius, 2014; González-Diaz, García-Fernandez, & López-
Díaz, 2012; González et al., 2008). In theory, media exposure could increase the pressure on actors to implement the
recommendations as put forward in the reports (Lonsdale, 1999). There might be risks too, however. Bringselius (2014)
found that the focus on maximizing media coverage by the Swedish SAI led to three risks: the performance audits
tended to focus more on compliance instead of performance; the focus shifted toward negative findings; and the SAI
encroached upon politics and policy-making. In the current article, a distinction is made between:

• The amount of media attention surrounding the forum's report and the findings it contains.
• The focus of this media attention (fair and balanced, or focused on scandals and drama).

2.8 Integration of the influential factors

This article tries to investigate all the factors mentioned before, and to see whether or not they influenced the imple-
mentation rate of the audited/investigated organizations.

3 METHODOLOGY AND CASES

Considering the number of factors involved in perceptions, human interaction, and relationships, we opted for inter-
views as our method of data gathering. Concepts such as trust and credibility are elusive and definitions can differ
strongly between persons and situations. These issues asked for nuance and context, hence the conclusion that sur-
veys were not the best suited method for this particular investigation.
The cases for the analysis here are the Dutch “Rekenkamer” (SAI), the Belgian “Rekenhof”/“Court des Comptes”
(SAI), and both the Flemish and Belgian federal ombudsmen. These organizations are shortly introduced further on.
Semi-structured interviews were conducted with respondents who were actively involved in the audit/investigation
process, and preferably led this process on behalf of the actor or the forum. Using a case survey methodology (Yin
& Heald, 1975), we were able to combine “cross-sectional analysis and in-depth, processual analysis” (Larsson, 1993,
p. 1515). This methodology can bridge the gap between large-N research, which lacks the in-depth insights of the audit
and investigatory process on the one hand, and the lack of generalizability of small-N case studies on the other hand.
In total 26 interviews were conducted, and in two cases the actor responded through filling in an adjusted, electronic
interview protocol, for a total of 14 cases. In order to give an overview of the results, the data from the interviews will
be greatly, and sometimes inelegantly, shortened. In sections 4.2 through 4.8, however, they have been discussed in
more detail. Interviewing actors who were directly involved in the process is a tricky endeavor, especially for audits and
investigations that have finalized a few years ago (Banister, 2005). It is possible for such respondents to be subjective
or defensive in their answers. At the same time, it is possible they might be trying to paint too pretty a picture of the
implementations, the process, or both. However, these issues are inherent to interview techniques, the only method
available for this type of investigation. Triangulation was used to resolve any discrepancies when it came down to facts.
VAN ACKER AND BOUCKAERT 61

It is up to the researchers, as well as the reader, to critically assess the responses and observations found through the
interviews.
Following are a few points on the operationalization of “impact.” In this article, impact is measured as the number
of recommendations that is implemented by the actor. Other research has taken a similar approach (e.g., Hatherly &
Parker, 1988; Kostadinova, 2015; Milgrom & Schwartz, 2008; Morin, 2001; Raudla, Taro, Agu, & Douglas, 2016; Torres,
Yetano, & Pina, 2016; Weets, 2011). At the same time, Pollitt et al. (1999) and Van Loocke and Put (2011) found that
many SAIs also use implementation rates as their measure of impact. However, this method has also been criticized.
Weets (2008) and Desmedt, Morin, Pattyn, and Brans (2017) pointed out that audits (and ombudsman investigations,
for that matter) have an impact on much more than just the recommendations that are or are not adopted by the actor.
It is also true that the total number of recommendations made can influence the percentage of actual implementations.
At the same time, the required efforts (money, time, Full-time equivalent (FTE), political capital) to implement recom-
mendations can differ from recommendation to recommendation, and thus the difficulty of implementation as well.
Pollitt et al. (1999) also note that counting implemented recommendations are a flawed operationalization, but con-
clude, much like we do, that research on performance audit and ombudsman investigation impact cannot avert looking
at the implementation or recommendations as a factor. Although flawed, it is still a reasonable measure of effective-
ness. It gives us a global view of the effectiveness of the ombudsman organizations and SAIs that were investigated.
At the same time, issues such as the required efforts for implementation can be discussed and observed during the
interviews. In the cases included here, this did not seem to be an issue.

3.1 Belgian cases

3.1.1 The supreme audit institutions

Named in Dutch the “Rekenhof” and in French the “Court des Comptes” is a constitutionally established institution with
the task of performing external audits on the financial and budgetary proceedings, as well as checking the accountancy
of the federal, communal, and regional governments, together with the related institutions and provinces2 (Desmedt
et al., 2017). The tasks of the SAI can be described in more detail as follows:

1. It subjects the operations of the federal government, the community, and regional governments and of the deputa-
tion of the provinces to an audit, a review of legality, and a check on the proper use of public funds. The audits both
consider the expenditures and the revenues.
2. It reports the results of the audits to the respective parliaments and provincial councils.
3. It checks the accounts of the general administration and those who are financially accountable toward the state
(Desomer & Van Loocke, 2010).

3.1.2 The ombudsmen

The focus in this research is on the federal ombudsman and the ombudsman responsible for the Flemish region
and Dutch-speaking community (hereafter called “Flemish ombudsman”). The federal ombudsman has the following
tasks3 :

1. She or he investigates complaints about the federal administration.

2. She or he investigates the workings of the federal administration, which are directed to her or him by the chamber
of representatives.
3. She or he submits recommendations for improvement.
4. She or he reports about its investigations to parliament.
5. She or he investigates reports of assumed violations of integrity.
62 VAN ACKER AND BOUCKAERT

The Flemish ombudsman is a parliamentary ombudsman like its federal colleague. His or her task can be summed up
as follows (Schram, 2013):

1. Investigating and mediating in complaints against the acts and workings of the Flemish administration and the
administration of the Dutch-speaking community.
2. Investigating and mediating concerning reports of breaches of the code of ethics of Flemish parliamentarians.
3. Investigating reports of negligence, misconduct, or criminal acts by personnel of the Flemish administration or the
Dutch-speaking community while carrying out professional tasks.

Besides investigating complaints and reports of misconducts, the ombudsman also has the duty to articulate and
propose cross-cutting recommendations to improve the Flemish administration in its yearly report to parliament
(Schram, 2013).

3.2 The Dutch case

3.2.1 The supreme audit institution

The Dutch SAI (Algemene Rekenkamer, or ARK) is responsible for the external audits of the national governmental
level. Other audit organizations are responsible for the audits of the provinces, municipalities, and water authorities.
Its board consists of three persons, one of whom is the president of the SAI (Troupin, 2012). The Dutch SAI has a wider
task than “only” checking the accountancy of the State's financial statements. As a matter of fact, the Dutch SAI has
a long history of performance audits, ever since the 1950s (Pollitt et al., 1999). Performance and recommendations
lay directly in its scope of work. Organizations that have been granted more independence in carrying out their tasks
(so-called ZBO's) are also under the scope of the SAI4 .

3.3 Audits and ombudsman investigations

Choosing cases for this research was a balancing exercise between two important criteria. First, the particular reports
that were chosen had to be published far enough in the past for the scrutinized organizations to implement the recom-
mendations, because the instrumental use of audit and investigation outcomes can be long term as well as short term
(Desmedt et al., 2017), but recent enough to limit the effect of personnel turnover and fading memories, which could
obstruct our research efforts. The time frame that was chosen consisted of the years 2009–2013. The bulk of the inter-
views were held in 2015. A possible negative consequence of this is that the process had taken place quite some time in
the past, making it potentially more difficult for interviewees to remember each detail. Second, the reports had to focus
on performance measures for a considerable policy, instead of just focusing on the (financial) legality of an organiza-
tion's conduct or on the mediation for one citizen (in the case of ombudsmen). Based on these criteria, 14 cases were
chosen. Fourteen cases is both enough to investigate a wide range of issues, as well as not too much so as to exceed the
possibilities of the investigation in terms of budget. Because anonymity was guaranteed to the involved respondents
in this research, these cases will be anonymized and randomized. The 14 cases were found to embody a representative
picture of the work field of the ombudsman and audit organizations.

4 RESULTS AND ANALYSIS

4.1 Implemented recommendations

Table 2 shows the number of implemented and not implemented recommendations per case. The cases with up to 100%
implementation are expected, based on the literature, to score high on the different factors that are discussed in Sec-
tion 2, as they have relatively high implementation rates. For the case with lower percentages the opposite is expected.
VAN ACKER AND BOUCKAERT 63

TA B L E 2 Implemented recommendations

Belgium Implemented Not implemented Percentage of implemented (%)

SAI 1 0 8 0
SAI 2 8 4 66
SAI 3 2 8 20
SAI 4 8 3 72
Ombudsman 1 2 4 33
Ombudsman 2 3 2 60
Ombudsman 3 3 4 43
Ombudsman 4 2 5 29
The Netherlands Implemented Not implemented Percentage of implemented (%)
SAI 1 2 0 100
SAI 2 2 2 50
SAI 3 3 2 60
SAI 4 5 0 100
SAI 5 3 0 100
SAI 6 3 2 60

Reputaon:
Collaboraon & Forum's Influence of
Topic Exit Meeting
Trust
Forum's Forum's Role
Criteria
Influence of Parliament
Media Percentage of
Experse Implementaon

Cases Actor Forum Actor Forum Actor Forum Actor Forum Actor Actor Forum Actor Forum

Belgium SAI 1 +/- +/- + + ? +/- 2 2 ? + +/- +/- - 72%

Belgium SAI 2 + + . + + + 3 3 ? - - - +/- 0%
Belgium SAI 3 - +/- + +/- +/- + 3 1 ? ? +/- - +/- 66%
Belgium SAI 4 - + +/- . - + 2 2 ? - +/- - +/- 20%
Belgium Ombud 1 ? + + + + + 3 3 ? +/- - + - 33%
Belgium Ombud 2 + + + + + + 4 4 + - +/- + - 60%
Belgium Ombud 3 + + + + + - 5 - + ? + - - 43%
Belgium Ombud 4 ? + + + + - 4 3 +/- - +/- + +/- 29%
Netherlands SAI 1 + . +/- . - +/- 2 3 - + +/- + +/- 100%
Netherlands SAI 2 + . +/- . + +/- 2 2 ? +/- +/- +/- - 50%
Netherlands SAI 3 +/- . + . - +/- 2 3 - +/- + + +/- 60%
Netherlands SAI 4 ? . +/- . + +/- 2 3 ? ? + +/- + 100%
Netherlands SAI 5 - . . +/- +/- 2 4 +/- - +/- +/- + 100%
Netherlands SAI 6 - . + . +/- + 3 2 - +/- +/- +/- +/- 60%

FIGURE 3 Summary of results [Colour figure can be viewed at wileyonlinelibrary.com]

Figure 3 summarizes the results on both the independent and dependent factors. A “?” indicates that the answer is
unknown to the respondent. A “.” means that the answer is unknown to the researchers. “+” indicates a positive view on
the issue at hand, “-” a negative view. Finally, the indicators for the forum's role diverge from 1 (watchdog) to 6 (advisor).

4.2 Exit meetings

First, it was expected that exit meetings lead to more clarity among the actors on what the ideas behind the recom-
mendations are. This, in turn, might make it more likely for the recommendations to be implemented. It is interesting to
see that there are significant differences between the respondents on both sides of the isle. Where all Belgian auditors
told us there were exit meetings, two respondents on the auditees side declared there was no such meeting. This could
mean several things: the respondents might have been inaccurate or forgetful, but it is more likely that the respondents
simply did not experience the exit meeting in the same way. Many auditors told us that the exit meeting is nothing more
64 VAN ACKER AND BOUCKAERT

than a statement of the conclusions, and perhaps a factual check, and not in any way a platform for feedback, let alone
negotiations about the recommendations. The ombudsman cases, on the other hand, were characterized by a contin-
uous dialogue. These cases constitute examples of a more collaborative process, where there is a genuine exchange
of views. As respondents mentioned, it is often not about being convinced by the forum, but at least understanding
the SAI's or ombudsmen's point of view. Argumentation on paper, apparently, does not suffice in many cases. Finally, it
should hardly come as a surprise by now that there was no correlation between the number of recommendations that
were implemented and the presence/quality of an exit meeting.

4.3 Collaboration and trust

Good, informal, and intensive communication, together with a close collaboration between both parties during the
audit, might well contribute to the learning processes in an organization under scrutiny. It is therefore hypothesized in
this article that a good relationship, based on cooperation and trust, should lead to a higher implementation rate than
the opposite. It is important to note that this collaboration and communication should be more elaborate than “just”
an exit meeting. We found in the interviews that the communication and discussions surrounding recommendations in
particular start way before the exit meeting is in sight.
Overall the cooperative and communicative process seems to have been rather well, considering the results that
were found. However, there is a notable difference in this category between Belgium and the Netherlands. The Dutch
respondents paint a more negative picture then their Belgian counterparts. Several respondents on the actor side
described how the relationship with the SAI went beyond the confines of a certain audit, further improving their con-
structive relationship:

It happened often that they call us now and then, in a different context with a question or inquiry. We definitely
try to help them then, or we just go over there and help them out. (Actor respondent SAI 2, Belgium)
And what I did at a certain moment, I just started to have coffee with [the actor] once a week with the contact
person. Of once every two weeks. Just to explain to them what was going on, how the investigation was progress-
ing and talk things through. (Forum respondent SAI 5, the Netherlands)

At the same time, both in the Netherlands and in Belgium, the auditors and ombudsmen most of the time paint
a more positive picture than their actor counterparts. However, there does not seem to be a correlation between
trust/collaboration during the process and the number of recommendations that are implemented afterward. Two
cases that, on the actor side, were characterized as bad relationships, implemented every single recommendation. Like-
wise, there were a number of cases where the relationship was deemed good by both sides, but the recommendations
were only partially implemented, or not at all.

4.4 Perceptions of forum's expertise

The perception of the forum's expertise is linked to the overall trust and collaboration during the audit and ombudsman
process. The expertise of the federal and Flemish ombudsmen was rated high, from both sides of the isle. The SAIs got
worse reviews, from both the actor as well as the forum itself. In the same fashion as the collaboration/trust, the Dutch
cases were rated worse than the Belgian SAI. The most important factor in expertise, for both countries, is said to be
the fact that it is not possible for SAIs to be an expert in every field. Most auditors have a specific field in which they
specialize, but this is still much broader than the public servants on the other side of the table. However, the initial
disadvantage can be overcome during the process through a steep learning curve. Although this might lead to some
delay in the process, and frustration on the side of the actor, forum respondents see this as one of the strengths of the
SAIs and ombudsmen.

You just need to get organized. […] If the information or expertise would be lacking, you just have to work as
hard as possible to make sure you get it. It's unacceptable that we as the SAI couldn't look into issues such as
VAN ACKER AND BOUCKAERT 65

healthcare, only because it's too complicated. If not by us, than by who else? (Forum respondent SAI 6, the
Netherlands. Translation by the authors)
What plays a role here is the enormous disadvantage in knowledge on the topic vis-a-vis people who work there.
[…] They are constantly thinking, at first, “What do they think they can accomplish here.” “Do they really think
we can learn anything from them?” […] But after a while you notice a change. (Forum respondent SAI 1, the
Netherlands. Translation by the authors)

Again, no direct or clear correlation was found between the perceived expertise of the SAI or ombudsman on the
one hand, and the implementation of recommendations on the other. For example, in “Belgium SAI 1″ the expertise was
deemed high by the actor, but no recommendations were implemented. Conversely, in a case where the expertise was
rated negatively, such as “Netherlands SAI 1,” all of the recommendations were implemented.

4.5 Perception of forum's role

Like the perception of the forum's expertise, how an audit organization sees itself (as controller or advisor) partly deter-
mines its attitude toward the organization under scrutiny. Contrariwise might the role the actor ascribes to the forum
determine the actor's attitude toward the latter: cooperative and open, or defensive and closed. The respondents were
asked to fill-in a Likert scale, with a score of 1 being a purely controlling role, and 5 being a purely advising role. The
perceptions of the auditor's role do not differ that much between cases (the majority note a score of 2 or 3).

Well, its somewhere in between the two, I think. Of course, they have their role to play which is partly to control
the terrain. To keep oversight. But on the other hand, for their recommendations they are clearly in an advisory
role. (Actor respondent SAI 3, Belgium)
I'm not automatically in agreement with either of the two extremes. It's somewhere in the middle. I think the
ombudsman has both a controlling task and an advisory task. […] But I don't feel like I'm being watched by my
mother-in-law and corrected all the time. (Actor respondent ombudsman 1, Belgium)

There are however some interesting differences between respondents within the same audit organization. This
implies that much depends on an auditor's personal perception of his or her job, and on the specific experience of
the respondents on the actors’ side. Many respondents noted in interviews that the discussion within the forum, on
whether they are more controller or advisor, is still going on. Overall, however, the perception on the side of the forum
is one of the moderations: they see themselves as being both a controller and an advisor. This role might also change
during the process, starting out as a controller, but becoming an advisor toward the end, when recommendations have
to be drawn up. As one of the respondents mentioned:

The traditional role is one of controller. But a good controller also gives advice. […] Our goal is to improve the
governmental system. You can improve it by constantly pointing at things that need improvement. But sooner or
later the other side is going to ask: but then what? (Forum respondent SAI 3, the Netherlands)

Much like the results with regard to the relationship between the actor and forum, the view on the forum's role
does not seem to be correlated with the number of recommendations that are implemented. On the controller side
of the Likert scale, there are directly contradictory results. The same goes for cases where the forum was assigned an
advisory role.

4.6 Appropriateness of criteria

As mentioned in the overview of the literature, criteria are the framework for the judgments the auditors and ombuds-
man draw-up after their investigation or audit. As such, they form a key part in understanding where the recommenda-
tions come from, and on what they are based. When the criteria are seen as irrelevant by the actors, the organizations
under scrutiny are unlikely to learn from the recommendations. When the criteria are unknown, it is harder to under-
stand for the organization why an auditor has come to certain conclusion, making it more difficult to learn from the
66 VAN ACKER AND BOUCKAERT

recommendations and conclusions. Our results paint a rather grim picture in this regard. Most organizations said that
the criteria were unknown, and for the Dutch cases even unsuitable. A Belgian respondent, after responding to the fact
that the criteria were in fact mentioned in the final report, noted:

Yes, but you discuss those at the start. This has been written up post-factum. You ought to discuss these before-
hand: “This is what we're going to be doing, this is the framework, these will be the questions, and these are the
possibilities.” (Actor respondent SAI 4, Belgium)
Their ideal picture doesn't make sense. It's not rooted in practice and the day to day workings of the field. Neither
is it rooted in the day to day workings of the political and bureaucratic reality. So they're not feasible. (Actor
respondent SAI 3, the Netherlands)

In these cases, it is easier for the actor to claim that the recommendations do not apply correctly to its organization,
and therefore reject the recommendations. However, this grim picture does not seem to directly translate into worse
implementation rates.

4.7 Influence of parliament

Parliamentary pressure, resulting from exposure of the SAI or ombudsman report, is assumed to increase the rate or
chance of the implementation of a recommendation.
Parliamentary influence was regarded as rather insignificant (although less so than the media) with regard to the
implementation of recommendations. In most cases, it was clear for the respondents from both sides that coalition
politics and the coalition agreement limit parliamentary pressure, with the exception of scandals and crises (where
media and parliamentary pressure tend to coincide). Even if there is a parliamentary committee where each and every
forum report is discussed, questions, coverage, and scrutiny were mostly dominated by coalition-opposition tactics
and politics. The fora usually have a close relationship with the parliamentary staff of these committees. They discuss
the agenda, and determine on the basis of that information when the right time to release the report would be. Direct
contact with members of parliament seated on these committees, or focused on the topic at hand, is avoided. The audit
organizations, although some thus had a close and intensive relationship with the administrative side of parliament,
stated that trying to collaborate or influence parliament is very risky business. The worst thing to happen for an SAI or
ombudsman, so it was almost universally claimed, is to be accused of partisanship. They are hence extremely careful in
encountering parliament. Leaning on parliament to increase pressure is equally questionable. If there is pressure from
parliament, it thus has to originate in parliament itself.

4.8 Influence of media

The results show that most media seem rather uninterested in most of the ombudsman and audit reports. Even when
the media does report on the reports, their influence is very limited, according to most respondents in both countries.

It could provide some extra pressure, but that doesn't mean that the recommendations are automatically imple-
mented. Everything depends on whether or not we agree with the recommendations, and if it is in line with the
logic and policy coming from the political hierarchy. (Actor respondent Ombudsman 1, Belgium)

Only when the media attention generates parliamentary attention, it can cumulate to significant pressure to actu-
ally influence the implementation.
The auditors seem to be divided into two camps: one with an extensive media strategy, and one with a rather mini-
malistic strategy toward the media. The latter group stated that it intentionally wanted to avoid media attention, even
though it (at least potentially) could influence the acceptance of its recommendations. The possibility that the media
would mostly focus on the negative parts of a report, even though the majority of findings might be (very) positive,
can seriously deteriorate the forum's relationship with the organization under scrutiny. The auditors and ombudsmen
declared to rather have a good, productive relationship with the actors, than to try to influence the implementation of
recommendations at all costs.
VAN ACKER AND BOUCKAERT 67

If it's at all possible we like to communicate the successes to the media. [..] But of course, the media isn't interested
in successes. (Forum respondent Ombudsman 2, Belgium)

There does not seem to be a link between media pressure and implementation on its own, much like the respondents
noted. It does seem to be the case that media and parliamentary pressure lead to higher implementation rates. How-
ever, the results found here on that point can only be seen as exploratory. Further, more focused research is necessary
to establish more firm conclusions on this issue.

5 CONCLUSION AND DISCUSSION

Contrary to our expectations, the research presented here did not find a clear link between the independent factors
identified from the literature and the impact of audits and ombudsman investigations. Especially the cases marked
with a very high and low implementation score in Figure 3 did not show a clear connection with the influential factors
investigated. One of the explanations might be that the civil servants on the actor side who were interviewed, and
who were part of the audit or investigation process, are not the ones who make the final decision about whether or
not to implement the recommendation. In fact, the involved civil servants are usually rather far removed from the top
decision makers; hence the fact that the process might not have the influence on implementation as is usually expected
in the literature. In the Belgian and Dutch cases, the decision to implement or not in some instances got as high as
the ministerial level. Political aspects naturally play a larger role in decisions made at the higher levels, even further
diminishing the influence of the audit or investigation process. This thus presents a clear window of opportunity for
further research: perhaps the focus of this field of research has been too much pointed toward the process, and not
enough on the politics of implementation.
In that sense, the research presented here contradicts a number of studies carried out before (e.g., parts of the
findings of Weets [2011] and Morin [2001]). It is more in line with research such as that of Pollitt et al. (1999), who
found that “the impact of a performance audit report is largely determined by environmental factors, which cannot be
easily altered by SAIs” (in Weets, 2008, p. 4). However, referring to the earlier made critical notes on the methodology,
the factors studied here are not only influential with regard to implementation, and implementation is only one type of
impact of an audit and ombudsmen. This indicates that the factors here might still be relevant for other types of impact,
and serve other purposes in the functioning of SAIs and ombudsmen. One prime example might be the quality of the
recommendations themselves. It is probably not too far-fetched to assume that the quality of the recommendations is
to a great extent dependent on the actor involved, and not just on the expertise and skills of the auditor or ombudsman.
The transparency of the process and the atmosphere in which the process takes place could greatly improve or inhibit
the quality of the information SAIs and ombudsmen receive. In the end, implementation for the sake of implementation
is not what SAIs, ombudsmen, governments, or societies are looking for. The recommendations have to be actually good
and reached in a proper manner. Second, the specific atmosphere in which an audit or investigation takes place (trusting
or distrusting, collaborative, or one-sided) could lead to differences in reporting from the actor's civil servants to the
minister about the report. The audit and investigatory process thus enter the political realm that is claimed here to be
of greater importance than was previously acknowledged in the literature.

6 RECOMMENDATIONS FOR FURTHER RESEARCH

This article represents one of the very first instances in which the implementation rate of SAIs is jointly investigated
with that of ombudsman institutions. Although a straight up comparison of the two institutions was not the goal of this
article, future research would do well in focusing on this issue. Both are engaged in performance audits, although an
ombudsman institution would probably never call it that. As both have a different legal basis, different historical roots,
68 VAN ACKER AND BOUCKAERT

and different pathways to their present culture and idiosyncrasies, it could be expected that they also perform these
“audits” in a different fashion.
In future research, considering the “negative” findings in this article, it might be wise to divert attention to other
factors (more political in nature, with special attention for the influence of media and parliamentary pressure on the
key decision makers) when investigating the implementation rate of audit and ombudsmen reports. The factors inves-
tigated here and in other articles perhaps serve a better purpose in explaining some other effects audits and investiga-
tions have on public sector organizations. That being said, the results in this article could not be seen as conclusive, and
generalization from qualitative research such as this is obviously flawed.
A particularly interesting as well as particularly challenging topic for further research would be the influence of
the factors presented here on the quality of the recommendations. In line with this, an intensified focus on the role of
criteria in audits and investigations would enable us to compare the quality of recommendations and processes from
different perspectives.
The issues of timing and financial/administrative burden of the implementation of the recommendations, already
mentioned in Section 2 as two factors this article was not able to focus on, both deserve further attention. Windows of
opportunity are important both for financial and political reasons. Costly recommendations in times of financial crisis
are badly timed. The same goes for a critical report for a government that is already receiving a lot of negative feedback,
or is undergoing the consequences of a scandal.
Furthermore, earlier research found that actors “tend to be become more negative if they are exposed to many
performance audits” (Reichborn-Kjennerud, 2015, p. 28). The interviews carried out for the current article showed
that many ministries and governmental organizations have specific people assigned to deal with audits and ombudsman
investigations. These two issues could interact with each other (perhaps positively, but perhaps also negatively), and
warrant further research as well. Finally, on a methodological note, the audit and investigatory process, and the link
with implementation is a complex one. Perhaps such a complex issue is well suited for not just a qualitative approach,
but a microscopic approach. It would therefore be well worth exploring if the influencing process could be investigated
with the use of a process-tracing methodology.

ACKNOWLEDGMENT
The research leading to these results has received funding from the European Union's Seventh Framework Program
under grant agreement no. 320090 (Project Learning from Innovation in Public Sector Environments, LIPSE), Socioe-
conomic Sciences and Humanities. LIPSE is a research program under the European Commission's Seventh Framework
Program as a Small or Medium-Scale Focused Research Project (2011–2014). The project focuses on studying social
innovations in the public sector (www.lipse.org).

NOTES
1
The rest of the text refers to fora (the SAIs and ombudsmen) and actors (the organizations under scrutiny; Bovens, 2005).
2 https://www.ccrek.be/EN/Presentation/Competences.html

3
http://www.federaalombudsman.be/nl/de-federale-ombudsman/opdrachten/wat-doet-hij
4
https://www.rekenkamer.nl/over-de-algemene-rekenkamer/positie-en-bevoegdheden

ORCID
Wouter van Acker http://orcid.org/0000-0002-3879-2096

REFERENCES
Arnaboldi, M., & Lapsley, I. (2008). Making management auditable: The implementation of best value in local government. Aba-
cus, 44, 22–47. https://doi.org/10.1111/j.1467-6281.2007.00247.x
VAN ACKER AND BOUCKAERT 69

Arnaboldi, M., Lapsley, I., & Steccolini, I. (2015). Performance management in the public sector: The ultimate challenge. Financial
Accountability & Management, 31, 1–22. https://doi.org/10.1111/faam.12049
Banister, F. (2005). Through a glass darkly: Fact and filtration in the interpretation of evidence. The Electronic Journal of Business
Research Methodology, 3, 11–24.
Bovens, M. (2005). Public accountability. In E. Ferlie, L. Lynne, & C. Pollitt (Eds.), The Oxford handbook of public management
(pp. 182–208). Oxford: Oxford University Press.
Bovens, M., Schillemans, T., & ‘t Hart, P. (2008). Does public accountability work? An assessment tool. Public Administration, 86,
225–242. https://doi.org/10.1111/j.1467-9299.2008.00716.x
Bringselius, L. (2014). The dissemination of results from supreme audit institutions: Independent partners with the media?
Financial Accountability & Management, 30, 75–94. https://doi.org/10.1111/faam.12028
Buck, T., Kirkham, R., & Thompson, B. (2011). The ombudsman enterprise and administrative justice. Farnham: Ashgate.
Desmedt, E., Morin, D., Pattyn, V., & Brans, M. (2017). Impact of performance audit on the administration: A Belgian study
(2005-2010). Managerial Auditing Journal, 32, 251–275. https://doi.org/10.1108/MAJ-04-2016-1368
Desomer, I., & Van Loocke, E. (2010). Performance audit van het Rekenhof: Bijdrage tot een efficiënte en effectieve overheid?
Vlaams Tijdschrift voor Overheidsmanagement, 1, 41–48.
Dowie, J. (1999). The “pathology of decision making in clinical practice and a framework for evaluations. In H. Selbmann (Ed.),
Guidelines for guidelines in healthcare. Baden-Baden: Nomos-Verlag.
Gay, O., & Winetrobe, B. K. (2008). Watchdogs in need of support. In O. Gay & B. K. Winetrobe (Eds.), Par-
liament's watchdogs: At the crossroads (pp. 11–16). UK Study of Parliament Group. Retrieved from https://
www.ucl.ac.uk/political-science/publications/unit-publications/144.pdf
Gill, C. (2011). Right first time: The role of ombudsmen in influencing administrative decision-making. Journal of Social Welfare
and Family Law, 33, 181–192. https://doi.org/10.1080/09649069.2011.617077
Gill, C. (2012). The impact of the Scottish Public Services Ombudsman on administrative decision-making
in local authority housing departments. Journal of Social Welfare and Family Law, 34, 197–217. https://
doi.org/10.1080/09649069.2012.718535
González, B., López, A., & García, R. (2008). Supreme audit institutions and their communication strategies. International Review
of Administrative Sciences, 74, 435–461. https://doi.org/10.1177/0020852308095312
González-Díaz, B., García-Fernandez, R., & López-Díaz, A. (2012). Communication as a transparency and accountability strat-
egy in supreme audit institutions. Administration & Society, 45, 583–609. https://doi.org/10.1177/0095399712438376
Hatherly, D. J., & Parker, L. D. (1988). Performance auditing outcomes: A comparative study. Financial Accountability & Manage-
ment, 4, 21–41. https://doi.org/10.1111/j.1468-0408.1988.tb00288.x
Hertogh, M. (2001). Coercion, cooperation, and control: Understanding the policy impact of administrative courts and the
ombudsman in the Netherlands. Law & Policy, 23, 47–67. https://doi.org/10.1111/1467-9930.00102
Justesen, L., & Skærbæk, P. (2010). Performance auditing and the narrating of a new auditee identity. Financial Accountability &
Management, 26, 325–343. https://doi.org/10.1111/j.1468-0408.2010.00504.x
Kastberg, G. (2016). Trust and control in network relations: A study of a public sector setting. Financial Accountability & Man-
agement, 32, 33–56. https://doi.org/10.1111/faam.12079
Kostadinova, P. (2015). Improving the transparency and accountability of EU institutions: The impact of the office of the Euro-
pean Ombudsman. Journal of Common Market Studies, 53, 1077–1093. https://doi.org/10.1111/jcms.12245
Larsson, R. (1993). Case survey methodology: Quantitative analysis of patterns across case studies. The Academy of Manage-
ment Journal, 36, 1515–1546. https://doi.org/10.2307/256820
Lonsdale, J. (1999). Impacts. In C. Pollitt, X. Girre, J. Lonsdale, R. Mul, H. Summa, & W. Waerness (Eds.), Performance or compli-
ance. Performance audit and public management in five countries (pp. 171–193). Oxford: Oxford University Press.
Mansbridge, J., Bohman, J., Chambers, S., Estlund, D., Follesdal, A., Fung, A., … Marti, J. L. (2010). The place of
self-interest and the role of power in deliberative democracy. Journal of Political Philosophy, 18, 64–100. https://
doi.org/10.1111/j.1467-9760.2009.00344.x
McMillan, J. (2010). Re-thinking the separation of powers. Federal Law Review, 38, 423–443.
Milgrom, T., & Schwartz, R. (2008). Israel's auditor as policy change agent: The case of water policy. International Journal of Public
Administration, 31, 862–877. https://doi.org/10.1080/01900690701696178
Modell, S. (2009). Institutional research on performance measurement and management in the public sector
accounting literature: A review and assessment. Financial Accountability & Management, 25, 227–303. https://
doi.org/10.1111/j.1468-0408.2009.00477.x
70 VAN ACKER AND BOUCKAERT

Morin, D. (2001). Influence of value for money audit on public administrations: Looking beyond appearances. Financial Account-
ability & Management, 17, 99–118. https://doi.org/10.1111/1468-0408.00123
Morin, D. (2011). Serving as magistrate at the French Cour de Comptes: Navigating between tradition and modernity. Account-
ing, Auditing & Accountability Journal, 24, 718–750. https://doi.org/10.1108/09513571111155528
Morin, D., & Hazgui, M. (2016). We are much more than watchdogs: The dual identity of auditors at the UK National Audit
Office. Journal of Accounting and Organizational Change, 12, 568–589. https://doi.org/10.1108/JAOC-08-2015-0063
Oomsels, P., & Bouckaert, G. (2014). Studying interorganizational trust in public administration. A conceptual and analytical
framework for “Administrational Trust”. Public Performance & Management Review, 37(4), 577–604.
Pollitt, C., Girre, X., Lonsdale, J., Mul, R., Summa, H., & Waerness, M. (1999). Performance or compliance? Performance audit and
public management in five countries. Oxford: Oxford University Press.
Pollitt, C., & Summa, H. (1997). Reflexive watchdogs? How supreme audit institutions account for themselves. Public Adminis-
tration, 75, 313–336. https://doi.org/10.1111/1467-9299.00063
Put, V. (2011). Norms used: Some strategic considerations from the Netherlands and the UK. In J. Lonsdale, P. Wilkins, & T. Ling
(Eds.), Performance auditing. Contributing to accountability in democratic government (pp. 75–94). Cheltenham: Edward Elgar.
Raudla, R., Taro, K., Agu, C., & Douglas, J. W. (2016). The impact of performance audit on public sector organizations: The case
of Estonia. Public Organization Review, 16, 217–233. https://doi.org/10.1007/s11115-015-0308-0
Reichborn-Kjennerud, K. (2014). Performance audit and the importance of the public debate. Evaluation, 20, 368–385.
https://doi.org/10.1177/1356389014539869
Reichborn-Kjennerud, K. (2015). Resistance to control—Norwegian ministries’ and agencies’ reactions to performance audit.
Public Organization Review, 15, 17–32. https://doi.org/10.1007/s11115-013-0247-6
Reichborn-Kjennerud, K., & Vabo, S. I. (2017). Performance audit as a contributor to change and improvement in public admin-
istration. Evaluation, 23, 6–23. https://doi.org/10.1177/1356389016683871
Schmidt, V. A. (2010). Discursive institutionalism: The explanatory power of ideas and discourse. Annual Review of Political Sci-
ence, 11, 303–326. https://doi.org/10.1146/annurev.polisci.11.060606.135342
Schram, F. (2013). Burger en Bestuur. Een introductie tot een complexe verhouding. Brussels: Politeia.
Tomkins, C. (2001). Interdependencies, trust and information in relationships, alliances and networks. Accounting, Organization
and Society, 26, 161–183. https://doi.org/10.1016/S0361-3682(00)00018-0
Torres, L., Yetano, A., & Pina, V. (2016). Are performance audits useful? A comparison of EU practices. Administration & Society,
30, 1–32. https://doi.org/10.1177/0095399716658500
Troupin, S. (2012). Professionalizing public administration(s)? The cases of performance audit in Canada and the Netherlands (PhD
dissertation). KU Leuven Public Governance Institute, Belgium.
Van Helden, J. G., Johnsen, A., & Vakkuri, J. (2006). Researching public sector transformation: The role of management account-
ing. Financial Accountability & Management, 21, 99–133. https://doi.org/10.1111/j.0267-4424.2005.00211.x
Van Loocke, E., & Put, V. (2011). The impact of performance audits of SAIs: A review of the existing evidence. In J. Lonsdale, P.
Wilkins, & T. Ling (Eds.), Performance auditing: Contributing to accountability in democratic government (pp. 175–208). Chel-
tenham: Edward Elgar.
Vanlandingham, G. R. (2006). A voice crying in the wilderness—Legislative oversight agencies’ efforts to achieve utilization.
New Directions for Evaluation, 112, 25-39. https://doi.org/10.1002/ev.205
Weets, K. (2008). September 3–5). How effective are performance audits? A multiple case study within the local audit office of Rot-
terdam. Paper presented at the fifth International Conference on Accounting, Auditing and Management in Public Sector
Reforms, Amsterdam, The Netherlands.
Weets, K. (2011). Impact at local government level: A multiple case study. In J. Lonsdale, P. Wilkins, & T. Ling (Eds.), Performance
auditing—Contributing to accountability in democratic government (pp. 248–267). Cheltenham: Edward Elgar.
Wettenhall, R. (2012). Integrity agencies: The significance of the parliamentary relationship. Policy Studies, 33, 65–75.
https://doi.org/10.1080/01442872.2011.601217
Wilkins, P. (2010). Viewing performance through different lenses—The role of organizational context and culture. Proceedings of the
2010 International Conference on Public Administration (pp. 495–502), Canberra, Australia.
Wilkins, P. (2012). Cultures of ombudsman and audit institutions: Effects on their evaluative activities. In J.-C. Barbier & P.
Hawkins (Eds.), Evaluation cultures: Sense-making in complex times (pp. 159–176). New Brunswick: Transaction Publishers.
Wilkins, P. (2015). Watchdogs as satellites of parliament. Australian Journal of Public Administration, 75, 18–27.
https://doi.org/10.1111/1467-8500.12142
VAN ACKER AND BOUCKAERT 71

Wilkins, P., Phillimore, J., & Gilchrist, D. (2017). Benefits and risks of collaboration between watchdogs: The Western Australian
experience. Policy Studies, 38, 291–310. https://doi.org/10.1080/01442872.2017.1301414
Yin, R. K., & Heald, K. A. (1975). Using the case survey method to analyze policy studies. Administrative Science Quarterly, 20,
371–381. https://doi.org/10.2307/2391997

How to cite this article: van Acker W, Bouckaert G. The impact of supreme audit institutions and ombudsmen
in Belgium and The Netherlands. Financial Acc & Man. 2019;35:55–71. https://doi.org/10.1111/faam.12182