The effects of continued

use of intelligent
decision aids upon
auditor procedural
knowledge
Micheal Axelsen
Research Colloquium 2009
Agenda
Overview


 Research problem
 Research questions
 Contributions
 Motivation
Theory development


 Background
 Theoretical model
 Hypothesis development
Methodology
Potential limitations

Appendix: References
Research problem

 The continued use of tools to support the audit

process is needed to help ensure audit quality and
consistency
 However, persistent use of this technology results in
auditors being deskilled in terms of applying their
judgment in the audit process
 So this research-in-progress is designed to examine
the impact of the continued use of technology-based
intelligent decision aids upon auditor skill levels
Motivation

 Practical:
 My practical experience in an audit firm for ten years is
partly my motivation for this (including ‘junior-burger’
effect – substituting ‘juniors’ for highly-paid senior staff
through use of an IDA)
 Dowling & Leech (2007) noted that practitioners did not
raise deskilling as an issue - confirmed by my
exploratory interviews
 Theoretical:
 The Theory of Technology Dominance (Arnold & Sutton
1998) outlines a deskilling proposition –tested only once
(Dowling, Leech & Moroney 2006) for declarative
knowledge
 Leech (2008) noted ‘alarm bells ringing’ - research into
design of audit support systems and long-term
Research questions

 Does continued use of IDAs reduce auditor declarative

knowledge?
 Does continued use of IDAs reduce auditor procedural
knowledge?
 Does ability, experience, motivation, or environment
have an impact upon this deskilling effect?
Contributions

 Contributions of this potential research to theory:

 Increase robustness through reconciliation with other
established theories (anchoring & adjustment heuristic,
cognitive load theory) to extend implications to other
professional areas
 Extend theoretical testing to the field
 Test proposition 7 (deskilling) in the context of
procedural knowledge (‘auditor know-how’)
 Contributions of this potential research to practice:
 Contributes towards establishing a basis for the design
of effective audit support systems that reduce the
deskilling effect
Background & theoretical
model
 Reliance by auditors on information systems in the
audit process is increasing (Dowling & Leech 2007;
Leech 2008; Mascha 2001), and this is confirmed by
experience and observation
 Many benefits of IDAs are recognised, including
training , efficiency, consistency (Elliott & Kielich
1985; O’Leary 1987; Rose & Wolfe 1998; Sutton &
Byington 1993)
 Managers also perceive that IDAs allow for the control
of junior staff and improve risk management (Dowling
& Leech 2007)
 There may be negative epistemelogical implications
of IDA use in the long term that affect accounting
expertise (Arnold & Sutton 1998; Dowling, Leech &
Anchoring & adjustment
heuristic

After Tversky & Kahnemann (1974); Epley & Gilovich

(1996)
Cognitive load theory

After Sweller (1988); Libby & Tan (1994)

Theory of technology
dominance
 Sutton (2006) – factors in reliance:
 At low to moderate level of experience, there is a negative
relationship between task experience and reliance on a
decision aid
 Positive relationship between task complexity and reliance
on a decision aid
 Positive relationship between decision aid familiarity and
reliance on the decision aid
 Positive relationship between cognitive fit and reliance on
the decision aid
Theory of technology
dominance
 Sutton (2006) – susceptibility to dominance by
technology:
 When user expertise and an IDA are mismatched, there is a
negative relationship between the user’s expertise level and
the risk of poor decision making
 When user expertise level and an IDA are matched, there is a
positive relationship between reliance on the aid and
improved decision making
 Sutton (2006) – long-term effects (deskilling):
 Positive relationship between continued use of an IDA and
the de-skilling of knowledge workers’ abilities for the domain
in which the aid is used
 Negative relationship between the broad-based, long term
use of an IDA in a given problem domain and the growth in
knowledge and advancement of the domain
Theoretical model
Hypothesis development

 The following hypotheses are proposed:

 H1: The longer an auditor has continuously used an IDA
the less declarative knowledge they possess
 H2: The longer an auditor has continuously used an IDA
the less procedural knowledge they possess
 H3a: The higher an auditor’s experience the less effect
the continuous use of an IDA has upon declarative
knowledge
 H3b: The higher an auditor’s experience the less effect
the continuous use of an IDA has upon procedural
knowledge
Hypothesis development

 Proposed hypotheses (continued):

 H4a: The higher an auditor’s ability the less effect the
continuous use of an IDA has upon declarative
knowledge
 H4b: The higher an auditor’s ability the less effect the
continuous use of an IDA has upon procedural
knowledge
 H5a: The higher an auditor’s motivation the less effect
the continuous use of an IDA has upon declarative
knowledge
 H5b: The higher an auditor’s motivation the less effect
the continuous use of an IDA has upon procedural
knowledge
Approach

 A survey of public sector auditors to test the

theoretical model is proposed - this will be supported
by a series of semi-structured interviews
 Participants are potentially 1,096 Australian public
sector audit staff – access may be possible to a
further 1,700 international participants
 Semi-structured interviews with senior staff will be
undertaken with seven public sector audit offices
 A survey will be undertaken with these participants; it
is expected that approximately 219 usable responses
across 7 offices (with differing implementations of
IDAs) will be received (20% response)
 The survey instrument will draw upon existing
instruments where possible, and new measures where
Potential limitations

 External validity
 Conclusions limited to Australian public sector – may not
generalise to private sector auditors
 Internal validity
 Potential bias in the auditor cohort as auditors leave
public sector – deskilled auditors may leave and thus not
be captured
 Construct validity
 Measuring procedural and declarative knowledge will be
difficult and necessarily perceptual
 Concerned that amount of procedural knowledge in tools
used is little or limited – may not see highly
intelligent/restrictive IDAs instantiated in any instances
we study, and declarative knowledge and structural
restrictiveness have previously been dealt with
Appendices
References
References
 Anderson, J. R. (1993). Rules of the Mind. Hillsdale, New Jersey:
Erlbaum.
 Anson, R., Bostrom, R., & Wynne, B. (1995). An experiment
assessing group support system and facilitator effects on meeting
outcomes. Management Science, 41(2), 189.
 Arnold, V., Clark, N., Collier, P., Leech, S., & Sutton, S. G. (2006).
The Differential Use and Effect of Knowledge-Based System
Explanations in Novice and Expert Judgment Decisions. MIS
Quarterly, 30(1), 79-97.
 Arnold, V., Collier, P., Leech, S., & Sutton, S. G. (2004). Impact of
intelligent decision aids on expert and novice decision-makers'
judgments. Accounting and Finance, 44(1), 1-26.
 Arnold, V., & Sutton, S. G. (1998). The Theory of Technology
Dominance: Understanding The Impact Of Intelligent Decision
Aids On Decision Makers’ Judgments. Advances in Accounting
Behavioral Research, 1(1), 175-194.
 Awasthi, V., & Pratt, J. (1990). The Effects of Monetary Incentives
References
 Chu, P. C., & Elam, J. J. (1990). Induced system restrictiveness: An
experimental demonstration. IEEE Transactions on Systems, Man
and Cybernetics, 20, 195-201.
 DeSanctis, G., & Poole, M. S. (1994). Capturing the complexity in
advanced technology use: Adaptive structuration theory.
Organization Science, 5(2), 121.
 Dowling, C., Leech, S., & Moroney, R. (2006). The Deskilling of
Auditors’ Abilities: An Empirical Test of the Theory of Technology
Dominance Paper presented at the Second Asia/Pacific Research
Symposium on Accounting Information Systems.
 Dowling, C., & Leech, S. (2007). Audit support systems and
decision aids: Current practice and opportunities for future
research. International Journal of Accounting Information Systems,
8(2), 92-116.
 Elliott, R. K., & Kielich, J. A. (1985). Expert systems for
accountants. Journal of Accountancy, 160(3), 126-134.
 Hampton, C. (2005). Determinants of reliance: An empirical test of
References
 Joyce, E. J., & Biddle, G. C. (1981). Anchoring and adjustment and
probabilistic inference in auditing. Journal of Accounting Research,
19(1), 120-145.
 Kowalczyk, T., & Wolfe, C. (1998). Anchoring effects associated
with recommendations from expert decision aids: an
experimental analysis. Behavioral Research in Accounting,
10(Supplement), 147-169.
 Leech, S. (2008, 1 November 2008). The use and misuse of
intelligent systems in accounting: the risk of technology
dominance. Insights: Melbourne Economics and Commerce
Retrieved 7th July 2009, 2009, from
http://insights.unimelb.edu.au/vol4/08_Leech.html
 Libby, R., & Tan, H.-T. (1994). Modeling the determinants of audit
expertise. Accounting, Organizations and Society, 19(8), 701-716.
 Mascha, M. F. (2001). The effect of task complexity and expert
system type on the acquisition of procedural knowledge: some
new evidence. International Journal of Accounting Information
References
 Mascha, M. F., & Smedley, G. (2007). Can computerized decision
aids do “damage”? A case for tailoring feedback and task
complexity based on task experience. International Journal of
Accounting Information Systems, 8(2), 73-91.
 Masselli, J. J., Ricketts, R. C., Arnold, V., & Sutton, S. G. (2002). The
Impact of Embedded Intelligent Agents on Tax Compliance
Decisions. Journal of the American Taxation Association, 24(2), 60-
78.
 McCall, H., Arnold, V., & Sutton, S. G. (2008). Use of Knowledge
Management Systems and the Impact on the Acquisition of
Explicit Knowledge. Journal of Information Systems, 22(2), 77-101.
 Mitroff, K., & Mason, R. (1989). Deep ethical and epistemological
issues in the design of information systems. Expert System
Review(Fall), 21-25.
 Noga, T., & Arnold, V. (2002). Do tax decision support systems
affect the accuracy of tax compliance decisions? International
Journal of Accounting Information Systems, 3(2), 125-144.
References
 O'Leary, D. (1987). Validation of expert systems with applications
to auditing and accounting expert systems. Decision Science,
18(3), 468-486.
 Presutti, A. H. (1995). Anchor and adjustment heuristic effect on
audit judgement. Managerial Auditing Journal, 10(9), 13-21.
 Rose, J. M., & Wolfe, C. J. (1998). Schema acquisition and the
effects of cognitive load imposed by decision aid design
alternatives: a tax setting. Paper presented at the Proceedings of
the Fifth Accounting Information Systems Conference, Scottsdale,
AZ.
 Silver, M. (1988). Descriptive analysis for computer-based
decision support. Operations Research, 36(6), 904-916.
 Slovic, P., & Lichtenstein, S. (1971). Comparison of Bayesian and
regression approaches to the study of information processing in
judgment. Organizational behaviour and Human Performance, 6,
649-744.
 Speier, C., & Morris, M. G. (2003). The influence of query interface
References
 Sutton, S. G., & Byington, J. (1993). An analysis of ethical and
epistemological issues in the development and implementation of
audit expert systems. Advanced Public Interest Accounting, 5,
234-243.
 Sutton, S. G. (2006). Theory of Technology Dominance:
Reflections on the Research to Date and the Mysteries
Unexplored. Paper presented at the Digital Accounting Research
Conference, University of Huelva, Spain.
 Sweller, J. (1988). Cognitive Load During Problem Solving: Effects
on Learning. Cognitive Science, 12(2), 257-285.
 Todd, P., & Benbasat, I. (1992). The Use of Information in Decision
Making: An Experimental Investigation of the Impact of Computer-
Based Decision Aids. MIS Quarterly, 16(3), 373.
 Todd, P., & Benbasat, I. (1994). The influence of decision aids on
choice strategies: An experimental analysis of the role of
cognitive effort. Organizational Behavior and Human Decision
Processes, 60(1), 36.
References
 Wheeler, B. C., & Valacich, J. S. (1996). Facilitation, GSS, and
training as sources of process restrictiveness and guidance for
structured group decision making: An empirical assessment.
Information Systems Research, 7(4), 429.