The effects of continued use of intelligent decision aids upon auditor procedural knowledgeMicheal AxelsenResearch Colloquium 2009

Overview Research problem Research questions Contributions Motivation

Theory development Background Theoretical model Hypothesis development

MethodologyPotential limitationsAppendix: References

Agenda

Research problem

The continued use of tools to support the audit process is needed to help ensure audit quality and consistency

However, persistent use of this technology results in auditors being deskilled in terms of applying their judgment in the audit process

So this research-in-progress is designed to examine the impact of the continued use of technology-based intelligent decision aids upon auditor skill levels

Motivation

Practical: My practical experience in an audit firm for ten years is partly my

motivation for this (including ‘junior-burger’ effect – substituting ‘juniors’ for highly-paid senior staff through use of an IDA)

Dowling & Leech (2007) noted that practitioners did not raise deskilling as an issue - confirmed by my exploratory interviews

Theoretical: The Theory of Technology Dominance (Arnold & Sutton 1998) outlines

a deskilling proposition –tested only once (Dowling, Leech & Moroney 2006) for declarative knowledge

Leech (2008) noted ‘alarm bells ringing’ - research into design of audit support systems and long-term consequences for knowledge development was needed

Research questions

Does continued use of IDAs reduce auditor declarative knowledge? Does continued use of IDAs reduce auditor procedural knowledge? Does ability, experience, motivation, or environment have an

impact upon this deskilling effect?

Contributions

Contributions of this potential research to theory: Increase robustness through reconciliation with other established

theories (anchoring & adjustment heuristic, cognitive load theory) to extend implications to other professional areas

Extend theoretical testing to the field Test proposition 7 (deskilling) in the context of procedural knowledge

(‘auditor know-how’) Contributions of this potential research to practice:

Contributes towards establishing a basis for the design of effective audit support systems that reduce the deskilling effect

Background & theoretical model

Reliance by auditors on information systems in the audit process is increasing (Dowling & Leech 2007; Leech 2008; Mascha 2001), and this is confirmed by experience and observation

Many benefits of IDAs are recognised, including training , efficiency, consistency (Elliott & Kielich 1985; O’Leary 1987; Rose & Wolfe 1998; Sutton & Byington 1993)

Managers also perceive that IDAs allow for the control of junior staff and improve risk management (Dowling & Leech 2007)

There may be negative epistemelogical implications of IDA use in the long term that affect accounting expertise (Arnold & Sutton 1998; Dowling, Leech & Moroney 2006; Leech 2008;McCall, McCall, Arnold & Sutton 2008; Mitroff & Mason 1989; Todd & Benbasat 1992; Todd & Benbasat 1994)

Anchoring & adjustment heuristic

After Tversky & Kahnemann (1974); Epley & Gilovich (1996)

Cognitive load theory

After Sweller (1988); Libby & Tan (1994)

Theory of technology dominance

Sutton (2006) – factors in reliance: At low to moderate level of experience, there is a negative relationship

between task experience and reliance on a decision aid Positive relationship between task complexity and reliance on a decision aid Positive relationship between decision aid familiarity and reliance on the

decision aid Positive relationship between cognitive fit and reliance on the decision aid

Theory of technology dominance

Sutton (2006) – susceptibility to dominance by technology: When user expertise and an IDA are mismatched, there is a negative

relationship between the user’s expertise level and the risk of poor decision making

When user expertise level and an IDA are matched, there is a positive relationship between reliance on the aid and improved decision making

Sutton (2006) – long-term effects (deskilling): Positive relationship between continued use of an IDA and the de-skilling of

knowledge workers’ abilities for the domain in which the aid is used Negative relationship between the broad-based, long term use of an IDA in

a given problem domain and the growth in knowledge and advancement of the domain

Theoretical model

Hypothesis development

The following hypotheses are proposed: H1: The longer an auditor has continuously used an IDA the less

declarative knowledge they possess H2: The longer an auditor has continuously used an IDA the less

procedural knowledge they possess H3a: The higher an auditor’s experience the less effect the continuous

use of an IDA has upon declarative knowledge H3b: The higher an auditor’s experience the less effect the continuous

use of an IDA has upon procedural knowledge

Hypothesis development

Proposed hypotheses (continued): H4a: The higher an auditor’s ability the less effect the continuous use

of an IDA has upon declarative knowledge H4b: The higher an auditor’s ability the less effect the continuous use

of an IDA has upon procedural knowledge H5a: The higher an auditor’s motivation the less effect the continuous

use of an IDA has upon declarative knowledge H5b: The higher an auditor’s motivation the less effect the continuous

use of an IDA has upon procedural knowledge

Approach

A survey of public sector auditors to test the theoretical model is proposed - this will be supported by a series of semi-structured interviews

Participants are potentially 1,096 Australian public sector audit staff – access may be possible to a further 1,700 international participants

Semi-structured interviews with senior staff will be undertaken with seven public sector audit offices

A survey will be undertaken with these participants; it is expected that approximately 219 usable responses across 7 offices (with differing implementations of IDAs) will be received (20% response)

The survey instrument will draw upon existing instruments where possible, and new measures where necessary

Potential limitations

External validity Conclusions limited to Australian public sector – may not generalise to

private sector auditors Internal validity

Potential bias in the auditor cohort as auditors leave public sector – deskilled auditors may leave and thus not be captured

Construct validity Measuring procedural and declarative knowledge will be difficult and

necessarily perceptual Concerned that amount of procedural knowledge in tools used is little or

limited – may not see highly intelligent/restrictive IDAs instantiated in any instances we study, and declarative knowledge and structural restrictiveness have previously been dealt with comprehensively

References

Appendices

References

Anderson, J. R. (1993). Rules of the Mind. Hillsdale, New Jersey: Erlbaum. Anson, R., Bostrom, R., & Wynne, B. (1995). An experiment assessing group support

system and facilitator effects on meeting outcomes. Management Science, 41(2), 189. Arnold, V., Clark, N., Collier, P., Leech, S., & Sutton, S. G. (2006). The Differential Use

and Effect of Knowledge-Based System Explanations in Novice and Expert Judgment Decisions. MIS Quarterly, 30(1), 79-97.

Arnold, V., Collier, P., Leech, S., & Sutton, S. G. (2004). Impact of intelligent decision aids on expert and novice decision-makers' judgments. Accounting and Finance, 44(1), 1-26.

Arnold, V., & Sutton, S. G. (1998). The Theory of Technology Dominance: Understanding The Impact Of Intelligent Decision Aids On Decision Makers’ Judgments. Advances in Accounting Behavioral Research, 1(1), 175-194.

Awasthi, V., & Pratt, J. (1990). The Effects of Monetary Incentives on Effort and Decision Performance: The Role of Cognitive Characteristics. The Accounting Review, 65(4), 797.

References

Chu, P. C., & Elam, J. J. (1990). Induced system restrictiveness: An experimental demonstration. IEEE Transactions on Systems, Man and Cybernetics, 20, 195-201.

DeSanctis, G., & Poole, M. S. (1994). Capturing the complexity in advanced technology use: Adaptive structuration theory. Organization Science, 5(2), 121.

Dowling, C., Leech, S., & Moroney, R. (2006). The Deskilling of Auditors’ Abilities: An Empirical Test of the Theory of Technology Dominance Paper presented at the Second Asia/Pacific Research Symposium on Accounting Information Systems.

Dowling, C., & Leech, S. (2007). Audit support systems and decision aids: Current practice and opportunities for future research. International Journal of Accounting Information Systems, 8(2), 92-116.

Elliott, R. K., & Kielich, J. A. (1985). Expert systems for accountants. Journal of Accountancy, 160(3), 126-134.

Hampton, C. (2005). Determinants of reliance: An empirical test of the theory of technology dominance. International Journal of Accounting Information Systems, 6(4), 217-240.

References

Joyce, E. J., & Biddle, G. C. (1981). Anchoring and adjustment and probabilistic inference in auditing. Journal of Accounting Research, 19(1), 120-145.

Kowalczyk, T., & Wolfe, C. (1998). Anchoring effects associated with recommendations from expert decision aids: an experimental analysis. Behavioral Research in Accounting, 10(Supplement), 147-169.

Leech, S. (2008, 1 November 2008). The use and misuse of intelligent systems in accounting: the risk of technology dominance. Insights: Melbourne Economics and Commerce Retrieved 7th July 2009, 2009, from http://insights.unimelb.edu.au/vol4/08_Leech.html

Libby, R., & Tan, H.-T. (1994). Modeling the determinants of audit expertise. Accounting, Organizations and Society, 19(8), 701-716.

Mascha, M. F. (2001). The effect of task complexity and expert system type on the acquisition of procedural knowledge: some new evidence. International Journal of Accounting Information Systems, 2(2), 103-124.

References

Mascha, M. F., & Smedley, G. (2007). Can computerized decision aids do “damage”? A case for tailoring feedback and task complexity based on task experience. International Journal of Accounting Information Systems, 8(2), 73-91.

Masselli, J. J., Ricketts, R. C., Arnold, V., & Sutton, S. G. (2002). The Impact of Embedded Intelligent Agents on Tax Compliance Decisions. Journal of the American Taxation Association, 24(2), 60-78.

McCall, H., Arnold, V., & Sutton, S. G. (2008). Use of Knowledge Management Systems and the Impact on the Acquisition of Explicit Knowledge. Journal of Information Systems, 22(2), 77-101.

Mitroff, K., & Mason, R. (1989). Deep ethical and epistemological issues in the design of information systems. Expert System Review(Fall), 21-25.

Noga, T., & Arnold, V. (2002). Do tax decision support systems affect the accuracy of tax compliance decisions? International Journal of Accounting Information Systems, 3(2), 125-144.

References

O'Leary, D. (1987). Validation of expert systems with applications to auditing and accounting expert systems. Decision Science, 18(3), 468-486.

Presutti, A. H. (1995). Anchor and adjustment heuristic effect on audit judgement. Managerial Auditing Journal, 10(9), 13-21.

Rose, J. M., & Wolfe, C. J. (1998). Schema acquisition and the effects of cognitive load imposed by decision aid design alternatives: a tax setting. Paper presented at the Proceedings of the Fifth Accounting Information Systems Conference, Scottsdale, AZ.

Silver, M. (1988). Descriptive analysis for computer-based decision support. Operations Research, 36(6), 904-916.

Slovic, P., & Lichtenstein, S. (1971). Comparison of Bayesian and regression approaches to the study of information processing in judgment. Organizational behaviour and Human Performance, 6, 649-744.

Speier, C., & Morris, M. G. (2003). The influence of query interface design on decision-making performance1. MIS Quarterly, 27(3), 397.

References

Sutton, S. G., & Byington, J. (1993). An analysis of ethical and epistemological issues in the development and implementation of audit expert systems. Advanced Public Interest Accounting, 5, 234-243.

Sutton, S. G. (2006). Theory of Technology Dominance: Reflections on the Research to Date and the Mysteries Unexplored. Paper presented at the Digital Accounting Research Conference, University of Huelva, Spain.

Sweller, J. (1988). Cognitive Load During Problem Solving: Effects on Learning. Cognitive Science, 12(2), 257-285.

Todd, P., & Benbasat, I. (1992). The Use of Information in Decision Making: An Experimental Investigation of the Impact of Computer-Based Decision Aids. MIS Quarterly, 16(3), 373.

Todd, P., & Benbasat, I. (1994). The influence of decision aids on choice strategies: An experimental analysis of the role of cognitive effort. Organizational Behavior and Human Decision Processes, 60(1), 36.

Tversky, A., & Kahnemann, D. (1974). Judgment Under Uncertainty: Heuristics and Biases. Science, 185(4157), 1124-1131.

References

Wheeler, B. C., & Valacich, J. S. (1996). Facilitation, GSS, and training as sources of process restrictiveness and guidance for structured group decision making: An empirical assessment. Information Systems Research, 7(4), 429.