why usability works for it audit iacis 2010
DESCRIPTION
TRANSCRIPT
WHY USABILITY TESTING WORKS FOR IT AUDIT
Voraphan Manomuth Raungpaka, PhD Kasetsart University, [email protected]
Usana Patramontree, CPAKasetsart University, [email protected]
Voraphan Manomuth Raungpaka, IACIS Conference, Las Vegas, USA,2010
1Presented by Voraphan M. Raungpaka7 October, 2010
Introduction and Background Research Methodology Results Recommendations & Practitioner’s take away
2
OUTLINE
Voraphan Manomuth Raungpaka, IACIS Conference, Las Vegas, USA,2010
We were expected to perform typical Computer Assisted Audit techniques (CAATs) such as the test data and general audit software techniques.
CAATs requires high computer skill. These techniques are under-utilized in auditing field
(Curtis & Payne, 2008; Celluro, 2003; IIA 2002; AICPA, 1998).
3
Voraphan Manomuth Raungpaka, IACIS Conference, Las Vegas, USA,2010
IN OUR ENGAGEMENT PLAN
• The impact of real users and computer interaction, especially when using a system interface on the data integrity, seems to have been overlooked or superficially commented on.
• It is an obvious risk made in database integrity when input data are incomplete and/ or inaccurate because of mistakes that real users made when using an interface. 4
TYPICAL IT AUDIT TECHNIQUES
5
Voraphan M
anomuth R
aungpaka, IAC
IS
Conference, Las V
egas, US
A,2010
Is used to discover weaknessesIn program controls.
Is used to discover data, database inaccuracy andIncompleteness.
user
ApplicationProgram
Test Data Audit Software
????
Data
Data
What IT techniques should be used to discover usability problems of usability problems of real users?
AN AUDITOR EVER KNOW ABOUT ANY SYSTEM USABILITY PROBLEMS OF REAL USERS?
Auditors seldom know about any system usability problems of real users since auditors are hired to audit company’s
historical data.
6
Voraphan Manomuth Raungpaka, IACIS Conference, Las Vegas, USA,2010
7
Voraphan Manomuth Raungpaka, IACIS Conference, Las Vegas, USA,2010
WHAT WE HAVE GOT FROM PRELIMINARY SURVEYS ?
management and staff were quite satisfied about the various aspects of
FMS, but there were also management and staff with negative user interface
experiences
So…… I came up with the idea of applying usability testing
techniques to gain better evidence on the users’ aspect when using
fund management system in
specified tasks. (In addition to administering the test data and the generalized audit package)
SOLUTION POSTED!
9
Voraphan M
anomuth R
aungpaka, IAC
IS
Conference, Las V
egas, US
A,2010
Is used to discover weaknessesIn program controls.
Is used to discover data, database inaccuracy andIncompleteness.
user
ApplicationProgram
Test Data Audit Software
Data
Data
Is used to discover usability problems of usability problems.
Usability testing?
USABILITY TESTING TECHNIQUE
10
Voraphan M
anomuth R
aungpaka, IAC
IS
Conference, Las V
egas, US
A,2010
User + ability to use ….
A technique used to evaluate a human-made product by testing it on users or acquiring direct information on how people use technology, such as web applications, software applications, computer interfaces, devices, or any other products.
involves watching people using the product to discover errors and areas for improvement.
Usability testing generally involves measuring how well test subjects respond in 5 areas:
1. Efficiency—how long does it take subjects to complete basic tasks?
2. Accuracy —how many mistakes do subjects make? 3. Recall—how much do the subjects remember
afterwards or after periods of non-use? 4. Emotional response—how do the subjects feel
about the task completed? Is the person confident or stressed? Would the user recommend this system to a friend?
5. Learnability —how easy is it for users to accomplish basic task the first time they encounter the design?
11
Voraphan Manomuth Raungpaka, IACIS Conference, Las Vegas, USA,2010
USABILITY TESTING
Usability Testing Techniques
12
Voraphan Manomuth Raungpaka, IACIS Conference, Las Vegas, USA,2010
METHODOLOGY USED TO EVALUATE USABILITY PROBLEMS
Tested participant’s qualification who were familiar with the use of computers and
some other software applications but never trained nor used the FMS software
application. In our pre-test briefing,
introduced them to the application. asked them to think-aloud and do the 10 specified
tasks Carefully observed them during the 60min test
session.
13
Voraphan Manomuth Raungpaka, IACIS Conference, Las Vegas, USA,2010
OBSERVATION
It was developed by using the Venkatesh and Morris, et al. (2003) questions and followed the UTAUT Theory.
This questionnaire consisted of 6parts 1. performance expectancy, 2. effort expectancy, 3. attitude toward using, 4. facilitating conditions, 5. self efficacy, 6. behavior intention
15
Voraphan Manomuth Raungpaka, IACIS Conference, Las Vegas, USA,2010
POST TEST QUSTIONNAIRE
The study results show that user acceptance of FMS software is generally low.
The high number of users who did the tasks unsuccessfully, and who gave up shows the high risk level of inaccuracy and incompleteness of input data.
The mean score of 13.8 minutes shows the inefficiency in using this software.
17
Voraphan Manomuth Raungpaka, IACIS Conference, Las Vegas, USA,2010
RESULTS
PROS AND CONS OF USING THE USABILITY TESTING TECHNIQUES IN IT AUDIT
It is not too difficult to perform,
It is relatively understandable by firm’s management and users ,
It provides ways to measure the users’ attitude and behavior,
the recommendation gained from the ‘Think Aloud’ process is beneficial for further improvement of the software.
it does not provide direct evidences on data inaccuracy and incompleteness;
it only indicates the potential audit risk areas and the risk likelihood.
PROs: CONs:
Voraphan M
anomuth R
aungpaka, IAC
IS
Conference, Las V
egas, US
A,2010
21
We still need other typical computer assisted audit techniques to detect the actual risk and to correct the problem.
help auditors to carry out their IT audit plan and engagement more effectively.
hope that our experience gained from this work can contribute beneficial knowledge to the IT audit profession.
22
Voraphan Manomuth Raungpaka, IACIS Conference, Las Vegas, USA,2010
CONTRIBUTION OF THIS REPORT
PRACTITIONER’S TAKE AWAY Individual usability testing gives you more
information on users’ behavior when using the application’s interface, which we rarely had any explicit audit evidence before.
Individual usability testing supports thinking aloud method, which allows you to understand how participant thinks and feels about whatever he/she is looking at and doing, as he/she goes about his/her tasks.
Individual usability testing gives you almost immediate feedback on your design and you will have much better understanding which parts of your interface are difficult to use.
23
Voraphan Manomuth Raungpaka, IACIS Conference, Las Vegas, USA,2010
PRACTITIONER’S TAKE AWAY (CONT.) The video-recorded test session of the
individual usability testing allows us to refer to what participant did and reacted so you can do an analysis of the video where the user identification of usability problems is emphasized.
Individual usability testing session followed by a individual interview, a focused discussion, enables us to clarify user’s comments made during the test session.
‘Zero users will give zero insights’ (Nielsen, 200b), but the usability testing with only five users will allow you to discover more than 75% of the usability problems in the design.
24
Voraphan Manomuth Raungpaka, IACIS Conference, Las Vegas, USA,2010
THANK YOU! Question?25