usability evaluation method through sus analysis for coin ... · usability explains the procedure...

1

Usability Evaluation Method through

SUS Analysis for Coin Party

UROOJ WAHEED M. SADIQ ALI KHAN HIRA ANWAR KHAN

ASSISTANT PROFESSOR ASSOCIATE PROFESSOR LECTURER

COCIS, PAF-KIET DCS, UoK COCIS, PAF-KIET

Karachi, Pakistan Karachi, Pakistan Karachi, Pakistan

[email protected] [email protected] [email protected]

Abstract

Usability explains the procedure about how a product or system is providing interaction to the user. While user is interacting with an API usability gives a view about the degree of user experience. Usability is about effectiveness, efficiency and the overall satisfaction of the user. The performance and satisfaction metrics are the base line for developing of Usability tasks. For making the methodology to evaluate the system a game name “Coin Party” is chosen. In this paper authors uses different techniques and methods to not just evaluate a design of the API but also factors like; system status visibility, ease of use, control of user, freedom of user, mapping among real world and system.

Keywords: Usability Evaluation Performance Metric, SUS and Self-Reported

Metrics. 1. Introduction

Usability defines the extent to which the system and product is usable to the user,

i.e.; the interaction between system and user. Through usability we can also measure

the degree of experience user achieve while interacting with the system. There are

three major components which describe usability: effectiveness, efficiency and the

overall satisfaction [5]. Evaluation of usability analyzes the grade to which user is

allowed for interaction with the API easily and enjoyably [4]. The design of the

system can be improved by using Usability evaluation methods.

1.1 Testing Software: Game

This paper applies Usability evaluation method on the API of a game named as Coin Party. Coin Party game is coin pushing game based on jungle story. The game is developed by group of Pakistani game developers known as Mind storm studios.

The game features are described below.

International Journal of Pure and Applied MathematicsVolume 118 No. 9 2018, 171-181ISSN: 1311-8080 (printed version); ISSN: 1314-3395 (on-line version)url: http://www.ijpam.euSpecial Issue ijpam.eu

171

2

1.1.1. Complete your Quests: Completing a quest has the ultimate rewards within the game. Each quest will transform an existing table power up in a special way and even unlock new Power Ups.

1.1.2. Compete on the Hall of Fame: This is the ultimate honor in Coin Party. Compete with players across the world to see who can bank the most coins.

1.1.3. Upgrade your Power Ups: When you upgrade your power ups by collecting prize recipes, you end up collecting coins faster and the game play just becomes a lot more fun with wild things happening on the table all the time.

1.1.4. Increase your Level: Level up to access higher level quests and each level gives makes you naturally more powerful with faster coin regeneration, more prizes and more magic chips.

1.1.5. Collect Prizes: Prizes help you unlock quests, upgrade magic chips and you can exchange them for coins and party bucks from the Prizes menu.

1.1.6. Unlock Achievements: Visit the Hall of Fame to see which achievements you can unlock. We will be adding plenty more achievements with new updates.

1.2 Description of Coin Party

When you start the game, it shows that the heaps of gold coins are dropping from

the top onto the dozer which later drops the coins and prizes from the play board. One

can appreciate the entire gambling club involvement with turn haggles machines [1].

To open more catalyst and mysterious chips in the quests, a player will use the earned

prizes and coins in exchange. The players can gather club turn tokens and opening

International Journal of Pure and Applied Mathematics Special Issue

172

3

gold bars that show up on the dozer table and spend them on turning the in-

amusement fortune prize haggle space machine for astounding prizes [8].

2. Method

To evaluate the system we focuses on the design of the system but also additional

features included like ease of use, visibility of game control, user control and freedom

to play game, synchronization between system and the real world. In this study we

have used self-reported metric and on the basis of it we have make our methodology

by looking at SUS Score.

2.1 Evaluation method for Usability

As the demand of the evaluation increases at industrial level, there emerges a need

of evaluation technique which is cost effective and also practically possible to carry

out on large scale. From this technique the system performance can be measured with

respect to predecessors or with competitors.

John Brooke states that: “The measure had to be capable of being administered

quickly and simply, but also had to be reliable enough to be used to make

comparisons of user performance changes from version to version of a software

product” [12]. If the test consists of questionnaire which includes more than 25

questions, firstly this will frustrate the user to complete the test, and if he left it

incomplete then it will be of no use in the evaluation. In return become hassle for

users to perform test. The user gets involved easily in the test which has simple and

less amount of questions and require less time for performing the test. In contrast to

the problems stated above the simple evaluation technique was discovered. This

method provides the global view of subjective assessments of usability.

2.2 System Usability Scale

In 1986 the new system usability evaluation method was created by John Brooke.

From then till now, SUS is considered to be well known rating scale for evaluation of

usability because of its simplicity and cost effectiveness. Through SUS we are able to

gather user’s subjective responses [6]. SUS is pocket friendly i.e. cost effective,

because it consists of questionnaire with 10 questions which requires less resource

with reliable result. SUS is quicker to implement because the template of the

questionnaire is already present you just need to upload it for initiating the work. SUS

gives reliable results even when the sample size is small. It also effectively

categorizes the system on usable and unusable classes. SUS gather the responses of

user in the form of Likert scale. SUS method asks users 10 questions with 5 response

options from strongly disagree to strongly agree.


173

4

The SUS questioner consists of following 10 questions:

System Usability Scale

Strongly Disagree

to

Strongly Agree

1 2 3 4 5

1 I think I would like to play this game frequently. 2 I found the game unnecessarily complex.

3 I thought the game was easy to use.

4 I think that I would need the support of a technical

person to be able to use this game.

5 I found that various options in this game were well

integrated.

6 I thought there was too much inconsistency in this

game.

7 I would imagine that most people would learn to use

this game very quickly.

8 I found the game very cumbersome to use. 9 I felt very confident playing the game.

10 I needed to learn a lot of things before I could get going

with this system.

Table 1: Questions for System Usability Scale

2.3 Calculating Usability Score using SUS

SUS gives a single numeric value which represents the combined usability of the

overall system. For calculating the SUS score the response of each question is

transformed to a number keeping in view the number of question [7], i.e. even number

question or odd number question. Let us assume that the response of user on the scale

value will be “X” then for all even number questions we will be using formula: “5-X”

and for all odd number questions we will be using formula “X-1” .

The answer we get from the above mentioned formula will be in the range of 0 –

40 so we will map this range to 0 – 100 by multiplying the SUS value to 2.5. The 0 –

100 a percentile range rather than percentage.


174

5

Figure 1: System Usability Scale

3. Procedure Used for Evaluation

3.1 Questioner Development For gathering the feedback of user about game and his experience of

interacting with the system we use System Usability scale Template which

consists of 10 questions mentioned in Table 2 [3]. As the SUS questions have

quantitative responses so will get accurate result. This quantitative result will be

helpful in improving the game [13].

3.2 Methodology

The evaluation tests for usability were conceded from the users’ age group 19

– 25, mostly students of all four years of BSCS. The survey was conducted using

Google Form Template for the SUS questionnaire [2]. The reason for choosing

Google Forms is that the test was conducted for the game on MOBILE phone

which also facilitates the user and lessens the hassle of paper pen work. The

Google Forms can be accessible on any MOBILE phone easily by opening the

link on the browser.

The test sessions were conducted in closed environment of PAF – KIET class

room with the help of volunteers in our supervision where the MOBILE phones

were provided to each participant with the initial briefing about the evaluation

test using SUS before playing the game and filling the survey.

3.3 Participants The samples of users are selected for the surveys are from the age group of

18 – 25. 22 participants are selected in total. The selection for the mentioned

above age group was made keeping in consideration that firstly we need to test a

game and secondly the game which we selected to test was available on android


175

6

and iOS i.e. touch screen mobiles/ tablets. So the above mentioned age group

posses both the qualities.

The participants were provided with the android / iOS device. They were

asked to comment on the usability and learn ability of the game. 5 volunteers

were selected for conducting the survey fairly. Firstly the 20 minutes session was

conducted to brief about the game and procedures of the survey to the

volunteers. Then each volunteer is assigned to 1 participant. Only 5 participants

and allowed in one time frame because of limited resources i.e. devices installed

with game. Only those participants are allowed to take part those who are willing

to be a part of this test so as to get the correct data.

3.4 Ethics

Following ethical guidelines are to be followed strictly by the participants of

the survey:

The participants fill in the survey honestly. Participants are required to

provide their age correctly. The participants were briefed about the intention and

method of survey, after the briefing participants have right to decide about

participating in the survey or not. The name of the participants were not

disclosed anywhere [14].

4. Result and Findings In this part we discuss the result and finding of the evaluation technique i.e.

SUS used to test the game: COIN PARTY. The usability goals were kept in

consideration while designing the usability evaluation test [14][15]. There are

three major components used to describe usability: effectiveness, efficiency and the

overall satisfaction [5]. The results of the survey are helpful for drawing attention and

highlighting the issues which needs improvement of game, based on the

recommendation of the respondents for better response in the future.

The usability test objectives are as follows:

Learnability

Various aspects of learnability while playing the game were

studied to know about the load game is putting on the respondent

to memorize the control if he want to use the system in near

future. [2]

Enjoyability Various aspects of user satisfaction while playing the game were

studied to know about the user experience of the game. [4]


176

7

GUI Various aspects of GUI were studied to know about the

consistency of the game design. [5]

Task

Performance

Various aspects of user performance were studied during the game

test to know about the Task Performance of the game. [6]

Task

Success

Various aspects of completion of task were studied during the

game test to know about the failure or successful completion of

Task of the game. [6]

Table 2: Usability Test Objectives

5. Results Analysis

5.1 SUS

25 responses were recorded for this survey.

The game was difficult to play for most of the students as in the start of game

there was no help was given, the options were not clearly mentioned, and even the

controls of the game are not clearly visible to them. Respondents start playing game

by hit and trail method, there was a need for instructions on how to play a game [10].

Figure 2: Responses from 25 students for SUS Questionnaire

Figure 3: Average score of students’ response for each question in SUS

12

2018 17

0

24

4

20

5

15

0

5

10

15

20

25

Q 1 Q 2 Q 3 Q 4 Q 5 Q 6 Q 7 Q 8 Q 9 Q 10

1

2

3

4

5

3.122.61

3.17

1.95

2.75

1.55

3.34

1.33

3.01

0.95

0

1

2

3

4

Q 1 Q 2 Q 3 Q 4 Q 5 Q 6 Q 7 Q 8 Q 9 Q 10

Average SUS Score


177

8

The game was inconsistent from the other games present. The integration between

levels were not also present, there is an ambiguity for the level on which students

were playing. The switching from lower level to upper level in game was not shown

properly [9]. The total lives present to avail for the game was also not mentioned on

the playing screen. The score in term of coin earned was not shown clearly [10]. This

game put burden on the minds of the user as it will be using memory to memorize the

whole options available for the future use [11]. The power jump controls was not

mentioned properly but when you become use to of playing you will get attach with

the game but for the initiators it was hassle to try different options in order to know

about the correct option for that control. The sounds of the system are too loud and

not pleasing. The color of the overall game is also very loud and not appealing.

Adjective Rating Acceptability Grade Scale

Learnability 50 low F

Enjoyability 65 high D

GUI 45 Not acceptable F

Task Performance 80 Acceptable B

Task success 65 High D

Table 3: SUS scores for Factors

6. Conclusion

Result analysis lead us to the conclusion that there are some points which needs improvement in near future to attract more users. These highlighted points are discussed below.


178

9

7. References

7.1. Journal Article J. W. Palmer, "Web site usability, design, and performance metrics," Information Systems Research,

vol. 13, no. 2, pp. 151–167, Jun. 2002.

[1]. A. Bangor, P. T. Kortum, and J. T. Miller, "An empirical evaluation of the system usability scale," International Journal of Human-Computer Interaction, vol. 24, no. 6, pp. 574–594, Jul. 2008.

[2]. A. Sonderegger and J. Sauer, "The influence of design aesthetics in usability testing: Effects on

user performance and perceived usability," Applied Ergonomics, vol. 41, no. 3, pp. 403–410, May

2010.

[3]. S. Wasserkrug et al., "Usability," in Encyclopedia of Database Systems. Springer Science + Business Media, 2009, pp. 3247–3251. [4]. Layla Hasan, Anne Morris, Steve Probets. (2012) A comparison of usability evaluation methods for evaluating e-commerce websites. Behaviour & Information Technology 31:7, pages 707-737. [5]. A. Mouloudi, P. Morizet-Mahoudeaux, A. Valentin. (2011) RAMSES: A Method for the Design Process of Interactive Information Systems. International Journal of Human-Computer Interaction27:2,pages 107-130. [6]. Christos Katsanos, Nikolaos Tselios, Nikolaos Avouris. (2010) Evaluating website navigability:

validation of a tool-based approach through two eye-tracking user studies. New Review of Hypermedia and Multimedia 16:1-2, pages 195-214.

[7]. Kasper Hornbæk. (2010) Dogmas in the assessment of usability evaluation methods. Behaviour &

Information Technology 29:1, pages 97-111.

[8]. Arnold P.O.S. Vermeeren, Jelle Attema, Evren Akar PhD, Huib de Ridder, Andrea J. von

Doorn, Çiğdem Erbuğ, Ali E. Berkman, Martin C. Maguire. (2008) Usability Problem Reports for

Comparative Studies: Consistency and Inspectability. Human–Computer Interaction 23:4, pages 329-

380. [9]. Rajesh, M. & Gnanasekar, J.M. Wireless Pers Commun (2017) 97: 1267. https://doi.org/10.1007/s11277-017-4565-9

[10].Wolmet Barendregt, Mathilde M. Bekker, Don G. Bouwhuis, Esther Baauw. (2007) Predicting effectiveness of children participants in user testing based on personality characteristics. Behaviour & Information Technology 26:2, pages 133-147. [11].Iain Connell, Ann Blandford, Thomas Green. (2004) CASSM and cognitive walkthrough: Usability issues with ticket vending machines. Behaviour & Information Technology 23:5, pages 307-320. [12]. Rolf Molich, Meghan R Ede, Klaus Kaasgaard, Barbara Karyukin. (2004) Comparative usability evaluation. Behaviour & Information Technology 23:1, pages 65-74.

[13]. Deborah Hix and H. Rex Hartsons. Developing User Interfaces: Ensuring Usability through Product and Process. New York: John Wiley and Sons, 1993. 7.2. Book [14]. P. W. Jordan, B. Thomas, I. L. McClelland, and B. Weerdmeester, Usability evaluation in industry. CRC Press, 1996.

[15]. B. Shackel and S. J. Richardson, Human factors for informatics usability. Cambridge University Press, 1991.

7.3. Patent [16]. hide, "Usability metrics," 1998. [Online]. Available:

https://www.nngroup.com/articles/usability-metrics/ . Accessed: Nov. 5, 2016.


179

10

Authors

Urooj Waheed

She is currently a Phd scholar at DCS, UoK, having MSCS (2016) in

Computer Science with specialization in Human Computer Interaction

and Intelligent System. She is currently working as Assistant Professor in

College of Computing and Information Sciences of Pakistan Air Force -

Karachi Institute of Economics and Technology and also as visiting faculty in

Department of Computer Science – UBIT. Her main research interests are

Security, Computer Networking, Human Computer Interaction.

Dr. M. Sadiq Ali Khan

He is Chairman and an Associate Professor of Department of Computer

Science – UBIT. He completed his Ph.D in 2011 under the supervision of

Prof. Dr. S.M. Aqil Burney. He had earned his Bachelors and Masters degree

from Sir Syed University of Engineering and Technology. His main areas of

interests are Network Security, Computer Networking, Software Engineering,

Educational Administration, Security Information Technology.

Hira Anwar Khan

She is having MS (2016) in Computer Science with specialization in

Human Computer Interaction and Intelligent System. She is also an MBA

in Supply Chain Management (2014) and MCS in Computer Science. She

is currently working as Lecturer at College of Computing & Information

Science – CoCIS, Pakistan Air Force - Karachi Institute of Economics and

Technology. She is working as academia for about 6 years. Has

researched different domains from CS. Research interest include interaction Design,

Information Systems, Enterprise Integration.


180

usability evaluation method through sus analysis for coin ... · usability explains the procedure...

Documents