a comparative study of user acceptance testing...

A COMPARATIVE STUDY OF USER

ACCEPTANCE TESTING BETWEEN MODIFIED

WATERFALL MODEL AND EXTREME

PROGRAMMING IN SMALL-SCALE PROJECT

IDA ARYANIE BAHRUDIN

UNIVERSITI TUN HUSSEIN ONN MALAYSIA

UNIVERSITI TUN HUSSEIN ONN MALAYSIA

STATUS CONFIRMATION FOR MASTER’S DISSERTATION

A COMPARATIVE STUDY OF USER ACCEPTANCE TESTING BETWEEN

MODIFIED WATERFALL MODEL AND EXTREME PROGRAMMING IN

SMALL-SCALE PROJECT

ACADEMIC SESSION: 2015/2016

I, IDA ARYANIE BINTI BAHRUDIN, agree to allow this Master’s Dissertation to be kept at the

library under the following terms:

1. This Master’s Dissertation is the property of the Universiti Tun Hussein Onn Malaysia.

2. The library has the right to make copies for educational purposes only.

3. The library is allowed to make copies of this report for educational exchange between higher

educational institutions.

4. ** Please Mark (√)

CONFIDENTIAL

RESTRICTED

FREE ACCESS

Approved by,

(WRITER’S SIGNATURE) (SUPERVISOR’S SIGNATURE)

Permanent Address:

NO 1C, JALAN LAMA, DR. NORAINI BINTI IBRAHIM

83700 YONG PENG, JOHOR

Date: ____________________ Date: ____________________

NOTE:

** If this Master’s Dissertation classified as CONFIDENTIAL or

RESTRICTED, please attach the letter from the relevant

authority/organization stating reasons and duration for such classifications.

(Contains information of high security or great

importance to Malaysia STIPULATED under the

OFFICIAL SECRET ACT 1972)

(Contains restricted information as determined by the

Organization/institution where research was

conducted)

This dissertation has been examined on date ......................... and is sufficient in

fulfilling the scope and the quality for the purpose of awarding the Degree of Master

of Computer Science (Software Engineering).

Chairperson:

PROF. DR. ISMAIL BIN ABDUL RAHMAN

Faculty of Civil and Environmental Engineering

DR. NOOR AZAH BINTI SAMSUDIN

Faculty of Computer Science and Information Technology

Universiti Tun Hussein Onn Malaysia

Examiners:

DR. HJ. MOHD ZAINURI BIN SARINGAT



MDM. RUHAYA BINTI AB. AZIZ



12th June 2016

ii

A COMPARATIVE STUDY OF USER ACCEPTANCE TESTING BETWEEN

MODIFIED WATERFALL MODEL AND EXTREME PROGRAMMING IN

SMALL-SCALE PROJECT


A dissertation submitted in partial

fulfillment of the requirement for the award of the

Degree of Master of Computer Science (Software Engineering)



JULY 2016

iii

STUDENT DECLARATION

I hereby declare that the work in this dissertation is my own except for quotations

and summaries which have been duly acknowledged

Student : ……………………………………


Date : ……………………………………

Supervisor : ……………………………………

DR. NORAINI IBRAHIM

iv

To Ezree, for the understanding, endless love and care

v

ACKNOWLEDGEMENT

In the Name of Allah, the Most Gracious, the Most Merciful, praise to Allah, the

Almighty, for granting me the opportunity to complete this study successfully. I

believe that I would not have been able to complete this journey without the help and

support of countless people over the past years. Hereby, I would like to express my

sincere gratitude to my supervisor, Dr Noraini Ibrahim, Faculty of Computer Science

and Information Technology, Universiti Tun Hussein Onn Malaysia for their excellent

guidance, enthusiasm, patience, and providing me with an excellent atmosphere for

doing this research. I have learned tremendously from her and her guidance on

practising excellent skills of writing articles, analysing and interpreting test results, as

well as polishing my presentation skills. I would also like to appreciate the member of

my research team, Mrs. Rafizah Mohd Hanifa for her helpful assistance and advice.

Thanks are extended to Mr. Muhammad Firdaus Kamarudin, Mr. Muhammad Aqil

Abdul Razak and Miss Nor Syafeeqa Abdullah Syukor for their assistance throughout

the research work. Most importantly, thank you to my dear husband, Mohd Ezree

Abdullah, for showing me the beauty and opposite side of life and calming me during

the hard times of the research work. Special thanks to all my beloved children, Durrah

Hannani, Farhah Amni, Nusaybah Auni and Ukasyah, for their patience and

understanding. Finally, I sincere wish to my beloved mother and father for their

endless love and prayers.

vi

ABSTRACT

The successfulness of software development relies on many factors but the selection

of software development methodology plays a crucial role in order to complete the

project hence to get high quality software. Since the advent of traditional waterfall

model, software development methodology is still evolving. There are two categories

of software development methodologies that had been implemented recently

considered as traditional and Agile methodology. At present, many researchers had

conducted comparative studies on different agile software development

methodologies. However, literature review had shown few studies investigating the

comparison of software development methodologies especially in software testing

between traditional methods and Agile modeling. Therefore, this research focused in

details on the comparison of user acceptance testing (UAT) in Modified Waterfall

model with eXtreme Programming (XP) methodology. The objectives of the research

are to perform UAT to the system modules in two case studies, to identify project size

and defects in two case studies and to carry out comparative analysis of the error

discovery rate from UAT result of Modified Waterfall and XP for selected case studies.

A Mann-Whitney U test was performed on XP and Modified Waterfall to compare the

difference between error discovery rate means. The analysis shows that there is no

significant difference of error discovery rate between both methodologies in the

selected case studies. The main contribution of this work is to bring empirical evidence

from real life cases that contribute understanding the adoption of both methodologies

in small scale project. This study concluded that one of the factor of choosing the

software methodology that best fit to the organization is depending on the individual

project and circumstances.

vii

ABSTRAK

Kejayaan pembangunan perisian bergantung kepada pelbagai faktor. Namun begitu,

pemilihan metodologi pembangunan perisian memainkan peranan yang sangat penting

untuk menyelesaikan projek seterusnya dapat menghasilkan perisian yang berkualiti.

Semenjak kemunculan model tradisional Air Terjun, metodologi pembangunan

perisian sentiasa berkembang. Terdapat dua kategori metodologi pembangunan

perisian yang telah dilaksanakan hingga kini iaitu metodologi tradisional dan Agil.

Pada masa kini, kebanyakan penyelidik melaksanakan kajian perbandingan terhadap

metodologi pembangunan Agil yang berlainan. Walaubagaimanapun, kajian literatur

menunjukkan bahawa masih kurang kajian berkaitan perbandingan metodologi

pembangunan sistem antara kaedah tradisional dengan permodelan Agil khususnya

dalam pengujian perisian. Oleh itu, kajian ini memfokus secara terperinci

perbandingan ujian penerimaan pengguna (UAT) antara model Air Terjun

Terubahsuai dengan Pengaturcaraan eXtreme (XP). Objektif penyelidikan ini adalah

untuk melaksanakan UAT terhadap modul sistem dalam dua kajian kes, untuk

mengenalpasti saiz projek dan kesilapan dalam kajian kes yang dipilih serta membuat

analisis perbandingan kadar penemuan ralat daripada keputusan UAT bagi kajian kes

model pembangunan Air Terjun Terubahsuai dan XP untuk kajian kes yang dipilih.

Ujian Mann-Whitney U telah dilaksanakan terhadap model Air Terjun Terubahsuai

dan XP untuk membandingkan perbezaan purata kadar penemuan ralat. Analisis

menunjukkan tiada perbezaan yang signifikan antara purata kadar penemuan ralat bagi

kedua-dua metodologi pembangunan untuk kajian kes yang dipilih. Sumbangan utama

kajian ini adalah untuk menunjukkan bukti empirikal daripada kes sebenar yang

menyumbang kepada pemahaman berhubung penggunaan kedua-dua metodologi

dalam projek berskala kecil. Kajian ini merumuskan bahawa salah satu faktor

pemilihan metodologi pembangunan sistem yang sesuai untuk organisasi adalah

bergantung kepada sesuatu projek itu sendiri dan kekangan dalam pembangunannya.

viii

CONTENTS

TITLE i

STUDENT DECLARATION iii

ACKNOWLEDGEMENTS v

ABSTRACT vi

ABSTRAK vii

CONTENTS viii

LIST OF TABLES xii

LIST OF FIGURES xiv

LIST OF SYMBOLS AND ABBREVIATIONS xvi

LIST OF APPENDICES xvii

LIST OF AWARDS AND PUBLICATIONS xviii

CHAPTER 1 INTRODUCTION 1

1.1 Research Background 1

1.2 Problem Statement 3

1.3 Research Objectives 4

1.4 Research Scope 4

1.5 Research Hypothesis 4

1.6 Significance of Research 5

1.7 Dissertation Outline 5

1.8 Chapter Summary 6

ix

CHAPTER 2 LITERATURE REVIEW 7

2.1 Introduction 7

2.2 Software Development Methodology 8

2.2.1 Heavyweight Methodology 9

2.2.2 Lightweight Methodology 13

2.2.3 Comparison between Heavyweight and

Lightweight Methodology 18

2.3 Software Testing 20

2.3.1 Unit Testing 21

2.3.2 Integration Testing 22

2.3.3 System Testing 22

2.3.4 User Acceptance Testing (UAT) 22

2.4 Designing User Acceptance Testing 23

2.4.1 The Importance of UAT 24

2.4.2 Testing Team 26

2.4.3 Designing Test Plan 27

2.4.4 Test Cases Design 28

2.4.5 Test Execution 31

2.4.6 Software Defects 31

2.5 User Acceptance Testing in Modified Waterfall

and Agile 33

2.5.1 UAT in Agile 34

2.5.2 UAT in Modified Waterfall Model 39

2.6 Comparison of UAT in Modified Waterfall and

eXtreme Programming 42


CHAPTER 3 RESEARCH METHODOLOGY 46

3.1 Introduction 46

x

3.2 Research Plan 47

3.2.1 Phase 1 49

3.2.2 Phase 2 51

3.2.3 Phase 3 53

3.2.4 Phase 4 55

3.2.5 Phase 5 56


CHAPTER 4 ANALYSIS AND DESIGN 58

4.1 Description of Modified Waterfall Model and

XP Project 58

4.2 Case Study 1 58

4.2.1 Analyzing Functional Requirements

of eDoc (Modified Waterfall model) 60

4.2.2 Test Case Design for eDoc (Modified

Waterfall Model) 66

4.2.3 Analyzing User Stories of eDoc (XP

Methodology) 69

4.2.4 Test Case Design for eDoc (XP

Methodology) 76

4.3 Case Study 2 85

4.3.1 Functional Requirements of PiTOS

(Modified Waterfall Model) 87

4.3.2 Test Case Design for PiTOS (Modified

Waterfall Model) 93

4.3.3 User Stories for PiTOS (XP

Methodology) 98

4.3.4 Test Case Design for PiTOS

(XP Methodology) 104


xi

CHAPTER 5 IMPLEMENTATION AND RESULT 112

5.1 Criteria for Case Selection 112

5.1.1 Project Size 112

5.1.2 Team Size 114

5.1.3 Duration 114

5.2 Test Case Execution for Modified Waterfall

Project 115

5.3 Analysis of Requirements and Defects in

Modified Waterfall Project 119

5.4 Test Case Execution for XP Project 121

5.5 Analysis of User Stories and Defects in XP Project 124

5.6 Comparison Between Modified Waterfall and XP

Projects 126

5.6.1 Rank Table 127

5.6.2 Test Statistic Table 127


CHAPTER 6 CONCLUSION AND RECOMMENDATIONS 130

6.1 Objectives Achievement 130

6.2 Conclusion 130

6.3 Future Works 132

6.4 Summary 132

REFERENCES 132

APPENDICES 142

xii

LIST OF TABLES

Table 2.1 Advantages and disadvantages of Modified Waterfall model 13

Table 2.2 Description of the Agile Methodology 15

Table 2.3 Advantages and disadvantages of XP 18

Table 2.4 Comparison of heavyweight and lightweight methodologies 19

Table 2.5 Summary of comparative study from different authors 20

Table 2.6 Traceability in user story 39

Table 2.7 Requirement traceability 42

Table 4.1 Functional requirements of eDoc 65

Table 4.2 Test case TCCS1MW100 description 66



Table 4.5 Test Case TCCS1MW400 description 68


Table 4.7 User authentication user story and acceptance criteria 70

Table 4.8 Add new folder user story with acceptance criteria 70

Table 4.9 Insert file user story with acceptance criteria 71

Table 4.10 View report user story with acceptance criteria 72

Table 4.11 View file user story with acceptance criteria 73

Table 4.12 Create folder or subfolder user story with acceptance criteria 75

Table 4.13 User story of additional requirement for user authentication 75

Table 4.14 Test case for user authentication 77

Table 4.15 Test case for add new folder 78

Table 4.16 Test case for insert file 79

Table 4.17 Test case for view report 80

Table 4.18 Test case for view files 81

Table 4.19 Test case for Iteration 1 83


xiii

Table 4.21 Summary of functional requirements of PiTOS 92







Table 4.28 User authentication user story 98

Table 4.29 User story for create industrial trainee profile 99

Table 4.30 View all results user story 100

Table 4.31 Modify personal information user story 100

Table 4.32 View result user story 101

Table 4.33 User story of submit evaluation feature 102

Table 4.34 User story and acceptance criteria for Iteration 1 103

Table 4.35 User story and acceptance criteria for Iteration 1 103

Table 4.36 Test case for user authentication 105

Table 4.37 Test case for creating industrial trainee profile 106

Table 4.38 Test case for view all results 107

Table 4.39 Test case for modify personal information 108

Table 4.40 Test case for view result 108

Table 4.41 Test case for submit evaluation 109



Table 5.1 Project size 114

Table 5.2 Result summary of Test case TCCS1MW100-TCCS1MW500 115

Table 5.3 Result summary of Test case TCCS2MW100-TCCS2MW600 117

Table 5.4 Result summary for test case TCCS1XP000 to TCCS1XP200 121

Table 5.5 Result summary for test case TCCS2XP000 to TCCS2XP200 122

Table 5.6 Project size and defects number for both Modified

Waterfall and XP 126

Table 5.7 Error discovery rate for each project 127

Table 5.8 Rank table 127

Table 5.9 Test statistic table 128

xiv

LIST OF FIGURES

Figure 2.1 Linear Sequential Model 10

Figure 2.2 Waterfall model 11

Figure 2.3 Modified Waterfall model 12

Figure 2.4 XP flow chart 16

Figure 2.5 Iterative project with possible documentation artifacts in Agile 17

Figure 2.6 Software testing strategy 21

Figure 2.7 UAT steps 24

Figure 2.8 Test case sample Error!

Bookmark not defined.

Figure 2.9 Software defects categories 32

Figure 2.10 Types of error and defect 33

Figure 2.11 AgileUAT framework 35

Figure 2.12 Sample of user story 36

Figure 2.13 Important consideration of writing user stories 37

Figure 2.14 Sample of user story with acceptance criteria 38

Figure 2.15 Requirement gathering techniques 40

Figure 2.16 SRS content 41

Figure 2.17 Comparison of Modified Waterfall and XP methodologies 44

Figure 3.1 Research activities 48

Figure 3.2 Use case for eDoc 50

Figure 3.3 Use case diagram of PiTOS 52

Figure 3.4 Index card 54

Figure 4.1 eDoc Framework 59

Figure 4.2 Process of using eDoc 60

Figure 4.3 User authentication use case 61

Figure 4.4 Use case for add new folder 62

Figure 4.5 Insert file use case 63

xv

Figure 4.6 View report use case 64

Figure 4.7 View file use case 65

Figure 4.8 PiTOS Workflow 86

Figure 4.9 PiTOS architecture 86

Figure 4.10 Login page use case 87

Figure 4.11 Create industrial trainee profile list use case 88

Figure 4.12 View all results use case 90

Figure 4.13 Modify personal information use case 90

Figure 4.14 View result use case 91

Figure 4.15 Submit evaluation use case 91

Figure 5.1 Size of selected project using Modified Waterfall model 113

Figure 5.2 Size of selected project using XP methodology 113

Figure 5.3 Project duration of selected Case Study 115

Figure 5.4 Proportions of functional requirement defects in eDoc 119

Figure 5.5 Proportions of functional requirement defects in PiTOS 120

Figure 5.6 Proportions of functional requirement defects in eDoc

and PiTOS 120

Figure 5.7 Proportions of functional requirement defects for each

iteration in eDoc 124

Figure 5.8 Proportions of functional requirement defects for each

iteration in PiTOS 125

Figure 5.9 Proportion of defects and user stories for eDoc and PiTOS 125

Figure 5.10 Mean number of defects for Modified Waterfall and XP 126

xvi

LIST OF SYMBOLS AND ABBREVIATIONS

XP - eXtreme Programming

UAT - User Acceptance Testing

ID - Identification

CeDS - Centre for Diploma Studies

eDoc - Electronic Document Management System

PiTOS - CeDS Industrial Online System

SDLC - Software Development Life Cycle

IS - Information system

SDM - Software development methodology

α - Constant

p - Probability Value

Ho - Null Hypothesis

xvii

LIST OF APPENDICES

APPENDIX TITLE PAGE

A Computer Software Configuration Item (CSCI) 142

Capability Requirement for Electronic Document

Online System (eDoc)

B Computer Software Configuration Item (CSCI) 151

Capability Requirement for Centre for Diploma Studies

Industrial Training Online System (PiTOS)

C Test Cases Result Details for Modified Waterfall 161

Case Study

D Test Cases Result Details for XP Case Study 171

xviii

LIST OF AWARDS AND PUBLICATIONS

Awards

1. 1st Prize of Technical Category for the product Electronic Document

Management System (e-Doc) at Innovative & Creative Circle Project

Presentation, UTHM 2014.

2. Silver Award Category for the product Electronic Document Management

System (e-Doc) at 10th Innovative & Creative Circle Convention, Universiti

Malaysia Sarawak, 2014.

3. Copyright for Electronic Document Management System (e-Doc).

4. Silver Medal Awarded at Research and Innovation Festival for the product

Centre for Diploma Studies Industrial Training Online System (PiTOS)

organized by Pusat Penyelidikan dan Inovasi, Universiti Tun Hussein Onn

Malaysia, 19-20 Disember 2013.

Journal Papers

1. Rafizah Mohd Hanifa, Shamsul Mohamad, Ida Aryanie Bahrudin and

Muhammad Firdaus Kamarudin “Development of Pusat Pengajian Diploma

Industrial Training Online System (PiTOS): A Step Forward” Applied

Mechanics and Materials Vols. 321- 324 (2013) pp 2528-2531 (Scopus

index)

2. Ida Aryanie Bahrudin, Rafizah Mohd Hanifa, Mohd Ezree Abdullah,

Muhammad Firdaus Kamarudin, “Adapting eXtreme Programming

Approach in Developing Electronic Document Online System (eDoc),”

Applied Mechanics and Materials Vols. 321-324 (2013) pp 2938-2941©

(2013) Trans Tech Publications, Switzerland

doi:10.4028/www.scientific.net/AMM.321-324.2938 (Scopus index)

CHAPTER 1

INTRODUCTION

The aim of this chapter is to emphasize the key components of this dissertation. It

provides a clear statement of the research topic which includes the research

background, problem statement, research objectives, scope, hypothesis and also the

significance of this research.

1.1 Research Background

The emergence of information system (IS) development history began in the year

1940. It was a shift from scientific or mathematical application to business application

where the viewpoint of software was more on general view [1]. IS development was

based on information technology (IT) pioneers’ individual knowledge up to the 1960s

thus the period is referred as pioneer era or heroic age [2]. A systematic software

development approach makes the first appearance when the software crisis occurred.

It was known earlier as approach to problem solving in other disciplines which was

applied to systems development at the moment. In 1970s, the structured approach

based on waterfall cycle was introduced [3]. Waterfall model is a method which the

software can be developed in a systematic manner by improving the probability of

completing the software project within the deadline while maintaining the quality of

the software product to conform certain standard [4].

Recently, there are two categories of software development methodologies

that had been implemented which considered as heavyweight and lightweight

methodologies. Heavyweight methodologies are known as the traditional methodology

of software development while lightweight methodologies can be considered as agile

2

modeling [5]. Heavyweight methodologies are basically based on a sequential series

of steps like requirements definition, solution building, testing and deployment [6]. It

is also requires defining and documenting a stable set of requirements at the beginning

of a project. The examples of software development approach that fall in heavyweight

category are waterfall model, prototyping and spiral model while the lightweight

methodologies consist of eXtreme Programming (XP) and Scrum [5]. Agile modeling

are stressed on two concepts which are the intolerant honesty of working code and the

effectiveness of people working together with goodwill [7]. Agile methodologies are

also focusing on the ability to embrace changes which are very likely to happen in

environments that lack predictability.

Software development methodologies are considered as key tools for all

software developers and also project manager in order to facilitate them in delivering

projects on time, within budget constraints while at the same time accomplish

customer’s requirements. The emergence of various methodologies provides the

developer the numerous choices to develop software that is able to produce satisfactory

results. A successful software development depends sturdily on the delivery of high

quality software attributes such as customer satisfaction [8]. Accordingly, one of the

first decisions that faced by the software developers for each of their project

implementations is which development methodology should they use because every

type of methodologies has their own uniqueness. This is a topic that gets a lot of

discussion yet open into ongoing debate regarding what methodologies suit to what

types of software development [9]. Both traditional and agile methodologies are

usable and established methodologies that have been involved in software

development projects for many years ago. Therefore, the decision of which

methodology to be utilized can refine the overall process that best suit to the project

goals. Eventually, although there are slight distinction between both methodologies in

terms of the software developing process, delivering software that satisfies customer

needs is what really takes into account.

During software development, there are number of phases involved throughout

the processes which begin with requirements capture and ending with maintenance

activities. Among all the phases involved, 50% of the development time is spent on

testing [10, 11]. According to [12], software testing is a resource-hungry, time-

consuming and also labor intensive work to be done. In other words, software testing

is a process which is used to measure the quality of software developed. There are

3

many types of software testing such as unit testing, functional testing, system testing,

integration testing, regression testing, usability testing and also user acceptance testing

(UAT) [13].

Among all the types of software testing, researchers [14, 15] found that the

UAT is a kind of software testing that can identify the successfulness of the developed

software because the risk of software failure will increase without UAT. It can be said

that UAT need to be conducted in order to test whether the developed software had

met the user requirement [16]. However, the UAT in waterfall model is slightly

different as compared to the agile method in terms of the way to perform the testing.

The discovery on the result of UAT in selected both traditional and agile can contribute

to a better consideration factors of choosing the software methodology to employ.

Therefore, explorations in detail on UAT in both methodologies can improve better

understanding and guarantee something beneficial findings in the future.

1.2 Problem Statement

At present, many researchers had conducted comparative studies on different agile

software development methodologies [8, 17, 18]. Moniruzzaman [18] stated that Agile

development approach improves software development process to meet the rapid

changing business environments while, traditional plan-driven developments fail to

meet up these requirements. According to Mahalakshmi [19], Scrum Methodology

have the great potential to meet the new requirements of the software development

companies. On the other hand, researchers [20] believed that waterfall model is best

suit for large scale software project development while XP is fit for small scale project.

Despite a number of previous studies, literature review had shown few studies

investigating the comparison of software development methodologies especially in

software testing between traditional methods and Agile modeling [18, 21]. Therefore,

this research focused in details on the comparison of user acceptance testing in

Modified Waterfall model with XP methodology.

4

1.3 Research Objectives

The aim of this study is to conduct the comparative analysis of UAT between Modified

Waterfall model and XP in small-scale project. In order to achieve this aim, the

objectives of the research are to:

(i) perform user acceptance testing to the system modules in selected case studies;

(ii) identify project size and defects in selected case studies;

(iii) carry out comparative analysis of the error discovery rate from UAT result of

Modified Waterfall and XP for selected case studies.

1.4 Research Scope

This research aims to analyze the output of UAT for the software developed using XP

as compared to the software developed using Modified Waterfall model. This study

involved two software that developed using Modified Waterfall which later

redeveloped using XP. The UAT result of each case study had been analyzed in order

to calculate its error discovery rate. This study focused on the XP practices mainly in

pair programming and on-site customer involvement for developing the software.

1.5 Research Hypothesis

The followings are the hypothesis of this research:

(i) H1: Error discovery rate in Modified Waterfall project is higher than the XP.

H1: μModifiedWaterfall Error Discovery Rate > μXP Error Discovery Rate

(ii) H0: Error Discovery Rate in Modified Waterfall projects is the same or lower

than the XP.

H0: μModifiedWaterfall Error Discovery Rate ≤ μXP Error Discovery Rate

where μXP Error Discovery Rate is the ratio of defects per test cases on XP

projects and μModifiedWaterfall No. of Defects is the ratio of defects per test

cases on Modified Waterfall projects.

5

1.6 Significance of Research

XP has recently become one of the major interests among researchers, software

developer and programmers in terms of new software development approach. Many

studies have been conducted on XP and most studies suggested XP as one of the best

agile methods that focusing on customer requirement [22] by bringing customer into

the development process. Studied by Woit [23] shown that XP project is built for the

customers, in accordance with their needs that stated during requirement gathering

process. Despite the importance of the on-site customer requirement, few XP

researches stated that XP's most problematic feature is the amount of on-site customer

involvement it requires [24]. There were also many studies regarding the advantages

of traditional method as compared to agile. According to Balaji [25], requirement

definition is clearly stated before development starts and each phases is properly

documented in waterfall model.

In order to measure how compliant the software with the business

requirements, UAT were performed. Since the issues regarding the factor of selecting

the suitable software development methodology that will increase the probability of

achieving software that conform the requirement has long been debated and is still

being discussed, more research in this area is required. Therefore, comparative study

between traditional methodology and agile methodology is required because

challenges in software development are somewhat different from other organizations

due to the cost, duration and also the involvement of stakeholder. Moreover, this study

provides the comparison of UAT between Modified Waterfall model and XP

especially in small-scale project in terms of its error discovery rate.

1.7 Dissertation Outline

This dissertation is organised into three chapters and each chapter consists of several

subsections, beginning with a brief of introductory passage and ended with the

research methodology. The dissertation structures begin with introduction in Chapter

1. Chapter 1 presents the overall perspectives related to this research, reviews of the

research background and description of the objectives. Chapter 2 gives a

comprehensive review from prior researches and practical experienced among

6

researchers and software developer while implementing both traditional and agile

software development methodology. Inside this chapter, the advantages and

disadvantages of traditional methodology especially in Modified Waterfall model and

agile modelling that mainly focus in XP were identified clearly. In addition, the

exploration regarding UAT in both methodologies were explained in detail including

the similarities, distinction and also the way of performing the test. In Chapter 3, there

were two case studies involved where UAT were performed to both case studies in

order to accomplish the objectives throughout the research. This chapter explains the

activities of each phases of research work plan including materials, supporting tools,

standards, and the software testing procedures applied in this research. Chapter 4

consists the analysis and design of all the test cases in the case studies. The

implementation and results of the UAT were described in Chapter 5 while the last

chapter concludes the research work along with the future recommendation for this

study.

1.8 Chapter Summary

This chapter consists of explanations about research background and also addresses

the problem, objectives and scope of this research as well as the importance of

performing comparative analysis of error discovery rate from UAT between XP and

Modified Waterfall model.

7

CHAPTER 2

LITERATURE REVIEW

The overall goals of this chapter were to elaborate a comprehensive summary of the

research particularly on UAT in both Modified Waterfall model and XP methodology.

This chapter described the comparative study of software development approach

between Agile and traditional methodology that performed by prior researchers. This

chapter was also highlighted the framework that used by Modified Waterfall Model

and XP in performing the UAT.

2.1 Introduction

Nowadays software has been an important part of the modern society. Since the

computer technology is rapidly increased, a great attention has been given to ensure

that the software program will perform as best as the computer hardware that it runs

on. Software has gone through the continuous redesign and also upgrading process to

adapt with the technology and also specification changes. Over the years, software

expertise or even software developer had introduced the numbers of rules, best

practices and also guidelines for developing the software. Research had shown that

software development can be considered as a complicated task [26-29]. According to

Yu [6], selecting software development methodology (SDM) is the most important

element in software development as it portrays the necessary phases in software

development. It can be said that the selection of software development methodology

is very important in order to deliver the required quality for business success.

There are number of phases involved throughout the software development

processes which begin with requirements capture and ending with maintenance

8

activities. Among all the phases involved in SDM, software testing is the most crucial

part. Basically, it is more than a half of the whole development process time is spent

on testing [10, 11]. There is a bit different way of performing software testing in

different type of SDM. This chapter provided a detailed explanation of software testing

particularly in UAT for Modified Waterfall model and XP.

2.2 Software Development Methodology

Basically, SDM is a process of developing or maintaining the software. It is also

considered as creation of a program by generating code along with various tools. It is

a challenge process when choosing the suitable SDM that really suits with the software

development situation because we have to ensure that the development is completed

on time and within reasonable budget. It is really important to select the methodology

wisely so that it is sufficient to deliver the quality required for business success while

avoiding the steps that can waste time. Every step taken along the software

development has its own risk and variety of available techniques for improving

development process and increase the quality of result.

The term SDM can be considered as a framework or as an approach in

developing software. A SDM is a framework purposely for structuring, planning and

controlling the software development process. Usually SDM includes the pre-

definition of specific deliverables that are created and completed by a project team to

develop or maintain an application [1]. In addition, the term SDM refers to an approach

used to apply the software development methodology framework. Every SDM

approach performs as a foundation for applying specific frameworks to develop and

maintain software. There are several software development approaches that have been

used since the origin of information technology. Broadly these are several established

SDM approaches [30]:

(i) Waterfall: a linear framework

(ii) Spiral: a combined linear-iterative framework

(iii) Incremental: a combined linear-iterative framework or V Model

(iv) Prototyping: an iterative framework

(v) Rapid application development (RAD): an iterative framework

(vi) Scrum: iterative and incremental agile SDM

9

(vii) eXtreme programming: is a pragmatic approach to program development

(viii) Adaptive software development (ASD): is a principle for creating software

that focuses on quickly making programs

(ix) Dynamic system development method (DSDM): is a robust Agile project

management and delivery framework

Basically all the SDM are categorized into two categories that known as heavyweight

methodology that is also known as traditional methodology and lightweight

methodology or called agile methodology [31].

2.2.1 Heavyweight Methodology

Software methodologies like Waterfall model, V-Model and Rational Unified Process

(RUP) are called traditional SDM and these are classified into the heavyweight

methodologies [32]. These methodologies are based on a sequential series of steps like

requirements definition, solution building, testing and deployment. Heavyweight

methodologies require defining and documenting a stable set of requirements at the

beginning of a project [6]. There are four main phases involved in heavyweight

methodology which consist of project requirements set up and determine the time

duration that will take to implement the various phases of development while at the

same time trying to forecast any troubles that may occur in the project.

After gathering the requirements, the next step is to design the architectural

planning phase in the form of diagrams or models. The next step is the development

phase where code is produced until the specific goals are reached. Development is

often broken down into smaller tasks that are distributed among various teams based

on skill. Usually the testing phase often overlaps with the development phase to ensure

issues are addressed early on. Once the project is nearly complete, the customer will

become part of the testing process and the project will be delivered after the customer

satisfies with it [6].

The heavyweight SDM are dependent on a set of predetermined processes and

also on-going documentation which is written as the work progresses and guides

further development [32]. The successful factor of a project depending on the

understanding of the requirements at the early phase of development and it means that

any requirement changes during the development phase can be somewhat problematic.

However by unpermitted requirement to change it makes easier to determine the costs

10

of the project set a schedule and allocate resources. The followings are the

characteristics of heavyweight SDM [20]:

(i) Heavyweight SDM is usually used to develop the traditional projects that use

procedural programming.

(ii) It also uses common processes likes: analysis, design, implementation, and

testing.

(iii) The implementation of heavyweight SDM depends on the size of projects and

type of projects

(iv) It sometimes needs the longer duration to develop the large projects.

Heavyweight SDM is considered as the linear sequential model. The linear

sequential model suggests a sequential approach to software development that starts at

the system level and progresses through analysis, design, coding, testing, and support

[33]. Figure 2.1 illustrates the linear sequential model for software engineering.

Figure 2.1: Linear Sequential Model [33]

As mentioned before, there are various SDM that fall into this category but this

research only focused on Modified Waterfall model.

2.2.1.1 Modified Waterfall Model

Historically, the waterfall model is the classical model of software engineering which

is widely used by the software developer. This model emphasizes proper planning at

the beginning of the stages and ensures there is no design fault before the developing

process [34]. Furthermore waterfall model does not permit any changes of requirement

in between the phases of development. The phases in waterfall model consist of six

Analysis Design Code Test

System/information

engineering

11

phases which are analysis, design, development, testing, implementation and

maintenance as shown in Figure 2.2.

Figure 2.2: Waterfall model [25]

Nowadays, the dynamic business environment makes organizations to

constantly change their software requirements in order to adjust with new

environment. The demand for fast delivery of software products is rapidly increase as

well as accepting changing requirements [18]. This aspect contributes to the failure of

meet up user requirement in this plan-driven development. Therefore, Modified

Waterfall model was introduced by Winston W. Royce which improvised the

sequential model or waterfall model that he developed before [2]. The Modified

Waterfall uses the same phases as the pure waterfall, but is not based on a

discontinuous basis. This enables the phases to overlap when needed which is

unacceptable in pure waterfall model [34].

In the Modified Waterfall model, each phase influences and depends on the

next and the previous phase respectively as shown in Figure 2.3. Verification and

validation process have been added to each of the phases making it is more flexible.

As a result, the defects and faults in software are removed in the development stage

which will reduce the overhead cost of making changes to a software project before

implementation stage [35].

Analysis

Design

Development

Testing

Implementation

Maintenance

12

Figure 2.3: Modified Waterfall model [2]

The overlap process contributes to give a lot of flexibility to this methodology

because at the same time a number of tasks can function concurrently. The other

advantage of this methodology is a more relaxed approach if compared to the

traditional waterfall model. There is also drawback found while implementing

Modified Waterfall model such as the flexibility of turning back to previous phase will

expand the allocation time of software development process. The development team

may be bored to always moving back and forth between the phases to ensure the

software error will be fixed early. Table 2.1 shows the advantages and disadvantages

of Modified Waterfall model.

System

Requirements

Software

Requirements

Analysis

Program Design

Coding

Testing

Operations

UAT is implement at

this stage

13

Table 2.1: Advantages and disadvantages of Modified Waterfall model [34]

Advantages Disadvantages

More flexible than the pure waterfall model. Milestones are more ambiguous than the pure

waterfall.

If there is personnel continuity between the

phases, documentation can be substantially

reduced.

Activities performed in parallel are subject to

miscommunication and mistaken assumptions.

Implementation of easy areas does not need to

wait for the hard ones.

Unforeseen interdependencies can create

problems.

Although heavyweight methodologies, such as Modified Waterfall model

continue to dominate the software development almost few decades and much research

has done in traditional methodologies, lightweight methodology brings its own

uniqueness in order to satisfy the customer through early and continuous delivery of

the valuable software [18].

2.2.2 Lightweight Methodology

Many years ago, when the software development is at the early stage, most of the

user’s requirements were quite stable where the software development mostly

followed the plans without major changes. However, as the technology growth,

software development involved more critical and dynamic industrial projects that

make new difficulties emerged accordingly [3]. The followings are the difficulties

emerged while using the previous heavyweight methodology:

(i) Frequently changing requirements: customer requirements are always

changing due to evolving business needs. Most of the customers do not have a

clear vision about the specifications of their requirements at the early stages

because they sometimes realize what their true requirements are only when

they use an application that does not really meet their needs. This usually led

to cause the major software failure which is incorrect assumptions with regard

to software requirements or it can be said as faulty user expectation [36, 37].

(ii) Customer involvement: lack of customer involvement will usually leads to

higher chances of project failure [38] because in heavyweight SDM customer

will only involve at the requirement phase .

(iii) Cost assumptions and deadlines: In any software development, customers often

do not accept failure. Apart from that, companies usually offer low budgets,

14

tight deadlines, while at the same time, requiring high demands because of

competition in the markets [39].

(iv) Miscommunications: one cause of the misunderstanding of requirements is the

miscommunication between developers and customers. It is hard to gather

requirements from customer as they were some barriers exist between the

developer and the customer as customer came from various background.

Therefore, the software practitioners are looking forward for a new SDM to

face the changes. Lightweight methodologies hold practices that enable programmers

to build solutions more quickly and efficiently, with better responsiveness to changes

in business requirements. Basically, it is mainly focused on development based on

short life cycles, involves the customer throughout the software development and also

strive for simplicity along with value the people [5]. Lightweight methodologies are

inherently flexible enough to be tailored to each project depending upon the

characteristics of each [40]. Lightweight methodology also known as Agile

methodology.

There have been several studies and ideas on improving and enhancing the

overall traditional software development process such as to defeat the rapid change in

organization and business need [41]. Basically, Agile methodology focuses on people,

customer’s satisfaction and rapid response to change [31, 42]. According to Agile

Manifesto [39], the followings are the major factors of Agile:

(i) Early customer involvement;

(ii) Iterative development;

(iii) Self-organizing teams; and

(iv) Adaptation to change.

There are various methodologies that categorized under lightweight

methodology. Currently, there are six methods that are identified as Agile

methodology, which are Crystal Methodologies, Dynamic Software Development

Method, Lean Software Development, Scrum, and eXtreme Programming [5, 6, 41,

42]. Table 2.2 shows the description of the Agile Methodology.

15

Table 2.2: Description of the Agile Methodology [42]

Agile Methodologies Description

Crystal methodologies The most agile method, Crystal Clear, focuses on communication in

small teams developing software that is not life-critical.

Dynamic Software

Development Method

It divides projects into three phases: pre-project, project life-cycle, and

post project

Lean Software

Development

It consists of seven principles: eliminate waste, amplify learning, and

decide as late as possible, deliver as fast as possible, empower the team,

build integrity, and see the whole.

Scrum Focuses on project management in situations where is difficult to plan

ahead, with an importance on feedback mechanisms.

eXtreme Programming

(XP)

Focuses on best practice for development which consists of twelve

practices.

All the listed methodologies have their own uniqueness that will suit on

different software development situation. This research focused in details on XP as

one of the latest lightweight SDM.

2.2.2.1 eXtreme Programming (XP)

XP is actually a deliberate and disciplined approach to Agile software development

which was found to be most successful at smaller companies. The successful of XP

especially at smaller companies basically it is focused on customer satisfaction [27].

XP allows developers to confidently respond to changing customer requirements in

any software development phase [43]. The aim of this methodology is to deliver the

software to the customer needs on time. The key element of XP is teamwork where

managers, customers and developers collaborate in a team. There are five ways that

improve a software project in XP which are communication, simplicity, feedback,

respect, and courage.

XP is based on four main values which are known as simplicity,

communication, feedback and courage while the XP process is characterized by short

development cycles, incremental planning, continuous feedback, and reliance on

communication and evolutionary design [44]. The core of XP is made up of a simple

set of common-sense practices. These practices are planning game, small releases,

metaphor, simple design, testing, refactoring, pair programming, collective ownership,

continuous integration, 40-hourweek, on-site customer, coding standards, open

workspace and just rules [43]. The XP software development process starts with

16

planning, and all iterations consist of four basic phases in its life cycle: designing,

coding, testing, and listening.

XP uses an iterative and incremental software process that carried out in

relatively short cycles. The life cycle of XP includes planning and also four basic

iterative development phases which are designing, coding, testing, and listening. In the

planning phase user stories are written, iterative implementation planning is

performed, efforts are estimated, and priorities of the features are decided [45].

Although simplicity is one of the XP’s principles, it doesn’t mean that it can exclude

designing process. Improper will lead to the complexity of certain project. Therefore

it is then important to create a design structure that organizes the logic in the system

to avoid too many dependencies in the system [46]. The other process in XP is coding

which is the most important phase where programmers begin to write the code at the

very beginning. Traditionally, testing is a phase of development that is carried out after

the main coding effort. In contrast, XP makes the programmers to write the tests before

the code. Every time new code is written on the XP project, a corresponding test case

must be written and implemented first. Listening is another important process in XP

lifecycle where good listening skill of the developer will enable them to understand

clearly what customer want thus will develop solutions based on customers’ need as

close as possible [43]. Figure 2.4 shows the flow chart of XP.

Figure 2.4: XP flow chart [43]

Based on the figure, it can be seen that at the starting point gathering

requirements is similar to traditional projects but different in the way it is documented.

It involves establishing business requirements and justification which results in a

vision document at a high level [47]. The iteration process is one that makes progress

17

through successive improvement. Figure 2.5 shows how the iteration occur in Agile

along with possible documentation artifacts [47].

Figure 2.5: Iterative project with possible documentation artifacts in Agile

The stage after initiation refer as Iteration 0. In this phase initial scoping and

requirements takes place. Preparation activities also occur such as assembling the

technical infrastructure, setting up test harness for test driven development and

assembling the team. After Iteration 0 comes a collection of iterations where each

iteration delivering a part of working functionality. Iterations are bound with time

allocations which tend to be between one to six weeks. The common concept in agile

methods is the concept that a deployment should be occurred as conclusion of each

iteration.

Although the advantages of XP had been proven but there are still some

disadvantages of XP. Table 2.3 shows the advantages and disadvantages of XP in

software development project.

Initiation

Iteration 0

(Requirements

Analysis/

Iteration Prep)

Iter

atio

n

(1)

Iter

atio

n

(2)

Iter

atio

n

(n)

DE

PL

OY

Iter

atio

n

(n+

1)

Iter

atio

n

(n+

2)

DE

PL

OY

- Project Charter

- Vision statement

- Business Requirement

Document (Light)

- Functional

Requirements

- Non-functional

Requirements

- Backlog

- Light Uses

Cases/Stories

- Non-functional

Requirements

- Test

- Design Models

- Wireframe

- Help

- Maintenance Plan

18

Table 2.3: Advantages and disadvantages of XP

Advantages Disadvantages

XP maximizes the communication among all

stakeholders [24].

XP’s most problematic feature is the amount of

on-site customer involvement it requires [24].

It welcome changing requirements, even late in

development [25].

The issue regarding what extent to which XP can

adapt to changing requirements is still open to

discussion between the researcher [44].

2.2.3 Comparison between Heavyweight and Lightweight Methodology

Many years ago, software development methodologies were basically limited to the

process of coding and fixing the error found in the coding. At that moment, these

heavyweight methodologies continue to dominate the systems development [18].

Despite the success it has a lot of drawbacks, like linearity, inflexibility in changing

requirements, and high formal processes irrespective of the size of the project [31]. As

the system grew, they became prone to bugs and also unable to run efficiently.

Therefore, the newer methodologies arise with the aim of improving the software

development flexibility and efficiency. As there are various models of software

development life cycle, each has its own advantages and disadvantages depending

upon which have to be decided and which model should be selected [4].

There are many different characteristics between the heavyweight and

lightweight methodologies. According to Dyba & Dingsoyr [48], the differences

between heavyweight and lightweight development basis on the an unpredictable

world, as well as emphasizing the value competent people and their relationships bring

to software development. Research by Moniruzzaman & Akhter [18] shown the reason

of lightweight methodology adoption is mainly because of the acceleration time to

market the product. Table 2.4 shows the comparison of heavyweight and lightweight

methodologies [49].

19

Table 2.4: Comparison of heavyweight and lightweight methodologies [49]

Issues Heavyweight Lightweight

Customers Committed, knowledgeable,

collocated, collaborative,

representative, empowered

Access to knowledgeable,

collaborative, representative,

empowered customers

Requirements Mainly emergent, rapid change. Knowable early, largely stable.

Architecture Designed for current requirements Designed for current and

foreseeable requirements

Size Smaller Team and Products Larger Team and Products

Primary objective Rapid value High assurance

Developers Knowledgeable, collocated,

collaborative

Plan-driven, adequate skills,

access to external knowledge.

Release cycle In phases (multiple cycles) Big bang (all functionality at

once)

Development life cycle Linear or incremental Incremental

Style of development Adaptive Anticipatory

Documentation Light (replaced by face to face

communication)

Heavy / detailed Explicit

knowledge

Team members Co-location of generalist senior

technical staff

Distributed teams of specialists

Client Involvement onsite and considered as a team

member Active/proactive

Low involvement

Measure of success Business value delivered Conformance to plan

There were also many researchers that conducted comparative studies between

specific heavyweight and lightweight methodologies. For example, Liu & Umphress

[50] conducted comparative study between IEEE 1074 and XP [50], Abrahamsson et

al. [17] had conducted comparative study between several agile methodologies itself,

Balaji & Murugayan had performed comparative study between Waterfall model, V-

Model and also Agile modeling while Moniruzzaman & Hossain [18] had conducted

comparative study between Waterfall and Agile Modeling. Table 2.5 summarizes the

authors’ research work results.

20

Table 2.5: Summary of comparative study from different authors

Author Comparative Study Result

Liu & Umphress, 2008

[50]

IEEE 1047 and XP XP is more suitable for developing grid

software.

Abrahamsson et al., 2010

[17]

Among difference

Agile modeling`

The better approach depends on different

situation of system development.

Balaji & Murugayan, 2012

[25]

Waterfall, V-Model and

Agile Modeling

The better approach is depends on how

frequent the user requirement change.

Moniruzzaman & Hossain,

2013 [18]

Waterfall, Agile

Modeling

Agile modeling significantly improves

software development process in many

ways.

Apoorva & Deepty, 2013

[4]

Waterfall, V-shaped,

Rapid Application

Development (RAD),

Incremental, Spiral

The hybrid of all these methodologies is

used with some modification will help to

improve the quality of developed software.

2.3 Software Testing

Software testing is an integral part of the software development process. All the

activities involved in developing software need to be carefully coordinated in order to

meet the requirements. According to Mustafa & Khan [16], it is not unusual for

developers to spend 40 percent of the total project time to perform software testing.

Testing can be considered as a process of planning, preparing, executing and analyzing

the difference between the actual and the required status [16]. According to Singh [51],

testing is the process of executing a program with the intent of finding faults. In other

words, software testing is a process which is used to measure the quality of software

developed. Any software which is produced without any testing can be dangerous to

the users. Therefore the importance of software testing in software development are as

follows [52, 53]:

(i) To improve software quality;

(ii) For reliability estimation;

(iii) To measure usefulness of software; and

(iv) For verification and validation.

In average 50% of the software failure are because of lack of cooperation,

weak task backlog, and lack of software testing resources were common bridge causes

[54]. For many years until now developers are still using the same testing techniques

21

some of which is crafted method rather than good engineering methods [55]. In order

to perform software testing, there are few software testing strategies that can be

followed such as unit testing, integration testing, system testing and user acceptance

testing. Figure 2.6 shows the relationship between all the software testing strategies

while performing testing activities [51]. However, this research focused on user

acceptance testing only where testing is done by the customer.

Figure 2.6: Software testing strategy [51]

2.3.1 Unit Testing

Unit testing can be considered as the basic level of software testing where it focuses

on smaller blocks of a program or system [56]. This process will give developer the

ability to verify whether the system unit functions as expected. Therefore, for any

function with a set of inputs, we can determine if the function is returning the proper

values. Although performing unit testing is to ensure that the unit implemented

structure matches the intended design structure but there are also problems with unit

testing [51]. The problem that usually occurred is how to run each unit independently

when there is a unit that may not be completely independent.

Unit testing is generally seen as a white box test class which purposely looking

and evaluating the code as implemented rather than conformance user requirements.

The followings are unit testing benefits according to Abhijit et al. [55]:

(i) Unit level testing is very cost effective.

(ii) It provides a much greater reliability improvement for resources expanded than

system level testing.

Unit Testing Integration

Testing System Testing

Acceptance

Testing

Any testing technique (verification plus validation) is applicable. Testing

is done by testers only.

Testing is done by

the customer

Software is ready for the

customer

22

(iii) Be able to test parts of a project without waiting for the other parts to be

available.

(iv) Achieve parallelism in testing by being able to test and fix problems

simultaneously by many engineers.

(v) Be able to detect and remove defects at a much less cost compared to other

later stages of testing.

2.3.2 Integration Testing

Integration testing is a systematic technique for constructing the program structure

while at the same time conducting tests to uncover errors associated with interfacing.

The objective is to take unit tested components and build a program structure that has

been dictated by design [55]. Integration testing can be divided into two categories

which are top down integration testing and also bottom up integration testing.

2.3.3 System Testing

System testing is performed once the unit testing and integration testing had been done.

In this testing all the developed software is test along with the expected environment

including hardware and other associated parts [51]. System testing is conducted on the

entire system in Software Requirement Specification (SRS). It tests the behaviour and

even the believed expectations of the customer. It is also intended to test up to and

beyond the bounds defined in the software/hardware requirements specification.

2.3.4 User Acceptance Testing (UAT)

UAT is a type of testing that carried out to verify whether the product is developed

according to the standards and specified criteria hence meets all the requirements that

specified by customer [57]. Generally, this testing is carried out by a user/customer

where the product is developed externally by another party. UAT categorized under

black box testing methodology where the user is only focusing on the overall system

function and compares it with the requirements specified by them. UAT is considered

23

to be one of the most important testing by user before the system is finally delivered

or handed over to the end user.

Testing accomplishes a variety of things especially the quality of the

application. UAT does three things for a software development project [58]:

(i) They measure how compliant the system is with the business requirements.

(ii) They expose functionality/business logic problems that unit testing and system

testing have missed out since unit testing and system testing do not focus much

on functionality/business logic testing.

(iii) They provide a means of determining how done the system is.

According to Bordo [59] and Lewis [56], UAT process can be divided into six

activities such as analyzing business requirement, identify UAT scenarios, define the

test plan, create test case, run the tests and record the results.

2.4 Designing User Acceptance Testing

The steps of performing UAT are test strategy, test development, test

execution, defect management and delivery [60]. The test strategy, schedule and also

resources need to be defined at the beginning of the UAT process by the manager. It

is vital to decide the test strategy as it will affect the delivery phase because the UAT

is done just before delivery process. After completing the test strategy, the test

development is performed. This step consists of test plans and business scenario test

cases. It is followed by the test execution step which includes defects, reports, and

metrics. The following step is the defect management which consists of defect tracking

and verification. The last step is the delivery phase that includes the UAT plan with

verification and testing strategy. Figure 2.7 shows the UAT steps [60].

24

Figure 2.7: UAT steps

One of the important questions in planning to perform UAT is what need to be

tested. This question is crucial as it will identify the objects that are needed for

planning the UAT. During planning the UAT, elements need to be considered are

documentation, business requirements, functional requirements, business processes,

workflows, transactions and the software [61]. It is impossible to test every possible

issue in each business system. Therefore, the most significant functionalities to be

tested are business functions. Business users should have the knowledge and

understanding of their business processes [62]. Even though choosing what to test is

so hard, but under certain circumstances a test stakeholder as well as the test team have

to consider the item to be tested [63].

2.4.1 The Importance of UAT

The importance of UAT is the ability to keep ongoing maintenance costs as low as

possible because the costs of fixing functionality and usability is cheaper if the

problem can be detected earlier. Performing UAT in various stages and with various

types of test respondents can provide ideal opportunities to identify and repair broken

a comparative study of user acceptance testing...

Documents