system usability scale for usability testing of ...santoso, & purwadi, kajian aplikasi pertanian...
TRANSCRIPT
- International Refereed Social Sciences Journal
■ E-ISSN: 2229-4686 ■ ISSN: 2231-4172 ■ http://www.researchersworld.com ■ Vol.–X, Issue–4, October 2019 [43]
DOI : 10.18843/rwjasc/v10i4/06
DOI URL : http://dx.doi.org/10.18843/rwjasc/v10i4/06
System Usability Scale for Usability Testing of
Agriculture E-Commerce Website
Joko Purwadi,
Department of Informatics,
Universitas Kristen Duta Wacana, Yogyakarta, Indonesia.
Rosa Delima,
Department of Informatics,
Universitas Kristen Duta Wacana,
Yogyakarta, Indonesia.
Novia Indrayono Rante Toding,
Department of Informatics,
Universitas Kristen Duta Wacana,
Yogyakarta, Indonesia.
Argo Wibowo,
Department of Information System,
Universitas Kristen Duta Wacana,
Yogyakarta, Indonesia.
Halim Budi Santoso,
Department of Information System,
Universitas Kristen Duta Wacana,
Yogyakarta, Indonesia.
(Received September 21, 2019; Accepted October 25, 2019)
ABSTRACT
A software development project should pass many stages starting from defining needs to
implementation. Before the process of system implementation, the development team will
conduct a usability test and a user acceptance test for the application. This stage is very
important to ensure the implementation process can be done well by minimizing problems
when users interact with the system. In this study, usability and user acceptance tests will be
carried out on the sale and purchase application of agricultural products (agricultural e-
commerce). The method used in the study consists of 6 main stages, such as preliminary
studies, participant studies, tool development, test implementation, data analysis, and result
formulation. The tools to be used are such as a questionnaire for participants’ data
collection, a System Usability Scale (SUS) questionnaire to measure the application
acceptance and task scenarios to measure system usability. This article specifically
discusses the design of task scenarios and the data taking process on the usability test on an
agricultural e-commerce system.
Keywords: Agriculture E-Commerce; Usability Test, User Acceptance, System Usability
Scale; SUS.
INTRODUCTION:
The field of agriculture in Indonesia continues to experience growth both in terms of agricultural
technology and information technology that supports agriculture. Various systems have been developed
to support increased productivity and expansion of agricultural product markets both in Indonesia and in
the world. Some applications that have been developed in Asian and African countries are e-Choupal
(India), Agricultural Market Information System (Bangladesh), and Infotrade (Uganda) (Delima,
Santoso, & Purwadi, Kajian Aplikasi Pertanian yang Dikembangkan di Beberapa Negara Asia dan
Afrika, 2016). Meanwhile, in Indonesia several information applications have been developed in
agriculture (Delima & Purwadi, Analisis Situs Web Pertanian Berbahasa Indonesia, 2015). One of the
- International Refereed Social Sciences Journal
■ E-ISSN: 2229-4686 ■ ISSN: 2231-4172 ■ http://www.researchersworld.com ■ Vol.–X, Issue–4, October 2019 [44]
agricultural applications that has been developed is a system for sales and puschasing agricultural product
(Delima, Santoso, Andriyanto, & Wibowo, 2018).
The Agricultural Product Sales and Purchasing System is a system developed to support economic
development and market expansion, particularly for farmers and people in rural areas. The system was
completed in 2018. Before the system is introduced to the users, one step is needed, such as testing the
user acceptance of the application. A test aimed at determining the level of user acceptance e-commerce
application. This test is very important to minimize failures in the system implementation process. The
test is carried out through the SUS questionnaire.
The paper consists of five parts. Part I is an introduction, literature review can be seen in Part II, while
the methodology and design of test equipment can be seen in Parts III and IV, and Part V is a conclusion
and future work.
LITERATURE REVIEW
System Usability Scale (SUS):
System Usability Scale (SUS) is a questionnaire developed by John Brooke in 1986 that was used to
evaluate the usability level of Blackboard (Brooke, 1996). The SUS questionnaire consists of 10
questions, with odd sequences for positive items and even sequences for negative items. Each question is
graded on a 5 scales from “Strongly Disagree” to “Strongly Agree”. The ten list of questions on SUS can
be seen in Table 1.
Table 1: Question in SUS (Brooke, 1996)
No Questions
1. I Think that I would like to use this system frequently.
2. I found the system unnecessarily complex.
3. I thought the system was easy to use.
4. I think that I would need the support of a technical person to be able to use this system.
5. I found the various functions in this system were well integrated.
6. I thought there was too much inconsistency in this system.
7. I would imagine that most people would learn to use this system very quickly.
8. I found the system very cumbersome to use.
9. I felt very confident using the system.
10. I needed to learn a lot of things before I could get going with this system.
To measure the application’s level of acceptance using the SUS questionnaire, we can use a measurement
developed by (Bangor, Kortum, & Miller, 2009) as shown in Table 2. Through the value of SUS, the level
of application’s acceptance is evaluated from the reusability aspect.
Table 2: Application Acceptance Rates on the SUS Questionnaire (Bangor, Kortum, & Miller, 2009)
Nilai SUS Interpretation
< 50 Not Acceptable
50 - 70 Marginal
> 70 Acceptable
SUS questionnaire has been widely used in the process of system reusability testing. Some studies using
the SUS questionnaire are (Puspita, Sibarani, Muslim, & Moch., 2016) (Baharrudin, Wardani, &
Herlambang, 2018) (Devy, Wibirama, & Santosa, 2017) (Devy, Wibirama, & Santosa, 2017), and (Al-
Omar, 2018).
- International Refereed Social Sciences Journal
■ E-ISSN: 2229-4686 ■ ISSN: 2231-4172 ■ http://www.researchersworld.com ■ Vol.–X, Issue–4, October 2019 [45]
E-Commerce:
In today’s global era, E-commerce is a promising new business model. This business model utilizes
information and communication technology (ICT) support. According to Turban et al (Turban, King,
Lee, Liang, & Turban, 2015), electronic commerce (abbreviated as EC, or E-commerce) is a trading
model that includes the process of buying, selling, transferring, and exchanging products, services or
information through computer networks, including the internet.
E-commerce business is one of the most important parts of the economy in Indonesia. According to
(Gentur, 2014) “The Ministry of Trade of the Republic of Indonesia estimated that the sale and purchase
of goods transactions via the internet (E-commerce) from Indonesia will penetrate the number of US $
10,08 billion. The average value of online shopping transactions is growing 40 percent every year”.
Further, according to Agus Tjandra in the News (Berita, 2014) as vice chairman & foreign relations of
the Indonesian E-commerce Association (idEA), on the signing of E-commerce cooperation between the
Indonesian Information Technology Federation (FTII) and the Taipei Computer Association (TCA) in
Jakarta stated that Indonesia’s total E-commerce transaction value in 2013 reached US $ 8 billion or
around Rp 94 trillion and is predicted to continue to increase to reach US$ 24 billion or around Rp 283
trillion in 2016. The number of people who shop online in the country is recorded at 4,6 million people
during 2013 and is expected to increase to 8,7 million in 2016.
E-commerce also provides many benefits for both entrepreneurs and customers. Some of the
benefits of e-commerce according to (Khan, 2016) are such as increasing entrepreneur income,
reducing company’s operating costs, saving time to do shopping for consumers, and having a
certain level of convenience in shopping.
Dodolantani: An Agriculture E-Commerce:
Agricultural Product Sales and Purchasing System is a system developed to support economic
development and market expansion for farmers in particular and people in rural areas (Delima, Santoso,
Andriyanto, & Wibowo, 2018).
Sales and Purchases are two processes of trading goods and services business. At present the business
model of trading in Information and Communication Technology (ICT) is known as e-Commerce. E-
Commerce is a trading model including the sale, purchase and exchange of goods and services through
computer networks and the internet. E-Commerce has several advantages for both sellers and buyers,
such as e-commerce can reduce operational costs and buyers can save time in shopping.
Dodolantani agricultural e-commerce has 2 main modules namely the purchase module and sales
module. The purchase module functions to manage the product’s purchasing process, this module
consists of the user’s login page (Figure 1). There are 2 types of logins, such as login as a buy er
and login as an admin.To be able to login as buyer, can be done by selecting the register button on
the right side of the login button. After log in, the buyer will see the product page (Figure 2). This
page shows the latest and most popular products so ld by the seller. Through the product page,
buyers can see, search and discuss the products they want to buy. After being sure to make a
purchase transaction, the user can continue to make purchases through the buy button on the
product details (Figure 3.). Then the buyer can make a purchase transaction (Figure 4). Purchase
transactions include determining the purchase amount, purchase address, and shipping expedition .
After making a product purchase, the buyer will then be on the shopping cart page (Figure 5) and
the purchase can be proceeded to the payment process. After making payment the buyer must
upload the proof of payment. After the buyer makes a payment, the system admin must verify the
payment made by the user. Verification can be done via the validation page (Figure 6). If the
payment does not match, the system admin can send a notification to the buyer regarding the
payment incompatibility. If the payment is correct and validated, the process continues with
sending goods via the specified courier. The purchase module sitemap can be seen in Figure 7
(Delima, Santoso, Andriyanto, & Wibowo, 2018).
- International Refereed Social Sciences Journal
■ E-ISSN: 2229-4686 ■ ISSN: 2231-4172 ■ http://www.researchersworld.com ■ Vol.–X, Issue–4, October 2019 [46]
Figure 1: User Login Page
Figure 2: Product Page
- International Refereed Social Sciences Journal
■ E-ISSN: 2229-4686 ■ ISSN: 2231-4172 ■ http://www.researchersworld.com ■ Vol.–X, Issue–4, October 2019 [47]
Figure 3: Detail Product Page
Figure 4: Shopping Page
- International Refereed Social Sciences Journal
■ E-ISSN: 2229-4686 ■ ISSN: 2231-4172 ■ http://www.researchersworld.com ■ Vol.–X, Issue–4, October 2019 [48]
Figure 5: Shopping Cart Page
Figure 6: Payment Validation Page
- International Refereed Social Sciences Journal
■ E-ISSN: 2229-4686 ■ ISSN: 2231-4172 ■ http://www.researchersworld.com ■ Vol.–X, Issue–4, October 2019 [49]
Figure 7: Site Map Purchase Module in Dodolantani
The sales module in dodolantani e-commerce functions to support the seller in conducting goods sales
transactions. The sales module of agriculture is divided into 2 menu features such as the seller display
which consists of the login page, the seller main page, product listing page, product added form page,
product offer form page, product listing page, product price check page, sales page, inbox page, seller
report page, profile page. Next, there is admin display which consists of admin main page, product
category list page, product list page, product offer list page, seller list page, seller validation page, seller
data page, product validation page, transaction data page, payment validation page and admin report
page. While on the product listing page, the details of product offerings, sales transactions, product
discussions, and each sale reports can be seen in Figure 8 until Figure 12 and Site map of the sales
module can be seen in Figure 13 (Delima, Santoso, Aditya, Purwadi, & Wibowo, 2018).
Figure 8: Product List Page
Purchase
Module
Home RegistrationProduct
List
Equipment
Substance
Agriculture Product
Product Checking
Purchasing
Purchase Data Entry
Product Discussion
Product Comment
Entry
Payment
Upload Payment Receipt
Report
Sales Transection
Sales Report
Profile
- International Refereed Social Sciences Journal
■ E-ISSN: 2229-4686 ■ ISSN: 2231-4172 ■ http://www.researchersworld.com ■ Vol.–X, Issue–4, October 2019 [50]
Figure 9: Detail Sales Product Page
Figure 10: List of Sales Transactions Page
- International Refereed Social Sciences Journal
■ E-ISSN: 2229-4686 ■ ISSN: 2231-4172 ■ http://www.researchersworld.com ■ Vol.–X, Issue–4, October 2019 [51]
Figure 11: Product Discussion Page
Figure 12: Sales Report Page.
- International Refereed Social Sciences Journal
■ E-ISSN: 2229-4686 ■ ISSN: 2231-4172 ■ http://www.researchersworld.com ■ Vol.–X, Issue–4, October 2019 [52]
Figure 13: Site Map Sales Module.
RESEARCH METHODOLOGY:
This research will apply a testing method which consists of six stages, namely a preliminary study,
participant/respondent study, test tool development, implementation, data analysis, and result
formulation. The order of the stages of the research can be seen in Figure 14.
Figure 14: Research Methodology Scheme.
The Preliminary Study is carried out to make plan, identify needs, and the availability of research
resources. The test would be conducted in May to July 2019. Data collection will be carried out directly
with agricultural stakeholders including farmers, sellers of agricultural products, and farmer groups. The
data are collected in the Central Java and Yogyakarta area during May to July 2019. Researchers have
cooperation with Farmer Group in these region.
The targeted respondents consist of two groups of respondents such as respondents who are end users
such as buyers and sellers and respondents who are administrators of the system. The total number of
respondents is targeted at around 35 respondents. Criteria for buyer and seller respondents are a
minimum age of 20 years, a minimum of junior high school graduate, and able to operate a computer.
The questionnaire used to test user’s acceptance is the SUS questionnaire and to support the data retrieval
process, the Google form is used. To measure the application’s level of acceptance using the SUS
questionnaire, we use acceptance rates as shown in table II.
ANALYSIS AND DISCUSSION:
SUS Questionnaire: Questionnaire is one of the data collection tools in form of a series of questions or statements that would
be submitted to the respondents. The questionnaire is based on and refers to the SUS (System Usability
Scale) Questionnaire, which is a questionnaire consisting of 10 statements (Brooke, 1996) (Lewis &
Sauro, 2017).
Google Form for Support Usability Testing:
To support the process of data retrieval during usability test, Google form is used for data collection.
There are 2 forms developed such as a form to get the respondent’s characteristics and a form for SUS
questionnaire. A form to get the respondent’s characteristics consists of questions that include personal
identity (name, address, gender, age, education), farming experience, experience in using computer or
mobile phone, experience related to social media and internet, and experience in conducting online
transactions. Meanwhile the SUS questionnaire was made using a Likert scale from 1 to 5, with 1 for the
category of “strongly disagree” and 5 for “strongly agree”.
Sales
Module
Home RegistrationProduct
List
Equipment
Substance
Agriculture Product
Product Checking
Selling
Sales List Information
Order Checking
Status
Inbox
Product Discussion
Product
Review
Product Entry
Equipment
Substance
Agriculture
Product
Report
Sales
Transection
Sales
Report
Profit
Preliminary
Study
Perticipants
Study
Develop
Tools
Conduct
Testing
Data
AnalysisResults
- International Refereed Social Sciences Journal
■ E-ISSN: 2229-4686 ■ ISSN: 2231-4172 ■ http://www.researchersworld.com ■ Vol.–X, Issue–4, October 2019 [53]
Data Collection:
The test was conducted from May to July 2019, before conducting the test a brief interview was
conducted to explore the respondents’ experience in the use of laptops and smartphones. This process is
carried out to ensure that the respondents are qualified as required. If the respondent meets the next
requirements, then the researchers gives an explanation of the purpose of data retrieval and provide a
brief explanation related to the system to be tested. Respondents were asked to fill out the agreement
form. Then the respondents are given time to try to run the system independently. If the time is
considered as enough, then the test data would be taken to complete the task in accordance with the
scenario that had been prepared. After finishing it, the process continued with filling SUS questionnaire
to find out the level of users’ acceptance.
Participants Characteristics:
Respondents in this study are 35 persons, with a distribution of 30 respondents who were buyers or
sellers and five respondents were admin users. The respondents came from Bantul and Salatiga. The
characteristics of the respondents can be seen in Table 3.
Table 3: Participants Characteristics.
Variable Class Total Percentage
Age
20 – 30 years 9 26%
31 – 40 years 12 34%
41 – 50 years 12 34%
> 50 years 2 6%
Gender Male 16 46%
Female 19 54%
Education Level
Junior High School 5 14%
Senior High School 1 3%
Diploma 3 4 11%
Under Graduate/ Bachelor 13 37%
Post Graduate 1 3%
Based on the data in Table 3, it is known that the majority of respondents aged between 31 to 50 years
old reached 78%. Based on gender, respondents of the research were quite balanced even though the
percentage of women is more than men, but the difference is not too significant. Based on the level of
education, the largest percentage was undergraduate education at 37%. The graph of the respondents’
characteristics can be seen in Figure 15 to Figure 17.
Figure 15: Graph based on Participants Age.
0
2
4
6
8
10
12
14
20 -30 thn 31-40 thn 41-50 thn > 50 thn
Respondent Distribution Based on Age
- International Refereed Social Sciences Journal
■ E-ISSN: 2229-4686 ■ ISSN: 2231-4172 ■ http://www.researchersworld.com ■ Vol.–X, Issue–4, October 2019 [54]
Figure 16: Graph based on Participants Gender.
Figure 17: Graph based on Participants Education
Result:
System Usability Scale Questionnaire consists of 10 questions as stated on Table I. There are two groups
of respondents. The first group is buyer and seller. The second group is System Administrator.
Questionnaire uses 5 Likert Scale which has following criteria as mentioned on Table 4.
Table 4: Likert Scale for SUS Questionnaire.
Skala Categories
1 Very disagree
2 Disagree
3 Neutral
4 Agree
5 Very Agree
Table 4 describe likert scale for System Usability Scale Questionnaire (SUS Questionnaire). Respondent
should answer the questionnaire using this scale. After respondent answer the questionnaire, the next step
is to calculate the value of SUS using normalization formula (1) and (2), where SUSValue i,j is defined
as a value of question i from respondent j and RVji is respondent j’s value for question i.
SUSvalueij = RVji − 1 for i is an odd questions (1)
SUSvalueij = 5 − RVji for i is an even question (2)
Acceptance value is got by using formula (3), where j is a respondent and n is the number of respondents.
Acceptance Value = ∑ (∑ SUSvalueij
10i−1 x 2.5)n
j=1
n (3)
Men
46%
Women
54%
Men
Women
02468
101214
Junior High
School
Senior High
School
Non Degree
Diploma
Qualification
Undergraduate Graduate
Respondents Education Level
- International Refereed Social Sciences Journal
■ E-ISSN: 2229-4686 ■ ISSN: 2231-4172 ■ http://www.researchersworld.com ■ Vol.–X, Issue–4, October 2019 [55]
User acceptance level is interpreted as acceptance value and value interpretation as shown in Table 3.
Based on SUS Questionnaire, Table 5 describes the response from buyer and seller. While Table 6
describes the response from system administrator.
Table 5: Result for Buyers and Seller Respondents.
R SUS Questions
Total Total x 2.5 1 2 3 4 5 6 7 8 9 10
1 4 0 4 1 3 2 2 1 3 0 21 50
2 2 2 2 2 2 2 3 2 3 2 22 55
3 2 2 2 1 2 2 2 1 2 2 18 45
4 2 2 3 2 3 3 4 2 3 2 26 65
5 4 2 3 2 3 2 4 3 3 2 28 70
6 2 1 3 2 3 2 4 3 4 2 26 65
7 3 2 3 1 3 2 3 3 3 2 25 62.5
8 2 2 3 4 2 3 4 3 4 3 30 75
9 2 3 2 2 2 3 2 3 2 3 24 60
10 4 2 4 3 2 3 3 3 2 3 29 72.5
11 2 2 4 4 2 2 3 3 4 2 28 70
12 3 2 3 3 3 2 3 3 3 2 27 67.5
13 3 2 1 2 2 2 2 2 2 1 21 47.5
14 4 1 2 2 2 3 2 3 2 1 22 55
15 3 2 2 2 2 3 2 3 2 2 23 57.5
16 3 2 3 2 2 2 2 3 3 2 24 60
17 4 2 3 3 2 2 3 2 3 2 26 65
18 3 2 2 3 2 3 2 3 2 3 25 62.5
19 3 2 3 3 2 3 2 3 2 3 26 65
20 4 2 2 2 2 2 2 3 3 2 24 60
21 2 3 2 3 2 3 2 3 2 3 25 62.5
22 2 2 3 2 1 2 1 2 1 2 18 45
23 3 2 1 2 1 2 1 2 1 2 17 42.5
24 2 2 2 3 2 3 2 3 2 3 24 60
25 2 2 1 2 1 2 1 2 1 2 16 40
26 3 2 1 2 1 2 1 2 1 2 17 42.5
27 2 2 1 1 2 3 2 2 2 2 19 47.5
28 3 2 1 1 2 2 2 2 3 1 19 47.5
29 3 2 2 1 2 2 2 2 2 1 19 47.5
30 2 2 3 3 2 3 4 3 3 2 27 67.5
Table 6: Result for Admin System Respondents.
R SUS Questions
Total Total x 2.5 1 2 3 4 5 6 7 8 9 10
1 3 2 3 3 2 3 3 3 3 2 27 67.5
2 2 2 4 3 2 3 3 3 3 3 28 70
3 3 2 3 3 2 3 4 4 3 3 30 75
4 3 2 1 2 2 2 2 3 2 2 21 52.5
5 3 1 2 2 2 2 2 2 2 3 21 52.5
- International Refereed Social Sciences Journal
■ E-ISSN: 2229-4686 ■ ISSN: 2231-4172 ■ http://www.researchersworld.com ■ Vol.–X, Issue–4, October 2019 [56]
As mentioned before, Table 5 shows the result for buyer and seller respondents and Table 6 shows the
result for System Administrator respondents. After calculating using formula (1) and (2), we got
acceptance value for buyer and seller is 57.8 and acceptance value for system administrator is 63.5.
Based on those acceptance rate from Table 2, this acceptance value can be categorized as marginal
acceptable. Furthermore, there are some improvement should be done to the system in order to increase
the acceptance value. We also conducted another analysis to test average value for each question in SUS
Questionnaire. During the design phase, we divided 10 questions in SUS Questionnaire into two groups:
Questions which give positive contributions and questions which give negative contributions.
Positive contribution question exists for the odd number questions and negative contribution question
exists for the even number questions. After have some normalization using formula (1) and formula (2),
we gain value range from 0 to 4 with interpretation as shown on Table 7.
Table 7: Data Range and Interpretation SUS Quistionnaire.
Data Value Interpretation Impact
4 ++
3 +
2 Neutral
1 -
0 --
Table 7 shows data range and interpretation of SUS Questionnaire. Based on the value range, we conducted
calculation analysis to calculate average response for each questions. The result is shown on Table 8.
Table 8: Average Value for SUS Value.
SUS Questions Average Value
Buyer and Saller Respondents Admin System Respondents
1 2.77 2.8
2 1.93 1.8
3 2.37 2.6
4 2.2 2.6
5 2.07 2
6 2.4 2.6
7 2.4 2.8
8 2.5 3
9 2.43 2.6
10 2.2 2.6
Table 8 shows the average value for SUS Value. From this analysis, it is found that 90% of each average
value above neutral but this value has not reached positive impact value to the acceptance value of the
application. This result supports the first finding that there are some improvements should be conducted
in order to satisfy user and increase user acceptance value.
CONCLUSION:
From the analysis, it can be concluded that user acceptance level for agriculture e-Commerce dodolantani
on middle acceptable level. This result is based on SUS Questionnaires that are distributed to the users. It
means that some system improvement should be done to satisfy user acceptance level. For future work, it
needs some user task test in order to get the detail feature that should be improved. Through usability
testing, researchers are expected to find some specific feature that needs to be improved and revised.
- International Refereed Social Sciences Journal
■ E-ISSN: 2229-4686 ■ ISSN: 2231-4172 ■ http://www.researchersworld.com ■ Vol.–X, Issue–4, October 2019 [57]
ACKNOWLEDGMENT:
Special thanks to Research and Community Services Institution and Faculty of Information Technology
Universitas Kristen Duta Wacana for providing facilities and funding for publishing this article.
REFERENCES:
Al-Omar, K. (2018). Evaluating the Usability and Learnability of the "Blackboard" LMS Using SUS and
Data Mining, 2nd Conference on Computing Methodologies and Communication (ICCMC) (pp.
386-390). Erode, India: IEEE.
Baharrudin, M., Wardani, N. H., & Herlambang, A. D. (2018). Analisis Usability Pada Sistem Informasi
FILKOM Apps Fakultas Ilmu Komputer Universitas Brawijaya, Jurnal Pengembangan
Teknologi Informasi dan Ilmu Komputer, Vol 2 No. 10, 4179 - 4183.
Bangor, A., Kortum, P., & Miller, J. (2009). Determining what individual SUS scores mean : Adding an
adjective rating scale, Usability Stud, Vol. 4, No. 3, pp 114-123.
Berita, S. (2014, September Rabu). 2016, Potensi Bisnis E-Commerce Indonesia Mencapai Rp 283
Triliun. Indonesia: Berita Satu. Retrieved from Berita Satu:
http://www.beritasatu.com/iptek/207256-2016-potensi-bisnis-ecommerce-indonesia-mencapai-
rp-283-triliun.html
Brooke, J. (1996). SUS-A quick and dirty usability scale Industrial usability evaluation. 189(194): p. 4-7.
Delima, R., & Purwadi, J. (2015). Analisis Situs Web Pertanian Berbahasa Indonesia, Seminar Nasional
Komputer dan Informatika Terapan (pp. 1-5). Jember: Politeknik Negeri Jember.
Delima, R., Santoso, H. B., & Purwadi, J. (2016). Kajian Aplikasi Pertanian yang Dikembangkan di
Beberapa Negara Asia dan Afrika, Seminar Nasional Aplikasi Teknologi Informasi. 2016, pp. pp.
B19-B26. Yogyakarta: Universitas Islam Indonesia.
Delima, R., Santoso, H. B., Aditya, G. H., Purwadi, J., & Wibowo, A. (2018). Development of Sales
Modules for Agricultural E-Commerce Using Dynamic Syatem Development Method,
International Journal of New Media Technology (IJNMT), 95-103.
Delima, R., Santoso, H. B., Andriyanto, N., & Wibowo, A. (2018). Development of Purchasing Module
for Agriculture E-Commerce using Dynamic System Development, International Journal of
Advanced Computer Science and Application, 86-96.
Devy, N. I., Wibirama, S., & Santosa, P. I. (2017). Evaluating Eser Experience of English Learning
Interace using User Experience Questionnaire and System Usability Scale, 1st International
Conference on Informatics and Computational Science (ICICoS) (pp. 101 -105). Semarang,
Indonesia: IEEE.
Devy, N. R., Wibirama, S., & Santosa, P. I. (2017). Evaluating user experience of english learning
interface using User Experience Questionnaire and System Usability Scale, 1st International
Conference on Informatics and Computational Sciences (ICICoS). Semarang, Indonesia.
Gentur, P. J. (2014, September Selasa). Transaksi e-Commerce Indonesia Capai US$ 10 Miliar, Retrieved
from CNN INDONESIA: http://www.cnnindonesia.com/ekonomi/20140923142423-92-
4073/transaksi-e-commerce-indonesia-capai-us--10-miliar/
Khan, A. G. (2016). Electronic Commerce: A Study on Benefits and Challenges in an Emerging
Economy, Global Journal of Management and Business Research: Beconomics and Commerce,
3-4.
Lewis, J. R., & Sauro, J. (2017). Can I Leave This One Out? The Effect of Dropping an Item From the
SUS, Journal of Usability Studies, 13(1), 38-46.
Puspita, M. A., Sibarani, T. M., Muslim, E., & Moch., B. N. (2016). Perancangan Ulang User Interface
Mesin Parkir Meter Jakarta, Jurnal Ergonomi dan K3, Vol. 1 No. 1, 25-33.
Turban, E., King, D., Lee, J., Liang, T., & Turban, D. (2015). Electronic Commerce : A Managerial and
Social Network Perspective, Cham, Switzerland: Springer International Publishing AG.
Retrieved from E-commerce: http://staff.uns.ac.id
----