mubarak alharbi-user experience dimensions, aspects and measures...
TRANSCRIPT
User Experience Dimensions, Aspects
and Measures: Systematic Literature
Review
A Thesis Submitted in Partial Fulfillment of the Requirements for the Degree of Master of Sciences in Software Engineering
At the College of Computer & Information Sciences At Prince Sultan University
Mubarak Saad Alharbi
March 2016
January 201
I
User Experience Dimensions, Aspects
and Measures: Systematic Literature
Review
Mubarak Saad Alharbi
This thesis was defended and approved on
Supervisor: Dr. Mohammad Zarour
Members of the Exam Committee
Dr. Mohammad Zarour Chair
Dr. Mamdouh Alenezi Member
Dr. Iman Almomani Member
II
Acknowledgments
In the name of Allah, the Most Gracious and the Most Merciful,
Alhamdulillah, all praises to Allah for the strengths and His blessing in completing this
thesis.
Firstly, I have to thank my parents for their love, pray and support through my life.
Thank you for everything that giving me the strength to reach this achievement. Special
thanks to my wife for her understanding and support during my study.
Secondly, I would like to express my sincere gratitude to my advisor Dr. Mohammad
Zarour for the continuous support of my Master study and related research, for his
patience, motivation, and immense knowledge. His guidance helped me in all the time
of research and writing of this thesis. I could not have imagined having a better advisor
and mentor for my Master study.
My sincere thanks also go to Mr. Tariq Alhuzaimi – eChannels Department Director in
STC, who gave me the time to complete this thesis.
I should not forget to acknowledge my brothers and sisters for their continuous prays,
support, understanding and good wishes.
Finally, I thank all those who have helped me directly or indirectly in the successful
completion of my thesis. Anyone missed in this acknowledgment are also thanked.
If I did not mention someone’s name here, it does not mean that I do not acknowledge
your support and help. Again, I would like to thank everyone who supported and helped
me during my Master study.
III
Table of Contents Abstract __________________________________________________________ V
البحث ملخـــــص _____________________________________________________ VI List of Tables _____________________________________________________ VII
List of Figures ___________________________________________________ VIII 1 Introduction ___________________________________________________ 1
1.1 Mobility Revolution Worldwide 1
1.2 Mobility in Saudi Arabia 2
1.3 Popularity of Mobile Applications 3
1.4 HCI and User experience activities 4
1.5 What is User Experience? 5
1.6 Research Problem 7
1.7 Goals 8
1.8 Research Approach 8
2 A Systematic Literature Review ___________________________________10
2.1 Systematic Literature Reviews Protocol 11 2.1.1 The need for the systematic review of this topic 11 2.1.2 The Research Questions 11 2.1.3 Search Strategy 11 2.1.4 Study Selection 12 2.1.5 Study Quality Assessment 13 2.1.6 Data Extraction Strategy 14 2.1.7 Data Synthesis Strategy 14
3 Conduct the SLR _______________________________________________15
3.1 The Need for a Systematic Review 15
3.2 Research Questions 15
3.3 Search Strategy 17
3.4 Study Selection 21 3.4.1 Inclusion Criteria 21 3.4.2 Exclusion Criteria 21 3.4.3 Primary study selection process 22
3.5 Quality Assessment 24
3.6 Data Extraction Strategy 25
3.7 Data Analysis 26 3.7.1 The Demographic Data 26 3.7.2 Research Questions’ Answers & Findings 32
3.8 Validity Threats 44
4 Proposed Framework ____________________________________________46
IV
4.1 UX Definitions 46
4.2 UX Dimensions 46
4.3 UX Aspects 48
4.4 UX Measurement Methods 52
4.5 Final Proposed Framework 53
5 Conclusion & Future Work _______________________________________55
5.1 Conclusion 55
5.2 Future Work 56
Appendix _________________________________________________________57
A. Selected Primary Studies _________________________________________57
B. UX Aspects’ Studies ____________________________________________58
C. UX Measurement Methods’ Studies ________________________________59
Bibliography ______________________________________________________60
V
Abstract
Technology is changing the way the people used to live. The new generation,
indisputably, is the generation of technology and more specifically the generation of
mobile. Kids started using the technology even before they entered the school.
Nowadays, the users of technology applications started developing their user experience
very early in their lives. Mobile applications have become one of the most used
products on mobile devices, 90% of the US mobile Internet usage time is spent using
applications while 10% of the time spent on browsing. The success or failure of a
software product is affected by the users of that product. Accordingly, studying,
measuring and improving user experience is crucial for the success of any software
product.
The purpose of this research is to scan various aspects that may affect the user
experience (UX) and how it can be measured in various applications and apply the
gained knowledge in mobile applications. To achieve this purpose, a systematic
literature review is conducted to collect necessary information. By analyzing the
extracted data, the possible UX dimensions that have an impact on the UX either during
the product development or the product usage are gathered, and four main dimensions
are identified. In addition to the UX dimensions, the relationship between those
dimensions and UX aspects that have a direct or indirect impact on the UX is scanned.
The UX aspects that have been identified are categorized into seven categories and
assigned to the corresponding UX dimension. Moreover, the possible methods that can
be used to measure UX aspects are scanned, and the generic names of these methods
have been identified and as a result of this step, fourteen generic methods are identified.
The generic methods are linked to the corresponding UX dimension based on the
applicability of measuring UX aspects in this UX dimension.
All these results have been analyzed and consolidated in one framework that explains
the relationship between UX dimensions, UX Aspects and UX measurement methods.
This framework can be used as starting step for understanding how the user experience
got affected in mobile applications and how to measure it.
VI
البحث صـــــملخ تغیر نمط حیاتنا الیومیة. فالجیل الجدید یعتبر، بال منازع، جیل التقنیة او باألحرى تعمل التقنیة على
جیل الجوال. فاالطفال یشرعون في استخدام الجوال وما یصاحبھ من تقنیات حتى قبل دخولھم
حل التقنیة بتطویر خبراتھم كمستخدمین للتطبیقات التقنیة في مرا االمدرسة. فالیوم، یبدأ مستخدمو
حیث تطبیقات الجوال من أكثر التطبیقات استخداماً خالل الفترة األخیرة تعتبر و .رة من حیاتھممبك
التصفح في حین إحتل، الكلیة٪ من فترة استخدام الجوال ٩٠التطبیقات مایقارب احتل استخدام ھذه
و نتیجةً رتبط بشكل رئیسي بمستخدمي ذلك التطبیق.نجاح أي تطبیق یتأثر وی . إن ٪١٠بقیة ال
، فإن دراسة وقیاس تجربة مستخدمي التطبیقات ھي جزء من األسباب المؤدیة لنجاح التطبیق.لذلك
تطبیقاتال في المستخدم تجربة في مایؤثرمانستطیع م جمع ھو البحث ھذا من لغرضاكان لذلك
.، ومن ثم تطبیقھا على تطبیقات الجوالقیاسھا یمكن كیف ومعرفة بشكل عام
تم ذلك من المستخرجة البیانات وتحلیل المجال ھذا في أبحاث من سبق لما علمیة مراجعة خالل من
بعد أو التطبیق تطویر فترة خالل سواءاً المستخدم تجربة في التأثیر ذات الرئیسیة اإلتجاھات حصر
حصر من ماسبق ىإل وباإلضافة ، وقد تم حصر أربعة اتجاھات رئیسیة.للمستخدمین ونشره اكتمالھ
تجاھاتاإل ھذه بین العالقة توضیح تم فقد المستخدم تجربة في المؤثرة الرئیسیة لإلتجاھات
بحیث حصلنا على سبع وقد قمنا بتصنیف ھذه العوامل .المستخدم تجربة في المؤثرة والعوامل
تجربة قیاس أدوات بحصر قمنا فقد ذلك على عالوةً اإلتجاھات المناسبة لھا. بربطھا تصنیفات وتم
ومن أداة عامة، أربع عشرةبحیث أصبح لدینا وإعادة تصنیفھا حسب نوع األداة الممكنة المستخدم
اعتماداً المستخدم تجربة في الموثرة العوامل وكذلك السابقة التأثیر باتجاھات ربطھا ثم تم ، وذلك
.محدد تأثیراتجاه على قابلیة استخدام ھذه األداة مع العوامل المؤثرة والمرتبطة ب
تجربة المستخدم قیاس وأدوات الموثرة والعوامل لإلتجاھات حصر من سبق مما اإلنتھاء بعد
الفھم زیادة لغرض وذلك ماسبق لكل شامل عمل إطار ببناء قمنا ، بینھا فیما العالقات وتوضیح
.قیاسھا وكیفیة الجوال تطبیقات في المستخدم تجارب طبیعة حول
VII
List of Tables
Table 1: Attributes required to document the research results ...................................... 12
Table 2: Research questions and the expected outcomes .............................................. 16
Table 3: Research questions and the extracted keywords ............................................. 17
Table 4: Research sources and the fields used to conduct the search ............................ 17
Table 5: Results of the first round of the research questions and the sources ................ 18
Table 6: Results of removing the duplicates from the sources ...................................... 18
Table 7: Results of applying the inclusion/exclusion criteria (title & abstract scanning)
.................................................................................................................................... 20
Table 8: New results using Mendeley DB to find related papers .................................. 20
Table 9: Included and excluded studies across the sources ........................................... 22
Table 10: Quality assessment checklist ........................................................................ 24
Table 11: Extracted attributes from the selected studies ............................................... 25
Table 12: Suggested categories/Dimensions for research disciplines ............................ 35
Table 13: UX Aspects' descriptions ............................................................................. 37
Table 14: Suggested categories for UX aspects and their relationship to UX dimensions
.................................................................................................................................... 39
Table 15: Categories of UX measurement methods from the SLR ............................... 40
Table 16: Extracted UX measurement methods from the selected studies .................... 41
Table 17: UX measurement Methods - General Names ................................................ 42
Table 18: Frameworks to measure the UX and the UX dimensions that are covered .... 43
Table 19: Suggested relationship between the UX measurement methods and the UX
dimensions .................................................................................................................. 52
Table 20: Framework measures UX in literature and the proposed one in addition to the
covered UX dimensions............................................................................................... 54
Table 23: Selected primary papers and their points ...................................................... 57
Table 24: UX aspects and the related studies ............................................................... 58
Table 25: UX measurement methods and the related studies ........................................ 59
VIII
List of Figures Figure 1: ICT growth during the last 15 years ...............................................................1
Figure 2 : Growth of ICT sector in Saudi Arabia ...........................................................2
Figure 3: Categories of usage on mobile devices ...........................................................3
Figure 4: Growth of Mobile Apps between Q2-2014 and Q2-2015 ...............................4
Figure 5: UX is elaboration of Usability ........................................................................6
Figure 6: Usability for Objective and UX for Subjective ................................................6
Figure 7: UX is an umbrella for everything ....................................................................6
Figure 8: Research approach flow and outcomes ...........................................................9
Figure 9: SLR Flow Chart ........................................................................................... 23
Figure 10: Sources of selected studies ......................................................................... 26
Figure 11: The growth of selected studies .................................................................... 27
Figure 12: Participation of countries in UX research .................................................... 28
Figure 13: Percentage of answered research questions in selected studies .................... 29
Figure 14: Percentage of research methods used in selected studies ............................. 30
Figure 15: Application domains that studied in this SLR papers .................................. 30
Figure 16: Output of selected studies ........................................................................... 31
Figure 17: Perspectives relationship ............................................................................ 33
Figure 18: Identified Dimensions from the SLR and its relationships ........................... 35
Figure 19: UX aspects and number of related papers ................................................... 36
Figure 20: The relation between value and UX dimensions.......................................... 47
Figure 21: UX Aspects Classifications......................................................................... 49
Figure 22: Relation between UX Aspects categories and UX dimensions .................... 50
Figure 23: Relation between UX aspects and all UX dimensions ................................. 51
Figure 24: Proposed Framework that covers gathered UX dimensions, aspects
categories, aspects, and measurement methods ............................................................ 53
1
1 Introduction
Technology is changing the way the people used to live. The new generation,
indisputably, is the generation of technology and more specifically the generation of
mobile. Kids started using the technology even before they entered the school.
Nowadays, the users of technology applications started developing their user experience
very early in their lives. One of the very popular types of applications used are those
running by mobile devices. Mobile applications have become some of the most-used
products on mobile devices [1].
1.1 Mobility Revolution Worldwide
The number of subscriptions to mobile cellular service is increasing. In 2015,
there are more than 7 billion subscriptions worldwide compared to 738 million in 2000,
as shown in Figure 1. “Globally 3.2 billion people are using the Internet, of which 2
billion are from developing countries”[1].
Figure 1: ICT growth during the last 15 years [1]
Figure-1 shows that the Internet usage is growing, and mobile broadband is one of the
Internet service’s facilitators. Worldwide, mobile broadband penetration has reached
47% in 2015, which is 12 times the value in 2007 [1]. Figure 1 also shows that the
number of Internet users has grown as well, from 6.5% in 2007 to 43% in 2015 [1].
2
Hence, in the world that moves faster than ever towards the digitalization of societies,
the use of mobile and smart devices become prominent. Nowadays, mobiles are used
heavily by individuals as well as business. Mobile usage in business was divided into
three categories as below [2]:
• Business to Enterprise (B2E): Apps to Employees
• Business to Business (B2B): Apps to Brokers
• Business to Consumer (B2C): Apps to Consumers
Mobile in B2C world is important, and was divided into three domains Commerce,
Intermediary, and Information. Commerce usage of mobiles covers the commercial
transactions of digital such as eBooks and physical products such as buying tickets.
Intermediary usage of mobiles covers the content delivery to the consumers such as
providing content about weather, while Information usage of mobiles covers sending
personalized information to the consumer either for marketing use or personal
information [3].
1.2 Mobility in Saudi Arabia The Information and communication technologies (ICT) sector in Saudi Arabia
has expanded over the last ten years and remains on a huge growth. In 2012, the ICT
sector spent 94 billion and reached 112 billion in 2013; the expected spending in 2017 is
138 billion as shown in Figure 2 .
Figure 2 : Growth of ICT sector in Saudi Arabia [4]
3
Mobility in Saudi Arabia is one of the most dynamic aspects of ICT, and it gained 64%
of the spending in 2013. Due to the high penetration of smartphones, mobile broadband
subscriptions have grown over the past five years. Saudi Arabia has one of the world’s
highest percentages of total Internet traffic consumed on mobile devices [4]. In Saudi
Arabia, mobile broadband penetration represented 47.6% by the end of 2013 while
fixed broadband penetration represented 45.5%. These facts refer to the importance of
the mobile role in keeping the consumers connected to others, as it is one of their
primary needs [4].
1.3 Popularity of Mobile Applications Mobile applications have become one of the most used products on mobile
devices [5], 90% of the mobile usage time is spent using applications while 10% of the
time spent on browsing [6] as shown in Figure 3. Accordingly, it is important to make
sure that these applications continue to satisfy the users’ needs and expectations.
Figure 3: Categories of usage on mobile devices [6]
Taking a closer look at Figure 3, notice that Social, Messaging and Entertainment
categories account for 51% of the time spent on mobile applications [6]. Messaging and
4
Social grew to 31% in Quarter 2 (Q2) 2015 comparing to 28% in Q2 2014, and
Entertainment Apps grew from 8% in Q2 2014 to 20% in Q2 2015. The unexpected
result that Gaming Apps decreased from 32% in Q2 2014 to 15% in Q2 2015 [6] as
shown in Figure 4. The success or failure of a software product is affected by the users
of that product. Accordingly, studying, measuring and improving user experience as
part of the human-computer interaction domain is crucial for the success of any
software product.
Figure 4: Growth of Mobile Apps between Q2-2014 and Q2-2015 [6]
1.4 HCI and User experience activities In the human-computer interaction (HCI) domain, an evolution in product
measurement is happening, and five generations of measurement identified. The first
generation referred to as product reliability, which started in the 1940s and focused on
ensuring that the product would perform without failure. The second generation was
emphasizing the data processing and how long the product would take to complete this
job, and this was in the 1950s. The third generation was in the 1960s when the time-
sharing concept became popular, and the measurement was to assess the user
5
performance and how long the task would take to be completed. The fourth generation
started in the 1980s when the number of products’ users increased, and the
measurement included more measures such as learnability and others, and this referred
to the usability. The fifth generation started late in the 1990s and shifted to measure the
user experience [7] [8]. Academic researchers of HCI are studying the usage and the
user experience of user interfaces through experimental testing of user interfaces in the
lab, or qualitative/ quantitative and observation of people using user interfaces in the
wild.
1.5 What is User Experience?
User experience (UX) studies in HCI fields have increased during last two
decades [8]. In these studies, a common agreement has been reached about the
dynamics of user experience and the subjectivity of its nature [9]. Several researchers
have stated that user experience is dynamic, context-dependent, and subjective. Due to
this fact, “the user experience is seen as something desirable without defining what
something means”, and this has led to difficulty in agreeing on a user experience
definition [10].
In literature, definitions for user experience were proposed, but those definitions were
affected by the researcher’s background and interest [10] and could not be used as a
common definition. ISO 9241-210 defined user experience as “A person's perceptions
and responses that result from the use or anticipated use of a product, system or
service”[11]. This definition is a promising one as stated by Law[10], The definition is
in line with the view by most researchers about the subjectivity of UX, but the used
terms require more explanation to list possible objects that affect user experience[10].
User experience a tightly coupled with usability [12], and it can be conceptualized in
different ways as it summarized in [13]:
• An elaboration of the usability from the satisfaction perspective as shown in
Figure 5. This is because the usability has different attributes such as efficiency,
effectiveness and satisfaction [14]. Satisfaction attribute refers to the likely,
pleasure trust and other similar attributes, and from this perspective the UX is
considered as elaboration of usability.
6
Figure 5: UX is elaboration of Usability
• Distinct from usability which emphasizes on objective measures, while user
experience emphasizes on subjective measures as shown in Figure 6. This
differentiation based on the usage of measurement methods. It is not necessary
that usability methods can measure UX as there is a clear difference between
usability characteristics and UX characteristics[15].
Figure 6: Usability for Objective and UX for Subjective
• An umbrella term for users’ perceptions, whether measured objectively or
subjectively as shown in Figure 7, it follows ISO 9241-210 definition which
covers perceptions from the user; either objectively or subjectively[10].
Figure 7: UX is an umbrella for everything
7
1.6 Research Problem Mobile applications have become one of the most popular products used by users
nowadays, and the need for measuring the acceptance of the mobile application is
growing as well [5]. Referring to the fact that current generation of measuring the
products measures the user experience with it, leads to the need for measuring user
experience with the mobile application. The concept of user experience has been
growing in the research community during the last two decades, and there is still no
agreement about its nature and how to measure it [16]. As stated by many studies in the
literature, the “UX is dynamic, context-dependent, and subjective” [10]. This fact led to
the need to scan this research area systematically to identify the possible dimensions
that related to the UX and that have an impact on it. Identifying these dimensions will
build boundaries around the UX and will give directions on how to recognize the impact
on the UX if it exists. Also, this may help to discover the methods that can be used to
measure such an impact or, at least, understand its nature. Once identified the possible
the dimensions that have an impact on the UX and the methods that have been used to
measure such dimensions, then a framework can be proposed to covers these
dimensions and apply the useful methods to measure it. As a summary, the UX is
difficult to be measured due to its nature, and a need for a framework that covers the
different UX dimensions and aspects will help to measure it using proper methods.
Accordingly the main research problem covered in this research work is to identify the
main user experience dimensions, aspects and measurement methods currently available
in the literature and relate them all together in one framework.
8
1.7 Goals In this research, a systematic approach will be applied to a reach the main
objective of understanding how to measure the user experience in mobile applications
and to build a framework for that. To reach this final goal, the below tasks need to be
completed:
1. Study the available UX literature to understand the current state, which will help us to know if a new framework need to be built or a common frameworks can be used.
2. Identify available frameworks and analyze the weaknesses.
3. Identify various dimensions that affect UX.
4. Build a comprehensive framework that covers the main UX dimensions and overcome the current weaknesses.
1.8 Research Approach Figure 8 shows the main stages of this research and the expected outcomes from
each phase, and the explanation as below.
1. Systematic Literature Review and the expected outcomes is a set of selected
primary studies which will be used to scan related disciplines.
2. Demographical Analysis and the expected outcome is a demographical data
about the topic and the selected primary papers.
3. Findings Analysis and the expected outcomes are UX definitions, UX Aspects,
UX Measurement Methods and UX frameworks used in the field to measure
UX. Moreover, in this phase, a more relational analysis will be conducted to
relate findings to each other, and more understanding should exist.
4. Build Framework and the expected outcome is a proposed framework that
contains findings and explains the relation between them.
5. Report Results and the outcomes are submitting papers to related conferences,
journals, and a full thesis draft will be delivered.
9
Figure 8: Research approach flow and outcomes
Systematic Literature
Review
Primary Studies
General Analysis
Demographic data
Data Analysis
UX Definition
UX Aspects
UX Measurement Methods
UX Frameworks
Build Framework
Proposed Framework
Report Results
Thesis Draft
Submit Paper
Outcome
Phase
10
2 A Systematic Literature Review
Evidence-based software engineering (EBSE) aims to improve decision-making
related to software development and maintenance by integrating current best evidence
from research with practical experience and human values [17]. An evidence-based
paradigm is used in medicine for decades[18]. The applicability of this paradigm to the
software engineering (SE) field studied in the literature [18], and it showed that EBSE
brings benefits to SE by improving the quality of software-intensive systems and
reassuring stakeholder groups that practitioners are using the best practice. To identify
the possible best practices, collecting evidence and synthesize practical experience, a
systematic literature review is needed.
A systematic literature review, is an identification and evaluation of available research
that is related to a research topic or research question. Any individual study that
contributes to a systematic review called a primary study while the systematic review is
known as a secondary study. The systematic review helps to summarize the available
evidence that is related to a topic or domain. Also, it helps identify any gaps in a
particular area of research to suggest areas of investigations. In addition to that, a
systematic review helps to provide background for new research work, in particular for
those who are new to the area [19].
It is recommended [18] that software engineers adopt more EBSE to treat systematic
reviews at their practitioner publications. Since then, software reviews have become
increasingly popular in software engineering [19], and the main purpose of those SRs is
to get the best quality of scientific studies [20]. Such systematic reviews should define
the search strategy that has been followed to conduct the review. This documented
strategy, which known as systematic review protocol, will help to assess the
completeness of the SR[19].
11
2.1 Systematic Literature Reviews Protocol
A review protocol specifies the methods and steps that will be followed to
implement the systematic review. Such a protocol will reduce the possibility of
researchers' bias [19]. The protocol contains components needed to complete the
review, which are as follows:
2.1.1 The need for the systematic review of this topic
Conducting systematic reviews will help in summarizing the findings documented
in the literature related to certain topic and it helps in exploring the weaknesses in that
topic that should gain more attention by the researchers in the domain. Hence, the
reviewer should document clearly the need to conduct the systematic review.
2.1.2 The Research Questions
In SLR, the research questions that are going to be answered need to be identified.
These research questions are used as bases to identify the suitable keywords to be used
as search strings to search the digital resources.
2.1.3 Search Strategy
It is necessary to follow a search strategy, and the strategy is an iterative process
that benefits from assessing existing systematic reviews and related studies. It also
benefits from regular searches using the terms derived from the research questions. In
addition to that, checking the results of the regular searches against a list of known
primary studies will add more value to the search terms that used. Finally, consulting
the experts in the search domain is a factor in building the search strategy [19].
Breaking the research question down to identify the search terms is a starting point, and
the result will be nouns. Using the combination of those terms and alternative
translations and abbreviations will lead to a list of possible search terms that can be used
to search for primary studies. Previous factors need to be considered in building the
search strategy and need to be documented and replicable. Suggested procedures to
document the search process are given in Table 1:
12
Table 1: Attributes required to document the research results
Data Source Documentation
Digital Library Name of database (e.g. IEEE, ACM, Citeseer)
Search strategy for the database
Date of search
Years covered by search
Other Sources Date searched/contacted
URL
Any specific conditions about the search
2.1.4 Study Selection
Once the primary studies obtained, then both inclusion and exclusion criteria
should be applied to identify the primary studies that provide evidence about the
research question [19]. The inclusion/exclusion criteria need to be applied based on
practical issues such as the following [19]:
• Journal
• Authors
• Subject
• Research methodology and design
• Date of publication
Another factor needing to be explained is how many assessors will be available to
evaluate the selected primary studies and how the conflicts will be resolved.
13
2.1.5 Study Quality Assessment
Adding a new layer for filtering the selected primary studies after applying the
inclusion and exclusion criteria is a necessary step to do the following [19]:
• Provide more detailed inclusion/exclusion criteria
• Investigate if the differences of quality provide an explanation for differences in
studies’ results
• Find the strength of studies' results and conclusions
• Give weights for the studies
Such a layer of filtering the studies needs to be applied against an appropriate checklist.
The choice of the checklist is affected by the type of primary studies and the stage of
implementing the study either design, conduct, analysis, or conclusions, and this will
lead to a complex process to apply the quality checklist [19].
One of the choices is the checklist that proposed by Kichenham [20] and below the
selected questions that will be answered during the SLR conducting:
• Is there a clear statement of the aims of the study? Yes/No. Score as 1, 0.5, 0.
• Is there an adequate description of the context in which the research or
observation was carried out? Yes/Partly/No. Score as 1, 0.5, 0.
• Was the research method appropriate to address the aims of the research?
Yes/Partly/No/Not applicable. Score as 1, 0.5, and 0 or mark NA.
• Was the recruitment strategy or experimental material or context appropriate to
the aims of the research? Yes/Partly/No/Not applicable. Score as 1, 0.5, and 0 or
mark NA.
• Has the relationship between researcher and participants been considered to an
adequate degree? Yes/Partly/No. Score as 1, 0.5, 0.
• Is there a clear statement of findings? Yes/Partly/No. Score as 1, 0.5, 0.
• Is the study of value for research or practice? Yes/Partly/No. Score as 1, 0.5, 0.
14
2.1.6 Data Extraction Strategy
After selecting the final primary studies and assessing the selection against a
proper quality instrument, specific data need to be extracted to obtain the information
from the primary studies. The process of extracting such data should be clear and can
extract required data that answer the research questions. The extracted data can be used
either for inclusion/exclusion criteria or data analysis. If the data required in
inclusion/exclusion criteria, then a separated form needs to be used; otherwise, one form
can be used to collect possible data.
Such data extraction needs to be reviewed to ensure the quality of extracted data. As
suggested by the SLR guidelines [19], the SLR conducted by postgraduate students in
their master or Ph.D. work is revised by the supervisor to ensure that the protocol
applied correctly. Hence, the selected studies will be reviewed by the supervisor for
further analysis and approval.
2.1.7 Data Synthesis Strategy
This stage of review protocol is the final phase. It involves collecting and
summarizing the results of primary studies. Synthesis can be quantitative and
qualitative. Using statistical techniques to obtain the quantitative results is called meta-
analysis. Synthesis activities need to be documented and explained in the protocol, and
some of the issues cannot be resolved until the data is analyzed since the protocol will
be updated by that.
15
3 Conduct the SLR
In this chapter, the systematic literature review guideline is applied and the details
of each step is explained.
3.1 The Need for a Systematic Review
A systematic literature review is needed to scan and assess the current status and
the work that has done in the user experience field. This literature review will have
good value if conducted following a systematic process, which can be repeated at any
time and assessed by the readers. This SLR is needed to scan relevant dimensions and
aspects that have an impact on user experience as the user experience is considered as a
part of satisfying the users’ needs [12]. Due to the numerous studies that conducted in
UX field, the need to synthesize these studies using SLR is prominent. Moreover, the
literature lacks the development of clear frameworks or measurement criteria to
measure the UX. The literature needs to be systematically reviewed to be able to
identify related factors, to develop such a framework.
3.2 Research Questions
To determine the aims of this study and get more understanding about it, seeking
answers to a set of research questions is mandatory. This study will focus on the main
milestones of the UX area, namely, UX definitions, aspects that affect the user
experience, methods that are used to measure UX and if there is any framework has
been used to measure U. The research questions and an explanation of the need for each
question and how it can add value to the study domain is following.
RQ1: What are the different definitions of user experience in literature?
This question concerned with identifying the possible definitions of UX in literature,
either defined by authors or any specialized organizations such as ISO. Identifying those
definitions will guide us in extracting the general dimensions of the aspects that affect
the UX, and this will help us to discover the relationship between the aspects, see Table
2.
16
RQ2: What are the aspects that affect the user experience?
This question concerned with identifying the aspects that affect the UX, which have
been studied or discovered in literature. Identifying these aspects, in addition to relating
them to the study domain and the UX dimension, will help us to study the effects of
these aspects on the UX as shown in Table 2.
RQ3: What are the methods identified to measure each of the specified aspects in
the literature?
This question concerned with identifying the measurement methods that have been used
in literature to measure the UX aspects as shown Table 2.
RQ4: What kind of framework can be developed to understand and measure the
user experience with mobile applications?
This question concerned with identifying any UX measurement framework that has
been documented in the literature. Identifying these frameworks will help us to know
how to build the frameworks that contain the UX dimensions, aspects, and measurement
methods and how to extract more dimensions, aspects, and measurement methods and
understand the relation between them as shown in Table 2Error! Reference source not
found..
Table 2: Research questions and the expected outcomes
Research
Question Expected Outcomes
Value for the study
RQ1 UX definitions
Gather possible
dimensions.
RQ2 The aspects that affect the UX and the relation
to UX dimensions
Gather affections
aspects.
RQ3 Methods used to measure the UX aspects Gather useful methods.
RQ4 Frameworks used to measure the UX and its
structure
Understand available
frameworks’ structure to
build a framework that
cover available
Dimensions, Aspects and
17
Measurement methods.
3.3 Search Strategy
In this study, the available sources will be scanned, and a list of keywords will be
used for that. The keywords derived from the research questions. The corresponding
keywords and their synonyms are used then to conduct the search.
The keywords that have been used to scan the sources were as follows; see Table
3Error! Reference source not found.:
Table 3: Research questions and the extracted keywords
Research Keywords
Research Question Extracted Keywords
RQ1 User experience, user experience definition, customer
experience, customer experience definition
RQ2 User experience aspects, customer experience aspects
RQ3 User experience measurement method, customer experience
measurement method
RQ4 Mobile application measurement framework, user experience
measurement framework
The resulted keywords are used to conduct the search. To retrieve relevant papers that
are focusing on the study domain, the publications’ title and keywords are the fields that
used to conduct the search; see Table 4Error! Reference source not found..
Table 4: Research sources and the fields used to conduct the search
Source Fields
IEEE
• Title
• Keywords
ACM
Citeseer
Google Scholar
18
RQ's keywords have been added into the search fields to scan a particular source. Four
RQs and four sources are used, and the result was 11,759 across sources and research
questions; see Table 5Error! Reference source not found..
Table 5: Results of the first round of the research questions and the sources
RQ/ Source IEEE ACM Citeseer Google
Scholar
Total
RQ1 2341 1452 254 5760 9807
RQ2 459 21 2 124 606
RQ3 36 0 0 2 38
RQ4 1306 0 1 1 1308
Total 4142 1473 257 5887 11759
To reach an accurate result and to minimize the duplication, the papers that answered
one RQ or more in each source are retrieved, and this was done using the OR condition
between the keywords. With this condition, the duplicate vertically and horizontally are
removed. So, there was no duplicate in the single source, and the result was as below
until August 2015; see Table 6Error! Reference source not found..
Table 6: Results of removing the duplicates from the sources
RQ/ Source IEEE ACM Citeseer Google
Scholar
Total
RQ1
574
1464
254
39
2331 RQ2
RQ3
RQ4
The papers that are related to the UX domain and its focus on computers and technology
have been selected. Within this scanning, a duplication inside some sources because the
papers’ titles were different while the papers’ contents or contribution were almost the
same was discovered, hence, the most recent paper has been chosen.
19
Table 7Error! Reference source not found. summarizes the resulted set of papers per
digital library that included in this study.
20
Table 7: Results of applying the inclusion/exclusion criteria (title & abstract scanning)
RQ/ Source IEEE ACM Citeseer Google
Scholar
Total
RQs 59 119 67 8 253
To store these papers and review them carefully, a Mendeley reference management
tool[21] has been used. This tool has its own database (DB), and there was an ability to
retrieve any papers related to the known papers. This feature added extra papers to the
selected papers which are related to UX domain, and those papers have been retrieved
from ACM and Google Scholar. Hence, those papers did not have any keywords from
the search keywords that were extracted from the RQs. Table 8Error! Reference
source not found. summarizes the final accepted papers that related to the UX domain
as per the conditions above.
Table 8: New results using Mendeley DB to find related papers
RQ/ Source IEEE ACM Citeseer Google Scholar Tot
al
RQs 59 124 67 10 260
By the end of this stage for gathering and selecting the accepted papers, another review
has been done on all selected papers to update their meta-data so it can be used in the
next phase during the inclusion and exclusion process.
The meta-data attributes that have been updated in Mendeley DB are as follows:
• Title
• Source
• Authors
• Publishing date
• Country – this depended on the first author's country
21
3.4 Study Selection
To select the primary studies, a new layer for filtering the selected papers that
gathered in the previous stage needs to be applied. This layer will have two categories
which are inclusion criteria and exclusion criteria.
3.4.1 Inclusion Criteria
The inclusion criteria will focus on the factors and conditions that need to be exist to
select the paper and include it in the primary studies. The inclusion criteria applied in
this study are below.
• The paper answers one of the RQs partially.
• The paper's publication date is 2005 and above.
• If a snowballing search happened, then the paper should exist on one of the
primary sources which are IEEE, ACM, Citeseer, or Google Scholar.
3.4.2 Exclusion Criteria The exclusion criteria will focus on the factors and conditions that cause the paper
to excluded from the primary studies list. The exclusion criteria applied in this study
are:
• The paper's publication date is 2004 or before.
• Full paper is not accessible
• The paper limited to the non-IT domain.
• The paper's author is not mentioned in the paper itself.
• The paper is calling for a workshop.
• It is a book or magazine.
• The paper is not answering any RQs, either with fully or partially.
• The paper is a duplicate of another paper with a different title, and the most recent
one has been chosen.
22
3.4.3 Primary study selection process
To apply the above inclusion and exclusion criteria, a Mendeley tool was used to
update the below meta-data, which will be used to filter the papers that gathered in the
previous stage.
The meta-data attributes that are added at this stage in addition to the previous attributes
are as follows:
• Answered RQs:
o This attribute is used to identify the RQs that answered in this paper.
o Example: RQ1RQ2; RQ1RQ4; RQ3.
• Application domain:
o This attribute is used to identify the application domain that related to the
paper.
o Example: General; Mobile App; Mobile Device; software, etc.
Table 9Error! Reference source not found. shows the final number of papers from the
previous stage after applying inclusion and exclusion criteria.
Table 9: Included and excluded studies across the sources
Criteria IEEE ACM Citeseer Google
Scholar
Total
Included Papers 26 68 15 5 114
Excluded Papers 33 56 52 5 146
Total 59 124 67 10 260
So, the total number of primary studies becomes 114 after applying the inclusion and
exclusion criteria. All studies' meta-data, either included or excluded, have been
updated to extract the data about it during the data synthesis stage, and Figure 9
summarize all previous stages.
24
3.5 Quality Assessment
In addition to using the general inclusion and exclusion criteria, it is required to
assess the quality of the primary studies. An initial difficulty is that there is no
agreement on the definition of quality of papers, but most quality checklists include a
question that aimed at assessing the bias and validity of papers [19].
Below a list of questions that are proposed to evaluate the quality of the selected
primary papers[20] as shown in Table 10Error! Reference source not found..
Table 10: Quality assessment checklist
No. Question No. of papers
answered the
Question by
Yes (Out of
114)
%
1 Is there a clear statement of the aims of the study? 60 52%
2 Is there an adequate description of the context in which the research or observation was carried out?
83 72%
3 Was the research method appropriate to address the aims of the research?
87 76%
4 Was the recruitment strategy or experimental material or context appropriate to the aims of the research?
62 54%
5 Has the relationship between researcher and participants been considered to an adequate degree?
66 57%
6 Is there a clear statement of findings? 103 90%
7 Is the study of value for research or practice? 86 75%
25
3.6 Data Extraction Strategy
In this phase, after selecting the primary studies, extracted data will be used in the data analyzing stage or even utilized in another round of the inclusion and exclusion stage. In previous stages, some meta-data attributes have been extracted, and Mendeley tool has been used to keep it, and more attributes will be added as listed in Table 11. This table willError! Reference source not found. explain the extracted attributes and the expected values for each one.
Table 11: Extracted attributes from the selected studies
Attribute Description Values Title The paper title Source The digital source for this paper IEEE, ACM, Citeseer, Google
Scholar Authors The paper authors Authors’ names Publishing Date
The date of publishing the paper 2005,2006, … ,2014
Country The country where this paper published
City attributes in Mendeley is the prime reference, but when this attribute is not available, the first author's country is selected as per his email or country name.
Answered RQs
The research questions (RQs) that answered in this paper.
RQ1, RQ2, RQ3, RQ4
Application Domain
The domain that this paper has studied
E.g.,: General, Software, Mobile.
Study Participants
This attribute identifies who participated in this paper
Academia, Industry, or Mixed
Research Method
What is the research method used in this paper?
E.g.,: Case Study, Action Research, Interview, and Questionnaire.
Research Output
What is the type of output from this paper?
E.g.,: New Framework, New Model, Usage of technique, usage of a tool.
UX Definition
The UX definition that was proposed in this paper
UX definition
UX Research Discipline
The research disciplines that the paper focuses on
E.g.,: Value, Technology, and User.
UX Aspects The aspects that mentioned in the paper and that have impacted on the UX levels
E.g.,: Emotions, usability, functionality, visual attractiveness.
UX Measurement Methods
The measurement methods that used in the primary paper to measure the UX aspect
E.g., The questionnaire, Survey, Persona, Focus groups.
UX The proposed UX framework that UX framework
26
Measurement Frameworks
has been studied or suggested in the primary paper
3.7 Data Analysis
As a result of conducting the SLR stages, extracted data from the primary studies
have been analyzed, and charts that explain the trends have been generated.
3.7.1 The Demographic Data
In next sections, findings related to demographical data are shown which may
help to get more understanding about the UX domain and the participants.
3.7.1.1 Source
Four primary sources in this SLR have been used, which are IEEE, ACM,
Citeseer, and Google Scholar. The majority of the selected papers were from the ACM
digital library, then IEEE, then Citeseer, and finally Google Scholar as shown in Figure
10. If any paper founded by snowballing, and it was not exist in one of these primary
source then it is excluded.
Figure 10: Sources of selected studies
IEEE25%
ACM59%
CiteSeer12%
Google4%
27
3.7.1.2 Publishing Date
In this SLR, the primary studies that related to the research topic and published between
2005 and 2015 are selected. As noticed in Figure 11, the publishing in this SLR domain
as per the agreed inclusion and exclusion criteria grew over the last ten years.
Figure 11: The growth of selected studies
The trend referring to the growth and the percentage is a high number. Since 2008, the
publishing in this area is increasing, and this explains the need for these UX-related
topics. Hence, that 2015 has only 1 selected primary paper, and this is due to the
inclusion and exclusion criteria. The publishing in 2015 was in the same level of
previous years, but the topics related to different domains; such as “Relating UX
activities to CMMI and Agile methodologies” [22].
3.7.1.3 Country and Sectors’ Participations
As per the conditions of extracting the countries of the primary studies in the
previous stage, Figure 12 explains and shows that the majority of the primary studies
were in the USA, by 24%, followed by the European countries. A few papers were from
the Asian countries and South American countries. In the USA, 24% which was the
high percentage of the studies, the academia’s participation was 82.8 %, while the
industry’s involvement was 3.4% and the mixed participation between them was 13.8%.
Finland, with 20%, came to the top of the European countries and was the second
country after the USA. The industry participation was higher than the USA's industry
participation, and it was 12%, while the mixed involvement and the academia were 12%
and 76%, respectively.
5 4 5
10
13 13
10
15
22
16
1
0
5
10
15
20
25
2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015
# of Papers
28
The reason behind this high percentage of industry participation in Finland was the full
participation of Nokia for both the industry percentage and the mixed percentage. The
period of this participation was between 2008 and 2010, and there was nothing after
that. Figure 12 shows the percentages of primary studies across the countries.
Figure 12: Participation of countries in UX research
3.7.1.4 Answered RQs
From the previous stage – data extracting –the answered research questions have
been identified in each primary study, and Figure 13 shows the percentages of each
answered RQ.
From Figure 13, the majority of the primary studies have answered RQ2 which was
53%. The question is about identifying the aspects that affect the user experience. Those
aspects related to the study's domain itself but can be applied to any other domain,
either fully or partially.
Germany12%
Ireland3%
Finland20%
Netherland4%
UK10%China
9%Malaysia
3%
Taiwan2%
USA24%
Portugal1%
Korea1%
India2%
Singapore1%
Sweden1%
Switzerland2%
Bangladesh1%
Belgium1%
Greece2%
Japan1%
Norway1%
Denmark2%Country
29
The aspects that affect user experience that identified in RQ2 need methods to prove
their effectiveness and those methods will be identified in RQ3, which seeks the answer
to what methods have been used to measure the UX's aspects. For this reason, the
percentage of studies that answered RQ3 is near RQ2's percentage, and it is 36%. By
these two RQs (RQ2 and RQ3), 89% of the studies identified the aspects and methods
of measuring the UX that related to study domains, and the remaining studies were
11%. 2% was for defining the user experience, and 9% was for gathering the aspects
and methods in one framework.
The relationship between these percentages expected because RQ1, which seeks for UX
definitions, is used in RQ2 to identify the aspects. In RQ3, the percentage is less than
RQ2 because the same method can be used to measure more than one aspect. This
analysis explains the need to work in the area of defining UX or the area of building
either a general framework to measure UX or a specific framework for a particular
domain.
Figure 13: Percentage of answered research questions in selected studies
2%
53%36%
9%
# of papers answered RQs
RQ1
RQ2
RQ3
RQ4
30
3.7.1.5 Research Method
Figure 14 shows the percentages of the various research methods adopted by different
publications. The high percentage was 18%, and it was for Case Study. The method has
been classified to be other if the author does not explicitly mention the method.
Figure 14: Percentage of research methods used in selected studies
3.7.1.6 Application Domain
Application domains that have been scanned in this SLR are shown in Figure 15,
which explains that the majority of primary studies done without the limitation to a
particular domain, which was 51%, and it was identified as a General Domain. In
addition to the general domain, a scanning was happened for the studies that related to
domains similar to the mobile application from a development perspective; such as web
portals and from the point of view of affecting the mobile application such as Mobile
Communication.
Figure 15: Application domains that studied in this SLR papers
Other41%
Case Study18%Experiment
6%
Interview3%
Questionnaire
3%
Survey8%
Discussion11%
Empirical Study
2%
Comparison2%
Experience Report
4%
Ethnographic Study
1%
Thinking Aloud
1%Research Method
General51% Mobile
Communication2%
Mobile Apps12%
Software9%
e-Commerce
System3%
Mobile Devices
6%
Mobile Software
11%
Web Portals
6%Application Domain
31
3.7.1.7 Research Output
The research output from the primary studies in this SLR shown in Figure 16. It
noticed that the majority was about guidelines and emphasizing the aspects that need to
be considered when measuring the UX in addition to the methods that can be used to
measure those aspects. The percentage for the guidelines was 44%, while the new
suggested frameworks, models/methods, tools, and techniques - which are the ways of
using the aspect or the method- were 18%, 17%, 3%, and 1%, respectively.
Those new frameworks and models were for particular domains, but none of them was a
comprehensive one which can be used as a generic one. However, those frameworks
and models will be used to build the appropriate structures of a framework to measure
the UX on mobile applications. The usage experiences of techniques, tools, and models,
were 3%, 3%, and 2%, respectively. The remaining percentage was 1%, and it was for a
modification of current techniques to use a method to measure an aspect of user
experience.
Figure 16: Output of selected studies
New Framework18%
Guidelines44%
New Tool3%
Usage Experience of Technique
11%
Usage Experience
of Tool3%
Usage Experience of
Model2%
New Technique
1%
Modification of Technique
1%
New Model17%
32
3.7.2 Research Questions’ Answers & Findings
In next sections, the findings related to research questions are explained, and the
relation between the answers will be used to build the framework.
3.7.2.1 UX Definitions
ISO 9241-210 [11] has defined UX as “A person's perceptions and responses
that result from the use or anticipated use of a product, system or service”. A survey
that gathered a view of UX from 275 researchers has been conducted in [10]. It is
concluded that: “ the draft UX definition by ISO is a promising one, but the used terms
and the list of the possible objects of UX will require further explanations, e.g.,
‘anticipated use’” [10]. In 2010, the ISO 9241-210 definition was confirmed and
published. Hassenzahl defined the UX [23] as “A momentary, primarily evaluative
feeling while interacting with a product or service”. Hassenzahl‘s definition focused on
evaluating feelings while interacting with the product. Schulze defined the UX in [24]
as: “The degree of positive or negative emotions that can be experienced by a particular
user in a specific context during and after product use and that motivates for further
usage.” Schulze’s definition focused on the emotions felt while interacting with the
product.
For each UX’s definition, there are one or more elements that build the UX definition,
and the main element that identified from the UX survey [10] is the Perspective.
Perspective will help identify what and who has been affected by the changes of the UX
and below are the UX Perspectives that taken from the UX definitions [10]:
• Organization: concerned about designing a good UX of its products
• User: concerned about emotions and usability.
• Value: concerned about gained values
• Design: concerned about product look and its experience’s quality.
• Evaluation: concerned about the context of use and how to evaluate it.
33
To identify the relationships between the above perspectives and the UX impact, a
consideration was taken to what stated in many studies that the main components of
interaction between technology and user are system, product, user, and the context of
use [25] [26] [5], and Figure 17 explains this relationship.
Figure 17: Perspectives relationship
From Figure 17, the UX-related elements are:
1. UXà Organization: This will look to provide valuable products for the users.
2. UX à User: This is the person who is looking for a valuable and efficient
usage of the product.
3. UX à Context of use: which is the intermediate bridge between organization
and user, and it has an impact on both of them.
4. UX à Product: This is concerned the product design.
So, as a summary, the UX is a value centric and all UX elements related to the value.
34
3.7.2.2 UX research disciplines
In this SLR, the researchers focused on different disciplines depending on their
background or their purpose for the study and below are the main disciplines that are
studied in this SLR:
• UX and values: This is related to the impact of the UX’s level on one of the below
parts [25][27][28][29]:
o User’s value from using the product.
o Organization’s value from providing the product to the users.
o Conditional values that are related to the context of use and linked to the user
or company or both of them.
• UX and brand: This is related to the image of the organization or the marketing of
the products and how the UX got affected by them [25][28][5].
• UX and user’s needs: This is related to the two types of needs of the user which
are Pragmatic & Hedonic, and how satisfying the users will lead to a change in
UX’s level [16][25][30] [26][24][31] [32].
• UX and development process: This is related to the impact of the development
process on UX [32] [33].
• UX and technologies: This is related to the technologies that have been used to
deliver the product, the level of operation after releasing the product to the users,
and how the provided operation may affect the UX’s level [28] [5] [24][31] [34].
35
Table 12Error! Reference source not found. summarizes the categories of these
disciplines and the relationships between them. These categories represented as
Dimensions, and Figure 18 explains the relationship between Disciplines and related
Dimension.
Table 12: Suggested categories/Dimensions for research disciplines
Discipline Dimension
Values & Experience Value & Overall Experience (VX)
Context
Brand & Experience Organization Brand Experience (BX)
User & Experience User Needs Experience (NX)
Development Process &
Experience
Technology Experience (TX)
Technologies & Experience Technology Experience (TX)
Also, taking into consideration the UX definitions’ elements from the previous section,
another dimension needs to be added, which is Context. Context is a dimension that
covers the interaction between all other dimensions as shown in Figure 18.
Figure 18: Identified Dimensions from the SLR and its relationships
36
3.7.2.3 UX Aspects
From this SLR, the main UX aspects have been identified which mentioned in
the primary studies, and have a direct or indirect impact on the UX [24]. These aspects
are related to either the user himself, the system that is providing the service or the
context of use between them. Those three components building the relation between the
user and the system [5][25] [26]. The same terms that have been used in these primary
studies were used as much as possible to keep the same terminologies and Table
13Error! Reference source not found. explains the meaning of these aspects.
Moreover, the terms Pragmatic quality which is related to doing the task itself [35] such
as usability ;and Hedonic which is related to the expression of self and user’s personal
values [36] such as emotions were identified. The user’s related aspects can be either
related to achieving explicit goals which are pragmatic aspects, or achieving implicit
goals which are hedonic aspects. Figure 19 explains all aspects and the number of
papers that mentioned it.
Figure 19: UX aspects and number of related papers
3 2
1 1
10
15
10
4
20
854432
23
9
8
1 3 59 2
# of Papers / UX AspectsBrandingEveryday OperationsMarketingBusiness CommunicationsContext of useSpatio-TemporalUser JourneyCulturalEmotionalHedonicTrustworthinessAestheticsFunPrivacySensualUsabilityFunctionalityUsefulnessPlatform TechnologyInfrastructureService Response timeVisual AttractivenessDevelopment Process
37
Table 13: UX Aspects' descriptions
UX Aspect Description
Branding [25] This referred to the Brand identity and the organization promises which have an impact on the trustworthiness.
Everyday Operations [25]
This referred to the real business work such as selling the product or serving the customer which has an effect on the usability of the service.
Marketing [25] This referred to communicating the Brand identity and building an emotional link between the client and the organization.
Business Communications [25]
This referred to communicate the Business goals clearly inside the organization to construct a valuable product for the client which meets the organization values at the same time.
Context of use [26] This referred to the situation and the environment around the user which he is using the product or the service and it is a dynamic and could not be explicitly defined.
Spatio-Temporal [37]
This referred to the time space, period of usage between user and product as it has an impact on the learnability and usability.
User Journey [9] This referred to the loyalty between user and the organization as it has impact on the emotions and trustworthiness.
Cultural [38] This referred to the user’s culture which has an impact on the usability and emotions as well.
Emotional [37] This referred to all emotions during the product usage, and it is explicitly mentioned in some studies.
Hedonic [30] This referred to the category of all non-goals aspects, and it is explicitly mentioned in some studies and use it as is.
Trustworthiness [30] This referred to the trust between the user and the organization during the usage of its products, and it has an impact on different aspects such as emotions.
Aesthetics [30] This referred to the sense of beauty and how the user feels about it.
Fun [30] This referred to the enjoyment of using the product, and it has an intersection with emotional aspects.
Privacy [39] This referred to the trust between the user and the product and how he is feeling safe, and it has an intersection with the trustworthiness.
Sensual [37] This referred to the experience derived from the sense when using the product.
Usability [30] This referred to the ease of use and learnability of using the product.
38
Table 14: UX Aspects' descriptions (continued)
Functionality [40] This referred to the state of being functional and can meet the goals of using the product.
Usefulness [30] This referred to the state of being useful.
Platform Technology [41]
This referred to the software technology that has been used to build the product which has an impact on the usability and the emotions as well.
Infrastructure [42]
This referred to the hardware technology that has been used to build the product from both sides (client and Server) as it has an impact on the usability and it has interaction with Service Response Time aspect.
Service Response time [43]
This referred to the time that the organization takes to response to the user by a valuable result, and it is mainly here related to the technology.
Visual Attractiveness [30]
This referred to User Interface, and it has an impact on the usability and emotions as well.
Development Process [44]
This referred to the development process and how it is covering the user’s needs and expectations.
Taking into consideration the UX disciplines’ dimensions, the aspects that affect the UX
can be categorized into one of the below types:
• User’s needs aspects: which is related to the User dimension and can be divided
into two categories[36] [35]:
o Pragmatic aspects: This is related to the usability and do-goal qualities.
o Hedonic aspects: This is related to the feelings and be-goal qualities.
• Brand aspects: This is related to the marketing and the business communications
between the user and the organization and can be linked to the Organization
Brand dimension.
• Technology aspects: This is related to the development and production
technologies that have an impact on the overall experience and can be linked to
the Technology dimension. It can be divided into four types:
o UX designs (UXD) aspects: This is related to the UI designs.
o Development technology aspects: This is related to development technologies
that are used in delivering the product.
39
o Hardware aspects: This is related to the infrastructure of either the client side
or the server side.
o Operation aspects: This is related to the technologies that are used to monitor
the production and response to the users.
• The context of use aspects: This is related to any aspect that is not related to one
of the previous aspects but has an impact on them and can be linked to the
Context dimension.
From the previous classifications and the identified UX aspects from the primary
studies, the below table is filled to show the relationship between the UX aspects, its
categories and UX dimensions as shown in Table 15.
Table 15: Suggested categories for UX aspects and their relationship to UX dimensions
UX Aspect Category Dimension
Branding Brand BX
Everyday Operations Brand BX
Marketing Brand BX
Business Communications Brand BX
Context of use Context BX
Spatio-Temporal Context BX
User Journey Context BX
Cultural Context NX
Emotional Hedonic NX
Hedonic Hedonic NX
Trustworthiness Hedonic NX
Aesthetics Hedonic NX
Fun Hedonic NX
Privacy Hedonic NX
Sensual Hedonic NX
Usability Pragmatic NX
Functionality Pragmatic NX
Usefulness Pragmatic NX
40
Table 16: Suggested categories for UX aspects and their relationship to UX dimensions
Platform Technology Development Technology TX
Infrastructure Hardware TX
Service Response time Operation TX
Visual Attractiveness UXD TX
Development Process Context TX
3.7.2.4 UX Measurement methods
Roto [15] has classified the measurement methods depending on which study type
each can be used for. Also, another classification are suggested by Rajeshkumar [12]
which depends on different perspectives, and Error! Reference source not found.
shows the overall classifications:
Table 17: Categories of UX measurement methods from the SLR
Classification Type Classification Author Study Type • Lab Studies
• Field Studies • Survey • Expert Evaluation • Mixed Methods
Roto [15]
Study Type • Lab Studies • Field Studies • Survey • Expert Evaluation
Rajeshkumar [12]
Development Phase • Concept • Analysis • Design • Prototype • Implement
Research Type • Qualitative • Quantitative
User/Evaluator • User-Oriented • Expert-Oriented
Time Restriction • Short Evaluation • Long Evaluation
Period of Experience • Moment of Whole Application • Episode of Whole Application
41
• Overall of Whole Application
An identification for the measurement methods that have been used to measure the UX
aspects in the primary studies has been done. Those methods are used to measure the
UX aspects, either alone as one method or mixed with other methods. Also, the same
method is used to measure more than one aspect in some cases.
In these primary studies, some of the methods are mentioned by the general name, such
as questionnaire, interview, etc., while the remaining methods are mentioned as part of
the commercial name, such as Attrakdiff, which is a Questionnaire. Table 18 shows the
measurement methods and number of related papers.
Table 18: Extracted UX measurement methods from the selected studies
Measurement Method # of papers Measurement Method # of
papers Questionnaire 6
Experience Report (Open-ended Questions) 1
Interview 5 User’s Feedback 1 Experience Report 5 Diary 1 Expert Review 4 Valence Method 1
Attrakdiff 4 Expressing Experiences and Emotions (3E) 1
Electromyography Responses (EMG) 3 Probe 1
Self-Assessment Manikin (SAM) 3 Fuzzy Cognitive Map ( FCM) 1
Persona 3 UNeeQ 1 User Profile 3 Survey 1
UX Curve 2 System Usability Scale questionnaire (SUS) 1
Observation 2 Sentence Compaction Survey 1
PLEX 2 Emocards 1
Day Reconstruction Method 2 Emotion Words Priming List (EWPL) 1
Lean UX 1 PrEmo 1
Prototype 1 Group Experience Evaluation Method 1
BadIdeas 1
Subjective Usability Measurement Inventory (SUMI) 1
Electro Dermal Activity (EDA) 1 SUXES 1 Specification Document 1 Attrak-Work 1 Think-Aloud 1 Focus Group 1
AHP 1 Experience Sampling Method (ESM) 1
42
Table 19: Extracted UX measurement methods from the selected studies
Tracking Realtime User Experience (TRUE) Method 1
Goal Question Metric (GQM)
1
DrawUX 1 Experience Evaluation Model (EEM)
1
UX Cards 1 User Experience Metric(UXM)
1
UX Concept Testing 1 SPUX 1
iScale 1 Self-Expression Template Method
1
An important point that needs to be mentioned is that those measurement methods are
not necessarily used to measure all UX aspects in the study mentioning the method, and
the goal was to understand the methods’ popularity along with the aspect.
The focus will be on the generic methods, so it can be a starting point to link the
methods with the aspects and Error! Reference source not found. shows the UX
measurement methods that are mentioned using the general names:
Table 20: UX measurement Methods - General Names
General Measurement Methods
Questionnaire
Interview
Expert Review
Persona
Survey
User Profile
Observation
Prototype
Specification Document
Focus Group
Think-Aloud
Experience Report
Users’ Feedback
Diary
43
3.7.2.5 UX Frameworks and related participants
In this SLR, there are participations from the authors about UX, but none of
them shows a structured concept about UX that can be used as a framework. The
available participations have been reviewed to understand how it is related to UX, and if
there is any discipline that can help to understand the UX relations. Gathered disciplines
in 3.7.2.2 which converted to be UX dimensions are used to validate the available
participations, and to check the UX dimensions coverage. This was applied to validate
the need for a new framework that gather available participations in one place. Error!
Reference source not found. Shows participations, and covered UX dimensions.
Table 21: Related UX participations and UX dimensions coverage
Participations From Year UX dimensions
Mahlke [30] 2005 User Needs Experience
Vyas [45] 2006 User Needs Experience
Jetter [25] 2007 User Needs Experience, Brand Experience
Thayer [46] 2009 User Needs Experience, Brand Experience
Moller [26] 2009 User Needs Experience, Technology Experience
Schulze [24] 2010 User Needs Experience
Gegner [27] 2011 Brand Experience
Zhi Chen [31] 2011 User Needs Experience, Technology Experience
Gross [16] 2012 User Needs Experience, Value Experience
Anitha [33] 2012 Technology Experience
Jian [28] 2012 Technology Experience, Brand Experience
Fuchsberger [29] 2012 Value Experience, User Needs Experience
Gao [34] 2013 User Needs Experience, Technology Experience
Kujala [47] 2013 Brand Experience
Fronemann [32] 2014 User Needs Experience, Technology Experience
Zhenzhen [5] 2014 Brand Experience
44
3.8 Validity Threats
As stated by Pereira in [48] that a key issue when performing the SLR is validating
the results. In this section, the possible threats and how to minimize its impact will be
discussed, and it will be categorized as follow[49].
• Investigator Bias
This research has been done by an individual researcher, and it could have more
threats than other SLRs which done by more than one researcher, and this may
have an impact on selected primary papers, extracted data, and its results.
However, this SLR was reviewed and validated by the supervisor who did too
many SLRs on various topics. In addition to that, all primary papers have been
reviewed more than one time during the various phases. For instance, abstract
and titles have been scanned at least three times as explained in section 3.3.
However, a claim that these have solved this type of bias could not be exist, but
it reduced its impact.
• Publication Bias
In this SLR, the primary papers have been selected from four digital libraries
using specific keywords derived from the research questions as explained in
section 3.3. All papers that were not accessible or not related to the domain were
excluded, and all excluded papers are tracked and can be reviewed by any other
researchers if needed. However, the primary studies were limited to papers, and
magazines and books were excluded to increase the reliability if possible.
Hence, that ACM has 59% from selected papers while IEEE has 25% which
refer to the majority of papers are related to a computer technology, not an
engineering. The research was limited to these digital libraries due to the
availability, and because of the popularity of these libraries. No claim that
coverage for all studies in this domain is happened because there are too
different digital libraries and source may contain relevant studies.
45
• Threats to identification of Primary Studies
Search strategy in this SLR has been designed and followed to cover as many as
possible UX related studies. Four digital libraries which are well known and
common sources in such topics are used, and a list of keywords derived from the
research questions have been used to conduct the search as explained in
section 3.3. Search strategy and all results are explained and can be repeated at
any time.
• Threats to Data extraction and its Results Consistency
Study Selection criteria as explained in Section 3.4, has been used to include the
related studies. The studies that are related to technology and UX at the same
time were included without any limitation to a particular product or technology.
However, meta-data attributes have been collected as per procedure explained in
sections 3.3 and 3.6 and it can be repeated and enhanced at any time. This is led
to increasing the reliability of the extracted data which are used in Data Analysis
stage.
46
4 Proposed Framework
In this chapter, the results of the extracted data from the SLR are considered in
detail aiming to define a framework that combine collected data.
4.1 UX Definitions Perspective [10] is the main element that used to build UX definition, and it will help
to identify who has been affected by the changes of the UX. From Figure 17, the UX-
related perspectives are explained as follow:
• Organization: This will look to provide valuable products for the users.
• User: The user is looking for a valuable and appropriate usage of the product.
• The context of use: This has an impact on both elements which are organization
and user, and is the intermediate bridge between them.
• Product: This is concerned the product design.
Also, the UX is a value centric, and all UX perspectives are related to the value.
4.2 UX Dimensions Dimension is the category of the research disciplines that has been followed in
the primary studies to study the UX. The UX dimensions that are related to the research
studies in the primary studies are listed below and was shown before in Error!
Reference source not found..
• Value: This is related to the studies that have been focused on the gained value.
• User Needs Experience (NX): This is related to the studies that have been
focused on user needs and gained qualities.
• Brand Experience (BX): This is related to the studies that have focused on the
organization’s brand image.
47
• Technology Experience (TX): This is related to the studies that have focused on
the technology that has been used to deliver the product or the service.
• Context: This is related to the studies that have focused on the context of use and
the interaction between the previous dimensions.
As a result of the UX definitions and UX dimensions, a relationship between these
elements can be built as shown in Figure 20. In this figure, Value is the centric point for
different UX dimensions, and surrounding Context is related all of them. By this way,
available UX dimensions are gathered in one place and the relation between them are
explained.
Figure 20: The relation between value and UX dimensions
48
4.3 UX Aspects Aspect is the quality attribute that has an impact on UX. In Error! Reference source
not found. a list of the aspects is identified. The classifications of these aspects, the
relationship between them and the UX dimensions are shown in Figure 21, and below
detailed descriptions of these classifications:
• User’s needs aspects: This is related to the User dimension and can be divided
into two categories [36] [35]:
o Pragmatic aspects: These are related to the usability and the do-goal
qualities.
o Hedonic aspects: These are related to the stimulation and be-goal qualities.
• Brand aspects: Aspects related to the marketing and the business
communications between the user and the organization and can be linked to the
Organization Brand dimension.
• Technology aspects: Aspects related to the development and production
technologies that have an impact on the overall experience and can be related to
the Technology dimension. This category can be divided into four types:
o UX designs (UXD) aspects: These are related to the UI designs.
o Development technology aspects: These are related to the development
technologies that are used in delivering the product.
o Hardware aspects: These are related to the infrastructure of either the client
side or the server side.
o Operation aspects: These are related to the technologies that are used to
monitor the production and response to the users.
• The context of use aspects: Aspects related to any aspect that is not related to
one of the previous aspects but has an impact on them, and this can be linked to
the Context dimension.
49
Figure 21: UX Aspects Classifications
A list of the aspects and all related categories explained in Error! Reference source
not found.. The Context dimension refers to the interaction between other dimensions,
and it has the following aspects:
o Spatio-Temporal
o User Journey
o Cultural
o Context of Use
o Development Process
50
The aspect Context of Use is a general aspect and could not be related to a particular
dimension, while the interaction between the technology dimension and context
dimensions can be represented by the Development Process and can be placed within
the Technology/Context interaction space. However, Cultural aspect is related to User
dimension and context at the same time. Also, User Journey and Spatio-Temporal
which is related to the period of experience that the user has with the product or the
organization can be placed in Brand dimension.
As a result of this aspect categorization and UX dimension, the relationship between
them can be rebuilt again to be as shown in Figure 22.
Figure 22: Relation between UX Aspects categories and UX dimensions
51
After the aspects are listed as explained in Error! Reference source not found., the
relationships between the aspects and the categorizations are reflected in the framework,
see Figure 23.
Figure 23: Relation between UX aspects and all UX dimensions
52
4.4 UX Measurement Methods Measurement Method is a method to measure the UX aspect and to get
information about the fulfillment level of this aspect. Too many methods have been
identified from the primary studies. Most of them belong to the categories (generic
methods) that were shown in Error! Reference source not found. but are identified by
another name – either a commercial name such as Attrakdiff [15] or a proposed name
because it was mixed methods such as UNeeQ [32]. After reviewing those generic
methods and their use in the primary studies, a categorization has been identified
according to the applicability of the method to the UX dimension in general. Error!
Reference source not found. shows the main measurement methods (general methods)
that have been proposed to be used with corresponding UX dimensions to measure its
Aspects.
Table 22: Suggested relationship between the UX measurement methods and the UX dimensions
Method UX Dimensions
NX BX TX
Questionnaire √ √
Interview √ √
Expert Review √ √
Persona √ √
User Profile √ √
Observation √
Prototype √ √
Specification Document √
Think-Aloud √
Experience Report √ √
Users’ Feedback √ √
Diary √
Survey √ √
Focus Group √
For the Context dimension, no measurement methods are assigned, because the UX
aspects in this dimension are not applicable to be measured.
53
4.5 Final Proposed Framework
The framework shown in Figure 24 covers gathered UX dimensions, aspects and
UX measurement methods which built based on the summary of Table 12, Table 15,
Table 20 and Table 22.
Figure 24: Proposed Framework that covers gathered UX dimensions, aspects categories, aspects, and measurement
methods
This framework is validated using the comparison with the identified frameworks from
the primary studies, and it is found that this framework covers gathered UX dimensions
54
compared to the remaining framework as shown in Error! Reference source not
found.. This framework can be used to understand the UX aspects and factors that
affect the UX in general. Also, it can be used as strategic guidelines for including the
UX activities in an organization.
Table 23: Related UX participations and UX dimensions coverage, and proposed framework
Participation From Year UX dimensions
Mahlke [30] 2005 User Needs Experience
Vyas [45] 2006 User Needs Experience
Jetter [25] 2007 User Needs Experience, Brand Experience
Thayer [46] 2009 User Needs Experience, Brand Experience
Moller [26] 2009 User Needs Experience, Technology Experience
Schulze [24] 2010 User Needs Experience
Gegner [27] 2011 Brand Experience
Zhi Chen [31] 2011 User Needs Experience, Technology Experience
Gross [16] 2012 User Needs Experience, Value
Anitha [33] 2012 Technology Experience
Jian [28] 2012 Technology Experience, Brand Experience
Fuchsberger [29] 2012 Value, User Needs Experience
Gao [34] 2013 User Needs Experience, Technology Experience
Kujala [47] 2013 Brand Experience
Fronemann [32] 2014 User Experience, Technology Experience
Zhenzhen [5] 2014 Brand Experience
Proposed Framework 2015 Value, Brand Experience, User Needs Experience,
Technology Experience
55
5 Conclusion & Future Work
As a result of the all previous stages, the conclusion of findings, and future work
that required to move forward will be explained.
5.1 Conclusion
In this research, UX related results that can help in measuring user experience with
mobile applications have been presented. The purpose of this research is exploring how
the user experience on a mobile application is affected and how to measure it. To gain
this purpose, the below questions that derived the understanding are identified:
1. What are the different definitions of user experience in literature?
2. What are the aspects that affect the user experience?
3. What are the methods identified to measure each of the specified aspects in the literature?
4. What kind of framework can be developed to understand and measure the user experience with mobile applications?
From the systematic literature review and the analysis of extracted data, the possible
UX dimensions that have an impact on the UX either during the development or after
the product use are gathered. In addition to the UX dimensions, the relationship between
those dimensions and UX aspects that have a direct or indirect impact on the UX is
scanned. However, the possible methods that can be used to measure those aspects are
scanned. All these results have been analyzed and consolidated in one framework that
explains the relationship between UX dimensions, UX aspects and UX measurement
methods as shown in Figure 24. This framework is a promising one and can be used as
starting step for understanding how the user experience got affected in mobile
applications and how to measure it.
The proposed framework can be used as strategic guidelines for anyone who wants to
include the UX activities in his or her organization, as it shows the main dimensions
that affect the UX and it can help strategically to know the roles and responsibilities of
these concerned dimensions.
56
5.2 Future Work Concerning the future work, more work is required on the below points:
• Use a cumulative method to measure the UX from a long-term usage perspective,
as the current methods in the proposed framework are mainly for short-term
usage. The UX Curve method is mentioned in literature and can be used in such
cases and needs to be included in this framework [50][51].
• Build a relationship and integrate the proposed framework with a common
maturity model such as CMMI. This will help to improve strategically the UX
work level and a study in literature referring to that [22].
• Identify the proper development process that brings more consistency between the
proposed framework and the Development Process aspect that exists in the
Context dimension. Currently, no specific process is suggested, while the Lean
UX is proposed as a UX-centric development process [52] in literature.
• For each UX measurement method, a template and general guidelines are required
to assure that the questions and statements used in this method are covering all
related aspects. This can be reached by identifying the goals that satisfy the
aspect, and the Analytic Hierarchy Process (AHP) [31] method can help in this
case.
• An Experimental work on the commercial UX methods to assure its applicability
to measuring the different UX aspects. This will help to simplify the proposed
framework and give directions about which method can be used.
• More analysis in the Context dimension is needed to identify more aspects, so it
can be related to other dimensions.
57
Appendix
A. Selected Primary Studies Table 24: Selected primary papers and their points
Study ID
Bibliography ID
Study ID
Bibliography ID
Study ID
Bibliography ID
1 [30] 41 [53] 89 [54] 2 [37] 42 [55] 91 [56] 3 [40] 43 [24] 92 [57] 4 [58] 45 [59] 93 [60] 6 [39] 47 [61] 94 [62] 7 [63] 48 [64] 95 [65] 8 [45] 49 [66] 96 [34] 9 [67] 51 [35] 97 [68]
10 [69] 52 [70] 98 [71] 11 [43] 53 [72] 99 [73] 12 [74] 54 [75] 100 [7] 13 [76] 55 [77] 101 [78] 14 [79] 57 [80] 102 [47] 15 [25] 58 [81] 103 [82] 16 [83] 59 [84] 104 [85] 17 [86] 61 [87] 107 [88] 18 [23] 62 [27] 108 [89] 19 [38] 63 [90] 109 [91] 20 [92] 64 [93] 110 [94] 21 [95] 65 [31] 111 [12] 22 [96] 66 [50] 112 [97] 23 [98] 67 [8] 113 [99] 24 [9] 69 [100] 114 [101] 25 [102] 71 [103] 115 [104] 26 [10] 73 [16] 116 [105] 27 [15] 74 [106] 117 [32] 28 [107] 75 [108] 118 [109] 29 [44] 76 [110] 119 [111] 30 [42] 78 [112] 120 [113] 31 [36] 80 [114] 121 [115] 32 [46] 81 [41] 122 [116] 34 [117] 82 [118] 123 [119] 35 [120] 83 [33] 124 [121] 36 [122] 84 [123] 125 [5] 37 [26] 85 [28] 126 [52] 38 [13] 86 [124] 127 [51] 39 [125] 87 [29] 128 [126] 40 [127] 88 [128] 131 [129]
58
B. UX Aspects’ Studies Table 25: UX aspects and the related studies
Aspect Related Studies (Study ID)
Usability 1,3,7,19,12,14,16,20,31,34,37,39,43,48,54,59,69,76,81,93,97,98,123
Emotional 2,4,6,8,9,14,17,28,36,37,54,57,74,92,99,100,103,113,115,131 Spatio-Temporal 2,20,21,26,30,35,66,84,92,94,102,118,119,127,128 Context of Use 37,49,55,57,64,75,91,97,110,123 User Journey 24,29,42,47,48,51,83,107,125 Functionality 3,8,20,22,30,36,37,97,104 Hedonic 1,15,31,34,37,43,51,93 Visual Attractiveness 1,7,8,37,43,89,97,112,121 Usefulness 1,7,12,16,37,43,119,123 Trustworthiness 1,9,16,34,88 Service Response time 11,16,20,26,37 Branding 15,20,26,129 Cultural 19,23,40,45 Aesthetics 1,14,37,82 Fun 1,9,12,13 Privacy 6,37,97 Infrastructure 30,34,37 Everyday Operations 15,20 Sensual 2,8 Marketing 15 Business Communications 15 Platform Technology 81 Development Process 29,51
59
C. UX Measurement Methods’ Studies Table 26: UX measurement methods and the related studies
Method Study ID Method Study ID
Questionnaire 30,32,37,43,92,96 Experience Report ( Open-Ended Questions ) 41
Interview 9,12,32,59,103 Users’ Feedback 124 Experience Report 32,55,59,63,95 Diary 127 Expert Review 30,34,37,122 Valence Method 52
Attrakdiff 27,29,59,119,128 Expressing Experiences and Emotions (3E) 100
Electromyography Responses (EMG) 14,115,131 Probe 23
Self-Assessment Manikin (SAM) 14,59,100 Fuzzy Cognitive Map ( FCM) 71
Persona 6,63,110 UNeeQ 117 User Profile 7,12,110 Survey 16,119
UX Curve 66,127 System Usability Scale Questionnaire (SUS) 96
Observation 27,28 Sentence Compaction Survey 40 PLEX 55,121 Emocards 27 Day Reconstruction Method 35,92 Emotion Words Priming List
(EWPL) 28
Lean UX 125 PrEmo 74
Prototype 63 Group Experience Evaluation Method 25
BadIdeas 42 Subjective Usability Measurement Inventory (SUMI) 14
Electro Dermal Activity (EDA) 14 SUXES 109
Specification Document 37 Attrak-Work 31
Think-Aloud 43 Focus Group 27
AHP 65 Experience Sampling Method(ESM) 100
Tracking Realtime User Experience (TRUE) Method
27 Goal Question Metric (GQM) 101
DrawUX 80 Experience Evaluation Model (EEM)
11
UX Cards 122 User Experience Metric (UXM) 22 UX Concept Testing 51 SPUX 48 iScale 119,128 Self-Expression Method 112
60
Bibliography
[1] ITU, “ICT Facts & Figures: The World in 2015.” p. 6, 2015. [2] Mubaloo, “Insurance Mobility : the vision.” Mubaloo-White Paper, 2014. [3] A. Haapajoki, “Entrepreneurial marketing in the B2C mobile application
business : A netnographic study across four expert blogs,” 2014. [4] CITC, “Mobility in Saudi Arabia,” vol. 1, 2015. [5] C. B. Zhenzhen Zhao, “A Design Framework of Branded Mobile Applications,”
in MobileHCI ’14 Proceedings of the 16th international conference on Human-computer interaction with mobile devices & services, 2014, pp. 507–512.
[6] S. Khalaf, “The Cable Industry Faces The Perfect Storm: Apps, App Stores and Apple,” Flurry Insights, 2015. [Online]. Available: http://flurrymobile.tumblr.com/post/128773968605/the-cable-industry-faces-the-perfect-storm-apps. [Accessed: 09-Apr-2016].
[7] L. Yong, “User experience evaluation methods for mobile devices,” in Innovative Computing Technology (INTECH), 2013 Third International Conference on, 2013, pp. 281–286.
[8] E. L.-C. Law, “The measurability and predictability of user experience,” in Proceedings of the 3rd ACM SIGCHI symposium on Engineering interactive computing systems - EICS ’11, 2011, p. 1.
[9] K. Väänänen-vainio-mattila, M. Hassenzahl, C. Landau, and I. Fort, “Towards Practical User Experience Evaluation Methods,” in 5th COST294-MAUSE Open Workshop on Valid Useful User Experience Measurement (2008), 2008, pp. 1–4.
[10] E. L. Law, V. Roto, and M. Hassenzahl, “Understanding, Scoping and Defining User eXperience: A Survey Approach,” in CHI ’09 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2009, pp. 719–728.
[11] ISO, “Ergonomics of human-system interaction - Part 210: Human-centred design for interactive systems,” Int. Stand. Organ., vol. 2010, pp. 1–32, 2010.
[12] S. Rajeshkumar, R. Omar, and M. Mahmud, “Taxonomies of User Experience (UX) evaluation methods,” in Research and Innovation in Information Systems (ICRIIS), 2013 International Conference on, 2013, pp. 533–538.
[13] N. Bevan, “What is the difference between the purpose of usability and user experience evaluation methods ?,” in INTERACT 2009, 2009.
[14] N. Bevan, “Extending quality in use to provide a framework for usability measurement,” Proc. HCI Int. 2009, San Diego, California, USA, vol. 5619 LNCS, no. 1991, pp. 13–22, 2009.
[15] V. Roto, M. Obrist, and K. Väänänen-vainio-mattila, “User Experience Evaluation Methods in Academic and Industrial Contexts,” in Interact 2009 conference, User Experience Evaluation Methods in Product Development (UXEM’09). ,2009 .
[16] A. Gross and S. Bongartz, “Why do I like it?: investigating the product-specificity of user experience,” in NordiCHI ’12 Proceedings of the 7th Nordic Conference on Human-Computer Interaction: Making Sense Through Design, 2012, pp. 322–330.
61
[17] S. Engineering, “Evidence-Based Software Engineering for Practitioners,” Software, IEEE 22.1, pp. 58–65, 2005.
[18] B. A. Kitchenham and T. Dybå, “Evidence-based Software Engineering,” in Proceedings of the 26th international conference on software engineering, 2004, pp. 273–281.
[19] S. Keele, “Guidelines for performing systematic literature reviews in software engineering,” EBSE Tech. Rep., vol. 2.3, 2007.
[20] B. Kitchenham and P. Brereton, “A systematic review of systematic review process research in software engineering,” Inf. Softw. Technol., vol. 55, no. 12, pp. 2049–2075, Dec. 2013.
[21] Mendeley.com, “Getting started with Mendeley Desktop.” 2016. [22] A. L. Peres and S. L. Meira, “Towards a framework that promotes integration
between the UX design and SCRUM, Aligned to CMMI,” in 2015 10th Iberian Conference on Information Systems and Technologies (CISTI), 2015, pp. 1–4.
[23] M. Hassenzahl, “User experience (UX): towards an experiential perspective on product quality,” in Proceedings of the 20th International Conference of the Association Francophone d’Interaction Homme-Machine on - IHM '08, 2008, pp. 11–15.
[24] K. Schulze and H. Krömker, “A framework to measure user experience of interactive online products,” in Proceedings of the 7th International Conference on Methods and Techniques in Behavioral Research - MB ’10, 2010, pp. 1–5.
[25] H. Jetter and J. Gerken, “A Simplified Model of User Experience for Practical Application,” in 2nd International Open Workshop on COST294-MAUSE (2006), 2007, pp. 2–3.
[26] S. Moller, K.-P. Engelbrecht, C. Kuhnel, I. Wechsung, and B. Weiss, “A taxonomy of quality of service and Quality of Experience of multimodal human-machine interaction,” in 2009 International Workshop on Quality of Multimedia Experience, 2009, pp. 7–12.
[27] L. Gegner, M. Runonen, and T. Keinonen, “Oscillating between extremes: a framework for mapping differing views on User eXperience,” in DPPI ’11 Proceedings of the 2011 Conference on Designing Pleasurable Products and Interfaces, 2011, p. Article No. 57.
[28] C. Jian, L. Wen-wang, Z. Wen-an, G. Hai-sheng, C. Zhang, and M. Shao-fu, “Customer experience oriented service quality management,” in Robotics and Applications (ISRA), 2012 IEEE Symposium on, 2012, pp. 298–301.
[29] V. Fuchsberger, C. Moser, and M. Tscheligi, “Values in action (ViA): combining usability, user experience and user acceptance,” in CHI EA ’12 CHI '12 Extended Abstracts on Human Factors in Computing Systems, 2012, pp. 1793–1798.
[30] S. Mahlke, “Understanding users’ experience of interaction,” in EACE ’05 Proceedings of the 2005 annual conference on European association of cognitive ergonomics, 2005, pp. 251–254.
[31] Zhi Chen and Shangshang Zhu, “The research of mobile application user experience and assessment model,” in Computer Science and Network Technology (ICCSNT), 2011 International Conference on, 2011, pp. 2832–2835.
62
[32] N. Fronemann and M. Peissner, “User experience concept exploration: user needs as a source for innovation,” in NordiCHI ’14 Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational, 2014, pp. 727–736.
[33] P. Anitha and B. Prabhu, “Integrating requirements engineering and user experience design in Product life cycle Management,” in 2012 First International Workshop on Usability and Accessibility Focused Requirements Engineering (UsARE), 2012, pp. 12–17.
[34] Gao Changyuan, Wang Shiying, and Zhong Chongran, “Research on user experience evaluation system of information platform based on web environment,” in Measurement, Information and Control (ICMIC), 2013 International Conference on, 2013, pp. 558–562.
[35] S. Sproll, M. Peissner, and C. Sturm, “From product concept to user experience: exploring UX potentials at early product stages,” in NordiCHI ’10 Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries, 2010, pp. 473–482.
[36] H. Väätäjä, T. Koponen, and V. Roto, “Developing Practical Tools for User Experience Evaluation: A Case from Mobile News Journalism,” in ECCE ’09 European Conference on Cognitive Ergonomics: Designing beyond the Product --- Understanding Activity and User Experience in Ubiquitous Environments, 2009, p. Article No. 23.
[37] R. I. A. Mercuri, “Technology as Experience,” interactions - Funology, pp. 42–43, 2005.
[38] I. Lee, J. Kim, and Y. An, “Cultural Dimensions for User Experience : Cross-Country and Cross-Product Analysis of Users’ Cultural Characteristics,” in British Computer Society Conference on Human-Computer Interaction (2008), 2008, pp. 3–12.
[39] D. Swallow, M. Blythe, and P. Wright, “Grounding Experience: Relating Theory and Method to Evaluate the User Experience of Smartphones,” in EACE ’05 Proceedings of the 2005 annual conference on European association of cognitive ergonomics, 2005, pp. 91–98.
[40] N. Mcnamara, “Functionality, Usability, and User Experience: Three Areas of Concern,” in International Journal (2005), 2005, pp. 26–28.
[41] N. P. Huy and D. VanThanh, “Evaluation of mobile app paradigms,” in Proceedings of the 10th International Conference on Advances in Mobile Computing & Multimedia - MoMM ’12, 2012, p. 25.
[42] H. Wigelius and H. Väätäjä, “Dimensions of Context Affecting User Experience in Mobile Work,” in INTERACT ’09 Proceedings of the 12th IFIP TC 13 International Conference on Human-Computer Interaction: Part II, 2009, pp. 604–617.
[43] W.-F. Tung and S.-T. Yuan, “Optimization of Collaborative Service Systems Using Experience Evaluation Model,” in Services Computing, 2007. SCC 2007. IEEE International Conference on, 2007, pp. 244–247.
[44] V. Roto, H. Rantavuo, and V. Kaisa, “EVALUATING USER EXPERIENCE OF EARLY PRODUCT CONCEPTS,” in Proc. DPPI. ,2009 , p. 9.
[45] D. Vyas and G. C. Van Der Veer, “Experience as Meaning: Some Underlying
63
Concepts and Implications for Design,” in Proceedings of the 13th Eurpoean conference on Cognitive ergonomics: trust and control in complex socio-technical systems. ACM. ,2006 .
[46] A. Thayer and T. E. T. E. Dugan, “Achieving design enlightenment: Defining a new user experience measurement framework,” in Professional Communication Conference, 2009. IPCC 2009. IEEE International, 2009, pp. 1–10.
[47] S. Kujala, M. Obrist, M. Vogel, and A. Pohlmeyer, “Lost in Time: The Meaning of Temporal Aspects in User Experience,” in CHI ’13 Extended Abstracts on Human Factors in Computing Systems on - CHI EA '13 (2013), 2013, pp. 559–564.
[48] J. Pereira, K. Constantino, and E. Figueiredo, “A Systematic Literature Review of Software Product Line Management Tools,” Softw. Reuse Dyn. Syst. Cloud Beyond SE - 6, vol. 8919, pp. 73–89, 2014.
[49] a. Krapp, S. Hidi, and K. a. Renninger, “Human Factors in Software Development : A Systematic Literature Review,” Role Interes. Learn. Dev., p. 368, 2010.
[50] S. Kujala and V. Roto, “Identifying hedonic factors in long-term user experience,” in DPPI ’11 Proceedings of the 2011 Conference on Designing Pleasurable Products and Interfaces, 2011, p. 17.
[51] F. Sahar, J. Varsaluoma, and S. Kujala, “Comparing the Effectiveness of Electronic Diary and UX Curve Methods in Multi-Component Product Study,” in AcademicMindTrek ’14: Proceedings of the 18th International Academic MindTrek Conference: Media Business, Management, Content & Services, 2014, pp. 93–100.
[52] L. Liikkanen, H. Kilpiö, L. Svan, and M. Hiltunen, “Lean UX - The Next Generation of User-Centered Agile Development?,” in NordiCHI ’14 Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational, 2014, pp. 1095–1100.
[53] K. Väänänen-Vainio-Mattila, H. Korhonen, and J. Arrasvuori, “Let Users Tell the Story: Evaluating User Experience with Experience Reports,” in CHI EA ’10 CHI '10 Extended Abstracts on Human Factors in Computing Systems, 2010, pp. 4051–4056.
[54] M. Nivethika, I. Vithiya, S. Anntharshika, and S. Deegalla, “Personalized and adaptive user interface framework for mobile application,” in Advances in Computing, Communications and Informatics (ICACCI), 2013 International Conference on, 2013, pp. 1913–1918.
[55] P. Silva and J. Read, “A methodology to evaluate creative design methods: a study with the BadIdeas method,” in OZCHI ’10 Proceedings of the 22nd Conference of the Computer-Human Interaction Special Interest Group of Australia on Computer-Human Interaction, 2010, pp. 264–271.
[56] X. Sun and A. May, “A Comparison of Field-Based and Lab-Based Experiments to Evaluate User Experience of Personalised Mobile Devices,” in Advances in Human-Computer Interaction, 2013, pp. 1–9.
[57] S. Kujala and T. Miron-Shatz, “Emotions, experiences and usability in real-life mobile phone use,” in CHI ’13 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2013, pp. 1061–1070.
64
[58] K. Battarbee and I. Koskinen, “Co-Experience – User Experience as Interaction,” CoDesign, no. 1.1, pp. 5–18, 2005.
[59] Q. Yan, “Cultural versioning of mobile user experience,” in Proceedings of the 28th of the international conference extended abstracts on Human factors in computing systems - CHI EA ’10, 2010, p. 2943.
[60] M. Salehi, M. Salimi, and A. Haque, “The Impact of Online Customer Experience (OCE) on Service Quality in Malaysia,” World Appl. Sci. J. , pp. 1621–1631, 2013.
[61] A. P. O. S. Vermeeren, E. L. Law, and V. Roto, “User Experience Evaluation Methods : Current State and Development Needs,” in Proceedings of the 6th Nordic Conference on Human-Computer Interaction Extending Boundaries - NordiCHI ’10 (2010), 2010, pp. 521–530.
[62] M. Vogel, “Temporal evaluation of aesthetics of user interfaces as one component of user experience,” in AUIC ’13 Proceedings of the Fourteenth Australasian User Interface Conference, 2013, pp. 131–132.
[63] Z. Pei and Z. Zhenxiang, “A Framework for Personalized Service Website based on TAM,” in Service Systems and Service Management, 2006 International Conference on, 2006, pp. 1598–1603.
[64] Y.-L. Huang, T. Marsh, and A. D. Cheok, “Investigation of software patterns of user experience,” in Proceedings of the 7th International Conference on Advances in Computer Entertainment Technology - ACE ’10, 2010, p. 116.
[65] S. Hedegaard and J. G. Simonsen, “Extracting usability and user experience information from online user reviews,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI ’13, 2013, p. 2089.
[66] J. Arrasvuori, H. Korhonen, and K. Väänänen-Vainio-Mattila, “Exploring playfulness in user experience of personal mobile products,” in Proceedings of the 22nd Conference of the Computer-Human Interaction Special Interest Group of Australia on Computer-Human Interaction - OZCHI ’10, 2010, p. 88.
[67] J. Schrammel, A. Geven, and M. Tscheligi, “Using Narration to Recall and Analyse User Experiences and Emotions Evoked by Today’s Technology,” in International Conference on Design and Emotion (In Press). ,2006 .
[68] L. Arhippainen, “A Tutorial of Ten User Experience Heuristics.,” in MindTrek, 2013, pp. 336–337.
[69] S. N. Junaini, “Towards a Framework for the Assessment of Online Store Product Catalogs,” in Information and Communication Technologies, 2006. ICTTA ’06. 2nd, 2006, pp. 245–250.
[70] M. Burmester, M. Mast, K. Jäger, and H. Homans, “Valence method for formative evaluation of user experience,” in Proceedings of the 8th ACM Conference on Designing Interactive Systems - DIS ’10, 2010, p. 364.
[71] D. Raptis, N. Tselios, J. Kjeldskov, and M. Skov, “Does size matter ? Investigating the impact of mobile phone screen size on users ’ perceived usability , effectiveness and efficiency .,” in Proceedings of the international conference on Human-computer interaction with mobile devices and services (MobileHCI ’13) (2013), 2013, pp. 127–136.
[72] K. Rodden, H. Hutchinson, and X. Fu, “Measuring the User Experience on a
65
Large Scale : User-Centered Metrics for Web Applications,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (2010), 2010.
[73] C. Von Saucken, C. von Saucken, I. Michailidou, and U. Lindemann, “Emotional mental model,” in Industrial Engineering and Engineering Management (IEEM), 2013 IEEE International Conference on, 2013, pp. 802–806.
[74] Y. S. Lee, B. M. Kleiner, P. D, and M. A. Nussbaum, “OLDER ADULTS ’ USER EXPERIENCES WITH MOBILE PHONES: IDENTIFICATION OF USER CLUSTERS AND USER REQUIREMENTS,” 2007.
[75] Wei Liu, Daoli Huang, and Yan Zhang, “Research on fuzzy comprehensive evaluation of user experience,” in Information Computing and Telecommunications (YC-ICT), 2010 IEEE Youth Conference on, 2010, pp. 122–125.
[76] J. Holt and S. Lock, “MARPLE Investigates: An ‘Adversarial’ Approach to Evaluating User Experience,” in BCS-HCI ’07 Proceedings of the 21st British HCI Group Annual Conference on People and Computers, 2007, pp. 141–144.
[77] H. Korhonen, J. Arrasvuori, and K. Väänänen-Vainio-Mattila, “Analysing User Experience of Personal Mobile Products Through Contextual Factors,” in Proceedings of the 9th International Conference on Mobile and Ubiquitous Multimedia, 2010, pp. 11:1–11:10.
[78] J. Tan, K. Ronkko, and C. Gencel, “A Framework for Software Usability and User Experience Measurement in Mobile Industry,” in Software Measurement and the 2013 Eighth International Conference on Software Process and Product Measurement (IWSM-MENSURA), 2013 Joint Conference of the 23rd International Workshop on, 2013, pp. 156–164.
[79] S. Mahlke and M. Thüring, “Studying antecedents of emotional experiences in interactive contexts,” in Proceedings of the SIGCHI conference on Human factors in computing systems - CHI ’07, 2007, p. 915.
[80] C. Tselios, I. Politis, M. Tsagkaropoulos, and T. Dagiuklas, “Valuing quality of experience: A brave new era of user satisfaction and revenue possibilities,” in FITCE Congress (FITCE), 2011 50th, 2011, pp. 1–6.
[81] J. Bargas-Avila and K. Hornbæk, “Old wine in new bottles or novel challenges: a critical analysis of empirical studies of user experience,” in CHI ’11 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2011, pp. 2689–2698.
[82] M. Winckler, C. Bach, and R. Bernhaupt, “Identifying User Experience Dimensions for Mobile Incident Reporting in Urban Contexts,” IEEE Trans. Prof. Commun., pp. 97–119, Jun. 2013.
[83] L. Patrício, R. P. Fisk, and J. F. E. Cunha, “Designing Multi-Interface Service Experiences The Service Experience Blueprint,” J. Serv. Res. , no. 10.4, pp. 318–334 ,2008 .
[84] C. Bach, N. Gauducheau, and P. Salembier, “Combining interviews and scales in the multidimensional evaluation of user experience: a case study in 3D games,” in ECCE ’11 Proceedings of the 29th Annual European Conference on Cognitive Ergonomics, 2011, pp. 157–160.
[85] A. Sonnleitner, M. Pawlowski, T. Kässer, and M. Peissner, “Experimentally Manipulating Positive User Experience Based on the Fulfilment of User Needs,”
66
in Human-Computer Interaction–INTERACT 2013. ,2013 , pp. 555–562. [86] Y. Lim, J. Donaldson, H. Jung, B. Kunz, D. Royer, S. Ramalingam, S.
Thirumaran, and E. Stolterman, “Emotional Experience and Interaction Design,” in Affect and Emotion in Human-Computer Interface Design (2008), 2008, pp. 116 – 129.
[87] C. Mulwa, S. Lawless, M. Sharp, and V. Wade, “The evaluation of adaptive and personalised information retrieval systems : a review,” in International Journal of Knowledge and Web Intelligence (2011), 2011, vol. 2, pp. 138–156.
[88] J. Mashapa, E. Chelule, D. Van Greunen, and A. Veldsman, “Managing User Experience – Managing Change,” in Human-Computer Interaction–INTERACT 2013. ,2013 , pp. 660–677.
[89] S. Wirtz and E. Jakobs, “Improving User Experience for Passenger Information Systems. Prototypes and Reference Objects,” IEEE Trans. Prof. Commun., pp. 120–137, 2013.
[90] M. Obrist, E. Beck, D. Wurhofer, and M. Tscheligi, “Experience characters: a design tool for communicating mobile phone experiences to designers,” in MobileHCI ’11 Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services, 2011, pp. 385–394.
[91] T. Keskinen, J. Hakulinen, T. Heimonen, M. Turunen, S. Sharma, T. Miettinen, and M. Luhtala, “Evaluating the experiential user experience of public display applications in the wild,” in Proceedings of the 12th International Conference on Mobile and Ubiquitous Multimedia - MUM ’13, 2013, pp. 1–10.
[92] P. Ketola and V. Roto, “Exploring User Experience Measurement Needs,” in Proc. of the 5th COST294-MAUSE Open Workshop on Valid Useful User Experience Measurement (VUUM). ,2008 , pp. 23–26.
[93] V. Roto, H. Väätäjä, S. Jumisko-Pyykkö, and K. Väänänen-Vainio-Mattila, “Best practices for capturing context in user experience studies in the wild,” in Proceedings of the 15th International Academic MindTrek Conference on Envisioning Future Media Environments - MindTrek ’11, 2011, p. 91.
[94] R. Jain, J. Bose, and T. Arif, “Contextual adaptive user interface for Android devices,” in India Conference (INDICON), 2013 Annual IEEE, 2013, pp. 1–5.
[95] E. Karapanos, M. Hassenzahl, and J.-B. Martens, “User experience over time,” in Proceeding of the twenty-sixth annual CHI conference extended abstracts on Human factors in computing systems - CHI ’08, 2008, p. 3561.
[96] A. Joshi and S. Tripathi, “User Experience Metric and Index of Integration: Measuring Impact of HCI Activities on User Experience,” in I-USED ,2008 .
[97] L. Arhippainen and M. Pakanen, “Utilizing Self-Expression Template Method in User Interface Design - Three Design Cases,” in Proceedings of International Conference on Making Sense of Converging Media - AcademicMindTrek ’13, 2013, pp. 80–86.
[98] A. Ahtinen, S. Ramiah, J. Blom, and M. Isomursu, “Design of mobile wellness applications: identifying cross-cultural factors,” in OZCHI ’08 Proceedings of the 20th Australasian Conference on Computer-Human Interaction: Designing for Habitus and Habitat, 2008, pp. 164–171.
[99] C. L. Paul and A. Komlodi, “Measuring user experience through future use and
67
emotion,” in Proceedings of the extended abstracts of the 32nd annual ACM conference on Human factors in computing systems - CHI EA ’14, 2014, pp. 2503–2508.
[100] R. De Oliveira and N. Oliver, “Influence of Usability on Customer Satisfaction : A Case Study on Mobile Phone Services,” in NordiCHI 2012, 2012, no. September.
[101] T. Yamakami, “Exploratory analysis of differences between social experience design and user experience design,” in Advanced Communication Technology (ICACT), 2014 16th International Conference on, 2014, pp. 769–773.
[102] Z. Wang, X. Zhou, H. Wang, H. Ni, and R. Wu, “A Quantitative Evaluation Model of Group User Experience,” in Computational Intelligence and Industrial Application, 2008. PACIIA ’08. Pacific-Asia Workshop on, 2008, pp. 918–923.
[103] V. C. Gerogiannis, S. Papadopoulou, and E. I. Papageorgiou, “A Fuzzy Cognitive Map for Identifying User Satisfaction from Smartphones,” in Informatics (PCI), 2012 16th Panhellenic Conference on, 2012, pp. 156–161.
[104] A. Aggarwal, G. Niezen, and H. Thimbleby, “User experience evaluation through the brain’s electrical activity,” in Proceedings of the 8th Nordic Conference on Human-Computer Interaction Fun, Fast, Foundational - NordiCHI ’14, 2014, pp. 491–500.
[105] E. L. Law and P. van Schaik, “To Measure or Not to Measure UX: An Interview Study,” in I-UxSED, 2014, pp. 58–63.
[106] V. Milanova, T. Mandl, and R. Kölle, “Design for emotion: a case study,” in MobileHCI ’12 Proceedings of the 14th international conference on Human-computer interaction with mobile devices and services companion, 2012, pp. 59–64.
[107] H. Petrie and C. Harrison, “Measuring users’ emotional reactions to websites,” in Proceedings of the 27th international conference extended abstracts on Human factors in computing systems - CHI EA ’09, 2009, p. 3847.
[108] T. Jokela, “The early phases of UX : Why they are important ( more than evaluation ), and what they are ?,” in NordiCHI 2012, 2012.
[109] C.-C. Huang and E. Stolterman, “Temporal anchors in user experience research,” in Proceedings of the 2014 conference on Designing interactive systems - DIS ’14, 2014, pp. 271–274.
[110] I. Moczarny, M. Villiers, and J. Biljon, “How can usability contribute to user experience? A study in the domain of e-commerce,” in SAICSIT ’12 Proceedings of the South African Institute for Computer Scientists and Information Technologists Conference, 2012, pp. 216–225.
[111] J. Varsaluoma and F. Sahar, “Usefulness of long-term user experience evaluation to product development: practitioners’ views from three case studies,” in NordiCHI ’14: Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational, 2014, pp. 79–88.
[112] K. Kunze and D. Strohmeier, “Examining subjective evaluation methods used in multimedia Quality of Experience research,” in Quality of Multimedia Experience (QoMEX), 2012 Fourth International Workshop on, 2012, pp. 51–56.
[113] R. Alves, P. Valente, and N. J. Nunes, “The state of user experience evaluation
68
practice,” in Proceedings of the 8th Nordic Conference on Human-Computer Interaction Fun, Fast, Foundational - NordiCHI ’14, 2014, pp. 93–102.
[114] J. Varsaluoma and V. Kentta, “Drawux: web-based research tool for long-term user experience evaluation,” in NordiCHI ’12 Proceedings of the 7th Nordic Conference on Human-Computer Interaction: Making Sense Through Design, 2012, pp. 769–770.
[115] Y. Li and X. Wang, “Mobile interface studies about style description and influential factors,” in Management Science & Engineering (ICMSE), 2014 International Conference on, 2014, pp. 578–583.
[116] C. Lallemand, V. Koenig, and G. Gronier, “How Relevant is an Expert Evaluation of User Experience based on a Psychological Needs-Driven Approach?,” in NordiCHI ’14 Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational, 2014, pp. 11–20.
[117] K. Väänänen-Vainio-Mattila and M. Wäljas, “Developing an Expert Evaluation Method for User eXperience of Cross-platform Web Services,” in Proceedings of the 13th International MindTrek Conference: Everyday Life in the Ubiquitous Era, 2009, pp. 162–169.
[118] M. Q. Abbasi, J. Weng, Y. Wang, I. Rafique, X. Wang, and P. Lew, “Modeling and Evaluating User Interface Aesthetics Employing ISO 25010 Quality Standard,” in Quality of Information and Communications Technology (QUATIC), 2012 Eighth International Conference on, 2012, pp. 303–306.
[119] C. M. Macdonald and M. E. Atwood, “What Does it Mean for a System to be Useful ? An Exploratory Study of Usefulness,” in Designing interactive systems (2014), 2014, pp. 885–894.
[120] E. Karapanos, J. Zimmerman, J. Forlizzi, and J.-B. Martens, “User Experience Over Time: An Initial Framework,” in CHI ’09 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2009, pp. 729–738.
[121] N. Seyff, G. Ollmann, and M. Bortenschlager, “AppEcho: a user-driven, in situ feedback approach for mobile platforms and applications,” in Proceedings of the 1st International Conference on Mobile Software Engineering and Systems - MOBILESoft 2014, 2014, pp. 99–108.
[122] S. Wang and Shence Wang, “Emotional design based on customer experience requirement,” in Computer-Aided Industrial Design & Conceptual Design, 2009. CAID & CD 2009. IEEE 10th International Conference on, 2009, pp. 158–161.
[123] S. Luojus, “Integrating Momentary and Long-term UX: A Theoretical Approach,” in OzCHI ’12 Proceedings of the 24th Australian Computer-Human Interaction Conference, 2012, pp. 353–356.
[124] D. Buskermolen, J. Tarken, and B. Eggen, “Informing User Experience Design About Users: Insights from Practice,” in CHI EA ’12 CHI '12 Extended Abstracts on Human Factors in Computing Systems, 2012, pp. 1757–1762.
[125] H. Petrie and N. Bevan, “The evaluation of accessibility , usability and user experience,” in The Universal Access Handbook (2009), 2009, pp. 10–20.
[126] T. Walsh, S. Kujala, H. Petrie, and C. Power, “Axe UX : Exploring Long-Term User Experience with iScale and AttrakDiff,” in AcademicMindTrek ’14: Proceedings of the 18th International Academic MindTrek Conference: Media Business, Management, Content & Services, 2014, pp. 32–39.
69
[127] T. Walsh, P. Nurkka, and R. Walsh, “Cultural differences in smartphone user experience evaluation,” in Proceedings of the 9th International Conference on Mobile and Ubiquitous Multimedia - MUM ’10, 2010, pp. 1–9.
[128] Z. Hasan, R. Gope, and M. Nasir Uddin, “Do Aesthetics Matter in Long-Established Trust?,” in International Journal of Computer Applications, 2013, p. 77.
[129] V. Carofiglio, G. Ricci, and F. Abbattista, “User brain-driven evaluation of an educational 3D virtual environment,” in Information Systems and Technologies (CISTI), 2015 10th Iberian Conference on, 2015, pp. 1–7.