redesign of a website for technical information and ...1252665/fulltext01.pdf ·...
TRANSCRIPT
IN DEGREE PROJECT COMPUTER SCIENCE AND ENGINEERING,SECOND CYCLE, 30 CREDITS
, STOCKHOLM SWEDEN 2018
Redesign of a website for technical information and service tools following a human centered design process to reflect the workflow of different users
MILDA NORKUTE
KTH ROYAL INSTITUTE OF TECHNOLOGYSCHOOL OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE
i
AbstractWeb-based services designed using human centered design methods containing in-structions how to self-complete various tasks and providing interfaces to do themhelped various organizations become more efficient. However, often software en-gineering teams have a lack of understanding of when and how users should beinvolved when designing such systems. They are often unsure how to measure theimpact of human centered design methods. This thesis presents a human centereddesign solution to designing a website for technical information and service toolsfor Windows services at CERN. This involved using a carefully chosen combinationof design research methods to study user, stakeholder groups and their workflowsas well as organizational requirements. I then used this data to inform the designdecisions and outputs such as sitemaps, wireframes and a high fidelity prototype.The study focused on the role of these methods in the design process and how theycould help to accomplish the goals of the project. Also, on evaluating the impactof these methods on the design outcome. It was found that interviews, card sortingand usability testing each had a role to play in the process with the new designperforming significantly better than the old one. Further to this, usability testingwas found to be a useful evaluation and development technique.
Keywords: Human centered design, Card sorting, Website tools, Documenta-tion, Windows
iii
SammanfattningWebbaserade tjänster utvecklade efter människocentrerade utvecklingsmetoder in-nehållande instruktioner för att utföra vissa uppgifter och förse gränssnitt på egen-hand har hjälpt ett flertal organisationer att bli effektivare. Dock har ofta mjuk-varuutvecklingsteam en brist på förståelse utav när och hur användare skall varainblandade i utvecklingen av dessa system. De är ofta osäkra på hur påverkan av enmänniskocentrerad utvecklingsprocess skall mätas. Denna avhandling presenteraren människocentrerad utvecklingslösning för utveckling av en hemsida avsedd förteknisk information och tjänsteverktyg till windows services åt CERN. Detta in-volverade användning utav en noggrant utvald kombination utav forskningsresultatangående utvecklingsmetoder för att studera användare, intressenter och deras ar-betsflöde samt organisations relaterade krav. Denna data användes sedan för att tautvecklings relaterade beslut och framta resultat så som webbplatskartor, wireframesand en realistisk prototyp. Denna studie fokuserar på rollen utav dessa metoder iutvecklingsprocessen och hur de kan bidra till att åstadkomma målen för projektet.Samt, för att evaluera påverkan av dessa metoder på utvecklings processen. Detkonstaterades att intervjuer, sortering utav kort och testning utav användbarhetalla hade en roll i varför den nya produkten presterade signifikant bättre än denäldre produkten. Samt test utav användbarhet visar sig vara ett bra verktyg ochutvecklingsmetod.
Keywords: Människocentrerad utveckling, kortsortering, webbplatsverktyg, doku-mentation, Windows
v
AcknowledgmentsI would like to thank Eneko, who was the best LATEX, cooking and thesis writingsupport in general ever. Also, my supervisors for their never-ending patience inanswering my questions and guiding me in writing this project. Everyone at CERNwho were involved for their input. And finally my family and friends who pushedme to do this masters degree when I had doubts whether this is the right path forme.
September 23, 2018
Milda Norkute
vii
Contents
Abstract i
Sammanfattning iii
Acknowledgments v
Contents vii
List of Figures xi
List of Tables xiii
Symbols xv
1 Introduction 11.1 Information sharing on the web . . . . . . . . . . . . . . . . . . . . . 11.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3 CERN case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.4 Problem and research question . . . . . . . . . . . . . . . . . . . . . . 31.5 Purpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.6 Goal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.6.1 Benefits, Ethics and Sustainability . . . . . . . . . . . . . . . 41.7 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.8 Delimitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.9 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2 Theoretic background 72.1 The human centred design processes in the web . . . . . . . . . . . . 7
2.1.1 Understanding and specifying the context of use . . . . . . . . 72.1.2 Specifying the user requirements . . . . . . . . . . . . . . . . . 82.1.3 Producing design solutions . . . . . . . . . . . . . . . . . . . . 82.1.4 Evaluating designs against requirements . . . . . . . . . . . . 9
2.2 Measuring design outcomes . . . . . . . . . . . . . . . . . . . . . . . 102.2.1 Defining usability . . . . . . . . . . . . . . . . . . . . . . . . . 102.2.2 Researching usability . . . . . . . . . . . . . . . . . . . . . . . 112.2.3 Measuring usability . . . . . . . . . . . . . . . . . . . . . . . . 122.2.4 The relationship between design and evaluation methods . . . 14
2.3 Web information architecture design . . . . . . . . . . . . . . . . . . 142.3.1 Card sorting . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3 The design process of the Windows services website 193.1 Understanding context of use . . . . . . . . . . . . . . . . . . . . . . 193.2 Acquiring user requirements . . . . . . . . . . . . . . . . . . . . . . . 193.3 Design realization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
viii
3.4 Design evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4 Understanding context of use, methods 234.1 User group and stakeholder analysis methods . . . . . . . . . . . . . . 23
4.1.1 Semi-structured interviews . . . . . . . . . . . . . . . . . . . . 234.2 Old site navigation analysis . . . . . . . . . . . . . . . . . . . . . . . 24
5 Understanding context of use, results 255.1 User groups and stakeholders analysis results . . . . . . . . . . . . . . 255.2 Old site navigation analysis results . . . . . . . . . . . . . . . . . . . 28
6 Acquiring user requirements, methods 316.1 Card sorting task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
6.1.1 Card sorting task set up . . . . . . . . . . . . . . . . . . . . . 316.1.2 Test participants . . . . . . . . . . . . . . . . . . . . . . . . . 336.1.3 Task procedure . . . . . . . . . . . . . . . . . . . . . . . . . . 346.1.4 Data collection and analysis . . . . . . . . . . . . . . . . . . . 34
6.2 Usability testing of the old website . . . . . . . . . . . . . . . . . . . 356.2.1 Test materials . . . . . . . . . . . . . . . . . . . . . . . . . . . 366.2.2 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366.2.3 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366.2.4 Data collection and analysis . . . . . . . . . . . . . . . . . . . 40
6.3 Investigating organizational requirements . . . . . . . . . . . . . . . . 41
7 Acquiring user requirements, results 437.1 Card sorting results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
7.1.1 Identifying the most commonly suggested categories . . . . . . 437.1.2 Card sorting results matrix . . . . . . . . . . . . . . . . . . . 447.1.3 Key qualitative findings . . . . . . . . . . . . . . . . . . . . . 47
7.2 Usability testing of the old website . . . . . . . . . . . . . . . . . . . 487.3 Website purpose and organizational requirements . . . . . . . . . . . 497.4 Use cases of the website . . . . . . . . . . . . . . . . . . . . . . . . . 50
8 Design realization, methods 538.1 Sitemaps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 538.2 Wireframes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 538.3 High fidelity prototyping . . . . . . . . . . . . . . . . . . . . . . . . . 54
9 Design realization, results 559.1 New site navigation map . . . . . . . . . . . . . . . . . . . . . . . . . 559.2 Results from wireframes development . . . . . . . . . . . . . . . . . . 559.3 Results from high fidelity prototype development . . . . . . . . . . . 56
ix
10 Evaluation, methods 5910.1 Usability testing for evaluation . . . . . . . . . . . . . . . . . . . . . . 59
10.1.1 Test participants . . . . . . . . . . . . . . . . . . . . . . . . . 6010.2 Feedback collection sessions . . . . . . . . . . . . . . . . . . . . . . . 60
11 Evaluation, results 6111.1 Usability testing for evaluation . . . . . . . . . . . . . . . . . . . . . . 61
11.1.1 Usability testing scores . . . . . . . . . . . . . . . . . . . . . . 6111.1.2 Old and new interface scores comparison . . . . . . . . . . . . 63
11.2 Feedback collection results . . . . . . . . . . . . . . . . . . . . . . . . 63
12 Discussion and Conclusion 6512.1 Designing tools for different workflows and levels of technical aware-
ness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6512.2 Evaluation methods for design process . . . . . . . . . . . . . . . . . 6612.3 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6712.4 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
Bibliography 71
Appendices 77
A Semi-structured interview questions for staff and organisational re-quirements specification 77
B Semi-structured interview questions for identified user groups rep-resentatives 79
C Cards used in card sorting test 81
D Information and consent form for the card sorting task 83
E Follow up interview questions for the card sorting task 85
F Information and consent form for the usability tests 87
G Questions asked and during usability tests 89
H System Usability Scale 91
I High fidelity prototype of new website screens 93
xi
List of Figures1 Interdependence of human-centred design activities . . . . . . . . . . 72 The process of the work to redesign the Windows services website . . 213 Sitemap starting from the homepage of the old Windows services
website . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294 Sitemap starting from the DFS section on the homepage of the old
Windows services website . . . . . . . . . . . . . . . . . . . . . . . . 295 Sitemap starting from the Windows computers section on the home-
page of the old Windows services website . . . . . . . . . . . . . . . . 306 Card sorting data initial visualization . . . . . . . . . . . . . . . . . . 357 Old Windows services website home page . . . . . . . . . . . . . . . . 388 Old Windows services website documentation . . . . . . . . . . . . . 389 Affinity diagram to identify the most commonly suggested categories
among all of the responses from the open card sort and relationshipof those to the categories used in the hybrid card sorting task . . . . 43
10 Common card sorting visualization techniques: A) dendogram fromhierarchical cluster analysis B) similarity matrix of card relationships,and C) word frequency cloud of participant-generated topics. . . . . . 44
11 Developed sitemap for the new website . . . . . . . . . . . . . . . . . 5512 Developed wireframe representing the home page . . . . . . . . . . . 5713 Home screen of the high fidelity UI which was tested in the second
usability test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5814 UI of the home screen . . . . . . . . . . . . . . . . . . . . . . . . . . 9315 UI of the tools screen . . . . . . . . . . . . . . . . . . . . . . . . . . . 9416 UI of the help screen . . . . . . . . . . . . . . . . . . . . . . . . . . . 9517 UI of the Bitlocker retrieval page . . . . . . . . . . . . . . . . . . . . 9618 UI of set local administrator page . . . . . . . . . . . . . . . . . . . . 9719 UI of the recreate account page . . . . . . . . . . . . . . . . . . . . . 9820 UI of the local password retrieval page . . . . . . . . . . . . . . . . . 9921 UI of the key computer facts page . . . . . . . . . . . . . . . . . . . . 10022 UI of the key computer facts page: extended view . . . . . . . . . . . 101
xiii
List of Tables1 Demographic data of participants that took part in both usability tests 372 Tasks given each participant during both rounds of usability testing . 393 Metrics of usability recorded in both usability testing rounds . . . . . 404 Each card assigned a number for further analysis . . . . . . . . . . . 455 Card sorting matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . 466 Overall SUS Scores after conversion . . . . . . . . . . . . . . . . . . . 617 Various usability measures for users 4 and 5 from usability tests with
old and new UIs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 628 Various usability measures for users 1, 2 and 3 from usability tests
with old and new UIs. . . . . . . . . . . . . . . . . . . . . . . . . . . 62
xv
SymbolsN Number of total pages visited while performing the taskS The total number of pages visited counting revisits to the same pageR The minimum number of pages that must be visited to accomplish the task
AcronymsCERN European Organisation for Nuclear Research. 1–6, 10, 17, 19, 20, 23–28,
33, 34, 36, 38, 41, 49, 50, 54, 59, 65, 67
CMF Computer Management Framework. 3, 27
DFS Distributed File System. xi, 29
IT-CDA Information Technology Collaboration Devices Applications. 2, 24, 25,28, 31, 41, 60, 63, 65
LDAP Lightweight Directory Access Protocols. 3
NICE Network Integrated Computing Environment. 2, 3, 28
SUS System Usability Scale. 40, 61
UI User Interface. xi, 20, 36, 45, 48, 58–61, 63, 66–68
1
1 Introduction
1.1 Information sharing on the web
The invention of the web had a significant impact on many aspects of our society:from business, education, government to social, personal lives and many other areas.The web provides a unique platform for software engineering with benefits thatinclude writing and pushing instant updates for all users as well as universal accessand execution from any device connected to the Internet. Many existing softwaresystems have been and continue to be migrated to the web thanks to the opennature of the web. A few of examples of services powered by the web include onlinebanking allowing people to manage their finances and paying their bills withoutphysically stepping into a bank or interacting with the government services onlineamong millions of other use cases [1].
The desire to provide all users at CERN, The European Organization for NuclearResearch, with the equal ability to access all documents was Tim Berners-Lee’s, theword wide web inventor’s primary goal. However, the web has evolved quite farahead of this original use case since its invention. And even this equality of access,also known as accessibility, is difficult to quantify, define, or agree upon. There aremany different definitions in the literature, all with a different perspective. Thismakes it difficult for the web creators communities to interact, reach agreement andshare understanding on how the web should be built [2].
One of the approaches to the development and design of systems including webservices is the human centered design process. Specifically, the ISO 9241-210:2010 [3]provides requirements and recommendations for human-centered design principlesand activities throughout the life cycle of computer-based interactive systems. It isprimarily intended to be used by those managing design processes. It is concernedwith ways in which both hardware and software components of interactive systemscan improve the human–system interaction and ensure they are usable. This isachieved by focusing on the needs of the user by employing various human centereddesign methods [3].
1.2 Background
The human centered design process in an iterative cycle which consists of four mainstages: understanding context of use, acquiring user requirements, design realizationand evaluation. Following the human centered practices when creating a new webservice can be very beneficial. Software and web systems with good usability havebeen documented to be economically beneficial in terms of increased sales and userproductivity, decreased training costs and decreased needs for user support. Inmany cases, the initial investment and costs needed to follow the human centereddesign process are outweighed in the long run [4]. However, it is important to choosesuitable development and evaluation methods for each of the human centered designprocess stages and define the term usability itself.
According to ISO 9241-210 [3], the term usability is “the extent to which a
2
system, product or service can be used by specified users to achieve specified goalswith effectiveness, efficiency and satisfaction in a specified context of use” (p.17).There is a wide range of different methods and metrics available to study usabilityand they may or may not involve users. Furthermore, according to Nielsen [5], theusability can be enhanced significantly by creating an information architecture thatreflects how users view the content. According to Nielsen’s intranet studies [5], someof the biggest productivity gains have occurred when companies restructured theirintranet to reflect employees’ workflow. Similarly, it was found that in ecommerce,sales increase when products appear in the categories where users expect to findthem [6]. To research the users’ view of an information space the primary methodis card sorting [7]. This method can help to identify the high-level categories basedon which the navigation of the websites can then be designed [8].
1.3 CERN case
CERN, The European Organization for Nuclear Research, has a large variety ofwebsites for various purposes. This includes representational ones dedicated to theoutside public, web services used internally to monitor its scientific experiments andweb services for helping staff to complete various administrative tasks and run theinstitution.
One such website is the Windows services website which belongs to CERN’s IT-CDA: Information Technology, Collaboration Devices and Applications department.It is also known as the Network Integrated Computing Environment or the NICEservices website. The official description of the NICE services is “Providing a con-sistent environment through which Windows users at CERN can obtain resources,applications and managed operating system services” [9]. This concept has beenlaunched at CERN in 1991. The NICE environment is primarily oriented towardsWindows operating system users and everyone who has a CERN account can use it.It can create a single desktop environment for all users from which they can accessall the facilities of the laboratory.
After four generations based respectively on Windows 3.0, 3.1, 95/NT, 2000,XP, Vista, Windows 7 and 8 the environment is currently based on Windows 10.All applications are centrally supported, installed, and managed. However, someoffices and laboratories are also using older Windows versions as in some cases thesoftware for specific tasks is available for only those versions of Windows [9]. Inother cases they are simply unwilling to spend time to update their computers withnewer Windows versions.
Besides the different versions of Windows that computers at CERN may have,typically, they also have one of the three Windows configurations:
• Centrally managed: Windows computers that receive updates once a month.The NICE team is responsible for those.
• Locally managed: A group of Windows computers that does not receive up-dates from the NICE team. These typically are computers used for experi-ments and for this reason they need to be in control of when the updates are
3
pushed as well as their content. There are different groups of locally managedcomputers with Windows. Usually each group is managed by one person whois responsible for the updates.
• Hardened: Windows computers that are configured to have a stronger re-silience to external attacks. One of the main features of this is that the mainuser of the computer does not have the elevated privileges to run applications[10]. For example, they are not able to open files received by email which maycontain malware. The NICE team is responsible for looking after these typeof Windows computers.
All the updates, regardless of the computer configuration type, are pushed usingthe Computer Management Framework also known as CMF. The end users of thecomputers receive updates through this system and may download the applicationsthey need using it. The purpose of having the system at CERN set up in this way isto avoid the user efforts to install the operating system and all necessary applicationsthemselves. Furthermore, this ensures that the applications are installed in a secureand safe environment as the user does not need to look for other sources from whichhe could install the applications.
This system has been built with the help of Microsoft’s Active Directory. It isa centralized and standardized system that automates network management of userdata, security, and distributed resources [11]. In the case of CERN, Active Directoryis a LDAP database containing all CERN user, group and computer objects. Itprovides LDAP access to information as well as Kerberos authentication services forCERN central services, Windows and Linux stations [12].
In the context of the existing Windows system at CERN, the Windows serviceswebsite is supposed to provide information about this ecosystem, the applicationsavailable and their installation process. In addition to this, the website has a setof tools allowing users and computer administrators do some actions. These in-clude requesting specific information about their Windows computer, recreatingtheir computer account and several others. In addition to this, it is supposed tooffer a catalogue of related services that also may be useful to the user. This webservice was the subject to redesign and redevelopment and therefore was used as acase study for this thesis.
1.4 Problem and research question
Web-based services providing information and interfaces to complete various taskswith good navigation, built using human centered design methods, helped variousorganizations to become more efficient [13]. However, the old website of Windowsservices was not built following these methods. This made it hard to use and findthe needed information. In addition to this, since the website was first created,the ecosystem of Windows at CERN has become even more complicated thanksto introduction of the hardened computers configuration in November 2016. TheWindows services website was never updated to reflect this.
4
CERN is not alone in this: many organisations still tend to underestimate theimportance of employing human-centered design methods when designing new orredesigning old websites [13]. Often software engineering teams have a lack of un-derstanding of when and how users should be involved in designing such systemsand how to measure the impact of human centered design methods [13].
Through the process of redesigning the Windows services website this thesis triedto answer the following research questions:
1. How to create a website containing technical information, tools forWindows computer management and troubleshooting reflecting theworkflow and concept familiarity levels of different users ?
2. How to measure and evaluate the impact of the redesign process ?
1.5 Purpose
The purpose of this project was to redesign a web service for Windows servicesfor CERN which contains information about the Windows infrastructure, providetools to manage Windows computers and instructions about how to use them. Inparticular, the focus was on creating good navigation so that the information andservices on the website become easy to find for the users.
This in turn, helped to carry out a study on how to accomplish the redesign taskand validate the design following the human centered design process and methods.
1.6 Goal
The goal of the project was to produce a new design of website for CERN’s Windowsservices, built with a human centered approach, well-organized information andservices.
1.6.1 Benefits, Ethics and Sustainability
First of all, web-based services containing a lot of information and interfaces todo certain tasks with good navigation helped many organizations to become moreproductive and cost-efficient. This is because users become less reliant on requiringprofessional assistance [13]. Therefore, in the case of CERN, a web-service createdwith human-centered approach should lower the amount of the tickets related toWindows computers that the service help desk receives. It could also help to positionCERN’s IT department as more approachable and user oriented.
However, the methods used in this project involved different interactions withthe users of the system including studying how they interacted with the old system.Thus, it was also important to consider the impact these interactions might have hadon the users. To ensure minimal stress to those that were involved in the researchprocess throughout the thesis, all tasks and tests were presented and conducted inan ethical manner, ensuring a stress free environment to the participants.
5
Further, to this, designing this type of new web service is highly likely to changethe current ways of working of those that are using the current website on a dailybasis [14]. Thus, to ensure a positive outcome, it was also important to identifyfrequent users of the website and involve them thoroughly in the design process.The reason for this was ensuring a smooth transition between the old and the newwebsite.
1.7 Methodology
This work followed a human centered design process when redesigning the Windowsservices website. Therefore, it consisted of four major stages in total: understand-ing and specifying the context of use of the website, gathering user requirements,producing design solutions and evaluating designs against the requirements. I useddifferent methodologies for each stage.
The first, specifying context of use stage, has mostly involved work to understandand research the user groups and stakeholders of the Windows services website andthe Windows ecosystem at CERN as a whole. I used semi-structured interviewsto gather the majority of the data about the user groups and stakeholders. I alsocreated a sitemap of the old website and analyzed it.
Next, I researched the user requirements and specified the organizational re-quirements. The main methods used for this were the card sorting [8] and usabilitytesting of the old website with follow up interviews. I also did a presentation to theinternal stakeholders of my findings and held a discussion with them to learn moreabout the organizational requirements.
In the design realization stage I used sitemaps, wireframes and finally created ahigh fidelity prototype. It was then tested in the second usability test conducted inthe evaluation phase of the work. It was done so that I could compare the perfor-mance of the output of the design process with the performance of the old Windowsservices website. I also took regular feedback from the internal stakeholders of theWindows services website to evaluate the work of this project.
1.8 Delimitations
One of the study delimitations is that a mock up was the subject to the secondusability test rather than a fully functional website. The are several reasons for this.The choice of software tools for the development of the website was made by theCERN IT department with a broader set of requirements taken into consideration.Making this decision took longer than expected. In addition to this, the chosentechnology for building the user interface was React.js. It is not yet adopted atCERN widely, so this required inputs from other departments such as the securityand the web-frameworks teams. In particular, I needed them to provide componentsto build the search and single sign on features. Therefore, the discussion on thesoftware tools and implementation is out of the scope of this thesis because of thetime constraints.
6
Furthermore, some of the graphical design decisions made during the projectwere subject to internal design guidelines that CERN has for its websites. Thus,the impact of these guidelines on the final result of the project is also beyond thescope of this thesis.
1.9 Outline
Chapter 1 is the introduction to the key parts of the thesis, what its goals are, howthe work was carried out, the scope of the work and its delimitations.
Chapter 2 presents the theoretical framework of the thesis and to make for easyreading it is divided into three sub chapters. First, the human centered designprocess theory is explained. Then different ways of evaluating and measuring designoutcomes are presented. Finally, different architecture of the web design techniquesincluding the introduction of the card sorting method are discussed.
Chapter 3 explains the overall process of the work carried out for this projectand gives an overview of the type of work that was conducted for each major projectphase.
Chapter 4 deals with the methodology I used for understanding the context ofuse.
Chapter 5 explains the analysis and findings after employing methods for re-searching the context of use.
Chapter 6 describes the methods used to gather user and organizational require-ments including card sorting and the usability test of the old Windows serviceswebsite.
Chapter 7 presents the results of the user requirements research stage and sum-marizes key findings including finalizing the use cases of the website.
Chapter 8 explains and motivates the design realization methods chosen for thisproject.
Chapter 9 presents the design artifacts created for this project using the chosendesign realization methods.
In Chapter 10 I present and motivate the choice of the methods for the evaluationphase of the project.
In Chapter 11 I discuss the evaluation of the design process results.Finally, in Chapter 12 the thesis is concluded with a summary to answer the
initial research question. It also includes suggestions and considerations for futurework.
7
2 Theoretic backgroundFirst, this thesis explains and summarizes the human centered design process prin-ciples as these were key in guiding the work throughout the project. Next, specifictheories and design methods relevant to the type of work the project required toredesign the Windows services website are discussed.
2.1 The human centred design processes in the web
To understand where the different techniques and terms might fit in the designprocess, it is important to understand the full picture and the theory behind howit works. According to the ISO 9241-210 [3] which offers detailed guidelines forhuman-centered design for interactive systems, figure 1 can be used to explain thehuman-centered design process:
Figure 1: Interdependence of human-centred design activities
As we can see, the human centered, also known as the user centered, designprocess starts with planning it. After that the iterative cycle, which may be repeatedas many times as needed, consists of four key parts. These are understanding contextof use, specifying user requirements, producing design solutions for the requirementsand evaluating designs against the requirements. Then if the requirements are met,the cycle closes, else it repeats itself again until it is decided that its outcome wassatisfactory. Next, I discuss the 4 main parts that are part of the iterative cycle ingreater depth including the kind of work and methods they involve to understandthem in greater detail.
2.1.1 Understanding and specifying the context of use
The first part of the iterative cycle deals with understanding and specifying thecontext of use for the system that is being designed. This is because a system thatis being designed will be used by a user population which has certain goals and
8
wishes to perform certain tasks. In addition to this, the system might also be usedwithin a certain range of technical, physical and social or organizational conditionsthat may affect how it is being used [15].
Key methods to understand and specify context of use include: stakeholderanalysis for each user and stakeholder group, their main roles, responsibilities andgoals of their tasks in relation to the system. For example, a public informationsystem in a local library should be designed to meet the contrasting needs of thegeneral public, the information service staff and information providers. All of themare the stakeholders in this system [15].
One method to study the context of use could be user cost-benefit analysis. Itcompares the costs and benefits for different user groups when considering a newsystem. It provides an overview of how acceptable each user group will find thenew system [16]. Other methods for studying the context include user requirementinterviews, focus groups and development of potential scenarios of use. This is bestdone together with the user personas and creating maps between task-functionality[15]. Choices for studying the context of use for this project are presented in chapter4.
2.1.2 Specifying the user requirements
After understanding the context of use comes the user requirements specification.One of the ways this can be done is by creating summary descriptions of the tasksthat the system will support and the functions that will be provided to supportthem. Another method could be creating example task scenarios and possible in-teraction steps for the future system, describing features of the system to meet thecontext-of-use characteristics. In addition to the user requirements, organizationalrequirements may be specified as well, taking into account the needs of an organi-zation the design might be produced for [17].
Finally, it is also possible to define user requirements as usability goals. Typically,these are effectiveness: the degree of success with which users achieve their taskgoals; efficiency: the time it takes to complete tasks, and satisfaction: user comfortand acceptability. These are most easily derived from the evaluation of an existingsystem and are independent of any specific implementations [17]. However, therecan also be other more detailed usability issues specific to design objectives. Chapter6 explains how this thesis dealt with specifying the user requirements phase of thedesign process.
2.1.3 Producing design solutions
After specifying the user requirements, the design solutions are produced. At thesimplest, they may consist of a series of user interface screens and a partial databaseallowing potential users to interact with, visualize and comment on the future design.Such simulations or prototypes can be produced both relatively quickly and easilyin the early stages of the system development cycle for evaluation. They then maybe evaluated by usability experts and members of the design team or even testedquickly by users. Changes to the design can be made rapidly in response to user
9
feedback, so that major problems with the design can be identified before systemdevelopment begins [18].
Some of the methods for prototyping the design solutions include an affinity di-agram: using sticky notes to organize screens or functions from a user perspective;paper prototyping which is a paper based simulation of an interface to test inter-action with a user; wizard-of-oz prototype which involves a user interacting with acomputer system that is actually operated by a hidden developer [19]. Another pop-ular tool is schematics or wireframes that act as a representation of the informationor content that should appear on a particular page. They are typically made usingsimple colors, sometimes monochrome or gray-scale, to indicate key features of apage [20]. The choice of the prototyping method depends on what is the purpose ofthe prototyping. Yet, ultimately the purpose of prototypes is to be tested by eitherusers or expert evaluators [19]. Specifically, choices for producing design solutionsfor this project are presented in chapter 8.
2.1.4 Evaluating designs against requirements
Finally, the solutions are evaluated against the user or organizational requirementsthat were produced at the beginning of the design cycle. This can be done eitherwith or without the user involvement. However, the key thing to note with regardsto human-centred design process as a whole is that if the designs do not meet therequirements, then a new set of designs needs to be produced and the cycle continues[19]. Chapter 10 discusses the work that was done to evaluate the design processoutcome of this thesis.
Often companies generally consider usability evaluation as a process that requireslarge investments of time and resources [21]. Bak et al. [22] combined questionnairesurvey and interview study of obstacles for deploying usability evaluation in softwaredevelopment organizations. The study revealed that 29 of 39 software developmentorganizations conducted some form of usability evaluation. However, the study alsoshowed that the understanding of usability evaluation is a major obstacle. Forinstance, the focus on user involvement is significant among the organizations’ un-derstanding of usability evaluation, with many respondents understanding the termas some form of user involvement. Yet, there was a general lack of awareness of thedifferent ways this could be done. Furthermore, the two most significant obstacleswere resource demands and the mindset of developers with many reporting theyfound it hard to think from the user’s perspective. These obstacles were not onlyan obstacle for more organizations to deploy usability evaluation, but also a con-cern for the software development organizations that had already deployed usabilityevaluations in their development process [22].
Next, different available evaluation methods and metrics are discussed to betterunderstand why using them them may be challenging for some of the developmentteams.
10
2.2 Measuring design outcomes
Design is an iterative process, therefore, to determine whether the produced de-sign is an improvement on the old one, ideally data should be collected at everymilestone. The usability measures and measurement techniques should typically bechosen based on what the company would like to achieve when designing or redesign-ing an existing system. In addition, it is also important to define the term usabilityitself.
2.2.1 Defining usability
According to ISO 9241-210 [3], the term usability is “the extent to which a system,product or service can be used by specified users to achieve specified goals witheffectiveness, efficiency and satisfaction in a specified context of use” (p.17). Hertzum[23] proposes that there are six types of usability: Universal, Situational, Perceived,Hedonistic, Organizational and Cultural.
Universal usability entails the challenge of designing systems for everybody touse. Universal usability is particularly relevant when designing systems such asATMs; or web-based commerce, government, health or learning e-platforms andsystems. However, designing these systems is extremely challenging, given the va-riety of users groups that would be using them, making it difficult to specify thecontext of use for the system. The second type, situational usability, is understoodin a complete specified situation with its users, tasks, and wider context of use.Similarly, perceived usability is always evolving around the user’s needs - as userslearn to use the system, it becomes more complex. Overall, perceived usability be-lieves in putting the user’s subjective experiences measures above the measures ofperformance such as task completion times [23].
The fourth, hedonistic usability type is also focused on users perceptions of thesystem, similarly to perceived usability. However, the key difference between theseis that the hedonistic usability aims for a very specific goal - to trigger emotionsof pleasure in the user which could be either physical, social, psychological or ide-ological, or focused on tastes and values. Thus, hedonistic usability is particularlyimportant and relevant for the development of gaming and similar systems that aimto award the user with the feeling of satisfaction and similar while using them. Bycontrast, organizational usability is concerned with a completely different type ofcontext, where pleasure is not important at all. Organizational usability focuses onmaking a match between a computer system and the structure and practices of anorganization, so that the system can be effectively used for work purposes by theorganization’s members [23]. This last type of usability is relevant for the work ofthis project since the website that was redesigned was a tool for storing informationand managing Windows computers at CERN.
It is often helpful to further break down the usability definition in order to comeout with metrics that can be measured during the design and development process.Nielsen [24] suggests that usability is a quality attribute that assesses how easy touse are user interfaces and proposes to define it by 5 key components:
11
• Learnability: How easily can the user accomplish the basic tasks the first timethey encounter the system?
• Efficiency: After users have learned the design, how fast can they performtasks using the system?
• Memorability: When users return to the design after a period of not using it,is it easy to become an efficient user again?
• Errors: How many errors do users make, how severe they are, and is it easyto recover from the errors?
• Satisfaction: How pleasant it is for the user to use the system?
In addition to this, there are various other important quality attributes such asutility, which refers to the design’s functionality: does the system do and accomplishwhat users need? [24].
Nielsen [24] argues that usability and utility are equally important and togetherdetermine whether something is useful. If the system is designed well but does notaccomplish what the user wants, it cannot be consider to have good usability. Inaddition to this, it is also no good if the system can in theory help the user do thetask he wants or needs to do, however, it is extremely difficult to make it happenbecause the user interface is too difficult to use. Therefore, the same user researchmethods that are used to determine how useful the system can also help to improvethe usability of the system which is being studied. This relationship between designand evaluation methods was relevant for this project as it aimed to demonstrate theresults of the design methods applied. Chapter 3 outlines in more detail how thisproject dealt with this.
2.2.2 Researching usability
Design reviews and usability research can be done either with or without users. Themost widely used evaluations without users include heuristic evaluation, cognitivewalkthrough and task analysis. In a heuristic evaluation, usability experts reviewthe site’s interface and compare it against accepted usability principles. The anal-ysis results in a list of potential usability issues [25]. In the cognitive walkthroughone or more evaluators work through a series of tasks and ask a set of questionsfrom the perspective of the user. The focus of the cognitive walkthrough is on un-derstanding the system’s learnability for new or infrequent users [26]. Finally, thetask analysis involves breaking down the tasks into as small components as possibleand completing them from the perspective of the user and measuring the time thattakes [27].
These type of evaluations are usually faster than conducting evaluation or re-search with users, hence why some companies favour some of these methods. Suchdesign reviews work best when the examiner not only has a deep knowledge of us-ability best practices and a large amount of past experience conducting usability
12
research but also is someone who was not involved in creating the design to be re-viewed. This is because a fresh perspective is more likely to be unbiased: an outsideris not emotionally invested in the design, is oblivious to any internal team politics,and can easily spot issues that may stay hidden to someone who has been lookingat the same design for too long [28].
However, heuristic evaluation cannot replace all evaluation methods involvingusers. One such method is contextual inquiry which includes observing the partici-pants interacting with a system in a natural environment. Another one is usabilitytesting where users are asked to complete tasks, typically while being observed by aresearcher, to see where they encounter problems and experience confusion. Accord-ing to Tan, Liu and Bishu [29] although the heuristics relate to criteria that affectyour site’s usability, the issues identified in a heuristic evaluation are different thanthose found in a usability test.
Silva et al. [30] compared the effectiveness of one of the usability testing meth-ods known as the rehearsal technique with users and heuristic evaluation. Bothmethods were applied with the same goal, to identify usability issues with emphasison navigation design. They were applied on the same product, a functional proto-type of a washing and drying machine’s control panel. The heuristic evaluation wasapplied by three analysts, and the usability testing was done with ten participants.Each of the methods resulted in usability issues, which were evaluated by criticalityand related to design fields [30]. It was found that the heuristic evaluation allowedthe verification of broader usability problems, detecting about 85% of the issuesidentified on the rehearsal technique. However, the rehearsal technique enabled theidentification of user feedback, as well as the study of actual behaviors, confirmingsome usability issues. Also, more issues were detected than with heuristic evalua-tion. Therefore, it was concluded that the two ways of studying usability providedifferent results and different types of data and the evaluation without users cannotbe considered as a replacement for the usability testing [30].
This project decided to employ the evaluation methods involving users and thechosen methods alongside the motivation are discussed in chapter 10.
2.2.3 Measuring usability
In addition to choosing a suitable method for collecting data it is also importantto be able to measure how the design performs. Thus, this section discusses keyusability metrics used in the industry that are used to evaluate different designs.
In the case of evaluations not involving users, for instance, heuristic evaluation,typically the design is evaluated against industry wide accepted design principlesand guidelines [27]. For example, one of the classic tools used are Nielsen’s 10usability heuristics that ask the expert evaluator to score the design against these10 principles, rating the severity of the violation of each error on a scale rangingfrom 0 to 4 where:
• 0 = I do not agree that this is a usability problem at all.
13
• 1 = Cosmetic problem only: does not need be fixed unless extra time is avail-able on project.
• 2 = Minor usability problem: fixing this should be given a low priority.
• 3 = Major usability problem: important to fix, so should be given high priority.
• 4 = Usability catastrophe: must be fixed before product can be released [31].
Similar principles are also applied when using different guidelines and scales.When it comes to usability metrics collected involving the users, typically, usabil-
ity is measured relative to users’ performance on a given set of test tasks. Accordingto Nielsen [32] the most basic measures are based on the definition of usability as aquality metric which includes measures of success rate (whether users can performthe task at all), the time a task requires (how much time is required to completea task), the error rate (the mistakes made during a task). Errors can be useful inpointing out particularly confusing or misleading parts of an interface, and users’subjective satisfaction.
Tullis and Albert [33] also propose to measure task success which can show howeffectively users are able to complete a given set of tasks, time on task, errors andsatisfaction. In addition to those, they also suggest measuring effectiveness whichcan be assessed by examining the amount of effort a user expends to complete a task,such as the number of clicks in a website or the number of button presses. Anothermetric they suggest is learnability, which shows how performance improves or failsto improve over time and the extent to which something can be learned efficiently[33].
According to Tullis and Albert [33] especially nowadays, learnability is muchmore important as the web has facilitated a move toward many more “self-service”applications than we previously had. At the same time, it has encouraged an ex-pectation that people should be able to use almost anything on the web withoutextensive training or practice. For instance, today when using one of the many dif-ferent websites that let users book airline flights we have to consider the learnabilityaspect because training user to use the site would be too costly. Tullis and Albert[33] argue that learnability is a key differentiator in today’s self-service economy.
Finally, Smith [34] has also proposed a lostness measure which can show howeasy or difficult it can be for the user to find the needed information on the website.This deals with “the lost in hyperspace” problem. In general, this problem refers toany of the following conditions:
• Users cannot make out where they are in the application.
• Users cannot return to previously visited information.
• Users cannot go to information that is believed to exist.
• Users cannot remember what they have covered.
• Users cannot remember the key points covered.
14
• Users are actually lost and it is not a superficial disorientation.
Many of the discussed measures, in particular those that are collected using meth-ods involving users are relevant for this project and the chosen ones are presentedin the chapter 6.
2.2.4 The relationship between design and evaluation methods
As previously mentioned, it can be a good idea to keep track of the metrics collectedthroughout the different stages of the design process. This way one is able to measurewhether there was an improvement. Therefore, to minimize the required design effortand required amount of work, it could be a good idea to choose methods that servea dual purpose. In particular usability testing has been used both as a developmentand evaluation method in previous work by other authors.
Becker and Yannotta [35] conducted four rounds of usability testing throughoutthe process of building a new academic library web site. The old site was tested sothat the scores it received could be used as a benchmark to establish how good thenew design was. In addition to this, several drafts and iterations of the new sitehave been tested as well, including a series of post-test questions that would allowparticipants to share their demographic information and give subjective feedback onthe drafts of the site. Participants were asked to perform tasks using a talk-aloudprotocol for all of the testing rounds. According to the authors, testing throughoutthe design process has proved to be an effective way to build a website that reflectsuser needs and preferences. It also helped to track the progress of the designsprocess thanks to having a benchmark for the old website. A similar approach hasbeen taken by Pendell and Bowman [36] when redesigning Portland State UniversityLibrary’s Mobile Website where several rounds of testing different versions of thewebsite including the old one prior to redesign have been conducted.
This type of setup was relevant for this project as well. The details of howusability testing was set up to act both as development and evaluation method arediscussed in more detail in chapter 3.
2.3 Web information architecture design
This project aimed to redesign a website where a lot of information is stored. There-fore, the information organization and how this affected the navigation of the websitewere important concerns of this project. Thus, next we take a look at the differ-ent design techniques for requirements gathering and producing design solutionsthat deal specifically with helping to organize content and website functions in auser-centered manner.
One of the classic mistakes is to structure the information space based on howthe designer or the developer views the content. This often can result in differentsubsites for each of the company’s departments or information providers. Since theusers typically do not have in-depth knowledge about the companies that createand provide the services to them, this type of information architecture can oftenprove to be problematic [37]. According to Nielsen [5], usability can be enhanced
15
significantly by creating an information architecture that reflects how users viewthe content. Nielsen’s intranet studies [5], have found that the biggest productivitygains have occurred when companies restructured their internal websites and toolsto reflect employees’ workflow. Similarly, it was found that in ecommerce, salesincrease when products appear in the categories where users expect to find them.Therefore, it can be argued that designing a well-structured information architectureof a web service could significantly improve the usability of the website as a whole.
Methods to design information architecture for the web include user flows anduser stories, affinity diagramming, performing content inventory, wireframing andseveral others [37]. User stories are concerned with fostering an understanding ofthe goals or needs users have and the tasks they need to complete. Ultimately, userflows, influenced by user stories, create a smooth path for users to achieve theirgoals. Designing the information architecture using this method typically involvesmeeting with stakeholders and users to identify the needs of both and turning thoseinto user stories [38]. Next, the user flows are prioritized in a way which allow theprotagonists of the user stories to achieve their goals. While this method does involveusers to some extent there is a risk that some user groups may be under representedin the final user flows by focusing on protagonists. Similarly, when using contentinventory, it is not clear to what extent the user views are represented. This isbecause this method usually involves listing all the content on the site which willtypically include text, images, documents, and applications [39].
Affinity diagrams are used to organize the data thematically based on the insightsit provides about users and their response to the design challenge. The individualpoints have to be grouped into meaningful categories that can be used to createaccurate personas, logical layouts, and intuitive interactions. The data to do this istypically gathered through interviews, contextual inquiry or user observation [40].Finally, wireframes act as representation of the information or content that shouldappear on a particular page and are typically made using simple colors, sometimesmonochrome or gray-scale, to indicate key features of a page [20]. These two toolsare useful for data analysis and turning research insights into designs. However,they do not fully answer the question on how it could be best to gather data whichcould be useful in informing the information architecture design.
2.3.1 Card sorting
To research the users’ view of an information space and where they think each itemshould go, the primary method is card sorting [41]. Typically, in this method a nametogether with a short description on what the item does are written on paper cards.The deck is given to the user and he is asked to sort the cards into piles placing itemsthat belong together in the same pile. Users can make as many or as few piles asthey want; some piles can be big, others small. Optional extra steps include askingusers to arrange the resulting piles into bigger groups, and to name the differentgroups and piles. The latter step can give ideas for words and synonyms to use fornavigation labels, links, headlines, and search engine optimization. Because cardsorting uses no technology, a 1995 card sort follows the same procedure as the ones
16
conducted today [41].Today it is possible to find digitized and slightly upgraded versions of card sorting
tasks for testing web architecture, however, the underlying core idea and principlesof them are the same. One example is a web based service named Treejack. It isa tool for testing information architecture, based on tree testing. More specifically,participants in a Treejack study navigate an information hierarchy to indicate wherein the hierarchy they would expect to find a given piece of information or performsome action. Participants can move down the hierarchy or, if needed, they can moveback up. This is done by using digitized cards through a web service provided byTreejack. An example task could be asking the participants to suppose that theyare not a broadband website customer and to find the lowest cost home broadbandsubscription using the digitized cards representing the site navigation. Therefore, atits core this method is based on the same premise as the physical card sorting task[42].
Card sorting tasks can be open, closed, or a hybrid, depending on what is beingresearched. In an open card sort participants sort cards into categories that makesense to them, and label each category themselves [43]. An open sort can help tounderstand how users naturally categorize the types of information on a websiteand therefore to gain insight for designing the initial information architecture ofthe website. In a closed card sort participants sort cards into categories that themoderator gives them. Thus, a closed card sort is good for getting people to rankand prioritize items and actions, like from Important to Unimportant, or Use Alwaysto Use Never. It is also possible to use closed card sorting to task if category labelsmake sense to people and by asking them to place items into predefined categories[44].
Finally, in a hybrid card sort participants sort cards into given categories but canalso create their own categories [44]. This means that running a hybrid card sortcan help to get user input for potential category labels for the website, but also tosee if pre-defined labels make sense to people. According to George [44] hybrid cardsorting is especially useful if one has information architecture partially complete butis interested in seeing how participants label and sort the rest of the structure.
Olmsted-Hawala [45] looks into how different card sorting techniques have beenemployed for re-designing the Census Bureau Web Site. One of the major problemsidentified by previous research for the users of the information rich Census BureauWeb site was locating or navigating to desired content. In particular this madethe team realize that they would like to focus on reorganizing the site to create amore usable information architecture. It was decided to conduct two rounds of cardsorting: first round was open and the second one was closed. 100 representativeterms of the Census.gov Web site have been created as cards and these terms weresupposed to be representative of information that users would be looking for whencoming to the Census Bureau Web site. Terms were a combination of high-levelterms, likely found on our introductory level pages as well as detailed informationlikely found on the lower level pages.
It was found that ten high-level categories, identified in Round 1 and then alsovalidated in Round 2, could potentially be located as link labels on the main Cen-
17
sus.gov Web site. This was followed up with low fidelity prototyping which tookinto account the findings of the card sorting results. It was noted that in additionto the specific work on categories and group labels, the card sorting was helpful foridentifying terms that were confusing or uncommon for the general Census user [45].This was taken account when creating the card sorting experiment for this thesis.
Another card sorting study worth mentioning was conducted by Lewis and Hep-burn [43]. It was done during the planning stages of a website redesign project at theUniversity of Illinois at Chicago. Lewis and Hepburn [43] performed an open cardsort with 15 participants using 93 cards labeled with content from the library’s website. The subjects were asked to think aloud and explain their rationale for sortingthe cards. The researchers used statistical analysis software to run a factor analysison the results and extracted 11 categories of cards that loaded together and 27 cardsthat did not fit a category. Cards that did not load were standalone categories, orwere redundant or meaningless to the subjects. Therefore Lewis and Hepburn [43]conclusions were that combination of the open card sort methodology and largenumber of cards resulted in cumbersome data that required specialized analysis andthat the qualitative data were critical to the interpretation of the quantitative data.
In fact, card sorting seems to be especially popular in redesigning library websitesthat include large scale catalogues. This makes sense given that typically electroniclibraries can consist of a large number of items. They need to be grouped in cate-gories that make sense for the readers to enable them to find what they are lookingfor quickly. In fact, in that sense the CERN service catalogue is somewhat similarto a library – it has many services that need to be categorized from users perspec-tive. In this context the Windows services website can be viewed as a catalogue fortopics related to Windows computers. Thus, employing card sorting as one of themethods to redesign the Windows services website was like designing a subsectionof an electronic library and therefore had to be adapted appropriately. How thiswas achieved is discussed in chapter 6.
19
3 The design process of the Windows services web-site
This project employed the human centered design process to redesign the Windowsservices website. This meant that the work for the project had to be divided intofour parts that corresponded to the human centered design process steps the readerwas introduced to earlier in chapter 2.
To follow this process, first, I had to research the context of use, after that I hadto acquire user requirements. After these two steps I also had to choose the toolsto realize and create my designs with. Finally, I also had to evaluate the designoutputs. The last stage was also the answer the second part of my research questionon how to measure the impact of the redesign process.
Each step of the human centred design process required its own methods andalso had its own findings. Therefore, for example, the findings from researching thecontext of use stage have informed how the user requirements were researched. Inaddition to this, the findings from the first two stages informed the design decisionstaken during the design realization stage. For this reason, I introduce the researchmethods and findings from each stage of research phase by phase before explainingthe next step in the process.
Next, before starting to introduce the work carried out in each phase of thehuman centered design process in detail I provide an overview of how the work wasapproached in each of the major stages.
3.1 Understanding context of use
For the purpose of redesigning the Windows services website, understanding contextof use meant understanding not only what population the website has but alsothe landscape and the population of the Windows services at CERN. I needed tounderstand what type of tasks the different user groups may wish to perform andtherefore the website was supposed to support. I also wanted to learn what typeof services lived on the old Windows services website and how they were organized.I needed to learn what they were both from end users and internal stakeholdersperspective to make sure the expectations of all sides are fulfilled.
To achieve this, first I needed to understand who were the users of the Windowsservices website. Therefore, the first part of the work for this project was focusedresearching the user groups of the Windows services website and identifying theinternal stakeholders as well as their needs. The methods chosen for this researchstage and the findings are presented in chapters 4 and 5.
3.2 Acquiring user requirements
After key user groups and internal stakeholders were identified, the project couldproceed to gather the user requirements from these groups. Methods to specify userrequirements include creating summary descriptions of the tasks that the systemwill support, creating example task scenarios or describing features of the system
20
to meet the context-of-use characteristics. In addition to the user requirements,organizational requirements can be specified as well, taking into account the needsof an organization the design might be produced for [17].
I decided to gather both user and organizational requirements. This was becausethe website was supposed to be a tool for accomplishing tasks which were relatedto how the Windows ecosystem at CERN was set up to work as a whole. However,it was important to understand the user requirements as well because the tasksneeded to be organized based on different users technical competence levels. I havechosen the card sorting and usability testing of the old Windows services websitetechniques for gathering user requirements. I motivate my choice against otherpossible methods and I discuss how I employed them for this project in chapters 6and 7.
3.3 Design realization
After gathering user requirements the project proceeded to design realization stage.The data collected and analysed in the previous stage was used to inform designdecisions. The chosen set of tools for realizing the designs included sitemaps, wire-frames and high fidelity prototyping. I motivate my choice for these methods andexplained how I used them in chapters 8 and 9. More importantly, the high fidelityinteractive prototype created in this stage was the subject to the second usabilitytest which I conducted for evaluation purposes at the last stage of the project.
3.4 Design evaluation
The last stage of the project consisted of the evaluation stage. With the help ofsecond a usability test I gathered data on the performance of the new design ofthe Windows services website. I also already had metrics I collected during thefirst usability test of the old website conducted in the requirements gathering phase.Therefore, I could compare these metrics and answer one of my research questionson how to measure and evaluate the impact of the redesign process this way. Sinceusability tests were conducted in both requirements gathering and evaluation stagesof the project I explain how I set up them up in chapters 6 as well as 10. Thecomparison and the performance of the two UIs is mostly discussed in the evaluationresults part (chapter 11). There I also discuss the findings of other methods I usedto evaluate the outcome of this project.
The overall process of the work carried out for this thesis can be summarizedwith figure 2. Having introduced my overall work approach to the project I expectthe reader to be able to follow the detailed descriptions of the work carried outin each of the project stages with more ease. I present these next starting withexplaining the methodology employed for researching the context of use (chapter 4).
21
Figure 2: The process of the work to redesign the Windows services website
23
4 Understanding context of use, methodsThe first phase of work for the website redesign project involved using a range ofmethods to understand and specify context of use of the old website. This researchalso informed the context of use and of the new website. This chapter discusses thechosen methods and steps taken to achieve this. In each subsection possible methodsfor achieving the project goals are discussed, the chosen methods are motivated andit is explained how they were used for specifically for this project.
4.1 User group and stakeholder analysis methods
First to understand the context of use of the website, I wanted to learn who wereits users. I also wanted to find out who were the stakeholders including the internalones that created and maintained the services for Windows computers that wereoffered through the website. Methods that can be used to collect data for the usergroup and stakeholder analysis include interviews, questionnaires and focus groups.Questionnaire is a fixed list of questions given to a selected sample of persons tolearn about the their attitudes, behaviours, opinions and/or behaviour with regardsto the target system [46]. Questionnaires are useful in gathering data from a largegroup of participants and if kept short they do not take long time for the participantsto complete. However, the success of the questionnaire is typically dependent onhow well the audience is selected. As I was new to CERN I did not have enoughknowledge on who should be targeted with the questionnaire questions.
The focus groups do involve direct interaction with participants. Focus groupis different people assembled to participate in a discussion about a a service andshould ensure all key user groups are being represented in it [46]. However, onceagain since I did not have a clear view of who should be included in this discussion,I decided to go with semi-structured interviews and their use is explained below.
4.1.1 Semi-structured interviews
Interviews are one of the most commonly used facts finding techniques. Their effec-tiveness depends almost entirely on the competence of the interviewer which makesit easy to control the quality of the results obtained [47]. Typically, it is a conver-sation between two people. The interview usually has a theme that is made clearto both parties and the interviewer is trying get information out of the intervieweeabout the subject matter that is being discussed.
In semi-structured interviews, the interviewer comes prepared with a set of ques-tions that he/she would like to find the answers to. Usually, each question is dis-cussed in depth and several why questions are asked in order to get to the truemotive behind the answers given by the interviewee [47]. This form of interviewhelps in gathering information which are previously unforeseen. This was especiallyhelpful in the case of the Windows services website that was supposed to cater formany different user groups and had various stakeholders involved in providing thewebsite functionalities. Users and stakeholders who were interviewed for the project
24
had their own understanding of what the website does and what its strengths andshortcomings are. This includes having own views on which functions are used oftenand services provided by them are clear and which ones are not relevant for some ofthe users. Having a free form discussion with these participants helped to identifythe user groups and other stakeholders of the Windows services website. It alsohelped to understand the Windows services at CERN ecosystem.
I interviewed the Windows services manager, one of the IT-CDA departmentsection leaders, an employee working at the service desk dealing with tickets relatedto Windows computers (for questions see appendix A). After that, once I had someidea who the user groups using the website may be, I also conducted interviews withthe representatives of the user groups (for questions see appendix B). These werelocal supporters of Windows computers, administrative staff, IT staff, infrastructureengineers and physicists. More about these user groups and interview results ispresented in chapter 5.
4.2 Old site navigation analysis
I also wanted to understand what services and information were offered on thecurrent website and how they were organized. Furthermore, since I was planningto use the card sorting method, which involves users sorting through a deck ofcards where each item represents a service on the website, I had to determine whatthe cards should contain. For this reason, I had to break the existing site downinto something for users to sort as cards. One way of doing current site analysisis performing a content inventory of the website. A thorough content inventorytypically lists all available content as a list breaking it down by unique content ID,title, URL, file format (HTML, PDF, DOC, TXT), author or provider and otherfeatures [37]. However, rather than creating an extensive inventory list, my aim wasalso to understand how the different services and contents were organized on the oldwebsite.
For this reason I decided to create a site map of the old website instead. Asitemap is a diagram showing the structure of a site using simple schematics [20].It is normally used to showcase the structure of the site as it is being built and theboundaries of information hosted in the site. In my case, I wanted to understand thestructure of the old website including key functionalities that it had. This helpedme to understand what type of services exist on this website and where they livedin the architecture of the old website. The site map was created by me trying tovisit every link listed on the homepage of the website.
25
5 Understanding context of use, results
5.1 User groups and stakeholders analysis results
User groups and stakeholders analysis was conducted with the help of the data col-lected using interviews. Below I summarize the findings and key facts about each ofthe identified user group from the first round of interviews with Windows servicesmanager, one of the IT-CDA section leaders and employee working at the servicedesk dealing with tickets related to Windows computers. I also added the findingsabout each of the user groups that I uncovered later, after conducting interviewswith the members representative of each of the groups. Every user group presentedis a also stakeholder group, because the users are directly affected by the outcomeof the system. However, not all the stakeholders are necessarily users.
User groups:
• Physicists:
Physicists are a very important user group at CERN since the it was createdto facilitate and enable groundbreaking physics research. However, most ofthem do not use Windows computers, opting for Linux or Mac OS computersystems instead. Their main interactions with Windows services at CERNinclude the need to use or access a Windows computer in their lab to be ableto run specific experiments or work with a particular software that is onlyavailable on a Windows operating system.
• Administrative staff:
Administrative staff includes human resources, finance and other departmentsthat are responsible for performing administrative and managerial duties atCERN. This user group mostly uses Windows computers and therefore shouldbe interacting with Windows services website every now and then, for exam-ple, in case of an issue with the machine. However, after conducting interviewswith two of the representatives of this user group I learned that they have localWindows computer support staff at their workplace. These people typicallyset up computers for the users and assist in solving issues. Therefore, manyadministrative staff members have felt little need so far to learn more aboutCERN’s Windows ecosystem, how they could get help elsewhere and be moreindependent users. Furthermore, typically this user group has hardened Win-dows computers. This means that they do not have administrator rights onthe machine. This was done to prevent them from executing files that maycontain malicious code received through email since their job involves a lot ofcommunication with the outside world.
• IT staff:
There are many different IT department subsections at CERN, thus the workof different IT specialists varies greatly: from software, web, app developers,
26
to those taking care of the infrastructure. Typically they use either Windowsor Linux machines, thus they do need the Windows service website supportguidelines at times. They do not have local Windows computer support staff attheir workplace and have to follow the standard procedure of creating a ticketin case of an issue with their computer that they are unable to solve them-selves. They usually have a standard, centrally managed Windows computerconfiguration if they use Windows operating systems.
• Infrastructure engineers:
From civil, mechanical, electrical and other type of engineering, the engineersat CERN build and test the machines and systems that physicists rely on.Their task is to keep these systems running smoothly, performing repairs andupgrades where necessary. They typically use Windows computers, althoughthey usually care little about the setup as long as the software needed for theirwork is available to them. Thus, they do occasionally use Windows serviceswebsite for Windows installation instructions, or guidelines on how to get theneeded software. They also may need to use some of the functions provided onthe website for troubleshooting and other tools such as recreating computeraccount. This is especially true if their work involves having a computer labto plan and test infrastructure related experiments and upgrades. They areless likely to have local Windows computer support around and therefore needto be more independent users.
Next, I interviewed the representatives of these user groups. This has helpedme identify two additional user groups. One of them was a rather frequent userof the Windows services website. I then held meetings and conducted additionalinterviews with the representatives of those user groups to learn more about them.
• Local computer support staff
Many of the departments, specifically where the staff does mostly adminis-trative work, hired local Windows computer support staff. These people aretasked with setting up the workstations for new employees, helping them down-load applications and troubleshoot in case any problems arise. To do this, thesepeople often use Windows services website because it can help them see whohas privileges to a computer they are trying to fix as well as how it is config-ured. In addition to this, in case a computer installation runs into problems,the website contains tools that can be used to fix them.
This user group is one of the key stakeholders in the current Windows servicesecosystem at CERN. This is because they stand in between the CERN’s ITdepartment and, for example, the administrative staff group users. Thus, tochange the current situation and get more end users to use the new Windowsservices site, people from the local computer support staff user group had tobe involved in introducing the new website and its functionalities to the usersthey are looking after. In addition to this, I expected them to remain primaryusers in some cases where the concepts and the use cases may remain too
27
advanced and specialized for the other end users such as the administrativestaff.
• IT service desk staff
However, CERN also has its own IT support desk. These people are taskedwith solving tickets that the users create through CERN’s Service Now plat-form. This is how all other users at CERN get the problems with their com-puters resolved if their department does not have a local supporter. Thisgroup may use Windows services website to work and solve the tickets thatthey receive if these are related to Windows computers.
From these interviews, I was able to get a more complete picture of the Windowssystem at CERN. The IT department’s intention has always been to have Windowsusers as more independent – able to troubleshoot and set up their workstations usingthe Windows services website. In case they ran into problems they could not solvethemselves they were expected to use CERN’s service desk. However, due to thecomplexity of the system and poor navigation of the website, the additional usergroup of local computer support staff has come into existance. This was not theway CERN’s IT department had intended for the system to work. After I presentedthem the results of the user groups research they told me that while they knew thatsome of the departments had local supporters, they never realized their role wasthat important and that the website they built had such little interaction with endusers of Windows services at CERN.
Next, in addition to physicists, administrative and IT staff, infrastructure en-gineers, local computer support and IT service desk staff, the following internalstakeholder groups have been identified:
• Computer privilege management system authors
This group holds the technical knowledge about some of the functions that cur-rently live on the website. For example, it is possible to retrieve the Bitlockerkey of one’s machine through the website for computers that are encrypted incase it is needed. To reform the current Windows services landscape at CERNthe priviledge authors had to be involved to understand these use cases betterand also be consulted in the process of simplifying them. The complete listof functions and use cases the website has to support will be presented in therequirements gathering results (chapter 7).
• Windows service manager
The person responsible for the maintaining the Windows service at CERN isalso an important stakeholder. While he is not a frequent user of the website,he is responsible for the write up of help articles as well as the updates pushedto the Windows computers through CMF. Further to this, he has a goodknowledge of how the CERN’s Windows ecosystem works, therefore, he wasconsulted frequently in design decision making process.
28
• Heads of CERN’s IT-CDA department
Another group of stakeholders were heads of CERN’s IT-CDA department.They were shown the new designs and also informed of the key findings in theprocess of design. This was done to help them become more informed of theactual Windows services landscape at CERN and also to make sure the newdesign was aligned with the future vision of Windows services that they have.
Analyzing user groups and internal stakeholders early on in the design processwas important because it helped me in next stages of the design process. Thanks tothis analysis I knew who I had to recruit for my usability tests and other researchwork that I did for the project. In addition to this, early interviews with usersand internal stakeholders gave me some sense of what were the issues with the oldWindows services website. In particular I learned that there were some user groupsthat used it not as much, contrary to what the creators of the services accessiblethrough the website expected. This meant that one of the things I needed to uncoverfurther in the process of my research was why this was the case and how this couldbe improved with the new design.
5.2 Old site navigation analysis results
After creating the sitemap of the old Windows services website and analyzing it, Ihave discovered that in addition to services that are available as core part of theold Windows services website, it was in total linked to 8 different websites. Theywere believed to be related to Windows computers by those who chose to createthese links. For example for of them was NICE Administrative Account Reviewwhich is dedicated for reviewing users with administrative privileges on differentcomputers. Another one was printing services containing the list of printers atCERN and instructions of different ways of accessing and using them. The figure 3illustrates the old site navigation starting from the homepage of the current Windowsservices website. The boxes coloured in grey had more links navigating to other pageswhich all could not fit on the map. These were Windows computers, distributedfile system and help documentation links. Thus, two more figures below show thesitemaps starting from distributed file system (figure 4) and Windows computers(figure 5) sections from the website’s homepage.
I noticed that many services not related to Windows computers were present onthe website. For example, the printing service is relevant not only Windows but alsothe users of Mac OS or Linux operating system based computers. This is also truefor the terminal services that help to establish a remote connection to computersthat may have various operating systems. Thus, the Windows services website hasbeen turned into a home and a connecting spot for many services not always clearlyrelated to Windows computers. In addition to this, it did not have clear organizationor any explanations about some of the services. This made the structure of the siteas well as the main purpose of it really unclear.
Doing the old site navigation analysis helped me to better understand the issueswith the old Windows services website, learn what purpose the Windows services
29
Figure 3: Sitemap starting from the homepage of the old Windows services website
Figure 4: Sitemap starting from the DFS section on the homepage of the old Win-dows services website
website was supposed to have and what kind of services were accessible through it.Furthermore, thanks to the old site navigation analysis I was able to prepare for thecard sorting task which I used to gather user requirements.
30
Figure 5: Sitemap starting from the Windows computers section on the homepageof the old Windows services website
31
6 Acquiring user requirements, methodsNext stage in the design process was gathering user requirements for the Windowsservices website. To do this I had several methods to choose from. They includedcreating summary descriptions of the tasks that the system can support, creatingexample task scenarios or describing features of the system to meet the context-of-usecharacteristics [17]. It is also possible to define user requirements as usability goals.Typically, these are effectiveness: the degree of success with which users achievetheir task goals; efficiency: the time it takes to complete tasks, and satisfaction:user comfort and acceptability. These can be derived from the evaluation of anexisting system [17].
To study the user requirements and to identify the areas of improvement ofthe old Windows services website I employed card sorting and usability testingmethods. The card sorting was chosen over the other existing methods such asaffinity diagrams, content inventory and user flows discussed in chapter 2, sinceit enables to collect data that shows how the different users view the informationspace and where they think each item should go [41]. This was especially importantsince the user group analysis has shown that Windows services manager, leadersof IT-CDA department and creators of the tools that were accessible through thewebsite were unaware of who the users of the different tools were. With thosefindings in mind, it was important to get as much direct input that could be used forreorganizing the site from the less well-known user groups perspectives as possible.
In addition to this, I decided to conduct usability testing of the old website. Itwas a way for me to learn about issues the old Windows services website had. Ialso could collect metrics during the tests and then use those as a way to measurewhether the new website design was an improvement of the old one. Next, I discusshow the card sorting and usability testing were adapted and used for this thesis.
6.1 Card sorting task
The card sorting was chosen as a tool to understand the requirements and use cases.In the case of the Windows services website, its content deals with instructions onhow to manage and set up Windows computers as well as interfaces to performsome tasks. Therefore, I expected that the card sorting should reveal not only howusers view content but also which services are relevant for the different user groups.I wanted to understand which of the services reflect the workflow of the differentuser groups. I also hoped that the card sorting task would reveal how familiar thedifferent user groups were with some of the technical concepts.
6.1.1 Card sorting task set up
The typical set up of the card sorting task includes a short description on what theitem does written on a paper card. The deck is given to the user and he is asked tosort the cards into piles placing items that belong together in the same pile [7].
It is also possible to conduct the card sorting task testing sessions remotelyonline using an online service. Usually these services can also help to recruit the
32
participants and obtain a larger sample. This typically ensures higher statisticalsignificance of the results and therefore can be a good solution when the researchangle and the purpose of card sorting is to look for statistical significance in thedata [48].
However, conducting the session remotely means that the researcher cannot ob-serve the participants’ reactions and ask follow up questions [41]. Being able to dothis was particularly important for the Windows services website redesign project.This is because the goal of card sorting was not only to find out how participantswould organize the content from their perspective but also to understand how theywork with different services and information available on the website. In particular,using the cards with all key services on the website enabled me, the researcher, usethese as prompts and find out how much interaction the participants have had theseservices in the past. Therefore, for the purpose of this project I decided to use aqualitative approach to card sorting, stick with face to face task session and a tradi-tional set up using paper cards. I chose not to include the explanation on the cardsbecause I wanted to know how familiar different user groups were with the variousservices on the website.
Regarding the number of cards, according to Brucker [41], while it can be tempt-ing to get the participants to sort all of the content turned into cards, it can easilyexhaust them and result in lack of concentration. It could also be difficult to recruitthe required amount of people since the task would be really long. While Lewis andHepburn [43] used 93 cards for their study and recommended using between 70-80,according to Brucker [41] depending on the project goal between 30 to 40 couldwork too. Thus, following this recommendation I have prepared 45 cards in totalcorresponding to key services and tasks from each category of the website as wellas the other websites linked to the Windows services website. To see the cards usedsee appendix C.
As discussed in chapter 2, the card sorting tasks can be open, closed, or a hybridcard sort. In an open card sort participants sort cards into categories that make senseto them, and label each category themselves [43]. In a closed card sort participantssort cards into categories that the moderator gives them which is is good for gettingpeople to rank and prioritize items and actions [44]. Finally, in a hybrid card sortparticipants sort cards into categories given to them, and can create their owncategories as well. According to George [44] hybrid card sorting is especially usefulif the information architecture is partially complete. Then it is possible to test howthe participants label and sort the rest of the site structure.
The goal of conducting card sorting for the Windows services website case wasto redesign an already existing information architecture. This meant that the opencard sorting might require too much time from the participants to go through all thecontent. This was likely to be the case especially for the users that are less familiarwith the concepts and functions available on the website. The closed card sort-ing could help to determine the relevance and importance of the different services,however, it was also important to determine how might the different user groupsunderstand and view content.
Thus, in the end, it was decided to conduct a hybrid card sort task testing sessions
33
with the participants from user groups that were identified to be less familiar withthe terminology and CERN’s Windows ecosystem during the user group analysispart. I also conducted open card sorting task sessions with the expert users who usethe website frequently. This way the non-expert users could use helper categoriesgiven to them by me if they were not sure how to start the task but still were able tocreate their own categories. Having the freedom to introduce an additional categorymeant that participants still had the space to interpret the items according to theirown level of experience and I could learn how they understood it. Meanwhile,presenting the expert users with the open card sorting task gave me the chanceto get insight into how the existing categories could be redesigned and renamedcompletely, without having the reminders of the old website structure.
6.1.1.1 Card sorting task pilot
To ensure that the key concepts were covered, before the main testing took place, apilot task was conducted with one participant. It was the Windows services managerwho is responsible for the website content and who was also recruited for one of theinterviews when identifying the key user groups.
To save time, he did not answer the demographic and the interview questions,he was only asked to complete the open card sorting task and think aloud whiledoing so. The session went smoothly, however, after this pilot session it was decidedto exclude printing services from the card sorting task to make it shorter. Anotherreason for this was the feeling that the printing services were out of the scope of themain Windows services concept. Therefore, the final number of cards used in realtesting sessions was 40.
6.1.2 Test participants
Nielsen assessed the outcome of smaller card sorting studies by looking at how wellits similarity scores correlate with the scores derived from testing a large user group.A correlation of 1 shows that the two datasets are perfectly aligned; 0 indicates norelationship; and negative correlations indicate datasets that are opposites of eachother. Nielsen [49] found that the correlations of 0.90 (for 15 users) and 0.93 (for20) are good enough for most practical purposes when conducting a card sortingexperiment. Therefore, according to Nielsen [49] while testing 30 people and reaching0.95 for a big, well-funded project with a lot of money at stake might make sense,for most projects with limited resources 15 users should be good enough.
Thus, this was the number of participants recruited for this study. In total3 physicists, 3 administrative staff members, 2 infrastructure engineers, 2 regularIT staff members, 3 local computer support staff members and 2 IT service deskmembers were recruited. All them were CERN based.
Seven out of fifteen participants completed the open card sort. They ended upintroducing their own categories and suggesting names for them. Eight participantscompleted the hybrid card sorting task with the help of the categories introducedby the moderator. Those who completed the open card sorting belonged to localcomputer support, IT service desk and IT staff member users groups. The rest of
34
the participants who completed the hybrid card sort were physicists, administrativestaff members and infrastructure engineers. All the participants who completedthe open card sort were from the user groups where the users were either using thewebsite more frequently and/or were more familiar with Windows services landscapeat CERN and technology in general.
6.1.3 Task procedure
Each card sorting task began with a short interview. The aim of this was to find outthe demographic information, the technical background of the participants as wellas to learn what their daily tasks might involve. They were also asked how oftenthey use Windows computers at CERN to gain insight on what to expect duringthe card sorting task.
After this the participants were presented with the cards mixed in a randomorder on the table, ensuring all of them were visible to them. They were explainedthat the cards were representing the content and available services of an existingCERN website dedicated to Windows services. Next, they were asked to group thecards in piles of services which they think belong together.
In the hybrid card sort the participants were told that they can either try to placethe cards into introduced categories or they can introduce their own categories if theyfeel that a card cannot be placed in either of these. The starter categories introducedby me, the task moderator, to them were “computer account”, “remote desktop” and“guides and help”. They were also told they could have as many categories as theywanted and that there were no right or wrong answers for this task. They wereasked to think aloud while sorting the cards – share their doubts, ask questions.After the task was completed they were asked to mark services they use frequently,services they never heard of and services that they are aware of but do not use.Finally, they were asked to share their thoughts if there were any services that theybelieved were missing. For used services, the participants were also asked how theyuse the services and if they use them through the website because some of theseservices could also be reached by other means. For the cards where participantsmarked that some concepts were unclear to them, the users were asked to explainhow they understand the concepts.
In the open card sort conducted with the expert users, the participants were notgiven the helper categories. They were instructed to place sort cards in piles thatmake sense to them and label these piles. Otherwise the procedure was essentiallythe same.
For full interview questions that were used please see appendix E. I also made theparticipants sign a simple consent form at the start of each task (appendix D) thatallowed me to use the observations and survey results anonymously in this thesis.
6.1.4 Data collection and analysis
All experiments took place at CERN, in meeting rooms. All the sessions were one-to-one to ensure the possibility to have in-depth interviews and also to make sureparticipants did not compare their performance to others. It also helped to ensure
35
they would create their own versions of the card categorization without the influenceof anyone else.
Further to this, each participant had their own deck of paper cards printed forthem. While this was not the most sustainable way in collecting data in terms ofthe paper waste, since I was conducting the testing sessions alone it helped me toensure I captured everything. For instance, I was able to ask participants to markthe concepts that were unclear to them and services they used a lot with differentcolours, which helped me to visualize and interpret the data, directly on the cards.
During the experiment, the data collection was done through observational notes.I also recorded the interviews and the sound during task to ensure all the commentsby the participants were fully captured. Also, this way as the task moderator andinterviewer I was able to focus my full attention on the process and asking thefollow up questions instead of worrying about taking notes. After each session, Itook photos of how cards were organized by participants as demonstrated in figure6.
Immediately after the session, the card sorting result of each participant wasvisualized. The main takeaways that came up during the task and interview werealso noted next to the visualization board. Further steps undertaken to analyze datawill be explained in chapter 7.
Figure 6: Card sorting data initial visualization
6.2 Usability testing of the old website
Usability testing was another method used for development in this project. Usabilitytesting is a technique used in user-centered interaction design to evaluate a productby testing it on users. This gives direct input on how real users use the system [44].
36
I have decided to conduct user tests of an old system to evaluate it and identify thekey pain points of a complete user journey to do certain tasks.
However, as this thesis also aimed to demonstrate the impact of design methodsapplied during the project on the design outcome, the usability testing was used forthis purpose too in addition to using it as a development method. Therefore, theusability tests were conducted both on the old system and the new system to be ableto collect metrics on both and measure the improvement. How this was achievedis presented in chapter 10, dedicated to evaluation methods. Here I discuss howusability testing was used as a development method.
6.2.1 Test materials
The object of the first round of usability testing was the old Windows services web-site before it was redesigned. It was developed in the year 2000 using ASP.NETframework: a software framework developed by Microsoft that runs primarily onMicrosoft Windows [50]. The other websites that currently are linked to the Win-dows services website were also developed with the same technology. Help articleson the right side of the homepage as seen in figure 7 were the only exception. Theywere hosted on Sharepoint. Sharepoint is a web-based collaborative platform knownas a document management and storage system, also developed and offered by Mi-crosoft [51]. This is why the UI for help looks slightly different than the homepageUI as demonstrated by figure 8.
All the participants did the user tests on my laptop - Macbook Air 13 inch screen.They were given some time to familiarize with the laptop before the task began sincenot all of them were used to using Mac OS X operating systems.
Material that was subject to the usability test in the second round of testingwhen collecting the comparison metrics was the high fidelity design produced. Itcan be viewed in design realization chapter 9.
6.2.2 Participants
In total 5 users were tested. According to the cost-benefit analysis of usability testingby Nielsen Norman Group [24] this number provides the optimal ratio and allowsto discover most of the errors. If a larger number of users is tested, the findingsstart to get repetitive. I tried to get all key user groups represented in the sampling,thus I had tested 1 physicist, 1 administrative staff member, 1 IT staff member, 1infrastructure engineer and 1 IT service desk member working on Windows support.More information about the participants can be seen in table 1. I also made theparticipants sign a simple consent form (appendix F) that allowed me to use theobservations and survey results anonymously in this thesis.
6.2.3 Procedure
The user test was started with a short demographic survey that collected data likeage, position at CERN, their technical background and what their daily tasks are.They were also all asked what type of computer operating systems they use. This
37
User number #1 #2 #3 #4 #5Gender Male Female Male Female MaleAge 23 26 26 25 30
User group Physicist Adminstaff IT staff Infrastructure
engineers IT support
Time at CERN 7 months 5 months 1 year 2 months 9 yearsWork PC Linux Windows Windows Windows Windows
Table 1: Demographic data of participants that took part in both usability tests
38
Figure 7: Old Windows services website home page
Figure 8: Old Windows services website documentation
helped me to learn more about participants’ background and if they have previouslyused the Windows services website.
After that the participants were given a task to complete using the website.The task they were given was chosen based on which user group they belongedto. Thanks to the user group analysis conducted earlier, I knew that, for instance,physicists do not normally use Windows computers a CERN, however, they need touse Microsoft office applications occasionally. The tasks given to the participantsduring tests can be seen in table 2. The complete list of interview questions theparticipants were asked can be viewed in appendix G.
39
User number Task#1 #2 #4 Find out how to install application you need#3 #5 Find out your computer’s Bitlocker recovery key
Table 2: Tasks given each participant during both rounds of usability testing
40
Category Measurable attributeEffectiveness Number of clicks to complete task, task completion (Yes/No?)Learnability System Usability Scale (SUS) items 4 and 10Satisfaction System Usability Scale (SUS) remaining itemsEfficiency Time to complete the task
Number of errors No. of errors the user made during the task, for examplehaving to navigate back to the start page.
Lostness
L =√(
NS−1
)2+(
RN−1
)2N: The number of different web pages visited whileperforming the taskS: The total number of pages visited while performingthe task, counting revisits to the same pageR: The minimum (optimum) number of pages that must bevisited to accomplish the task
Table 3: Metrics of usability recorded in both usability testing rounds
During the test, the users completing the task were encouraged to think aloudand comment their actions. After the task the users were asked a few questionsabout the task and I also asked them to fill out a feedback form containing a 5-pointLikert scale. I decided to use System Usability Scale (SUS) (appendix H), as it hasbecome an industry standard and gives reliable results even with small sample sizes[52].
6.2.4 Data collection and analysis
The data collection was carried out by recording the screen of the device on whichthe task was performed and the sound was recorded as well. Observational noteswere also taken by me. The metrics collected during usability testing were based onISO 9241-210 definition of usability metrics [3]. The metrics that were collected areshown in table 3.
Efficiency was calculated based on the times it took to complete the task. Datafor effectiveness was collected based on how many errors the users made, how muchassistance they needed and whether they could complete the task at all. While SUSoriginally was only intended to measure perceived ease-of-use as a single dimension,recent research shows that it provides a global measure of system satisfaction andsub-scales of usability and learnability. In particular, items 4 and 10 provide thelearnability dimension and the other eight items provide the usability dimension[52]. This means it is possible report on both subscales as well as global SUS score.Therefore, to measure learnability items 4 and 10 were recorded, whereas for themeasure of system satisfaction the SUS scores from all the remaining items werecollected. Also, since the aim of the study was to redesign the service and one ofthe key aims of card sorting is to improve the navigation of the system, the metrics
41
to calculate lostness were also collected as seen in table 3.
6.3 Investigating organizational requirements
In addition to studying user requirements, I also wanted to understand the organi-zational requirements for the Windows services website. I wanted to specify whatCERN would like to achieve with this website from organizational point of view toclarify its purpose and vision. To investigate these I chose to have a meeting and adiscussion with the Windows services manager, IT-CDA group leaders and authorsof the services that were accessible through the Windows services website. I didthis after I conducted the card sorting and the usability tests and had a clearer ideaof the user requirements as well as the issues the different user group were facingwhen using the website and Windows services at CERN. I presented the key findingsabout the user groups and results from card sorting and usability testing and thenheld a discussion about them. I also asked some specific questions about each of theavailable services on the website as well as how the authors of these services haveenvisioned their future in relation to the website.
43
7 Acquiring user requirements, resultsThis chapter presents the results from the data collected and analyzed using thecard sorting method and usability testing to gather the user requirements. It is alsodiscussed how these findings informed the design decisions when creating the designartifacts in the following stage of the design process.
7.1 Card sorting results
The data gathered with the help of card sorting went through several stages ofanalysis. I motivate the chosen methods of analysis and present the obtained resultsusing these analysis methods.
7.1.1 Identifying the most commonly suggested categories
The first step of the analysis was to identify the most commonly suggested cate-gories in the open card sorting task and see how those compare to those that wereused in the hybrid card sorting task. This was done by creating the affinity dia-gram displayed in figure 9. In the hybrid card sorting the categories given to theparticipants were only three. Yet, in the open card sort 5 categories emerged. In par-ticular, two categories, “applications/apps” and “security”, that were part of “guidessection” previously seemed to be perceived as standalone categories by some of theparticipants.
Figure 9: Affinity diagram to identify the most commonly suggested categoriesamong all of the responses from the open card sort and relationship of those to thecategories used in the hybrid card sorting task
44
7.1.2 Card sorting results matrix
Next, a spreadsheet, listing the cards in the rows and the categories in the columnswas created with the aim to turn it into a histogram matrix. It was used to vi-sualize participant agreement of cards within specific topics in card sorting [53].This method was chosen instead of the other available ones such as dendograms,similarity matrix or word frequency cloud as seen in figure 10.
The histogram/similarity matrix was chosen because creating dendograms isdone with statistical analysis techniques to explore the way participants group thingsaccording to their own logic, and so work best when participants face no constraintswhile doing so. Thus, in short dendograms work the best with an open card sort.Meanwhile, I employed both open and hybrid approaches where some categories thatparticipants sorted the cards into were already defined. Therefore, this would haveintroduced bias to the dendogram. Further to this, dendograms would also providelittle additional value due to the small sample size I had and relatively small numberof cards used. Regarding the word cloud, while it helps to see most popular cate-gories easily, it is rather hard to see relationships between the different categoriesusing this visualization method. Hence, studying proportions of different cards be-ing placed among various categories with the help of histogram matrix seemed themost appropriate for this project.
Figure 10: Common card sorting visualization techniques: A) dendogram fromhierarchical cluster analysis B) similarity matrix of card relationships, and C) wordfrequency cloud of participant-generated topics.
First, all cards were assigned numbers (table 4) since it was easier to referencethem across the different metrics – from quantities to percentages this way.
Next, the data of how many participants added a specific card to each categorywas added and was then turned into percentages. The categories were: computeraccount, windows guides, remote access, security, applications and I also left a notsure category in the spreadsheet. This is because some participants were not familiarwith the terms or for some other reasons felt they could not place the card into anycategory and felt they could not create a new category either.
After that the card placements which had a percentage equal to or greater than33% were selected and are presented in table 5. This cut off point was chosen giventhat the number of cards and participants was not too high, therefore, accepting re-
45
Computer account status Card 1 Card 21 View computer softwareinventory
Recreate computer account Card 2 Card 22 Get Bitlocker recovery keySet local admin accounts Card 3 Card 23 About spywaresSet local admin password Card 4 Card 24 About hoaxesNICE administrative accountreview Card 5 Card 25 Microsoft office
Linux Client Card 6 Card 26 Main user of the computerMac Client Card 7 Card 27 Recommended applicationsWindows Client Card 8 Card 28 Manage accountRequest more space with DFS Card 9 Card 29 Status of Windows 10 at CERNWindows Installation Card 10 Card 30 Status of Windows 8 at CERNWindows Desktop settings Card 11 Card 31 Hard disc encryptionAntivirus Card 12 Card 32 Remote applications
How to verify computer name? Card 13 Card 33 Manage remote Desktopconnection for a client pc
Moving or renaming a computer Card 14 Card 34 Remote connection via RemoteDesktop Service
Windows remote desktopservice Card 15 Card 35 Setup local Remote Desktop
configuration
Virtual machine on Windows Card 16 Card 36 Defining RDP file to connect toRemote Desktop
Add or remove software Card 17 Card 37 Mozilla firefoxSee patch status details Card 18 Card 38 Internet explorerAvailable antivirus software Card 19 Card 39 About terminal servicesAbout viruses Card 20 Card 40 Login to terminal services
Table 4: Each card assigned a number for further analysis
ally low percentage values could mean having one card appearing in many categories.The final results after doing this appears in table 5.
From table 5 we can see that some of the cards in computer account and remoteaccess categories were assigned to those categories by all the participants that tookpart in the card sorting tasks. Thus, in the new sitemap and the new UI designthese categories should be kept separately. We also see that some of the cards thatare part of the Windows guide category also belong in security and applications witha higher percentage of participants placing them in the latter two.
At this point it is also important to note that most of the participants whoparticipated in the open card sorting task have placed these cards in the securityand applications categories. Those that placed them in the Windows guide categorywere mostly participants who completed the hybrid card sorting task. In additionto this, two cards from the Windows guide category have been placed many timesin the uncertain category. Those cards are “Request more space with DFS” and“Virtual machine on Windows”.
46
WindowsGuides Security Applications Computer
accountRemoteaccess Not sure
Card 9 33% 67%Card 10 100%Card 11 40%Card 12 40% 60%Card 13 67%Card 14 53% 47%Card 16 47% 53%Card 18 47%Card 20 40% 60%Card 22 40%Card 23 40%Card 24 40%Card 25 40% 60%Card 27 40% 60%Card 28 100%Card 29 100%Card 30 100%Card 31 67% 33%Card 37 40% 60%Card 38 40% 60%Card 19 60% 40%Card 26 33% 67%Card 1 100%Card 2 100%Card 3 100%Card 4 100%Card 5 67%Card 21 80%Card 6 47%Card 7 47%Card 8 47%Card 32 67%Card 33 100%Card 34 100%Card 35 100%Card 36 100%Card 39 67%Card 40 100%
Table 5: Card sorting matrix
47
7.1.3 Key qualitative findings
The findings of the matrix can be better explained with the qualitative data thathas been collected after participants completed the card sorting tasks. They havebeen asked about the reasoning behind some of their choices, in particular on thosewhich according to them were hard to place anywhere. Questions I asked can beviewed in appendix E. These were the main findings from the qualitative part of thetask:
• Windows Guides:
Many of the participants were not sure how to name this section. Partly thisis because in the current website many topics are covered there. This is alsoone of the reasons why in the open card sort several other categories emerged.Some explanation cards overlap with the computer account section. This isbecause the participants felt that explanations were needed in order to beable to perform some of the actions available listed in the account section.Therefore, probably having clear links to information in the guides sectionwould be beneficial for users who are less familiar with the concepts and tasksthat the computer account section deals with such as local computer accountsand similar.
• Security:
Some participants were not sure about the meaning of the word hoax. How-ever, everyone that took part in an open card sort task grouped the viruses,hoaxes and antiviruses cards in the security section which they also all namedsimilarly. In the old version of the website it was a subsection of the Windowsguides section. Thus, it could be argued that security items should be groupedtogether more distinctly based on the results of the card sorting task.
• Applications:
Similarly to the security section, information about available applications forWindows computers and how to download them was mostly a part of theWindows guides section on the old website. However, the findings of thecard sorting task would also suggest that this information should be presentedmore clearly to the participants and may potentially deserve its own section orsubsection. This is because the participants in the post-interview discussionsaid this information currently was difficult to find.
• Computer Account:
For the computer account, one of the major hints even for the participantswho were not always sure what some of the services such as recreate computeraccount were supposed to do, was the word account or computer. It hasinfluenced their decision to put these cards in the computer account category.Also, thanks to the interviews it has been verified that administrative staffand physicists do not really have to deal with any of the actions available
48
under this category themselves. This is usually the task of their local ITsupport dedicated person. Unsurprisingly these people were unfamiliar withthe terms. This means that to increase the usage of functions living under thissection, renaming may be required as well as facilitation and educational toolshelping less experienced users become more familiar with tools for computeraccounts.
• Remote Access:
Most card sorting task participants made the same choices on the placementsof the cards that in their opinion should belong in remote access category.However, some of the cards and tasks available in the remote access sectionwere not clear to all users such as “Manually defining the RDP file to con-nect using the Remote Desktop Gateway”. Therefore, while it is clear whichservices should belong in this section, many of them require renaming andmuch clearer instructions for the users to be able to configure remote desktopconnection successfully. However, many participants felt that remote accesswas not purely Windows computers related service, in particular the physicistsoften need to connect remotely to the services of their labs that may also oftenbe Linux based. Thus, users that needed to connect to a computer remotelywere not only Windows operating system users. Instead, Windows computersusers were just some of the users of the remote access services.
• Not sure:
Regarding Virtual machines related information on Windows services, someof the participants said they were sure what this meant or how this servicelooks like. These were mostly administrative staff members. Others who didknow the meaning of this said it did not belong in the Windows guides sinceyou could run the virtual machine from any computer and it also did not haveanything to do with all the other remaining categories. This was also similarwith DFS. Many of the participants were aware of the service and have usedit, as it is a separate website which allows to request more storage and manageit using an online website for their CERN computer account. However, theyfelt this was a related service rather than something which could be placed ineither of the other categories.Therefore, perhaps a clear section on the websitecontaining links to related services may be beneficial.
7.2 Usability testing of the old website
In this section I present the main problems uncovered thanks to the usability testconducted with the old website. It focuses on key observations and qualitative find-ings as these have informed the design decision making the most. The quantitativemetrics that I have collected as well such as efficiency, error rate and others arepresented in the evaluation section (chapter 11) where the scores between the oldUI and the new UI are compared.
49
The observations made during the test coupled with think aloud protocol usedduring the usability tests and quick follow interview helped to uncover the mainproblems with the old website. These were:
• Troubles using search because of not knowing what terms to search for.
• Separation between the help contents and tools to complete task across differ-ent websites did not help participants to navigate themselves back to the taskafter finding instructions.
• Various tools and help being part of different websites made navigation diffi-cult.
• Instructions found in the help section did not have any links which would helpto point the user back to the task starting point. The tasks and instructionsfor them were not well connected on the old website.
• Many of the instructions were outdated and it was not clear if one should befollowing them.
• Too much information present on the screen, participants felt overloaded.
• Unclear purpose of the website due to poor naming and too much information.
• Information hard to follow as it is too long and uses many technical terms.
• Unclear what information to expect in which help category, organization notintuitive.
These were kept in mind when creating the new design. In particular they gavethe new design the idea to display relevant information articles for users next toeach tool. This would save users time looking for information on what the tool doesand how it works in the help section. How this was realized the design is presentedin more detail in chapter 9.
7.3 Website purpose and organizational requirements
The discussion I had with the internal stakeholders about my research findingshelped me learn that the intended purpose of the Windows services website wasto provide information about Windows services at CERN. This includes teachingusers how to set up their computers for work, install applications safely, providinginformation about what privileges each machine has. For example, accessing in-formation on who is the computer managed by, does the user have permissions toinstall applications that are not provided at CERN or not and similar.
One of the main organizational requirements for the website were making usersmore independent in managing their machines. Providing functions to troubleshootin case of failed installation of Windows or required Bitlocker key and providingfunctions to manage permissions for Windows computers. More importantly, after
50
the discussion and the research I conducted it was clear that the old website hasfailed in achieving these organizational goals. Because of this although many peopleat CERN were users of the Windows operating system, very little of them usedthe Windows services website. They relied on on support desk or local supportersif they had those instead of trying to accomplish Windows computer managementtasks independently. Therefore, one of the main goals of the new website was toreduce this dependence on the support where this was possible.
7.4 Use cases of the website
Based on the card sorting results, usability test and interviews both with users andinternal stakeholders, it has been decided that some parts of the website would re-main mostly used by advanced users: that is those who know which functionality ishelpful in each situation. This is because some use cases were really specific and ad-vanced. They also required some knowledge about how to use some of the functionssafely. However, over time and with proper set up and instructions they could alsobe used by more users. Further to this, I also identified a set of features and usecases which could be used by all users that do not require as much specialist knowl-edge and therefore where the reliance on supporters could be reduced immediately.Overall, the following use cases of the website have been identified:
For all users:
• Retrieving one’s Bitlocker key in case it is needed.
The goal is to get many users to do this independently, without support or ser-vice desk involvement. As many users may be unfamiliar with the terminology,help articles should accompany the functionality.
• Being able to find out what type of computer configuration you have.
Knowing this would answer many questions about the user experience andgive users more power over the decision making especially in the cases wherethe local support decides that the users should have hardened computers forthem.
• Being able to learn how to install applications.
As not all departments have hands on local support available it is importantto help the end users be more independent.
• Being able to install and set up Windows.
Once again, as not all departments have hands on local support available it isimportant to held end users be more independent.
• Being able to find relevant services such as for instance setting up printers orconnecting to a computer remotely easily.
51
Advanced users:
• Being able to recreate a computer account in the domain if the Windowsinstallation fails.
• Being able to retrieve local administrator password if it is needed to troubleshoot.
• Being able to set someone as a local administrator as a computer in case theserights are needed to perform some actions on the computer.
• Being able to see someone else’s computer configuration so you can help them.
Overall, the identified use cases and findings from card sorting as well as usabilitytests helped to define a list of features the new website should have. It also helpedto identify key problem areas of the old website. This information was used whenmaking layout decisions and creating design artifacts. I present the methods forcreating these in the following chapter.
53
8 Design realization, methodsAfter gathering user and organizational requirements I was ready to produce thenew designs. I have used several design realization methods to turn the insightsdiscovered by different research methods into designs. I present the methods used,discuss the motivation for using those and explain how I used them.
8.1 Sitemaps
A sitemap is a diagram showing the structure of a site using simple schematics[20]. It is primarily used to showcase the structure of the site as it is being builtand the boundaries of information that is to be contained in the site. The sitemap reflects the understanding of the site’s goals and structure. Typically, the sitestructure is used to represent each page of the website in a menu which is usuallyaccessible on the main page of the site. Sitemaps consist of labeled blocks and linesto display the relation between the pages represented by the blocks. The pagesare usually connected by several redundant links. This is designed to make theprocess of navigating between the pages easier but sitemaps generally showcase onlythe primary paths between the pages. One of the key reasons why sitemaps areoften used during the process of designing a website is because very often it canbe translated into a menu structure. In addition to this, it helps in creating theinformation architecture of the website.
For these reasons I previously created a sitemap of the old website to under-stand its structure and goals better. This was also needed to prepare for the cardsorting task. However, I also created a sitemap of the new website before creatingthe wireframes and further design outputs. This was done to understand how thenavigation on the new site should work and decide how to structure the access todifferent services on the Windows services website in terms of navigation.
8.2 Wireframes
Wireframes are representation of the information or content that should appear ona particular page of the website [20]. Typically, wireframes have no media besidestext and placeholders representing where a image or a video is supposed to go inthe page layout. Wireframes are predominantly made using simple colors, some-times monochrome or gray-scale to communicate key features of a page. The maingoal is to represent the content structure of individual pages rather than graphicdesign as they represent the information organization and the elements that sup-port navigation on the page. This is done so that the information architect creatingthe wireframes can indicate what content should go where meanwhile the graphicdesigner determines how to make the information visually appealing. They can alsoact as tools for communication with the clients.
Wireframes were used in this project to communicate the information organiza-tion of the new website with the internal stakeholders of the project. They werecreated after analyzing the collected user requirements data. They were shown to
54
the internal project stakeholders several times until the agreement about the struc-ture of the new site was reached before proceeding to make decisions about thevisuals of the new website.
8.3 High fidelity prototyping
According to Nielsen [24], we cannot afford not to test prototypes. This is becausethe design gets tested anyway: once your system goes live and people begin to useit, they are testing it. And rather than collecting feedback in a low-risk researchsetting, where you can learn, and then react and change the design, you may haveactual unhappy users or customers on your hands. In addition to this, changingcode typically is very expensive.
Since the Windows services website was going to be completely rebuilt with anew stack of technology, I decided there was value in creating a testable prototypewith the new design of the website. Further to this, test scores of the prototypecould be compared with the scores of the old design and therefore test whether newdesign is an improvement on the old one.
To receive realistic feedback, ideally one should have a high fidelity prototype[54]. Typically a testable, high fidelity prototype should have clickable links andmenus, automatic responses to user’s actions, realistic visual hierarchy, priority ofscreen elements, and screen size and somewhat realistic content. Some of the populartools for high fidelity prototyping including Sketch, InVision app, Axure and AdobeXD.
My choice was impacted by what CERN had available: I had access to a fullypaid for Adobe Creative Suite including Adobe XD and therefore decided to use it.It enables to create colorful prototypes and make them clickable, therefore, it fittedwell with what I wanted to.
55
9 Design realization, results
9.1 New site navigation map
The sitemap of the new Windows services website based on the card sorting andinterview results was created as follows:
• Home is the central entry point of the site. It features only key services, helparticles to conveys what the website is about. This is clearly different from theold site where the homepage contained links to many different services thatoften were not even Windows related.
• Tools is a section which did not exist previously. I found it to be a useful wayto categorize the service actions that the Windows services website providesinstead of using the vague terms such as computer account.
• Help section contains articles relevant for the website, divided into subcate-gories.
Figure 11: Developed sitemap for the new website
9.2 Results from wireframes development
Figure 12 shows the wireframe designed for the main page of the new Windowsservices website. Some of the key components used in the wireframe are describedand explained below.
• CERN header - is a compulsory element that must be included according toCERN’s web design guidelines.
56
• Landing page - this component is the first visible section of a page. It ischaracterized by an image or animation in the background and some textdescribing what the page is about on top of it. In figure 12, this is representedby the first component on the page with the search bar on it. It also includesa navigation menu on the side for the page.
• Body – main content of the page which can contain images, text. In the caseof the Windows services homepage it is divided into 3 main sections.
• Help section- the space is reserved for getting started and frequently askedquestions space. Only limited amount of articles are permitted in this space.
• Tools section - the space is reserved for featured tools that the website contains.
• Related services - the space in the page is reserved for displaying linked andpotentially useful services to Windows users, but these services are not partof the Windows services website.
• Footer - this component is typically fixed and visible in all pages of a website.It usually has links to the main sections of the website and contact informationof the organization. This is the last component seen in figure 12.
Footer, CERN header, related services and landing page served as reusable com-ponents throughout the website in the needed places.
9.3 Results from high fidelity prototype development
Next, a high fidelity prototype was created using Adobe XD. The structure of thepages remained largely the same as decided during the wireframing stage. As seenfrom figure 13 I decided to use neutral colours, since this website is supposed tocontain tools and I wanted the user to be able to focus at the task rather than beingdistracted by bright colours and images [55].
Helpful but not too distracting infographics is considered to be good practicefor this type of websites [56]. Therefore, I introduced the icons which correspondedwith the purpose of the link or tool included. The complete high fidelity prototypecan be viewed in appendix I.
Overall at the end of design realization I had a design that was ready to bemoved into development stage. Since I had a complete high fidelity prototype withcolors, if the development team were to take over at this stage, they could alreadystart creating the styling and the basic interaction for the page. Meanwhile, thesitemap also helped to get a picture on how the application routes should be createdand how many of these would be needed. However, since implementation was notthe focus of this research work, it is not discussed further and instead the evaluationwork is presented. This is because one of the research question this work had setout to answer originally was evaluating the outcome of the human centred designprocess.
57
Figure 12: Developed wireframe representing the home page
58
Figure 13: Home screen of the high fidelity UI which was tested in the secondusability test
59
10 Evaluation, methodsThe final stage of the project was to evaluate the output of the design work that wascarried out throughout the project. Vredenburg et. al [57] collected data from 100human centered design practitioners to research how they measure the success ofimplementing its practices in their organization’s works. They found that there wasa lot of variance, but there was an agreement that the most effective metrics wereinternal design satisfaction, external satisfaction of users measured by user feedbackand to what extent the user requirements were represented in designs.
Therefore, I used the same principle to evaluate the work of this project anddecided to focus on the first two: external satisfaction of users and internal stake-holders. The motivation for each chosen method is briefly discussed in this chapter.Evaluation results and the analysis are presented in chapter 11.
10.1 Usability testing for evaluation
The collected usability metrics during the usability test of the old UI describedin chapter 6 were used as the external user satisfaction metric for evaluating thedesigns. These were: effectiveness, learnability, satisfaction, efficiency, number oferrors and lostness. I was not alone in choosing this method. Becker and Yannotta[35] conducted four rounds of usability testing throughout the process of buildinga new academic library website. The old site was tested so that the scores it re-ceived could be used as a benchmark to establish how good the new design was. Inaddition to this, several drafts and iterations of the new site have been tested aswell. According to the authors, testing throughout the design process has provedto be an effective way to build the website and also a way to measure progress ofthe design process. A similar approach has been taken by Pendell and Bowman [36]when redesigning Portland State University Library’s Mobile Website.
Therefore, following these examples, this project has also employed usabilitytesting both as a development and evaluation method. Two rounds of usabilitytesting in total were conducted to be able to compare the difference in collectedscores between the new and the old design. The UIs that got tested and comparedwere:
• The old Windows services website
• The high fidelity prototype of the new website
The reason a prototype was tested over a working new site was delay in softwaredevelopment method choice for the new website on the CERN side. However, therewas also value in testing the prototype: being able to fix more serious issues with thedesign in case there were any. This could prove helpful in saving time and resourceson the development side.
60
10.1.1 Test participants
In both testing rounds, 5 users were tested. I tested the same participants whotook part in the the first round of usability testing conducted with the old UIduring requirements gathering stage. Their demographics were already presented inchapter 6, table 1. Since there was a 3 month time gap between the first and thesecond usability test participants had little chance to remember what they had todo in the first testing round. This is because the new UI presented to them wasredesigned and re-organised completely. Therefore, even if they could rememberhow they previously completed the task, the process to achieve the same goal wasdifferent in the new design. Finally, since all participants agreed to be tested againin the first usability testing session and I had a sample which ensured all key usergroups were represented, it was time efficient to set up the tests with the samepeople again.
The collected metrics, the tasks given to the participants and the test proce-dure for the first usability test conducted with the old Windows services websitehave already been discussed in requirements gathering phase, chapter 6. They haveremained essentially the same for the second usability test with the high fidelityprototype. Results and analysis comparing the scores of the two UIs are presentedin chapter 11, evaluation results.
10.2 Feedback collection sessions
The internal design satisfaction is also a way to measure the effectiveness of employ-ing human centered design methods in software development [57]. Internal designsatisfaction was measured by involving internal stakeholders identified during theuser and stakeholder group analysis stage. The stakeholders involved were: Localcomputer support, Windows service manager, heads of IT-CDA department and Ialso occasionally asked for feedback from IT service desk staff. I was getting feed-back from them regularly through unstructured interviews during progress updatemeetings.
61
11 Evaluation, resultsIn this chapter, I present the results of the evaluation of the design outcomes of thisproject. I compare the usability scores between new and the old Windows serviceswebsite interfaces. I also present the collected feedback from internal stakeholdersthroughout the design process.
11.1 Usability testing for evaluation
11.1.1 Usability testing scores
The overall SUS scores after each testing session were calculated before splittingthem to learnability measure and the system satisfactions scale. The recommendedapproach for calculating the overall SUS score is for each of the odd numberedquestions to subtract 1 from the score and for each of the even numbered questions,subtract their value from 5. After that you take these new values which you havefound and add up the total score. Then you multiply this by 2.5. This helps tosee the score out of 100 [52]. Therefore, first, before any further analysis, the scoreswere converted accordingly using the presented approach, as seen in table 6, to beable to see the differences in the scores more clearly.
Old UI New UIParticipant 1 22.5 65Participant 2 17.5 82.5Participant 3 20 82.5Participant 4 22.5 80Participant 5 40 67.5Average score 24.5 75.5
Table 6: Overall SUS Scores after conversion
In terms of the interpretation of the SUS scores, according to research into valid-ity of these scores across many design research projects, the average SUS score is 68[58]. Therefore, according to the authors, a SUS score above 68 would be consideredabove average and anything below 68 is below average. As we can see from table 6,individual scores for all the participants for the old UI were more than half below theaverage. By contrast, the new UI scored above average for all participants exceptone. However, that one is only 3 points below the average. Therefore, based on theSUS scores it can be said that the new UI has scored significantly better than theold one, as the average score of the new UI for almost all participants is two timeshigher.
Next, to conduct a more detailed comparison, items 4 and 10 from the SUS scale,which according to research can provide learnability dimension, and the other eightitems which show the overall system usability satisfaction scale were extracted intotwo separate items: learnability and satisfaction. They are presented in tables 7and 8 alongside other metrics.
62
User 4 User 5Old UI New UI Old UI New UI
Efficiency (time) 02:25 00:59 02:15 01:05Effectiveness (clicks) 8 3 10 3Learnability 10 15 12.5 15Satisfaction 12.5 65 27.5 52.5N: The number of differentweb pages visited whileperforming the task
4 3 6 3
S: The total number of pagesvisited while performing thetask, counting revisits to thesame page
7 4 9 3
R: The minimum number ofpages that must be visited toaccomplish the task
4 3 3 3
Lostness 0.184 0.06 0.36 0Errors 2 1 2 0Assistance No No No No
Table 7: Various usability measures for users 4 and 5 from usability tests with oldand new UIs.
User 1 User 2 User 3Old UI New UI Old UI New UI Old UI New UI
Efficiency 04:50 03:10 02:30 00:55 04:35 01:58Effectiveness 6 5 6 4 18 4Learnability 0 10 5 17.5 10 15Satisfaction 22.5 55 12.5 65 10 67.5N 5 4 4 3 9 3S 8 4 6 3 11 3R 4 3 4 3 3 3Lostness 0.361 0.063 0.18 0 0.23 0Errors 1 0 2 1 5 1Assistance Yes No Yes Yes Yes No
Table 8: Various usability measures for users 1, 2 and 3 from usability tests withold and new UIs.
63
11.1.2 Old and new interface scores comparison
Overall, the new UI did better on many of the usability metrics than the old UI.In terms of lostness, while a perfect lostness score would be 0, Smith [34] foundthat participants with a lostness score less than 0.4 did not exhibit any observablecharacteristics of being lost. However, she reported that participants with a lostnessscore greater than 0.5 did appear to be lost. From tables 7 and 8 we can see that forall participants the scores for lostness were higher for the old UI than the new UI. Infact, in 3 cases the new UI received the perfect lostness score. This is encouragingand meant that the new design once implemented would enable many users of thewebsite to find the information more easily and perform tasks more effectively.
Furthermore, participants made less errors with the new UI and needed lessassistance to do the tasks. The efficiency and satisfaction scores for the new UIwere also higher than for the old UI. In addition to this, we can see that the newUI scored better than the old one on learnability metrics.
11.2 Feedback collection results
The people I held meetings with and interviewed to collect feedback shared theiropinions both on the design and also the process of it. Many of them were not aroundwhen the old website was built as it was almost 15 years ago. Thus, unfortunately,I was not able to compare their feelings about the previous development process tothe feelings about the current one. This was with the exception of two people whowere the leaders of the IT-CDA department.
Based on their comments, the old website was built solely based on the technicalrequirements and the decisions of the developer with no users involved. They saidthat they were much happier with both the development process which followed thehuman centered design approach and the outputs that came out of it. They alsofelt that thanks to all the learnings throughout the project, it brought them closerto the end users of the IT services they provide.
Finally, the people who did not know much about the development process of theprevious website but were active users of it have evaluated the new design positivelyas well.
65
12 Discussion and ConclusionOverall, using a human centered process for the redesign of the old Windows serviceswebsite led to a better, more intuitive end-product to the user as demonstrated bythe evaluation results. This is the goal of any design process [57].
There were substantial improvements in the representation of the varied audienceof the website in the final design output. It took into account their different technicalskills levels as well as the relevance of difference tools for them available on thewebsite. This resulted in improvements in the performance during user tests, and ahigher internal design satisfaction.
Through the process of redesigning this website, this thesis tried to answer thefollowing research questions:
1. How to create a website containing technical information and toolsfor Windows computer management and troubleshooting reflectingthe workflow and concept familiarity levels of different users?
2. How to measure and evaluate the impact of the redesign process ?
I discuss how these two questions were dealt with in the scope of this projectand how that relates back to the bigger picture. The conclusion for each question ispresented. I also present the limitations of the work and the areas for future work.
12.1 Designing tools for different workflows and levels of tech-nical awareness
Overall, this project benefited greatly from following the human centered designprocess in achieving its goals.
Dedicating enough time and effort to understand the different user groups provedto be very important for understanding what types of services on the website mightbe relevant for different users groups. In particular CERN’s IT-CDA departmenthad practically no real knowledge of who were the real users of the website. Theold website had been built to serve various user groups and let them take care oftheir computers autonomously. However, in reality, it was being used only by a fewpeople because of poor design and navigation.
Furthermore, this research has helped to uncover hidden stakeholders that werean important part of the Windows ecosystem at CERN. In particular, these werelocal supporters for some Windows users at CERN who had direct influence onusers’ habits with regards to Windows computers. They were also active users ofthe old website. Thus, I involved them in the design process to learn how the newwebsite could benefit both them and the users they were looking after.
In addition to this user research, organizational requirements have played animportant role too in making some decisions. One concrete example would be theBitlocker key retrieval functionality. The CERN department wanted users to becomemore independent in using it. On the old website this function was previously hidden
66
in computer account section making it hard to find. In the new design it receivedan independent page.
Further to this, usability testing has proved to be useful in identifying key prob-lems in the old UI and that has helped in making some of the decisions on how thetools should be displayed in the new UI. For instance, in the case of the Bitlockerkey, during the test of the old UI, after learning how to use the tool in the helpsection, the users struggled to return to the tool. This has influenced me to createa space for relevant featured help articles next to each tool in the new design.
Card sorting has helped me to understand how the different services could becategorized from different users perspectives. However, one known issue with cardsorting is that there may be multiple entry points and one item should not belongto only category only [45]. This has proved to be the case for this project too. Thismeans that this method works well together with other methods and one should notbe relying on card sorting only in the design process to make the decisions.
However, the card sorting has proved to be extremely useful in learning howvarious tools and help topics were perceived by different users. The cards served asprompts to get the users to open up on how relevant different services were to themas well as to learn more about their knowledge about those. Finally, this also hasgiven me some ideas on how to rename certain sections of the website. Therefore, Iused the card sorting to reorganize the information architecture from users’ point ofview. But I also used this method to study the workflow of the different users andtheir relationship with available tools. I would argue that this was a valuable wayof gaining insight about this for the project.
Overall, all these design methods I used in the human centered design processproved useful in different ways in informing the redesign of the Windows serviceswebsite. This answers the research question of this thesis - designing websites fortools and information for different users can be achieved by following the humancentered design method process, using design methods such as interviews, card sort-ing and usability testing to inform design decisions in the process. In particular cardsorting and interviews with user groups and internal stakeholders have been usefulfor making design decisions.
12.2 Evaluation methods for design process
Software engineering teams sometimes do not employ human centered design meth-ods because they fail to understand how to measure the outcome of using them.Also using these methods can take time and be expensive [13]. In this project byconducting two rounds of usability testing on the old and new UIs I was able tocollect and provide concrete quantitative metrics both designs could be comparedon. This way I could measure the outcome of the human centered design processthat this project followed.
In terms of the metrics chosen to do this, the classic ones - that is efficiency,effectiveness and satisfaction did a pretty good job. Therefore, if short on time and ifused solely for the purpose of comparing different design outputs, these metrics couldbe sufficient and some other studies have also chosen to use them [59]. However, I
67
believe there is value in also collecting other metrics to compare the design outputson. In this case, lostness and learnability were also used. Lostness was relevantsince I was designing a website where creating good navigation was important, thismetric helped to show whether the new design made it easier for the users to findthe information they were looking for. The new UI in some cases got the lostnessscore 0 which is the ideal score according to Smith [34]. Yet it is also worth noting,that according to her, participants with a lostness score less than 0.4 do not exhibitany observable characteristics of being lost. However, this was not the case in theusability tests I conducted with the old UI. Many of participants reported feelinglost in the follow up interviews I conducted. I also observed them navigating backand forth before finding what they were looking for, yet, their lostness scores werenot greater than 0.4. Thus, perhaps, the boundaries for what are considered to beacceptable lostness scores should be revisited. Quantitative studies comparing thescores from a large variety of research projects that used the lostness metric couldbe useful in researching this further.
For learnability, originally, I wanted to use this metric since I was building toolsfor troubleshooting and I wanted to make these accessible and used by more users.Therefore, measuring how easy it would be to learn to use these tools made sense.On the other hand, some of the tools were used only occasionally to accomplishcertain tasks rather than being used for everyday purposes. This is because some ofthe use cases of the website were really specialized. Therefore, the users would notspend sufficient time on the website to learn how to work with these tools. Thus,perhaps for this type of website using the learnability metric was not that relevant.Nonetheless, it could be relevant for similar types of websites being built where thetools provided are being used more frequently.
Finally, feedback collection sessions could have been more useful for comparingthe design process effectiveness and outcomes if more people that took part in thedevelopment processes of both, the old and the new websites, were still at CERNand could be involved in these sessions. Thus, I would argue that because of the cir-cumstances, usability testing proved to be a more straightforward way of measuringthe outcome of the human centered design process when redesigning the Windowsservices website.
12.3 Limitations
The current study also has its limitations. One such limitation is the sample size ofthe usability testing. According to the cost-benefit analysis of user testing by NielsenNorman Group [24], testing five participants provides the optimal ratio and allowsto discover most of the errors. Later, with larger number of users tested, the findingsstart to get repetitive. However, in my case, since the project had pretty diverseuser groups, it may have benefited from a larger sample size. Nonetheless, I believeI was able to catch most of the usability errors. Since I was the only designer anddeveloper working on this project, a larger sample size would have required muchmore recruitment effort, testing and analysis time.
This was also one of the reasons why after some period of time I tested the same
68
users for the two rounds of usability tests I conducted for this project. It couldbe argued that results would be even more robust if different users were recruited.However, all the participants reported not being able to remember the structureof the site and how to accomplish task they had to do during the previous test.This is because they either used the website occasionally (basic users) or the taskwas a specialized and rare problem (advanced users). Therefore, I can say that thepossibility that the better results for the new UI were achieved due to participantslearning was accounted for at least to some extent. Furthermore, ideally, I shouldhave also tested a fully working new website too. However, this was not possibledue to time constraints for being able to write up this thesis in time.
Finally, this thesis demonstrates a clear difference between the performances ofthe new and the old designs thanks to various usability measures it employed. How-ever, it could be argued that these improvements may have also been accomplishedusing a much simpler process than this work did. In other words, it is hard todemonstrate how and why the methods used in each of the stages of the humancentered design process were important. To address this, in the write up of the workthat was done I explained how the findings of each of the stages contributed towardsthe next one and their importance. For example, researching the user groups in un-derstanding context of use stage has informed who should be tested in the usabilitytests and the card sorting task in the following, user requirements gathering phase ofthe work. The card sorting then helped to understand which of the services neededrenaming and gave me ideas how to organize the content for the new website designwhich was then produced in the design realization phase. Finally, the design whichwas produced as a result of the work in the previous stages demonstrated a betterperformance than the old UI. Nonetheless, it is difficult to prove a direct relationshipbetween the methods used in the design process and the design outcome which wasevaluated and compared with the old website.
12.4 Future work
Regarding the future work specifically for this project, to demonstrate the outcomeof the human centered design process this project has followed, once the new websiteis developed and launched an additional usability test could be conducted to comparethe performance between the old and the new websites. Furthermore, it would beuseful to observe how the new website evolves and is used after the launch to see ifbroadening access for more user groups through the new design was effective.
Other projects could try other methods to study the workflow of the users suchas the contextual inquiry where users are first asked a set of standard questions andthen observed and questioned while they work in their own environments [5]. Theycould, for instance, compare if it is more useful to use that method instead of usingcard sorting or interviews. How these methods could work in combination for similartype of projects could also be researched further. In addition to this, it may bebeneficial to experiment with collecting more varied metrics using usability tests tomeasure the design progress. Finally, as previously mentioned, further quantitativestudies could be useful in researching how to interpret the lostness scores. This could
69
provide useful insights on how to use and interpret this metric for other designersand developers.
71
Bibliography[1] Ali Mesbah. Software analysis for the web: Achievements and prospects. vol-
ume 5, pages 91–103. IEEE, 2016.
[2] Yeliz Yesilada, Giorgio Brajnik, Markel Vigo, and Simon Harper. Under-standing web accessibility and its drivers. In Proceedings of the InternationalCross-Disciplinary Conference on web accessibility, W4A ’12, pages 1–9. ACM,2012.
[3] ISO DIS. 9241-210: 2010. ergonomics of human system interaction-part 210:Human-centred design for interactive systems. International StandardizationOrganization (ISO). Switzerland, 2009.
[4] Roger R Hall. Prototyping for usability of new technology. International Journalof Human - Computer Studies, 55(4):485–501, 2001.
[5] Nielsen. Intranet Information Architecture (IA) Trends. NielsenNorman Group, 2014. https://www.nngroup.com/articles/intranet-information-architecture-ia/, Accessed on 2018-03-30.
[6] Michael Bernard. Constructing user-centered websites: The early design phasesof small to medium sites. Usability News, 2(1):Article–8, 2000.
[7] Card sorting prototyping-chapter 14. In Effective Prototyping for SoftwareMakers, pages 250–271. 2007.
[8] Donna Spencer. Card sorting: Designing usable categories. Rosenfeld Media,2009.
[9] Windows services help pages. https://espace.cern.ch/winservices-help/NICEEnvironment/Pages/Mandate.aspx. Accessed on 2018-03-30 .
[10] Cern service portal, configure windows 10 to be hardened. https://cern.service-now.com/service-portal/article.do?n=KB0004951. Accessed on2018-03-30 .
[11] Brian Desmond, Joe Richards, Robbie Allen, and Alistair G Lowe-Norris.Active Directory: Designing, Deploying, and Running Active Directory. "O’Reilly Media, Inc.", 2008.
[12] Cern service portal. https://cern.service-now.com/service-portal/function.do?name=active-directory. Accessed on 2018-03-30 .
[13] Jack Moffett. Bridging UX and Web Development: Better Results ThroughTeam Integration. Morgan Kaufmann, 2014.
72
[14] Zarla Ludin. On ethical problem solving in user-centered research: an analysis.In CHI’11 Extended Abstracts on Human Factors in Computing Systems, pages791–798. ACM, 2011.
[15] H Beyer and K Holtzblatt. Contextual design: Defining customer-centereddesign, 1998.
[16] Jonathan Earthy, Brian Sherwood Jones, and Nigel Bevan. The improvementof human-centred processes—facing the challenge and reaping the benefit ofiso 13407. International Journal of Human-Computer Studies, 55(4):553–585,2001.
[17] Suzanne Robertson and James Robertson. Mastering the requirements process:Getting requirements right. Addison-wesley, 2012.
[18] Carolyn Snyder. Paper prototyping: The fast and easy way to design and refineuser interfaces. Morgan Kaufmann, 2003.
[19] Martin Maguire. Methods to support human-centred design. Internationaljournal of human-computer studies, 55(4):587–634, 2001.
[20] Mark W Newman and James A Landay. Sitemaps, storyboards, and spec-ifications: a sketch of web site design practice. In Proceedings of the 3rdconference on Designing interactive systems: processes, practices, methods,and techniques, pages 263–274. ACM, 2000.
[21] Carmelo Ardito, Paolo Buono, Danilo Caivano, Maria Francesca Costabile,Rosa Lanzilotti, Anders Bruun, and Jan Stage. Usability evaluation: a surveyof software development organizations. In SEKE, pages 282–287, 2011.
[22] Jakob Otkjær Bak, Kim Nguyen, Peter Risgaard, and Jan Stage. Obstacles tousability evaluation in practice: a survey of software development organizations.In Proceedings of the 5th Nordic conference on Human-computer interaction:building bridges, pages 23–32. ACM, 2008.
[23] Morten Hertzum. Images of usability. Intl. Journal of Human–ComputerInteraction, 26(6):567–600, 2010.
[24] Nielsen. Usability 101: Introduction to Usability. NielsenNorman Group, 2012. https://www.nngroup.com/articles/usability-101-introduction-to-usability/, Accessed on 2018-03-30.
[25] Elif Kılıç Delice and Zülal Güngör. The usability analysis with heuristic eval-uation and analytic hierarchy process. International Journal of IndustrialErgonomics, 39(6):934–939, 2009.
[26] Péter Balázs Polgár. Using the cognitive walkthrough method in software pro-cess improvement. e-Informatica Software Engineering Journal, 9(1), 2015.
73
[27] Chauncey Wilson. User interface inspection methods: a user-centered designmethod. Newnes, 2013.
[28] Adrian Fernandez, Emilio Insfran, and Silvia Abrahão. Usability evaluationmethods for the web: A systematic mapping study. Information and SoftwareTechnology, 53(8):789–817, 2011.
[29] Wei-siong Tan, Dahai Liu, and Ram Bishu. Web evaluation: Heuristic evalua-tion vs. user testing. International Journal of Industrial Ergonomics, 39(4):621–627, 2009.
[30] Caio Marcio Silva, Vanessa Macedo, Rafaela Lemos, and Maria Lúcia LR Oki-moto. Evaluating quality and usability of the user interface: A practical studyon comparing methods with and without users. In International Conference ofDesign, User Experience, and Usability, pages 318–328. Springer, 2014.
[31] Jakob Nielsen. Reliability of severity estimates for usability problems found byheuristic evaluation. In Posters and short talks of the 1992 SIGCHI conferenceon Human factors in computing systems, pages 129–130. ACM, 1992.
[32] Jakob Nielsen. Usability metrics: Tracking interface improvements. IeeeSoftware, (6):12–13, 1996.
[33] William Albert and Thomas Tullis. Measuring the user experience: collecting,analyzing, and presenting usability metrics. Newnes, 2013.
[34] Pauline A Smith. Towards a practical measure of hypertext usability.Interacting with computers, 8(4):365–381, 1996.
[35] Danielle A Becker and Lauren Yannotta. Modeling a library web site re-design process: Developing a user-centered web site through usability testing.Information Technology and Libraries, 32(1):6–22, 2013.
[36] Kimberly D Pendell and Michael S Bowman. Usability study of a library’smobile website: An example from portland state university. Informationtechnology and libraries, 31(2):45–62, 2012.
[37] Louis Rosenfeld and Peter Morville. Information architecture for the world wideweb. " O’Reilly Media, Inc.", 2002.
[38] Richard Saul Wurman. Information architect. Information Architects, 1996.
[39] Roy T Fielding and Richard N Taylor. Principled design of the modern webarchitecture. ACM Transactions on Internet Technology (TOIT), 2(2):115–150,2002.
[40] Jean Anderson, F Fleak, Kathi Garrity, and Fred Drake. Integrating usabilitytechniques into software development. IEEE Software, 18(1):46–53, 2001.
74
[41] James Brucker. Playing with a bad deck: The caveats of card sorting as aweb site redesign tool. Journal of Hospital Librarianship, 10(1):41–53, January2010.
[42] Cheri Mullins. Responsive, mobile app, mobile first: untangling the ux designweb in practical experience. In Proceedings of the 33rd Annual InternationalConference on the Design of Communication, page 22. ACM, 2015.
[43] Krystal M Lewis and Peter Hepburn. Open card sorting and factor analysis: ausability case study. The electronic library, 28(3):401–416, 2010.
[44] Carole George. User-centred library websites: usability evaluation methods.Elsevier, 2008.
[45] Erica Olmsted-Hawala. Card sorting, information architecture and usability:Adding in our users’ perspective to re-design the census bureau web site. InInternational Professional Communication Conference, pages 27–33, 2006.
[46] Alan Bryman. Social research methods. Oxford university press, 2016.
[47] Steinar Kvale. Doing interviews. SAGE Publications Sage CA: Los Angeles,CA, 2008.
[48] Miranda G Capra. Factor analysis of card sort data: an alternative to hierar-chical cluster analysis. In Proceedings of the Human Factors and ErgonomicsSociety Annual Meeting, volume 49, pages 691–695. SAGE Publications SageCA: Los Angeles, CA, 2005.
[49] Jakob Nielsen. Card sorting: How many users totest, 2004. Disponible online a< http://www. nngroup.com/articles/card-sorting-how-many-users-to-test/>[Data de consulta: 06.06.2015], 2008.
[50] Dino Esposito. Programming Microsoft ASP .NET 4. Developer Reference.Pearson Education, 2011.
[51] Bill English. Microsoft Sharepoint Products and Technologies Resource Kit(Pro-resource Kit). WA, USA: Microsoft Press Redmond, 2004.
[52] Aaron Bangor, Philip T Kortum, and James T Miller. An empirical evaluationof the system usability scale. Intl. Journal of Human–Computer Interaction,24(6):574–594, 2008.
[53] Carol Righi, Janice James, Michael Beasley, Donald L Day, Jean E Fox, JenniferGieber, Chris Howe, and Laconya Ruby. Card sort analysis best practices.Journal of Usability Studies, 8(3):69–89, 2013.
[54] Jim Rudd, Ken Stern, and Scott Isensee. Low vs. high-fidelity prototypingdebate. interactions, 3(1):76–85, 1996.
75
[55] Eileen G Abels, Marilyn Domas White, and Karla Hahn. Identifying user-basedcriteria for web pages. Internet research, 7(4):252–262, 1997.
[56] Banu Inanc Uyan Dur. Data visualization and infographics in visual com-munication design education at the age of information. Journal of Arts andHumanities, 3(5):39–50, 2014.
[57] Karel Vredenburg, Ji-Ye Mao, Paul W Smith, and Tom Carey. A survey of user-centered design practice. In Proceedings of the SIGCHI conference on Humanfactors in computing systems, pages 471–478. ACM, 2002.
[58] Aaron Bangor, Philip Kortum, and James Miller. Determining what individualsus scores mean: Adding an adjective rating scale. Journal of usability studies,4(3):114–123, 2009.
[59] Daniel Cunliffe. Developing usable web sites–a review and model. InternetResearch, 10(4):295–308, 2000.
77
AppendicesA Semi-structured interview questions for staff and
organisational requirements specificationInterview questions for Windows services staff:
To identify user groups and stakeholders:
• Who would you say are the main different users groups at CERN using Win-dows services and the Windows services website?
• Can you tell me what you know about what kind of tasks and work do thesedifferent user groups you named have? independently
• Can you tell me more about specific functionality (X) on the website? Whoare the users and what is the intended purpose?
• Can you tell me about the different types of users that use Windows services?
• How are the user group 1 different from the user group 2 at CERN?
• How does the user group 1 uses Windows services listed on the website incomparison to user group 2?
To specify organizational requirements:
• How does the current problem solving with Windows computer as well as othercomputers work at CERN at the moment based on your experience?
• Can you tell me more about the role of CERN’s service desk and who helpsthe users?
• Until what extent are the users expected to set up their Windows computersand troubleshoot independently?
• What is the vision of Windows services at CERN?
• Is there anything else that is important for me to know with regards to or-ganisational requirements CERN has for Windows ecosystem? What shouldit look like?
• How independent would you like users of Windows to be in the future?
• What do you think is missing in the old website so that would be possible ?Why?
79
B Semi-structured interview questions for identi-fied user groups representatives
Interview questions for identified user groups:
• What type of computer operating system do you use at work?
• If you have a problem with your computer where do you look for help?
• How do you install the applications you need?
• Have you ever used NICE website, what for? Do you know of anyone who usesthese type of services on this website? Who? Tell me about them, what theydo they use these services for?
• Tell me about the time you used NICE website. What did you use it for?
• Can you show me how you used it? What was the process of finding the answerto the problem like?
• How was the experience, what were the good/bad points of it?
• What were the most confusing parts of the website?
• Can you tell me what you would use X and Y (or each?) category for? Whatkind of information do you think you will find there?
• Do you ever access this site content on mobile?
• What type of computing services ar CERN do you use most frequently/ Howand why?
• What is your experience with computers inside/outside work?
• How often do you submit tickets to the CERN service desk, what kind oftickets do they tend to be?
Questions for advanced users:
• Have you ever changed local computer administrators on your Windows ma-chine?
• Have you ever looked up computer account information through the windowsservices website? What kind of information did you look for?
• Have you ever needed to use Windows remote desktop service? How did youuse it?
• Did you ever need to install or reinstall windows or antivirus? How hard/easywas it to find documentation and follow the steps?
81
C Cards used in card sorting test
83
D Information and consent form for the card sort-ing task
Information Sheet
Investigation into windows services website use cases
You are invited to complete a card sorting experiment. The cards given to yourepresent various functions and content that are available on the windows serviceswebsite at CERN. You are invited to group them based on how you think theyshould be categorized. It does not matter if you have not used the website before,there are no wrong or right answers here. The findings will be used to inform thedevelopment of the new windows services website.
The sound and discussion during the test will be recorded. Pictures of yoursolution will also be taken. The testing will also be followed up by a short interview.Your identity will be kept anonymous and all recorded content will be deleted afterthe results have been analyzed.
If you have any concerns after the testing or you wish to get more informationabout the project, feel free to contact: Milda Norkute, [email protected]
Name
Date
Signature
85
E Follow up interview questions for the card sortingtask
• Could you explain me why you have introduced these categories? (for opencard sort and in case additional categories were introduced in the closed cardsort)
• Were there any cards you would have liked to place in more than one category?Why?
• Which one of those represented services familiar to you?
• Which one of those represented services you use often? Is it through Windowsservice website or somewhere else?
• Were there any cards presenting terms you have never heard about before?Which ones?
• Were there any terms that were hard to understand? Which ones and why?
• Were there any cards you have liked to have thinking of them as services on awebsite related to Windows computers? Why? And Which ones?
• Anything else you would like to share?
87
F Information and consent form for the usabilitytests
Information Sheet
Investigation into windows services website user experience
You are invited to complete a task using the Windows services website. The aim isto investigate how is the user experience when using it.
Your actions (but not face) will be recorded. The testing will also be followed upby a short interview. Your identity will be kept anonymous and all recorded contentwill be deleted after the results have been analyzed.
If you have any concerns after the testing or you wish to get more informationabout the project, feel free to contact: Milda Norkute, [email protected]
Name
Date
Signature
89
G Questions asked and during usability tests1) Demographic questions
• How old are you?
• How long have you worked at CERN?
• What is your computer operating system?
• What do your daily tasks involve?
• What is your experience with computers inside/outside work?
• What type of computing services ar CERN do you use most frequently/ Howand why?
2) Interview questions
• Do you ever access this site content on mobile?
• How was the experience of completing the task?
• What were the main pain points of the process? Why?
• Was there anything good about it? Why?
• What part of the site would you say was the most confusing? Why?
• Did you feel lost at any point? Why and Why not?
91
H System Usability Scale
93
I High fidelity prototype of new website screens
Figure 14: UI of the home screen
94
Figure 15: UI of the tools screen
95
Figure 16: UI of the help screen
96
Figure 17: UI of the Bitlocker retrieval page
97
Figure 18: UI of set local administrator page
98
Figure 19: UI of the recreate account page
99
Figure 20: UI of the local password retrieval page
100
Figure 21: UI of the key computer facts page
101
Figure 22: UI of the key computer facts page: extended view
TRITA EECS-EX-2018:562
www.kth.se