online user tracking and usability tools - a mapping study759122/fulltext01.pdf · tracking user...

Royal Institute of Technology & StockholmUniversity

Master thesis

Online user tracking and usability tools -a mapping study

Author:

Niklas Nystrom

Supervisors:

Linda Kann

Tanja Pelz-Wall

Examiner:

Olle Balter

5 oktober 2014

http://www.kth.se

http://www.kth.se

“Always try the problem that matters most to you.”

Andrew Wiles

Abstract

Tracking user activities online can be done in order to improve systems usability, un-

derstand user behaviour or to increase monetary profit, to mention a few things. The

software tools for this purpose can vary, for example in terms of usability of the tool

itself, features, security, price, accessibility and support options. In this study I have de-

veloped a methodology to evaluate a selection of modern tools and map them depending

on various properties. I have also analysed learning theories to gain an understanding of

how they impact on the educational challenges in how to use these tools. This is all done

within the context of public procurement. I conclude that learning theories can be used

to further understand the complexity of designing educational material for this type

of tools. Findings also show that tools designed for usability studies do not all follow

usability design guidelines, both in design of the tool itself and the design of educational

material. This study shows an approach to how to map the selected tools based on their

main area of use, and how to expand the mapping in the future.

Sammanfattning

Sparning av anvandares aktiviteter pa internet gors av manga skal. De mjukvaruverktyg

som anvands for dessa andamal varierar i termer av anvandbarhet, funktioner, sakerhet,

pris, tillganglighet och support. I denna studie har jag utvecklat en metodik for att

utvardera ett urval av dessa verktyg och kartlagga dem beroende pa olika egenskaper.

Jag har ocksa analyserat larandeteorier for att fa forstaelse for hur de paverkar de

pedagogiska utmaningarna i att lara sig anvanda dessa verktyg. Allt detta gors inom

ramen for offentlig upphandling. Jag drar slutsatsen att larandeteorier kan anvandas for

att ytterligare forsta komplexiteten i att utforma utbildningsmaterial for dessa typer av

verktyg. Resultaten visar ocksa att verktyg avsedda for anvandbarhetsstudier inte alla

foljer riktlinjerna for anvandbarhetsutformning, bade i design av sjalva verktyget och i

utformningen av laromedel. Denna studie visar en strategi for hur man ska kartlagga de

valda verktygen utifran deras huvudsakliga anvandningsomrade, och hur man kan utoka

kartlaggningen i framtiden.

Acknowledgements

Special thanks to my supervisors Linda Kann and Tanja Pelz-Wall. Your support has

been indispensable!

iv

Contents

Abstract ii

Sammanfattning iii

Acknowledgements iv

Contents v

List of figures vii

List of tables viii

List of code snippets ix

Abbreviations x

1 Introduction 1

1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.2 About public procurement . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.2.1 About Visma TendSign . . . . . . . . . . . . . . . . . . . . . . . . 2

1.3 User tracking and usability studies . . . . . . . . . . . . . . . . . . . . . . 3

1.4 Problem motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.5 Study objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.6 Needs and boundaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.6.1 Privacy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

1.6.2 Security and data management . . . . . . . . . . . . . . . . . . . . 6

1.6.3 Usability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

1.7 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2 Methodology 9

2.1 Qualitative and quantitative methodologies in general . . . . . . . . . . . 9

2.2 Customize . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.3 Adapt and augment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.3.1 Grounded theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.3.2 Expert testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

v

Contents vi

3 Evaluation of tools 15

3.1 smt2ε . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.1.1 Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.1.2 Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.1.3 Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3.2 UserZoom . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

3.2.1 Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

3.2.2 Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3.2.3 Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

3.3 Mixpanel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

3.3.1 Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

3.3.2 Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

3.3.3 Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

3.4 Loop11 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

3.4.1 Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

3.4.2 Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

3.4.3 Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

4 Educational challenges 32

4.1 Synchronous vs asynchronous communication . . . . . . . . . . . . . . . . 32

4.2 Video, text and images - a didactic design approach . . . . . . . . . . . . 34

4.2.1 Educational videos overview . . . . . . . . . . . . . . . . . . . . . . 38

5 Results 41

5.1 Technical aspects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

5.1.1 Cloud solution or local solution . . . . . . . . . . . . . . . . . . . . 42

5.2 Educational aspects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

5.3 Financial aspects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

6 Conclusion & Discussion 45

6.1 Technical aspects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

6.2 Educational aspects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

6.3 Financial aspects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

6.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

6.5 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

Bibliography 50

List of Figures

1.1 Visma TendSign. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.2 Proxy implementation model. . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.1 Adapted grounded theory. . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.1 smt2ε . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.2 smt2ε administration interface . . . . . . . . . . . . . . . . . . . . . . . . . 19

3.3 UserZoom interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

3.4 UserZoom task. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3.5 UserZoom clickstream . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

3.6 Mixpanel funnel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

3.7 Loop11 setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

3.8 Loop11 dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

4.1 Educational text design - traditional . . . . . . . . . . . . . . . . . . . . . 35

4.2 Educational text design - API . . . . . . . . . . . . . . . . . . . . . . . . . 36

4.3 Unwanted activity in UZ tutorial video . . . . . . . . . . . . . . . . . . . . 39

4.4 Another unwanted activity in UZ tutorial video . . . . . . . . . . . . . . . 39

5.1 Categorization graph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

vii

List of Tables

2.1 Grounded theory translation. . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.1 Description of each main feature in Mixpanel . . . . . . . . . . . . . . . . 25

4.1 Synchronous communication vs asynchronous communication. . . . . . . . 33

4.2 Guidelines for creating a tutorial video. . . . . . . . . . . . . . . . . . . . 37

viii

List of code snippets

3.1 smt2ε tracking using JavaScript . . . . . . . . . . . . . . . . . . . . . . . . 17

3.2 Database query in smt2ε with aliases. . . . . . . . . . . . . . . . . . . . . 17

3.3 Database query in smt2ε without aliases. . . . . . . . . . . . . . . . . . . . 18

3.4 Line causing a bug in smt2ε . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.5 Mixpanel tracking using JavaScript . . . . . . . . . . . . . . . . . . . . . . 25

ix

Abbreviations

API Application Programming Interface

HTML Hyper Text Markup Language

KTH Royal Institute of Technology

LOU Swedish Public Procurement Act

PHP Hypertext Preprocessor

SSL Secure Sockets Layer

SU Stockholm University

UEM User Evaluation Method

UI User Interface

URL Uniform Resource Locator

URUT Unmoderated Remote User Testing

UX User eXperience

UZ User Zoom

x

Chapter 1

Introduction

In this chapter I will introduce the background of usability testing and user tracking and

why this is an interesting topic. I will also review related work and finally justify why

automated user analysis is a useful technique to consider as a complement to traditional

user studies.

1.1 Background

There are many reasons why companies want to perform usability studies on a product

or in other ways track user behaviours within their products. Rubin and Chisnell [2008]

argues for five main benefits gained from usability testing:

• The improvement of future products.

• Reduce support costs.

• Positive customers are more likely to share their experience with others.

• Increase market share.

• Minimize release risks.

The conclusion is that usability and user tracking related information is important and as

corporations revolve around monetary profit it is crucial to gain an edge in this field. To

do this - they must get to know their users. It is important to realize that the perspective

1

Chapter 1. Introduction 2

of whose needs we address, will dramatically change the way to approach the subject.

As mentioned, corporation are interested in monetary profit while developers working

for companies might be interested in a completely different matter. They, e.g might

get satisfaction from delivering a product of high standards using modern technology.

Finally, the customers are interested in a usable product that satisfy their needs. So

depending on how we look upon the needs for user tracking and usability studies, the

perspective changes.

To conduct an evaluation of modern user tracking- and usability tools we must first

investigate the actual needs of such a study. Do we really need to investigate the matter

further? To answer this question we need to explore the current situation and previous

research. This will help us justify further investigation within the area.

1.2 About public procurement

Public procurement is an area within the trade of goods, services and works primarily

between central government, local governments and private companies. According to

Swedish and European legislation, contracts above certain threshold amounts require

advertisement on the national or European level. These thresholds vary from about

SEK 1.1 million for goods and services, up to SEK 45 million for works, depending

on contracting authority. To ensure fair use of public funds all public contracting au-

thorities in Sweden must follow the Swedish Public Procurement Act (LOU). Public

procurement requires public notices that a number of basic principles apply such as

non-discrimination, equal treatment, proportionality and transparency [Konkurrensver-

ket, 2014]. A procurement process can range from months to a couple of years.

1.2.1 About Visma TendSign

Visma TendSign (https://tendsign.com) is a tool used in the majority of Sweden’s

public procurement. Swedish government units create and publish their procurements

here, invite tenders from the market, evaluate and assign winners. The winner signs

agreements that can be published in agreement directories for each business.

https://tendsign.com


Figure 1.1: Visma TendSign default start page for procurers.

TendSign is a 100% web-based system that supports fully electronic procurement. Spec-

ifications are created electronically in the system, including support for automatic evalu-

ation models. Bidders are given electronic documentation to fill in online with validation

that all requirements are met and all relevant questions are answered. During tender

evaluation, the system automatically determines who the winner is. Agreements can

also be created automatically including support for electronic signing. Figure 1.1 shows

Visma TendSign default start page.

1.3 User tracking and usability studies

Traditionally, usability analysis is cumbersome, costly and time consuming [Atterer

et al., 2006, Ivory and Hearst, 2001, Leiva and Vivo, 2013, Nebeling et al., 2013]. It often

requires the use of labs and special equipment, thus separating users from their natural

environment. This can lead to false results [Lazar et al., 2010] when analysing users’

way of interacting with software compared with their everyday work, as it is infeasible

to reproduce this environment in laboratory settings [Leiva and Vivo, 2013]. It is also

worth to note that most free Web tools fail to provide accurate information regarding

the user’s interaction with the website [Poggi et al., 2013].


With the development of new Web standards (Web 2.0 and forth) as well as client-side

technologies we can see new possibilities within the domain of usability analysis. In-

creasing use of modern technologies gives us the ability to define new ways of automated

remote user tracking, that give advantages compared to classical user studies [Leiva and

Vivo, 2013]. With automatic tracking tools we can gather a large amount of data [Carta

et al., 2011] which gives us a more legitimate user sample. Dingli and Mifsud [2011] and

Ivory and Hearst [2001] also state many advantages with automatic usability testing of

which some include cost reduction, increased consistency and coverage, and increased

efficiency.

1.4 Problem motivation

Corporations abilities to analyse their users’ activities in Web applications is of utmost

importance in order to gain knowledge of how to improve their software [Fernandez

et al., 2011]. Visma Commerce AB needed to gain information regarding how their

users behaved within their e-solutions for public procurement, particularly in TendSign.

This information would serve as a basis to improve the user experience (UX) and to

optimize the application design.

As the area of usability testing and user tracking is big and because many frameworks

and cloud solutions have been developed it is useful to evaluate a modern set of tools

and evaluate and compare them with each other. The development in this field is rapid

and even though studies have been done within the field, even a few years absence of

updated studies can make huge difference in how modern tools work.

The context of this comparison is within public procurement and within the boundaries

mentioned below. Visma needs a tool that has the capability to analyse users in Web

applications in order to extract, summarize and visualize quantified data. It is also

required to be easy to use and implement in order to be cost effective and usable by

employees with a lower level of technical skill. While these are all Vismas requirements, it

is indeed an interesting topic for other organisations wanting to understand their online

users better as well. In fact, the topic is even interesting in a sociological perspective; it

shows, among other things, how easy it is to monitor the digital community - without

the common man knowing about it. Now, it is important to understand that while this


can be of great use for organisations, it can be a big problem regarding personal integrity

in our future society.

There are different systems in which to conduct user tracking and analysis, and these

systems vary in complexity, i.e some require programming skills, while others do not.

This could in practice mean that in order to implement a certain user study without

knowing which tool to use, and its capabilities, companies could end up with lack of

resources. Indeed, some tools require almost no technical skill, while others are inoper-

able without a software developer. This leads to educational challenges and the need to

investigate what needs to be incorporated by those who intend to use these tools. This

will be discussed in detail in chapter 4.

1.5 Study objectives

In this thesis I have studied research and educational challenges regarding automatic

user tracking and data collection, using modern usability tools. With restriction to the

domain of public procurement, the aim of this thesis is to investigate the following areas

and questions:

1. Investigate and map a subset of tools within the area of user tracking.

2. Make a security and privacy overview regarding each tool.

3. Make a brief financial overview.

4. What educational challenges lies within the area?

By investigating these topics we can further categorize user tracking tools based on

their properties. We can also further understand the complexity behind the educational

aspects in designing learning material for this sort of tools.

1.6 Needs and boundaries

In this section I will discuss requirements and needs from Visma in contrast with the

rules and laws of public procurement.


1.6.1 Privacy

Public procurement is a field with comprehensive privacy requirements, so this is both a

need and a restriction for the tracking tool to fulfil, as well as being a requirement from

LOU. Lazar et al. [2010] suggests that a website should include clear policies regarding

what information is being collected, alternatively allowing users to opt in on whether or

not to allow data collection [Atterer et al., 2006]. Nguyen-Tuong et al. 2005, Wall et al.

1996 and Martin et al. 2006 also suggest adding dynamic information tainting [Kiciman

and Livshits, 2010] as this would prevent sensitive information from being reported.

Privacy becomes an issue in order to maintain full transparency. If an end user has

to agree on some security policy before using a specific Web application it might affect

the users behaviour [Leiva and Vivo, 2013]. There is thus a balance between respecting

personal integrity and business secrets and gaining as much genuine value as possible

from your collected data.

1.6.2 Security and data management

As the information regarding procurements is sensitive while within a process, Visma

also has high security requirements issued through LOU. This in turn places demands

on data management to be safe and stored by trusted partners only. The company has

previously denied the use of Google Analytics due to the security issues of storing highly

sensitive information on external servers. However this decision was made several years

ago, and today Visma has migrated more of their work flow to the cloud. In general,

cloud-based solutions have grown increasingly in the last couple of years, and there have

been increased security concerns from the community [Roberts and Al-Hamdani, 2011].

This clearly motives taking a second look at cloud solutions - regardless of previous

decisions.

1.6.3 Usability

In order for the UX department to be able to use the product it is desirable if it has

a low technical learning curve. It is understandable if the product is required to be

implemented by a software developer at first, however it should not require developer-

like skills to be used on a daily basis.


1.7 Related work

Research by Fernandez et al. [2011] presents a mapping study of user evaluations meth-

ods (UEMs) and claims the need to integrate user evaluation in all phases of Web

application development. The study shows that most UEMs were implemented towards

the end of the application development cycle, which renders them unable to adopt the

implementation in early stages. The authors also find that the studies (206 in total)

they selected give little or no direction in how to handle usability issues, but merely

identify and list them.

A study presented by [Ivory and Hearst, 2001] discusses the use of automated usability

evaluation methods for both Web UI’s and WIMP (windows, icons, mouse, pointer) UI’s.

They conclude that automated usability testing is indeed of good value and provides with

relevant data to help designers improve their software. However - this study is published

in 2001, making most of the tools obsolete and thus motivating the need of an updated

study with modern tools.

In order to understand code-level behaviour of Web applications Kiciman and Livshits

[2010] present AjaxScope. This tool allows for in-depth analysis from error reporting to

memory leak detection. While this low level analysis is not in focus for this paper it can

still provide useful information about application performance and thus ultimately UX.

Figure 1.2: Theoretical model for proxy implementation.

Another popular method to track end-users is to let traffic go through a proxy server

before leaving or entering the end-users machine. The proxy can annotate accessed Web

pages with JavaScript by modifying the HTML before sending it back to the client.


This has been done in Carta et al. [2011] or similarly in Arroyo et al. [2006] and [Atterer

et al., 2006]. Once the JavaScript is in place it provides numerous data that can be of

interest while tracking user activity. Figure 1.2 shows a theoretical implementation of

the proxy approach.

Chapter 2

Methodology

In this chapter I will describe the methodologies with which the comparison was con-

ducted. I will examine implementation details as well as methods for analysing the use

of these tools. Each tool will also be evaluated based on learning theories to gain an

understanding of the educational challenges each tool brings. These findings, regarding

educational challenges, will be presented later, in chapter 4. The reason for a later

discussion is because challenges lie within information gained from each evaluation, and

the reader needs to be familiar with each tool prior to discussion about the educational

challenges they bring.

2.1 Qualitative and quantitative methodologies in general

Depending on what kind of information we are interested in, the choice of method

reflects what we can extract from a study. However, it is important to understand that

qualitative and quantitative methods advantageously work together [Holme and Solvang,

1997]. It is a matter of combining the two approaches with the correct parameters. Even

though Holme and Solvang [1997] discuss qualitative and quantitative methods within

social- and behavioural science education, these methodology still applies to the current

subject.

Quantitative methods can be used to make a statement from a selection of a group. It

can extract a cross section from the phenomenon we are investigating and detect connec-

tions and extents of phenomenons [Holme and Solvang, 1997]. When using quantitative

9

Chapter 2. Methodology 10

methods the researcher is interested in a fair reflection of variations in data. This will

give a more structured approach of the situation compared to qualitative studies.

Qualitative methods focus to capture individual aspects of a certain unit. This means

that each unit’s condition will be considered and the researcher needs to adapt to each

of them. It is clear that with this approach, all details regarding an investigation are

set up in advance. In this study however, a global scope is defined which will guide the

rest of the study of each individual tool. So without knowing the exact features of each

tool it is still valid to qualitatively examine each.

Grønmo [1982] in Holme and Solvang [1997] presents four strategies for combining qual-

itative and quantitative approaches where one is of particular interest:

Qualitative studies can act as a preperation to quantitative. The qualita-

tive part is about understanding, where we in the best possible way prepare

for the actual investigation. In this way we establish an empirical ground

from which we can construct better tools for our main investigation. The

qualitative part thus gains the characteristics of a preliminary investigation.

The following sections in this chapter describe smaller qualitative steps towards building

a mapping of tested tools. The process as a whole has quantitative influences since it is

indeed characterized by distance and selection [Holme and Solvang, 1997]. However as

discussed above, the individual steps are truly of qualitative nature.

2.2 Customize

While there are methods and research on how to implement usability studies and user

tests of various kinds, I have yet to find any specific research regarding evaluation and

comparison of the actual tools used in these tests. According to Dillon, 2001 in Celik

[2012], user based evaluations are considered to give the most valid results when evalu-

ating usability of software. However, due to lack of resources this option is not possible

in this study and we will instead rely on an expert-based evaluation.

Some of these tools might have similar capabilities in general (i.e they can perform mouse

tracking), but it is hardly likely for them to share UI-based similarities such as work


flow. Differences in UI implementation will impact on the usability of the tool itself,

thus making this an interesting factor for the people using it. It seems logical to base

this comparative study on the needs presented in section 1.6 and base the result on an

expert review. Based on previous research I will adapt and augment methods to best fit

the goals of this study.

2.3 Adapt and augment

As suggested by Kobsa et al. [2009], results from comparative usability studies are

improved by normalizing test conditions. In some cases this might be hard to achieve

based on the supported metrics given by each tool. However, we can define a set of base

patterns and paths that we can follow in TendSign while testing each individual tool.

This will lead to a more uniform set of data, thus making it easier to draw conclusions.

A few definitions are required to make it easier to keep track of involved individuals:

• Administrator - this is a person who administer tools and tests.

• Coder - If the tool requires programming skills, beyond trivial scenarios, the coder

will be responsible for writing the necessary code.

• End-user - a user who interacts with the web solution that is undergoing tests.

A typical administrator can be a member from the following groups within a corporation:

UX, market or sales. It is natural that administrators and coders work close together,

in order to simplify work processes.

2.3.1 Grounded theory

Grounded theory, proposed by Glaser and Strauss in 1967, describes a method for quali-

tative research based on data rather than a hypothesis [Lazar et al., 2010]. It is suggested

to keep an open mind and urge creativity in order to succeed. Grounded theory, unlike

traditional experimental research, is based on data to formulate a theory instead of a

hypothesis. The method is generally based on four steps which I will adapt and redefine

to a more appropriate terminology compared to the description in Lazar et al. [2010].


Original terminology Translation Description

Open coding Exploration Explore and analyse the tools andidentify interesting features such asmouse tracking, click paths etc.

Development of concepts Feature-wisegrouping

Each tool consists of a set of fea-tures that will define it’s main areaof use. Tools can be grouped uponthese features.

Grouping concepts intocategories

Domain-wisegrouping

Each category will describe in whichhigher level category the tool be-longs, whether it is a cloud solutionor a local solution. Two other do-mains for categorization are analyt-ics and usability.

Formation of a theory Recommendation In this study the theory will be re-placed by a set of recommendationsbased on the outcome of each eval-uation.

Table 2.1: A custom approach of grounded theory.

Table 2.1 describes the mapping with adapted terminology. The new translation will

better suit the needs of this study.

Since this study only includes four tools, it is arguable that grouping is excessive and

not necessarily provide anything useful. However, future work will hopefully include

additional tools and these definitions might come in handy. Based on this, I will concen-

trate on exploration and recommendations. A similar approach with the use of grounded

theory was used by Bang et al. [2013].

Based on my translation, figure 2.1 shows the adapted model of this theory.

Figure 2.1: Model of the adapted version of grounded theory.

2.3.2 Expert testing

There are times when it is hard to gather large user groups to evaluate computer sys-

tems. It is both costly and time consuming to perform such tests. Expert-based tests

are performed by users who are experts within the domain of use. This method can


and should be used in conjunction with user-based tests at early stages within software

development [Lazar et al., 2010]. However, in this case, it is justified to perform an

expert-based review on each tool of interest to expose things like usability, and perfor-

mance issues. While, as mentioned, expert reviewing should be performed by experts

this is also a matter of resources and in this study I will perform the review myself. This

is the only option since there are no other experts available with knowledge in this area.

According to Shneiderman [1986] there are eight golden rules of interface design. Out

of these eight I will include the following in my evaluation:

1. Strive for consistency

2. Offer informative feedback

3. Prevent errors

4. Permit easy reversal of actions

5. Reduce short-term memory load

Rule strive for consistency is included because it is the most violated [Shneiderman,

1986]. Rule offer informative feedback is included because it speeds up daily work. If

users of a system do not receive feedback, much time is spent figuring out whether or

not the system responded. Rule prevent errors is included since it is essential for using

a tool of any kind. Without it, users of the system can make serious errors and may

cause the system to stop functioning. Permit easy reversal of actions is included due to

the fact that many of these tools will eventually handle informations collected during

long periods. Ensuring users of these tools the ability to reverse actions will result in

reduced anxiety, especially during the learning phase of each tool. This rule will also

encourage exploration of new features in future releases. Reduce short-term memory load

is included since it is required in order to simplify everyday use. Reducing heavy memory

processing done by the end-user also reduces the time it takes to complete tasks. This

leaves the following rules excluded from this study: cater to universal usability, design

dialogs to yield closure and support internal locus of control.

Cater to universal usability was excluded because the target group of the tools in this

study is limited to UX employees at Visma, and thus does not apply to the conditions of


this rule. Rule design dialogs to yield closure was excluded since there is no guarantee

that tools have a flow-based usage as stated by this rule. Rule support internal locus of

control was excluded because it is partly included in the rule offer informative feedback.

Chapter 3

Evaluation of tools

As described briefly in section 1.6 there are requirements from Visma that the selected

tools and techniques need to fulfil in order to qualify for further usage. This will impact

on the comparison of tools of interest. The tools were selected based on discussions

with usability expert Tom Airaksinen from the Visma Commerce UX-team, search on

Google and inventory of scientific articles in the area. The selection is believed to

represent modern, up to date tools from both commercial industry as well as open-

source solutions and tools from the research community. In the following sections I will

present the selected tools along with each step of the evaluation. Each tool will be

evaluated regarding setup, execution and usability. I will also provide an assessment

section for each tool. Setup describes the installation process, technical challenges and

requirements. Execution describes how well the tool performs its designated tasks, and

how it can be customized. Assessment summarizes each evaluation by weighing together

all the steps and will serve as a basis for final recommendations. Learning theories and

educational challenges will be discussed in chapter 4. The selected tools are smt2ε,

UserZoom, Mixpanel and Loop11.

As development tool, I use Visual Studio 2012 Premium whenever coding is needed.

However, for smaller tasks, any text editor with syntax highlighting will suffice such as

Notepad++.

15

Chapter 3. Evaluation of tools 16

3.1 smt2ε

smt2ε is an open-source tool developed by Leiva and Vivo [2013], where smt stands for

’simple mouse tracking’. Some features include unobtrusive mouse activity tracking, real

time playback, instant logging and visualization using ActionScript. It is built on PHP

and JavaScript. Figure 3.1 shows the smt2ε software architecture. While smt2ε can

be completely transparent, similar to Mixpanel, it allows the user to setup notifications

telling the user it’s being tracked.

Figure 3.1: smt2ε architecture. When a user (1) visits a monitored Web page,the server (2) adds the mouse tracking tool in background. Data are sent asyn-chronously back to the server (3) where they are processed and stored (4). A copyof the browsed page is cached (5) as well as the tracking data (6) for later studies.Source: http://smt.speedzinemedia.com/architecture.php. Image by Luis Leiva. Icons

by crystalXP.

3.1.1 Setup

Getting this tool to work is not a trivial task and requires a great deal of technical skill.

It requires a web server, a MySQL database and PHP5 or above - something users of

smt2ε will have to install themselves. The lack of proper and up to date documentation

contributes to the complexity of the installation process. For the environment I used

XAMPP (https://www.apachefriends.org/) which contains all compontents needed.

https://www.apachefriends.org/


Once the hosting environment is running, including smt2ε on your website is simple:

<script type="text/javascript"

src="http :// localhost :5850/ smt2/core/js/smt2e.min.js">

</script >

<script type="text/javascript">

try {

smt2.record ();

} catch (err) { }

</script >

Listing 3.1: smt2ε tracking using JavaScript

The tool comes with a web based administration interface allowing users of smt2ε to

access all recordings and logs. When replaying user interaction on a web site smt2ε

needs access to the original url, something TendSign does not support due to security

precautions. Luckily the tool caches HTML on the server and this may be used to access

the content directly. Since the code is open source, it is possible to change desired parts

of the code which in turn allows for solving a number of issues with the setup process.

To solve the url issue, all that was needed was to send the base-64 encoded ASCII string

based on the serialized HTML, then access this in the playback function.

Another issue was that the communication with the database was malfunctioning, caus-

ing the application to throw an error message instead of retrieving the individual logs.

“Error: User log #xx was not found on database.” was displayed where xx is the id of

the accessed log. This code originally lives in analyze.php:

$log = db_select($tables ,

$r.".* AS record ,

".$c.".* AS cache ,

".$b.". name AS browser ,

".$o.". name AS os",

TBL_PREFIX.TBL_RECORDS .".id = ’".$id."’");

Listing 3.2: Database query in smt2ε with aliases.

The above code use aliases which can cause problems on some servers. The exact details

regarding this issue is beyond the scope of this thesis. The reader is encouraged to visit

the XAMPP website (https://www.apachefriends.org/) for more details.

Removing the use of aliases (credits to Luis Leiva for tip) solved the issue. The above

code is changed to this:

https://www.apachefriends.org/


$log = db_select($tables ,

$r.".*, ".$c.".*,

".$b.".name , ".$o.". name",

$r.".id = ’".$id."’");

Listing 3.3: Database query in smt2ε without aliases.

When users navigate on tracked websites, smt2ε creates log entries in the database based

on a user id. This allows the tool to keep track of users visiting different links; the user

id connects user actions and generates a click path. Using browsers Chrome or Firefox,

a bug in smt2ε caused by cookie management causes every link visited to generate a

unique user id. This disables, among other things, the click path functionality because

the system can no longer keep track of the user. The bug is not present in Internet

Explorer 11, and was fixed by removing the following line from gateway.php:

$_POST[’client ’] = get_client_id ();

Listing 3.4: Line causing a bug in smt2ε

After removal smt2ε now handles creation of database entries correctly, thus making the

tool function work as intended. The author of smt2ε has updated the source code of the

framework based on findings during the setup phase.

3.1.2 Execution

Once installed smt2ε provides the ability to track end-user activities and save these in

the MySQL database. The tool provides a replay function that replays the user’s mouse

activity in real-time as well as details about each session. As shown in figure 3.2, smt2ε

also provides a list of every tracked session from every individual user along with filtering

options based on various parameters. Each log contains detailed information regarding

mouse movement, interacted HTML elements, interaction frequency and more. The

click path feature allows administrators to follow end-users paths throughout tracked

pages.

The real time playback offers useful information about actual end-user interaction. A

configuration menu can be accessed within the playback feature allowing users to change

what kind of metrics that will display in the video. Several options are available such


as setting visual layers, colors, coordinates, time chart views. For full detail about these

options, refer to [Leiva and Vivo, 2013]. The playback function works well for most of the

time, although there is a remarkable delay at the beginning and end of each recording.

This can be fixed, however the function’s potential is clear and spending time on this

bug is not justified at this point.

As an end-user, with the notification option disabled, user tracking is completely trans-

parent. Interaction with the website is unaffected by smt2ε, and the tool does not affect

performance notably.

Figure 3.2: smt2ε administration interface.

3.1.3 Assessment

Based on findings during the exploratory phase, smt2ε contains many interesting features

as mentioned above. The tool is open-source and belongs in the the domains of local

solutions and analytics. Feature-wise, the tool is categorized as mouse tracking tool.

According to my selected rules of interface design smt2ε performs acceptably. The tool

follows rule 1, 2 and 5 and violates rule 3 and partially rule 4. An example of a rule

3 violation is that administrators can enter illegal characters in the e-mail field when

adding new users. Although the illegal characters are not technically a violation of the

input field, they are clearly a violation of legal e-mail characters. While many actions

do comply with rule 4, actions such as deleting logs and database entries do not; most


database actions are in general permanent, so this is no surprise. Actions regarding the

user interface however are reversible.

The biggest strength of smt2ε is being open source. To understand why this is of

such importance, refer to section 3.1.1, the setup. Although it is arguable that errors

described here should not have been around in the first place, it still shows the power of

open source-code. If technical resources are available, i.e. developers, open source tools

can be used as a foundation of custom implementations.

The level of security regarding data from smt2ε is as safe as you make it. Since hosting

can be provided by anyone meeting the requirements, it is possible to handle all data

internally. Some companies and organisations handle data that are of such sensitive

nature that this is the only option.

Regarding end-user integrity it is up to the administrator to notify end-users that they

are being tracked. In fact, it is even illegal in some countries not to.

Regarding educational challenges, most lie in the setup process and I recommend it to

be handled by a developer or someone else with knowledge of PHP and JavaScript. It

is also useful to have knowledge of basic web server behaviour. The interface itself, the

part that is intended to be used by UX-staff is easy to understand, although it might

not be the prettiest in terms of design.

3.2 UserZoom

UserZoom (UZ) is a cloud based URUT tool providing numerous functions to perform

user surveys and tracking with features such as video recording, mouse tracking and

click paths. All studies in UZ are grouped within projects. A project contains everything

about a specific study, including editing, recruitment, start, stop and so forth. The data

collected from a study can be visualized and summarized with tables, graphs and video

playback.

End-users interact with UZ through intercepted pop-up windows, asking them whether

or not to participate in the survey. If they accept, UZ offers various options to describe

the different tasks to be performed in the test. Tasks can vary from navigation to card


sorting and questionnaires. UZ also offers high data security standards using 128-bit

SSL encryption.

3.2.1 Setup

Most setup done in UZ is performed in a web interface, shown in figure 3.3. Tracking

and invitation triggers are enabled either through JavaScript or browser add-ons. The

add-ons support Internet Explorer, Firefox and Chrome. It is worth to note that in

order to use the video recording feature, the project requires the UZ browser add-

on. Using JavaScript, administrator or coders manually add the scripts to each page.

The projects can be configured in a number of ways and it is possible to add custom

HTML within tasks to further customize the look and feel of a project. UZ offers

several options to invite survey participants; invitation links can be sent by e-mail, a

survey can be triggered based on a visited URL or end-users can use a feedback tab to

participate. During and before setup, UZ offers help and support in various ways. There

are instructional videos, the option to submit tickets, live chat and a help center.

Figure 3.3: Using UserZooms interface to customize the look and feel of a project.

In order to restrict access to projects it is possible to include or exclude IP addresses.

Administrators can also restrict access for specific browsers. This is especially useful

during the development stage of a website, in order to test specific browser behaviour.


Tasks are easy to set up and complex logic and conditions can be added to control the

flow of the study. E.g a task can be included or excluded based on the answer from a

questionnaire from another task.

For websites with static URL address, UZ offers options to tag individual pages using

JavaScript which is especially useful when dealing with older websites that use frames.

Otherwise there is no way for UZ to keep track of the pages that are visited, and

click paths among other things become unavailable. In fact, UZ even provides tracking

of dynamic content using JavaScript. This enables administrators to see if end-users

interact with specific HTML elements within a page.

3.2.2 Execution

From an end-user perspective, UZ offers a pleasant user experience on tracked websites.

There is no notable performance decrease, however UZ launches its own browser window

in order to keep the end-user notified with information regarding tasks. This causes the

main browser window to shrink slightly, making room for the UZ task window. Figure

3.4 shows an end-users perspective during a UZ study.

Figure 3.4: An end-users view of interacting with UserZoom. Notice the secondwindow at the bottom describing the current task.


Before a task is accepted, UZ will notify the user and ask for permission prior to starting

data collection. Questionnaires are easy to follow and through the options of customiza-

tion during setup, designers can make these windows have the same feel as the rest of

the web page.

The administrator can access results from a study as soon as tracking has begun. The

project administration interface is solid and bug free, providing a rich user experience.

Project results are visualized using various chart types, and reveal detailed information

regarding end-users systems, task summaries, questionnaire answers and so forth. Click-

streams are available, and through generated graphs administrators can see statistics of

the different paths taken during the study, and also if the path lead to completing the

task or not. Figure 3.5 shows an example of a clickstream.

Video recording, although only available using UZ browser add-on, provide administra-

tors with the options to review user interaction in real time. The video playback is

accurate and the interface makes it easy to sort videos by various options. Recording

videos requires large disk storage, so it is fortunate that all data is stored at external

servers provided by UZ. If desired, administrators can also export data for local storage.

Figure 3.5: Example of a clickstream. The various percentage can be replaced byshowing the actual number taking a certain path.

3.2.3 Assessment

Having explored UZ it is clear that it offers many interesting and powerful features.

To summarize the most interesting, these are: mouse tracking with real time playback,

clickstream graphs, data visualization, statistics and strong security. Combined with


a simple web administration interface, easy setup and a rich end-user experience, UZ

proves to be a strong contestant among URUT tools.

TendSign is an old web application built on old technologies, without a well defined

architecture. There is a lot of code executing just by navigating, and this leads to

results in the clickstreams that are difficult to interpret, generated by UZ. Even if an

end-user only tries to navigate to a specific link, behind the scenes TendSign actually

accesses many different pages causing UZ to map them in the clickstreams. However,

this has nothing to do with UZ, but the tested application itself and the technologies

behind it.

Feature-wise it is fair to put UZ among URUT tools due to its task based work flow.

Domain-wise it clearly belongs to the category of cloud solutions, even thought it sup-

ports, and sometimes requires, JavaScript code to be inserted at pages.

As for educational challenges for UX employees, most lie in learning the setup process

and the interface itself. While UX staff might not have access to source code, the only

part where a developer or someone similar will have to be included is when including

JavaScripts in the code, or when attempting to do advanced tracking. The rest is handled

by the UZ web interface.

UZ follows all usability guidelines without remark.

3.3 Mixpanel

Mixpanel is an analytics platform for web and mobile applications. It is built on

JavaScript technology and is driven by events and properties. In Mixpanel, a collec-

tion of features are grouped together defining a main feature. These main features

including a description of each are presented in table 3.1. Mixpanel offers a cloud based

web administration interface where administrators can set up projects. This whole eval-

uation is based on analysing a web application, and therefore features regarding mobile

applications are excluded. All tracking will be done with JavaScript. All assumptions

and decisions made are based on these criteria.


Segmentation This feature allows administrators to filter and sort data. It offers

a visual way to write SQL-like queries, removing the need to write

code.

Funnels This feature allows administrators to visually measure how end-users

move through their system based on selected events. These can be

defined at any time, given that the events themselves were defined at

that point.

Retention With this feature, administrators can setup detailed views of recur-

ring events within applications. Usage can be grouped by end-users

that join at specific times, with specific conditions, and administra-

tors can filter groups by a series of actions made in the application,

i.e those who created an account and created a procurement.

People The people analytics feature can be used to sort end-users by different

criteria. All end-user actions can be viewed and it is possible to tie

revenues to each user. Communication between end-users can be set

up via messaging.

Notification Administrators can set up Mixpanel to engage specific users or user

groups based on predefined conditions. In practice this could be

asking end-users if they need help, or offering a whole group a special

deal of some sort.

Table 3.1: Description of each main feature in Mixpanel

3.3.1 Setup

As Mixpanel is built around events and properties, fundamental knowledge about JavaScript

and HTML are required in order to be able to do anything interesting. Projects are set

up and configured using the web interface, and the proper JavaScript code is generated

to enable Mixpanel’s interaction with the desired page. Once the JavaScript is included

in the page tracking can begin.

mixpanel.track(" createdProcurement", {" Municipality" : "Ulricehamn",

"AccountType ": "Basic "});

mixpanel.track_links ("# myProcurements", "Clicked ’My Procurements ’ link");

Listing 3.5: Mixpanel tracking using JavaScript


This code will track an event called ’createdProcurement’ and include data about the

municipality and account type. This will be sent to Mixpanels web interface and can be

viewed there. Typically, data is not hard coded, but instead retrieved from a real data

source, e.g using AJAX (Asynchronous JavaScript and XML) technology.

Figure 3.6: Example of a simple funnel using Mixpanels web interface. It was gener-ated using events from code snippet 3.5. The diagram shows how many users created

a procurement and then went to the ’my procurement’ page.

Shortly after registration, Mixpanel sends an e-mail to new customers asking them if

they would like to participate in a call. The call offers an opportunity to ask for help or

discuss Mixpanel in general.

3.3.2 Execution

Mixpanel gathers data and can be completely transparent to end-users, similar to smt2ε.

In the same manner, end-user experience is not affected by the tracking framework.


Working with the Mixpanel web interface is a pleasant experience. The UI is clean

and simple to follow, and when visiting a new project for the first time, tutorials and

help sections are presented automatically. The learning sections are short and concise,

speeding up the start-up process for new users. There is also a help section that provides

developer-documentation, common questions and a collection of how-to-videos. The UI

clearly indicates what elements can be interacted with, thus providing a fluent and solid

experience overall.

As stated in section 3.3.1 JavaScript and HTML knowledge is required and in the devel-

oper documentation section developers can access the full API (Application Program-

ming Interface) reference. Some examples are included which can be of value to less

experienced developers.

3.3.3 Assessment

The true strength of Mixpanel is that it combines an elegant administrative cloud solu-

tion with a powerful and flexible client-side framework. The fact that it is based around

events and properties allows for detailed tracking of most scenarios. Even though Mix-

panel is not classified as a usability tool primary, it can certainly be used to enhance

user experience through the valuable data it can generate. One missed feature is video

recording. Video recording for behavioural analysis can be a valuable complement to vi-

sualizing events and properties. It is not trivial to categorize Mixpanel into a domain as

it consists of elements from both cloud solutions and local solutions. Thus, it is natural

to define it as a hybrid between the two. Feature-wise, Mixpanel belongs the category

of analytic tools.

As it is not likely that administrators will write code themselves, Mixpanel will require

more resources due to the need for coders to implement tracking. However, the web in-

terface offers a smooth and easy learning curve, and the help center contains all necessary

information.

During the use of Mixpanel, and specifically when certain events trigger, e.g when first

sending data from an application to Mixpanel, Mixpanel staff sends an e-mail to the

administrator. The e-mail notifies the administrator that Mixpanel staff has noticed the


activity and offers help and the opportunity to give feedback. The overall experience of

receiving e-mails like this is positive and encourages the administrator to ask for help.

All connections to the Mixpanel API is encrypted with SSL and data is stored redun-

dantly in a compressed binary format and this is fragmented over multiple clusters. Even

though the information itself is stored unencrypted the data structure would make it

virtually infeasible to extract information from it (source Mixpanel support).

Mixpanel follows all five usability guidelines resulting in a great user experience.

3.4 Loop11

Loop11 is a cloud based URUT tool and focuses on organisations with limited budget. It

offers features such as heatmaps and clickstreams, questionnaires, and custom themes.

Loop11 does not require any technical skills prior to get familiar with its own web

interface. The only JavaScript is provided when needed. This is if the owner of the

study decides to use pop-ups, asking end-users to participate. Loop11 also offers the

ability to evaluate remote websites, that is websites which the tester does not own.

3.4.1 Setup

Setting up projects in Loop11 is a quick process, of course depending on how many

questions and tasks that needs to be defined. Creating tasks and questions is easy and

the work flow is obvious. To set up a task, simply enter the task name, description, start-

and end URL. When evaluating, I used a local URL address and this caused Loop11

to warn me about it being invalid. Ignoring this warning proved to work out just fine.

Being an external web application this behaviour is understandable, yet not what was

expected. Local addresses are bound to be used for test scenarios and should not raise

a warning. Figure 3.7 shows a view from the setup process. If desired, it is possible to

customize the Loop11 appearance to better fit your website. This would be visible for

end-users who participate in your study.


Figure 3.7: Setting up Loop11 user test project.

3.4.2 Execution

Loop11 can be activated by navigating to a specific page, in which end-users will get a

pop-up window asking them to participate in the survey. The Loop11 toolbar appears

and can either be expanded or folded to compact mode. The toolbar contains task

information as well as the option to cancel or mark a task as completed. End-users will

manually have to mark tasks as complete as Loop11 does not offer any way to trigger

this by other events, e.g navigation and questions.

When evaluating Loop11 there was severe communication delay while browsing. The

cause of this is unknown, although an educated guess is that traffic needs to be routed

through Loop11 servers.

Results from studies are available instantly through the Loop11 project page which can

be useful if progress reports are of interest. It is cumbersome to perform studies on local


addresses, e.g for educational purposes, since Loop11 has poor support to track activity

within these addresses. This results in a loss of information regarding path analysis and

heatmaps - some of the core features of the tool.

3.4.3 Assessment

Loop11 consists of features such as path analysis, clickstreams, heatmaps and question-

naires. It is also possible to see some statistics regarding completion rate, as shown from

the Loop11 dashboard in figure 3.8. This includes data like rate of completion, time and

number of participants. It is also possible to export results to various file formats for

local storage. Feature-wise, Loop11 fits best among URUT tools and its domain is cloud

solutions.

Figure 3.8: Loop11 dashboard presenting statistics from a user survey.

The end-user experience is not what is expected from a commercial tool. The simple

toolbar contains the information needed to complete or abandon tasks. However, using


Loop11 to evaluate websites produces severe delays while browsing, thus affecting the

end-user experience significantly. This might have to do with local network setup or ISP

(Internet Service Provider) latency, but the problem remains even when testing against

several reliable websites. As soon as tracking was disabled the latency disappeared, thus

isolating it to the Loop11 service.

Loop11 offers support from a help center, containing an FAQ (Frequently Asked Ques-

tions). This covers most of the common question, however if something breaks within

the application or the need for help is urgent the only communication channel for help

is via a support form or e-mail. There is an option for marking the support form post

as ”urgent” speeding up the process. There is a section for educational videos although

it requires logged in users to log out and visit the resource page.

Another problem is that because TendSign uses frames, it is difficult to provide accurate

URL tracing, and this ultimately affects the overall experience. The survey pop-up

does not work as intended, resulting in end-users not being able to participate in the

survey. This is mainly a problem with TendSign and not Loop11, although one could

have wished for better support for frame-enabled websites, or alternative ways to work

around this.

Regarding usability guidelines, Loop11 only performs well enough on rule 5. Since it is

not possible to change anything in a user test once it has started, i.e pause and change,

this violates rule 4. The setup process violates rule 1, with inconsistent layout, menus

and buttons. The lack of feedback and inability to prevent errors when analysing (and

creating projects) results from a local address violates rule 2 and 3.

Chapter 4

Educational challenges

In this chapter I will discuss the educational challenges from various perspectives using

learning theories and models. Each subsection will present a specific challenge. Please

note that literature and references used in this section will often be directed towards

the relation between teachers and students. However, its is still justified to use the

same theories when describing learning situations within the corporate world. Indeed,

employees often take on the role of a student when learning from a colleague or someone

else with more knowledge in a specific field. In fact the roles of students and teachers

are interchangeable, and this is encouraged by employers in order to increase the spread

of knowledge. The interchange is natural in the exchange of knowledge; students can

become teachers and vice versa, depending on who is the most educated in a specific

area. Both text and video related scenarios will be discussed.

4.1 Synchronous vs asynchronous communication

Let us start by defining the terms naming this subsection and identify advantages and

disadvantages in using them. Synchronous, or direct communication is characterized by

occurring in real time. This could be communicating by phone (extreme long distant

calls might be an exception) or an instant chat over the internet. The key essence is

that it is bound by time. A historical note is that before internet, telecommunication

and radio, true synchronous communication was also bound to space. Benefits of us-

ing direct communication is discussed by Hrastinski et al. [2009] and there are several

32

Chapter 4. Educational challenges 33

recommendations regarding when to use this communication type. One main benefit is

that it motivates and enables discussion and thus is great to use when establishing new

contacts.

Asynchronous, or delayed communication is characterized by being separated in time

and space. It could be discussions over blog posts or communicating through forum

posts. This method, which is the most researched [Hrastinski et al., 2009], is more

suitable for deeper learning and for studying more complex questions. Indeed, more

time is given for reflection due to the nature of this communication type, a prerequisite

to solve more complex problems. Table 4.1 shows when, how and why along with an

example of when to use these two communication types (adapted from Hrastinski et al.

[2009]).

Synchronous communication Asynchronous communication

When? Getting to know support staff. Solv-

ing less complex problems regarding

setup or other basic usage.

Solving complex question or prob-

lem. Discussing code-related issues.

How? Phone calls, video calls, live chat. Forum posts, blogs and e-mail.

Why? Through direct responses it is easier

to establish contact. Less complex

problems are easier to understand,

thus making them more suitable for

direct communication. An example

question can be ”Where do I find the

task setup menu?”

Because more time is given to re-

flection and thought, complex prob-

lems are easier to address through

delayed communication. Code, for

example, can be difficult to discuss

directly in a chat or phone call.

Table 4.1: Synchronous communication vs. asynchronous communication: when, howand why.

Mixpanel offers synchronous communication in several ways: via webinar (a video con-

ference much like a seminar, but over the internet), regular call (over the internet)

and asynchronous via e-mail. UserZoom offers synchronous communication through a

live chat, and delayed using e-mail and forum support. Loop11 did not support any

synchronous methods; the only way to communicate was through e-mail and ticket sub-

missions. Regarding smt2ε, the only communication channel is via e-mail. It is worth


to note that since it is an open source project, the online community can be a great

resource for help.

Hrastinski et al. [2009] makes the distinction that direct communication relates to per-

sonal communication, and that delayed communication relates to educational communi-

cation. While this certainly seems reasonable, I argue that in order to learn a new tool,

a chat can act as a primary channel of educational communication. This is mainly due

to the fact that in order to get familiar with a tool, most of the actions are on a basic

level.

4.2 Video, text and images - a didactic design approach

The steps of designing content in order for others to understand it includes many steps

and decisions. It comes down to the choice of words, phrases, positioning, colouring and

so forth. Note that images can be included in a text and it is assumed, in this case, that

they on occasion are. Selander and Rostvall [2008] discuss the design of pedagogical

texts in depth and conclude that they depend on many factors in order to be successful.

One conclusion is that with words, its easy to define and with images its easy to see

the correlation between objects [Selander and Rostvall, 2008]. As a result of this, they

also conclude that text and images can enhance or undermine each other depending on

usage.

According to Selander and Rostvall [2008] it is up to the writers to decide what audience

they expect and adjust the text design based on this assumption. So depending on

who the reader is, texts might look totally different even though they represent the

same content basis. This explains why API documentation and explanatory texts look

different. While an API reference is often intended to be used as an encyclopaedia, a

tutorial text might not be. Figure 4.1 presents an example of a traditional educational

text layout, adapted from Selander and Rostvall [2008].

Figure 4.2 shows a typical example of an API structure. As mentioned above, an API

is mainly used as a resource to look up functionality and does not rely much on images.

It is interesting to note that all of the tools present much of their tutorial text in a

”Learning-by-doing” way, a concept introduced by John Dewey in the late 19th century.


Figure 4.1: Educational text with an image as center.

Often a scenario is described as a website link, e.g ”Creating your first survey”, and

is then followed by a step by step flow encouraging the reader to follow along and

work towards the goal. This technique is especially used in how-to sections, for obvious

reasons. Mixpanel, UZ and smt2ε all use the same approach, much like shown in figure

4.2 with the adding of pictures, while Loop11 has a plan questions and answer section.

To begin understanding why Mixpanel, UZ and smt2ε all use the same approach when

it comes to educational text design, we need to bring in a concept from Selander and

Rostvall [2008]. The textual meta function describes the fact that texts are coherent

with present textual practices within its own use. Different text practices represent

social conventions that state how a text is supposed to be, what it should contain and

what a reader can expect from it [Selander and Rostvall, 2008] (freely translated).

Both video and plain text is used to educate administrators in how to use each tool.

This leads us to discuss the differences between the two options. Text based tutorials

benefit from being easy to access, allow administrators to read at their own pace, and

are easy to overview [Van Der Meij and Van Der Meij, 2014]. Video on the other

hand takes advantage of both visual and auditive elements, thus providing multiple

modalities. According to Clark & Paivio, 1991 and Paivio, 1986 in Van Der Meij and

Van Der Meij [2014] these two modalities strengthen each other, thus providing an

advantage compared to traditional text. Using video tutorials also allow the users to


Figure 4.2: An example of how an API documentation can be structured. Sam-ple code from Mixpanel API documentation. Note that it is the structure that is of

importance, not the text itself.


follow along, and given that both narrator and learner have the same system setup,

the experience of the learning scenario can be almost identical to that of the narrators.

Based on the conclusion regarding the benefits of videos, according to previous research,

video will be in focus in this study. This decision is also based upon the fact that

today, in general, more people have access to high speed Internet, which enables them

to access video based content in a greater extent. The topic of video based tutorials is

also of special interest with the spreading of video related websites such as YouTube,

TwitchTV, TED and so forth.

Van Der Meij and Van Der Meij [2014] presents eight guidelines for the creation of

tutorial videos and they suit well for discussing educational challenges regarding videos

from each tool. Table 4.2 present these guidelines.

1: Provide easy access. Craft titles carefully, avoiding jargon.

2: Use animation with narra-

tion

Be faithful to the actual interface in the animation.

Use a spoken human voice for the narration.

Use a conversational style.

Action and voice must be synchronized.

3: Enable functional interac-

tivity

Pace the video carefully.

Enable user control.

4: Preview the task Sell the goal.

Introduce new content by showing its use in context.

5: Concentrate on giving pro-

cedural information

Include deliberate pauses to stimulate reflection.

6: Make tasks clear and sim-

ple

Follow the user’s mental plan in describing an action

sequence.

Emphasize the interconnection of user actions and sys-

tem reactions.

Use signalling techniques to guide attention.

7: Keep videos short

8: Strengthen demonstration

with practice

Table 4.2: Guidelines for creating a tutorial video.

These guidelines provide enough pointers to form an understanding of how pedagogical


the instructional videos are regarding the tools tested in this study. I will distinguish

between demonstrational videos and educational videos. A demonstrational video is used

merely for demonstration product features, and not to educate users. An educational

video is where the intention is to actually educate uses in how product features work,

and this type of video will be the focus in the following section.

4.2.1 Educational videos overview

This section will provide an overview of the educational video content of each tool and

how it performs.

UserZoom offers a variety of videos from Webinars, introduction videos as well as longer

in depth video for universities, describing the main features of UZ. The overall content

of the instructional video for universities provides a great first look at UZ and follow

guidelines 2, 3, 4, 5 and 6 for tutorial video design. The video is almost 50 minutes

long so it fails to satisfy guideline 7, and is not publicly available nor does it contain

indexing, thus violating guideline 1. The video does not promote viewers to actively

practice and this violates guideline 8. Although the video performs well in the aspect

of content and narration, some criticism is justified. An instructional video should be

designed with care and should not contain recording flaws such as unrelated activities

happening on the recording machine. Figure 4.3 and 4.4 show two frames from the video

where activities from the recording machine disturb the purpose of the video. Although

it might be a decoder specific issue, the right part of the video is clearly not presented

correctly.

On the other hand, during webinar videos it is understandable that technical issues

remain in videos after recording due to the fact that webinars are indeed live recorded

sessions with interaction from viewers. It is worth to notice that these webinars do not

only include UZ specific topics, but also general usability topics and guidelines. Also,

in respect to other introductory videos from UZ, they all perform well according to the

tutorial video design guidelines.

Loop11 provides excellent initial tutorial videos according to the guidelines in table 4.2.

Most follow guideline 1 to 6 with some exceptions for guideline 7 which is not always

followed. Rule 7 is not fulfilled due to the length of some clips. Some of these videos


Figure 4.3: Unwanted activity in a tutorial video. Activities such as performanceissues and Skype messages (visible later in the video) does not belong in a tutorial

video.

Figure 4.4: Another unwanted activity in a tutorial video. Session time out and othertechnical related issues should be removed from the video during editing.


are, like UZ videos, from webinars and will thus include interaction from participants

from each individual webinar.

Mixpanel videos follow tutorial video design guideline 1 to 8. The lengths vary from one

to a few minutes and they are structured in a clear way so viewers will have an easy way

finding their way around.

Being an open-source tool with limited resources and developed by a single person, it is

understandable why smt2ε does not provide any tutorial videos. A few demonstrational

videos are available giving a basic understanding of what smt2ε can be used for. Because

there are no educational videos available there is no way of evaluating this tool based

on the usability guidelines for tutorial videos.

Chapter 5

Results

In this chapter I will summarize the results from my assessment and present a mapping

graph of the evaluated tools. I will also include results regarding educational aspects.

5.1 Technical aspects

During my evaluation it became clear that tools can be categorized based on their main

features. To illustrate the results I will introduce a graph in which to place tools based

on their categorization. Figure 5.1 presents a graph illustrating the properties of each

of the evaluated tools. Each tool is represented by a circle, where the area corresponds

to the versatility of the tool, i.e a large circle represents a tool with a wide area of use.

To clarify the axes, the horizontal shows if a tool is more focused on analytics or usability

testing. Analytics means more focused on gathering data, and usability testing is more

focused on offering surveys that end-users can interact with. Analytic tools in this survey

offer more transparency than usability testing tools. Indeed, in order to conduct a user

survey based on tasks and questionnaires it is necessary to inform the users about it.

The vertical axis shows if the tool bases its workflow on a cloud solution or if it requires

programming in order to work. E.g, Mixpanel requires the use of its API but also offers

administrators to setup metrics etc. within their website. So this is a combination of a

local solution and a cloud solution. Smt2ε, while offering an administrative website, is

more of a stand-alone tool since everything is hosted manually, and requires use of its

API.

41

Chapter 5. Results 42

Figure 5.1: Each circle represents an evaluated tool. The circle areas correspond tohow broad a tool is. The horizontal axis represents how analytic-based or focused onusability testing a tool is. The vertical axis represents to what extent a tool is more

cloud-based or a local solution.

5.1.1 Cloud solution or local solution

One clear difference between cloud solutions and local solutions is that they differ in

the need for technical skill. Both UZ and Loop11 can be used basically without any

programming knowledge, except for knowing how to include a JavaScript on a page

which is a trivial task. Mixpanel and smt2ε however, require moderate to high technical

skill. Using an open source tool like smt2ε, turned out to require not only knowledge of

how to use the tool itself, but how to install it. This was far from trivial and to engage

in someone else’s code can be very time consuming.


Regarding time spent on tasks, there is a significant difference between using cloud

solutions and local solutions.

Expert testing has revealed possible flaws in the UI design within the tested tools ad-

ministration interface, based on usability guidelines defined by Shneiderman [1986].

5.2 Educational aspects

We have seen how the evaluated tools differ in terms of support options and the design

of tutorial videos. It was shown that in the case of UZ, even though the product

if profiled towards usability studies, it fails to deliver material according to usability

guidelines defined by van der Meij and van der Meij [2013]. Loop11 performed well

according to most usability guidelines and Mixpanel was the only tool to comply to

all. Regarding smt2ε, there was no educational video material to be found. However,

regarding educational aspects, we saw that Mixpanel, UZ and smt2ε used the same

approach when designing tutorial and educational sections based on text. They base

it on a flow where it is expected from the reader to go through each section from top

to bottom in order to follow the content, with the exception of the API documentation

which can serve as a pure dictionary. Loop11 uses a questions and answer style which

can be used to look up questions and related answers.

The difference in terms of communication between administrators and support staff are

varying. Synchronous and asynchronous communication should be used for different

purposes according to Hrastinski et al. [2009]. The optimal solution would be to offer

end-users and administrators the ability to use both methods, but in the evaluation it

was shown that only Mixpanel and UZ offer both methods.

These finding can be summarized to that even though companies who deliver tools and

products that are supposed to be used in order to improve systems usability, the related

material to these products can still fail in terms of usability. We have also seen that not

all tools adapt communication channels in order to address different situations. These

results will impact the educational challenge of learning these tools.


5.3 Financial aspects

To start with, the evaluated tools differ hugely in term of cost and price models. UZ

offers annual subscriptions starting at $25,000 per year for corporations. Mixpanel uses

a model where subscribers pay per triggered event. An example is a monthly cost of

$2,0000 for a maximum of 20,000,000 triggered events. Loop11 offers either a monthly

cost depending on the size of the business or to pay per use. An enterprise account costs

$825 per month compared to $350 per project. smt2ε is free of charge according to the

MIT Licence (http://opensource.org/licenses/mit-license.php).

This study also showed that the time spent upon setup and installation, and the amount

of technical skill required, differed between local solutions and cloud solutions. This will

impact on the final cost.

http://opensource.org/licenses/mit-license.php

Chapter 6

Conclusion & Discussion

In this chapter I will discuss the results of this study, both from a technical and an

educational perspective and formulate a conclusion. I will also bring up aspects regarding

future work and how to expand this study and/or investigate specific matters deeper.

6.1 Technical aspects

As we can see from the results in chapter 5 it is clear that in order to conduct successful

user tracking, usability studies or user studies, whatever the reason or intention might

be, selecting the correct tool is important. The different quadrants of figure 5.1 show

how this selection can be done, based on how organisations prefer to work and also based

upon available resources. Even though price and price models are subject to change it

is worth to note that cloud solutions that offer various services to support its users will

in many cases cost more than downloadable tools.

Organizational resources in terms of technically qualified personnel will also impact on

the choice of tool. Local tools, such as smt2ε, requires more technical expertise than e.g

a cloud solution like UserZoom, because of implementation details. Also, open-source

solutions obviously provide more options for customization and give more control of

what is going on within the application. This brings us to the security question.

Depending on what kind of information that is desired regarding end-users and how

this information needs to be collected, the choice of tool varies. A usability study

45

Chapter 6. Conclusion & Discussion 46

usually has the purpose of improving some part of a system, while user tracking might

have a completely different one. User tracking can be conducted in order to investigate

users’ general behaviour. In fact, transparent tracking is the only way to see how users

naturally act within a system. This study has shown that tools offer various ways of

collecting information, both transparent and through questions, task descriptions and

so forth. This study has also shown that it is possible to track users completely without

their consent, an action that can be illegal in some countries.

Can we meet the security and privacy requirements of public procurement? This, again,

depends on how secure the information needs to be. E.g, can a third party organisation

be trusted enough to allow them access to your tracked data? First off, companies that

want to use a tool must review the tool’s privacy policy and decide whether or not it

fulfils the desired requirements. If the policy fulfils requirements, the next step would be

to investigate technical security implementations, e.g encryption, firewalls and so forth.

When both of these security related questions have been dealt with, a fair decision can

be made. In the context of this study, the only option to obtain true control over traffic,

data and data storage security is to use a local tool, and customize it to meet the desired

needs. SSL is today one of the most popular solutions to secure Internet connections.

We have seen that some of the tools in this study use this technology and while this all

looks good on paper, it can still mean a security risk is imminent. Another concern is

how data is handled and stored. Using a cloud solution will require information to be

transmitted over the Internet and storing it externally. This might impact on security

requirements.

We have seen how grounded theory can be customized and applied to give us the data

needed to formulate these conclusions.

We have also seen that not even tools designed to evaluate usability always follow general

usability design guidelines defined by Shneiderman [1986].

6.2 Educational aspects

Educational challenges depend on the tool’s categorization. It is clear that companies

with more resources offer more ways to support their users. However this often means,

as mentioned earlier, higher costs of the tool itself. Open-source tools can provide


support through communities but it is highly unlikely to encounter live chat support or

video calls within this community. It is also shown that cloud solutions offered by large

companies can afford the development of more learning material, both text and video.

However, it is not certain that more is equals better. As shown in the case with UZ,

there can be flaws even in material provided by larger companies. For example, more

available resources given to the design of video tutorials does not mean they will follow

the guidelines presented by Van Der Meij and Van Der Meij [2014]. So depending on

what kind of help and support that is desired, the selection of tools should be made with

this in consideration.

Tools in this study offered different ways of communicating with their respective devel-

oper. As discussed in chapter 4, we can see that the theories developed by Hrastinski

et al. [2009] can be applied in the analysis of educational challenges. Companies use

different methods when communicating with their customers, which leads to different

experiences. My own experience is that depending on the situation, the ability to choose

form of communication can be very convenient. Also combining these forms of commu-

nication can be of great value. For example, discussing code can be cumbersome over

the phone or a webinar, so combining this with e-mail and blog posts is ideal. Both the

sender and receiver can initially take time to formulate thoughts regarding the problem,

which then can be discussed in real time, synchronously.

A difference between traditional educational material from the school world and online

tutorial material is the difference in design. By using the textual meta function from Se-

lander and Rostvall [2008], we find that the design differs from examples from traditional

learning material, and this is fine; it is supposed to. An educational text describing steps

to be taken in an application in order to perform an action is expected to do just that.

This is what is expected by the readers. They are not expecting a pedagogical attempt

to teach them how the underlying technology works. So it is indeed justified to design

tutorial texts in the manner found in this study.

6.3 Financial aspects

As we saw from the results, time spent on getting these tools up and running vary. Local

tools, foremost smt2ε had a steep learning curve, and required a few bug fixes in order


to work properly. Cloud tools on the other hand had a quick and smooth setup. Also

the technical skill required to operate these tools differ.

So, based on these findings we can conclude that they both impact on the final cost.

To start with, solving bugs takes time which leads to an increasing cost. A lack of

proper documentation can also lead to an increasing cost, since the people operating the

tool must figure out how it works on their own. Also, based on the finding regarding

synchronous and asynchronous communication, this will impact the cost. Offering ways

for administrators and coders to directly get in touch with support staff will reduce down

time when facing problems. Based on the finding regarding video tutorials and usability

guidelines, we can conclude that this too will impact the final cost. More thoroughly

planned educational material will bring administrators and coders up to speed quicker,

reducing the costs.

6.4 Summary

This thesis shows that it is possible to map usability and user tracking tools based on

their properties. The mapping is useful when deciding upon which tool to use. I have

shown how to adapt and customize existing theories in order to apply to the context of

this thesis. Secondly, I have made a security and privacy overview regarding each tool,

and of the subject in general. This showed that in order to gain a firm understanding

about security, one must further investigate the matter. The only option to gain full

control of data is to store it locally. Thirdly, I have made a brief financial overview

where I show that the time aspect impacts on the final cost. Finally, I have analysed

educational challenges within these tools. This analysis showed that challenges mainly

lie within communication and the design of tutorial video material, and this will impact

on how much resources the tools require.

6.5 Future work

For future work, it is desirable to expand this study and include more tools. This means

expanding the categorization from my adaptation of grounded theory, both feature-wise

and domain-wise to establish more definitions. Also, expanding this study would make it


possible to draw more general conclusions regarding the capabilities of local tools versus

cloud solutions. All this would certainly make the methodology used in this study more

sustainable, more valid and more acceptable on a larger scale.

Another approach would be to make a deeper investigation of security implementations of

tools in this area. As Fahl et al. [2012, 2013], Georgiev et al. [2012] show, there have been

issues with SSL and even though they are centred around non-browser applications, this

indeed justifies a study with security focus. Even though SSL is still the most popular

approach to secure connections over the Internet, it would be interesting to investigate

the benefits TLS (Transport Layer Security) has to offer. Not only are the technical

details of interest but also how information handling is defined in privacy agreements.

Regarding the future of investigating educational challenges, it is desired to expand the

overview of video and text and how these were used as tutorial resources. It would

be interesting with an in-depth study of videos in order to reveal even more challenges

that lie within the pedagogical area of video design. Future work could also include

deeper investigation of designing tutorials based on images and text, utilizing learning

theories similar to the ones presented by Selander and Rostvall [2008]. As shown in this

study, the only true local tool, smt2ε, did not offer much educational material. There is

reason to believe this has to do with resources regarding money and time given that it

was developed by a single individual. However, not all open-source tools are developed

by single individuals and by expanding research of open-source tools, more information

regarding this matter would hopefully be revealed.

It would also be interesting to investigate how monetary resources impact on support,

and more specifically if it affects the pedagogical quality of the educational material.

Bibliography

Ernesto Arroyo, Ted Selker, and Willy Wei. Usability tool for analysis of web designs

using mouse tracks. In CHI ’06 Extended Abstracts on Human Factors in Computing

Systems, CHI EA ’06, pages 484–489, New York, NY, USA, 2006. ACM. ISBN 1-

59593-298-4. doi: 10.1145/1125451.1125557. URL http://doi.acm.org/10.1145/

1125451.1125557.

Richard Atterer, Monika Wnuk, and Albrecht Schmidt. Knowing the user’s every move:

User activity tracking for website usability evaluation and implicit interaction. In

Proceedings of the 15th International Conference on World Wide Web, WWW ’06,

pages 203–212, New York, NY, USA, 2006. ACM. ISBN 1-59593-323-9. doi: 10.1145/

1135777.1135811. URL http://doi.acm.org/10.1145/1135777.1135811.

Soon K. Bang, Sam Chung, Young Choh, and Marc Dupuis. A grounded theory analysis

of modern web applications: Knowledge, skills, and abilities for devops. In Proceedings

of the 2Nd Annual Conference on Research in Information Technology, RIIT ’13,

pages 61–62, New York, NY, USA, 2013. ACM. ISBN 978-1-4503-2494-6. doi: 10.

1145/2512209.2512229. URL http://doi.acm.org/10.1145/2512209.2512229.

Tonio Carta, Fabio Paterno, and Vagner Figueredo de Santana. Web usability probe: A

tool for supporting remote usability evaluation of web sites. In Proceedings of the 13th

IFIP TC 13 International Conference on Human-computer Interaction - Volume Part

IV, INTERACT’11, pages 349–357, Berlin, Heidelberg, 2011. Springer-Verlag. ISBN

978-3-642-23767-6. URL http://dl.acm.org/citation.cfm?id=2042283.2042319.

Serkan Celik. Development of usability criteria for e-learning content develop-

ment software. Turkish Online Journal of Distance Education, 13(2):336 –

345, 2012. ISSN 1302-6488. URL https://ezp.sub.su.se/login?url=http:

50

http://doi.acm.org/10.1145/1125451.1125557

http://doi.acm.org/10.1145/1125451.1125557

http://doi.acm.org/10.1145/1135777.1135811

http://doi.acm.org/10.1145/2512209.2512229

http://dl.acm.org/citation.cfm?id=2042283.2042319

https://ezp.sub.su.se/login?url=http://search.ebscohost.com.ezp.sub.su.se/login.aspx?direct=true&db=eric&AN=EJ983666&site=eds-live&scope=site


Bibliography 51

//search.ebscohost.com.ezp.sub.su.se/login.aspx?direct=true&db=eric&

AN=EJ983666&site=eds-live&scope=site.

Alexiei Dingli and Justin Mifsud. Useful: A framework to mainstream web site usability

through automated evaluation. International Journal of Human Computer Interaction

(IJHCI), 2(1):10, 2011.

Sascha Fahl, Marian Harbach, Thomas Muders, Lars Baumgartner, Bernd Freisleben,

and Matthew Smith. Why eve and mallory love android: An analysis of android ssl

(in)security. In Proceedings of the 2012 ACM Conference on Computer and Commu-

nications Security, CCS ’12, pages 50–61, New York, NY, USA, 2012. ACM. ISBN

978-1-4503-1651-4. doi: 10.1145/2382196.2382205. URL http://doi.acm.org/10.

1145/2382196.2382205.

Sascha Fahl, Marian Harbach, Henning Perl, Markus Koetter, and Matthew Smith.

Rethinking ssl development in an appified world. In Proceedings of the 2013 ACM

SIGSAC Conference on Computer & Communications Security, CCS ’13, pages

49–60, New York, NY, USA, 2013. ACM. ISBN 978-1-4503-2477-9. doi: 10.1145/


Adrian Fernandez, Emilio Insfran, and Silvia Abrahao. Usability evaluation methods

for the web: A systematic mapping study. Information and Software Technology, 53

(8):789–817, 2011.

Martin Georgiev, Subodh Iyengar, Suman Jana, Rishita Anubhai, Dan Boneh, and

Vitaly Shmatikov. The most dangerous code in the world: Validating ssl certificates

in non-browser software. In Proceedings of the 2012 ACM Conference on Computer

and Communications Security, CCS ’12, pages 38–49, New York, NY, USA, 2012.

ACM. ISBN 978-1-4503-1651-4. doi: 10.1145/2382196.2382204. URL http://doi.

acm.org/10.1145/2382196.2382204.

I.M. Holme and B.K. Solvang. Forskningsmetodik: om kvalitativa och kvantitativa

metoder. Studentlitteratur, 1997. ISBN 9789144002118. URL http://books.google.

se/books?id=tJ6YHAAACAAJ.

Stefan Hrastinski, Mats Deutschmann, and David Richardson. Framatblick. In

Natbaserad utbildning : En introduktion, pages 117–121. 1 edition, 2009.




http://doi.acm.org/10.1145/2382196.2382205

http://doi.acm.org/10.1145/2382196.2382205

http://doi.acm.org/10.1145/2508859.2516655

http://doi.acm.org/10.1145/2382196.2382204

http://doi.acm.org/10.1145/2382196.2382204

http://books.google.se/books?id=tJ6YHAAACAAJ

http://books.google.se/books?id=tJ6YHAAACAAJ

Bibliography 52

Melody Y. Ivory and Marti A Hearst. The state of the art in automating usability

evaluation of user interfaces. ACM Comput. Surv., 33(4):470–516, December 2001.

ISSN 0360-0300. doi: 10.1145/503112.503114. URL http://doi.acm.org/10.1145/

503112.503114.

Emre Kiciman and Benjamin Livshits. Ajaxscope: A platform for remotely monitoring

the client-side behavior of web 2.0 applications. ACM Trans. Web, 4(4):13:1–13:52,

September 2010. ISSN 1559-1131. doi: 10.1145/1841909.1841910. URL http://doi.

acm.org/10.1145/1841909.1841910.

Alfred Kobsa, Rahim Sonawalla, Gene Tsudik, Ersin Uzun, and Yang Wang. Serial hook-

ups: A comparative usability study of secure device pairing methods. In Proceedings

of the 5th Symposium on Usable Privacy and Security, SOUPS ’09, pages 10:1–10:12,

New York, NY, USA, 2009. ACM. ISBN 978-1-60558-736-3. doi: 10.1145/1572532.

1572546. URL http://doi.acm.org/10.1145/1572532.1572546.

Konkurrensverket. Om offentlig upphandling, June 2014. URL http://www.

konkurrensverket.se/t/Page____6735.aspx.

Jonathan Lazar, Jinjuan Heidi Feng, and Harry Hochheiser. Research Methods

in Human-Computer Interaction. Wiley Publishing, 2010. ISBN 0470723378,

9780470723371.

Luis A. Leiva and Roberto Vivo. Web browsing behavior analysis and interactive hy-

pervideo. ACM Trans. Web, 7(4):20:1–20:28, November 2013. ISSN 1559-1131. doi:

10.1145/2529995.2529996. URL http://doi.acm.org/10.1145/2529995.2529996.

Michael Nebeling, Maximilian Speicher, and Moira C. Norrie. Crowdstudy: General

toolkit for crowdsourced evaluation of web interfaces. In Proceedings of the 5th ACM

SIGCHI Symposium on Engineering Interactive Computing Systems, EICS ’13, pages

255–264, New York, NY, USA, 2013. ACM. ISBN 978-1-4503-2138-9. doi: 10.1145/


Nicolas Poggi, Vinod Muthusamy, David Carrera, and Rania Khalaf. Business process

mining from e-commerce web logs. In Business Process Management, pages 65–80.

Springer, 2013.

John C. Roberts, II and Wasim Al-Hamdani. Who can you trust in the cloud?: A review

of security issues within cloud computing. In Proceedings of the 2011 Information

http://doi.acm.org/10.1145/503112.503114

http://doi.acm.org/10.1145/503112.503114

http://doi.acm.org/10.1145/1841909.1841910

http://doi.acm.org/10.1145/1841909.1841910

http://doi.acm.org/10.1145/1572532.1572546

http://www.konkurrensverket.se/t/Page____6735.aspx

http://www.konkurrensverket.se/t/Page____6735.aspx

http://doi.acm.org/10.1145/2529995.2529996

http://doi.acm.org/10.1145/2494603.2480303

Bibliography 53

Security Curriculum Development Conference, InfoSecCD ’11, pages 15–19, New York,

NY, USA, 2011. ACM. ISBN 978-1-4503-0812-0. doi: 10.1145/2047456.2047458. URL

http://doi.acm.org/10.1145/2047456.2047458.

Jeffrey. Rubin and Dana. Chisnell. Handbook of usability testing : how to plan, design,

and conduct effective tests. Wiley Publ., Indianapolis, Ind., 2008. ISBN 9780470185483

0470185481. URL http://www.worldcat.org/search?qt=worldcat_org_all&q=

0470185481.

Staffan Selander and Anna-Lena Rostvall. Design for larande. Nordstedts forlag, 2008.

ISBN 9789113028590.

Ben Shneiderman. Designing the User Interface: Strategies for Effective Human-

computer Interaction. Addison-Wesley Longman Publishing Co., Inc., Boston, MA,

USA, 1986. ISBN 0-201-16505-8.

H. van der Meij and J. van der Meij. Eight guidelines for the design of instructional

videos for software training. Technical Communication, 60(3):205–228, 2013. URL

www.scopus.com. Cited By (since 1996):2.

H. Van Der Meij and J. Van Der Meij. A comparison of paper-based and video tutorials

for software learning. Computers and Education, 78:150–159, 2014. URL www.scopus.

com.

http://doi.acm.org/10.1145/2047456.2047458

http://www.worldcat.org/search?qt=worldcat_org_all&q=0470185481

http://www.worldcat.org/search?qt=worldcat_org_all&q=0470185481

www.scopus.com

www.scopus.com

www.scopus.com

online user tracking and usability tools - a mapping study759122/fulltext01.pdf · tracking user...

Documents