remote testing report

Findings: Loop 11Remote User testing report

1

Contents

2

Methodology 3

Summary 4

Detailed analysis 6

Task 1 6

Task 2 10

Task 3 13

Task 4 16

Task 5 21

Methodology

3

Remote user testing was conducted via an online testing toolkit to cast a wider net across users of the current site within NSW health. Five tasks were devised to target key user activities as well as explore problem areas identified in the emerging research from the Heuristic evaluation, Google analytics data and user interviews. These tasks were worded using a combination of natural language and terms that match exactly the content.

A total of 43 staff from across the state participated in the test. Participants were given the choice to abandon or complete each task. Users who spend less than 10 seconds on any task were filtered out from that task only.

The data available from the online testing toolkit* allowed us to view total time taken by each participant as well as a screen-by-screen breakdown of their user journey through each task. Findings focus on the following attributes from each participants’ journey as a way of breaking down the experience:

• Initial search behaviour (Eg, Quick search Vs All Indicators)• Lateral search behaviour (related indicators or change view)• Total time taken• Total pages viewed• Total indicators viewed

The following report delivers a summary of the data gathered, and identifies a set of opportunities that present themselves as solutions to problem areas.

* Limitations in the data available include an ability to see which tabs were opened or closed on a page and data relating to the Quick Search. The My Report feature, which is php driven was also difficult to track using remote testing.

The majority of tasks had around a 50% success rate. The ‘success’ group was more likely to access more pages and more indicators in less time.

The ‘grey zone’ lies between the success and fail-abandon groups and tell us about particularly interesting patterns where users are almost able to successfully complete the task. The ‘grey zone’ group typically spends the longest on tasks, yet arrives at the least pages and indicators.

The ‘fail/abandon’ group on average spent less time and accessed less pages than the ‘success’ group, suggesting that they came across barriers to content or had difficulty accessing the right content.

4

Summary:

Task 1

Task 2

Task 3

Task 4

Task 5

0 25 50 75 100

Success Grey zone Fail/Abandon

2.9 mins

4 mins

2 mins

9 pages5 indicators

5 pages2 indicators

6 pages3 indicators

5

2.9 mins

4 mins

2 mins

9 pages

5 indicators

5 pages

2 indicators

6 pages

3 indicators

6

Themes for failure, abandon and time-consuming user journeys:

High bounce rate at the home page - users not knowing how or where to begin task.

High bounce rate at Indicator pages - users not knowing how or where to begin lateral searching or data slicing.

Users leaving Indicator pages and searching/browsing then repeating the process to arrive at data slices or related content.

Users spending the overwhelming percentage of their task navigating lists of links in ‘Browse’ methods such as All Indicators, Topics etc.

Users arriving at many indicators that • were related to the indicator that they

were after, but didn’t contain the precise information they needed

• included the precise information they needed among other superfluous data

Opportunities for an improved user experience:

Improving pathways to content from the home page

Creating a stronger design focus for lateral searching techniques (related indicators and data slices) on Indicator pages.

Create more efficient navigation for information categories in pages that use lists of links.

Increasing visibility of filter and other sub-categories of data slice tools at a higher navigation level

Improve match between data slice titles and Indicator content

7

“You are researching Antenatal care across NSW. Download a graph that shows the percentage of mothers who made an antenatal visit before 14 weeks in 2010.”

Task 1: Find specific Indicator

AIM:•to observe participant methods when conducting a targeted search.•determine barriers and accelerators to arriving at specific content.

•NB. due to the limitations of the data available it was not possible to see when the download tab was opened on the site. ‘Success’ was therefore defined as having arrived at the correct Indicator.

62% Successfuly found the Indicator9.5% Found the Indicator by LGA (‘Grey’ succcess area)28.5% Abandoned the task

20 Participants

8

The ‘Grey zone’

The ‘Grey zone’ of success in this task denotes participants who arrived at the indicators to suit their purpose, but not precisely. This shows barriers to navigating related content. ‘Grey zone’ participants spent longer on their task and viewed more pages and indicators than other participants, yet were not able to find the precise indicator.

How they got there

• Both browse and search are highly represented among the ‘successful’ and ‘grey zone’ participants.• Evidence suggests that browsing participants were using the browsers’ find-on-page function to arrive

at indicators instead of the website’s navigation. This explains why related indicators such as ‘by LGA’ and ‘by Aboriginality’ were accessed from the Mothers and Babies report.

• Those that abandoned the task largely did so on the home page, however, journeys of up to 13 minutes were taken by those who browsed All Indicators.

0

20

40

60

80

Mothers and Babies reportQuick Search Topics All IndicatorsNone - Home PageQuickSearch

Mothersand

Babies

Topics NoneAllIndicators

%

%

%

%

9

1.8 mins

5.1 mins

2.5 mins

5 pages viewed1 indicator viewed

6 pages viewed2 indicators viewed


Main reasons for longer journeys

• Successful participants - navigating home page and navigating topics • Grey zone participants - navigating home page and navigating mothers and babies report• Abandon participants - navigating all indicators and navigating mothers and babies report

10

Opportunities for improvement

• More transparent pathways to related content on Indicator pages• More efficient navigation for information categories in on pages with lists of links (E.g, html reports,

All Indicators and Topics).• Stronger design focus on pathways to content from homepage

11

“You are researching Antenatal care over NSW, and you have found this indicator trend shown below. Now download related indicators that might be useful to your research.”

Task 2: Finding related content

AIM:•determine the availability of related content.•determine the usability of the download function.


45% Successfuly found content at Related Indicator tab55% Failed to find content and Related Indicator tab

20 Participants

The majority of participants failed to find content through the Related Indicator tab. Although less pages were accessed by this group, they typically spent longer on the task, indicating an issue with the visibility of related content at Indicator pages.

12

1 min

1.6 mins

5 pages viewed

3 pages viewed

0

25

50

75

100

related indicatortopicsQuick Searchchange viewnothingQuickSearch

ChangeView

Topics NoneRelatedIndicator

Tab

%

%

%

%

How they got there

• Although all successful participants used the related indicator tab, a few continued to search for related content elsewhere.

• The majority of those that failed the task took no action at all. Others looked to Change View or went out of the Indicator page, back to the home screen to conduct a Quick Search - showing a lack of awareness for lateral or context-driven search methods.


• Increase visibility of related content and lateral search techniques at indicator views• Improve taxonomy relating to ‘Change View’ and ‘Related Indicators’ so that it is clear that

‘Change View’ does not list all indicators relating to the one visible.

13

14

“You are researching Antenatal care over NSW, and you have found this indicator trend shown below. Now you want to compare antenatal trends of ‘Major Cities’ against the rest of NSW. Find the graph that shows this.”

Task 3: Data slicing with Change View

AIM:•observe how users are investigating specific findings through data slicing•determine the usability of ‘change view’ drop down menu

58% Successfuly used Change View to find the Indicator21% Used Change View to find an incorrect Indicator21% Abandoned the task

20 Participants

15

The ‘Grey zone’

There ‘Grey zone’ of success in this story explains the proportion of participants who went directly to Change View as a way of accessing different data slices, but who did not arrive at the correct Indicator. This tells us that users were either not able to match the natural wording of the question to the drop down titles, and/or that the drop down titles do not effectively match the content shown in the Indicator slices that they point to.

0

25

50

75

100

correct sliceincorrect slicequick search TopicsNone - Home Pagedirectlyto correct

slice

via anumberof slices

Topics NoAction

Quick Search

%

%

%

%

How they got there

• 58% of the Success group went directly to the correct Indicator via the Change View menu, followed by 33% viewing a number of indicators in the Change View menu first.

• The ‘Grey’ success group are characterized by looking only at incorrect indicators in the Change View menu

• The fail group are halved between not acting at all, and navigating to ‘topics’ in order to find the Indicator described in the question.

1.4 mins

2 mins

1 min





• Successful participants - difficulty perusing Change View drop downs and filters.• Grey zone participants - difficulty perusing Change View drop downs and filters. • Abandon participants - not aware of Change View functionality at all. Time spent on Topics page.


• Improved match of taxonomy to Indicator content• Improved match of taxonomy to users language• Visibility of filter and other sub-categories of data slice tools at a higher navigation level Improved

match of taxonomy to Indicator content• Stronger design focus on data slice tools at Indicator page

16

17

“You are giving a presentation on Smoking in NSW. Tailor and download a PDF report that contains all content from the following 3 indicators: ‘Smoking Status, trends’, ‘Smoking Status, area trends’, and ‘Current Smoking by Age, trends.’”

Task 4: Using ‘My Report’

AIM:•To determine usability of ‘My Report’ feature•To observe how users search for a number of related Indicators

NB. due to the technology behind the ‘My Report’ feature, results did not contain the data necessary to collect data on participants success in using this feature. Success was therefore redefined as having access all 3 of the said Indicators as well as ‘My Report’ page at some point in their journey.

53% Accessed all 3 indicators and My Report47% Abandoned the task

16 Participants

All of the 53% that successfully accessed all indicators were also able to access My Report. Only 1 participant from the Abandoned-task group had accessed My Report in their task journey.

0

25

50

75

100

found all 3 found 2 found 1 found noneFoundall 3

indicators

Found only 1

indicator

Foundno

indicators

Foundonly 2

indicators

%

%

%

%

0

25

50

75

100

accessed my reportAccessed My Report

%

%

%

%

18

How they got there

• Majority of both groups used Quick Search as the initial method• Success group was more likely to search laterally from the Indicator page using both Change View

and Related Indicator tab. • Abandon group was more likely to browse through Topics and All Indicators, and less likely to

search laterally from the Indicator page using Change View and Related Indicator tab.

UsedChange

View

UsedRelatedIndicator

tab

%

%

%

%

0

25

50

75

100

Lateral Search Methods

QuickSearch

Topics All Indicators

%

%

%

%

0

20

40

60

80

Initial Search Methods

19

6.7 mins

4.5 mins

18 pages viewed

14 pages viewed

11 indicators viewed

6 indicators viewed

The minimum time taken amongst Success-group was 1.5 minutes, suggesting that task-completion can be improved by the opportunities listed above.

There is a level of ambiguity in this data relating to how much time was spent on the My Report section. However, looking at the Abandon-group alone, who did not access My Report - we can see that there is a barrier to arriving at content easily for those users who don’t use lateral-search methods. This high ratio between pages viewed and indicators viewed could be responsible for abandon rate at an overall shorter time threshold.

Interestingly, 21% of all participants accessed ‘Using This Report’ during this task.

21%

21% of participants accessed ‘Using This Report’ during this task.

20

21


• Improved category navigation for browse-like searches such as Topics and All Indicators• Stronger design focus on lateral searching for related content from Indicator pages.

22

“You are investigating data relating to Smoking in NSW and it’s impact on the hospital system. Find out the total hospitalizations in 2011-12 caused by smoking in: 1. all of NSW, and 2. just the Sydney Local Government Area. Download the 2 graphs.”

Task 5: Drilling down to specific data

AIM:•observe methods of arriving at 2 related Indicators based on location.•determine barriers and accelerators to arriving at specific content.


61% Successfully accessed both indicators39% Abandoned the task

13 Participants

0

25

50

75

100

found all 3 found 1 found noneFoundboth

indicators

Foundno

indicators

Foundonly 1

indicator

%

%

%

%

How they got there

• Majority of both groups used Quick Search as the initial method, closely followed by Topics page in both groups.

• Success group was only slightly more likely to search laterally from the Indicator page.• Abandon group was more likely to browse All Indicators page.

UsedChange View

%

%

%

%

0

20

40

60

80

Lateral Search Methods

QuickSearch

Topics All Indicators

%

%

%

%

0

17.5

35

52.5

70Initial Search Methods

LHD Indicators

NoAction

23

24

3.6 mins

2.3 mins

14 pages viewed

9 pages viewed

7 Indicators viewed

5 indicators viewed


• All participants - difficulty finding specific indicators in Topics, All Indicators and Indicators by LHD pages. Barriers to content are evident in 2:1 ratio above of pages to Indicator content


• Improved match of taxonomy to Indicator content• Simplified navigation through information categories on pages with lists of links (E.g, html reports,

All Indicators and Topics)

remote testing report

Design