user interface testing - segmentation 22-03-2013 · 1. create an account on dotmailer with...

User Interface TestingSegmentation

Date: 22/03/2013Version: 0.1Author: Xavier Akram (xavier.akram@gmail.com)

Recap(phase 1) To deploy the ability to create variations (dynamic content) with transactional data within the current system to create immediate revenue streams.

(phase 2) Develop the new dynamic content variation builder. With consideration of using a templating language for the underlying foundation of the new solution.

Imported from: Dynamic Content - Variation Builder Stakeholder Review 26-02-2013 v0.1

RecapUsers need to easily segment data and personalize emails with dynamic content using their transactional data.

With the current system it is not possible to create dynamic content using transactional data and there is currently no visibility in the current system for rules.

Previous sprint we tested users with limited knowledge on how they would build logical statements.

Imported from: User Testing Dynamic Content Rule Builder Review 12-03-2013

The Issue

Users need to easily segment data and personalize emails with dynamic content using their transactional data.

With the current system it is not possible to create dynamic content using transactional data and there is currently no visibility in the current system for rules.

The ObjectivesTo measure if users understand the data structure within dotMailerContacts > My Contacts.

To measure if users can understand and translate predefined segments within dotMailer.

To measure if users can create segments from a written requirement.To measure if users understand the terminology used within the Segmentation Tool

The Approach

To create a test with gradable criteria that can be tested against real users to measure their experience with the dotMailersegmentation platform.

The Tasks1. Create an account on dotMailer with transactional

data.2. To create a test that can measure how users will

interact with the segmentation tool & create a test that will measure if users understand terminology.

3. To test with dotMailer staff.4. Results & Conclusions.

1. The Client

The client an IT company that retails software, hardware, peripherals, computers and laptops. The company also provides training in the form of events.

2. Test Structure

1. Introduction / Explanation 2. T1 C1 – Terminology3. T2 C1 – Understanding Segments4. T2 C2 – Creating segments

TC1 C1 Terminology Test

The purpose

To test users if they could differentiate between Aggregate Filters and Normal Filters, and to see how they address the two.

Test Candidates

• 11 Candidates tested• 5 from IT, 2 from Admin, 4 Sales• 4 Females / 7 Males• 7 between 20-30 / 4 candidates 31+

Test sheet

Purchases.datePurchased is equal to 06/03/2013

Purchases.items.name is one of ( Banana, Orange, Apple, Pear, Strawberry)

Purchases.items.price is greater than or equal to 17.14

Purchases.dateDelivered is greater than 10/03/2012 AND is less than 11/03/2013

Purchases.item.name contains ‘berry’

For Purchases.datePurchased the max is equal to 06/03/2013

For Purchases the count is equal to 2

For Purchases.items.price the average is greater than or equal to 17.14

For Purchases.dateDelivered the min is greater than 10/03/2012 AND is less than 11/03/2013

For Purchases.totalprice the sum is greater than 500

Test Question

Can you identify why each column is different, then try to give a heading / description to each column.

Test Results

Filters Aggregate FiltersSpecific Values Range ValuesFilter Aggregate Filter1 Group Large GroupPurchase Item Type (Specific Data) Purchase Price Total (Range of Data)Single Data Multiple DataNon-Comparative Amalgamations (Comparative)Singular Multiple up to rangeData Field Value Data field attribute valueSingle Purchases Multiple Purchases

Test Question

Sort the cards into two groups, one for ____ and the other for _____.You can use the previous sheet to aid your task

Test ResultsStatement Type Score

Event.DateBooked is greater than 01/01/2013 Filter 6/11

Event.concertName is not one of ( Justin Bieber, One Direction, Nicki Minaj ) Filter 8/11

Event.seats.price is less than or equal to 120.99 Filter 9/11

Event.concertDate is equal to 10/03/2012 Filter 10/11

Event.seats.name starts with ‘W’ Filter 8/11

For Event.DateBooked the min is equal to 06/03/2013 Aggregate 7/11

For Event.seats.number the average is less than 3 Aggregate 10/11

For Event.totalspend the sum is greater than or equal to 100 Aggregate 8/11

For Event.concertDate the max is greater than 10/03/2012 AND is less than 11/03/2013

Aggregate 7/11

For Event the count is greater than 5 Aggregate 9/11

TC2 C1 Understanding Segments

The test

Have the user explain the pre-defined segments on screen.

Basic Segment

Intermediate Segment

Advanced Segment

Results

0

10

20

30

40

50

60

70

80

90

100

Understood Didn't Understand

Basic Segement Intermediate Segment Advanced Segment

Attempt Breakdown

1st Attempt 2nd Attempt 3rd Attempt Didn't Understand

Problem areas

Explanation 1st

Attempt2nd

Attempt3rd

AttemptDidn’t Understand

Purchase has a minimum of one transaction 11 0 0 4

AND (statement connector) 12 0 0 3

Explanation 1st

Attempt2nd

Attempt3rd

AttemptDidn’t Understand

Purchase total greater than 150 6 4 1 4

AND (statement connector) 11 0 0 4

Events has a minimum of one transaction 8 1 0 6

Explanation 1st

Attempt2nd

Attempt3rd

AttemptDidn’t Understand

Purchase has a minimum of one transaction 13 0 0 2

OR (statement connector) 11 0 0 4

Events has a minimum of one transaction 7 1 0 7

Contact booked a course 9 1 0 5

TC1 C2 Building Segments

Interface Navigation

EasyMedium

Hard

Very Hard

Note 6 candidates have not used the segmentation tool before

Interface Navigation Breakdown

0

2

4

6

8

10

12

Basic Segment Intermediate Segment Advanced Segment

Easy Medium Hard Very Hard

The task

Handed a sheet with three requirements to the user. The task, to build the segments from the requirements on the sheet.

Requirement # 1

We would like to target our customers that have made a software purchase in the excess of £10.

Requirement # 1 Results

0

1

2

3

4

5

6

Requirement # 1

1st Attempt 2nd Attempt 3rd Attempt 4+ Attempts Couldn't Validate

Assisted user in building segment

Requirement # 2We would like to target all women that have spent no more than 500 pounds in total on printers, peripherals and smartphones between 1st of January 2010 and the 31st of December 2011.

Requirement # 2 Results

0

1

2

3

4

5

6

Requirement # 2

1st Attempt 2nd Attempt 3rd Attempt 4+ Attempts Couldn't Validate

Assisted user in building segment

Requirement # 3

We want to target our customers that have spent no more than 300 pounds in total on Acer products [other than computers] before the 1st of February 2013 and who have attended an Adobe course since the 18th of March 2012.

Requirement # 3

Requirement # 3 Results

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

Requirement # 3

1st Attempt 2nd Attempt 3rd Attempt 4+ Attempts Couldn't Validate

Assisted user in building segment

Validation Results (all)

1st Attempt2nd Attempt

3rd Attempt

4+ Attempts

Couldn't Validate

Findings

• Users don’t understand the terminology within the segmentation tool specifically ‘Aggregate Filter’ & ‘Filters’.

• Users instinctively clicked ‘Aggregate Filter’ first when building a requirement.

• Users looked for a ‘between’ comparison operator on price(number) and date fields.

• Users had no clear visual feedback on validation from comparing wrong data types (i.e. Number to String).

• To understand the rule, users would close the modal and view the human readable rule within segment editor then re-open the rule in edit view to continue working on their rule.

• Users could not visually determine that there are dropdowns contained within the modal that initiate data & field selection.

• Users did not know that there was autocomplete on item and category fields.

• When an entry in an autocomplete field did not return a result, users couldn’t understand that their criteria was non-existent in the data.

• Users expected autocomplete on ‘Data Fields’ such as Gender.

• Users could benefit from multi-select within the autocomplete feature.

• Users did not understand aggregation and couldn’t work out why certain data fields (i.e. Category “string”) wasn’t available in the dropdown.

• Users understood the tool relatively quickly and improved as the test went on.

• Users didn’t understand the transactional data contained within Contacts > My Contacts.

• Users looked for account related data (i.e. gender) within the transactional modal.

• Users tried to add a manual count method on rules by using ID is greater than 0.

• Users looked for a ‘begins with’ comparison operator on requirement 3 (i.e. Product Item begins with ‘Acer’)

• Users entered a list of categories as ‘Software, Laptops, Printers’ against the equal to comparison operator

• Users showed annoyance once they switched from ‘equal to’ comparison operator to ‘is one of’ and input data was cleared.

• Users showed annoyance once they changed a field and input data was cleared.

Next stepsAnalyse terminology with marketing and architect a new interface that addresses the aggregate data misunderstanding.

Introduce new comparison operators such as ‘between’ and ‘begins with’, and architect the interface to accommodate these changes.

Analyse and re-architect visual validation within the modal experience, with the view to apply across dotMailereventually.

Analyse and re-architect interface validation within the modal experience to ensure no annoyances are had by the user.

Wireframe the segmentation tool based on phase 1 proposal with findings taken into consideration.

Phase 2 Re-design the dropdowns to show more emphasis on the function of the control & update pattern library.

Phase 2 Re-work the autocomplete experience to give emphasis on what it does, and lookup feedback.

Phase 2 Lookup / checkbox style selection options on small data fields. (i.e. Category with 5 items show as a check list instead of is one of).

Phase 2 Architect a method to show data structure & sample data in a tree format etc. directly within the segmentation modal.

Phase 2 Look at parsing of formatted lists (i.e. apple, pear, banana,..) and the best way to handle them on input

Phase 2 Look at re-introducing based per wireframes visibility of rules within the modal experience.

Thank You