user interface testing - segmentation 22-03-2013 · 1. create an account on dotmailer with...
TRANSCRIPT
User Interface TestingSegmentation
Date: 22/03/2013Version: 0.1Author: Xavier Akram ([email protected])
Recap(phase 1) To deploy the ability to create variations (dynamic content) with transactional data within the current system to create immediate revenue streams.
(phase 2) Develop the new dynamic content variation builder. With consideration of using a templating language for the underlying foundation of the new solution.
Imported from: Dynamic Content - Variation Builder Stakeholder Review 26-02-2013 v0.1
RecapUsers need to easily segment data and personalize emails with dynamic content using their transactional data.
With the current system it is not possible to create dynamic content using transactional data and there is currently no visibility in the current system for rules.
Previous sprint we tested users with limited knowledge on how they would build logical statements.
Imported from: User Testing Dynamic Content Rule Builder Review 12-03-2013
The Issue
Users need to easily segment data and personalize emails with dynamic content using their transactional data.
With the current system it is not possible to create dynamic content using transactional data and there is currently no visibility in the current system for rules.
The ObjectivesTo measure if users understand the data structure within dotMailerContacts > My Contacts.
To measure if users can understand and translate predefined segments within dotMailer.
To measure if users can create segments from a written requirement.To measure if users understand the terminology used within the Segmentation Tool
The Approach
To create a test with gradable criteria that can be tested against real users to measure their experience with the dotMailersegmentation platform.
The Tasks1. Create an account on dotMailer with transactional
data.2. To create a test that can measure how users will
interact with the segmentation tool & create a test that will measure if users understand terminology.
3. To test with dotMailer staff.4. Results & Conclusions.
1. The Client
The client an IT company that retails software, hardware, peripherals, computers and laptops. The company also provides training in the form of events.
2. Test Structure
1. Introduction / Explanation 2. T1 C1 – Terminology3. T2 C1 – Understanding Segments4. T2 C2 – Creating segments
TC1 C1 Terminology Test
The purpose
To test users if they could differentiate between Aggregate Filters and Normal Filters, and to see how they address the two.
Test Candidates
• 11 Candidates tested• 5 from IT, 2 from Admin, 4 Sales• 4 Females / 7 Males• 7 between 20-30 / 4 candidates 31+
Test sheet
Purchases.datePurchased is equal to 06/03/2013
Purchases.items.name is one of ( Banana, Orange, Apple, Pear, Strawberry)
Purchases.items.price is greater than or equal to 17.14
Purchases.dateDelivered is greater than 10/03/2012 AND is less than 11/03/2013
Purchases.item.name contains ‘berry’
For Purchases.datePurchased the max is equal to 06/03/2013
For Purchases the count is equal to 2
For Purchases.items.price the average is greater than or equal to 17.14
For Purchases.dateDelivered the min is greater than 10/03/2012 AND is less than 11/03/2013
For Purchases.totalprice the sum is greater than 500
Test Question
Can you identify why each column is different, then try to give a heading / description to each column.
Test Results
Filters Aggregate FiltersSpecific Values Range ValuesFilter Aggregate Filter1 Group Large GroupPurchase Item Type (Specific Data) Purchase Price Total (Range of Data)Single Data Multiple DataNon-Comparative Amalgamations (Comparative)Singular Multiple up to rangeData Field Value Data field attribute valueSingle Purchases Multiple Purchases
Test Question
Sort the cards into two groups, one for ____ and the other for _____.You can use the previous sheet to aid your task
Test ResultsStatement Type Score
Event.DateBooked is greater than 01/01/2013 Filter 6/11
Event.concertName is not one of ( Justin Bieber, One Direction, Nicki Minaj ) Filter 8/11
Event.seats.price is less than or equal to 120.99 Filter 9/11
Event.concertDate is equal to 10/03/2012 Filter 10/11
Event.seats.name starts with ‘W’ Filter 8/11
For Event.DateBooked the min is equal to 06/03/2013 Aggregate 7/11
For Event.seats.number the average is less than 3 Aggregate 10/11
For Event.totalspend the sum is greater than or equal to 100 Aggregate 8/11
For Event.concertDate the max is greater than 10/03/2012 AND is less than 11/03/2013
Aggregate 7/11
For Event the count is greater than 5 Aggregate 9/11
TC2 C1 Understanding Segments
The test
Have the user explain the pre-defined segments on screen.
Basic Segment
Intermediate Segment
Advanced Segment
Results
0
10
20
30
40
50
60
70
80
90
100
Understood Didn't Understand
Basic Segement Intermediate Segment Advanced Segment
Attempt Breakdown
1st Attempt 2nd Attempt 3rd Attempt Didn't Understand
Problem areas
Explanation 1st
Attempt2nd
Attempt3rd
AttemptDidn’t Understand
Purchase has a minimum of one transaction 11 0 0 4
AND (statement connector) 12 0 0 3
Explanation 1st
Attempt2nd
Attempt3rd
AttemptDidn’t Understand
Purchase total greater than 150 6 4 1 4
AND (statement connector) 11 0 0 4
Events has a minimum of one transaction 8 1 0 6
Explanation 1st
Attempt2nd
Attempt3rd
AttemptDidn’t Understand
Purchase has a minimum of one transaction 13 0 0 2
OR (statement connector) 11 0 0 4
Events has a minimum of one transaction 7 1 0 7
Contact booked a course 9 1 0 5
TC1 C2 Building Segments
Interface Navigation
EasyMedium
Hard
Very Hard
Note 6 candidates have not used the segmentation tool before
Interface Navigation Breakdown
0
2
4
6
8
10
12
Basic Segment Intermediate Segment Advanced Segment
Easy Medium Hard Very Hard
The task
Handed a sheet with three requirements to the user. The task, to build the segments from the requirements on the sheet.
Requirement # 1
We would like to target our customers that have made a software purchase in the excess of £10.
Requirement # 1 Results
0
1
2
3
4
5
6
Requirement # 1
1st Attempt 2nd Attempt 3rd Attempt 4+ Attempts Couldn't Validate
Assisted user in building segment
Requirement # 2We would like to target all women that have spent no more than 500 pounds in total on printers, peripherals and smartphones between 1st of January 2010 and the 31st of December 2011.
Requirement # 2 Results
0
1
2
3
4
5
6
Requirement # 2
1st Attempt 2nd Attempt 3rd Attempt 4+ Attempts Couldn't Validate
Assisted user in building segment
Requirement # 3
We want to target our customers that have spent no more than 300 pounds in total on Acer products [other than computers] before the 1st of February 2013 and who have attended an Adobe course since the 18th of March 2012.
Requirement # 3
Requirement # 3 Results
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
Requirement # 3
1st Attempt 2nd Attempt 3rd Attempt 4+ Attempts Couldn't Validate
Assisted user in building segment
Validation Results (all)
1st Attempt2nd Attempt
3rd Attempt
4+ Attempts
Couldn't Validate
Findings
• Users don’t understand the terminology within the segmentation tool specifically ‘Aggregate Filter’ & ‘Filters’.
• Users instinctively clicked ‘Aggregate Filter’ first when building a requirement.
• Users looked for a ‘between’ comparison operator on price(number) and date fields.
• Users had no clear visual feedback on validation from comparing wrong data types (i.e. Number to String).
• To understand the rule, users would close the modal and view the human readable rule within segment editor then re-open the rule in edit view to continue working on their rule.
• Users could not visually determine that there are dropdowns contained within the modal that initiate data & field selection.
• Users did not know that there was autocomplete on item and category fields.
• When an entry in an autocomplete field did not return a result, users couldn’t understand that their criteria was non-existent in the data.
• Users expected autocomplete on ‘Data Fields’ such as Gender.
• Users could benefit from multi-select within the autocomplete feature.
• Users did not understand aggregation and couldn’t work out why certain data fields (i.e. Category “string”) wasn’t available in the dropdown.
• Users understood the tool relatively quickly and improved as the test went on.
• Users didn’t understand the transactional data contained within Contacts > My Contacts.
• Users looked for account related data (i.e. gender) within the transactional modal.
• Users tried to add a manual count method on rules by using ID is greater than 0.
• Users looked for a ‘begins with’ comparison operator on requirement 3 (i.e. Product Item begins with ‘Acer’)
• Users entered a list of categories as ‘Software, Laptops, Printers’ against the equal to comparison operator
• Users showed annoyance once they switched from ‘equal to’ comparison operator to ‘is one of’ and input data was cleared.
• Users showed annoyance once they changed a field and input data was cleared.
Next stepsAnalyse terminology with marketing and architect a new interface that addresses the aggregate data misunderstanding.
Introduce new comparison operators such as ‘between’ and ‘begins with’, and architect the interface to accommodate these changes.
Analyse and re-architect visual validation within the modal experience, with the view to apply across dotMailereventually.
Analyse and re-architect interface validation within the modal experience to ensure no annoyances are had by the user.
Wireframe the segmentation tool based on phase 1 proposal with findings taken into consideration.
Phase 2 Re-design the dropdowns to show more emphasis on the function of the control & update pattern library.
Phase 2 Re-work the autocomplete experience to give emphasis on what it does, and lookup feedback.
Phase 2 Lookup / checkbox style selection options on small data fields. (i.e. Category with 5 items show as a check list instead of is one of).
Phase 2 Architect a method to show data structure & sample data in a tree format etc. directly within the segmentation modal.
Phase 2 Look at parsing of formatted lists (i.e. apple, pear, banana,..) and the best way to handle them on input
Phase 2 Look at re-introducing based per wireframes visibility of rules within the modal experience.
Thank You