how to master ux testing in an agile design process
Post on 27-Jan-2015
123 Views
Preview:
DESCRIPTION
TRANSCRIPT
How to Master UX Testing in an Agile Design Process
Jon Innes
Jeff Sauro
March 29, 2012
Measuring Usability
2UX INNOVATION LLC
Introductions
MeasuringUsability.comjeff@measuringusability.com
Twitter @MsrUsability‘The UX Measurement MASTER’
Measuring Usability
www.uxinnovation.comTwitter @innes_jon‘The Agile UX GURU’
Founder at UserZoomalfonso@userzoom.com
@userzoom‘The Moderator’
3UX INNOVATION LLC
Overview
• Agile and UX• Jon Innes, approx 35 min
• Agile Metrics An Example• Jeff Sauro, approx 15 min
• Q&A session
• Everyone, approx 10 min
• Twitter hashtag #uzwebinar
4UX INNOVATION LLC
Myths I Hope to Dispel Today
• Agile and user testing timelines are incompatible
• You need working code to test with users
• It’s hard to scale the usability testing process
• Automating usability testing is not possible
5UX INNOVATION LLC
Things Change
6UX INNOVATION LLC
Factors Influencing User Research
• Competitive forces make user experience far more critical than the past
• Software development processes have changed
• Product development cycles are faster, yearly release cycles are almost extinct
• Modern technology allows us to do things we couldn’t do before
7UX INNOVATION LLC
The Old Way: Too Big & Too Late
8UX INNOVATION LLC
Design Refine
Combine
The 3x3 Way: Test Multiple Concepts Early
http://www.carolrighi.com/documents/Righi%203x3.pdf
• 3 concepts, 3 pages deep
• 3 iterations of design & test
• 3 or more users per iteration
• Test low fidelity first, then high
• Improve concept each iteration
• Combine best elements into 1 concept & iterate ? Refine
Design
9UX INNOVATION LLC
The RITE Way: Refine One Concept Faster
Schedule image from Brian Keith Sullivanhttp://bigdesignevents.com/2011/08/5-ways-to-improve-your-next-agile-project-with-the-rite-method/RITE: Method http://en.wikipedia.org/wiki/RITE_Method
REFINEDESIGN
REFINEDESIGN
REFINEDESIGN?
10UX INNOVATION LLC
The A/B Way: Refine Details At End
Design A Build & Launch A Test A
Design B Build & Launch B Test B
• Design & build multiple designs and compare details
• By nature less subject to test condition biases
• Easy to get large sample sizes & statistically reliable metrics
• Not so great at providing deeper insights on “why”
Update
Repeat?
11UX INNOVATION LLC
The Challenge: Testing in an Agile Way
• Traditional testing does not fit well into agile:• Recruiting can take up to two weeks• Reports and test plans take too long• You often don’t have a stable UI until too late
• What can we do? Adapt existing methods?• Use old methods planning outside of agile timeframes?• 3x3 solves the “waiting till the end” for working UI problem• RITE is a step in the right direction, it speeds up iterations• A/B split tests are effective for tweaking final details
12UX INNOVATION LLC
What is Agile?
A collection of best practices for developing software including:
• Holding regular status meetings in a standup format
• Requirements in the form of user stories or use cases
• Defining tests for code before writing the code itself
• Working in short iterations to create incremental improvements
Common variations include Scrum & XP and all have a mindset that is best described as “lean”
13UX INNOVATION LLC
What is Scrum?
Product Backlog
Sprint Backlog
Potentially Shippable Product
Increment
Daily Scrum Meeting
24 hrs
2-4 weekSprint
Image from: http://www.mountaingoatsoftware.com/
14UX INNOVATION LLC
What is Lean?
The western interpretation of Taiichi Ohno’s set of values and practices for removing three types of inefficiencies:
• 無駄 = muda wasted outputs
• 斑 = mura unnecessary inconsistencies
• 無理 = muri wasted efforts
Lean Startups and Lean UX are based on these values and practices
15UX INNOVATION LLC
What is Lean UX?
As response to the overall trend of using agile and lean methods, Lean UX applies the lean principles to UX:
• Breaking deliverables down to fit into agile timelines so they are “just in time”
• Reducing wasted effort associated with internal deliverables
• Reducing the variability in UX deliverables that create waste
• Collaborating more closely with non-UX team members
• Getting user feedback earlier and on a more regular basis
For more on Lean UX see:http://www.slideshare.net/balancedteam/02-sat-janice-lean-ux-landscapehttp://joshuaseiden.com/blog/2011/09/what-makes-it-lean/ http://uxdesign.smashingmagazine.com/2011/03/07/lean-ux-getting-out-of-the-deliverables-business/
16UX INNOVATION LLC
Where’s The Waste?
• Large formal reports that nobody reads
• Studies that find big problems after it’s too late
• Gathering unnecessarily large sample sizes
• Retesting stuff over and over the manual way
• Developing and launching real stuff just to learn what you could learned from simpler prototypes
17UX INNOVATION LLC
What’s Changed?
• Product development cycles are getting faster
• Businesses are now more UX savvy
• In the US 79% of the population is now online
• Remote automated usability testing solutions have become more:• More capable• Less costly• Easier to use
18UX INNOVATION LLC
What is Remote Automated User Testing?
• Not just site analytics or surveys
• Task oriented vs. impressions and time on site
• Not limited to existing users of your site
• Can be used on prototypes or working code
• No moderator required
• Combines the benefits of lab testing with the scalability of A/B testing
19UX INNOVATION LLC
Remote Automated Testing Supports
• Task completion rates vs. just click counts
• Time on tasks vs. simple time on site
• Click path data by task
• Heat maps showing where users click on a page
• Task specific and overall comments
• Structured survey data (such as SUS)
• Video highlights (of user’s screens)
20UX INNOVATION LLC
Advantages of Automated Testing
Automation of moderation allows you to:• Conduct large scale studies before having working code• Easily test competitive sites against your site• Compare functionality from past designs with minimal effort
including changes of any size with ease• Gather large sample sizes like A/B testing and actionable
insights comparable to traditional usability data• Gather data from users who do not currently visit your site• Reduce both testing costs and associated development
21UX INNOVATION LLC
Tips for More Agile Testing
• Have personas agreed to by team before sprinting
• Develop an appropriate recruiting strategy early
• Define hypotheses & target metrics for stories in the backlog working with your product owner
• Track usability metrics in a public place
• Test early design concepts not just final details
• Automate user testing for stories as you go & retest
• Waiting until the end is the waterfall way
22UX INNOVATION LLC
Tips for Remote Automated Testing
• Always run small pilots before large scale tests, in many cases this can provide valid data anyway
• Allow at least a week for tests with panel participants
• Combine intercept recruiting with panels to minimize costs while reaching both new and existing users
• Keep the participant tasks to a small number to minimize drop out rates
• Use content analysis techniques to analyze open ended survey data
23UX INNOVATION LLC
Some Things Stay The Same
• Questions to ask on your team• Have we validated these stories and personas? • Did the iteration incorporate user feedback?
• Can they use it?• Do they like it? • Would they recommend it to a friend?• Did we make it measurably better?
• UX principles remain the same in agile• Iterations without user tests are a lost learning opportunity• Waiting until the end to user test is the waterfall way• The definition of “done” can only be determined by users
24UX INNOVATION LLC
Agile Metrics: An Example
Measuring UsabilityJeff Sauro
25UX INNOVATION LLC
Design Wireframes: Don’t Guess Test
• Which Design Comp is better ?• Operationalize into Metrics
26UX INNOVATION LLC
Trick Test: Tree Test + Click Test
• Use Photoshop Mockups with Core Nav/Content
• Create a set of realistic task scenarios• For example, where would click to download drivers for your
printer?
• Recruit participants online to try and have them find the information.
27UX INNOVATION LLC
The Metrics
• Time to locate (task)
• Confidence
• Task Difficulty
• Preference
28UX INNOVATION LLC
3 Design Comps: 2 Navigation Comps
A B C
1 2
29UX INNOVATION LLC
Add CSS & HTML to Create Clickable Hotspots
30UX INNOVATION LLC
Audio & Video With YouEye.com + UserZoom
• UserZoom ensures tasks and comps appear in counterbalanced order• UserZoom records time and responses to questions
31UX INNOVATION LLC
Click Map from UserZoom
• 73% click on top navigation
• 90% confidence interval (50% - 93%)
32UX INNOVATION LLC
Which Design / Tasks Are Faster? (13 Users)
Comp A is Easier About The Same Comp B is Easier
33UX INNOVATION LLC
Which Design is Preferred ?
Strongly Prefer A Strongly Prefer BNeutralMean +
80% Confidence Int.
A B
34UX INNOVATION LLC
Recommendations
Iterate Next Designs with Comp B and Nav 2
B2
35UX INNOVATION LLC
Find out More
www.LeanUXDenver.comSept 19-21st 2012
Hyatt Convention CenterDenver, CO
@LeanUXDenver
MeasuringUsability.comjeff@measuringusability.com
Twitter @MsrUsability
Measuring Usability
www.uxinnovation.comTwitter @innes_jon
36UX INNOVATION LLC
Q&A
37UX INNOVATION LLC
Thank you for your time!
Thank you for your time!
Stay tuned for new webinars!
Measuring Usability
top related