reaching for your quality stretch goals: testing at realtor.com
TRANSCRIPT
© 2015 Move, Inc. All rights reserved. Do not copy or distribute.
Reaching your Quality Stretch@ realtor.comKlaus SalchnerVP Engineeringhttps://www.linkedin.com/in/ksalchner@ksalchner
April/5, 2017
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 2
Agenda
• A/B Testing to measure customer impact – 5-6 min
• Group activity – 5-6 min
• Where in the stack to invest in test automation – 5-6 min
• Group activity – 5-6 min
• Testing for reliability, resilience und recovery – 5-6 min
• Group activity – 5-6 min
• Q&A – 5-6 min
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 3
A/B Testing to measure customer impact
• End-to-end quality goes beyond shipping software
• It is about outcome, value for the customer
• How do you measure whether you solved a real customer need?
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 4
A/B Testing to measure customer impact
• Start at the story level, instead of WHAT you build you talk about
what customer need you solve and WHY that matters
• Come up with a Hypothesis and what you will measure to make it
quantative, meaningful?
E-commerce industry is a great example
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 5
A/B Testing to measure customer impact
By VWO Blog
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 6
A/B Testing to measure customer impact
Optimizey Blog
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 7
A/B Testing to measure customer impact
By VWO Blog
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 8
A/B Testing to measure customer impact
By VWO Blog
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 9
A/B Testing to measure customer impact
By VWO Blog
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 10
Some of the A/B Testing tools
• Adobe Test & Target
• Optimizely
• Maxymizely
• and more https://conversionsciences.com/blog/ab-testing-tools
• or home-grown
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 11
The 8 rules of A/B Testing by Search Engine Watch
• Hypothesis
• One Variable at a time
• Clear and aligned success criteria
• Volume and statistical significance (also risk)https://www.optimizely.com/optimization-glossary/statistical-significance
• Test Groups and Splits (can be 50/50, 10/90, etc.)
• Randomization
• Always be testing, with common sense
• Documentation
© 2016 Move, Inc. All rights reserved. Do not copy or distribute.
Q&Ahttp://www.topproducer.com/careers
12
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 13
Agenda
• A/B Testing to measure customer impact – 5-6 min
• Group activity – 5-6 min
• Where in the stack to invest in test automation – 5-6 min
• Group activity – 5-6 min
• Testing for reliability, resilience und recovery – 5-6 min
• Group activity – 5-6 min
• Q&A – 5-6 min
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 14
Where in the stack to invest in test automation
• Always go to the lowest levels
• Always write small, reusable, fast tests
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 15
Where in the stack to invest in test automation
Experiencebehavior
Functionalbehavior
Databehavior
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 16
Where in the stack to invest in test automation
Services
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 17
Where in the stack to invest in test automation
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 18
Test automation best practices
• Test Early and Test often
Have strong acceptance criteria – is an outline for your testing
Write your unit testing as components are created and iterate – full
or lightweight TDD
Integrate the tests early on into your CI/CD so they run all the time
Run them all the time!
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 19
Test automation best practices
• Create good quality test data
Managing test data, especially at scale, is a hard problem to
manage
Many times there is a pre-set environment with pre-set data which
lives for long-times and as time passes gets messy and in the way
for teams to be efficient
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 20
Test automation best practices
• Create good quality test data
The nirvana is that test data gets generated as tests run, and
cleaned up afterwards
And that you have sets of test data you can load into an
environment as it gets stood up
And that you can spin up and tear down environments as needed
– really hard to achieve
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 21
Test automation best practices
• Create UI tests which are resistant to change
Use names and class IDs to identify objects in the UI
Never use text labels, they always change
Never use positions, they always change
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 22
Test automation best practices
• Decide what test cases to automate, like
Frequently used (key) features
Tests which tend to cause human errors
Repetitive tests run with each CI/CD build / deployment
Tests you need to run on multiple platforms / browsers
Tests which are very time consuming
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 23
Test automation best practices
• Review your test cases with PO / PM and Engineers
Get many eyes on it and understand the main flows and edge
cases
Iterate on the testing like you iterate on the development
Get it out to customers for validation and use the feedback
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 24
Test automation best practices
• BDD – Behavior Driven Dev
Test the behavior, business use case
• TDD – Test Driven Dev
Be pragmatic, define the test for a component and then write the
code, not the whole thing
https://codeutopia.net/blog/2015/03/01/unit-testing-tdd-and-bdd
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 25
Test automation best practices
• What are other doing or not doing?
© 2016 Move, Inc. All rights reserved. Do not copy or distribute.
Q&Ahttp://www.topproducer.com/careers
26
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 27
Agenda
• A/B Testing to measure customer impact – 5-6 min
• Group activity – 5-6 min
• Where in the stack to invest in test automation – 5-6 min
• Group activity – 5-6 min
• Testing for reliability, resilience und recovery – 5-6 min
• Group activity – 5-6 min
• Q&A – 5-6 min
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 28
Testing for reliability, resilience and recovery
• Reliability is a concern owned by your Services & APIs
Clustering, fail-over
Elastic scaling (up and down)
Error handling
More
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 29
Basic steps for reliability testing…
• Set up a test environment where ideally all services are set up for
elastic scaling (cloud)
• Run a set of automated tests to establish a baseline (can be really
any automated tests like Selenium or JMeter)
• Scale up and down the load and observe how the various services
will start to scale up and down
• It will also tell you which services are the most stressed parts of
your application
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 30
Basic steps for reliability testing – expected results
• You want to see that your various services scale up – if they don’t
then they will break (unless they are super-fast/efficient)
• You want to see that the services scale down again – if not then
your costs will go up and up
• During the scale up and down you want to see that the functionality
doesn’t break (validates things like state-less)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 31
More reliability testing…
• You can also set the maximum for the scaling up and down and
observe how the system behaves as it reaches the maximum load
• It will point out the weaknesses in your architecture and where to
optimize
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 32
Testing for reliability, resilience and recovery
• Resilience is a concern owned by the consumer / client
Error handling
Graceful degradation of features, so if one or more services are
down the user can still use your application:
For example: If the Ratings & Review service is down I can still
find and view products, or purchase products, without delays
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 33
Testing for reliability, resilience and recovery
By Nextflix Blog
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 34
Testing for reliability, resilience and recovery
By Nextflix Blog
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 35
Testing for reliability, resilience and recovery
By Nextflix Blog
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 36
Testing for reliability, resilience and recovery
• The “golden standard” is Hystrix, former circuit-breaker, by Netflix
https://github.com/Netflix/Hystrix/wiki
• Calls to services are wrapped in commands, and you can define
thresholds to these commands
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 37
Testing for reliability, resilience and recovery
By Nextflix Blog
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 38
Basics of resilience testing…
• Run automated tests, like Selenium or JMeter, and while these run
start shutting off non-essential services
• Then observe that your experience still functions, fast and reliable
• And observe that some of the features depending on these services
degrade – can be either not showing up or showing relevant
messages to users
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 39
Testing for reliability, resilience and recovery
• Recovery testing is the exercise of what it takes to recover from a
major fault or an outage
Troubleshooting – like Logs, instrumentation, etc.
Restoring services
Worst case, restoring data
Root causing is part of it
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 40
Basics of recovery testing…
• Run automated tests, like Selenium or JMeter, and while these run
start shutting off essential services
• Then observe that your experience stops functioning or slows down
for users
• Then run through the manual and automated steps to recover,
without making the situation worse for users
© 2016 Move, Inc. All rights reserved. Do not copy or distribute.
Q&Ahttp://www.topproducer.com/careers
41
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 42
Agenda
• A/B Testing to measure customer impact – 5-6 min
• Group activity – 5-6 min
• Where in the stack to invest in test automation – 5-6 min
• Group activity – 5-6 min
• Testing for reliability, resilience und recovery – 5-6 min
• Group activity – 5-6 min
• Q&A – 5-6 min
© 2016 Move, Inc. All rights reserved. Do not copy or distribute.
Q&Ahttp://www.topproducer.com/careers
43