zero to tested
TRANSCRIPT
![Page 1: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/1.jpg)
Zero to Tested “Bullet proof test automation in hostile environments.”
![Page 2: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/2.jpg)
The test automation promised land
Small team of full stack developers (with specialists in specific areas)
BDD/TDD
Strong adherence to the Test Pyramid
Isolation of different layers (good use of stubs, mocks, fakes)
Thin layer of e2e tests
CONTINUOUS DELIVERY!
![Page 3: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/3.jpg)
For most projects
Fairly unrealistic, at least not anytime soon...
![Page 4: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/4.jpg)
But this means...
“Often reliant on end to end automated tests which have a terrible reputation!”
![Page 5: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/5.jpg)
Why?
People acknowledge e2e tests are essential
Tests are flaky
CI infrastructure is poorly managed (configuration drift)
Development of tests lags behind the development cycle (“test
sprints”)
Often ignored by most of the team
Wallboard is more often RED than GREEN
![Page 6: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/6.jpg)
Worth reading...
http://googletesting.blogspot.co.uk/2015/04/just-say-no-to-more-end-to-end-tests.html
http://www.alwaysagileconsulting.com/articles/end-to-end-testing-considered-harmful/
http://devblog.songkick.com/2012/07/16/from-15-hours-to-15-seconds-reducing-a-crushing-build
-time/
![Page 7: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/7.jpg)
But I prefer
https://www.symphonious.net/2015/04/30/making-end-to-end-tests-work/
10,000 end to end acceptance tests, primary form of quality control
complete in 50 minutes, including deploying/starting and tearing down the
servers
all required to pass before we consider a version releasable
included in our information radiators to ensure the team has constant visibility
are owned by the whole team
![Page 8: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/8.jpg)
My theory.
Us “test automation” folk have let ourselves down
Even if we are not in an agile utopia…
Have we exhausted every avenue to make tests reliable?
Need to prove we are doing everything we can to write bulletproof
tests
![Page 9: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/9.jpg)
Now for the practical stuff!Even with minimal influence on budget/project direction we can usually hugely improve on:
CI Infrastructure automation
Automation code and architecture
Let’s take a look at a thin vertical slice through a solid automation
setup...
![Page 10: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/10.jpg)
CI Infrastructure automation
![Page 11: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/11.jpg)
Infrastructure automationStart from an empty EC2 account and finish with:
Fully functioning Jenkins instance
Automated tests running successfully against a “flaky” test
application
Deployed in < 5 mins, very little code, fully automated and
“Ephemeral”
![Page 12: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/12.jpg)
DevOps Toys - Chef, Packer, Terraform, AWS, Docker
BAKE SPIN UP RUN TESTS
![Page 13: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/13.jpg)
Chef
![Page 14: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/14.jpg)
Packer
![Page 15: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/15.jpg)
DevOps Toys - Chef, Packer, Terraform, AWS, Docker
BAKE SPIN UP RUN TESTS
![Page 16: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/16.jpg)
Terraform
![Page 17: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/17.jpg)
DevOps Toys - Chef, Packer, Terraform, AWS, Docker
BAKE SPIN UP RUN TESTS
![Page 18: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/18.jpg)
Jenkins Job DSL Plugin
![Page 19: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/19.jpg)
Docker
![Page 20: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/20.jpg)
Docker
![Page 21: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/21.jpg)
Has it finished?
![Page 22: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/22.jpg)
Recap...
Automated provisioning of AWS infrastructure
Automated build of ‘pre-baked’ AMI
Automatically provisioned Jenkins with Docker
Docker allows clients to have complete control over dependencies
Job DSL for building jobs
![Page 23: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/23.jpg)
Take awayEverything is in source control, no more snowflake CI boxes, no more
handcrafted Jenkins jobs.
“But it’s not scalable!” - switch to agent model with auto scaled
agents
“But it’s not secure!”- not at the moment, but easy to add e.g. github
auth
More importantly It’s ephemeral - you can destroy if compromised
![Page 24: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/24.jpg)
Automation Code & Architecture
![Page 25: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/25.jpg)
Tooling choiceIn Ruby Land the “go-to” libraries are:
Capybara
Selenium Webdriver
Poltergeist/PhantomJS
SitePrism
RSpec
![Page 26: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/26.jpg)
Automation Code
What are the biggest failings of most end to end regression test suites?
![Page 27: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/27.jpg)
My thoughts
Inappropriate synchronisation strategies (page load, async JS)
Inability to deal with non-deterministic applications and environments
Inability to deal with “eventual consistency”
Poor reporting of failure when it occurs
![Page 28: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/28.jpg)
RETRY ALL THE THINGS!
![Page 29: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/29.jpg)
A flaky app...
![Page 30: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/30.jpg)
Page Load
Handled by Selenium Webdriver!
![Page 31: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/31.jpg)
Async JSAvoid this...
![Page 32: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/32.jpg)
Async JSGood libraries have first class support for this:
![Page 33: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/33.jpg)
VisibilityElements exist in DOM but CSS means they are not visible.
Good libraries have first class support for this:
![Page 34: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/34.jpg)
VisibilityWorth remembering…
● display:none
● visibility:hidden
● Opacity:0
● Dimensions
And remember there is a W3C spec for Webdriver:
https://w3c.github.io/webdriver/webdriver-spec.html#dfn-element-displayed
![Page 35: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/35.jpg)
Non determinism
Should we be expected to deal with this?
Maybe, maybe not
Either way don’t just give up, at least detect, log and report on it!
![Page 36: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/36.jpg)
Non determinismDon’t just rely on re-run - be proactive and report on failure
Libraries probably won’t help here!
![Page 37: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/37.jpg)
Eventual Consistency
Caching, load balancing, asynchronous architectures etc.
TL;DR - You might need to wait a bit...
![Page 38: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/38.jpg)
Eventual ConsistencyYou might have to ‘roll your own’ wait helpers from time to time
![Page 39: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/39.jpg)
Logging
Configurable levels: debug, info, warn, error etc
Pretty colours - makes debugging broken builds easier
Have log aggregation for your application? (RSyslog, Logstash,
Kibana)
Now you can combine test and application logs for super debugging
power!
![Page 40: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/40.jpg)
Test the tests
We should unit test any “library code” that has significant complexity.
![Page 41: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/41.jpg)
Static Analysis
Helps maintain consistency, best practice and readability of code.
![Page 42: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/42.jpg)
Reporting
![Page 43: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/43.jpg)
Reporting
![Page 44: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/44.jpg)
Wrapping up
We need to up our game when writing end to end tests
Make the provision of CI infrastructure automated and ephemeral
Write automation code that can handle hostile conditions
Report on your findings to effect change
![Page 46: Zero to tested](https://reader035.vdocuments.us/reader035/viewer/2022062503/587606e21a28ab4a508b6bb3/html5/thumbnails/46.jpg)
Questions?