stabilizing se build - selenium conf 2013
DESCRIPTION
Talk at 2013 Selenium Conference on the topic of stabilizing a build and making it green over allTRANSCRIPT
![Page 3: Stabilizing SE Build - Selenium conf 2013](https://reader038.vdocuments.us/reader038/viewer/2022102814/54b69cce4a7959b5588b4593/html5/thumbnails/3.jpg)
Love My Job
• Love testing• Love hacking• Love talking about above
![Page 4: Stabilizing SE Build - Selenium conf 2013](https://reader038.vdocuments.us/reader038/viewer/2022102814/54b69cce4a7959b5588b4593/html5/thumbnails/4.jpg)
![Page 5: Stabilizing SE Build - Selenium conf 2013](https://reader038.vdocuments.us/reader038/viewer/2022102814/54b69cce4a7959b5588b4593/html5/thumbnails/5.jpg)
Partner In Crime
Darko MarinovAssociate ProfessorUniversity of Illinois
![Page 6: Stabilizing SE Build - Selenium conf 2013](https://reader038.vdocuments.us/reader038/viewer/2022102814/54b69cce4a7959b5588b4593/html5/thumbnails/6.jpg)
Stability is a Journey not destination
![Page 7: Stabilizing SE Build - Selenium conf 2013](https://reader038.vdocuments.us/reader038/viewer/2022102814/54b69cce4a7959b5588b4593/html5/thumbnails/7.jpg)
![Page 8: Stabilizing SE Build - Selenium conf 2013](https://reader038.vdocuments.us/reader038/viewer/2022102814/54b69cce4a7959b5588b4593/html5/thumbnails/8.jpg)
Culture
![Page 9: Stabilizing SE Build - Selenium conf 2013](https://reader038.vdocuments.us/reader038/viewer/2022102814/54b69cce4a7959b5588b4593/html5/thumbnails/9.jpg)
Culture Changes
• Convince Management that flakey tests waste a lot of money
• Discourage developers from relying on tests• Prevent red build from merging into
master/trunk• Red master is NOT OK!
![Page 10: Stabilizing SE Build - Selenium conf 2013](https://reader038.vdocuments.us/reader038/viewer/2022102814/54b69cce4a7959b5588b4593/html5/thumbnails/10.jpg)
Fix Ajax Waiting
• jQuery– evaluate_script("jQuery.active") == 0– evaluate_script('$(":animated").length') == 0
![Page 11: Stabilizing SE Build - Selenium conf 2013](https://reader038.vdocuments.us/reader038/viewer/2022102814/54b69cce4a7959b5588b4593/html5/thumbnails/11.jpg)
Flakey Tests
![Page 12: Stabilizing SE Build - Selenium conf 2013](https://reader038.vdocuments.us/reader038/viewer/2022102814/54b69cce4a7959b5588b4593/html5/thumbnails/12.jpg)
Self contained tests
• Separate Databases• Self reliant tests– Setup/Teardown
![Page 13: Stabilizing SE Build - Selenium conf 2013](https://reader038.vdocuments.us/reader038/viewer/2022102814/54b69cce4a7959b5588b4593/html5/thumbnails/13.jpg)
Faster Feedback Loop
• Fight to get build to 10 mins• Notify of failures before build is finished– Rspec/Cucumber Early Notification Formatters– Blog post to follow– engineering.groupon.com
![Page 14: Stabilizing SE Build - Selenium conf 2013](https://reader038.vdocuments.us/reader038/viewer/2022102814/54b69cce4a7959b5588b4593/html5/thumbnails/14.jpg)
Flakey Detector
• Check Jenkins or Git diff• If a test was modified, kick of flake detector
build– Run changed test 5k times in 10 mins– In random order– They should always be green!
• engineering.groupon.com
![Page 15: Stabilizing SE Build - Selenium conf 2013](https://reader038.vdocuments.us/reader038/viewer/2022102814/54b69cce4a7959b5588b4593/html5/thumbnails/15.jpg)
Fix Environment Problems!!
• Tests get blamed if environment fails• Standardize test boxes– Chef, Puppet, w/e
![Page 16: Stabilizing SE Build - Selenium conf 2013](https://reader038.vdocuments.us/reader038/viewer/2022102814/54b69cce4a7959b5588b4593/html5/thumbnails/16.jpg)
Flakey Environment
![Page 17: Stabilizing SE Build - Selenium conf 2013](https://reader038.vdocuments.us/reader038/viewer/2022102814/54b69cce4a7959b5588b4593/html5/thumbnails/17.jpg)
Windows Tips
• Don’t use IE• Move mouse out of browser in IE• Maximize test window• Don’t use RPD/RDC– Use VNC
![Page 18: Stabilizing SE Build - Selenium conf 2013](https://reader038.vdocuments.us/reader038/viewer/2022102814/54b69cce4a7959b5588b4593/html5/thumbnails/18.jpg)
Linux Tips
• Run Firefox in separate Desktop
![Page 19: Stabilizing SE Build - Selenium conf 2013](https://reader038.vdocuments.us/reader038/viewer/2022102814/54b69cce4a7959b5588b4593/html5/thumbnails/19.jpg)
Speed up tests
• Parallel more• Does it have to be Selenium?– Can this be a unit test?– Can it be headless?– Can you run Jasmine instead?
![Page 20: Stabilizing SE Build - Selenium conf 2013](https://reader038.vdocuments.us/reader038/viewer/2022102814/54b69cce4a7959b5588b4593/html5/thumbnails/20.jpg)
Black Hole Proxy
• Capture any external requests, return 200• Speeds tests up• Reliability goes up• em-proxy– https://gist.github.com/dimacus/5757573
![Page 21: Stabilizing SE Build - Selenium conf 2013](https://reader038.vdocuments.us/reader038/viewer/2022102814/54b69cce4a7959b5588b4593/html5/thumbnails/21.jpg)
Gem Cacher
• Cache any external dependencies locally• Don’t rely on rubygems.org to always be up!– github.com/dimacus/gem-cacher
• Local Maven repo
![Page 22: Stabilizing SE Build - Selenium conf 2013](https://reader038.vdocuments.us/reader038/viewer/2022102814/54b69cce4a7959b5588b4593/html5/thumbnails/22.jpg)
Credit on Grid idea
• David Burns• @AutomatedTester
![Page 23: Stabilizing SE Build - Selenium conf 2013](https://reader038.vdocuments.us/reader038/viewer/2022102814/54b69cce4a7959b5588b4593/html5/thumbnails/23.jpg)
Selenium Grid Extras
• github.com/groupon/Selenium-Grid-Extras
![Page 24: Stabilizing SE Build - Selenium conf 2013](https://reader038.vdocuments.us/reader038/viewer/2022102814/54b69cce4a7959b5588b4593/html5/thumbnails/24.jpg)
Basic Idea
• Help maintain the Environment to produce stable tests
• Turn Selenium Grid Nodes into dumb browsers
![Page 25: Stabilizing SE Build - Selenium conf 2013](https://reader038.vdocuments.us/reader038/viewer/2022102814/54b69cce4a7959b5588b4593/html5/thumbnails/25.jpg)
Control OS from tests
• Clean up old sessions• Kill browsers• Take full OS Screenshots• Move mouse• Reboot windows
![Page 26: Stabilizing SE Build - Selenium conf 2013](https://reader038.vdocuments.us/reader038/viewer/2022102814/54b69cce4a7959b5588b4593/html5/thumbnails/26.jpg)
Future plan
• Record videos• Share files with grid nodes• Spin up/down VMs on the fly• Image comparison• Auto update Selenium WebDriver• Black-hole proxy integration• Improve Grid API• Restart SE Node• Idle machine utilization
![Page 27: Stabilizing SE Build - Selenium conf 2013](https://reader038.vdocuments.us/reader038/viewer/2022102814/54b69cce4a7959b5588b4593/html5/thumbnails/27.jpg)
Need your help
• https://github.com/groupon/Selenium-Grid-Extras
![Page 28: Stabilizing SE Build - Selenium conf 2013](https://reader038.vdocuments.us/reader038/viewer/2022102814/54b69cce4a7959b5588b4593/html5/thumbnails/28.jpg)
Questions?
42!