selenium at mozilla: an essential element to our success
If you can't read please download the document
TRANSCRIPT
Selenium at Mozilla: An Essential Element to our Success
Stephen Donner
Web QA
March 25, 2013
Overview:
History
2013 Scale
Infrastructure
Custom Plugins/Configs
Lessons Learned from the Firefox 4 Era
Along Came Continuous Deployment
Community Collaboration
Our Key Takeaways
History, part 1
1st IDE tests in 2007; written by me for the Rock Your Firefox Facebook application
Started in earnest in May 2010Custom test-runner (w/concurrency)
1 Mac Mini
In November, same year, still only 2 suites (addons.mozilla.org and support.mozilla.org); same stack and setup
And, still:No POM
No screenshots
No Selenium 2 / WebDriverWhich meant no mobile...
History, part 2
October 2010, we hired David Burns, and really started maturing our page-object model
Around June 2011 we started migrating RC WebDriver
August 15, 2011, got full HTML reports
2013 scale
13 projects w/WebDriver automation
August 16, 2012, we officially dropped RC support2 with mobile support
Environments:Dev/trunk/master (synonymous)
Staging
Production
Infrastructure
Hardware:7 Mac Minis4 VMs
Browsers/OSs:Latest Firefox + Beta + Aurora, Opera, Chrome, Safari, IE 6 10
Mac OS X + Windows XP/7
Mobile (through Sauce Labs)
Custom Plugins/Configs
pytest-mozwebqaOur mainstay integrates with py.test (test runner), creates HTML reports, Sauce Labs integration, flexibility, test-data safety
moz-grid-configQuickly deploy new Selenium and browser versions to our local Grid
pytest-rerunfailuresComes close to solving staging-server/WebDriver intermittent test failures
(Quick) Infra Demo
Lessons learned from the Firefox 4 era
Page object model really helped us scale and maintain
Selenium+ - augmented what we already did well with WebDriver
Re-usable templates (for style-guide/community)
Destructive vs. non-destructiveOtherwise known as don't spam addons.mozilla.org or any other site in production, ever again
And Along Came Continuous Deployment...
Shock + fear, which segued into...Risk-assessment
Shift in where/how; not what coverage
Healthy, reset view of Selenium's role in Web testing @ Mozilla
Community Collaboration
Always striving to anticipate and provide for an answer to what can I work on?
Open GitHub Issues for both paid and community contributors
Heavy community participation, and we'd love to see more!
RedHat and some startups have used pytest-mozwebqa
Fork our repositories
Our key takeaways
Use POM if it makes sense for your apps/websites
Strive for consistency wherever and whenever possible/practical
Be tenacious in evangelizing + improving your tests' value
Look at readying for and/or doing Continuous Delivery, if you can
Focus on your core strengths
Don't test everything just because you can
Reach out to us!
Please, ask lots of questions; likewise, make suggestions, compare notes, etc.
[email protected] (mailing list)
#mozwebqa on irc.mozilla.org
Homepage: https://quality.mozilla.org/teams/web-qa/
Text