css regression tests

CSS RegressionTests

CSS Testing: 4 Types1. Syntax

Did you literally make mistakes in your CSS?

2. Project

Does this CSS meet the standards you set?

3. Reference

Does the rendered page look as you expect it to? Cross-browser as well.

4. Regression

After actions are performed, does the page look as you expect it to?

Credits: CSS Tricks: https://css-tricks.com/automatic-css-testing/

Breaking css is easy,testing it is hard.

1. Headless BrowsersA headless browser is a web browser without a graphical user interface.

They provide automated control of a web page in an environment similar to popular web browsers, but are executed via a command line interface or using network communication.

PhantomJS - WebKit based (Safari, Chrome)

SlimerJS - Gecko based (Firefox)

* SlimerJS is not yet truly headless. You see windows and it needs a graphical environment. You can use a tool like xvfb on a Linux/MacOS system to have this headless feature and to execute SlimerJS on Linux boxes that do not have xorg installed.

2. Headless Browsers AutomationProvides navigation scripting (scenarios) & testing for Headless Browsers.

CasperJS - Navigation scripting & testing for PhantomJS and SlimerJS

3. Image Analysis and Comparison

ResembleJS - JavaScript / HTML5 library for making image comparisons.

Case Study: OneStudio

BackStopJS (v1.0.3)

Wrapper for:

1. PhantomJS or SlimerJS (Headless Browsers)

2. CasperJS (navigation scripting & testing for Headless Browsers)

3. ResembleJS (Image analysis and comparison)

Configuration: Viewports{ "viewports": [ { "name": "phone-portrait-6s", "width": 375, "height": 480 }, ... ], "scenarios": [ ... ], "paths": { ... }, "engine": "phantomjs", "report": ["browser"]}

Configuration: Scenarios{ "scenarios": [ ... { "label": "[Not Logged-in] Home / Studios", "url": "http://one.dev", "removeSelectors": [ ".studio-list-view .feed-entry:nth-child(n+3)", ".footer", "#navbar" ], "selectors": [ "body" ], "delay": 300 }, ... ]}

Configuration: Repeatable Components{ "scenarios": [ ... { "label": "[Not Logged-in] Navbar", "url": "http://one.dev", "selectors": [ "#navbar" ], "delay": 300 }, ... ]}

Configuration: Scenarios with CasperJS{ "scenarios": [ ... { "label": "[Not Logged-in] Login Popup", "url": "http://one.dev", "removeSelectors": [ "#view-home", ".footer" ], "selectors": [ "body" ], "delay": 300, "onReadyScript": "trigger-login-modal.js" }, ... ]}

trigger-login-modal.js:

module.exports = function(casper, scenario) { casper.evaluate(function(){

// Show Login popup $('#login--modal"]').trigger('click');

});

// Wait a bit, to complete the CSS transition casper.wait(500);};

Summary: One Studio Regression TestsViewports: 6 (phone-portrait-5s, phone-portrait-6s, tablet-portrait, tablet-portrait-ipad, tablet-landscape-ipad, laptop)

Scenarios: 25

gulp reference - Creates 150 reference screenshots. ( ~5 min)

gulp test - Creates 150 test screenshots (~5 min)

and compares them with the reference screenshots (~1 min)

Total time consumed: 11min.

Workflowgit checkout developgulp reference

git checkout -b feature/button-styles… implement it… make a Pull Request

gulp testSee results → → → → → → → → →

Merge itlike a boss! :)

ConclusionsLearning curve: 1 weekend.

For smaller projects: configuration efforts are too much.

Screenshots are not always all right. I had some issues with:

- Iframes: https://github.com/garris/BackstopJS/issues/116

- Position: fixed elements, some absolute positioned elements

- JavaScript-related DOM visual changes

Tips from my experience:

- Try to have reasonable amount of ViewPorts

- Separate repeating components (header, footer, menu) on single scenarios

- CLI tool for reporting (for CI)

AlternativesPhantomCSS: Require writing long form CasperJS tests - great for testing complex UI interactions, but clumsy for more simple applications like static CMS templates or lots and lots of app states at different screen sizes. Uses PhantomJS / SlimerJS, CasperJS and Resemble.js too. https://github.com/Huddle/PhantomCSS

Wraith (by BBC-News). Uses PhantomJS / SlimerJS, CasperJS and ImageMagick. https://github.com/bbc-news/wraith

Succss (new). Uses PhantomJS / SlimerJS, CasperJS, Imagediff or ResembleJS. http://succss.ifzenelse.net/

Good ReadsAutomating CSS Regression Testing with BackStopJS:

https://css-tricks.com/automating-css-regression-testing/ and http://blog.bartos.me/css-regression-testing/

Visual Regression Testing with PhantomCSS: https://css-tricks.com/visual-regression-testing-with-phantomcss/

Using Wraith for Visual Regression Testing: https://pantheon.io/docs/guides/visual-diff-with-wraith/

Lullabot Regression QA: https://www.lullabot.com/articles/css-regression-testing-with-resemblejs

How To Create Your Own Front-End Website Testing Plan: https://www.smashingmagazine.com/2014/11/how-to-create-your-own-front-end-website-testing-plan/

Visual Test-Driven Development For Responsive Interface Design (Galen Framework): https://www.smashingmagazine.com/2015/04/visual-test-driven-development-responsive-interface-design/

SiteEffect.io Startup (in private beta): http://siteeffect.io/

Thank you. :)