Building better websites with automated quality assurance

Cheaper, Faster, more Reliable

Agenda

• Introductions• Drupal SimpleTest• Drupal.org usage• Drupal.org Future• Business Value of Quality • Assurance• Examiner Development• Examiner Development -

Selenium• Examiner Benefits

• Testing in the Wild• Contributed Module • Development• Tools for Site Owners• ReviewDriven.com• Contributed Back• Demo

Introductions

Jimmy Berry is the lead developer for the automated quality assurance platform used in the Drupal core development process. Jimmy works for Examiner.com where he develops automated quality assurance tools for examiner.com's migration to Drupal 7. Kieran Lal is the project manager for the quality assurance platform. Kieran is the Drupal community adventure guide for Acquia where he helps organizations understand the benefits of Drupal and Acquia's products to be successful with their Drupal sites.

Drupal SimpleTest

• Based on SimpleTest.org API

• Provides seemless integration with Drupal

• Extensive Drupal core suiteo Provides many exampleso ~19,000 assertions 

• Introduction - http://drupal.org/simpletest

• Primarily used for "functional" testingo Simulate usero Start from scratch

environment

Drupal.org Usage

Number of tests reviewed 24,840

Number of reviews performed 132,773

Average number of times a test is reviewed 5

* http://qa.drupal.org/pifr/statistics

Business Value of Quality Assurance

• Confidence to make large changes and refactor as needed.• Can prevent bugs from reoccurring.• Provides the confidence to know the system is ready to

deploy.• Removes the need to for large scale human testing with

every change set.

Drupal.org Future

• Drupal 8o Clean code to perfection early in cycle

Combination of manual cleanup and automated tools like the (PHP) Grammar Parser.

o Remove any false positives from Codero Enable Coder review on all patches

• Code coverage reports on commit• Additional automated reviews

o Textual analysiso Security reviews

Examiner Development

• On commito Coder reviews - e-mail results to developers

• Dailyo Pre-configured tests against shared dev database

Import clone of dev database. Additional copies made during testing

o Atomic tests that require no database Run against an external server such as production or

staging Ensure that rolled out code is functioning properly Could be used for Drupal.org

Examiner Development - Selenium

• Selenium IDEo Workflow

Record test "marcos" Export as PHP Selenium output Convert using simpletest_selenium (d.o project)

o Benefits Distribute test writing workload by allowing non-tech

savy users to record tests Lower initial learning curve requird to include testing

into a development process

Examiner Benefits

• Code reviewo Makes it code reviews second natureo Encourages developers to clean up code

• Testingo Lighten manual test loado Still fairly early in development cycle to access benefit

Testing in the Wild

• Project Issue File Request (PIFR)o ClassicGraphicso Drupal.orgo Examiner

• Non-PIFRo Consumersearch.como Economist - Hudsono myLifetime - Script

Contributed Module Development

• SimpleTest documentation• 365 contributed modules that already have • 46 contributed modules on http://qa.drupal.org • Assistance in writing tests • Infrastructure issues

o Finish info file parsing update* Allow any project maintainer to enable testing Provides mechanism to specify dependency versions

o Interface improvements to enhancing integration Links to related review results Aggregation of summary results on project pages Coverage ratings for projects Project specific filters and settings

http://drupal.org/project/project_issue_file_review http://drupal.org/project/project_issue_file_test* http://drupal.org/node/102102

Tools for Site Owners

• SimpleTesto Works with Drupal 6 & 7o Clone of a database

Drupal 6 module - simpletest_clone Drupal 7 patch - node #666956

o Atomic tests – node #758662• PIFR 2

o Automate running of tests and reviewso Used with trigger script or project management

integrationo Works with any combinations of version control systems

Currently supported: bzr, cvs, git, svn

ReviewDriven.com

• QA as a serviceo Add site repositoryo Schedule reviewso Define triggerso Configure notifications

• Make QA as easy as possible to improve the Drupal ecosystem as a whole 

• Focus on enhancing tools available to Drupal developers • Public beta coming soon

Contributing Back

• Write tests for core• Write tests for contrib and custom modules• Run tests for your modules • Run tests for your site• Run tests for your development process• Write the testing framework

o Drupal.org Project module integration (PIFT)o qa.drupal.org (PIFR)o SimpleTest testing framework

Drupal core 6.x-2.x backport

Demo

class DrupalRedesignTestCase extends DrupalStageTestCase { [...]

  protected function setUp() {    $this->stagingUrl = 'http://staging6.drupal.org';    $this->httpauth_credentials = 'drupal:drupal';

    parent::setUp();  }

  protected function testFoo() {    $this->drupalGet('about');    $this->assertText('Documentation');  }}

* http://drupal.org/node/758662

QA Sprint Drupalcon SF • Sunday 9AM, April 18th•     Coder Lounge• Learn how to write tests for Drupal core• Learn how to contribute tests to modules you use • Learn how to write tests for your contrib modules• Learn how to run simpletest for your custom site

Contributing to the QA project• Add features to the QA platform for core

– E.g. Load testing

• Add features that your consulting company needs for clients

• Add features that your company needs for it’s QA process

• Add features to integrate with your development and test workflow

• Contact: http://qa.drupal.org/contact

Appendix

 

How to Reach Our Audience

• Developers - front pageo Coder reviewso Involve most popular module developers

• Assemble knowledgeable QA team o [Dave Reid]

• [List of people who have corresponeded]• [List of projects with SimpleTests]• Site owners

o Maintainable benefit that will increase in value Additional plugins and services

Schedule for Webinar and Sprint

• Announce Webinar and Sprint on March 22nd• Practice run on the March 25th • Webinar on March 31st• Sprint on April 18th

Business Value of Quality Assurance

• Confidence to make large changes and refactor as needed.• Frees developers to focus on features and functionality

instead of ensuring that functionality is not broken.• Helps prevent changes from having a cascade effect.• Can prevent bugs from reoccurring.• Provides the confidence to know the system is ready to

deploy.• Removes the need to for large scale human testing with

every change set.• Provides a consistent mechanism to ensure the system is in

a certain level of stability.• Ensures that testing is not neglected, but instead occurs on

a regular basis.• Provides a way to ensure that configuration changes

specific to one site do not have unwanted effects.