continuous test automation

Continuous Test Automation

Test Automation Framework using Jenkins CI to drive the execution and

present the results

[email protected] 1Feb 2015

About

• This presentation details the thinking behind how I managed to integrate Test Automation with Continuous Integration for a client in the Broadcasting technology sector.


Take away message

“Test automation alone is useful but test automation hooked up with a Continuous

Integration Platform is damn powerful”Viresh Doshi – February 2015

Feb 2015 [email protected] 3

About Viresh Doshi

• Test Specialist with over 15 years industry experience.

• Started out testing Air Traffic Management Systems for Lockheed Martin.

• Gained Test Consultancy working for one of the first pure play test consultancies.

• Now help businesses/projects mature the QA, Agile and Test process.


The problems

• The client was only manually testing with some adhoc automated unit testing.

• The team was transforming to Agile.

• Team was using Jenkins CI for building software.

• The test environments were not controlled.

• Multiple teams from the UK, USA and Canada updating the main code base and difficult to pinpoint regressions.


The biggest problem


• The software gets to the client and…

– Software does not install.

– The software is missing a database change.

– The client can not login

– Basic functionality is broken

Pressure to Deliver


Hold on!

• Jenkins is a build tool and not a testing tool.


The system

Made up of three key componentsTest

automation tool

Jenkins CI

Jenkins Plugins

GIT –source code Control


Test automation tool

• This is any test automation tool that best compliments the system under test e.g.

– Test Complete

– Selenium

– jMeter

– Quick Test Professional

– Home Grown Code and Tools


GIT source control

• GIT is open source

• Easy to use and manage

• Client tools allow for easy usage

• Integrates with Jenkins CI

• Web management system

• Ideal choice for management of all test configurable items and test code.


What does it look like?


In more detail?

The radiator view is what we stare at daily!

Each block is a labeled “job” that does something.

Green indicates success and Red/Yellow indicates failure or issues.

Running progress is shown. Job run time and version is

shown. Red blocks need investigating

immediately.


What “things” does it do?

• Smoke tests – functional test coverage of the key functions of the system

• End to End business scenarios

• New features

• Software installation

• Configuration

• Monitoring of servers , database server and log files


Test Starting point

• A new production development build is the initial trigger point

• That build is copied across to the test automation system and then installed and then the suite of automation tests are run and reported.


System Vision and Values

• Continuous installation

• Continuous testing

• Continuous reporting

• Easy to manage

• Source Controlled

• Open Source Technology

• Scalable


JENKINS

Continuous Integration Platform


About

• This section contains information on the Jenkins plugins used that achieve Continuous Test Automation.


Jenkins

What is Jenkins?

• A CI Build Platform that allows the user to create build process jobs that do “things”.

• Jenkins Plugins allow the integration and ability to enrich the “things” being done.

• When those “things” are not successful then a red fail is reported and it’s possible to drill down further and investigate problems.


Concepts

• Automate Repetitive tasks

• Report Pass and Fails

• Provide easy access to Log files

• Allow for Parameters

• Integration to existing systems

• Establish a Workspace


Plugin: Build Monitor

What does it do?

• Provides a visual “radiator” view of configured jobs.

Why?

• The view shows the jobs key features like: how long it took to execute, running state, pass/fail and build number.


Plugin: Build Name Setter

What does it do?

• Allows for a user set a build name

Why?

• The standard build name is just an incremented number.


Plugin: Build Pipeline

What does it do?

• Provides a visual view of upstream and downstream connected jobs.

Why?

• Builds and jobs can get complicated, so it is useful to have a visual view of connected jobs.


Plugin: Config slicing

What does it do?

• Allows multiple projects settings to be adjusted conveniently.

Why?

• A great way to manage configuration settings when you have 100s of similar Jenkins jobs.


Plugin: Copy Artifact

What does it do?

• Allows artifacts to be copied from other build projects

Why?

• Useful for taking successful executables from other projects for usage.


Plugin: Green Balls

What does it do?

• Simply shows a green for Pass and Red for Fail

Why?

• Jenkins automatically uses a colour blind scheme – orange and yellow.


Plugin: GIT Plugin

What does it do?

• Integrates into GIT source code control.

Why?

• Needed to pull the latest code changes from source control.


Plugin: HTML Publisher

What does it do?

• A HTML report is published per run

Why?

• It was used to publish the HTML execution results as created by Test Complete Automation Tool


Plugin: PowerShell

What does it do?

• Integrates the ability to run PowerShell scripts.

Why?

• PowerShell is a very powerful scripting language that adds endless possibilities for windows based servers.


Plugin: jUnit

What does it do?

• A nice graph with a pass/fail report in jUnitstyle is presented per run.

Why?

• A test is not a test without a pass/fail result. Any failures can also display stack traces or error screenshots.


Plugin: MSBuild

What does it do?

• Allows MSBuild to build .NET projects

Why?

• The executable isbuilt consistently using Jenkins instead of adhocly on a developer’s machine.


Plugin: Sidebar Link

What does it do?

• Simply adds a HTML anchor link to the Jenkins Sidebar

Why?

• To simplify our life!


Plugin: Workspace Cleanup

What does it do?

• Cleans up the workspace before a job is executed.

Why?

• Only that run’s current artifacts are stored.


CONCLUSION

Lessons learned and tips


Tips

• Try and limit the full suite of tests to 2 hours execution time.

• Too many builds provided by the developers –Limit the execution to twice a day.

• Failures could be with problems with the tests and not the code.

• The system needs to be tested. Watch the test execution and double check that it is doing what you expect.


Next Steps

• Introduce Maven or equivalent to manage dependencies.

• Introduce a first time install test on virtualized servers.

• Introduce Continuous Delivery


Thank You

I am looking for a new challenge!

[email protected]

07970 291828

UK


mailto:[email protected]

continuous test automation

Business