the difference that makes the difference
Post on 30-Dec-2015
44 Views
Preview:
DESCRIPTION
TRANSCRIPT
The difference that makes the difference
Make your test automation project a success story
Magnus NilssonLemontree
Agenda
The Test automation concept– Success factors– Components– Test cases– Framework
Walkthrough of a test automation project– Analysis phase and the business case– Maintenance of an automated solution
Test automation - success factors
Develop functional components
Decouple in- and output data from the script
Central data storage
Development project – strategy and – center of
excellence
Combine components to test cases by actions as input data
Functionalcomponents
Test automation
In-/outdata
Central data
storage
Project
Test Automation
Concept
Test automation - building up the components
Functionality available in test – Development of the functionality
as it comes available
Requirement
Design and development
Test
Phase 1 Phase 2
Function 1
Function 2
Function 1
Function 2
Function 1
Func. 2
Each functionality is part of a component (or a component itself)
Func. 3
Func. 4
Test Automation
Concept
6
Test automation - building up the components
CRM System
Function 1: Create customer using CRM
CRMCustomer
7
Test automation - building up the components
http://www.webshop.se
Function 1: Create customer using Webshop
WebCustomer
Function 1: Create customer using mobile application
8
Test automation - building up the components
AppCustomer
Test automation – test cases
Login
Customer Account[new]
Inventory
Login Credit check
[create]
Subscription Order
[new]
Account [new]
Customer [open]
Billing
Test case 1:
Test case 2:
Test Automation
Concept
Test automation – test cases
Login
Customer
Inventory
Login Credit check
[create]
Subscription Order
[new]
Account [new]
Customer [open]
Billing
Test case 1:
Test case 2:
Account [new]
ver 2.0
ver 1.0
Account[new]
ver 1.0
Account [new]
ver 2.0
Account[new]
ver 2.0
Test Automation
Concept
1st generation script– Using automation tool for capture and replay only– Hardcoded data
2nd generation script– Code refactoring for maintainable scripts– Uses generic functions for iterations and error handling
3rd generation script– Data driven test (DDT) where dynamic data is taken from data table rows– Keyword driven test (KDT) where action words controls the actions– Use of driver scripts for executing multiple test cases– SME for test designs and requirement– More complex scripting by developers
Test automation - development of a framework
Framework
Development of a
Test automation - development of a framework
HP QuickTest Professional/Functional tester– Framework itself with
• test management connection ALM, • keyword driven action scripts with • data table,• function libraries,• Repositories, • BPT,• recovery scenario
– Generates code Library definitions, template of standard functions Structured data handling, input and output data
Repository
objects
Application GUI layer
List object
Text object
Table object
Object recognition
engine
Data table
M
O Business ID will be validated if provided
Out
N/A
CustomerID_Out
BusinessID
FamilyName
FirstName
In
In
In
M
N/A The created CRM customer ID
Family name
First name of customer
MOfficialName
In Private or Business
OutScriptstate_Out
Status of execution
LinkEntity_In
In O Entity linked from previous component
Function library A
Function library B
ALM
Test resources
Test plan
Test lab
Dashboard
Recovery scenario
Test action script
Framework
Development of a
Test automation - development of a framework
HP QuickTest Professional/Functional tester
Library definitions, template of standard functions– Application specific libraries– Standard actions in applications should be in application library– Component libraries enabling Configuration Management
• Branched Release Management for ST, AT and dev• Multiple checkout and merge• Only driver scripts
– Framework libraries handling framework and best practices Structured data handling, input and output data
Framework
Development of a
HP QuickTest Professional/Functional tester Library definitions, template of standard functions
Structured data handling, input and output data– Central data source– Decouple structure and data– Central structured data source
Test automation - development of a framework
Data table
Component A sheet
M
O Business ID will be validated if provided
Out
N/A
CustomerID_Out
BusinessID
FamilyName
FirstName
In
In
In
M
N/A
The created CRM customer ID
Family name
First name of customer
MOfficialName
In Private or Business
OutScriptstate_Out
Status of execution
LinkEntity_In
In O Entity linked from previous component
Component B sheet
M
O Business ID will be validated if provided
Out
N/A
CustomerID_Out
BusinessID
FamilyName
FirstName
In
In
In
M
N/A
The created CRM customer ID
Family name
First name of customer
MOfficialName
In Private or Business
OutScriptstate_Out
Status of execution
LinkEntity_In
In O Entity linked from previous component
CreateCustomer sheet
M
O Business ID will be validated if provided
Out
N/A
CustomerID_Out
BusinessID
FamilyName
FirstName
In
In
In
M
N/A
The created CRM customer ID
Family name
First name of customer
MOfficialName
In Private or Business
OutScriptstate_Out
Status of execution
LinkEntity_In
In O Entity linked from previous component
ALM
Test resources
Test plan
Test lab
DashboardData &Structure
Framework
Development of a
Function library A
Master DB
Data
Data &Structure
Test action script
The test automation projectPhase 1
Development
System test
Acceptance test
Week
Analysis
Detailed design
Requirement
Plan next phase
Documentation & Training
Phase 2
Detailed design
Development
…
Walkthrough
Project
Phase 0 The Business Case
Test automation project - business case
Why do we need a business case?– Test automation cost time and money– The automated solution must at least ensure the time spent in development, maintenance
and execution is less than a manual solution would cost to accomplish the same goal.
What is a test automation business case?– List of all costs and savings associated with test that possibly can be covered by test
automation.– Not all drivers are possible to calculate directly as quality. But quality can be expressed as an
estimation of the number of testing hours to ensure the desired level of quality.– Additional test automation drivers as repeated executions is calculated indirectly as increased
quality.
How can a business case be used?– Business case can be used to predict effectiveness as automation deliveries should follow the
calculated schedule.– ROI should be calculated at regular audits based on the business case.
Walkthrough
Project
Test automation project - business caseCost for manual regression tests
Cost for error handling
External and business costs for errors
ProductionAT
Number of errors found per year
Handling time per error (report, reproduce, administer, test data, close defect)
Number of FTE's for error handling per year
Error handling cost per hour
Annual error handling costs
1000
2
2
100
200000
Error source
500
4
2
100
200000
Total test case executions per cycle
Number of test cycles
Execution time per test, h
Execution cost per hour
10004
0.5
100
Annual cost for execution of regression test
200000
Grand total
Annual error correction costs for errors
Business costs for productions errors1)
0 100000
2
400000
Cost multiplier for Production Errors
1)Goodwill, compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test
1100000
Test automation project - Business case
Savings - Manual Test Labour
Savings - Error Reduction
Implementation Cost (Year 1)
Operational costs (running)
Description Reduction ratio
Annual reduction/cost
Acc reduction
100000
70000
70000
Phase 1
Phase 2
Phase 3
Total savings
270000
90000
180000
Year 1
Year 2
Year 3
10%
20%
30% 540000
90000
270000
Year 1 120000
160000
140000Year 2
Year 3
60%
70%
80%
120000
420000
260000
Year 1 proj support, SME
Year 2 TAC OP
Year 3 TAC OP
50000
60000
60000 Acc savings
Year 1
Year 2
Year 3
-80 000
260000
370000
180000
550000
-80 000
Walkthrough
Project
Test automation project - Business case
Walkthrough
Project
Technology2
1
1
Mixed
Yes
5%
1.5
2
3
Create
5
1
2
Maintenance
Reusability
Simple TC
Complex TC
Framework
Test data
Integrations
Selection Value
45
61
New
Implementation cost calculations– Initial estimates– Design estimates
CRM Customer
20
10
10
10
15
10 20
Create
Terminate
WS Customer
Create
App Customer
Create
Analyis&Design Development
20
Component/Functions
Business case – follow up short term
Analysis
Detailed design
Development
System test
Acceptance test
Plan next phase
Phase 1
Phase 2
Detailed design
Development
Week
DeliveredPlannedTest cases
Week
Walkthrough
Project
Business case – follow up long term
”Test automation strategy”
Description of the standard approach on how to use the tools Company specific set of rules on how automation should be done Generic solution that is not dependent on individuals Maintenance focused Encapsulated by test automation centre Enforced by framework Ensured by the automation project
But we need to know what we are testing!
We should not test manually what we
execute automatically!
Walkthrough
Project
Analysis Analysis
– Analysis based on existing documentation, test cases, use cases and requirements (static test review)– Provides the prioritization to an analysis structure template
Disposition– Decision making: if we go live what does and does not work?– What business processes are working and not working or not tested?
• Often details on number of test cases passed and failed on integration and system level is not vital• Test and defect status itself often only aids prioritizing of resources/focus
Structure– Common and clear reporting on a level that enables correct business decisions– Structured test design and coverage analysis– Will control the test automation deliveries– Used in test management to create common priorities for both manual and automated tests
Implementation– After the analysis and approved structure design, the structure may be implemented using
available tools in HP QC/ALM
Walkthrough
Project
Analysis structure - disposition
what is the system used for?– how is it done?• which are the requirements on process level?
»which data conditionsare covered by use cases?
Use case
Process
Requirement
Data condition 1
Data condition 2
Walkthrough
Project
Example
Customer
Web Self Care
New
Private
Business
Analysis structure - disposition
Requirements
Folder area Process Description Prio
Use cases/product
Customer Account Subscription
Create Terminate New New
Sales Web self care 1
Modify Cancel
3 1 10Web shop
Sales Customer operations
1 3 1 1 3 1 15CRM
Sales Device 2 3App
Sales Device 2 1 1 1Online
1
Product Inventory 1Product configurations
3
1
11
Walkthrough
Project
Requirement and component coverage
Requirements
Folder area
Requirement Process
Data conditionTest case Prio
Components
Customer Account Customer
Create Terminate New Create
SalesWeb self care R1.1
1
Modify Cancel
Create Customer
Sales Customer op R2.3.4
1
Sales 2Terminate customer
Sales 2Create bank account
1
CRM
Create Customer
Web Shop
X
Customer op R2.4.1
X
Customer op R4.1.1
X
X
SalesCustomer op R4.1.1
Create invoice account
X
Test case
TC23
TC12
TC1
TC45
TC6
Walkthrough
Project
X
Coverage analysis and status report; BC follow up
Customer
Customer operations
New
Terminate
New
2 1
0
3
2 1
1
No runNot
completedFailed PassedStatus Not covered
1
3 2 71
3Passed
1
1
1
3
Not covered
Not completed
Web self care
Mobile device
New Failed 0
1 3
2
2 3
2
Coverage analysis and status report, HP ALM
Coverage in ALM Test either
– manual,– automated or– business
componentWalkthrough
Project
Maintenance
Test cases created Test data prepared Result reporting arranged Press the button
Walkthrough
Project
Then what? X
Automation Engineer
Functional tester
Automation stakeholder
Subject Matter Expert
Test Automation Center
Project organisation - Use roles to enforce competence
Automation Engineer
Automation Analyst
Automation Solution Architect
Automation Project Lead
Walkthrough
Project
Project A
Automation stakeholder
Subject Matter Expert
Project B
Automation stakeholder
Subject Matter Expert
Project C
Automation stakeholder
Subject Matter Expert
Project organisation - Test automation center
Development
Execution
Maintenence
Walkthrough
Project
CommunicatingReportingPlanned activities
Test automation center
TAC Service Management
Management
Test management
Release management
Project planning
Change request
TAC Execution
TAC Maintenance
TAC Developmen
t
Request
Result
Operations
ST SIT AT
Project AProject B
Project C
Maintenance
Supplier
Defect and Issue Management
Resolution
Short about Lemontree
Swedish company established 1999
Awarded Gasell and Superföretag 2008-2012
Turnover +98 MSEK 2011
AAA-rated
About 100 employees, growing intensive
Owners are actively working in the company, no external capital
Offices in Stockholm and Oslo
QM trainee program since 2012
Focus areas
• Quality Management
• eGovernment
• Enterprise Solutions
Quality Management AQUA solutions
Thanks!Magnus Nilsson
Lemontree Solution Managermagnus.nilsson@lemontree.se
top related