test automation - business v2
TRANSCRIPT
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 1/36
Test AutomationTest Automation
Let¶s talk businessLet¶s talk business
Igor GershovichIgor GershovichConnected Testing, Inc.Connected Testing, Inc.www.connectedtesting.comwww.connectedtesting.comigershovich@[email protected]
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 2/36
Over view
Test Automation Myths
Return on Inv
estment (RO
I) Frameworks
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 3/36
Test Automation Myths
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 4/36
Myth #1 - Test Automation is
simple, that every tester can do it This myth is promoted by the tool sales people.
They are trying to promote the following testautomation process: Record the script Enhance the script by adding functions and data
driving Run the scripts Report results
Under the influence of this myth the QAmanager can proudly report: All our testers aredeveloping test automation.
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 5/36
Reality - Test automation is a
software development task Automation should be designed, developed and
tested
You need to have some kind of aprogramming background to implement testautomation. Test Automation is not as complexas C++/C#/Java development.
Test automation standards should be developed
Automated test components are assets thatshould be treated like application source code
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 6/36
Myth #2 ± Commercial test tools
are expensive Under the influence of this myth some
companies, especially the small ones:
Try to develop their own test automation tools
Use scripting languages like Perl and Ruby
Use shareware test tools
Do not consider test automation at all
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 7/36
Reality ± Commercial tools are
cheap Per seat license for the most expensive automation tool
is $8K
This tool will be used for 5 years.
Maintenance/Support fees are 20% of tool cost or $1,800 per year
The cost of this tool is $8K/5+$1,800 = $3,100 per year
The automation developer cost with overhead is $100K
per year The cost of this tool is just 3% of the person who uses it,
but productivity gain can be very significant
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 8/36
Commercial Tool Benefits: Customer support. Many of the open source tools come and go with
little to no support Most commercial tools are constantly being updated as technologies
change Most commercial tools usually have more functionality (QTP can test
various GUI applications: Web, .Net, Java, VB, C/C++,PowerBuilder, etc. vs. WATIR ± Web only)
Commercial tools usually have a large community of users, whichtranslates into better availability of qualified resources
Commercial tools require less advanced programming More test automation frameworks are available for commercial tools Commercial tools are integrated with Test Management tools which
makes reporting and execution much simpler.
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 9/36
Over 300 Test Tools are available
(http://www.softwareqatest.com) Load/Performance tools ± 54
Web Functional/Regression ± 60
Java Test tools - 48
Other Web tools ± 76
Which tool is right for you?
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 10/36
Absolutely Unscientific research:
Employer demand by tool expertise(May, 2008)
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 11/36
Dice search results across US (30 days)
Tool Search string Matches
QTP Qtp OR quicktest OR "quick test³ 613
Functional
Tester
"rational robot" OR "functional
tester´
118
SilkTest silktest OR "silk test´ 86
TestComplete testcomplete OR "test complete´ 26
TestPartner testpartner OR "test partner³ 19
VSTE VSTE AND test 45
WATIR watir 26
Selenium Selenium 108
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 12/36
De a e l
HP P
60
Open ou ce
I
2c osoft
4
uto ated
est o p ete
2
o pu a eestPar tner
2
orland
lk est
8
Iunct onal
ester
12
Open ource
elen u
10P
Funct onal ester
Silk est
est o plete
estPar tner
VSTE
WATI
Seleniu
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 13/36
Excerpt from IDC Report:
³Worldwide AutomatedSoftware Quality 2007-2011Forecast and 2006 Vendor
Shares«´
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 14/36
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 15/36
Think of yourself like a business Facts:
Average IT job tenure is about 2 years Competitive intensity is rising
What you should do: Invest your time/money in learning mainstream
automation Create a niche for yourself - add Configuration
Management, Performance Testing or ProjectManagement to your skillset Market yourself
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 16/36
Return On Investment (ROI)
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 17/36
Classic ROI Calculation
ROI = BENEFIT/COST
Automation Cost = Price Of HW + Price of SW + Development Cost +Maintenance Cost + Execution Cost
Manual Testing Cost = Development Cost + Maintenance Cost +Execution Cost
ROI = (Manual Testing Cost - Automation Cost)/Automation Cost
Looks right, Doesn¶t it?
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 18/36
Problems with Classic ROI Calculation
You can¶t compare Automated Testingand Manual Testing. They are not the
same and they provide differentinformation about the AUT.
You can¶t compare cost of multipleexecution of automated tests vs. manual
tests. You would never dream of executing that many test cases manually.
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 19/36
Automation Real RO
I ROI value IS NOT the value of Automationvs. Cost of executing these tests manually
Automation ROI value IS the benefit of this type of testing, and it can be:
Reducing Time to Market
Increased Test Efficiency (Productivity)
Increased Test Effectiveness
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 20/36
Reduced Time to Market Can get a greater market share
Makes people available to work onother projects
Higher margins, if no competitive
products are currently available
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 21/36
Productiv
ity and Effectiv
eness More testing gets done faster, increasing the odds of
finding defects
Defects found early have better chances of being fixed Manual Testers can concentrate on clever ways to
finding defects, instead of typing test inputs and verifyoutput.
About 7% of bug fixes create new bugs, sometimes inalready tested parts of the system. With automation youcan rerun tests for those modules. This almost never happens when testing is done manually.
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 22/36
RO
I summary Each project requires different types of
automation - there is often no easy
formula available to calculate ROI
Performing ROI calculation can help todetermine upfront what type of
automation, what level of skills, what toolswill be required.
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 23/36
Automation Frameworks
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 24/36
Test Automation Framework A Test Automation Framework is a set
of assumptions, concepts and tools that
provide support for Automated SoftwareTesting.
A correctly implemented Test AutomationFramework can further improve ROI byreducing the development andmaintenance costs.
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 25/36
Types of Test Automation
Frameworks Modular
Data-Driven
Keyword±Driven
Model-Based
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 26/36
Modular framework The Modular framework is the natural
progression derived from Record-and-Playback
The modular framework seeks to minimizerepetition of code by grouping similar actionsinto ³modules´ (e.g.: login)
Test Data is in a script or in an internal table
This is what tool sales people are promoting.
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 27/36
Data-Driven/Keyword Driven
These frameworks are similar in that thedata is separated from the test script
The script is just a "driver" or deli
verymechanism for the data.
The difference:
In keyword-driven testing, the navigation dataand test data are contained in the data source
In data-driven testing, only test data iscontained in the data source.
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 28/36
Model-Based testing
Model-based testing is software testingin which test cases are derived in whole or
in part from a model that describes some(usually functional) aspects of the system.
Model-based testing for complex software
systems is still an evolving field.
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 29/36
We¶ll be concentrating onKeyword-Driven testing since it
is the most beneficial frameworkfor large-scale test automation.
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 30/36
Advantages of Keyword-Driven
Test automation This Framework addresses the most common problem
with test automation: Automation Engineers do not have domain knowledge
and the End Users (Subject Matter Experts/TestEngineers) usually do not have automation expertise. When properly implemented and maintained, it presents
a superior ROI because each business event isdesigned, automated and maintained as a discreteentity.
Keywords can then be used to design test cases, but thedesign and automation overhead for the keyword hasalready been paid.
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 31/36
Advantages of Keyword-Driven
Test automation Reduced the cost and time spent maintaining
and updating tests The modular structure of keyword-driven testing
means that new tests can easily be created frompre-existing modules
The test team is capable of entirely automatingtests, even without programming knowledge
Can be easily modified to use with different testtool Reusability across different projects
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 32/36
Classic Keyword-Driven
ExampleObject Action Data
Textfield (username) Enter Text <username>
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 33/36
Recent Success with Keyword-Driven
approach Estimate using Modular Framework: 1 year
Estimate with Keyword-Driven Framework: 6 months
Project was completed in 4 months by 1 person, 2 monthahead of schedule.
This was the first project for the automation developer utilizing the Keyword-Driven Framework.
170 components were developed (1 component per Webpage).
109 Regression Test cases were automated.
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 34/36
Example of Test Data for Keyword-Driven testautomation developed by Connected Testing Inc.
for TestComplete and QTP
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 35/36
Resources
³Everything you want to know about Test Automation«´ by Brian LeSuer
³Manager¶s Guide to GUI Test Automation´ by Yury Makedonov ³The ROI of Test Automation´ by Michael Kelly ³What is my ROI?´ by mVerify Corporation SQA Forums ³An Over view of Test Automation Frameworks´ by Nathaniel
Ritmeyer ³The benefits of keyword-based software test automation´ by David
W. Johnson Wikipedia
8/6/2019 Test Automation - Business v2
http://slidepdf.com/reader/full/test-automation-business-v2 36/36
Igor Gershovich
Company: Connected Testing, Inc.
E-Mail: [email protected]
Phone: 720-933-9395
Website: www.connectedtesting.com