test automation success: choosing the right people & process
DESCRIPTION
Test Automation Success: Choosing the Right People & Process. Kiran Pyneni, Automation Manager Aetna, Inc. Agenda. Company Background. Automation Team. Automation Objectives. Our Journey…. Strategy Team. Delivery Team. Benefits of Teams Strategy. Automation Lifecycle. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Test Automation Success: Choosing the Right People & Process](https://reader036.vdocuments.us/reader036/viewer/2022070419/56815a6f550346895dc7d1cd/html5/thumbnails/1.jpg)
Test Automation Success: Choosing the Right People & Process
Kiran Pyneni, Automation ManagerAetna, Inc.
![Page 2: Test Automation Success: Choosing the Right People & Process](https://reader036.vdocuments.us/reader036/viewer/2022070419/56815a6f550346895dc7d1cd/html5/thumbnails/2.jpg)
Agenda
Automation Lifecycle
Automation Team
Metrics Of Our Success
Questions
Automation Coverage, Efforts Automated, Utilization & Cost Avoidance
Company Background
Strategy Team
Delivery Team
Benefits of Teams Strategy
Benefits of Process Integration
Our Journey…
Automation Objectives
![Page 3: Test Automation Success: Choosing the Right People & Process](https://reader036.vdocuments.us/reader036/viewer/2022070419/56815a6f550346895dc7d1cd/html5/thumbnails/3.jpg)
Aetna Background• Aetna is one of the nation's leading diversified health care benefits
companies, serving members with information and resources to help them make better informed decisions about their health care.
• Products and management Programs: – Aetna offers a broad range of insurance and employee benefits products. – The first national, full-service health insurer to offer a consumer-directed health
plan, Aetna continues to lead the way with its Aetna HealthFund2 line of products, including HSA, HRA and RRA options.
– Aetna offers a wide array of programs and services that help control rising employee benefits costs while striving to improve the quality of health care, such as case management; disease and patient safety programs; integrated medical, dental, pharmaceutical, behavioral health and disability information.
– Aetna provides members with access to convenient tools and easy-to-understand information that can help them make better-informed decisions about their health and financial wellbeing.
3
![Page 4: Test Automation Success: Choosing the Right People & Process](https://reader036.vdocuments.us/reader036/viewer/2022070419/56815a6f550346895dc7d1cd/html5/thumbnails/4.jpg)
Automation Team Improves efficiencies and quality at our organization by automating
consistent and repeatable manual tasks using industry standard technologies / tools / frameworks.
AutomationTeam
Assessments & DesignsReports & Metrics
Standards & Best Practices
Automation Development &
Maintenance
![Page 5: Test Automation Success: Choosing the Right People & Process](https://reader036.vdocuments.us/reader036/viewer/2022070419/56815a6f550346895dc7d1cd/html5/thumbnails/5.jpg)
Automation Objectives
AutomationObjectives
Automating Right Tasks
Sustaining and Growing Automation Suite
Utilization of existing automation
![Page 6: Test Automation Success: Choosing the Right People & Process](https://reader036.vdocuments.us/reader036/viewer/2022070419/56815a6f550346895dc7d1cd/html5/thumbnails/6.jpg)
Our Journey…
• Benefits– Easy identification of
automation opportunities– Higher collaboration between
application & automation teams• Challenges
– Automation development has become expensive
– Usage of different tools & solutions
– Standards & Best practices are not shared across applications
• Benefits– Standardization of automation tool
set at enterprise level– Sharing of coding best practices,
templates and standards– Cost of automation development
is lowered• Challenges
– Slower automation coverage progression
– Lower automated scripts utilization
– Funding Challenges
Decentralized Approach Centralized Approach
![Page 7: Test Automation Success: Choosing the Right People & Process](https://reader036.vdocuments.us/reader036/viewer/2022070419/56815a6f550346895dc7d1cd/html5/thumbnails/7.jpg)
Our Journey Cont’d. – Challenges & Solutions
Challenges• Identification of automation opportunities
• Prioritization of automation projects based on ROI
• Cost of automation
• Implementation of optimized designs
Solution• Structuring automation as strategy &
delivery team
Challenges• Early completion of maintenance for
automation scripts
• Funding for automation projects
• Collaboration between automation & business teams
Solution• Create automation life cycle process
and integrate with company's overall software development life cycle
![Page 8: Test Automation Success: Choosing the Right People & Process](https://reader036.vdocuments.us/reader036/viewer/2022070419/56815a6f550346895dc7d1cd/html5/thumbnails/8.jpg)
Automation Team Structure
Automation Team
Automation Strategy Team
AutomationDelivery Team
![Page 9: Test Automation Success: Choosing the Right People & Process](https://reader036.vdocuments.us/reader036/viewer/2022070419/56815a6f550346895dc7d1cd/html5/thumbnails/9.jpg)
Automation Strategy Team
• Definition: – To provide automation strategic direction and identify
potential new automation opportunities
• Characteristics:– Project management experience– Various automation frameworks/tools knowledge– Ability to manage multiple projects in parallel– Very strong written and verbal communication skills– Very strong leadership and analytical skills
• Activities:– Process & Best Practices– Assessment & Strategies– Design Solutions & Techniques– Estimations & Planning– ROI Cost/Benefit Analysis– Reports & Metrics– Tool Technical Support & Analysis
![Page 10: Test Automation Success: Choosing the Right People & Process](https://reader036.vdocuments.us/reader036/viewer/2022070419/56815a6f550346895dc7d1cd/html5/thumbnails/10.jpg)
Automation Delivery Team
• Definition: – To develop, deliver, and maintain automation solutions
• Characteristics:– Understanding of programming principles and best
practices– Knowledge of various automation tools– Knowledge of multiple programming languages– Strong communication skills
• Activities:– Scripts Development– Scripts Maintenance– Coding Standards– Scripts Planning & Execution – Day to Day interactions w/End User
![Page 11: Test Automation Success: Choosing the Right People & Process](https://reader036.vdocuments.us/reader036/viewer/2022070419/56815a6f550346895dc7d1cd/html5/thumbnails/11.jpg)
Benefits of Teams Strategy
Benefits
• Easy identification of automation opportunities• Prioritization of automation projects based on ROI• Improved automation design• Optimized the cost of automation• Clearly defined roles & responsibilities• Better communication process• Optimized automation project planning
![Page 12: Test Automation Success: Choosing the Right People & Process](https://reader036.vdocuments.us/reader036/viewer/2022070419/56815a6f550346895dc7d1cd/html5/thumbnails/12.jpg)
• Preliminary Estimations• Automation Test Strategy• Business Case - ROI• Selected automation tool• Detailed Estimations• Go, No Go
• Automation Project Plan• Framework documentation –
Re-usable components, Portability, test data grid design, etc.
• Business Process Test (BPT) components
• Automation Scripts• Test Data
• Tested automation scripts
• Automated test execution results
• Lessons Learned• Sign Off
G1 G2 G3
Automation Evaluation
• Understand functionality of the project / application
• Understand testing processes
• Understand target environments
• Understand test data details
• Verify automation tool compatibility
• Identify Automation opportunities
Phas
esAc
tiviti
esDe
liver
able
s
Automation Preparation
• Identify the resources needed.
• Understand the requirements with the help application demos.
• Create automation framework / design
• Identify timelines and milestones
• Training required for automation tools
• Tool installation if required
• Access & Environment Setup
Automation Generation
• Create Business Process Test (BPT) components
• Parameterize test data in BPT components
• Review components for quality and correctness
• String the BPT components together to create a test script
• Develop automation scripts for BPT components
• Identify test data to conduct unit & acceptance testing
• Unit testing of automation
• Deliver automation to application team
• Execute automation by application team
• Validation of ROI
• Retrospection
• Sign off
Execution & Acceptance
Automation Lifecycle
![Page 13: Test Automation Success: Choosing the Right People & Process](https://reader036.vdocuments.us/reader036/viewer/2022070419/56815a6f550346895dc7d1cd/html5/thumbnails/13.jpg)
Team Collaboration
Automation
EvaluationAutomation
Preparation
Automation
Generation
Execution
& Acceptance
Automation Strategy Team
AutomationDelivery Team
![Page 14: Test Automation Success: Choosing the Right People & Process](https://reader036.vdocuments.us/reader036/viewer/2022070419/56815a6f550346895dc7d1cd/html5/thumbnails/14.jpg)
Automation ROI Calculator
![Page 15: Test Automation Success: Choosing the Right People & Process](https://reader036.vdocuments.us/reader036/viewer/2022070419/56815a6f550346895dc7d1cd/html5/thumbnails/15.jpg)
Automation ROI Calculator
![Page 16: Test Automation Success: Choosing the Right People & Process](https://reader036.vdocuments.us/reader036/viewer/2022070419/56815a6f550346895dc7d1cd/html5/thumbnails/16.jpg)
Integration of Automation Lifecycle &Software Development Lifecycle
Automation Evaluation Feasibility/Analysis
Automation Preparation Design
Automation Generation Implementation
Execution & Acceptance
Testing & Maintenance
![Page 17: Test Automation Success: Choosing the Right People & Process](https://reader036.vdocuments.us/reader036/viewer/2022070419/56815a6f550346895dc7d1cd/html5/thumbnails/17.jpg)
Benefits of Process Integration
Benefits
• Improved understanding on automation process across the entire organization
• Better collaboration between business and automation teams
• Funding challenges are removed• Improved on-time delivery of automation scripts• Helped to promote the automation across entire
organization• Formal escalation process developed• Initialized the creation of automation reports & metrics
![Page 18: Test Automation Success: Choosing the Right People & Process](https://reader036.vdocuments.us/reader036/viewer/2022070419/56815a6f550346895dc7d1cd/html5/thumbnails/18.jpg)
Automation Coverage – Regression Testing
42%Q1-2009
45%Q2-2009
47%Q3-2009
52%Q4-2009
35%Year-2008
![Page 19: Test Automation Success: Choosing the Right People & Process](https://reader036.vdocuments.us/reader036/viewer/2022070419/56815a6f550346895dc7d1cd/html5/thumbnails/19.jpg)
Manual Testing Efforts Automated
7,047 hrs
8,570 hrs
10,200 hrs
11,750 hrs
13,803 hrs
Year As of 2008 As of 2009 2009 only effortsTotal Efforts Automated 7047 hrs 13803 hrs 6756 hrs
Year-2008 Q1-2009 Q2-2009 Q3-2009 Q4-2009
![Page 20: Test Automation Success: Choosing the Right People & Process](https://reader036.vdocuments.us/reader036/viewer/2022070419/56815a6f550346895dc7d1cd/html5/thumbnails/20.jpg)
Utilization & Cost Avoidance
Q1-2009 Q2-2009 Q3-2009 Q4-2009
Cost Avoidance
7,200 hrs
10,810 hrs9,618 hrs 9,867 hrs
Q1-2009 Q2-2009 Q3-2009 Q4-2009
Utilization
45%
Year-2008
87%
150% 134% 130%
![Page 21: Test Automation Success: Choosing the Right People & Process](https://reader036.vdocuments.us/reader036/viewer/2022070419/56815a6f550346895dc7d1cd/html5/thumbnails/21.jpg)
Savings Vs. Cost Avoidance
• Cost Savings– Efforts difference between manual and automation testing– Manual (Not automation) execution frequency– Produced by automation strategy team after assessment– Automation ROI Calculator
• Cost Avoidance– Test coverage difference between manual & automation testing– Efforts difference between manual and automation testing– Automation execution frequency– Generated from Quality Center after release implementation
Helped to improve the credibility of automation metrics significantly
![Page 22: Test Automation Success: Choosing the Right People & Process](https://reader036.vdocuments.us/reader036/viewer/2022070419/56815a6f550346895dc7d1cd/html5/thumbnails/22.jpg)
Automation Metrics – Best Practices
Automation is a way of Testing and process followed for manual testing can be applied
• Best Practices followed in Test Management Tool – Quality Center– Define automation work flow
• Regression Testing identifier• Automation Assessment status• Automation Development status
– Store automation inventory – Manage automated test scripts execution
![Page 23: Test Automation Success: Choosing the Right People & Process](https://reader036.vdocuments.us/reader036/viewer/2022070419/56815a6f550346895dc7d1cd/html5/thumbnails/23.jpg)
Questions