BIO PRESENTATION

International Conference On Software Test Analysis

And Review May 14-18, 2007 Orlando, FL USA

W8

5/16/2007 1:45:00 PM

"A PROCESS FOR SCOPING AND

PLANNING TEST AUTOMATION

PROJECTS"

Jennifer Seale Nationwide Insurance

Jennifer Seale Jennifer Seale holds Master’s degrees in English and Computer Science as well as active CSTE, PMP, and Mercury CPC-QTP designations. She has seven years of experience in the software industry as a developer, tester, test coordinator, and test automation solution architect. Jennifer is a top-rated adjunct faculty member at Franklin University and has taught a variety of undergraduate programming classes over the past four years. For the past two years Jennifer has contributed to the development of the Test Automation Center of Excellence at Nationwide Insurance.

Test Automation Centers of ExcellenceJennifer Seale

2

Impediments to Automation Projects

• Lack of experience and knowledge• Lack of resources• Environment instability• Project Process Immaturity• Costs

– Initial investment may be large (tools, knowledge, training)

– Return on Investment may be long term– Maintenance required

3

Nationwide’s Solution

• Centralized Test Automation Team• Experts in designing automation

solutions and using automation tools• Partner with projects• Provide various services

4

Developing a Center of Excellence

• Set up Shop• Document Review Process• Process for engagements• Templates to support process• Process to manage requests for services• Process to track status• Continuous Improvement• Best Practices and Standards

5

Customer Engagement Process (CEP)

Six Phase Process1. Scoping 2. Planning 3. Development4. Execution5. Wrap Up6. Maintenance

6

Templates Supporting our Process

• Initial Screening Survey• Readiness Assessment• Complexity Assessment• Statement of Work (SOW)

– Schedule– Cost Benefit Analysis (CBA)– Training and Knowledge Transfer Plan

• Proof of Concept (POC)• Customer Development Plan• Test Automation Design Document• Test Automation User Guide• Project Log

7

CEP Phase 1: Scoping

• Learn about project and needs• Complete Assessments• Determine scope of engagement• Develop preliminary schedule• Conduct POC• Get approval to continue to planning

8

Scoping: Assessments

• Initial Screening Survey– Does automation make sense for the project?– Focus on goals, application, environments, builds

• Readiness Assessment– Is the project ready for a test automation effort?– Focus on resources, processes, artifacts, funding

• Complexity Assessment– How much effort will it take to automate this

project?– Focus on scope, dependencies

9

Scoping: Determine Scope

We typically use the following method to determine the scope of the project:

1. Determine automation needs.2. Break down automation needs into a list of well-

defined, manageable units of work.3. Prioritize the list in terms of functional

dependencies.4. If the prioritized list is long, divide the list into

logical sections. These sections will be used during planning to allow a phased approach to automation development.

10

Scoping: Preliminary SOW

• Using template as base, add details as they are revealed through conversations with the customer.

• Helps to have information in writing• Will become a complete description of all the

work involved in the automation effort.• Need to determine preliminary schedule to

help in resource planning.

11

Scoping: Proof of Concept (POC)

• A POC is a prototype of the automation that demonstrates the technical feasibility of the automation project.

• Use Complexity Assessment results to help determine an interesting subset of scenarios to cover in POC.

• Automate this subset of test scenarios using the selected automation tool(s) and report results.

12

Scoping: Benefits of a POC

• Increased confidence in the project• Early identification of challenges and risks• Early set up of automation environment• Provides useful input to estimating effort

13

Scoping: Customer Development Plan

When we say “No” to an automation project, we offer to provide the customer with a document that outlines what needs to be addressed on their project before an automation effort can begin.

Most useful when:• the project is interested in automation• automation makes sense for the project• other factors exist that put the success of an automation effort

at risk

14

CEP Phase 2: Planning

• Refine SOW• Refine Schedule• Perform Cost Benefit Analysis• Training and Knowledge Transfer Plan• Test Automation Design Document• Approval for SOW

15

Planning: Refining SOW

• Finalize communication plan• Finalize details of objectives and

deliverables• Finalize details of schedule• Finalize cost estimates

16

Planning: Cost Benefit Analysis

• Using template as a base, tailor to fit project specifics.

• Estimates all costs associated with automation effort.

• Predicts benefits of automation effort based on how often the project will run the automation once it is complete.

• Predicts the breakeven point, when the benefit is equal to the cost.

17

Planning: Automation Costs

When planning for an automation, consider the following costs:– Staff Resources– Training– Learning curve– Initial Development– Execution and Analysis– Maintenance– Enhancements– Hardware– Support

18

Planning: Automation Benefits

There are three types of benefit that can be realized through Test Automation:

• Reduction –benefit realized when a test suite that is currently being executed manually is replaced by an automated suite.

• Coverage –benefit realized when an automated suite is created or extended to perform testing that is not currently being executed manually due to lack to resources or funds.

• Quality –benefit realized by detecting defects earlier in the software development life cycle due to the automation.

Return on InvestmentGeneric Automation Savings Profile

Quality

Reduction

CoverageInitial

Investment

-1000-500

0500

1000150020002500300035004000

Jan-

06

Mar

-06

May

-06

Jul-0

6

Sep-

06

Nov

-06

Jan-

07

Mar

-07

May

-07

Jul-0

7

Sep-

07

Nov

-07

Jan-

08

Mar

-08

May

-08

Jul-0

8

Sep-

08

Nov

-08

Jan-

09

Mar

-09

May

-09

Jul-0

9

Sep-

09

Nov

-09

Months

Hou

rs

Break Even

20

Planning:Training and Knowledge Transfer Plan

• It is important that we prepare key members of the project team to take over the automation.

• We tailor this plan specifically to meet the needs of the project.

• Often this plan contains its own schedule.

• Typically produce a Test Automation User Guide.

21

Planning:Test Automation Design Document

• Documents the automation design for the project.

• Provides an overview of exactly what was automated for Test leads to use when planning future testing efforts.

• Documents all design decisions and workarounds that were employed.

22

Approval of SOW

• The completed SOW is sent to the customer stakeholders for review.

• We conduct a face-to-face meeting where we walk the customer through the SOW to ensure we are in agreement.

• Once the SOW is approved, development begins.

23

CEP Phase 5: Wrap Up

• Complete Training and Knowledge Transfer and Plan

• Update SOW with results of engagement

• SOW Complete approval meeting• Customer Satisfaction Survey• Engagement Retrospective

24

CEP Phase 6: Maintenance

• Follow up with projects every 6 months for life of project

• Update CBA with actuals• Address noteworthy issues

25

Managing Requests for Services

• Spreadsheet of requests established• Tab per request with overview page• Log all pre-engagement contact• Used for tracking and forecasting• Available on shared drive, all team

members responsible for keeping up-to-date.

26

Tracking Status: The Project Log

• Spreadsheet with multiple tabs• Tab per document to track change requests• Tab to track technical issues• Tab to track resource issues• Tab for lessons learned

• Always available for review• Wealth of historic data for future projects

27

Continuous Improvement

• Document Change Request (CR) System– Survey Results– Lessons Learned– Retrospectives

• Quarterly reviews to address CRs and improve the process and associated templates

28

Best Practices and Standards

• Perfectly positioned to tailor theoretical best practices to specific business

• Reuse solution models across applications

• Develop standards for specific technologies used by the business