automate virtualize and smart test the new testing realities
DESCRIPTION
Presentation on changes occurring in the testing space due. Presented in Star East 2011.TRANSCRIPT
©2011, Cognizant
by
Manoj NarayananDirector –Testing Services, Cognizant
Automate, Virtualize & Smart Test
Understanding the new realities of Testing
@manoj7698
http:\\everydaytesting.blogspot.com
Requirements Gathering
Architecture & Design
Development
Testing Release
1
What does traditional testing offer?
Discrete, independent
Waterfall Methodology
Focused on defect identification
©2011, Cognizant
Reducing risk by preventing poor quality
from impacting satisfaction
Quickly responding to
customer requests and requirements
22%Reducing costs by improving
our development and testing productivity
17%Delivering
applications on time and on
budget
14%Improving
collaboration between
development and QA
13%Delivering
applications before
competition
7%Compliance
and regulatory
issues
5%
2
• 32% of organizations say they release software with too many defects
• 38% of organizations believe they lack an adequate quality assurance program
• 27% of organizations do not conduct any formal quality reviews
What does business want today?
Source- QAI –Edista Testing More than 60%
24%
©2011, Cognizant
Reducing risk by preventing poor quality
from impacting satisfaction
Quickly responding to
customer requests and requirements
22%Reducing costs by improving
our development and testing productivity
17%Delivering
applications on time and on
budget
14%Improving
collaboration between
development and QA
13%Delivering
applications before
competition
7%Compliance
and regulatory
issues
5%
3
• 32% of organizations say they release software with too many defects
• 38% of organizations believe they lack an adequate quality assurance program
• 27% of organizations do not conduct any formal quality reviews
What does business want today?
Source- QAI –Edista Testing
Reducing Risk
Responding Fast
Beat Competitors
More than 60%
24%
©2011, Cognizant
Requirements Gathering
Architecture & Design
Development
Testing Release
4
Traditional testing approach
Discrete, independent
Waterfall Methodology
Focused on defect identification
©2011, Cognizant
We need to move testing to the beginning of SDLC – focus on prevention than identification
5
Requirements Gathering
Architecture & Design
Development Testing Release
From Discrete and Independent Testing to…
…Continuous and Integrated Testing
Waterfall Methodology
Scrum
©2011, Cognizant
The “Shift-Left” Philosophy
6
Identify QA skill sets needed
and initial effort estimate
Early execution of QA, Non-GUI testing
Automated end to end regression
testing
Non-functional testing
Focus on Test Driven Development or a
variant
Performance & Security PoCs
©2011, Cognizant
Shift-Left Philosophy is accompanied by the need for a new testing approach
7 Automation becomes critical
Need to move away from traditional
approach focusing on GUI based testing..…to an approach focusing on increased
defect capture through Unit and Service
Layer Testing
Unit
Testing
Service Layer Testing
GUI Testing
Unit Testing
Service Layer Testing
GUI
Testing Perc
en
tag
e
eff
ort
sp
en
t
©2011, Cognizant
This results in the need for seeding Quality Engineering into testing
8
QE Approach QA Approach
• A QE approach focused on testing components
• A QA approach focused on regression and UAT
©2011, Cognizant
And “Smart Testing” dissolving boundaries of traditional system & integration testing
9
QE Approach
Unit Testing
Service Layer Testing
Functional Testing
(automation)
Regression Testing
(automation)
Beta Testing
User Acceptance
Testing
QA Approach
Fast merging into a social media and
“crowd” aided Community Testing
©2011, Cognizant
And “Smart Testing” dissolving boundaries of traditional system & integration testing
10
QE Approach
Unit Testing
Service Layer Testing
Functional Testing
(automation)
Regression Testing
(automation)
QA Approach
Community Testing
©2011, Cognizant
Leverage optimal mix of automation across the lifecycle
11
Unit Testing
Service Layer Testing
Regression Testing
• Re-usable, automated test cases used by developers• Leveraging xUnit Frameworks
• Leverage re-usable frameworks like RIDE, Fitnesse, Cucumber
• Web Services testing • Leverage Service Virutalization
• >90% automated regression test cases• 100% automated smoke testing• Continuous validation of test flow relevancy
Manual Testing
©2011, Cognizant
Prioritizing Automation: ROI driven approach
12
Repetitive Testing
Verify application paths that have high degree of frequency and other high volumeactivities where software failures would occur
Validate applications with a high degree of risk which define or control the core ofa company’s business; Failure can cause extreme disruptions in critical operations
Automating applications that are or will be in production for a long time
Greater benefits can be reaped from Test Automation if the test executionfrequency is high
Test Automation will be an ideal testing solution if the testing has to be performed onmultiple environments
Any flow that needs to be run repeatedly in a build
©2011, Cognizant
Prioritizing Automation: ROI driven approach
13
Repetitive Testing
Verify application paths that have high degree of frequency and other high volumeactivities where software failures would occur
Validate applications with a high degree of risk which define or control the core ofa company’s business; Failure can cause extreme disruptions in critical operations
Automating applications that are or will be in production for a long time
Greater benefits can be reaped from Test Automation if the test executionfrequency is high
Test Automation will be an ideal testing solution if the testing has to be performed onmultiple environments
Any flow that needs to be run repeatedly in a build
©2011, Cognizant
The need for early testing and automation also triggers environment related challenges
14
OrderingSwing App
Processing Web App
COM App
ESB, JMS, MQ
Mainframe
Business Process App
Additional Apps Now & Future
Database
Database
App ServerEJB, WS,
Environment availability challenges –productivity losses, inability to duplicate
defects, performance testing
Need for testing individual components as part of “shift-left”
philosophy
©2011, Cognizant
Work Around: Virtualization & Service Virtualization
15
OrderingSwing App
Processing Web App
COM App
ESB, JMS, MQ
Mainframe
Business Process App
Additional Apps Now & Future
Database
Database
App ServerEJB, WS,
5
LISA Virtualize
UI Testing no longer the dominant testing method
Defects identified at each component level
Early testing is now feasible. Enables organization to adopt
“Shift Left” testing
Environment availability challenges mitigated – simulate performance
environment, component transactions
Re-usability increases, limited dependency on development teams
Picture: Courtesy ITKO
©2011, Cognizant
Explaining Service Virtualization
• Mainframe applications are consumed
through queues – in this case, MQ
• The MQ sends messages to the
application which gets transformed
through the adapter
“Listener”
©2011, Cognizant
This has to be complemented by a centralized Test Environment Management approach
17
Cloud too is an option – even after the Amazon EC2 fiasco
18
And an organizational transition plan based on current maturity level
Focus on• Process & Methodology• Tools• Infrastructure• Operating Model• People: Skill set/Roles
Evaluate Current QA
State
Define Future State
Stakeholder Discussions
Artifact Analysis
Inputs
Industry best practices
Methodology Adopted
Inputs
Identify implementation priorities
Focus on• Risk appetite• Cost constraints• Available time• Priority areas• Current & Future
state gap
Draw Implementation
Roadmap
Stakeholder Validation
External dependencies
Inputs
©2011, Cognizant
But do not let the total cost creep up…
19
• The QA organization needs to keep a constant watch on the
Total Cost of Quality as it makes the transition
• During the transition phase, the Cost of Testing goes up due to
the increased focus in areas like skill ramp-up, automation and
process re-alignment
• The resultant increase in Total Cost of Quality needs to be
continuously adjusted to ensure that it falls within best practice
levels of 25–30% within 3 to 6 months of transition
Cost of TestingCost of Production
Defects
Cost of Compliance Audits
Total Cost of Quality
And most importantly, do not forget to communicate
20
• Ensure Executive & Stakeholder buy-in on the
change
• Establish a communication channel for regular
updates and feedback
Summary
21
• Traditional testing approaches are being replaced by a “shift-
left” approach to accommodate current business realities
• Shift away from GUI based testing to component based testing
• Focus on both QE and QA based testing – prevention rather
than identification: Need for “Smart Testing”
• Automation becomes critical to maintain the ever reducing
testing window
• New facilitators have risen: Community Testing, Service
Virtualization, Test Environment Management
• However, it is very critical that the organization has a well
thought out transition plan to accommodate these changes
And, DO NOT forget to communicate!
©2011, Cognizant
@manoj7698
http:\\everydaytesting.blogspot.com
www.cognizant.com