product qa
Post on 30-Nov-2014
1.202 Views
Preview:
DESCRIPTION
TRANSCRIPT
Product QA – A Test engineering perspective
Ensuring products are tested the way they will be used
Product QA: Why it is Tough
1. Product is more than software• Software application• Platform and variations• Documentation: online help, manuals, tech briefs, examples, training• Add-ins and third party product integrations• Installation experience• Handling upgrades and updates
2. Quality is more than lack of bugs• Does it solve the problem when it works?• Is it practical to use? Is it designed for efficient use?• Does it keep working or works only under ideal conditions?• Is it economical to build, maintain and respond rapidly to
customer/user needs?
Role of QA in Product Company
• QA is everything you do to minimize risk of failure and promote excellence
• Help producing products that satisfy target customer base
• Broad functions of QA– Risk management: release planning, release approval– Bring customer involvement: early release programs– Act as skillful developers: mimic tough customer!– Process definition and improvement– Inspection and testing– Experience-based improvement: contribute to product
definition, many finer points• Should play an important role throughout The Life of a
Product
Hawk-Eye Methodology-Precision Delivered
5 4
3
21
6
Planning Design
Coding
TestingRelease
Post-Release
Process Definition
Process Improvement
QA Planning
Risk Assessment
QA Certification
Release Inspection
Test Development
Test Suite Design
Product DefinitionInputs
Early ReleaseProgram
Test Execution
Code Review
Design Review
Our offerings
Services
Business landscape
Non-testing Functions
QA Planning
• What QA is required to do• QA procedures, expectations from other
teams• Effort estimation and team planning• Resource planning, tools and software
procurement
Test Suite Design
• Design the tests in parallel to software design• Tests need organization into independent test
suites• Build the tests before you build the software• If you find a bug, write a test that proves the
bug exists, fix the bug and then rerun the test to prove it is fixed
Test Suite Design Review
• Goal: Clarification and accept/reject design decisions
• Starts with a presentation of the design for the component/module to a selected panel of developers
• Peer review is commonly used, with most relevant engineers from other parts of the product included
• Discussion and coordination of comments incorporation
Check-in Review & Code Walkthrough
• Check-in review done with peers of the dev team– Including new features and bug fixes
• Code walkthrough by dev across streams, including QA– review meeting with prepared participants– Sample source files are selected for review
• Focus is on finding defects, before they slip through to testing phases– Lower costs by catching them early
Early Release Programs
• Need to ensure smooth porting of current customers
• Test ISV applications available internally• Public program to increase the test base– Alpha and Beta testing
Release Readiness
• QA evaluates quality, independent of developers– Developers should not determine how much
quality is sufficient
• QA acts as internal customers and verifies release readiness
Our Test ClassificationBased on Class Description
How a test is written
Black box Test does not use knowledge of product code and design
White box Test uses product internals, even calls unexposed functions
When a test is being run
Sanity testing Basic set of tests run after every nightly build, to ensure that nothing is broken
Alpha testing Testing done on code release to QA
Beta testing Testing done with the help of external partners, early release customers, etc.
Final testing Final set of testing done by QA to approve GA release
What purpose the test is serving
Functional Verify that functional requirements of the product are as per specifications
Compliance Test suite for compliance to industry standards
Load Verify performance and scalability of the product; also produce data on installation requirements
Reliability Negative testing to ensure that the product functions gracefully under unexpected conditions
Usability Testing the product for practicality, efficiency of use and ease of learning of the product interface
Functional Testing
• Key aspects to cover (they are endless!)– Correctness– Interoperability– Compatibility– Compliance– Security– …
• Think of permutations– Functions vs. input data vs. states– Product vs. platforms– System vs. external factors– Testing vs. versions of product
Compliance Testing
• Emphasis on when product is expected to be declared “compliant” with Industry Standard (such as J2EE™)
• Compliance test suite (TCK for J2EE) might be made available
• Compliance to standards does not mean the product is functionally tested for use
Load Testing
• Performance testing– Performance that can be achieved on a configuration setup– Deterioration with load– Variations across platforms– Writing a sizing guide with suggested deployment
configurations– Generating a near real-world load
• Scalability testing– How does the system scale with increase in platform capacity– Fail over and load balancing under clustered configurations
• Benchmarking– Comparative performance testing and analysis with competitor
products– Public benchmarking, such as SPECjAppserver™ or ECperf™
Performance Reports
Reliability Testing
• Aspects to test– Maturity: how the product expects erroneous user handling
– Fault-tolerance: Ability to perform under faulty hardware/platform conditions with degraded performance but near full functionality
– Integrity: How the system cannot be broken by malicious users
– Recoverability: ability of the product to recover to a consistent state when power outage, network fault or other disaster happens
– Safety: protection of user data, backup, logging, etc.
Usability Testing
• Aspects to cover– Understandability: clarity of interface, intuitiveness– Learnability: ease of learning, matches user’s current
understanding of other products– Consistency: across versions, all across the product
• Its not only about GUI, its also about file formats, XML, alternatives such as command line tools
• We normally work with external consultants who are specialized in usability engineering
Other Aspects for Testing
• Some of these aspects may not be important to end-customers, but very important for engineering
• Key aspects include– Efficiency
• Storage• Processing
– Maintainability• Code organization, branching• Modularity for ease of changes, maintenance, refactoring
– Portability• Platform support• Reusability of code
Making Testing Easier
• Document the design
• Use internal error checking
• Test units before integrating them
• Tell the tester what’s new or flaky
• Test the build yourself, first
• Evolve product in functional layers
Tools and Infrastructure
Test Automation
• Test automation helps repeatability of testing• Useful test automation is a major software project• Engineers are able to detect more problems, more accurately
Industry standard skills
In-House Tools
BrighTest Automation tool developed in-house Leverages Selenium and Java API based approach Easily customizable to suite specific project
requirements Ability to store reports in our own format
for future use. Easy to install. Easily redefined parameters Customized Alerts and Reports. Weekly and Nightly build integrations Applicable to testing of web based
applications and products as well as Social Platforms
Pilot usage on for a couple of happy customer engagements.
Extensively used to test Pramati’s Enterprise 2.0 product Qontext.
Test Case Management System
J2EE™ Compliance Testing• J2EE CTS Kit
– From Sun
• Standards Verification• Automated Harness
Test Management Framework• Augment CTS
– For Complex scenarios
• Built on Studio Framework– Homegrown
• Ease of writing new tests• Automated Discovery of new tests• Extensive tests metadata
– Compose new plans/suites
• Integrated into the Build System– For automated post-build testing
Reliability Testing
Blade- Distributed Stress Testing System
Thank you.
info@imaginea.com
top related