Acceptance Testing for Acceptance Testing for ROMEROME

Pete CastlePete Castle

Test & Quality ManagerTest & Quality Manager

AgendaAgenda

• What is software testing/ Who does it?What is software testing/ Who does it?• Why software testing is importantWhy software testing is important• Some fundamentals of testingSome fundamentals of testing• Test Plans & ScriptsTest Plans & Scripts• Sample Testing TechniquesSample Testing Techniques

What is software Testing?• ““Software testing is an Software testing is an empiricalempirical technicaltechnical investigationinvestigation

conducted to provide stakeholders with information conducted to provide stakeholders with information about the quality of the product or service under test” about the quality of the product or service under test” Professor Cem KanerProfessor Cem Kaner - - Director of Florida Tech's Center Director of Florida Tech's Center for Software Testing Education & Researchfor Software Testing Education & Research

• Empirical - derived from experiment, experience, and Empirical - derived from experiment, experience, and observationobservation

• Technical - Technical - Having special skill or practical knowledgeHaving special skill or practical knowledge• Investigation - A detailed inquiry or systematic Investigation - A detailed inquiry or systematic

examinationexamination

Why testing is ImportantWhy testing is Important

• All Software has defects (bugs)All Software has defects (bugs)• All software products are ‘prototypes’ (in my view)All software products are ‘prototypes’ (in my view)• Software products are getting larger and more Software products are getting larger and more

complicated - Vista 40% larger than XP @ over 50 complicated - Vista 40% larger than XP @ over 50 million LOCmillion LOC

• Software Engineering is not as mature as other Software Engineering is not as mature as other disciplines e.g. Civil Engineeringdisciplines e.g. Civil Engineering

• Software is written by people – people make mistakesSoftware is written by people – people make mistakes• Software testing looks to find the most important defects Software testing looks to find the most important defects

as early as possible – increasing confidence that the as early as possible – increasing confidence that the software meets specificationsoftware meets specification

Who’s involved in testing?Who’s involved in testing?

• Requirements Analysts – Inspections, Peer ReviewsRequirements Analysts – Inspections, Peer Reviews• Developers – Code Inspection, Unit TestingDevelopers – Code Inspection, Unit Testing• Testers – System & Integration TestingTesters – System & Integration Testing• Trainers – Training materials productionTrainers – Training materials production• Users – User Acceptance TestingUsers – User Acceptance Testing• Project Managers – Scheduling, Resourcing, Risks, Project Managers – Scheduling, Resourcing, Risks,

Issues, Defect StatsIssues, Defect Stats• Everybody is responsible for quality - NASAEverybody is responsible for quality - NASA

Fundamentals of Software TestingFundamentals of Software Testing

• Software testing needs planning, tests need specifying, Software testing needs planning, tests need specifying, once executed they need results recording, and post once executed they need results recording, and post

completion should be easily auditablecompletion should be easily auditable

Plan Specify

Execute Record Check

The importance of a planned approachThe importance of a planned approach

• Important to map out a strategy that will give the greatest Important to map out a strategy that will give the greatest level of confidence in the product level of confidence in the product

• ‘‘Ad hoc’ testing may find errors, but may not be cost Ad hoc’ testing may find errors, but may not be cost effectiveeffective

• Testing should focus on areas where defects are most Testing should focus on areas where defects are most likely likely

• All testing should have a reasonAll testing should have a reason• Question “Is a test that doesn’t find an error a good test or not?”Question “Is a test that doesn’t find an error a good test or not?”

• Essential to plan what needs to be done and then Essential to plan what needs to be done and then itemise how it is to be achieved.itemise how it is to be achieved.

Testing MantraTesting Mantra• Mantra - Spiritual conduit, words or vibrations that instil Mantra - Spiritual conduit, words or vibrations that instil

concentration in the devotee.concentration in the devotee.

• Test as early as possibleTest as early as possible

• Gather as much knowledge of the application under test as possibleGather as much knowledge of the application under test as possible

• Look for vulnerabilitiesLook for vulnerabilities

• Build ‘Bug Taxonomies’ (Classification)Build ‘Bug Taxonomies’ (Classification)

• Use Quicktests (and publicise the fact)Use Quicktests (and publicise the fact)

Testing MantraTesting Mantra• You can always think of another test – but should youYou can always think of another test – but should you??

• Concept of ‘Good enough Testing’Concept of ‘Good enough Testing’• Practicality over dogmaPracticality over dogma• Everybody has responsibility for ‘shipping’ the productEverybody has responsibility for ‘shipping’ the product

• Record all tests/defects/issues/recommendationsRecord all tests/defects/issues/recommendations

• Testers are not the sole arbiters of qualityTesters are not the sole arbiters of quality• Testing only shows problems exist – not their absenceTesting only shows problems exist – not their absence

• Never, Never, ever, everever, ever make it personal make it personal• Defects are issues with products and process not peopleDefects are issues with products and process not people• Good working relationship is essential for good productsGood working relationship is essential for good products

Document Hierarchy - Test PlanDocument Hierarchy - Test Plan

TestPolicy

Test Strategy

Project Test Plan

UnitPhase

IntegrationPhase

Acceptance Phase

What is a Test Plan - 1What is a Test Plan - 1

Test plan isTest plan is tool to help plan the testing activitytool to help plan the testing activity product to inform others of test processproduct to inform others of test process

IncludesIncludes Document controlDocument control ObjectivesObjectives ScopeScope Approach – Schedule, Priorities, Deliverables, Approach – Schedule, Priorities, Deliverables,

Resources, ResponsibilitiesResources, Responsibilities Risks/ContingencesRisks/Contingences Sign-off/ApprovalSign-off/Approval

What is a Test Plan - 2What is a Test Plan - 2

• Produced by Test Lead/Project ManagerProduced by Test Lead/Project Manager• Published to Project/ProgrammePublished to Project/Programme• Not constrained by format – living documentNot constrained by format – living document• Enough information to be used by anyone to test Enough information to be used by anyone to test

the productthe product

Rome Test PlanRome Test Plan

Ready for reviewReady for review Written by Tim WellsWritten by Tim Wells

Document Hierarchy -Test ScriptsDocument Hierarchy -Test Scripts

TestPolicy

Test Strategy

Project Test Plan

UnitPhase

IntegrationPhase

Acceptance Phase

Test ScriptsTest Scripts

• Test Script - Is a collection of test cases for the Test Script - Is a collection of test cases for the software under test (manual or automated)software under test (manual or automated)

• Test Case - A set of inputs, execution Test Case - A set of inputs, execution preconditions, and expected outcomes preconditions, and expected outcomes developed for a particular objective, such as to developed for a particular objective, such as to exercise a particular program path or to verify exercise a particular program path or to verify compliance with a specific requirement. compliance with a specific requirement.

Pre-conditions/InformationPre-conditions/Information

• Browsers – IE, Firefox, SafariBrowsers – IE, Firefox, Safari• O/S – Linux, WindowsO/S – Linux, Windows• Access Control – Logins, RolesAccess Control – Logins, Roles• Test Data requirementsTest Data requirements• Date/Time considerationsDate/Time considerations• Other document referencesOther document references

Example Test Script - 1Example Test Script - 1

Ref. Field/Button Action Input Expected Result Pass/Fail

001 Month Enter Data 0 Data rejected. Error Message 'Invalid Month' Fail

002 Month Enter Data 1 Data Accepted, January Displayed Pass

003 Month Enter Data 06 Data Accepted, June Displayed Pass

004 Month Enter Data 12 Data Accepted, December Displayed Pass

005 Month Enter Data 13 Data rejected. Error Message 'Invalid Month' Fail

• System Test of input of numeric month into data System Test of input of numeric month into data fieldfield

Example Test Script - 2Example Test Script - 2Search Researcher Page          

Test Ref.

ReqsRef. Function Inputs Expected Result Actual Result

Pass/Fail

2.001 REF003 Search Researchers

1. Forenames = John2. Surname = <Blank>3. eMail = <Blank>

All UCL researchers with forenames starting Pete displayed in alphabetic order, 23 records per pageList comprises Name, Department, Occupation TypeAll data items hyperlinked

427 matches - paging working correctly, data displayed correctly and in reasonable time (5 secs)

Pass

2.002 REF003 Search Researchers

1. Forenames = <Blank>2. Surname = Smith3. eMail = <Blank>

All UCL researchers with surnames starting Smith displayed in alphabetic order, 23 records per pageList comprises Name, Department, Occupation TypeAll data items hyperlinked

61 matches - paging working correctly, data displayed correctly and in reasonable time (5 secs)

Pass

Example Test Script - 3Example Test Script - 3

No. Type Scenario  Test To test?New/ Change?

Covered (Y/N)

Pass/ Fail Who

Date tested

39 Admissions Basic Licence Admissions

Basic Licence Admissions - Setup

Y  Y  Y  P SM  06/01/08

40     Basic Licence Admissions - Criterion Setup

Y Y  Y  P  SM 06/01/08

41     Basic Licence Admissions - Admissions Policy

Y Y  Y  P  SM 06/01/08

42     Basic Licence Admissions - ADT Import from EMS

Y Y  Y  P  SM 06/01/08

43     Basic Licence Admissions - ASL Export to EMS

Y Y  Y  F  SM 06/01/08

Test ScriptsTest Scripts

ReadableReadable AccessibleAccessible Usable by anyone – standardsUsable by anyone – standards Can vary depending upon the type of Can vary depending upon the type of

testing being undertakentesting being undertaken

Testing TechniquesTesting Techniques

• QuicktestsQuicktests• Negative TestingNegative Testing• Integration TestingIntegration Testing

Techniques 1- Quick TestsTechniques 1- Quick Tests

• Quicktests - Investigation more important than Quicktests - Investigation more important than ConfirmationConfirmation

• A A quicktest quicktest is a cheap test that has some value but is a cheap test that has some value but requires little preparation, knowledge, or time to performrequires little preparation, knowledge, or time to perform

• Focus on common coding errorsFocus on common coding errors

Techniques 1- Quick TestsTechniques 1- Quick Tests

• Things that have failed before – Defect dataThings that have failed before – Defect data• Tab order (particularly when adding new functions)Tab order (particularly when adding new functions)• Addresses (BFPO, new Post Codes)Addresses (BFPO, new Post Codes)• Short cut keysShort cut keys

• Boundaries – Ages, Dates, Values, Increments, Page BreaksBoundaries – Ages, Dates, Values, Increments, Page Breaks

• Interrupts, Duplications, Ordering of tasksInterrupts, Duplications, Ordering of tasks• Generate Order before setup – no cost codes, cost centres, suppliers, Generate Order before setup – no cost codes, cost centres, suppliers,

budgetsbudgets• Exit/Interrupt before completionExit/Interrupt before completion

• Consume resource (Dog Piling)Consume resource (Dog Piling)• Never close a windowNever close a window• Never exit an optionNever exit an option

Techniques 1- Quick TestsTechniques 1- Quick Tests

• Force all error messagesForce all error messages• Informative messages - SpellingInformative messages - Spelling• Debug information?Debug information?

• Capacity/Files – Files to fill all available space, large files, empty Capacity/Files – Files to fill all available space, large files, empty files, incorrect formatfiles, incorrect format

• Dependencies – e.g. same student many functionsDependencies – e.g. same student many functions• Integration QuicktestIntegration Quicktest

• Comparisons – screens, data, reportsComparisons – screens, data, reports• Tools e.g. Beyond Compare, Screen Capture, Redgate Toolset, InCtrl3Tools e.g. Beyond Compare, Screen Capture, Redgate Toolset, InCtrl3

Negative TestingNegative Testing

• Testing the system using negative data – to generate Testing the system using negative data – to generate exceptions that cause the software to failexceptions that cause the software to fail

• Going against knowledge of ‘How the system should Going against knowledge of ‘How the system should work’work’

• For Example - Testing the password where it should be For Example - Testing the password where it should be minimum of 8 characters - so test it using 7 and 9 minimum of 8 characters - so test it using 7 and 9 characters characters

• Emphasis on Emphasis on breakingbreaking not confirmation not confirmation

Negative TestingNegative Testing• Embedded single quote and other ‘special’ characters e.g. John’s Embedded single quote and other ‘special’ characters e.g. John’s

Car, John & Erin, 99%, Straße (German Addresses)Car, John & Erin, 99%, Straße (German Addresses)• Required Data Input – Don’tRequired Data Input – Don’t• Field Size – Shoe test (also Quick Test)Field Size – Shoe test (also Quick Test)• Field ‘Types’ – Characters in numeric fieldField ‘Types’ – Characters in numeric field• Boundaries (Upper/Lower) – underage job applications, 101 lines on Boundaries (Upper/Lower) – underage job applications, 101 lines on

an order with a maximum of 100 linesan order with a maximum of 100 lines• Invalid dates e.g. 31/04/08Invalid dates e.g. 31/04/08• Addresses – BFPO, Hong Kong Addresses, ‘New Post Codes’Addresses – BFPO, Hong Kong Addresses, ‘New Post Codes’• Web Session Testing – Access web page but not logged inWeb Session Testing – Access web page but not logged in• Switch off during upgrade – what happens, does the application Switch off during upgrade – what happens, does the application

know there is a problem?know there is a problem?

Integration Testing (In the large)Integration Testing (In the large)

• ““Testing performed to expose faults in the interfaces and Testing performed to expose faults in the interfaces and in the interaction between integrated components and in the interaction between integrated components and products” Sue Myler – Integration Team Leadproducts” Sue Myler – Integration Team Lead

• Usually scenario based rather than low level test casesUsually scenario based rather than low level test cases• Relies upon testers having system knowledge & testing Relies upon testers having system knowledge & testing

expertise – ability to think ‘outside of the box’, develop expertise – ability to think ‘outside of the box’, develop new tests during testingnew tests during testing

• Relies on successful unit, integration in the small and Relies on successful unit, integration in the small and system testingsystem testing

• Can mimic business processes Can mimic business processes

Integration Testing (In the large)Integration Testing (In the large)

Integration Test CasesIntegration Test Cases• 3 Applicants 3 Applicants

• 1 applies for 1 post1 applies for 1 post• 1 applies for 2 posts - also applies for the same 1 applies for 2 posts - also applies for the same

post twice (by accident)post twice (by accident)• 1 applies for 3 posts1 applies for 3 posts• do their records appear correctly across ROMEdo their records appear correctly across ROME

• Delete a Vacancy – what happens to that applicant Delete a Vacancy – what happens to that applicant records?records?

Integration Testing (In the large)Integration Testing (In the large)

• Short list applicant for post he entered twice, deleting Short list applicant for post he entered twice, deleting one applicationone application

• Invite for interview but candidate withdrawsInvite for interview but candidate withdraws• Candidate then re-appliesCandidate then re-applies• Data exported, ROME updated, then re-exported, does Data exported, ROME updated, then re-exported, does

data appear correctly in target applicationdata appear correctly in target application

Test ScenariosTest Scenarios

No. Type Scenario  Test To test?New/ Change?

Covered (Y/N)

Pass/ Fail Who

Date tested

39 Admissions Basic Licence Admissions

Basic Licence Admissions - Setup

Y  Y  Y  P SM  06/01/08

40     Basic Licence Admissions - Criterion Setup

Y Y  Y  P  SM 06/01/08

41     Basic Licence Admissions - Admissions Policy

Y Y  Y  P  SM 06/01/08

42     Basic Licence Admissions - ADT Import from EMS

Y Y  Y  P  SM 06/01/08

43     Basic Licence Admissions - ASL Export to EMS

Y Y  Y  F  SM 06/01/08

44     Basic Licence Admissions - ATF Import from EMS

Y Y  Y  F  SM 06/01/08

45     Basic Licence Admissions - ATF Re-Import from EMS with additions and amendments

Y Y  Y  F  SM 06/01/08

ReviewReview

• Software Testing is important for increasing confidence that the Software Testing is important for increasing confidence that the software meets specificationsoftware meets specification

• To get the best results from testing certain fundamentals should be To get the best results from testing certain fundamentals should be followedfollowed

• Testing is part of software developmentTesting is part of software development• Different software testing techniques enhance our ability to testDifferent software testing techniques enhance our ability to test• Many different types of software testing exist – which we can Many different types of software testing exist – which we can

combine into single test cases/scenarioscombine into single test cases/scenarios

Test Example – Data Entry ScreenTest Example – Data Entry Screen

• Create Test cases to negatively test (break) the Create Test cases to negatively test (break) the data entry screendata entry screen

Description Data/Validation

Title 20 Chars, mandatory, pick list provided

Forename 25 Chars, mandatory

Surname 25 Chars, mandatory

email Address 50 Chars, mandatory, validated

Home Telephone Number 25 Chars

Next StepsNext Steps

ROME - Kick off meetingROME - Kick off meeting Testing required who/whenTesting required who/when Test Script TemplateTest Script Template Mantis - Issue/Defect LoggingMantis - Issue/Defect Logging