formal software testing
TRANSCRIPT
![Page 1: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/1.jpg)
Formal Software Testing
Terri Grenda, CSTEIV&V Testing Solutions, LLCwww.ivvts.com
![Page 2: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/2.jpg)
Scope of Testing
Find defects earlyRemove defects prior to productionIdentify RisksUnbiased opinion
![Page 3: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/3.jpg)
When Should Testing Occur
In all phases of the Software Development Life CyclesRequirements
Determine Verification ApproachDetermine Adequacy of RequirementsGenerate Test Plan Outline
![Page 4: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/4.jpg)
When Should Testing Occur
DesignDetermine consistency of design with requirementsDetermine adequacy of designGenerate structural and functional outline Test Plan
Build (Code)Determine consistency of design
![Page 5: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/5.jpg)
When Should Testing Occur
TestSystem Test the applicationCareful control and management of test environment
InstallationTest installation processEnsure correct versions of the programLessons learned passed onto supporting parties
![Page 6: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/6.jpg)
When Should Testing Occur
Neither development nor verification is a straight-line processAnalyze the structures produced at this phase for internal testability and adequacyDetermine that the structures are consistent with structures produced during previous phasesRefine or redefine test scripts generated in previous phases
![Page 7: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/7.jpg)
Development of Test Plan
Identify the system development processSelect and rank test objectivesIdentify the business risks associated with the system under developmentPlace risks in matrixDetermine test approach
![Page 8: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/8.jpg)
Common Problems Associated with Testing
Failing to define testing objectivesUsing ineffective test techniquesIncomplete Statement of Work “The Goal”Changes in TechnologyLimited Tester Skills
![Page 9: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/9.jpg)
Technological Development
IntegrationSystem ChainsDomino EffectReliance on Electronic EvidenceMultiple Users
![Page 10: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/10.jpg)
Common Software Risks
Unauthorized transactions can be accepted by the systemSecurity of the system will be compromisedProcessing will not comply with organizational policy or government regulationsResults of the system will be unreliable
![Page 11: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/11.jpg)
Common Software Risks
System will be difficult to usePrograms will not be maintainableSystems will be able to interconnect with other computer systemsPerformance level will be unacceptable
![Page 12: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/12.jpg)
Why are Defects Hard to Find
Not LookingTest often are not performed because a particular test condition was unknown.Some parts of a system go untested because developers assume software changes don’t affect them
Looking, But Not SeeingLosing your car keys only to discover they were in plain sight the entire time
![Page 13: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/13.jpg)
Levels of Testing
Verification Testing (static testing development process)Unit TestingIntegration TestingSystem TestingUser Acceptance Testing
![Page 14: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/14.jpg)
Testing Techniques
Structural versus FunctionalVerification versus ValidationStatic versus Dynamic TestingExamples of Specific Testing Techniques
![Page 15: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/15.jpg)
Structural versus Functional
Both structural and functional analysis should be performed to ensure adequate testingStructural tend to uncover errors during codingFunctional is not concerned with how processing occurs, but with the results of processing
![Page 16: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/16.jpg)
Structural Testing Techniques
StressExecutionRecoveryOperationsCompliance (To Process)Security
![Page 17: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/17.jpg)
Stress Testing
Determine system performs with expected volumes
Normal or above-normal volumes of transactionStructurally able to process large volumes of dataSufficient resources available to meet expectationsUser perform their tasks in desired turnaround time
![Page 18: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/18.jpg)
Execution Technique
System achieves desired level of proficiency
Performance level of the system structure.Optimum use of hardware and softwareResponse time to online user requestsTransaction processing turnaround time
![Page 19: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/19.jpg)
Recovery Testing
System can be returned to an operational status after a failure
Evaluate process, procedures and documentationEstimate potential loss time spans and backup recoverySimulate Disaster
![Page 20: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/20.jpg)
Operations Testing
System can be executed in a normal operational status
Completeness of computer operator documentationEnsure necessary support, such as job control are prepared and functionalCompleteness of operator trainingEnsure prepared documentation can, in fact, operate the system
![Page 21: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/21.jpg)
Compliance
System is developed in accordance with standards and procedures
System development and maintenance methodologies are followedDepartment standards, procedures and guidelinesCompleteness and reasonable of application system documentation
![Page 22: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/22.jpg)
Security Testing
System is protected in accordance with importance to organization
Resources are protected and access levels are definedSecurity procedures have been implemented and function in accordance with the specifications.System can identify and prevent unauthorized access
![Page 23: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/23.jpg)
Functional Testing Techniques
RequirementsRegressionError HandlingManual SupportIntersystemControlParallel
![Page 24: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/24.jpg)
Requirements
System performs as specifiedProven system requirementsCompliance to policies and regulationsMaintain correctness over extended processing periodsRecord retention
![Page 25: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/25.jpg)
Regression
Verifies that anything unchanged still performs correctly
Unchanged system segments functionUnchanged manual procedures correctFocus on areas of exchange between new functionality and oldVery time-consuming and tedious operation
![Page 26: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/26.jpg)
Error Handling
Errors can be prevented or detected and then corrected
Requires to think negativelyExpected error conditions are handled by the applicationErrors are properly handledReasonable controls are maintained during correction process
![Page 27: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/27.jpg)
Manual Support
The people-computer interaction works
Support procedures are documented and completeSupport staff are adequately trainedSupport tools are implementedAutomated segment are properly interfaced
![Page 28: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/28.jpg)
Intersystem
Data is correctly passed from system to system
Proper parameters have been setData correctly passes between applicationsCoordination and timing functions exists between systemsSystems are accurate and complete
![Page 29: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/29.jpg)
Control
Controls reduce system risk to an acceptable level
Data validationFile IntegrityAudit TrailBackup and recoveryTransactions are authorizedProcess is efficient, effective and economical
![Page 30: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/30.jpg)
Parallel
Old system and new system are run and the results compared to detect unplanned differences
System CalculationsData processingEnsure scheduled background tasksEnsure operational status
![Page 31: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/31.jpg)
Verification versus Validation
Verification ensures the system complies with the organization’standards and processes. Did we build the right system?
Validation physically ensures the system operates according to plan. Did we build the system right?
![Page 32: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/32.jpg)
System Verification Examples
Requirements ReviewDesign ReviewCode WalkthroughsCode Inspections
![Page 33: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/33.jpg)
System Validation Examples
Unit TestingIntegration TestingSystem TestingUser Acceptance Testing
![Page 34: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/34.jpg)
Static versus Dynamic Testing
Static testing is performed using the software documentation. The code is not executing during static testing.
Dynamic testing requires the code to be in an executable state to perform the tests.
![Page 35: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/35.jpg)
Static Testing
Verification techniques are static tests
Feasibility ReviewsRequirement ReviewsDesign Reviews
![Page 36: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/36.jpg)
Dynamic Testing
Validation tests are dynamic testsUnit TestingIntegration TestingSystem TestingUser Acceptance
![Page 37: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/37.jpg)
Examples of Testing Techniques
White-Box TestingBlack-Box TestingIncremental TestingThread TestingRequirements TracingBoundary Value AnalysisError Guessing and Special Value AnalysisCause-Effect Graphing
Design-Based Functional TestingCoverage-Based TestingComplexity-Based TestingStatistical Analyses and Error SeedingMutation AnalysisFlow Analysis
![Page 38: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/38.jpg)
White-Box Testing
Assumes the path of logic in a unit or program is know. Consists of testing paths, branch by branch, to produce predictable results.Examples: Statement Coverage, Decision Coverage and Multiple Condition Coverage
![Page 39: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/39.jpg)
Black-Box Testing
Focuses on testing the function of the program or application against its specificationsOften called Functional TestingDetermines whether combinations of inputs and operations product expected results
![Page 40: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/40.jpg)
Incremental Testing
Discipline method of testing the interfaces between unit tested programsTop Down
Use interim stubs to simulate lower interfacing modules
Bottom-UpDrives to provide test input that call module or programs
![Page 41: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/41.jpg)
Thread Testing
Demonstrates key functional capabilities by testing a string of units that accomplish a specific function in the applicationThread testing and incremental testing are usually utilized together
![Page 42: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/42.jpg)
Requirements Tracing
Primary goal of software testing is to prove requirements are delivered in the final productTrace functional and non-functional requirements through out analysis and design models, class and sequence diagrams, code and testing
![Page 43: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/43.jpg)
Boundary Value Analysis
Technique that consists of developing test cases and data that focus on the input and output boundaries of a given function
Low boundary +/- one ($9,999 and $10,001)On the boundary ($10,000 and $15,000)Upper boundary +/- one ($14,999 and $15,001)
![Page 44: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/44.jpg)
Error Guessing and Special Value Analysis
Certain test data seem highly probable to catch errors
February 29, 2000Zero inputsNegative value inputsSpecial CharactersData Type ValuesRandomized selections
![Page 45: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/45.jpg)
Cause-Effect Graphing
Technique for developing test cases for programs form the high-level specificationsLimited entry decision table is derived
Examples: Causes in 2 and 3 result in effect 4 or Causes in 2, 3 and 5 result in 6
![Page 46: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/46.jpg)
Design-Based Functional Testing
Top Down Approach (Functional Design Tree)Necessary to invent other smaller functions from requirementsA requirement become the root nodeCorresponding design functions become the second level of the tree
![Page 47: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/47.jpg)
Coverage-Based Testing
Top down designKnowledge and access to code baseNumber of statements, branches or paths in the programTest data should cause execution of all paths
![Page 48: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/48.jpg)
Complexity-Based Testing
McCabe metricsCyclomaticEssentialActual
Cyclomatic and Essential are calculated from a program graph; linearly independent program pathsActual is a runtime metric
![Page 49: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/49.jpg)
Statistical Analyses and Error Seeding
Most common type of test data analysisInsert known errors in the codeExecute test data and determine number of actual errors
![Page 50: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/50.jpg)
Mutation Analysis
Derived from branch coverage and statement coverageSeed the program to be tested with errors, creating several mutants of the original problemDetermines how deep will the error pass through the code
![Page 51: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/51.jpg)
Flow Analysis
Control flowAnalyze program behaviorLocate instrumentation breakpointsIdentify paths
Data flowDiscover program anomalies such as undefined or unreferenced variables
![Page 52: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/52.jpg)
Combining Testing Techniques
Forms more powerful and efficient testing techniqueMerge standard testing techniques with formal verificationTradeoffs between testing and formal methods are efficient and effective
![Page 53: Formal Software Testing](https://reader034.vdocuments.us/reader034/viewer/2022052410/555a62c4d8b42a972b8b457a/html5/thumbnails/53.jpg)
Questions?