Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 19981Testing Across the Lifecyclepresented atNinth Annual BorlandConference August 11, 1998Denver ColoradobyTimothy D. Korson, Ph. d.Senior Partner, Software ArchitectsandDean, School of ComputingSouthern Adventist University(423) 238-3288korson@software-architects.comwww.software-architects.comEmail: info@software-architects.comPhone: (423) 238-3288Web: www.software-architects.comFax: (423) 238-3289Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 19982Section 1Course OverviewIn this section we will discuss:Course GoalsTesting ContextRisk AnalysisCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-3Development Goals Affect the Testing Process Reuse - requires thorough testing across the complete specification. Shorter development time - requires an automated technique for class, system, and regression testing. Higher quality - requires systematic testing over the entire development process. Extensibility - requires the inclusion of hypothetical test cases to search for problems encountered in extending the design. Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-4Why Do We Test?Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-5Testing Testing consists of all activities that increase our confidence that the system will do what it should and not do what it shouldnt. More specifically, testing is a process of reducing risk by comparing what is and what should be. In the object oriented paradigm, faults may be in: The analysis model The design model The implementation Those faults are the result of defective development processes which allow (and in some cases even foster): Improper analysis Improper design Improper implementationCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-6Risk Analysis Risk analysis is the process used to decide where to allocate limited testing resources. In the object oriented paradigm, risk analysis becomes the basis for determining the level of testing classes, clusters and systems. Use risk analysis to determine the types of errors that are more critical to system success and focus resources in these areas. Develop a standard risk scale that developers can use to categorize their components.Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-7Risk Analysis Establish risk criteria: Complexity of idea Stability of specification Maturity level of class Risk of injury - financial, safety, etc. Use risk analysis to: Prioritize component tests Allocate testing resources Choose the number of test cases Decide which features to cover more completely Re-evaluate as classes mature Reallocate resources over iterations Risk should be reduced over iterationsCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-8Classic Waterfall DevelopmentCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-9System TestingDomainAnalysisApplicationAnalysisArchitecture &High Level DesignDetailed DesignClass DevelopmentApplicationAssemblySystem TestingDomainAnalysisApplicationAnalysisArchitecture &High Level DesignDetailed DesignClass DevelopmentApplicationAssemblySystem TestingDomainAnalysisApplicationAnalysisArchitecture &High Level DesignDetailed DesignClass DevelopmentApplicationAssemblyIncrementsIterationIterative IncrementalDevelopment ProcessCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-10The Object-Oriented Process Model The iterative incremental process model is a widely used approach for building object-oriented software systems. The dual goals of the object-oriented development process are: To generate applicationsAND To develop an inventory of components and frameworks for use in future projects How do these dual goals affect testing?Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-11Myths About Testing Object-Oriented SystemsMyth:Object-orientedprogramsareeasiertotestbecause once a method is tested, it never needs tobe tested again.Reality: Not necessarily so.Myth: Object-oriented programs are harder to testbecause the use of dynamic binding means that anyobject can send a message to any other object.Reality: Not necessarily so.Testing object-oriented systems is not nec-essarily more or less difficult, it is simply DIFFERENT!Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-12What is Different Adapting to a use case requirements process Understanding and interacting with domain and application level class diagrams Adapting to the impact of OO language constructs on unit testing Increased importance of unit testing Establishing a new metrics program Learning how to create a parallel architecture for component testing Learning how to use orthogonal arrays to test framework interactions Implementing support for hierarchical incremental testing for inheritance trees Adjusting to an incremental/iterative software development process Creating a certification process for standard components and frameworks Learning the impact of object distribution mechanisms on testing distributed and web-enabled applications Adapting ISO requirements to the new OO development processes New ways of interacting with software development teams which may involve organizational adjustments. Defining an effective regression testing process to match the iterative development process.Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-13What do We Test? Requirements Models Architectures and Frameworks Designs ComponentsClassCluster Increments Applications ProcessesCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-14Testing Requirements Prototypes Use CasesUnfortunately most testing organizaions take requirements as a given. Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-15Testing ModelsDo we understand the business?Pedestrian TCS Vehicle TCSClockControllerPhase Plan PhaseIntersectionRoadDetectorLaneTrafficTraffic Control SignalPressureDetectorMagneticDetector Push Button Detector RadioDetectorPedestrian VehicleTraffic ControlOfficerCrosswalkLeft Turn ThruOther Detector Types.. .Other Types of Lanes...2+1+1+1+1+1+1+TrafficIntersection Domain Analysis Object ModelCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-16Testing FrameworksConsider all possible combinations of Polymorphic Substitutionsm(P)ABCXYZPQ RCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-17Testing DesignsWhat If1. Meets best current practices and guidelines2. is the most robust against What If?Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-18Testing Components PACT-- Parrallel Architecture for Component Testing IT--Hierarchiacal Incremntal TestingSoftware developers should do the unit testing.Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-19Testing Increments Stubs SimulationGrey Box TestingIncrement 5System?Unit?Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-20Testing ApplicationsNot much new here...Except Use Case ---> Test Case First N increments already testedCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 199821Testing Process Testing effectiveness is determined by the percentage of total defects found at each step in the testing phase Our goal is to find a larger percentage in the early stages and smaller percentages in later phases. Effectiveness of the overall testing process is defined as: Faults found during system testing divided by total number of faults found by all testing activities (including the testing done by customers that we call production).Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 199822Section 2Testing Object-Oriented ModelsIn this sectionwe will discuss: Static Testing Reviews Inspections Walkthroughs Inspecting Requirements Inspecting Analysis ModelsCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-23Static Testing Static testing is the testing of a product without actually executing it. Static testing is used to identify errors in requirements, analysis and design models, coding, plans, and tests. Static tests consist of: Reviews Walkthroughs Inspections (the primary focus of this chapter) The goals of static testing are: To detect defects as early aspossible in the developmentprocess To improve the software development processCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-24Inspections inObject-Oriented Projects Good news: The use of domain terminology in the development process makes it easier for clients to effectively participate in inspections. The iterative/incremental development process produces smaller chunks of work products allowing inspections to be more thorough in their coverage. Bad news: The iterative/incremental approach requires careful coordination and scheduling of inspections.Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-25C3 - Criteria for Testing Models Correctness It is judged to be equivalent to a reference standard believed to be an infallible source of truth, i.e. a domain expert. Completeness No required elements are missing. It is judged complete if the knowledge recorded is sufficient to support the goals of the current portion of the system being developed. Consistency There are no contradictions among the elements within the work product (internal) or between work products (external).Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 199826Create Domain-level Object Interaction Diagrams A sequence diagram1 is a diagram that formally describes a scenario. Each class is shown as a vertical line; each event as a horizontal arrow from the sender class to the receiver class; time flows from the top to the bottom of the diagram.1.Event trace or object interaction diagramInstructor Student Work GraderStudentGradeCreateSubmit (Student Work)DetermineRecord (Grade)CreateAssignmentBookGradeAssign (Assignment)Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 199827Domain Analysis Summary The minimal set of UML deliverables includes use cases, class diagrams, and sequence diagrams. Of these, the class diagram is core.n The sequence diagrams show how the use cases are played out in the class diagramCategoryStudent WorkActual Student Work AnnotationGuardianStudentAssignmentGrade******Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 199828Section 3Testing ComponentsIn this section we will discuss: A philosophy of testing Class testing- Functional testing- Structural testing- Interaction testing Subclass testing Cluster testing Framework testing-Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-29Levels of Testing Object-Oriented Systems Class testing - the smallest unit that should be tested. This level of testing combines traditional unit testing with some aspects of integration testing. Cluster testing - a set of closely interacting classes. The focus at this level is the interactions between objects in the cluster. System testing - a complete system that could be represented as a single class. The focus at this level is demonstrating the required functionality. Regression testing - becomes an integral part of the development process because each iteration refines products from the previous iteration and these products should be retested.Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-30Goals for a Testing Process The first goal in developing a testing process is to reduce the number of test cases that must be generated. The second goal is to minimize the number of test cases that must be executed and the results validated. Testing algorithms should provide a volume control approach that supports increasing the level of confidence in a systematic way.Minimal ExhaustiveAdequate RepresentativeTest Coverage Volume ControlWeightedRepresentativeCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-31Types of Objects Passive Also known as Abstract Data Types (ADT) or primitives No change in behavior based on state Active Also known as Finite State Machines (FSM) Significant changes in behavior depending upon stateCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-32Types of Test Cases Three types of test cases allow the tester to know that various aspects of the component or system under test are correct: Functional test cases are created to verify the product against the specification. Structural test cases are created to fully exercise the code. Interaction test cases are created to determine the correctness of class interaction.Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-33Definition of Terms A test case is a sequence of messages and an expected result. A test script first creates the environment for the execution of test case(s); it creates the OUT (object under test), tests it, and finally deletes it. A test suite is a grouping of test scripts that achieve some level of test coverage.For example, the functional suite covers the class specification.Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-34Topics A philosophy of testing Class testing Functional testing Structural testing Interaction testing Subclass testing Cluster testing Framework testingCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-35Functional Test Cases Constructed by analyzing the specification of the class - the aggregation of the specification of each method. Coverage may be expressed in terms of the percentage of post-conditions, or the percentage of transitions in the state representation, covered with the selected test cases. Impossible to guarantee any level of coverage of the underlying implementation. Synonyms: (1) Specification-based; (2) Black-box testing.Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-36Constructing Functional Test Cases Each pre-condition is used to establish the appropriate testing environment for object. Each post-condition is a logical statement constructed as a sequence of if-then clauses. These may be linked together with disjunctive (or) clauses. Each disjunctive clause may have several conjunctive (and) clauses. Example for the pop method of a Stack class:(If the stack contains items, then pop will return the last item added to the stack and the stack will contain one less item) or (if the stack is empty initially, then pop will raise an UnderFlow exception.)Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-37Constructing Functional Test Cases Create a test case for each or clause. For each or clause check the resulting environment to determine that every and clause within the or clause has been satisfied. Create a test case for each exception. Create test cases for obvious boundary conditions such as empty stacks, full stacks, a stack with only one element.Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-38Topics A philosophy of testing Class testing Functional testing Structural testing Interaction testing Subclass testing Cluster testing Framework testingCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-39Structural Class Testing Use techniques similar to those for procedurally-oriented analysis to identify test cases that exercise the paths through the code. Use partially implemented classes to speed integration testing. Complete stubbing is difficult since many lines of code will be messages to other objects.Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-40Topics A philosophy of testing Class testing Functional testing Structural testing Interaction testing Subclass testing Cluster testing Framework testingCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-41Interaction Test Cases - Intraclass Test cases are identified by considering methods that access a common attribute or send messages to other methods within the object. Coverage would be expressed as percentage of interactions tested.int widthset_width()get_width()Class RectangleHere the two methods are in the same classCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 199842Interaction Test Planning Matrix Intra-class interactions are identified as either one method invoking another (M), or as two methods messaging the same object (O), or two methods sharing data (D).startstartstopstopread_read_ lapsed lapsedresetresetSimple TimerCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-43Basic Test Case Execution Sequence Develop a functional test suite that covers the complete class specification. Develop state-based test cases for all of the transitions in the dynamic model. Develop structural test cases to cover every line of code and every conditional. Develop test cases that test the interactions between methods within the class. Develop test cases that cover the interactions between classes.Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-44Topics A philosophy of testing Class testing Functional testing Structural testing Interaction testing Subclass testing Cluster testing Framework testingCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-45Constructing Test Cases for a Set of SubclassesTestcaseconstructionstartsatthetopoftheinheritance hierarchy and progresses downward.Thisexpandsthenumberoftestcasesastheinterfaceoftheclassgrows.Thisprovidesopportunities for reuse of test cases.ListQueue StackDequeCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-46Testing Abstract Classes An abstract class should not be, and in some languages can not be, instantiated. Test cases should be constructed even for abstract classes. For each method, use the pre- and post-conditions of the methods to construct functional test cases. Fully implemented methods in an abstract class can have complete structural test suites developed and executed. It should be possible to static check (i.e. compile) the abstract class to test for interface errors. In C++, abstract (pure virtual) methods can be changed to provide a null or stubbed implementation, an object can be created, and some portion of the class can be tested.Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-47Questions Regarding Subclass Testing Questions: Can the test cases from the parent classes be used as test cases for the subclass? What parts of the subclass must be tested or retested? Important Question: Does the subclass conform to strict inheritance?Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-48Hierarchical Incremental Testing (HIT) Once a superclass is tested, its subclasses can benefit from that testing. HIT provides a technique for determining just which pieces of the subclass must be retested.Superclass Not Defined Redefinable Redefinable Abstract AbstractSubclass New Code No New CodeNew Code No New CodeNew CodeFunctionalTestsWrite? yes no no no noExecute? yes no yes no yesStructuralTestsWrite? yes no yes no yesExecute? yes no yes no yesInteractionTestsWrite? yes no*yes*no yesExecute? yes yes yes no yesm(){...} m(){...}m(){...}m(){...}m() m(){...} m()Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-49Topics A philosophy of testing Class testing Functional testing Structural testing Interaction testing Subclass testing Cluster testing Framework testingCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-50Cluster Interface A cluster is a set of closely related classes that interact more with each other than other classes. The specification for a cluster could take the same form as that of a class. There are a limited number of messages that come into the cluster from outside. These are the methods in the specification of the cluster.ClusterInterfaceClusterRest ofSystemCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-51Clusters Pre- and post-conditions can be derived for each method in the interface by using the conditions stated for those methods in their respective classes. The test cases are derived in the same way as for a class. Sometimes it is more sensible for cluster testing to be the first level of testing rather than testing the individual classes.Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-52Topics A philosophy of testing Class testing Functional testing Structural testing Interaction testing Subclass testing Cluster testing Framework testingCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-53Using Frameworks Frameworks reverse the usual idea of the direction of reuse: In reusing a component, our code calls the reused component. In reusing a framework, the framework calls our code that specializes some operation for which the framework contains a default or virtual placeholder..frameworkapplicationCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-54Testing When Using Frameworks A framework contains many abstract classes with many partially implemented methods. A framework contains many alternatives so there are many possible paths through the classes. The classes that we add to the framework are tested individually prior to the framework being tested (but remember, they inherit from the framework classes so they are not testable in isolation). Techniques such as HIT become important since the developer of an application will subclass from classes written by the framework developers. HIT assists in determining what portion of the behavior inherited from the framework should be retested.Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 199855Too Many Tests Consider the following system structure. How many test cases do we need to cover all of the combinations?m(P)ABCXYZPQ RCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 199856OATS to the Rescue Orthogonal Array Testing is a statistical technique that has been borrowed from manufacturing. The purpose of OATS is to assist in the selection of appropriate combinations of factors to provide maximum coverage from a test case with a minimum number of cases. OATS selects test cases so as to test the interactions between independent measures, called factors. Each factor has a finite set of possible values, called levels. Each column in the array corresponds to a factor. Each row corresponds to a test case. The rows are created to provide all possible pairwise combinations of possible levels for the factors.Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 199857Section 4Testing State-Based ClassesIn this section we will discuss: Testing Strategies Case Study - Timer ClassesCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 199858State-Based Testing All systems have state, but object-oriented systems decompose states into sufficiently small state machines to be manageable.Global state decomposedinto encapsulated states. Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-59Many Classes Are State Machines Here is a Queue classEmptyNotEmptyaddremove/itemaddremove/itemcreateremove/exceptionCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-60Adequacy Criteria for State-Based Structural Testing All methods exercised.Thisleveloftestingassuresthatnonecessarymethodsaremissing,butdoesnotguaranteethatallrequired states are present. All states visited.This level of testing assures that no states are missing, but does not guarantee that all required methodsare present. All transitions exercised.This level of testing would identify any missing states or methods. It would also identify some extrastates, if they are present. N-way switch cover.This level tests combinations of transitions. It will find additional extra states and some corruption. All paths followed.This level of testing would identify all missing and extra states and methods. It is seldom practical andoften not possible.Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 199861Section 5Parallel Architecture for Component TestingIn this section we will discuss: An Architecture for Managing Test CasesCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-62Architecture for Managing Test Cases Test Cases as Data Store sequence of messages as a file Interpret the messages and apply Test Cases as Methods Each test case is a method in a test harness class Take advantage of inheritance and other object-oriented featuresClassTesterOfClassmethod1()method2()testMethod1()testMethod2()testAll()Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-63Automating Testing A simple main method may be used or a more sophisticated one could be built:void main() {TesterOfX foo;foo.testAll();}orvoid main() {TesterOfX bar;bar.functionalSuite();bar.structuralSuite();bar.interactionSuite();}Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-64Parallel Architecture For Component TestingTesterOfClass1TesterOfClass2TesterOfClass3Class1Class2 Class3GenericTest HarnessTesterOfClass4TesterOfClass5TesterOfClass6Class4Class5Class6ProductionArchitecturePACTCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-65An Alternative ApproachClassmethod1()testMethod1()method2()testMethod2()...testAll() Advantages? DisadvantagesCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-66Generic Test Harness Class Methods in the Generic Test Harness include: Test scripts that sequence test execution Logging and reporting mechanisms Methods that catch exceptions Methods that watch for memory leakage Methods that access test hardwareCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 199867Section 6System TestingIn this session we will discuss: Strategies for System Testing Constructing System Test Cases Measuring the Effectiveness of System TestingCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-68Object-Oriented System Testing System testing is basically independent of the paradigm used to create the system. Why? However, system testing of object-oriented systems is different in the following ways: Because modeling is a vital part of the object-oriented development process, system testing can begin very early in development as those models are tested. The use cases built as part of analysis are tested early in the development process. These use cases then serve as the foundation for system test cases. Testing of models and use cases involves a closer interaction between clients and developers and testers.Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-69Object-Oriented System Testing System testing of object-oriented systems is different in the following ways (continued): The creation of PACT structures serves as the foundation for system and regression test suites. The use of frameworks may mean that a significant portion of system testing has already been done by the framework author.Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-70Role of the System Tester System testing can begin much earlier in the development cycle. System testers should: Develop the project test strategy during project planning phase. Participate in all system-level tests including tests of requirements models, design models, and the system implementation. Develop system test cases from use cases and from additional system requirements such as performance and capacity requirements. Audit developers class and cluster test processes. Take the lead in building regression tests.Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-71Use Cases and Test CasesSystemTest CasesDesign TestCasesUseCases Object-oriented projects often employ a use case model to represent the majority of system requirements. This model represents both the users of the system and their requirements for the system.Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-72Constructing Test CasesUse CaseScenario Scenario Scenario ... ...Test Case ... Test Case ... Test Case Each use case may produceseveral scenarios. Each scenario in turn may produceseveral test cases.Requirement RequirementCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-73Percentage of Defects Found Testing effectiveness is determined by the percentage of total defects found at each step in the testing phase Our goal is to find a larger percentage in the early stages and smaller percentages in later phases. Effectiveness of the overall testing process is defined as: Faults found during system testing divided by total number of faults found by all testing activities (including the testing done by customers that we call production).Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-74Iterative Measurement of Effectiveness The effectiveness ratios should be calculated after the testing of each iteration. The progression of the values from one iteration to another provides trends that indicate the effectiveness of the testing process. Expect more defects to be found in the early testing phases and fewer in the later testing phases as the increment matures. Do the number of errors always approach zero as the component matures?Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 199875Section 7SummaryCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-76Higher Reliance onFunctional Testing Functional tests are derived from the class/method specifications. Classes and methods are specified using pre-conditions, post-conditions, and invariants. At least one test should be created for each post-condition statement. Use equivalence classes to minimize the number of test cases created. Select test cases to verify the correct operation at the boundary conditions.Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-77Subclass Testing Follow strict inheritance Pre-conditions same or less Post-conditions same or more Invariants same or more Hierarchical Incremental TestingSuperclassNot DefinedVirtual Virtual Abstract AbstractSubclassNew CodeNo New CodeNew CodeNo New CodeNew CodeFunctionalTestsWrite? Yes No No No NoExecute? Yes No Yes No YesStructuralTestsWrite? Yes No Yes No YesExecute? Yes No Yes No YesInteractionTestsWrite? Yes No Yes No YesExecute? Yes Yes Yes No YesCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-78Parallel Test Harness DevelopmentTesterOfClass1TesterOfClass2TesterOfClass3Class1Class2 Class3GenericTest HarnessTesterOfClass4TesterOfClass5TesterOfClass6Class4Class5Class6ProductionArchitecturePACTCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-79Interaction Testing Use OATS to select a random but equally distributed selection of test cases.L4 (23)1 2 31 1 1 12 1 2 23 2 1 24 2 2 1L8 (27)1 2 3 4 5 6 71 1 1 1 1 1 1 12 1 1 1 2 2 2 23 1 2 2 1 1 2 24 1 2 2 2 2 1 15 2 1 2 1 2 1 26 2 1 2 2 1 2 17 2 2 1 1 2 2 18 2 2 1 2 1 1 2L9 (34) 1 2 3 41 1 1 1 12 1 2 2 23 1 3 3 34 2 1 2 35 2 2 3 16 2 3 1 27 3 1 3 28 3 2 1 39 3 3 2 1Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-80System Testing System testing is less dependent on the paradigm used in creating the system than component testing. The difference in system testing of object-oriented systems is the availability and applicability of use cases. Structured use cases become the basis of structured test cases. System testers should be involved from the beginning in: Creating and executing system tests Auditing the developers ongoing testing Emphasizes the importance of regression testing. Measure the effectiveness of system testing as input to the process improvement process.Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-81Organizing for Testing Must have a complete testing architecture defined.TestGroupDeveloper-ledTestingTestGroupTimeFocusCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-82Thanks On behalf of Software Architects, thanks for attending this tutorial. Remember us for future object-oriented training or consulting. Let us know about your testing experience.Wed like to hear about your successes and your difficulties. My e-mail address is: korson@software-archtiects.comCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 199883Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 199884BibliographyCopyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-85Bibliography[1]SoftwareQualityManagementII.BuildingQualityintoSoftware.Comp.Mech.Publications,Southampton, UK, 1994.[2] Marick, Brian. Testing Object-oriented Software. ACM, New York, NY, USA, 1995.[3]R.D.Ammann,M.H.;Cameron.Measuringprogramstructurewithinter-modulemetrics.IEEEComput. Soc. Press, Los Alamitos, CA, USA, 1994.[4] Thomas R. Arnold and William A. Fuson.Testing in a perfect world. CACM, 37(9):78-86, Sep-tember 1994.[5] R.J.; Wright C.; Zweig-D. Banker, R.D.; Kauffman. Automating output size and reuse metrics in arepository-basedcomputer-aidedsoftwareengineering(case)environment.IEEETransactionsonSoftware Engineering, 20(3):169-87, March 1994.[6]A.Barbey,S.;Strohmeier.Theproblematicsoftestingobject-orientedsoftware.Comp.Mech.Publications, Southampton, UK, 1994.[7]B.Beizer.SoftwareTestingTechniques.VanNostrandReinholdCompany,Inc.,NewYork,1990.[8] Robert V. Binder. Design for testability in object-oriented systems. CACM, 37(9):87-101, Septem-ber 1994.[9]J.ProwseBrownlie,RandM.S.Phadke.RobusttestingofAT&Tpmx/starmailusingoats.AT&T Technical Journal, pages 41-47, May-June 1992.[10]D.M. Chao, B.P.; Smith. Applying software testing practices to an object-oriented software devel-opment. OOPS Messenger, 5(2):49-52, April 1994.[11]Thomas J. Cheatham and Lee Mellinger.Testing object-oriented software systems. In 1990 ACMEighteenth Annual Computer Science Conference Proceedings, pages 161-165, 1990.[12][13]In Sang Chung. Methods of comparing test criteria for object-oriented programs based on sub-sumption,powerrelationandtestreusability.JournaloftheKoreaInformationScienceSociety,22(5):693-704, May 1995.[14]M.-C.Chung,C.-M.;Lee.Object-orientedprogrammingtestingmethodology.InternationalJournal of Mini and Microcomputers, 16(2):73-81.[15]Z.E.Cusack.Inheritanceinobjectorientedz.InEuropeanConferenceonObject-OrientedPro-gramming, pages 167-179, 1991.[16]S.R. Davis.Armor cladding c++ classes.C++ Report vol.6, no.8, p.36-9, 41,Oct. 1994.[17]P. K. Devanbu, D. S. Rosenblum, and A. L. Wolf. Automated construction of testing and analysistools.In Proceedings of 16th International Conference on Software Engineering, pages 241-250, LosAlamitos,CA,1994.IEEEComput.Soc.Tech.CommitteeonSoftwareEng.,IEEEComput.Soc.Press.Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-86[18]R.K. Doong and Phyllis G. Frankl. Case studies on testing object-oriented programs. In Proceed-ings of the Fourth Symposium on Testing, Analysis and Verification(TAV4), pages 165-177, 1991.[19]R. J. DSouza and R. J. Leblanc Jr.Class testing by examining pointers. Journal of Object-Orient-ed Programming, 7(4):33-39, July-Aug 1994.[20]D. Duke and Roger Duke.Towards a semantics for object-z.In VDM and Z-Formal Methods inSoftware Development, pages 244-259, 1990.[21]DawnStockbridge Dunaway andEugenia Gillan. Applying object-orienteddesign principles todeveloping a test strategy. In 11th International Conference and Exposition on Testing Computer Soft-ware, pages 341-368. ASQC and STSC, June 13-16 1994.[22]M.FelderandP.SanPietro.Testingbyexecutinglogicspecification.InA.D.Halang,W.A.;Stoyenko, editor, Proceedings of the NATOAdvanced Study Institute on Real Time Computing, pag-es 683-684, Berlin, Germany, 1994. Springer-Verlag.[23]S.P. Fielder. Object-oriented unit testing. Hewlett-Packard Journal, April 1989.[24]S. Frost. Modeling for the rdbms legacy. Object Magazine, 4(5):43-4, 46, 48-51, Sept. 1994.[25]T.J. Gattis, G.F.; Cheatham. Testing object-oriented software. ACM, New York, NY, USA, 1995.[26]MaryJeanHarroldandJohnD.McGregor.Incrementaltestingofobject-orientedclassstruc-tures.InProceedingsoftheInternationalConferenceonSoftwareEngineering,pages68-80,May1992.[27]P. A. Hausler, R. C. Linger, and C. J. Trammell. Adopting cleanroom software engineering with aphased approach. IBM Systems Journal, 33(1):89-109, 1994.[28]J.H. Hayes. Testing of object-oriented programming systems (oops), 1994.[29]Ivar Jacobson, Magnus Christerson, Patrik Jonsson, and Gunnar Overgaard. Object-Oriented Soft-ware Engineering. Addison-Wesley, 1991.[30]PaulC.JorgensenandCarlErickson.Object-orientedintegrationtesting.CACM,37(9):30-38,September 1994.[31]P. Juettner, P. Zimmerer, U. Naumann, and S. Kolb.A complete test process in object-orientedsoftware development. In The 7th International Software Quality Week in San Francisco, pages 3-A-2.Software Research, Inc., May 17-20 1994.[32]P. Juttner, S. Kolb, S. Sieber, and P. Zimmerer. Testing major object-oriented software systems.Siemens Review (Special Issue), pages25-29, Spring 1994.[33]S. Kanjulal, S. T. Chakradhar, and V. D. Agrawal. A test function architecture for interconnectedfinite state machines. In Proceedings of 7th International Conference on VLSI Design, pages 113-116,Los Alamitos, CA, 1994. VLSI Soc. India, IEEE Comput. Soc. Press.[34]Mark Lorenz.Object-Oriented Software Development:a practical guide. PTR Prentice Hall, En-glewood Cliffs, N.J., 1993.[35]T. J. McCabe and A. H. Watson.Combining comprehension and testing in object-oriented devel-opment. Object Magazine, 4(1):63-64,March-April 1994.Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-87[36]R.L. McGarvey. Object-oriented test development in ABBET. IEEE,New York, NY, USA, 1994.[37]J.D. McGregor and D. Dyer.A note on inheritance and state machines. Technical report, ClemsonUniversity, 1993.[38]John McGregor. Constructing functional test cases using incrementally derived state machines.In11th International Conference and Exposition on Testing Computer Software, pages 377-386. ASQCand STSC, June 13-16 1994.[39]John D. McGregor. Selecting functional test cases for a class. In The7th International SoftwareQuality Week in San Francisco, pages 2-T-2.Software Research, Inc., May 17-20 1994.[40]John D. McGregor and Douglas M. Dyer.Selecting specification- based test cases for object-ori-ented systems.Technical Report, Clemson University, 1993.[41]John D. McGregor and Tim Korson. Understanding object-oriented: A unifying paradigm.Com-munications of the ACM, 33(9):40-60, September 1990.[42]John D. McGregor and Timothy D. Korson.Integrating object-oriented testing and developmentprocesses.CACM, 37(9):59-77, September 1994.[43]M. D. Smith and D.J. Robson. A framework for testing object-oriented programs. Journal of Ob-ject-Oriented Programming, 5(3):45-53, June 1992.[44]Gail C. Murphy, Paul Townsend, and Pok Sze Wong.Experiences with cluster and class testing.CACM, 37(9):39-47, September 1994.[45]Udo Naumann.Experiences in testing object-oriented software.In 11th International Conferenceand Exposition on Testing Computer Software, pages 373-376. ASQC and STSC, June 13-16 1994.[46]A. Parrish and D. Cordes. Applying conventional unit testing techniques to abstract data type. In-ternational Journal of Software Engineering and Knowledge, 4(1):103-122, March 1994.[47]A. Parrish, D. Cordes, and M. Govindarajan. Removal from object- oriented modules.In The 7thInternational Software Quality Week in San Francisco, pages 2-T-1. Software Research, Inc., May 17-20 1994.[48]D.E. Perry and G.E. Kaiser.Adequate testing and object-oriented programming.Journal of Ob-ject-Oriented Programming, 2:13-19,Jan/Feb 1990.[49]M.; Staples-G. Petruv, V.; Ross. Testing for tomorrows professionals, 1994.[50]Robert M. Poston. Automated testing from object models. CACM, 37(9):48-58, September 1994.[51]Marc Rettig. Testing made palatable. Communications of the ACM,34(5):25-29, May 1991.[52]P.G.Roong-KoDoong;Frankl.Theastootapproachtotestingobject-orientedprograms.ACMTransactions on Software Engineering and Methodology, 3(2):101-30, April 1994.[53]M.J.HarroldandRothermel,G.;Selectingregressiontestsforobject-orientedsoftware.IEEEComput. Soc. Press, Los Alamitos, CA, USA, 1994.[54]S.J. Samadzadeh, M.H.; Khan.Stability, coupling, and cohesion ofobject-oriented software sys-tems. ACM, New York, NY, USA, 1994.[55]A. Serra and P. Nesi.Object-oriented approach for a non-invasive, automatic testing tool.In The7th International Software Quality Week in San Francisco, pages 4-A-3. Software Research, Inc., May17-20 1994.Copyright 1993-1998 Timothy Korson. All rights reserved.Borland Conference, Denver, August 11, 1998-88[56]S. Siegel. Automating integration testing for oo projects. In The 7th International Software QualityWeek in San Francisco, pages 4-A-4. Software Research, Inc., May 17-20 1994. Paper not included inthe conference proceedings.[57]A.R. Siepmann, E.; Newton.Tobac:a test case browser for testing object-oriented software. SIG-SOFT Software Engineering Notes special issue, pages 154-68, 1994.[58]M.D. Smith and D.J. Robson.Object-oriented programming - theproblems of validation. In Pro-ceedings of the Conference on Software Maintenance, pages 272-281, 1990.[59]Bob Stahl. How to test object-oriented software. In 11th International Conference and Expositionon Testing Computer Software, pages 369-372. ASQC and STSC, June 13-16 1994.[60]S. Subramanian, Wei-Tek Tsai, and S. H. Kirani. Hierarchical dataflow analysis for o-o programs.Journal of Object-Oriented Programming, 7(2):36-46, May 1994.[61]C.D. Turner and D.J. Robson.The testing of object-oriented programs. Technical report, Univer-sity of Durham, England, 1992.[62]J.; Mraz R. Von Mayrhauser, A.; Walls.Sleuth : a domain basedtesting tool. In Int. Test Confer-ence, Altoona, PA, USA, 1994.[63]Chi-Ming Chung; Ming-Chi Lee; Ching-Chian Wang.Inheritance testing for object-oriented pro-gramming by transitive, closure strategies. Advances in Modelling & Analysis B, 31(2), 1994.[64]K.; Srivastava A.; Krueger-J. Weber, R.; Thelen. Automated validation test generation. 1994.[65]StuartH.Zweben.Testingformallyspecifieddata-orientedmodulesusingprogram-basedtestdata adequacy criteria.Technical report, Ohio State University, 1989.