testing terminology project group org. review terminology oo concepts development products team...

Testing Terminology

• Project Group Org.• Review• Terminology• OO Concepts• Development Products• Team Meeting

CEN 5076 Class 2 – 09/12

CEN 5076 Class 2 - 09/12 2

What is software?

• Software is the instruction codes and data necessary to accomplish some task on a computer.

• Includes all representations and artifacts of the code and data i.e.,– program source code and data files

– models created during analysis and design activities.

CEN 5076 Class 2 - 09/12 3

Software Process

• S/w Specification – requirements elicitation (func. & non-func.) and analysis.

• S/w Development – systems design, detailed design (OO design), implementation.

• S/w Validation – validating system against requirements (testing).

• S/w Evolution – meet changing customer needs and error correction (maintenance).

CEN 5076 Class 2 - 09/12 4

Unified Software Development Process

Analysis model

Use case model

Design model

Deployment model

Implementation model

Test model

specified by

realized by

distributed by

verified by

implemented by

Process is use case driven!

Requirementscaptured as a set of use cases.

All models are relatedthrough traceability dependencies.

System Development

CEN 5076 Class 2 - 09/12 5

View of UDSP from a Testing Perspective

Analysis model

Use case model

Design model

Deployment model


Test model

specified by

realized by

distributed by

Verified and/orvalidate

implemented by

Process is use case driven!

Requirementscaptured as a set of use cases.

All models are relatedthrough traceability dependencies.

System Development

CEN 5076 Class 2 - 09/12 6

What is software testing?

• Software testing is the process of operating software under specified conditions, observing or recording the results and making an evaluation of some aspect of the software.

(IEEE/ANSI std 610.12-1990)

CEN 5076 Class 2 - 09/12 7

What is software testing?

• Software testing is the process of uncovering evidence of defects in software systems.A defect can be introduced during any phase of development or maintenance and results from one or more “bugs” – mistakes, misunderstandings, omissions or even misguided intent on the part of the developers.

(McGregor & Sykes 2001)

CEN 5076 Class 2 - 09/12 8

Testing Terminology

The following defns are taken from Binder 2000 and McGregor & Sykes 2001.

• Failure – is the manifested inability of a system or component to perform a required function within specified limits e.g. abnormal termination, or unmet time and space constraints of the software.

• A fault is missing or incorrect code.• An error is a human action that produces a fault.

CEN 5076 Class 2 - 09/12 9

Testing Terminology cont

• A bug is an error or a fault.• Debugging is the process of tracking down the

source of failures (errors and/or faults) and making repairs.

• A test case specifies the pretest state of the component under test (CUT) and its environment, the test inputs and conditions, and the expected results.

• A test point is a specific value for test case input state variables.

CEN 5076 Class 2 - 09/12 10

Testing Terminology cont

• A test suite is a collection of test cases, typically related to some testing goal or criterion.

• A test driver is a class or utility program that applies test cases to a CUT.

• A stub is a partial, temporary implementation of a component.

• A test harness is a system of test drivers and other tools to support test execution.

CEN 5076 Class 2 - 09/12 11

Testing Approach (McGregor & Sykes)

• Kinds of testing– Model testing– Class testing, replaces unit testing– Interaction testing, replaces integration testing– System (and subsystem) testing– Acceptance testing– Deployment/self testing

• Motto: Test early, Test often, Test enough

CEN 5076 Class 2 - 09/12 12

The Testing Perspective • Testing perspective - looking at any

development product and questioning its validity.• Software testing accomplished by a combination

of inspections, reviews, and test executions.

• Inspection: – Defn: A formal small group procedure for detailed

scrutiny of a technical work product.– Cost effective and efficient for defect prevention.– Process has a formal entry-exit criteria, defined

checklist, captures process and product metrics, has a follow up session.

CEN 5076 Class 2 - 09/12 13

The Testing Perspective cont • Review:

– Defn: A meeting at which a work product or set of work products is presented to project personnel, managers, users, customers, or other interested parties for commitment and approval. [IEE 610]

– Types of review: client, project, peer (inspections and walkthroughs), status.

– Examines the software with the purpose of finding errors and faults even before the software is executed.

– Deeper interest in s/w than do inspections.

CEN 5076 Class 2 - 09/12 14

The Testing Perspective cont

• Test execution:– Defn: Testing the s/w in the context of a running

program.– Challenges:

• identifying suitable input (test data selection criteria and test adequacy criteria),

• determining correct outputs (oracle problem), • determining how to observe the outputs

• Walkthrough (informal review)– A group meeting to conduct an informal peer

review of a technical work product.

CEN 5076 Class 2 - 09/12 15

OO Concepts

Before testing software it is important to know the paradigm used to develop the software. Some OO concepts are:– Object– Message– Interface– Class– Inheritance– Polymorphism

CEN 5076 Class 2 - 09/12 16

OO Concepts - Definitional and Operational Semantics

Dynamic

Operational

Definitional

Static

Bound atruntime

Bound at compile time

Object

Bound atAnalysis time

Interface/ClassDefinitions

CEN 5076 Class 2 - 09/12 17

OO Concepts cont

• Object - an operational entity that encapsulates both specific data values an the code that manipulates those values.

– Basic computational entity.

– Community of objects collaborate to solve some problem.

– During program execution objects are created, modified, accessed, and/or destroyed.

CEN 5076 Class 2 - 09/12 18

OO Concepts cont

• Object (testing perspective) Encapsulation - makes the complete defn of an object

“easy” to identify, easy to pass around the system, and easy to manipulate.

Information hiding – sometimes makes observation of changes to object difficult i.e., checking the state of the object.

State – can become inconsistent and become source of errors.

Lifetime – must check state during life time.

CEN 5076 Class 2 - 09/12 19

OO Concepts cont

• Message (member function call - C++, method invocation - Java) – is a request (from sender) that an operation be performed by some object (receiver).

– Pass data via parameters (value and/or reference).

– Receiver can return value to sender.

– Collaboration achieved by sending messages between objects.

CEN 5076 Class 2 - 09/12 20

OO Concepts cont

• Message (testing perspective)– Sender - may send the message at the

incorrect time.– Receiver - may not be ready for a specific

message and take corrective action.– Actual parameters – used to update the state

of the receiver; must be in correct state before and after message is processed.

– Sender – must use correct interface for intended message.

CEN 5076 Class 2 - 09/12 21

OO Concepts cont

• Interface – is an aggregation of behavioral declarations, related by a single concept.– Building blocks for specifications. – Specification defines the total set of public

behaviors for a class.– Java has interface, C++ has abstract base

class with all public pure virtual member functions.

CEN 5076 Class 2 - 09/12 22

OO Concepts cont

• Interface (testing perspective)– Encapsulates operation specifications.– Used to incrementally build the spec. of

larger grouping e.g., classes, packages.– Closely related to the design model.– Relationships with other interfaces and

classes identifies most dependencies between classes.

CEN 5076 Class 2 - 09/12 23

OO Concepts cont

• Class – set of objects that share a common conceptual basis.

– Template for creating objects.

– Type (user-defined or built-in) = set + operations define over the set

– Used to define OO programs

– Object instance of a class.

– Class specification: the what

– Class implementation: the how

CEN 5076 Class 2 - 09/12 24

OO Concepts – Class Spec.

• Class Specification – describes what the class represents and what an instance of the class can do.

– Includes a specification for each operation of an instance.

– Operation is an action that can be applied to an object to obtain a certain effect.

– Operations are categorized as follows:

CEN 5076 Class 2 - 09/12 25

OO Concepts – Class Spec. cont

Accessor (or inspector) – provides info about an object

Modifier (or mutator) – changes the state of an object.

Constructor (class operation) - creates a new object and initializes the new instance (data and methods).

Destructor – performs processing on object prior to the end of the object’s lifetime.

CEN 5076 Class 2 - 09/12 26

OO Concepts – Class Spec. cont Operation semantics are specified using

precondition, postconditions, and invariants.

• Preconditions – prescribe conditions that must hold before the operation can be performed.

Stated in terms of:

1. attributes of receiver object, and/or

2. actual parameters of the message requesting the operation.

CEN 5076 Class 2 - 09/12 27


Specifying operation semantics:

• Postconditions – prescribe conditions that must hold after the operation is performed.

Stated in terms of:

1. attributes of the receiver object,

2. actual parameters in the sender’s message,

3. value of any return value, and/or

4. exceptions that might be raised.

CEN 5076 Class 2 - 09/12 28


Specifying operation semantics:

• Invariants – prescribe conditions that must always hold during the lifetime of an object. A class invariant describes a set of operating boundaries.

- Treated as an implied postcondition for each operation.

- Must hold whenever an operation completes.

- May violate invariants during execution of operation

- Can define interface invariants.

CEN 5076 Class 2 - 09/12 29


• In this course we will be using OCL to specify operation semantics.

• See handout given in class.

• When you submit the final project all classes and operations should be specified by invariants, preconditions and postconditions.

• We will use the OCL statement to generate unit test cases.

• Start writing OCL statements to specify classes now!!!

CEN 5076 Class 2 - 09/12 30


• Contracts are constraints on a class that enable a sender (caller) and a receiver (callee) to share the same assumptions about the class (Meyer ’97).

• A contract specifies constraints that the caller must meet before using the class as well as constraints that are ensured by the callee when used.

• Operation semantics are specified using precondition, postconditions, and invariants.

CEN 5076 Class 2 - 09/12 31


• Two basic approaches to write specs for operations between the receiver and sender:

1. Contract – emphasizes preconditions with simple postconditions. Example P. 26

2. Defensive programming – emphasizes the postconditions with simple preconditions. Example P. 28.

• The decision on which approach(es) is used to write specs is made at the design phase.

CEN 5076 Class 2 - 09/12 32

OO Concepts – Class Spec. cont • Design by contract – views the relationship

between a class and its clients as a formal agreement, expressing each party’s rights and obligations (Meyer ’97).

– The precondition is an obligation for the client (sender) and a benefit for the supplier (receiver).

– The postcondition is a benefit for the client and an obligation for the supplier.

CEN 5076 Class 2 - 09/12 33

OO Concepts – Class Spec. cont • Defensive design – intended to ensure the

continuing function of a piece of software in spite of unforeseeable usage of said software (Wikipedia).

– Results in defensive programming.

– Supports security e.g., buffer overflow

– Operation typically returns some indication concerning the status of the result of the request – success or failure.

– Uses exception handling mechanism to support above

CEN 5076 Class 2 - 09/12 34

OO Concepts – Class Spec. cont Contract Defensive Prog.

• Mutual responsibility between sender and receiver.

• Eliminate need of receiver to verify preconditions.

• How to enforce contract?•Simplifies class

testing.•Complicates

interaction between classes.

• Lack of trust between sender and receiver.

• Increase size of code and execution time.

• “Garbage in eliminate garbage out.”

•Complicates class testing.

•Complicates interaction testing.

CEN 5076 Class 2 - 09/12 35

OO Concepts – Class Spec. cont • Contract

– Complicated interaction testing: must ensure that any sender meets preconditions.

• Defensive Programming

– Complicated class testing: test cases must address all possible outcomes

– Complicated interaction testing: must ensure all possible outcomes are produced and that they are properly handled by a sender.

CEN 5076 Class 2 - 09/12 36

OO Concepts – Class Impl. • Class implementation - describes how an

object represents attributes and carries out operations.

Components:

– Data members (instance variables or attributes): store the state of an instance.

– Methods (member functions or routines): code used to implement some algorithm that accomplishes one operation.

– The data members and methods exhibit properties that can affect the testing process.

CEN 5076 Class 2 - 09/12 37

OO Concepts – Class Impl. • Properties of data members and methods

(features):

– Types (built-in, user-defined)

– Accessibility (private, protected, private)

– Shared class features (static)

– Polymorphism

– Dynamic binding

– Deferred features

– Exception handling

– Concurrency

CEN 5076 Class 2 - 09/12 38

OO Concepts – Testing perspective

Potential causes of failure within design and impl.:

• Initialization of the attributes in a new instance.

• Implementation of operations.

• Does implementation “satisfy” the specification?

• Implementation may not support all required operations.

• Constraints need to be written in a “testable” manner. This is why we will use OCL.

CEN 5076 Class 2 - 09/12 39

OO Concepts – Inheritance

Inheritance is a relationship between classes that allows for the definition of a new class based on the definition of an existing class.– Allows the reuse of both specification and

implementation.

– New class referred to as subclass or derived class (descendents).

– Existing class is referred to as superclass or base class (ancestors).

– The inheritance relation is represented as an inheritance hierarchy.

CEN 5076 Class 2 - 09/12 40


– Some languages allow multiple inheritance.

– Abstract classes?

– Concrete subclasses?

– Deferred features?

– Public, protected and private inheritance?

– Virtual inheritance?

• Focus of inheritance in object design is to reduce redundancy and enhance extensibility.

CEN 5076 Class 2 - 09/12 41


• Factoring all redundant behavior into a single superclass reduces the risk of inconsistencies when changes are made.

• Use of abstract classes and interfaces allows for extensibility i.e., new specialized behavior can be written in new subclasses.

• Inheritance is a powerful mechanism and should be carefully considered before use.

CEN 5076 Class 2 - 09/12 42

OO Concepts – Inheritance • Note there is usually strong coupling between

classes in the inheritance hierarchy.

• There are two types of inheritance:

1. Implementation inheritance – sole purpose of reusing code.

2. Specification inheritance (or interface inheritance) – classification of concepts into type hierarchies.

CEN 5076 Class 2 - 09/12 43


Liskov Substitution Principle:

• If an object of type D can be substituted in all places where an object of type C is expected then D is a subtype of C (strict inheritance).

– Models the is a (or is a kind of) relationship.

– Promotes specification inheritance.

– Allowed behavior for a new subclass D derived from C:• Preconditions in D must be the same or

weaker than in C.

CEN 5076 Class 2 - 09/12 44

OO Concepts – Inheritance • Postconditions in D must be the same or stronger

than in C.

• Class invariant in D must be the same or stronger than in C.

• Viewed from perspective of observable states:- All observable states and transitions must be

preserved between D and C.

- D may add transitions between these states.

- D may add observable states as long as each is either concurrent or a sub-state of an existing state.

CEN 5076 Class 2 - 09/12 45


Testing perspective:

• Bugs can be propagated.

• May reuse test cases, because derived classes inherits part of its spec. and impl. from its superclass.

• Models is a kind of relationship i.e., specification inheritance.

• Proper use of inheritance in design leads to benefits in execution and testing of classes.

CEN 5076 Class 2 - 09/12 46

OO Concepts – Polymorphism Polymorphism is the ability to treat an object as

belonging to more than one type.

There are different types of polymorphism: inclusion and parametric.

1. Inclusion polymorphism – the occurrence of different forms of the same class. Gives the programmer the ability to substitute an object of the same type in a request for an operation.– Polymorphism arises from inheritance.

– C++ inheritance of classes

– Java inheritance of classes and interfaces

CEN 5076 Class 2 - 09/12 47

OO Concepts – Polymorphism

Testing perspective for inclusion inheritance:

– Allows systems to be extended incrementally by adding classes; unanticipated behavior may occur.

– Allows any operation to have one or more parameters of a polymorphic reference; increases number of possible kinds of actual parameters to be tested.

– Allows the operation to specify replies that are polymorphic references; referent could be incorrect or unanticipated by the sender.

CEN 5076 Class 2 - 09/12 48

OO Concepts – Polymorphism

2. Parametric polymorphism – is the capability to define a type in terms of one or more parameters.

- C++ templates: instantiation of a “new” class at compile-time.

Testing perspective:

- Supports a different type of relationship from inheritance.

- Must make sure operations work for all possible instances of the “new” class. Should be done during static analysis e.g., walkthrough.

CEN 5076 Class 2 - 09/12 49

Development Products - UML Diagrams

• A diagram is a graphical representation of a set of elements, most often rendered as a connected graph of vertices (things) and arcs (relationships).

• Diagrams in UML – structural diagrams (static) Class – shows a set of classes, interfaces, and

collaborations and their relationships. Object – class diagram that includes only instances. Component – class diagrams that focus on system’s

components e.g., executables, libraries, tables, files … Deployment – shows the configuration of run-time

processing nodes and the components that live on them

CEN 5076 Class 2 - 09/12 50

Diagrams in UML cont• Diagrams in UML – behavioral diagrams

(dynamic) Use case - represents the actors and uses of the

system and relationships between the uses. Sequence - interaction diagram that focuses on the

time ordering of messages. Collaboration - interaction diagram that focuses on

the structural organization of objects that send and receive messages

Statechart – shows a state machine that shows different states (data-attribute values) of the object driven by events (messages).

Activity – shows flow of control from activity to activity, emphasizes flow of control among objects.

CEN 5076 Class 2 - 09/12 51

Development Products - Analysis model Good documentation is critical for successful

development and successful testing.

Analysis model

• Purpose to define the problem to be solved and determine the requirements for a solution to the problem.

• Two levels of analysis:– Domain analysis focuses on understanding the

problem domain.

– Application analysis focuses on a specific problem and the requirements for a solution.

CEN 5076 Class 2 - 09/12 52

Development Products - Analysis model • Analysis model represents a system from the

perspective of what it is suppose to do.

• Analysis model provides an understanding for the various stakeholders.

• The diagrams in the analysis model should be checked for:

– Accuracy

– Consistency

– Completeness

CEN 5076 Class 2 - 09/12 53

Development Products - Analysis model • Models in the analysis model:

Functional model – represented by use cases and scenarios.

Analysis object model – represented by class and object diagrams.

Dynamic model – represented by statechart and sequence diagrams.

• Entity, boundary and control objects are identified.

• Generalization and specification activities performed.

CEN 5076 Class 2 - 09/12 54

Development Products – Design Model

Design model

• Represents how the software meets the requirements.

• Refines and extends the analysis model.

• Can reuse and extend test cases.

• System design: Identification of design goals

• Performance – response time, throughput, memory.

• Dependability – robustness, reliability, availability, fault tolerance, security, safety.

• Maintenance – extensibility, modifiability, adaptability, portability, readability, traceability of reqs.

CEN 5076 Class 2 - 09/12 55

Development Products – Design Model

Software architecture describing subsystem decomposition: responsibilities, dependencies, mapping to h/w, policy decisions – control flow, access control and data storage.

Boundary use cases identified.

• Object design Reuse - COTS identified, class libraries and other

components selected, design patterns identified. Interface specification – operations, arguments,

type signatures, exceptions (Here we use OCL).

CEN 5076 Class 2 - 09/12 56

Development Products – Design Model Restructuring – manipulate system model to meet

reuse and other design goals. Optimization – changing algorithms, optimize data

associations.

• Many of the diagrams used in the analysis model are used, but with an emphasis on the solution and not the problem.

CEN 5076 Class 2 - 09/12 57

Development Products – Impl. Model


• Involves mapping design models to source code i.e., a number of transformations are performed on the model.

• Transformations include: Optimization of the system model to address

performance issues. Realizing associations – map associations to source

code constructs such as references and collections of references.

CEN 5076 Class 2 - 09/12 58


Mapping contracts to exceptions – what happens when the contract is broken i.e., raising exceptions, throwing exceptions.

Mapping class models to a storage schema – select persistent data storage e.g. DBMS

• Mapping concepts Model transformations – e.g. attribute address to

a class address. Refactorings – improve a single aspect of the

source code

CEN 5076 Class 2 - 09/12 59


Forward engineering – production of source code templates.

Reverse engineering – produces a model form the code.

CEN 5076 Class 2 - 09/12 60

Laws of Testing Handbook of SSE

• Online debugging is more efficient than offline debugging (Sackman’s).

• Testing can show the presence but not the absence of errors (Dijkstra).

• A developer is unsuited to test his or her code (Weinberg’s).

• Approximately 80 percent of defects come from 20 percent of modules (Paret-Zipf-type)

• Performance testing benefits significantly from system-level benchmarks (Gray-Serlin)

CEN 5076 Class 2 - 09/12 61

Laws of Testing Handbook of SSE• Usability is quantifiable (Nielsen-Norman law).• Partition testing is more effective than random

testing (Gutjahr’s hypothesis).• The adequacy of a coverage criterion can

only be intuitively defined (Weyuker’s hypothesis).

• The test suite needed to verify an arithmetic path expression can be determined (Endres-Glatthaar hypothesis).

• Suspicion-based testing can be more effective than most other approaches (Hamlet’s hypothesis).

testing terminology project group org. review terminology oo concepts development products team...

Documents

software testing

software systems

test cut

test point

test harness

test inputs

test execut

system of test drivers