software quality: testing and verification ii. 2 1.a failure is an unacceptable behaviour exhibited...
Post on 21-Dec-2015
213 views
TRANSCRIPT
Software Quality:Testing and Verification II
2
1.A failure is an unacceptable behaviour exhibited by a system
— The frequency of failures measures software reliability Low failure rate = high reliability
— Failures result from violation of a requirement
2.A defect is a flaw that contributes to a failure— It might take several defects to cause one
failure
3.An error is a software decision that leads to a defect
Software Flaws are identified at three levels
3
Eliminating Failures: Testing vs Verification
Testing = running the program with a set of inputs to gain confidence that the software has few defectsGoal: reduce the frequency of failures When done: after the programming is completeMethodology: develop test cases; run the program with
each test caseVerification = formally proving that the software has no defects
(in this case, the program is said to be “correct”)Goal: eliminate failuresWhen done: before, during and after the programming is
completeMethodology: write separate specifications for the
code; prove that the code and the specifications are mathematically equivalent
4
Program Correctness
The correctness of a program is based on a specific standard.
That standard is called a specification.
int max (int a, int b) { int m; if (a >= b) m = a; else m = b; return m;}
E.g., a specification for the above program might be that it “finds the maximum value of any two integers.”
5
Formalizing a Specification
A Formal specification is written as a logical expression called an assertion.
An assertion describes the state of the program’s variables.Two key assertions are the program’s precondition and its
postcondition.
int max (int a, int b) { int m; if (a >= b) m = a; else m = b; return m;}
A domain is a set of values over which a variable is well defined.
The primitive types (int, float, boolean, etc.) and standard Java classes (String, Vector, HashMap, etc.) provide domains for reasoning about programs.
Postcondition Q = what’s ensured tobe true when the program finishes
Precondition P = what’s requiredfor the program to begin its run.
6
Pre- and Postconditions
Postconditions describes what it will compute. For max, a postcondition is Q = m = max(a, b), where max is a mathematical function that delivers the larger of two integers.
Preconditions describe minimum requirements for the program to run.For max, a and b can be any integers, so the precondition is P = true.
Before proving a program’s correctness, we first write its specifications:{true}int max (int a, int b) { int m; if (a >= b) m = a; else m = b; return m;}
{m = max(a, b)}
7
Another Example – computing a factorial (n!)
{n > 1} int Factorial (int n) { int f = 1; int i = 1; while (i < n) { i = i + 1; f = f * i; } return f;}
{f = n!}
This example raises two issues. What happens if:1. the program has a loop that never terminates?2. The program terminates abnormally (e.g., an exception is raised)?
Precondition P
Postcondition Q
8
Defining away the problem: “partial correctness”
Let’s assume that nothing bad ever happens…• programs always terminate after a finite number of steps, and
• termination is always normal.
Then such a program is partially correct if, for every set of input values that satisfies precondition P, the program computes a result that satisfies postcondition Q.
E.g., Factorial is partially correct if for every value of n that satisfies n > 1, it computes f =n!
9
(Partial) Correctness Proofs
Let’s think generally about any program or sequence of statements s, whose pre- and postconditions are P and Q.
A “Hoare triple” is a predicate of the form {P} s {Q}
which asserts that “execution of statements s, beginning in a state that satisfies P, results in a state that satisfies Q.”
If we can prove that this Hoare triple is valid, (i.e., it is true for all assignments of values to variables in P, Q, and s) then the program s is said to be correct.
But how can we construct such a proof?
10
Constructing a Correctness Proof
If the program is a series of statements: s1; s2; …; sn
We start with the Hoare triple {P} s1; s2; …; sn {Q}, and • use inference rules for programs to • derive an intermediate triple {Pi} si {Pi+1} for every statement si in this sequence.
When done, we also ensure that
Note: This process is similar to a direct proof in logic, where we• use inference rules to• derive a series of assertions that logically link the premises to the conclusion.
So, what are the inference rules for programs?
€
P ⇒ P1 and Pn+1⇒ Q.
11
Inference Rules are tied to program statement types
€
12
Formal methods and real programs
Major question: Where do formal methods fit in the software process? That is,
1. How do we integrate them into object oriented programs?
2. How do we write pre- and postconditions P and Q for
– methods?– classes?– systems?
3. Once written, how are these used to ensure that software is correct?
13
Recall the spiral model for software design
Requirements
Specifications(use cases)
Design
Coding/Integration
Testing andVerification
Deployment (v 1.0)Review
Prototype
v 1.1v 1.0
14
Formal methods in the object oriented design process
In OO design, we focus on classes and objectsMethods and messages are subordinate
Collections of objects have state, which is the set of all active objects and the values of their variables at any moment of run time.
Formal specifications P and Q are therefore logical expressions about each object’s state.
Tools for the formal design process• Specifications : Java Modeling Language (JML)• Design: Unified Modeling Language (UML and JML)
• Coding: Java and JML• Verification: JML
15
Correctness of OO systems
Where?• Method level: pre- and post-conditions, loop invariants
• Class level: class invariant (class state)• System level: intra-class invariants (system state)
When?• Specification and design phases:
Write specifications for all classes and methods (UML/JML)
• Coding phase:Develop code from the specifications (UML/JML/Java)
• Validation phase:Prove (mathematically) that specifications and code are logically equivalent (JML <==> Java)
16
What is JML? (www.jmlspecs.org)
History– Emerged in early 2000s out of ESC/Java2
Goals— Integration of formal methods throughout the software process
— Formal specification accessible to programmers— Direct support for design by contract— Integration with a real language (Java)
JML allows us to mix specifications directly with the Java code— Preconditions— Postconditions— Loop invariants— Class invariants
17
JML Basics
JML specifications are special comments in a Java program:
//@ for one-liners
/*@ …. @*/ for multiple-liners
The Hoare triple
{P} s1; s2; …; sn {Q}
is written in JML/Java as
(P and Q are written as Javaboolean expressions, and useparameters, local, and classvariables as arguments.)
/*@ requires P ; ensures Q ;@*/type method (parameters) { local variables s1; s2; …; sn
}
18
JML Language Summary
Note: p and e are also good old fashioned Java boolean expressions, possibly augmented by JML-specific operations.
19
Here’s a simple example, first as a Hoare triple:
{n > 1} P (precondition)int Factorial (int n) { int f = 1; int i = 1;
{1 < i i < n f = i!} R (loop invariant) while (i < n) { i = i + 1; f = f * i; } return f;}
{f = n!} Q (postcondition)
20
And again with JML expressions for P, Q, and R:
/*@ requires 1 <= n ;
ensures \result == (\product int i; 1<=i && i<=n; i) ;
@*/
static int Factorial (int n) {
int f = 1;
int i = 1;
/*@ loop_invariant i <= n && f == (\product int j; 1 <= j && j <= i; j);
@*/
while (i < n) {
i = i + 1;
f = f * i;
}
return f;
}
P Q R
21
JML-based software tools (1 and 3 are built into Eclipse)
1. Compiling (use jmlc instead of javac)• Does syntactic and type checking, and byte code generation for all JML assertions and Java code
2. Static checking (ESC/Java2)3. Runtime assertion checking (use jmlrac instead of java)
• Checks truth of precondition P at entry to every call• Checks truth of postcondition Q at exit from every call• Checks truth of loop invariant R before every iteration• Issues a Java Exception when any of these is not true
Note: this is not formal verification (Checking truth for one instance of a call is not the same as checking truth for all instances. The latter is proof of correctness!)
4. Proof assistance tools (Daikon, LOOP)
22
JML Eclipse Environment
3. Runtime assertion checking
1. Compiling
23
Continuing our JML example, let’s wrap Factorial inside a simple class:
public class myFactorial {/*@ requires 1 <= n; ensures \result == (\product int i; 1<=i && i<=n; i); @*/static int Factorial (int n) {…}public static void main(String[] args) { int n = Integer.parseInt(args[0]); System.out.println("Factorial of " + n + " = " +
Factorial(n));}}
24
… compile it with jmlc, and run it twice with jmlrac:
% jmlc -Q myFactorial.java
% jmlrac myFactorial 3
Factorial of 3 = 6
% jmlrac myFactorial -5
Exception in thread "main”
org.jmlspecs.jmlrac.runtime.JMLEntryPreconditionError:
by method myFactorial.Factorial regarding specifications at
File "myFactorial.java", line 3, character 15 when
'n' is -5
at myFactorial.checkPre$Factorial$myFactorial(myFactorial.java:240)
at myFactorial.Factorial(myFactorial.java:382)
at myFactorial.main(myFactorial.java:24)
normal run
abnormal run(throws a JML exception)
25
JML Exceptions
26
JML helps identify errors
Example 1: Suppose we change the while loop from while (i < n)
to while (i <= n) so that n! will be computed
incorrectly. Here’s the result:
% jmlrac myFactorial 3Exception in thread "main" org.jmlspecs.jmlrac.runtime.JMLLoopInvariantError: LOOP INVARIANT: by method myFactorial.Factorial regarding specifications at File "myFactorial.java", line 9, character 24 when 'n' is 3 at myFactorial.internal$Factorial(myFactorial.java:102) at myFactorial.Factorial(myFactorial.java:575) at myFactorial.main(myFactorial.java:211)
invariant not satisfied
27
JML Example 2
Suppose we change the while loop from while (i < n) to while (i <=n )
and also remove the JML loop invariant. Now we get:
% jmlrac myFactorial 3Exception in thread "main" org.jmlspecs.jmlrac.runtime.JMLNormalPostconditionError: by method myFactorial.Factorial regarding specifications at File "myFactorial.java", line 4, character 23 when 'n' is 3 '\result' is 24 at myFactorial.checkPost$Factorial$myFactorial(myFactorial.java:321) at myFactorial.Factorial(myFactorial.java:392) at myFactorial.main(myFactorial.java:24)
postcondition not satisfied
28
JML Example 3
Disagreement between a JML specification and a program may signal an error in the specification.
E.g., if the loop invariant had specified j <= i rather than j < i the following outcome would occur:
% jmlrac myFactorial 3
Exception in thread "main"
org.jmlspecs.jmlrac.runtime.JMLLoopInvariantError: LOOP INVARIANT:
by method myFactorial.Factorial regarding specifications at
File "myFactorial.java", line 9, character 24 when
'n' is 3
at myFactorial.internal$Factorial(myFactorial.java:101)
at myFactorial.Factorial(myFactorial.java:573)
at myFactorial.main(myFactorial.java:209)
invariant not satisfied
29
But beware… JML is no panacea
jmlrac doesn’t trap all errors… here are two “normal” runs:
% jmlrac myFactorial 21Factorial of 21 = -1195114496% jmlrac myFactorial 32Factorial of 32 = -2147483648
Recall: (1) Java has no ArithmeticOverflow exception, but (2) Factorial(n) for n > 12 should give a result > 231-1
Note: jmlrac computes the same wrong result when it checks the postcondition as the Factorial method computes, so that this error goes undetected.
Conclusion: the program and its specifications are both wrong.
wrong results!
30
Exception Handling in JML
/*@ requires P ; ensures Q ; signals (exception) expression;@*/type method (parameters) { locals s1; s2; …; sn
(includes “throw new exception ;”)}
We can throw Java Exceptions, and then validate their circumstances in JML whenever they occur.
whenever this happens
JML executes this
Two outcomes: 1) expression is true and normal Java exception handling proceeds, or 2) expression is false and JMLExceptionalPostconditionError is reported.
31
JML Example 4: Throwing and Checking Exceptions
/*@ requires 1 <= n; ensures \result == (\product int i; 1<=i && i<=n; i); signals (ArithmeticException) n > 12; @*/ static int Factorial (int n) { if (n > 12) throw new ArithmeticException(); else { …
% jmlrac myFactorial 22Exception in thread "main" java.lang.ArithmeticException at myFactorial.internal$Factorial(myFactorial.java:9) at myFactorial.Factorial(myFactorial.java:610) at myFactorial.main(myFactorial.java:213)
Normal Java exceptionhandling occurs, sincen > 12 is true.
32
Additional Points about JML
1. We can sometimes avoid a signals clause by strengthening the precondition. E.g., for Factorial, we could have said:
requires 1 <= n && n < 13 ;2. Specifications are always declarative; they never
affect the state of the program.3. Runtime assertion checking is not proof, but it
does provide a rigorous framework for debugging.4. JML provides a language in which formal methods
and Java programs can be integrated.5. There’s a lot more to JML. We are particularly
interested in:a. class level specifications?b. Tools for static checking of specifications?c. Tools for proving correctness?
33
Readings (at the course web site)
Hoare, An axiomatic basis for computer programming, Communications of the ACM 12(10):576-580.
Leino, Hoare-Style Program Verification ILeino, Hoare-Style Program Verification IIGordon, Specification and Verification I Dwyer et al., Software SpecificationsLeavens and Cheon, Design by Contract with JMLPoll, Kiniry, and Cok, Introduction to JMLBurdy et al., Overview of JML ToolsJML Reference Manual