Introduction Metamorphic testing (MT) Assertion checking Oracles generated with Machine Learning (ML) Discussion

Techniques for Testing Scientific ProgramsWithout an Oracle

Upulee Kanewala and James M. Bieman

Computer Science DepartmentColorado State University

May 18, 2013

This work is supported by the National Institute of General Medical Sciences of the N.I.H

Upulee Kanewala and James M. Bieman Colorado State University

Techniques for Testing Scientific Programs Without an Oracle 1 / 27

Introduction Metamorphic testing (MT) Assertion checking Oracles generated with Machine Learning (ML) Discussion

Outline

1 Introduction

2 Metamorphic testing (MT)

3 Assertion checking

4 Oracles generated with Machine Learning (ML)

5 Discussion

Upulee Kanewala and James M. Bieman Colorado State University

Techniques for Testing Scientific Programs Without an Oracle 2 / 27

Introduction Metamorphic testing (MT) Assertion checking Oracles generated with Machine Learning (ML) Discussion

Outline

1 Introduction

2 Metamorphic testing (MT)

3 Assertion checking

4 Oracles generated with Machine Learning (ML)

5 Discussion

Upulee Kanewala and James M. Bieman Colorado State University

Techniques for Testing Scientific Programs Without an Oracle 3 / 27

Introduction Metamorphic testing (MT) Assertion checking Oracles generated with Machine Learning (ML) Discussion

Introduction

Program Under Test

t1

t2

tn

...

o1

o2

on

...

Oracle

Pass/fail?

Non-testable programsAn oracle does not exist or too hard to implementMany scientific programs are non-testable

Ad-hoc approaches used for implementing oracles

Upulee Kanewala and James M. Bieman Colorado State University

Techniques for Testing Scientific Programs Without an Oracle 4 / 27

Introduction Metamorphic testing (MT) Assertion checking Oracles generated with Machine Learning (ML) Discussion

Test oracle precision

Oracle precision: tolerance for errors in the oracleNull oracle: detects whether program crashes or hangsIdeal oracle: provides correct pass/fail judgment for anypossible executionPseudo-oracle: independent implementation of theprogram under test

Three methods fall between the null oracle and the idealoracle

Upulee Kanewala and James M. Bieman Colorado State University

Techniques for Testing Scientific Programs Without an Oracle 5 / 27

Introduction Metamorphic testing (MT) Assertion checking Oracles generated with Machine Learning (ML) Discussion

Oracle Properties

Completeness

Ifresult of a test t is correct according to specification s

Thenoracle o will indicate that t has passed

Soundness

Iforacle o indicates that a test t has passed

Thenprogram p is correct w.r.t t and specification s

Upulee Kanewala and James M. Bieman Colorado State University

Techniques for Testing Scientific Programs Without an Oracle 6 / 27

Introduction Metamorphic testing (MT) Assertion checking Oracles generated with Machine Learning (ML) Discussion

Oracle Properties

Completeness

Ifresult of a test t is correct according to specification s

Thenoracle o will indicate that t has passed

Soundness

Iforacle o indicates that a test t has passed

Thenprogram p is correct w.r.t t and specification s

Upulee Kanewala and James M. Bieman Colorado State University

Techniques for Testing Scientific Programs Without an Oracle 6 / 27

Introduction Metamorphic testing (MT) Assertion checking Oracles generated with Machine Learning (ML) Discussion

Outline

1 Introduction

2 Metamorphic testing (MT)

3 Assertion checking

4 Oracles generated with Machine Learning (ML)

5 Discussion

Upulee Kanewala and James M. Bieman Colorado State University

Techniques for Testing Scientific Programs Without an Oracle 7 / 27

Introduction Metamorphic testing (MT) Assertion checking Oracles generated with Machine Learning (ML) Discussion

Metamorphic testing (MT)

MT checks whether the program behaves according to aset of metamorphic relations (MRs)

MR specifies how the output changes according to achange made to the input

Upulee Kanewala and James M. Bieman Colorado State University

Techniques for Testing Scientific Programs Without an Oracle 8 / 27

Introduction Metamorphic testing (MT) Assertion checking Oracles generated with Machine Learning (ML) Discussion

MT Process

Identify MRs that need to be satisfied by the PUTCreate a set of initial test casesCreate follow-up test cases based on input transformationsExecute test case pairs and check whether the MR holds

Initial test case a1 a2 a3 a4 a5

Follow-up

test case 1a4 a3 a5

Follow-up

test case 2a1+k a2+k a3+k a4+k a5+k

=

==

S

S S+5*k

Permute Add a constant

a2a1

Upulee Kanewala and James M. Bieman Colorado State University

Techniques for Testing Scientific Programs Without an Oracle 9 / 27

Introduction Metamorphic testing (MT) Assertion checking Oracles generated with Machine Learning (ML) Discussion

MT Process

Identify MRs that need to be satisfied by the PUTCreate a set of initial test casesCreate follow-up test cases based on input transformationsExecute test case pairs and check whether the MR holds

Initial test case a1 a2 a3 a4 a5

Follow-up

test case 1a4 a3 a5

Follow-up

test case 2a1+k a2+k a3+k a4+k a5+k

=

==

S

S S+5*k

Permute Add a constant

a2a1

Upulee Kanewala and James M. Bieman Colorado State University

Techniques for Testing Scientific Programs Without an Oracle 9 / 27

Introduction Metamorphic testing (MT) Assertion checking Oracles generated with Machine Learning (ML) Discussion

MT Process

Identify MRs that need to be satisfied by the PUTCreate a set of initial test casesCreate follow-up test cases based on input transformationsExecute test case pairs and check whether the MR holds

Initial test case a1 a2 a3 a4 a5

Follow-up

test case 1a4 a3 a5

Follow-up

test case 2a1+k a2+k a3+k a4+k a5+k

=

==

S

S S+5*k

Permute Add a constant

a2a1

Upulee Kanewala and James M. Bieman Colorado State University

Techniques for Testing Scientific Programs Without an Oracle 9 / 27

Introduction Metamorphic testing (MT) Assertion checking Oracles generated with Machine Learning (ML) Discussion

MT Process

Identify MRs that need to be satisfied by the PUTCreate a set of initial test casesCreate follow-up test cases based on input transformationsExecute test case pairs and check whether the MR holds

Initial test case a1 a2 a3 a4 a5

Follow-up

test case 1a4 a3 a5

Follow-up

test case 2a1+k a2+k a3+k a4+k a5+k

=

==

S

S S+5*k

Permute Add a constant

a2a1

Upulee Kanewala and James M. Bieman Colorado State University

Techniques for Testing Scientific Programs Without an Oracle 9 / 27

Introduction Metamorphic testing (MT) Assertion checking Oracles generated with Machine Learning (ML) Discussion

Applications of MT

Applications:Testing ML classifiers: k-nearest neighbor (k-NN) andNaı̈ve Bayes classifier (NBC) 1

Unit and integration testing of a jpeg2000 encoder2

MT oracles are complete and not sound

1X. Xie, J. W. Ho, C. Murphy, G. Kaiser, B. Xu, and T. Y. Chen, “Testing and validating machine learning

classifiers by metamorphic testing,” Journal of Systems and Software, 2011.2

R. Just and F. Schweiggert, “Automating unit and integration testing with partial oracles,” Software QualityJournal, 2011

Upulee Kanewala and James M. Bieman Colorado State University

Techniques for Testing Scientific Programs Without an Oracle 10 / 27

Introduction Metamorphic testing (MT) Assertion checking Oracles generated with Machine Learning (ML) Discussion

Outline

1 Introduction

2 Metamorphic testing (MT)

3 Assertion checking

4 Oracles generated with Machine Learning (ML)

5 Discussion

Upulee Kanewala and James M. Bieman Colorado State University

Techniques for Testing Scientific Programs Without an Oracle 11 / 27

Introduction Metamorphic testing (MT) Assertion checking Oracles generated with Machine Learning (ML) Discussion

Assertion Checking

long square_root(long x) in { assert(x >= 0);} out (result) {

assert((result * result) <= x && (result+1) * (result+1) >= x);}body {

return cast(long)std.math.sqrt(cast(real)x); }

Assertions: expected range of output, variablerelationships within the programAssertions can be more effective than the null oracle3

Oracle is complete but not sound

3K. Shrestha and M. Rutherford, “An empirical evaluation of assertions as oracles,” IEEE 4th Int. Conf. on

Software Testing, Verification and Validation (ICST), 2011

Upulee Kanewala and James M. Bieman Colorado State University

Techniques for Testing Scientific Programs Without an Oracle 12 / 27

Introduction Metamorphic testing (MT) Assertion checking Oracles generated with Machine Learning (ML) Discussion

Outline

1 Introduction

2 Metamorphic testing (MT)

3 Assertion checking

4 Oracles generated with Machine Learning (ML)

5 Discussion

Upulee Kanewala and James M. Bieman Colorado State University

Techniques for Testing Scientific Programs Without an Oracle 13 / 27

Introduction Metamorphic testing (MT) Assertion checking Oracles generated with Machine Learning (ML) Discussion

Oracles generated with Machine Learning (ML)

Upulee Kanewala and James M. Bieman Colorado State University

Techniques for Testing Scientific Programs Without an Oracle 14 / 27

Introduction Metamorphic testing (MT) Assertion checking Oracles generated with Machine Learning (ML) Discussion

Oracles generated with Machine Learning (ML)

Upulee Kanewala and James M. Bieman Colorado State University

Techniques for Testing Scientific Programs Without an Oracle 15 / 27

Introduction Metamorphic testing (MT) Assertion checking Oracles generated with Machine Learning (ML) Discussion

Oracles generated with Machine Learning (ML)

Upulee Kanewala and James M. Bieman Colorado State University

Techniques for Testing Scientific Programs Without an Oracle 16 / 27

Introduction Metamorphic testing (MT) Assertion checking Oracles generated with Machine Learning (ML) Discussion

Oracles generated with Machine Learning (ML)

Upulee Kanewala and James M. Bieman Colorado State University

Techniques for Testing Scientific Programs Without an Oracle 17 / 27

Introduction Metamorphic testing (MT) Assertion checking Oracles generated with Machine Learning (ML) Discussion

Oracles generated with Machine Learning (ML)

Upulee Kanewala and James M. Bieman Colorado State University

Techniques for Testing Scientific Programs Without an Oracle 18 / 27

Introduction Metamorphic testing (MT) Assertion checking Oracles generated with Machine Learning (ML) Discussion

Oracles generated with Machine Learning (ML)

Upulee Kanewala and James M. Bieman Colorado State University

Techniques for Testing Scientific Programs Without an Oracle 19 / 27

Introduction Metamorphic testing (MT) Assertion checking Oracles generated with Machine Learning (ML) Discussion

Applications of oracles generated with ML

Applications:Mesh simplification program 4

Used another similar program to create training dataImage segmentation program 5

Used previous versions of the same program to createtraining data

Oracles are not complete and not sound

4W. K. Chan, S. C. Cheung, J. C. F. Ho, and T. H. Tse, “Pat: A pattern classification approach to automatic

reference oracles for the testing of mesh simplification programs,” Journal of Systems and Software, 20085

K. Frounchi, L. C. Briand, L. Grady, Y. Labiche, and R. Subramanyan, “Automating image segmentationverification and validation by learning test oracles,” Inf. Softw. Technol., 2011

Upulee Kanewala and James M. Bieman Colorado State University

Techniques for Testing Scientific Programs Without an Oracle 20 / 27

Introduction Metamorphic testing (MT) Assertion checking Oracles generated with Machine Learning (ML) Discussion

Outline

1 Introduction

2 Metamorphic testing (MT)

3 Assertion checking

4 Oracles generated with Machine Learning (ML)

5 Discussion

Upulee Kanewala and James M. Bieman Colorado State University

Techniques for Testing Scientific Programs Without an Oracle 21 / 27

Introduction Metamorphic testing (MT) Assertion checking Oracles generated with Machine Learning (ML) Discussion

Fault finding effectiveness

Two studies compared MT vs assertion checking 6,7

Determined MT is more effective but used different test sets

Oracles developed using ML were not compared

Adding more test cases alone will not improve theeffectiveness

6Z. Zhang, W. K. Chan, T. H. Tse, and P. Hu, “Experimental study to compare the use of metamorphic testing

and assertion checking,” Journal of Software, 20097

C. Murphy and G. E. Kaiser, “Empirical evaluation of approaches to testing applications without test oracles,”Dep. of Computer Science, Columbia University, Tech. Rep. CUCS-039-09, 2010

Upulee Kanewala and James M. Bieman Colorado State University

Techniques for Testing Scientific Programs Without an Oracle 22 / 27

Introduction Metamorphic testing (MT) Assertion checking Oracles generated with Machine Learning (ML) Discussion

Domain knowledge

Domain expert is required to develop MRs/assertions

ML oracles:ML knowledgeSelecting suitable features

Upulee Kanewala and James M. Bieman Colorado State University

Techniques for Testing Scientific Programs Without an Oracle 23 / 27

Introduction Metamorphic testing (MT) Assertion checking Oracles generated with Machine Learning (ML) Discussion

Automation

MT: techniques for automatically identifying likely MRs

Assertion checking: automatic invariant detection tools

Disadvantage: Can make these oracles incomplete

Upulee Kanewala and James M. Bieman Colorado State University

Techniques for Testing Scientific Programs Without an Oracle 24 / 27

Introduction Metamorphic testing (MT) Assertion checking Oracles generated with Machine Learning (ML) Discussion

Unsolved Problems

Metamorphic testing:Develop techniques to automatically detect likely MRsDevelop techniques to identify effective MRs

Assertion checking:Minimize spurious invariants detected by automaticinvariant detection methods

Oracles generated with ML:Investigate applicability of ML based oracles in otherdomains

Upulee Kanewala and James M. Bieman Colorado State University

Techniques for Testing Scientific Programs Without an Oracle 25 / 27

Introduction Metamorphic testing (MT) Assertion checking Oracles generated with Machine Learning (ML) Discussion

Unsolved Problems

Metamorphic testing:Develop techniques to automatically detect likely MRsDevelop techniques to identify effective MRs

Assertion checking:Minimize spurious invariants detected by automaticinvariant detection methods

Oracles generated with ML:Investigate applicability of ML based oracles in otherdomains

Upulee Kanewala and James M. Bieman Colorado State University

Techniques for Testing Scientific Programs Without an Oracle 25 / 27

Introduction Metamorphic testing (MT) Assertion checking Oracles generated with Machine Learning (ML) Discussion

Unsolved Problems

Metamorphic testing:Develop techniques to automatically detect likely MRsDevelop techniques to identify effective MRs

Assertion checking:Minimize spurious invariants detected by automaticinvariant detection methods

Oracles generated with ML:Investigate applicability of ML based oracles in otherdomains

Upulee Kanewala and James M. Bieman Colorado State University

Techniques for Testing Scientific Programs Without an Oracle 25 / 27

Introduction Metamorphic testing (MT) Assertion checking Oracles generated with Machine Learning (ML) Discussion

Summary

Many scientific programs are non-testable

Three techniques for testing programs that do not haveoracles

Technique MT Assertion Checking ML based Oracles

Applications ML Classifiers jpeg encoder

JDK 1.4.2 classes Mesh simplification Image segmentation

Complete X

Sound X X X

Fault finding ability, automation and domain knowledgediffers

Upulee Kanewala and James M. Bieman Colorado State University

Techniques for Testing Scientific Programs Without an Oracle 26 / 27

Introduction Metamorphic testing (MT) Assertion checking Oracles generated with Machine Learning (ML) Discussion

Thank you!

Upulee Kanewala and James M. Bieman Colorado State University

Techniques for Testing Scientific Programs Without an Oracle 27 / 27