icst2016 keynote - what got us here won't get us there: trends and challenges in testing...

What got us here won't get us thereTrends and challenges in testing tomorrow’s systems

Software testing research has been tremendously successful

But yesterday’s hot computing technology is today’s museum curiosity.

What will tomorrow’s software systems look like?

1985Graduate student at UCI

Researchers were laying the foundations for the entire field.

ICST 2016: Daniel Liew, Alastair Donaldson and CristianCadar, Symbooglix: A Symbolic Execution Engine for

Boogie Programs

Cites: J. C. King, “Symbolic execution and program testing,” Communications of the Association for

Computing Machinery (CACM), vol. 19, no. 7, pp. 385–394, 1976.

1995

1985

2005

2015

Exploration over

application

Research lifecycle

Laying Foundations

1995Asst. Prof. at UMD

Collaborating with AT& T

Empirical evaluation became a standard part of research

A. Avritzer and E. J. WeyukerThe Automatic Generation of Load Test Suites and the

Assessment of the Resulting SoftwareIEEE Transactions on Software Engineering, 1995

Cited in ICST2016: Ruoyu Gao, Zhen Ming Jack Jiang, Cornel Barna and Marin Litoiu

A Framework to Evaluate the Effectiveness of Different Load Testing Analysis Techniques

1995

1985

2005

2015

Evaluating practice

Deepening theory

Demanding Data

Exploration over

application

Research lifecycle

Laying Foundations

2005Assoc. Prof. at UMD

Testing takes off in industryMoore’s law makes old techniques practical

“Historically, what have been called ‘testing’ and ‘model checking’ have been considered polar opposites … recently testing and model checking have been moving toward each other, testing becoming more algorithmic and using model checking as a subroutine.”

Robert Kurshan, Cadence Design SystemsISSTA 2004

Patrice Godefroid, Nils Klarlund, and Koushik SenDART: directed automated random testing PLDI '05

Koushik Sen, Darko Marinov, and Gul Agha. 2005. CUTE: a concolic unit testing engine for C. ESEC/FSE-13

ICST2016: Abdulmajeed Alameer, Sonal Mahajan and William G. J. Halfond

Detecting and Localizing Internationalization Presentation Failures in Web Applications

Industry adopts

testing

Incorporating outside

research advances

Looking Outward

1995

1985

2005

2015

Evaluating practice

Deepening theory

Demanding Data

Exploration over

application

Research lifecycle

Laying Foundations

2015Full Prof. at UMD

Basili Postdoctoral Fellowship Programhttps://www.cs.umd.edu/basili-postdoc

Research area specializes into multiple, focused subcommunities

Industry adopts

testing

Incorporating outside

research advances

Looking Outward

1995

1985

2005

2015

Significant prior work

Improvements are

more incremental

Applicability expected

Reaching for Maturity

Evaluating practice

Deepening theory

Demanding Data

Exploration over

application

Research lifecycle

Laying Foundations

Some reasons for field’s

success

Respecting the research

lifecycle

Demanding data

Looking outward


A lot to be proud of!

But we also need to look to the future

Because in technology, success doesn’t last

Recently became Executive DirectorMust guide organization towards the future

Every leader of every organization needs to do the same

Some major trends will change software systems and how we test them

Ubiquity

Features

• Computers and physical world merging

• Sensor rich and context aware

• Operating in, on and around humans

Testing Implications

• Physical world is a test case parameter

• Must test over continuous domains

• Many systems become safety critical

Complexity

Features

• Very large code bases

• Increasingly supported by models

–Multiple system wide properties of interest


• Source code analysis can be too low level

• Integrate model-based engineering and testing throughout the lifecycle

• More model-level testing

Uncertainty

This autonomous vehicle drove from SF to NYC

Features

• Intended behavior is under specified

• Decision making is autonomous

• Sensor and system limitations

• Imprecise requirements

–e.g., legal policies


• Incorporating notions of uncertainty

–Test case generation

–Uncertainty injection and response

–Oracles

• Supporting / documenting regulatory compliance

Rapid Change

Some drones are designed to learn at runtime.What does it mean to test a system like this?

Features

• Designed for change at multiple scales

–Over the air updates

–Runtime adaption / reconfiguration

–Unconstrained learning at runtime


• Greater focus on deployment time

–Runtime monitoring and verification

–Disciplined rollout and A/B testing

–Big data for telemetry streams

• Built-in test capabilities needed

–Periodic runtime audits

Decentralization

Each node represents a separate system independently managed and tested

ICST2016: Teng Long, Ilchul Yoon, Adam Porter, Atif Memon and Alan Sussman

Coordinated Collaborative Testing of Shared Software Components

Features

• Systems of systems with distributed component ownership

• Cloud infrastructure and execution environments

• Web-scale IT technologies


• Testing crosses ownership boundaries

• Cloud performance hard to understand and predict

• Technology semantics changing

Respecting the research lifecycle

Demanding data

Looking outward


Ubiquity – Testing for the physical world

Complexity –Integrating testing with models

Uncertainty – Treating uncertainty explicitly

Rapid Change – More deployment time testing

Decentralization – Make testing collaborative

Adam PorterProfessor: University of Maryland

Executive Director: Fraunhofer CESE

[email protected]

Credits

• Janus - http://i.kinja-img.com/gawker-media/image/upload/dvtgwrtxzhzhh2ztbfyc.jpg

• Win race - https://www.flickr.com/photos/68978695@N00/6152736494/• Arithmometer -

https://upload.wikimedia.org/wikipedia/commons/b/b1/Brunsvika_arithmometer.JPG

• Question marks - https://www.flickr.com/photos/87099998@N05/14713228051/• Zot on - https://www.flickr.com/photos/50854518@N05/10201180195/• Construction site - https://www.flickr.com/photos/47997385@N00/16908422032/• UMD aerial - https://www.flickr.com/photos/126497846@N03/14601582999/• Bell Labs - https://www.flickr.com/photos/21266163@N00/763433063/• Test tube - https://tanyagrove.files.wordpress.com/2011/05/test-tubes.jpg• Crash test - https://www.flickr.com/photos/54519810@N02/7920091208/• Iribe Center - http://iribe.cs.umd.edu/home• Galaxies - https://www.flickr.com/photos/24354425@N03/22492430541/

Credits

• Success - https://www.flickr.com/photos/zionfiction/16507837706

• Toast - https://www.flickr.com/photos/22016744@N06/23678867029/

• Penny - https://www.flickr.com/photos/26556146@N07/16254669028/in/album-72157650084451358/

• Wearables - https://www.tractica.com/newsroom/press-releases/wearable-device-shipments-to-reach-560-million-units-annually-by-2021/

• Complexity -http://41.media.tumblr.com/cf1fe0bbb5d530cc072024e7d70c16dc/tumblr_nogkpbhiSk1u51dj2o2_1280.jpg

• Connected - http://www.forbes.com/sites/salesforce/2014/08/30/5-ways-iot-marketing-smarter/

• Uncertainty - https://www.flickr.com/photos/38692570@N00/19699096321/

• Delphi - http://www.extremetech.com/wp-content/uploads/2015/04/Delphi-autonomous-driving-vehicle-parked-with-san-francisco-in-background-3-640x353.jpg

• Rapid Change - https://www.flickr.com/photos/30576334@N05/16638657367/

• Drone - https://www.flickr.com/photos/59663349@N00/23685673702/

• Decentralized - https://gigaom.com/wp-content/uploads/sites/1/2013/09/screen-shot-2013-09-03-at-2-52-20-pm.png