safe runtime validation of behavioral adaptations in autonomic software

NORTH DAKOTA STATE UNIVERSITY DEPARTMENT OF COMPUTER SCIENCE

© NDSU SOFTWARE TESTING RESEARCH GROUP

Safe Runtime Validation of Behavioral Adaptations in

Autonomic SoftwarePresenter: Tariq M. King

ATC 2011September 2-4, Banff, Alberta, Canada

© NDSU SOFTWARE TESTING RESEARCH GROUP 2

Outline Motivation Introduction Testing Approach Component Design Prototype Related Work Conclusion

© NDSU SOFTWARE TESTING RESEARCH GROUP

Motivation Self-* features in autonomic computing

systems may incorporate dynamic software adaptation System adds, removes, replaces its own

components at runtime

Raises reliability concerns asnew faults may be introduced due to dynamic adaptation

Most AC research neglects that self-test should be an integral part of these types of systems

3

A NEGLECTED SELF-* CHARACTERISTIC

Self-Configure

Self-Optimize

Self-Heal

Self-Protect

Self-Test

FAULTS? FAILURES?


To address this issue, we introduced the notion of Autonomic Self-Testing (King et. al 2007)

Idea: Investigate howAutonomic Managers(AMs) can be tailored for purpose of testing the system itself

Test Managers (TMs)Monitor, intercept and validate the adaptive changes requests of AMs

Managed Resource

Introduction AUTONOMIC SELF-TESTING

Monitor

Analyze Plan

ExecuteKnowledge

EffectorSensorAutonomic Manager

Monitor

Analyze Plan

ExecuteKnowledge

Autonomic ManagerTest Manager


Initial work on AST concentrated on: Defining validation strategies for TMs

1. Replication with Validation (RV) – tests changes using separate copies of managed resources

2. Safe Adaptation with Validation (SAV) – tests changes in-place, directly on managed resources

Developing prototypes to investigate the feasibility of the RV strategy Lightweight, focused on localized validation

Introduction AUTONOMIC SELF-TESTING (CONT.)


In this research, we extend previous work on AST by overcoming two of its limitations.

Major Contributions:

1. Addresses the need for system-wide validation in autonomic software through the description of an automated runtime integration testing approach

2. Investigates AST of a real-world application in which it is too expensive to maintain copies of managed resources for testing – Validates SAV

Introduction CONTRIBUTIONS


System-Wide Validation of Behavioral Adaptations Using Self-Testable Autonomic

Components (STACS)

Overview of Testing Approach


State-Based Safety Model

1. Extends Zhang (2004) “Enabling Safe Component-Based Software Adaptation” in WADS ‘04


Self-Testable Autonomic Component (T, A, R, I, K)

Component Design STAC DEFINITION


Abstract MAPE Generics

Managers (T, A)

Multi-Threading Safety Mechanisms


Full Access to Source Code Derive suspend, resume, save

and restore state operations from AbstractResource class

Loose coupling of concreteobjects to facilitate stubinjection at runtime

Limited Access to Source Code Use wrapper class (carefully) Research on improving COTS testability

Resources (R)


Test Interface (IT)

Test Case Execution Code Coverage Analysis

Performance


Communication Virtual Machine (CVM) is a model-driven platform for realizing user-centric communication services Collaboration: FIU & Miami Children’s

Hospital

CVM separates the concerns of modeling, synthesis, coordination, and delivery of communication services into self-contained layers

Layer in which communication in communication takes place is called the Network Communication Broker (NCB).

Prototype COMMUNICATION VIRTUAL MACHINE


Allen (2009) leverage AC and open-platform APIs for integrating communication services into NCB

Prototype COMMUNICATION VIRTUAL MACHINE

CVMNCB LAYER

OF


Conducted two sets of experimental runs as part of the prototype evaluation: Test Quality Experiments – compared

the effectiveness of test sets in revealing faults and exercising the NCB program code

Performance Experiments – investigates ways to reduce the performance overhead of having a runtime testing process interleaved with the NCB

In both cases, a Mutation Analysis technique was used to simulate faulty change requests,

induce self-testing, and produce the results

Experiment Overview


39 mutants were generated for the prototype by planting artificial faults into the interfaces and/or implementations of Skype and Smack 15 created manually, 24 automatically

generated

Test Support Tools: JUnit, Cobertura, MuJava, Eclipse TPTP

Experiments were performed on a windows-based, webcam-enabled Intel® Core 2 Duo 2.4GHz PCs with 2 GB RAM or greater

Setup Environment


Fault Detection 35 out of the 39 mutants were detected,

producing a mutation score of 89.7% (Threat, CVM Dev/Test)

Code Coverage Recorded 63% statement coverage and 57%

branch coverage of the NCB implementation 100% method coverage of the communication

service interfaces (SkypeAdapter, SmackAdapter)

Results TEST QUALITY


Results (cont.) PERFORMANCE

SKYPE TWO-WAY

TOSMACK

THREE-WAY


System-wide SAV is feasible and beneficial for preventing runtime faults in the context of a real-world application

Testing adaptive changes in-place using TMs was challenging: high complexity, synchronization Significant performance overhead, further

research Safety mechanisms (locks) were instrumental

during development and runtime testing

Selecting and tailoring open-source testing tools presented unnecessary difficulties

Lessons Learned


Da Costa et al. (2010) JAAF+T: Java Self-Adaptive Framework +

Test Modify MAPE structure to include a self-test

activity

Stevens et al. (2007) and Ramirez et al. (2008) Prototypes of Replication with Validation Autonomic Container & Autonomic Job

Scheduler

Zhang et al. (2009) and Zhao et al. (2005) Runtime verification of self-* systems

Related Work


Presented a system-wide approach to runtime integration testing in autonomic software

Still many open research issues: Dynamic test generation Dynamic regression test scheduling Runtime test case maintenance Reducing overhead of runtime testing

Latest Direction: Cloud (CASCON 2011)

Conclusion


Acknowledgements Dr. Masoud Milani, FIU

Dr. S. Masoud Sadjadi, FIU

Participants of the REU Summer 2006 and 2007 Programs on Autonomic Computing at FIU

Graduate students and research collaborators on CVM project

This work has been supported in part by the NSF under grant IIS-0552555.


Thank You!Questions?

¿Preguntas?問題Sawwalвопросы質問domandeερωτήσεις

safe runtime validation of behavioral adaptations in autonomic software

Documents

system wide validation

selfheal self

validation strategies

validation rv tests

selftest faults

validation sav tests

autonomic softwarepresenter

rv strategylightweight