PLAN ORIENTED TESTING OF AGENT BASED SYSTEMS USING FAULT

MODELS

1.INTRODUCTION

Softwaretesting remainsthe most widelyusedapproachtoverification inindustry

today, consumingbetween30-50%oftheentire developmentcost. Test

inputselectionforintelligentagentspresentsaproblem due totheveryfactthat the

agentsare intendedtooperaterobustly under conditions

inwhichdevelopersdidnotconsider,and wouldthereforebeunlikely totest.

Usingmethodstoautomaticallygenerate andexecutetestsisone waytoprovide

coverageofmany conditions without significantlyincreasingcost.However,

oneproblem usingautomaticgenerationandexecutionoftestsistheoracle

problem:how canweautomaticallydecideifobservedprogrambehavioriscorrect

withrespecttoitsspecification.

2. RESEARCH QUESTIONS

Based on the issues associated with agent testing and the limitations of existing work,

thefollowing research questions addressed in my projects are:

What is the scope of unit testing for agent systems?This involves the

identification of the basic units in agent systems as units to be tested.

How can an automated testing framework be developed to effectively capture

errors? There are three parts to this question: How do dependencies between

units/agents affect testing? An agent system usually contains multiple agents and

an agent contains multiple units. Do the dependencies between these components

affect testing?

Our approach for testing a plan does not involve the understanding of internal

logic of the plan; therefore we do not test the coverage of the execution paths and

code branches of the plan. The tester may however specify value ranges of

inputvariables as certain values in order to cover a plan’s internal logic

thoroughly, but this requires manual effort and understanding of the plan’s logic

There is another limitation in the algorithm of test input generation. Although combination of

input variable values and comparisons between variables have been taken into account in test

input generation, combinations of comparative relationships (e.g “x>y” and “y>z”) are not

considered

An effective way for revealing faults is to define for a SUT a fault model, which specifies

the assumptions about under what situation a fault is likely to be found in the SUT .Each

assumption introduces the occurrence of a software failure and such an occurrence in the

SUT can be identified as a fault that exists in the system. In this work we using a test case

but it is not covering the whole test process so we are introducing a fault model to

overcome the existing process to identify the total number faults present in the agent

system.

3.TESTINGFRAMEWORK

In thissection, we providean overviewof the test frameworkinwhich our oracle

resides.Wenote that oncethe designdocuments have had testing

descriptorsaddedtocapture those aspects of implementation necessaryfor test

inputgenerationand appropriatecodeaugmentation, thetesting process

isautomatic,supportingthegeneration, executionand reportingofcomprehensive test suites

withoutinvolvementfrom the test engineer.

Thesystemmodel thatweuseastheoracleconsistsof the artifacts producedduringthe

detaileddesign phase ofthe Prometheusmethodology.These arethe

overviewdiagramsof agents and their capabilities, along with the detailed descriptorsof

the internal components. Itisofcourse thecasethat errors found may bethe result oferrors

inthe designdocuments, which arecorrectedintheimplementation.However,

intheinterestsofmaintainingup todate and correct documentation of a system, it is as

important to correct these astocorrect implementationerrors.

3.1 General Working Procedure of Testing Framework

Thetestingtoolperformsanautomatedprocessfortestinganagentsystem,asoutlined

here .Themainstepsinthetestingprocessareasfollows:

•Step1: When thetestingprocess starts,thetestingtoolaccessesthePrometheus

designdocumentationofthesystemand extracts thelistofallagents..

•Step2: For each agent,thetestingtoolidentifies thetestunits withintheagent(plans andevents),and

determines theorder inwhichtheseunits are tested

•Step3: For eachunit,thetestingtoolimplements atestautomationvia aprocess

ofcodeaugmentation.Thedetailsregarding testautomation ofdifferenttypesofunitsare discussed.

•Step4:Thetestingtoolthenexecutesthetestharnesstoperformaunittestingprocess

fortestingtheassociatedunit. Inaunittestingprocess, thetestharness

initializestheruntimeenvironmentthatisneeded fortheexecutionoftestcases,generatesand

executesa setoftestcases, collectstestoutcomesand compares themagainstthe

informationextractedfromthedesigndocumentationandidentifiesfaultsif theyexist.

•Step6: Ifanyfaultis detectedintheunittestingprocess,one of

thefollowingthreeactionswillbeperformeddepending onthedifferentlevelsof the fault:

•Step7: Attheendof thetestingprocess,thetestingtoolgeneratesatestreportin

HTMLformat.Thereportindicatesallthe detailsof the testingprocessandthetest

resultsofalltheunitsthathavebeentested

Compared withotherexistingtechniques fortestingagentsystems , thetestingtoolinour

testingframework performs acompleteautomatedprocessforunit

testinganagentsystem.Intheprocess,userinterventionisnotmandatoryastestautomation

areautomaticallyimplementedand eachtestcase isautomaticallyexecutedtotestthe

associatedunit.Userinterventionisonlynecessary insomepreparatorytasksfortheauto-

matedtestingprocess,suchasdefinitionofinputvariables andspecification ofinitializationprocedures .

The usercan alsomanually add additional testcasesifnecessary.

Figure 3.1 Testing Process

4.FAULT MODELS

An effective way for revealing faults is to define for a SUT a fault model, which specifies the

assumptions about under what situation a fault is likely to be found in the SUT. Each assumption

introduces the occurrence of a software failure and such an occurrence in the SUT can be

identified as a fault that exists in the system. So by writing all the faults in the db as the rules and

by leveling all the rules such as level 1 as exception, level 2 is errorand level 3 is errors so by

defining it all the test cases will be passing it so all the test cases will be tested and also the test

engineer knows what behavior it will exhibit using the test process using fault models.

So I conclude it by adding the fault models in the plan oriented testing of agent systems will be

more effective testing process and the result also will be compared with existing testing

approach.

.

Figure 4.1 Overall System Architecture

Agents in MAS

Fault Modes Final Result

Test Report Plan Oriented

Testing

Units in Agents

ie.Plan