A studyof PROforma,adevelopmentmethodologyforclinical procedures

AI in Medicine,1999.

Arjen Vollebregt JohanvanderLeiAnnettetenTeije

�MeesMosseveld

FrankvanHarmelenDepartmentof ComputerScience Departmentof MedicalInformatics

andMathematicsVrije UniversiteitAmsterdam ErasmusUniversiteitRotterdam�

annette,frankh� @cs.vu.nl�vanderlei,mosseveld� @mi.fgg.eur.nl

Abstract

Knowledgeengineeringhasshown thatbesidesthegeneralmethodologiesfrom softwareengineeringit is usefulto developspecialpurposemethodologiesfor knowledgebasedsystems(KBS). PROformaisa newly developedmethodologyfor a specifictypeof knowledgebasedsystems.PROformais intendedfor decisionsupportsystemsandin particularfor clinical proceduresin themedicaldomain.

Thispaperreportsonanevaluationstudyof PROforma,andon thetrade-off thatis involvedbetweengeneralpurposeandspecialpurposedevelopmentmethodsin KnowledgeEngineeringandMedicalAI.Ourmethodfor evaluatingPROformais basedonre-engineeringarealisticsystemin two methodologies:the new andspecialpurposeKBS methodologyPROformaandthe widely accepted,andmoregeneralKBS methodologyCommonKADS.

The four mostimportantresultsfrom our studyareasfollows. Firstly, PROformahassomestrongpointswhicharealsostrongrelatedto requirementsof medicalreasoning.Secondly, PROformahassomeweakpoints,but noneof themarein any wayrelatedto thespecialpurposenatureof PROforma.Thirdly,a moregeneralmethodlike CommonKADSworks betterin the analysisphasethan the morespecialpurposemethodPROforma.Finally, to supportacomplementaryuseof themethodologies,weproposeamappingbetweentheir respective languages.

1 Moti vation

Over theyears,muchresearchin AI & Medicinehasbeenconcernedwith developingmethodologiesforapplyingAI to medicalknowledge[11]. Thisresearchparallelsmuchwork in bothKnowledgeEngineeringandSoftwareEngineering.

In the literatureof softwareengineeringmany methodologiesareproposedfor building complex systems,suchasOMT [14] or MSA [22]. Thesemethodologiesenablesusto developsoftwarein a structuredway.

Knowledgebasedsystemsareof coursea particularkind of softwaresystems,andresearchin KnowledgeEngineeringin the lastdecadehasshown that it is usefulto developmethodologieswhich arespecialised�Supportedby theNetherlandsComputerScienceResearchFoundationwith financialsupportfrom theNetherlandsOrganisation

for ScientificResearch(NWO) project:612-32-006

1

for this type of systems.Examplesof specialpurposeKnowledgeEngineeringmethodologiesareCom-monKADS[20, 16], MIKE [1], GenericTasks[3], DESIRE[17]. In particularCommonKADSis oneof themethodologiesthat is acceptedbothin researchandindustry. Many variantsbasedon theCommonKADSmethodologyareusedin industry[9, 2].

PROforma[6] is a methodologyfor an evenmorespecifictype of systems,namelyfor decisionsupportsystemsand in particularfor clinical proceduresin the medicaldomain. This domainis of coursemorenarrow thanKnowledgeEngineeringin general,althoughclinical proceduresin the medicaldomainstillconstitutesalargeclassof differentproblems(e.g.diagnosis,monitoring,treatmentplanning).Furthermorethe medicaldomainis a large domain,containingmany different typesof knowledge(causal,temporal,uncertain,etc.),eachof whichconcernslargevolumesof knowledge.

PROformais intendedfor supportingthecompleteprocessfrom earlyknowledgeacquisitionto the con-structionof an executablesystem. PROforma containsa semi-formallanguagefor describingmedicaldecision-makingprocedures,a tool for constructingknowledge-level modelsof suchprocedures,and atool for executingsuchmodel. The fact that the modelcanbe executedis an importantcharacteristicofPROformamethodology. ThePROformamethodologyis in developmentat theImperialCancerResearchFundin London.Underits new nameArezzo,PROformais currentlyproposedasa Europeanstandardforspecifyingclinical procedures.

In this paper, we areconcernedwith comparingthespecialpurposePROformamethodologywith a moregeneralpurposemethodologyfrom KnowledgeEngineering,namelyCommonKADS.Sucha comparisonis importantfor a numberof reasons.First, PROformais a newly proposedmethod,so it is relevant tocomparePROformawith someof theexistingalternatives,in particularwith amethodlikeCommonKADSwhichhasalreadyfoundwidespreadacceptancein practicalapplications.Secondly, PROformais a specialpurposemethod,aimedat a narrower (albeit still very broad)applicationareathanCommonKADS.Asa consequence,a comparisonbetweenPROformaandCommonKADSmight tell us somethingaboutthetrade-off thatis involvedbetweengeneralpurposeandspecialpurposedevelopmentmethodsin KnowledgeEngineeringandMedicalAI.

Thestructureof thispaperis asfollows: In section2, webriefly outlinetheapproachwehave takento thiscomparisonstudy. This is followedby two sectionsdevotedto a brief descriptionof bothPROformaandCommonKADS(sections3 and4). We thendescribetheactualcase-studywe performed(section5) andtheconclusionswedraw from thiscase-study(section10).

2 Evaluation Method

Evaluationof KnowledgeEngineeringmethodsis a hotly debatedtopic in the KnowledgeAcquisitioncommunity[4, 8, 15, 10]. The main problemof suchan evaluationis the difficulty of designinggoodexperimentsfor suchan evaluation,becausetherearetoo many variablesthat play a role. For instance,thebackgroundof theusersinvolvedin theexperiment,ambiguousand/orincompleterequirementson thesystemsthatmustbebuilt in anexperiment,learningeffectswith userswho repeatedlyperform(partsof)anexperiment,etc.

For our comparison-studyof PROformaandCommonKADSwe have chosenfor a re-engineeringexperi-ment:ratherthanbuilding new systemswith thetwo methodsandcomparingtheresults,wehavechosentocomparetheefficacy of PROformaandCommonKADSby re-engineeringanexistingsystemin bothmeth-ods.Theadvantageof sucha re-engineeringexperimentis that thespecificationof thesystemwhich is tobebuilt usingPROformaandCommonKADSis perfectlyclear, becausebehaviour of thesystemis alreadyavailable.Thiseliminatesat leastsomeof theproblemswhichhavehamperedotherevaluationexperiments[8, 15].

As we will describein section5, our re-engineeringstudyconcernsa realisticsystemwhich is already

2

in fieldedusewith a largenumberof generalpractitionersin The Netherlands.This ensuresthat we areapplyingthe methodsto a realisticcase,therebyavoiding the problemsof otherevaluationexperimentswhichwereaimingatartificial (or eventoy) problems[8].

Finally, we also tried to minimise noisein the experimentwith respectto the user’s background. Thesystemis modeledby anintended-userof bothmethodologies(apersonknowledgablein AI andKnowledgeEngineeringtechniques).Furthermore,this personwasa novice in both methodologies,ensuringthat nohiddenlearningeffectswereintroduced.

Notwithstandingall this,ourexperimentis far from ideal.Themostobviouslimitation is thatwehaveonlyperformeda singlere-engineeringexperiment(a singleexisting applicationwhich wasre-engineeredby asingleperson).As hasbeenobservedbefore[13], thesizeandcostof theseevaluationstudiesin termsofrequiredcalender-time andman-power prohibitslarger scaleevaluations.Secondly, our evaluationstudywasperformedusinga ratherearly versionof the PROformatools. We will have moreto sayabouttheeffectson this in ourconclusions(section10).

3 The PROforma methodology

Severalattemptshavebeenmadeto designknowledgerepresentationsystemsthatmake it possibleto sup-port clinical procedures.As alsomentionedin [6] the currenttechniqueshave limitations. The currenttechniquesarenot reliableor expressiveenough.For examplethereis oftennosupportto dealwith uncer-taintyor temporalaspects.ThePROformamethodis designedto meettheseproblems.Themostimportantelementof themethodis thePROformalanguage,this is a knowledgerepresentationlanguagewhich has,amongothers,decisionandprocessmanagementtasks. The PROformalanguageis alsoa specificationlanguage:adeclarative formalismthatcanbeusedto designsoftware.

How to usePROforma Building aclinical protocolin PROformais donein two phases.In thefirst phasea high level descriptionof theprotocolis madewith thegraphicaleditor. In thesecondphasethegraphicaldesignis instantiatedwith knowledgeneededfor theenactmentof theprotocol.

The’ task’is thetoplevel structurein PROforma,all theotherstructuresarederivedfrom it. ThePROformalanguageis designedto givesupportto awidevarietyof clinical tasks,likedecisions,schedulingof actionsover time,monitoringetc.PROformasupportsfour basicclassesof tasks(seefigure1):

� plan: a sequenceof severalsub-tasksor components.Plancomponentsusuallyhave anorderingtospecifytemporal,logicalor sourceconstraints.� decision:is usedwhereacandidatemustbechosenfrom agivensetusingargumentsproandcontra.� action:a procedurethathasto beexecutedoutsideof thecomputersystem,like aninjection.� enquiry:requestsinformationneededto executeacertainprocedure.

Suchprotocolscanbeconstructedwith agraphicaleditor, asshown in figure1

The protocolsmadewith the graphicaleditor have to be instantiatedwith specific data, such as pre-conditions,post-conditions,abort-conditionsanddatadefinitions. The differenttasksin PROformahavea commoninternal structure(commontask attributes)and a specificstructure(task specificattributes).For the commontaskattributesso calledtemplatesareavailable,which easethe enteringof data. Thesetemplatesareequalfor all tasks.Eachtaskhasthefollowing attributes(thesetablesarebasedon[5]):

Commonattrib utesof tasks

3

Figure1: A graphicalrepresentationof PROformaplansin thePROformaeditor: diamondsareenquiries,roundedrectanglesareplans,normalrectanglesareactions,andcirclesaredecisions.Thearrows indicateschedulingconstraints.

Attribute Description

Title Descriptive title of thetaskIdentifier IdentifiernameandparametersDescription Textualdescriptionof thetaskGoal Goalof thetaskTriggerconditions Conditionsthathave to bemetbeforea taskcanbestartedoutsidethe

schedulingconstraints.The triggerconditionscanstarta taskwithoutmeetingtheschedulingconstraintsof its task.

Preconditions Conditionsthathave to bemetbeforea taskcanbestarted.Postconditions Conditionswhichbecometruewhena taskis finishedVersioninfo Author, dateof design,modificationsetc.

As mentionedabove,every taskhasits own specificattributes:

Specificattrib utesof a plan task

4

Attribute Description

Components List of thesubtasksSchedulingconstraintsonsubtasks Constraintswhichspecifytheorderingof thetasksTemporalconstraintsonsubtasks Constraintswhichspecifythetiming of thetasksOptionaltasks List of tasksthatareoptionalNonconfirmatorytasks Tasksthatdonot requireconfirmationNumberof cycles Numberof timesa tasksshouldberepeatedCycleuntil conditions OnwhichconditionshouldtherepetitionstopRepetitioninterval Time interval betweentherepetitionof thetaskAbort conditions Conditionswhichaborta planwhenthey aremetTerminateconditions Conditionswhichspecifywhenaplanis finished

Specificattrib utesof a decisiontask

Attribute Description

Candidates Candidatesfor theresultof thedecisionArguments Ruleswith argumentsfor thevariouscandidatesCommitmentrules Rulesto choosea certaincandidateSources InformationsourceswhichareneededMandatoryInfo Informationneededto confirma decision

Arguments:Thepro’s andcon’s of a candidatearespecifiedusingthearguments.Therearefour differentpossibilities:

�: thesupportfor a candidateis increasedwith oneargument.� : thesupportfor a candidateis decreasedwith oneargument.���

: thecandidateis certainlychosen.��� : thecandidateis certainlynotchosen.

Thedecisionis madein the following simpleway: the supportvalueof a candidateis its numberof�

’sminusits numberof � ’s. Thecandidatesareorderedbasedontheir supportvalue,but a

���or ��� always

hasthe highestpriority. Using the commitmentrulesa decisionis made. This decisionis madeusingasocalledthreshold:whenfor a certaincandidatea certainthresholdis reachedthecandidategetschosen.Whendifferentcandidateshave thesameamountof argumentsthefirst oneis chosen.

Specificattrib utesof an action task

Attribute Description

Procedure Theprocedurethathasto beexecutedContext Context, specialconditionsetc.

In thecurrentversionof PROformatheactionprocedureis onlydisplayedonscreen.Theuserhastoexecutetheprocedure.In thefuturesomeactionscouldbeexecutedautomatically, by callingexternalsoftware.

Specificattrib utesof an enquiry task

Attribute Description

Datarequired Dataitem(s)required.Thedataitemsarefor examplerequestedfrom theuseror hostsystem.

PROformahasa dataexpressionsub-languagethatallows theconstructionsof arbitrarily complex expres-sionsandconditions.This languagecanbeusedwhereverexpressionsor conditionsmustbespecified,e.g.

5

preconditions,decisionargumentrules.Dataitemsmayhavemultiplevalues,whichcanbemanipulatedasa setof values,andcanalsobemanipulatedby specifyingtemporalaspectsof thedata.Thedatalanguageincorporatesall theusualarithmeticoperators,aswell asvarioussetfunctionslikesum,average,maximumandminimum.Furthermoretemporalpredicatesandvariouscomparisonoperatorsareprovided.

DatadefinitionsenablePROformato communicatewith externaldatabasesor hostsystems.With thehelpof datadefinitionsit is possibleto definefor thedataitemsfrom theprotocolwhatthedatatypeof thedataitem is, which valuesit canacceptandwherethedatacomesfrom. Thedatadefinitionsarespecifiedin aseparatefile thatis loadedwhenstartingtheprotocol.

The versionof PROformausedin our casestudywasan evaluationcopy availableto interestedparties.Currently, anew releaseof PROformais preparedunderthecommercialname“Arezzo”.

PROformahasbeenusedto developa numberof clinical guidelines,includingseveral for usein primarycare.SeveralGPsin a Londonsurgeryhave beendoinga studyusinga guidelinefor Dyspepsiawritten inPROforma. A guidelinefor managingpaincontrolhasbeendevelopedin collaborationwith St. Christo-pher’s Hospicein London. Anotherresearchprojectat the ICRF, calledRAGS,hasusedPROformafordecisionsupportin assessingcancerrisks. TheEuropeanprojectMACRO hasembeddedPROformain itsclinical trialsapplication,which is currentlyin usein at leastfour differentcentresin Europe.

It is hopedthatArezzowill becomethestandardfor representingguidelinesin medicine.We have plansfor commercialexploitationandarealreadyin consultationwith potentialcommercialpartners.theICRFexpectto formally publishtheArezzoLanguagesoon,andmeanwhilewecontinuefurtherdevelopment.

4 The CommonKADS methodology

TheCommonKADSmethodologyis a structuredapproachto analysis,designandmanagementof knowl-edgebasedsystems[20], [16]. CommonKADSis basedon the following principles: a modeldrivenapproach (sincemodelsarepurposefulabstractions); knowledge-level principle (modellingknowledgeindependentof implementationdetails); limited-role concept(structuringproblemsolving by identifyingknowledge-types).

CommonKADSconsistsof anumberof models:

� OrganisationModel: concernedwith theorganisationin which theKBS mustfunction� TaskModel: modellinginputs,outputs,preconditionsandperformancecriteriaof theglobaltask� AgentModel: assigningtasksto variousagentssuchashumans,computers,KBS, etc.� Knowledge-Model:a detailedbut implementationindependentdescriptionof theknowledgeusedtoperformthedesiredtasks,andtherolethatdifferentknowledgecomponentsplay in problem-solving.� CommunicationModel: statingthecommunicationbetweenthevariousagents.� DesignModel: describingthetechnicalspecificationof thesystem.

A CommonKADSknowledgemodelconsistsof threecategoriesof knowledge(oftenreferredto as“lay-ers”): domain,inferenceandtaskknowledge(for thepurposesof thispaperwe ignorethestrategic knowl-edge).Thefirst categorycontainsadescriptionof thedomainknowledgeof aKBS application.It describesobjectsproperties,relations,conceptsetcin theapplicationdomain.Thisdescriptionshouldbeasmuchaspossibleindependentfrom therolethisknowledgeplaysin thereasoningprocess.Theinferenceknowledgeof aCommonKADSknowledgemodeldescribesthereasoningsteps(or: inferenceactions)thatcanbeper-formedusingthedomainknowledge,aswell astheway thedomainknowledgeis usedin theseinferencesteps.Theinputsandoutputsof suchinferencestepsarecalled“knowledgeroles”. CommonKADSrepre-sentstherelationsbetweenthe inferencestepsthroughtheir sharedinput/outputroles,with a dependencygraphamonginferencestepsandknowledgeroles. Sucha graph,calledan inferencestructure,specifies

6

only datadependenciesamongthe inferences,not the order in which they shouldbe executed. The useof domain-knowledgeby inferencesis describedby mappingsuchdomain-knowledgeto input-rolesofinferences.The executionorderof the inferencestepsis specifiedas taskknowledge. For this purpose,CommonKADSusesa simpleprocedurallanguagewith proceduraldecompositionto executeinferencestepsandpredicatesto testthecontentsof knowledgeroles. Theseprocedurescanbeorganisedin a task-decompositiontree,andcanbecombinedusingsequences,conditionalsanditerations.TheCommonKADSarchitectureis illustratedin figure2.

subtask

task

primitivetask

TaskLayer

InferenceLayer

DomainLayer

inputknowledge

role

outputknowledge

roleinference

decomposition

invocation

mapping

domaintheory

control knowledge+

Figure2: structureof theCommonKADSknowledgemodel

As shown in figure2, theCommonKADSarchitectureconsistsof separatelayers,with explicit connectionsbetweentheselayers.Primitive proceduresat thetask-layeraremappedto inferencestepsat theinferencelayer, andknowledge-rolesat the inferencelayer aremappedto knowledgeat the domainlayer. Theseseparationswereintroducedto maximisethepossibilitiesfor re-use.For example,a combinationof task-andinferencelayer thatmodela reasoningstrategy for treatmentplanningcouldbereusedandappliedtodifferentsetsof domainknowledge(e.g. treatmentplanningfor heartdiseasesor asthma).Vice versa,thesamedomainknowledge(e.g knowledgeaboutasthma)could be usedfor different inference-andtask-layers(e.g.to eitherdiagnoseor to treatheartdiseases).

7

5 The case-studyBloedLink

5.1 BloedLink

In TheNetherlandsgeneralpractitioners(GPs)yearlyspend75 million guilderson lab research[12], andtheGPoftendoesnothaveenoughknowledgeabouttheright indicationsto choosethevarioustests.Recentdatafrom the Nationalstudyof diseasesandactionsin the office of theGP’s shows a largevariationperworking hypothesisin therequestsof tests[7]. On average,asmuchas45 differentcombinationsof testsarerequestedperworkinghypothesisExcessiverequestof lab researchdoesnotonly generateunnecessarycosts,but canalsobe thecauseof deviating results(e.g. falsepositives),which canby themselvesbe thereasonfor more(unnecessary)diagnosticactivity. Oneway of alteringtherequestbehaviour of GP’s is toaltertherequestform. Therearetwo strategiesfor doingthis:

1. Theuseof a restrictedrequestform, onwhich rarelyusedtestshavebeenleft out.

2. Problemorientedrequests.A formonwhichfor thespecificproblem,only therelatedtestsareshown.(thesewereelectronicforms).

The ErasmusUniversity carriedout researchwith 60 GP’s in the region Delft/Westland/OostlandusingBloedLink with the GP InformationSystem’Elias’to find out how many testswereincludedin eachlabrequest[19]. The useof the restrictedrequestform showed an averageof 6.9 testsper request,whereastheproblemorientedrequestform showedanaverageof 5.5testsperrequests.On averageit turnsout thatthereis a decrease20%in testsper requestwhentheproblemorientedform is used.Thesefigurescanbeexplainedby thefactthattheproblemorientedform is a furtherrestrictionon therestrictedform, sincetheproblemorientedform is basedonaspecificproblem,excludingnotonly rarelyusedtests,but alsorequeststhatwereirrelevantfor thespecificproblem.

The problemorientedrequestforms have to be basedon a standard.Researchhasshown that from theguidelinesof theDutchCollegeof GeneralPractitioners(DCGP)concerningbloodresearch,unambiguousadviceassociatedto working hypothesescanbegenerated[18]. In otherwords,problemorientedrequestscanbederivedfrom theDCGP-guidelines.A possibility is to useanelectronicform, basedon theDCGP-guidelines,to presentworking hypothesesandtheassociatedrequests.SinceGP’s areincreasinglyusingtheelectronicalpatientrecord(EPR)in daily practiceandtheEPRis veryusefulto supportprotocolbasedcare,this is aninterestingoption.By incorporatingthedecisionsupportprogramsin theEPR,informationlike DCGP-labguidelinescan be madedirectly available to the GP. Theseresultshave beenthe mainmotivationfor developingamodulethatsupportstheGPwith requestingbloodtestsdirectlyfrom theEPR.This moduleis calledBloedLink. BloedLink shows theright laboratoryprotocolindexedto thereasonofrequest,basedon a working hypothesischosenby theGP. An (optional)helptext (asmuchaspossibleanabstractfrom the text of the relatedDCGP-standard)givesinformationon the selectionin the protocol.ThemodulegivestheGPtheopportunityto deviatefrom theprotocolby leaving testsout or addingteststo the protocol. Whenthe GP hasfinishedthe requestform it is eitherprintedor sentto the laboratoryelectronically(althoughat themomentthelatteroptionis supportedby only a few laboratories).

BloedLink is a decisionsupportsystemthatis integratedwith theelectronicpatientrecordof two systems(Elias and MicroHIS). BloedLink operatesas follows: When the GP wantsto requestblood tests,sheactivatesthe system. The systemqueriesthe GP aboutthe reasonsfor requestingthe tests. The systemcontinuesaskingquestionsuntil theworkinghypothesisis identified.To determinetheworkinghypothesis,theGP hasto answerup to four questionsthatdealwith the reasonfor requestingbloodtests.Whentheworkinghypothesisis identified,thesystemproposestherelevanttests.

Thecurrentversionof BloedLinkhassomelimitations:

8

� superfluousinformationis presentedto thegeneral practitionerFor exampleif ahypothesisis notconfirmed,oneshouldn’t havethepossibilityto choosea monitor-ing protocol,but only confirmor excludethehypothesis.Anotherexampleis thatwhena patientismaletheguidelineaboutpregnancy shouldn’t beshown.� nosupportfor monitoringIn thecurrentversionof BloedLink whentheworking hypothesisturnsout to bea monitoringtask,theGPis responsiblefor rememberingwhencertaintestsshouldbeadministered.BloedLinkshouldalsoprompttheGPthefor appropiatetestsat theright moment.

Besidesre-engineeringtheexistingBloedLinksystemwehavedecidedto dealwith theselimitationsin thefollowing ways:

� usingknowledgeaboutthepatientTo preventthepresentationof superfluousdatato theGP, BloedLinkshouldusedataaboutthepatientto determinewhich datashouldbe presented.This canbe doneby queryingthe electronicpatientrecordfor dataaboutthepatient.Thenew versionof BloedLinkshouldmake useof moreadvancedreasoningmethods.� supportfor monitoringBloedLinkshouldsupportmonitoring.Thecurrentversionof BloedLinkdoesnotsupportautomatedmonitoring.Thenew versionof BloedLinkshouldhave thispossibility.

Theproposedextensionsto BloedLinkareincludedin thenew versionof BloedLink.

6 CommonKADS modelof BloedLink

This sectiondiscussesthe threelayersof a CommonKADSknowledge-modelof BloedLink. We discussthemodeltop-down,startingwith thetask-layer, followedby inferenceanddomainlayer.

6.1 Task layer

In figure3 thetaskstructureof BloedLinkis presented.Thistaskstructureis ataskdecompositiontree.Theroundedboxesaretasks,tasksaredefinedin termsof their input andoutput. Thesquaredboxesaretask-methods.A task-methoddescribeshow a taskcanberealizedthrougha decompositioninto sub-functionsplus a control regimeover the executionof the sub-functions[21]. The taskandthe task-methodcanbebestunderstoodasrespectively the”what (needsto bedone)”andthe”how (is it done)”view on reasoningtasks[21]. Thecombinedtasks(fig. 3) carryout thetaskof BloedLink,finding theright setof testsfor anabstracthypothesis.

An exampleof a taskdescriptionis:

task find tests;domain name : find-test-by-select-and-match;goal : "find the right tests with hypothesis and objectives";roles:

input: abstract-hypothesis: "abstract hypothesis from user";objectives: "the objective for performing the tests";

output: tests: "the adviced set of tests for the hypothesis";end task find tests;

Below we describethegoalsof eachof thetasksthatcorrespondto thesquaresin thedecompositiontree(fig. 3) andto theroundedboxesin theinferencestructure(seefig. 4).

9

Figure3: Thetasklayerof BloedLink

TASK GOAL

find tests find theright testsusinganabstracthypothesisandobjectivesmonitoring monitor thetreatmentby applyingtheright testsover time. For exam-

ple, to checktheglucoselevel afterfour weeksof treatment.find hypothesis find hypothesiswith abstracthypothesis,patienthistory andselection

of user. For example:”confirm monocytair anemia”find testwith hypothesis find advicedsetof testsfor concretehypothesis

Wegivetwo examplesof thecontrolstructureof thetaks-layerof BloedLink: thetaskcontrol(task-method)of “monitoring” andof the taskcontrolof “find tests”. Thefirst onespecifiesan iterative structureover asubtask,theseconda sequentialstructure.

task-method repeat testing;realizes: monitoringdecomposition :

tasks : find test;roles:intermediate:control structure:

while time < set-time dofind test;

end whileend task-method repeat testing;

task-method use hypothesis to find test;realizes: find test;decomposition :

10

tasks : find hypothesis, find test with hypothesis;roles:intermediate: find hypothesiscontrol structure:find hypothesis(abstract hypothesis + select hypothesis + patienthistory + patient data + select objective -> concrete hypothesis);find test with hypothesis(concrete hypothesis -> tests);

end task-method use hypothesis to find test;

6.2 The inferencelayer

Figure4: Theinferencestructureof BloedLink.

Figure4 shows theinferencestructureof BloedLink. Theboxesaretheinputandoutputroles,theellipsesaretheinferences.Theitemsbetweento horizontallinesaretherule schema’s of thedomainlayerwhichareusedin theinferences.Only thefirst inference“selectabstracthypothesis”is describedin detailbelowasillustration.

inference select-abstract-hypothesis;roles: input abstract-hypothesis;

output abstract-hypothesis;dynamic obtain;

11

domain-mappings:abstract-hypothesis -> abstract-hypothesis;

specification"User selects an abstract hypothesis based on a set ofabstract hypothesis";

end inference select-abstract-hypothesis;

Theuserselectsanabstracthypothesisbasedonasetof abstracthypotheses.Thedomainmappingsspecifytheconceptsof thedomainthataremappedontoroles.

A brief descriptionof theotherinferencesis asfollows:

� inferenceselect-hypothesisTheuserselectsahypothesisbasedonanabstracthypothesis.� inferenceselect-underlying-diseaseTheuserselectsanunderlyingdiseasefrom a setof underlyingdiseasesgeneratedfrom thehypoth-esis.� inferencegenerateObjectivesaregeneratedbasedonthehypothesis,patienthistoryandpatientdata.� inferenceselect-objectiveTheuserselectsanobjective from ageneratedlist of objectives.� inferencespecialiseA concretehypothesisis selectedbasedonobjective,patientdataandthehypothesis.� inferencematchTheinference’match’ selectstestsusingconcretehypothesisandobjective.

6.3 The domain layer

Figure5: Thedomainstructureof BloedLink.

Figure5 showsthedomainstructureof BloedLink. Herethedatadependenciesof theinferencesareshown,e.g. which domainknowledgeis usedfor which inference.The boxesarethe concepts,the ellipsestherule -schema’s andthe itemswithout a borderaroundit arethe inferences.TheconceptsarethedataforBloedLink andthe rule-schema’s theoperationson thedata. The inferencescorrespondto the inferences

12

of the inferencelayer. For exampletheinference”generate”usestheconcepts“abstracthypothesis”,“hy-pothesis”,“patienthistory” andthe rule-schema“generaterules”. An exampleof a conceptcanbefoundbelow.

concept abstract hypothesis;"abstract hypothesis selected by user"

propertiesabstract hypothesis: liver, thyriod hypercholesterolemy, etc.;

end concept abstract hypothesis;

Below is anexampleof a ruleschema.

rule-schema generate-rules;antecedent:

hypothesis;cardinality : 1+;

consequent:objective;cardinality : 1;

symbol:generate;

examples:abstract_hypothesis = liver_diseases and hypothesis = hepatitis_Bindicates objective = confirm;

end rule-schema generate rules;

In thissectionwehavegivenanimpressionof theCommonKADSmodelof BloedLinkof ourcasestudy.

7 PROforma modelof BloedLink

In thissectionwepresentthePROformaspecificationof BloedLink.

Figure6 below givesa high level overview of theBloedLinkprotocol.

Figure6: The“BloedLink” Plan:High level PROformastructurefor theBloedLinkprotocol.

Firstof all thepatientdatais requested:ageandsex of thepatient.Thesewill bestoredin thetaskattributes“age” and“sex”. Thepatientdatawill berequestedfrom thehostsystem.Thentheuseris askedif hewantsto monitor the diseaseof a patientdirectly or wantsto confirm/excludea diseasefirst. In the latter case,whatfollowsis anenquiry“abstracthypothesis”,aplancontainingonly theenquiry“selecthypothesis”andthe plan “find tests”. Theenquiry“abstracthypothesis”requestsfor an abstracthypothesis.Theenquiry“selecthypothesis”is put insidea planto makesurethattheprotocolcontinuesevenif thepreconditionof

13

theenquiry(“are therehypothesesfor theselectedabstracthypothesis?”)is not true,sincea planalwayssucceeds.Theenquiryrequestsfor a hypothesis.Only thehypothesesthatareappropriatearepresented.Finally a plan“find tests”is startedto find theadvicedtestsfor thehypothesis.

Figure7: Thecomponentsandschedulingconstraintsof “find tests”plan. The“find tests”is a subplanof“BloedLink” plan.

Figure7 shows the “find tests”plan. Thesubplansof “find tests”arediscussedseparatelybelow. First aplan “first time” (figure8) is startedto find the teststhatareadvicedwhenno testshave beenperformedpreviously. Thedecision“decidemonitoring”decideswhichmonitoringplanshouldbestarted.

Figure8: The “first time” plan. The “first time” is a subplanof “find tests”,which is itself a subplanof“BloedLink” plan.

Thedecision“ageimportant”decideswhethertheageof thepatientis importantfor thenext selectionof anobjective. If theageis importanttheprotocolwill selecttheplan“selectageobjective plan” thatpresentstheobjectivesthatareappropriatefor a patientwith thespecifiedage. Theselectedobjective is storedinthe dataitem “objectivea”. If the ageis not importantthe protocolwill selectthe plan “selectobjectiveplan”. thatdoesthesame,but storestheselectedobjective in thedataitem “objective”. Lateron thedataitem “objectiva” is copiedto the dataitem ’objective’ to allow the remainsof theprotocolto refer to thedataitem “objective”. Theenquiry“presenttests”presentstheteststhatareappropriategiventheselectedobjective. Finally theplan“test result” is started.It requestsfor thetestresultsof theselectedtests.

After thefirst time plan thesecondtime plan(figure9) is started.For someguidelineslike “Dysfunctionof the thyroid gland” and“Asthma-COPD”a secondstepis neededbecauseif the testresultsof the firststepwereabnormala secondtest is neededto confirm the hypothesis.The protocoladvisesa differenthypothesisif the resultsof the secondtime plan conflict with the chosenhypothesis.If for the chosenhypothesisanunderlyingdiseasecouldbethecauseof theillness,theunderlyingpathologyplanis started:

In theunderlyingpathologyplan(figure10)thegeneralpractitioneris askedif she/hesuspectsanunderlyingdisease.If so the useris asked to selectan underlyingdisease.Only the diseasesthat areappropriateat

14

Figure9: The “secondtime” plan. The “secondtime” plan is a subplanof “first tests”,which is itself asubplanof “BloedLink”.

Figure10: The“underlyingpathology”plan.The“underlyingpathology”is asubplanof “find tests”,whichis itself a subplanof “BloedLink” plan.

the momentarepresented.Thenthe testsneededto confirm the underlyingdiseasearepresented.Theusercandeselecttestsif she/hewantsto. Next thetestresultsarerequestedandif thetestresultsconfirmtheunderlyingdiseasetheuseris warnedthat theselectedunderlyingdiseaseis theprobablecauseof theillness.Theusercanproceedwith thecurrentprotocolthough.Theusercanalsoselectanotherunderlyingdiseaseto test.After theunderlyingdiseaseplanthedecision“DecideMonitoring” decidesif amonitoringplanis appropriatefor thecurrenthypothesis.If soamonitoringplanis started(for instancemonitor”heartfailure” of fig. 7), if not theprotocolends.

Figure11: An exampleof amonitoringplan(for instance“monitor heartfailure”). Sucha monitoringplanis a subplanof “find tests”,which is itself a subplanof “BloedLink” plan.

If in figure6 theuserchoosesfor themonitoringbranch,thedecision“start monitor” of figure11 decideswhichmonitortasksto startor, if nomonitoringis needed,to endtheprotocol.

15

Figure12: A specificmonitoringplan(for instance“monitor yearly” or “monitor specific”)of a particularprotocol(for instance“monitor heartfailure”). Suchspecificmonitoringplan is a sub-subplan of “findtests”,andthusa sub-sub-subplanof “BloedLink”

Figure12 shows a monitoringplan for the plan chosenin figure 11. The enquiry“selectmonitor tests”presentsthe teststhatareadvicedfor themonitoringtaskandstoresthemin a specificdataitem for thatdisease.Theplan“test resultof monitor...’ is generallythesameastheplan“test result”of the“first time”plan. The only differencesare the namesof the dataitemsand tasks. The decision“continue” decideswhetherthemonitoringtaskshouldcontinueor endbecausethegoalof theprotocolis reached.

Figure13: The“monitoringplans”is a subplanof BloedLinkplan.

Figure 13 shows the plan for “monitoring plans” of figure 6. First the useris asked which monitoringprotocolhewantsto bestarted.This is donewith theenquiry“choosemonitoringprotocol”. Thencertainconditionshaveto befulfilled. Thedataitem“abstracthypothesis”with thecorrectvaluehasto beassertedin thePROformadatabasebecausethemonitoringplansusethisdataitem.

8 Evaluation of PROforma

In this sectionwe will discussthe conclusionsconcerningboth methodsthat canbe drawn from the ex-perimentdescribedabove. We will first discussa numberof shortcomingsthatwe observedin PROformaandCommonKADS(section8.1). It will turn out that themethodshave rathercomplementaryproperties,leadingusto thesuggestionthatthey shouldbecombinedin differentstagesof theKnowledgeEngineeringprocess(section9). Sucha combinationrequiresa mappingbetweenthetwo methods,for which we willoutlineaproposal.

8.1 Shortcomingsof PROforma

We haveidentifiedfivemayorshortcomingsof PROformaasfollows:

1. necessityof usingglobaldatarepresentations

16

2. data-dependenciesaredifficult to track

3. nooverview of theglobalstructure

4. somemissingfunctionality

5. performanceproblems

Necessityof using global data representations PROformacontainsno mechanismfor encapsulatingdatalocally within a task.Everyvariable(dataitem) is global. This meansthatall dataof onetaskcanbeoverwrittenby anothertask,for exampledatausedin pre-andpost-conditionsof tasks.This is contrarytocommonlyacceptedprinciplesin SoftwareEngineering,whichpromotelocalencapsulationof dataasmuchaspossible,in orderto preventproblemssuchasname-clashes,illegal read-andwrite-access.This is allthemoretruesincePROformacontainsbuilt-in predicatesto addandremovesuchglobaldatawithoutanyprotection.This makesit very hardto developindependenttasksandhampersreusabilityof PROforma’sknowledgecomponents.Theonly way currentlyavailablein PROformato avoid this problemis to ensurethatall dataitemshavegloballydifferentnames.

Data-dependenciesare difficult to track Theonly dependenciesbetweentaskswhich arecurrentlyvi-sualisedin PROformaarethetemporalschedulingconstraints.However, besidesthesetemporaldependen-cies,tasksalsohave importantdata-dependencies:onetaskusesdataproducedby others. For example,“select-hypothesis”from figure6 producesahypothesiswhich is usedby “select-objective-plan”occurringin “first-time plan” (figure8), which is a subplanof “find-testsplan” (figure7). Thesedata-dependenciesarenotvisualisedin PROforma.Theonly wayto view thedatausedby thetaskis to inspectall task-specificattributesandall commontaskattributesassociatedwith a giventask.Thisproblemis compoundedby thefactthatall data-structuresareglobal,asdiscussedabove.Besidesindicatingthatadata-dependency exists,it shouldalsobeindicatedwhich datais involvedin thisdependency (ie which typeof datafrom onetaskisusedby another).It is easyto imaginehow suchdata-linkscanbevisualised,andthis is beingconsideredfor futureversionsof PROforma.

No overview of the global structur e PROformaonly visualisesa protocolat a single level of detail,namelyall tasksthat occur in the top-level of a plan. Whendescendinginto a subplan,the surroundingpartsof theprotocolarenotseen.Thismakesit difficult to geta globaloverview of thenestedstructureoftheplan.

Somemissingfunctionality In someplaceswefoundsomefunctionalityor expressivenessmissingfromthe PROformalanguage.An exampleof this is in figure 6. There,the enquiryselect-hypothesisis putinsidea plan only to make surethat the protocolcontinueseven if the preconditionof the enquiry(”aretherehypothesesfor the selectedabstracthypothesis?”)is not true, sincea plan is alwaysguaranteedtosucceed.It would have beenmorenaturalto specifythis fail-safebehaviour directly in the enquiry-task.This, andsimilar problemsarerelatedto the fact that we wereusingan early versionof the PROformatoolkit, andareexpectedto improvein futureversions.

Performanceproblems SincePROformais explicitly intendednot only asananalysistool, but alsoasa delivery vehiclefor thefinal operationalsystem,its computationalperformanceis crucial. In particular,theBloedLinksystemis intendedto beusedby generalpractitionerswhohavegenerallyonly very limitedhardwareavailable. Consequently, the currentperformanceof the PROformasystemis a problemfor itsuseasdeliveryplatform.

17

Comparisonwith CommonKADS It wasrathera surpriseto usthatnoneof theabovepointsarein anyway relatedto the specialpurposenatureof PROforma. Secondly, in comparisonwith CommonKADS,we notice that the above points (1), (2) and (3) are handledbetter in CommonKADS:CommonKADSprovidesencapsulationof datawithin tasksandinferences,data-dependenciesarevisualisedat theCom-monKADS inference-layer(the CommonKADStasklayer suffers from the samelack of visualisationofdata-dependencies),andtheglobalstructureof aCommonKADSmodelis typically depictedasin figure2.

8.2 Strong points of PROforma

We now discusssomestrongpointsof PROforma,which canconverselybeseenascorrespondingshort-comingsof CommonKADS.

Temporal constraints In medicalreasoning,timeis oftenanimportantaspect(e.gduringtreatmentmon-itoring). Thesetemporalaspectsoccurfrequentlyin BloedLink,butCommonKADShasnoeasywayto rep-resenttheseaspects.Unlike in PROforma,reasoningin CommonKADSabouttime-points,time-intervalsandtime-constraintsmustbemodelledfrom scratchif required.

Ordering constraints among tasks BloedLink sometimeshasto go backto an earlierinferencewhenthe chosenoption turnedout to be incorrectwhen more information becomeavailable. Under certaincircumstances,it is alsonecessaryto immediatelyabortthereasoning(for instancewhenduringmonitoringthe test resultsbecomenormal). However, both the graphicalnotationand the control-languageof theCommonKADStask structureassumea continuouscontrol-flow during task-execution,which makes ithard(if not impossible)to representsuchdiscontinousjumpsin thecontrol-structure.

Executablemodel After the graphicalPROformamodelhasbeenextendedwith all the requiredtask-attributes, the result is an executableprotocol. No further implementationeffort is neededto obtain arunning system. This is in contrastwith KADS, wherethe knowledgemodel is not executable,and asignificanteffort is oftenneededto transformthis to anexecutableform.

We noticethat thefirst two points(temporalreasoningandnon-localcontrolflows) arerelatedto require-mentsof medicalreasoning.It is perhapsherethattheadvantageof usinga morespecialpurposelanguagelike PROformaover a generalpurposemethodlike CommonKADSpaysoff. On the otherhand,thesefeaturesarerequiredfor many differentdomains,andshouldperhapssimply beregardedasshortcomingsof CommonKADSperse.

9 CommonKADS/PROforma combination

In thebeginningof ourstudyweconsideredCommonKADSasageneralmethodologyfor knowledgebasedsystems,whereasPROformais a methodologyfor a specifictypeof knowledgebasedsystems.However,it turns out that the methodologiesarecomplementary. Our study hasshown that CommonKADSis amethodologythat is veryusefulin theanalysis-phase.In this phasethemorespecificPROformalanguageis too implementationoriented.Theanalysisof severaltypesof knowledge,andwherethesetypesareusedin thevariousreasoningstepsis verydifficult in PROforma,becausereasoningstepshaveto berepresentedasaparticularkind of task(e.g.decision,plan)andtheinstancesof theknowledgetypesmustbeencodedintheattributesof thesetasks.However, in theanalysisphaseit is notall clearwhattypeof taskthereasoningstepsshouldbe,andthereforeit is evenmoredifficult to encodeinstancesof theknowledgetypesin the

18

attributesof PROformatasks. A moregeneralmethodlike CommonKADSworks betterin the analysisphasethenaspecialpurposelanguage.This is a resultof ourstudythatwedid notatall expect.

Ontheotherhand,whenwehavebuilt aPROformamodelthenwealsohaveanactualimplementationof asystem,whereasin CommonKADSwestill haveto implementthemodelin someotherlanguage.Ourstudyhastaughtus that CommonKADSand PROforma arecomplementary:useCommonKADSas analysismethodologyandusePROformaaftertheanalysisphasefor building a clinical procedureapplication.It isthereforeusefulto comeupwith guidelinesfor mappingaCommonKADSmodelontoaPROformamodel.

A mappingof CommonKADSinto PROformaconsistsof a mappingof every layerof a CommonKADSmodel:thetasklayer, theinferencelayerandthedomainlayer. We will discusseachmappingbelow.

Mapping of the task layer Thetasklayerconsistsof thetaskdecompositiontreeandthecontrolstructure.Theprimitive inferencesof the taskstructure(the leavesof thedecompositiontree)have to be translatedto plansin PROforma. The control structurespecifiesthe orderingof the tasks. This orderingwill ingeneralbethesamein CommonKADSasin PROformamodel,thereforecreatescheduling-linksbetweenthePROformaplansaccordingto thecontrolstructureof the tasklayer. Theresultof themappingof thetask-layeris thetop-structureof thePROformamodelincludingtheschedulinglinks. Thestructureof theCommonKADStask-decompositiontreecanbemodelledby a correspondingnestingof PROformaplans,althoughthis nestingcannotbevisualisedin thecurrentPROformagraphicalenvironment.Figure14 andthefollowing controlstructureillustratethemappingof a tasklayerinto apartialPROformamodel.

Figure14: MappingCommonKADStasklayerto PROforma.

task-method evaluate by select and evaluation;....

control structure:select(hypotheses -> hypothesis);evaluate(hypothesis -> decision);

end task-method evaluate by select and evaluation;

Mapping of the inferencelayer The PROformaplan madewith the task layer is refinedwith the in-ferencelayer. Static rolesandinput rolesof the inferencesaretranslatedto enquirytaskof PROforma.The instancesof the rolesbecomedata-itemsof PROformatask(seefigure15. A CommonKADSinfer-encestructureexplicitly representsthe rolesof the domainknowledgeandthedependency of theseroles

19

in the several reasoningsteps.However, both the abstractionof domain-knowledgeto rolesandthe datadependency betweeninferencesarelost in aPROformamodel.

Figure15: MappingCommonKADSinferencelayerto PROforma.

Mapping of the domain layer The conceptsof the domainlayermapdirectly to the dataattributesofPROformatasks.A mappingof thedomainstructureis notpossible.Thedomainknowledgeis distributedover several placeslike templates,datadefinitions,actions,etc. For instance,rule schemascanmaptothe templatesof decisions,a knowledgebase,actionsandenquiries.Onerule in CommonKADScanbemodelledwith severaltasksin PROformawhichhavetheirdatastoredin differentplaces.

10 Conclusion

In the last decadeknowledgeengineeringhasshown that specialpurposemethodologiesfor knowledgebasedsystemsareuseful,besidesthe moregeneralmethodologiesof softwareengineering.This studyshows that an even morespecialisedmethodologylike PROforma is useful. PROforma is intendedforclinical proceduresin themedicaldomain,which still constitutesa largeclassof differentproblems(e .g.diagnosis,monitoring,treatmentplanning)within a largedomain.

For our evaluationstudy, we chosea methodbasedon re-engineeringa realisticsystemin two methodolo-gies:thenew andspecialpurposeKBS methodologyPROformaandthewidely accepted,andmoregeneralKBS methodologyCommonKADS.

Both the CommonKADSandPROformamethodologiesturnedout to be usefulandeasyto learnmod-elling tools. Both methodologieshave their limitations. Problemsduring themodelingprocessusingtheCommonKADSmethodologyoccuronly at thetasklayer, for instancethelackof temporalreasoning.Theoppositeseemsto bethecasewith PROforma. In thePROformamodelingprocessproblemsoccursat thelevel of the inference-anddomainlevel. As pointedout earlierPROformacould be improvedat severalplacesin particular: the lack of dataencapsulation,the hiddendatadependenciesin the model,and itsperformanceasplatformfor theapplication.

Our studyhastaughtus that CommonKADSandPROformaarecomplementary:useCommonKADSasanalysismethodologyandusePROformaafter theanalysisphasefor building a clinical procedureappli-

20

cation.We gaveguidelinesfor mappinga CommonKADSmodelontoa PROformamodel,sinceapplyingthemhelpsin building thefinal PROformamodel(which is theactualsystem).

As mentionedabove, it seemsthatCommonKADShasits strengthson thedomainandinferencelevel andPROformaonthetasklevel,at leastin thecaseof ourBloedLinkstudy. Thisdifferencein strengthshashadits implicationson themappingfrom CommonKADSto PROforma. It waseasyto mapthetaskstructureof CommonKADSto PROforma,but the inferenceanddomainlayer raisedproblems.It hasbeena littledisappointingthatit wasnotpossibleto specifyclearrulesto mapaCommonKADSinference-anddomainlayerto partsof a PROformaspecification

Our study was surprisingto us at threepoints. Firstly, the strongpoints of PROforma were relatedtorequirementsof medicalreasoning,but they arerequiredfor many differentdomains,andthereforshouldbeconsideredasshortcomingsof themoregeneralCommonKADSapproach.Secondly, theweakpointsofPROformaarenot in any wayrelatedto thespecialpurposenatureof PROforma.Finally, it turnsout thatintheanalysisphasefor building a clinical procedureapplicationCommonKADSwasmoreappropriatethenthespecialpurposemethodologyPROforma.ThereforeCommonKADSandPROformaarecomplementaryfor building suchclinical procedureapplications.

References

[1] J. Angele,D. Fensel,D. Landes,andR. Studer. Developingknowledge-basedsystemswith MIKE.Journalof AutomatedSoftwareEngineering, 1998.To appear.

[2] C. BauerandW. Karbach,editors. ProceedingsSecondKADSUserMeeting, ZFE BT SE21, Otto-HahnRing6, D-8000Munich83,17–18February1992.SiemensAG.

[3] B. Chandrasekaran.Generictasksin knowledgebasedreasoning:High level building blocks forexpertsystemdesign.IEEEExpert, 1(3):23–30,1986.

[4] C. Corbridge,N.P. Major, andN.R. Shadbolt.Modelsexposed:An empiricalstudy. In Proceedingsof the9thAAAI-SponsoredBanff KnowledgeAcquisitionfor KnowledgeBasedSystems, 1995.

[5] J.Fox, N. Johns,andA. RahmanzadehA. PROMPT,protocolsfor medicalprodeduresandtherapies,PROMPT design. TechnicalReportEC 4FPProjectNummerHC1043,ImpericalCancerResearchFund,April 1997.

[6] J. Fox, N. Johns,andA. RahmanzadehA. Protocolsfor medicalproceduresandtherapies- a provi-sionaldescriptionof thePROformalanguageandtools. In Proceedingof AIME 97. Springer1997,1997.

[7] I. Kluijt, J.O.M Zaat,J. van der Velden,J.Th.M. van Eijk, andF.J. Schellevis. ”voor eenprikje”,generalpracticionersandbloodtests.resultsof thenationalstudyon diseasesandtreatmentin first-line care.HuisartsenWetenschap, 34:67–71,1991.(in Dutch).

[8] M. Linster. Sisyphus’91/92:Modelsof problemsolving. Int. J. of HumanComputerStudies, 40(3),1994.Editorialspecialissue.

[9] C. Lockenhoff, D. Fensel,andR. Studer, editors.Proceedingsof theThird KADSUserMeeting, ZFEBT SE21,Otto-HahnRing6, D-8000Munich83,8–9March1993.SiemensAG.

[10] D. Marques,G. Dallemagne,G. Kliner, J. McDermott,andD. Tung. EasyProgramming:Empowering Peopleto Build Theirown Applications”. IEEEExpert, pages16–29,june1992.

[11] M. Musen,S.Tu,A. Das,andY. Shahar. A component-basedarchitecturefor automationof protocol-directedtherapy. In Proceedingsof AIME 95,Lecture Notes934, pages5–13.Springer, 1995.

21

[12] DenHaag:Housesof Parliament.Financialsurvey medicalcare. Parliamentarysession1993-1994,23407,nrs1-2,pag.243,(in Dutch).

[13] R.J.Offen andR. Jeffery. Establishingsoftwaremeasurementprograms.IEEE Software, pages45–53,march/april1997.

[14] J. Rumbaugh,M. Blaha,W. Premerlani,F. Eddy, andW. Lorensen.Object-OrientedModellingandDesign. PrenticeHall, EnglewoodCliffs, New Jersey, 1991.

[15] A. Th. Schreiberand W. P. Birmingham. The Sisyphus-VTinitiative. International Journal ofHuman-ComputerStudies, 43(3/4):275–280,1996.Editorialspecialissue.

[16] A. Th. Schreiber, B. J. Wielinga, J. M. Akkermans,W. Van de Velde,andA. Anjewierden. CML:TheCommonKADSconceptualmodellinglanguage.In L. Steels,A. Th. Schreiber, andW. VandeVelde, editors,A Future for Knowledge Acquisition.Proceedingsof the 8th EuropeanKnowledgeAcquisitionWorkshopEKAW’94, volume867of Lecture Notesin Artificial Intelligence, pages1–25,Berlin/Heidelberg,September1994.Springer-Verlag.

[17] I. A. vanLangevelde,A. W. Philipsen,andJ. Treur. Formalspecificationof compositionalarchitec-tures.In B. Neumann,editor, ProceedingsECAI’92,Vienna, pages272–276,Chichester, 1992.Wiley.Longerversionavailableas: ReportIR-282,MathematicsandComputerScience,FreeUniversityofAmsterdam.

[18] M.A.M. van Wijk, A.M. Bohnen,and J. van der Lei. Advice on blood-testsin the standardsofthe dutchassociationof generalpracticioners:sufficiently consistentfor useby the practicionerinproblem-orientedrequest-protocols.NedTijdschrift voor KlinischeChemie, 21:285–289,1996. (inDutch).

[19] M.A.M vanWijk, B.M.T. Mosseveld,A.M. Bohnen,andJ.vanderLei. Requestsof laboratorytests:experienceswith bloedlink in the delft region. In P.C.G.M. Sollet, editor, Huisarts, SpecialistenhetElektronischeMedisch Dossier, Werkenaantransmuralezorg, pages13–25.Rotterdam:ErasmusUniversiteit,1997.(in Dutch),ISBN 90-74479-06-5.

[20] B. J. Wielinga andJ. A. Breuker. Modelsof expertise. In ProceedingsECAI–86, pages306–318,1986.

[21] B. J.Wielinga,A. Th.Schreiber, andJ.A. Breuker. KADS: A modellingapproachto knowledgeengi-neering.KnowledgeAcquisition, 4(1):5–53,1992.Specialissue‘The KADS approachto knowledgeengineering’.Reprintedin: Buchanan,B. andWilkins, D. editors(1992),Readingsin KnowledgeAcquisitionandLearning, SanMateo,California,MorganKaufmann,pp.92–116.

[22] E. Yourdon.ModernStructuredAnalysis. PrenticeHall, EnglewoodCliffs, New Jersey, 1989.

22