skonetzki: helen, a modular framework for representing and ...€¦ · the first three cycles focus...

© 2004 Schattauer GmbH

Methods Inf Med 4/2004

413

place. Local recommendations have tobe made due to the fact that for certainillnesses different prevalences should betaken into account, and because everyhealth institution has its own resourcesand experiences [6, 16-19]. Furthermorethe explicit adaptation invites future users to actively participate during anessential part of the development pro-cess.

● Providing patient-specific recommenda-tions during patient encountering is often recommended but only possiblethrough individually tailored computer-based decision support systems [20 to22].

● Rapid prototyping for effective author-ing and testing during the developmentprocess [23]. This is a possible way tosimulate the usability and effect of therecommendations and to systematicallytest and refine the achieved evidencefoundation.

The HELEN-Project was established toface these problems and to systematicallyintroduce CPGs at the Department of Neonatology of the Heidelberg UniversityMedical Center.

The main focus of this project is to at-tempt the following:● to represent existing CPGs from various

sources by preserving their current pres-entation format and structure;

● to provide an editor that easily allowsour development team to enter existingCPGs as narrative text and/or as struc-tured algorithms and to apply necessarychanges during adaptation in an audit-able manner;

● to generate a web-based view for brows-ing the adapted CPG and the underlyingCPG source;

HELEN, a Modular Framework for Representingand Implementing Clinical Practice GuidelinesS. Skonetzki1, H.-J. Gausepohl2, M. van der Haak1, S. Knaebel2, O. Linderkamp2, T. Wetter1

1Department of Medical Informatics, Heidelberg University Medical Center, Germany2Department of Neonatology, Heidelberg University Medical Center, Germany

1. IntroductionWithin the past two decades, hundreds ofclinical practice guidelines (CPGs) havebeen developed, aiming at assisting clinicalpractitioners and patients in the decisionabout the appropriate treatment in specificclinical circumstances. However, CPGs often fail to affect clinical practice [1-3].Therefore systematic research is done tofind out why physicians do not follow CPGs[4-7]. Besides, several researchers deal withidentifying strategies for improving physi-cians’ compliance with the recommenda-tions of the CPG [8-12]. Some of these aspects are:● Consideration of different needs caused

by different roles and experiences of theusers [13, 14].This is necessary in order to avoid usersfinding information on the wrong levelof experience, who therefore becomemore and more frustrated by using theCPG.

● A flexible implementation strategybased upon the specific focus of theCPG and individual barriers of the target care setting.This is necessary due to the wide varietyof topics addressed within CPGs (e.g.,standardization of diagnostic proce-dures, assistance during planning of cer-tain therapies, or reducing avoidablerisks) and to the fact that barriers in onesetting may not be present in others(e.g., organizational constraints, lack ofresources) [4, 15].

● The explicit adaptation of CPGs.This is necessary to gain user acceptanceand to provide recommendations notonly specific to the patient, but also tothe clinical setting and to the specific institution in which the treatment takes

SummaryObjectives: In order to implement clinical practiceguidelines for the Department of Neonatology of theHeidelberg University Medical Center we developed amodular framework consisting of tools for authoring,browsing and executing encoded clinical practiceguidelines (CPGs).Methods: Based upon a comprehensive analysis of literature, we set up requirements for guideline repre-sentation systems. Additionally, we analyzed furtheraspects such as the critical appraisal and known bridges and barriers for implementing CPGs. Thereafterwe went through an evolutionary spiral model to develop a comprehensive ontology. Within this modeleach cycle focuses on a certain topic of managementand implementation of CPGs.Results: In order to bring the resulting ontology intopractice we developed a framework consisting of atool for authoring, a server for web-based browsing,and an engine for the execution of certain elements ofCPGs. Based upon this framework we encoded andimplemented several CPGs in varying medical domains.Conclusions: This paper shall present a practicalframework for both authors and implementers ofCPGs. We have shown the fruitful combination of different knowledge representations such as narrativetext and algorithm for implementing CPGs. Finally, weintroduced a possible approach for the explicit adapta-tion of CPGs in order to provide institution-specific recommendations and to support sharing with othermedical institutions.

KeywordsClinical practice guidelines, representation, acquisition,implementation, local adaptation, execution, decisionsupport

Methods Inf Med 2004; 43: 413–26

● to automatically generate computer-based decision support components forproviding patient specific recommenda-tions, reminders, or alerts based upon thisCPG also during patient encountering;

● to facilitate sharing of CPGs with othercooperating institutions, even if these in-stitutions are only interested in parts ofthe text or algorithms or in the case thatthey wish to apply minor changes to theCPGs (e.g., different critical values orother preferences for certain therapyoptions).

Different approaches and tools have beenpresented in the literature for representingCPGs. Some of them focus initially on thetransfer from paper-based to computer-based formats in order to handle and dis-tribute CPGs in an easy way [24-26]. Thesetypes of approaches mainly improve the accessibility to the guideline itself.

Other approaches such as GEM addi-tionally focus on computer-interpretableelements for appraisal issues, eligibility de-termination or on presenting certain partsas single-step guidelines by means of deci-sion variables. [27-29]. With these kinds ofapproaches it is hard to represent complexplans for diagnosis, therapy or patient man-agement. As a complement to GEM theGLARE project developed a comprehen-sive representation of clinical algorithmswith respect to several aspects of executionsuch as sequencing, handling of concurrenttasks and errors [30]. Moreover within theGLARE project a flexible engine for exe-cuting encoded CPGs was developed. Themain problem of GLARE as of the SIEG-FRIED representation is the internal stor-age of the encoded CPG within a relationaldatabase system without export capabilities[31]. Therefore, sharing CPGs with other institutions is still problematic.

The Guideline Interchange Format (GLIF) symbolizes an intermediate con-cept between narrative CPG formats andcomputer executable formats [32]. GLIFmainly focuses on sharing CPGs among different institutions. Until version 3 the ex-pressiveness of the language used to formu-late decision criteria was therefore rathervague [33]. Furthermore GLIF 2.3 encodedCPGs are only executable by extensions as

shown by [34], or [35]. To overcome thisweakness following versions of GLIF (3.x)were extended by several new constructswhich allow a formal definition of decisioncriteria, action specification and patientdata. Therefore a more expressive object-oriented expression language (GELLO),as well as medical vocabularies (UMLS), astandard medical data model (HL-7 RIM),and elements of the “virtual medicalrecord” (vMR) are now included [36, 37].

As an extension to the GLIF concept,P-CAPE refined amongst others elementarydecision criteria resulting in a computer-executable CPG representation, thus over-coming this weakness.Additionally P-CAPEincludes a high-level tool for entering guide-line parameters in the form of algorithms.The output of this tool are MUMPS codeand data structures used within the Brighamand Women’s Hospital Information System(BWH) [38]. The core elements of P-CAPE“Navigator” and “Notifier” were speciallyimplemented for the use within the BWHalerting system [39].

Other published concepts for represent-ing CPGs are PRODIGY, Asbru, G-CARE, Opade, Prestige, and PROforma[40-44]. All of these concepts are customtailored to the requirements of specialclasses of CPGs, such as clinical protocols,single-step decision rules or reasoningabout temporal abstraction data. Thereforethey often form additional barriers forsharing CPGs across different institutions.

Therefore EON developed a flexible approach for modeling different classes ofCPGs (e.g. one-shot decision, complex mul-ti-encounter CPG, clinical trial protocol).The EON approach is based upon a com-mon conceptual model of time and patientdata [33, 45]. The author can choose amongdifferent modeling primitives that can bemixed and matched in order to constructprocesses of decision-making, action se-quencing and data interpretation as neededby the different classes of guidelines. In ad-dition both an authoring and execution en-vironment were developed to complete thisapproach [46]. Unfortunately this modularand flexible approach is not able to dealwith narrative guideline elements (e.g., forexplanation or educational purposes) nordoes it include information needed by

systematic appraisal. Because it is not in thescope of this article we would like to referto www.openclinical.org where a compre-hensive overview with detailed descrip-tions of many CPG representation systemscan be found. Additionally systematic com-parisons of certain features and the inter-connections between different guidelinerepresentation models can be found in [22,47], and [48].

2. Methods – Development of the Knowledge RepositoryConsidering the amount of proven repre-sentation concepts for CPGs, we initiallyaimed at using one of these. We made sev-eral attempts to rearrange or extend suchrepresentation schemes in order to fulfillthe requirements of the German MedicalSocieties and the German Guideline Clear-inghouse to which our CPGs have to besubmitted [49]. These modifications includ-ed a.o. a systematic labeling of the underly-ing medical evidence for each statementand structuring the considerable amount ofadditional information which is neededduring appraisal, classification, and clearingprocess of the CPG. Especially our at-tempts to rearrange and insert new ele-ments without destroying the core system(thus allowing us to benefit from future de-velopments of the underlying system) ledto really complex and sometimes confusingtools for our authoring team. Therefore wedecided to develop a new representationscheme custom-tailored to the needs withinour setting. However we tried to build onproven concepts instead of developing newones as much as possible.

Considering the aim of developing aflexible, shareable, and computable de-scription of the CPG, we identified the con-cept of ontologies to be useful.An ontologyis a formal specification of terms in the target domain and relations among them[50].This formal representation can be usedto share a common understanding of thestructure of information among people orsoftware agents [51].

Within an ontology terms are stated assingle elements such as frames or objects

Skonetzki et al.

414


with their associated slots or attributes.Theseelements are usually structured within a hierarchy. The relations that are used withinthe hierarchy must be specified by the author(e.g., ‘is-a’, ‘consist-of’, or ‘is-part-of’).

As a methodology for building the ontology we used the process model of theModel-based Incremental Knowledge En-gineering Process (MIKE) [52]. This modelis based upon the Spiral Model of SoftwareDevelopment as presented in [53].

The main focus of this model lies in thecyclic processing of the following four tasks:● analysis of requirements● design or refinement of the ontology● implementation of an initial knowledge

acquisition tool and its extensions● evaluation by entering examples and

test usage of the encoded information.

Within the HELEN project we have chosena development process with six cycles.The first three cycles focus on certain topicsof the domain such as management/ap-praisal, document structure, and algorithm,whereas the last three cycles are dedicatedto the development of the authoring tool,the guideline viewer and the execution engine.

2.1 Analysis of RequirementsEach cycle starts with a careful collection ofthe specific requirements. As the first cyclefocuses mainly on management aspects westarted with an initial concept for the ontol-ogy. This concept is based upon require-ments derived from the literature abouttools for critical appraisal (e.g., AGREE,SIGN, NGC, AWMF) [54-58], the Guide-line Element Model (GEM) [59], and fromour own requirement analysis at the De-partment of Neonatology [60].

The second cycle focuses on identifyingelements for preserving the documentstructure of existing guidelines as publishedby e.g., SIGN [61] and DEGAM [62]. Thiswas necessary to enable the mapping of existing text-based CPGs. The third cyclewas dedicated to the requirements of acomputable algorithm format according toa.o. EON [33].

One of the intentions of the HELEN-Project was to also include medical domain

experts during the development of comput-able algorithms. However, one has to con-sider that medical domain experts usuallysubconsciously evoke scripts of actionsrather than explicitly think in formal rules,necessary for inference machines usedwithin common decision support (e.g.,[63]). Therefore, we conceived representa-tion elements based upon concepts as usedby procedural programming languagessuch as C or Pascal.

Within the last three cycles we used stan-dard software engineering methodologiesfor collecting and analyzing requirements ofthe developed tools and for recording themwithin a functional specification. By imple-menting these tools we also applied system-atic refinements to the ontology such asmodifications to the forms for entering theCPG (authoring tool), an additional labelingfor user-specific presentation (GuidelineViewer), or a more detailed description ofcertain elements for automatic processing(Guideline Execution Engine).

2.2 Design of the OntologyAfter the exploratory process applied forthe analysis of requirements we chose atop-down development process for design-ing the ontology according to [64]. There-fore we started with the definition of themost general concept of each cycle (hereguideline, pragmatics, knowledge modules,and adaptation). Subsequently, we refinedthis concept by means of a hierarchy using‘is-a’ relations. This kind of relation repre-sents the fact that every subclass inheritsthe descriptive attributes from its super-class. Consequently, instances of the sub-classes can be stated as a specialization ofinstances of the superclass.

In order to achieve more expressivenessof the ontology we used specialized attrib-utes which refer to other instances basedupon the ‘consist-of’ relation. These attrib-utes are labeled with a name followed by ‘_I’(e.g. Developer_I which could be interpretedas: ‘developer consists of a list of people,groups or organizations’). To preserve theability to merge with other ontologies theprefix ‘HELEN_’* as a kind of namespacewas used for all classes that are specializedto the context of CPGs. For classes not spe-

cialized for the representation of CPGs weused standard names without any prefixes.

Wherever possible values for attributeswere based upon existing classification orindexing schemes such as provided byNGC or other stakeholders [57, 65].

After each development cycle wechecked the resulting ontology accordingto the methodology as published in [66].This task is necessary for ensuring the cor-rectness regarding:● the correct use of the ‘is-a’ relation with-

in the ontology and ‘consist-of’ relationwithin specialized attributes;

● avoiding cycles in the class definitionlike A has a subclass B and at the sametime B is a superclass of A;

● analyzing siblings within the hierarchy;● consistent use of classes, property values

and instances.

As a result of the iterative processing of thesix cycles we obtained an ontology with fivemajor trees containing 83 classes and over200 descriptive attributes.As a main class weintroduced the class HELEN_Guideline toaccess the guideline and to instantiate theother classes. HELEN_Knowledge_Modulerepresents all the content produced duringthe development process (e.g. text, graphicsor algorithm). The life-cycle of the guide-line is reflected by the next two classes.HELEN_Adaption contains the localizedcontent produced during the adaption pro-cess plus a documentation about this pro-cess. HELEN_Pragmatics is a special classcontaining the necessary accompanying doc-uments of a guideline, the documentationabout the development process, and also aset of additional information needed for theclearing process. HELEN_Modules containsseveral classes which can be understood ascomplex data types used to organize all the content which is linked with the majorclasses. These classes are not directly access-ible as top level classes since they can onlybe instantiated from one of the major classesor their components.

HELEN, a Practical Framework for CPGs

415


* Names written in italic such as Guideline Viewer refer to a developed tool within theHELEN framework, names written in italicand conjunct by the prefix HELEN_ such asHELEN_Decision refer to an entity of thedeveloped ontology.

The full annotated ontology and someexamples for the usage can be downloadedat: http://www.med.uni-heidelberg.de/mi/research/dss/helen/helen.htm.

2.3 Implementation of the InitialKnowledge Acquisition ToolFor the knowledge acquisition task we chosethe platform-independent and configurableProtégé-2000 toolset [67]. This tool consistsof an ontology-based knowledge model, aknowledge acquisition tool, and a processmodel similar to our own approach. Theinternal format for the knowledge model isMODEL, a frame-like representation lan-guage which is an extension of the CLIPSlanguage. Also part of this toolset is a formgenerator which allows a flexible and semi-

automatic configuration of entry formsbased upon the encoded ontology. This fea-ture especially meets our requirements toreduce the effort for the necessary develop-ment of a knowledge acquisition environ-ment during each cycle.

In addition, Protégé-2000 supports sev-eral formats such as RDF, Standard TextFile, XML as well as the ability to exportthe acquired knowledge base to an externalJDBC Database.This allows a smooth tran-sition from the acquisition to the test use ofthe encoded knowledge.

2.4 Evaluation of the OntologyA detailed test after each cycle is needed in order to verify whether the target set before has been met. Since the first threecycles focus on representation issues, we

regularly checked the ability to encode allinformation provided by published guide-lines from DEGAM and SIGN. Further-more we tested whether it was possible tocarry out a systematic appraisal based uponthe encoded information. Therefore weused the checklist for methodological qual-ity of guidelines as published by [68].

Since there is no normative frameworkto determine the necessary componentsand required expressiveness of the comput-able algorithm format, we derived elemen-tary features both from published descrip-tions and examples of the above mentionedmodels as well as from comprehensivecomparisons of computer-interpretableguideline models such as [47] and [48].These comparisons deal with the type, thelevel of detail, and the number of givenprimitives used for assembling the algo-

Skonetzki et al.

416


Fig. 1 Definition of a therapy cycle processed until new data arrives

rithms. By examining these, we found simi-lar primitives for representing several typesof actions, decisions, branching, and nestedsubplans. Other dimensions that were com-pared deal with the expressiveness of theimplemented language used for definingdecision criteria, setting goals, achievingdata interpretation/abstraction, and man-aging the control flow. Since there is no objective criteria as to which type of ex-pression language is superior to the otherwe examined HELEN’s ability to encodeseveral types of clinical actions (choosing adiagnostic procedure, refining a diagnosis,planning and carry out of an appropriatetreatment) and how to use these encodedexpressions within several modes of execu-tion (decision support, workflow support,

watchdog, or reminder) during severaltests.

During the 4th cycle we customized theProtégé-2000 authoring environment andimplemented, if necessary, additional entry-wizards. This task itself checked the con-sistency and completeness of the ontologyagain and additionally, it allowed to test theability to encode CPGs in a user-friendlymanner. Within the 5th cycle we developeda web-based tool to browse encoded CPGs.This task checked the correctness of the en-coded information with regard to the im-port of the XML and XSD files which areproduced by the authoring environment.We also verified the correct presentation ofguideline documents as encoded and thecomplete presentation of information as

needed for appraisal. The last cycle wasdedicated to the implementation of the execution engine. Here we carried out asystematic check of the implementation ofeach element that is used within an algo-rithm. Afterwards we checked the correct-ness of encoded Boolean and arithmeticexpressions. Another check was aimed atthe correct processing of encoded test algo-rithms by means of an execution as expect-ed. Finally we applied medical algorithms totest cases derived from real cases at the De-partment of Neonatology. Some examplesabout the CPGs which were used for thisevaluation can be found under section 4 ofthis paper. In the next section we will give ashort overview of the developed tools of theframework and how they interact.


417


Fig. 2 Major tools for authoring and execution of the HELEN framework and their communication via file, RMI or HTTP

3. Architecture of the SystemBased upon an assumed life-cycle whichconsists of authoring a national or interna-tional CPG, systematic appraisal, adapta-tion for local use, and implementation inclinical practice we identified an initial setof three tools (Fig. 2). The authoring envi-ronment is called the Guideline Editorwhich is a Protégé-2000 based tool for encoding CPGs according to the HELENOntology. This Guideline Editor helps theauthors in structuring, entering, and modi-fying recommendations during the initialdevelopment or adaptation. The result ofthese processes can be stored either in thenative format of Protégé-2000 or as XML-Schema [69]. In order to handle XML/XSDfiles and other accompanied files such aspictures of the included algorithm in an

easier way, all data can be stored within azip-archive.

The second tool of this framework is a WWW-Server (Guideline Viewer) forbrowsing encoded CPGs via a standard web-browser. For this purpose we developeda Java Servlet processed within an Apache-Tomcat web server [70, 71].This servlet readsXML-encoded CPGs from the earlier men-tioned zip-archive and produces differentviews for presenting the content either to theappraisal team or to different groups ofhealth care professionals according to theirspecific needs. The Guideline Viewer natural-ly focuses on the text-based elements of theCPG, but it is also possible to browse imagesof included algorithms and to request furtherinformation on certain elements such asviewing sub-algorithms, specifications of rec-ommended therapies or browsing through

associated background material. For mainte-nance purposes, the Guideline Viewer addi-tionally provides a web-based interface foruploading and managing CPGs on the serv-er. It also has the capability to synchronizeactive CPGs with the third tool of our frame-work, i.e. the Guideline Execution Engine.This is necessary to ensure that all activetools carry out the same version of the CPGthereby avoiding mistakes. As a complemen-tary tool to the Guideline Viewer, the Guide-line Execution Engine is specialized in tra-versing and processing included algorithmsfor certain patients during encountering.Therefore, the engine administrates addi-tional information necessary for executingCPGs, amongst others data about registeredusers (e.g., user name, position, role, avail-ability, and period of service). The GuidelineExecution Engine also communicates via

Skonetzki et al.

418


Fig. 3 Modular document model of the HELEN_Ontology. Beside pragmatics (e.g., information for systematic appraisal) each CPG contains at least one guideline document which is composed of knowledge modules in order to represent the recommendations of the guideline.

RMI (Remote Method Invocation) with sev-eral clients for retrieving data, providingchoices to the users, and for informing themabout necessary tasks. It can also invoke theGuideline Viewer to present associated back-ground material suitable for the current taskof the algorithm.

In order to ensure the platform inde-pendency when connecting the GuidelineViewer and the Guideline Execution Engineto a hospital information system (HIS) anadditional custom-tailored mediator com-ponent is necessary. This mediator compo-nent is responsible for communicating vari-able requests between the guideline toolsand the local HIS as well as for receivingADT-data and for generating trigger eventsfor the Guideline Execution Engine to acti-vate, abandon, or manipulate the guidelineflow control. Despite the explicit task shar-ing, the software has proved to be a frame-work which is easily to customize to differ-ent scenarios of use by omitting a tool,replacing one with a more sophisticatedversion, or by introducing a new tool withadditional features.

3.1 Guideline AuthoringIn matching the life-cycle model the Guide-line Editor aims at assisting authors duringthe initial development and adaptation.Therefore the editor allows authors tocreate new CPGs, to edit existing CPGs,and to store them in Protégé or XML for-mat. A suitable set of additionally devel-oped plug-ins and widgets assists the au-thors in formulating guideline recommen-dations as well as in checking the complete-ness, the syntactic correctness, and withinnarrow confines also the semantic correct-ness of entered values.

One of the advantages of the underlyingProtégé-2000 environment in combinationwith the developed ontology is the fact thatthe developer team can choose a top-down,as well as a bottom-up or a middle-out ap-proach to develop the guideline. By usingthe bottom-up approach, the authors gen-erate suitable knowledge modules whichrepresent the evidence found in the litera-ture. By structuring these knowledge mod-ules, documents of the CPG are generated.In combination with the pragmatic’s sec-

tion and in certain cases with the adapta-tion section these documents form thecomplete CPG (Fig. 3).

Alternatively the authors start with thetop-level class HELEN_Guideline and instantiate from this class the necessarydocuments in a top-down approach. Duringthe following step each document must berefined by the use of suitable knowledgemodules. Finally the development teamcompletes the pragmatics section with in-formation about the development processand the life-cycle of the guideline. The middle out approach starts with the mostimportant and ends with the least impor-tant elements. Here the authors instan-tiate classes of HELEN_Documents and HELEN_Knowledge_Module which theyidentify to be important and form the CPGby adding missing parts.

In order to support the authors duringencoding and to provide them with an easyto use tool we customized the pre-config-ured Protégé entry forms for each class. Forcertain slots of these forms (attributes ofthe classes) we used specialized widgets as provided by the web site of Protégé for viewing images or for preventing the author from repetitive mechanic routine orfault prone tasks such as entering the dateand the author for each created or editedinstance [72].

Special attention is given to the custom-ization of the entry forms for the KnowledgeModules. As mentioned earlier we distin-guish between text, graphics and algorithms.

Beside standard text formats such asASCII or HTML, the HELEN tools sup-port a slightly modified version of HTMLcalled HELEN_HTML in order to repre-


419


Fig. 4Protégé form for enteringa text-based knowledgemodule by using the HELEN_HTML syntax

sent narrative text elements.This version ofHTML contains three new tags for linkingto the other HELEN_Knowledge_Moduleswithin a CPG. Consequently we developeda specialized edit widget for HELEN_HTML to prevent the author from manual-ly encoding text in this format (Fig. 4).

Although HTML supports the represen-tation of images we used a specializedknowledge module instead. This decisionwas made due to the fact that images couldbe used within non-html documents. Fur-thermore, as they are knowledge resources,images must directly be accessible and anno-tated with descriptive attributes correspond-ing to other HELEN_Knowledge_Modules.The acquisition form for images is the same

as shown in Figure 4 despite the fact that theeditor for HELEN_HTML is replaced by animage-viewer widget.

The third knowledge module representsan algorithm which can be used to illustratethe constellation of clinical actions and de-cisions on a high level of abstraction. Alter-natively executable algorithms can be mod-eled by specifying sequences and dependen-cies between algorithmic elements on a lower level of abstraction. For editing suchdiagrams we used the built-in diagram wid-get of Protégé. This widget provides an easyto use graphical interface that allows encod-ing of certain parts of the formal guideline representation by means of icons such as diamonds, circles and rectangles as shown in

Figure 5. For several reasons we developeda special slot widget which produces an im-age file from the content of the diagramwidget and stores the corresponding file-name in a specialized slot. These images arebeneficial during the development processfor communication and presentation andare also used by the Guideline Viewer.

Finally we developed an XML-Schemaexport filter called HELEN_XML_Back-end in order to integrate the Guideline Editor within the presented framework.This backend extends Protégé with theability to produce an XML-Schema defini-tion according to the ontology and to storethe encoded CPG as a valid XML-file.

3.2 The Guideline ViewerOne basic use of encoded CPGs is viewingthem via a standard web browser (Fig. 5).Therefore we developed the GuidelineViewer which also assists the systematic ap-praisal of CPGs.This is a Java Servlet whichcan be carried out by a servlet containersuch as Apache-Tomcat [71]. In order to incorporate the Guideline Viewer in dailyroutine, some elementary features besidedisplaying, searching and navigating mustbe provided. Such features address man-agement tasks such as uploading, updatingor deleting the encoded CPG. Thereforethe Guideline Viewer comes with a separatepassword protected entry point which isprovided via a generic URL. This URL canalso be used to access the viewer by differ-ent roles such as physician, nurse, or pa-tient. Depending on the chosen entry pointthe system presents a pre-selected viewwith a list of guidelines and included docu-ments dedicated to the corresponding userrole as encoded within the CPG. By gener-ating such user specific views the systemhas also taken care of adapted documentsand knowledge modules. Therefore theprovided web-pages are either based onadapted documents or modules or, in casethat these do not exist, on original ele-ments. For each displayed document orknowledge module the Guideline Viewer also provides an INFO-Button. By usingthis button additional background informa-tion as well as the underlying original re-source will be displayed.

Skonetzki et al.

420


Fig. 5 Protégé form for entering an algorithm-based knowledge module by means of diamonds, circles and rectangles

In order to contribute to the frameworkthe Guideline Viewer provides a dedicatedentry point additionally to the alreadymentioned synchronization feature. Thisfeature can be used by the hospital infor-mation system and other tools to invokethe viewer for displaying requested Knowl-edge Modules. It will be used by e.g.,the Guideline Execution Engine providingbackground information about certain de-cisions or recommended tasks.Another useof this functionality can be found in trigger-ing the printing of patient handouts orpaper-based prescription forms via an algo-rithm. In return the Guideline Viewer caninvoke the Guideline Execution Engine tostart a displayed algorithm for a certain patient-physician combination.

3.3 The Guideline Execution EngineThe third tool of our framework is dedicat-ed to the execution of algorithms as part ofCPGs. The intended use of this engine is toexecute one or more CPGs for a certain pa-tient by traversing encoded algorithms andcommunicating with physicians, nurses, andpatients about decisions, recommendations,or scheduled actions.

As a server-based tool the Guideline Execution Engine (GEE) is basically re-sponsible for traversing the algorithms,sequencing clinical actions, and evaluatingcontained logic or arithmetic statements.Additionally, administration tasks such asreading the encoded CPG, management ofinformation about users (e.g., username,role, responsibilities), and management ofinformation about the patients (e.g., demo-graphic and ADT-data) are necessary andperformed by the GEE.

Due to the fact that the GEE interpretsflexible programming logic it supports several modes of use which depend on theencoded algorithm. A possible applicationcan be an algorithm for sequencing clinicalactions depending on information aboutthe patient, external trigger events, and user´s choices. Other applications can be reminders, several kinds of decision sup-port, or triggering the Guideline Viewer ascontext sensitive information resource.

Algorithms are represented as directedgraphs assembled from predefined entitieswhich are connected by ‘followed-by’ rela-tions. Each algorithm strictly requires aHELEN_Start_Step and each leaf of the algorithm must be terminated by a HELEN_Diagnose. Such diagnoses can – depending on their use – be understood as ‘intermediate result’, ‘return to the

calling algorithm’, ‘final achievement’, or‘abandoning the execution’. Beside theseessential entities the author can use HELEN_Actions, HELEN_Messages, orHELEN_Decisions to produce the re-quired functionality. An action can be aphysical examination, history taking, labor-atory request, technical examination, thera-py or a prescription. Decisions are used as


421


Fig. 6 Using a standard web browser to access documents within the CPG as provided by the ‘Guideline Viewer’

branching steps where either the user can choose (HELEN_Evaluate_by_users_choice) or where the system automaticallydecides depending on logic statements(HELEN_Evaluate_automatic) which stepto take next. Beside basic Boolean logicstatements we extended the language tocompare variables and constants also with the results of arithmetic expressions or specialized functions accessing certaininformation such as current time or date(HELEN_Binary_Arithmetic_Operator).

Additionally, a specialized set of entitiesis dedicated to control the execution. Theseentities are obtained from procedural pro-gramming languages to generate condition-al, synchronizing or wait statements. Be-sides, several types of cycles such as ‘do-while’, ‘repeat-until’ or cyclic statements are

possible. Another concept obtained fromprocedural programming is the precept util-ization of variables and constants. Con-stants are used amongst others to achieve amaintainable set of thresholds. The system-atic use of variables also provides a flexibleand easily adjustable architecture for ac-cessing information about the patientstored in external databases, hospital infor-mation systems, or other systems as long asthe value is accessible by a database connec-tion (Fig. 7). In case the value is not storedyet the GEE provides the ability to requestthe value just in time when it is needed bysending a predefined question to a statedsingle user or a group of professionals. Forthis purpose we also developed a JAVA-based communication tool available fordesktop systems as well as for PDAs to as-

Skonetzki et al.

422


Fig. 7 Definition of a recommended therapy and calculation of the appropriate dosage

sist the involved health care professionals(Fig. 8). This tool can be understood as akind of e-mail client where the user receivesand answers messages. Therefore the GEEpushes requests, hints and all the other in-formation on a server-based agenda.

The client receives and presents all themessages for which the user takes respon-sibility. Whereas some of these messagesare dedicated for requesting current infor-mation about the patient or performedtasks, used to inform about suitable clinicalactions or to remind about scheduled tasks,others are used to receive an approval toautomatic decisions of the system or to request decisions from the user. In re-sponding to any message the user can alsosuspend, re-enact or abandon the executionof a CPG for a certain patient.

By checking the expiry date of eachmessage at regular intervals the GEEsupervises the correct processing and inter-venes by withdrawing or reassigning a mes-sage if necessary. This is only one of theprincipal tasks to ensure that the guidelinecan be processed in a proper form evenover a longer period of time. Other tasksare necessary to ensure that minor errorswithin the structure of the algorithms orwithin formal statements may not lead toan automatic termination of the execution.So far the GEE is not able to correct com-plex errors such as missing ‘followed-by’ re-lations nor can it detect deadlocks duringexecution. For handling such problems wefocus on preventing errors within the algo-rithms such as the mentioned wrong ormissing ‘followed-by’ relations, incompleteor inconsistent specification of entities aswell as certain well known typical model-ing faults already during authoring of theCPG.

4. Applied MaterialFollowing the objectives of our work we applied the developed framework duringsome projects focusing on implementingCPGs within the Department of Neonatol-ogy. As frequently concluded an effectiveimplementation of CPGs should be basedon multifaceted interventions [73]. There-fore the presented tools here constitute on-ly one supportive element to our strategy.This strategy is based upon an initial selec-tion of an area with urgent problems,high-risk, high-volume, or problem-proneprocesses.After careful collection and eval-uation of relevant data we tried to deter-mine causes. If indicated we established aguideline authoring team with members ofall apparently affected groups. This author-ing group is responsible for local tailoringof guideline recommendations by adapt-ing existing or developing new guidelines.Since the Guideline Editor is just a tool thatassists during encoding guideline recom-mendations the general strategy to identify,synthesize and formulate evidence is basedupon proven concepts from SIGN [61] andDEGAM [62]. To involve a broad range ofaffected users we presented and discussed

major recommendations during regularstaff meetings. These processes resulted inan initial version of the encoded CPG avail-able by means of the GuidelineViewer.Closely connected to formulation of rec-ommendations we also focused on changesregarding organization, workflow, techni-cal equipment, and education to promotethe implementation. In trying to supportthese changes we developed several typesof algorithms executed by the GuidelineExecution Engine. Due to the current defi-nition of the HIS used at the HeidelbergUniversity Medical Center, which is basi-cally designed to assist during administra-tive tasks such as admission, discharge,transfer, and order entry for technical pro-cedures, we could not practice regular dataexchange concerning patient-specific clini-cal parameters [74]. The connection is pres-ently limited to ADT-data and some labor-atory results, but within the bounds of aseparate project we also focus on a strong-er integration by using scheduling, docu-mentation and order entry functionality ofthe HIS.

Within the next two sections we shallpresent some examples of our work focusing especially on the usage of frame-work.

4.1 In-depth InvestigationBased upon the developed framework, weimplemented two CPGs together with theDepartment of Neonatology as part of theproject. The first one is a guideline for themanagement of hyperbilirubinemia in thehealthy newborn as published by theAmerican Academy of Pediatrics [75]. Thisis a typical example of a text-based CPG in-tended for clarifying treatment indications.The text is complemented by a non-execut-able clinical algorithm used for illustratingthe indications of the treatment options.During implementation the adaptationteam decided to use this CPG basically aseducational material for junior physicians.We also included a separate document withmaterial for nurses who initiate the diagno-sis process and carry out several tasks suchas the recommended phototherapy. Con-sidering the needs of experienced physi-cians we developed a single paged summar-

ization of thresholds for the treatment indi-cations as a printable pocket version. Final-ly we completed the implementation withan algorithm used as a simple computer-based training application for junior physi-cians.

In contrast the second CPG (Manage-ment of apnea in pre-term newborns [76,77]) focuses on correct sequencing and tim-ing of clinical actions. The core element is acomplex set of nested algorithms whichcovers several therapy decisions as well asthe timing of the execution. We distinguishbetween five therapy levels ranging fromstandard care such as food adaptation overcaffeine or doxapram medication to severaltypes of mechanical ventilation in emer-gency cases. Narrative text elements areused here as the explanatory resource ofcertain decisions, therapy options, and ad-vice with regard to the accomplishment oftasks.The text resources can also be used togive an introduction to the topic or in cas-es where the physician requests guidanceonly on an advanced state of treatment. Itprovides him with alternative entry pointsto the algorithm. During the execution thesystem basically acts as a reminder forchecking and reconsidering the currenttreatment for a certain patient. With thisCPG we have extensively tested the algo-rithmic features and the linkage with text-based elements.


423


Fig. 8 HELEN PDA client for communicating with the GEE

4.2 In-breadth ExperimentsWe also worked on several other CPGswhich currently are not implemented in alldetail but yet modeled on an advanced lev-el. This work has been done in cooperationwith experts to verify the flexibility of ourapproach and to broaden the range of con-sidered modes of use. One example usedfrom ophthalmology is the diagnosis ofuveitis [78, 79]. With this example wechecked the ability for inserting and con-necting a new knowledge module. This newmodule represents a decision table for aheuristic classification of certain diagnoses.Additionally, we checked the ability to han-dle a multi-professional (ophthalmologistand rheumatologist) as well as a multi-cen-tered treatment team. The last example tomention here is derived from oncology anddeals with the diagnosis and treatment ofsmall-cell lung cancer [80]. The focus herelies on the intended implementation by using algorithms as navigation aids. Fortu-nately, the representation that we devel-oped is able to deal with this kind of infor-mation as well as with the intended use.Slight modifications are necessary to en-able the Guideline Viewer to present theCPG by using a frame-based concept. Thefirst frame contains alternating images ofthe algorithms for navigation and the otherframe contains the selected documents andknowledge modules as hitherto.

5. DiscussionIt is not the intention and it would go beyond the means of the framework pre-sented here to supersede the variety of pro-posed and implemented approaches forformal representation of clinical guidelines.We rather directly focus on managementand implementation specific topics to bringCPGs into clinical practice. As reported bymany scientific groups the complexity ofthe authoring process has proven to be areal bottleneck (e.g. [33]). Therefore it isworthwhile to share such computerizedCPGs among different institutions (e.g.,[14]). We have taken up both topics and developed a framework for authoring andimplementing CPGs that is as easy and flex-

ible as possible to use. During the implemen-tation of the CPGs within the Departmentof Neonatology a cooperative rapid proto-typing process involving medical specialists,future users, and knowledge engineers hasarisen and proven itself.

Especially the interconnection betweenthe authoring and execution environmentencourages the authors to debate. Besides, ithelps to verify unclear or ambiguous topicswith test cases, and assists in testing the dif-ferent representation modules for best suit-able descriptions with respect to the specialcharacteristics of each topic. Fortunatelythis often leads to manifold documents withdouble tracked descriptions (both text andalgorithm) to achieve an optimal accuracyand to satisfy the range of needs of furtherusers. The usual procedure was based uponan initial encoding of the sources as skeletalCPG as provided by national or interna-tional medical societies. After that the CPGwas completed and adjusted with respect tospecial requirements of users, environment,and functional organization by using the ad-aptation feature for each encoded knowl-edge module.Wherever possible we derivedthe adapted recommendations from theunderlying guideline resources. In case thatthis wasn’t possible we introduced newmodules and documents. It is still under ex-amination to what extent the adaptationfeature can be useful by sharing the CPGwith other institutions.

Last but not least we have to concludethat additional to the already mentionedfactors for successful implementations ofCPGs we assumed that a confinement to alimited and carefully selected number ofurgent areas per department is also a sub-stantial factor for motivation. It is neces-sary to consider not only the often volun-tary and extensive work done during devel-opment, adaptation and implementationbut also the persistent amount of work forthe regular necessary updating of the CPGto keep track with current scientific re-search as well as with changed basic condi-tions of providing health care.

5.1 Future Work After developing the basic fundamentalswe will focus on enhancing the capabilities

of our framework regarding more flexibil-ity and usefulness. The first problem is howto strengthen and prove the evidence foun-dation of the guideline. Therefore we con-sider the presentation of additional infor-mation and annotations about the litera-ture used during the development process.Furthermore, intermediate results such asgenerated evidence tables and informationabout the process of how the evidence wasderived and combined to form a certainstatement within a knowledge modulecould be represented.

By reasoning about this information weattempt an automatic detection of weak-nesses within the use of the evidence andwhether the evidence foundation is alsopreserved during adaptation of a certainmodule. In combination with another fieldof our ongoing research focusing on attach-ing rules for detecting typical errors andmistakes made during modeling as well asdetecting syntactic and marginal semanticinconsistencies within the guideline thiscould lead to an advanced detection of con-tradictions or weak evidence rationale.Thismight help the authors to keep focused onseeking better evidence. This purposemakes it necessary to include standardizedmedical terminology and conceptual mod-els (see also [81, 82]). Such terminologiesdo not only offer a more sophisticated deci-sion support with explicit abstraction ofdata and actions as well as standardized pa-tient states and clinical diagnosis but are al-so essential for comprehensive access toelectronic patient records, and for our in-tention of checking advanced semanticrules.

Besides, we would like to spend some effort to extend our framework with a sep-arate tool for an automatic eligibility deter-mination of a CPG for certain patientspreferably based upon the yet limited clini-cal data available within our HIS. There-fore, we have to clarify which information is essentially needed for an automaticallygenerated recommendation to start orabandon a CPG and above all, how to ob-tain it preferably without additional effortfor the health care professionals.A possiblesolution may lie in the use of methodolo-gies from case-based reasoning. Other ex-tensions concern the automatic export and

Skonetzki et al.

424


import feature to other guideline represen-tation systems.

Besides the technical aspects alreadymentioned, we consider systematically collecting information on the benefits andefforts by using our CPGs in clinical prac-tice. Last but not least we are focusing onimproving the adaptation feature in coop-eration with other medical institutions. Weconsider the introduction of an automati-cally generated more detailed documenta-tion about necessary adjustments, recurringchanges, or extensions to certain elementsof the CPG during the adaptation process.Such information might be helpful to opti-mize CPGs because it offers the initial authoring team structured feedback aboutimprecise, ambiguous, or infeasible recom-mendations. By supporting these crucialtasks the adaptation feature hopefully be-comes an accepted and useful part of implementing CPGs.

5.2 ConclusionsWe have developed a working set of toolsfor authoring, browsing and executing clin-ical guidelines. This framework provides anopen and flexible architecture which caneasily be extended by adding new knowl-edge modules to the ontology and by im-plementing corresponding classes with therequired additional functionality to the existing tools.

We have shown the benefits of combin-ing different knowledge representationssuch as narrative text, graphic illustrationsand algorithms. Finally we have introduceda possible approach for an explicit adapta-tion process of documented and auditableCPGs in order to provide institution-specif-ic recommendations and to support sharingwith other medical institutions.

In the end we would like to point to thefact that all tools were developed under theGNU public licence so that we do not onlypermit the sharing of formally representedguidelines, but also make use of theseguidelines as a shared resource, as otherprojects did before.

AcknowledgmentsThe authors would like to thank Dr. Becker forhis contribution in developing CPGs for the Department of Ophthalmology of the UniversityMedical Center Heidelberg. Furthermore we aremost grateful to Jan-Peter Klippel and OliverMerker for their commitment during the devel-opment of the HELEN tools.

References1. Grimshaw JM, Russell IT. Effect of clinical

guidelines on medical practice: a systematic review of rigorous evaluations. Lancet 1993;342 (8883): 1317-22.

2. Hibble A, Kanka D, Pencheon D, et al. Guide-lines in general practice: the new Tower of Babel? BMJ 1998; 317 (7162): 862-3.

3. Woolf SH, Grol R, Hutchinson A, et al. Clinicalguidelines: potential benefits, limitations, andharms of clinical guidelines. BMJ 1999; 318(7182): 527-30.

4. Cabana MD, Rand CS, Powe NR, et al. Whydon’t physicians follow clinical practice guide-lines? A framework for improvement. JAMA1999; 282 (15): 1458-65.

5. Dahlberg K. Reasons physicians do not followclinical practice guidelines. JAMA 2000; 283(13): 1686.

6. Davis DA,Taylor-Vaisey A. Translating guide-lines into practice. A systematic review of theoretic concepts, practical experience and research evidence in the adoption of clinicalpractice guidelines. CMAJ 1997; 157 (4): 408-16.

7. Haynes B, Haines A. Barriers and bridges to evidence based clinical practice. BMJ 1998;317 (7153): 273-6.

8. Grimshaw J, Russell I. Achieving health gainthrough clinical guidelines. I: Developing scien-tifically valid guidelines. Qual Health Care,1993. 2 (4): 243-8.

9. Grimshaw JM, Russell IT, Achieving healthgain through clinical guidelines II: Ensuringguidelines change medical practice. QualHealth Care 1994; 3 (1): 45-52.

10. Keffer JH. Guidelines and algorithms: percep-tions of why and when they are successful andhow to improve them. Clin Chem 2001; 47 (8):1563-72.

11. Duff L, Casey A. Implementing clinical guide-lines: how can informatics help? J Am Med Inform Assoc 1998; 5 (3): 225-6.

12. Frankel HL, FitzPatrick MK, Gaskell S, et al.Strategies to improve compliance with evi-dence-based clinical management guidelines.J Am Coll Surg 1999; 189 (6): 533-8.

13. Patel VL,Arocha JF, Diermeier M, et al. Cogni-tive psychological studies of representation anduse of clinical practice guidelines. Int J Med Inf2001; 63 (3): 147-67.

14. Boxwala AA,Tu S, Peleg M, et al.Toward a rep-resentation format for sharable clinical guide-lines. J Biomed Inform 2001; 34 (3): 157-69.

15. Corb GJ, Liaw Y, Brandt CA, et al. An object-oriented framework for the development ofcomputer-based guideline implementations.Methods Inf Med 1999; 38 (3): 148-53.

16. Silagy C, Weller D, Lapsley H, et al. The effec-tiveness of local adaptation of nationally pro-duced clinical practice guidelines. Fam Pract2002; 19 (3): 223-30.

17. Tierney WM, Overhage JM, Takesue BY,et al. Computerizing guidelines to improve careand patient outcomes: the example of heartfailure. JAMIA 1995; 2 (5): 316-22.

18. Smith WR. Evidence for the effectiveness oftechniques To change physician behavior.Chest 2000; 118 (2 Suppl): 8S-17S.

19. Greco PJ, Eisenberg JM. Changing physicians’practices. N Engl J Med 1993; 329 (17): 1271-3.

20. Johnston ME, Langton KB, Haynes RB, et al.Effects of computer-based clinical decisionsupport systems on clinician performance andpatient outcome. A critical appraisal of re-search. Ann Intern Med 1994; 120 (2): 135-42.

21. Shea S, DuMouchel W, Bahamonde L. A meta-analysis of 16 randomized controlled trials toevaluate computer-based clinical remindersystems for preventive care in the ambulatorysetting. JAMIA 1996; 3 (6): 399-409.

22. Shiffman RN, Liaw Y, Brandt CA, et al. Com-puter-based guideline implementation systems:a systematic review of functionality and effec-tiveness. JAMIA 1999; 6 (2): 104-14.

23. Greenes RA, Peleg M, Boxwala A, et al. Shar-able computer-based clinical practice guide-lines: rationale, obstacles, approaches, and pros-pects. Medinfo 2001; 10 (Pt 1): 201-5.

24. Lukoff JY, Dolin RH, McKinley CS, et al. Vali-dation of an XML-based process to automati-cally web-enable clinical practice guidelines:experience with the smoking cessation guide-line. Proc AMIA Symp 1999; 311-4.

25. Hoelzer S, Schweiger R, Dudeck J, Representa-tion of practice guidelines with XML – model-ing with XML schema. Methods Inf Med 2002;41 (4): 305-12.

26. Dufour JC, Fieschi M, Fieschi D, et al. A Plat-form to Develop and to Improve Effectivenessof Online Computable Guidelines. Stud HealthTechnol Inform 2003; 95: 445-51.

27. Agrawal A, Shiffman RN. Evaluation of guide-line quality using GEM-Q. Medinfo 2001; 10(Pt 2): 1097-101.

28. Agrawal A, Shiffman RN. Using GEM-encod-ed guidelines to generate medical logic mod-ules. Proc AMIA Symp 2001; 7-11.

29. Gershkovich P, Shiffman RN. An implementa-tion framework for GEM encoded guidelines.Proc AMIA Symp 2001: 204-8.

30. Terenziani P, Molino G, Torchio M. A modularapproach for representing and executing clini-cal guidelines. Artif Intell Med 2001; 23 (3):249-76.

31. Lobach DF, Gadd CS, Hales JW. Structuringclinical practice guidelines in a relational data-base model for decision support on the Inter-net. Proc AMIA Annu Fall Symp 1997; 158-62.

32. Ohno-Machado L, Gennari JH, Murphy SN,et al. The guideline interchange format: a mod-el for representing guidelines. J Am Med In-form Assoc 1998; 5 (4): 357-72.

33. Tu SW, Musen MA. A flexible approach toguideline modeling. Proc AMIA Symp 1999;420-4.


425


34. Boxwala AA, Greenes RA, Deibel SR. Archi-tecture for a multipurpose guideline executionengine. Proc AMIA Symp 1999; 701-5.

35. Greenes RA, Boxwala A, Sloan WN, et al. Aframework and tools for authoring, editing,documenting, sharing, searching, navigating,and executing computer-based clinical guide-lines. Proc AMIA Symp 1999; 261-5.

36. Peleg M, Boxwala AA, Ogunyemi O, et al.GLIF3: the evolution of a guideline representa-tion format. Proc AMIA Symp 2000; 645-9.

37. Peleg M, Ogunyemi O, Tu S, et al. Using fea-tures of Arden Syntax with object-orientedmedical data models for guideline modeling.Proc AMIA Symp 2001; 523-7.

38. Zielstorff RD, Teich JM, Paterno MD, et al.P-CAPE: a high-level tool for entering and pro-cessing clinical practice guidelines. PartnersComputerized Algorithm and Editor. ProcAMIA Symp 1998; 478-82.

39. Kuperman GJ, Teich JM, Bates DW, et al. De-tecting alerts, notifying the physician, and offer-ing action items: a comprehensive alertingsystem. Proc AMIA Annu Fall Symp 1996;704-8.

40. Purves IN, Sugden B, Booth N, et al. ThePRODIGY project – the iterative developmentof the release one model. Proc AMIA Symp1999; 359-63.

41. Johnson PD, Tu S, Booth N, et al. Using sce-narios in chronic disease management guide-lines for primary care. Proc AMIA Symp 2000;389-93.

42. Shahar Y, Miksch S, Johnson P. The Asgaardproject: a task-specific framework for the appli-cation and critiquing of time-oriented clinicalguidelines. Artif Intell Med 1998; 14 (1-2):29-51.

43. Barahona P,Azevedo F,Veloso M, et al.A com-puterised guideline for the management of diabetes. Stud Health Technol Inform 2000; 77:600-4.

44. Fox J, Johns N, Rahmanzadeh A. Disseminatingmedical knowledge: the PROforma approach.Artif Intell Med 1998; 14 (1-2): 157-81.

45. Musen MA, Tu SW, Das AK, et al. EON: a component-based approach to automation ofprotocol-directed therapy. J Am Med InformAssoc 1996; 3 (6): 367-88.

46. Tu SW, Musen MA. From guideline modelingto guideline execution: defining guideline-based decision-support services. Proc AMIASymp 2000; 863-7.

47. Elkin PL, Peleg M, Lacson R, et al. Toward thestandardization of electronic guidelines. MDComput 2000; 17 (6): 39-44.

48. Peleg M,Tu S, Bury J, et al. Comparing comput-er-interpretable guideline models: a case-studyapproach. JAMIA 2003; 10 (1): 52-68.

49. AeZQ, Checkliste zur Bewertung der Metho-dischen Qualität von Leitlinien. Dt Ärzteblatt2000; 97 (17): A1170-1172.

50. Gruber TR. A Translation Approach to Port-able Ontology Specification. Knowledge Ac-quisition 1993; (5): 199-220.

51. Musen MA. Dimensions of knowledge sharingand reuse. Comput Biomed Res 1992; 25 (5):435-67.

52. Angele J, Decker S, Studer R. Domain and TaskModeling in MIKE. In: Domain Knowledge for Interactive Systems Design, Proceedings of IFIP WG 8.1/13.2 Joint Working Conference.1996. Genevam Nay: Chapman & Hall.

53. Boehm BW.A Spiral Model of Software Devel-opment and Enhancement. ACM SIGSOFT1988; 4 (11): 21-42.

54. AGREE, Guideline development in Europe.An international comparison. Int J Technol Assess Health Care 2000; 16 (4): 1039-49.

55. Burgers J, Grol R, Klazinga N, et al. Interna-tional comparison of 19 clinical guideline pro-grams – a survey of the AGREE Collaboration.Z Arztl Fortbild Qualitatssich 2003; 97 (1): 81-8.

56. Cluzeau FA, Littlejohns P, Grimshaw JM, et al.Development and application of a genericmethodology to assess the quality of clinicalguidelines. Int J Qual Health Care 1999; 11 (1):21-8.

57. Bernstam E, Ash N, Peleg M, et al. Guidelineclassification to assist modeling, authoring,implementation and retrieval. Proc AMIASymp 2000: 66-70.

58. Helou A, Perleth M, Bitzer EM, et al. Method-ological quality of clinical practice guidelines inGermany: results of a systemic assessment ofguidelines presented on the Internet. Z ÄrztlFortbild Qualitätssich 1998; 92 (6): 421-8.

59. Shiffman RN, Karras BT, Agrawal A, et al.GEM: a proposal for a more comprehensiveguideline document model using XML. JAMIA2000; 7 (5): 488-98.

60. Mayerhoff L. Optimal embedding of a knowl-edge based system in an existing clinical envi-ronment. Specification of requirements and design based upon the CommonKADS meth-odology. Institute for Medical Biometry and Informatics, Department of Medical Informat-ics, University of Heidelberg: Heidelberg 2000;151.

61. SIGN, SIGN 50: A guideline developers’ hand-book. 1992, Scottish Intercollegiate GuidelinesNetwork, http://www.sign.ac.uk/guidelines/full-text/50/index.html Last Access: January 10,2004.

62. Gerlach FM, Beyer M, Berndt M, et al. The DEGAM-concept – development, dissemina-tion, implementation and evaluation of guide-lines for general practice. Z Ärztl FortbildQualitätssich 1999; 93 (2): 111-20.

63. de Clercq PA, Blom JA, Hasman A, et al. Anontology-driven approach for the acquisitionand execution of clinical guidelines. StudHealth Technol Inform 2000; 77: 714-9.

64. Uschold M, Grunninger M. Ontologies: Princi-ples, Methods and Applications. KnowledgeEngineering Review 1996; 1(2): 93-155.

65. NGC, NGC Classification for CPGs. 2002,http://www.guideline.gov/FRAMESETS/static_fs.asp?view=about.overview.classify,Last Access: December 15, 2003.

66. Noy FN, McGuiness DL. Ontology Develop-ment 101: A Guide to Creating Your First Ontology. 2001, Stanford Medical Informatics:Stanford; 25.

67. Gennari JH, Musen MA, Fergerson RW, et al.The Evolution of Protégé: An Environment for

Knowledge-Based Systems Development. 2002.Stanford Medical Informatics: Stanford; 32.

68. Ollenschlager G, Helou A, Kostovic-Cilic L,et al. Checklist for methodological quality ofguidelines.A contribution to quality promotionof medical guidelines. Z Ärztl Fortbild Quali-tätssich 1998; 92 (3): 191-4.

69. Fallside DC. XML Schema Part 0: Primer. 2001,W3C Recommendation.

70. Sun, JAVA Servlet Technology. 2003, http://java.sun.com/products/servlet/index.html. LastAccess: 21.01.2004.

71. Jakarta, The Apache Jakarta Project. 2003,http://jakarta.apache.org/tomcat/. Last Access:January 21, 2004.

72. Protege, Protégé Plug-ins Library. 2002,http://protege.stanford.edu/plugins.html. LastAccess: January 21, 2004.

73. Grol R, Grimshaw J. From best evidence to bestpractice: effective implementation of change inpatients’ care. Lancet 2003; 362 (9391): 1225-30.

74. Gell G, Schmucker P, Pedevilla M, et al. SAPand partners: IS-H and IS-H* MED. MethodsInf Med 2003; 42 (1): 16-24.

75. AAP, Practice parameter: management of hy-perbilirubinemia in the healthy term newborn.American Academy of Pediatrics. ProvisionalCommittee for Quality Improvement and Sub-committee on Hyperbilirubinemia. Pediatrics1994; 94 (4 Pt 1): 558-65.

76. Bhatia J. Current options in the management of apnea of prematurity. Clin Pediatr (Phila),2000; 39 (6): 327-36.

77. Emborn C, Humphrey JA. Apnea of prematur-ity. 1999, National Association of NeonatalNurses – Professional Association.

78. Biswas J. Practical concepts in the managementof uveitis. Indian J Ophthalmol 1993; 41 (3):133-41.

79. Becker MD, Rosenbaum JT. Essential labora-tory tests in uveitis. Dev Ophthalmol 1999; 31:92-108.

80. Becker H, Drings P, Havemann K, et al. Treat-ment of small-cell lung cancer. In: Quality as-surance in oncology. Diagnosis and therapy malignany diseaes, I.f.S.i.d.O. (ISTO), Editor:Deutsche Krebsgesellschaft e.V.: Muenchen1999; 218-229.

81. Beeler GWJ. Taking HL7 to the next level. MDComput 1999; 2 (16): 21-4.

82. Cornet R, Abu-Hanna, A. Usability of expres-sive description logics – a case study in UMLS.Proc AMIA Symp 2002; 180-4.

Correspondence to:Dipl.-Inform. Med. S. SkonetzkiInstitute for Medical Biometry and InformaticsDepartment of Medical InformaticsUniversity of HeidelbergIm Neuenheimer Feld 40069120 HeidelbergGermanyE-mail: [email protected]

Skonetzki et al.

426
