solving safety implications in a case based decision-support system in medicine

8/9/2019 Solving Safety Implications in a Case Based Decision-Support System in Medicine

1/81

Solving Safety Implications in a Case Based Decision-

Support System in Medicine

Isabelle Bichindaritz

University of Washington, Tacoma, Institute of Technology, 1900 Commerce Street, Box#358426, Tacoma, WA 98402-3100, USA

[email protected]

Abstract.Case-based reasoning systems applied to safety-critical environmentsjustify specific measures to ensure that the assistance provided is not dangerousto human life. Taking example on a case-based reasoning system developed formedical decision-support in stem cell post-transplant long-term follow-up, thisarticle stresses the importance to differentiate between reliability and safety,and how case-based reasoning steps can be refined to ensure system safety. Aknowledge-based approach has enabled to model what types of information aresafety-critical, and to adapt the case-based decision-support advice using intro-spective reasoning.

1 Introduction

CARE-PARTNER is a case-based computerized decision-support system on theWorld-Wide Web (WWW). It is applied to the long-term follow-up (LTFU) of pa-tients having undergone a stem-cell transplant(SCT) at the Fred Hutchinson CancerResearch Center (FHCRC) in Seattle, after their return in their home community [7].Home care providers use CARE-PARTNER to place contacts with LTFU on theInternet, and receive from the system decision-support advice in a timely manner forthe care of post-transplant patients. An essential characteristic of CARE-PARTNERis that it proposes to implement evidence-based medical practice [4] by applyingclinical guidelines developed by FHCRC. Thus CARE-PARTNER knowledge-basecontains representation of clinical guidelines, protocols, clinical pathways and cases,and resorts to a multimodal reasoning framework for the cooperation of case-basedreasoning (CBR) and rule-based reasoning [5].

Providing decision-support advice in a medical domain is not without risk. As a

matter of fact, CARE-PARTNER is a safety critical system, since the medical adviceit provides can have serious consequences on a patients health. The safety criticalityof the system has been the major issue to overcome in clinical practice. Based uponthe study of similar medical systems, a safety insurance plan has been designed forCARE-PARTNER. Important components of this plan are a procedural level plan, asoftware engineering level plan, and a knowledge level plan involving introspectivereasoning. The increasing reliability of the system, shown in the evaluation results,does not prevent the risk that the system may be unsafe because it is not possible toascertain that it will not make a mistake [6]. This general acknowledgement isparticularly true in case-based reasoning, where the notion of convergence of the
mailto:[email protected]:[email protected]


2/81

ticularly true in case-based reasoning, where the notion of convergence of the systemis not pertinent since the improvement in competence comes from local properties those of the cases close to a new case to solve.

This article presents in sequence: an evaluation of CARE-PARTNER, showingwhy safety aspects need to be taken into account; a review of the set of problems en-countered in safety critical systems in medicine; the process of planning and imple-menting a safety critical system, taking CARE-PARTNER as an example; a conclu-sion.

Table 1. CARE-PARTNER CDSS Evaluation Form Inter-Rater Agreement and Summary

Ratings for Two Raters over Three Patients

Applicable Cases Concordant Cases

Number

PercentAgreement

Rating

Kappacoefficient

ofagreement

Number

Fails tomeet

standards AdequateMeets allstandards

Labs 57 94.7 .71 54 3.7% 3.7% 92.6%

Procedures 70 95.7 .83 67 8.9% 3.0% 88.1%

Diagnosis 79 86.1 .74 68 16.2% 13.2% 70.6%

Treatment 77 92.2 .81 71 9.9% 11.3% 78.8%

Pathways 53 88.6 .71 47 8.5% 8.5% 83.0%

OverallAppreciation

178 91.6 .77 163 5.5% 12.3% 82.2%

2 CARE-PARTNER Evaluation

A sample evaluation of CARE-PARTNER decision-support performance is providedin Table 1. It shows the rating of the system by two independent expert clinicians ac-cording to the criteria Fails to meet standards / Adequate / Meets all standards. On163 different clinical situations or cases, corresponding to contacts between the sys-tem and a clinician about three patients, the system was rated 82.2% as Meets allstandards, and 12.3% as Adequate, for a total of 94.5% of results judged clinicallyacceptable. Table 1 also shows that the advice provided by the system covers most ofthe clinicians tasks: labs and procedures results interpretation, diagnosis, treatment,and pathways information retrieval.

Another part of the evaluation dealt with measuring the progress of the systemwhen solving new contact cases. As noted earlier, case-based reasoning gives the sys-tem the ability to learn. This important characteristic of the system has been evaluatedon three patients complete charts (see Table 2). We see that the performance of thesystem has significantly improved between patient 1 and 3, to reach 98.6% of satisfy-ing results.

The good evaluation results of the system at 98.6% adequate indicate that CARE-PARTNERs knowledge-base has reached an excellent state of completeness. Be-


3/81

cause learning is a constant in a medical environment, where atypical cases continueto occur, and because medical recommendations are frequently updated, the learningability of the system permits the knowledge-base to evolve and keep up-to-date. Nev-ertheless, the use of CARE-PARTNER in clinical practice would not be possiblewithout a 100% rate of at least adequate results over a much larger set of cases. Thisis a reachable goal on hundreds or thousands of contact cases, but still would not en-sure that the system would be safety critical. Even if the rate of error occurrence is ex-tremely low, it is not tolerable in a system in clinical practice. The example andanalysis of Therac-25 system in the next section explains why reliability is notenough to ensure safety.

Table 2.Improvement in CARE-PARTNER CDSS Decisions over TimePatient Fails to

meet standardsAdequate Meets all

standards

Labs 1 14.3% 14.3% 71.4%

2 0.0% 0.0% 100.0%

3 0.0% 0.0% 100.0%

Procedures 35.7% 7.1% 57.1%

2 5.6% 0.0% 94.4%

3 0.0% 2.9% 97.1%

Diagnosis 1 30.0% 25.0% 45.0%

2 12.5% 12.5% 75.0%

3 8.3% 4.2% 87.5%

Pathways 1 10.5% 0.0% 89.5%

2 6.7% 6.7% 86.7%

3 7.7% 23.1% 69.2%Overall Appre-

ciation1 6.4% 27.7% 66.0%

2 11.1% 2.2% 86.7%

3 1.4% 8.5% 90.1%

3 Safety-Critical Systems in Medicine

Medical applications of software systems have generated considerable attention in themedical and software development communities, in particular for their potential dan-

ger. Even if in comparison with the volume of medical software, the number ofsafety-critical accidents has been quite low, a few such accidents have raised the levelof quality required for medical software.

Probably the most famous case of medical software accident is the Therac-25 [6].The Therac-25 is a computer-controlled radiation therapy machine that massivelyoverdosed six patients, leading to several deaths, which has been described as theworst accident in the 35-year history of medical accelerators.

One of the main lessons learnt from Therac-25 is that it is necessary to put inplace safeguards against potential software errors. Even if, like with Therac-25, the


4/81

medical device or software is used safely in 100,000 cases (100% safety rate), therestill can happen some hazards that need to be prevented.

Extensive analysis has been performed of the causes of the malfunction of this sys-tem, and has lead to defining a set of guidelines for safety-critical system.

The set of causal factors identified in Therac-25 is the following[6]:1. Overconfidence in Software: both users, and software developers are over-

confident in software, and less confident in hardware.2. Confusing Reliability with Safety: Therac-25 was extremely reliable,

working almost 100,000 times with no fault, but still it was not safe.3. Lack of Defensive Design: Therac-25 did not contain error-detection and er-

ror-handling features; audit trails were limited.4. Failure to Eliminate Root Causes: Like all software, Therac-25 had soft-

ware bugs. But focusing on particular software design errors is not how tomake a system safe, because any software could behave in an unexpectedfashion given an abnormal set of conditions. If software errors cannot beeliminated, the system should be built with a control mechanism to safeguardagainst its own errors.

5. Complacency: People have a tendency to trust medical technology becausethey think that the developers have safety guards in place. When an accidenthappens, they then realize that safety is not ensured with medical technol-ogy.

6. Unrealistic Risk Assessment: As for Therac-25, risk assessment methodsquantify factors often independently of each other. When an accident didhappen, a risk analysis showed, based on probabilistic calculations that itcould not be because of Therac-25, but related to an unknown factor, such ashuman failure. The inquiry did confirm that it came from Therac-25 failure.

7. Inadequate Investigation or Follow-up on Accident Reports: Everysafety-critical systems should have audit trails and incident analysis proce-dures to apply when any hint of a problem is found that might lead to an ac-cident.

8. Inadequate Software Engineering Practices: Therac-25 development didnot meet the highest standards in software development, such as softwarespecifications and documentations, designs kept simple, dangerous codingpractices avoided, audit trails to detect errors, careful design of computerdisplays. Software must be designed to be testable, including unit testing,software testing, and system integrated testing.

9. Software Reuse: reusing software or off-the-shelf software, does not guar-antee safety. Same testing and safeguard procedures need to be in place forreused components.

10. Safe versus Friendly User Interfaces: Therac-25 made the machine as easyas possible for the users, but eliminated at the same time important controlsand validation mechanisms.

11. User and Government Oversight and Standards: Once starting to investi-gate the Therac-25, the FDA put in place a reporting system, procedures andguidelines for software. These provide an important lesson for users in theindustry.

Leveson and Turnwe [6] note that if this set of factors is impressive, it also hints tothe fact that not a single factor could have prevented the accidents: it was a conjunc-


5/81

tion of these software development practices that caused different types of failure.Lessons learnt on safety-critical systems in medicine led to the development of asafety insurance plan for CARE-PARTNER.

4 CARE-PARTNER Safety Insurance Plan

CARE-PARTNERs role is to provide decision-support assistance to home-care pro-viders and to facilitate communication between LTFU specialists and home-care pro-viders. Caring for a transplanted patient may be quite cumbersome for the home-careprovider. Patients are immuno-compromised, and the fight between the graft and thehost may cause a dangerous immune response. Moreover, patients take time develop-

ing their new immune system, and during the first months and even years are verysensitive to infections. Some symptoms specific of post-transplant diseases may bemistaken for other classical diseases, and the severity of a common disease such asbronchitis may be rapidly fatal in a matter of a day or two - to a transplant patient.

Fig. 1.Contact data entry uses a controlled medical vocabulary

Thus this decision-support system is a safety-critical system in medicine. Evalua-tion results from tables 1 and 2 show that the validity of CARE-PARTNER advice isno perfect. From what we have learnt with Therac-25, it does not in fact make muchdifference if results are 98% satisfying, or 99%, or even 100%. Reliability of the sys-tem will not ensure safety. Following we designed a safety insurance plan for CARE-


6/81

PARTNER to safe-proof it against any accident. The focus shifted during the testingphase to building safeguards into the system, instead of spending equivalent timereaching a 100% level of reliability in clinical decision-support advice, since thislevel would anyway be reached after some time in this kind of case-based learningsystem.

CARE-PARTNER safety insurance plan takes into account several types of fac-tors: procedural, software engineering, and knowledge level.

Fig. 2.Planning actions hierarchy

Procedural Level

It is well explained to medical users that the decision-support system should not beused in case of life-threatening situation. A paging system is available through LTFUphone answering system. Nevertheless, the system must detect such situations in casethe home care provider has misjudged the criticality of a patient condition, which isexplained in the knowledge level plan. The system detects such situations very pre-cisely because in particular it uses a controlled vocabulary defined in its ontology (seeFig. 1). Another procedure in place is that patients need to agree to be treated in partby the system by filling in an informed consent form explaining well the system re-sults and consequences.


7/81

Software Engineering Level

CARE-PARTNER follows recommendations for safety critical systems listed above.Some specific aspects are that a comprehensive audit trail is in place, monitoringevery action from the user, mostly those generating database access. A fine-grainedaudit trail logs every access to the system, identifying the type of access, the subjectand object of the access, and the role of the user accessing the system. This level ofprecision in the audit trail is required by current legislation about patient identifiablemedical information.

Software development has been well documented, and followed Unified ModelingLanguage (UML) method for object-oriented development.

Another aspect is testing. Testing has been performed first on selected patientcharts to compare paper-based staff recommendations and system recommendations.

Then took place a pilot testing, following which a clinical trial was authorized.

Knowledge Level

Building safeguards into the system takes place mostly at the knowledge level. AfterCBR reuse step an adaptation - and before presenting the adapted case to the user,CARE-PARTNER evaluates the case recommendations for safety by introspectivereasoning.

The different dimensions of the system recommendations present different risk lev-els: Interpretation for each laboratory test or procedure result: these are not

monitored for safety, but certainly they are monitored for errors. It is common inmedicine for physician to review labs and procedure results interpretations giventhat they depend upon the reference population.

List of differential diagnoses, ranked by likelihood: this list may be safety criti-cal. For instance, if some of the signs and symptoms that should be covered by adiagnosis are not, this may be dangerous. Thus the system monitors that all criti-cal signs and symptoms have been covered by at least a diagnosis, otherwise aLTFU staff is paged to handle the case. Moreover some signs and symptoms aretagged as safety critical, such as Temperature level=Very-elevated, correspond-ing to a high fever.

List of steps of laboratory tests and/or procedures for diagnosis assessment:these are not considered as safety critical because after review by the experts,they were not assessed as life-threatening.

List of steps of planning actions for treatment : Planning actions (see Fig 2) maybe safety critical. Placing a patient on a drug, or removing a drug may cause ahealth risk for a patient. So certain planning actions are tagged as safety critical.

List of pertinent documents hyperlinked to the previous elements, such asguidelines, treatment protocols, or textbook excerpts: these are linked to pre-vious elements for documentation and explanation purposes, and thus need notbe monitored for safety.

In summary, certain knowledge elements are tagged as critical in the knowledge-base, such as certain signs and symptoms (for exampleBleedingNOS, or Temperatu-reElevated when the elevation is high), and certain planning actions (for exampleStopMedication, or StartMedication for certain types of medication marked as dan-gerous). When any of this safety critical knowledge shows in an adapted case, a


8/81

LTFU staff is paged to review the system advice and take further charge of the case(see Fig. 3) immediately.

In any case, the system recommendations provided to the hometown care providerare also transmitted to LTFU, and they can review the advice given, modify it, and/orcontact the hometown care provider. Having all information about patient contacts inelectronic format is already an advance in LTFU clinical work, and CARE-PARTNER contact management system itself is very valuable over paper-based con-tact management.

No

Safety critical Sign

or Symptom?

Safety critical

Planning Action?

No

Page LTFU

specialist

Provide non safety c ritical

recommendations

End of

decision-support

advice for that

contact

Beginning of

decision-support

advice self-critique,after case solution is

found by the system

Yes

Yes

Fig. 3System introspective reasoning to filter out safety critical clinical situations

In addition, the decision-support system speeds the response time from LTFU,

making it instantaneous in most cases, all those that are not safety critical. By com-parison, response speed for phone contact takes several days, because the nurse tak-ing phone calls first reports the case to the clinicians during regularly scheduled meet-ings, then gets back with an answer. Even for safety critical cases, the systemprovides non-safety critical answers right away to the provider, and pages LTFU cli-nician for additional recommendations. Thus the system advocates a close coopera-tion between LTFU nurses and clinicians, and home care providers in the best interestof the patient, and still speeds up considerably the response time of LTFU.


9/81

5 Conclusion

The main issue that faced CARE-PARTNER to go into clinical practice was its safetycriticality. By implementing a procedural, software engineering, and knowledge levelsafety insurance plan, and particularly through introspective reasoning, CARE-PARTNER has built-in safeguards to prevent errors that may lead to life-threateningaccidents. Spending time incorporating these safeguards into the system is more valu-able than spending the same time improving the reliability of the system to perfection,because reliability does not ensure safety.

Case-based reasoning systems generally do not deal with the safety criticality as-pect of the recommendations that they provide because of one of the following rea-sons:

Either they assume that their users will take their recommendations aspure advice, and will make their own judgment about whether or not tofollow the system recommendations. Taking this argument for granted isdangerous in a safety critical environment (see reasons 5 and 1 in Levesonand Turnwe [6], Section 3 above). It is very likely that such an expecta-tion will be defeated one day, and may lead, in addition to irremediablepersonal injury, to discrediting the system and like systems. Moreover, itlimits the use of the system to experts, while the main benefit of CARE-PARTNER is to provide case-based decision-support to non experts in thedomain.

Or they provide cases to users mostly to support their own decision-making process, thus leaving the reuse step to the user.

It is most of the time possible to find a non safety critical way of using a system

that may be potentially safety critical. For instance, CARE-PARTNER has beenextended into an intelligent tutoring system (ITS) as a teaching assistant [3].Nevertheless, it is often very valuable to provide active and timely decision-

support advice in domains that may be safety critical. CARE-PARTNER for in-stance provides improvement of patients care along several dimensions. It is anelectronic contact management system that records detailed contact information,with their complete clinical context, for care follow-up as well as for clinical re-search purposes, and is thus quite valuable as a medical knowledge managementtool. It also provides just in time, evidence-based decision support recommenda-tions for physicians not specialized in the stem-cell transplant domain. It fostersand monitors the application of evidence-based medicine, which is valued as ad-vancement in medical scientific practice. Finally, with its safeguards in place andits introspective reasoning, it is capable of discriminating when to resort to human

expert for recommendations, and when to provide these recommendations itself,enabling the cooperative work between the system, the home care provider, and thespecialist.

References

1. Aamodt, A., Plaza, E.: Case-Based Reasoning: Foundational Issues, MethodologicalVariations, and Systems. AI Communications. 7(1) (1994) 39-59


10/81


11/81

A Case-Based Approach to Gene Finding

Edwin Costello and David C. Wilson

Computer Science Department, University College Dublin, Dublin 4, Irelandedwin [email protected],[email protected]

Abstract. Advances in molecular biology and the tools and techniquesavailable for analysing molecular structure are providing a rapidly in-creasing deluge of information that maps fundamental genetic structuresin humans and other organisms. Intelligent support is essential for man-aging and interpreting this overwhelming amount of data, and one ofthe most important tasks faced currently is the analysis of sequences ofnucleotides in order to locate the areas of DNA that actually encode

functional biological information. Such analysis has a substantial impacton the health sciences as the foundation of identifying genetic structuresthat are related to disease and that can serve as the basis for pharma-cological or gene-therapeutic response. This paper describes our initialresearch in developing a CBR approach to the problem of finding regionsin mammalian DNA that code proteins essential for life.

1 Introduction

Bioinformatics incorporates the fields of biology, computer science and informa-tion technology with the goal of discovering new biological insights and the en-hancement of diagnostic and pharmaceutical medicine. Sequence analysis, which

involves the study of an organisms DNA in an effort to understand its molecu-lar structure and underlying functionality, is of major importance to the area ofgene therapy, which has led to the discovery of mutations in DNA and chromo-somal abnormalities indicative of diseases such as cancer. Thus the analysis ofnucleotide sequences, in particular the identification of DNA segments that en-code functional biological information, can provide the medical profession withinvaluable insight into the pathology of disease state and treatment.

Sequence analysis first involves determining the basic molecular structure ofan organisms DNA, which is simply a molecule made up of two strands, witheach strand comprised of nucleotides from a finite set. There are four differentnucleotidesadenine, guanine, thiamine and cytosine, and the first letter ofeach provides an alphabet {A,G,T,C} for representing DNA. A nucleotide-to-nucleotide bond holds the two strands together with each nucleotide being

bonded to its complementary match, A only bonding to T and C only bondingtoG. Therefore, given one strand (i.e., half a DNA molecule), the complementarystrand can be reconstructed relatively easily.

Determining the basic molecular DNA structure sequence is a well under-stood task that is providing a deluge of information for interpretation. The task


12/81

of gene finding, identification of coding regions in an organisms DNA, is thenext essential step in analysing an organisms genome. These coding regions arecalled exonsand when these are put together they form an entire gene. It is thegenes that tell the body how to create proteins, and it is these that give rise tobiological function. Exons are continuous sequences in a strand that the bodyuses to replicate proteins; the parts in between these exons do not contribute toprotein replication and are called introns, or noncoding regions. A human gene,for example, can consist of up to 2000 base-pairs (bps), with the gene beingsplit up into, on average, 10 exons. The aspect that makes the finding of theseexons most difficult is that the exons can be spread at random in a string of upto one million base pairs. In fact the human DNA is made up of almost 97% ofso-called junk DNA that does not code any proteins [Pevzner 2000]. It mustbe noted however, that this intron-exon structure only applies to mammaliantype DNA, which we focus on in this research.

We are interested in applying CBR to the gene-finding problem by employing

a case library of nucleotide segments that have previously been categorised ascoding (exon) or non-coding (intron), in order to locate the coding regions ofa new DNA strand. The primary research issues addressed in this work areestablishing a similarity metric for nucleotide segments and combining the resultsof multiple cases to categorise entire new DNA strands. This paper describes ourinitial research in developing a CBR approach to the problem of finding regions inmammalian DNA that code proteins essential for life. Section 2 gives an overviewof related work, section 3 describes our similarity metric, section 4 describes anevaluation of the initial approach, and we conclude with a discussion of futurework.

2 Gene Finding Methods

The structure of genes is well understood and their characteristics are used inmany of the techniques that are used for exon prediction and gene classifica-tion. While we employ only direct nucleotide comparison, it is important to notethat other information may be available, such as Promoter and Terminator sig-nals that occur before the first and after the last exon respectively, and Donorand Acceptor sites that can help to indicate intron-exon boundaries [Brunak,Engelbrecht, & Knudsen 1997].

A number of approaches have been applied to signal detection and gene-finding, including neural networks (e.g., [Towell, Shavlik, & Noordewier 1990;Farber, Lapedes, & Sirotkin 1992]), Bayesian approaches (e.g., [Staden & McLach-lan 1982]), and Hidden Markov Models (e.g., [Krogh 1998; Kulp et al. 1996;Burge & Karlin 1997]).

Overton and Haas [1998] describe a case-based system that used grammars,describing features of genes such as promoter regions and other signals. Thecases that these grammars described were mapped to the unknown sequencein an attempt to generate predictions for exons, introns and certain regulatorysignals. The cases used were generally full genes. For the system proposed here,


13/81

the way an identified gene was broken up in its original DNA sequence is used,i.e. the individual component exons. These exons will be used as the cases in thecase-base.

Shavlik [1991] discusses a system called FIND-IT, which translates a querysequence of DNA into its possible protein translations based on the different read-ing frames. It then performs a search for matching cases from a large databaseof protein sequences. Our approach matches in nucleotide space, rather thanprotein space; while matching in protein space may have some advantages, itcan also suffer from frame-shift errors in translation.

3 Sequence Similarity

For a CBR system to work effectively, it is necessary to be able to compare the

case (i.e., a known exon), with the query strand of DNA in which we want toidentify exons. We have investigated a number of possible approaches to similar-ity, including Longest Common Subsequence [Cormen, Leiserson, & Rivest 1992]and sequence alignment methods [Jiang b], but we have chosen an edit distancemethod for our initial work.

The edit distance between two sequences is defined as the minimum cost oftransforming one sequence to the other with a sequence of the following oper-ations: deleting a character, inserting a character or substituting one characterfor another, with no character taking part in more than one operation [Hoang1993]. Each of these operations can be given a weight in order to penalise cer-tain operations. Most implementations [Jiang a] seem to apply a weight of 1 toan insertion, 1 to a substitution and 0 for a nucleotide match. Figure 1 showsan example edit distance computation between a case exon and a target DNA

sequence segment, with which the case is aligned. The minimal transformationcost between the exon case and the target segment requires a deletion (d) fromthe target, two substitutions (s), and an insertion (i) in the case, giving anedit distance of 4. This type of similarity is also conceptually appealing, as itcomputes similarity using adaptability (e.g., [Smyth & Keane 1995]).

... G T A G C C G A A T C G ... Target SequenceA C G A A G A T C Case Exon

... G T A - G C C G A A T C G ...A C G A A G - A T C

d s s i

1 1 1 1 = 4 Edit Distance

Fig. 1.Example edit distance computation.


14/81

Computing the edit distance can be expensive, and we use a dynamic pro-gramming approach. Memoization is employed by creating a table t of size(m+ 1)(n+ 1) to store values for subproblems of the original. The edit dis-tance algorithm matches each character in a sequence with all the charactersin the comparison sequence. At each comparison it assigns a score based onthe previous scores kept in the table. When generating a score for a charactercomparison at index (i, j), if the characters match it is given the score fromindex (i1, j1), otherwise it is given the minimum score of (i1, j) + 1 or(i, j1) + 1. This way, the smallest score is propagated through the table.

3.1 Sequence Scoring

Given a measure of sequence similarity, we need to employ the case librarysegments in a way that will enable one to isolate regions of a sequence of DNAand point to them as potential protein coding regions. Since library exons arelikely to be much shorter than a new strand, we adopt an approach that combinesmany retrieved cases (e.g., [Ram & Francis 1996]) in order to arrive at the newsolution.

We assign an activation to all the nucleotides involved in an optimal editdistance alignment. When all the case exons have been compared to the querystrand, each nucleotide of the strand will have an activation that can be used todecide whether it is involved in a coding region or not. The scoring of a nucleotideis straightforward, the edit distance calculated for the best scoring subsequenceof the query strand is applied to each of the bases in that subsequence, as shownin Figure 2.

Fig. 2.Example sequence scoring.

As more exons line up with a strand region, the activation increases. Howeverkeeping track of the cumulative edit distance would not be enough, as a large

distance score would skew the results. To compensate for this, we also store avalue that indicates how many times an individual nucleotide has been involvedin an optimal alignment. In the example shown in Figure 2, the three nucleotideswith a score of 7 would have a participation score of 2, with a score of 1 for theothers.


15/81

Given scores for activation and participation, we employ three measures forclassifying an individual nucleotide:

Measure 1 nucleotide activation score, normalised by the maximum nu-cleotide activation score

Measure 2 number of optimal matches the nucleotide participates in, nor-malised by the maximum nucleotide participation score

Measure 3 the first two measures combined as a product A parameterizedthreshold is applied to the metric value, in order to determine coding status.

When the analysis of a given test DNA sequence is finished, the result can bevisualized, as shown in Figure 3. The base (wrapped) line represents the sequencebeing analysed, the thickness represents the degree of nucleotide activation, andthe segments above the baseline show the position of the actual exons in the testsequence.

Fig. 3.Visualization of nucleotide activation.

4 Evaluation

We were interested in comparing our approach with other types of analysis,and we chose to evaluate the system with a test dataset developed by Bursetand Guigo [1996] to evaluate gene-finding programs. It consists of 570 sequencesobtained from the vertebrate divisions of GenBank release 85.0 [Burset & Guigo

1996] and contains a total of 2649 coding exons.In choosing a set of sequences to act as the case-base for our system, we

wanted to select a dataset such that none of the sequences in the test set werethe same as any of those in the case-base. The dataset that was chosen was onethat was constructed specifically for the evaluation of seven recently developed


16/81

programs for gene finding in mammalian sequences [Rogic, Mackworth, & Ouel-lette 2001]. The name of the dataset is HMR195 and consists altogether of 195strands of human, rat and mouse DNA.

In order to evaluate the system, the entire set of test sequences was analysed.Before the results are reviewed however, it is necessary to present the accuracymeasures used. Burset and Guigo, [1996] outlined the measures used in theirevaluation of gene finding programs, which will be utilised for the purposes ofthis project. These measures are:

Sensitivity (Sn) proportion of coding nucleotides that are correctly classifiedas coding

Specificity (Sp) proportion of noncoding nucleotides that are correctly clas-sified as noncoding

Correlation Coefficient (CC) combine the values for sensitivity and specificity

Approximate Correlation (AC) approximates CC, but is defined for all values

In this work, we use the AC as our primary measure of accuracy, since it inte-grates Sn and Sp and is defined for all values.

4.1 Results

The system was allowed to run on the entire test set of sequences. This involvedcomparing the whole case-base of exons to each of the test set sequences.

Cumulative Results In order to determine the average behaviour of the system

across the entire set and to see how each new case affects the overall result, wekept track of cumulative results. This involves calculating the average, acrossthe entire test set, of each of the accuracy measures after each successive librarycase has been applied. The measure provides not only the average accuracy trendacross all the test sequences, but also, for the final case added it gives the averageof each accuracy statistic for all the tested strands. Figure 4 shows the AC valuesfor Measure 1 (middle curve), Measure 2 (top curve), and Measure 3 (bottomcurve). We can see a number of things in these results. First, Measure 2 providesthe most accurate method of scoring nucleotides. Second, there is an indicationthat there may be critical points in terms of numbers of cases necessary foranalysis, as positive correlations begin only around the 50th case and take asignificant turn around the 850th case.

The correlation itself is perhaps most informative. A correlation value of

zero would indicate random behaviour by the approach. Our method, while rel-atively weak when compared to some others, does provide a significant measureof predictive power. Given that other methods can employ thousands of trainingexamples, we can achieve a reasonable accuracy with 948 exons from our 195base strands with a clear upward trend continuing.


17/81

Fig. 4. Approximate correlation cumulative accuracy results for Measure 1 (middlecurve), Measure 2 (top curve), and Measure 3 (bottom curve).

Strand Results Another way of viewing the results is to examine how wellthe approach performs on individual strands over the set. For example, of thesequences analysed, 23% have a CC value greater than 0.4 and over 43% have a

value greater than the average of 0.28. Figure 5 compares the results of each ofthe strands tested against the results for the same strand as returned by the arepresentative program from the [Burset & Guigo 1996] analysis, GeneId. Theresults are ordered by increasing GeneId accuracy, represented by the smoothcurve, along with the results from our approach that appear more dispersed giventhe ordering. This arrangement provides a visualization of the proportion of thetest population for which our approach outperforms GeneId, given by the pointsthat lie above the GeneId curve. There is a significant segment of the population(13%) for which the CBR approach performs better than GeneId. While thiscomparison argues that GeneId would be the better single choice, there is clearlya segment of the population that would benefit from a complementary CBRapproach.

4.2 Metric Comparison

Table 1 shows a comparison of our average Metric 2 accuracy results with thetested programs from [Burset & Guigo 1996]. The CBR approach outlined here


18/81

Fig. 5.Strand results in increasing order of GeneId (smooth curve) accuracy.

demonstrates that although it may not be as accurate as the other programs, itdoes suggest that its value should not be overlooked.

In summary, having tested the initial system and evaluated the results it isevident that a simple case-based reasoning approach to the recognition of coding

regions is certainly possible. The results show that there are sequences in thetest set that can be classified more accurately by this approach. The statisticalmeasure used (AC) rules out the affect of random influence on the results andthat the accuracy levels are indeed indicative of the emergence of the codingregion features through comparison to example exons.

Program Sn Sp AC CC

FgeneH 0.77 0.88 0.78 0.80GeneId 0.63 0.81 0.67 0.65

GeneParser2 0.66 0.79 0.67 0.65

GenLang 0.72 0.79 0.67 0.65GRAILII 0.72 0.87 0.75 0.76

SORFIND 0.71 0.85 0.73 0.72Xpound 0.61 0.87 0.68 0.69

CBR Measure 20.77 0.30 0.49 0.35Table 1. Average Accuracy Measures for Nucleotide Level.


19/81

5 Conclusion and Future Work

We have presented our initial work in applying a case-based approach to theproblem of gene-finding in mammalian DNA. The results obtained from the ap-proach taken here indicate that it is certainly feasible to do DNA-to-DNA com-parisons in order to isolate relevant coding regions. Using DNA sequences, avoidsthe need for the translation of the sequence to the different protein sequence pos-sible. This in turn, it is believed, will reduce the effect of any frame-shift errorson the final results. Our first iteration has employed a very straightforward edit-distance method for similarity comparison in nucleotide space, ignoring othercontext such as promoter/terminator signals and donor/acceptor sites. Using arelatively small library in comparison to the training sets from other approaches,we have (1) achieved significant levels of accuracy, albeit low in comparison toother approaches, and (2) achieved higher accuracy in a significant segment ofthe test population than other approaches. This argues that our CBR approach

can be useful, certainly as a complement to other methods. We intend to testupdated versions of the system that take into account additional contextualinformation, such as signals and protein encodings, as well as larger case repos-itories. Given the simple approach taken here, we expect that results can beimproved dramatically and that a CBR approach to gene finding will prove aviable complement, or even alternative, to other methods.

References

Brunak, S.; Engelbrecht, J.; and Knudsen, S. 1997. Prediction of human mRNAdonor and acceptor sites from the DNA sequence. Journal of Molecular Biology220:4965.

Burge, C., and Karlin, S. 1997. Prediction of complete gene structures inhuman genomic DNA. Journal of Molecular Biology268:7894.

Burset, M., and Guigo, R. 1996. Evaluation of gene structure prediction pro-grams. Genomics34:353367.

Cormen, T. H.; Leiserson, C. E.; and Rivest, R. L. 1992. Introduction toAlgorithms. MIT Press.

Farber, R.; Lapedes, A.; and Sirotkin, K. 1992. Determination of eukaryoticprotein coding regions using neural networks and information theory. Journalof Molecular Biology226:471479.

Hoang, D. 1993. Searching genetic databases in splash 2. IEEE Workshop onFPGAs for Custom Computing Machines185191.

Jiang, T. Approximation algorithms for multiple sequence alignment. URL:

http://www.iis.sinica.edu.tw/ hil/summer/jiang2.ppt. University of CaliforniaLecture Notes.

Jiang, T. Fundamental algorithmic problems and techniques in sequence align-ment. URL: http://www.iis.sinica.edu.tw/hil/summer/jiang1.ppt. Universityof California Lecture Notes.


20/81

Krogh, A. 1998. An introduction into hidden markov models for biological se-quences. In Salzberg, S.; Searls, D.; and Kasif, S., eds., Computational Methodsin Molecular Biology. Elsevier Science. chapter 4.

Kulp, D.; Haussler, D.; Reese, M.; and Eeckman, F. 1996. A generalized hiddenmarkov model for the recognition of human genes in DNA. In Proceedings ofISMB-96, 134142.

Overton, C. G., and Haas, J. 1998. Case-based reasoning gene annotation. InSalzberg, S.; Searls, D.; and Kasif, S., eds.,Computational Methods in MolecularBiology. Elsevier Science. chapter 5.

Pevzner, P. A. 2000. Computational Molecular Biology: An Algorithmic Ap-proach. The MIT Press. chapter 4,6.

Ram, A., and Francis, A. 1996. Multi-plan retrieval and adaptation in anexperience-based agent. In Leake, D., ed., Case-Based Reasoning: Experiences,Lessons, and Future Directions. Menlo Park, CA: AAAI Press.

Rogic, S.; Mackworth, A.; and Ouellette, B. 2001. Evaluation of gene-finding

programs on mammalian sequences. Genome Research11(5):817832.Shavlik, J. 1991. Finding genes by case-based reasoning in the presence of noisycase boundaries. InProceedings of the 1991 DARPA Workshop on Case-BasedReasoning, volume 14, 861866.

Smyth, B., and Keane, M. 1995. Experiments on adaptation-guided retrieval incase-based design. In Proceedings of First International Conference on Case-Based Reasoning.

Staden, R., and McLachlan, A. 1982. Codon preferences and its use in iden-tifying protein coding regions in long DNA sequences. Nucleic Acids Research10(1):141156.

Towell, G. G.; Shavlik, J. W.; and Noordewier, M. O. 1990. Refinement of ap-proximate domain theories by knowledge-based neural networks. InProceedingsof the Eighth National Conference on Artificial Intelligence, 861866.


21/81

Matching SMARTHOUSETechnology to Needs of the

Elderly and Disabled

Genevieve Davis

, Nirmalie Wiratunga

, Bruce Taylor

, and Susan Craw

School of Computing,

The Robert Gordon University,

Aberdeen, AB25 1HG, Scotland, UK.

nw s.craw @comp.rgu.ac.uk

Scott Sutherland School,

The Robert Gordon University,

Aberdeen, AB10 7QB, Scotland, UK.

b.taylor @rgu.ac.uk

Abstract. SMARTHOUSEtechnology comprises of devices that help the elderly

and people with disabilities to live independently in their homes. This paper

presents our experiences from a three month pilot project involving the applica-

tion of CBR techniques to match the needs of the elderly and those with disabili-

ties to SMARTHOUSEtechnology. The SMARTHOUSEproblem was decomposed

into sub-tasks and generalised concepts were added for each sub-task. This de-

composition and generalisation enabled the use of a standard decision tree based

iterative retrieval strategy. Documented real situations were used to create a small

case base and a prototype implemented using R ECAL Lwith TCLscript was eval-

uated on three test cases. Results showed that the generated solutions to be com-

parable to that of a domain experts solutions.

1 Introduction

Smart House Technology has been around for about twenty years and aims to encourage

independent living for the elderly and disabled. This technology may consist of some-

thing as simple as a telephone amplifying unit for those with hearing impairments, toa field bus system, where large numbers of sensors and actuators each with their own

microprocessors are connected together, for people with severe mobility problems.

Occupational Therapists usually match assistive technology to peoples needs by

observing people engaged in day to day tasks. If they have difficulty doing these tasks

then a technological solution is suggested, often based on similar past observations. The

solution may also consists of non technological elements e.g. secure by landscape de-

sign. In recent years the range of technology available has increased considerably. This

additionally makes it harder for occupational therapist to keep abreast with S MART-

HOUSEdevices, thereby creating a real need for automated tools to help match S MART-

HOUSEtechnology to peoples needs.

In the rest of this paper we will describe how a prototype was built to match peoples

needs to SMARTHOUSEtechnology. Section 2 discusses case representation issues, and

an iterative retrieval strategy is presented in Section 3. Evaluation on three test cases


22/81

are presented in Section 4 followed by related work in Section 5, and conclusions in

Section 6.

2 Case Representation

The EU funded Custodian project (1999-2001) created a SMARTHOUSE technology

simulation tool to illustrate how different S MARTHOUSEdevices function in the house.

More importantlyfor thework presented in this paper elevenreal scenarios consisting of

people needs and matching technology solutions were acquired during this project [3].

The first task for developing our prototype CBR system was to manuallytranslate

the documented real cases into a more structured representation of feature-value pairs.

This involved identifying relevant features that belonged to the problem space and the

solution space; and establishing the data values with which to instantiate these features.

For example

... Ms M was a powered wheelchair user with cerebral

palsy. She was a tenant in her own ground-floor flat, and

whilst living moderately independently, did have support

from care workers to assist her in getting dressed andbathed... The case goes on to describe specific problem

areas: Door and Lock operation; Telephone operation; etc.

So translating from this text we can extract the following to describe the problem:

wheel-chair-indoor Y, wheel-chair-outdoor N, house-type ground-floor-flat, care-staff

Y, able-into-bath with-help; and the solution described by: contacts-powered-external-

door Y, electrically-operated-locks-external-door Y, intercom-front-door video-hands-

free. Both the problem and solution features are nominal but establishing quantifiable

values to instantiate these features with required much time and effort (e.g. the feature

able-into-bath can have values Y, N or with-help). Missing values was another common

problem (because the cases were documented for an entirely different purpose from that

of developing a CBR tool) and although the domain expert could have made informed

guesses in some situations often it was left as unknown.A total of 108 features were identified of which 64 describe the problem in terms

of the person, their home, abilities and needs; and a further 44 features describe the

solution (the SMARTHOUSEdevices). With increased numbers of features and instan-

tiation values it made sense to group these features (particularly the solution features)

according to their context forming 10 problem space and 14 solution space groups (see

Figure 1). The solution space consists of the SMARTHOUSEdevices organised into 14

groups, the number within brackets denote the number of devices in each group e.g.

5 MovementRelateddevices. We will refer to solution space groups as device ob-

jects while features such asRemoteAlarm,FieldBusArchitectureand Pow-

eredWindowsforming singleton groups will be referred to simply as a device. Often

grouping devices is generally a straightforward task for the domain expert and in dif-

ficult instances a pragmatic decision was made, e.g. would it make a difference if the

devices associated withDoorsis split into exterior and interior doors.


23/81

Fig. 1.SMARTHOUSEcase representation

3 Retrieval Strategy

Several commercial CBR tools (e.g. RECALL, REMINDand KATE) use decision trees

(DTs) to index the case base in order to improve retrieval efficiency [1]. Although ef-

ficiency is a primary reason for using an index, a DT formed using C4.5s information

gain ratio heuristic [5] additionally provides a useful means to explain the underly-

ing reasoning behind the retrieval; a desirable feature for CBR applied to the S MART-

HOUSEdomain. Figure 2 illustrates a case base index created by inducing a C4.5 DTfor the sub-taskPoweredWindows (i.e. power windows required or not). When a new

problem (i.e. a person description) is encountered the tree is traversed and depending

on the persons ability to open internal doors the relevant leaf node and its cases are se-

lected. These selected cases form the neighbourhood and the majority solution obtained

using weighted Nearest Neighbour (K-NN) instantiates the PoweredWindows part

of the solution for the new problem. Notice that the DT is built with the aim of parti-

tioning cases according to a specified concept (e.g. the concept is PoweredWindows

and can be instantiated with values or ).

3.1 Using Multiple Indices

In previous work, we have shown how multiple DTs can be employed to solve design

problems, where each tree concentrates on retrieving cases pertinent to solve a different


24/81

part of a design task [2]. The main advantage of this approach is that it enables the

CBR system to use different parts of cases in the case base to solve different sub-tasks

given a new problem. A similar approach can be adopted for the S MARTHOUSEdo-

main, where identifying a suitable SMARTHOUSE device is a sub-task and is solved

in conjunction with a separate DT index. Traversal of each tree given the new per-

sons description identifies a different subset of cases to solve each sub-task. However,

apart from RemoteAlarms,PoweredWindowsand FieldBusArchitecture

the remaining 41 devices (grouped under 11 device objects) would require inducing a

further 41 DTs, which from an efficiency point of view does not make sense. Instead

we exploit the fact that there are 11 device objects identified and create a generalised

concept feature, gen-concept, per object. This would mean inducing 11 DTs (in-

stead of 41), in addition to the 3 DTs for RemoteAlarm, PoweredWindowsand

FieldBusArchitecture.

Fig. 2.Index used when retrieving cases to solve the PoweredWindows sub-task

3.2 Creating Generalised Concepts

We will now look at how a value can be assigned to a gen-conceptfeature created

for a device object. Let us consider two cases from the case base and their instantiations

for just the CookerRelateddevice object (see Table 1). With CaseXwe have a

solution involving a timer based automatic cooker shut off together with a detector thatcandetect gas, heat andsmoke, and three alarms; while CaseY has comparatively fewer

alarms. So how do we establish a value for gen-concept? The simplest approach is

to assign a binary value; where a majority ofN values for associated devices suggests

0, otherwise1. In our example the gen-conceptfeature will be instantiated with 1

forCaseXand0forCaseY.

In this manner all cases in the case base are modified by adding a binary valued

gen-conceptfeature per device object. We can then induce a DT per device object

with the gen-conceptfeature set as the class. For example if we have to establish

suitable devices related with cookers for a given new problem, theCookerRelated

DT (induced for the CookerRelated gen-conceptfeature) is traversed to iden-

tify the leaf node cases and then K-NN is applied to access the nearest neighbours.

The majority solutions for cooker detectors and alarms from these cases are reused

and instantiates the cooker related solution part of the new problem.


25/81

Table 1.Case values for the CookerRelated device object

CookerRelated-Devices CaseX: Value CaseY: Value

auto-cooker-shut-off

gas, heat, timer

gas, heat

detectors-over-cooker

gas, heat, smoke

gas, heat

alarm-for-carers N N

cooker-isolation-alarm Y N

fire-alarm Y N

gas-alarm Y N

gen-concept ? ?

4 Evaluation

In our experiments we used the RECALL CBR shell together with a TCL script to

implement the SMARTHOUSE prototype. The 11 real-cases were supplemented with

another 12 cases invented by our domain expert, forming a case base of 23 cases. A

further 3 test cases were supplied by the domain expert where test case A and B were

created by making minor variations based on existing cases and test case C created

from scratch. Importantly these test cases were generated to portray realistic problem

scenarios with different levels of similarity to existing cases in the case base.

The graphs indicate the percentage correct for each of the device objects, and 3

of the (non-object) devices. So 100% correct forCookerRelatedindicates that the

systems suggestions for the 6 devices grouped under this object (see Figure 1) matched

the domain experts solution. ForRemoteAlarm,PoweredWindowsand Field-

BusArchitecture the percentage correct can either be 100 or 0 (because these

represent a single device each).

4.1 Test Case A

Generally there was a significant overlap between the experts and systems solutions.

With Controllers the expert solution suggests a lighting-controller while

the system solution failed to include this device. The reason for this is that the DT

for Controllersselected ambulant-mobilityas a discriminatory feature to

partition the case base. Since the only case with a lighting-controllerdevice

in the case base differed from test case A in ambulant-mobility, it did not fall

under the same leaf node as test case A. This is an obvious problem with the limited

number of cases in the case base and the poor performance of 3-NN in general supports

this conclusion. However, the domain expert was satisfied with the use ofambulant-

mobility as the decision node for the Controllers device object. The expert also

indicated that with test case A the person, although having good ambulant mobility

required a lighting-controllerdevice because of the inability to operate light

switches.


26/81

Fig. 3.Percentage correct for Test Case A

Fig.4.Percentage correct for Test Case B

4.2 Test Case B

Here we see a mismatch with the Doorsdevice object, where the system suggested

door-contacts-on-exterior-doors and alarm-for-exterior-door-when-open, while the do-

main expert suggested only an alarm-for-exterior-door-when-open.Again 3-NN perfor-

mance is worse compared to 1-NN, for instance it suggested a telephone amplifier while

the experts solution did not contain any devices associated with hearing.

4.3 Test Case C

Unlike test cases A and B, C was not a modified case from the existing case base and

so was more challenging for the system. With 1-NN for CookerRelateddevices


27/81

Fig.5.Percentage correct for Test Case C

the expert solution consisted of a fire-alarm and an alarm-for-carers, which were not

picked up by the system. Next with devices associated with the Doorsobject, the

expert solution included door contacts on the exterior doors, and in the bedroom door,and an alarm on the exterior door. These were also not included in the systems solution.

With MovementRelated devices the system suggestedall of themovement detectors

while the expert suggested a single movement detector in the bedroom with a movement

detection alarm. However, it was thought that this result although not exactly the same

as the experts solution was nevertheless plausible. With the Phonerelated devices

the system suggested the use of a mobile phone while the expert did not. This was

considered by the expert to be a trivial error.

5 Related Work

A prototype developed within the Auguste Project aids decision making with regards

to prescribing a neuroleptic drug for Alzheimers patients [4]. The case representationconsists of a grouping of features where the top level view contains: Personal Infor-

mation, Physical Health, Behavioural Aspects etc. We have also adopted this general

approach of grouping the problem and solution space, although with S MARTHOUSE

devices the solutions space being more complex benefited from concept generalisation.

The use of weights for K-NN was ascertained by the domain expert, but unlike the

domain dependent similarity metric employed by the Auguste Project we applied the

standard overlap metric (due mainly to project time constraints).

The NIRMANI system [6] uses a hierarchical case representation to model the prob-

lem domain. The problem space is decomposed into sub-tasks as a means to enforce

context guided retrieval, where parts of cases are retrieved in stages. We also advo-

cate task decomposition particularly with complex cases, additionally the structuring of

the solution space and concept generalisation enabled the SMARTHOUSEprototype to

apply an iterative DT-based retrieval strategy.


28/81

The work discussed here is most similar to the retrieval approach adopted for the

tablet formulation domain [2], where instantiation of each component of a tablet is

treated as a sub-task and a DT-based retrieval is triggered for each sub-task. The main

difference with the SMARTHOUSEdomain is that here the sub-tasks are too numerous

to be dealt with separately and hence a grouping of sub-tasks into device objects and the

addition of concept generalisation is required before multiple retrieval is manageable.

6 Conclusion

We have presented our experiences with developing a prototype CBR system for the

SMARTHOUSE problem domain, where the task involved matching SMARTHOUSE

technology to the needs of the elderly and people with disabilities.

Organising the solution space into device objects decomposed the SMARTHOUSE

task in to manageable sub-tasks. Task decomposition with concept generalisation en-

abled the use of a standard DT-based iterative retrieval strategy. Importantly the use of

multiple indices enabled re-use of different cases to solve different parts of the same

test case, thereby encouraging best use of the relatively small case base. Initial results

on three test cases have been promising and show a significant overlap between the ex-

perts and systems solutions. However we are conscious of the fact that these results

were purely based on cases invented by the domain expert. Therefore a more realisticevaluation is needed before feasibility of this approach can be justified.

The ability to explain the reasoning behind the proposed systems solution is a de-

sirable feature and with the SMARTHOUSEprototype the expert found that the display

of both index trees and similarity of cases at leaf nodes to be a useful step in this di-

rection. Finally it is worth mentioning that it was the prototypes ability to use multiple

cases to solve different parts of a given S MARTHOUSEproblem that most impressed

our domain expert.

References

1. Aha, D.W., Breslow, L.A.: Comparing simplification procedures for decision trees on an eco-

nomics classification task. Technical Report AIC-98-009, Navy Center for Applied Researchin AI, Washington DC, 1998

2. Craw, S., Wiratunga, N., Rowe, R.: Case-based design for tablet formulation. InProceedings

of the 4th European Workshop on Case-Based Reasoning, pages 358369, Dublin, Eire,

1998. Springer

3. Dewsbury, G., Bonner, S., Taylor, B., Edge, M.: Final Evaluation of Tools Developed. CUS-

TODIAN Project, CUSTODIAN/RGU/WP7.3/RE/004 Vb, EU Telematics Initiative for Dis-

abled and Elderly People (TIDE), DE4004, 2001

4. Marling, C., Whitehouse, P.: Case-Based Reasoning in the Care of Alzheimers disease

patients. InProceedings of ICCBR 2001, pages 702-715, 2001, Springer

5. Quinlan, J.R.: C4.5: Programs for Machine Learning. 1993, Morgan Kaufmann

6. Watson, I., Perera, S.: A hierarchical case representation using context guided retrieval.

Knowledge-Based Systems, 11:285292, 1998


29/81

FM-Ultranet: a Decision Support System using Case-

Based Reasoning, Applied to Ultrasonography

Ziad El Balaa1, Anne Strauss1, Philippe Uziel1, Kerstin Maximini2and Ralph

Traphner3

1University Pierre and Marie Curie,

75005 Paris, France{elbalaa, strauss}@ccr.jussieu.fr

[email protected] and Knowledge Management Group, University of Hildesheim, PO Box 101363,

31113 Hildesheim, [email protected] GmbH arvato Knowledge management, part of Bertelsmann, Europaallee 10,

D-67657 Kaiserslautern, [email protected]

Abstract.Case-based reasoning (CBR) is a recent approach for many applica-

tions in medicine, particularly in decision support. In this paper we describe the

CBR system FM-Ultranet, used in the domain of ultrasonography. Due to the

hierarchical structure of the model and the definition of a similarity measure,

we achieve an excellent retrieval of similar cases from the case base. Medical

experts have evaluated FM-Ultranet in daily practice and the system has met

their expectations.

1 Introduction

In almost all European countries, the surveillance of pregnant women is done

through ultrasonographic examinations, usually 3 during the 9 months pregnancy.

This technique started to be used in 1975 and there are still few experts in this field;

depending on the country, between60% and 75% of the malformations remain unno-

ticed in Europe [1]. This is due to the fact that, as compared to almost all other ultra-

sonographic examinations, the examination of the fetus necessitates to carefully

watch not only for the anatomical features of the organs but also for their relationship

as well as the movements of the fetus. The detection of malformations or abnormali-ties is consequently very difficult and they often remain unnoticed until the birth of

the child. The consequences are manifold; the most common ones are unscheduled

medical or surgical interventions - which might have consequences on the future

health of the child - up to medical termination of pregnancy. To reduce these prob-

lems, reference centers in prenatal diagnosis have been established to offer first level

ultrasonographists, i.e. thepractitioners who perform routine screening ultrasounds of

pregnant women, the possibility to ask for expert advice. This adds extra costs and is

far from solving the problem; the number of experts is insufficient and sometimes the


30/81

expert is not available where the practitioner needs real-time expertise about a mal-

formation.

In France only, fetal malformations represent about 3%-4% of the cases [2], i.e. 40

000 to 50 000 cases every year. At the European level, this problem concerns a hun-

dred thousandpregnancies [3].

The problems described above are addressed by our work, partly funded by the

European Union within the IST research program under contract no. IST-1999-

20865: Foetal Malformation Ultrasonography training networking platform (FM-

Ulranet).

2 Case-Based Reasoning in Ultrasonography

When an ultrasonographist detects an abnormality during the morphological examina-

tion, his or her problem is to decide whether this abnormality is a dangerous malfor-

mation; if pregnancy should be terminated, or just a particularity without importance.

To come to his decision, the ultrasonographist can use several knowledge sources: his

textbooks medical knowledge in this domain, his personal memory of previous

cases (personal experience), and/or the advice and second opinion of an expert. FM-

Ultranet relates those ways to support the ultrasonographist by using Case based

reasoning (CBR) and Expert networking technology.

We improve ultrasound scans interpretation and diagnosis through comparison of

past existing similar cases stored in a database of reference cases and with guidance

byan expert.

The goal of our work within the project was the development of: a reference database of clinical cases of foetal malformation,

a multimedia support dedicated to these malformations making available the data-

base for consultation by first level ultrasonographists and offering them the possi-

bility to submit cases to experts for advice,

a decision support system that will exploit image analysis and pattern recognition

techniques to provide diagnosis and training to the practitioner.

In this paper we describe how we have achieved this goal and present the architecture

of the FM-Ultranet system. It is an application based on CBR, because this technol-

ogy is able to combine the capacity of the ultrasonographist to solve a problem with

the capacity of an information system. The efficiency of CBR in medicine is proven

in many domains [4,5,6,7] especially in decision support systems [8,9,10]

3 Knowledge Representation

For case representation, we have developed an object-oriented, hierarchical model,

which consists of about 140 attributes, organized in 39 concepts and sub concepts to

represent characteristics in a medical sense. Most of the attributes store knowledge

about the anatomical structure of the fetus, like the urinary tract and the morphology


31/81

of the head, thorax, or rachis. But to retrieve preferably similar cases to a present case

from the reference case base, the case representation is completed by clinical data of

the mother (medical history), ultrasound imaging, bibliographic references, outcome

of the pregnancy, the status of the child at birth, the result of the autopsy, and the

international classification of diseases code (ICD10).

Each attribute has a data type assigned. The majority of data types are given by

special values like Seen, Not seen, It cannot be seen, and Not yet examined. Numeri-

cal values are used for biometrical information. Some attributes can carry symbolic

value enumerations, e.g., the type Change carries decrease, unchanged, and increase

to indicate the change between the current examination and the last examination for

the same patient.

Figure 1 is an excerpt from the complete hierarchical model. The concept Urogeni-

tal System consists of left and right kidney, adrenal gland and ureter as well as sex,bladder and the information, whether the renal vessel is found or not. The concept

Kidney and its sub-concepts are shown in detail. Figure 2 shows the decomposition of

our main concept FM. The concept has 23 attributes with different data types: we

distinguish attributes with simple data types (written in italics), attributes with data

types defined by us (written in italics and framed with a grey box) and attributes

whose data type is a concept itself (written bold and having a "+" or "-" in front).

-RightKidney : Kidney

-LeftKidney : Kidney

-Sex : Sex

-RightAdrenalGland : Normal/Abnormal

-LeftAdrenalGland : Normal/Abnormal

-RightUreter : Ureter-LeftUreter : Ureter

-RenalVesselFound : Boolean

-Bladder : Bladder

Urogenital System

+Normal

+Abnormal

enumeration

Normal/Abnormal

+true

+false

enumeration

Boolean

+decrease

+unchanged+increase

enumeration

Change

+small

+normal+big

enumeration

SimpleBiometry

Types

-AspectOfKidney : SimpleBiometry

-BiometryOfKidney : BiometryOfKidney

-KidneyPelvis : Normal/Abnormal

-ChangeOfAspectOfKidney : Change

-ChangeOfKidneyPelvis : Change

Kidney

-KidneyShape : Normal/Abnormal

-Cortico-medullarDifferentiation : Normal/Abnormal

-ChangeOfShapeOfKidney : Change

-ChangeOfCortico-medullarDifferentiation : Change

-ChangeOfDuplication : Change

-ChangeOfStatus : Change

-Duplication : Boolean

-Status : Normal/Abnormal

StatusOfKidney

-...

Vascularisation

Fig. 1.Excerpt from Aggregation and Specialization Hierarchy of Concepts


32/81

Fig. 2.FM main concept and the different attributes

For each attribute a similarity measure is defined by a mathematical function or a

table, depending on the attributes type. For example the similarity measure for the

gestational age is a function that reflects the usual time frame of eight weeks in which

the regular examinations take place, i.e. around week 14, 22 and 34. Figure 3 depicts

this function, where e.g. a case that is from a week that is 4 weeks later than the pre-

sent one is considered as being 50% similar.


33/81

Fig. 3.Similarity Measure for Gestational Age

Additional medical background knowledge is expressed in rules. There are rules to

derive the value of attributes from others, ex: if the practitioner enters the value Ab-

normal for the attribute Kidney Shape the system then automatically assigns the

value Abnormal for the attribute Status. There are rules to show any changes

between the present examination and the last one of the same patient and others to

generate an alert message if the practitioner fills an error value.

4 Results

FM-Ultranet has been implemented with CBR-Works, a tool provided by empolis, a

firm based in Germany. We have chosen 130 reference cases about malformations of

the urinary tract with their corresponding imagesas a starting point to evaluate our

system and to build upon; these malformations being very common, most frequent

and often difficult to detect.

To represent these cases we have defined a model, consisting of 140 attributes like

those described in Section 3. The knowledge representation fully covers sane and

normal fetus; according to the representation of malformations, the urinary tract is

fully described, 25% of the heart diseases and 10% of abdomen diseases are covered.

Using an easy interface with five screens only, the ultrasonographist can enter the

necessary data for a normal examination. For an abnormal examination another inter-

face containing all the attributes (Fig. 4), most of the values are automatically derivedby the system, and providing access to the International Classification of Diseases

(ICD10) can be used.


34/81


35/81

An evaluation questionnaire filled out after their perusal of the FM-Ultranet sys-

tem. The questions are assembled into nine groups (General information, Usage of

the tool, Satisfaction analysis and perceived usefulness, Impact on your work prac-

tices, Relevance of the data base, On-line training, Technical analysis, Economic

impact of the tool, Conclusion)

The objective of this trial was to experiment an on-the-job training platform for ultra-

sonographists in the field of pregnancy surveillance to improve early detection of

foetal malformation. The training platform aims at helping ultrasonographists to im-

prove their diagnosis skills by comparison with existing clinical cases, exploitation of

their own past experience and means to contact external experts.

The trial addresses the problem of improving foetal malformation detection by the

practitioner, which is a very difficult task for a non-specialist in this domain.

With

a unanimous yes answer to the question on usefulness and With 45% of thetesting population having acquired new knowledge, the FM-Ultranet system is meet-

ing ultrasonographers needs. Not only training needs directly related to their practice

but also basic organisational needs such as record keeping, statistical, and report

writing.

The possibility offered by the tool to compare situations from both an image stand-

point and clinical data standpoint, the FM-Ultranet system has proven useful in

guiding its users as regards: helping manage the cases, refining the selection of im-

ages, refining diagnosis and prognosis and determining the type of pathology. thus

demonstrating its capacity to improve ultrasonographers practice

The system is perceived as pedagogic, its use is offering diagnosis making sup-

port and decision making support. It is also perceived as a good tool for continuing

education.

Concerning the role of the experts the trial has clarified two key points. The ultra-sonographers wish to have access, on line, to an expert and one of the conclusions is

the role of the expert will become more important over time. This appears to be espe-

cially true with field practitioners whose main activity is not ultrasonography ; such

as MDs specialising in medical imagery working in small urban centres or rural areas.

5 Conclusions and Future

The FM-Ultranet system has proven its feasibility in the work with subject matter

experts and its usability in day-to-day practice.

With the help of the system, data collection turned out to be more complete than

before. In particular, the creation of the reference case base produced a new asset initself. The presentation of a relevant reference case during a present examination

leads to quicker and safer decisions, hence improving the confidence level of the

ultrasonographist.

After the trial period the ultrasonographists had two requests: the first one was to

extend the representation of malformations of the fetus, e.g., for the heart, head and

abdomen. The present structure already forms an important step towards standardiz-

ing the description of malformations. The second request is to integrate FM-Ultranet


36/81


37/81

P r e s c r i b i n g E x e r c i s e R e g i m e n s f o r C a r d i a c a n d

P u l m o n a r y D i s e a s e P a t i e n t s w i t h C B R

K a t h l e e n E v a n s - R o m a i n e a n d C i n d y M a r l i n g

S c h o o l o f E l e c t r i c a l E n g i n e e r i n g a n d C o m p u t e r S c i e n c e

O h i o U n i v e r s i t y , A t h e n s , O h i o 4 5 7 0 1 , U S A

f e v a n s - r k , m a r l i n g g @ o h i o . e d u

A b s t r a c t . T h i s p a p e r d e s c r i b e s a p r o t o t y p i c a l s y s t e m t h a t e m p l o y s

c a s e - b a s e d r e a s o n i n g t o h e l p s p o r t s m e d i c i n e s t u d e n t s l e a r n t o p r e s c r i b e

e x e r c i s e r e g i m e n s f o r c a r d i a c a n d p u l m o n a r y d i s e a s e p a t i e n t s . T r a d i t i o n -

a l l y , s t u d e n t s h a v e b e e n t a u g h t a m a n u a l , r u l e - b a s e d a p p r o a c h t o s e -

l e c t i n g w o r k o u t s f o r t h e s e p a t i e n t s . T h e t r a i n i n g s y s t e m p r o d u c e s t w o

r e c o m m e n d a t i o n s f o r e a c h p a t i e n t , o n e b a s e d o n t h e t r a d i t i o n a l r u l e s ,

a n d o n e b a s e d o n e x p e r i e n c e w i t h p a s t p a t i e n t s . I n p r e l i m i n a r y t e s t s , t h e

c a s e - b a s e d r e c o m m e n d a t i o n s m o r e c l o s e l y m a t c h e d t h e e x e r c i s e r e g i m e n s

p r e s c r i b e d f o r a c t u a l p a t i e n t s b y e x p e r i e n c e d s p o r t s m e d i c i n e p r o f e s s i o n -

a l s t h a n t h e r u l e - b a s e d r e c o m m e n d a t i o n s .

1 I n t r o d u c t i o n

T h e W e l l w o r k s / H e a r t w o r k s A d v i s o r / T r a i n e r ( W H A T ) i s a p r o t o t y p i c a l t r a i n i n g

s y s t e m f o r b e g i n n i n g s p o r t s m e d i c i n e s t u d e n t s i n t h e W e l l w o r k s a n d H e a r t w o r k s

p r o g r a m s a t O h i o U n i v e r s i t y . W e l l w o r k s i s a h e a l t h a n d l i f e s t y l e p r o g r a m f o r

f a c u l t y a n d s t a o f t h e u n i v e r s i t y . H e a r t w o r k s i s a p h y s i c a l r e h a b i l i t a t i o n p r o -

g r a m f o r p a t i e n t s w i t h c a r d i a c a n d / o r p u l m o n a r y d i s o r d e r s . T h e t w o p r o g r a m s

u s e s i m i l a r m e t h o d o l o g y a n d s h a r e s t a .

S t u d e n t s i n b o t h p r o g r a m s l e a r n t o p r o d u c e e x e r c i s e p r e s c r i p t i o n s f o r p a r -

t i c i p a n t s b a s e d o n t h e r e s u l t s o f m e d i c a l h i s t o r i e s a n d p h y s i o l o g i c a l t e s t s . T h e y

a r e t a u g h t s y s t e m a t i c r u l e s t o d o t h i s 1 ] . B e c a u s e t h e c o n s e q u e n c e s o f o v e r e x -

e r t i o n m a y b e s e v e r e , t h e e x e r c i s e p r e s c r i p t i o n s a r e h i g h l y c o n s e r v a t i v e . M a n y

p a r t i c i p a n t s c o u l d b e n e t f r o m a m o r e r i g o r o u s p r o g r a m , a n d e x p e r i e n c e d s t a

m e m b e r s w i l l r e c o m m e n d a c c e l e r a t e d r e g i m e n s f o r p a t i e n t s i n r e l a t i v e l y g o o d

c o n d i t i o n . K n o w i n g w h e n t o b e n d t h e p r e s c r i p t i o n r u l e s , w h i c h r u l e s t o b e n d ,

a n d b y h o w m u c h t o b e n d t h e m i s t h e m a r k o f a n e x p e r t . S i n c e i t i s n o t c o d i -

e d , n o v i c e s m u s t a c q u i r e t h i s k n o w l e d g e b y t r a i n i n g w i t h e x p e r t s a n d o b s e r v i n g

o u t c o m e s o v e r t i m e . W H A T i s d e s i g n e d t o h e l p s t u d e n t s a c q u i r e n u a n c e d s k i l l s

q u i c k l y b y s i m u l t a n e o u s l y e n h a n c i n g t h e i r a b i l i t y t o p r o d u c e a n e x e r c i s e p r e s c r i p -

t i o n \ b y t h e b o o k " a n d t h e i r a b i l i t y t o e v a l u a t e a n d a d j u s t t h a t p r e s c r i p t i o n t o

s u i t i n d i v i d u a l n e e d s .


38/81

2 T h e W e l l w o r k s / H e a r t w o r k s A d v i s o r / T r a i n e r

2 . 1 O v e r v i e w

U p o n e n t e r i n g t h e W e l l w o r k s o r H e a r t w o r k s p r o g r a m , a p a r t i c i p a n t p r o v i d e s

a m e d i c a l h i s t o r y a n d u n d e r g o e s a s e r i e s o f p h y s i o l o g i c a l t e s t s . B a s e d o n t h e

d a t a a n d t e s t r e s u l t s , a s t a m e m b e r o r s t u d e n t d e s i g n s a n e x e r c i s e p r e s c r i p t i o n

a n d p r o v i d e s h e a l t h r e c o m m e n d a t i o n s f o r t h e p a r t i c i p a n t . T h e p r e s c r i p t i o n s a r e

d r a w n f r o m a t a b l e o f p r o g r e s s i v e l y h a r d e r e x e r c i s e r e g i m e n s , s h o w n i n T a b l e 1 .

L e v e l P r o g r a m L e v e l P r o g r a m L e v e l P r o g r a m

0 6 x 1

1 3 x 2 5 4 x 4 9 2 0 @ 7 5 %

2 5 x 2 6 3 x 6 1 0 3 0 @ 5 5 %

3 3 x 4 7 2 0 @ 5 5 % 1 1 3 0 @ 6 5 %

4 3 x 5 8 2 0 @ 6 5 % 1 2 3 0 @ 7 5 %

T a b l e 1 . E x e r c i s e R e g i m e n s b y L e v e l . ( A x B i s A r e p e t i t i o n s a t B m i n u t e s e a c h . A @

B % i s A m i n u t e s a t B % o f m a x i m u m h e a r t r a t e . )

E x e r c i s e r e g i m e n s a r e r a n k e d i n l e v e l f r o m 0 t o 1 2 . P a r t i c i p a n t s a r e a s s i g n e d

a n i n i t i a l l e v e l a n d t h e n e n c o u r a g e d t o m o v e u p t o p r o g r e s s i v e l y h a r d e r w o r k o u t s .

P a r t o f t h e s t u d e n t s ' r e s p o n s i b i l i t y i s t o t e l l p a r t i c i p a n t s h o w f a s t a n d h o w f a r

t h e y s h o u l d m o v e u p . T h e g o a l i s t o g e t e a c h p a t i e n t t o a m a i n t e n a n c e l e v e l a s

q u i c k l y a s p o s s i b l e w i t h o u t o v e r t a x i n g t h e m . T h e m a i n t e n a n c e l e v e l i s 3 0 m i n u t e s

a t 6 5 % o f m a x i m u m h e a r t r a t e f o r W e l l w o r k s p a r t i c i p a n t s a n d 3 0 m i n u t e s a t 7 5 %

o f m a x i m u m h e a r t r a t e f o r H e a r t w o r k s p a r t i c i p a n t s .

T h e c u r r e n t v e r s i o n o f W H A T p r o v i d e s t h e i n i t i a l e x e r c i s e p r e s c r i p t i o n o n l y .

F u t u r e v e r s i o n s o f t h e p r o g r a m w i l l h a n d l e p r e s c r i p t i o n a d j u s t m e n t f o r p a r t i c -

i p a n t s i n m i d - p r o g r a m , a s e x p l a i n e d i n S e c t i o n 2 . 6 . A l t h o u g h W H A T c o u l d b e

u s e d f o r W e l l w o r k s o r H e a r t w o r k s p a r t i c i p a n t s , a s b o t h p r o g r a m s u s e t h e s a m e

p r e s c r i p t i v e p r o c e s s , o n l y H e a r t w o r k s p a r t i c i p a n t s a r e c u r r e n t l y i n c l u d e d i n t h e

c a s e b a s e .

T h e r s t t a s k o f t h e s y s t e m i s t o i d e n t i f y p a r t i c i p a n t s t o o i l l t o p a r t i c i p a t e

i n t h e p r o g r a m . T h i s i s a c c o m p l i s h e d b y a s i m p l e c r i t i c . I f a p a t i e n t h a s c h e s t

p a i n , h e a r t p a l p i t a t i o n s , c o u g h i n g o n e x e r t i o n , e d e m a , s h o r t n e s s o f b r e a t h , l i g h t -

h e a d e d n e s s , c o u g h i n g b l o o d , o r f a i n t i n g , t h e y a r e d i s q u a l i e d a n d r e f e r r e d t o a

d o c t o r . Q u a l i e d p a r t i c i p a n t s p e r f o r m a b r i e f e x e r c i s e t e s t , t h e h a l l w a y - w a l k

t e s t , w h i l e t h e s t a g a t h e r s b a s i c p h y s i o l o g i c a l d a t a . B a s e d o n t h e r e s u l t s , t h e

s t a g e n e r a t e s a n i n i t i a l e x e r c i s e p r e s c r i p t i o n .

2 . 2 T h e E x e r c i s e P r e s c r i p t i o n D e c i s i o n M a k i n g P r o c e s s

T h e p r o c e s s f o r m a k i n g t h e i n i t i a l r e c o m m e n d a t i o n i s s i m p l e a n d c o n s e r v a t i v e :

f o r e v e r y r i s k f a c t o r , l i k e o b e s i t y o r h i g h b l o o d p r e s s u r e , t h e p a r t i c i p a n t r e c e i v e s


39/81

o n e r i s k p o i n t . S t u d e n t s a r e i n s t r u c t e d t o s t a r t p a r t i c i p a n t s w h o h a v e n o r i s k

f a c t o r s a t L e v e l 1 0 a n d p a r t i c i p a n t s w h o h a v e o n e o r m o r e r i s k f a c t o r s a t L e v e l 2 .

U p w a r d a d j u s t m e n t s m a y b e m a d e f o r p a r t i c i p a n t s i n a b o v e a v e r a g e c o n d i t i o n .

G i v e n t h e a g e o f t h e H e a r t w o r k s p o p u l a t i o n a n d t h e f a c t t h a t a l l p a r t i c i p a n t s

h a v

solving safety implications in a case based decision-support system in medicine

Documents