a bayesian network coding scheme for annotating biomedical information presented to genetic...

www.elsevier.com/locate/yjbin

Journal of Biomedical Informatics 38 (2005) 130–144

A Bayesian network coding scheme for annotatingbiomedical information presented to genetic counseling clients

Nancy Green*

Department of Mathematical Sciences, University of North Carolina at Greensboro, Greensboro, NC 27402-6170, United States

Received 13 July 2004Available online 11 November 2004

Abstract

We developed a Bayesian network coding scheme for annotating biomedical content in layperson-oriented clinical genetics doc-uments. The coding scheme supports the representation of probabilistic and causal relationships among concepts in this domain, at ahigh enough level of abstraction to capture commonalities among genetic processes and their relationship to health. We are using thecoding scheme to annotate a corpus of genetic counseling patient letters as part of the requirements analysis and knowledge acqui-sition phase of a natural language generation project. This paper describes the coding scheme and presents an evaluation of inter-coder reliability for its tag set. In addition to giving examples of use of the coding scheme for analysis of discourse and linguisticfeatures in this genre, we suggest other uses for it in analysis of layperson-oriented text and dialogue in medical communication.� 2004 Elsevier Inc. All rights reserved.

Keywords: Content annotation; Content analysis; Intercoder agreement; Bayesian network; Genetic counseling; Clinical genetics; Patient commu-nication; Natural language generation; Natural language processing; Discourse annotation

1. Introduction

As information about human genetics has increasedrapidly in the last few years, so have genetic testing op-tions such as newborn screening for inherited disorders,testing for genetic predispositions to certain types ofcancer, and testing for genetic variations that may deter-mine the effectiveness of certain medications. In theUSA, genetic counselors meet with clients to discusstesting options, risk of complications, interpretation oftest results, diagnosis of inherited conditions, and recur-rence risks. An important function of the counselor is toprovide educational counseling, i.e., to provide informa-tion in terms that are comprehensible to a lay person.The patient letter is a standard document written by a

1532-0464/$ - see front matter � 2004 Elsevier Inc. All rights reserved.

doi:10.1016/j.jbi.2004.10.001

* Fax: +1 336 334 5949.E-mail address: [email protected].

genetic counselor to her client summarizing the servicesand information provided to the client [1].

We are analyzing a corpus of patient letters writtenby genetic counselors to gain knowledge for the designof a computer-supported health communication systemfor clinical genetics. Employing various artificial intelli-gence methods, including natural language generation(NLG) techniques [2], the proposed system will auto-matically produce the first draft of a patient letter usinggeneral information on clinical genetics from a knowl-edge base and documentation about the client�s casefrom her healthcare providers. By generating a firstdraft, the system could save the counselor time, as wellas provide her with easy access to information on relatedresearch and patient resources. Note that it is not thegoal of the proposed system to automate problem-solv-ing tasks such as recommending testing, performingdiagnosis, or calculating risk, but rather to provide sup-port to healthcare professionals in performing patient-tailored communication tasks.

mailto:[email protected].

2 In clinical genetics, genotype refers to the two copies, or alleles, of

N. Green / Journal of Biomedical Informatics 38 (2005) 130–144 131

Many NLG-capable systems synthesize texts in a tar-get language, such as English, starting from a nonlinguis-tic representation of information stored in a knowledgebase (KB); also, the systems may separate the task of dis-course planning, automatically selecting relevant informa-tion from a KB and determining how to organize it intostructural units of discourse, from the task of linguisticrealization, automatically synthesizing sentences to con-vey the selected information [2]. Our analysis of a corpusof clinical genetics patient letters follows a related three-way division. First, we are analyzing the biomedical con-tent of the letters to help design a nonlinguistic KB for thehealth communication system and to provide an abstrac-tion of the letters� contents to facilitate the next two levelsof analysis. Second, we are analyzing (discourse) struc-tural devices in the letters. Third, we are analyzing their(sentential) linguistic features.

Analyzing a corpus of sample documents as one ofthe first steps towards design of the generation compo-nents of a system is a standard methodology in theNLG field for understanding user requirements andcharacteristics of the target genre1. In addition to thegoal of developing this NLG system for genetic counsel-ors, another, broader, objective of our research is toanalyze problems in communication of biomedicalinformation to lay persons and to investigate potentialsolutions for use in NLG systems. For example, chal-lenges in clinical genetics patient communication thatare shared with other areas of medicine include explana-tion of risk and explanation of the diagnostic processand its use of evidential reasoning and statistical data.

For the biomedical content level of analysis of this cor-pus, we have devised a coding scheme for representingcausal and probabilistic relationships among conceptsat a level of abstraction that captures commonalitiesamong genetic processes and their relationship to health.The result of applying the coding scheme to apatient letteris tomodel its content in aBayesian network (BN) formal-ism [3,4]. Although systems using BNs have been devel-oped for automated genetic risk analysis [5] and otherbiomedical decision support applications, e.g., [6–8], weknow of no previous work in which a BN-based approachhas been used for content annotation of text or dialoguecorpora. While the initial motivation for developing thecoding scheme was to support our NLG research efforts,it provides a way of encoding biomedical content in thisgenre that should be useful for future research on patientcommunication in clinical genetics. Furthermore, ourgeneral approach to representing biomedical content ina BN formalism may be useful for applications requiring(manual or automated) analysis of patient-oriented docu-ments in other areas ofmedicine. In this paper, we present

1 While corpus analysis often is useful for knowledge acquisition atthe beginning of an NLG project, it does not obviate the need forsystem evaluation by users [2].

the biomedical content coding scheme and a formal eval-uation of the intercoder reliability of its initial tag set,whichwe found tobe very good.Then,webriefly illustratehow we have used the coding scheme for analysis of thecorpus. In addition, we propose how to automate the con-tent annotation process and explore other uses of the cod-ing scheme.

2. A Bayesian network coding scheme

2.1. Overview

Essentially, the coding scheme consists of a relativelysmall set of tags relevant to clinical genetics and con-straints on their inter-relationships. Although there aremore than 5000 known human genetic disorders [9], withmany different direct and indirect effects, our codingscheme provides a more abstract representation that isbased on commonalities among genetic processes andtheir relationship to health. For example, two completelydifferent genotypes discussed in the letters, such as thegenotype related to Velocardiofacial (VCF) syndromeand the genotype related to Neurofibromatosis (NF),would both be tagged as referring to the more abstractconcept of genotype.2 Also, different health problems,such as a birth defect or a developmental delay, wouldeach be tagged as referring to the more abstract conceptof symptom. In addition to classifying the concepts de-scribed in a letter, the encoding tracks which family mem-ber is described by a tagged phrase. For example, patientletters often discuss a known or hypothesized genotype ofthe mother and father of a patient. The potential causalrelationships among these concepts can be representedin a graphical format as shown in Fig. 1, where arrows de-pict possible direct causal relationships.

The graph shown in Fig. 1 can be viewed as a simpleBayesian network (BN), where each node (depicted as arectangle) of the network graph represents a discrete-valued random variable. The text inside the rectanglesin Fig. 1 includes several types of information. The la-bels genotype and symptom are tags in our codingscheme denoting variable types. A tag appended with anumeral, e.g., genotype-1, is a variable name. (As we dis-cuss later, due to repeated mentions of the same conceptin a text, a BN variable may be identified by more thanone name.) For illustrative purposes, Fig. 1 also showsthe domain of each variable, e.g., 0, 1, or 2 copies of

VCF mutation3. Also, the text inside a rectangle indi-

a particular gene in an individual�s genome. One copy is inherited fromthe mother, and the other from the father. In Mendelian autosomaldominant or recessive disorders, the presence of one or two abnormalcopies of a gene, respectively, may result in health problems.

3 i.e., zero, one, or two abnormal copies of the gene related to VCF.

Fig. 1. Simplified view of concepts and inter-causal relationships.

132 N. Green / Journal of Biomedical Informatics 38 (2005) 130–144

cates which family member (e.g., the patient�s mother) isdescribed by the variable. Directed arcs (depicted as ar-rows) in a BN represent dependencies among randomvariables. For example, the probability that the childhas inherited a copy of the VCF mutation is condition-ally dependent on whether or not his mother and fatherare carriers of the mutation, and the parents� genotypeshave a causal influence on their child�s genotype.

The coding scheme enables an analyst to constructBN graphs representing the biomedical content of a let-ter in the corpus at this level of abstraction. To illus-trate, first an encoded excerpt from a letter in thecorpus is given in Fig. 2. In the figure, numbers in paren-theses on the left are sentence identification numbers.Text in upper case enclosed in square brackets has beensubstituted for original text in places to preserve clientanonymity. Each annotation added by the analyst is

Fig. 2. Excerpt of coder-annotated sentences from letter VCF.

shown in bold font immediately following a left anglebracket. Following current convention in markup lan-guages, the angle brackets enclose the phrase4 in the textto which the tag applies. For example, according to theanalyst�s judgment the phrase Velocardiofacial syndrome

in sentence 4 represents a random variable of typegenotype, to which the analyst has assigned the namegenotype-4. In other words, the tagged phrase isevidence that the letter is about a concept, labeledgenotype-4, that is a subtype of the concept of genotype,where genotype-4 is the pair of alleles in the patient�sgenome that is associated with VCF syndrome.5

Also, as illustrated in Fig. 2, family relationship termsare appended by the analyst following a delimiter sym-bol (�/�) to variable names. In this excerpt, only the childof the client, encoded by the analyst as proband, wasdiscussed.6 However, rather than applying to a specificindividual in the client�s family, a tagged phrase may re-fer to a family member�s counterpart in the general pop-ulation. In the coding scheme, this type of phrase isdually annotated with population and with the familymember annotation that is most relevant to the client�scase. For example, in (5) in Fig. 2, several tags includethe suffix population/proband, indicating that the taggedphrase is about the proband�s counterpart in the generalpopulation, rather than the specific person discussed inthe rest of the excerpt. In addition to assigning familymember and population annotations, the analyst identi-fies which tagged phrases corefer to the same attribute ofan individual. For example, the phrases tagged asresult-7/proband and result-8/proband in Fig. 2 coreferto the proband�s same test result.

A BN diagram constructed from the encoding of thefull letter containing the excerpt given in Fig. 2 is givenin Fig. 3. This BN diagram shows only variables that re-fer to the client�s specific family. Another BN diagramwould be constructed to show the variables bearingthe population annotation, as in sentence (5). Also, notevery tag in in Fig. 2 referring to the client�s familyappears as a separate node in the BN diagram; e.g.,result-7/proband and result-8/proband do not appear asseparate nodes in Fig. 3 since in the analyst�s judgment

4 The coder is instructed to place the closing (right) angle bracketimmediately after the syntactic head of a tagged phrase; thus, post-head modifiers such as prepositional phrases and relative clauses willnot appear inside the brackets even though they are understood ascontributing to the specification of the tagged concept.

5 In the corpus as well as in the related literature, we have observeduse of the same string of words to refer to a kind of symptom or to thegenotype associated with that symptom; thus, an analyst may need totake context into consideration when deciding which of the tags isappropriate.

6 Proband is a term in genetics that refers to an individual ofinterest, i.e., in this context, the client�s child. Following convention ingenetics, in the coding scheme family relationship terms such as mother

and brother always are specified relative to the proband.

Table 1BN tags in coding scheme

Tag Brief description Initial set Extended set

History Demographic or predispositional factor (e.g., gender, ethnicity, age,family history of specific mutation, environmental risk factor)

+ +

Genotype A pair of alleles of a gene + +Karyotype The full set of chromosomes in an individual�s cells (normally 23 pairs of chromosomes) � +Mutation-event Event that results in a change in an individual�s genotype or karyotype

at different points in his/her life cycle and/or in different cells+ +

Biochemistry Manifestation of an individual�s genotype at the biochemical level, e.g., a shortened protein + +Physiology Manifestation of an individual�s genotype at the physiological level, e.g., in metabolism + +Symptom Manifestation of an individual�s genotype that is observable without

performing a test, e.g., some types of birth defects+ +

Test Event of performing a test + +Result Result of test + +Complication Undesirable side-effect of test, e.g., fetal damage � +Probability Quantitative or qualitative probability value + +

Fig. 3. A BN diagram for letter VCF. Solid arcs represent BN dependencies. UML notation represents semantic dependencies (group–member, type-subtype, association). Reprinted from author�s coding manual [10].


they corefer with the concept tagged as result-1/probandthat has been used as a node label. Associated with eachnode is a list of coreferences giving the identificationnumbers of coreferring variables, e.g., the list forresult-1/proband in the figure indicates that it coreferswith result-7/proband and result-8/proband.

Figs. 2 and 3 include some variable types and othernotation that have not been discussed yet but which willbe covered in the rest of Section 2. For a completedescription of the coding scheme, see [10].7

7 For example, the manual describes the constraints on dependen-cies between variable types.

2.2. Tag set

Table 1 gives the full set of BN tags in the codingscheme. The initial tag set was intended to cover sin-gle-gene autosomal disorders (i.e., non-sex-linked disor-ders following Mendelian inheritance patterns),representing over 4500 diseases and affecting 1% of thepopulation [9]. The initial tag set was developed basedon qualitative analysis of three letters from the corpusand clinical genetics textbooks. An evaluation of theintercoder reliability of the initial tag set is given in Sec-tion 3. The extended tag set (the initial set and two addi-tional tags) is designed to cover chromosomal and


multifactorial disorders, which represent more than 100diseases each, and which affect 0.5% and 5–10% of thepopulation, respectively [9]. The extended set was devel-oped based on qualitative analysis of four more lettersfrom the corpus (two on chromosomal disorders andtwo on multifactorial disorders) and textbooks, buthas not yet been formally evaluated for intercoder reli-ability. All of the tags except probability, whose role isdiscussed in Section 2.3, represent BN variable types.

As stated above, the tag set was designed to cover awide range of genetic disorders but at a level of abstrac-tion that enables regularities in communication to bestated independently of particular genes, types of muta-tions, proteins, etc. We are not claiming that this level ofabstraction is sufficient for representation of content inthe genetics or biomedical research literature, but it doesseem sufficient for our purposes, i.e., for the generationof information about human genetics and its relation tohealth as portrayed to the layperson, e.g., in patient let-ters.8 The tags were chosen with the goal of providingenough categories to be useful for describing patient-tai-lored communication without compromising intercoderreliability. For example, two categories, finding andsymptom, were merged into one, symptom, after infor-mal testing suggested that coders would have difficultyin reliably discriminating the two classifications.

Note that the BN models developed for applicationsin medicine, e.g., [5–8], differ in several respects frommodels constructed using our tag set. First, the Bayesiannetworks (i.e., the particular nodes and topology) inthose systems were designed for medical reasoning tasksrather than for analysis of communication; thus, the de-sign of those BNs was subject to different constraints,including efficiency constraints. Second, the developersof those systems were not constrained as we are by thegoal of intercoder reliability. Despite the apparent sim-plicity of our coding scheme, the BN graphs constructedfrom letters in our corpus using this tag set sometimesdemonstrate a high degree of complexity of content tobe communicated to the layperson.

2.3. Probabilistic and causal dependencies

A Bayesian network is defined formally as a directedacyclic graph whose nodes represent discrete-valued ran-dom variables and whose arcs represent direct depen-dencies between variables; the relationship betweendirectly connected nodes is described by a conditionalprobability distribution given with each node in a condi-tional probability table (CPT) [4]. For example, in a

8 In a pilot study reported in [11], we coded three letters andcompared the total number of BN tags assigned (51) to sentences (102),which gives a rough idea of the high degree of relevance of the tag setto the letters.

fully specified BN with the structure shown in Fig. 1,the CPT associated with genotype-3, would give

P(genotype-3|genotype-1, genotype-2)

for all values of genotype-1 and genotype-2, as in Table2. (The values in this case are based on a Mendelianmodel of autosomal recessive inheritance. In general,CPT values also may be supplied by domain expertsor based on epidemiological data.) In addition to prob-abilistic relationships, BN dependencies often corre-spond to causal relationships. In Fig. 1 for example,all of the arcs may be interpreted causally.

Although it is not possible to acquire most of the nu-meric probability values required to fully specify a CPTfrom a typical letter in the corpus, the letters do providesome of this information. Usually, however, mainlyqualitative descriptions of probability values are givenin the letters, e.g., the expression (�could�) tagged asprobability-8 in (8) of Fig. 2. Also, while a CPT onlyspecifies probabilities in the form P(Y|X1, . . . ,Xn), whereX1, . . . ,Xn are the direct predecessors of Y in the BN,letters in the corpus express other related conditionalprobability statements. For example, (8) in Fig. 2 couldbe analyzed as conveying the conditional probability

P(genotype-8|symptom-8, result-8),

which (although related by Bayes theorem) is not equiv-alent to

P(symptom-8, result-8|genotype-8).

When annotating a letter with the tags shown in Ta-ble 1, a coder also constructs BN diagrams depictingrelationships among variables in the letters. Reflectinga layperson-oriented model of clinical genetics, the cod-ing scheme specifies the dependencies between the vari-ous node types. All dependencies are probabilistic andmost but not all are causal. For example, a history nodefor an individual I may be directly linked to a karyotype

node for I; such an arc is probabilistic but not causal,i.e., as the age of the mother increases, the risk of chro-mosomal abnormalities increases, but age does notcause the abnormalities.

The formal properties of a BN determine the updatedposterior probability of any of its variables as evidenceis acquired about other variables. As healthcare profes-sionals involved in a case gather evidence, the posteriorprobabilities of hypotheses about the diagnosis or pre-dictions may change. Thus, a BN can be used to modelthe process of diagnosis and prediction of health effectsand recurrence risks that are reported in patient letters.Initially, a BN (which we term BNpopulation) can be usedto model background knowledge relevant to a case as re-ported in a letter. For an example of BNpopulation, see

Fig. 4. Other BN diagram for letter VCF. Reprinted from author�s coding manual [10].

Table 2P(genotype/proband | genotype/mother, genotype/father) according to a Mendelian inheritance model

Father: 0 copies Father: 1 copy Father: 2 copies

Mother: 0 copies 0 copies 100% 0 copies 50% 0 copies 0%1 copy 0% 1 copy 50% 1 copy 100%2 copies 0% 2 copies 0% 2 copies 0%

Mother: 1 copy 0 copies 50% 0 copies 25% 0 copies 0%1 copy 50% 1 copy 50% 1 copy 50%2 copies 0% 2 copies 25% 2 copies 50%

Mother: 2 copies 0 copies 0% 0 copies 0% 0 copies 0%1 copy 100% 1 copy 50% 1 copy 0%2 copies 0% 2 copies 50% 2 copies 100%

9 Strictly speaking, the test node type in our coding scheme is a kindof intervention variable [4].


Fig. 4, which was constructed by a coder just from vari-ables affixed with population derived from the same let-ter as the one from which Fig. 3 was constructed.Another BN (which we term BNpre-test) can be used tomodel the beliefs of the healthcare professionals aboutthe case after revising BNpopulation based upon all obser-vations made prior to testing reported in the letter, e.g.,the proband�s history and physical findings. AnotherBN (which we term BNpost-test) can be used to modelthe revised beliefs of the healthcare professionals afterthe test results reported in the letter were available.

In addition to BN arcs, an analyst may represent sev-eral types of possible semantic relationships betweenconcepts in the diagram. For example, in Fig. 3, a linerepresents the relationship between test-4.2 andgenotype-4, which we term semantic association (not tobe confused with statistical association). In other words,

we are not interested in modeling the likelihood of a test,given genotype; instead, we wish to model which testenables a genotype to lead to a particular result.9

Semantic notation is used to represent group–member

relationships when a letter contains text referring toindividual symptoms and other text referring to the col-lection of these symptoms; for example, in the letter de-picted in Fig. 3, the symptoms tagged as symptom-2.1

and symptom-2.2 comprise the group of symptomstagged as symptom-8. Also, semantic notation is usedto represent type-subtype relationships; e.g., in the letterdepicted in Fig. 3, genotype-2 (the set of genotypestested) is interpreted by the analyst as a subset of geno-


type-8 (the set of all genotypes that could be responsiblefor the proband�s symptoms).

Although each relationship shown in a BN diagramcould be encoded just in written notation by the analyst,we have found that a pictorial rendering of the informa-tion is useful.10 A BN diagram provides the analyst witha concise pictorial representation of what types of con-cepts and what probabilistic and (sometimes) causalinterrelationships among them the writer intended tocommunicate. In using a variable numbering scheme re-lated to text position and in including coreferring vari-able names in the diagram, the diagram enables theanalyst to view the order of presentation of conceptsand the frequency with which concepts were covered.It also provides the analyst with a visualization of therelative complexity of different text segments.11

2.4. Related work in natural language generation and

discourse analysis

There has been previous NLG research on generationof arguments from domain knowledge and user modelsrepresented in Bayesian networks [12,13]. However,those projects did not develop or use BN based codingschemes for analyzing communication in the targetgenre. Also, the BNs used in those projects were not de-signed to model clinical genetics.

In the last decade, there has been considerable inter-est in development of coding schemes for discourse anal-ysis, such as schemes for classification of speech acts indialogue (e.g. [14]), subjectivity and affect in newspaperarticles (e.g. [15]), and rhetorical moves in scientific arti-cles [16]. However, none have addressed the encoding ofan underlying biomedical domain model as we have. Acommon methodological concern in research on dis-course analysis has been the formal evaluation of thereliability of the coding schemes used. We address thisissue with respect to our coding scheme in the nextsection.

3. Evaluation of intercoder reliability

3.1. Overview

In this section, we report the results of a formal eval-uation of the intercoder reliability (reproducibility) ofthe initial tag set, which covers single-factor autosomaldisorders. Our goal was to measure to what extent, givensufficient training, different coders would agree on therepresentation of the content of a genetic counseling pa-

10 For a discussion on automatic construction of the BN diagrams,see Section 4.3.

11 In Section 5.2, we discuss how complexity metrics might bedefined in terms of the BN representing a text.

tient letter using our coding scheme. Verifying inter-coder reliability is important to ensure that the corpuswill be annotated consistently.12 A basic measure ofintercoder agreement on classification tasks in computa-tional linguistics was used for this purpose, the Kappacoefficient [17].

3.2. Coders and training procedure

A coding manual [10] was written for training pur-poses by the author following a pilot study of the codingscheme [11]. Two paid coders (R and S) had just com-pleted the first year of study in a Master of Science inGenetic Counseling program. They each received a totalof 20 h of training. Training included an orientationmeeting, self-study of the coding manual, practice cod-ing of two sample letters, and two other meetings to dis-cuss the practice coding exercises. The third coder (K), agraduate student in computer science with an under-graduate degree in biology, had acquired expert knowl-edge of the coding scheme through work with the authorduring the writing of the coding manual. That work in-cluded reviewing the manual for clarity and consensuscoding with the author of several letters used for codertraining.

3.3. Testing procedure

Information about the letters used in development ofthe coding scheme, training, and the evaluation (testing)is shown in Table 3. The four letters from the corpusused for the formal evaluation (a total of 238 sentences,or 4670 words) are identified in Table 3 as GL, OI, MN,and AC. These letters had not been read by the authorof this paper or by the coders before the evaluation.

The coders used standard desktop computer pro-grams to insert annotations into the letters and to drawBN diagrams. The coders were instructed that theycould consult the coding manual, coded letters used intraining, or genetics reference materials as needed, butnot to discuss the work with each other. They were al-lowed to take as much time as needed and were askedto keep track of time spent on each letter. The total timereported by each of the genetic counseling student cod-ers (R and S) was 18.25 and 17.5 h.

3.4. Segmentation issues

To compute intercoder agreement on the annotationof the text, it is necessary to count how many times apair of coders assigns the same tag to the same unit (seg-ment) of a text. A potential problem is that two codersmay agree that a sentence in a letter conveys the same

12 It is also needed to provide a standard against which automatedtagging results can be compared.

Table 3Letters used in developing the coding scheme, for coder training, and as the test set of the intercoder agreement evaluation

Letter Disorder Author Sentences (words) Use

VCF Velocardiofacial syndrome A 24 (446) Development/trainingNF Neurofibromatosis Baker et al. 38 (814) Development/trainingHL Hearing loss A 40 (756) Development/trainingCF Cystic fibrosis Constructed Training onlySC Sickle cell anemia Constructed Training onlyGL Galactosemia B 68 (1342) Test setOI Osteogenesis imperfecta A 46 (948) Test setMN Marfan syndrome B 76 (1537) Test setAC Achondroplasia C 48 (843) Test set

‘‘Constructed’’ letters, used only for training, were written by the author of this paper and coder K. Authors A, B, and C are professional geneticcounselors. Letter NF appeared in [1].

Table 4Kappa for segmentation (95% confidence limit)

Letter OI GL MN ACCoders R and S

Kappa 0.6720 0.7230 0.7102 0.7311Lower 0.5930 0.6527 0.6360 0.6520Upper 0.7510 0.7933 0.7843 0.8101

Coders K and S


Coders K and R



concept with reference to the same individual, but maynot place brackets for the tag around the same stringof words in the sentence. For example, one coder mayplace angle brackets around a region of the sentence thatoverlaps with the region bracketed by the other coder.In other cases, each coder may assign the tag for thesame concept to one of two non-overlapping regions.In either case, the coders essentially agree on the corre-sponding concept in the BN, which is what we wouldlike the evaluation to reflect. To give an example, con-sider sentence (20) from letter HL: ‘‘One allele for theGJB2 gene is inherited from our mother while the otherallele is inherited from our father.’’ While the readerprobably would agree with us that the sentence discussesthe three concepts, the GJB2 genotype of a mother,father, and child, one coder may tag our mother, whileanother coder may tag one allele, one allele of the

GJB2 gene, or the GJB2 gene, as the segment referringto the mother�s genotype; similar problems may arisein deciding where to place tags in the sentence for seg-ments referring to the genotypes of the father and child.

As Carletta et al. [14] note13, when one coding judg-ment (such as assigning a tag in our coding scheme toa segment) depends on another (in our case, choosingthe string of words to bear the tag) the second judgment‘‘cannot be reasonable unless the first also is’’ (p. 25).Therefore, following Carletta et al.�s approach, wedecided to measure the intercoder agreement of ourtag set only for those text segments tagged by both cod-ers. To identify segments tagged by both coders, first,the author of this paper segmented each sentence in aletter based on syntactic criteria into minimal syntacticconstituents (MSC). Then we identified which MSCwere tagged by both coders. Only those segments taggedby both were used in the evaluation described in the nextsection.

Before reporting the main results in the next section,in the rest of this section we provide supplementary

13 They attribute this point to Krippendorf [18].

information on an analysis of the extent to which codersagreed on which particular segments (MSC) in a sen-tence to tag. To do so, we constructed a contingency ta-ble for each letter representing agreement onsegmentation. For example, coders R and S tagged thesame 80 segments in letter OI. (Those 80 segments werethe only segments used for the intercoder reliability eval-uation of the tag set described in the next section forthat pair of coders for that letter.) The Kappa statisticfor segmentation was then computed from these contin-gency tables for all letters (see Table 4). Although thesegmentation results indicate a level of agreement whoselower range is in some cases below the current norm14 incomputational linguistics, the lower range would be con-sidered to be moderate to substantial by medicalresearchers [19].

3.5. Intercoder reliability of tag set

Having identified the segments as described in thepreceding section, we proceeded to compute the Kap-pa statistic for the tag set by counting the number of

14 Following Krippendorf [18], over 0.8.

Table 5Kappa for initial tag set (95% confidence limit)

Coders R and S

Letter OI GL MN AC(Segments) (80) (88) (77) (72)Kappa 0.9220 0.9168 0.9426 0.9791Lower 0.8562 0.8540 0.8783 0.9382Upper 0.9115 0.9414 1.0069 0.9691

Coders K and S


Coders K and R


The number of segments tagged by both coders (i.e., the number onwhich Kappa was computed) are given in parentheses.


times each pair of coders assigned the same tag andfamily member designator (e.g., proband, mother, etc.)to the same segment, for segments that both coderstagged.15 The final results, shown in Table 5, indicatethat the initial tag set can be applied with a high de-gree of intercoder agreement, ranging between 0.8290and 1.16

Fig. 5. Role of corpus annotation in development of NLG system.

4. Uses of coding scheme

Fig. 5 depicts the role of the annotated letters in ourprocess of NLG system development.17 As mentionedin Section 1, our goal in annotating the biomedical con-tent of letters in the corpus is to help design the KB andto analyze the structural (discourse-level) devices andlinguistic (sentence-level) features of this genre. Thecoded letters and other sources of information are beingused to design the KB as a BN.18 The role of the KB inthe system is to store information in a way that enablesthe system to model the reasoning of the genetic coun-selor when the system is composing the first draft of the

15 For segments that were annotated with more than one tag, eachinstance of agreement was counted separately.

16 This level is characterized as almost perfect on Landis and Koch�s[19] scale and as reliable in Krippendorf�s [18] scale.

17 Subprocesses are denoted by rectangles. Currently, coding andanalysis are performed manually, but may be partially automated asdiscussed in Section 4.3.

18 Technically speaking, the KB is implemented as a hybrid BN/qualitative probabilistic network (QPN) [20] to avoid the requirementto acquire numerical probability values for all variables in the network,and since in many cases only qualitative assessment of probability isrequired [21].

patient letter for the counselor. For example, supposethat the counselor informs the system that the resultof the proband�s test for a change in the gene GJB2 ispositive and that the clinic�s diagnosis is that thischange is likely to be responsible for the proband�shearing loss. Now, in composing the first draft of thepatient letter, the system can use its generic causal mod-el in the KB for this type of hearing loss to compose aclient-tailored explanation of the bearing of the evi-dence from the test results on the diagnosis. Also, thesystem could include additional information from theresearch literature stored in the KB related to thediagnosis.

The purpose of performing an analysis of discourse-level devices and linguistic features of this genre is two-fold: to ensure that the letters drafted by the system areconsistent with current practice in terms of topics, orga-nization, writing style, etc.; and to provide the systemwith knowledge that it can generalize to draft letterscovering many scenarios, not just those represented inthe corpus. Since the biomedical content coding schemecaptures commonalities among genetic processes and


their relationship to health, it enables structural and lin-guistic regularities to be stated at a higher level ofabstraction than in terms of specific terminology (or spe-cific genes and health problems under discussion) foundin the corpus. This facilitates generalizing communica-tion strategies found in the corpus to new scenarios,i.e., other genetic disorders. Examples are given in thenext two sections.

4.1. Analysis of discourse

Genre analysis identifies major discourse structures,called moves, that characterize a genre [22]. For exam-ple, our analysis of the corpus has identified a very com-mon move, occurring after the opening of a letter, whosecommunicative goal is to document the diagnostic pro-cess. The steps of this move, which we call Document

Diagnostic Process, track successive transformations ofthe BN, starting from an initial state (BNpopulation)describing the general population based on epidemiolog-ical studies and probability ‘‘laws’’ of Mendelianinheritance, and ending in a state that reflects allpatient-specific observations including test results(BNpost-test). For example, this move may include thefollowing steps:

� Describe diagnostic reasoning leading to initial set ofcandidate diagnoses after BNpopulation has beenupdated with symptoms and history observed byreferring doctor (resulting in BNpre-test).

� Describe diagnostic reasoning leading to final set ofcandidate diagnoses after BNpre-test has been updatedwith test results (resulting in BNpost-test).

Note that the moves are described at the level ofabstraction of the coding scheme, rather than in termsof specific symptoms, tests, and genotypes mentionedin particular letters in the corpus, so that they can be ap-plied to new scenarios in this domain.

In addition to defining moves required for the currentapplication, we have been analyzing the encoded corpusto identify general forms of causal argumentation usedin patient-tailored communication. Knowledge gainedfrom the analysis will contribute to our broader researchgoals of developing computer systems to help lay audi-ences to understand medical and other scientific infor-mation [23].

4.2. Analysis of linguistic features for communication of

uncertainty

To understand better how probability is conveyed inthis lay-oriented genre, we are using the coding scheme�stag set to encode the probability statements given ineach letter. For example, (5) in Fig. 2 could be encodedas

P(symptom-5.1, symptom-5.2|genotype-5) =probability-5,

where probability-5 is a qualitative description of fre-quency (�often�).

This is an example of a retrospective probability state-ment, which provides the frequency of an observablecondition (e.g., symptom-5.1) for a population in whichthe cause is known (e.g., genotype-5). Eddy [24] distin-guishes retrospective from predictive probability state-ments, in which the likelihood of a hypothesis is statedgiven knowledge of observables. For example, (8) inFig. 2 could be analyzed as conveying the predictiveprobability statement

P(genotype-8/symptom-8, result-8) =probablity-8,

where probability-8 is a qualitative description of likeli-hood (�could�).

Typically, retrospective statements are the formfound in the research literature (and correspond to theform of the conditional probability table of a BN),whereas predictive statements are in a form that maybe more useful to a patient interested in his diagnosis.In a pilot study reported in [11], we found a surprisinglyhigh proportion of probability statements to sentencesin a sample of three patient letters from the corpus.Moreover, in those three letters the ratio of retrospectiveto predictive statements ranged from roughly equalnumbers of each type to roughly half as many retrospec-tive as predictive statements. Since even experts mayconfuse retrospective and predictive probabilities [24],it is important to understand how they are used andthen, perhaps, to offer the writer an alternative formula-tion in the generated first draft of the patient letter.

In addition, we classify probability statements in thecorpus as progressive when the order of presentation ofvariables in the text matches the causal direction in theBN, or regressive when the variables are given in the re-verse of the causal direction. For example, (8) in Fig. 2would be classified as regressive. We are interested intracking this aspect of usage since presentation ordermay influence comprehensiblity. In the same pilot study,we found the ratio of regressive to progressive state-ments to range from roughly equal to roughly half asmany regressive as progressive forms.

In addition to linguistic expression of uncertainty, weare using the encoded corpus to facilitate analysis ofother linguistic features [25]. In particular, we have iden-tified different perspectives that may be signaled bychoice of wording and syntactic variation. For example,parts of a letter included for purposes of medical docu-mentation and which reflect the referring doctor�s per-spective are characterized by use of medicalterminology. However, parts of a letter reflecting the


educational counseling perspective are characterized byuse of non-technical and (so-called) non-stigmatizinglanguage (e.g., �alteration� instead of �mutation�). Thus,the encoded corpus can be used to study the differentways in which the same kinds of information may beconveyed for different purposes.

4.3. Automating tagging

Although the coding scheme was created initially formanual analysis of our corpus, in the future we wouldlike to automate the process as much as is feasible.19

In this section, we outline a proposal for automatingthe tagging process for documents in this genre. The firststep would be to identify biomedical terms in the docu-ment using supplementary resources, some of which weshall describe briefly here.

The goal of the Gene Ontology Consortium (GOC)[26] project is to develop controlled vocabularies forthe description of genes and gene products to supportcross-database queries in collaborating databases. Theproject�s three controlled vocabularies describe geneproducts in terms of biological processes, cellular com-ponents, and molecular function. The vocabularies areorganized as directed acyclic graphs, whose arcs repre-sent semantic dependencies.

The National Library of Medicine�s Unified MedicalLanguage System (UMLS) provides knowledge sourcesfor biomedicine and health-related concepts [27]. TheUMLS Metathesaurus is a comprehensive vocabularydatabase; all of its concepts are assigned a semantic typein the UMLS Semantic Network, containing 135 seman-tic types and 54 relationships. In addition, UMLS pro-vides SPECIALIST, a lexicon of general Englishwords and biomedical terms, and associated lexical pro-cessing computer programs.

To compare our coding scheme to these resources,our tag set can be viewed as a subset of possible conceptsin the UMLS Semantic Network, and a tagged phrasefrom a patient letter could correspond to a term in aGOC or UMLS thesaurus. Also, the non-BN relations(association, group–member, and subtype) encoded inour approach are semantic relationships. However, akey difference is that our approach is based on a BN for-malism. Thus, it is possible to model causal and proba-bilistic dependencies supporting causal probabilisticinference patterns.20 Another key difference is that ourtag set has been designed for and demonstrated to havegood intercoder reliability.

19 Coding letters manually is time-consuming and limits the size ofthe corpus that is feasible to study.

20 For a discussion of types of inferences supported by Bayesiannetworks, see e.g., [4]. Examples include reasoning from cause to effect(prediction), from effect to cause (diagnosis), and explaining away. Incontrast, semantic networks support taxonomic inference.

Nevertheless, these resources could be invaluable inautomating the tagging process. As noted above, thefirst step would be to identify biomedical terms in thedocument using supplementary resources such as theUMLS Thesaurus. Next, the recognized biomedicalterms could be associated with their superordinate cate-gories in the UMLS Semantic Network. Hand-craftedrules could be specified mapping the superordinate cat-egories to the tags in our coding scheme. Thus, it seemsit will be relatively straightforward to automate this as-pect of tagging.

However, what if the goal were to automatically de-rive a BN diagram representing the content of the doc-ument? After tagging the type of variable, thefollowing issues would still need to be addressed:

� Deciding to which member of the patient�s family (orcounterpart in the general population) a taggedphrase refers. This would require natural languageprocessing, e.g., to identify possessive modifiers andto resolve anaphoric references. To give an exampleillustrating the challenge in automating this step, con-sider the problem of tagging sentence (6) in letter HL:‘‘As with most children with genetic hearing loss,[PROBAND] has no unusual features and has whatis called �isolated� or �non-syndromic� hearing loss.’’The subordinate clause, as with most children with

genetic hearing loss, indirectly conveys that the actualpatient is a child with hearing loss, though directlyreferring only to a non-specific child with hearingloss. Conversely, the main clause, [PROBAND] has

no unusual features and has what is called �isolated�or �non-syndromic� hearing loss, directly refers to fea-tures of the actual patient but indirectly refers tothose same features in the generic child. The codingshould reflect that the sentence describes analogousconcepts with reference both to the actual patientand his generic counterpart. Thus for example,according to our coding manual, a coder should tagthe subordinate clause as follows: As with<probability-6 most><history-6.1/population/proband//history-6.2/proband children> with<genotype-6.1/population/proband//genotype-6.2/proband genetic><symptom-6.1/population/proband//symptom-6.2/proband hearing loss>.21

� After family relationships have been determined,building the network diagram from the tagged text.This would be straightforward to do using the con-straints specified by the coding scheme.

21 To explain notation used in the example, when the same phrase isassigned more than one tag, the delimiter �//� is used to separate thetwo tags.


In conclusion, our short-term strategy is to do auto-mated tagging of variable types, then use a human coderto verify the tags and assign family member descriptors,and then do automated BN diagram construction fromthe tagged text.

4.4. Related work on automatic BN construction from

document collections

Although having different goals from ours, there hasbeen research on automatic construction of Bayesiannetworks from electronic document collections for bio-medical decision support and information retrieval(IR) applications. Antal et al. [28] partially automatedconstruction of a knowledge base to be used in a deci-sion support system for classification of ovarian tumors.Starting with a BN graph of 31 variables created bymedical experts, the system applied text mining methodsto a document collection to derive the conditional prob-ability tables of the BN as an alternative to the labor-in-tensive process of eliciting the probabilities from domainexperts. The goal of our work described in the precedingsection, however, is not to automatically construct thesystem knowledge base (labeled KB in Fig. 5) from thecorpus, but to automate the process of analysis of eachletter as much as possible.22

De Campos et al. [29] describe the automatic con-struction of a BN representing collocational relation-ships among terms in a document collection. The BNwas used as a thesaurus for query expansion to improverecall in IR tasks. Their approach is based on theassumption that terms frequently found in the same doc-uments as terms in the original query should be added toa query to increase the probability of finding documentsrelevant to the user�s information need. The resultingstructure and conditional probabilities of a BN con-structed for their goals is different from a BN suitablefor representing the diagnostic reasoning, inheritancerelationships, and recurrence risks discussed in a letterfrom our corpus.

5. Other applications

5.1. Analysis of genetic counseling dialogue

Although our coding scheme was developed for anal-ysis of biomedical content in text, in this section webriefly explore its application to dialogue.

Two coding schemes have been proposed in the geneticcounseling literature for the analysis of genetic counselingsessiondialogue. The goal of those researcherswas to pro-

22 Nevertheless, analysis of the corpus will contribute to the designof our KB, since we would like the KB to model the variables andcausal relationships discussed in the letters.

vide tools for quantitative analysis of factors influencingcounseling effectiveness. Kessler [30] used his proposedscheme to analyze a transcript of a genetic counseling ses-sion (first published in [31]). To each interaction unit inthe transcript,Kessler assigned one of 12 categories devel-oped by Bales [32], a content descriptor, and one of nineattribution scores. Bales� classifications are based on lin-guistic form (e.g., asking or giving a suggestion), affect(e.g., showing antagonism or laughter), and politeness(e.g., interactions raising the other�s status). Kessler didnot describe how content descriptors were assigned or ifthey had been evaluated for intercoder reliability; exam-ples given include amniocentesis, Down syndrome, family

history, genetics and genetic risk. The attribution scoreindicates the relative roles of the participants in an inter-action; e.g., an interaction originating from the counselorand directed towards the husband but not the wife, whoalso was present at the session.

Liede et al. [33] developed and formally evaluated theManchester Observation Code (MOC), a coding schemefor analyzing the counselor�s statements in transcripts ofvideotaped genetic counseling sessions. Each of thecounselor�s statements is classified on four dimensions:grammatical form (e.g., declarative sentence vs. openquestion), function, i.e., the counselor�s reason for mak-ing the statement (e.g., to elicit vs. to give information),content (using seven tags such as genetics vs. social andfamily factors), and cue source (e.g., whether the patientor counselor initiated the topic). Intercoder reliability ofMOC was evaluated formally.

The above investigations followed a simpler ap-proach to representing dialogue content than what wehave adopted in our analysis of patient letters. It is pos-sible that future studies of counseling dialogue effective-ness would benefit from use of our coding scheme. Thus,as an experiment, we informally analyzed the transcriptof a genetic counseling session published in [31] andpresent a BN diagram for part of it in Fig. 6. Despitethe length of the transcript, the BN diagram depictsthe informational content of the session concisely. Withadditional annotation, such as coreference lists andspeaker, the diagram presents the analyst with a visual-ization of the flow of information during the session.23

5.2. Analysis of complexity and comprehension

It is possible to define a number of complexity metricsin terms of this coding scheme for use in studies of com-municative effectiveness. For example, the complexityscores of two texts might be the same according to met-rics based upon vocabulary and syntactic properties.However, one text might be more complex than theother for a variety of reasons that would be reflected

23 Because of the length of the transcript, we have not includedcoreference lists in the drawing.

Fig. 6. BN diagram for part of preamniocentesis counseling session transcript from (Kessler 1981).


in differences in BN diagrams representing the two textssuch as the lengths of causal paths discussed or the num-ber of alternative causal paths discussed. Furthermore,two texts may be about information represented by thesame BN diagram, but one text may be more complexthan the other because of how the information is con-veyed: e.g., a text presenting information in regressiveorder (in reverse of causal order), or presenting a recur-rence risk to the reader as a predictive probability state-ment rather than as a retrospective probabilitystatement, or presenting a diagnosis to the reader as aretrospective probability statement rather than as a pre-dictive probability statement.

Also, it is possible to define comprehension metrics interms of the coding scheme. For example, one way oftesting comprehension in a controlled experiment wouldbe to ask the reader to draw, in essence, a BN represent-ing the causal chain of concepts described in a letter orin a genetic counseling session and comparing that tothe BN constructed by a coder from the letter or sessiontranscript. This would be a way of assessing whether thesubject only understood and remembered isolated facts,or whether he had acquired a deeper understanding ofthe causal, probabilistic model underlying the informa-tion that was presented.

6. Conclusions

We developed and evaluated a coding scheme forclinical genetics patient letters that enables their biomed-ical content to be modeled in a Bayesian network for-malism. An advantage of representing this content in aBN formalism is that it enables one to model causaland probabilistic dependencies and inferences. The cod-ing scheme is at a high enough level of abstraction tocapture commonalities among genetic processes andtheir relationship to health. We are using the codingscheme to analyze a corpus of patient letters writtenby genetic counselors, as part of a natural language gen-eration project to create a computer system to help thecounselors with client communication tasks. The en-coded letters are being used to help design the systemknowledge base, which models the knowledge and rea-soning of a genetic counselor as she writes the first draftof a patient letter. Also, the encoded letters are beingused to identify discourse-level devices and linguisticfeatures of this genre. The discourse-linguistic knowl-edge gained is being used to help ensure that lettersdrafted by the system are consistent with current prac-tice, and to enable the system to draft letters coveringmany scenarios, not just those represented in the corpus.


The coding scheme was designed with several goals inmind.We selected tags representing concepts at a high en-ough level of abstraction to enable generalizations aboutcommunication strategies to be stated independently ofparticular genes, proteins, health problems, etc. Anothergoal was to model the causal and probabilistic model ofhuman genetics presented to a lay audience. Finally, thetag set has good intercoder agreement, as demonstratedby a formal evaluation reported in this paper. Althoughexpensive, verifying intercoder agreement is importantnot only for ensuring that a corpus will be annotated con-sistently by human coders, but also to provide a reliablestandard to which automated tagging results can be com-pared. We described a proposal for automating the tag-ging process in this paper.

This approach enables a representation of biomedicalcontent in written and spoken communication to layper-sons that goes beyond informal topic identification oreven a more sophisticated mapping of phrases to termsin a controlled language. For example, we showedhow this approach enables us to analyze the expressionof probability statements in the corpus in terms of retro-spective-predictive and regressive-progressive distinc-tions. In addition, we suggested how this approachenables new metrics of complexity and comprehensionto be defined. Although developed for lay audience-tai-lored communication in clinical genetics, this approachmay be useful in analysis of lay audience-tailored com-munication in other medical fields as well.

Acknowledgments

This material is based upon work supported by theNational Science Foundation under Career Grant No.0132821. We wish to thank Karen Jirak, JeutonneBrewer, and several students and faculty of the UNCGGenetics Counseling Program for their assistance duringdevelopment and evaluation of the coding scheme; ScottRichter and Ronnie Smith for consultation on the de-sign of the intercoder reliability evaluation; and theanonymous genetic counselors for providing letters forthe corpus.

References

[1] Baker DL, Eash T, Schuette JL, Uhlmann WR. Guidelines forwriting letters to patients. J Genetic Couns 2002;11(5):399–418.

[2] Reiter E, Dale R. Building natural language generation sys-tems. Cambridge, UK: Cambridge University Press; 2000.

[3] Pearl J. Probabilistic reasoning in intelligent systems. San Mateo,CA: Morgan Kaufmann; 1988.

[4] Korb K, Nicholson AE. Bayesian artificial intelligence. BocaRaton, Florida: Chapman & Hall/CRC; 2004.

[5] Szolovits P, Pauker SP. Pedigree analysis for genetic counseling.In: Lun KC et al., editors. MEDINFO 92: Proceedings of theSeventh Conference on Medical Informatics. p. 679–83.

[6] Shwe, et al.. Probabilistic diagnosis using a reformulation of theINTERNIST-1/QMR knowledge base. Methods Inf Med1991;30(4):241–55.

[7] Liebovici, et al.. A causal probabilistic network for optimaltreatment of bacterial infections. IEEE Trans Knowl Data Eng2000;12(4):517–28.

[8] Lucas PJF, van derGaag LC,Abu-HannaA. Bayesian networks inbiomedicine and health-care. Artif Intell Med 2004;30(3):201–14.

[9] Wilson GN. Clinical genetics: a short course. NY: Wiley-Liss,Inc.; 2000.

[10] Green N. Coding Manual. Available from: http://www.uncg.edu/~nlgreen/GenIE-web-page.html.

[11] Green N. Towards an empirical model of argumentation inmedical genetics. In: Workshop Proceedings of CMNA 2003:IJCAI 2003 Workshop on Computational Models of NaturalArgument: 2003. p. 39–44.

[12] Zukerman I, McConachy R, Korb K, Pickett DA. Bayesianreasoning in an abductive mechanism for argument generationand analysis. In: Proceedings of the Fifteenth National Confer-ence on Artificial Intelligence; 1998. p. 833–8.

[13] Carofiglio V, de Rosis F. Combining logical with emotionalreasoning in natural argumentation. In: Conati C, Hudlicka E,Lisetti C, editors. 9th International Conference on User Model-ing. Workshop Proceedings. Assessing and Adapting to UserAttitudes and Effect: Why, When and How: 2003. p. 9–15.

[14] Carletta J, Isard A, Isard S, Kowtko JC, Doherty-Sneddon G,Anderson AH. The reliability of a dialogue structure codingscheme. Computat Linguist 1997;23(1):13–32.

[15] Bruce RF, Wiebe JM. Recognizing subjectivity: a case study ofmanual tagging. Nat Lang Eng 1999;1(1):1–16.

[16] Teufel S, Moens M. Summarizing scientific articles: experimentswith relevance and rhetorical status. Comput Linguist2002;28(4):409–45.

[17] Carletta J. Assessing agreement on classification tasks: the Kappastatistic. Comput Linguist 1996;22(2):249–54.

[18] Krippendorf K. Content analysis: an introduction to its method-ology. Beverly Hills, California: Sage Publications; 1980.

[19] Landis JR, Koch GG. The measurement of observer agreementfor categorical data. Biometrics 1977;33:159–74.

[20] Druzdzel MJ, Henrion M. Efficient reasoning in qualitativeprobabilistic networks. Proceedings of the 11th National Confer-ence on AI (AAAI-93): 1993. p. 548–53.

[21] Green N, Britt T, Jirak K. Communication of uncertainty inclinical genetics patient health communication systems. In: AAAI2004 Fall Symposium on Dialogue Systems for Health Commu-nication (to appear).

[22] Swales J. Genre analysis: English in academic and researchsettings. Cambridge, UK: Cambridge University Press; 1991.

[23] Green N. Argumentation in a causal probabilistic humanisticdomain. In: International Journal of Intelligent Systems (toappear).

[24] Eddy D. Probabilistic reasoning in clinical medicine: problemsand opportunities. In: Kahneman D, Slovic P, Tversky A, editors.Judgment under uncertainty: heuristics and biases. Cambridge,England: Cambridge University Press; 1982. p. 249–67.

[25] Green N. Analysis of linguistic features associated with point ofview for generating stylistically appropriate text. In: ShanahanJG, Qu Y, Wiebe, J, editors. Computing attitude and affect intext. Dordrecht, The Netherlands: Springer; 2005.

[26] Gene Ontology Consortium. Available from: http://www.geneontology.org/index.shtml.

[27] National Library of Medicine. UMLS knowledge sources. 5thedition—April release. 2004AA documentation. Available from:http://www.nlm.nih.gov/research/umls/UMLSDOC.HTML.

[28] Antal P, Fannes G, Timmerman D, Moreau Y, De Moor B.Using literature and data to learn Bayesian networks as clinicalmodels of ovarian tumors. Artif Intell Med 2004;30:257–81.

http://www.uncg.edu/~nlgreen/GenIE-web-page.html

http://www.uncg.edu/~nlgreen/GenIE-web-page.html

http://www.geneontology.org/index.shtml

http://www.geneontology.org/index.shtml

http://www.nlm.nih.gov/research/umls/UMLSDOC.HTML


[29] De Campos LM, Fernandez JM, Huete JF. Query expan-sion in information retrieval systems using a Bayesiannetwork-based thesaurus. In: Uncertainty in Artificial Intel-ligence; Proceedings of the Fourteenth Conference: 1998. p.53–60.

[30] Resta RG, editor. Psyche and helix: psychological aspects ofgenetic counseling: essays by Seymour Kessler, Ph.D. New York:Wiley-Liss; 2000.

[31] Kessler S. Psychological aspects of genetic counseling: analysis ofa transcript. Am J Med Genetic 1981;8:137–53 (Reprinted in ch.1,Resta 2000).

[32] Bales RF. Interaction process analysis. Cambridge, MA: Addi-son-Wesley Press; 1950.

[33] Liede A, Kerzin-Storrar L, Crauford D. A method for analyzingvideotaped genetic counseling sessions. J Genet Couns2000;9(2):117–36.

a bayesian network coding scheme for annotating biomedical information presented to genetic...

Documents