A Modal Characterization ofDefeasible Deontic Conditionals and Conditional Goals

Craig BoutilierDepartment of Computer Science

University of British ColumbiaVancouver, British Columbia

CANADA, V6T 1Z2emaih cebly@cs.ubc.ca

Abstract

We explore the notions of permission and obli-gation and their role in knowledge representa-tion, especially as guides to action for plan-ning systems. We first present a simple condi-tional deontic logic (or more accurately a pref-erence logic) of the type common in the litera-ture and demonstrate its equivalence to a num-ber of modal and conditional systems for de-fault reasoning. We show how the techniquesof conditional default reasoning can be used toderive factual preferences from conditional pref-erences. We extend the system to account forthe effect of beliefs on an agent’s obligations,including beliefs held by default. This leads usto the notion of a conditional goal, goals to-ward which an agent should strive accordingto its belief state. We then extend the system(somewhat naively) to model the ability of anagent to perform actions. Even with this simpleaccount, we are able to show that the deonticslogan "make the best of a bad situation" givesrise to several interpretations or strategies fordetermining goals (and actions). We show thatan agent can improve its decisions and focus itsgoals by making observations, or increasing itsknowledge of the world. Finally, we discuss howthis model might be extended and used in theplanning process, especially to represent plan-ning under uncertainty in a qualitative manner.

1 Introduction

In the usual approaches to planning in AI, a planningagent is provided with a description of some state of af-fairs, a goal state, and charged with the task of discover-ing (or performing) some sequence of actions to achievethat goal. This notion of goal can be found in the ear-liest work on planning (see [26] for a survey) and per-sists in more recent work on intention and commitment[8]. In most realistic settings, however, an agent willfrequently encounter goals that it cannot achieve. Aspointed out by Doyle and Wellman [10] an agent pos-sessing only simple goal descriptions has no guidance forchoosing an alternative goal state toward which it should

strive. Some progress has been made toward providingsystems with the power to express sub-optimal goals.Recently, Haddawy and Hanks [14] have provided a veryrestricted model for partial fulfillment of deadline goals.Their goals are propositions that must be made true by agiven deadline. If a goal cannot be met, utility is assignedto partial fulfillment, for instance, making "more" of theproposition true or getting close to the deadline. Butwe should not expect goals to be so well-behaved in gen-eral. There are many situations in which, say, missinga deadline by some small margin is worse than ignoringthe task altogether; or in which an agent that fails tosatisfy one conjunct of a conjunctive goal should not at-tempt to satisfy the rest of the goal. Indeed, goals canchange drastically depending on the situation in whichan agent finds itself. If it is raining, an agent’s goalstate is one it which it has its umbrella. If it’s not rain-ing, it should leave its umbrella home. Moreover, goalsdo not change simply because some ideal goal cannotbe satisfied: a robot may be able to leave its umbrellahome when it is raining. Circumstances play a crucialrole in determining goals. We want the ability to ex-press such goals as "Pick up a shipment at wholesalerX by 9AM," but state exceptions like "If it’s a holiday,go to wholesaler Y." Permitting explicit exceptions togoal statements allows goals to be expressed naturally.Clearly, just as conditional plans allow circumstances todictate the appropriate sequence of actions for satisfyinga goal, conditional goals allow circumstances to dictatejust what goal state an agent "desires."

It is profitable to think of an agent’s goal as an obliga-tion. It ought to do what it can to ensure the goals set forit are achieved. There have been a number of systemsproposed to model the notions of permission and obli-gation, what ought to be the case [28, 7]. The earliestmodal logics for obligation were unsuccessful, for theyseem unable to represent conditional obligations. Condi-tional deontic logics have been developed to account forthe fact that obligations can vary in different circum-stances [15, 19, 27]. This indicates that such systemsare suitable for the representation of conditional goals.

In this paper we present a simple modal semantics fordefeasible deontic conditionals and extend this in sev-eral ways to account for the influence of default knowl-edge, incomplete knowledge and ability on obligationsand goals. Many systems have a rather complex seman-

3O

From: AAAI Technical Report SS-93-05. Compilation copyright © 1993, AAAI (www.aaai.org). All rights reserved.

tics that attempts to capture the defeasibility of suchconditionals [20, 17]. In Section 2, we present a fam-ily of simple modal logics that can be used to repre-sent defeasible deontic conditionals. These logics deal:only with preferences on deontic alternatives. This is acrucial abstraction for it allows us to concentrate solelyon the structure of deontic preferences, ignoring impor-tant but separate concerns that must be considered inany true logic of obligation (see below). We show thatour logic captures the semantic system of Hansson [15],and demonstrate that one can capture conditional obli-gations with a unary modal operator, contrary to "con-ventional wisdom" in the conditional logic literature.

Recently, the mechanisms of default reasoning havebeen applied to problems in deontic logic [16, 18]. In-deed, our conditional/modal logic was originally devel-oped for applications in default reasoning [1, 2]; we showin Section 3 how techniques for conditional default rea-soning can be used for deontic reasoning. ..... .........

In Section 4, we turn our attention to the influence ofbeliefs on an agent’s goals. Since goals are conditionalon circumstances, the goals adopted by an agent willdepend crucially (and defeasibly) on the agent’s beliefsabout the world. We (tentatively) adopt the deonticstrategy that an agent should strive to bring about thebest situations consistent with its beliefs. This providesa different conception of goals from that usually foundin AI (e.g., the goals of [8]). Included in this analysisis a model of default beliefs. This takes the first stepstoward defining a "qualitative decision theory."

Since an agent cannot be expected to do things be-yond its ability, we introduce a naive model of abilityand action in Section 5. We describe goals to be thosepropositions within an agent’s control that ensure opti-mal outcomes. It turns out, however, that this modelis sufficient to render useless the deontic slogan "bringabout the best situations possible." Incomplete knowl-edge of the world gives rise to different strategies for de-riving goals, in the tradition of game theory. We describeseveral of these and how observations can be used to im-prove decisions and focus goals.

Finally, we conclude with a brief summary and somefuture directions in which this work may be pursued,including applications to planning under uncertainty.

2 Conditional Preferences and DeonticLogic

Deontic logics have been proposed to model the con-cepts of obligation and permission [28, 29, 7, 15]. It isclear that a sentence "It ought to be the case that A"has non-extensional truth conditions, for what is actuallythe case need not have any influence on what (ideally)ought to be. Deontic logics usually have a modal con-nective O where the sentence Oa means that the propo-sition a is obligatory, or it ought to be the case that a.1Semantically, the truth of such statements are usually

1When discussing the obligations of a particular agent,we will often say that the agent has an obligation to "doa." Though ot is a proposition, we take "do a" to be some(unspecified) action that brings about

determined with respect to a given set of ideal possibleworlds, a being obligatory just when a is true in all idealsituations. While this set of ideal worlds (obligation) often taken to be determined by some code of moral orethical conduct, this need not be the case. "Obligations"may simply be the goals imposed on an agent by its de-signer, and the ideal worlds simply those in which anagent fulfills its specified duties.~ We take the notion ofideality or preference in what follows to be determinedby any suitable metric, and allow "obligation" to referto the satisfaction of design goals, moral imperatives oranything similar.

It has been widely recognized that deontic logic, de-fined using a unary modal connective O, has certain lim-itations. In particular, it is difficult to represent condi-tional obligations [7]. For this reason, conditional deonticlogic (CDL) has been introduced to represent the depen-dence of obligations on context [29]. Obligation is thenrepresented by a two-place conditional connective. Thesentence O(BIA) is interpreted as "It ought to be that Bgiven A" or "IfA then it is obligatory that B," and indi-cates a conditional obligation to do B in circumstancesA. These logics can be interpreted semantically using anordering on worlds that ranks them according to somenotion of preference or ideality [15, 19]. Such a rankingsatisfies O(B[A) just in case B is true at all most pre-ferred of those worlds satisfying A. Thus, we can thinkof B as a conditional preference given A. Once A is true,the best an agent can do is B.

In this section, we present a simple modal logic and se-mantics for the representation of conditional obligations.The logic CO (and related systems) are presented belowfor this purpose. The presentation is brief and we refer to[1, 4] for further technical details and an axiomatization.

2.1 The Bimodal Logic CO

We assume a propositional bimodal language LB over aset of atomic propositional variables P, with the usualclassical connectives and two modal operators [] and ~.Our Kripkean possible worlds semantics for deontic pref-erence will be based on the class of CO-models, triplesof the form M = (W, _<, ~o) where W is a set of possibleworlds, or deontic alternatives, ~, is a valuation functionmapping P into 2W (~(A) is the set of worlds where is true), and < is a reflexive, transitive connected binaryrelation on I~. 3 Thus < imposes a total preorder onW: W consists of a set of <-equivalence classes, thesebeing totally ordered by <. We take < to represent anordering of deontic preference: w < v just in case v is atleast as preferable as w. This ordering is taken to reflectthe preferences of an agent about complete situations,however these are to be interpreted (e.g., an ordering ofmoral acceptability, personal utility, etc.). We will some-times speak of preferred situations as being more idealor more acceptable than others. Each equivalence class,or cluster of worlds, consists of a set of equally preferredsituations. Figure 1 illustrates a typical CO-model. Thetruth conditions for the modal connectives are

2Cohen and Levesque [8] analyze goals similarly. An agenthas a goal a just in case a is true at all worlds that are "goal

31

©l

Worlds

@Figure 1: A CO-model

1. M ~w rla iff for each r such that w _< r, M ~r a.

2. M ~w ~a iff for each v such that to ~ v, M ~ a.Da is true at a world to just in ease a is true at all

worlds at least as preferred as w, while ~a holds justwhen a holds at all less preferred worlds. The dual con-nectives are defined as usual: <>a --df ~Q-’ot means ais true at some equally or more preferred world; and~a =dr -’~-’a means a is true at some less preferred

world. Oa ~df I-IO~ h OO~ and ~a ----dr <>a V ~a mean ais true at all worlds and at some world, respectively.

2.2 Deontic Conditionals

We now define a conditional connective /(-I-) to ex-press conditional preferences. Intuitively, I(B[A) shouldhold just when B holds at the most ideal worlds sat-isfying A. Of course, nothing in our models forces theexistence of such minimal A-worlds (see Lewis [19] on theLimit Assumption). However, we may simply say thatthere should be some world satisfying A A B such thatA D B holds at all "accessible" (equally or more ideal)worlds. We also let the conditional hold vacuously whenthe antecedent A is false in all situations. These truthconditions can be expressed in Lv as follows:

I(B]A) -~=df ~-A V ~(A ^ f"l(a ~ B)).

I(BIA) can be read as "In the most preferred situationswhere A holds, B holds as well," or "If A then ideallyB." This can be thought of, as a first approximation, asexpressing "If A then an agent ought to ensure that B,"for making B true (apparently) ensures an agent endsup in the best possible A-situation. There are problemswith such a reading, as we discuss shortly; hence, weusually adopt the former reading. However, we will oc-casionally lapse and read I(BIA) as "IfA then it ought tobe that B." We note that an absolute preference A can

be expressed as I(AIT), or equivalently, ~nA. We ab-breviate this as I(A) ("ideally A"). We can also expressthe (analog of) the notion of conditional permission. -~I(-’AIB) holds, then in the most preferred B-situationsit is not required that --A. This means there are ideal

accessible."s <_ is connected iff w <_ v or v _< W for each v, w E W.

B-worlds where A holds, or that A is "tolerable" givenB. We abbreviate this sentence T(AIB). Loosely, wecan think of this as asserting that an agent is permittedto do A if B. Unconditional toleration is denoted T(A)and stands for -’I(-’A), or equivalently, ~<>A.

Using CO we can express the conditional prefer-ences involved in a number of classic deontic puzzles.Chisholm’s paradox of contrary-to-duty imperatives [7]is one such puzzle. The preferences involved in the fol-lowing account [20] cannot be adequately captured inwith a unary modal deontic connective:

(a) It ought to be that Arabella buys a train ticketto visit her grandmother.

(b) It ought to be that if Arabella buys the ticketshe calls to tell her she is coming.

(c) If Arabella does not buy the ticket, it ought tobe that she does not call.

(d) Arabella does not buy the ticket.

We can, however, represent these sentences conditionallyas I(V), I(CIV), I(-,C]--,V) and -,V. These give rise tono inconsistency in CO, and induce a natural orderingon worlds where only V A C-worlds are most acceptable.Less preferred are certain -,V A -,C-worlds, and still lesspreferred is any -’V A C-world:

VC < VC < VC

(The relative preference of V A -’C-worlds is left un-specified b_y this account, though we are assured thatVC < VC.) Notice that from this set we can deriveI(C). More generally, CO satisfies the principle of deon-~ic detachment [20]:

I(BIA) ̂ I(a) Another principle sometimes advocated in the deonticliterature is that of factual detachment:

X(BIA) ^ A D I(B)

This expresses the idea that if there is a conditional obli-gation to do B given A, and A is actually the case,then there is an actual obligation to do B. No exten-sion of standard deontic logic can contain both prin-ciples as theorems [20]. Given our reading of I(BIA),factual detachment should not be (and is not) valid CO. However, clearly we require some mechanism for in-ferring "actual preferences" from factual statements andconditional preferences. In the next section we describehow techniques from conditional default reasoning canbe used for just this purpose.

Another "puzzling" theorem of CO (as well as mostconditional and standard deontic logics) is the following:

I(A) D I(A V B)

If we read I(A) as "an agent is obligated to do A," thistheorem seems somewhat paradoxical. From "It is oblig-atory that you help X" one can infer that "It is obliga-tory that you help X or kill X." This suggests that onemay fulfill this obligation by killing X [15]. Of course,if we adhere strictly to the reading of I as "In the most

32

ideal situations you help X or kill X" this is less prob- to great advantage. In particular, it allows us to ex-lematic, for it does not suggest any means for fulfilling press various assumptions about our premises and usethese obligations. Furthermore, seeing to it that X is the techniques of conditional default reasoning to infer

.killed (presumably) removes one from the realm of ideal. . actual preferences. : ~ ......... ,: ........situations and did nothing to fulfill the original obliga- Loewer and Belzer [20] have criticized Lewis’s seman-tion (help X). In Sections 4 and 5 we suggest a means tics "since it does not contain the resources to expresscapturing this distinction. This puzzle is closely related actual obligations and no way of inferring actual obli-to the notion of free choice permission, gations from conditional ones." Clearly, in our exam-

ple above, we should somehow be able to conclude that2.3 Representation Results Arabella ought not call her grandmother; but we cannotThe idea of using an ordering on possible situations to infer logically in CO that I(-,C). This is to be expected,reflect deontic preference and capture conditional obli- for the actual fact -,V does not affect the form takengations is not new. Hansson [15] provided a semantics by ideal situations. However, once -,V is realized, onefor CDL that is much like ours. 4 While Hansson does ought to attempt to make the best of a bad situation. Innot present an axiomatization of his system DSDL3, we other words, actual preferences (or obligations) shouldcan show that our modal semantics extends his and that be simply those propositions true in the most preferreda fragment of CO provides a sound and complete proof worlds that satisfy the actual facts.theory for his system. Let CO- denote the set of the- Let KB be a knowledge base containing statements oforems in CO restricted to those sentences whose only .... conditional preference and actual facts. Given that suchnon-classical connective is I.5 facts actually obtain, the ideal situations are those mostTheorem 1 }-co- ~ iff ~DSDL3 ~. preferred worlds satisfying KB. This suggests a straight-

forward mechanism for determining actual preferences.Furthermore, the power of the second modal operator is We simply ask for those c~ such thatnot required. The results of [1, 3] show that the purelyconditional fragment of CO can be captured using only }-co I(~IKB)the operator D, that is, using the classical modal logic$4.3, simply by defining If KB is the representation of Chisholm’s paradox above,

we have that I(-,CIKB ) is valid. Thus, we can quiteI(BIA) ~--’df Q--A V O(A A D(A D B)). (2) reasonably model a type of factual detachment simply

Let $4.3- denote the conditional fragment of $4.3. by using nested conditionals of this sort.We notice that this is precisely the preliminary scheme

Theorem 2 }-s4.3- a iff ~DSDL3 a. for conditional default reasoning suggested in [9, 2]. ThisWe note also that our system is equivalent to Lewis’s con-" mechanism unfortunately has a serious drawback: seem-ditional deontic logic VTA [19]. This shows that CDL, ingly irrelevant factual information can paralyze theas conceived by Hansson and Lewis, can be captured us- "default" reasoning process. For instance, let Kff =ing only a unary modal operator. The "trick", of course, KB U {R}, where R is some distinct propositional atomlies in the fact that O is not treated as unconditional (e.g., "it will rain"). We can no longer conclude thatobligation. Arabella ought not call, for I(-,C[KB~) is not valid. In-

Using a modal semantics of this sort suggests a number tuitively, R has nothing to do with Arabella’s obligationof generalizations of Lewis’s logics. For instance, by us- to call, yet, from a logical perspective, there is nothinging the modal logic $4, we can represent partially ordered in the premises that guarantees that raining cannot af-or preordered preferences. Furthermore, we can define feet Arabella’s obligations. The following (incomplete)a version of the conditional that allows us to explicitly ordering is satisfies with the original premises:(and consistently) represent conflicting preferences of the

VCR < VCR < VCRform I(B[A) and I(’,B[A) (see [1, 6]). The logic CO*,an extension of CO presented in [2, 4], is based on the ~Hence, it could bethat Arabella should call if it’s rainingclass of CO-models in which every propositional valua- (whether she visits or not).tion (every logically possible world) is represented. Using Several solutions have been proposed for the problemCO* we can ensure that each world is ranked according of irrelevance in default systems. We briefly describe one,to our preference relation. Pearl’s [24] System Z. Roughly, we want to assume that

worlds are as preferred or as ideal as possible, subject3 Defensible Reasoning to the constraints imposed by our theory of preferences.

For instance, a world where it is raining, Arabella failsWhile a standard modal logic ($4.3) suffices to representto buy a ticket and doesn’t call her grandmother violatesHansson’s and Lewis’s notion of conditional obligation,no more obligations (in KB) than a world where it is notthe added expressiveness of the logic CO can be usedraining and the other conditions obtain. It is consistent

4One key difference is the fact that Hansson invokes the to assume that raining situations are no less ideal thanLimit Assumption: there must be a minimal (or most pre- non-raining situations. System Z provides a mechanismferred) set of A-worlds for each satisfiable proposition A. This for determining the consequences of such assumptions,has no impact on the results below [4]. and it will allow us to conclude from Kff that Arabella

5Occurrences of Q and o must conform to the pattern in ought not call (when it is raining). Roughly, Systemthe definition of I. Z chooses a preferred model from those satisfying KB.

33

This model is the most "compact" model of KB, a modelwhere worlds are "pushed down" in the preference or-dering as far as possible (consistent with the constraintsimposed by KB). For simple conditional theories6 there~ ::is a unique preferred model, the Z-model. The Z-modelfor KB (or Kff) above is

{vcR, vc- } < V-OR, VU- , V-CA} <VCR}

Notice that I(C[V A R) is satisfied in this model. Alsonotice that visiting without calling is no "worse" thannot visiting: since our premises do not specify exactlyhow bad failing to call is, it is assumed to be as "good"as possible (though it cannot be ideal, since it violatesthe imperative I(C[V)).

Goldszmidt and Pearl [12, 13] have developed algo-rithms that compute (often efficiently) the conclusionsthat can. be derived in this way. In [2] we describe howthe expressive power of CO can be used to axiomatizethe assumptions of System Z and describe an explicitpreference relation on CO-models that captures the cri-terion of "compactness". Of course, any problems witha default reasoning scheme must be expected to carryover to our deontie approach. System Z has the draw-back of failing to count situations that violate more con-ditional preferences as less ideal than those that violatefewer (at least, within a a fixed priority level). A sim-ple solution has been proposed in [5] for the logic CO.Other solutions to the problem of irrelevance for condi-tional reasoning have also been proposed. We do notcatalogue these here, but simply point out that deriving

¯ factual obligations from a conditionalKB has exactlythe same structure as deriving default conclusions froma conditional KB (see the next section); any problemsand solutions for one will be identical for the other.

We note that recently other techniques for default rea-soning have been proposed for deontie systems. Jonesand P6rn [18] have also proposed an extension of theirearlier deontic logics that uses some ideas from Del-grande’s work in default reasoning. However, their sys-tem is rather complex and quite distinct from ours.Makinson [21] has made some preliminary suggestionsfor incorporating aspects of normality and agency in theformalization of obligations that,relate to work in default~reasoning. Horty [16] proposes representing imperativesand deriving obligations using Reiter’s default logic. Sit-uations are "scored" according to the imperatives theyviolate. This too is different from our approach, for wetake preferences to be primitive and use these to de-rive (conditional) obligations. Naturally, constraints a preference ordering can be derived from imperatives aswell should we chose to view a statement I(BIA) as animperative. In fact, System Z can be viewed as rank-ing situations according to the priority of the impera-tives violated by a situation. This is one advantage ofthe conditional approach over Horty’s system: prioritieson imperatives are induced by a conditional theory. Incontrast, should one situation violate two rules of equal

~Such theories may have propositions and conditionals ofthe form I(BIA) where B, A are propositional.

priority while a second violates just one, System Z viewsboth situations as equally preferred (this is the drawbackcited above), while Horty’s system clearly "prefers" thatfewer imperatives be :violated. ~Again, the proposal in [5]for default reasoning can be seen as combining these twoapproaches.7

The view of conditionals (in the sense of CO) as im-peratives is tenable. However, imperatives reflect muchmore information than the preferences of an agent. Is-sues of action and ability must come into play, so it ishard to see just how a preference ordering can be derivedfrom imperatives without taking into account such con-siderations. In the following sections, we explore theseissues, showing how one might determine actions appro-priate for an agent in various circumstances (the agent’s"imperatives" ).

4 Toward a Qualitative Decision Theory

4.1 A Logic of Goals

While deontie logics concentrate on preferences relatedto notions of moral acceptability or the like, it is clearthat preferences can come from anywhere. In particu-lar, a system designer ought to be able to convey to anartificial agent the preferences according to which thatagent ought to act. From such preferences (and otherinformation) and agent ought to be able to derive goalsand plan its actions accordingly.

Doyle and Wellman [10] have investigated a preferen-tial semantics of goals motivated by a qualitative notion

’ of’utility. Roughly, P is a goal just when any "fixed" P-¯ ’~ situation is preferred to the corresponding --P-situation. :

A loose translation into our framework would correspondto a sentence schema I(PIa) for all propositions ~ where

~t _,p. This type of goal is unconditional in the sensethat it ensures that P is ideally true: I(P). We canexpress conditional goals in their framework simply byfixing certain other propositions, but these can certainlynot be defeasible.

Asserting P as a goal in this sense is a very strongstatement, in fact, so strong that very few goals willmeet this criterion. For P to be a goal, it must be thatno matter what else is true the agent will be better off

, if P. We call such a goal absolute (whether conditional¯ ;.’or unconditional).; However, even such, strong moralim ......peratives as "Thou shalt not kill" typically have excep-tions (for instance, self-defense). Certainly, it is the easethat we can postulate an absolute goal by consideringall exceptional situations for a goal and making the goalconditional on the absence of exceptions. Unfortunately,just as with the qualification problem for default reason-ing, this leads to goals that must be expressed in a veryunnatural fashion. We should note that absolute goalshave an advantage over the defeasible goals expressible inCO. If Q is an absolute goal conditional on P, then onceP is known by an agent it is guaranteed that ensuring

rWhile based on default logic, Horty’s system shares someremarkable similarities because of the simple theories he usesand their connection to Poole’s Theorist system, which inturn is related to our conditional logic [5].

34

Q is the best course of action. No matter what contin-gencies occur, Q is better than --Q. However, as we willsee below, such goals might provide very little guidancefor appropriate behavior, especially in the presense of in-complete knowledge. Once again, very few realistic goalsmatch this specification. To use an example developedbelow, an agent might have to decide whether or not totake its umbrella to work: if it rains, taking the umbrellais best; if not, leaving the umbrella is best. Taking orleaving the umbrella cannot be an absolute goal for anagent, yet clearly there may be other considerations thatmake one or the other action a "real" goal. Such consid-erations include action, ability, expected outcomes andgame-theoretic strategies.

4.2 Default Knowledge

As we have seen, the belief set of an agent provides thecontext in which its goals are determined. However, we

¯ should not require that goals be based only on "certain".beliefs, but on any reasonable default conclusions as well.For example, consider the following preference orderingwith atoms R (it will rain), U (have umbrella) and (it’s cloudy). Assuming C A R is impossible, we mighthave the following preferences:

{C--R-g, C~-U} < CRU < {CRU, C-~U} < CR’U

Suppose, furthermore, that it usually rains when itscloudy. If KB = {C}, according to our notion of obli-

l~tion in the last section, the agent’s goals are R andU. Ideally, the agent ought to ensure that it doesn’train and that it doesn’tbring its umbrella. Ignoring theabsurdity of the goal R (we return to this in the nextsection), even the goal U seems to be wrong. Given C,the agent should expect R and act accordingly.

Much like in decision theory, actions should be basednot just on the utilities (preferences) over outcomes, butalso on the likelihood (or typicality or normality) of out-comes. In order to capture this intuition in a qualitativesetting, we propose a logic that has two orderings, onerepresenting preferences over worlds and one represent-ing the degree of normality or expectation associated witha world. The presentation is again brief and we refer to[6] for further motivation and technical details.

The logic QDT, an attempt at a qualitative decisiontheory, is characterized by models of the form M =(W, _<p, _<N, ~o), where W is a set of worlds (with valua-tion function ~o), _<p is a transitive, connected preferenceordering on W, and _<N is a transitive, connected nor-mality ordering on W. We interpret w <p v as above,and take w _<N v to mean w is at least as normalasituation as v (or is at least as expected). The submod-els formed by restricting attention to either relation areclearly CO-models. The language of QDT contains fourmodal operators: Qp, ~p are given the usual truth con-ditions over <_p and ON, DN are interpreted using <_N.The conditional I(BIA) is defined as previously, using1:3p, ~p. A new normative conditional connective =~ isdefined in exactly the same fashion using I::]N, ON:

A ~ B =dr 5N-~A V 5N(A ̂ nN(A Z n)).

35

The sentence A =~ B means B is true at the most nor-mal A-worlds, and can be viewed as a default rule. Thisconditional is exactly that defined in [2, 4], and the as-sociated logic is equivalent to a number of other systems(e.g., the qualitative probabilistic logic of [23, 13]).s

Given a QDT-model and a (finite) set of facts KB, wedefine the default closure of KB to be (where LcpL isour propositional sublanguage)

CI(KB) = {a E LCPL : KB ~ a}That is, those propositions a that are normally truegiven KB form the agent’s set of default conclusions. Itseems natural to ask how the ordering <:N is determined.Typically, we will have a set of conditional premises ofthe form A ::~ B, plus other modal sentences that con-strain the ordering. We note that conditional deonticsentences may also be contained in KB; but these im-pose no constraints on the normality ordering. Unlessthese premises form a "complete" theory, there will be aspace of permissible normality orderings. Many defaultreasoning schemes will provide a "preferred" such order-ing and reason using that ordering. System Z, describedabove, is one such mechanism, forming the most com-pact (normality) ordering consistent with the KB. Wemake no commitment to the default reasoning schemewe use to determine the ordering, simply that the clo-sure Cl(KB) is semantically well-defined. We assume forsimplicity (though this is relaxed in [6]), that the defaultclosure CI(KB) is finitely specifiable and take it to be asingle propositional sentence.9

We remark that similar considerations apply to thepreference ordering <p. One can take the ordering tobe the most compact ordering satisfying the premisesKB or use some other strategy to determine allowablemodels. However, we do not require the use of a singleordering -- the definitions presented below can be re-interpreted to capture truth in all permissible orderingsor all QDT-models of a given theory [6]. It is sufficient,though, to define goals relative to a single model, andthen use simple logical consequence (truth in all QDT-models of KB) to derive goals, if desired.

An agent ought to act not as if only KB were true, butalso these default beliefs Ci(KB). Assume a particularQDT-model M. As a first approximation, we define anideal goal (w.r.t. KB) to be any a E LCpL such that

M ~ I(aICl(gS))

The ideal goal set is the set of all such a. In our previousexample, where KB - {C}, we have that CI(KB) --C ̂ R and the agent’s goals are those sentences entailedby CARAU. It should be clear that goals are conditionaland de feasible; for instance, given C ̂ R, the agent nowhas as a goal ~’.10

SQDT can be axiomatized using the axioms of CO foreach pair of connectives [::]p, Dp and DN, DN, plus the sin-gle interaction axiom ~pa _--- D~a. Thus, _<v and _<Iv arecompletely unrelated except that they must be defined overthe same set of worlds.

9A sufficient condition for this property is that each "clus-ter" of equally normal worlds in _<N corresponds to a finitelyspecifiable theory. This is the case in, e.g., System Z [2].

1°The "priority" given to defaults can be thought of as

5 Ability and Incomplete Knowledge

The definition of ideal goal in the previous section issomewhat unrealistic as it fails ~to adequately accountfor the ability of an agent. Given C in our exampleabove, the derived goal U seems reasonable while thegoal R seems less so: we should not expect an agentto make it rain! More generally, there will be certainpropositions a over which the agent has no control: it isnot within its power to make a true or false, regardlessof the desirability of a. We should not require an agentto have as a goal a proposition of this type. Ideal goalsare best thought of as the "wishes" of an agent that findsitself in situation KB (but cannot change the fact thatit is in a KB situation).

5.1 Controllable Propositions

To capture distinctions of this sort, we introduce a naivemodel of action and ability and demonstrate its influ-ence on conditional goals. While this model is certainlynot very general (see the concluding section), it is suffi-cient to illustrate that conditional goals will require morestructure than we have suggested above using ideal goals.

We partition our atomic propositions into two classes:P = C U C. Those atoms A E C are controllable, atomsover which the agent has direct influence. We assumethat the only actions available are do(A) and do(A),which make A true or false, for each controllable A. Weassume also that these actions have no effects other thanto change the truth value of A. The atom U (ensureyou have your umbrella) is an example of a controllableatom. Atoms in ~ are uncontrollable. R (it will rain) an example of an uncontrollable atom.

Definition For any set of atomic variables ~o, let V(P)be the set of truth assignments to this set. Ifv E vCP) and w E V(O.) for distinct sets 7~, Q,then v; w E V(P U Q) denotes the obvious extendedassignment.

We can now distinguish three types of propositions:

Definition A proposition a is controllable iff, for everyu e V(C), there is some v e Y(C) and w E such that v; u ~ a and w; u ~ -~a.A proposition a is influenceable iff, for some u EV(~), there is some v E V(C) and w e V(C) that v; u ~ a and w; u ~ --a.Finally, a is uninfluenceable iff it is not influence-able.

Intuitively, since atoms in C are within complete con-trol of the agent, it can ensure the truth or the falsityof any controllable proposition a, according to its desir-ability, simply by bringing about an appropriate truthassignment. If A, B E C then A V B and A A B are con-trollable. If a is influenceable, we call the assignment uto C a context for a; intuitively, should such a contexthold, a can be controlled by the agent. If A E C, X Ethen A V X is influenceable but not controllable: in con-text X the agent cannot do anything about the truth of

assuming arbitrarily high conditional probabilities. We arecurrently investigating the ability to give priority to certainpreferences (e.g., infinitely low or high utility).

A V X, but in context X" the agent can make A V X trueor false through do(A) or do(A). Note that all control-lables are influenceable (using a tautologous context). this example, X isuninfluenceable, It is easy to see thatthese three types of propositions are easily characterizedaccording to properties of their prime implicates [6].

5.2 Complete Knowledge

Given the distinction between controllable and uncon-trollable propositions, we would like to modify our defi-nition of a goal so that an agent is obligated to do onlythose things within its control. A first attempt mightsimply be to restrict the goal set as defined in the lastsection to controllable propositions. In other words, wedetermine the those propositions that are ideally truegiven CI(KB), and then denote as a goal any such propo-sition within the agent’s control (i.e., any propositionthat is influenceable in context KB). The following ex-ample shows this to be inadequate.

Consider three atoms: L (my office thermostat is setlow); W (I want it set low); and H (I am home morning). My robot has control only over the atom (it can set the thermostat), and possesses the followingdefault information: T =~ W, T =v W and H A L" =¢, W.(I normally want the thermostat set low; but if it’s high, probably set it myself and want it high -- unless I stayedat home and the caretaker turned it up.) The robot hasthe "factual" knowledge KB = {L, H}, namely, that thethermostat setting is high and I’m at home. The defaultclosure of its knowledge is CI(KB) - {-L, H, W}: mostlikely I want the thermostat set low, even though it iscurrently high. Finally, the robot’s preference orderingis designed to respect my wishes:

{WL, WL} < WL < WL

(we assume H does not affect preference).It should be clear that the robot should not deter-

mine its goals by considering the ideal situations satis-fying CI(KB). In such situations (since L is known), is true and the robot concludes that L should be true.11

This is clearly mistaken, for considering only the bestsituations in which one’s knowledge of controllables istrue prevents one from determining whether changingthose controllables could lead to a better situation. Sinceany controllable proposition can be changed if required,we should only insist that the best situations satisfyinguninfluenceable known propositions be considered. Weshouldn’t allow the fact that L is known unduly influencewhat we consider to be the best alternatives -- we canmake L true if that is what’s best. But notice that weshould not ignore the truth of controllables when makingdefault predictions. The prior truth value of a control-lable might provide some indication of the truth of anuncontrollable; and we must take into account these un-controllables when deciding which alternatives are pos-sible, before deciding which are best. In this example,the fact L might provide an indication of my (uncontrol-lable) wish W (though in this case defeated by H).12

alThis is a simple theorem of our conditional logic: I(ala).l~If a controllable provides some indication of the truth

36

This leads to the following formulation of goals that 5.3 Incomplete Knowledgeaccount for ability. We again assume a QDT-model M The goals described above seem reasonable, in accordand sets 17, 17. The closure of KB is defined as usual, with the general deontic principle "do the best thing

.... The nninflnenceable belief set of an agent is ...... ’ possible consistent with your knowledge." We dubbed

UI(KB) = {or E Cl(KB) a is uninfluenceable}

This set of beliefs is used to determine an agent’s goals.We say a is a complete knowledge goal (CK-goal) iff

M ~ I(a[ UI(KB)) and c~ is controllable

In our example above, the only atomic goal the robothas is L (low thermostat). In the earlier example, givenC (cloudy) the goal will be U (take umbrella).

As with ideal goals, the set of CK-goals is deductivelyclosed. We can think of CK-goals as necessary conditionsfor an agent achieving some ideal state. Usually we areinterested in sufficient conditions, some sentence that,if known, guarantees that the agent is in an ideal state(given UI(KB)).13 This can be captured in modal terms.We say proposition G is CK-snfficient with respect toKB (or "guarantees ideality")just when G UI(KB) issatisfiable and

M ~ ~p(UI(KB) D "Dp(UI(KB) D

This simply states that any world satisfying G A UI(KB)is at least as preferred as any other UI(KB)-world. Thusensuring G is true (assuming as usual that UI(KB) can-not be affected) guarantees the agent is among the bestpossible UI( KB)-worlds.

Of course, changes in the world can only be effectedthrough atomic actions, so we are most interested in suf-ficient conditions described using atomic actions. We sayan (atomic) action set is any set of controllable literals(drawn from (7). If A is such a set we use it also to note the conjunction of its elements. An atomic goal setis any action set ,4 that guarantees each CK-goal (see [6]for further details). We can show that any atomic goalset determines a reasonable course of action.

Theorem 3 £et ,4 be an atomic goal set for KB. Then.4 is CK-sufficient for KB.

We note that usually we will be interested in minimalatomic goal sets, since these require the fewest actionsto achieve ideality. We may wish to impose other metricsand preferences on such goals sets as well (e.g., associat-ing costs with various actions).

of an uncontrollable or another controllable, (e.g., T :~ W)we should think of this as an evidential rule rather than acausal rule. Given our assumption about the independenceof atoms in C, we must take all such rules to be evidential(e.g., changing the thermostat will not change my wishes).We discuss this further in the concluding section. Note theimplicit temporal aspect here; propositions should be thoughtof as fluents. The theorem I(ala ) should not be viewed asparadoxical for precisely this reason. It does not suggest thata ought to be true if it is true, merely that it must be truein the best situations where it is true. As we see below, -,acan be a goal or obligation even if a is known.

13Hector Levesque (personal communication) has sug-gested that this is the crucial "operator."

such goals "CK-goals" because they seem correct whenan agent has complete knowledge of the world (or atleast uncontrollables). But CK-goals do not always de-termine the best course of action if an agent’s knowledgeis incomplete. Consider our preference ordering abovefor the umbrella example and an agent with an emptyknowledge base. For all the agent knows it could rain ornot (it has no indication either way). According to ourdefinition ofa CK-goal, the agent ought to do(U), for thebest situation consistent with its KB is RU. Leaving itsumbrella at home will be the best choice should it turnout not to rain; but should it rain, the agent has broughtabout the worst possible outcome. It is not clear thatU should be a goal. Indeed, one might expect U to bea goal, for no matter how R turns out, the agent hasavoided the worst outcome.

It is clear, in the presence of incomplete knowledge,that there are various strategies for determining goals.CK-goals (the "deontic strategy") form merely one al-ternative. Such a strategy is opportunistic, optimisticor adventurous. Clearly, it maximizes potential gain, forit allows the possibility of the agent ending up in thebest possible outcome. In certain domains this mightbe a prudent choice (for example, where a cooperativeagent determines the outcome of uncontrollables). Ofcourse, another strategy might be the cautious strategythat minimizes potential loss. This corresponds preciselyto the minimax procedure from game theory, describedformally here.

Complete action sets (complete truth assignments tothe atoms in C) are all of the alternative courses of actionavailable to an agent. To minimize (potential) loss, must consider the worst possible outcome for each ofthese alternatives, and pick those with the "best" worstoutcomes. If .Aa, ‘42 are complete action sets, we say .Aais as good as ,42 (,41 < ,42) iff

M ~ <~P(,42 A UI(KB) -’ 5p(,41 A UI(KB)))

Intuitively, if ,41 < ,42 then the worst worlds satisfying,41 are at least as preferred in <p as those satisfying,42 (considered, of course, in the context UI(KB)). Itis not hard to see that < forms a transitive, connectedpreference relation on complete action sets. The bestactions sets are those minimal in this ordering <. Todetermine the best action sets, however, we do not needto compare all action sets in a pairwise fashion:

Theorem 4 ,4i is a best action set iff M ~ ,4i < ",41.

This holds because the negation of a complete action setis consistent with any other action set. We say a is acautions goal iff

V{,4i : ,4i is a best action set } ~ a

In this way, if (say) A A B and A A --,B are best actionsets, then A is a goal but B is not. Simply doing A (andletting B run its natural course) is sufficient. This notionof goal has controllability built in (ignoring tautologies).

37

In our example above, U is a cautious goal. Of course,it is the action sets that are most important to an agent.We cannot expect best action sets, in general, to be suf-

’ficient in the same sense that CK-goal sets are.~Thepotential for desirable and undesirable outcomes makesthis impossible. However, we can show that if there doesexist some action set that is sufficient for KB that it willbe a best action set.

Theorem 5 If some complete action set ,4 is CK-sufficient for KB, then every best action set is CK-sufficient.

Note that the concept of CK-sufficiency can be appliedeven in the case of incomplete knowledge. When it ismeaningful, it must be that possible outcomes of un-known uncontrollable have no influence on preference(given best action sets): all relevant factors are known.

In [6] we describe various properties of these twostrategies.. In particular, we show that the adventur-ous and cautious strategies do indeed maximize poten-tial gain and minimize potential loss, respectively. Thecautious strategy seems applicable in a situation whereone expects the worst possible outcome, for example, ina game against an adversary. Once the agent has per-formed its action, it expects the worst possible outcome,so there is no advantage to discriminating among thecandidate (best) action sets: all have equally good worstoutcomes. However, it’s not clear that this is the beststrategy if the outcome of uncontrollables is essentially"random." If outcomes are simply determined by thenatural progression of events, then one should be moreselective. We think of nature as neither benevolent (a co-operative agent) nor malevolent (an adversary). There-:fore, even if we decide to be cautious (choosing frombest action sets), we should account for the fact thatthe worst outcome might not occur: we should choosethe action sets that take advantage of this fact. We arecurrently investigating such strategies in relation to, forinstance, the absolute goals of Doyle and Wellman [10].Indeed, it’s not hard to see that being a cautious goal is anecessary condition for being an absolute goal, but notsufficient. Other strategies provide useful (closer) ap-proximations to absolute goals. Such considerations alsoapply to games where an opponent might not be ableto consistently determine her best moves and an agentwants to exploit this fact. It should be clear, that thecombination of ability and incomplete knowledge alsohas a profound impact on how obligations (in the tradi-tional deontic sense) must be defined.

5.4 Observations

It should be clear that if an agent can observe the truthvalues of certain unknown propositions before it acts, itcan improve its decisions. Eliminating situations cannotmake the worst outcomes of action sets any worse (allcontingencies are accounted for in cautious goals); butin many cases, it will make the worst outcomes betterand change the actions chosen. To continue our pre-vious example, suppose R and C are unknown. Theagent’s cautious goal is then U. If it were in the agent’spower to determine C or C before acting, its actionscould change. Observing C indicates the impossibility

of R, and the agent could then decide to do(-U). In [6]we formalize this notion by distinguishing two types ofuncontrollable atoms: observables (like "cloudy") andunobservables (like "it will rain soon"). We describe thevalue of observations in terms of their effect on decisionsand possible outcomes (much like "value of information"in decision theory [23]).

6 Concluding Remarks

We have presented a modal logic of preferences that cap-tures the notion of "obligation" defined in the usual de-ontic logics. We have extended this logic with the abil-ity to represent normality and added to the system anaive account of action and ability. Within this quali-tative framework we have proposed various methods fordetermining the obligations and goals of an agent thataccount for the beliefs of an agent (including its defaultbeliefs), the agent’s ability, and the interaction of thetwo. We have shown that goals and obligations cannotbe uniquely defined in the presence of incomplete knowl-edge, rather that strategies for determining goals arise.

There are a number of ways in which this frameworkmust be extended. Clearly, the account of actions isnaive. True actions have preconditions, default or un-certain effects, and so on.14 We are currently extendingthe framework to include a representation of events andactions that have such properties. In this way, we can ac-count for planning under uncertainty with a qualitativerepresentation, and plan for conditional goals.

We would like to include observations as actions them-selves [11] and use these to characterize (dynamic) con-ditional goals and conditional plans. We hope to charac-terize the changes in an agent’s belief set, due to obser-vations and its knowledge of the effects of actions, usingbelief revision and update semantics. One drawback ofthis system is the extralogical account of action and abil-ity. We hope to embed the account of action and abilitydirectly in the object language (e.g., using the methodsof dynamic logic). Temporal notions also have a role toplay (see, e.g., McCarty [22]).

Finally, we need to explore extensions in which cer-tain preferences or "utilities" can be given precedence

14The default ."effects" of actions need not be causal. Con-.~ sider the the preference relation induced by the payoff ma-trix for the classic Prisoner’s Dilemma (where A means "ouragent" cooperates and O means the other agent cooperates):

AO < AO < AO < AO

Our agent’s best course of action .for any given choice by theother agent is not to cooperate. However, on the assumptionthat the other agent reasons similarly, our agent ends up ina suboptimal situation AO. Hence the dilemma: mutual co-operation would have been better (AO). To incorporate theassumption that the other agent reasons similarly, our agentmight hold two defaults: A =~ O and "-A =~ -~O. For thesedefaults to fill the appropriate role, they must be "applied"after the agent has made its choice: though apparently "unin-fluenceable," the truth value O must not persist after action.Given this, (should both agent’s use the same defaults) wehave a model of the Prisoner’s Dilemma that accounts formutual cooperation.

38

over expectations or "probabilities." The definition ofgoal in our system has the property that an agent shouldact as if every belief (including default beliefs) is true.This seems to be reasonable for the most part. But theconsequences of being wrong for certain acts may out-weigh the "probability" of being right (in the classic de-cision theoretic sense). For example, even if I believethat I can safely run across the busy freeway, the drasticconsequences of being wrong greatly outweigh the ben-efit of being right. We would like to capture this typeof trade-off in a qualitative way. Possible methods in-clude the "stratification" of propositions or situations togive priority to preferences in some cases, expectationsin others; or explicitly using the ranking information im-plicit in the ordering structure (as is done by Goldszmidtand Pearl [12, 13]) and comparing the qualitative de-gree of belief/expectation to the qualitative degree ofpreference. 15 This may lead to a concrete qualitativeproposal for decision-theoretic defaults [25], where a de-fault rule A ---* B means that "acting as if B" has higherexpected utility than not, given A.

References[1] Craig Boutilier. Conditional logics of normality as modal

systems. In Proc. of AAAI-90, pages 594-599, Boston,1990.

[2] Craig Boutilier. Inaccessible worlds and irrelevance:Preliminary report. In Proc. of IJCAI-91, pages 413-418, Sydney, 1991.

[3] Craig Boutilier. Conditional logics for default reason-ing and belief revision. Technical Report KRR-TR-92-1, University of Toronto, Toronto, January 1992. Ph.D.thesis.

[4] Craig Boutilier. A logic for revision and subjunctivequeries. In Proc. of AAAI-9P, pages 609-615, San Jose,1992.

[5] Craig Boutilier. What is a default priority? In Pro-ceedings of Canadian Society for Computational Studiesof Intelligence Conference, pages 140-147, Vancouver,1992.

[6] Craig Boutilier. Beliefs, ability and obligations: Aframework for conditional goals. Technical report, Uni-versity of British Columbia, Vancouver, 1993. (Forth-coming).

[7] Roderick M. Chisholm. Contrary-to-duty imperativesand deontic logic. Analysis, 24:33-36, 1963.

[8] Philip R. Cohen and Hector J. Levesque. Intention ischoice with commitment. Artificial Intelligence, 42:213-261, 1990.

[9] James P. Delgrande. An approach to default reasoningbased on a first-order conditional logic: Revised report.Artificial Intelligence, 36:63-90, 1988.

[10] Jon Doyle and Michael P. Wellman. Preferential seman-tics for goals. In Proc. of AAAI.91, pages 698-703, Ana-heim, 1991.

[11] Oren Etzioni and Steve Hanks. An approach to planningwith incomplete information. In Proc. of KR-92, pages115-125, Cambridge, 1992.

15This final suggestion is due to Judea Pearl (personalcommunication).

[12] Mois~s Goldszmidt and Judea Pearl. System Z+: Aformalism for reasoning with variable strength defaults.In Proc. o] AAAI-91, pages 399-404, Anaheim, 1991.

[13] Moisds Goldszmidt and Judea Pearl. Reasoning withqualitative probabilities can be tractable. In Proceedingsof the Eighth Conference on Uncertainty in AI, pages112-120, Stanford, 1992.

[14] Peter Haddawy and Steve Hanks. Representations fordecision-theoretic planning: Utility functions for dead-line goals. In Proc. of KR.9P, pages 71-82, Cambridge,1992.

[15] Bengt Hansson. Aa analysis of some deontic logics.Noas, 3:373-398, 1969.

[16] John F. Horty. Moral dilemmas and nonmonotonic logic.J. of Philosophical Logic, 1993. To appear.

[17] Andrew J. I. Jones and Ingmar PSrn. Ideality, sub-ideality and deontic logic. Synthese, 65:275-290, 1985.

[18] Andrew J. I. Jones and Ingmar Pfrn. On the logic ofdeontic conditionals. In Workshop on Deontic Logic inComputer Science, Amsterdam, 1991. To appear.

[19] David Lewis. Counterfactuals. Blackwell, Oxford, 1973.

[20] Barry Loewer and Marvin Belzer. Dyadic deontic de-tachment. Synthese, 54:295-318, 1983.

[21] David Makinson. Five faces of minimaiity. Studia Logica,1992. To appear.

[22] L. Thorne McCarty. Defensible deontic reasoning. InFourth International Workshop on Nonmonotonic Rea-soning, pages 139-147, Plymouth, VT, 1992.

[23] Judea Pearl. Probabilistic Reasoning in Intelligent Sys-tems: Networks of Plausible Inference. Morgan Kauf-mann, San Mateo, 1988.

[24] Judea Pearl. System Z: A natural ordering of de-faults with tractable applications to default reasoning.In M. Vardi, editor, Proceedings of Theoretical Aspectsof Reasoning about Knowledge, pages 121-135. MorganKaufmann, San Mateo, 1990.

[25] David Poole. Decision-theoretic defaults. In Proceedingsof Canadian Society for Computational Studies of Intel-ligence Conference, pages 190-197, Vancouver, 1992.

[26] Austin Tate, James Hendler, and Mark Drummond. Areview of AI planning techniques. In J. Allen, J. Hendler,and A. ’rate, editors, Readings in Planning, pages 26-49.Morgan-Kaufmann, San Mateo, 1990.

[27] Bas C. van Fraassen. The logic of conditional obligation.Journal of Philosophical Logic, 1:417-438, 1972.

[28] Georg Henrik yon Wright. Deontic logic. Mind, 60:1-15,1951.

[29] Georg Henrik von Wright. A new system of deonticlogic. In Risto Hilpinen, editor, Deontic Logic: Introduc-tory and Systematic Readings, pages 105-120. D.Reidel,Dordecht, 1964. 1981.

Acknowledgements

Thanks to Jon Doyle, Jeff Horty, Keiji Kanazawa, YggyKing, Hector Levesque, David Makinson, Thorne Mc-Carty, Judea Pearl, David Poole and Michael Well-man for very valuable discussions and suggestions. Thisresearch was supported by NSERC Research GrantOGP0121843.

39