92 AI MAGAZINE

Articles

Logic can be a powerful tool for reasoning about multiagentsystems. First of all, logics provide a language in which to speci-fy properties — properties of an agent, of other agents, and of theenvironment. Ideally, such a language then also provides ameans to implement an agent or a multiagent system, either bysomehow executing the specification, or by transforming thespecification into some computational form. Second, given thatsuch properties are expressed as logical formulas that form partof some inference system, they can be used to deduce other prop-erties. Such reasoning can be part of an individual agent’s capa-bilities, but it can also be done by a system designer or thepotential user of the agents.

Third, logics provide a formal semantics in which the sen-tences from the language are assigned a precise meaning: if onemanages to come up with a semantics that closely models thesystem under consideration, one then can verify properties eitherof a particular system (model checking) or of a number of simi-lar systems at the same time (theorem proving). This, in a nut-shell, sums up the three main characteristics of any logic (lan-guage, deduction, semantics), as well as the three main roleslogics play in system development (specification, execution, andverification).

We typically strive for logics that strike a useful balancebetweenexpressiveness on the one handand tractabilityon the oth-er: what kind of properties are interesting for the scenarios ofinterest, and how can they be “naturally” and concisely

Copyright © 2012, Association for the Advancement of Artificial Intelligence. All rights reserved. ISSN 0738-4602

Logics for Multiagent Systems

Wiebe van der Hoek and Michael Wooldridge

n We present a brief survey of logics for rea-soning about multiagent systems. We focus ontwo paradigms: logics for cognitive models ofagency, and logics used to model the strategicstructure of a multiagent system.

SUMMER 2012 93

Articles

dynamics is where things really become intriguing:there is not just “a future,” or a “possible futuredepending on an agent’s choice,” but how thefuture will look depends on the choices of severalagents at the same time. We will come across lan-guages in which one can express the following.

Let δ denote the fact that i and j are at differentfloors: δ = Vn≠ m, i≠ j (at(i, n) � at(m, j). Then ¬��i�� Fδ� ¬��j�� Fδ � ��i, j�� Fδ expresses that although i can-not bring about that eventually both lifts are ondifferent floors, and neither can j, by cooperatingtogether, i and j can guarantee that they both inthe end serve a different floor.

Logics for multiagent systems are typically inten-sional (in contrast to propositional and first-orderlogics, which are extensional). A logic is extension-al if the truth-value of a formula is completelydetermined by the truth-value of all its compo-nents. If we know the truth-value of p and q, wealso know that of (p � q), and of ¬p → (q → p). Forlogics of agency, extensionality is often not realis-tic. It might well be that “rain in Oxford” and “rainin Liverpool” are both true, while our agent knowsone without knowing the other. Even if one is giv-en the truth value of p and of q, one is not guaran-teed to be able to tell whether Bi (p � q) (agent ibelieves that p � q), whether F (p � q) (eventually,both p and q), or whether Bi G(p → Bhq) (i believesthat it is always the case that as soon as p holds,agent h believes that q also holds).

Those examples make clear why extensional log-ics are so successful for reasoning about multia-gent systems. However, perhaps the most com-pelling argument for the use of modal logics formodeling the scenarios we have in mind lies in thesemantics of modal logic. This is built around thenotion of a “state,” which can represent the state ofa system, of a processor, or some situation. Consid-ering several states at the same time is then rathernatural, and usually, they are related: some becausethey “look the same” for a given agent (they definethe agent’s beliefs), some because they are veryattractive (they comprise the agent’s desires), orsome of them may represent some state of affairs inthe future (they model possible evolutions of thesystem). Finally, some states are reachable onlywhen certain agents take certain decisions (thosestates determine what coalitions can achieve).States and their relations are mathematically repre-sented in Kripke models, to be defined shortly.

In the remainder of this section we demonstratesome basic languages, inference systems andsemantics that are foundational for logics ofagency. The rest of this overview is then organizedalong two main streams, reflecting the followingtwo key trends in multiagent systems research: cog-nitive models of rational action and models of thestrategic structure of the system.

The first main strand of research in logics for

expressed? How complex are the formalisms, interms of how costly is it to use the formalismwhen doing verification or reasoning with them?

In multiagent research, this complexity oftendepends on a number of issues. Let us illustratethis with a simple example, say the modeling of aset of lifts. If there is only one agent (lift)involved, the kind of things we would like to rep-resent to model the agent’s sensing, planning andacting could probably be done in a simple propo-sitional logic, using atoms like pos(i, n) (currentlylift i is positioned at floor n), callf(n) (there is a callfrom floor n), callt(n (there is a call within the liftto go to destination n), at(i, n) (lift i is currentlyat floor n), op (i) (the door of lift i is open) andup(i) (the lift is currently moving up). However,taking the agent perspective seriously, one quicklyrealizes that we need more: there might be a differ-ence between what is actually the case and whatthe agent believes is the case, and also betweenwhat the agent believes to hold and what hewould like to be true (otherwise there would be noreason to act!). So we would like to be able to saythings like ¬callf(n) � Bicallf (n) � Di(callf(n) →up(i)) (although there is no call from floor n, theagent believes there is one, and desires to establishthat in that case the agent moves up).

Obviously, things get more interesting whenseveral agents enter the scene. Our agent i needsnot only a model of the environment but also amodel of j’s mental state, the latter involving amodel of i’s mental state. We can then expressproperties like BiBjcallf(n) → ¬up(i) (if i believes thatj believes there is call from floor n, i will not go up).Higher-order information enters the picture, andthere is no a priori limit to the degree of nestingthat one might consider (this is for instance impor-tant in reasoning about games: see the discussionon common knowledge for establishing a Nashequilibrium [Aumann and Brandenburger 1995]).In our simple lift scenario, if for some reason lift iwould like not to go up to floor n, we might haveBicallf(n) � Bjcallf(n) � BiBjcallf(n) � DiBj ¬Bicallf(n)(although both lifts believe there is a call fromfloor n, i desires that j believe that i is not aware ofthis call).

Another dimension that complicates multiagentscenarios is their dynamics. The world changes,and the information, desires, and goals of theagents change as well. So we need tools to reasoneither about time, or else about actions explicitly.A designer of a lift is typically interested in proper-ties like ∀G¬(up(i) � op(i)) (this is a safety proper-ty, requiring that in all computations, it is alwaysthe case that the lift is not going up with an opendoor), and (callf(n) → ∀F Vi�Ag at(i, n)) (a livenessproperty, expressing that if there is a call from floorn, one of the lifts will eventually arrive there).Combining such aspects of multiagents and

94 AI MAGAZINE

Articles

state that an agent knows what it knows (Kn4) andknows what it doesn’t know (Kn5), respectively.Modus ponens (MP) is a standard logical rule, andnecessitation (Nec) guarantees that it is derivablethat agents know all tautologies.

Moving on to the semantics of such a logic,models for epistemic logic are tuples M = 〈S, Ri�Ag,V〉(also known as Kripke models), where S is a setof states, Ri ⊆ S × S is a binary relation for eachagent i, and V : At → S gives for each atom p � Atthe states V(p) where p is true. The fact that (s, s�)� Ri is taken to mean that i cannot tell states s ands� apart. The truth of � in a model M with state s,written as M, s �, is standard for the classical con-nectives (compare with figure 2), and the clause M,s Ki� means that for all t with Rist, M,t � holds.In other words, in state s agent i knows � iff � istrue in all states t that are indistinguishable to s fori. Ki is called the necessity operator for Ri. M �

means that for all states s � S, M, s �. Let S5 beall models in which each Ri is an equivalence rela-tion. Let S5 � mean that in all models M � S5, wehave M �. The system S5 is complete for thevalidities in S5, that is, for all �, S5 � iff S5 �.

Notice that it is possible to formalize groupnotions of knowledge, such as E� (“everybodyknows �,” that is, K1� � … � Km�), D� (“it is dis-tributed knowledge that �,” that is, if you wouldpool all the knowledge of the agents together, �would follow from it, like in (Ki (�1 → �2) � Kj �1)→ D�2), and C� (‘it is common knowledge that ��,which is axiomatized such that it resembles theinfinite conjunction E� � EE� � EEE� � …).

In epistemic logics, binary relations Ri on the setof states represent the agents’ ignorance; in tem-poral logics, however, they represent the flow oftime. In the most simple setting, we assume thattime has a beginning, and advances linearly anddiscretely into an infinite future: this is linear-timetemporal logic (LTL) (Pnueli 1977). So a simplemodel for time is obtained by taking as the set ofstates the natural numbers N, and for the accessi-bility relation “the successor of,” that is, R = {(n, n+ 1) : n � N} and V, the valuation, can be used tospecify specific properties in states. In the lan-guage, we then would typically see operators forthe “next state,” (X), for “all states in the future (G)and for “some time in the future” (F). The truthconditions for those operators, together with anaxiom system for them, are given in figure 2. Notethat G is the reflexive transitive closure of R, and Fis its dual: F� = ¬G¬�.

Often, one wants a more expressive language,adding for instance an operator (for “until”), say-ing M, n �Uψ iff �m ≥ n(M, m ψ & ∀k(n ≥ k ≥m ⇒M, k �)).

A rational agent deliberates about choices, andto represent those, branching time seems a moreappropriate framework than linear time. To under-

multiagent systems, cognitive models of rationalaction, focuses on the issue of representing the atti-tudes of agents within the system: their beliefs,aspirations, intentions, and the like. The aim ofsuch formalisms is to derive a model that predictshow a rational agent would go from its beliefs anddesires to actions. Work in this area builds largelyon research in the philosophy of mind. The logicalapproaches presented in the section on cognitivestates focus on this trend.

The second main strand of research, models ofthe strategic structure of the system, focuses on thestrategic structure of the environment: what agentscan accomplish in the environment, either togeth-er or alone. Work in this area builds on models ofeffectivity from the game theory community, andthe models underpinning such logics are closelyrelated to formal games. In the section “Represent-ing the Strategic Structure,” presented later on inthis article, we present logics that deal with thistrend.

A Logical ToolkitWe now very briefly touch upon the basic logics toreason about knowledge, about time and aboutaction. Let i be a variable over a set of agents Ag ={1, …, m}. For reasoning about knowledge or beliefof agents (we will not dwell here on their distinc-tion), one usually adds, for every agent i, an oper-ator Ki to the language, where Ki� then denotesthat agent i knows �. In the best-known epistemiclogics (Fagin et al. 1995; Meyer and van der Hoek1995), we see the axioms as given in figure 1.

This inference system is often referred to as S5.Axiom Kn1 is obvious, Kn2 denotes that an agentcan perform deductions upon what he or sheknows, Kn3 is often referred to as veridicality: whatone knows is true. If Kn3 is replaced by the weakerconstraint Kn3�, saying ¬Ki�, the result is a logicfor belief (where “i believes �” is usually writtenBi�) called KD45. Finally, Kn4 and Kn5 denote pos-itive and negative introspection, respectively: they

Knowledge Axioms Kn1 where is a propositional tautologyKn2 Ki ( →ψ) → (Ki → Kiψ ) Kn3 Ki → Kn4 Ki → Ki Ki Kn5 ¬Ki → Ki¬Ki

Rules of Inference MP , ( →ψ) ⇒ ψ Nec ) ⇒ Ki

Figure 1. An Inference System for Knowledge.

SUMMER 2012 95

Articles

Representing Cognitive StatesIn attempting to understand the behavior ofagents in the everyday world, we frequently makeuse of folk psychology, by which we predict andexplain the behavior of agents with reference totheir beliefs, desires, and intentions. The philoso-pher Dennett coined the phrase intentional systemto refer to an entity that can be understood interms of folk-psychology notions such as beliefs,desires, and the like (Dennett 1987). The inten-tional stance is essentially nothing more than anabstraction tool. If we accept the usefulness of theintentional stance for characterizing the propertiesof rational agents, then the next step in developinga formal theory of such agents is to identify thecomponents of an agent’s state. There are manypossible mental states that we might choose tocharacterize an agent: beliefs, goals, desires, inten-tions, commitments, fears, hopes are just a few. Wecan identify several important categories of suchattitudes, for example:

Information attitudes: those attitudes an agent has

stand branching time-operators though, an under-standing of linear time operators is still of benefit.Computational tree logic (CTL), (Emerson 1990) isa branching time logic that uses pairs of operators;the first quantifies over paths, the second is an LTLoperator over those paths. Let us demonstrate thisby mentioning some properties that are true in theroot ρ of the branching time model M of figure 3.Note that on the highlighted path, in ρ, the for-mula G¬q is true. Hence, on the branching modelM, ρ, we have EG¬q, saying that in ρ, there exists apath through it, on which q is always false. AF�

means that on every path starting in ρ, there issome future point where � is true. So, in ρ, AF¬pholds. Likewise, EpUq is true in ρ because there is apath (the path “up,” for example) in which pUq istrue. We leave it to the reader to check that in ρ, wehave EF(p � AG¬p).

Notice that in LTL and CTL, there is no notionof action: the state transition structure describeshow the world changes over time, but gives noindication of what causes transitions betweenstates. We now describe frameworks in which wecan be explicit about how change is brought about.The basic idea is that each state transition islabeled with an action — the action that causes thestate transition. More generally, we can label statetransitions with pairs (i, �), where i is an agent and� an action. The formalism discussed is based ondynamic logic (Harel, Kozen, and Tiuryn 2000).

Actions in the set Ac are either atomic actions (a,b, …) or composed (�, �, …) by means of test of for-mulas (�?), sequencing (�; �), conditioning (if �then � else �) and repetition (while � do �). Theinformal meaning of such constructs is as follows:

�? denotes a “test action” �, while �; � denotes �followed by �. The conditional action if � then �else � means that if � holds, � is executed, else �.Finally, the repetition action while � do � meansthat as long as � is true, � is executed.

Here, the test must be interpreted as a test by thesystem; it is not a so-called knowledge-producingaction (like observations or communication) thatcan be used by the agent to acquire knowledge.

These actions � can then be used to build newformulas to express the possible result of the exe-cution of � by agent i (the formula 〈doi(�)〉�denotes that � is a result of i’s execution of �), andthe opportunity for i to perform � (that is, 〈doi(�)〉T). The formula [doi(�)}� is shorthand for¬[doi(�)]¬�, thus expressing that one possible resultof performance of � by i implies �. In the Kripkesemantics, we then assume relations Ra for indi-vidual actions, where the relations for composi-tions are then recursively defined: for instance R�;�st iff for some state u, R�su and R�ut. Indeed, [doi(�)]is then the necessity operator for R�.

Truth Conditions of LTL M, n p iff n V(p) M, n ¬ iff not M, n M, n ψ iff M, n and M, n ψ M, n X iff M, n + 1 M, n G iff ∀m ≥ n, M, m M, n F iff m ≥ n, M, m

Figure 2. Semantics of Linear Temporal Logic.

p

p

qp

ρ

p

p q

Figure 3. A Branching Time Model.

96 AI MAGAZINE

Articles

knowing an action is rather important, and it isrelated to the distinction between knowledge de re(which involves knowing the identity of a thing)and de dicto (which involves knowing that some-thing exists) (Fagin et al. 1995, p. 101). In the safeexample, most people would have knowledge dedicto to open the safe, but only a few would haveknowledge de re.

It is often the case that actions are ontic: theybring about a change in the world, like assigning avalue to a variable, moving a block, or opening adoor. However, dynamic epistemic logic (DEL)(van Ditmarsch, van der Hoek, and Kooi 2007)studies actions that bring about mental change:change of knowledge in particular. So in DEL theactions themselves are epistemic. A typical exam-ple is publicly announcing � in a group of agents:[�]ψwould then mean that after announcement of�, it holds that ψ. Surprisingly enough, the formu-la [�]Ki� (after the announcement that �, agent iknows that �), is not a validity, a counterexamplebeing the infamous Moore (1942) sentences � =(¬Ki p � p): “although i does not know it, p holds”).Public announcements are a special case of DEL,which is intended to capture the interactionbetween the actions that an agent performs and itsknowledge.

There are several variants of dynamic epistemiclogic in the literature. In the language of van Dit-marsch, van der Hoek, and Kooi (2007), apart fromthe static formulas involving knowledge, there isalso the construct [�]�, meaning that after execu-tion of the epistemic action �, statement � is true.Actions � specify who is informed by what. Toexpress “learning,” actions of the form LB� areused, where � again is an action: this expresses thefact that “coalition B learns that � takes place”. Theexpression LB(� ! �), means the coalition B learnsthat either � or � is happening, while in fact �takes place.

To make the discussion concrete, assume wehave two agents, 1 and 2, and that they common-ly know that a letter on their table contains eitherthe information p or ¬p (but they don’t know, atthis stage, which it is). Agent 2 leaves the room fora minute, and, when he or she returns, is unsurewhether or not 1 read the letter. This action wouldbe described as

L12(L1 ?p ∪ L1 ?¬p ∪ !T)

which expresses the following. First of all, in factnothing happened (this is denoted by !T). Howev-er, the knowledge of both agents changes: theycommonly learn that 1 might have learned p, andhe or she might have learned ¬p.

We now show how the example can be inter-preted using the appealing semantics of Baltag andMoss (2004). In this semantics, both the uncer-tainty about the state of the world and that of theaction taking place are represented in two inde-

towards information about its environment. Themost obvious members of this category are knowl-edge and belief.

Pro attitudes: those attitudes an agent has that tendto lead it to perform actions. The most obviousmembers of this category are goals, desires, andintentions.

Normative attitudes: including obligations, permis-sions and authorization.

Much of the literature on developing formal the-ories of agency has been taken up with the relativemerits of choosing one attitude over another, andinvestigating the possible relationships betweenthese attitudes.

Knowledge and ChangeHaving epistemic and dynamic operators, one hasalready a rich framework to reason about agent’sknowledge about doing actions. For instance, aproperty like perfect recall

Ki[doi(�)]� → [doi(�)]Ki�,

which semantically implies some grid structure onthe set of states: If R�st and Ritu then for some v, wealso have Risv and R�vu. For temporal epistemiclogic, perfect recall is characterized by the axiomKiX� → XKi�, while its converse, no learning, isXKi � → KiX�. It is exactly such interaction prop-erties that can make a multiagent logic complex,both conceptually and computationally.

For studying the way that actions and knowl-edge interact, Robert Moore (1977, 1990) arguedthat one needs to identify two main issues. Thefirst is that some actions produce knowledge, andtherefore their effects must be formulated in termsof the epistemic states of participants. The secondis that of knowledge preconditions: what an agentneeds to know in order to be able to perform anaction. A simple example is that in order to unlocka safe, one must know the combination for thelock. Using these ideas, Moore formalized a notionof ability. He suggested that in order for an agentto be able to achieve some state of affairs �, theagent must either know the identity of an action �,(that is, have an “executable description” of anaction �) such that after � is performed, � holds; orelse know the identity of an action � such thatafter � is performed, the agent will know the iden-tity of an action �� such that after �� is performed,� holds.

The point about “knowing the identity” of anaction is that, in order for me to be able to becomerich, it is not sufficient for me simply to know thatthere exists some action I could perform thatwould make me rich; I must either know what thataction is (the first clause above), or else to able toperform some action that would furnish me withthe information about which action to perform inorder to make myself rich. This subtle notion of

SUMMER 2012 97

Articles

ing an action, and likewise, if two indistinguish-able actions � and � take place in a state s, they willgive rise to new states that can be distinguished.

Intention LogicOne of the best known, and most sophisticatedattempts to show how the various components ofan agent’s cognitive makeup could be combined toform a logic of rational agency is discussed inCohen and Levesque (1990). The logic has provedto be so useful for specifying and reasoning aboutthe properties of agents that it has been used in ananalysis of conflict and cooperation in multiagentdialogue (Galliers 1988), as well as in several stud-ies in the theoretical foundations of cooperativeproblem solving (Levesque, Cohen, and Nunes1990). This subsection will focus on the use of thelogic in developing a theory of intention. The firststep is to lay out the criteria that a theory of inten-tion must satisfy.

When building intelligent agents — particularlyagents that must interact with humans — it isimportant that a rational balance be achievedbetween the beliefs, goals, and intentions of theagents. For example, the following are desirableproperties of intention: An autonomous agentshould act on its intentions, not in spite of them;adopt intentions it believes are feasible and foregothose believed to be infeasible; keep (or commit to)intentions, but not forever; discharge those inten-tions believed to have been satisfied; alter inten-tions when relevant beliefs change; and adopt sub-sidiary intentions during plan formation (Cohenand Levesque 1990, p. 214).

Following Bratman (1987), Cohen and Levesqueidentify seven specific properties that must be sat-isfied by a reasonable theory of intention. Giventhese criteria, they adopt a two-tiered approach tothe problem of formalizing a theory of intention.First, they construct the logic of rational agency,“being careful to sort out the relationships amongthe basic modal operators” (Cohen and Levesque1990, p. 221). On top of this framework, theyintroduce a number of derived constructs, which

pendent Kripke models. The result of performingan epistemic action in an epistemic state is thencomputed as a “cross-product” (see figure 4). Mod-el N in figure 4 represents that it is common knowl-edge among 1 and 2 that both are ignorant aboutp. The triangular-shaped model N is the actionmodel that represents the knowledge and igno-rance when L12 (L1 ?p ∪ L1 ?¬p ∪ !T) is carried out.The points a, b, c of the model N are also calledactions, and the formulas accompanying the nameof the actions are called preconditions: the condi-tion that has to be fulfilled in order for the actionto take place. Since we are in the realm of truthfulinformation transfer, in order to perform an actionthat reveals p, the precondition p must be satisfied,and we write pre(b) = p. For the case of nothinghappening, only the precondition T need be true.Summarizing, action b represents the action thatagent 1 reads p in the letter, action c is the actionwhen ¬p is read, and a is for nothing happening. Aswith “static” epistemic models, we omit reflexivearrows, so that N indeed represents that p or ¬p islearned by 1, or that nothing happens: moreover itis commonly known between 1 and 2 that 1 knowswhich action takes place, while for 2 they all lookthe same

Now let M, w = (W, R1, R2, … Rm, π), w be a stat-ic epistemic state, and M, w an action in a finiteaction model. We want to describe what M, w⊕M,w = (W�, R�1, R�2 , … R�m, π�), w�, looks like — theresult of “performing” the action represented byM, w in M, w. Every action from M, w that is exe-cutable in any state v � W gives rise to a new statein W�: we let W� = {(v, v) | v � W, M, v pre(v)}.Since epistemic actions do not change any objec-tive fact in the world, we stipulate π�(v, v) = π(v).Finally, when are two states (v, v) and (u, u) indis-tinguishable for agent i? Well, agent i should beboth unable to distinguish the originating states(Riuv), and unable to know what is happening(Riuv). Finally, the new state w� is of course (w, w).Note that this construction indeed gives N, s ⊕ N,a = N�, (s, a), in our example of figure 4. Finally, letthe action � be represented by the action modelstate M, w. Then the truth definition under theaction model semantics reads that M, w [�]� iffM, w pre(w) implies (M, w) ⊕ (M, w) �. In ourexample: N, s [L12 (L1 ?p∪ L1 ?¬p∪ !T)]� iff N�, (s,a) �.

Note that the accessibility relation in the result-ing model is defined as

Ri (u, u)(v, v) � Riuv & Riuv (1)

This means that an agent cannot distinguish twostates after execution of an action �, if and only ifthe agent could not distinguish the “sources” ofthose states, and he does not know which actionexactly takes place. Put differently: if an agentknows the difference between two states s and t,then they can never look the same after perform-

1,2s tp p

N

2

a

b cp p

N2 2 =

(s, b)

(s, a)

(t, c)

(t, a)

2 2

N

2

2

p

p

p

p

Figure 4. Computing ⊕.

98 AI MAGAZINE

Articles

to actions, Happens and Done. The standard futuretime operators of temporal logic, “G” (always), and“F” (sometime) can be defined as abbreviations,along with a “strict” sometime operator, Later:

F� �x · (Happens x; �?)G� ¬F ¬�

(Later p) ¬p � Fp

A temporal precedence operator, (Before p q) canalso be derived, and holds if p holds before q. Animportant assumption is that all goals are eventu-ally dropped:

F ¬(Goal x (Later p))

The first major derived construct is a persistentgoal (figure 5). So, an agent has a persistent goal ofp if (1) it has a goal that p eventually becomes true,and believes that p is not currently true; and (2)before it drops the goal, one of the following con-ditions must hold: (a) the agent believes the goalhas been satisfied; or (b) the agent believes the goalwill never be satisfied.

It is a small step from persistent goals to a firstdefinition of intention, as in “intending to act.”Note that “intending that something becomestrue” is similar, but requires a slightly different def-inition (see Cohen and Levesque [1990]). An agenti intends to perform action � if it has a persistentgoal to have brought about a state where it had justbelieved it was about to perform �, and then did �.

(Intend i �) (P-Goal i[Done i (Bel i (Happens �))?; �])

Basing the definition of intention on the notion ofa persistent goal, Cohen and Levesque are able toavoid overcommitment or undercommitment. Anagent will only drop an intention if it believes thatthe intention has either been achieved, or isunachievable.

BDI LogicOne of the best-known approaches to reasoningabout rational agents is the belief desire intention(BDI) model (Bratman, Israel, and Pollack 1988).The BDI model gets its name from the fact that itrecognizes the primacy of beliefs, desires, andintentions in rational action.

Intuitively, an agent’s beliefs correspond to infor-mation the agent has about the world. Thesebeliefs may be incomplete or incorrect. An agent’sdesires represent states of affairs that the agentwould, in an ideal world, wish to be brought about.(Implemented BDI agents require that desires beconsistent with one another, although humandesires often fail in this respect.) Finally, an agent’sintentions represent desires that it has committedto achieving. The intuition is that an agent willnot, in general, be able to achieve all its desires,even if these desires are consistent. Ultimately, anagent must therefore fix upon some subset of itsdesires and commit resources to achieving them.

constitute a partial theory of rational action; inten-tion is one of these constructs.

Syntactically, the logic of rational agency is amany-sorted, first-order, multimodal logic withequality, containing four primary modalities (seetable 1). The semantics of Bel and Goal are giventhrough possible worlds, in the usual way: eachagent is assigned a belief accessibility relation, anda goal accessibility relation. The belief accessibilityrelation is Euclidean, transitive, and serial, giving abelief logic of KD45. The goal relation is serial, giv-ing a conative logic KD. It is assumed that eachagent’s goal relation is a subset of its belief relation,implying that an agent will not have a goal ofsomething it believes will not happen. A world inthis formalism is a discrete sequence of events,stretching infinitely into past and future. The sys-tem is only defined semantically, and Cohen andLevesque derive a number of properties from that.In the semantics, a number of assumptions areimplicit, and one might vary them. For instance,there is a fixed domain assumption, giving us suchproperties as∀x(Bel i �(x)) → (Bel i ∀x�(x)) (2)

The philosophically oriented reader will recog-nize a Barcan formula in equation 2, which in thiscase expresses that the agent is aware of all the ele-ments in the domain. Also, agents “know whattime it is,” from which we immediately obtain thevalidity of formulas like 2.30pm/3/6/85 → (Bel i2.30pm/3/6/85).

Intention logic has two basic operators to refer

Operator Meaning

(Bel i ) agent i believes

(Goal i ) agent i has goal of

(Happens ) action will happen next

(Done ) action has just happened

Table 1. Atomic Modalities in Cohen and Levesque’s Logic.

(P -Goal i p (=̂) Goal i (Later p))(Bel i ¬p)

Before((Bel i p) (Bel i G¬p))¬(Goal i (Later p))

Figure 5. Persistent Goal.

SUMMER 2012 99

Articles

We write B(w, t, i) to denote the set of worlds acces-sible to agent i from situation (w, t): B(w, t, i) = {w�| (w, t, w�) � B(i)}. We define D(w, t, i) and I (w, t, i)in the obvious way. The semantics of belief, desireand intention modalities are then given in theconventional manner, for example,

(w, t) (Bel i �) iff (w�, t) � for all w� � B(w, t, i).

The primary focus of Rao and Georgeff’s earlywork was to explore the possible interrelationshipsbetween beliefs, desires, and intentions from theperspective of semantic characterization. Forinstance, we can also consider whether the inter-section of accessibility relations is empty or not.For example, if B(w, t, i) ∩ I(w, t, i) ≠ �, for all i, w,t, then we get the following interaction axiom.

(Intend i �) → ¬(Bel i ¬�)

This axiom expresses an intermodal consistencyproperty.

But because the worlds in this framework have arich internal structure, we can undertake a morefine-grained analysis of the basic interactionsamong beliefs, desires, and intentions by consider-ing this structure, and so we are also able to under-take a more fine-grained characterization of inter-modal consistency properties by taking intoaccount the structure of worlds. Without going inthe semantic details, syntactically this allows oneto have properties like (Bel i A(�)) → (Des i A(�)): arelation between belief and desires that only holdsfor properties true on all future paths. Using thismachinery, one can define a range of BDI logicalsystems (see Rao and Georgeff [1998], p. 321).

Representing the Strategic StructureThe second main strand of research that wedescribe focuses not on the cognitive states ofagents, but on the strategic structure of the envi-ronment: what agents can achieve, either individ-ually or in groups. The starting point for such for-malisms is a model of strategic ability.

Over the past three decades, researchers frommany disciplines have attempted to develop a gen-eral-purpose logic of strategic ability. Within theartificial intelligence (AI) community, it was under-stood that such a logic could be used in order togain a better understanding of planning systems(Fikes and Nilsson 1971, Lifschitz 1986). The mostnotable early effort in this direction was Moore’sdynamic epistemic logic, referred to earlier (Moore1977, 1990). Moore’s work was subsequentlyenhanced by many other researchers, perhapsmost notably Morgenstern (1986). The distinctionsmade by Moore and Morgenstern also informedlater attempts to integrate a logic of ability intomore general logics of rational action inautonomous agents — see Wooldridge and Jen-nings (1995) for a survey of such logics.

These chosen desires, to which the agent has somecommitment, are intentions (Cohen and Levesque1990). The BDI theory of human rational actionwas originally developed by Michael Bratman(Bratman 1987). It is a theory of practical reason-ing — the process of reasoning that we all gothrough in our everyday lives, deciding momentby moment which action to perform next.

There have been several versions of BDI logic,starting in 1991 and culminating in the work byRao and Georgeff (1998); a book-length survey waswritten by Wooldridge (2000). We focus on the lat-ter.

Syntactically, BDI logics are essentially branch-ing time logics (CTL or CTL*, depending on whichversion one is reading about), enhanced with addi-tional modal operators Bel, Des, and Intend, forcapturing the beliefs, desires, and intentions ofagents respectively. The BDI modalities are indexedwith agents, so for example the following is a legit-imate formula of BDI logic:

(Bel i (Intend j AFp)) → (Bel i (Des j AFp))

This formula says that if i believes that j intendsthat p is inevitably true eventually, then i believesthat j desires p is inevitable. Although they sharemuch in common with Cohen and Levesque’sintention logics, the first and most obvious dis-tinction between BDI logics and the Cohen-Levesque approach is the explicit starting point ofCTLlike branching time logics. However, the dif-ferences are actually much more fundamental thanthis. The semantics that Rao and Georgeff give toBDI modalities in their logics are based on the con-ventional apparatus of Kripke structures and possi-ble worlds. However, rather than assuming thatworlds are instantaneous states of the world, oreven that they are linear sequences of states, it isassumed instead that worlds are themselvesbranching temporal structures: thus each worldcan be viewed as a Kripke structure for a CTLlikelogic. While this tends to rather complicate thesemantic machinery of the logic, it makes it possi-ble to define an interesting array of semantic prop-erties, as we shall see next.

We now summarize the key semantic structuresin the logic. Instantaneous states of the world aremodeled by time points, given by a set T; the set ofall possible evolutions of the system being mod-eled is given by a binary relation R⊆ T× T. A world(over T and R) is then a pair (T�, R�), where T� ⊆ Tis a nonempty set of time points, and R� ⊆ R is abranching time structure on T�. Let W be the set ofall worlds over T. A pair (w, t), where w = (Tw, Rw) �W and t � Tw, is known as a situation. If w � W,then the set of all situations in w is denoted by Sw.We have belief accessibility relations B, D, and I,modeled as functions that assign to every agent arelation over situations. Thus, for example:

B : Ag → ℘(W × T × W)

100 AI MAGAZINE

Articles

properties of the games and axioms of the logic,gave complete axiomatizations of the variousresulting logics, determined the computationalcomplexity of the satisfiability and model check-ing problems for his logics, and in addition,demonstrated how these logics could be applied tothe formal specification and verification of socialchoice procedures. The basic modal operator inPauly’s logic is of the form [C]�, where C is a set ofagents (that is, a subset of the grand coalition Ag),and � is a sentence; the intended reading is that “Ccan cooperate to ensure that �.”

The semantics of cooperation modalities are giv-en in terms of an effectivity function, whichdefines for every coalition C the states that C cancooperate to bring about; the effectivity function E: S → (P(Ag) → P(P(S))), gives, for any state t andcoalition C a set of sets of end-states EC(t), with theintended meaning of S � EC(t) that C can enforcethe outcome to be in S (although C may not beable to pinpoint the exact outcome that emergeswith this choice; this generally depends on thechoices of agents outside C, or “choices” made bythe environment). This effectivity function comeson a par with a modal operator [C] with truth defi-nition

t [C ]� iff for some S � EC(t) : for all s(s � iff s � S)

In words: coalition is effective for, or can enforce� if there is a set of states S that it is effective for,that is, which it can choose, which is exactly thedenotation of �: S = [|�|]. It seems reasonable tosay that C is also effective for � if it can choose a setof states S that “just” guarantees �, that is, forwhich we have S ⊆ [|�|]. This will be taken care ofby imposing monotonicity on effectivity func-tions: we will discuss constraints on effectivity atthe end of this section.

In games and other structures for cooperativeand competitive reasoning, effectivity functionsare convenient when one is interested in the out-comes of the game or the encounter, and not somuch about intermediate states, or how a certainstate is reached. Effectivity is also a level in whichon can decide whether two interaction scenariosare the same. The two games G1 and G2 from figure6 are “abstract” in the sense that they do not leadto payoffs for the players but rather to states thatsatisfy certain properties, encoded with proposi-tional atoms p, q and u. Such atoms could refer towhich player is winning, but also denote otherproperties of an end-state, such as some distribu-tion of resources, or “payments.” Both games aretwo-player games: in G1, player A makes the firstmove, which he choses from L (Left) and R (Right).In that game, player E is allowed to chose betweenl and r, respectively, but only if A plays R: other-wise the game ends after one move in the state sat-isfying p. In game G2, both players have the same

In a somewhat parallel thread of research,researchers in the philosophy of action developeda range of logics underpinned by rather similarideas and motivations. A typical example is that ofBrown, who developed a logic of individual abilityin the mid-1980s (Brown 1988). Brown’s mainclaim was that modal logic was a useful tool for theanalysis of ability, and that previous — unsuccess-ful — attempts to characterize ability in modal log-ic were based on an oversimple semantics. Brown’saccount (Brown 1988, p. 5) of the semantics ofability was as follows:

[An agent can achieve A] at a given world iff thereexists a relevant cluster of worlds, at every world ofwhich A is true.

Notice the �∀ pattern of quantifiers in thisaccount. Brown immediately noted that this gavethe resulting logic a rather unusual flavor, neitherproperly existential nor properly universal (Brown1988, p. 5):

Cast in this form, the truth condition [for ability]involves two metalinguistic quantifiers (one exis-tential and one universal). In fact, [the character ofthe ability operator] should be a little like each.

More recently, there has been a surge of interest inlogics of strategic ability, which has been sparkedby two largely independent developments: Pauly’sdevelopment of coalition logic (Pauly 2001), andthe development of ATL by Alur, Henzinger, andKupferman (Alur, Henzinger, and Kupferman2002; Goranko 2001). Although these logics arevery closely related, the motivation and back-ground to the two systems is strikingly different.

Coalition LogicPauly’s coalition logic was developed in an attemptto shed some light on the links between logic – andin particular, modal logic – and the mathematicaltheory of games (Osborne and Rubinstein 1994).Pauly showed how the semantic structures under-pinning a family of logics of cooperative abilitycould be formally understood as games of varioustypes; he gave correspondence results between

ρ1

L R

l r

G1

l r

L R L R

A

Ep

uq

a

c d

E

A A

p q p q

ρ2G2

Figure 6. Two Games G1 and G2 That Are the Same in Terms of Effectivity.

SUMMER 2012 101

Articles

to � means that no matter what the other agentsdo, C have a choice such that, if they make thischoice, then � will be true. Note the “∀∃” patternof quantifiers: C are implicitly allowed to maketheir choice while being aware of the choice madeby Cc. We will come back to the pairs of � and �ability in the CL-PC subsection.

We end this section by mentioning some prop-erties of �-abilities. The axioms for [C]� based on �-effectivity (or effectivity, for short) are summarizedin figure 7; see also Pauly (2002). The two extremecoalitions � and the grand coalition Ag are of spe-cial interest. [Ag]� expresses that some end-statesatisfies �, whereas [�]� holds if no agent needs todo anything for � to hold in the next state.

Some of the axioms of coalition logic corre-spond to restrictions on effectivity functions E: S→ (P(Ag) → P(P(S))). First of all, we demand that� � EC (this guarantees axiom �). The function Eis also assumed to be monotonic: For every coali-tion C ⊆ Ag, if X ⊆ X� ⊆ S, X � E(C) implies X� �E(C). This says that if a coalition can enforce anoutcome in the set X, it also can guarantee the out-come to be in any superset X� of X (this corre-sponds to axiom (M)). An effectivity function E isC-maximal if for all X, if Xc � E(Cc) then X � E(C).In words, if the other agents Cc cannot guaranteean outcome outside X (that is, in Xc), then C is ableto guarantee an outcome in X. We require effectiv-ity functions to be Ag-maximal. This enforcesaxiom (N) — Pauly’s symbol for the grand coalitionis N: if the empty coalition cannot enforce an out-come satisfying �, then the grand coalition Ag canenforce �. The final principle governs the forma-tion of coalitions. It states that coalitions can com-bine their strategies to (possibly) achieve more: E issuperadditive if for all X1, X2, C1, C2 such that C1 ∩C2 = �, X1 � E(C1) and X2 � E(C2) imply that X1 ∩X2 � E(C1 ∪ C2). This obviously corresponds toaxiom (S).

Strategic Temporal Logic: ATLIn coalition logic one reasons about the powers of

repertoire of choices, but the order in which theplayers choose is different. It looks like in G1 play-er A can hand over control to E, while the converseseems to be true for G2. Moreover, in G2, the play-er who is not the initiator (that is, player A), will beallowed to make a choice, no matter the choice ofhis opponent.

Despite all these differences between the twogames, when we evaluate them with respect towhat each coalition can achieve, they are thesame! To become a little more precise, let us definethe powers of a coalition in terms of effectivityfunctions E. In game G1, player A’s effectivity givesEA (ρ1) = {{a}, {c, d}}. Similarly, player E’s effectivityyields {{a, c}, {a, d}}: player E can enforce the gameto end in a or c (by playing l), but can also force theend-state among a and d (by playing r). Obviously,we also have E{A,E}(ρ1) = {{a}, {c}, {d}}: players A andE together can enforce the game to end in any end-state. When reasoning about this, we have torestrict ourselves to the properties that are true inthose end states. In coalition logic, what we havejust noted semantically would be described as:

G1 [A]p � [A](q � u) � [E](p � q) � [E](p � u)�[A, E]p � [A, E]q � [A, E]r

Being equipped with the necessary machinery, itnow is easy to see that the game G2 verifies thesame formula; indeed, in terms of what proposi-tions can be achieved, we are in a similar situationas in the previous game: E is effective for {p, q} (byplaying l) and also for {p, u} (play r). Likewise, A iseffective for {p} (play L) and for {q, u} (play R). Thealert reader will have recognized the logical law (p� (q � r)) � ((p � q) � (p � u)) resembling the“equivalence” of the two games: (p � (q � r)) cor-responds to A’s power in G1, and ((p � q) � (p � u))to A’s power in G2. Similarly, the equivalence of E’spowers is reflected by the logical equivalence (p �(q � r)) � ((p � q) � (p � u)).

At the same time, the reader will have recog-nized the two metalinguistic quantifiers in the useof the effectivity function E, laid down in its truth-definition. A set of outcomes S is in EC iff for somechoice of C, we will end up in S, under all choicesof the complement of C (the other agents). Thisnotion of so-called �-effectivity uses the ∃∀-orderof the quantifiers: what a coalition can establishthrough the truth-definition above, their �-ability,is sometimes also called ∃∀-ability. Implicit with-in the notion of �-ability is the fact that C have noknowledge of the choice that the other agentsmake; they do not see the choice of Cc (that is, thecomplement of C), and then decide what to do,but rather they must make their decision first. Thismotivates the notion of �-ability (that is, “∀∃”-ability): coalition C is said to have the �-ability for� if for every choice �D available to Cc, there existsa choice �C for C such that if Cc choose �D and Cchoose �C, then � will result. Thus C being �-able

¬[C(N) ¬[ ]¬w → [Ag]w(M) [C](w ψ ) → [C]ψ(S) ([C1]w1 [C2] w2) → [C1 ∪ C2](w1 ∧ w2 )

where C1 C2 =

(MP) from wand w infer(Nec ) from w infer [C]w

(⊥)

Figure 7. The Axioms and Inference Rules of Coalition Logic.

coalitions with respect to final outcomes. Howev-er, in many multiagent scenarios, the strategic con-siderations continue during the process. It wouldbe interesting to study a representation languagefor interaction that is able to express the temporaldifferences in the two games G1 and G2 of figure 6.Alternating-time temporal logic (ATL) is intendedfor this purpose.

Although it is similar to coalition logic, ATLemerged from a very different research communi-ty, and was developed with an entirely different setof motivations in mind. The development of ATLis closely linked with the development of branch-ing-time temporal logics for the specification andverification of reactive systems (Emerson 1990;Vardi 2001). Recall that CTL combines path quan-tifiers “A” and “E” for expressing that a certainseries of events will happen on all paths and onsome path respectively, and combines these withtense modalities for expressing that something willhappen eventually on some path (F), always onsome path (G) and so on. Thus, for example, usingCTL logics, one may express properties such as “onall possible computations, the system never entersa fail state” (AG¬fail). CTLlike logics are of limitedvalue for reasoning about multiagent systems, inwhich system components (agents) cannot beassumed to be benevolent, but may have compet-ing or conflicting goals. The kinds of properties wewish to express of such systems are the powers thatthe system components have. For example, wemight wish to express the fact that “agents 1 and 2can cooperate to ensure that the system neverenters a fail state.” It is not possible to capture suchstatements using CTLlike logics. The best one cando is either state that something will inevitablyhappen, or else that it may possibly happen: CTL-like logics have no notion of agency.

Alur, Henzinger, and Kupferman developed ATLin an attempt to remedy this deficiency. The keyinsight in ATL is that path quantifiers can bereplaced by cooperation modalities: the ATLexpression ��C���, where C is a group of agents,expresses the fact that the group C can cooperateto ensure that �. (Thus the ATL expression ��C���corresponds to Pauly’s [C]�.) So, for example, thefact that agents 1 and 2 can ensure that the systemnever enters a fail state may be captured in ATL bythe following formula: ��1, 2�� G¬fail. An ATL for-mula true in the root ρ1 of game G1 of figure 6 is��A�� X ��E�� �Xq: A has a strategy (that is, play R in ρ1)such that in the next time, E has a strategy (play l)to enforce u.

Note that ATL generalizes CTL because the pathquantifiers A (“on all paths… ”) and E (“on somepaths …”) can be simulated in ATL by the cooper-ation modalities ����� (“the empty set of agents cancooperate to …”) and ��Ag�� (“the grand coalitionof all agents can cooperate to …”).

One reason for the interest in ATL is that itshares with its ancestor CTL the computationaltractability of its model-checking problem (Clarke,Grumberg, and Peled 2000). This led to the devel-opment of an ATL model-checking system calledMOCHA (Alur et al. 2000).

ATL has begun to attract increasing attention asa formal system for the specification and verifica-tion of multiagent systems. Examples of such workinclude formalizing the notion of role using ATL(Ryan and Schobbens 2002), the development ofepistemic extensions to ATL (van der Hoek andWooldridge 2002), and the use of ATL for specify-ing and verifying cooperative mechanisms (Paulyand Wooldridge 2003).

A number of variations of ATL have been pro-posed over the past few years, for example to inte-grate reasoning about obligations into the basicframework of cooperative ability (Wooldridge andvan der Hoek 2005), to deal with quantificationover coalitions (Agotnes, van der Hoek, andWooldridge 2007), adding the ability to refer tostrategies in the object language (van der Hoek,Jamroga, and Wooldridge 2005), and adding theability to talk about preferences or goals of agents(Agotnes, van der Hoek, and Wooldridge 2006b;2006a).

CL-PCBoth ATL and coalition logic are intended as gen-eral-purpose logics of cooperative ability. In partic-ular, neither has anything specific to say about theorigin of the powers that are possessed by agentsand the coalitions of which they are a member.These powers are just assumed to be implicitlydefined within the effectivity structures used togive a semantics to the languages. Of course, if wegive a specific interpretation to these effectivitystructures, then we will end up with a logic withspecial properties. In a paper by van der Hoek andWooldridge (2005b), a variation of coalition logicwas developed that was intended specifically toreason about control scenarios, as follows. Thebasic idea is that the overall state of a system ischaracterized by a finite set of variables, which forsimplicity are assumed to take Boolean values.Each agent in the system is then assumed to con-trol some (possibly empty) subset of the overall setof variables, with every variable being under thecontrol of exactly one agent. Given this setting, inthe coalition logic of propositional control (CL-PC), the operator FC� means that there exists someassignment of values that the coalition C can giveto the variables under its control such that, assum-ing everything else in the system remainsunchanged, then if they make this assignment,then � would be true. The box dual GC� is definedin the usual way with respect to the diamond abil-ity operator FC. Here is a simple example:

Articles

102 AI MAGAZINE

Suppose the current state of the system is thatvariables p and q are false, while variable r is true,and further suppose then agent 1 controls p and r,while agent 2 controls q. Then in this state, wehave for example: F1(p � r), ¬F1q, and F2(q � r).Moreover, for any satisfiable propositional logicformula ψ over the variables p, q, and r, we haveF1,2ψ.

The ability operator FC in CL-PC thus capturescontingent ability, rather along the lines of “classi-cal planning” ability (Lifschitz 1986): ability underthe assumption that the world only changes by theactions of the agents in the coalition operator FC.Of course, this is not a terribly realistic type of abil-ity, just as the assumptions of classical planningare not terribly realistic. However, in CL-PC, wecan define � effectivity operators ��C����, intendedto capture something along the lines of the ATL��C��X�, as follows:

��C���� FCGD�, where D = Cc

Notice the quantifier alternation pattern �∀ inthis definition, and compare this to our discussionregarding �- and �-effectivity. Building on thisbasic formalism, van der Hoek and Wooldridge(2005a) investigate extensions into the possibilityof dynamic control, where variables can be“passed” from one agent to another.

Conclusion and Further ReadingIn this article, we have motivated and introduceda number of logics of rational agency; moreover,we have investigated the roles that such logicsmight play in the development of artificial agents.We hope to have demonstrated that logics forrational agents are a fascinating area of study, atthe confluence of many different research areas,including logic, artificial intelligence, economics,game theory, and the philosophy of mind. We alsohope to have illustrated some of the popularapproaches to the theory of rational agency.

There are far too many research challenges opento identify in this article. Instead, we simply notethat the search for a logic of rational agency posesa range of deep technical, philosophical, and com-putational research questions for the logic com-munity. We believe that all the disparate researchcommunities with an interest in rational agencycan benefit from this search.

We presented logics for MAS from the point ofview of modal logics. A state-of-the-art book onmodal logic was written by Blackburn, van Ben-them, and Wolter (2006), and, despite its maturity,the field is still developing. The references Fagin etal. (1995) and Meyer and van der Hoek (1995) arereasonably standard for epistemic logic in com-puter science: the modal approach modelingknowledge goes back to Hintikka (1962) though.In practical agent applications, information is

more quantitative then just binary represented asknowledge and belief though. For a logicalapproach to reasoning about probabilities seeHalpern (2003).

In the Representing Cognitive States section,apart from the references mentioned, there aremany other approaches: in the KARO framework(van Linder, van der Hoek, and Meyer 1998) forinstance, epistemic logic and dynamic logic arecombined (there is work on programming KAROagents [Meyer et al. 2001] and on verifying them[Hustadt et al. 2001]). Moreover, where we indicat-ed in the logical toolkit above how epistemicnotions can have natural “group variants,” Aldew-ereld, van der Hoek, and Meyer (2004) define somegroup proattitudes in the KARO setting. And, inthe same way as epistemics becomes interesting ina dynamic or temporal setting (see our toolkit),there is work on logics that address the temporalaspects and the dynamics of intentions as well(van der Hoek, Jamroga, and Wooldridge 2007)and, indeed, on the joint revision of beliefs andintentions (Icard, Pacuit, and Shoham 2010).

Where our focus in the section on cognitivestates was on logics for cognitive attitudes, due tospace constraints we have neglected the social atti-tude of norms. Deontic logic is another example ofa modal logic with roots in philosophy with workby von Wright (1951), which models attitudes likepermissions and obligations for individual agents.For an overview of deontic logic in computer sci-ence, see Meyer and Wieringa (1993), for a propos-al to add norms to the social, rather than the cog-nitive aspects of a multiagent system, see, forexample, van der Torre (2001).

There is also work on combining deontic con-cepts with for instance knowledge (Lomuscio andSergot 2003) and the ATL-like systems we present-ed in the section on strategic structures: (Agotneset al. 2009) for instance introduce a multidimen-sional CTL, where roughly, dimensions correspondwith the implementation of a norm. The formalstudy of norms in multiagent systems has arguablyset off with work by Shoham and Tennenholtz(1992b, 1992a). In normative systems, norms arestudied more from the multiagent collective per-spective, where questions arise like: which normswill emerge, why would agents adhere to them,when is a norm “better” than another one.

There is currently a flurry of activity in logics toreason about games (see van der Hoek and Pauly[2006] for an overview paper) and modal logics forsocial choice (see Daniëls [2010] for an example).Often those are logics that refer to the informationof the agents (“players,” in the case of games), andtheir actions (“moves” and “choices,” respective-ly). The logics for such scenarios are composedfrom the building blocks described in this article,with often an added logical representation of pref-

Articles

SUMMER 2012 103

erences (van Benthem, Girard, and Roy 2009) orexpected utility (Jamroga 2008).

ReferencesAgotnes, T.; van der Hoek, W.; Rodríguez-Aguilar, J.; Sier-ra, C.; and Wooldridge, M. 2009. Multi-Modal CTL: Com-pleteness, Complexity, and an Application. Studia Logica92(1): 1–26.

Agotnes, T.; van der Hoek, W.; and Wooldridge, M. 2006a.On the Logic of Coalitional Games. In Proceedings of the5th International Joint Conference on Autonomous Agentsand Multiagent Systems. New York: Association for Com-puting Machinery.

Agotnes, T.; van der Hoek, W.; and Wooldridge, M.2006b. Temporal Qualitative Coalitional Games. In Pro-ceedings of the 5th International Joint Conference onAutonomous Agents and Multiagent Systems. New York:Association for Computing Machinery.

Agotnes, T.; van der Hoek, W.; and Wooldridge, M. 2007.Quantified Coalition Logic. In Proceedings of the 20thInternational Joint Conference on Artificial Intelligence. Men-lo Park, CA: AAAI Press.

Aldewereld, H.; van der Hoek, W.; and Meyer, J.-J. 2004.Rational Teams: Logical Aspects of Multi-Agent Systems.Fundamenta Informaticae 63(2-3): 159–183.

Alur, R.; de Alfaro, L.; Henzinger, T. A.; Krishnan, S. C.;Mang, F. Y. C.; Qadeer, S.; Rajamani, S. K.; and Tasiran, S.2000. MOCHA User Manual. Technical Report, Depart-ment of Electrical Engineering and Computer Science,University of California, Berkeley.

Alur, R.; Henzinger, T. A.; and Kupferman, O. 2002. Alter-nating-Time Temporal Logic. Journal of the ACM 49(5):672–713.

Aumann, R., and Brandenburger, A. 1995. EpistemicConditions for Nash Equilibrium. Econometrica 63(5):1161–1180.

Baltag, A., and Moss, L. S. 2004. Logics for Epistemic Pro-grams. Synthese 139(2): 165–224.

Blackburn, P.; van Benthem, J.; and Wolter, F., eds. 2006.Handbook of Modal Logic. Amsterdam: Elsevier.

Bratman, M. E.; Israel, D. J.; and Pollack, M. E. 1988.Plans and Resource-Bounded Practical Reasoning. Com-putational Intelligence 4(4): 349–355.

Bratman, M. E. 1987. Intention, Plans, and Practical Rea-son. Cambridge, MA: Harvard University Press

Brown, M. A. 1988. On the Logic of Ability. Journal ofPhilosophical Logic 17(1): 1–26.

Clarke, E. M.; Grumberg, O.; and Peled, D. A. 2000. Mod-el Checking. Cambridge, MA: The MIT Press.

Cohen, P. R., and Levesque, H. J. 1990. Intention IsChoice with Commitment. Artificial Intelligence 42(2–3):213–261.

Daniëls, T. R. 2010. Social Choice and the Logic of Sim-ple. Journal of Logic and Computation 20(4).

Dennett, D. C. 1987. The Intentional Stance. Cambridge,MA: The MIT Press.

Drimmelen, G. 2003. Satisfiability in Alternating-timeTemporal Logic. In Eighteenth Annual IEEE Symposium onLogic in Computer Science (LICS 2003), 208–217. Piscat-away, NJ: Institute of Electrical and Electronics Engineers.

Emerson, E. A. 1990. Temporal and Modal Logic. InHandbook of Theoretical Computer Science Volume B: For-mal Models and Semantics, ed. J. van Leeuwen, 996–107.Amsterdam: Elsevier.

Fagin, R.; Halpern, J. Y.; Moses, Y.; and Vardi, M. Y. 1995.Reasoning About Knowledge. Cambridge, MA: The MITPress

Fikes, R. E., and Nilsson, N. 1971. STRIPS: A NewApproach to the Application of Theorem Proving to Prob-lem Solving. Artificial Intelligence 2(3–4): 189–208.

Galliers, J. R. 1988. A Theoretical Framework for Com-puter Models of Cooperative Dialogue, AcknowledgingMulti-Agent Conflict. Ph.D. Dissertation, Open Universi-ty, UK.

Goranko, V. 2001. Coalition Games and Alternating Tem-poral Logics. In Proceedings of the 8th Conference on Theo-retical Aspects of Rationality and Knowledge, 259–272. SanFrancisco: Morgan Kaufmann Publishers.

Halpern, J. 2003. Reasoning about Uncertainty. Cambridge,MA: The MIT Press.

Harel, D.; Kozen, D.; and Tiuryn, J. 2000. Dynamic Logic.Cambridge, MA: The MIT Press.

Hintikka, J. 1962. Knowledge and Belief. Ithaca, NY: Cor-nell University Press.

Hustadt, U.; Dixon, C.; Schmidt, R.; Fisher, J.-J. M.; andvan der Hoek, W. 2001. Verification with the KARO AgentTheory (extended abstract). In Formal Approaches toAgent-Based Systems, First International Workshop, FAABS2000, Lecture Notes in Computer Science 1871, 33–47.Berlin: Springer.

Icard, T.; Pacuit, E.; and Shoham, Y. 2010. Joint Revisionof Beliefs and Intention. In Proceedings of the 12th Inter-national Conference on Principles of Knowledge Representa-tion and Reasoning, 572–574. Menlo Park, CA: AAAI Press.

Jamroga, W. 2008. A Temporal Logic for Markov Chains.In Proceedings of the 7th International Joint Conference onAutonomous Agents and Multiagent Systems, 697–704. Rich-land, SC: International Foundation for AutonomousAgents and Multiagent Systems.

Levesque, H. J.; Cohen, P. R.; and Nunes, J. H. T. 1990. OnActing Together. In Proceedings of the 8th National Confer-ence on Artificial Intelligence, 94–99. Menlo Park, CA: AAAIPress.

Lifschitz, V. 1986. On the Semantics of STRIPS. In Rea-soning About Actions and Plans: Proceedings of the 1986Workshop, ed. M. P. Georgeff and A. L. Lansky, 1–10. SanMateo, CA: Morgan Kaufmann Publishers.

Lomuscio, A., and Sergot, M. 2003. Deontic InterpretedSystems. Studia Logica 75(1): 63–92.

Meyer, J.-J. C., and van der Hoek, W. 1995. Epistemic Log-ic for AI and Computer Science. Cambridge: CambridgeUniversity Press.

Meyer, J.-J. C., and Wieringa, R. J., eds. 1993. Deontic Log-ic in Computer Science — Normative System Specification.New York: John Wiley & Sons.

Meyer, J.-J. C.; de Boer, F.; van Eijk, R.; Hindriks, K.; andvan der Hoek, W. 2001. On Programming KARO Agents.Logic Journal of the IGPL 9(2): 245–256.

Moore, G. E. 1942. A Reply to My Critics. In The Philoso-phy of G. E. Moore, The Library of Living Philosophers, vol 4,535–677. Evanston, Illinois: Northwestern University.

Moore, R. C. 1977. Reasoning about Knowledge and

Articles

104 AI MAGAZINE

Action. In Proceedings of the 5th International Joint Confer-ence on Artificial Intelligence. Los Altos, CA: William Kauf-mann, Inc.

Moore, R. C. 1990. A Formal Theory of Knowledge andAction. In Readings in Planning, ed. J. F. Allen, J. Hendler,and A. Tate. San Mateo, CA: Morgan Kaufmann Publish-ers, 480–519.

Morgenstern, L. 1986. A First-Order Theory of Planning,Knowledge, and Action. In Proceedings of the 1st Confer-ence on Theoretical Aspects of Reasoning about Knowledge,99–114. San Mateo, CA: Morgan Kaufmann Publishers.

Osborne, M. J., and Rubinstein, A. 1994. A Course in GameTheory. The MIT Press: Cambridge, MA.

Pauly, M., and Wooldridge, M. 2003. Logic for Mecha-nism Design — A Manifesto. Paper presented at the 2003Workshop on Game Theory and Decision Theory inAgent Systems, Melbourne, Australia, 14 July.

Pauly, M. 2001. Logic for Social Software. Ph.D. Disserta-tion, University of Amsterdam. ILLC Dissertation Series2001-10. Amsterdam, The Netherlands.

Pauly, M. 2002. A Modal Logic for Coalitional Power inGames. Journal of Logic and Computation 12(1): 149–166.

Pnueli, A. 1977. The Temporal Logic of Programs. In Pro-ceedings of the Eighteenth IEEE Symposium on the Founda-tions of Computer Science, 46–57. Piscataway, NJ: Instituteof Electrical and Electronics Engineers.

Rao, A. S., and Georgeff, M. 1998. Decision Procedures forBDI Logics. Journal of Logic and Computation 8(3):293–344.

Ryan, M., and Schobbens, P.-Y. 2002. Agents and Roles:Refinement in Alternating-Time Temporal Logic. In Intel-ligent Agents VIII, Lecture Notes in Computer Science2333, ed. J. Meyer and M. Tambe, 100–114. Berlin:Springer.

Shoham, Y., and Tennenholtz, M. 1992a. Emergent Con-ventions in Multi-Agent Systems. In Proceedings of the 3rdInternational Conference on Principles of Knowledge Repre-sentation and Reasoning, 225–231. San Francisco: MorganKaufmann.

Shoham, Y., and Tennenholtz, M. 1992b. On the Syn-thesis of Useful Social Laws for Artificial Agent Societies.In Proceedings of the Tenth National Conference on ArtificialIntelligence. Menlo Park, CA: AAAI Press.

van Benthem, J.; Girard, P.; and Roy, O. 2009. Every-thing Else Being Equal: A Modal Logic Approach toCeteris Paribus Preferences. Journal of Philosophical Logic38(1): 83–125.

van der Hoek, W., and Pauly, M. 2006. Modal Logic forGames and Information. In Handbook of Modal Logic, ed.P. Blackburn, J. van Benthem, and F. Wolter, 1077–1148.Amsterdam: Elsevier.

van der Hoek, W., and Wooldridge, M. 2002. TractableMulti-Agent Planning for Epistemic Goals. In Proceedingsof the First International Joint Conference on AutonomousAgents and Multiagent Systems, 1167–1174. New York:Association for Computing Machinery.

van der Hoek, W., and Wooldridge, M. 2005a. On theDynamics of Delegation, Cooperation, and Control: ALogical Account. In Proceedings of the 4th InternationalJoint Conference on Autonomous Agents and Multiagent Sys-tems, 701–708. New York: Association for ComputingMachinery

van der Hoek, W., and Wooldridge, M. 2005b. On theLogic of Cooperation and Propositional Control. ArtificialIntelligence 164(1-2): 81–119.

van der Hoek, W.; Jamroga, W.; and Wooldridge, M. 2005.A Logic for Strategic Reasoning. In Proceedings of the 4thInternational Joint Conference on Autonomous Agents andMultiagent Systems, 157–153. New York: Association forComputing Machinery.

van der Hoek, W.; Jamroga, W.; and Wooldridge, M. 2007.Towards a Theory of Intention Revision. Synthese 155(2):265–290.

van der Torre, L. 2001. Contextual Deontic Logic: Nor-mative Agents, Violations and Independence. Annals ofMathematics and Artificial Intelligence 37(1): 33–63.

van Ditmarsch, H.; van der Hoek, W.; and Kooi, B. 2007.Dynamic Epistemic Logic. Berlin: Springer.

van Linder, B.; van der Hoek, W.; and Meyer, J.-C. 1998.Formalizing Abilities and Opportunities of Agents. Fun-dameta Informaticae 34(1, 2): 53–101.

Vardi, M. Y. 2001. Branching Versus Linear Time: FinalShowdown. In Tools and Algorithms for the Constructionand Analysis of Systems, Lecture Notes in Computer Sci-ence, Volume 2031, 1–22. Berlin: Springer-Verlag.

von Wright, G. 1951. Deontic Logic. Mind 60(237): 1–15.

Wooldridge, M., and Jennings, N. R. 1995. IntelligentAgents: Theory and Practice. The Knowledge EngineeringReview 10(2): 115–152.

Wooldridge, M., and van der Hoeck, W. 2005. On Obli-gations and Normative Ability. Journal of Applied Logic3(3–4): 396–420.

Wooldridge, M. 2000. Reasoning about Rational Agents.Cambridge, MA: The MIT Press: Cambridge, MA.

Zalta, E. N. Stanford Encyclopedia of Philosophy. Stanford,CA: Stanford University (plato.stanford.edu).

Wiebe van der Hoek is a professor in the Agent,Research and Technology (ART) Group in the Depart-ment of Computer Science, University of Liverpool, UK.He studied mathematics in Groningen, and earned hisPhD at the Free University of Amsterdam. He is inter-ested in logics for knowledge and interaction. He is cur-rently associate editor of Studia Logica, editor of Agent andMulti-Agent Systems, and editor-in-chief of Synthese. He canbe contacted at wiebe@csc.liv.ac.uk.

Michael Wooldridge is a professor of computer science atthe University of Oxford, UK. His main research interestsare in the use of formal methods of one kind or anotherfor specifying and reasoning about multiagent systems,and computational complexity in multiagent systems. Heis an associate editor of the Journal of Artificial IntelligenceResearch and the Artificial Intelligence journal and he serveson the editorial boards of the Journal of Applied Logic, Jour-nal of Logic and Computation, Journal of Applied ArtificialIntelligence, and Computational Intelligence. He can be con-tacted at mjw@cs.ox.ac.uk.

Articles

SUMMER 2012 105