Welcome!!!Welcome!!!

Fifth Americas Agents SchoolFifth Americas Agents School

Harvard UniversityHarvard University

Roger MaillerSRI International

David SarneHarvard University

Introduction to Agents and Introduction to Agents and Multi-agent SystemsMulti-agent Systems

What is an agent?What is an agent?

Gross DefinitionGross Definition– An agent is any objectAn agent is any object

SensesSensesDecidesDecidesActsActs

In the eye of the beholderIn the eye of the beholder– Thermostat can be considered an agentThermostat can be considered an agent– Not very intelligent!!!Not very intelligent!!!

Refining the DefinitionRefining the Definition

What makes an agent interesting is What makes an agent interesting is intelligenceintelligence

An intelligent agentAn intelligent agent– Senses its environmentSenses its environment– Makes decisions based onMakes decisions based on

What it senses (situated)What it senses (situated)Past experience (learning)Past experience (learning)To achieve a goal (goal driven)To achieve a goal (goal driven)

– Acts on those decisions without human Acts on those decisions without human intervention (autonomy)intervention (autonomy)

SensingSensing Sensing can be thought of as the act of collecting raw input Sensing can be thought of as the act of collecting raw input

and turning it into usable informationand turning it into usable information

The line between sensing and deciding is blurryThe line between sensing and deciding is blurry

Camera

Microphone

Laser

Text

ComputerVision

SpeechUnderstanding

Mapping

NLP

Images

Sounds

Points

Bytes

Decide Act

Error, Error, EverywhereError, Error, Everywhere

Sensing is an error-prone processSensing is an error-prone process– At the sensor levelAt the sensor level

CalibrationCalibration Insufficient ResolutionInsufficient Resolution Environmental InterferenceEnvironmental Interference

– Background noiseBackground noise– Multi-pathing Multi-pathing – Reflection and echoingReflection and echoing

– At the interpretation levelAt the interpretation level Incorrect translation induced by sensor level errorsIncorrect translation induced by sensor level errors Inaccurate and incomplete modelsInaccurate and incomplete models Translation ResolutionTranslation Resolution

Control

DecidingDeciding

Sense ActRules

Learning

Search

Control

Rule Based SystemsRule Based Systems Simplest way to act intelligently is a Rule-Based SystemSimplest way to act intelligently is a Rule-Based System Basically a large set of If-Then like rulesBasically a large set of If-Then like rules

If wind_speed > 30 thenIf wind_speed > 30 then increase thrust by 10%increase thrust by 10%

Highly engineeredHighly engineered Complicated to anticipate all conditions in the environmentComplicated to anticipate all conditions in the environment In some cases, it is impossible to have every solution pre-In some cases, it is impossible to have every solution pre-

programmedprogrammed

X Y

Z

If Y=blue and Z = greenX = red

If Y=blue and Z = redX = green

If Y=red and Z = blueX = green

And many more…

SearchSearch Useful when Useful when

– the number of possible solutions is largethe number of possible solutions is large– problem changes over timeproblem changes over time

Constraint satisfaction or Constraint satisfaction or optimization, scheduling, planningoptimization, scheduling, planning

X Y

Z

X

Y YY

Z Z Z

Y

Z Z Z

Search TechniquesSearch Techniques

Depends largely on the problemDepends largely on the problem Some methodsSome methods

– BacktrackingBacktracking– OR techniques like SimplexOR techniques like Simplex– Flow TechniquesFlow Techniques– Local SearchLocal Search

Hill ClimbingHill ClimbingSimulated AnnealingSimulated Annealing

LearningLearning Simplest form of learning is lookup tablesSimplest form of learning is lookup tables

– Store past experience and outcomeStore past experience and outcome– Assume that a repeated experience has the Assume that a repeated experience has the

same outcomesame outcome ProsPros

– Easy to implementEasy to implement– UnbiasedUnbiased

ConsCons– Memory intensiveMemory intensive– Only can predict something that you’ve Only can predict something that you’ve

previously seenpreviously seen– Not robust to ambiguityNot robust to ambiguity

Other learning techniquesOther learning techniques Decision TreesDecision Trees

– Pros: Easy to understand and implementPros: Easy to understand and implement– Cons: Biased and tend to over-fit dataCons: Biased and tend to over-fit data

Neural NetworksNeural Networks– Pros: Can fit almost any functionPros: Can fit almost any function– Cons: Slow to train, tend to over-fit, impossible to figure out Cons: Slow to train, tend to over-fit, impossible to figure out

what they’ve learned what they’ve learned Reinforcement LearningReinforcement Learning

– Pros: Guaranteed to find an optimal policy in the limitPros: Guaranteed to find an optimal policy in the limit– Cons: Slow to train, need a fairly robust model of the Cons: Slow to train, need a fairly robust model of the

environment (states and actions)environment (states and actions) Genetic AlgorithmsGenetic Algorithms

– Pros: Great for learning parametersPros: Great for learning parameters– Cons: Hard to setup, very slowCons: Hard to setup, very slow

Inductive Logic ProgrammingInductive Logic Programming– Pros: Can understand what’s learned, knowledge is consistentPros: Can understand what’s learned, knowledge is consistent– Cons: Has a hard time dealing with inconsistent knowledge Cons: Has a hard time dealing with inconsistent knowledge

ControlControl

Levels of controlLevels of control– Single execution threadSingle execution thread– Finite State MachineFinite State Machine– ThreadsThreads– ProcessProcess

Meta-level reasoningMeta-level reasoning– How long should I take to reason about this How long should I take to reason about this

input?input?– How much time should I spend searching How much time should I spend searching

versus learning?versus learning?

ActingActing

Sense Decide

Effectors

Graphics

Speech

Text

More ErrorsMore Errors

Inaccurate modelsInaccurate models– KinematicsKinematics– Grammar and languageGrammar and language– PronunciationPronunciation

ResolutionResolution– Details matterDetails matter

CoordinationCoordination In many environments, agents’ actions effect one In many environments, agents’ actions effect one

anotheranother For example, if I decide to drive through an For example, if I decide to drive through an

intersection, my actions effect any other agent intersection, my actions effect any other agent trying to drive through at the same timetrying to drive through at the same time

Reasons to coordinateReasons to coordinate– Shared resource – two cars in the same intersectionShared resource – two cars in the same intersection– Shared goal requiring multiple agents – building a houseShared goal requiring multiple agents – building a house– Overlapping goals, my action helps you out without Overlapping goals, my action helps you out without

costing me anythingcosting me anything

Forms of CoordinationForms of Coordination Implicit (without communication)Implicit (without communication)

– Social policies, norms, rulesSocial policies, norms, rules Stay on the right hand side of the roadStay on the right hand side of the road Steam (Tambe et al.)Steam (Tambe et al.)

– Observation (communicating through the environment)Observation (communicating through the environment) I start driving forward so you better stopI start driving forward so you better stop Swarms (Bonabeau, Eberhart, Shi, Kennedy )Swarms (Bonabeau, Eberhart, Shi, Kennedy )

Explicit (with communication)Explicit (with communication)– CompetitiveCompetitive

Auctions (Wellman, Sandholm)Auctions (Wellman, Sandholm)– CooperativeCooperative

Negotiation (Lesser, Durfee)Negotiation (Lesser, Durfee) Mediation (Cammerata ,Shen, Mailler)Mediation (Cammerata ,Shen, Mailler)

Implicit CoordinationImplicit Coordination Social norms and policies: simply adding Social norms and policies: simply adding

rules to follow in social settingsrules to follow in social settings– Pros: No need to communicate, fast to Pros: No need to communicate, fast to

compute a solutioncompute a solution– Cons: Like rule-based system, require a lot of Cons: Like rule-based system, require a lot of

pre-planning and design, can interfere with pre-planning and design, can interfere with internal rulesinternal rules

Swarms (everyone do there own thing)Swarms (everyone do there own thing)– Pros: Work well when individual optimality is Pros: Work well when individual optimality is

not importantnot important– Cons: Hard to create micro behaviors that Cons: Hard to create micro behaviors that

create a macro solution create a macro solution

Market-based MechanismsMarket-based Mechanisms Contract Net (Smith) – Announce contract, Contract Net (Smith) – Announce contract,

agents bid, contract is awardedagents bid, contract is awarded AuctionsAuctions

– English, Dutch, Vickery, Double, CombinatorialEnglish, Dutch, Vickery, Double, Combinatorial When they workWhen they work

– Utility of the bidding agents is independent Utility of the bidding agents is independent from one another (utility is not super-additive)from one another (utility is not super-additive)

– Market is designed to prevent collusion (often Market is designed to prevent collusion (often hard to do)hard to do)

– Auctioneer is trustedAuctioneer is trusted– Agents are not willing to reveal a lot of Agents are not willing to reveal a lot of

informationinformation

NegotiationNegotiation

Semi-cooperativeSemi-cooperative Willing to exchange some information in Willing to exchange some information in

order to resolve an impasseorder to resolve an impasse Usually involves multiple steps to Usually involves multiple steps to

converge on a solutionconverge on a solution Works well when Works well when

– dependencies are localized (no complex multi-dependencies are localized (no complex multi-linking)linking)

– Agents are willing to concede local utility for Agents are willing to concede local utility for the greater goodthe greater good

Cooperative MediationCooperative Mediation

Cooperative methodCooperative method Willing to eventually reveal all Willing to eventually reveal all

information to converge on a solutioninformation to converge on a solution Works well whenWorks well when

– Complex interdependencies between Complex interdependencies between agentsagents

– Agents are willing to exchange local Agents are willing to exchange local utility for global utilityutility for global utility

ConclusionsConclusions

An intelligent agent is a goal-directed, An intelligent agent is a goal-directed, autonomous object which is capable of autonomous object which is capable of sensing, deciding, and acting.sensing, deciding, and acting.

Agents represent the pinnacle of current Agents represent the pinnacle of current and past AI researchand past AI research– LearningLearning– SearchSearch– CoordinationCoordination– SensingSensing– EffectingEffecting

ScheduleSchedule Friday July 14thFriday July 14th

– 11:00-12:30 11:00-12:30 Computational Mechanism DesignComputational Mechanism Design David David ParkesParkes

– 2:00-3:30 2:00-3:30 Autonomous Bidding Agents: Lessons from the Autonomous Bidding Agents: Lessons from the Trading Agent CompetitionTrading Agent Competition Amy Amy GeenwaldGeenwald

– 4:00-5:30 4:00-5:30 Bio-inspired Multi-Agent SystemsBio-inspired Multi-Agent Systems RadhikaRadhika NagpalNagpal

– 6:00 -8:00 6:00 -8:00 Reception and Poster SessionReception and Poster Session Saturday July 15thSaturday July 15th

– 9:00-10:30 9:00-10:30 Graphical Models for Decision-Making in MASGraphical Models for Decision-Making in MAS Gal Gal KobiKobi

– 11:00-12:30 11:00-12:30 Ant RobotsAnt Robots Sven Koenig Sven Koenig– 2:00-3:30 2:00-3:30 Key Issues in Multi-Agent CoordinationKey Issues in Multi-Agent Coordination Victor Victor

LesserLesser– 4:00-5:30 4:00-5:30 Understanding Issues in Multi-Agent System Understanding Issues in Multi-Agent System

Design Through the Blackboard System Model.Design Through the Blackboard System Model. Zack Zack RubinsteinRubinstein