intelligent environments

66
Intelligent Environments 1 Intelligent Environments Computer Science and Engineering University of Texas at Arlington

Upload: ferdinand-cameron

Post on 03-Jan-2016

39 views

Category:

Documents


2 download

DESCRIPTION

Intelligent Environments. Computer Science and Engineering University of Texas at Arlington. Decision-Making for Intelligent Environments. Motivation Techniques Issues. Motivation. - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Intelligent Environments

Intelligent Environments 1

Intelligent Environments

Computer Science and Engineering

University of Texas at Arlington

Page 2: Intelligent Environments

Intelligent Environments 2

Decision-Making forIntelligent Environments Motivation Techniques Issues

Page 3: Intelligent Environments

Intelligent Environments 3

Motivation An intelligent environment

acquires and applies knowledge about you and your surroundings in order to improve your experience. “acquires” prediction “applies” decision making

Page 4: Intelligent Environments

Intelligent Environments 4

Motivation Why do we need decision-making?

“Improve our experience” Usually alternative actions

Which one to take? Example (Bob scenario: bedroom

?) Turn on bathroom light? Turn on kitchen light? Turn off bedroom light?

Page 5: Intelligent Environments

Intelligent Environments 5

Example Should I turn on the bathroom light? Issues

Inhabitant’s location (current and future) Inhabitant’s task Inhabitant’s preferences Energy efficiency Security Other inhabitants

Page 6: Intelligent Environments

Intelligent Environments 6

Qualities of a Decision Maker Ideal

Complete: always makes a decision Correct: decision is always right Natural: knowledge easily expressed Efficient

Rational Decisions made to maximize

performance

Page 7: Intelligent Environments

Intelligent Environments 7

Agent-based Decision Maker Russell & Norvig “AI: A Modern

Approach” Rational agent

Agent chooses an action to maximize its performance based on percept sequence

Page 8: Intelligent Environments

Intelligent Environments 8

Agent Types Reflex agent Reflex agent with state Goal-based agent Utility-based agent

Page 9: Intelligent Environments

Intelligent Environments 9

Reflex Agent

Page 10: Intelligent Environments

Intelligent Environments 10

Reflex Agent with State

Page 11: Intelligent Environments

Intelligent Environments 11

Goal-based Agent

Page 12: Intelligent Environments

Intelligent Environments 12

Utility-based Agent

Page 13: Intelligent Environments

Intelligent Environments 13

Intelligent Environments

Decision-Making Techniques

Page 14: Intelligent Environments

Intelligent Environments 14

Decision-Making Techniques Logic Planning Decision theory Markov decision process Reinforcement learning

Page 15: Intelligent Environments

Intelligent Environments 15

Logical Decision Making If Equal(?Day,Monday)

& GreaterThan(?CurrentTime,0600)& LessThan(?CurrentTime,0700)& Location(Bob,bedroom,?

CurrentTime)& Increment(?CurrentTime,?NextTime)

Then Location(Bob,bathroom,?NextTime)

Query: Location(Bob,?Room,0800)

Page 16: Intelligent Environments

Intelligent Environments 16

Logical Decision Making Rules and facts

First-order predicate logic Inference mechanism

Deduction: {A, A B} B Systems

Prolog (PROgramming in LOGic) OTTER Theorem Prover

Page 17: Intelligent Environments

Intelligent Environments 17

Prolog location(bob,bathroom,NextTime) :- dayofweek(Day), Day = monday, currenttime(CurrentTime), CurrentTime > 0600, CurrentTime < 0700, location(bob,bedroom,CurrentTime), increment(CurrentTime,NextTime). Facts: dayofweek(monday), ... Query: location(bob,Room,0800).

Page 18: Intelligent Environments

Intelligent Environments 18

OTTER (all d all t1 all t2

((DayofWeek(d) & Equal(d,Monday) &

CurrentTime(t1) &

GreaterThan(t1,0600) &

LessThan(t1,0700) & NextTime(t1,t2)

& Location(Bob,Bedroom,t1)) ->

Location(Bob,Bathroom,t2))). Facts: DayofWeek(Monday), ... Query: (exists r (Location(Bob,r,0800)))

Page 19: Intelligent Environments

Intelligent Environments 19

Actions If Location(Bob,Bathroom,t1) Then

Action(TurnOnBathRoomLight,t1) Preferences among actions

If RecommendedAction(a1,t1) & RecommendedAction(a2,t1) & ActionPriority(a1) > ActionPriority(a2) Then Action(a1,t1)

Page 20: Intelligent Environments

Intelligent Environments 20

Persistence Over Time If Location(Bob,room1,t1) & not Move(Bob,t1) & NextTime(t1,t2) Then Location(Bob,room1,t2)

One for each attribute of Bob!

Page 21: Intelligent Environments

Intelligent Environments 21

Logical Decision Making Assessment

Complete? Yes Correct? Yes Efficient? No Natural? No Rational?

Page 22: Intelligent Environments

Intelligent Environments 22

Decision Making as Planning Search for a sequence of actions to

achieve some goal Requires

Initial state of the environment Goal state Actions (operators)

Conditions Effects (implied connection to effectors)

Page 23: Intelligent Environments

Intelligent Environments 23

Example Initial: location(Bob,Bathroom) &

light(Bathroom,off) Goal: happy(Bob) Action 1

Condition: location(Bob,?r) & light(?r,on)Effect: Add: happy(Bob)

Action 2 Condition: light(?r,off) Effect: Delete: light(?r,off), Add: light(?r,on)

Plan: Action 2, Action 1

Page 24: Intelligent Environments

Intelligent Environments 24

Requirements Where do goals come from?

System design Users

Where do actions come from? Device “drivers” Learned macros

E.g., SecureHome action

Page 25: Intelligent Environments

Intelligent Environments 25

Planning Systems UCPOP (Univ. of Washington)

Partial Order Planner with Universal quanitification and Conditional effects

GraphPlan (CMU) Builds and prunes graph of possible

plans

Page 26: Intelligent Environments

Intelligent Environments 26

GraphPlan Example

(:action lighton :parameters (?r) :precondition

(light ?r off)) :effects

(and (light ?r on) (not (light ?r off))))

Page 27: Intelligent Environments

Intelligent Environments 27

Planning Assessment

Complete? Yes Correct? Yes Efficient? No Natural? Better Rational?

Page 28: Intelligent Environments

Intelligent Environments 28

Decision Theory Logical and planning approaches

typically assume no uncertainty Decision theory = probability theory +

utility theory Maximum Expected Utility principle

Rational agent chooses actions yielding highest expected utility

Averaged over all possible action outcomes Weight utility of an outcome by its probability of

occurring

Page 29: Intelligent Environments

Intelligent Environments 29

Probability Theory Random variables: X, Y, … Prior probability: P(X) Conditional probability: P(X|Y) Joint probability distribution

P(X1,…,Xn) is an n-dimensional table of probabilities

Complete table allows computation of any probability

Complete table typically infeasible

Page 30: Intelligent Environments

Intelligent Environments 30

Probability Theory Bayes rule

Example

More likely to know P(wet|rain) In general, P(X|Y) = * P(Y|X) * P(Y)

chosen so that P(X|Y) = 1

)(

)()|()|(

YP

XPXYPYXP

)(

)()|()|(

wetP

rainPrainwetPwetrainP

Page 31: Intelligent Environments

Intelligent Environments 31

Bayes Rule (cont.) How to compute P(rain|wet & thunder)

P(r | w & t) = P(w & t | r) * P(r) / P(w & t) Know P(w & t | r) possibly, but tedious as

evidence increases Conditional independence of evidence

Thunder does not cause wet, and vice versa P(r | w & t) = * P(w|r) * P(t|r) * P(r)

Page 32: Intelligent Environments

Intelligent Environments 32

Where Do Probabilities Come From?

Statistical sampling Universal principles Individual beliefs

Page 33: Intelligent Environments

Intelligent Environments 33

Representation of Uncertain Knowledge

Complete joint probability distribution

Conditional probabilities and Bayes rule Assuming conditional independence

Belief networks

Page 34: Intelligent Environments

Intelligent Environments 34

Belief Networks Nodes represent random variables Directed link between X and Y implies

that X “directly influences” Y Each node has a conditional

probability table (CPT) quantifying the effects that the parents (incoming links) have on the node

Network is a DAG (no directed cycles)

Page 35: Intelligent Environments

Intelligent Environments 35

Belief Networks: Example

Page 36: Intelligent Environments

Intelligent Environments 36

Belief Networks: Semantics Network represents the joint probability

distribution

Network encodes conditional independence knowledge Node conditionally independent of all other

nodes except parents E.g., MaryCalls and Earthquake are

conditionally independent

n

iiinnn XParentsxPxxPxXxXP

1111 ))(|(),...,(),...,(

Page 37: Intelligent Environments

Intelligent Environments 37

Belief Networks: Inference Given network, compute

P(Query | Evidence) Evidence obtained from sensory percepts

Possible inferences Diagnostic: P(Burglary | JohnCalls) = 0.016 Causal: P(JohnCalls | Burglary) P(Burglary | Alarm & Earthquake)

Page 38: Intelligent Environments

Intelligent Environments 38

Belief Network Construction Choose variables

Discretize continuous variables Order variables from causes to effects CPTs

Specify each table entry Define as a function (e.g., sum, Gaussian)

Learning Variables (evidential and hidden) Links (causation) CPTs

Page 39: Intelligent Environments

Intelligent Environments 39

Combining Beliefs with Desires Maximum expected utility

Rational agent chooses action maximizing expected utility

Expected utility EU(A|E) of action A given evidence E

EU(A|E) = i P(Resulti(A) | E, Do(A)) * U(Resulti(A)) Resulti(A) are possible outcome states after

executing action A U(S) is the agent’s utility for state S Do(A) is the proposition that action A is executed in

the current state

Page 40: Intelligent Environments

Intelligent Environments 40

Maximum Expected Utility Assumptions

Knowing evidence E completely requires significant sensory information

P(Result | E, Do(A)) requires complete causal model of the environment

U(Result) requires complete specification of state utilities

One-shot vs. sequential decisions

Page 41: Intelligent Environments

Intelligent Environments 41

Utility Theory Any set of preferences over possible outcomes

can be expressed by a utility function Lottery L = [p1,S1; p2,S2; ...; pn,Sn]

pi is the probability of possible outcome Si

Si can be another lottery

Utility principle U(A) > U(B) A preferred to B U(A) = U(B) agent indifferent to A and B

Maximum expected utility principle U([p1,S1; p2,S2; ...; pn,Sn]) = i pi * U(Si)

Page 42: Intelligent Environments

Intelligent Environments 42

Utility Functions Possible outcomes

[1.0, $1000; 0.0, $0] [0.5, $3000; 0.5, $0]

Expected monetary value $1000 vs. $1500

But depends on value $k Sk = state of possessing wealth $k EU(accept) = 0.5 * U(Sk+3000) + 0.5 * U(Sk) EU(decline) = U(Sk+1000) Will decline for some values of U, accept for

others

Page 43: Intelligent Environments

Intelligent Environments 43

Utility Functions (cont.)

Studies show U(Sk+n) = log2n Risk-adverse agents in positive part of curve Risk-seeking agents in negative part of curve

Page 44: Intelligent Environments

Intelligent Environments 44

Decision Networks Also called influence diagrams Decision networks = belief

networks + actions and utilities Describes agent’s

Current state Possible actions State resulting from agent’s action Utility of resulting state

Page 45: Intelligent Environments

Intelligent Environments 45

Example Decision Network

Page 46: Intelligent Environments

Intelligent Environments 46

Decision Network Chance node (oval)

Random variable and CPT Same as belief network node

Decision node (rectangle) Can take on a value for each possible action

Utility node (diamond) Parents are those chance nodes affecting utility Contains utility function mapping parents to

utility value or lottery

Page 47: Intelligent Environments

Intelligent Environments 47

Evaluating Decision Networks Set evidence variables according to

current state For each action value of decision

node Set value of decision node to action Use belief-net inference to calculate

posteriors for parents of utility node Calculate utility for action

Return action with highest utility

Page 48: Intelligent Environments

Intelligent Environments 48

Sequential Decision Problems No intermediate utility on the way to the

goal Transition model

Probability of reaching state j after taking action a in state i

Policy = complete mapping from states to actions Want policy maximizing expected utility Computed from transition model and state

utilities

aijM

Page 49: Intelligent Environments

Intelligent Environments 49

Example

P(intended direction) = 0.8 P(right angle to intended) = 0.1 U(sequence) = terminal state’s value -

(1/25)*length(sequence)

Page 50: Intelligent Environments

Intelligent Environments 50

Example (cont.)

Optimal Policy Utilities

Page 51: Intelligent Environments

Intelligent Environments 51

Markov Decision Process (MDP) Calculating optimal policy in fully-

observable, stochastic environment with known transition model

Markov property satisfied depends only on i and not

previous states Partially-observable environments

addressed by POMDPs

aijM

aijM

Page 52: Intelligent Environments

Intelligent Environments 52

Value Iteration for MDPs Iterate the following for each state i

until little change

R(i) is the reward for entering state i -0.04 for all states except (4,3) and (4,2) +1 for (4,3) -1 for (4,2)

Best policy policy*(i) is

j

aij

ajUMiRiU )(max)()(

j

aij

ajUMipolicy )(maxarg)(*

Page 53: Intelligent Environments

Intelligent Environments 53

Reinforcement Learning Basically MDP, but learns policy without

the need for transition model Q-learning with temporal difference

Assigns values Q(a,i) to action-state pairs Utility U(i) = maxa Q(a,i) Update Q(a,i) after each observed transition

from state i to state jQ(a,i) = Q(a,i) + * (R(i) + maxa’ Q(a’,j) - Q(a,i))

action in state i = argmaxa Q(a,i)

aijM

Page 54: Intelligent Environments

Intelligent Environments 54

Decision-Theoretic Agent Given

Percept (sensor) information Maintain

Decision network with beliefs, actions and utilities

Do Update probabilities for current state Compute outcome probabilities for actions Select action with highest expected utility

Return action

Page 55: Intelligent Environments

Intelligent Environments 55

Decision-Theoretic Agent Modeling sensors

Page 56: Intelligent Environments

Intelligent Environments 56

Sensor Modeling Combining evidence from multiple

sensors

Page 57: Intelligent Environments

Intelligent Environments 57

Sensor Modeling Detailed model of lane-position

sensor

Page 58: Intelligent Environments

Intelligent Environments 58

Dynamic Belief Network (DBN)

Reasoning over time Big for lots of states But really only need two slices at a time

Page 59: Intelligent Environments

Intelligent Environments 59

Dynamic Belief Network (DBN)

Page 60: Intelligent Environments

Intelligent Environments 60

DBN for Lane Positioning

Page 61: Intelligent Environments

Intelligent Environments 61

Dynamic Decision Network (DDN)

Page 62: Intelligent Environments

Intelligent Environments 62

DDN-based Agent Capabilities

Handles uncertainty Handles unexpected events (no fixed plan) Handles noisy and failed sensors Acts to obtain relevant information

Needs Properties from first-order logic

DDNs are propositional Goal directedness

Page 63: Intelligent Environments

Intelligent Environments 63

Decision-Theoretic Agent Assessment

Complete? No Correct? No Efficient? Better Natural? Yes Rational? Yes

Page 64: Intelligent Environments

Intelligent Environments 64

Netica www.norsys.com Decision network simulator

Chance nodes Decision nodes Utility nodes

Learns probabilities from cases

Page 65: Intelligent Environments

Intelligent Environments 65

Bob Scenario in Netica

Page 66: Intelligent Environments

Intelligent Environments 66

Issues in Decision Making Rational agent design

Dynamic decision-theoretic agent Knowledge engineering effort Efficiency vs. completeness

Monolithic vs. distributed intelligence

Degrees of autonomy