q j k a u v c p r g k a j decision-theoretic...
Post on 07-Sep-2018
233 Views
Preview:
TRANSCRIPT
Decision-theoretic Clustering of Strategies
Nolan Bard, Deon Nicholas, Csaba Szepesvári, and Michael Bowling
AAAI Spring Symposium 2015on Applied Computational Game Theory
March 23th, 2015
(appearing in AAMAS 2015)
U
VA!
A!
CK"
K"
PQ#
Q#
RJ$
J$
G10!
10!
University of AlbertaComputer Poker Research Group
Motivation
Given: Knowledge of agents/entities
Goal: Maximize utility by exploiting data
Problem: Limited response personalization
• Resource constrained
• Online learning cost
2
Motivation
3
...Portfolio P
Utility
Solution
• Cluster agents/entities into groups
• Tailor responses to the aggregate clusters
• But…
4
One of these things…
5
Rock
Paper Scissors
E2
3
1
One of these things…
6
Rock
Paper Scissors
P
S
RE2
3
1
Objective
7
4
7
2
9
3
5(E
)ntit
ies
(R)esponsesk-element partition of rows:
argmax
P2Partk(E)
X
C2P
max
r2R
X
e2C
u(e, r)
P = {C1, . . . , Ck} 2 Partk(E)
Segmentation Problems
8
4
7
2
9
3
5(E
)ntit
ies
(R)esponsesk-element partition of rows:
argmax
P2Partk(E)
X
C2P
max
r2R
X
e2C
u(e, r)
P = {C1, . . . , Ck} 2 Partk(E)
Cluster based on actionability. [Kleinberg et al.]
Maximum Coverage
9
4
7
2
9
3
5(E
)ntit
ies
(R)esponsesk-element subset of columns:
argmax
R0✓R|R0|=k
X
e2E
max
r2R0u(e, r)
4
7
2
9
3
5
(E)n
titie
s
(R)esponsesSegmentation Problems
10
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
argmax
R0✓R|R0|=k
X
e2E
max
r2R0u(e, r)
argmax
P2Partk(E)
X
C2P
max
r2R
X
e2C
u(e, r)
Segmentation Problems
11
argmax
R0✓R|R0|=k
X
e2E
max
r2R0u(e, r)
argmax
P2Partk(E)
X
C2P
max
r2R
X
e2C
u(e, r)
argmax
le1 ,...,lem|S
e2E le|=k
X
e2E
u(e, rle)
Segmentation Problems
12
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
Segmentation Problems
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
Segmentation Problems
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
Segmentation Problems
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
Segmentation Problems
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
Segmentation Problems
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
Segmentation Problems
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
Segmentation Problems
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
Segmentation Problems
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
Segmentation Problems
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
Segmentation Problems
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
O(|E||R|)
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
Segmentation Problems
23
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
Segmentation Problems
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
Segmentation Problems
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
Segmentation Problems
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
Segmentation Problems
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
Segmentation Problems
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
Segmentation Problems
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
Segmentation Problems
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
Segmentation Problems
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
Segmentation Problems
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
Segmentation Problems
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
Segmentation Problems
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
Segmentation Problems
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
Segmentation Problems
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
Segmentation Problems
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
Segmentation Problems
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
Segmentation Problems
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
Segmentation Problems
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
Segmentation Problems
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
Segmentation Problems
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
Segmentation Problems
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
Segmentation Problems
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
Segmentation Problems
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
Segmentation Problems
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
Segmentation Problems
O(k|E|)
Maximum Coverage
48
4
7
2
9
3
5
(E)n
titie
s
(R)esponses
argmax
R0✓R|R0|=k
X
e2E
max
r2R0u(e, r)
Exact: NP-hard
Approximation[Nemhauser et al.]
• Greedy submodular• -approximation• Complexity:
(1� 1/e)
O(k|E||R|)
Nemhauser’s Greedy
1 2 5
5 4 1
3 4 1
(E)n
titie
s
(R)esponses
argmax
R0✓R|R0|=k
X
e2E
max
r2R0u(e, r)
Nemhauser’s Greedy
1 2 5
5 4 1
3 4 1
(E)n
titie
s
(R)esponses
X
e2E
max
r2R0u(e, r)
0
0
0
0
R’{}
Nemhauser’s Greedy
1 2 5
5 4 1
3 4 1
(E)n
titie
s
(R)esponses
X
e2E
max
r2R0u(e, r)
0
0
0
0
1
5
3
9
MarginalGain
R’{}
Nemhauser’s Greedy
1 2 5
5 4 1
3 4 1
(E)n
titie
s
(R)esponses
X
e2E
max
r2R0u(e, r)
0
0
0
0
2
4
4
10
MarginalGain
R’{}
Nemhauser’s Greedy
1 2 5
5 4 1
3 4 1
(E)n
titie
s
(R)esponses
X
e2E
max
r2R0u(e, r)
0
0
0
0
5
1
1
7
MarginalGain
R’{}
Nemhauser’s Greedy
1 2 5
5 4 1
3 4 1
(E)n
titie
s
(R)esponses
X
e2E
max
r2R0u(e, r)
2
4
4
10
R’{r2}
Nemhauser’s Greedy
1 2 5
5 4 1
3 4 1
(E)n
titie
s
(R)esponses
X
e2E
max
r2R0u(e, r)
2
4
4
10
R’{r2}
2
5
4
1
MarginalGain
Nemhauser’s Greedy
1 2 5
5 4 1
3 4 1
(E)n
titie
s
(R)esponses
X
e2E
max
r2R0u(e, r)
2
4
4
10
R’{r2}
5
4
4
3
MarginalGain
Nemhauser’s Greedy
1 2 5
5 4 1
3 4 1
(E)n
titie
s
(R)esponses
X
e2E
max
r2R0u(e, r)
5
4
4
13
R’{r2,r3}
Problem
Infinitely/exponentially large response space?
• Nemhauser et al.’s greedy is infeasible
58
Structured Utility
• May be able to exploit structure in utility
59
argmax
P2Partk(E)
X
C2P
max
r2R
X
e2C
u(e, r)
Structured Utility
• May be able to exploit structure in utility
60
argmax
P2Partk(E)
X
C2P
max
r2R
X
e2C
u(e, r)
f(C) ⌘ argmax
r2R
X
e2C
u(e, r)
Response Oracle
61
f(C) ⌘ argmax
r2R
X
e2C
u(e, r)
Example: Sequence-form games
• # Responses: at least exponential in infosets
• Best response: linear in infosets
Greedy Heuristic
Greedy agglomerative (“bottom up”) clustering
62
Initialize: singletonsRock
Paper Scissors
2
3
1
Greedy Heuristic
Greedy agglomerative (“bottom up”) clustering
63
Initialize: singletons
Clustering Loss
Iteration: merge with min marginal loss
Ci, Cj 2 P
X
e2E
max
r⇤2Ru(e, r⇤)�
X
C2P
max
r2R
X
e2C
u(e, r)
Rock
Paper Scissors
2
3
1
Greedy Heuristic
Greedy agglomerative (“bottom up”) clustering
64
Rock
Paper Scissors
2
3
1
Initialize: singletons
Clustering Loss
Iteration: merge with min marginal loss
Ci, Cj 2 P
X
e2E
max
r⇤2Ru(e, r⇤)�
X
C2P
max
r2R
X
e2C
u(e, r)
Greedy Heuristic
Greedy agglomerative (“bottom up”) clustering
• oracle calls (using memoization)
• Feasible given efficient oracle
• k not needed in advance
• Lazy evaluations and parallelizable
65
O(|E|2)
Results
66
Worst-case Approximation Bounds
67
u(Gk) � max
✓1
k,k
m
◆u⇤k � 1p
mu⇤k
u(Gk)� " 2pmu⇤k
Lower:
Upper:
Experimental Design
• Sampled 200 static strategies uniformly
• Compared to k-means/Lloyd’s
• k-means++ seeding [Arthur and Vassilvitskii]
• Feature vectors: sequence-form
• 50 random restarts
68
Qualitative: Kuhn Poker
69
K-means
70
0.0 0.2 0.4 0.6 0.8 1.0
⌘
0.0
0.2
0.4
0.6
0.8
1.0⇠
Greedy
71
0.0 0.2 0.4 0.6 0.8 1.0
⌘
0.0
0.2
0.4
0.6
0.8
1.0⇠
Quantitative
72
Kuhn Poker
73
2 4 6 8 10 12k (number of clusters)
0
10
20
30
40
50
60
Mea
n lo
ss v
s. s
ingl
eton
resp
onse
s (m
bb/g
)
Greedyk-meansOptimal
Leduc Hold’em
74
10 20 30 40 50 60k (number of clusters)
0
50
100
150
200
250
300
350
Mea
n lo
ss v
s. s
ingl
eton
resp
onse
s (m
bb/g
)
Greedyk-means
Questions?75
U
VA!
A!
CK"
K"
PQ#
Q#
RJ$
J$
G10!
10!
University of AlbertaComputer Poker Research Group
poker.cs.ualberta.ca
Contact: nolan@cs.ualberta.ca
top related