![Page 1: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/1.jpg)
Distributed Optimization
Yen-Ling KuoDer-Yeuan Yu
May 27, 2010
![Page 2: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/2.jpg)
Outline [Yu]
• Optimized Sensing: From Water to the Web• Distributed Dynamic Programming• Distributed Solutions to Markov Decision
Problems
![Page 3: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/3.jpg)
Optimized Sensing
• Problem Statement• Greedy Algorithms and Submodularity• Robust Sensing Optimization with Saturate
Algorithm• Application in Blogs
![Page 4: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/4.jpg)
Problem Statement
• How do we detect contamination in drinking water distribution networks?
• Which blogs should we read to learn about the biggest, newest stories on the Web?
• Fundamental Question: How can we get the most useful information at minimum cost (limited resources)?
![Page 5: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/5.jpg)
Solutions to Optimized Sensing
• Covers fields of statistics, machine learning, sensor networks, and robotics
• With partially observable Marko decision processes, we can get optimal solutions
• But it is difficult to scale POMDP to large problems
• Introducing a new algorithm based on submodularity
![Page 6: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/6.jpg)
Formulation
• Sensing quality function F(A)– A: the set of sensor locations Si (i=1~k)– V: the set of all locations
• We can also have cost constraints– Total cost of sensor deployment no greater than
the budget• Goal: Find A*
– This is NP-hard already
![Page 7: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/7.jpg)
Greedy Algorithm
• Iteratively find Si
• This naïve algorithm actually performs pretty well– Why? Submodularity– We get near-optimal solutions
• Submodularity: diminishing returns
![Page 8: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/8.jpg)
Diminishing Returns
![Page 9: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/9.jpg)
Cost-Effective Lazy Forward-Selection (CELP)
• Greedy algorithm• Lazy evaluations
– Delaying computation until the result is required– A computational technique
![Page 10: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/10.jpg)
Robust Sensing Optimization
• Idea: Protect system against adversaries that know of our deployment of sensors
• Goal: Maximize the worst-case detection performance
• Approach
• Unfortunately, this naïve extension can fail
![Page 11: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/11.jpg)
Failure of Greedy Algorithm on Worst-Case Scenarios
• I1, I2: two contamination events• S1, S2, S3: three possible sensor locations
– S1: detect I1 immediately, but never I2– S2: detect I2 immediately, but never I1– S3: detect both I1 and I2, but only after a long time
• We can only place two sensors• Greedy would pick S3 first and then either S1 or S2• But we know the optimal solution should be S1 and
S2• Solution? Saturate algorithm
![Page 12: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/12.jpg)
Saturate Algorithm
• Idea: reduce the non-submodular worst-case objective to a submodular optimization problem– Transform non-submodular to submodular
• Transformation– Guess optimal solution value C using binary search– Try to find A such that F(A) is no less than C
![Page 13: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/13.jpg)
Performance of Saturate
![Page 14: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/14.jpg)
From Water to the WebBlog Reading
• Problem: Information cascading
![Page 15: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/15.jpg)
Improvements
• Number-of-posts (NP) model– Reading a big blog can be time-consuming, so they define
the cost to be the number of posts
• CELP tends to choose blogs with many posts• NP model tends to choose summarizer blogs
– But stories appear in summarizer blogs a little late
![Page 16: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/16.jpg)
Other Thoughts
• What if we are looking for stories to read instead of blogs to read?– We can reverse our information management goal– Find posts instead of blogs– Ref. 10
• End of Paper
![Page 17: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/17.jpg)
Distributed Dynamic Programmingfor Path Planning
• Asynchronous Dynamic Programming• Learning Real-Time A*
![Page 18: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/18.jpg)
Asynchronous Dynamic Programming
• Propagate costs from target to start locations
![Page 19: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/19.jpg)
Learning Real-Time A* (LRTA*)
![Page 20: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/20.jpg)
LRTA*(n)
• LRTA with n agents• Faster
– Agents break ties differently– They can share the same h-value table
![Page 21: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/21.jpg)
LRTA*(2)
![Page 22: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/22.jpg)
Distributed Solutions to Markov Decision Problems
• As previously mentioned in the Water to Web paper, MDPs can be difficult to scale to big problems
• Solution: Exploit independence properties• We address the modularity of actions
![Page 23: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/23.jpg)
Action Selection in multiagent MDPs
![Page 24: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/24.jpg)
Implementation
![Page 25: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/25.jpg)
Subtask Distribution
• A global problem is broken down into subtasks
• Subtasks are distributed among agents
• Each agent has different capabilities
Problem
25
![Page 26: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/26.jpg)
Contract Net
• Stages– Recognition– Announce– Bidding– Awarding & Expediting
• Initial assignment: Not optimal• Anytime property
– Improve assignment in negotiation process
26
![Page 27: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/27.jpg)
Assignment problem
• Problem definition– A set N of n agents– A set X of n objects– A set M N × X of possible assignment pairs, and⊆– A function v : M → R
• Find optimal assignment
X NM
27
![Page 28: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/28.jpg)
Corresponding Linear Program
• Linear program (LP) formulation
Profit maximization
Resource constraint
Optimal solution
• Any LP can be solved in polynomial time O(n3)
28
![Page 29: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/29.jpg)
Competitive Equilibrium
• Consider a price vector p = (p1, …, pn)– The utility from an assignment j to agent i is
u(i, j) = v(I, j) - pj
• A feasible assignment S and a price vector p are in competitive equilibrium when for every pairing (i, j)
S it is the ∈ case that ∀ k, u(i, j) ≥ u(i, k)
29
Every agent will not change its selection
S is a optimal solution
![Page 30: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/30.jpg)
Naïve Auction Algorithm
• Round-robin style• Bid increment is the difference between the
utility to i of the best and second-best object
30
The agent will not overbid
![Page 31: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/31.jpg)
Problem in Naïve Auction
• When more than one object offers maximal utility for an agent– Bid increment is zero
31
![Page 32: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/32.jpg)
Terminating Auction Algorithm
• Modify the bid increment–
32
ε-competitive equilibrium: u(i, j) + ε ≥ u(i, k)Agents may overbid some objects
![Page 33: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/33.jpg)
Scheduling Problem
• Problem definition– N is a set of n agents– X is a set of m discrete and consecutive time slots– q = (q1, . . . , qm) is a reserve price vector
– v = (v1, . . . , vn), where vi is the valuation function of agent I
• Find optimal allocation
33
F
![Page 34: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/34.jpg)
Corresponding Integer Program
• Integer program (IP) formulation
• IPs are not solvable polynomial time
34
![Page 35: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/35.jpg)
Competitive Equilibrium – General Form
• Definition– For all i N it is the case that ∈
Fi = argmaxT X ⊆ (vi(T) − ∑j|xj T ∈ pj)
– For all j such that xj F∈ ∅ it is the case that pj = qj
– For all j such that xj F∈ ∅ it is the case that pj ≥ qj
• May not exist competitive equilibrium
35
Has a competitive equilibrium solution ↕
The LP relaxation of the associated integer program has a integer solution.
![Page 36: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/36.jpg)
Ascending Auction Algorithm
• Center advertise an ask price• Bid increment is constant
36
![Page 37: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/37.jpg)
Problem in Ascending Auction
• If the increment is too large
• May not converge to optimal solution37
![Page 38: Distributed Optimization Yen-Ling Kuo Der-Yeuan Yu May 27, 2010](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b457f8b9ab0599a2f90/html5/thumbnails/38.jpg)
Social Laws and Conventions
• Social law– A restriction on the given strategies of the agents– Induce a sub-game
• Social convention– The sub-game consists of a single strategy for all agent
• Other topics– Social goal negotiation– Social norm negotiation– ….
38