sensor task manager (stm) v.s. subrahmanian university of maryland joint work with: f. ozcan, ibm...
TRANSCRIPT
Sensor Task Manager (STM)
V.S. SubrahmanianV.S. Subrahmanian
University of MarylandUniversity of Maryland
Joint work with:Joint work with:F. Ozcan, IBM AlmadenF. Ozcan, IBM Almaden
T.J. Rogers, University of MarylandT.J. Rogers, University of Maryland
Jan. 02 Darpa SenseIT PI Meeting 2
Scaling task handling Users specify tasks of interest:Users specify tasks of interest:
Where to monitorWhere to monitor When to monitorWhen to monitor Monitoring conditions to check forMonitoring conditions to check for What to do when monitoring conditions arise.What to do when monitoring conditions arise.
Data on the ground changes continuously.Data on the ground changes continuously. Monitoring conditions need to be evaluated Monitoring conditions need to be evaluated
continuously.continuously. LOTS of conditions, LOTS of sensed data. Scalability LOTS of conditions, LOTS of sensed data. Scalability
is key.is key.
Jan. 02 Darpa SenseIT PI Meeting 3
How to Handle lots of tasks
Three pronged strategy:Three pronged strategy: Merge: Merge: Merge tasks to eliminate any redundancy using Merge tasks to eliminate any redundancy using
a cost model. Such merging only works well for a cost model. Such merging only works well for relatively small sets of tasks (or conditions to evaluate).relatively small sets of tasks (or conditions to evaluate).
Task Assignment:Task Assignment: Select sensors (and/or data Select sensors (and/or data sources) to handle merged tasks so as to optimize sources) to handle merged tasks so as to optimize performance criteria.performance criteria.
Partition: Partition: Given a large set of tasks (or conditions) to Given a large set of tasks (or conditions) to process, determine ways of partitioning into smaller process, determine ways of partitioning into smaller sets of manageable size.sets of manageable size.
For time reasons, only the last is discussed today.For time reasons, only the last is discussed today.
Jan. 02 Darpa SenseIT PI Meeting 4
Task Partitioning Goal:Goal: Partition large number of tasks into disjoint Partition large number of tasks into disjoint
sets and minimize the total cost of executing the sets and minimize the total cost of executing the taskstasks
Cost estimation function (cost): Cost estimation function (cost): approximatesapproximates the the cost of executing a set of tasks together. Any cost of executing a set of tasks together. Any function satisfying the axioms:function satisfying the axioms:
TTi i T Tjj cost cost(T(Tii) ) costcost((TTjj)) cost(Ø) = 0cost(Ø) = 0
Jan. 02 Darpa SenseIT PI Meeting 5
Partitions
Partition:Partition: A partition P of a set T of tasks is a set A partition P of a set T of tasks is a set
{ P{ P11,…,P,…,Pnn}, where each P}, where each Pi i is non-empty, iis non-empty, ij j PPi i
PPjj = = and and
Each PEach Pii is called a is called a componentcomponent of P. of P.
P is a sub-partition of Q ifP is a sub-partition of Q if
Jan. 02 Darpa SenseIT PI Meeting 6
Task Partitioning Problem (TP)
Formal Problem Definition:Formal Problem Definition: Given as input a set T Given as input a set T of tasks, and a cost estimation function of tasks, and a cost estimation function costcost ,, find a find a partition P = { Ppartition P = { P11,…,P,…,Pnn} such that } such that
Need to balance execution time of tasks vs. Need to balance execution time of tasks vs. optimization time of tasks.optimization time of tasks.
)(cost1
n
i
iP is
minimized
Jan. 02 Darpa SenseIT PI Meeting 7
TP Algorithms
Theorem:Theorem: The task partitioning problem is NP-The task partitioning problem is NP-complete. complete.
Proposed multiple types of algorithms to solve TPProposed multiple types of algorithms to solve TP A*-basedA*-based : Finds optimal solution : Finds optimal solution Branch-and-BoundBranch-and-Bound (BAB): Finds optimal solution (BAB): Finds optimal solution GreedyGreedy: Is not guaranteed to find optimal solution, has : Is not guaranteed to find optimal solution, has
polynomial running time – several variants proposed.polynomial running time – several variants proposed.
Jan. 02 Darpa SenseIT PI Meeting 8
Adaptation of the A* Algorithm
State: A State: A sub-partitionsub-partition of T; P’ = of T; P’ = {P{P11,…,P,…,Pmm}} Start state: Empty partitionStart state: Empty partition Goal state: A Goal state: A partitionpartition of Tof T
Ex:Ex: T = {t T = {t11, t, t22, t, t33, t, t44, t, t55}} Example state s = {{tExample state s = {{t11, t, t33}, {t}, {t22, t, t55}}}} Goal State = {{tGoal State = {{t11, t, t3 3 , t, t44}, {t}, {t22, t, t55}}}}
g(s) = g(s) = PiPiP’ P’ cost(Pcost(Pii)) Ex:Ex: g(s) = cost({t g(s) = cost({t11, t, t33}) + cost ({t}) + cost ({t22, t, t55})})
Jan. 02 Darpa SenseIT PI Meeting 9
Adaptation of the A* Algorithm
Expansion functionExpansion function
Pick a task t and insert it into each component PPick a task t and insert it into each component P ii of of
P’P’
Create a new component PCreate a new component Pm+1 m+1 containing only tcontaining only t
Ex :Ex : {{t {{t11}, {t}, {t22}} and we pick t}} and we pick t44, then , then
{{t{{t11,t,t44}, {t}, {t22}}, }},
{{t{{t11}, {t}, {t22, qt, qt44}} }}
{{t{{t11}, {t}, {t22}, {t}, {t44}}, }},
Jan. 02 Darpa SenseIT PI Meeting 10
Adaptation of the A* Algorithm
h(s) = min{incr(t,s) | t h(s) = min{incr(t,s) | t P} P} incr(t,s) = min{cost(t), min{cost(t incr(t,s) = min{cost(t), min{cost(t P Pii) - cost(P) - cost(Pii) | P) | PiiP’}}P’}}
Ex:Ex: s = {{ts = {{t11}, {t}, {t22, t, t55}}}}
h(s) = min{incr(th(s) = min{incr(t33,s), incr(t,s), incr(t44, s)}, s)}
incr (tincr (t44, s) = min{cost(t, s) = min{cost(t44), (cost({t), (cost({t11, t, t44}) - cost(t}) - cost(t11)), )),
(cost ({t (cost ({t22, t, t5 5 , t, t44}) - cost ({t}) - cost ({t22, t, t55}))}}))}
Theorem:Theorem: The function The function h h is admissible and satisfies the monotone is admissible and satisfies the monotone restriction.restriction.
Theorem:Theorem: hence, A* finds an optimal partition. hence, A* finds an optimal partition.
Jan. 02 Darpa SenseIT PI Meeting 11
Cluster Graphs
Canonical Cluster GraphCanonical Cluster Graph (T): Undirected weighted (T): Undirected weighted
graph wheregraph where
V = { {tV = { {tii} | t} | tii T } T }
E = { ({tE = { ({tii},{t},{tjj})| t})| tii, t, tj j T and w({t T and w({tii},{t},{tjj}) }) > > 0 0
w({tw({tii},{t},{tjj}) = cost(t}) = cost(tii) + cost(t) + cost(tjj) - cost ({t) - cost ({tii,t,tjj})})
Jan. 02 Darpa SenseIT PI Meeting 12
Cluster Graph Example
T = {tT = {t11, t, t22, t, t33, t, t44, t, t55}}
cost(tcost(tii) =5, ) =5, cost({tcost({t11, t, t22}) = 8, cost({t}) = 8, cost({t33, t, t44}) = }) =
7 and cost({t7 and cost({t33, t, t55}) = 6}) = 6
t1 t2
2 t3
t4 t5
3 4
Jan. 02 Darpa SenseIT PI Meeting 13
Greedy Partitioning Algorithm
Builds the partition iteratively using a cluster graph Builds the partition iteratively using a cluster graph representationrepresentation
In each iteration, finds the edge (tIn each iteration, finds the edge (t ii,t,tjj) with the ) with the
maximum weight and removes from the graphmaximum weight and removes from the graph Terminates when all edges are processedTerminates when all edges are processed Running time : O(|V|.|E|)Running time : O(|V|.|E|)
Jan. 02 Darpa SenseIT PI Meeting 14
Greedy Partitioning Algorithm
At each step, four possible casesAt each step, four possible cases Case 1:Case 1: Both t Both tii and and ttj j are in the same component; are in the same component;
do nothingdo nothing Case 2:Case 2: One of t One of tii or or ttj j is in a component; insert the is in a component; insert the
other one into the same componentother one into the same component Case 3:Case 3: Neither is in any of the components; Neither is in any of the components;
create a new component with tcreate a new component with tii and and ttj j
Case 4:Case 4: t tii and and ttj j are in different components; are in different components; move one of them into the other component, or move one of them into the other component, or leave as it isleave as it is
Jan. 02 Darpa SenseIT PI Meeting 15
t1 t2
2
t3
t4 t5
3
t1 t2
2 t3
t4 t5
3 4
Running ExampleT = {t1, t2, t3, t4, t5}
P = {{t3, t5}}
P={}
Jan. 02 Darpa SenseIT PI Meeting 16
t1 t2
2
t3
t4 t5
Running Example, cont.P = {{t3, t4, t5}}
P = {{t3, t4, t5}, {t1, t2}}
t1 t2
t3
t4 t5
Jan. 02 Darpa SenseIT PI Meeting 17
Variants of the Greedy Algorithm
Several variants of the basic greedy algorithm (5 in all we Several variants of the basic greedy algorithm (5 in all we worked with, 2 examples below) worked with, 2 examples below) Greedy with weight update (Greedy with weight update (Greedy w/ WUGreedy w/ WU))
After inserting tasks into components, it updates the After inserting tasks into components, it updates the weights of adjacent edges weights of adjacent edges
Greedy with no move around (Greedy with no move around (Greedy w/ NMAGreedy w/ NMA))Once a task is inserted into a component, it stays there.Once a task is inserted into a component, it stays there.
Jan. 02 Darpa SenseIT PI Meeting 18
Running TimesNo of tasks
A*-based
BAB Greedy Basic
Greedy w/NMA
Greedy w/WU
5 68.3 81.1 2.2 0.3 1
6 328.5 417.1 2.75 0.35 1.55
7 1420.7 1570.5 2.8 0.4 1.7
8 9941.2 6752 3.3 0.45 3
9 18281.9 5243.4 4.7 0.5 4.8
10 44782.6 10109 6 0.5 6.6
Execution times (millisecs) (Cost-limit = 100, Overlap-degree=0.6, Overlap-prob = 0.4,0.6)
Only 10 tasks above as A* runs out of space. BAB cando 11 or 12. Greedy methods can handle thousands (see next slides).
Jan. 02 Darpa SenseIT PI Meeting 19
Scalability of The Greedy Algorithms Cost-limit=100
Overlap- degree=0.2
Overlap- prob=0.4,0.6
0
2
4
6
8
10
12
14
16
18
20
0 300 600 900 1200 1500 1800
no of tasks
op
tim
izat
ion
tim
e (s
ecs)
GreedyBasic
Greedyw/NMA
Gredy w/WU
e
Jan. 02 Darpa SenseIT PI Meeting 20
Cost Reduction
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
5 6 7 8 9 10
no of tasks
cost
red
uct
ion
per
cen
tag
e
greedy
A*-based
Cost-limit=100
Overlap-
degree=0.6
Overlap-
prob=0.4,0.6
Jan. 02 Darpa SenseIT PI Meeting 21
Cost Reduction of Greedy AlgorithmsCost limit =
100
Overlap degree =
0.2
Overlap prob=0.4,
0.6
11
11.5
12
12.5
13
13.5
14
14.5
100
200
300
400
500
600
700
800
900
1000
1100
1200
1300
1400
1500
1600
1700
1800
no of tasks s
co
st
red
uc
tio
n p
erc
en
tag
e
GreedyBasic
Greedyw/NMA
Greedyw/WU
Jan. 02 Darpa SenseIT PI Meeting 22
Bottom Line
Both A*-based and the BAB algorithm finds optimal Both A*-based and the BAB algorithm finds optimal solution, but do not scalesolution, but do not scale
Greedy algorithms find “good” solutions and scale up wellGreedy algorithms find “good” solutions and scale up well Greedy w/NMA scales very well, but achieves smaller Greedy w/NMA scales very well, but achieves smaller
cost reduction percentagescost reduction percentages Greedy w/WU achieves very large cost savings; it Greedy w/WU achieves very large cost savings; it
becomes the clear winner as the overlap degree becomes the clear winner as the overlap degree increasesincreases
Partitioning algorithms, in conjunction with merging Partitioning algorithms, in conjunction with merging algorithms promise substantial scalability algorithms promise substantial scalability improvements.improvements.
Jan. 02 Darpa SenseIT PI Meeting 23
Other key contributions
Solved task assignment problem efficiently Solved task assignment problem efficiently despite NP-completeness. (Golubchik,Ozcan, despite NP-completeness. (Golubchik,Ozcan, Subrahmanian).Subrahmanian).
Temporal probabilistic relational DBs on top of Temporal probabilistic relational DBs on top of ODBC (TODS 2001)ODBC (TODS 2001)
Solved problem of scaling temporal probabilistic Solved problem of scaling temporal probabilistic databases –Built cost models and query optimizer. databases –Built cost models and query optimizer. (Dekhtyar, Ross, Ozcan, Subrahmanian)(Dekhtyar, Ross, Ozcan, Subrahmanian)
Probabilistic object base models (TODS 2001)Probabilistic object base models (TODS 2001) Temporal probabilistic object base models (sub)Temporal probabilistic object base models (sub)
Jan. 02 Darpa SenseIT PI Meeting 24
SenseIT group demos
Developed gateway framework for Developed gateway framework for communicating with on-node cache maintained by communicating with on-node cache maintained by Fantastic data. Fantastic data.
STM provides data conduit behind Va Tech GUI.STM provides data conduit behind Va Tech GUI. Participated in Nov. 2002 SITEX experiments at Participated in Nov. 2002 SITEX experiments at
29 Palms. UMD gateway and conduit used there.29 Palms. UMD gateway and conduit used there. UMD Gateway and STM also to be used in joint UMD Gateway and STM also to be used in joint
demo with BBN, Va Tech, and other team demo with BBN, Va Tech, and other team members tomorrow.members tomorrow.
Jan. 02 Darpa SenseIT PI Meeting 25
Contact Info V.S. SubrahmanianV.S. Subrahmanian Dept. of Computer ScienceDept. of Computer Science
AV Williams BuildingAV Williams BuildingUniversity of MarylandUniversity of MarylandCollege Park,MD 20742.College Park,MD 20742.
Tel: (301) 405-2711Tel: (301) 405-2711 Fax: (301) 405-8488Fax: (301) 405-8488 Email: [email protected]: [email protected] URL: www.cs.umd.edu/users/vs/index.htmlURL: www.cs.umd.edu/users/vs/index.html