randomized algorithms eduardo laber loana t. nogueira
TRANSCRIPT
Randomized Algorithms
Eduardo LaberLoana T. Nogueira
Quicksort
Objective
Quicksort
Objective
– Sort a list of n elements
An Idea
An Idea
Imagine if we could find an element y S such that half the members of S are smaller than y, then we could use the following scheme
An Idea
Imagine if we could find an element y S such that half the members of S are smaller than y, then we could use the following scheme
Partition S\{y} into two sets S1 and S2
An Idea
Imagine if we could find an element y S such that half the members of S are smaller than y, then we could use the following scheme
Partition S\{y} into two sets S1 and S2
S1: elements of S that are smaller than y S2: elements of S that are greater than y
An Idea
Imagine if we could find an element y S such that half the members of S are smaller than y, then we could use the following scheme
Partition S\{y} into two sets S1 and S2
S1: elements of S that are smaller than y S2: elements of S that are greater than y
Recursively sort S1 and S2
Suppose we know how to find y
Suppose we know how to find y
Time to find y: cn steps, for some constant c
Suppose we know how to find y
Time to find y: cn steps, for some constant c
we could partition S\{y} into S1 and S2 in n-1 additional steps
Suppose we know how to find y
Time to find y: cn steps, for some constant c
we could partition S\{y} into S1 and S2 in n-1 additional steps
T(n) 2T(n/2) + (c+1)n
The total number os steps in out sorting procedure would be given by the recurrence
Suppose we know how to find y
Time to find y: cn steps, for some constant c
we could partition S\{y} into S1 and S2 in n-1 additional steps
T(n) 2T(n/2) + (c+1)n
The total number os steps in out sorting procedure would be given by the recurrence
c’nlogn
What´s the problem with the scheme above?
Quicksort
What´s the problem with the scheme above?
Quicksort
How to find y?
Deterministic Quicksort
Let y be the first element of S
Deterministic Quicksort
Let y be the first element of S
Split S into two sets: S< and S>
Deterministic Quicksort
Let y be the first element of S
Split S into two sets: S< and S>
– S< : elements smaller than y
– S> : elements greater than y
Deterministic Quicksort
Let y be the first element of S
Split S into two sets: S< and S>
– S< : elements smaller than y
– S> : elements greater than y
Qsort ( S< ), Qsort ( S> )
Performance
– Worst Case: O( n2 )
– Avarage Case: O( nlogn )
Performance
– Worst Case: O( n2 ) (The set is already sorted)
– Avarage Case: O( nlogn )
Performance
– Worst Case: O( n2 ) (The set is already sorted)
– Avarage Case: O( nlogn )
An Randomzied Algorithm
An Randomzied Algorithm
An algorithm that makes choice (random) during the algorithm execution
Randomized Quicksort (RandQS)
Randomized Quicksort (RandQS)
Choose an element y uniformly at random of S
Randomized Quicksort (RandQS)
Choose an element y uniformly at random of S – Every element of S has equal probability fo being
chosen
Randomized Quicksort (RandQS)
Choose an element y uniformly at random of S – Every element of S has equal probability fo being
chosen
By comparing each element fo S with y,
determine S< and S>
Randomized Quicksort (RandQS)
Choose an element y uniformly at random of S – Every element of S has equal probability fo being
chosen
By comparing each element fo S with y,
determine S< and S>
Recursively sort S< and S>
Randomized Quicksort (RandQS)
Choose an element y uniformly at random of S – Every element of S has equal probability fo being
chosen
By comparing each element fo S with y,
determine S< and S>
Recursively sort S< and S>
– OUTPUT: S<, followed by y and then S>
Intuition
For some instance Quicksort works very bad O( n2 )
Intuition
For some instance Quicksort works very bad O( n2 )
Randomization produces different executions for the same input. There is no instance for which RandQS works bad in avarage
Analysis
Analysis
For sorting algorithms: we measure the running time of RandQS in terms of the number of comparisons it performs
Analysis
For sorting algorithms: we measure the running time of RandQS in terms of the number of comparisons it performs
This is the dominat cost in any reasonable implementation
Analysis
For sorting algorithms: we measure the running time of RandQS in terms of the number of comparisons it performs
This is the dominat cost in any reasonable implementation
Our Goal: Analyse the expected number of comparisons in an execution of RandQS
Analysis
Si: the ith smallest element of S
Analysis
Si: the ith smallest element of S
S1 is the smallest element of S
Analysis
Si: the ith smallest element of S
S1 is the smallest element of SSn is the largest element of S
Analysis
Si: the ith smallest element of S
Define the random variable
S1 is the smallest element of SSn is the largest element of S
xik = 1, if Si and Sj are compared
0, otherwise
Analysis
Si: the ith smallest element of S
Define the random variable
S1 is the smallest element of SSn is the largest element of S
xik = 1, if Si and Sj are compared
0, otherwise
Dado um experimento aleatório com espaço amostral S, uma variável aleatória é uma função que associa a cada elemento amostral um número real
Analysis
Xij is a count of comparisons between Si and Sj:
The total numberof comparisons:
i = 1
n
j > iXij
Analysis
Xij is a count of comparisons between Si and Sj:
The total numberof comparisons:
i = 1
n
j > iXij
We are interested in the expected number of comparisons
E[ ] i = 1
n
j > iXij
Analysis
Xij is a count of comparisons between Si and Sj:
The total numberof comparisons:
i = 1
n
j > iXij
We are interested in the expected number of comparisons
E[ ] = i = 1
n
j > iXij
i = 1
n
j > iE[Xij]
By the linearity of E[]
Analysis
Pij: the probability that Si and Sj are compared in an execution
Analysis
Pij: the probability that Si and Sj are compared in an execution
Since Xij only assumes the values 0 and 1,
ikikikik PPPXE )1(01ij ij ij ij
Analysis – Binary Tree T of RandQS
Each node is labeled with a distinct element of S
y
T
Analysis – Binary Tree T of RandQS
Each node is labeled with a distinct element of S
y
T
S<
Analysis – Binary Tree T of RandQS
Each node is labeled with a distinct element of S
y
T
S< S>
Analysis – Binary Tree T of RandQS
Each node is labeled with a distinct element of S
y
T
S< S>
The root of T is compared to the elements in the two sub-trees,but no comparisons is perfomed between an element of the left and righ sub-trees
Analysis – Binary Tree T of RandQS
Each node is labeled with a distinct element of S
y
T
S< S>
The root of T is compared to the elements in the two sub-trees,but no comparisons is perfomed between an element of the left and righ sub-trees
There is an comparison betweenSi and Sj if and only if one of these elements is an ancestor of the other
Analysis – Binary Tree T of RandQS
Consider the permutation obtained by visiting the nodes of T in increasing order of the level numbers, and in a lef-to-rigth order within each level
Example: S=(3, 6, 2, 5, 4,1)
Example: S=(3, 6, 2, 5, 4,1)
2
Example: S=(3, 6, 2, 5, 4,1)
2
1
{3, 6, 5, 4}{1}
Example: S=(3, 6, 2, 5, 4,1)
2
1 5
{3, 6, 5, 4}
{6}{3, 4}
{1}
Example: S=(3, 6, 2, 5, 4,1)
2
1 5
{3, 6, 5, 4}
4 6
3
{6}{3, 4}
{3}
{1}
Example: S=(3, 6, 2, 5, 4,1)
2
1 5
{3, 6, 5, 4}
4 6
3
{6}{3, 4}
{3}
{1}
Example: S=(3, 6, 2, 5, 4,1)
2
1 5
{3, 6, 5, 4}
4 6
3
{6}{3, 4}
{3}
{1}
Example: S=(3, 6, 2, 5, 4,1)
2
1 5
{3, 6, 5, 4}
4 6
3
{6}{3, 4}
{3}
{1}
= (2, 1, 5, 4, 6, 3)
Back to the Analysis
To compute pij we make two observations:– There is a comparison between Si and Sj if and only
if Si or Sj occurs earlier in the permutation than any element Sl such that i < l < j
Any of the elements Si, Si+1, ..., Sj is likely to be the first of theses elements to be chosen as a partitioning element, and hence to appear first in
The probability that this first element is either Si or Sj is exactly 2/(j-i+1)
Back to the Analysis
To compute pij we make two observations:– There is a comparison between Si and Sj if and only
if Si or Sj occurs earlier in the permutation than any element Sl such that i < l < j
Any of the elements Si, Si+1, ..., Sj is likely to be the first of theses elements to be chosen as a partitioning element, and hence to appear first in
The probability that this first element is either Si or Sj is exactly 2/(j-i+1)
Back to the Analysis
To compute pij we make two observations:– There is a comparison between Si and Sj if and only
if Si or Sj occurs earlier in the permutation than any element Sl such that i < l < j
Any of the elements Si, Si+1, ..., Sj is likely to be the first of theses elements to be chosen as a partitioning element, and hence to appear first in
The probability that this first element is either Si or Sj is exactly 2/(j-i+1)
Back to the Analysis
To compute pij we make two observations:– There is a comparison between Si and Sj if and only
if Si or Sj occurs earlier in the permutation than any element Sl such that i < l < j
Any of the elements Si, Si+1, ..., Sj is likely to be the first of theses elements to be chosen as a partitioning element, and hence to appear first in
The probability that this first element is either Si or Sj is exactly 2/(j-i+1)
Análise
Therefore,
Pij = 2/(j-i+1)
Análise
Therefore,
Pij = 2/(j-i+1)
i = 1
n
j > iE[Xij] =
i = 1
n
j > iPij =
Análise
Therefore,
Pij = 2/(j-i+1)
i = 1
n
j > iE[Xij] =
i = 1
n
j > iPij = 2/(j-i+1)
i = 1
n
j > i
Análise
Therefore,
Pij = 2/(j-i+1)
i = 1
n
j > iE[Xij] =
i = 1
n
j > iPij = 2/(j-i+1)
i = 1
n
j > i
2/ki = 1 k=1
n n-1+1
Análise
Therefore,
Pij = 2/(j-i+1)
i = 1
n
j > iE[Xij] =
i = 1
n
j > iPij = 2/(j-i+1)
i = 1
n
j > i
2/ki = 1 k=1
n n-1+1
2i = 1 k=1
n n
1/k
Análise
Therefore,
Pij = 2/(j-i+1)
i = 1
n
j > iE[Xij] =
i = 1
n
j > iPij = 2/(j-i+1)
i = 1
n
j > i
2/ki = 1 k=1
n n-1+1
2i = 1 k=1
n n
1/kSérie Harmônica
Análise
Therefore,
Pij = 2/(j-i+1)
i = 1
n
j > iE[Xij] =
i = 1
n
j > iPij = 2/(j-i+1)
i = 1
n
j > i
2/ki = 1 k=1
n n-1+1
2i = 1 k=1
n n
1/kSérie Harmônica 2 nln n
RandQs x DetQs
Expected time of RandQs: O( nlogn ) A certain expected value may not garantee a reasonable
probability of success. We could have, for example, the following probabilities
nnlog of executing O( n2 ) operations
of executing O( nlogn ) operations1- n
nlog
RandQs x DetQs
For n=100 => in 7 % cases, the algorithm would execute in O( n2 ).
Some times we want to garantee that the algorithm performance will not be far from its avarage one
RandQs x DetQs
For n=100 => in 7 % cases, the algorithm would execute in O( n2 ).
Some times we want to garantee that the algorithm performance will not be far from its avarage one
Objective: Prove that with high probability the RandQS algorithm works well
High Probability Bound
High Probability Bound
The previous analysis only says that the expected running time is O(nlog n)
High Probability Bound
The previous analysis only says that the expected running time is O(nlog n)
This leaves the possibility of large “deviations” from this expected value
High Probability Bound
RECALL:– Quicksort choses a pivot at random from the input
array
High Probability Bound
RECALL:– Quicksort choses a pivot at random from the input
array– Splits it into smaller and larger elements
High Probability Bound
RECALL:– Quicksort choses a pivot at random from the input
array– Splits it into smaller and larger elements– Recurses on both subarrays
High Probability Bound
RECALL:– Quicksort choses a pivot at random from the input
array– Splits it into smaller and larger elements– Recurses on both subarrays
Fix an element x in the input
High Probability Bound
RECALL:– Quicksort choses a pivot at random from the input
array– Splits it into smaller and larger elements– Recurses on both subarrays
Fix an element x in the inputx belogs to a sequence of subarrays
High Probability Bound
RECALL:– Quicksort choses a pivot at random from the input
array– Splits it into smaller and larger elements– Recurses on both subarrays
Fix an element x in the inputx belogs to a sequence of subarrays
x´s contribution to the running time is proportional to the number of different subarrays it belongs
High Probability Bound
Every time x is compared to a pivot, its current subarray is split and x goes to one of the subarrays
High Probability Bound
Every time x is compared to a pivot, its current subarray is split and x goes to one of the subarrays
With high probability, x is compared to O(log n) pivots
High Probability Bound
Every time x is compared to a pivot, its current subarray is split and x goes to one of the subarrays
With high probability, x is compared to O(log n) pivots
With probability 1- n-c, for some pos. constant c
GOOD and BAD Splits
GOOD and BAD Splits
We say that a pivot is good if each of the subarrays has size at most ¾ (equivalently,at least ¼) of the size of the split subarray
GOOD and BAD Splits
We say that a pivot is good if each of the subarrays has size at most ¾ (equivalently,at least ¼) of the size of the split subarray
Otherwise, it is bad
GOOD and BAD Splits
We say that a pivot is good if each of the subarrays has size at most ¾ (equivalently,at least ¼) of the size of the split subarray
Otherwise, it is bad The probability of a good and of a bad split is ½
GOOD and BAD Splits
We say that a pivot is good if each of the subarrays has size at most ¾ (equivalently,at least ¼) of the size of the split subarray
Otherwise, it is bad The probability of a good and of a bad split is ½
X can participate in at most log4/3 n good splits
High Probability Bound
Upper bound the probability of less than M/4 good splits in M splits
High Probability Bound
Upper bound the probability of less than M/4 good splits in M splits
Set M so that log4/3 n M/4
High Probability Bound
Upper bound the probability of less than M/4 good splits in M splits
Set M so that log4/3 n M/4 Let M = 32 ln n:
High Probability Bound
Upper bound the probability of less than M/4 good splits in M splits
Set M so that log4/3 n M/4 Let M = 32 ln n log4/3 n 8 ln n
Good choice
High Probability Bound
Upper bound the probability of less than M/4 good splits in M splits
Set M so that log4/3 n M/4 Let M = 32 ln n log4/3 n 8 ln n
exp(-M/8) 1/n4
Good choice
High Probability Bound
Upper bound the probability of less than M/4 good splits in M splits
Set M so that log4/3 n M/4 Let M = 32 ln n log4/3 n 8 ln n
exp(-M/8) 1/n4
The probablility of participating in more than M = 32 ln n splits is less than 1/n4
Good choice
High Probability Bound
The probability that any element participates in more than M=32 ln n splits is less than 1/n3
High Probability Bound
The probability that any element participates in more than M=32 ln n splits is less than 1/n3
With probability at least 1-1/n3, the running time of quicksort is O(n log n)
Advantages of Randomized Algorithms
For many problems, randomized algorithms run faster than the best know deterministic algorithms
Many randomized algorithms are simpler to describe and implement than deterministic algorithms for comparable performance
Advantages of Randomized Algorithms
For many problems, randomized algorithms run faster than the best know deterministic algorithms
Many randomized algorithms are simpler to describe and implement than deterministic algorithms for comparable performance
Advantages of Randomized Algorithms
For many problems, randomized algorithms run faster than the best know deterministic algorithms
Many randomized algorithms are simpler to describe and implement than deterministic algorithms for comparable performance
Minimum Cut Problem
Entrada:
– Grafo G=(V,E)
Saída:
– Conjunto S V que minimiza , ou seja, o
número de arestas que tem uma extremidade em S
e outra em
SSe ,
S
Minimum Cut Problem
S S 3, SSe
Notação– d(v) : grau do vértice v no grafo– N(v) : vizinhança de v
Minimum Cut Problem - Applications
Network Reliability: If a graph has a small min-cut, then it is poorly connected
Clustering– Web pages = nodes– Hyperlinks = edges
Divide the graph into cluster with little connection between different clusters.
Contração de arestas
Dado um grafo G=(V,E) e uma aresta e=(u,v), a contração da aresta e produz o grafo G/e=(V’, E’), onde
,,' uvvuVV
evuvuEE de esextremidad sãoe|,'
EvxEuxvuxuvx ,ou,e,|,
Contração de arestas - Exemplo
G G/ea
b
c
d
uv
f
g
a
b
c
d
u v
f
g
e
Lema 1
Lema 1: o tamanho do corte mínimo em G/e é maior ou igual ao tamanho do corte mínimo em G
Lema 1
Lema 1: o tamanho do corte mínimo em G/e é maior ou igual ao tamanho do corte mínimo em G
Prova: podemos associar cada corte em G/e a um corte em G com mesmo tamanho. Basta substituir em S os nós obtidos por contração pelos nós originais.
Lema 2
Lema 2: Se o corte mínimo em um grafo tem
tamanho k, então d(v) k, para todo v V.
Lema 2
Lema 2: Se o corte mínimo em um grafo tem
tamanho k, então d(v) k, para todo v V.
Prova: Caso contrário S={ v } seria um corte de
tamanho menor que k.
Corolário
Corolário 1: Se o corte mínimo em um grafo
tem tamanho k, então |E| k.n/2
Corolário
Corolário 1: Se o corte mínimo em um grafo tem tamanho k, então |E| k.n/2
Prova: Segue do lema 2 e de que
2
Vv
vdE
Randomized MinCut
G0 G
Para i=1 até |V|-2
– selecione aleatoriamente ei em Gi-1
– faça Gi = Gi-1 / ei
Retorne os vértices de um dos supervértices obtidos
Probabilidade
Lema 3: Seja t1, t2, ..., tk uma coleção de eventos. Temos que
k
i
i
jji
k
ii ttt
1
1
11
PrPr Prova:
Base:
1
122 Pr
PrPrt
tttt i okPrPrPr 12121 ttttt
Assuma que vale para k, provar para k+1.
Teorema
Seja C = { e1, e2, ..., ek } um corte mínimo no
grafo G=(V,E). Se nenhuma aresta de C é
escolhida pelo RndMinCut, então as arestas
que sobram no grafo final são as arestas de C.
Teorema - prova
Sejam A e B os dois supervértices obtidos e seja AC e BC as componentes conexas obtidas retirando C.
Como nenhuma aresta de C foi escolhida:
CC
CC
BBABBAAA
oue,ou
Teorema - prova
Logo, A=AC e B=BC
De fato, assuma que a AC e b BC tal que a,b A . Neste caso, existe um caminho em A entre a e b que utiliza somente arestas escolhidas pelo algoritmo. Como qualquer caminho entre a e b tem que utilizar arestas de C, logo uma aresta de C é escolhida. Contradição!
Análise
Seja C o conjunto de arestas de um corte mínimo em G.
Calculamos a probabilidade do algoritmo não escolher nenhuma aresta de C para contração. Se isto acontece, o algoritmo retorna um corte mínimo.
Análise
Sortei : evento indicando que o algoritmo não sorteou uma aresta de C na i-ésima iteração.
2
1
PrCencontrar Pr
mínimo corteencontrar Prn
iisorte
Análise
Temos:
2
1
1
1
2
1
PrPrn
i
i
jji
n
ii sortesortesorte
EC
sorte 1Pr 1
Segue da relação entre o tamanho do corte e o número de arestas (corolário 1) que:
n
nsorte 2Pr 1
Análise
Na segunda iteração o grafo G1 tem n-1 vértices e seu corte mínimo tem tamanho |C|.
Logo,
13
12
1Pr 12
nn
CnC
sortesorte
Análise
Em geral,
Segue que,
121Pr
1
1
insortesorte
i
jji
12
121Pr
2
1
2
1
nninsorte
n
i
n
ii
Análise
Logo, conseguimos um min-cut com probabilidade maior ou igual a
12nn
n=100 => 0,02 % (RUIM)
O que fazer ?
Randomized MinCut 2
Repetir o processo RndMinCut várias vezes e devolver o melhor corte encontrado
Repetindo K vezes a probabilidade de encontrar o corte mínimo é
K
nn
1
211
Análise
Repetindo n2/2 vezes a probabilidade de sucesso é (e-1)/e 64 %
K repetições => O (K.m) tempo
Complexidade do Algoritmo mais rápido de Karger
Running time: O(n2 log n)
Space: O(n2)
Este algoritmo encontra um min-cut com probabilidade
(1/log n) [D. Karger e C. Stein, Stoc 1993]
Quem se habilita ?
Minimum Cut Problem – Deterministic Algorithm Complexity
O(nm log n2/m)
J. Hao and J. B. Orlin[1994] (baseado em fluxo em redes)
Dois tipos de algoritmos randomizados
Algoritmos Las Vegas– Sempre produzem a resposta correta– Tempo de execução é uma variável aleatória
Exemplo: RandQs– Sempre produz seqüência ordenada– Tempo de término varia de execução para
execução em uma dada instância
Dois tipos de algoritmos randomizados
Algoritmos Monte-Carlo– Podem produzir respostas incorretas– A probabilidade de erro pode ser cotada– Executando o algoritmo diversas vezes podemos
tornar a probabilidade de erro tão pequena quanto se queira
Exemplo: Min-Cut
MAX SAT
Entrada n variáveis booleanas : x1,... Xn
m cláusulas : C1,... Cm
Pesos wi >= 0 para cada clausula Ci
Objetivo : Encontrar uma atribuição de verdadeiro/falso para xi que maximize a soma dos pesos das clausulas
satisfeitas
MAX SAT
Algoritmo Aleatorizado
Para i=1,...,nSe random(1/2) = 1
xi true
Senãoxi false
Com probabilidade ½ dizemos que uma variável é verdadeira ou falsa
MAX SAT
Teorema : O algoritmo tem aproximação ½Prova: Considere a variável aleatória xj
Logo,
)()()( jj
jjj
j xEwxwEwE
jj
j xww
contrário caso 0,
satisfeita é j clausula a se ,1jx
MAX SAT
E(xj) = Pr(clausula j ser satisfeita)
Lj : número de literais na clausula j
Obs: A clausula j não é satisfeita somente se todos literais forem 0
Cj = (x1 v x3 v x5 v x6)
Devemos ter x1=0, x3 = 1, x5 =0 e x6 =0
Probabilidade = (1/2)4
Caso Geral=(1/2)Lj
MAX SAT
Probabilidade da clausula j ser satisfeita é 1-(1/2)Lj
Logo,
0,5-aproximação
Obs: é um limite superior
22211)( OPT
wwwE j
j
j
L
j
j
j
jw
MAX SAT
O que aconteceria se toda clausula tivesse exatamente 3 literais?
7/8-aproximação
Hastad 97) Se MAXE3SAT tem aproximação (7/8 + para algum > 0, P = NP
jj
jj wwwE
87
211)(
3
MAX SAT: Desaleatorização
C1 = (x1 v x2 v x3) w1
C2 = ( x2 v x3 ) w2
Cada folha da árvore corresponde a um atribuição:Cada folha esta associada a um peso (soma dos pesos
das clausulas satisfeitas pela atribuição correspondente a folha)
MAX SAT: Desaleatorização
E(c(I)) = 7/8 w1 + w2 /2