lecture 4: selection and median - stanford...

Lecture4:SelectionandMedian

Whileyouarewaitingforclasstostart:

GotoPiazza,findthisnote,filloutthepoll!

Orjustgotothelinkdirectly:

https://marykw.typeform.com/to/SeFSMa

Doitnow!Youcanevendoitonyourphone!(andifyoucan’tdoitnow,thenpleasedoitlater!)

Notenough!

Announcements!

• CheckouttheWiCS hackathon:

• http://web.stanford.edu/group/wics/hackoverflow/spr2017/

• April15

• RecitationSections:

• LocationmightchangetoGatesBasement.

• (ChecktheGoogleCalendarbeforeheadingtosection.)

• PleasefilloutthesurveyonPiazza.

• I’llupdatemyteachingstylebasedonit,soevenifyouarehappywithhowthingsare,youshouldregisterthat!

• HW1dueFriday.

• (AndHW2alsopostedFriday).

Somefinalremarksaboutthemastertheorem

• Suppose� � = � ⋅ � &' + � �* .Then

A powerful

theorem it is…

JedimasterYoda

Threeparameters:

a:numberofsubproblems

b:factorbywhichinputsizeshrinks

d:needtodond worktocreateallthe

subproblems andcombinetheirsolutions.

Lasttime…

• Endedontheboardwith� �+,-. &• TheTheoremsays� �+,-. /• Isitatypo?

• Goodexercise:

�+,-. / = �+,-. & Makes many

typos, Mary

does.

but in this case

that’s just how

logs work

Someintuitionaboutthecases

• Case1:a=bd

• Therecursiontreehasthesameamountofworkateverylevel.(LikeMergeSort).

• Case3:a>bd

• Thetreebranchesreallyquicklycomparedtoworkperproblem!Thebulkoftheworkisdoneatthebottomofthetree.(LikeKaratsuba).

• Case2:a<bd

• Theworkdoneshrinkswayfasterthanwebranchnewproblems.Thebulkoftheworkisdoneattherootofthetree.(Wehaven’tseenthisyetbutwewilltoday).

Workatlevelt:O(nd (a/bd )t )

Today:morerecursion,beyondtheMasterTheorem.

• TheMasterTheoremonlyworkswhenallsub-problemsarethesamesize.

• That’snotalwaysthecase.

I can handle all the recurrence

relations that look like

� � = � ⋅ � &' + � �* .

Before this theorem I was but

the learner. Now I am the

master.Only a master of evil*, Darth.

*Moreprecisely,onlya

masterofsame-sizesub-

problems…stillpretty

handy,actually.

• Todaywe’llseeanexample

wheretheMasterTheorem

won’twork.

• We’llusesomethingcalled

thesubstitutionmethod

instead.

Theproblemwewillsolve

• SELECT(A,k):

• Returnthek’th smallestelementofA.

Aisanarrayofsizen,kisin{1,…,n}

7 4 3 8 1 5 9 14

• SELECT(A,1)=MIN(A)

• SELECT(A,n/2)=MEDIAN(A)

• SELECT(A,n)=MAX(A)

• SELECT(A,1)=1

• SELECT(A,2)=3

• SELECT(A,3)=4

• SELECT(A,8)=14

Assumingniseven!

Otherwiseneed

someceilingsor

floors.

We’regonna doitintimeO(n)

• Let’sstartwithMIN(A)akaSELECT(A,1).

• MIN(A):

• ret=∞• For i=1,..,n:

• IfA[i]<ret:

• ret=A[i]

• Return ret

• TimeO(n).Yay!

ThisstuffisO(1)

ThislooprunsO(n)times

HowaboutSELECT(A,2)?

• SELECT2(A):

• ret=∞• minSoFar=∞• For i=1,..,n:

• IfA[i]<retandA[i]<minSoFar:

• ret=minSoFar

• minSoFar =A[i]

• ElseifA[i]<retandA[i]>=minSoFar:

• ret=A[i]

• Return ret

(Theactualalgorithmhereis

notveryimportantbecause

thiswon’tendupbeingavery

goodidea…)

StillO(n)SOFARSOGOOD.

SELECT(A,n/2)akaMEDIAN(A)?

• MEDIAN(A):

• ret=∞• minSoFar=∞• secondMinSoFar =∞• thirdMinSoFar =∞• fourthMinSoFar =∞• ….

• Thisisnotagoodideaforlargek(liken/2orn).

• BasicallythisisjustgoingtoturnintosomethinglikeINSERTIONSORT…andthatwasO(n2).

Amuchbetterideaforlargek

• SELECT(A,k):

• A=MergeSort(A)

• return A[k]

• RunningtimeisO(nlog(n)).

• Sothat’sthebenchmark….

Canwedobetter?We’rehopingtogetO(n)

Idea:recursion

9 8 3 6 1 4 2Saywewantto

findSELECT(A,k)

First,picka“pivot.”

We’llseehowtodo

thislater.

Howabout

thispivot?

Next,partitionthearrayinto

“biggerthan6”or“lessthan6”

9 8 3 6 1 4 2

L=arraywiththings

smallerthanA[pivot]

R=arraywiththings

largerthanA[pivot]

ThisPARTITIONsteptakes

timeO(n).(Noticethat

wedon’tsorteachhalf).

Ideacontinued…

9 83

6

1 4 2pivot

L=arraywiththings

smallerthanA[pivot]

R=arraywiththings

largerthanA[pivot]

Saywewantto

findSELECT(A,k)

• Ifk=5=len(L)+1:

• WeshouldreturnA[pivot]

• Ifk<5:

• WeshouldreturnSELECT(L,k)

• Ifk>5:

• WeshouldreturnSELECT(R,k– 5)

Thissuggestsa

recursivealgorithm

(stillneedtofigureout

howtopickthepivot…)

Let’smakethatabitmoreformal

• PARTITION(A,p):

• L=newarray

• R=newarray

• For i=1,…,n:

• If i==p:

• continue

• ElseifA[i]<=A[p]:

• L.append(A[i])

• ElseifA[i]>A[p]:

• R.append(A[i])

• Return L,A[p],R

• ThisistheO(n)PARTITION

algorithmthatwesawbefore.

• Forclarity,I’mjustgoingto

initializetwonewarrays,LandR.

(Assumetheyaredynamically

sized,andthatwecanappend

stuffintimeO(1),andaccessany

indexintimeO(1)).

• However,youcanimplementthis

(andeverythingelsewewilldoin

thislecture)in-place,withoutany

oftheseconsiderations. (Fun

exercise!OrseeCLRS.)

initialize

LandR

gothrough

elts oneata

time…put

smallonesinL,

bigonesinR.

MoreformalpartII

• PARTITION(A,p):

• L=newarray

• R=newarray

• For i=1,…,n:

• If i==p:

• continue

• elseIfA[i]<=A[p]:

• L.append(A[i])


• R.append(A[i])

• Return L,A[p],R

• SELECT(A,k):

• Choosepin{1,…,n}

• L,A[p],R=PARTITION(A,p)

• Iflen(L)=k- 1:

• ReturnA[p]

• ElseIflen(L)>k- 1:

• Return SELECT(L,k)

• Elseiflen(L)<k- 1:

• returnSELECT(R,k– len(L)– 1)

• Iflen(A)<=50:

• A=MergeSort(A)

• ReturnA[k]

We’llseewhyI

chose50later.

It’spretty

arbitrary.

(pictureonboard)

Correctness(illustrationonboard)

• Recursioninvariant:

Atthereturnofeachrecursivecallofsize<n,SELECT(A,k)returnsthek’th

smallestelementofA.

• Basecase(“Initialization”):

• Iflen(A)<=50,thentheMergeSortapproachis“clearly”correct.

• SELECT(A,p=k):

• Iflen(A)<=50:

• A=MergeSort(A)

• ReturnA[k]



• Iflen(L)=k- 1:

• ReturnA[p]





Note:SoonI’mgoingtostopproving

correctnessinclass– eventuallyallthese

argumentswillstarttolookthesame.

• Inductivestep:(“Maintenance”)• Supposethattherecursioninvariantholdsforn.

• Wanttoshowthatitholdsforn+1.

• Threecases:

• iflen(L)=k-1,thenA[p]isthecorrectthingtoreturn.

• Iflen(L)>k-1,thenthek’th smallestelementofListhecorrectthingtoreturn

• Andbyinduction,thisisindeedwhatwereturn.

• Iflen(L)<k-1,thenthe(k– (len(L)- 1)’st smallestelt ofRisthecorrectthingtoreturn.

• Andbyinduction,thisisindeedwhatwereturn.

• Conclusion(“Termination”)• Byinduction,therecursioninvariant

holdsforn+1,whichmeansthat

SELECT(A,k)iscorrect.

Note:somethinglikethisistotally

acceptableonyourHW(maybewithone

moresentence,oranexample,sayingwhy

thosearetherightthingstoreturn.)

Thereseems

tobesome

whitespace

notyetused.

That’sbetter.

Howaboutruntime?

• Let’stryarecurrencerelation…

• � � = � �� + � � �� < � − 1� �� + � � �� > � − 1� � �� = � − 1

• SELECT(A,p=k):

• Iflen(A)<=150:

• A=MergeSort(A)

• ReturnA[k]



• Iflen(L)=k- 1:

• ReturnA[p]

• ElseIflen(L)<k- 1:


• Elseiflen(L)>k- 1:


• Whatislen(L),len(R)?

• Let’spretendthatlen(L)isaboutn/2.

• � � ≤ � &< + �(�)

RecalltheMasterTheoremwhichtotallydoesn’tapplyhere,wearecheatingbypretendingweknowtheproblemsize.

• Suppose� � = � ⋅ � &' + � �* .Then

LuckytheLackadaisical

Lemur

• � � ≤ � &< + �(�)

• Soa=1,b=2,d=1

• � � ≤ � �* = � �

Inourcase:

Note:Thisisa

rhetoricalpointfor

intuitioninlecture.It

isNOTOKAYasafinal

solutiononyourHW.

Howaboutruntime?


• � � = � �� + � � �� < � − 1� �� + � � �� > � − 1� � �� = � − 1

• SELECT(A,p=k):

• Iflen(A)<=150:

• A=MergeSort(A)

• ReturnA[k]



• Iflen(L)=k- 1:

• ReturnA[p]






• Let’spretend thatlen(L)isaboutn/2.

• � � ≤ � &< + �(�)

• � � = �(�)• Thatwouldbegreat!

7n/10 (wecan

even

assume

something

alittle

weaker)


• Suppose� � = � ⋅ � &' + � �* .Then

Luckythe

LackadaisicalLemur

• � � ≤ � ?&@A + �(�)

• Soa=1,b=10/7,d=1

• � � ≤ � �* = � �

Inourcase:

Howaboutruntime?


• � � = � �� + � � �� < � − 1� �� + � � �� > � − 1� � �� = � − 1

• SELECT(A,p=k):

• Iflen(A)<=150:

• A=MergeSort(A)

• ReturnA[k]



• Iflen(L)=k- 1:

• ReturnA[p]







• � � ≤ � &< + �(�)

• � � = �(�)• Thatwouldbegreat!

7n/10

!!!!!

Howaboutruntime?


• � � = � �� + � � �� < � − 1� �� + � � �� > � − 1� � �� = � − 1

• SELECT(A,p=k):

• Iflen(A)<=150:

• A=MergeSort(A)

• ReturnA[k]



• Iflen(L)=k- 1:

• ReturnA[p]







• � � ≤ � &< + �(�)

Canweget

awaywithn-1?


• Suppose� � = � ⋅ � &' + � �* .Then

Luckythe

LackadaisicalLemur

• � � ≤ � � − 1 + �(�)• Soa=1,b=1/(1-1/n),d=1

• � � ≤ � � still?

• NO!!!bneedstobeindependentofnforthemaster

thm towork.ActualrunningtimeisO(n^2).

Inourcase:

Howaboutruntime?


• � � = � �� + � � �� < � − 1� �� + � � �� > � − 1� � �� = � − 1

• SELECT(A,p=k):

• Iflen(A)<=150:

• A=MergeSort(A)

• ReturnA[k]



• Iflen(L)=k- 1:

• ReturnA[p]







• � � ≤ � &< + �(�)

• � � = � �<• NotgoodenoughL!

Canweget

awaywithn-1?

Moralofthisextremelyshadylogic

• IfwecanpickapivotsothatLandRsomewhatbalanced(evenlike7n/10),thenwe’redoinggreat.Otherwise,nogood.

• Try1:Let’spickthepivottobethemedian!

• ThenLandRarealwaysn/2.(or&< or

&< ).

• Problem:That’sexactlytheproblemwe’retryingtosolvetobeginwith.

• Solution:• Wecan’tfindthemedianofnthings(yet),butwecanrecursively findthemedianofn/5things…

• thatwillgiveussomething“closeenough”tothemedianthatwecan(rigorously)applythepreviousanalysis.

Howtopickthepivot• CHOOSEPIVOT(A):

• SplitAintom=&B groups,ofsize<=5each.

• For i=1,..,m:• Findthemedianwithinthei’th group,callitpi

• p= SELECT([p1,p2,p3,…,pm ],m/2 )

• return p

• SELECT(A,p=k):

• Iflen(A)<=50:

• A=MergeSort(A)

• ReturnA[k]

• p=CHOOSEPIVOT(A)


• Iflen(L)=k- 1:

• ReturnA[p]

• ElseIflen(L)<k- 1:


• Elseiflen(L)>k- 1:

• returnSELECT(R,k–len(L)– 1)

5 9 1 3 41 8 9 3 15 12 2 1 5 20 15 13 2 4 6 12 1 15 22 3

ThistakestimeO(1),

sinceeachgrouphas

size5

8

4

5

6

12PivotisSELECT(,3)=6:8 4 5 6 12

5 9 1 3 41 8 9 3 15 12 2 1 5 20 15 13 2 46

12 1 15 22 3

5 91 3 41 8 93 15 122 1 5 20 15 132 4

6

121 15 223

PARTITIONaroundthat5:

ThispartisL ThispartisR:it’salmostthesamesizeasL.

Sothisgivesthewholealgorithm• PARTITION(A,p):

• L=newarray

• R=newarray

• For i=1,…,n:

• If i==p,continue

• ElseIfA[i]<=A[p]:

• L.append(A[i])


• R.append(A[i])

• Return L,A[p],R

• SELECT(A,p=k):

• Iflen(A)<=50:

• A=MergeSort(A)

• ReturnA[k]

• p=CHOOSEPIVOT(A)


• Iflen(L)=k- 1:

• ReturnA[p]





• CHOOSEPIVOT(A):

• SplitAintom=&B groups,ofsize<=5each.

• For i=1,..,m:• Findthemedianwithinthei’th group,callitpi

• p= SELECT([p1,p2,p3,…,pm ],m/2 )

• return p

• Doesitwork?

• Yes,ourproofbefore

workedforanypivoting

strategy.

Note:Weuse

recursionintwo

ways!Bothin

SELECT itself,andin

CHOOSEPIVOT.

What’stheruntime?Outlinefortherestoftoday

• Lemma:eachofthearraysLandRareprettybalanced.

• Thesubstitutionmethod:• A“guess-and-check”approachtosolvingrecurrencerelations.

• WorksevenwhentheMasterTheoremdoesn’t.

• WewillusethesubstitutionmethodtoseethatSELECTindeedrunsintimeO(n).

• (Timepermitting,we’llalsoseehowNOTtousethesubstitutionmethod.Ifwedon’thavetimeinclass,lookaheadattheslidesonyourown.)

Lemma:thatmedian-of-mediansthingisagoodidea.

• Lemma:IfLandRareasinthealgorithmSELECTgivenabove,then

� ≤ 7�10 + 5

and

� ≤ 7�10 + 5

• Whyisthisgood?

• Itmeansthatthingsareprettybalanced.

• Wewillseeaproofbypicture.

• SeeCLRSortheLectureNotesforproofbyproof.

Arbitrarynumbers!

Ithoughtthewholepointof

O()wasthatwe’dneverhave

todothatagain.

We’reonlydoingitsowecan

geta(correct)O()bound

later…soourchoiceof

numberswill bepretty

arbitrary.

grumpycat

ProofbypicturegenerallynotokayonHW– needatleastafew

words.(Butpicturesareencouragedifitmakesthingsclearer!)

Proofbypicture

1

8

9

3

15

5

18

4

6

35

2

10

7

12

11

3

13

70

4

2

6

7

17

22

Saytheseareourm=[n/5]sub-arraysofsizeatmost5.

5

m

Proofbypicture

1

3

8

9

15

4

5

6

18

35

2

7

10

11

12

2

3

4

13

70

6

7

17

22

Inourhead,let’ssortthem.

5

m

Thenfindmedians.

8 6 10 4

7

Proofbypicture

1

3

8

9

15

4

5

6

18

35

2

7

10

11

12

2

3

4

13

70

6

7

17

22

5

m

Thenlet’ssortthembythemedian

Proofbypicture

1

3

8

9

15

4

5

6

18

35

2

7

10

11

12

2

3

4

13

70

6

7

17

22

5

m

Themedianofthemediansis7.That’sourpivot!

Proofbypicture

1

3

8

9

15

4

5

6

18

35

2

7

10

11

12

2

3

4

13

70

6

7

17

22

5

m

HowmanyelementsareSMALLERthanthepivot?

Proofbypicture

1

3

8

9

15

4

5

6

18

35

2

7

10

11

12

2

3

4

13

70

6

7

17

22

5

m

Atleasttheseones:everythingaboveandtotheleft.

Proofbypicture

1

3

8

9

15

4

5

6

18

35

2

7

10

11

12

2

3

4

13

70

6

7

17

22

5

mHowmanyofthosearethere?

atleast3 ⋅ H< − 2

3 ⋅ H< − 1 ofthese,but

thenoneofthemcouldhave

beenthe“leftovers”group.

Proofbypicture

1

3

8

9

15

4

5

6

18

35

2

7

10

11

12

2

3

4

13

70

6

7

17

22

5

m

SohowmanyareLARGERthanthepivot?Atmost

� − 1 − 3 �2 − 2 ≤ 7�

10 + 5Remember

� = �5

(derivation

onboard)

Thatwasonepartofthelemma

• Lemma:IfLandRareasinthealgorithmSELECTgivenabove,then

� ≤ 7�10 + 5

and

� ≤ 7�10 + 5

Theotherpartisexactlythesame.

Whyisthisuseful?

Thesubstitutionmethod

• Recursiontreescangetprettymessyhere,sincewehavearecurrencerelationthatdoesn’tnicelybreakupourbigproblemintosub-problemsofthesamesize.

• Instead,wewilltryto:

• Makeaguess

• Checkusinganinductiveargument

• Thisiscalledthesubstitutionmethod.

Substitutionmethod

• Supposethat� � ≤ � ⋅ � � + ∑ �(�N)ONP@

• Let’sguessthesolutionis

� � ≤ Q� ⋅ � �A �� ≤ �A� ⋅ � � �� > �A (*)

• (aka,guessingT(n)=O(g(n)))

• We’llprovethisbyinduction,withtheinductivehypothesis(*)forallsmallern’s.

worktocallrecursive

sub-problemsand

mergethemback

Workinrdifferent

sub-problems,which

mighthavedifferent

sizes.

Inourcase

• � � ≤ � ⋅ � + � &B + � ?&

@A + 5

• Inductivehypothesis:

• � � ≤ Q� ⋅ 50�� ≤ 50� ⋅ �� > 50 (aka,T(n)=O(n)).

Thecn istheO(n)work

doneateachlevelfor

PARTITION

TheT(n/5)isforthe

recursivecalltogetthe

medianinFINDPIVOT

TheT(7n/10+5)isfor

therecursivecallto

SELECTforeitherLorR.

ford=10c.

Solet’sprovethis.• Basecase:

• ifn<=50,ouralgorithmwas:runMergeSort on<=50things.

• Bymaybeallowingctobeabitbigger,thattakestimeatmost50d.(WHYISTHISOKAY?)

• Inductivestep:Suppose(*)holdsforallsizes<n.Then

• � � ≤ � ⋅ � + � &B + � ?&

@A + 5≤ � ⋅ � + � ⋅ UB + � ⋅ ?&

@A + 5≤ � � + *

B + ?*@A + 5�

≤ � � + @AVB + ?A⋅V

@A + 50�= 9� + 50 �≤ 10� ⋅ � = � ⋅ � whenevern>50.

∗ � � ≤ Q� ⋅ 50�� ≤ 50� ⋅ �� > 50ford=10c.

Thisisaprettypedantic

proof!Butit’sworthbeing

carefulabouttheconstants

forthesubstitutionmethod,

aswe’llseeinabit.

Thisiswhywe

hadthebasecase

at50before.

Nearlythere!

• Byinduction,thisshowsthattheinductivehypothesis(*)appliesforalln.

• Termination:Observethatthisisexactlywhatwewantedtoshow!

• Thereexistsaconstantd(whichdependsontheconstantcfromtherunningtimeofPARTITION…)andann0 (aka50)sothatforalln>n0, T(n)<=dn.

• Bydefinition,T(n)=O(n).

• Hooray!

• Conclusion:WecanimplementSELECT(andinparticular,MEDIAN)intimeO(n).

∗ � � ≤ Q� ⋅ 50�� ≤ 50� ⋅ �� > 50ford=10c.

Thatwasprettypedantic

• Can’twejustusetheniceO()notation?

• � � ≤ � ⋅ � + � &B + � ?&

@A + 5• Inductivehypothesis:T(n)=O(n).

• Thentheinductivestepisjust

T n ≤ � ⋅ � + � &B + � ?&

@A + 5≤ � ⋅ � + � &

B + � ?&@A + 5 = �(�)

� � ≤ Q� ⋅ 50�� ≤ 50� ⋅ �� > 50vs.

Actuallythatdoesn’twork

• Considerthis“proof”thatMERGESORT runsintimeO(n).(Itdoesn’t).

• � � ≤ 2� &< + � ⋅ �

• Inductivehypothesis:T(n)=O(n).

• Thentheinductivestepisjust

T n ≤ 2� &< + � ⋅ � ≤ 2 ⋅ � &

< + � ⋅ � = � � .• What’swrong???

• (Itturnsoutthebasecaseisfine).

TheproblemisbeingsloppywithO()

• WhenweuseO()intheinductivehypothesis,itmighthaveadifferentconstant“c”inthedefinitionofO()eachtimethehypothesisiscalled.Asaresult,this“constant”mightdependonn,whichisnotallowedinthedefinitionofO().

• Tryrigorously:

• Supposethat� � ≤ � ⋅ � + 2� &<

• Let’sguessthesolutionis � � ≤ Q� ⋅ �A�� ≤ �A� ⋅ �� > �A• Thentheinductiveargumentwouldgo….

• � � = 2� &< + � ⋅ � ≤ 2 ⋅ *&< + � ⋅ � = � + � �

• Weneedthattobesmallerthandn fortheinductiontowork.

• Nowaythat’sgoingtohappen,sincec>0.

• Sothisargumentwon’twork.(Whichisgood,sincethestatementisfalse).

Toldyouso.

Recap

• Wesawa(prettyclever)algorithmtodoSELECT intimeO(n).

• WeprovedthatitworkedusingtheSubstitutionMethod.

• TheMasterTheoremwouldn’thaveworkedforthis.

• Inpracticeforthisalgorithm,it’softenbettertojustchoosethepivot randomly.You’llseeananalysisofthatifyoutakeCS265(randomizedalgorithms).

• We’llalsoseesomerandomizedalgorithms…

…nexttime

lecture 4: selection and median - stanford...

Documents