Computational Complexity

Dr. Colin CampbellCourse: EMAT20531

Order Notation for Functions

The order notation is used to give a bound on the limiting behaviour of a function.

We write f(x) is O(g(x)) iff

xgmxfmx that such 00

Order example f=100x3+50x2+10

x+100 g=x3

f-mg

m=110

m=120

m=130

m=140

m=150

The big O Notation for Algorithms

We can represent the complexity of an algorithm A by a function

fA(n)=number of steps (or time) it takes for A to complete given an input of size n.

The size of input may be measured in a number of different ways.

Big O NotationA=Do i=1,..,nPrint “hello world”ClearRepeatPrint “End” f(n)=2n+1 O(f(n))=n A is linear in n

Big O Notation: 2

A=Do i=1,…,nDo j=1,…,nPrint “Hello World”ClearRepeatRepeatPrint “End” f(n)=2n2+1 O(f(n) is n2

The Quicksort Algorithm

Input: a list L=(a1,…,an) of numbers Choose an number p at random from L Determine L1 all numbers ai less than p

and L2 all numbers ai greater than or equal to p

Recursively sort L1 and L2

Solution=concatenate L1 and L2

Quicksort Example

L=(1,5,2,7,6,4) sort p=4 [(1,2)(4,5,6,7)=(1,2,4,5,6,7)]L1=(1,2) L2=(5,7,6,4) sort L=(5,7,6,4) p=6 [(4,5)(6,7)=(4,5,6,7)] L1=(5,4), L2=(7,6) sort L=(5,4) [(4,5)] sort L=(7,6) [(6,7)]

Sort the list (1,5,2,7,6,4)

Quicksort: The Worst Case

Given p partitioning into L1 and L2 is O(n) The worst possible case would be that

every time we partition the list we have L1 length n-1 and L2 length 1

For the i’th partition we have a computation O(n-i)

Now (n-i)=n2. Hence the entire computation is O(n2).

Quicksort: On Average On average the split between L1

and L2 will result in two lists of n/2. This is based on the assumption that

the elements of the list are selected from an interval [x,y] according to a uniform distribution

Consider P(pai), the expected number of elements in L2 is nP(pai)

Quicksort: On Average (2) p is a uniformly distributed random

variable taking values in [x,y] Therefore

xy

xaapP ii

ai is also a uniformly distributed random variable from [x,y] with expected value (x+y)/2.

Therefore, the expected value of P(pai) is ½ and the expected number of elements in L2 is n/2

Quicksort: On Average (3)

Consider, the number of such partitions before we end up with strings of length 1. i.e. find k for which n/2k=1

Log(n/2k)=0 log(n)=log(2k)=k Hence, there are on average log(n)

partitions each requiring O(n) computations

This gives O(nlog(n))

Polynomial Time

An algorithm runs in polynomial time if fA(n) is O(nk) for some positive integer k.

E.g. Quicksort always runs in polynomial time even in the worst case when it is O(n2)

The class P refers to those algorithms which run in polynomial time on a Turing Machine as described in the earlier section.

Non-deterministic Turing Machines

A Non-deterministic Turing machine is a Turing machine where for some state qi and symbol S there is more than one instruction beginning qiS.

q1

q2

q3

1:R

0:R

Computation Paths A deterministic Turing machine follows

a computation path. A Non-Deterministic generates a tree of

computations. We can either think of a NDTM as

following all paths simultaneously until it finds an accepting state

Or being a very lucky guesser between when choosing between states

Accepting States and Paths

For NDTM we usually think of binary yes no responses from a computation.

We define a set of accepting state AQ. An accepting path is a path which

terminates in an accepting state. A NDTM accepts an input x if there is

an accepting path for that input.

NDTM Example Let TMP be a deterministic Turing machine

that given a string of 1s and 0s terminates in state q+ if there are an even number of 1 and q- otherwise. AQ={q+}

Further suppose that the start state of TMP is q3

Given a string of 1s we mean substrings to be a string of the same length where some or all of the 1s have been switched to 0.

Eg 111 has substrings 111,110,101,100, 011,010,001,000

NDTM Example: 2

q1 q2

q3

0:R

1:0

1:RTMP

1111

111

10211

011

11111

102

1

11131103 1011

Several deterministic steps1013

1003

0111

0011

0113

0103

0013

0003

q- q+

q+ q- q+ q- q- q+

Non-Deterministic Polynomial

Intuitively, we see that a NDTM is more efficient than a DTM since it can explore many computational paths simultaneously.

NP refer to the class of algorithms which run in polynomial time on a NDTM.

Clearly PNP but does P=NP?

NP Complete

NP-Complete are a collection of decision problems for which there is an algorithm in NP but no known algorithm in P.

Formally, a problem in NP is NP-Complete if any other problem in NP can be reduced to it.

Problem C is reducible to NP-Complete problem D if there is an algorithm A which solves C with a subroutine A+ that solves D, such if A+ ran in polynomial time so would A.

SAT: An NP Complete Problem

SAT: Given a sentence of Propositional Logic S in CNF, is there an allocation of truth values for which S is true?

There is no known algorithm in P which solves SAT

This is basically because all (known) approaches require us (in the worst case) to check all possible truth value allocations. If S has n propositional variables there are 2n possible allocations of truth values.

A NDTP algorithm for SAT Simplify things by assuming that S is in

CNF and each clause has exactly k literals for k>2. (k-SAT)

Notice that for any given allocation of truth values we can check if satisfies S in O(nk) – n is the number of variables in S.

There are 2n(2n-1)..(2n-k+1)/3! Different clauses.

Assume we can check if each is satisfied in 1 step then the computation is O(nk)

SAT Algorithm: 2

Algorithm• For each variable guess either t or f. [O(n) on NDTM]• Check if allocation of truth values satisfies S [O(nk)]

S=(PR Q)(PRQ)

Pt f

Q Qt f t f

R R R Rt t t tf f f f

acc

ept

acc

ept

reje

ct

acc

ept

acc

ept

reje

ct

acc

ept

acc

ept

Proving NP-Completeness

Usually a problem D is shown to be NP-complete by proving that a known NP-complete problem C reduces to it.All NP problems

reduce

NP complete C

D

All NP problems

reduce

Dreduce

CLIQUE

In graph theory a CLIQUE is a set of vertices (nodes) where each is connected to the others.

6

4

3

5

2

1

5

2

1

Clique of size 3

The CLIQUE Problem

For a given graph decide whether or not there is a clique of size k.

This has application in chip design where for example nodes are wires on a chip and edges signify that two wires can overlap.

CLIQUE tells the designer something about how much space is required on the chip.

We shall show that SAT reduces to CLIQUE So CLIQUE is NP-Complete.

CLIQUE and SAT

Let S be a sentence in CNF with k clauses. S=C1C2…Ck We construct a graph G where Vertices={Li:LiCi} i.e. every literal from

every clause Edges={<Li,Lj> LiLj and ij} i.e. edges

between literals from different clauses provides they do not conflict.

S is satisfiable if and only if G has a clique of size k.

CLIQUE and SAT: 2

Example S=(PQ)(PQ)(PR) L1=P, L2=Q, L3=P, L4=Q, L5=P, L6=R

L2

L5

L3

L6

L4

L1Graph G

CLIQUE and SAT: 3 (If) Suppose there is a clique of size k then there

are k non-conflicting literals, one from each clause in S.

Since they are non-conflicting there is a truth allocation that makes each true and hence each clause in S true.

(Only If) Suppose there is an allocation of truth values for which S is true.

At least one literal from each clause must be true

These cannot be conflicting, hence they must form a clique of size k in G

CLIQUE and SAT: 4

L2

L5

L3

L6

L4

L1Graph G

L2

L5L4

L2L6

L4

L6

L4

L1

L5L4

L1

L2

L3

L6

P=t,Q=t,R={t,f}

P=t,Q=t,R=t

P=t,Q=t,R={t,f}P={t,f},Q=t,R=tP=f,Q=t,R=t

Other NP-Complete Problems: The Knapsack Problem

500g100

Weight: wi

Utility: ui

1kg200

300g400

750g80

Maximize 1i

iiux

Quantity: xi

subject to Cwx ii

i

Decision problem: is there a solution for which Vuxi

ii 1and Cwx i

ii ?

Other NP-Complete Problems: The Travelling Salesman Problem

A B

C D

E

F

10

8 718

12

16 25

4

9

Starting at A find the shortest route which visits all towns and then returns to A.