Applied Computer Science IIChapter 7: Time Complexity

Prof. Dr. Luc De Raedt

Institut für InformatikAlbert-Ludwigs Universität Freiburg

Germany

Overview

• Measuring complexity• The class P• The class NP• NP-completeness• Additional NP-complete problems

Measuring complexity

• How much time needed to decide a language

• Use Machine M1•

{0 1 | 0}k kA k

Measuring complexity

• Worst-case versus average-case

• We typically use big-O and small-o notation (cf. ACS I); asymptotic analysis

The time complexity of a TM is the function

: , where ( ) is the max number of steps

that uses on any input of length .

M

f f n

M n

Big-O

0

0

Let , : be two functions. We say

that if positive integers and exist

so that for every integer holds that

( )

(

is ( ( ))

( ( )) is an upper bound for ( )

) . ( )

f g R

c n

n n f n

f n O g n

c g n

O g n f n

3 2

30

4 2

( ) 5 2 22 5

= ( ) choose 6 and 10

is also ( ) but not ( )

f n n n n

f O n c n

f O n O n

2 2 2( ) 3 .log 5 .log log 2

( ) ( .log )

f n n n n n

f n O n n

pb

b b b

b b b

p p q (pq)b b

b a b

log x = p if and only if b = x (definition)

log x*y = log x + log y

log x/y = log x - log y

log x = p log x which implies that (x ) = x

log x = log x * log a

Small-o

0

Let , : be two functions. We say

that if

( )lim 0, i.e. for any real number 0,

( )

there is a number where ( ) . ( ) for

( ) is o( ( ))

all

n

o

f g R

f nc

g n

n

f n g

f n c g n n n

n

Time Complexity

k 2

Let : be a function. Define the

( ( )) ={ | is a language decided by an ( ( )) time Turing machine}

So, {0 1 | 0} ( )

Can we find a machine that decides as

time complexity c

symptotic

lass

k

t

TIME t n L L O t n

k TIME n

A

2

ally more quickly ?

i.e. is ( ( )) for ( ) ( ) ?A TIME t n t n o n

Analyzing algorithms

Analyzing algorithms

• 2-tape machine

Complexity relationship among models

2

7.8

Let ( ) be a function where ( ) .

Then every ( ) time multitape Turing machine has

an equivalent ( ( ) ) time single-tape Turing machine.

t n t n n

t n

O t n

Theorem

2

elements

Let be a tape TM that runs in ( )

Construct (single-tape TM) that runs in ( ( ) ) time

Analyze simulation

To simulate each step of ,

makes two scans over active input tape

M k t n

S O t n

M

S

Proof

each of the active parts of the tapes are of length at most ( )

i.e. . ( ) ( ( ))

and possibly shifts (for inserting a new cell)

shifts take ( ( )) time

First phase

k t n

k t n O t n

k

O t n

2

2 2

: copying inputs from to

takes ( ) steps

Second phase : simulate the ( ) steps of

takes ( ). ( ( )) ( ( ) ) steps

( ) ( ( ) ) ( ( ) ) when ( )

M S

O n

t n M

t n O t n O t n

O n O t n O t n t n n

• Definition 7.9 including figure 7.1

Let be an NTM that is a decider. The running

time of is the function : , where ( ) is the

maximum number of steps that uses on any branch of its

computation of any input of length as

N

N f f n

N

n

Definition

shown below

( ( ))

7.10

Let ( ) be a function where ( ) .

Then every ( ) time nondeterministic Turing machine has

an equivalent 2 time deterministic single-tape Turing machine.O t n

t n t n n

t n

Theorem

2 ( )

elements

Every branch of has length at most ( )for input of length

Every node in tree has at most children (determined by )

Exploration tree is breadth first :

1+ ... nodes and let n

N t n n

b

b b b

Proof

( )

( ) ( ( ))

aves

= ( )

Time for processing one node (or leaf) starting from root

( ( ))

Running time of deterministic 3-tape TM

( ( )). ( ) 2

Use transformation to deterministic single tape

t n

t n O t n

O b

O t n

O t n O b

( ( )) 2 ( ( )) gives (2 ) 2O t n O t n

Differences in models

• Polynomial difference between single- and multiple tape deterministic TM

• Exponential difference between deterministic and non-deterministic TM

• Compare exponential versus polynomial behaviour !

Towards the class P

Distinction between :• Exponential algorithms

– Brute force search– Impractical

• Polynomial– Practical – Robust for reasonable models of computation

(I.e. polynomially equivalent)

Here, we ignore differences between,e.g. , level of abstraction5( ) and ( )O n O n

• P is invariant for all models of computation that are polynomially equivalent to a single tape deterministic TM

• P roughly corresponds to the class of problems that are realistically solvable on a computer

The class P

P is the class of languages that are decidable in polynomial time

on a deterministic single -tape TM, i.e.

P= ( )k

k

TIME n

Definition

Conventions

• High-level descriptions of algorithms• Abstraction of specific model used (as

long as polynomially equivalent)• Algorithms proceed in numbered stages• Analysis in two steps :

– A polynomial bound on the number of stages– Each stage can be implemented in

polynomial time on deterministic single tape TM (or poly. Equivalent model)

Conventions

• Encoding method for problems• We use <.> notation as before• Reasonable methods require

encoding and decoding of objects into other (internal ?) representation to require polynomial time.

• E.g. 111111111 (unreasonable) instead of 9 (or other base k-notation)

Encoding Graphs

• Option 1 : – List of nodes– List of edges

• Option 2 :– Adjacency matrix

1 iff there is an edge from node to node Entry ( , )

0 otherwise

i ji j

• Figure 7.2

{ , , | is a directed graph that has a directed path from to }PATH G s t G s t

• Insert algorithm

PPATH Theorem

• Insert algorithm

{ , | and are relative prime}

i.e. the greatest common divisor of and is 1

P

RELPRIME x y x y

x y

RELPRIME

Theorem

Every context-free language is a member of P

Theorem

Assume language in Chomsky Normal Form

Then each derivation of a string has 2 1steps

(Generating all

Dynamic programming

derivations is polynomial!)

accumulate information about smaller problem

:

s

w n

not

keep track of table of size

( , ) the set of variables generating ...

with

i j

n n

table i j w w

i j

Towards the class NP{ , , | is a directed graph with a hamiltonian path from to }

i.e. it is directed and contains all nodes in

HAMPATH G s t G s g

G

HAMPATH

• Easy to obtain exponential time algorithm :– Generate all paths of length m (no. of nodes)

and check whether they are hamiltonian

• Polynomial verifiability :– Checking whether a given path is hamiltonian

can be done in polynomial time

• Verifying existence of hamiltonian path is easier than determining the existence

• Consider also • Similar for COMPOSITES

{ | . for integers , 1}COMPOSITES x x p q p q

HAMPATH

The class NP

A verifier for a language is an algorithm s.t.

{ | accepts , for some string }

: certificate, proof

Time of verifiers is measured in terms of size of

(implies that is of size polyn

A V

A w V w c c

c

w

c

Definition

omial in size )

Polynomial time verifiers

is the class of languages that have

NP comes from Nondeterministic polyno

polynomial time

mial

verifiers

time

w

Definition

NP

NTM for HAMPATH

• Algorithm

• Proof

A language is in NP iff it is decided by some nondeterministic

polynomial time algorithm

Theorem

idea

NTIME( ( )) { | is a language decided by an ( ( )) time NTM}

NP= ( )k

k

t n L L O t n

NTIME n

Definition

Corollary

{ , | is an undirected graph with a -clique}CLIQUE G k G k

CLIQUE is in NP

Theorem

1 1 1{ , | { ,..., } and for some { ,..., } { ,..., }we have }

The and are multi-sets.

is in NP

k l k ii

SUBSET SUM S t S x x y y x x y t

y x

SUBSET SUM

Theorem

P versus NP

P versus NP

• P= the class of languages where membership can be decided quickly

• NP=the class of languages where membership can be verified quickly

• NP-complete problems : if a polynomial time (determ.) algorithm exists for one NP-complete problem, then P=NP

NP EXPTIME= (2 )kn

k

TIME

true is denoted by 1

false is denoted by 0

and

or

not

0 0 0 0 0 0 0 0 1

0 1 0 0 1 1 1 1 0

1 0 0 1 0 1

1 1 1 1 1 1

A Boolean formula is an expression involving Boolean variables and operations

E.g.

Boolean Logic

=( ) ( )

Assignment of values (0 and 1) to variables

For each assignment one can evalue a formula

If the formula evaluates to 1, we say that the assignment satisfies the formula.

T satisfiabilie yh t

x y x z

problem is to test whether a formula is satisfiable

(i.e. whether there exists an assignment that satisfies the formula)

{ | is a satisfiable boolean formula}

(Cook-Levin)

P iff P=NP

SAT

SAT

Theorem

Polynomial time reducibility

• Fig 7.6

* * A function : is a

if some polynomial time TM exists that halts with

just

polyno

( ) on its tape, when started on any inp

mial time computable func

ut

n

tiof

M

f w w

* *

Language is to language

written , if a polynomial time computable function :

exists where for every

polynomial time (mapping) reducible

polynomial time red

: iff ( )

is a uction

P

A B

A B f

w w A f w B

f

• Proof: N works in polynomial time

If and P, then PPA B B A Theorem

3 { | is a satisfiable 3cnf formula}

cnf : conjunctive normal form, i.e. a conjunction of disjunctions

(disjunctions are clauses, elements of disjunctions are literals)

3cnf : each clause/di

SAT

1 1 1

sjunction has exactly three literals

e.g. ( ) ( )

3 is polynomial time reducible to

Let be of the form ( ) ... ( )

( ) , where

all nodes in are org

k k k

x y z y x w

SAT CLIQUE

a b c a b c

f G k

G

Theorem

Proof

1anised in triples ,...,

each triple corresponds to a cluase and has 3 nodes

The edges of connect all nodes except for :

1. edges between the nodes of the same triple

2. nodes with contradict

kt t

G

ory label, i.e. and

Now prove that is a polynomial time reduction

x x

f

• Fig 7.7

NP-completeness

A language is NP-complete if

1. is in NP, and

2. every in NP is polynomial time reducible to

If is NP-complete and , then P=NP

If is NP-complete and for in NP, P

B

B

A B

B B P

B B C C

Definition

Theorem

Theorem

then is NP-completeC

Cook-Levin

is NP-complete

1. is in NP (easy)

2. any language in NP is polynomial time reducible to

Let be a NTM that decides in time

Define an (accepting) tableau for on

k

SAT

SAT

A SAT

N A n

N w

Theorem

Proof

, ,

define reduction from to

on input , define ( )

let us first define the boolean variables.

Let {#}

for 1 , , for :

define ( , ) contains symbol

Now define

k

i j s

cell start mov

f A SAT

w f w

C Q

i j n s C

x cell i j s

0 1

, , , , , ,, ,1 ,

1,1,# 1,2, 1,3, 1, 2, 1, 3, 1, 1, 1, ,#

( )

"each cell contains exactly one symbol"

... ...

" the first row correspo

k

k kn

e accept

cell i j s i j s i j ts C s t C s ti j n

start q w n w n n n

x x x

x x x x x x x

nds to the starting situation"

, ,1 ,

1 1 1 2 2

"at least one of the configurations is an accept state"

concept of legal windows

e.g. ( , ) {( , , )} and ( , ) {( , , ), ( , , )}

if the top row of the table is t

k acceptaccept i j qi j n

x

q a q b R q b q c L q a R

Claim he start configuration

and if every window (2x3) is legal then each row legally follows the preceding one.

1 2 3 4 5 61 6

1 ,1

, 1, , , , 1, 1, 1, 1, , 1, 1,,..., is a legal window

2

"the , window is legal"

for each window :

( )

3.Analyze the complexity

tableau has ce

k kmovei n j n

i j a i j a i j a i j a i j a i j aa a

k k k

i j

x x x x x x

n n n

2

lls

each cell has variables (for symbols in )

thus number of variables is ( )

Now investigate different components of

(All ( ) or ( ); indices can be encoded in (log

k

cell start move accept

k k

l C

O n

O n O n O n

)

( ) is of polynomial size and can be computed in polynomial timef w

3 is NP-complete

modify construction in proof

first rewrite in cnf form.

second rewrite clauses such that they have 3 literals

1. if less than 3 literals e.g. rewrite to e.g.

SAT

SAT

a b a b

Theorem

Proof

1

1 2 1

1 3 2

3 1

2. if more than 3 literals ( ... )

rewrite into

( )

( )

...

( )

l

l l l

b

a a

a a z

z a z

z a a

is NP-completeCLIQUE

Corollary

{ , | is an undirected graph that has a -node vertex cover}

Let be an undirected graph ( , ) and a number

A is a set of nodes ' such ve thrtex at

for all edges

cover

( , ) holds that

VERTEX COVER G k G k

G V E k

V V

u v E

'or '

is NP-complete

produce clause gadget and variable gadget

set 2 where has variables and clauses

then prove satisfiability if and only if graph has a vertex cover

u V v V

VERTEX COVER

k m l m l

Theorem

Proof

with nodesk

1 1 1

is NP-complete

we show that 3

Let be of the form ( ) ... ( )P

k k k

HAMPATH

SAT HAMPATH

a b c a b c

Theorem

Proof

{ | is an undirected graph that contains a hamiltonian path}

is NP-complete

( ) '

for all nodes in , introduce nodes , , in '

for and in

P

in mid out

UHAMPATH G G

UHAMPATH

HAMPATH UHAMPATH

f G G

u G u u u G

s t

Theorem

Proof

1

1 1

, introduce only ,

introduce edges between and and between and

connect also to if there is an edge from to

each path , ,..., , in has a counterpart

, ,

out in

in mid mid out

out in

k

in miout

G s t

u u u u

u v u v

s u u t G

s u u 1, ,..., and vice versa

d out inu t

1 1 1{ , | { ,..., } and for some { ,..., } { ,..., }we have }

The and are multi-sets.

is NP-complete.

1. NP

2.3

Let be in 3cnf and h

k l k ii

P

SUBSET SUM S t S x x y y x x y t

y x

SUBSET SUM

SUBSET SUM

SAT SUBSET SUM

Theorem

Proof

1 1ave vars ,..., and clauses ,...,l kx x c c

Set Covering

1

1

j

SetCover={< ,..., , , | (finite set) and there exist

sets ,..., such that }

Example

<{a,b,c},{b,c,d},{c.d},{a,b,c,d},2> SetCover

<{a,b,c},{b,c,d},{c.d},{a,b,c,d},1> SetCover

n j

k i

i i i

T T M n T M n

T T T M

BinPacking

1BinPacking={< ,..., , , | (all in ) and the objects

be divided over the bins of size such that no bin overflows}

Example

<4,5,7,2,2,9> BinPacking

because 4+5 and 7+2 are both smaller than

n i ia a k b a b a

k b

9

2 bins of size 9

<4,5,7,2,2,8> BinPacking

Traveling Salesman Problem

TSP={< , | contains distance between city and

and there exists a path through all cities of size less than }

0 5 5

5 0 5

5 5 0

10

1,2,3 is a solution

for 9 instead of 10, no solution

ijM k M i j

k

path

Coloring

Coloring={< , | is a graph and there

exists a coloring with colors so

that no two adjacent have the same color}

G k G

k

Key Reductions

SAT

3SAT

SetCover Clique SubsetSum Hampath Coloring

VertexCover Partition

BinPacking

UHampath

TSP

Conclusions

• The class P• The class NP• NP-completeness(e.g. 3SAT)• Polynomial time reductions• Two further classes :

– coNP : – NP-hard:

A language is NP-hard if

every in NP is polynomial time reducible to

B

A B

coNP={ | NP}A A

Standard Reference

• Garey and Johson, Computers and Intractability – A guide to the theory of NP-completeness, W.H.Freeman, 1979.