• Professor Jeanne Ferrante
1
http://www.jflap.org/jflaptmp/
CSE 105 Theory of
Computation
Today’s Agenda • More Undecidability • Time Complexity
– Big-O Asymptotic Analysis – Time Complexity Classes – The Complexity Class P
Reminders and announcements: • HW 7 due Friday May 27 by 11:59 pm • Reading Quiz 9 due Mon May 30 by 11:59 pm • Final Exam Study Guide is out!! Note: Computable Functions (HW 7) • REVIEW SESSION (Nathan Speidel):
– Thurs Jun 2 7 pm – 8:50 pm in Peterson 108 • Final Exam: If you need a LEFT handed seat, make a private post to
instructors on Piazza with your section number and PID ASAP • Final: Sat. Jun 4, 11:30 am – 2:29 pm Room WLH 2001 • CAPE AND TA Evals: OPEN!
2
Review: Reduction • For 2 problems P1 and P2, P1 reduces to P2 if
any solution for P2 can be used to solve P1
3
P1 P2 Yes No
Yes No (May be)
Harder
(May be) Easier
In other words: Using a solution for P2 as a subroutine gives a solution for P1.
ETM is undecidable Proof by contradiction: Assume that ETM is decidable, with TM R. We show that ATM reduces to ETM. But we know that ATM is undecidable, so ETM must be undecidable. • Construct TM MATM and show that it decides ATM: MATM = “ // Use R as a subroutine to get decider for ATM Correctness: MATM is a decider since R is, and accepts <M,w> iff L(X) is nonempty iff M accepts w.
What does R(<M>) do on TM M when L(M) = φ ? A. Accept D. Don’t Know B. Reject C. Fails to halt
ETM is undecidable Proof by contradiction: Assume that ETM is decidable, with TM R. We show that ATM reduces to ETM. But we know that ATM is undecidable, so ETM must be undecidable. • Construct TM MATM and show that it decides ATM: MATM = “ // Use R as a subroutine to get decider for ATM Correctness: MATM is a decider since R is, and accepts <M,w> iff L(X) is nonempty iff M accepts w.
What does R(<M>) do on TM M when L(M) ≠ φ ? A. Accept D. Don’t Know B. Reject C. Fails to halt
ETM is undecidable
Proof: Assume that ETM is decidable, with TM R. We show that ATM reduces to ETM. But we know that ATM is undecidable, so ETM must be undecidable. • Using R, we construct a TM MATM and show that it decides ATM: MATM =
Correctness: MATM is a decider since R is, and accepts <M,w> iff L(X) is nonempty iff M accepts w.
“ On input <M,w>: 1. Construct TM X, X = “On input y
a) If y ≠ w, reject b) If y = w, runs M on w. If M accepts, accept “
// Note that L(X) is nonempty iff M accepts w. 2. Run R on <X>. If R accepts, accept. If R rejects, reject. “
What is L(X) ? A. ATM B. {w} C. w or φ D. {w} or φ E. None of the
above
Only one of these can be recognizable: ETM or ETM . WHY?
Qu: Which one is recognizable? A. ETM
B. ETM
C. Neither of them
We define TM C = “On input <M>: For positive integers i = 1,2, 3.. 1. Run M for i steps on the first i input strings (in
lexicographical order). 2. If M every accepts a string, then ACCEPT. “
Review Theorem 4.22: A language L is decidable iff L is both TR and co-TR
• L Decidable -> TR and co-TR – Easy proof! Just use the decider to recognize L,
so it is TR. Then use the decider again to build a recognizer for L, by flipping the accept/reject result.
• L TR and co-TR -> Decidable – We can use the same run-in-parallel method we
did for ATM and ATM to build a decider for L: • Run recognizer TM’s for L and L in parallel • If the TM for L accepts, accept. If the TM for L
accepts, reject. • One of these must accept, so we can conclude this
is a decider, and accepts ATM . 8
Status: Language Hierarchy
Regular
Context-Free
Decidable
Turing-Recognizable
CF but not Regular: {0 n1n| n > 0} By pumping lemma (Regular) Decidable but not CF: { anbncn | n > 0} By pumping lemma (CF) Turing-Recognizable but Not Decidable: ATM By Diagonalization ETM By Reduction Not Turing Recognizable: ATM TH 4.22 ETM
ATM ATM
ETM ETM
MEASURING TIME COMPLEXITY OF DECIDABLE PROBLEMS BIG-O
10
What is “Theory of Computation”?
11
What is a Computer? A Computation? • We’ve looked at a number of computational
models: • Finite Automata (DFA, NFA), PDA, Turing Machines
Computability: What problems are computers capable of solving?
• By Church-Turing thesis, Computations on TM’s • There are many problems that can be shown
unsolvable (undecidable) Complexity: How easy or hard is a given problem
to solve? Are some problems inherently harder than others?
• What resources are needed to solve a problem?
Decidability vs. Complexity • Decidability gives the answer (eventually)
but gives no idea about how long it takes to compute the answer
• Complexity measures the resources (time, space, power…) needed to get the answer
• We will concentrate on TIME(n) as a function of length of input string, n
Decidability What can conceptually be solved, given unlimited resources?
Complexity What can practically be solved, given limited resources?
12
Example: Travelling Salesperson Problem Given: Collection {C1,…Cn} of cities, Distance k, Distance Matrix D(i,j): distance between Ci and Cj Problem: Is there a tour of all cities with total distance < k? • Good news: Its solvable! How?
Complexity of enumerating all tours to check total distance < k:
With n cities, there are (n-1)! choices of tours E. g., n = 50, 49! ≈ 1045
Not practically computable to enumerate all tours! • But complexity of checking if a given tour has
distance less than k is easy!
13
Time Complexity Def. Let M be a (deterministic) TM that always halts. The running time or time complexity of M is the function f: N R+ where f(n) is the maximum number of steps of the TM M on any input of length n. • We say that “M runs in time f(n)”
This is worst-case analysis •A different approach: average-case
Because it may be difficult to figure out exact number of steps, we will estimate it
14
Big-O Notation • To estimate the asymptotic running time, ignore:
– Constant coefficients – Lower order terms
Def. Let f and g be functions from N to R+. f(n) = O(g(n)) if there are positive integers c and n0 such that, for every n ≥ n0,
f(n) ≤ c g(n). • Example: f(n) = 2 n3 + 17 n2 + 3n + 10 Claim f(n) = O(n3 ): Let c = 2 + 17 + 3 + 10= 32, and n0 = 2. Then f(n) = 2 n3 + 17 n2 + 3n + 10 ≤ 32 n3 for n ≥ 2
, ,
IGNORE IGNORE
15
Big-O Questions Recall f(n) = O(g(n)) if there are positive integers c and n0 such that for every n ≥ n0,
f(n) ≤ c g(n).
Is f(n) = 2 n3 + 17 n2 + 30n + 1000 = O(n2 )? A. No B. Yes C. Maybe
Is f(n) = 2 n3 + 17 n2 + 30n + 1000 = O(n4 )? A. No B. Yes C. Maybe
16
More on Big-O • When f(n) = O(g(n)) we say g(n) is an
(asymptotic) upper bound for f(n). • If f(n) = logb (n)
– changing the value of base b corresponds to multiplication by a constant
– We can ignore the base b and write f(n) = log(n).
• We can add up Big-O terms: – f(n) = O(n2 ) + O(n) + O(log(n)) = O(n2 ) WHY?
• O(nc ) for constant c is a polynomial bound • O(2nc ) for c > 0 is an exponential bound
17
Example: TM M accepts {0k1k | k ≥ 0} M = “On input w:
1. Scan across the tape and reject if w not of form 0*1*
2. Repeat the following while both 0’s and 1’s: a) Scan across tape, crossing off single 0 and single 1
3. If 0’s remain after all 1’s checked off, reject. If 1’s remain after all 0’s checked off, reject. Otherwise, accept.”
18
Example: M accepts {0k1k | k ≥ 0} M = “On input w:
1. Scan across the tape and reject if w not of form 0*1*
2. Repeat the following while both 0’s and 1’s:
a) Scan across tape, crossing off single 0 and single 1
3. If 0’s remain after all 1’s checked off, reject. If 1’s remain after all 0’s checked off, reject. Otherwise, accept.”
Total is O(n) + O(n2 ) + O(n) = O(n2 )
2n steps
O(n) steps
O(n) steps
n/2 scans, each with
19
TIME COMPLEXITY CLASSES OF DECIDABLE PROBLEMS
Have we got the time?
20
Time Complexity Classes Def. Let t: N R+ be a function.
The time complexity class TIME(t(n)) is the collection of languages L that are decidable by a O(t(n)) TM. {0k1k | k ≥ 0} is in TIME(n2 ) because there is a TM M that decides it in O(n2 ) Can we do better? Would have to find a TM algorithm that is O(n) or O(n log n)…
21
A Different TM M’ decides {0k1k | k ≥ 0} M’ = “On input w:
1. Scan across the tape and reject if w not of form 0*1* O(n) steps
2. Repeat the following while both 0’s and 1’s: a) Scan across tape, checking whether total number
of 0’s and 1’s remaining is even or odd. If odd, reject //Must be even if same number of 0’s and 1’s O(n) steps
b) Scan across tape, crossing off every other 0 starting with first 0, and then crossing off every other 1, starting with first 1 O(n) steps
3. If no 0’s and no 1’s remain on the tape, accept. Otherwise, reject.” O(n) steps
How often is the loop at 2 repeated? A. O(n) B. O(log(n)) C. Neither of these
22
TM M’’ with 2 tapes decides {0k1k | k ≥ 0} M’’ = “On input w:
1. Scan across the tape 1 and reject if w not of form 0*1*
2. Scan across the 0’s on tape 1 until first 1. At same time, copy the 0’s on tape 2.
3. Scan across the 1’s on tape 1 until end of input. For each 1 read on tape 1, cross of 0 on tape 2. If all 0’s crossed off before all 1’s read, reject.
4. If all 0’s now crossed off, accept. If any 0’s remain, reject.”
What is the running time of M’’? A. O(n) B. O(n log(n)) C. O(n2 )
23
Decidability vs. Time Complexity Decidability • M always halts but
don’t know how long to halt
• Does not depend on variant of TM model: – Number of tapes – Nondeterminism
Time Complexity T(n) • M will halt within time
T(n) on all inputs of length n
• Depends on variant of TM model: – t(n) multitape TM
equivalent to O(t 2(n) ) single tape – t(n) nondeterministic
single tape equivalent to
2O(t(n)) deterministic single tape
24
Running times of decider TM’s
Deterministic q0
Nondeterministic q0
f(n)= MAX number of steps with input length n
f(n) = MAX number of steps on any branch with input length n
qrej
qacc
qrej
qrej qacc
25
CLASS P (DETERMINISTIC POLYNOMIAL TIME)
Most useful!
26
P = U DTIME(nk ) • P is the class of languages that can be
decided in polynomial time on a deterministic, single-tape TM
• Why do we study this particular class? – There is a dramatic difference in growth rate
between polynomial and exponential functions • n = 1000, n3 = 1 billion, 2n > no. of atoms in universe
– It contains the “feasibly” solvable problems • “Cobham’s Thesis”
– P the same for all models that are polynomially equivalent to deterministic, 1-tape TM’s
• t(n) multitape equivalent to O(t 2(n) ) single tape TM
27
k
Encodings Matter in Class P • We’ll assume polynomial time encodings
– Ex. Multiplication of two numbers • Matters how we encode the input and answer:
– Unary encoding of n exponentially larger than binary encoding of n
• Graph encoding choices – Encode nodes and edges, n nodes, ≤ n2
edges – Use n by n adjacency matrix to determine if
edge from node i to node j – Analyze graph problems in number n of
nodes; still polynomial in size of graph
28
Analyzing Problems in Class P • We’ll use high-level algorithm descriptions • We’ll describe the algorithm in stages
– We’ll need to show that there are only a polynomial number of stages for input of size n
– We’ll need to show that each stage has only polynomial number of steps for input of size n
• We can’t use nondeterminism or guessing • The algorithm needs to be implementable
on a deterministic TM in polynomial time
29
Example: PATH is in P PATH = {<G,s,t> | G is a directed graph with n nodes, and there is a directed path from node s to node t} • Enumerating all paths in G is exponential in n • Better?
Constant time
Constant time
O(n2 )
O(n)
M = “On input <G,s,t>: 1. Mark node s 2. Repeat until there are no unmarked nodes:
a) Scan all edges of G, and if edge (a,b) is found with a marked and b unmarked, mark b.
3. If t is marked, accept; otherwise, reject. “
30
More Examples in Class P • RELPRIME = { <x,y> | x and y are relatively prime}
– Enumerating through all possible common divisors would yield exponential time!
– Uses Euclidean Algorithm to see if GCD(x,y) = 1 • GCD(x,y) is the largest integer that divides both x and y • Ex. GCD (10, 6) = 2
• Every CFL in P: {w | w is generated by CFG G} in P – Every CFL is decidable, but
• Enumerating all derivations of w where |w|=n, is exponential at best
– Use Dynamic Programming to show in P • To be in P, must usually find way to avoid brute-
force searches!
Qu: Is sorting n numbers in P?
A. Yes B. No C. Depends on the method used