the strange link between incompressibility and complexity

Eric AllenderRutgers University

The Strange Link between Incompressibility and

Complexity

The Strange Link between Incompressibility and

Complexity

China Theory Week, Aarhus

August 13, 2012

Eric Allender: The Strange Link between Incompressibility and Complexity < 2 >

Today’s Goal:Today’s Goal:

To present new developments in a line of research dating back to 2002, presenting some unexpected connections between

– Kolmogorov Complexity (the theory of randomness), and

– Computational Complexity Theory

Which ought to have nothing to do with each other!


Complexity ClassesComplexity Classes

P

NP

BPP

PSPACE

NEXP

EXPSPACE

P/poly


A Jewel of DerandomizationA Jewel of Derandomization

[Impagliazzo, Wigderson, 1997]: If there is a problem computable in time 2n that requires circuits of size 2εn, then P = BPP.


RandomnessRandomness

Which of these sequences did I obtain by flipping a coin?

0101010101010101010101010101010 0110100111111100111000100101100 1101010001011100111111011001010 1/π =0.3183098861837906715377675267450

Each of these sequences is equally likely, in the sense of

probability theory – which does not provide us with a way to

talk about “randomness”.


RandomnessRandomness

Which of these sequences did I obtain by flipping a coin?

0101010101010101010101010101010 0110100111111100111000100101100 1101010001011100111111011001010 1/π =0.3183098861837906715377675267450

How many bits of information does each of these sequences

contain?


Kolmogorov ComplexityKolmogorov Complexity

C(x) = min{|d| : U(d) = x}

– U is a “universal” Turing machine K(x) = min{|d| : U(d) = x}

– U is a “universal” prefix-free Turing machine Important property

– Invariance: The choice of the universal Turing machine U is unimportant (up to an additive constant).

x is random if C(x) ≥ |x|.


Kolmogorov ComplexityKolmogorov Complexity

C(x) = min{|d| : U(d) = x}

– U is a “universal” Turing machine K(x) = min{|d| : U(d) = x}

– U is a “universal” prefix-free Turing machine Important property

– Invariance: The choice of the universal Turing machine U is unimportant (up to an additive constant).

x is random if C(x) ≥ |x|, or K(x) ≥ |x|.


K, C, and RandomnessK, C, and Randomness

K(x) and C(x) are “close”:

– C(x) ≤ K(x) ≤ C(x) + 2 log |x| Two notions of randomness:

– RC = {x : C(x) ≥ |x|}

– RK = {x : K(x) ≥ |x|} …actually, infinitely many notions of

randomness:

– RCU = {x : CU(x) ≥ |x|}



K(x) and C(x) are “close”:

– C(x) ≤ K(x) ≤ C(x) + 2 log |x| Two notions of randomness:

– RC = {x : C(x) ≥ |x|}

– RK = {x : K(x) ≥ |x|} …actually, infinitely many notions of

randomness:

– RCU = {x : CU(x) ≥ |x|}, RKU

= {x : KU(x) ≥ |x|}



When it makes no difference, we’ll write “R” instead of RC or RK.

Basic facts:

– R is undecidable

– …but it is not “easy” to use it as an oracle.

– R is not NP-hard under poly-time ≤m reductions, unless P=NP.

– Things get more interesting when we consider more powerful types of reducibility.


Three Bizarre InclusionsThree Bizarre Inclusions

NEXP is contained in NPR. [ABK06] PSPACE is contained in PR. [ABKMR06] BPP is contained in {A : A is poly-time ≤tt R}.

[BFKL10]

– A ≤tt reduction is a “non-adaptive” reduction.

– On input x, a list of queries is formulated before receiving any answer from the oracle.



NEXP is contained in NPR. [ABK06] PSPACE is contained in PR. [ABKMR06] BPP is contained in Ptt

R. [BFKL10]

“Bizarre”, because a non-computable “upper bound” is presented on complexity classes!

We have been unable to squeeze larger complexity classes inside. Are these containments optimal?



NEXP is contained in NPR. [ABK06] PSPACE is contained in PR. [ABKMR06] BPP is contained in Ptt

R. [BFKL10]

“Bizarre”, because a non-computable “upper bound” is presented on complexity classes!

If we restrict attention to RK, then we can do better…



NEXP is contained in NPRK.

– The decidable sets that are in NPRK for every U are in EXPSPACE. [AFG11]

PSPACE is contained in PRK.

BPP is contained in PttRK.

– The decidable sets that are in PttRK for every

U are in PSPACE. [AFG11]



NEXP is contained in NPRK (for every U).


PSPACE is contained in PRK (for every U).

BPP is contained in PttRK (for every U).

– The decidable sets that are in PttRK for

every U are in PSPACE. [AFG11]

– [CELM] The sets that are in PttRK for every U

are decidable.







– The sets that are in PttRK for every U are in

PSPACE. [AFG11]




– The sets that are in NPRK for every U are in EXPSPACE. [AFG11]




PSPACE. [AFG11]





Conjecture: This should hold for RC, too.



PSPACE. [AFG11]





This holds even for sets in EXPttRK for all U!



PSPACE. [AFG11]





Conjecture: This class is exactly NEXP.



PSPACE. [AFG11]

Conjecture: This class is exactly BPP.





Conjecture: This class is exactly NEXP.



PSPACE. [AFG11]

Conjecture: This class is exactly BPP P.


K-Complexity and BPP vs PK-Complexity and BPP vs P



PSPACE.

Conjecture: This class is exactly P. Some support for this conjecture [ABK06]:– The decidable sets that are in Pdtt

RK for every U are in P.

– The decidable sets that are in Pparity-ttRK for

every U are in P.





PSPACE.

Conjecture: This class is exactly P. New results support a weaker conjecture: Conjecture: This class is contained in

PSPACE ∩ P/poly. More strongly: Every decidable set in Ptt

R is in P/poly.





PSPACE.

Conjecture: This class is exactly P. New results support a weaker conjecture : Conjecture: This class is contained in

PSPACE ∩ P/poly. More strongly: Every decidable set in Ptt

R is in P/poly (i.e., for every U, and for both C and K).


The Central ConjectureThe Central Conjecture

Conjecture: Every decidable set in PttR is in

P/poly. What can we show?




P/poly. What can we show? We show that a similar statement holds in the

context of time-bounded K-complexity.


Time-Bounded K-complexityTime-Bounded K-complexity

Let t be a time bound. (Think of t as being large, such as Ackermann’s function.)

Define Kt(x) to be min{|d| : U(d) = x in at most t(|x|) steps}.

Define RKt to be {x : Kt(x) ≥ |x|}. Define TTRT = {A : A is in Ptt

RKt for all large enough time bounds t}.

Vague intuition: Poly-time reductions should not be able to distinguish between RKt and RK, for large t.




P/poly. We show that a similar statement holds in the

context of time-bounded K-complexity:

– TTRT is contained in P/poly [ABFL12]. If t(n) = 22n, then RKt is NOT in P/poly. …which supports our “vague intuition”,

because this set is not reducible to the time-t’-random strings for t’ >> t.




P/poly. We show that a similar statement holds in the

context of time-bounded K-complexity:

– TTRT is contained in P/poly [ABFL12]. BUT – The same P/poly bound holds, even if

we consider PRKt instead of PttRKt.

…and recall PSPACE is contained in PR. So the “vague intuition” is wrong!


The Central Conjecture: An Earlier Approach



P/poly. We give a proof of a statement of the form:

An

AjΨ(n,j)

such that: if for each n and j

there is a proof in PA of Ψ(n,j)

then the conjecture holds.


Basic Proof TheoryBasic Proof Theory

Recall that Peano Arithmetic cannot prove the statement “PA is consistent”.

Let PA1 be PA + “PA is consistent”. Similarly, one can define PA2, PA3, … “PA is consistent” can be formulated as “for all

j, there is no length j proof of 0=1”. For each j, PA can prove “there is no length j

proof of 0=1”.





P/poly. We give a proof (in PA1) of a statement of the

form: An

AjΨ(n,j)

such that: if for each n and j

there is a proof in PA of Ψ(n,j)

then the conjecture holds.


The Central Conjecture: The Earlier Approach FailsThe Central Conjecture:

The Earlier Approach Fails The connections to proof theory were

unexpected and intriguing, and seemed promising…

But unfortunately, it turns out that many of the statements Ψ(n,j) are independent of PA (and a related approach yields statements Ψ(n,j,k) that are independent of each system PAr).


A High-Level View of the “Earlier Approach”

A High-Level View of the “Earlier Approach”

Let A be decidable, and let M be a poly-time machine computing a ≤tt-reduction from A to R. Let Q(x) be the set of queries that M asks on input x. Let the size of Q(x) be at most f(|x|).

Then there is a d such that for all x,

there is a V containing only strings of length at most d+log f(|x|), such that

MV(x) = A(x).

Note: V says “long queries are non-random”.


A Warm-UpA Warm-Up



there is a V containing only strings of length at most d+log f(|x|), such that

MV(x) = A(x).

Note: If some V works for all x of length n, then A is in P/poly.


ProofProof

Assume that for each d there is some x such that, for all V containing strings of length at most d+log f(|x|), MV(x)≠A(x).

Consider the machine that takes input (d,r) and finds x (as above) and outputs the rth element of Q(x).

This shows that each element y of Q(x) has C(y) ≤ log d + log f(|x|) + O(1)


ProofProof



This shows that each element y of Q(x) has C(y) ≤ log d + log f(|x|) + O(1) < d + log f(|x|).

Thus if we pick V* to be R∩{0,1}d+log f(|x|), we see that MV*(x) = MR(x)


ProofProof




Thus if we pick V* to be R∩{0,1}d+log f(|x|), we see that MV*(x) = MR(x) = A(x). Contradiction!


Cleaning Things UpCleaning Things Up



there is a V containing only strings of length at most d+log f(|x|) gA(|x|), such that

MV(x) = A(x).





there is a V containing only strings of length at most d+log f(|x|) gA(|x|), such that

MV(x) = A(x).




Then for all x,

there is a V containing only strings in R of length at most gA(|x|) such that

MV(x) = A(x).


A RefinementA Refinement


Then for all x,

there is a V containing only strings in R of length at most gA(|x|) such that

MV(x) = A(x).


Approximating RApproximating R

We can obtain a series of approximations to R (up to length gA(n)) as follows:

Rn,0 = all strings of length at most gA(n). Rn,i+1 = Rn,i minus the i+1st string of length at

most gA(n) that is found, in an enumeration of non-random strings.

Rn,0, Rn,1, Rn,2, … Rn,i* = R∩{0,1}gA(n)


A RefinementA Refinement


Then for all xє{0,1}n, for all i,

there is a V containing only strings in Rn,i such that

MV(x) = A(x).


ProofProof

Assume that for each d there is some x,i such that, for all V containing strings in Rn,i of length at most d+log f(|x|), MV(x)≠A(x).

Consider the machine that takes input (d,r) and finds x,i (as above) and outputs the rth element of Q(x).


Thus if we pick V* to be R∩{0,1}d+log f(|x|), we see that MV*(x) = MR(x) = A(x). Contradiction!


Where does PA come in??Where does PA come in??


Then for all xє{0,1}n, for all i,

there is a V containing only strings in Rn,i such that

MV(x) = A(x)

And for all k, there is not a length-k proof that “for all i, V is not equal to Rn,i”.


What went wrong with the earlier approach.

What went wrong with the earlier approach.

We have shown: For all xє{0,1}n, for all i, there is a V containing only short strings in Rn,i such that MV(x) = A(x).

We were aiming at showing that one can swap the quantifiers, so that for all n, there is a V containing only short strings in Rn,i such that, for all x of length n, MV(x) = A(x).

But there is a (useless) reduction M for which this is false. (M already knows the outcome of its queries, assuming that the oracle is R.)


Open Questions:Open Questions:

Decrease the gap (NEXP vs EXPSPACE) between the lower and upper bounds on the complexity of the problems that are in NPRK for every U.

Some of our proofs rely on using RK. Do similar results hold also for RC?

– Disprove: The halting problem is in PttRC.

Can the PSPACE ∩ P/poly bound (in the time-bounded setting) be improved to BPP?

Is this approach relevant at all to the P=BPP question?


P vs BPPP vs BPP

Our main intuition for P=BPP comes from [Impagliazzo, Wigderson]. Circuit lower bounds imply derandomization.

Note that this provides much more than “merely” P=BPP; it gives a recipe for simulating any probabilistic algorithm.

Goldreich has argued that any proof of P=BPP actually yields pseudorandom generators (and hence a “recipe” as above)…

– …but this has only been proved for the “promise problem” formulation of P=BPP.


P vs BPPP vs BPP

Recall that TTRT sits between BPP and PSPACE ∩ P/poly.

A proof that TTRT = P would show that BPP = P – but it is not clear that this would yield any sort of recipe for constructing useful pseudorandom generators.

Although it would be a less “useful” approach, perhaps it might be an easier approach?


How to use R as an OracleHow to use R as an Oracle


Circuit ComplexityCircuit Complexity

Let D be a circuit of AND and OR gates (with negations at the inputs). Size(D) = # of wires in D.

Size(f) = min{Size(D) : D computes f} We may allow oracle gates for a set A, along

with AND and OR gates. SizeA(f) = min{Size(D) : DA computes f}


Oracle GatesOracle Gates

An “oracle gate” for B in a circuit:

B


K-complexity ≈ Circuit ComplexityK-complexity ≈ Circuit Complexity

There are some obvious similarities in the definitions.

C(x) = min{|d| : U(d) = x} Size(f) = min{Size(D) : D computes f}



There are some obvious similarities in the definitions. What are some differences?

A minor difference: Size gives a measure of the complexity of functions, C gives a measure of the complexity of strings.

– Given any string x, let fx be the function whose truth table is the string of length 2log|x|, padded out with 0’s, and define Size(x) to be Size(fx).





A more fundamental difference:

– C(x) is not computable; Size(x) is.


Lightning Review: ComputabilityLightning Review: Computability

Here’s all we’ll need to know about computability:– The halting problem, everyone’s favorite

uncomputable problem:– H = {(i,x) : Mi halts on input x}– Every r.e. problem is poly-time many-one

reducible to H.

– One such r.e. problem: {x : C(x) < |x|}. – There is no time-bounded many-one

reduction in other direction, but there is a (large) time t Turing reduction (and hence C is not computable).





A more fundamental difference:

– C(x) is not computable; Size(x) is. In fact, there is a fascinating history, regarding

the complexity of the Size function.


MCSPMCSP

MCSP = {(x,i) : Size(x) < i}. MCSP is in NP, but is not known to be NP-

complete. Some history: NP-completeness was

discovered independently in the 70s by Cook (in North America) and Levin (in Russia).

Levin delayed publishing his results, because he wanted to show that MCSP was NP-complete, thinking that this was the more important problem.


MCSPMCSP

MCSP = {(x,i) : Size(x) < i}. Why was Levin so interested in MCSP? In the USSR in the 70’s, there was great

interest in problems requiring “perebor”, or “brute-force search”. For various reasons, MCSP was a focal point of this interest.

It was recognized at the time that this was “similar in flavor” to questions about resource-bounded Kolmogorov complexity, but there were no theorems along this line.


MCSPMCSP

MCSP = {(x,i) : Size(x) < i}. 3 decades later, what do we know about

MCSP? If it’s complete under the “usual” poly-time

many-one reductions, then BPP = P. MCSP is not believed to be in P. We know:

– Factoring is in BPPMCSP.

– Every cryptographically-secure one-way function can be inverted in PMCSP/poly.


So how can K-complexity and Circuit complexity be the same?So how can K-complexity and

Circuit complexity be the same? C(x) ≈ SizeH(x), where H is the halting

problem. For one direction, let U(d) = x. We need a

small circuit (with oracle gates for H) for fx, where fx(i) is the i-th bit of x. This is easy, since {(d,i,b) : U(d) outputs a string whose i-th bit is b} is computably-enumerable.

For the other direction, let SizeH(fx) = m. No oracle gate has more than m wires coming into it. Given a description of D (size not much bigger than m) and the m-bit number giving the size of {y in H : |y| ≤ m}, U can simulate DH and produce fx.


So how can K-complexity and Circuit complexity be the same?So how can K-complexity and

Circuit complexity be the same? C(x) ≈ SizeH(x), where H is the halting

problem. So there is a connection between C(x) and

Size(x) … …but is it useful? First, let’s look at decidable versions of

Kolmogorov complexity.


Time-Bounded Kolmogorov Complexity


The usual definition: Ct(x) = min{|d| : U(d) = x in time t(|x|)}. Problems with this definition

– No invariance! If U and U’ are different universal Turing machines, Ct

U and CtU’ have

no clear relationship.

– (One can bound CtU by Ct’

U’ for t’ slightly larger than t – but nothing can be done for t’=t.)

No nice connection to circuit complexity!




Levin’s definition: Kt(x) = min{|d|+log t : U(d) = x in time t}. Invariance holds! If U and U’ are different

universal Turing machines, KtU(x) and KtU’(x) are within log |x| of each other.

And, there’s a connection to Circuit Complexity:

– Let A be complete for E = Dtime(2O(n)). Then Kt(x) ≈ SizeA(x).




Levin’s definition: Kt(x|φ) = min{|d|+log t : U(d,φ)=x in time t}. Why log t?

– This gives an optimal search order for NP search problems.

– This algorithm finds satisfying assignments in poly time, if any algorithm does:

– On input φ, search through assignments y in order of increasing Kt(y|φ).




Levin’s definition: Kt(x) = min{|d|+log t : U(d) = x in time t}. Why log t?

– This gives an optimal search order for NP search problems.

– Adding t instead of log t would give every string complexity ≥ |x|.

…So let’s look for a sensible way to allow the run-time to be much smaller.


Revised Kolmogorov ComplexityRevised Kolmogorov Complexity

C(x) = min{|d| : for all i ≤ |x| + 1, U(d,i,b) = 1 iff b is the i-th bit of x} (where bit # i+1 of x is *).

– This is identical to the original definition. Kt(x) = min{|d|+log t : for all i ≤ |x| + 1, U(d,i,b)

= 1 iff b is the i-th bit of x, in time t}.

– The new and old definitions are within O(log |x|) of each other.

Define KT(x) = min{|d|+t : for all i ≤ |x| + 1, U(d,i,b) = 1 iff b is the i-th bit of x, in time t}.


Kolmogorov Complexity is Circuit Complexity


C(x) ≈ SizeH(x). Kt(x) ≈ SizeE(x). KT(x) ≈ Size(x). Other measures of complexity can be

captured in this way, too:

– Branching Program Size ≈ KB(x) = min{|d|+2s : for all I ≤ |x| + 1, U(d,i,b) = 1 iff b is the i-th bit of x, in space s}.




C(x) ≈ SizeH(x). Kt(x) ≈ SizeE(x). KT(x) ≈ Size(x). Other measures of complexity can be

captured in this way, too:

– Formula Size ≈ KF(x) = min{|d|+2t : for all I ≤ |x| + 1, U(d,i,b) = 1 iff b is the i-th bit of x, in time t}, for an alternating Turing machine U.


…but is this interesting?…but is this interesting?

The result that Factoring is in BPPMCSP was first proved by observing that, in PMCSP, one can accept a large set of strings having large KT complexity: RKT = {x : KT(x) ≥ |x|}.

(Basic Idea): There is a pseudorandom generator based on factoring, such that factoring is in BPPT for any test T that distinguishes truly random strings from pseudorandom strings. RKT is just such a test T.


This idea has many variants.This idea has many variants.

Consider RKT, RKt, and RC. RKT is in coNP, and not known to be coNP hard. RC is not hard for NP under poly-time many-one

reductions, unless P=NP.

– How about more powerful reductions?

– Is there anything interesting that we could compute quickly if C were computable, that we can’t already compute quickly?

– Proof uses PRGs, Interactive Proofs, and the fact that an element of RC of length n can be found in

But RC is undecidable! Perhaps H is in P relative to RC?


This idea has many variants.This idea has many variants.

Consider RKT, RKt, and RC. RKT is in coNP, and not known to be coNP hard. RC is not hard for NP under poly-time many-one

reductions, unless P=NP.

– How about more powerful reductions? We show:

– PSPACE is in P relative to RC.

– NEXP is in NP relative to RC.

– Proof uses PRGs, Interactive Proofs, and the fact that an element of RC of length n can be found in poly time, relative to RC [BFNV].

But RC is undecidable! Perhaps H is in P relative to RC?


Relationship between H and RCRelationship between H and RC

Perhaps H is in P relative to RC? This is still open. It is known that there is a

computable time bound t such that H is in DTime(t) relative to RC [Kummer].

…but the bound t depends on the choice of U in the definition of C(x).

We also showed: H is in P/poly relative to RC. Thus, in some sense, there is an “efficient”

reduction of H to RC.


Playing the Game in PSPACEPlaying the Game in PSPACE

We can also define a space-bounded measure KS, such that Kt(x) < KS(x) < KT(x).

RKS is complete for PSPACE under BPP reductions (and, in fact, even under “ZPP” reductions, which implies completeness under NP reductions and P/poly reductions).

[This makes use of an even stronger form of the Impagliazzo-Wigderson generator, which relies on the “hard” function being “downward self-reducible and random self-reducible”.]


Thank you!Thank you!

the strange link between incompressibility and complexity

Documents

strange link

cx kx cx

universal turing machine

universal turing machinekx

theory of randomness

log xtwo notions of

kolmogorov complexitycx

additive constant