An Empirical Study of the PerformanceAn Empirical Study of the Performanceof Preprocessing and Look-ahead Techniquesof Preprocessing and Look-ahead Techniques

for Solving Finite Constraint Satisfaction Problemsfor Solving Finite Constraint Satisfaction Problems

Zheying Jane YangZheying Jane Yang

Under the supervision of Prof. B.Y. Choueiry

July 30, 2003July 30, 2003

Constraint Systems LaboratoryDepartment of Computer Science and Engineering

University of Nebraska-Lincoln

2

Outline

1. Motivation & results2. Background

3. Experimental design and empirical study 4. Results and analysis5. Conclusions & relation to previous work

6. Summary of contributions7. Future work

3

1. Motivation

CSP used to model NP-complete decision problems

Search (exponential) is necessary, improved with Preprocessing algorithms:

remove inconsistent combinations prior to search Look-ahead algorithms:

remove inconsistencies during search

Preprocessing Search w/ look-ahead

CSP Smaller CSP Solution(s)

4

Algorithms studied & goal

Several competing algorithms Preprocessing: removes inconsistencies prior to search

Arc-consistency: AC3, AC2001 Neighborhood inverse consistency (NIC), requires search

Look-ahead: filters search space during search Forward checking (FC) Maintaining arc-consistency (MAC)

Controversies about their relative performance exist

Our goal is to characterize empirically the relative performance of combinations of preprocessing and look-ahead schemes as a function of the CSP’s constraint probability & tightness

Current beliefs & results

Our results: We specify when the above claims hold or not

Bessière & Régin 2001AC3 vs AC2001 AC2001 is better

ClaimsIssue

Freuder & Elfe 1996NIC is betterAC vs NIC

MAC vs FC

Haralick & Elliott 1980

Nadel 1989

Bessière & Régin 1996

FC is better

MAC is much better

MAC is the winner in large spase CSPsFC is the winner in dense CSPs

Sabin & Freuder 1994

Grant 1997

Gent & Prosser 2000

Pre

pro

cess

ing

Loo

k-a

hea

d

No winner performs extremely well on all types of CSPs

6

2. Background

7

What is a CSP?

A CSP problem is defined as a triple P=(V,D,C)

One solution All solutions

The goal is to find

ZY

X

{(b,c), (b,d)} {(b,e), (a,f)}

{(c,e), (d,f)}{c,d} {e,f}

{a,b}

8

Solving CSPs with systematic BT search

Backtrack search is sound and complete, but exponential expands a partial solution explores every combination systematically

Improve performance of search Preprocessing (constraint filtering/propagation) Look-ahead (constraint filtering during search) Backtracking: chronological/intelligent Variable & value ordering: static/dynamic, heuristic etc.

9

Solving a CSP

We study the performance of Preprocessing Hybrid search = preprocessing x look-ahead algorithms

Preprocessing Look-ahead

Algorithms: AC3, AC2001, NIC Algorithms: FC & MAC

Following an assignment, removes values from the domains of the future variables that are inconsistent with the current assignment.

Enforces different levels of local consistency by deleting inconsistent values from variable domains

V3{R, G} {G}

{R, G, B}V1

V2

{R, G} {G}

{R, G, B}V1

V3

V2

{R} {G}

{R, B}

V1

V3

V2

{R} {G}

{B} V1

V3

V2

{R, G} {G}

{R, B}

V1

V3

V2

Preprocessing a graph coloring problem

{R, G} {G}

{R}V1

V3V2

V1

V2

V3

Start

Solving CSP using forward checking (FC)

{G} {G}

{R} V1

V3

V2

V1

V2

V3

Start

Domain wiped-outbacktrack

Solving CSP using forward checking (FC)

{R, G} {G}

{G} V1

V3

V2

V1

V2

V3

Start

Domain wiped-outbacktrack

Solving CSP using forward checking (FC)

{R, G} {G}

{B} V1

V3

V2

V1

V2

V3

Start

Solving CSP using forward checking (FC)

{R} {G}

{B} V1

V3

V2

V1

V2

V3

Start

Solving CSP using forward checking (FC)

{R} {G}

{G}V1

V3

V2

V1

V2

V3

Start

Solution!

Solving CSP using forward checking (FC)

17

3. Experimental design & empirical study

18

Algorithms tested

Preprocessing Look-ahead

AC3AC2001

NIC FCMAC-AC3MAC-AC2001

FCMAC-AC3MAC-AC2001

Preprocessing: 5 algorithms

Look-ahead: 3 algorithms

Hybrid search: 7 combinations

19

Considered only binary constraints Restricted to finding first solution

Restricted to chronological backtracking

Use least domain (LD) as variable ordering heuristic Variable ordering done dynamically No value ordering heuristic, too costly in general

Working assumptions

20

Random CSPs We need many instances with similar characteristics in order to analyze the performance of search Real-world problems cannot be controlled by explicit parameters to enable statistical analysis of the performance

Generated instances connected graphs instances guaranteed solvable

Problems tested

21

Control parameters < n, d, C, t >

Number of variables n

We choose n = 50

Domain size d (uniform)

We choose d = 10. Thus, problem size is 1050, relatively large

Constraint tightness t = (uniform)

We vary t = [0.1, 0.2, …, 0.9] and t = [0.05, 0.1, …, 0.95]

Number of constraints C determines constraint probability p= , Cmax= n(n-1)/2

We vary C = [20, 490] corresponding to p = [0.024, 0.4] We report C = 30, 50, 80, 120, 130, 245, 340, 490

Disallowed tuplesAll possible tuples

CCmax

22

Comparisons

Evaluation criteria Filtering effectiveness measures reduction of CSP size Number of constraint checks #CC measures filtering effort Number of nodes visited #NV measures backtracking effort CPU time [ms] measures overall performance

Since constraints are defined in extension, CPU time reflects #CC

Preprocessing: Filtering effectiveness, #CC, CPU time

Hybrid search: #CC, #NV, CPU time

23

Evaluated performance by running each algorithm on 30 CSP instances of equal characteristics, calculating average & standard deviation values for each evaluation criterion

Generated approximately 6,000 CSP instances

Implemented in JAVA: 21 JAVA classes 4,000 lines of code

Experiments carried out on PrairieFire.unl.edu

Sampling and code characteristics

24

Part I: PreprocessingPart II: Hybrids

4. Results and analysis

Preprocessing: filtering effectiveness

NIC reduces search space at a large amount

After NIC-FC check Ln (CSPsize) of instance <50 10 C T>

0

20

40

60

80

100

120

140

0.05

0.15

0.25

0.35

0.45

0.55

0.65

0.75

0.85

0.95

Tightness

Ln C

SPsi

ze

p=0.08p=0.11p=0.15p=0.2p=0.28

After AC3 check LN(CSPsize) of instance <50 10 C T>

0

20

40

60

80

100

120

140

0.05

0.15

0.25

0.35

0.45

0.55

0.65

0.75

0.85

0.95

TightnessLn

CS

Psi

ze

p=0.08p=0.11p=0.15p=0.2p=0.28

26

Preprocessing results

p < 0.05 0.05 p < 0.1 0.1 p < 0.2 p 0.2

Filtering effectiveness

Filtering effectiveness increases with p

NIC-MAC-x » NIC-FC » AC3 AC2001

» Better than, comparable

Preprocessing: #CC, sparse CSPs

CC (AC3) > CC (AC2001)CC (NICx) >> CC (AC3, AC2001)

Preprocessing on <50 10 60 T>p = 0.049

0

5000

10000

15000

20000

25000

30000

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9Tightness

#CC

AC3 AC2001 NIC-FC NIC-MAC3 NIC-MAC2001

Preprocessing on <50 10 60 T> p = 0.049

0

1000

2000

3000

4000

5000

6000

7000

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

Tightness

#CC

AC3 AC2001

Preprocessing: #CC, denser CSPs

When p=0.2, NICx are all costly NIC should never be combined with MAC

Preprocessing on <50 10 245 T>p = 0.2

0

200000

400000

600000

800000

1000000

1200000

1400000

Tightness

#CC

AC3

AC2001

NIC-FC

NIC-MAC3

NIC-MAC2001

Preprocessing on <50 10 245 T> p = 0.2

0

5000

10000

15000

20000

25000

Tightness

#CC

AC3

AC2001

29

Preprocessing results

p < 0.05 0.05 p < 0.1 0.1 p < 0.2 p 0.2

Filtering effectiveness

Filtering effectiveness increases with p

NIC-MAC-x » NIC-FC » AC3 AC2001

#CCAC2001 » AC3 » NICx

Never use MAC with NIC

» Better than, comparable

Preprocessing on <50 10 C 30> T = 0.3

0

5000

10000

15000

20000

25000

30000

35000

40000

0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

Probability

CPU t

ime (

ms)

AC3

AC2001

NIC-FC

NIC-MAC3

NIC-MAC2001

Preprocessing on <50 10 C 50> T = 0.5

0

5000

10000

15000

20000

25000

30000

35000

Probability

CPU t

ime (

ms)

AC3

AC2001

NIC-FC

NIC-MAC3

NIC-MAC2001

Preprocessing on <50 10 C 70> T = 0.7

05000

100001500020000250003000035000400004500050000

0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4Probability

CPU

time (

ms)

AC3

AC2001

NIC-FC

NIC-MAC3

NIC-MAC2001

Preprocessing: CPU time

When p > 0.2, NIC-x is too costly

31

Preprocessing results

p < 0.05 0.05 p < 0.1 0.1 p < 0.2 p 0.2

Filtering effectiveness

Filtering effectiveness increases with p

NIC-MAC-x » NIC-FC » AC3 AC2001

#CCAC2001 » AC3 » NICx

Never use MAC with NIC

CPU time AC3 » AC2001 » NICx

» Better than, comparable

32

Preprocessing: summary

When to use what

p < 0.05 0.05 p < 0.1 0.1 p < 0.2 p 0.2

ACx Preprocessing not effective

Not effective Effective Effective

NIC-xQuite effective Effective, but

MAC too costlyToo costly,

avoid

AC2001 is not significantly better than AC3, and is not worth the extra data structures

In general, we disagree with Bessière and Régin

NIC-x: powerful filtering, but too costly.

Use it on large problems when checking constraints is cheap

Hybrids: #CC, p = 5%, 11%

When p < 0.05: ACx is not effective,ACx-FC are costly, NIC-FC-FC is OK, NIC-MAC-x is bestWhen p > 0.10: avoid MAC, stick with FC

Search on <50 10 60 T>p = 0.049

0

50000

100000

150000

200000

250000

300000

350000

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9Tightness

#CC

AC3-FC

AC2001-FC

AC3-MAC3

AC2001-MAC2001

NIC-FC-FC

NIC-MAC3-MAC3

NIC-MAC2001-MAC2001

Search On <50 10 140 T>p = 0.114

0

5000000

10000000

15000000

20000000

25000000

Tightness

#CC

AC3-FC

AC2001-FC

AC3-MAC3

AC2001-MAC2001

NIC-FC-FC

NIC-MAC3-MAC3

NIC-MAC2001-MAC2001

34

Hybrids results

p < 0.05 0.05 p < 0.10 0.10 p < 0.15 0.15 p < 0.2 p 0.2

NIC-MAC-Acx

» NIC-FC-FC

» AC-MAC-x

» ACx-FC

NIC-FC-FC

» NIC-MAC-ACx

» ACx-FC

AC-MAC-x

» Better than, comparable

When to use what

p < 0.05 0.05 p < 0.10 0.10 p < 0.15 0.15 p < 0.2 p 0.2

FC Avoid FC FC dominates

MAC MAC dominates

Avoid MAC

NIC NIC helps NIC still helps

ACx ACx useless

#CC &

CPU time

As p increases: MAC deteriorates, NIC becomes expensive, use ACx-FC

Hybrids: #CC, p= 15%, 28%Search on <50 10 185 T>

p = 0.15

0

2000000

4000000

6000000

8000000

10000000

12000000

14000000

16000000

18000000

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9Tightness

#CC

AC3-FC

AC2001-FC

AC3-MAC3

AC2001-MAC2001 NIC-FC-FC

NIC-MAC3-MAC3 NIC-MAC2001-MAC2001

Search on <50 10 340 T> p = 0.28

0

50000

100000

150000

200000

250000

Tightness

#CC

AC3-FC

AC2001-FC

AC3-MAC3

AC2001-MAC2001

NIC-FC

NIC-MAC3-MAC3

NIC-MAC2001-MAC2001

36

Hybrids results

p < 0.05 0.05 p < 0.10 0.10 p < 0.15 0.15 p < 0.2 p 0.2

NIC-MAC-Acx

» NIC-FC-FC

» AC-MAC-x

» ACx-FC

NIC-FC-FC

» NIC-MAC-ACx

» ACx-FC

AC-MAC-x

NIC-FC-FC

» ACx-FC

» NIC-MAC-ACx

» AC-MAC-x

ACx-FC

» NIC-FC-FC

» NIC-MAC-ACx

» AC-MAC-x

ACx-FC

» AC-MAC-x

» NIC-FC-FC

» NIC-MAC-Acx

» Better than, comparable

When to use what

p < 0.05 0.05 p < 0.10 0.10 p < 0.15 0.15 p < 0.2 p 0.2

FC Avoid FC FC dominates

MAC MAC dominates Avoid MAC

NIC NIC helps NIC still helps NIC deteriorates Avoid NIC

ACx ACx useless AC-x starts to work ACx helps

#CC &

CPU time

Search on <50 10 60 T> p = 0.049

0

500

1000

1500

2000

2500

3000

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

Tightness

#NV

AC3-FC

AC3-MAC3

NIC-FC-FC

NIC-MAC3-MAC3

Search on <50 10 185 T> p = 0.10

0

50000

100000

150000

200000

250000

300000

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9Tightness

#NV

A3-FC

AC3-MAC3

NIC-FC

NIC-MAC3-MAC3

Search on <50 10 123 T> p = 0.15

0100002000030000400005000060000700008000090000

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9Tightness

#NV

A3-FC

AC3-MAC3

NIC-MAC3-MAC3

NIC-FC-FC

Hybrids: #NV

At phase transition, NIC and MAC do powerful filtering but influence of MAC is stronger

38

Hybrids: summary

p < 0.05 0.05 p < 0.10 0.10 p < 0.15 0.15 p < 0.2 p 0.2

NIC-MAC-Acx

» NIC-FC-FC

» AC-MAC-x

» ACx-FC

NIC-FC-FC

» NIC-MAC-ACx

» ACx-FC

AC-MAC-x

NIC-FC-FC

» ACx-FC

» NIC-MAC-ACx

» AC-MAC-x

ACx-FC

» NIC-FC-FC

» NIC-MAC-ACx

» AC-MAC-x

ACx-FC

» AC-MAC-x

» NIC-FC-FC

» NIC-MAC-Acx

#NVNIC-MAC-Acx » AC-MAC-x

» NIC-FC-FC » ACx-FC

AC-MAC-x » NIC-MAC-ACx

» ACx-FC » NIC-FC-FC

» Better than, comparable

When to use what

p < 0.05 0.05 p < 0.10 0.10 p < 0.15 0.15 p < 0.2 p 0.2

FC Avoid FC FC dominates

MAC MAC dominates Avoid MAC

NIC NIC helps NIC still helps NIC deteriorates Avoid NIC

ACx ACx useless AC-x starts to work ACx helps

#CC &

CPU time

39

5. Conclusions

Preprocessing AC3 vs. AC2001: AC2001 is not significantly better than AC3, and

is not worth the extra data structures

In general, we disagree with Bessière and Régin NIC-x: powerful filtering, but too costly

Use it on large problems when checking constraints is cheap

Look-ahead MAC vs. FC: performance depends on constraint probability and

tightness. MAC only wins in low p and high t

In general we disagree with results

of Sabin, Freuder, Bessière & Régin

40

Relation to previous work (I)

NIC is better than AC Freuder & Elfe 1996

The instances tested were all with low probability (p < 0.05). In this region, AC is ineffective.

MAC is better than FC Sabin & Freuder 1994

They tested CSPs with low probability (p = 0.018-0.09), and relatively high constraint-tightness (t = 0.15 – 0.675). In our study, MAC is effective in this region, but not outside it.

MAC is better than FC Bessière & Régin 1996

The instances tested were also in the region of low probability (p = 0.017, 0.024, 0.074, 0.08, 0.1, 0.12, 0.15), except instance#1 and instance#2, with relative high probability (p = 0.3, and 0.84). But here they test only 2 instances

41

Relation to previous work (II)

Gent & Prosser 2000 questioned validity of previous results on MAC. They concluded that: in large, sparse CSPs with tight constraints, MAC is winner in dense CSPs with loose constraints, FC is winner.

Grant 1997 showed that FC is winner on small CSPs with all range of probabilities

All concluded that: “A champion algorithm which perform extremely well on all types of problems does not exist.”

Our characterizations are more thorough and precise.

42

6. Summary of contributions

Random generator that guarantees solvability

Empirical evaluation of the performance of 7 combinations of preprocessing and look-ahead

Uncovered (restricted) validity conditions of previously reported results

Summarized best working conditions for preprocessing and look-ahead algorithms

Developed a Java library with 7 hybrid algorithms

43

7. Directions for future work

Compare to other, less common filtering algorithms, e.g. SRPC, PC, SAC, Max-RPC Debryune & Bessière 2001

Combining these preprocessing algorithms with intelligent backtrack search algorithms

Validate results on larger CSPs, real-world applications, non-binary

Test and analyze the effect of the topology of constraint networks on the performance of search

44

Acknowledgments

Dr. Berthe Y. Choueiry (advisor)Dr. Sebastian ElbaumDr. Peter Revesz

Ms. Catherine L. AndersonMr. Daniel BuettnerMs. Deborah Derrick (proof reading)Mr. Eric MossMr. Lin XuMs. Yaling Zheng Mr. Hui Zou

THANK YOU!