Efficient Closed Pattern Mining in the Presence of Tough Block Constraints

Krishna GadeComputer Science &

Engineeringgade@cs.umn.edu

Outline Introduction Problem Definition and Motivation Contributions

Block Constraints Matrix Projection based approach Search Space Pruning Techniques

Experimental Evaluation Conclusions

Introduction to Pattern Mining

What is a frequent pattern ? Why is frequent pattern mining a

fundamental task in data mining ? Closed, Maximal and Constrained

Extensions State-of-the-art algorithms Limitations with the current

solutions

What is a frequent pattern ? A frequent pattern can be a set of items,

sequence, graph that occurs frequently in a database

It can also be a spatial, geometric or topological pattern depending on the database chosen

For a given a transaction database and a support threshold min_sup, an itemset X is frequent if Sup(X) >= min_sup

Sup(X) is the support of X, defined as the fraction of the transactions in the database which contain X.

Why is frequent pattern mining so fundamental to data mining ?

Foundation for several essential data mining tasks Association, Correlation and Causality Analysis Classification based on Association Rules Pattern-based and Pattern-preserving Clustering.

Support is a simple, yet effective (in many cases) measure to determine the significance of a pattern, which also correlates with most of the other statistical measures.

Closed, Maximal & Constrained Extensions to Frequent Patterns

A frequent pattern X is said to be closed, if no superset of X has the same supporting set.

It is said to be maximal, if no superset of X is frequent.

It is said to be constrained, if it satisfies some constraint defined on the items

present or on the transactions that support it. E.g., length(X) >= min_l

State-of-the-art algorithms for Pattern Discovery Itemsets

Frequent : FP-Growth, Apriori, OP, Inverted Matrix

Closed : Closet+, Charm, FPClose, LCM Maximal : Mafia, FPMax Constrained : LPMiner, Bamboo

Sequences SPAM, SLPMiner, BIDE etc.,

Graphs FSG, gFSG, gSpan etc.,

Limitations Limitations of Support

May not capture the semantics of user interest. Too many frequent patterns if the support

threshold is too low. Closed and Maximal frequent patterns fix this but there

may be loss of information (in case of maximal). Support is ‘a’ measure of interestingness of a

pattern, there can be others such as ‘length’ etc. E.g., One may be interested in finding patterns whose

length decreases with the support.

Definitions Block is a 2-tuple B = (I,T), consisting of itemset I

and its supporting set T. Weighted block is a block with a weight function

w, where w: I x T ->R+

B is a Closed block iff there exists no block B’ = (I’,T’) where I’ is the superset of I. (If such B’ exists then it is the super-block of B and B its sub-block.)

The size of the block B is defined as

The sum of the corresponding weighted block, is defined as

||||)( TIBBSize

IiTtitwBBSum

,),()(

Example of a Block

T1 {a,b,c,e}

T2 {b,c,d,e}

T3 {a,d,e}

T4 {c,d,e}

a b c d e

T1 1 1 1 0 1

T2 0 1 1 1 1

T3 1 0 0 1 1

T4 0 0 1 1 1

B1 = ({a,b},{T1}), B2 = ({c,d},{T2,T4}) are examples of a block.

Red sub-matrix is not a block.

Example Database

Matrix Representation of the Database

Block Constraints Let be the set of all transactions in the database and

be the set of all items. A block constraint C is a predicate, C : 2 x 2-> {true,

false} A block B is a valid block for C if B satisfies C or C(B) is

true. C is a tough block constraint if there is no dependency

between the satisfaction (violation) of by a block B and the satisfaction (violation) of its super or sub-blocks.

In this thesis explore 3 different tough block constraints.

Block-size Block-sum Block-similarity

Monotonicity and Anti-montononicty of Constraints Monotone Constraint

C is monotone iff C(X) = true, then for every Y such that Y is a superset of X, C(Y) = true.

E.g. : Sup(X) <= v is monotone. Benefit : Prune all Y if Sup(X) > v.

Anti-monotone Constraint C is anti-monotone iff C(X) = true, then for

every Y such that Y is a subset of X, C(Y) = true.

E.g. : Sup(X) >= v is anti-monotone. Benefit : Prune all Y if Sup(X) < v.

Why Block-size is a tough constraint – An Illustration

a b c d e

T1

1 1 1 0 1

T2

0 1 1 1 1

T3

1 0 0 1 1

T4

0 0 1 1 1

For the constraint BSize >= 4,

•({b,c},{T1,T2} is a valid block, but ({b,c,d},{T2}) is invalid.

Block-size constraint is not monotone.

• neither of ({b},{T1,T2}), ({c},{T1,T2,T4}) is valid.

Block-size constraint is not anti-monotone.

Block-size, Block-sum Constraints Block-size Constraint

Motivation : Find set of itemsets each of which accounts for a certain fraction of overall number of transactions performed in a period of time.

Block-sum Constraint Motivation : Identify

product groups that account for a certain fraction of the overall sales, profits etc.

t

length(t)N

NvBBSize

and 10 where

,)(

it

itwW

WvBBSum

,

),( and 10 where

,)(

Block-similarity Definition Motivation: Finding groups of thematically related

words in large document datasets. Importance of a group of words can be measured

by their contribution to the overall similarity between the documents in the collection.

Here is the set of tf-idf scaled and normalized unit-length document vectors and is the set of distinct terms in the collection.

Block-similarity of a weighted block B is defined as Loss in the aggregate pairwise similarity of the

documents in resulting from zeroing-out entries corresponding to B.

BSim(B) = S – S’, where S, S’ are the aggregate pairwise similarities before and after removing B.

Block Similarity - Illustration

a b c d e

D1

.1 .1 .1 0 .1

D2

0 .1 .1 .3 .4

D3

.1 0 0 .3 .3

D4

0 0 .3 .2 .1

a b c d e

D1

.1 0 0 0 .1

D2

0 0 0 .3 .4

D3

.1 0 0 .3 .3

D4

0 0 .3 .2 .1({b,c},{D1,D2}) is removed here to calculate its block-similarity, by measuring the loss in the aggregate similarity.

Block-similarity contd., Similarity of any two documents is measured as the dot-

product of their unit-length vectors. (cosine) For the given collection , we define a composite vector to be

the sum of all document vectors in

We define the composite vector BI for a weighted block B = (I,T) to be the vector formed by adding all the vectors in T only along the dimensions in I. Then,

Block-similarity constraint is now defined as

ddD

III

II

BBBD

BDBD

DD

2')(

)()(',

SSBBSim

S

ddddSjiji d

j

d

i

dd

ji

10 where,)( SvBBSim

Key Features of the Algorithm

Follows widely used projection-based pattern mining paradigm. Adopts a depth-first search traversal on the lattice of complete set of

itemsets, with the items ordered non-decreasingly on their frequency. Represents the transaction/document database as a matrix,

transactions (documents) as rows and items (terms) as columns. Employs efficient compressed sparse matrix storage and access

schemes like to achieve high computational efficiency. Matrix-projection based pattern enumeration shares ideas from the

recently developed array-projection based method H-Mine. Prunes potentially invalid rows and columns at each node during the

traversal of the lattice (shown in the next page) as determined by our row-pruning and column-pruning and matrix-pruning tests.

Adopts various closed itemset mining optimization techniques, like column fusing, redundant pattern pruning from CHARM and Closet+ to the block constraints.

The hash-table consists of only closed patterns hashed by the sum of the transaction-ids of the transactions in their supporting sets.

Ø

a b c d

ab ac ad bc bd cd

abc abd acd bcd

abcd

Level 1

Level 2

Level 3

Level 4

Pattern Enumeration Visits each node in the lattice in a dept-first order. Each node represents a

distinct pattern p. At a certain node labeled p in the lattice, we report and store p in the hash

table, as a closed pattern if p is closed and valid under the given block constraint.

We build a p-projected matrix by pruning any potentially invalid columns and rows determined by our pruning tests.

Matrix-Projection A p-projected matrix

is the matrix containing only the rows that contain p, and the columns that appear after p, in the predefined order.

Projecting the matrix is linear on the number of non-zeroes in the projected matrix.

a b c d e

T1 1 1 1 0 1

T2 0 1 1 1 1

T3 1 0 0 1 1

T4 0 0 1 1 1

c d e

T1 1 0 1

T2 1 1 1{b}-projected matrix

given matrix

Compressed Sparse Representation CSR format utilizes two one-

dimensional arrays : First stores the actual non-zero elements

of the matrix in a row (or column). Second stores the indices corresponding

to the beginning of each row (or column).

We maintain both row- and column-based representation for efficient projection and frequency counting.

CSR format for the example matrix

a b c d e

T1 1 1 1 0 1

T2 0 1 1 1 1

T3 1 0 0 1 1

T4 0 0 1 1 1

T1 T2 T3 T4 end

0 4 8 11 14

o 1 2 3 4 5 6 7 8 9 10

11

12

13

a b c e b c d e a d e c d e

Index Array

Pointer Array

Row-based CSR

Search Space Pruning Column Pruning

Given a pattern p and its p-projected matrix.

Necessary condition for the columns which can form a valid block with p.

Eliminate all columns in the p-projected matrix that do not satisfy it.

Block-Size A= local supporting set of

x. rlen(t)= local rowlength of

t.

c d e

T1 1 0 1

T2 1 1 1

{b}-projected matrix

Let BSize >= 5 be the constraint.

‘d’ will get pruned as it can never form a block of size >= 5 with its prefix {b}, since the maximum block-size possible with ‘d’ is 4.

vtrlenApBSize x At

)(),( ifPrune

Search Space Pruning contd., Column Pruning

Block-sum rsum(t) = local rowsum of t.

Block-similarity maximum value of vector D. local maximum rowsum. freq = frequency. a = 2D.BP

vtrsumApBSize x At

)(),( ifPrune

2/)( ifPrune avfreq(x) x

Search Space Pruningcontd., Row Pruning

Smallest Valid Extension, SVE SVE(p) is the length of the smallest possible extension q

to p, such that resulting block formed by p and q, is valid. Prune rows whose length is smaller than SVE in the p-

projected matrix. SVE for generic block constraint BSxxx is given below.

Block-size z = size of the supporting set of p.

Block-sum z = maximum column sum in the p-projected matrix.

Block-similarity z = maximum column similarity in the p-projected matrix.

zBBSxxxvpSVE /))(()(

Search Space Pruning contd., Row Pruning Example Matrix Pruning

Prune p-projected matrix if Block-size

Sum of the row-lengths in the projected matrix is insufficient.

Block-sum Sum of the row-sums and

is insufficient. Block-similarity

Sum of the column- similarities is insufficient.

to form a valid block with p.

c d e

T1 1 0 1

T2 1 1 1

{b}-projected matrix

Let BSum >= 7 be the constraint.

Since SVE >= 3, T1 gets pruned.

Pattern Closure Check and Optimizations Closure Check

Hash-table consists of closed patterns Hash-keys are sum of transaction-ids

At a certain node p in the lattice (shown before), Column Fusing

Fuse the fully dense columns of the p-projected matrix to p. Also fuse columns to one another that have identical

supporting sets. Redundant Pattern Pruning

If p is a proper subset of an already mined closed pattern with the same support, it can be safely pruned. Also any pattern extending it need not be explored as it has already been done. Hence p is a redundant pattern.

Experimental Setup

Data #Trans #Items

A.(M.)tran.len

Gazelle 59601 498 2.5(267)

Pumsb* 49046 2089 50.5(63)

Big-market

838466 38336 3.12(90)

Sports 8580 126373

258.3(2344)

T10I4Dx 200k-1000k

10000 10(31)

Notation :

CBMiner – Closed Block Miner Algorithm

CLOSET+ - State-of-the-art closed frequent itemset mining algorithm

CP – Column Pruning, RP – Row Pruning, MP – Matrix Pruning.

Experimental Results

Comparisons with Closet+ on Gazelle

Experimental Resultscontd.,

Comparisons with Closet+ on Sports

Experimental Results contd.,

Comparisons of Pruning Techniques on Gazelle (left) and Pumsb*(right)

No Pruning Gazelle : 1578.48 , BSize >= 0.1; Pumsb* = 1330.03 , BSum >= 6.0

Experimental Results contd.,

Comparisons Closed & All Valid Block Mining Big-Market

Experimental Results contd.,

Comparison of Pruning Techniques on Big-Market

Scalability Test on T10I4Dx

Time for No Pruning : 3560 seconds

Micro Concept Discovery Scaled the document vectors using tf-idf. Normalized using L2-norm. Applied the CBMiner algorithm for each of

the three constraints. Chose the top-1000 patterns ranked on the

constraint function value. Compute the entropies of the documents

that form the supporting set of the block. Also ran CLOSET+ to get the top-1000

patterns ranked on frequency.

Micro Concept Discovery contd.,

Data #docs

#terms

#classes

Classic 7089 12009 4

Sports 8580 18324 7

LA1 3204 31472 6

Average entropies of the four schemes are pretty low.

Block Similarity outperforms the rest as it leads to lowest entropies or purest clusters.

Block-size and itemset frequency constraints do not account for the weights associated with the terms and hence are inconsistent.

But, Block-sum performs reasonably well as it accounts for the term weights provided by tf-idf and L2-norm.

Micro Concept Discovery contd.,

Conclusions Proposed a new class of constraints called

“tough” block constraints. And a matrix-projection based framework

CBMiner for mining closed block patterns. Block Constraints discussed : Block-size, Block-

sum, Block-similarity 3 novel pruning techniques column pruning,

row pruning and matrix pruning. Order(s) magnitude faster than traditional

closed frequent itemset mining algorithms Finds much fewer patterns.

Thank You !!