process mining: understanding and improving desire lines in big data

Process MiningUnderstanding and Improving Desire Lines in Big Data

prof.dr.ir. Wil van der Aalstwww.processmining.org

Let’s Play: Play-Out, Play-In, ReplayBig DataDesire LinesProcess MiningHow Good is My Model?

Process DiscoveryConformance Checking

Food for Thought: Lasagna and SpaghettiGoogle Maps and TomTom

How to Get Started?Conclusion

On the different roles of (process) models …

Play-Out

PAGE 4

event logprocess model

A

B

C

DE

p2

end

p4

p3p1

start

Play-Out (Classical use of models)

PAGE 5

A B C D

A C B DA B C D

A E D

A C B D

A C B D

A E D

A E D

Play-In

PAGE 6

event log process model

A

B

C

DE

p2

end

p4

p3p1

start

Play-In

PAGE 7

A C B DA B C D

A E D

A C B D

A C B D

A E D

A E DA B C D

Example Process Discovery(Vestia, Dutch housing agency, 208 cases, 5987 events)

PAGE 8

Example Process Discovery(ASML, test process lithography systems, 154966 events)

PAGE 9

Example Process Discovery(AMC, 627 gynecological oncology patients, 24331 events)

PAGE 10

Replay

PAGE 11

event log process model

· extended model showing times, frequencies, etc.

· diagnostics· predictions· recommendations

A

B

C

DE

p2

end

p4

p3p1

start

Replay

PAGE 12

A B C D

A

B

C

DE

p2

end

p4

p3p1

start

Replay

PAGE 13

A E D

A

B

C

DE

p2

end

p4

p3p1

start

Replay can detect problems

PAGE 14

AC D

Problem!missing token

Problem!token left behind

Conformance Checking (WOZ objections Dutch municipality, 745 objections, 9583 event, f= 0.988)

PAGE 15

A

B

C

DE

p2

end

p4

p3p1

start

Replay can extract timing information

PAGE 16

A5B8 C9 D13

5

8

9

13

3

4

5

43

265

8

764

7

74

3

Performance Analysis Using Replay(WOZ objections Dutch municipality, 745 objections, 9583 event, f= 0.988)

Big Data

Big Data

PAGE 19

Source: “Big Data: The Next Frontier for Innovation, Competition, and Productivity” McKinsey Global Institute, 2011.

“Enterprises globally stored more than 7 exabytesof new data on disk drives in 2010, while consumers stored morethan 6 exabytes of new data on devices such as PCs and notebooks.”

“All of the world's music can be stored

on a $600 disk drive.”

“Indeed, we are generating so much data today that it is

physically impossible to store it all. Health

care providers, for instance, discard 90

percent of the data that they generate.”

Hilbert and Lopez. The World's Technological Capacity to Store, Communicate, and Compute Information. Science, 332(6025):60-65, 2011.

www.olifantenpaadjes.nl

http://www.olifantenpaadjes.nl/wp-content/uploads/2011/06/Koen-Olsthoorn.jpg

http://www.olifantenpaadjes.nl/wp-content/uploads/2011/06/Peter-Huiskamp-Apeldoorn-a.jpg

Evidence-Based Business Process Management

PAGE 25

Process Mining

Event Data + Processes

Process Mining =

Data Mining + Process Analysis

Machine Learning + Formal Methods

Process Mining

software system

(process)model

eventlogs

modelsanalyzes

discovery

records events, e.g., messages,

transactions, etc.

specifies configures implements

analyzes

supports/controls

enhancement

conformance

“world”

people machines

organizationscomponents

businessprocesses

Starting point: event log

PAGE 30

XES, MXML, SA-MXML, CSV, etc.

Simplified event log

PAGE 31

a = register request, b = examine thoroughly, c = examine casually, d = check ticket,e = decide, f = reinitiate request, g = pay compensation, and h = reject request

Processdiscovery

PAGE 32

a

start register request

b

examine thoroughly

c

examine casually

d

check ticket

decide

pay compensation

reject request

reinitiate request

e

g

h

f

end

c1

c2

c3

c4

c5

Conformance checking

PAGE 33

a


b

examine thoroughly

c

examine casually

d

check ticket

decide

pay compensation

reject request

reinitiate request

e

g

h

f

end

c1

c2

c3

c4

c5

case 7: e is executed without being

enabled

case 8: g or h is missing

case 10: e is missing in second

round

Extension: Adding perspectives to model based on event log

PAGE 34

a


b

examine thoroughly

c

examine casually

d

check ticket

decide

pay compensation

reject request

reinitiate request

e

g

h

f

end

c1

c2

c3

c4

c5

Performance information (e.g., the average time between two subsequent activities) can be extracted from the event log and visualized on top of the model.

A

A

A

A

A

E

M

M

Pete

Mike

Ellen

Role A:Assistant

Sue

Sean

Role E:Expert

Sara

Role M:Manager

Decision rules (e.g., a decision tree based on data known at the time a particular choice was made) can be learned from the event log and used to annotated decisions.

The event log can be used to discover roles in the organization (e.g., groups of people with similar work patterns). These roles can be used to relate individuals and activities.

How good is my model?

Four Competing Quality Criteria

PAGE 36

process discovery

fitness

precisiongeneralization

simplicity

“able to replay event log” “Occam’s razor”

“not overfitting the log” “not underfitting the log”

Example: one log four models

PAGE 37

astart register

request

bexamine thoroughly

cexamine casually

d checkticket

decide

pay compensation

reject request

reinitiate requeste

g

hf

end

astart register

request

cexamine casually

dcheckticket

decide reject request

e hend

N3 : fitness = +, precision = -, generalization = +, simplicity = +

N2 : fitness = -, precision = +, generalization = -, simplicity = +

astart register

request

bexamine

thoroughly

cexamine casually

dcheck ticket

decide

pay compensation

reject request

reinitiate request

e

g

h

f

end

N1 : fitness = +, precision = +, generalization = +, simplicity = +

astart register

request

cexamine casually

dcheckticket


e hend

N4 : fitness = +, precision = +, generalization = -, simplicity = -

aregister request

dexamine casually

ccheckticket


e h

a cexamine casually

dcheckticket

decide

e g

a dexamine casually

ccheckticket

decide

e g

register request

register request

pay compensation

pay compensation

aregister request

b dcheckticket


e h

aregister request

d bcheckticket


e h

a b dcheckticket

decide

e gregister request

pay compensation

examine thoroughly

examine thoroughly

examine thoroughly

… (all 21 variants seen in the log)

acdeh

abdeg

adceh

abdeh

acdeg

adceg

adbeh

acdefdbeh

adbeg

acdefbdeh

acdefbdeg

acdefdbeg

adcefcdeh

adcefdbeh

adcefbdeg

acdefbdefdbeg

adcefdbeg

adcefbdefbdeg

adcefdbefbdeh

adbefbdefdbeg

adcefdbefcdefdbeg

455

191

177

144

111

82

56

47

38

33

14

11

9

8

5

3

2

2

1

1

1

# trace

1391

process discovery

fitness


simplicity



Model N1

PAGE 38

acdeh

abdeg

adceh

abdeh

acdeg

adceg

adbeh

acdefdbeh

adbeg

acdefbdeh

acdefbdeg

acdefdbeg

adcefcdeh

adcefdbeh

adcefbdeg

acdefbdefdbeg

adcefdbeg

adcefbdefbdeg

adcefdbefbdeh

adbefbdefdbeg

adcefdbefcdefdbeg

455

191

177

144

111

82

56

47

38

33

14

11

9

8

5

3

2

2

1

1

1

# trace

1391

astart register

request

bexamine

thoroughly

cexamine casually

dcheck ticket

decide

pay compensation

reject request

reinitiate request

e

g

h

f

end


Model N2

PAGE 39

acdeh

abdeg

adceh

abdeh

acdeg

adceg

adbeh

acdefdbeh

adbeg

acdefbdeh

acdefbdeg

acdefdbeg

adcefcdeh

adcefdbeh

adcefbdeg

acdefbdefdbeg

adcefdbeg

adcefbdefbdeg

adcefdbefbdeh

adbefbdefdbeg

adcefdbefcdefdbeg

455

191

177

144

111

82

56

47

38

33

14

11

9

8

5

3

2

2

1

1

1

# trace

1391

astart register

request

cexamine casually

dcheckticket


e hend


Model N3

PAGE 40

acdeh

abdeg

adceh

abdeh

acdeg

adceg

adbeh

acdefdbeh

adbeg

acdefbdeh

acdefbdeg

acdefdbeg

adcefcdeh

adcefdbeh

adcefbdeg

acdefbdefdbeg

adcefdbeg

adcefbdefbdeg

adcefdbefbdeh

adbefbdefdbeg

adcefdbefcdefdbeg

455

191

177

144

111

82

56

47

38

33

14

11

9

8

5

3

2

2

1

1

1

# trace

1391

astart register

request

bexamine thoroughly

cexamine casually

d checkticket

decide

pay compensation

reject request

reinitiate requeste

g

hf

end


Model N4

PAGE 41

acdeh

abdeg

adceh

abdeh

acdeg

adceg

adbeh

acdefdbeh

adbeg

acdefbdeh

acdefbdeg

acdefdbeg

adcefcdeh

adcefdbeh

adcefbdeg

acdefbdefdbeg

adcefdbeg

adcefbdefbdeg

adcefdbefbdeh

adbefbdefdbeg

adcefdbefcdefdbeg

455

191

177

144

111

82

56

47

38

33

14

11

9

8

5

3

2

2

1

1

1

# trace

1391

astart register

request

cexamine casually

dcheckticket


e hend


aregister request

dexamine casually

ccheckticket


e h

a cexamine casually

dcheckticket

decide

e g

a dexamine casually

ccheckticket

decide

e g

register request

register request

pay compensation

pay compensation

aregister request

b dcheckticket


e h

aregister request

d bcheckticket


e h

a b dcheckticket

decide

e gregister request

pay compensation

examine thoroughly

examine thoroughly

examine thoroughly


http://tlc.howstuffworks.com/style/style/celebrity-fashion-disaster-pictures.htm

Process Discovery

Process Discovery (small selection)

PAGE 43

α algorithm

α++ algorithm

α# algorithm

language-based regions

state-based regionsgenetic mining

heuristic mining

hidden Markov models

neural networks

automata-based learning

stochastic task graphs

conformal process graph

mining block structures

multi-phase miningpartial-order based mining

fuzzy mining

LTL mining

ILP mining

distributed genetic mining

http://www.sxc.hu/browse.phtml?f=download&id=45559

Petri net view: Just discover the places …

PAGE 44

a1

...

a2

am

b1

b2

bn

p(A,B) ...

A={a1,a2, … am} B={b1,b2, … bn}

process discovery

fitness


simplicity



Adding a place limits behavior:•overfitting ≈ adding too many places •underfitting ≈ adding too few places

Example: Process Discovery UsingState-Based Regions

PAGE 45

[ ]

a

d

b d

c

e

c

b

[a,b,c,d][a,b,c][a,c]

[ a,b][a,e] [a,d,e]

[a]

a

b

c

de

p2

end

p4

p3p1

start

event log

01011001101101001011111101101000110110011110111000001101101001001100

Example of State-Based Region

PAGE 46

[ ]

a

d

b d

c

e

c

b

[a,b,c,d][a,b,c][a,c]

[ a,b][a,e] [a,d,e]

[a]

a

b

c

de

p2

end

p4

p3p1

start

enter: b,eleave: ddo-not-cross: a,c

Example: Process Discovery UsingLanguage-Based Regions

PAGE 47

a1

a2

b1

b2

d

pR

e

c1

c

f

YX

A place is feasible if it can be added without disabling any of thetraces in the event log.

Example of Language-Based Regions

PAGE 48

b

a

d

e

c

YX

1. accd

2. bd

3. bce

4. ace

5. acd

6. bcce

7. ade

↓accd : 0 + 0 - 0 ≥ 0

a↓ccd : 0 + 1 - 1 ≥ 0

ac↓cd : 0 + 2 - 2 ≥ 0

acc↓d : 0 + 3 - 3 ≥ 0

↓ade : 0 + 0 - 0 ≥ 0

a↓de : 0 + 1 - 1 ≥ 0

ad↓e : 0 + 1 - 2 < 0

Example of a completely different process discovery technique: Genetic Mining

PAGE 49

http://www.foxnews.com/story/0,2933,316375,00.html?sPage=fnc/scitech/evolution

Genetic process mining: Overview

PAGE 50

next generationcomputefitness

elitism

parents

crossover

children

mutation

create initial population

“dead” individuals

tournament

select best individual

event log

termination

Example: crossover

PAGE 51

a


b

examine thoroughly

c

examine casually

d

check ticket

decide

pay compensation

reject request

reinitiate request

e

g

h

f

end

a


b

examine thoroughly

c

examine casually

d

check ticket

decide

pay compensation

reject request

reinitiate request

e

g

h

f

end

a


b

examine thoroughly

c

examine casually

d

check ticket

decide

reinitiate request

e

f

a


b

examine thoroughly

c

examine casually

d

check ticket

decide

pay compensation

reject request

reinitiate request

e

g

h

f

end

pay compensation

reject request

g

h

end

Example: mutation

PAGE 52

a


b

examine thoroughly

c

examine casually

d

check ticket

decide

pay compensation

reject request

reinitiate request

e

g

h

f

end

a


b

examine thoroughly

c

examine casually

d

check ticket

decide

pay compensation

reject request

reinitiate request

e

g

h

f

end

remove place

added arc

Conformance Checking

astart register

request

bexamine

thoroughly

cexamine casually

dcheck ticket

decide

pay compensation

reject request

reinitiate request

e

g

h

f

end

Replaying trace “abeg”

PAGE 54

m=1

r=1

a b e g

6

11

6= 0.83333

Can be lifted to log level

PAGE 55

astart register

request

bexamine

thoroughly

cexamine casually

dcheck ticket

decide

pay compensation

reject request

reinitiate request

e

g

h

f

end

p3p1

p2 p4

N1

p5

astart register

request

bexamine

thoroughly

cexamine casually

checkticket

decide

pay compensation

reject request

reinitiate request

e

g

hf

endp1 p2

N2

dp3

p4

astart register

request

cexamine casually

dcheckticket


e hend

p1

p2

p3

p4

p5

N3

astart register

request

bexamine thoroughly

cexamine casually

d checkticket

decide

pay compensation

reject request

reinitiate requeste

g

hf

end

N4

p1

acdeh

abdeg

adceh

abdeh

acdeg

adceg

adbeh

acdefdbeh

adbeg

acdefbdeh

acdefbdeg

acdefdbeg

adcefcdeh

adcefdbeh

adcefbdeg

acdefbdefdbeg

adcefdbeg

adcefbdefbdeg

adcefdbefbdeh

adbefbdefdbeg

adcefdbefcdefdbeg

455

191

177

144

111

82

56

47

38

33

14

11

9

8

5

3

2

2

1

1

1

# trace

1391

From “playing the token game” to optimal alignments …

a b » e g

a b d e g

PAGE 56

astart register

request

bexamine

thoroughly

cexamine casually

dcheck ticket

decide

pay compensation

reject request

reinitiate request

e

g

h

f

end

observed trace: “abeg”

move in model only

Another alignment

a b c d e g

a b » d e g

PAGE 57

astart register

request

bexamine

thoroughly

cexamine casually

dcheck ticket

decide

pay compensation

reject request

reinitiate request

e

g

h

f

end

observed trace: “abcdeg”

move in log only

Moves in an alignment

PAGE 58

a b » d e g

a » c d e g

trace in event log

possible run of model

move in log

move in model move in both

Optimal alignment describes modeled behavior closest to observed behavior

Moves have costs

• Standard cost function:

−c(x,») = 1

−c(»,y) = 1

−c(x,y) = 0, if x=y

−c(x,y) = ∞, if x≠yPAGE 59

… a …

… » …… » …

… a …

… a …

… a …

… a …

… b …

Non-fitting trace: abefdeg

PAGE 60

astart register

request

bexamine

thoroughly

cexamine casually

dcheck ticket

decide

pay compensation

reject request

reinitiate request

e

g

h

f

end abefdeg

a b » e f d » e g

a b d e f d b e g2

2a b e f d e g

a b » » d e g

Any cost structure is possible

PAGE 61

… send-letter(John,2 weeks, $400)

…

… send-email(Sue,3 weeks,$500)

…

• Similar activities (more similarity implies lower costs).• Resource conformance (done by someone that does

not have the specified role).• Data conformance (path is not possible for this

customer). • Time conformance (missed the legal deadline)

Fitness

PAGE 62

astart register

request

bexamine thoroughly

cexamine casually

d checkticket

decide

pay compensation

reject request

reinitiate requeste

g

hf

end

astart register

request

cexamine casually

dcheckticket


e hend



astart register

request

bexamine

thoroughly

cexamine casually

dcheck ticket

decide

pay compensation

reject request

reinitiate request

e

g

h

f

end


astart register

request

cexamine casually

dcheckticket


e hend


aregister request

dexamine casually

ccheckticket


e h

a cexamine casually

dcheckticket

decide

e g

a dexamine casually

ccheckticket

decide

e g

register request

register request

pay compensation

pay compensation

aregister request

b dcheckticket


e h

aregister request

d bcheckticket


e h

a b dcheckticket

decide

e gregister request

pay compensation

examine thoroughly

examine thoroughly

examine thoroughly


acdeh

abdeg

adceh

abdeh

acdeg

adceg

adbeh

acdefdbeh

adbeg

acdefbdeh

acdefbdeg

acdefdbeg

adcefcdeh

adcefdbeh

adcefbdeg

acdefbdefdbeg

adcefdbeg

adcefbdefbdeg

adcefdbefbdeh

adbefbdefdbeg

adcefdbefcdefdbeg

455

191

177

144

111

82

56

47

38

33

14

11

9

8

5

3

2

2

1

1

1

# trace

1391

1.0

0.8

1.0

1.0

Our A* algorithm exploits the Petri net marking equation and uses other “tricks” to prune the search space.

Aligned event log is starting point for other types of analysis.

Alignments are essential!•conformance checking to diagnose deviations•squeezing reality into the model to do model-based analysis

Food for Thought

We applied ProM in >100 organizations

PAGE 65

• Municipalities (e.g., Alkmaar, Heusden, Harderwijk, etc.)• Government agencies (e.g., Rijkswaterstaat, Centraal

Justitieel Incasso Bureau, Justice department)• Insurance related agencies (e.g., UWV)• Banks (e.g., ING Bank)• Hospitals (e.g., AMC hospital, Catharina hospital)• Multinationals (e.g., DSM, Deloitte)• High-tech system manufacturers and their customers

(e.g., Philips Healthcare, ASML, Ricoh, Thales)• Media companies (e.g. Winkwaves)• ...

How can process mining help?

PAGE 66

• Uncover bottlenecks• Detect deviations• Performance

measurement• Auditing/compliance• Business Process

Redesign (BPR)• Continuous improvement

(Six Sigma)• Operational support (e.g.,

recommendation and prediction)

• Provide new insights• Highlight important

problems• An organization’s

mirror (in two ways)• Helps to avoid ICT

failures• Avoid “management by

PowerPoint” • From “politics” to

“analytics”

Example of a Lasagna process: WMO process of a Dutch municipality

PAGE 68

Each line corresponds to one of the 528 requests that were handled in the period from 4-1-2009 until 28-2-2010. In total there are 5498 events represented as dots. The mean time needed to handled a case is approximately 25 days.

WMO process(Wet Maatschappelijke Ondersteuning)

• WMO refers to the social support act that came into force in The Netherlands on January 1st, 2007.

• The aim of this act is to assist people with disabilities and impairments. Under the act, local authorities are required to give support to those who need it, e.g., household help, providing wheelchairs and scootmobiles, and adaptations to homes.

• There are different processes for the different kinds of help. We focus on the process for handling requests for household help.

• In a period of about one year, 528 requests for household WMO support were received.

• These 528 requests generated 5498 events.PAGE 69

C-net discovered using heuristic miner (1/3)

PAGE 70


PAGE 71


PAGE 72

Conformance check WMO process (1/3)

PAGE 73


PAGE 74


PAGE 75

The fitness of the discovered process is 0.99521667. Of the 528 cases, 496 cases fit perfectly whereas for 32 cases there are missing or remaining tokens.

Bottleneck analysis WMO process (1/3)

PAGE 76


PAGE 77


PAGE 78

flow time of approx. 25 days with a standard deviation of approx. 28

Two additional Lasagna processes

PAGE 79

RWS (“Rijkswaterstaat”)

process

WOZ (“Waardering Onroerende Zaken”)

process

RWS Process

PAGE 80

• The Dutch national public works department, called “Rijkswaterstaat” (RWS), has twelve provincial offices. We analyzed the handling of invoices in one of these offices.

• The office employs about 1,000 civil servants and is primarily responsible for the construction and maintenance of the road and water infrastructure in its province.

• To perform its functions, the RWS office subcontracts various parties such as road construction companies, cleaning companies, and environmental bureaus. Also, it purchases services and products to support its construction, maintenance, and administrative activities.

C-net discovered using heuristic miner

PAGE 81

Social network constructed based on handovers of work

PAGE 82

Each of the 271 nodes corresponds to a civil servant. Two civil servants areconnected if one executed an activity causally following an activity executed by the other civil servant

Social network consisting of civil servants that executed more than 2000 activities in a 9 month period.

PAGE 83

The darker arcs indicate the strongest relationships in the social network. Nodes having the same color belong to the same clique.

WOZ process

• Event log containing information about 745 objections against the so-called WOZ (“Waardering Onroerende Zaken”) valuation.

• Dutch municipalities need to estimate the value of houses and apartments. The WOZ value is used as a basis for determining the real-estate property tax.

• The higher the WOZ value, the more tax the owner needs to pay. Therefore, there are many objections (i.e., appeals) of citizens that assert that the WOZ value is too high.

• “WOZ process” discovered for another municipality (i.e., different from the one for which we analyzed the WMO process).

PAGE 84

Discovered process model

PAGE 85

The log contains events related to 745 objections against the so-called WOZ valuation. These 745 objections generated 9583 events. There are 13 activities. For 12 of these activities both start and complete events are recorded. Hence, the WF-net has 25 transitions.

Conformance checker:(fitness is 0.98876214)

PAGE 86

Performance analysis

PAGE 87

bottleneck detection: places are colored based on average durations

information on total flow time

time required to move from one activity to another

Resource-activity matrix(four groups discovered)

PAGE 88

clique 2

clique 1

clique 3

clique 4

Example of a Spaghetti process

PAGE 90

Spaghetti process describing the diagnosis and treatment of 2765 patients in a Dutch hospital. The process model was constructed based on an event log containing 114,592 events. There are 619 different activities (taking event types into account) executed by 266 different individuals (doctors, nurses, etc.).

Fragment18 activities of the 619 activities (2.9%)

PAGE 91

Another example(event log of Dutch housing agency)

PAGE 92

The event log contains 208 cases that generated 5987 events. There are 74 different activities.

Google Maps and TomTom

Process models should be treated as electronic maps

PAGE 95

Business process movies

PAGE 96

Business information systems should offer “TomTom” functionality

PAGE 97

Predict: When will I be home? At 11.26!

Recommend: How to get home ASAP? Take a left turn!

Detect: You drive too fast!

How to get started?

Hundreds of plug-ins available covering the whole process mining spectrum

PAGE 99Download from: www.processmining.org

open-source (L-GPL)

How to Get Started?

Collect event data• Minimal requirement:

events referring to an activity name and a process instance.

• Good to have: timestamps, resource information, additional data elements.

• Challenges: scoping and sometimes correlation.

Collect questions

• What kind problems would you like to address (cost, time, risk, compliance, service, etc.)?

• Related to discovery, conformance, enhancement?

• Iterative process: can be “curiosity driven” initially.

PAGE 100

Conclusion

Conclusion

PAGE 102

www.processmining.org

www.win.tue.nl/ieeetfpm/

process mining: understanding and improving desire lines in big data

Business

reinitiaterequest177

p3 dstart

rejectcheck

dutch housing

pay payc

process discovery

wmo process

lognot underfitting