fighting groundhog days: expert-guided automatic diagnosis of performance problems in enterprise...

Fighting Groundhog Days:

Expert-Guided Automatic Diagnosis of

Performance Problems in Enterprise Applications

Symposium on Software Performance (SSP)

November 05, 2015 @ Munich

Christoph Heger, Dušan Okanović, Stefan Siegl, André van Hoorn and Alexander Wert

@diagnoseIT_apm https://diagnoseIT.github.io

(htt

p:/

/go

o.g

l/Q

3O

Zp

A)

2/ 2015/11/05

Performance Problems are Omnipresent

Fighting Groundhog Days: Expert-Guided Automatic Diagnosis of


An unexpected

error occured

Temporarily

not available

… more capacity

is on the way

Try again later

Please visit us

again later

We are

experiencing

heavy demand

3/ 2015/11/05

Manual Diagnosis



Adapt

instrumentation Component

Deep-Dive

Interpretation of

measurements

Detect problem

4/ 2015/11/05

Manual Configuration



Adapt

instrumentation

5/ 2015/11/05

Manual Reasoning



Component

Deep-Dive

Interpretation of

measurements

6/ 2015/11/05

diagnoseIT Goals



Reduce maintainance effort by 50%

Detect 100% of all problems automatically

Identify 80% of the root causes automatically

No APM vendor lock-in

7/ 2015/11/05

Overview



Node 1

JVM

App 1

App 2

Node n

SUT

Traces

CTA

Instrumentation

Refinement

Request

Instrumentation

Language Result

Result Description

Language

Monitoring Tool

Dynamic

Instrumentation

API Labelling

Location

Identification

Instrumentation

Quality Manager

Result Query

8/ 2015/11/05

Diagnosis



Trace

Traces

CTA

Rule

Retrieve

information from

Notify on tagging

Problem

Instance

Repository

Manage

Tag

Instrumentation Refinement Request

Instrumentation

Language

Insight

Enrichment

Information

Request

Maintain

Problem

Instance

Problem

Instance

Create/Update

Result Description

Language

Mo

nito

rin

g T

oo

l

9/ 2015/11/05

diagnoseIT Characteristics



Adaptive Configuration

Continuous Learning

APM Process Automation

Formalized Expert

Knowledge

10/ 2015/11/05

Rules



Root

Cause

Problem

Context

Global

Context

…

Root Cause

Tag

Problem

Context Tag

Global

Context Tag

…

…

…

Isolate

Root Cause

Semantify

Root Cause

Generic Analysis

Domain- and Technology-

specific Analysis

11/ 2015/11/05

DVDStore Example



doFilter(…) 47.67 186,259.50

searchTitleAndDescription(…) 22,961.16 184,897.25

searchTitleAndDescriptionWithOneWord(…) 0.14 158,054.08

list(…) 152,620,13 158,036.29

executeQuery() 1,793.75 1,793.75

executeQuery() 0.36 0.36

…



list(…) 2,625.22 3,881.70

executeQuery() 1,257.47 1,257.47 {SQL: Select PROD_ID …}

{SQL: Select PROD_ID …}

Real Trace

(Simplified)

{SQL: Select INV_ID …}


Response Time

[ms]

Exclusive Execution

Time [ms]

12/ 2015/11/05

Root Cause Isolation



list(…) 152,620,13 158,036.29


list(…) 2,625.22 3,881.70

executeQuery() 1,793.75 1,793.75

executeQuery() 1,257.47 1,257.47

doFilter(…) 47.67 186,259.50





Real Trace

(Simplified)

Response Time

[ms]

Exclusive Execution

Time [ms]





Descending Order

Cause

Cause

Cause

Cause

Cause

13/ 2015/11/05

Problem Instance 1



C P

doFilter(…) 47.67 186,259.50



list(…) 152,620,13 158,036.29

executeQuery() 1,793.75 1,793.75


…



list(…) 2,625.22 3,881.70


Cause


Real Trace

(Simplified)

Problem Context



Response Time

[ms]

Exclusive Execution

Time [ms]

14/ 2015/11/05

Problem Instance 2



C P

doFilter(…) 47.67 186,259.50



list(…) 152,620,13 158,036.29

executeQuery() 1,793.75 1,793.75


…



list(…) 2,625.22 3,881.70

executeQuery() 1,257.47 1,257.47

Cause


Cause


Real Trace

(Simplified)

Problem Context



Response Time

[ms]

Exclusive Execution

Time [ms]

15/ 2015/11/05

Problem Instance 3



C P

Response Time

[ms]

doFilter(…) 47.67 186,259.50



list(…) 152,620,13 158,036.29

executeQuery() 1,793.75 1,793.75


…



list(…) 2,625.22 3,881.70

executeQuery() 1,257.47 1,257.47 Cause {SQL: Select PROD_ID …}

Cause {SQL: Select PROD_ID …}

Real Trace

(Simplified)

Problem Context



Exclusive Execution

Time [ms]

16/ 2015/11/05

Global Context



Response Time

[ms]

doFilter(…) 47.67 186,259.50



list(…) 152,620,13 158,036.29

executeQuery() 1,793.75 1,793.75


…



list(…) 2,625.22 3,881.70



Real Trace

(Simplified)



Exclusive Execution

Time [ms]

3x Problem Context Global Context

17/ 2015/11/05

diagnoseIT Prototype



Problems

Overview

Natural language

description

Problem instance

details

18/ 2015/11/05

diagnoseIT Prototype



Problem

Highlighting

19/ 2015/11/05

Common Trace API



https://goo.gl/UuoZkN

Join us

APM vendor independent data access

Goal

Monitoring data format is tool-specific

Motivation

APM vendor lock-in limits

interoperability and data exchange

Drawback

Importers

Planned

(http://goo.gl/5S337v)

(http://goo.gl/AAwdge)

(https://goo.gl/eDuA9R)

(http://goo.gl/3xM2ur) (http://goo.gl/KCWimU)

(http://goo.gl/z18Tuj) (http://goo.gl/B3rJt4)

20/ 2015/11/05

diagnoseIT-as-a-Service



Evaluation of diagnosis rules

Motivation

Diagnosis service for traces

Idea

Collection of real traces and problems

Benefit

Support us

https://ditaas.inspectit.rocks

21/ 2015/11/05

Future Steps



Anti-Pattern Detection

Overhead-aware Instrumentation

Case Studies System

measures

22/ 2015/11/05

Case Studies Outlook



Level of Distribution

Le

ve

l of C

om

ple

xity

DVDStore

(http://goo.gl/dPc0YQ)

(http://goo.gl/yDHkW6)

(http://goo.gl/8PnaqE)

(https://goo.gl/oqvy4K)

23/ 2015/11/05

Conclusion



Demo Bundle

https://goo.gl/M1VQA9 @diagnoseIT_apm https://diagnoseIT.github.io

Problem Detection

Automatic Diagnosis

APM Process

Automation

No APM vendor lock-in