the art of performance evaluation

The Art of Performance Evaluation

Yuto Hayamizu (The University of Tokyo)

2015/06/19 1

PGCon 2015 @ University of Ottawa

Q: What is the role of IT?

2015/06/19 2

The Role of Information Technology

•  “Transforming” computing power into business/social values

3

Tabulating Machine (Herman Hollerith, 1980)

Performance•  The key criterion of “transformation”

efficiency

4

Tabulating Machine (Herman Hollerith, 1980)

National Population Census in U.S1880: 8 years (by humans)1890: 13 years (by humans, expected) → 18 months

And Today

5Data Is The Source of Values

✓ Big Data✓ Internet of Things✓ Cyber Physical System

Database System•  Foundation of “transformation”•  The value of an engineer

== Knowledge of how “transformation” works

6Hardware

App App App App

Database System

Performance of Database Systems

•  End-to-end study–  From physics of hardware to logics of applications

•  Learning every component technology is NOT enough– A component technology changes rapidly– An architecture could change drastically

•  Acquire the way of learning how “transformation” works

2015/06/19 7

Performance Evaluation•  The key process to understand

performance of systems–  Identify performance goals which contribute

to user value– Design and execute an evaluation procedure

2015/06/19 8

Common Misconception

2015/06/19 9

The Heart of Performance Evaluation Is Not Individual Skills

2015/06/19 10

Kernel probing and tracingHardware simulation

Statistical analysis

Workload characterizationWorkload generation

Factorial experiments design

Analytical modelingMetrics selection

Application monitoring

Middleware monitoringOS monitoringDB benchmarking

CPU benchmarking

Storage benchmarking

Memory benchmarking

Designing network

Designing storage

Integrating cloud environments

Kernel tuningDB tuning

Network monitoring

But A Philosophy of Orchestrating Individual Skills

2015/06/19 11

Kernel probing and tracing

Hardware simulation

Statistical analysis

Workload characterizationWorkload generation

Factorial experiments design

Analytical modelingMetrics selection

Application monitoring

Middleware monitoring

OS monitoring

DB benchmarkingCPU benchmarking

Storage benchmarking

Memory benchmarking Designing networkDesigning storage

Kernel tuningDB tuning Network monitoring

Example Story•  “Find the best price/performance SSD for our

PostgreSQL database”

2015/06/19 12

List allpossible SSDs

Build test systemsfor all SSDs

Run pg_bench onall test systems

“XX tps/$ with SSD Z!”

Smarter Way•  “Find the best price/performance SSD for our

PostgreSQL system”

2015/06/19 13Characterize

workload

Model performancewith some metrics

Estimate performancewith datasheets

Validate estimationw/ pg_bench on several SSDs

Narrow downcandidate SSDs

Run pg_bench on candidatesand get the result

Nature of Performance Evaluation

•  A goal could change during evaluation– Progress of evaluation deepens insight into

target systems

•  A philosophy will guide you

2015/06/19 14

2015/06/19 15

Contrary to common belief, performance evaluation is an art. Like a work of art, successful evaluation cannot be produced mechanically. Every evaluation requires an intimate knowledge of the system being modeled and a careful selection of the methodology, workload, and tools.

— Raj Jain

How can one develop his/her own philosophy?

2015/06/19 16

Learn principles and individual techniques

Put into practice andget experiences

The First Step to Performance Evaluation

•  Fundamentals of Performance Evaluation– Principles – Basic Techniques

•  Modeling•  Measurement•  Simulation

... with my practices and experiences

2015/06/19 17

Principles•  Define a goal first– Understand the system and identify the

problem to be solved•  The most important, and usually the most difficult

part

•  Stay goal-oriented– Do not start with a tool or a technique– Evaluation could change a goal itself

•  Redesign the process for a new goal

2015/06/19 18

Principles are so simple :)

Basic Techniques

2015/06/19 19

Modeling

MeasurementSimulation

Modeling

2015/06/19 20

Modeling

MeasurementSimulation

Modeling•  Building a model: approximation– Understand how it works– Select variables which effect performance–  Ignore unnecessary details– Formulate performance

•  The core of scientific approach

2015/06/19 21

Example: Throughput of HDD

•  Hard disk drive– Platters rotates at constant speed– A cylinder is selected by moving

heads•  Important variables– Rotating speed: R [rpm]– Radius of a cylinder: r [m]– Density of a cylinder: D [byte/m]

•  Performance model

2015/06/19 22

2πr × D × R / 60 [byte/sec]

Seq. Read Throughput of HDD

2015/06/19 23

Outer platter Inner platter

Example: Latency of HDD•  Important variables– Rotating speed: R [rpm]– Avg. rotational latency: Δt = 60/R [sec]– Max seek time: ts [sec]

•  Latency model:

2015/06/19 24

ts < (latency) < ts+Δt [sec]

I/O Latency of HDD•  a

2015/06/19 25

~ 6ms (rotational latency of 10k)

Queueing Model•  Simple queueing model– Avg. arrival rate of customer: λ– Avg. service latency: µ

•  Modeling computer system performance– CPU : customer = CPU instructions, server =

execution units in CPU–  Storage: customer = I/O requests, server = disks

and controllers– Database system: customer = queries, server =

query executors–  ...

2015/06/19 26

Performance Modeling with Queueing Model

•  M/M/1 : simple but useful model– 1 server and 1 infinite queue– Customer arrives at rate λ according to

Poission process

2015/06/19 27

Average response timeλ

μ

Waiting time

Response time (sojourn time)

Modeling OLTP Performance with Various CPU Frequencies

• 

28

Theoretical response time Measured response time

Modeling Deepens Understanding

•  A good model provides good view– Performance values– Scaling trends– Highlight bugs

•  A bad model is also informative– Missing key variables– Complexity of systems

2015/06/19 29

Measurement

2015/06/19 30

Modeling

MeasurementSimulation

Measurement•  Measurement without modeling is

pointless– Modeling validates measurement– Measurement validates modeling

2015/06/19 31

Model

Guide For Fruitful Measurement

•  Selecting workload–  Does it characterize your application well?

•  Selecting performance metrics–  Which metrics define your goal?

•  Modeling performance•  Selecting environments

–  Is it designed for measuring your metrics?•  Selecting measurement methods and tools•  Conduct measurements•  Validating results with models

2015/06/19 32

Goalden rule: Decide methods and tools based on your goal

Measurement != just picking up a benchmark tool and run it

Example story: Storage for OLTP System

•  Goal– Build a storage system for OLTP system which

can attain XXXX tps

•  Workload– Random I/O from many clients

•  Metric–  IOPS of storage system (random access)

2015/06/19 33

Example story: Storage for OLTP System

•  Modeling performance– What affects your metrics?–  IOPS (random access)

•  I/O block size: 8KB ~ 16KB•  I/O concurrency: 1 ~ 1000•  The length of I/O queue in:

–  I/O scheduler of OS– HBA driver–  Storage controller–  Storage device

2015/06/19 34

Example story: Storage for OLTP System

•  Measurement environment– Carefully design the flow of resources– Your metrics: IOPS of the storage system

•  Ensure whether I/O requests really reach the storage system

2015/06/19 35

Server Storage

Proc

esso

rs

Memo

ry

Stor

age C

ontro

ller

(with

cach

e)

Example story: Storage for OLTP System

•  Measurement methods and tools–  Select an appropriate tool or make it yourself

•  What should be measured?•  What can be measured?

–  How measured value represent your metric?•  Ex) iostat value != IOPS of storage devices

2015/06/19 36

Server Storage

Proc

esso

rs

Memo

ry

Stor

age C

ontro

ller

(with

cach

e)

Example story: Storage for OLTP System

•  Conduct measurement– Plan your experiments with minimal steps

•  Exhaustive parameter combinations ...?•  A good model tells you where should be measured

2015/06/19 37

Next measurement should be ...

Example story: Storage for OLTP System

•  Validate measurement results with your model– Results match the model

•  Right understanding of system performance

– Results do not match the model•  Missing something•  Improve your model or

measurement method

2015/06/19 38

Simulation

2015/06/19 39

Modeling

MeasurementSimulation

Simulation•  Simulating unavailable systems– Coming hardware devices– Unimplemented software components

•  Useful technique for– More detailed behaviour analysis than

modeling

2015/06/19 40

I/O Replay•  Performance evaluation with realistic I/O

workload– Trace I/O requests during actual execution– Replay traced I/O patters on hypothetical systems

2015/06/19 41TPC-C benchmark (SF=100)

I/O Replay (TPC-C) on Various Devices

2015/06/19 42

HDD

SATA SSD

PCIe SSD

Tips on Simulation•  Technique of what-if analysis based on

some assumptions– Underlying models, workload generation

(random number distribution), omitted details, ...

– Only proper assumptions can lead to meaningful results

•  Validating assumptions and results with modeling/measurement is important

2015/06/19 43

Basic Techniques

2015/06/19 44

Modeling

MeasurementSimulation Validate

Wrap Up•  Why performance evaluation matters– The key process to understand systems

•  Stay goal-oriented•  Use basic techniques effectively– Modeling, measurement and simulation– Validate each other

•  Experiences develop your philosophy

2015/06/19 45