Download - The Art of Performance Evaluation
![Page 1: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/1.jpg)
The Art of Performance Evaluation
Yuto Hayamizu (The University of Tokyo)
2015/06/19 1
PGCon 2015 @ University of Ottawa
![Page 2: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/2.jpg)
Q: What is the role of IT?
2015/06/19 2
![Page 3: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/3.jpg)
The Role of Information Technology
• “Transforming” computing power into business/social values
3
Tabulating Machine (Herman Hollerith, 1980)
![Page 4: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/4.jpg)
Performance• The key criterion of “transformation”
efficiency
4
Tabulating Machine (Herman Hollerith, 1980)
National Population Census in U.S1880: 8 years (by humans)1890: 13 years (by humans, expected) → 18 months
![Page 5: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/5.jpg)
And Today
5Data Is The Source of Values
✓ Big Data✓ Internet of Things✓ Cyber Physical System
![Page 6: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/6.jpg)
Database System• Foundation of “transformation”• The value of an engineer
== Knowledge of how “transformation” works
6Hardware
App App App App
Database System
![Page 7: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/7.jpg)
Performance of Database Systems
• End-to-end study– From physics of hardware to logics of applications
• Learning every component technology is NOT enough– A component technology changes rapidly– An architecture could change drastically
• Acquire the way of learning how “transformation” works
2015/06/19 7
![Page 8: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/8.jpg)
Performance Evaluation• The key process to understand
performance of systems– Identify performance goals which contribute
to user value– Design and execute an evaluation procedure
2015/06/19 8
![Page 9: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/9.jpg)
Common Misconception
2015/06/19 9
![Page 10: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/10.jpg)
The Heart of Performance Evaluation Is Not Individual Skills
2015/06/19 10
Kernel probing and tracingHardware simulation
Statistical analysis
Workload characterizationWorkload generation
Factorial experiments design
Analytical modelingMetrics selection
Application monitoring
Middleware monitoringOS monitoringDB benchmarking
CPU benchmarking
Storage benchmarking
Memory benchmarking
Designing network
Designing storage
Integrating cloud environments
Kernel tuningDB tuning
Network monitoring
![Page 11: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/11.jpg)
But A Philosophy of Orchestrating Individual Skills
2015/06/19 11
Kernel probing and tracing
Hardware simulation
Statistical analysis
Workload characterizationWorkload generation
Factorial experiments design
Analytical modelingMetrics selection
Application monitoring
Middleware monitoring
OS monitoring
DB benchmarkingCPU benchmarking
Storage benchmarking
Memory benchmarking Designing networkDesigning storage
Kernel tuningDB tuning Network monitoring
![Page 12: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/12.jpg)
Example Story• “Find the best price/performance SSD for our
PostgreSQL database”
2015/06/19 12
List allpossible SSDs
Build test systemsfor all SSDs
Run pg_bench onall test systems
“XX tps/$ with SSD Z!”
![Page 13: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/13.jpg)
Smarter Way• “Find the best price/performance SSD for our
PostgreSQL system”
2015/06/19 13Characterize
workload
Model performancewith some metrics
Estimate performancewith datasheets
Validate estimationw/ pg_bench on several SSDs
Narrow downcandidate SSDs
Run pg_bench on candidatesand get the result
![Page 14: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/14.jpg)
Nature of Performance Evaluation
• A goal could change during evaluation– Progress of evaluation deepens insight into
target systems
• A philosophy will guide you
2015/06/19 14
![Page 15: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/15.jpg)
2015/06/19 15
Contrary to common belief, performance evaluation is an art. Like a work of art, successful evaluation cannot be produced mechanically. Every evaluation requires an intimate knowledge of the system being modeled and a careful selection of the methodology, workload, and tools.
— Raj Jain
![Page 16: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/16.jpg)
How can one develop his/her own philosophy?
2015/06/19 16
Learn principles and individual techniques
Put into practice andget experiences
![Page 17: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/17.jpg)
The First Step to Performance Evaluation
• Fundamentals of Performance Evaluation– Principles – Basic Techniques
• Modeling• Measurement• Simulation
... with my practices and experiences
2015/06/19 17
![Page 18: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/18.jpg)
Principles• Define a goal first– Understand the system and identify the
problem to be solved• The most important, and usually the most difficult
part
• Stay goal-oriented– Do not start with a tool or a technique– Evaluation could change a goal itself
• Redesign the process for a new goal
2015/06/19 18
Principles are so simple :)
![Page 19: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/19.jpg)
Basic Techniques
2015/06/19 19
Modeling
MeasurementSimulation
![Page 20: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/20.jpg)
Modeling
2015/06/19 20
Modeling
MeasurementSimulation
![Page 21: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/21.jpg)
Modeling• Building a model: approximation– Understand how it works– Select variables which effect performance– Ignore unnecessary details– Formulate performance
• The core of scientific approach
2015/06/19 21
![Page 22: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/22.jpg)
Example: Throughput of HDD
• Hard disk drive– Platters rotates at constant speed– A cylinder is selected by moving
heads• Important variables– Rotating speed: R [rpm]– Radius of a cylinder: r [m]– Density of a cylinder: D [byte/m]
• Performance model
2015/06/19 22
2πr × D × R / 60 [byte/sec]
![Page 23: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/23.jpg)
Seq. Read Throughput of HDD
2015/06/19 23
Outer platter Inner platter
![Page 24: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/24.jpg)
Example: Latency of HDD• Important variables– Rotating speed: R [rpm]– Avg. rotational latency: Δt = 60/R [sec]– Max seek time: ts [sec]
• Latency model:
2015/06/19 24
ts < (latency) < ts+Δt [sec]
![Page 25: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/25.jpg)
I/O Latency of HDD• a
2015/06/19 25
~ 6ms (rotational latency of 10k)
![Page 26: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/26.jpg)
Queueing Model• Simple queueing model– Avg. arrival rate of customer: λ– Avg. service latency: µ
• Modeling computer system performance– CPU : customer = CPU instructions, server =
execution units in CPU– Storage: customer = I/O requests, server = disks
and controllers– Database system: customer = queries, server =
query executors– ...
2015/06/19 26
![Page 27: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/27.jpg)
Performance Modeling with Queueing Model
• M/M/1 : simple but useful model– 1 server and 1 infinite queue– Customer arrives at rate λ according to
Poission process
2015/06/19 27
Average response timeλ
μ
Waiting time
Response time (sojourn time)
![Page 28: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/28.jpg)
Modeling OLTP Performance with Various CPU Frequencies
•
28
Theoretical response time Measured response time
![Page 29: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/29.jpg)
Modeling Deepens Understanding
• A good model provides good view– Performance values– Scaling trends– Highlight bugs
• A bad model is also informative– Missing key variables– Complexity of systems
2015/06/19 29
![Page 30: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/30.jpg)
Measurement
2015/06/19 30
Modeling
MeasurementSimulation
![Page 31: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/31.jpg)
Measurement• Measurement without modeling is
pointless– Modeling validates measurement– Measurement validates modeling
2015/06/19 31
Model
![Page 32: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/32.jpg)
Guide For Fruitful Measurement
• Selecting workload– Does it characterize your application well?
• Selecting performance metrics– Which metrics define your goal?
• Modeling performance• Selecting environments
– Is it designed for measuring your metrics?• Selecting measurement methods and tools• Conduct measurements• Validating results with models
2015/06/19 32
Goalden rule: Decide methods and tools based on your goal
Measurement != just picking up a benchmark tool and run it
![Page 33: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/33.jpg)
Example story: Storage for OLTP System
• Goal– Build a storage system for OLTP system which
can attain XXXX tps
• Workload– Random I/O from many clients
• Metric– IOPS of storage system (random access)
2015/06/19 33
![Page 34: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/34.jpg)
Example story: Storage for OLTP System
• Modeling performance– What affects your metrics?– IOPS (random access)
• I/O block size: 8KB ~ 16KB• I/O concurrency: 1 ~ 1000• The length of I/O queue in:
– I/O scheduler of OS– HBA driver– Storage controller– Storage device
2015/06/19 34
![Page 35: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/35.jpg)
Example story: Storage for OLTP System
• Measurement environment– Carefully design the flow of resources– Your metrics: IOPS of the storage system
• Ensure whether I/O requests really reach the storage system
2015/06/19 35
Server Storage
Proc
esso
rs
Memo
ry
Stor
age C
ontro
ller
(with
cach
e)
![Page 36: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/36.jpg)
Example story: Storage for OLTP System
• Measurement methods and tools– Select an appropriate tool or make it yourself
• What should be measured?• What can be measured?
– How measured value represent your metric?• Ex) iostat value != IOPS of storage devices
2015/06/19 36
Server Storage
Proc
esso
rs
Memo
ry
Stor
age C
ontro
ller
(with
cach
e)
![Page 37: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/37.jpg)
Example story: Storage for OLTP System
• Conduct measurement– Plan your experiments with minimal steps
• Exhaustive parameter combinations ...?• A good model tells you where should be measured
2015/06/19 37
Next measurement should be ...
![Page 38: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/38.jpg)
Example story: Storage for OLTP System
• Validate measurement results with your model– Results match the model
• Right understanding of system performance
– Results do not match the model• Missing something• Improve your model or
measurement method
2015/06/19 38
![Page 39: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/39.jpg)
Simulation
2015/06/19 39
Modeling
MeasurementSimulation
![Page 40: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/40.jpg)
Simulation• Simulating unavailable systems– Coming hardware devices– Unimplemented software components
• Useful technique for– More detailed behaviour analysis than
modeling
2015/06/19 40
![Page 41: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/41.jpg)
I/O Replay• Performance evaluation with realistic I/O
workload– Trace I/O requests during actual execution– Replay traced I/O patters on hypothetical systems
2015/06/19 41TPC-C benchmark (SF=100)
![Page 42: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/42.jpg)
I/O Replay (TPC-C) on Various Devices
2015/06/19 42
HDD
SATA SSD
PCIe SSD
![Page 43: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/43.jpg)
Tips on Simulation• Technique of what-if analysis based on
some assumptions– Underlying models, workload generation
(random number distribution), omitted details, ...
– Only proper assumptions can lead to meaningful results
• Validating assumptions and results with modeling/measurement is important
2015/06/19 43
![Page 44: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/44.jpg)
Basic Techniques
2015/06/19 44
Modeling
MeasurementSimulation Validate
![Page 45: The Art of Performance Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022032503/55c03316bb61eb85148b45d1/html5/thumbnails/45.jpg)
Wrap Up• Why performance evaluation matters– The key process to understand systems
• Stay goal-oriented• Use basic techniques effectively– Modeling, measurement and simulation– Validate each other
• Experiences develop your philosophy
2015/06/19 45