survey of web performance analysisprincetonacm.acm.org/tcfpro/web_performance.pdf · two different...
TRANSCRIPT
![Page 1: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/1.jpg)
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
copyright 2009 Guy Ferraiolosome rights reserved
This work is licensed under the Creative Commons Attribution-No Derivative Works 30 License To view a copy of this license visit httpcreativecommonsorglicensesby-nd30 or send a letter to Creative Commons 171 Second Street Suite 300 San Francisco California 94105 USA
Introduction
Who am I
Consultant Principal Software Engineer 8 years at a major internet presence 30 years in the software industry
Ran performance testing effort two sophisticated performance test labs
Committer and contributor to Apache Software Foundation
My Environment
In a performance group
Work with engineering organizations to facilitate and empower their performance analysis efforts
Responsibilities clearly divided among organizations
tiered architecture mostly
Approach of This Talk
Find a conceptual basis
the task is complex
Provide some advice based on experience
Focus of This Talk
Survey only much more information is available
Primarily concepts some discussion of tools
Heuristics and work processes
General using web examples HTTP
Will Provide
Definite advice
Amusing anecdotes
Some tool recommendations
Wonrsquot Provide
Magic bullets
Simple answers
Performance Analysis
Goals
Know what performance results are
Understand reasons for performance results
Save money
Save time
Two Different Web Performance Aspects
Page delivery time
Resource cost to deliver a page
Not entirely unrelated but commonly confused
Why Performance Matters
Poor performance means less traffic and therefore less revenue
This relationship is surprizingly strong perhaps 20 less traffic for an additional 01 seconds
Infrastructure is very expensive
Why Resource Cost Matters
A highly dynamic page may be very lightweight but require enormous effort to assemble - the example is Google
Even if a page is fast to deliver the cost of the infrastructure to create it can be a major business problem
Performance Analysis
is primarily analytic
but testing is essential
works best when integrated into the development process
Performance Analysis Consists of
Detecting Performance Issues
some sort of monitoring is needed
Localizing Performance Issues
usually done with analysis
Resolving Performance Issues
often easy but testing can be critical
Performance Analysis
is not code monkey work
is not fast
is not highly rigorous
Types of Performance Analysis
Capacity planning
Acceptance testing
Design and algorithm selection
Good amp Bad News
Meaningful results can be obtained
Results sometimes not expected
Requires a serious investment
Is inherently iterative
Where to Start
A hierarchy of
concepts - most important
policies - derived from concepts
tools - necessary but not dominant
Concepts
The Big Concept
Reduction of Uncertainty
overarching concept
provides answers to why what and do I care
How to Apply the Big Concept
Relate performance to business needs
provide enough information to allow management to make meaningful business decisions
gain enough information to allow extrapolation of trends
Other Concepts
Intuition is likely to be wrong
The process is inherently iterative
business process
technical process
An early start is very helpful
Policies
Policies
Analyze and test enough to make a business decision
Have a clearly defined process
Define your purpose
Make a quantitative estimate of the traffic
Let those who know something do it
More Policies
Communicate openly and often
Compartmentalize responsibility
Do not use performance testing to expose functional problems this is not QA (quality assurance)
Processes
Process Attributes
Defined
Documented
Public
Includes pre and post testing
Pre-Test Process
Assign responsibilities be definite about who
Declare goals
Record configuration
Assess traffic quantity and type
Post-Test Process
Meet definite end criteria
Report results
Make recommendations
Communicate issues
Test Process
Dynamic goal seeking
Interactive
Iterative
Duration depends on behavior
Test Options
Live or lab
End-to-end or component
The Most Common Error
Unjustified belief that the problem is code related usually architecture data traffic
Common Pitfalls
Unclear goals
N-dimensional matrix of tests
Emphasis on the tool
Emphasis on the code
More Common Pitfalls
Focus on a single tool for testing
Belief in a single answer to the problem
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 2: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/2.jpg)
Introduction
Who am I
Consultant Principal Software Engineer 8 years at a major internet presence 30 years in the software industry
Ran performance testing effort two sophisticated performance test labs
Committer and contributor to Apache Software Foundation
My Environment
In a performance group
Work with engineering organizations to facilitate and empower their performance analysis efforts
Responsibilities clearly divided among organizations
tiered architecture mostly
Approach of This Talk
Find a conceptual basis
the task is complex
Provide some advice based on experience
Focus of This Talk
Survey only much more information is available
Primarily concepts some discussion of tools
Heuristics and work processes
General using web examples HTTP
Will Provide
Definite advice
Amusing anecdotes
Some tool recommendations
Wonrsquot Provide
Magic bullets
Simple answers
Performance Analysis
Goals
Know what performance results are
Understand reasons for performance results
Save money
Save time
Two Different Web Performance Aspects
Page delivery time
Resource cost to deliver a page
Not entirely unrelated but commonly confused
Why Performance Matters
Poor performance means less traffic and therefore less revenue
This relationship is surprizingly strong perhaps 20 less traffic for an additional 01 seconds
Infrastructure is very expensive
Why Resource Cost Matters
A highly dynamic page may be very lightweight but require enormous effort to assemble - the example is Google
Even if a page is fast to deliver the cost of the infrastructure to create it can be a major business problem
Performance Analysis
is primarily analytic
but testing is essential
works best when integrated into the development process
Performance Analysis Consists of
Detecting Performance Issues
some sort of monitoring is needed
Localizing Performance Issues
usually done with analysis
Resolving Performance Issues
often easy but testing can be critical
Performance Analysis
is not code monkey work
is not fast
is not highly rigorous
Types of Performance Analysis
Capacity planning
Acceptance testing
Design and algorithm selection
Good amp Bad News
Meaningful results can be obtained
Results sometimes not expected
Requires a serious investment
Is inherently iterative
Where to Start
A hierarchy of
concepts - most important
policies - derived from concepts
tools - necessary but not dominant
Concepts
The Big Concept
Reduction of Uncertainty
overarching concept
provides answers to why what and do I care
How to Apply the Big Concept
Relate performance to business needs
provide enough information to allow management to make meaningful business decisions
gain enough information to allow extrapolation of trends
Other Concepts
Intuition is likely to be wrong
The process is inherently iterative
business process
technical process
An early start is very helpful
Policies
Policies
Analyze and test enough to make a business decision
Have a clearly defined process
Define your purpose
Make a quantitative estimate of the traffic
Let those who know something do it
More Policies
Communicate openly and often
Compartmentalize responsibility
Do not use performance testing to expose functional problems this is not QA (quality assurance)
Processes
Process Attributes
Defined
Documented
Public
Includes pre and post testing
Pre-Test Process
Assign responsibilities be definite about who
Declare goals
Record configuration
Assess traffic quantity and type
Post-Test Process
Meet definite end criteria
Report results
Make recommendations
Communicate issues
Test Process
Dynamic goal seeking
Interactive
Iterative
Duration depends on behavior
Test Options
Live or lab
End-to-end or component
The Most Common Error
Unjustified belief that the problem is code related usually architecture data traffic
Common Pitfalls
Unclear goals
N-dimensional matrix of tests
Emphasis on the tool
Emphasis on the code
More Common Pitfalls
Focus on a single tool for testing
Belief in a single answer to the problem
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 3: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/3.jpg)
Who am I
Consultant Principal Software Engineer 8 years at a major internet presence 30 years in the software industry
Ran performance testing effort two sophisticated performance test labs
Committer and contributor to Apache Software Foundation
My Environment
In a performance group
Work with engineering organizations to facilitate and empower their performance analysis efforts
Responsibilities clearly divided among organizations
tiered architecture mostly
Approach of This Talk
Find a conceptual basis
the task is complex
Provide some advice based on experience
Focus of This Talk
Survey only much more information is available
Primarily concepts some discussion of tools
Heuristics and work processes
General using web examples HTTP
Will Provide
Definite advice
Amusing anecdotes
Some tool recommendations
Wonrsquot Provide
Magic bullets
Simple answers
Performance Analysis
Goals
Know what performance results are
Understand reasons for performance results
Save money
Save time
Two Different Web Performance Aspects
Page delivery time
Resource cost to deliver a page
Not entirely unrelated but commonly confused
Why Performance Matters
Poor performance means less traffic and therefore less revenue
This relationship is surprizingly strong perhaps 20 less traffic for an additional 01 seconds
Infrastructure is very expensive
Why Resource Cost Matters
A highly dynamic page may be very lightweight but require enormous effort to assemble - the example is Google
Even if a page is fast to deliver the cost of the infrastructure to create it can be a major business problem
Performance Analysis
is primarily analytic
but testing is essential
works best when integrated into the development process
Performance Analysis Consists of
Detecting Performance Issues
some sort of monitoring is needed
Localizing Performance Issues
usually done with analysis
Resolving Performance Issues
often easy but testing can be critical
Performance Analysis
is not code monkey work
is not fast
is not highly rigorous
Types of Performance Analysis
Capacity planning
Acceptance testing
Design and algorithm selection
Good amp Bad News
Meaningful results can be obtained
Results sometimes not expected
Requires a serious investment
Is inherently iterative
Where to Start
A hierarchy of
concepts - most important
policies - derived from concepts
tools - necessary but not dominant
Concepts
The Big Concept
Reduction of Uncertainty
overarching concept
provides answers to why what and do I care
How to Apply the Big Concept
Relate performance to business needs
provide enough information to allow management to make meaningful business decisions
gain enough information to allow extrapolation of trends
Other Concepts
Intuition is likely to be wrong
The process is inherently iterative
business process
technical process
An early start is very helpful
Policies
Policies
Analyze and test enough to make a business decision
Have a clearly defined process
Define your purpose
Make a quantitative estimate of the traffic
Let those who know something do it
More Policies
Communicate openly and often
Compartmentalize responsibility
Do not use performance testing to expose functional problems this is not QA (quality assurance)
Processes
Process Attributes
Defined
Documented
Public
Includes pre and post testing
Pre-Test Process
Assign responsibilities be definite about who
Declare goals
Record configuration
Assess traffic quantity and type
Post-Test Process
Meet definite end criteria
Report results
Make recommendations
Communicate issues
Test Process
Dynamic goal seeking
Interactive
Iterative
Duration depends on behavior
Test Options
Live or lab
End-to-end or component
The Most Common Error
Unjustified belief that the problem is code related usually architecture data traffic
Common Pitfalls
Unclear goals
N-dimensional matrix of tests
Emphasis on the tool
Emphasis on the code
More Common Pitfalls
Focus on a single tool for testing
Belief in a single answer to the problem
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 4: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/4.jpg)
My Environment
In a performance group
Work with engineering organizations to facilitate and empower their performance analysis efforts
Responsibilities clearly divided among organizations
tiered architecture mostly
Approach of This Talk
Find a conceptual basis
the task is complex
Provide some advice based on experience
Focus of This Talk
Survey only much more information is available
Primarily concepts some discussion of tools
Heuristics and work processes
General using web examples HTTP
Will Provide
Definite advice
Amusing anecdotes
Some tool recommendations
Wonrsquot Provide
Magic bullets
Simple answers
Performance Analysis
Goals
Know what performance results are
Understand reasons for performance results
Save money
Save time
Two Different Web Performance Aspects
Page delivery time
Resource cost to deliver a page
Not entirely unrelated but commonly confused
Why Performance Matters
Poor performance means less traffic and therefore less revenue
This relationship is surprizingly strong perhaps 20 less traffic for an additional 01 seconds
Infrastructure is very expensive
Why Resource Cost Matters
A highly dynamic page may be very lightweight but require enormous effort to assemble - the example is Google
Even if a page is fast to deliver the cost of the infrastructure to create it can be a major business problem
Performance Analysis
is primarily analytic
but testing is essential
works best when integrated into the development process
Performance Analysis Consists of
Detecting Performance Issues
some sort of monitoring is needed
Localizing Performance Issues
usually done with analysis
Resolving Performance Issues
often easy but testing can be critical
Performance Analysis
is not code monkey work
is not fast
is not highly rigorous
Types of Performance Analysis
Capacity planning
Acceptance testing
Design and algorithm selection
Good amp Bad News
Meaningful results can be obtained
Results sometimes not expected
Requires a serious investment
Is inherently iterative
Where to Start
A hierarchy of
concepts - most important
policies - derived from concepts
tools - necessary but not dominant
Concepts
The Big Concept
Reduction of Uncertainty
overarching concept
provides answers to why what and do I care
How to Apply the Big Concept
Relate performance to business needs
provide enough information to allow management to make meaningful business decisions
gain enough information to allow extrapolation of trends
Other Concepts
Intuition is likely to be wrong
The process is inherently iterative
business process
technical process
An early start is very helpful
Policies
Policies
Analyze and test enough to make a business decision
Have a clearly defined process
Define your purpose
Make a quantitative estimate of the traffic
Let those who know something do it
More Policies
Communicate openly and often
Compartmentalize responsibility
Do not use performance testing to expose functional problems this is not QA (quality assurance)
Processes
Process Attributes
Defined
Documented
Public
Includes pre and post testing
Pre-Test Process
Assign responsibilities be definite about who
Declare goals
Record configuration
Assess traffic quantity and type
Post-Test Process
Meet definite end criteria
Report results
Make recommendations
Communicate issues
Test Process
Dynamic goal seeking
Interactive
Iterative
Duration depends on behavior
Test Options
Live or lab
End-to-end or component
The Most Common Error
Unjustified belief that the problem is code related usually architecture data traffic
Common Pitfalls
Unclear goals
N-dimensional matrix of tests
Emphasis on the tool
Emphasis on the code
More Common Pitfalls
Focus on a single tool for testing
Belief in a single answer to the problem
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 5: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/5.jpg)
Approach of This Talk
Find a conceptual basis
the task is complex
Provide some advice based on experience
Focus of This Talk
Survey only much more information is available
Primarily concepts some discussion of tools
Heuristics and work processes
General using web examples HTTP
Will Provide
Definite advice
Amusing anecdotes
Some tool recommendations
Wonrsquot Provide
Magic bullets
Simple answers
Performance Analysis
Goals
Know what performance results are
Understand reasons for performance results
Save money
Save time
Two Different Web Performance Aspects
Page delivery time
Resource cost to deliver a page
Not entirely unrelated but commonly confused
Why Performance Matters
Poor performance means less traffic and therefore less revenue
This relationship is surprizingly strong perhaps 20 less traffic for an additional 01 seconds
Infrastructure is very expensive
Why Resource Cost Matters
A highly dynamic page may be very lightweight but require enormous effort to assemble - the example is Google
Even if a page is fast to deliver the cost of the infrastructure to create it can be a major business problem
Performance Analysis
is primarily analytic
but testing is essential
works best when integrated into the development process
Performance Analysis Consists of
Detecting Performance Issues
some sort of monitoring is needed
Localizing Performance Issues
usually done with analysis
Resolving Performance Issues
often easy but testing can be critical
Performance Analysis
is not code monkey work
is not fast
is not highly rigorous
Types of Performance Analysis
Capacity planning
Acceptance testing
Design and algorithm selection
Good amp Bad News
Meaningful results can be obtained
Results sometimes not expected
Requires a serious investment
Is inherently iterative
Where to Start
A hierarchy of
concepts - most important
policies - derived from concepts
tools - necessary but not dominant
Concepts
The Big Concept
Reduction of Uncertainty
overarching concept
provides answers to why what and do I care
How to Apply the Big Concept
Relate performance to business needs
provide enough information to allow management to make meaningful business decisions
gain enough information to allow extrapolation of trends
Other Concepts
Intuition is likely to be wrong
The process is inherently iterative
business process
technical process
An early start is very helpful
Policies
Policies
Analyze and test enough to make a business decision
Have a clearly defined process
Define your purpose
Make a quantitative estimate of the traffic
Let those who know something do it
More Policies
Communicate openly and often
Compartmentalize responsibility
Do not use performance testing to expose functional problems this is not QA (quality assurance)
Processes
Process Attributes
Defined
Documented
Public
Includes pre and post testing
Pre-Test Process
Assign responsibilities be definite about who
Declare goals
Record configuration
Assess traffic quantity and type
Post-Test Process
Meet definite end criteria
Report results
Make recommendations
Communicate issues
Test Process
Dynamic goal seeking
Interactive
Iterative
Duration depends on behavior
Test Options
Live or lab
End-to-end or component
The Most Common Error
Unjustified belief that the problem is code related usually architecture data traffic
Common Pitfalls
Unclear goals
N-dimensional matrix of tests
Emphasis on the tool
Emphasis on the code
More Common Pitfalls
Focus on a single tool for testing
Belief in a single answer to the problem
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 6: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/6.jpg)
Focus of This Talk
Survey only much more information is available
Primarily concepts some discussion of tools
Heuristics and work processes
General using web examples HTTP
Will Provide
Definite advice
Amusing anecdotes
Some tool recommendations
Wonrsquot Provide
Magic bullets
Simple answers
Performance Analysis
Goals
Know what performance results are
Understand reasons for performance results
Save money
Save time
Two Different Web Performance Aspects
Page delivery time
Resource cost to deliver a page
Not entirely unrelated but commonly confused
Why Performance Matters
Poor performance means less traffic and therefore less revenue
This relationship is surprizingly strong perhaps 20 less traffic for an additional 01 seconds
Infrastructure is very expensive
Why Resource Cost Matters
A highly dynamic page may be very lightweight but require enormous effort to assemble - the example is Google
Even if a page is fast to deliver the cost of the infrastructure to create it can be a major business problem
Performance Analysis
is primarily analytic
but testing is essential
works best when integrated into the development process
Performance Analysis Consists of
Detecting Performance Issues
some sort of monitoring is needed
Localizing Performance Issues
usually done with analysis
Resolving Performance Issues
often easy but testing can be critical
Performance Analysis
is not code monkey work
is not fast
is not highly rigorous
Types of Performance Analysis
Capacity planning
Acceptance testing
Design and algorithm selection
Good amp Bad News
Meaningful results can be obtained
Results sometimes not expected
Requires a serious investment
Is inherently iterative
Where to Start
A hierarchy of
concepts - most important
policies - derived from concepts
tools - necessary but not dominant
Concepts
The Big Concept
Reduction of Uncertainty
overarching concept
provides answers to why what and do I care
How to Apply the Big Concept
Relate performance to business needs
provide enough information to allow management to make meaningful business decisions
gain enough information to allow extrapolation of trends
Other Concepts
Intuition is likely to be wrong
The process is inherently iterative
business process
technical process
An early start is very helpful
Policies
Policies
Analyze and test enough to make a business decision
Have a clearly defined process
Define your purpose
Make a quantitative estimate of the traffic
Let those who know something do it
More Policies
Communicate openly and often
Compartmentalize responsibility
Do not use performance testing to expose functional problems this is not QA (quality assurance)
Processes
Process Attributes
Defined
Documented
Public
Includes pre and post testing
Pre-Test Process
Assign responsibilities be definite about who
Declare goals
Record configuration
Assess traffic quantity and type
Post-Test Process
Meet definite end criteria
Report results
Make recommendations
Communicate issues
Test Process
Dynamic goal seeking
Interactive
Iterative
Duration depends on behavior
Test Options
Live or lab
End-to-end or component
The Most Common Error
Unjustified belief that the problem is code related usually architecture data traffic
Common Pitfalls
Unclear goals
N-dimensional matrix of tests
Emphasis on the tool
Emphasis on the code
More Common Pitfalls
Focus on a single tool for testing
Belief in a single answer to the problem
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 7: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/7.jpg)
Will Provide
Definite advice
Amusing anecdotes
Some tool recommendations
Wonrsquot Provide
Magic bullets
Simple answers
Performance Analysis
Goals
Know what performance results are
Understand reasons for performance results
Save money
Save time
Two Different Web Performance Aspects
Page delivery time
Resource cost to deliver a page
Not entirely unrelated but commonly confused
Why Performance Matters
Poor performance means less traffic and therefore less revenue
This relationship is surprizingly strong perhaps 20 less traffic for an additional 01 seconds
Infrastructure is very expensive
Why Resource Cost Matters
A highly dynamic page may be very lightweight but require enormous effort to assemble - the example is Google
Even if a page is fast to deliver the cost of the infrastructure to create it can be a major business problem
Performance Analysis
is primarily analytic
but testing is essential
works best when integrated into the development process
Performance Analysis Consists of
Detecting Performance Issues
some sort of monitoring is needed
Localizing Performance Issues
usually done with analysis
Resolving Performance Issues
often easy but testing can be critical
Performance Analysis
is not code monkey work
is not fast
is not highly rigorous
Types of Performance Analysis
Capacity planning
Acceptance testing
Design and algorithm selection
Good amp Bad News
Meaningful results can be obtained
Results sometimes not expected
Requires a serious investment
Is inherently iterative
Where to Start
A hierarchy of
concepts - most important
policies - derived from concepts
tools - necessary but not dominant
Concepts
The Big Concept
Reduction of Uncertainty
overarching concept
provides answers to why what and do I care
How to Apply the Big Concept
Relate performance to business needs
provide enough information to allow management to make meaningful business decisions
gain enough information to allow extrapolation of trends
Other Concepts
Intuition is likely to be wrong
The process is inherently iterative
business process
technical process
An early start is very helpful
Policies
Policies
Analyze and test enough to make a business decision
Have a clearly defined process
Define your purpose
Make a quantitative estimate of the traffic
Let those who know something do it
More Policies
Communicate openly and often
Compartmentalize responsibility
Do not use performance testing to expose functional problems this is not QA (quality assurance)
Processes
Process Attributes
Defined
Documented
Public
Includes pre and post testing
Pre-Test Process
Assign responsibilities be definite about who
Declare goals
Record configuration
Assess traffic quantity and type
Post-Test Process
Meet definite end criteria
Report results
Make recommendations
Communicate issues
Test Process
Dynamic goal seeking
Interactive
Iterative
Duration depends on behavior
Test Options
Live or lab
End-to-end or component
The Most Common Error
Unjustified belief that the problem is code related usually architecture data traffic
Common Pitfalls
Unclear goals
N-dimensional matrix of tests
Emphasis on the tool
Emphasis on the code
More Common Pitfalls
Focus on a single tool for testing
Belief in a single answer to the problem
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 8: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/8.jpg)
Wonrsquot Provide
Magic bullets
Simple answers
Performance Analysis
Goals
Know what performance results are
Understand reasons for performance results
Save money
Save time
Two Different Web Performance Aspects
Page delivery time
Resource cost to deliver a page
Not entirely unrelated but commonly confused
Why Performance Matters
Poor performance means less traffic and therefore less revenue
This relationship is surprizingly strong perhaps 20 less traffic for an additional 01 seconds
Infrastructure is very expensive
Why Resource Cost Matters
A highly dynamic page may be very lightweight but require enormous effort to assemble - the example is Google
Even if a page is fast to deliver the cost of the infrastructure to create it can be a major business problem
Performance Analysis
is primarily analytic
but testing is essential
works best when integrated into the development process
Performance Analysis Consists of
Detecting Performance Issues
some sort of monitoring is needed
Localizing Performance Issues
usually done with analysis
Resolving Performance Issues
often easy but testing can be critical
Performance Analysis
is not code monkey work
is not fast
is not highly rigorous
Types of Performance Analysis
Capacity planning
Acceptance testing
Design and algorithm selection
Good amp Bad News
Meaningful results can be obtained
Results sometimes not expected
Requires a serious investment
Is inherently iterative
Where to Start
A hierarchy of
concepts - most important
policies - derived from concepts
tools - necessary but not dominant
Concepts
The Big Concept
Reduction of Uncertainty
overarching concept
provides answers to why what and do I care
How to Apply the Big Concept
Relate performance to business needs
provide enough information to allow management to make meaningful business decisions
gain enough information to allow extrapolation of trends
Other Concepts
Intuition is likely to be wrong
The process is inherently iterative
business process
technical process
An early start is very helpful
Policies
Policies
Analyze and test enough to make a business decision
Have a clearly defined process
Define your purpose
Make a quantitative estimate of the traffic
Let those who know something do it
More Policies
Communicate openly and often
Compartmentalize responsibility
Do not use performance testing to expose functional problems this is not QA (quality assurance)
Processes
Process Attributes
Defined
Documented
Public
Includes pre and post testing
Pre-Test Process
Assign responsibilities be definite about who
Declare goals
Record configuration
Assess traffic quantity and type
Post-Test Process
Meet definite end criteria
Report results
Make recommendations
Communicate issues
Test Process
Dynamic goal seeking
Interactive
Iterative
Duration depends on behavior
Test Options
Live or lab
End-to-end or component
The Most Common Error
Unjustified belief that the problem is code related usually architecture data traffic
Common Pitfalls
Unclear goals
N-dimensional matrix of tests
Emphasis on the tool
Emphasis on the code
More Common Pitfalls
Focus on a single tool for testing
Belief in a single answer to the problem
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 9: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/9.jpg)
Performance Analysis
Goals
Know what performance results are
Understand reasons for performance results
Save money
Save time
Two Different Web Performance Aspects
Page delivery time
Resource cost to deliver a page
Not entirely unrelated but commonly confused
Why Performance Matters
Poor performance means less traffic and therefore less revenue
This relationship is surprizingly strong perhaps 20 less traffic for an additional 01 seconds
Infrastructure is very expensive
Why Resource Cost Matters
A highly dynamic page may be very lightweight but require enormous effort to assemble - the example is Google
Even if a page is fast to deliver the cost of the infrastructure to create it can be a major business problem
Performance Analysis
is primarily analytic
but testing is essential
works best when integrated into the development process
Performance Analysis Consists of
Detecting Performance Issues
some sort of monitoring is needed
Localizing Performance Issues
usually done with analysis
Resolving Performance Issues
often easy but testing can be critical
Performance Analysis
is not code monkey work
is not fast
is not highly rigorous
Types of Performance Analysis
Capacity planning
Acceptance testing
Design and algorithm selection
Good amp Bad News
Meaningful results can be obtained
Results sometimes not expected
Requires a serious investment
Is inherently iterative
Where to Start
A hierarchy of
concepts - most important
policies - derived from concepts
tools - necessary but not dominant
Concepts
The Big Concept
Reduction of Uncertainty
overarching concept
provides answers to why what and do I care
How to Apply the Big Concept
Relate performance to business needs
provide enough information to allow management to make meaningful business decisions
gain enough information to allow extrapolation of trends
Other Concepts
Intuition is likely to be wrong
The process is inherently iterative
business process
technical process
An early start is very helpful
Policies
Policies
Analyze and test enough to make a business decision
Have a clearly defined process
Define your purpose
Make a quantitative estimate of the traffic
Let those who know something do it
More Policies
Communicate openly and often
Compartmentalize responsibility
Do not use performance testing to expose functional problems this is not QA (quality assurance)
Processes
Process Attributes
Defined
Documented
Public
Includes pre and post testing
Pre-Test Process
Assign responsibilities be definite about who
Declare goals
Record configuration
Assess traffic quantity and type
Post-Test Process
Meet definite end criteria
Report results
Make recommendations
Communicate issues
Test Process
Dynamic goal seeking
Interactive
Iterative
Duration depends on behavior
Test Options
Live or lab
End-to-end or component
The Most Common Error
Unjustified belief that the problem is code related usually architecture data traffic
Common Pitfalls
Unclear goals
N-dimensional matrix of tests
Emphasis on the tool
Emphasis on the code
More Common Pitfalls
Focus on a single tool for testing
Belief in a single answer to the problem
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 10: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/10.jpg)
Goals
Know what performance results are
Understand reasons for performance results
Save money
Save time
Two Different Web Performance Aspects
Page delivery time
Resource cost to deliver a page
Not entirely unrelated but commonly confused
Why Performance Matters
Poor performance means less traffic and therefore less revenue
This relationship is surprizingly strong perhaps 20 less traffic for an additional 01 seconds
Infrastructure is very expensive
Why Resource Cost Matters
A highly dynamic page may be very lightweight but require enormous effort to assemble - the example is Google
Even if a page is fast to deliver the cost of the infrastructure to create it can be a major business problem
Performance Analysis
is primarily analytic
but testing is essential
works best when integrated into the development process
Performance Analysis Consists of
Detecting Performance Issues
some sort of monitoring is needed
Localizing Performance Issues
usually done with analysis
Resolving Performance Issues
often easy but testing can be critical
Performance Analysis
is not code monkey work
is not fast
is not highly rigorous
Types of Performance Analysis
Capacity planning
Acceptance testing
Design and algorithm selection
Good amp Bad News
Meaningful results can be obtained
Results sometimes not expected
Requires a serious investment
Is inherently iterative
Where to Start
A hierarchy of
concepts - most important
policies - derived from concepts
tools - necessary but not dominant
Concepts
The Big Concept
Reduction of Uncertainty
overarching concept
provides answers to why what and do I care
How to Apply the Big Concept
Relate performance to business needs
provide enough information to allow management to make meaningful business decisions
gain enough information to allow extrapolation of trends
Other Concepts
Intuition is likely to be wrong
The process is inherently iterative
business process
technical process
An early start is very helpful
Policies
Policies
Analyze and test enough to make a business decision
Have a clearly defined process
Define your purpose
Make a quantitative estimate of the traffic
Let those who know something do it
More Policies
Communicate openly and often
Compartmentalize responsibility
Do not use performance testing to expose functional problems this is not QA (quality assurance)
Processes
Process Attributes
Defined
Documented
Public
Includes pre and post testing
Pre-Test Process
Assign responsibilities be definite about who
Declare goals
Record configuration
Assess traffic quantity and type
Post-Test Process
Meet definite end criteria
Report results
Make recommendations
Communicate issues
Test Process
Dynamic goal seeking
Interactive
Iterative
Duration depends on behavior
Test Options
Live or lab
End-to-end or component
The Most Common Error
Unjustified belief that the problem is code related usually architecture data traffic
Common Pitfalls
Unclear goals
N-dimensional matrix of tests
Emphasis on the tool
Emphasis on the code
More Common Pitfalls
Focus on a single tool for testing
Belief in a single answer to the problem
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 11: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/11.jpg)
Two Different Web Performance Aspects
Page delivery time
Resource cost to deliver a page
Not entirely unrelated but commonly confused
Why Performance Matters
Poor performance means less traffic and therefore less revenue
This relationship is surprizingly strong perhaps 20 less traffic for an additional 01 seconds
Infrastructure is very expensive
Why Resource Cost Matters
A highly dynamic page may be very lightweight but require enormous effort to assemble - the example is Google
Even if a page is fast to deliver the cost of the infrastructure to create it can be a major business problem
Performance Analysis
is primarily analytic
but testing is essential
works best when integrated into the development process
Performance Analysis Consists of
Detecting Performance Issues
some sort of monitoring is needed
Localizing Performance Issues
usually done with analysis
Resolving Performance Issues
often easy but testing can be critical
Performance Analysis
is not code monkey work
is not fast
is not highly rigorous
Types of Performance Analysis
Capacity planning
Acceptance testing
Design and algorithm selection
Good amp Bad News
Meaningful results can be obtained
Results sometimes not expected
Requires a serious investment
Is inherently iterative
Where to Start
A hierarchy of
concepts - most important
policies - derived from concepts
tools - necessary but not dominant
Concepts
The Big Concept
Reduction of Uncertainty
overarching concept
provides answers to why what and do I care
How to Apply the Big Concept
Relate performance to business needs
provide enough information to allow management to make meaningful business decisions
gain enough information to allow extrapolation of trends
Other Concepts
Intuition is likely to be wrong
The process is inherently iterative
business process
technical process
An early start is very helpful
Policies
Policies
Analyze and test enough to make a business decision
Have a clearly defined process
Define your purpose
Make a quantitative estimate of the traffic
Let those who know something do it
More Policies
Communicate openly and often
Compartmentalize responsibility
Do not use performance testing to expose functional problems this is not QA (quality assurance)
Processes
Process Attributes
Defined
Documented
Public
Includes pre and post testing
Pre-Test Process
Assign responsibilities be definite about who
Declare goals
Record configuration
Assess traffic quantity and type
Post-Test Process
Meet definite end criteria
Report results
Make recommendations
Communicate issues
Test Process
Dynamic goal seeking
Interactive
Iterative
Duration depends on behavior
Test Options
Live or lab
End-to-end or component
The Most Common Error
Unjustified belief that the problem is code related usually architecture data traffic
Common Pitfalls
Unclear goals
N-dimensional matrix of tests
Emphasis on the tool
Emphasis on the code
More Common Pitfalls
Focus on a single tool for testing
Belief in a single answer to the problem
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 12: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/12.jpg)
Why Performance Matters
Poor performance means less traffic and therefore less revenue
This relationship is surprizingly strong perhaps 20 less traffic for an additional 01 seconds
Infrastructure is very expensive
Why Resource Cost Matters
A highly dynamic page may be very lightweight but require enormous effort to assemble - the example is Google
Even if a page is fast to deliver the cost of the infrastructure to create it can be a major business problem
Performance Analysis
is primarily analytic
but testing is essential
works best when integrated into the development process
Performance Analysis Consists of
Detecting Performance Issues
some sort of monitoring is needed
Localizing Performance Issues
usually done with analysis
Resolving Performance Issues
often easy but testing can be critical
Performance Analysis
is not code monkey work
is not fast
is not highly rigorous
Types of Performance Analysis
Capacity planning
Acceptance testing
Design and algorithm selection
Good amp Bad News
Meaningful results can be obtained
Results sometimes not expected
Requires a serious investment
Is inherently iterative
Where to Start
A hierarchy of
concepts - most important
policies - derived from concepts
tools - necessary but not dominant
Concepts
The Big Concept
Reduction of Uncertainty
overarching concept
provides answers to why what and do I care
How to Apply the Big Concept
Relate performance to business needs
provide enough information to allow management to make meaningful business decisions
gain enough information to allow extrapolation of trends
Other Concepts
Intuition is likely to be wrong
The process is inherently iterative
business process
technical process
An early start is very helpful
Policies
Policies
Analyze and test enough to make a business decision
Have a clearly defined process
Define your purpose
Make a quantitative estimate of the traffic
Let those who know something do it
More Policies
Communicate openly and often
Compartmentalize responsibility
Do not use performance testing to expose functional problems this is not QA (quality assurance)
Processes
Process Attributes
Defined
Documented
Public
Includes pre and post testing
Pre-Test Process
Assign responsibilities be definite about who
Declare goals
Record configuration
Assess traffic quantity and type
Post-Test Process
Meet definite end criteria
Report results
Make recommendations
Communicate issues
Test Process
Dynamic goal seeking
Interactive
Iterative
Duration depends on behavior
Test Options
Live or lab
End-to-end or component
The Most Common Error
Unjustified belief that the problem is code related usually architecture data traffic
Common Pitfalls
Unclear goals
N-dimensional matrix of tests
Emphasis on the tool
Emphasis on the code
More Common Pitfalls
Focus on a single tool for testing
Belief in a single answer to the problem
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 13: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/13.jpg)
Why Resource Cost Matters
A highly dynamic page may be very lightweight but require enormous effort to assemble - the example is Google
Even if a page is fast to deliver the cost of the infrastructure to create it can be a major business problem
Performance Analysis
is primarily analytic
but testing is essential
works best when integrated into the development process
Performance Analysis Consists of
Detecting Performance Issues
some sort of monitoring is needed
Localizing Performance Issues
usually done with analysis
Resolving Performance Issues
often easy but testing can be critical
Performance Analysis
is not code monkey work
is not fast
is not highly rigorous
Types of Performance Analysis
Capacity planning
Acceptance testing
Design and algorithm selection
Good amp Bad News
Meaningful results can be obtained
Results sometimes not expected
Requires a serious investment
Is inherently iterative
Where to Start
A hierarchy of
concepts - most important
policies - derived from concepts
tools - necessary but not dominant
Concepts
The Big Concept
Reduction of Uncertainty
overarching concept
provides answers to why what and do I care
How to Apply the Big Concept
Relate performance to business needs
provide enough information to allow management to make meaningful business decisions
gain enough information to allow extrapolation of trends
Other Concepts
Intuition is likely to be wrong
The process is inherently iterative
business process
technical process
An early start is very helpful
Policies
Policies
Analyze and test enough to make a business decision
Have a clearly defined process
Define your purpose
Make a quantitative estimate of the traffic
Let those who know something do it
More Policies
Communicate openly and often
Compartmentalize responsibility
Do not use performance testing to expose functional problems this is not QA (quality assurance)
Processes
Process Attributes
Defined
Documented
Public
Includes pre and post testing
Pre-Test Process
Assign responsibilities be definite about who
Declare goals
Record configuration
Assess traffic quantity and type
Post-Test Process
Meet definite end criteria
Report results
Make recommendations
Communicate issues
Test Process
Dynamic goal seeking
Interactive
Iterative
Duration depends on behavior
Test Options
Live or lab
End-to-end or component
The Most Common Error
Unjustified belief that the problem is code related usually architecture data traffic
Common Pitfalls
Unclear goals
N-dimensional matrix of tests
Emphasis on the tool
Emphasis on the code
More Common Pitfalls
Focus on a single tool for testing
Belief in a single answer to the problem
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 14: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/14.jpg)
Performance Analysis
is primarily analytic
but testing is essential
works best when integrated into the development process
Performance Analysis Consists of
Detecting Performance Issues
some sort of monitoring is needed
Localizing Performance Issues
usually done with analysis
Resolving Performance Issues
often easy but testing can be critical
Performance Analysis
is not code monkey work
is not fast
is not highly rigorous
Types of Performance Analysis
Capacity planning
Acceptance testing
Design and algorithm selection
Good amp Bad News
Meaningful results can be obtained
Results sometimes not expected
Requires a serious investment
Is inherently iterative
Where to Start
A hierarchy of
concepts - most important
policies - derived from concepts
tools - necessary but not dominant
Concepts
The Big Concept
Reduction of Uncertainty
overarching concept
provides answers to why what and do I care
How to Apply the Big Concept
Relate performance to business needs
provide enough information to allow management to make meaningful business decisions
gain enough information to allow extrapolation of trends
Other Concepts
Intuition is likely to be wrong
The process is inherently iterative
business process
technical process
An early start is very helpful
Policies
Policies
Analyze and test enough to make a business decision
Have a clearly defined process
Define your purpose
Make a quantitative estimate of the traffic
Let those who know something do it
More Policies
Communicate openly and often
Compartmentalize responsibility
Do not use performance testing to expose functional problems this is not QA (quality assurance)
Processes
Process Attributes
Defined
Documented
Public
Includes pre and post testing
Pre-Test Process
Assign responsibilities be definite about who
Declare goals
Record configuration
Assess traffic quantity and type
Post-Test Process
Meet definite end criteria
Report results
Make recommendations
Communicate issues
Test Process
Dynamic goal seeking
Interactive
Iterative
Duration depends on behavior
Test Options
Live or lab
End-to-end or component
The Most Common Error
Unjustified belief that the problem is code related usually architecture data traffic
Common Pitfalls
Unclear goals
N-dimensional matrix of tests
Emphasis on the tool
Emphasis on the code
More Common Pitfalls
Focus on a single tool for testing
Belief in a single answer to the problem
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 15: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/15.jpg)
Performance Analysis Consists of
Detecting Performance Issues
some sort of monitoring is needed
Localizing Performance Issues
usually done with analysis
Resolving Performance Issues
often easy but testing can be critical
Performance Analysis
is not code monkey work
is not fast
is not highly rigorous
Types of Performance Analysis
Capacity planning
Acceptance testing
Design and algorithm selection
Good amp Bad News
Meaningful results can be obtained
Results sometimes not expected
Requires a serious investment
Is inherently iterative
Where to Start
A hierarchy of
concepts - most important
policies - derived from concepts
tools - necessary but not dominant
Concepts
The Big Concept
Reduction of Uncertainty
overarching concept
provides answers to why what and do I care
How to Apply the Big Concept
Relate performance to business needs
provide enough information to allow management to make meaningful business decisions
gain enough information to allow extrapolation of trends
Other Concepts
Intuition is likely to be wrong
The process is inherently iterative
business process
technical process
An early start is very helpful
Policies
Policies
Analyze and test enough to make a business decision
Have a clearly defined process
Define your purpose
Make a quantitative estimate of the traffic
Let those who know something do it
More Policies
Communicate openly and often
Compartmentalize responsibility
Do not use performance testing to expose functional problems this is not QA (quality assurance)
Processes
Process Attributes
Defined
Documented
Public
Includes pre and post testing
Pre-Test Process
Assign responsibilities be definite about who
Declare goals
Record configuration
Assess traffic quantity and type
Post-Test Process
Meet definite end criteria
Report results
Make recommendations
Communicate issues
Test Process
Dynamic goal seeking
Interactive
Iterative
Duration depends on behavior
Test Options
Live or lab
End-to-end or component
The Most Common Error
Unjustified belief that the problem is code related usually architecture data traffic
Common Pitfalls
Unclear goals
N-dimensional matrix of tests
Emphasis on the tool
Emphasis on the code
More Common Pitfalls
Focus on a single tool for testing
Belief in a single answer to the problem
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 16: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/16.jpg)
Performance Analysis
is not code monkey work
is not fast
is not highly rigorous
Types of Performance Analysis
Capacity planning
Acceptance testing
Design and algorithm selection
Good amp Bad News
Meaningful results can be obtained
Results sometimes not expected
Requires a serious investment
Is inherently iterative
Where to Start
A hierarchy of
concepts - most important
policies - derived from concepts
tools - necessary but not dominant
Concepts
The Big Concept
Reduction of Uncertainty
overarching concept
provides answers to why what and do I care
How to Apply the Big Concept
Relate performance to business needs
provide enough information to allow management to make meaningful business decisions
gain enough information to allow extrapolation of trends
Other Concepts
Intuition is likely to be wrong
The process is inherently iterative
business process
technical process
An early start is very helpful
Policies
Policies
Analyze and test enough to make a business decision
Have a clearly defined process
Define your purpose
Make a quantitative estimate of the traffic
Let those who know something do it
More Policies
Communicate openly and often
Compartmentalize responsibility
Do not use performance testing to expose functional problems this is not QA (quality assurance)
Processes
Process Attributes
Defined
Documented
Public
Includes pre and post testing
Pre-Test Process
Assign responsibilities be definite about who
Declare goals
Record configuration
Assess traffic quantity and type
Post-Test Process
Meet definite end criteria
Report results
Make recommendations
Communicate issues
Test Process
Dynamic goal seeking
Interactive
Iterative
Duration depends on behavior
Test Options
Live or lab
End-to-end or component
The Most Common Error
Unjustified belief that the problem is code related usually architecture data traffic
Common Pitfalls
Unclear goals
N-dimensional matrix of tests
Emphasis on the tool
Emphasis on the code
More Common Pitfalls
Focus on a single tool for testing
Belief in a single answer to the problem
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 17: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/17.jpg)
Types of Performance Analysis
Capacity planning
Acceptance testing
Design and algorithm selection
Good amp Bad News
Meaningful results can be obtained
Results sometimes not expected
Requires a serious investment
Is inherently iterative
Where to Start
A hierarchy of
concepts - most important
policies - derived from concepts
tools - necessary but not dominant
Concepts
The Big Concept
Reduction of Uncertainty
overarching concept
provides answers to why what and do I care
How to Apply the Big Concept
Relate performance to business needs
provide enough information to allow management to make meaningful business decisions
gain enough information to allow extrapolation of trends
Other Concepts
Intuition is likely to be wrong
The process is inherently iterative
business process
technical process
An early start is very helpful
Policies
Policies
Analyze and test enough to make a business decision
Have a clearly defined process
Define your purpose
Make a quantitative estimate of the traffic
Let those who know something do it
More Policies
Communicate openly and often
Compartmentalize responsibility
Do not use performance testing to expose functional problems this is not QA (quality assurance)
Processes
Process Attributes
Defined
Documented
Public
Includes pre and post testing
Pre-Test Process
Assign responsibilities be definite about who
Declare goals
Record configuration
Assess traffic quantity and type
Post-Test Process
Meet definite end criteria
Report results
Make recommendations
Communicate issues
Test Process
Dynamic goal seeking
Interactive
Iterative
Duration depends on behavior
Test Options
Live or lab
End-to-end or component
The Most Common Error
Unjustified belief that the problem is code related usually architecture data traffic
Common Pitfalls
Unclear goals
N-dimensional matrix of tests
Emphasis on the tool
Emphasis on the code
More Common Pitfalls
Focus on a single tool for testing
Belief in a single answer to the problem
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 18: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/18.jpg)
Good amp Bad News
Meaningful results can be obtained
Results sometimes not expected
Requires a serious investment
Is inherently iterative
Where to Start
A hierarchy of
concepts - most important
policies - derived from concepts
tools - necessary but not dominant
Concepts
The Big Concept
Reduction of Uncertainty
overarching concept
provides answers to why what and do I care
How to Apply the Big Concept
Relate performance to business needs
provide enough information to allow management to make meaningful business decisions
gain enough information to allow extrapolation of trends
Other Concepts
Intuition is likely to be wrong
The process is inherently iterative
business process
technical process
An early start is very helpful
Policies
Policies
Analyze and test enough to make a business decision
Have a clearly defined process
Define your purpose
Make a quantitative estimate of the traffic
Let those who know something do it
More Policies
Communicate openly and often
Compartmentalize responsibility
Do not use performance testing to expose functional problems this is not QA (quality assurance)
Processes
Process Attributes
Defined
Documented
Public
Includes pre and post testing
Pre-Test Process
Assign responsibilities be definite about who
Declare goals
Record configuration
Assess traffic quantity and type
Post-Test Process
Meet definite end criteria
Report results
Make recommendations
Communicate issues
Test Process
Dynamic goal seeking
Interactive
Iterative
Duration depends on behavior
Test Options
Live or lab
End-to-end or component
The Most Common Error
Unjustified belief that the problem is code related usually architecture data traffic
Common Pitfalls
Unclear goals
N-dimensional matrix of tests
Emphasis on the tool
Emphasis on the code
More Common Pitfalls
Focus on a single tool for testing
Belief in a single answer to the problem
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 19: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/19.jpg)
Where to Start
A hierarchy of
concepts - most important
policies - derived from concepts
tools - necessary but not dominant
Concepts
The Big Concept
Reduction of Uncertainty
overarching concept
provides answers to why what and do I care
How to Apply the Big Concept
Relate performance to business needs
provide enough information to allow management to make meaningful business decisions
gain enough information to allow extrapolation of trends
Other Concepts
Intuition is likely to be wrong
The process is inherently iterative
business process
technical process
An early start is very helpful
Policies
Policies
Analyze and test enough to make a business decision
Have a clearly defined process
Define your purpose
Make a quantitative estimate of the traffic
Let those who know something do it
More Policies
Communicate openly and often
Compartmentalize responsibility
Do not use performance testing to expose functional problems this is not QA (quality assurance)
Processes
Process Attributes
Defined
Documented
Public
Includes pre and post testing
Pre-Test Process
Assign responsibilities be definite about who
Declare goals
Record configuration
Assess traffic quantity and type
Post-Test Process
Meet definite end criteria
Report results
Make recommendations
Communicate issues
Test Process
Dynamic goal seeking
Interactive
Iterative
Duration depends on behavior
Test Options
Live or lab
End-to-end or component
The Most Common Error
Unjustified belief that the problem is code related usually architecture data traffic
Common Pitfalls
Unclear goals
N-dimensional matrix of tests
Emphasis on the tool
Emphasis on the code
More Common Pitfalls
Focus on a single tool for testing
Belief in a single answer to the problem
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 20: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/20.jpg)
Concepts
The Big Concept
Reduction of Uncertainty
overarching concept
provides answers to why what and do I care
How to Apply the Big Concept
Relate performance to business needs
provide enough information to allow management to make meaningful business decisions
gain enough information to allow extrapolation of trends
Other Concepts
Intuition is likely to be wrong
The process is inherently iterative
business process
technical process
An early start is very helpful
Policies
Policies
Analyze and test enough to make a business decision
Have a clearly defined process
Define your purpose
Make a quantitative estimate of the traffic
Let those who know something do it
More Policies
Communicate openly and often
Compartmentalize responsibility
Do not use performance testing to expose functional problems this is not QA (quality assurance)
Processes
Process Attributes
Defined
Documented
Public
Includes pre and post testing
Pre-Test Process
Assign responsibilities be definite about who
Declare goals
Record configuration
Assess traffic quantity and type
Post-Test Process
Meet definite end criteria
Report results
Make recommendations
Communicate issues
Test Process
Dynamic goal seeking
Interactive
Iterative
Duration depends on behavior
Test Options
Live or lab
End-to-end or component
The Most Common Error
Unjustified belief that the problem is code related usually architecture data traffic
Common Pitfalls
Unclear goals
N-dimensional matrix of tests
Emphasis on the tool
Emphasis on the code
More Common Pitfalls
Focus on a single tool for testing
Belief in a single answer to the problem
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 21: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/21.jpg)
The Big Concept
Reduction of Uncertainty
overarching concept
provides answers to why what and do I care
How to Apply the Big Concept
Relate performance to business needs
provide enough information to allow management to make meaningful business decisions
gain enough information to allow extrapolation of trends
Other Concepts
Intuition is likely to be wrong
The process is inherently iterative
business process
technical process
An early start is very helpful
Policies
Policies
Analyze and test enough to make a business decision
Have a clearly defined process
Define your purpose
Make a quantitative estimate of the traffic
Let those who know something do it
More Policies
Communicate openly and often
Compartmentalize responsibility
Do not use performance testing to expose functional problems this is not QA (quality assurance)
Processes
Process Attributes
Defined
Documented
Public
Includes pre and post testing
Pre-Test Process
Assign responsibilities be definite about who
Declare goals
Record configuration
Assess traffic quantity and type
Post-Test Process
Meet definite end criteria
Report results
Make recommendations
Communicate issues
Test Process
Dynamic goal seeking
Interactive
Iterative
Duration depends on behavior
Test Options
Live or lab
End-to-end or component
The Most Common Error
Unjustified belief that the problem is code related usually architecture data traffic
Common Pitfalls
Unclear goals
N-dimensional matrix of tests
Emphasis on the tool
Emphasis on the code
More Common Pitfalls
Focus on a single tool for testing
Belief in a single answer to the problem
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 22: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/22.jpg)
How to Apply the Big Concept
Relate performance to business needs
provide enough information to allow management to make meaningful business decisions
gain enough information to allow extrapolation of trends
Other Concepts
Intuition is likely to be wrong
The process is inherently iterative
business process
technical process
An early start is very helpful
Policies
Policies
Analyze and test enough to make a business decision
Have a clearly defined process
Define your purpose
Make a quantitative estimate of the traffic
Let those who know something do it
More Policies
Communicate openly and often
Compartmentalize responsibility
Do not use performance testing to expose functional problems this is not QA (quality assurance)
Processes
Process Attributes
Defined
Documented
Public
Includes pre and post testing
Pre-Test Process
Assign responsibilities be definite about who
Declare goals
Record configuration
Assess traffic quantity and type
Post-Test Process
Meet definite end criteria
Report results
Make recommendations
Communicate issues
Test Process
Dynamic goal seeking
Interactive
Iterative
Duration depends on behavior
Test Options
Live or lab
End-to-end or component
The Most Common Error
Unjustified belief that the problem is code related usually architecture data traffic
Common Pitfalls
Unclear goals
N-dimensional matrix of tests
Emphasis on the tool
Emphasis on the code
More Common Pitfalls
Focus on a single tool for testing
Belief in a single answer to the problem
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 23: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/23.jpg)
Other Concepts
Intuition is likely to be wrong
The process is inherently iterative
business process
technical process
An early start is very helpful
Policies
Policies
Analyze and test enough to make a business decision
Have a clearly defined process
Define your purpose
Make a quantitative estimate of the traffic
Let those who know something do it
More Policies
Communicate openly and often
Compartmentalize responsibility
Do not use performance testing to expose functional problems this is not QA (quality assurance)
Processes
Process Attributes
Defined
Documented
Public
Includes pre and post testing
Pre-Test Process
Assign responsibilities be definite about who
Declare goals
Record configuration
Assess traffic quantity and type
Post-Test Process
Meet definite end criteria
Report results
Make recommendations
Communicate issues
Test Process
Dynamic goal seeking
Interactive
Iterative
Duration depends on behavior
Test Options
Live or lab
End-to-end or component
The Most Common Error
Unjustified belief that the problem is code related usually architecture data traffic
Common Pitfalls
Unclear goals
N-dimensional matrix of tests
Emphasis on the tool
Emphasis on the code
More Common Pitfalls
Focus on a single tool for testing
Belief in a single answer to the problem
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 24: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/24.jpg)
Policies
Policies
Analyze and test enough to make a business decision
Have a clearly defined process
Define your purpose
Make a quantitative estimate of the traffic
Let those who know something do it
More Policies
Communicate openly and often
Compartmentalize responsibility
Do not use performance testing to expose functional problems this is not QA (quality assurance)
Processes
Process Attributes
Defined
Documented
Public
Includes pre and post testing
Pre-Test Process
Assign responsibilities be definite about who
Declare goals
Record configuration
Assess traffic quantity and type
Post-Test Process
Meet definite end criteria
Report results
Make recommendations
Communicate issues
Test Process
Dynamic goal seeking
Interactive
Iterative
Duration depends on behavior
Test Options
Live or lab
End-to-end or component
The Most Common Error
Unjustified belief that the problem is code related usually architecture data traffic
Common Pitfalls
Unclear goals
N-dimensional matrix of tests
Emphasis on the tool
Emphasis on the code
More Common Pitfalls
Focus on a single tool for testing
Belief in a single answer to the problem
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 25: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/25.jpg)
Policies
Analyze and test enough to make a business decision
Have a clearly defined process
Define your purpose
Make a quantitative estimate of the traffic
Let those who know something do it
More Policies
Communicate openly and often
Compartmentalize responsibility
Do not use performance testing to expose functional problems this is not QA (quality assurance)
Processes
Process Attributes
Defined
Documented
Public
Includes pre and post testing
Pre-Test Process
Assign responsibilities be definite about who
Declare goals
Record configuration
Assess traffic quantity and type
Post-Test Process
Meet definite end criteria
Report results
Make recommendations
Communicate issues
Test Process
Dynamic goal seeking
Interactive
Iterative
Duration depends on behavior
Test Options
Live or lab
End-to-end or component
The Most Common Error
Unjustified belief that the problem is code related usually architecture data traffic
Common Pitfalls
Unclear goals
N-dimensional matrix of tests
Emphasis on the tool
Emphasis on the code
More Common Pitfalls
Focus on a single tool for testing
Belief in a single answer to the problem
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 26: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/26.jpg)
More Policies
Communicate openly and often
Compartmentalize responsibility
Do not use performance testing to expose functional problems this is not QA (quality assurance)
Processes
Process Attributes
Defined
Documented
Public
Includes pre and post testing
Pre-Test Process
Assign responsibilities be definite about who
Declare goals
Record configuration
Assess traffic quantity and type
Post-Test Process
Meet definite end criteria
Report results
Make recommendations
Communicate issues
Test Process
Dynamic goal seeking
Interactive
Iterative
Duration depends on behavior
Test Options
Live or lab
End-to-end or component
The Most Common Error
Unjustified belief that the problem is code related usually architecture data traffic
Common Pitfalls
Unclear goals
N-dimensional matrix of tests
Emphasis on the tool
Emphasis on the code
More Common Pitfalls
Focus on a single tool for testing
Belief in a single answer to the problem
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 27: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/27.jpg)
Processes
Process Attributes
Defined
Documented
Public
Includes pre and post testing
Pre-Test Process
Assign responsibilities be definite about who
Declare goals
Record configuration
Assess traffic quantity and type
Post-Test Process
Meet definite end criteria
Report results
Make recommendations
Communicate issues
Test Process
Dynamic goal seeking
Interactive
Iterative
Duration depends on behavior
Test Options
Live or lab
End-to-end or component
The Most Common Error
Unjustified belief that the problem is code related usually architecture data traffic
Common Pitfalls
Unclear goals
N-dimensional matrix of tests
Emphasis on the tool
Emphasis on the code
More Common Pitfalls
Focus on a single tool for testing
Belief in a single answer to the problem
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 28: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/28.jpg)
Process Attributes
Defined
Documented
Public
Includes pre and post testing
Pre-Test Process
Assign responsibilities be definite about who
Declare goals
Record configuration
Assess traffic quantity and type
Post-Test Process
Meet definite end criteria
Report results
Make recommendations
Communicate issues
Test Process
Dynamic goal seeking
Interactive
Iterative
Duration depends on behavior
Test Options
Live or lab
End-to-end or component
The Most Common Error
Unjustified belief that the problem is code related usually architecture data traffic
Common Pitfalls
Unclear goals
N-dimensional matrix of tests
Emphasis on the tool
Emphasis on the code
More Common Pitfalls
Focus on a single tool for testing
Belief in a single answer to the problem
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 29: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/29.jpg)
Pre-Test Process
Assign responsibilities be definite about who
Declare goals
Record configuration
Assess traffic quantity and type
Post-Test Process
Meet definite end criteria
Report results
Make recommendations
Communicate issues
Test Process
Dynamic goal seeking
Interactive
Iterative
Duration depends on behavior
Test Options
Live or lab
End-to-end or component
The Most Common Error
Unjustified belief that the problem is code related usually architecture data traffic
Common Pitfalls
Unclear goals
N-dimensional matrix of tests
Emphasis on the tool
Emphasis on the code
More Common Pitfalls
Focus on a single tool for testing
Belief in a single answer to the problem
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 30: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/30.jpg)
Post-Test Process
Meet definite end criteria
Report results
Make recommendations
Communicate issues
Test Process
Dynamic goal seeking
Interactive
Iterative
Duration depends on behavior
Test Options
Live or lab
End-to-end or component
The Most Common Error
Unjustified belief that the problem is code related usually architecture data traffic
Common Pitfalls
Unclear goals
N-dimensional matrix of tests
Emphasis on the tool
Emphasis on the code
More Common Pitfalls
Focus on a single tool for testing
Belief in a single answer to the problem
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 31: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/31.jpg)
Test Process
Dynamic goal seeking
Interactive
Iterative
Duration depends on behavior
Test Options
Live or lab
End-to-end or component
The Most Common Error
Unjustified belief that the problem is code related usually architecture data traffic
Common Pitfalls
Unclear goals
N-dimensional matrix of tests
Emphasis on the tool
Emphasis on the code
More Common Pitfalls
Focus on a single tool for testing
Belief in a single answer to the problem
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 32: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/32.jpg)
Test Options
Live or lab
End-to-end or component
The Most Common Error
Unjustified belief that the problem is code related usually architecture data traffic
Common Pitfalls
Unclear goals
N-dimensional matrix of tests
Emphasis on the tool
Emphasis on the code
More Common Pitfalls
Focus on a single tool for testing
Belief in a single answer to the problem
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 33: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/33.jpg)
The Most Common Error
Unjustified belief that the problem is code related usually architecture data traffic
Common Pitfalls
Unclear goals
N-dimensional matrix of tests
Emphasis on the tool
Emphasis on the code
More Common Pitfalls
Focus on a single tool for testing
Belief in a single answer to the problem
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 34: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/34.jpg)
Common Pitfalls
Unclear goals
N-dimensional matrix of tests
Emphasis on the tool
Emphasis on the code
More Common Pitfalls
Focus on a single tool for testing
Belief in a single answer to the problem
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 35: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/35.jpg)
More Common Pitfalls
Focus on a single tool for testing
Belief in a single answer to the problem
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 36: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/36.jpg)
Tools
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 37: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/37.jpg)
Tools Issues
Open source or proprietary
Proxy capture of user actions
Scriptable
Randomizable
Enforce policies
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 38: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/38.jpg)
Two Types of Tools
Load generators
Test environment managers
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 39: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/39.jpg)
Load Generators
flood
JMeter
many others
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 40: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/40.jpg)
Load Generator Issues
Synchronous
Randomizable
Script or proxy
Load ramping
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 41: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/41.jpg)
Load Generator Obsession
What people usually mean when they talk about performance testing tools
Necessary
Far from sufficient
There are many
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 42: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/42.jpg)
Environment Managers
Unlike load generators few if any exist
But you can get lots of what is needed through policies and procedures
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 43: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/43.jpg)
Test Environment
Controlled
Isolated
Automated
Recording
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 44: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/44.jpg)
Test Facility Components
Manager
Load generator
System under test
Data recorder
Report generator
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 45: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/45.jpg)
Performance Tests
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 46: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/46.jpg)
Test Design
Much more in common with experiment design than statistical analysis
Statistics are good but
You donrsquot need statistics to know which tire is flat
Good test design can make it that obvious
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 47: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/47.jpg)
Test Components
Two parts
load side
system under test side
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 48: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/48.jpg)
Test Results
Also have two sides
Load side
requestss srequest
timing of parts of the request
System under test side
system and application metrics
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 49: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/49.jpg)
Test Analysis
Reality check results and test parameters
Compare to actual or derive a baseline
Follow good analytic methodology
multiple tests etc
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 50: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/50.jpg)
Pet Peeves
Load ramping
if you want different loads run different tests
actual load changes hard to match
Proxy capture
better to get real traffic from logs
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 51: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/51.jpg)
Wrap Up
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 52: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/52.jpg)
Key Points
Strong concepts
Well-defined process
Experiment design
Iterative
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 53: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/53.jpg)
How to Solve Your Problem
Irsquom available for consulting
Introductory talks for Managers and Engineers
tools instruction procedures policies
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 54: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/54.jpg)
Further Info - General Information
httpopensourcetestingorgperformancephp
httpwwwsoftwareqatestcomqatweb1htmlLOAD
search for lsquoweb performance testingrsquo
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 55: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/55.jpg)
Further Info - Proprietary
TestRunner
OpenLoad
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 56: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/56.jpg)
Further Info - Load Generators
JMeter
httpjakartaapacheorgjmeter
Flood
httphttpdapacheorgtestflood
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 57: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/57.jpg)
Further Info - Environment Managers
OpenSTA
httpopenstaorg
Faban
httpfabansunsourcenet
The Grinder
httpgrindersourceforgenet
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 58: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/58.jpg)
Excellent Book
Programming Pearls Jon Bentley
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 59: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/59.jpg)
Thanks
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 60: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/60.jpg)
Survey of Web Performance Analysis
Guy Ferraiolo Norstrilia Software Incgdfnorstriliasoftwarecom
contact information
![Page 61: Survey of Web Performance Analysisprincetonacm.acm.org/tcfpro/Web_Performance.pdf · Two Different Web Performance Aspects Page delivery time Resource cost to deliver a page ... Two](https://reader034.vdocuments.us/reader034/viewer/2022042213/5eb761e91b747d18234916b8/html5/thumbnails/61.jpg)
contact information