Download - Software performance testing_overview
Software Performance Testing Overview
Rohan Bhattarai
Contents What is Performance Testing? What are the Terminologies I should know? Why Performance Testing? What are the Performance Bottlenecks? Why do we have bottlenecks ? What are the contributors? What are the types of Performance Testing? When is it required? What should be tested? What should be my Performance Testing Process? Why should I use Tool ? What should be my tools combination? What tools are available and what should I use for Performance
Testing? What are the Performance Metrics? What should be my best approach?
What is Performance Testing? Performance testing is
the process of determining the speed or effectiveness of a computer, network, software program or device.
Let’s talk about Software Performance Testing Before going into the details, we should
understand the factors that governs Software Performance Testing:
Throughput Response Time Tuning Benchmarking
What are the Terminologies I should know? Application Architecture HTTP vs HTTPS Request Vs Response GET and POST Methods HTTP Status Codes Throughput Response Time Tuning Benchmarking Transaction Mix User Mix User Think Time Ramp-up Pattern
What are the Terminologies I should know? Application Architecture
Client-Server 2-Tier 3-Tier N-Tier
What are the Terminologies I should know?HTTP vs HTTPS The Hypertext Transfer Protocol (HTTP) is a networking
protocol for distributed, collaborative, hypermedia information systems .
HTTP functions as a request-response protocol in the client-server computing model.
In HTTP, a web browser, for example, acts as a client, while an application running on a computer hosting a web site functions as a server. The client submits an HTTP request message to the server. The server, which stores content, or provides resources, such as HTML files, or performs other functions on behalf of the client, returns a response message to the client.
HTTPS is simply HTTP using Secure Sockets Layer (SSL) to encrypt and decrypt data for secure transactions
What are the Terminologies I should know?Request Vs Response Client Request
GET /index.html HTTP/1.1
Host: www.example.com Server Response
HTTP/1.1 200 OK
Date: Mon, 23 May 2005 22:38:34 GMT
Server: Apache/1.3.3.7 (Unix) (Red-Hat/Linux)
Last-Modified: Wed, 08 Jan 2003 23:11:55 GMT
Etag: "3f80f-1b6-3e1cb03b"
Accept-Ranges: bytes
Content-Length: 438
Connection: close
Content-Type: text/html; charset=UTF-8
What are the Terminologies I should know?GET and POST Methods GET is the simplest HTTP method, and its main job in life is to
ask the server to get a resource and send it back. That resource might be an HTML page, a JPEG, a PDF, etc. Doesn't matter. The point of GET is to get something back from the server.
POST is a more powerful request. It's like a GET++. With POST, you can request something and at the same time send form data to the server
GET and POST basically allow information to be sent back to the webserver from a browser (or other HTTP client for that matter). Imagine that you have a form on a HTML page and clicking the "submit" button sends the data in the form back to the server, as "name=value" pairs.
What are the Terminologies I should know?GET and POST Methods Choosing GET as the "method" will append all of the data to the
URL and it will show up in the URL bar of your browser. The amount of information you can send back using a GET is restricted as URLs can only be 1024 characters.
http://www.google.co.uk/#hl=en&source=hp&q=Rational+Performance+Tester&oq=Rational+Performance+Tester&aq=f&aqi=g10&aql=&gs_sm=e&gs_upl=1296l6140l0l6328l33l25l5l6l6l0l297l2062l2-8l8l0&bav=on.2,or.r_gc.r_pw.&fp=950be6bfd6a5c219&biw=1024&bih=563
A POST on the other hand will (typically) send the information through a socket back to the webserver and it won't show up in the URL bar. You can send much more information to the server this way - and it's not restricted to textual data either. It is possible to send files and even binary data such as serialized Java objects!
What are the Terminologies I should know?
GET Method: All the name value pairs are submitted as a
query string in URL. It's not secured as it is visible in plain text
format in the Location bar of the web browser.
As the data transfers through address bar (URL) there are some restrictions in using space, some characters like ampersand (&) etc in the GET method of posting data. We have to take special care for encoding data if such special characters are present.
Length of the string is restricted. If method is not mentioned in the Form tag,
this is the default method used. If get method is used and if the page is
refreshed it would not prompt before the request is submitted again.
One can store the name value pairs as bookmark and directly be used while sharing with others - example search results.
Data is always submitted in the form of text
If the response of the page is always same for the posted query then use GET example database searches
POST Method: 1. All the name value pairs are submitted in
the Message Body of the request. 2. Length of the string (amount of data
submitted) is not restricted. 3. Post Method is secured because Name-
Value pairs cannot be seen in location bar of the web browser.
4. If post method is used and if the page is refreshed it would prompt before the request is resubmitted.
5. If the service associated with the processing of a form has side effects (for example, modification of a database or subscription to a service), the method should be POST.
6. Data is submitted in the form as specified in enctype attribute of form tag and thus files can be used in FileUpload input box.
What are the Terminologies I should know?HTTP Status Codes In HTTP/1.0 and since, the first line of the HTTP
response is called the status line and includes a numeric status code (such as "404") and a textual reason phrase (such as "Not Found").
1xx Informational 2xx Success 3xx Redirection 4xx Client Error 5xx Server Error
URL: http://en.wikipedia.org/wiki/List_of_HTTP_status_codes
What are the Terminologies I should know?Throughput Capability of a product to handle
multiple transactions in a give period.
Throughput represents the number of requests/business transactions processed by the product in a specified time duration.
As the number of concurrent users increase, the throughput increases almost linearly with the number of requests. As there is very little congestion within the Application Server system queues.
What are the Terminologies I should know?Response Time It is equally
important to find out how much time each of the transactions took to complete.
Response time is defined as the delay between the point of request and the first response from the product.
The response time increases proportionally to the user load.
What are the Terminologies I should know?Tuning Tuning is the procedure
by which product performance is enhanced by setting different values to the parameters of the product, operating system and other components.
Tuning improves the product performance without having to touch the source code of the product.
What are the Terminologies I should know?Tuning It is a process of identifying and eliminating
bottlenecks Performance tuning of Webbased systems is a
multivariable and iterative process It is a collaborative effort between cross functional
teams Performance tester is an enabler since he provides
the necessary information in an efficient manner
What are the Terminologies I should know?Benchmarking A very well-improved
performance of a product makes no business sense if that performance does not match up to the competitive products.
A careful analysis is needed to chalk out the list of transactions to be compared across products so that an apple-apple comparison becomes possible.
What are the Terminologies I should know?Transaction Mix
CRUD Operations Mission Critical Transactions High Frequency Transactions
Search
Buy
Hack
What are the Terminologies I should know? User Mix
User level and their roles No. of users accessing the application
Admin
Power
Normal
What are the Terminologies I should know? User Think Time
Time that user think to perform any event How long do user think while navigating from one page to
another?
5 sec
random
fixed
What are the Terminologies I should know? Ramp-up Pattern
How many user should login to application at a time?
Time
User Load
Defining Performance Testing Again The testing to evaluate the response time (speed),
throughput and utilization of system to execute its required functions in comparison with different versions of the same product or a different competitive product is called Performance Testing.
Performance testing is done to derive benchmark numbers for the system.
Heavy load is not applied to the system
Tuning is performed until the System Under Test (SUT) achieves the expected levels of performance.
Why Performance Testing? The week of Feb 6, 2000:
Hackers delivered over 1-Billion transactions concurrently to each one of these sites.
YahooeBayBuy.comAmazoneTrade
How did you think they performed ?
Why Performance Testing? Is your application scalable ? How reliable is it? How fast is it? How secure is it? What are other performance bottlenecks?
“Only conducting Performance testing at the conclusion of system or functional
testing is like ordering a diagnostic blood test after the patient is dead.”
Scott Barber
Why Performance Testing?
Why Performance Testing? Identifies problems early on before they become
costly to resolve. Reduces development cycles. Produces better quality, more scalable code. Prevents revenue and credibility loss due to poor
Web site performance. Enables intelligent planning for future expansion. To ensure that the system meets performance
expectations such as response time, throughput etc. under given levels of load.
Expose bugs that do not surface in cursory testing, such as memory management bugs, memory leaks, buffer overflows, etc.
What are the Performance Bottlenecks?
Why do we have bottlenecks ? People
Reactive than proactive, Not Thinking About “Load” or “Scalability” or
“Performance” During Des & Dev Limited Expertise in System Design / Development
Process Poor Architecture and System Design Changing requirements Changing project plans
Technology Inadequate Hardware Inadequate Network Capacity Poor Implementation / Tuning of Databases and
components
What are the contributors?
What are the types of Performance Testing? Load Vs Stress Testing Goal based Performance Testing Reactive and Proactive Performance
Engineering Performance Validation, Performance Testing
and Performance Engineering/Tuning Other Types - Soak Testing / Endurance
Testing, Spike Testing, Scalability Vs Stability Testing, Application Speed Vs Application Scalability, Scaling Up Vs Scaling Out
What are the types of Performance Testing? Load Vs Stress Testing
What are the types of Performance Testing?
Load Testing Process of exercising the system
under test by feeding it the largest tasks it can operate with.
Constantly increasing the load on the system via automated tools to simulate real time scenario with virtual users.
Examples: Testing a word processor by
editing a very large document. For Web Application load is
defined in terms of concurrent users or HTTP connections.
What are the types of Performance Testing?
Stress Testing Trying to break the system under
test by overwhelming its resources or by taking resources away from it.
Purpose is to make sure that the
system fails and recovers gracefully.
Example: Double the baseline number for
concurrent users/HTTP connections. Randomly shut down and restart
ports on the network switches/routers that connects servers.
What are the types of Performance Testing? Goal based Performance Testing
Reactive and Proactive Performance Engineering
What are the types of Performance Testing? Performance Validation, Performance Testing
and Performance Engineering/Tuning
DB
Web Server (s)
App Server (s)
What are the types of Performance Testing?
When is it required?Design Phase:
Pages containing lots of images and multimedia for reasonable wait times. Heavy loads are less important than knowing which types of content cause slowdowns.
Development Phase:
To check results of individual pages and processes, looking for breaking points, unnecessary code and bottlenecks.
Deployment Phase:
To identify the minimum hardware and software requirements for the application.
What should be tested? High frequency transactions: The most frequently used
transactions have the potential to impact the performance of all of the other transactions if they are not efficient.
Mission Critical transactions: The more important transactions that facilitate the core objectives of the system should be included, as failure under load of these transactions has, by definition, the greatest impact.
Read Transactions: At least one READ ONLY transaction should be included, so that performance of such transactions can be differentiated from other more complex transactions.
Update Transactions: At least one update transaction should be included so that performance of such transactions can be differentiated from other transactions.
What should be my Performance Testing Process?
What should be my Performance Testing Process?
Determine the performance testing objectives
Describe the application to test using a application model
1. Describe the Hardware environment
2. Create a Benchmark (Agenda) to be recorded in Phase 2.
A. Define what tasks each user will perform B. Define (or estimate) the percentage of users per task.
1.Planning1.Planning
What should be my Performance Testing Process?
Record Record the defined testing activities that will be used as a
foundation for your load test scripts. One activity per task or multiple activities depending on user
task definition
Modify Modify load test scripts defined by recorder to reflect more
realistic Load test simulations. Defining the project, users Randomize parameters (Data, times, environment) Randomize user activities that occur during the load test
3.Modify3.Modify2.Record2.Record1.Planning
1.Planning
What should be my Performance Testing Process?
Virtual Users (VUs): Test Goals
Start: 5 Max Response Time <= 20 Sec
Incremented by: 5
Maximum: 200
Think Time: 5 sec
Test Script:
One typical user from login through completion.
4. Execute4. Execute
What should be my Performance Testing Process?
Monitoring the scenario: We monitor scenario execution using the various online runtime monitors.
Analysing test results: During scenario execution, the tool records the performance of the application under different loads. We use the graphs and reports to analyse the application’s performance.
6. Analyze6. Analyze5.Monitor5.Monitor
Why should I use Tool ? Manual Testing Limitation
Testers
Load Generation System Under Test
Do you have the testing resources?• Testing personnel• Client machines
How do you coordinate and synchronize users?
How do you collect and analyze results?
How do you achieve test repeatability?
Analysis?
123.20
All of you, click the GO button again
Web server Database server
Coordinator
Why should I use Tool ?
Manual Testing LimitationsExpensive, requiring large amounts of both
personnel and machinery.Complicated, especially co-ordinating and
synchronising multiple testers Involves a high degree of organization,
especially to record and analyse results meaningfully
Repeatability of the manual tests is limited
Why should I use Tool ? Benefit of Tool
Load Generation System Under Test
Web server Database server
Vuser host
• Controller manages the virtual users• Analyze results with graphs and
reports
• Replaces testers with virtual users
Solves the resource limitations
• Runs many Vusers on a few machinesAnalysis
Controller
Why should I use Tool ?Using Automated Tools Reduces personnel requirements by replacing human
users with virtual users or Vusers. These Vusers emulate the behaviour of real users
Because numerous Vusers can run on a single computer, the tool reduces the amount of hardware required for testing.
Monitors the application performance online, enabling you to fine-tune your system during test execution.
It automatically records the performance of the application during a test. You can choose from a wide variety of graphs and reports to view the performance data.
Because the tests are fully automated, you can easily repeat them as often as you need.
What should be my tools combination?
What tools are available and what should I use for Performance Testing?
ToolVendor Test Suite - Companion Tools
QALoadCompuware QACenter Enterprise Edition+
LoadRunnerMercury Quality Center
e-LoadEmpirix e-TEST suite
Rational Performance Tester
IBM Rational Suite
WebLOADRadView TestView Suite
SilkPerformerBorland(Segue)
SilkCentral Test Manager
Apache JMeter™Apache ?
Rational Performance Tester
IBM Rational Suite
What are the Performance Metrics? Client-side V/s Server-side Performance
Metrics Server-side Metrics – Application level and
OS level Client side – Response Time, Pass Fail
Statistics, Throughput, Hits/sec, Customized Functionality Validation Counters, User Abandonment
Server-side – OS – CPU, Memory, Network and Disk I/O
Server-side – Performance related metrics and observations for Web Server (s), Application Server (s) and/or Database Server (s)
What should be my best approach? How do we find and fix the problems? The
framework? PE FRAMEWORK:
Performance engineering is the process by which software is reviewed, tested and tuned with the intent of realizing the required performance. This process aims to optimize the most important application performance trait, user experience.
What should be my best approach?
Let’s Re-cap What is Performance Testing? What are the Terminologies I should know? Why Performance Testing? What are the Performance Bottlenecks? Why do we have bottlenecks ? What are the contributors? What are the types of Performance Testing? When is it required? What should be tested? What should be my Performance Testing Process? Why should I use Tool ? What should be my tools combination? What tools are available and what should I use for Performance
Testing? What are the Performance Metrics? What should be my best approach?
THANK YOU
Rohan Bhattarai