cloud-based performance testing
Post on 14-Nov-2014
541 Views
Preview:
DESCRIPTION
TRANSCRIPT
DISC IPL INED INNOVATION
TEST PERFORMANCE USING CLOUD
Abhinav Maheshwari (Chief Scientist & Director, 3Pillar Labs)
Sayantam Dey (Architect, 3Pillar Labs)
OBJECTIVES
1. What are the benefits of using the cloud for testing performance over traditional means?
2. What are the various tools available for cloud based performance testing?
3. How can we use open source load testing frameworks in the cloud?
4. How to design a custom framework for rapid testing with large number of concurrent users?
TRADITIONAL PERFORMANCE TESTING
Load GenerationServer (High end
or multiple servers)
Application Server
Workstation
1. Deploy replica of the production environment in the LAN
2. Deploy a high-end server for generating load (on the same LAN)
3. Run agents of a load test tool like JMeter, Grinder or WAPT
4. Connect remotely to load generation server
5. Load generation server collects metrics and generates reports
CLOUD BASED PERFORMANCE TESTING
Load GenerationServer (High end
or multiple servers)
Application Server
Workstation
Cloud machinesin different geographies
Application Server
Workstation
BENEFITS OF USING THE CLOUD
Traditional Approach Cloud based Approach
Provisioning physical machines is costly, specially for medium sized companies
Large number of small, virtual machines running in bursts, ideal usage of cloud from a cost perspective
Manual provisioning of load generation server requires time and effort
Simpler provisioning, can even be automated using API
Application needs to be deployed in the same LAN
Application may be hosted at any public IP, also more realistic from latency point of view
Not dynamic to address any significant increase in the performance requirement
Distribute load generation over multiple instances which can be increased dynamically
Extremely difficult to simulate geographically distributed load
Cloud clusters in different geographic locations can simulate geographically distributed load
CLOUD APPROACH – CHOICES ?
Commercial Products
1 Record or upload scripts for usage scenarios using the web based interface
2 Configure testing parameters like number of virtual users, time for test, etc. through interface
3 Start the test through web based interface
4 Download reports and analyze
Open source frameworks
1 Create the required number of VM instances on the cloud, deploy the chosen load testing framework
2 Configure all the parameters for the framework and upload scripts written for the chosen framework
3 Run agents on the system under test and then start the test
4 Download reports and analyze
CLOUD APPROACH – CHOICES ?
Commercial Products
1 Record or upload scripts for usage scenarios using the web based interface provided by the vendor
2 Configure testing parameters like number of virtual users, time for test, % allocation to usage scenarios etc. through interface
3 Start the test through web based interface
4 Download reports and analyze
Open source frameworks
• Very popular, large user base• Very little programming
knowledge required• Handles many protocols• Plugin architecture for
extension
• Requires programming• Scripting in Jython or Clojure• Distributed framework
CLOUD APPROACH – CHOICES ?
Commercial Products
• JMeter compatible• Real time reports• 10K virtual users x 25 hours
@ $1000
Web Performance
• Selenium IDE scripts • Real browser or simple
request playback (virtual user)
• 10K virtual users x 25 hours @ $3750
• Custom scripts• Real time analytics• Custom pricing
Open source frameworks
Only few frameworks are mature enough to work over the cloud
Only the cost of cloud infrastructure, which is much less expensive
Completely flexible as scripting is done through open APIs
Can only be used through command line, user has to care about calculating VM requirement, creation and closure
Reports are basic and sometimes not helpful
A CUSTOM FRAMEWORK?
Commercial Products
Not many choices since there are only a few products in the market
May prove to be expensive for repeated load testing
May use recording or proprietary scripting, which may not be fully flexible
Provide web-based interface to directly run tests without worrying about virtual machines
Provide easy to use reports
X
Sizable investment - Suitable for organizations needing repeated performance tests
CASE STUDY - HAILSTORMConfiguration JMeter Amazon SUT
Provisioning Manager
JMeterScripts
Script Manager
JMeter Controller
Data Collector
Report Engine- JFreeChart
AMI- JMeter- JRE
Agent
System Under Test (SUT)
Agent
Agent
HTTP
11
CASE STUDY - HAILSTORM
Observations
Sharp rise in response time for over 50 users
“Detail Page” degrades significantly for 75 users and above
System is stable but peaks out at 75 concurrent users and starts degrading at 100 users
Maximum throughput of 3.2 transactions per second at 75 users is low
© Copyright 2012 3PILLAR GLOBAL All rights reserved
Sample Report Summary
12
CASE STUDY - HAILSTORM
© Copyright 2012 3PILLAR GLOBAL All rights reserved
Page Response time Avg. (ms)
Response time 90tile (ms)
% errors
Home Page 1384 2293 0.0
Login Page 559 764 0.0
Login 1050 1740 0.0
Browse 1996 2898 0.25
Category Browsing
2553 3601 0.0
Detail Page 5368 7231 0.27
Search 2054 2948 0.0
Sample Report for one iteration
CASE STUDY - HAILSTORM
• Application– A leading provider of online classroom and instruction
solutions
• Key Challenge– Required simulation of 40,000 concurrent users– Rapid and cost effective solution to capture load
metrics for large number of concurrent users
• The Solution– Different JMeter scripts/plans seamlessly integrated– Engaging reports pointing out bottlenecks generated
top related