tools of the trade: load testing - ignite session at webperfdays ny 14

20

Click here to load reader

Upload: alexander-podelko

Post on 26-May-2015

422 views

Category:

Technology


3 download

DESCRIPTION

Tools of the Trade: Load Testing - an Ignite session at WebPerfDays NY 2014. Some consideration about load testing and selecting load testing tools - as much as could be squeezed into 5 min / 20 slides.

TRANSCRIPT

Page 1: Tools of the Trade: Load Testing -  Ignite session at WebPerfDays NY 14

Tools of the Trade: Load Testing

Alexander Podelko [email protected]

alexanderpodelko.com/blog@apodelko

WebPerfDays NY 2014

September 18, 2014

Page 2: Tools of the Trade: Load Testing -  Ignite session at WebPerfDays NY 14

2

About Me• Specialized in performance for the last 17 years

• Currently performance testing and optimization of Hyperion products at Oracle

• Board director at CMG http://cmg.org – Organization of performance and capacity

professionals

– Conference November 3-6, 2014 in Atlanta, GADisclaimer: The views expressed here are my personal views only and do not necessarily represent those of my current or previous employers. All brands and trademarks mentioned are the property of their owners.

Page 3: Tools of the Trade: Load Testing -  Ignite session at WebPerfDays NY 14

3

Front- vs. Back-End• Back-end response times depend on load

Thanks to Andy Hawkes for the graph

Page 4: Tools of the Trade: Load Testing -  Ignite session at WebPerfDays NY 14

4

Load Testing

Applying multi-user synthetic load to the system

– Load testing– Performance testing– Stress testing– Scalability testing– Volume testing– Reliability testing

– Concurrency testing– Realistic Testing– Endurance testing– Longevity testing– Soak testing– Stability testing

Page 5: Tools of the Trade: Load Testing -  Ignite session at WebPerfDays NY 14

5

Can System Handle Peak Load?• You can’t know without testing:

Page 6: Tools of the Trade: Load Testing -  Ignite session at WebPerfDays NY 14

6

Verify Multi-User Performance

• Single-user improvement may lead to multi-user performance degradation

Page 7: Tools of the Trade: Load Testing -  Ignite session at WebPerfDays NY 14

7

What Else Load Testing Adds

• Performance optimization– Apply exactly the same load– See if the change makes a difference

• Debugging/verification of multi-user issues

• Testing self-regulation functionality– Such as auto-scaling or changing the level of

service depending on load

Page 8: Tools of the Trade: Load Testing -  Ignite session at WebPerfDays NY 14

8

Commercial Tools

Gartner Magic Quadrant for Quality Suites:• Leaders:

– HP (LoadRunner)– IBM (Rational Performance Testing)– Microsoft (Visual Studio Web Performance…)– Oracle (Application Testing Suite)– SOASTA(CloudTest)– Borland (SilkPerformer)

• Peformance: Neotys (NeoLoad), Blazemeter

Page 9: Tools of the Trade: Load Testing -  Ignite session at WebPerfDays NY 14

9

Open Source Tools

• 53 tools listed at

http://www.opensourcetesting.org/performance.php• Jmeter

– Clear leader in popularity• Gatling

– Getting popularity recently• Grinder, OpenSTA

– Were popular/looks like no development recently

Page 10: Tools of the Trade: Load Testing -  Ignite session at WebPerfDays NY 14

10

Mixed Business Models

• Advanced functionality– With small tests for free

• Freemium models – LoadRunner, NeoLoad (up 50 users free)– CloudTest (Lite up to 100 users free)

• BlazeMeter service using open source JMeter• Important when more people get involved

– Continuous Integration, Agile development

Page 11: Tools of the Trade: Load Testing -  Ignite session at WebPerfDays NY 14

11

Load Testing ToolsDiffer drastically in:

– Supported approaches / protocols– Functionality / extendibility – Supported environments– Scalability– Result analysis– Integration (monitoring, APM, CI, etc.)– Cost/Licensing– Available support and skills

Page 12: Tools of the Trade: Load Testing -  Ignite session at WebPerfDays NY 14

12

Record and Playback: Protocol Level

Load Testing Tool

Virtual Users

ServerLoad Generator

Application

Network

Page 13: Tools of the Trade: Load Testing -  Ignite session at WebPerfDays NY 14

13

Record and Playback: UI Level

Load Testing Tool

Virtual Users

ServerLoad Generator

ApplicationNetwork

Browsers

Page 14: Tools of the Trade: Load Testing -  Ignite session at WebPerfDays NY 14

14

Programming

Load Testing Tool App.

Virtual Users

ServerLoad Generator

Application

NetworkAPI

Page 15: Tools of the Trade: Load Testing -  Ignite session at WebPerfDays NY 14

15

Environments

• Lab vs. Service (SaaS) vs. Cloud (IaaS)– For both the system and load generators

• Test vs. Production

• No best solution, depends on your goals / system

Page 16: Tools of the Trade: Load Testing -  Ignite session at WebPerfDays NY 14

16

Scenarios

• System validation for high load– Outside load (service or cloud), production system– Wider scope, lower repeatability

• Performance optimization / troubleshooting– Isolated lab environment– Limited scope, high repeatability

• Testing in Cloud– Lowering costs (in case of periodic tests)– Limited scope, low repeatability

Page 17: Tools of the Trade: Load Testing -  Ignite session at WebPerfDays NY 14

17

Integration

• Reporting and Analysis• Monitoring / APM

– System– Application insights (APM)

• AppDynamics, New Relic, Dynatrace, etc.

• Continuous Integration– CI servers (Jenkins, Hudson, etc.)– Adding to automatic build process

Page 18: Tools of the Trade: Load Testing -  Ignite session at WebPerfDays NY 14

18

Tool’s Ecosphere

Name Number of found documents by Google

Number of found US positions at Monster.com

[HP] LoadRunner 894,000 170

[Apache] JMeter 688,000 90

[Borland] SilkPerformer 138,000 12

[Neotys] NeoLoad 87,100 3

[SOASTA] CloudTest 51,500 2

LoadStorm 18,000 -

Page 19: Tools of the Trade: Load Testing -  Ignite session at WebPerfDays NY 14

19

Summary

• There is no best tool – it depends on your needs

• Almost every tool works for a small and simple web site

• If using more sophisticated technologies, always check if the tool / approach supports it

Page 20: Tools of the Trade: Load Testing -  Ignite session at WebPerfDays NY 14

Questions?Alexander [email protected]

alexanderpodelko.com/blog@apodelko

Need More? Come to the Performance and Capacity conference http://www.cmg.org

• November 3-6, 2014 in Atlanta, GA

• Perfect complement to Velocity / WebPerfDays