performance testing with 100,000 concurrent users in aws
DESCRIPTION
M-Square build an easy scalable performance test solution on AWS, using open source tools & CI servers, to allow cost-effective testing at scale. The solution is suitable for any organisation type, from startup to enterprise. The talk covers VPC, EC2, S3, ELB’s, AWS API scripting, automation and interesting performance issues when running massive workloads on AWS.TRANSCRIPT
Performance Testing@
With 100,000 concurrent users
WHO ARE WE
• AWS for most client projects and our own• Enterprise/Solution Architecture +
Development• Performance Engineering, Profiling,
Optimization• "DevOps"• AWS for most client projects and our own
WHAT WE WANTED
Performance test applications with 100,000 concurrent users
(Websocket + HTTP)
• Easy to use• Traceable history of tests• Good fit with modern DevOps
processes• Low maintenance and cost
99 Problems
Quickly max out a single test node(or even 10 of them)
Performance tests should be available to whole team(not just the guru)
Shouldn't have to write another App(or pay A LOT for one)
REQUIREMNTS
SOLUTION
(one of many)
STEP 1
Build App(Compile, Test, Package)
STEP 2
Build your environment (Puppet, Chef, etc)Deploy new application version
STEP 3
Create load test slave instances
STEP 4
Start running the test plan against new environment
STEP 5
• Test slaves return results to Jenkins.
• Then clean up• Shut down instances• Snapshot new AMI
PIPELINE
Build app -> Create test env + Deploy -> Test new environment
LESSONS LEARNED 1/2
• Performance problems found and fixed Garbage Collection, Serialization and encoding
• Very helpful when sizing infrastructure Webservice tier network was bottleneck Different instance sizes picked
• Scaling the test clients can be hard
LESSONS LEARNED 2/2
• Performance testing via ELB• Ramp up slowly• Make sure client see DNS
changes• ELB's not great for Websocket
• Linux tuning• Ephemeral port range• Buffers everywhere• File handles, flood protection, etc• It does a pretty good job by
default
Results & Deep Insight
Deep Insight with dynaTrace APM beyond simple JMeter results
Clearly understand the difference between:• Load generation• Real User Monitoring • Always-on deep transaction tracing
TEST AUTOMATION / CI
• Unit Tests are fully integrated as part of CI• Each build also measures performance for each testcase
BONUSES
Easily re-used for different types of build/testing
• BYO load test tool• Spot instances are cheaper• Not just for load testing• Test Chef/Puppet scripts• OS updates
• Browser based (Webdriver, PhantomJS, etc)
• Auto-scale testing• Snapshot AMI after successful tests
Questions ?
Thanks !
Twitter: msquareau
http://m-square.com.au